konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
CI fails recurrently and randomly on Windows

As reported by @lhoestq,

The windows CI is currently flaky: some dependencies like `aiobotocore`, `multiprocess` and `seqeval` sometimes fail to install.
In particular it seems that building the wheels fail. Here is an example of logs:

```
Building wheel for seqeval (setup.py): started
  Running command 'C:\tools\miniconda3\envs\py37\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-h55pfgbv\\seqeval_d6cdb9d23ff6490b98b6c4bcaecb516e\\setup.py'"'"'; __file__='"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-h55pfgbv\\seqeval_d6cdb9d23ff6490b98b6c4bcaecb516e\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\circleci\AppData\Local\Temp\pip-wheel-x3cc8ym6'
  No parent package detected, impossible to derive `name`
  running bdist_wheel
  running build
  running build_py
  package init file 'seqeval\__init__.py' not found (or not a regular file)
  package init file 'seqeval\metrics\__init__.py' not found (or not a regular file)
  C:\tools\miniconda3\envs\py37\lib\site-packages\setuptools\command\install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
    setuptools.SetuptoolsDeprecationWarning,
  installing to build\bdist.win-amd64\wheel
  running install
  running install_lib
  warning: install_lib: 'build\lib' does not exist -- no Python modules to install

  running install_egg_info
  running egg_info
  creating UNKNOWN.egg-info
  writing UNKNOWN.egg-info\PKG-INFO
  writing dependency_links to UNKNOWN.egg-info\dependency_links.txt
  writing top-level names to UNKNOWN.egg-info\top_level.txt
  writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  reading manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  Copying UNKNOWN.egg-info to build\bdist.win-amd64\wheel\.\UNKNOWN-0.0.0-py3.7.egg-info
  running install_scripts
  creating build\bdist.win-amd64\wheel\UNKNOWN-0.0.0.dist-info\WHEEL
  creating 'C:\Users\circleci\AppData\Local\Temp\pip-wheel-x3cc8ym6\UNKNOWN-0.0.0-py3-none-any.whl' and adding 'build\bdist.win-amd64\wheel' to it
  adding 'UNKNOWN-0.0.0.dist-info/METADATA'
  adding 'UNKNOWN-0.0.0.dist-info/WHEEL'
  adding 'UNKNOWN-0.0.0.dist-info/top_level.txt'
  adding 'UNKNOWN-0.0.0.dist-info/RECORD'
  removing build\bdist.win-amd64\wheel
  Building wheel for seqeval (setup.py): finished with status 'done'
  Created wheel for seqeval: filename=UNKNOWN-0.0.0-py3-none-any.whl size=963 sha256=67eb93a6e1ff4796c5882a13f9fa25bb0d3d103796e2525f9cecf3b2ef26d4b1
  Stored in directory: c:\users\circleci\appdata\local\pip\cache\wheels\05\96\ee\7cac4e74f3b19e3158dce26a20a1c86b3533c43ec72a549fd7
  WARNING: Built wheel for seqeval is invalid: Wheel has unexpected file name: expected 'seqeval', got 'UNKNOWN'
```
Smooth-BLEU bug fixed

Hi,

the current implementation of smooth-BLEU contains a bug: it smoothes unigrams as well. Consequently, when both the reference and translation consist of totally different tokens, it anyway returns a non-zero value (please see the attached image). 

This however contradicts the source paper suggesting the smooth-BLEU _(Chin-Yew Lin, Franz Josef Och. ORANGE: a method for evaluating automatic evaluation metrics for machine translation. COLING 2004.)_ :

> Add one count to the n-gram hit and total ngram count for n > 1. Therefore, for candidate translations with less than n words, they can still get a positive smoothed BLEU score from shorter n-gram matches; however if nothing matches then they will get zero scores. 

This pull request aims at fixing this bug.

I made a pull request in the target repository `tensorflow/nmt`, which implements this script, yet the last commit there is dating 19.02.2019 and I doubt whether this will be fixed promptly. Yet, this bug is critical, for instance for summarization datasets with short summaries (e.g. AESLC), since smoothing needs to be applied there. Therefore, the easiest solution that I found is to fork the repo and download this script directly from the forked fixed repo.

Kind,
Akim Tsvigun

<img width="516" alt="Снимок экрана 2022-06-29 в 17 49 27" src="https://user-images.githubusercontent.com/36672861/176466935-ac579e6d-6a93-4111-ab41-9b33056e7d47.png">
 

Thanks @Aktsvigun for your fix.

However, metrics in `datasets` are in deprecation mode:
- #4739

You should transfer this PR to the `evaluate` library: https://github.com/huggingface/evaluate

Just for context, here the link to the PR by @Aktsvigun on tensorflow/nmt:
- https://github.com/tensorflow/nmt/pull/488
Streaming issue for financial_phrasebank

### Link

https://huggingface.co/datasets/financial_phrasebank/viewer/sentences_allagree/train

### Description

As reported by a community member using [AutoTrain Evaluate](https://huggingface.co/spaces/autoevaluate/model-evaluator/discussions/5#62bc217436d0e5d316a768f0), there seems to be a problem streaming this dataset:

```
Server error
Status code:   400
Exception:     Exception
Message:       Give up after 5 attempts with ConnectionError
```

### Owner

No

cc @huggingface/datasets: it seems like https://www.researchgate.net/ is flaky for datasets hosting (I put the "hosted-on-google-drive" tag since it's the same kind of issue I think)

Let's see if their license allows hosting their data on the Hub.

License is Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0).

We can host their data on the Hub.
Dataset Viewer issue for universal_dependencies

### Link

https://huggingface.co/datasets/universal_dependencies

### Description

invalid json response body at https://datasets-server.huggingface.co/splits?dataset=universal_dependencies reason: Unexpected token I in JSON at position 0

### Owner

_No response_

Thanks, looking at it!

Finally fixed! We updated the dataset viewer and it fixed the issue.

https://huggingface.co/datasets/universal_dependencies/viewer/aqz_tudet/train

<img width="1561" alt="Capture d’écran 2022-09-07 à 13 29 18" src="https://user-images.githubusercontent.com/1676121/188867795-4f7dd438-d4f2-46cd-8a92-20a37fb2d6bc.png">
Dataset Viewer issue with False positive PII redaction

### Link

https://huggingface.co/datasets/cakiki/rosetta-code

### Description

Hello, I just noticed an entry being redacted that shouldn't have been:

`RootMeanSquare@Range[10]` is being displayed as `[email protected][10]`

### Owner

_No response_

The value is in the data, it's not an issue with the "dataset-viewer".

<img width="1161" alt="Capture d’écran 2022-06-29 à 10 25 51" src="https://user-images.githubusercontent.com/1676121/176389325-4d2a9a7f-1583-45b8-aa7a-960ffaa6a36a.png">

 Maybe open a PR: https://huggingface.co/datasets/cakiki/rosetta-code/discussions


This was indeed a scraping issue which I assumed was a display issue; sorry about that!
Generalize meta_path json file creation in load.py [#4540]

# What does this PR do?

## Summary

*In function `_copy_script_and_other_resources_in_importable_dir`, using string split when generating `meta_path` throws error in the edge case raised in #4540.*

## Additions
-

## Changes
- Changed meta_path to use `os.path.splitext` instead of using `str.split` to generalize code.

## Deletions
-
## Issues Addressed : 

Fixes #4540

_The documentation is not available anymore as the PR was closed or merged._

@albertvillanova,  Can you please review this PR for Issue #4540 

@lhoestq  Thank you for merging the PR . Is there a slack channel for contributing to the datasets library. I would love to work on the library and make meaningful contributions.

Hi ! Sure feel free to join our discord ^^ 
https://discuss.huggingface.co/t/join-the-hugging-face-discord/11263 so that we can discuss together mor eeasily. Otherwise everything happens on github ;)
Host head_qa data on the Hub and fix NonMatchingChecksumError

This PR:
- Hosts head_qa data on the Hub instead of Google Drive
- Fixes NonMatchingChecksumError

Fix https://huggingface.co/datasets/head_qa/discussions/1

_The documentation is not available anymore as the PR was closed or merged._

Hi @albertvillanova ! Thanks for the fix ;)
Can I safely checkout from this branch to build `datasets` or it is preferable to wait until all CI tests pass?
Thanks 🙏 

@younesbelkada we have just merged this PR.
Add binary classification task IDs

As a precursor to aligning the task IDs in `datasets` and AutoTrain, we need a way to distinguish binary vs multiclass vs multilabel classification.

This PR adds binary classification to the task IDs to enable this.

Related AutoTrain issue: https://github.com/huggingface/autonlp-backend/issues/597

cc @abhishekkrthakur @SBrandeis 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4584). All of your documentation changes will be reflected on that endpoint.

> Awesome thanks ! Can you add it to https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts first please ? This is where we define the cross libraries tasks taxonomy ;)

Thanks for the tip! Done in https://github.com/huggingface/hub-docs/pull/217

I don't think we need to update this file anymore. We should remove it IMO, and simply update the dataset [tagging app](https://huggingface.co/spaces/huggingface/datasets-tagging)

I'm closing this PR.
add_column should preserve _indexes

https://github.com/huggingface/datasets/issues/3769#issuecomment-1167146126

doing `.add_column("x",x_data)` also removed any `_indexes` on the dataset, decided this shouldn't be the case.

This was because `add_column` was creating a new `Dataset(...)` and wasn't possible to pass indexes on init.
with this PR now can pass 'indexes' on init through `IndexableMixin`

- [x] Added test

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4582). All of your documentation changes will be reflected on that endpoint.
Dataset Viewer issue for pn_summary

### Link

https://huggingface.co/datasets/pn_summary/viewer/1.0.0/validation

### Description

Getting an index error on the `validation` and `test` splits:

```
Server error
Status code:   400
Exception:     IndexError
Message:       list index out of range
```

### Owner

No

linked to https://github.com/huggingface/datasets/issues/4580#issuecomment-1168373066?

Note that I refreshed twice this dataset, and I still have (another) error on one of the splits

```
Status code:   400
Exception:     ClientResponseError
Message:       403, message='Forbidden', url=URL('https://doc-14-4c-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/pgotjmcuh77q0lk7p44rparfrhv459kp/1656403650000/11771870722949762109/*/16OgJ_OrfzUF_i3ftLjFn9kpcyoi7UJeO?e=download')
```

Like the three splits are processed in parallel by the workers, I imagine that the Google hosting is rate-limiting us.

cc @albertvillanova 



Exactly, Google Drive bans our loading scripts.

When possible, we should host somewhere else.
Dataset Viewer issue for multi_news

### Link

https://huggingface.co/datasets/multi_news

### Description

Not sure what the index error is referring to here:

```
Status code:   400
Exception:     IndexError
Message:       list index out of range
```

### Owner

No

Thanks for reporting, @lewtun.

I forced the refreshing of the preview and it worked OK for train and validation splits.

I guess the error has to do with the data files being hosted at Google Drive: this gives errors when requested automatically using scripts.
We should host them to fix the error. Let's see if the license allows that.

I guess we can host the data: https://github.com/Alex-Fabbri/Multi-News/blob/master/LICENSE.txt
Support streaming cfq dataset

Support streaming cfq dataset.

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq I've been refactoring a little the code:
- Use less RAM by loading only the required samples: only if its index is in the splits file
- Start yielding "earlier" in streaming mode: for each `split_idx`:
  - either yield from buffer
  - or iterate over samples and either yield or buffer the sample
  
  The speed gain obviously depends on how the indexes are sorted in the split file:
    - Best case: indices are [1, 2, 3]
    - Worst case (no speed gain): indices are [3, 1, 2] or [3, 2, 1]

Let me know what you think.

I have to update the dummy data so that it aligns with the real data (inside the archive, the samples file `dataset.json` is the last member).

There is an issue when testing `test_load_dataset_cfq` with dummy data:
- `MockDownloadManager.iter_archive` yields FIRST `'cfq/dataset.json'`
- [`Streaming`]`DownloadManager.iter_archive` yields LAST `'cfq/dataset.json'` when using real data tar.gz archive

Note that this issue arises only with dummy data: loading the real dataset works smoothly for all configurations: I recreated the `dataset_infos.json` file to check it (it generated the same file).

This PR should be merged first:
- #4611

Impressive, thank you ! :o 

feel free to merge master into this branch, now that the files order is respected. You can merge if the CI is green :)
[Multi Configs] Use directories to differentiate between subsets/configurations

Currently to define several subsets/configurations of your dataset, you need to use a dataset script.

However it would be nice to have a no-code way to to this. 

For example we could specify different configurations of a dataset (for example, if a dataset contains different languages) with one directory per configuration.

These structures are not supported right now, but would be nice to have:


```
my_dataset_repository/
├── README.md
├── en/
│   ├── train.csv
│   └── test.csv
└── fr/
    ├── train.csv
    └── test.csv
```

Or with one directory per split:

```
my_dataset_repository/
├── README.md
├── en/
│   ├── train/
│   │   ├── shard_0.csv
│   │   └── shard_1.csv
│   └── test/
│       ├── shard_0.csv
│       └── shard_1.csv
└── fr/
    ├── train/
    │   ├── shard_0.csv
    │   └── shard_1.csv
    └── test/
        ├── shard_0.csv
        └── shard_1.csv
```

cc @stevhliu @albertvillanova 

This can be specified in the README as YAML with

```
configs:
- config_name: en
  data_dir: en
- config_name: fr
  data_dir: fr
```

I want to be able to create folders in a model.

How to set new split names, instead of train/test/validation? For example, I have a local dataset, consists of several subsets, named "A", "B", and "C". How can I create a huggingface dataset, with splits A/B/C ?

The document in https://huggingface.co/docs/datasets/dataset_script only tells me how to create datasets with subsets that is hosted on another server. How to do it if my datasets are local?

> The document in https://huggingface.co/docs/datasets/dataset_script only tells me how to create datasets with subsets that is hosted on another server. How to do it if my datasets are local?

It works the same - you just need to use local paths instead of URLs
Include `metadata.jsonl` in resolved data files

Include `metadata.jsonl` in resolved data files.

Fix #4548 

@lhoestq ~~https://github.com/huggingface/datasets/commit/d94336d30eef17fc9abc67f67fa1c139661f4e75 adds support for metadata files placed at the root, and https://github.com/huggingface/datasets/commit/4d20618ea7a19bc143ddc5fdff9d79e671fcbb95 accounts for nested metadata files also, but this results in more complex code. Let me know which one of these two approaches you prefer.~~ Maybe https://github.com/huggingface/datasets/commit/d94336d30eef17fc9abc67f67fa1c139661f4e75 is good enough for now (for the sake of simplicity). https://github.com/huggingface/datasets/commit/4d20618ea7a19bc143ddc5fdff9d79e671fcbb95 breaks the imagefolder tests due to duplicates in the resolved metadata files. One way to fix this would be to resolve the metadata pattern only on parent directories, but this adds even more logic to `_get_data_files_patterns`, so not sure if this is what we should do.

_The documentation is not available anymore as the PR was closed or merged._

I still don't know if the way we implemented data files resolution could support the metadata.jsonl file without bad side effects for the other packaged builders. In particular here if you have a folder of csv/parquet/whatever files and a metadata.jsonl file, it would return 
```
split: patterns_dict[split] + [METADATA_PATTERN]
```
which is a bit unexpected and can lead to errors.

Maybe this logic can be specific to imagefolder somehow ? This could be an additional pattern `["metadata.jsonl", "**/metadata.jsonl"]` just for imagefolder, that is only used when `data_files=` is not specified by the user.

I guess it's ok to have patterns that lead to duplicate metadata.jsonl files for imagefolder, since the imagefolder logic only considers the closest metadata file for each image.

What do you think ?

Yes, that's indeed the problem. My solution in https://github.com/huggingface/datasets/commit/4d20618ea7a19bc143ddc5fdff9d79e671fcbb95 that accounts for that (include metadata files only if image files are present; not ideal): https://github.com/huggingface/datasets/blob/4d20618ea7a19bc143ddc5fdff9d79e671fcbb95/src/datasets/data_files.py#L119-L125.
Perhaps a cleaner approach would be to check for metadata files after the packaged module type is inferred as `imagefolder` and append metadata files to already resolved data files (if there are any). WDYT?

@lhoestq 

> Perhaps a cleaner approach would be to check for metadata files after the packaged module type is inferred as imagefolder and append metadata files to already resolved data files (if there are any). WDYT?

I decided to go with this approach.

 Not sure if you meant the same thing with this comment:

> Maybe this logic can be specific to imagefolder somehow ? This could be an additional pattern ["metadata.jsonl", "**/metadata.jsonl"] just for imagefolder, that is only used when data_files= is not specified by the user.


It adds more code but is easy to follow IMO.


The CI still struggles but you can merge since at least one of the two WIN CI succeeded
Dataset Viewer issue for mlsum

### Link

https://huggingface.co/datasets/mlsum/viewer/de/train

### Description

There's seems to be a problem with the download / streaming of this dataset:

```
Server error
Status code:   400
Exception:     BadZipFile
Message:       File is not a zip file
```

### Owner

No

Thanks for reporting, @lewtun.

After investigation, it seems that the server https://gitlab.lip6.fr does not allow HTTP Range requests.

We are trying to find a workaround...
move under the facebook org?

### Link

https://huggingface.co/datasets/gsarti/flores_101

### Description

It seems like streaming isn't supported for this dataset:

```
Server Error
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol for TAR archives like 'https://dl.fbaipublicfiles.com/flores101/dataset/flores101_dataset.tar.gz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.
```

### Owner

No

Related to https://github.com/huggingface/datasets/issues/4562#issuecomment-1166911751

I'll assign @albertvillanova 

I'm just wondering why we don't have this dataset under:
- the `facebook` namespace
- or the canonical dataset `flores`: why does this only have 2 languages?

fwiw: the dataset viewer is working. Renaming the issue
Dataset sharding non-contiguous?

## Describe the bug
I'm not sure if this is a bug; more likely normal behavior but i wanted to double check.
Is it normal that `datasets.shard` does not produce chunks that, when concatenated produce the original ordering of the sharded dataset? 

This might be related to this pull request (https://github.com/huggingface/datasets/pull/4466) but I have to admit I did not properly look into the changes made.

## Steps to reproduce the bug
```python
max_shard_size = convert_file_size_to_int('300MB')
dataset_nbytes = dataset.data.nbytes
num_shards = int(dataset_nbytes / max_shard_size) + 1
num_shards = max(num_shards, 1)
print(f"{num_shards=}")
for shard_index in range(num_shards):
    shard = dataset.shard(num_shards=num_shards, index=shard_index)
    shard.to_parquet(f"tokenized/tokenized-{shard_index:03d}.parquet")
os.listdir('tokenized/')
```

## Expected results
I expected the shards to match the order of the data of the original dataset; i.e. `dataset[10]` being the same as `shard_1[10]` for example

## Actual results
Only the first element is the same; i.e. `dataset[0]` is the same as `shard_1[0]`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-4.15.0-176-generic-x86_64-with-glibc2.31
- Python version: 3.10.4
- PyArrow version: 8.0.0
- Pandas version: 1.4.2



This was silly; I was sure I'd looked for a `contiguous` argument, and was certain there wasn't one the first time I looked :smile:

Sorry about that.

Hi! You can pass `contiguous=True` to `.shard()` get contiguous shards. More info on this and the default behavior can be found in the [docs](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Dataset.shard).

EDIT: Answered as you closed the thread 😄 

Hahaha I'm sorry; my excuse is: it's Sunday. (Which makes me all the more grateful for your response :smiley: 

@mariosasko Sorry for reviving this, but I was curious as to why `contiguous=False` was the default. This might be a personal bias, but I feel that a user would expect the opposite to be the default. :thinking: 

This project started as a fork of TFDS, and `contiguous=False` is the default behavior [there](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shard).
Dataset Viewer issue for sst2

### Link

https://huggingface.co/datasets/sst2

### Description

Not sure what is causing this, however it seems that `load_dataset("sst2")` also hangs (even though it downloads the files without problem):

```
Status code:   400
Exception:     Exception
Message:       Give up after 5 attempts with ConnectionError
```

### Owner

No

Hi @lewtun, thanks for reporting.

I have checked locally and refreshed the preview and it seems working smooth now:
```python
In [8]: ds
Out[8]: 
DatasetDict({
    train: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 67349
    })
    validation: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 872
    })
    test: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 1821
    })
})
```

Could you confirm? 

Thanks @albertvillanova - it is indeed working now (not sure what caused the error in the first place). Closing this :)
Document link #load_dataset_enhancing_performance points to nowhere

## Describe the bug
A clear and concise description of what the bug is.

![image](https://user-images.githubusercontent.com/11674033/175752806-5b066b92-9d28-4771-9112-5c8606f07741.png)


The [load_dataset_enhancing_performance](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#load_dataset_enhancing_performance) link [here](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Dataset.load_from_disk.keep_in_memory) points to nowhere, I guess it should point to https://huggingface.co/docs/datasets/v2.3.2/en/cache#improve-performance?


Hi! This is indeed the link the docstring should point to. Are you interested in submitting a PR to fix this?

https://github.com/huggingface/datasets/blame/master/docs/source/cache.mdx#L93

There seems already an anchor here. Somehow it doesn't work. I am not very familiar with how this online documentation works.
Add UFSC OCPap dataset

## Adding a Dataset
- **Name:** UFSC OCPap: Papanicolaou Stained Oral Cytology Dataset (v4)
- **Description:** The UFSC OCPap dataset comprises 9,797 labeled images of 1200x1600 pixels acquired from 5 slides of cancer diagnosed and 3 healthy of oral brush samples, from distinct patients.
- **Paper:** https://dx.doi.org/10.2139/ssrn.4119212
- **Data:**  https://data.mendeley.com/datasets/dr7ydy9xbk/1
- **Motivation:** real data of pap stained oral cytology samples

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


I will add this directly on the hub (same as #4486)—in https://huggingface.co/lapix
Dataset Viewer issue for allocine

### Link

https://huggingface.co/datasets/allocine

### Description

Not sure if this is a problem with `bz2` compression, but I thought these datasets could be streamed:

```
Status code:   400
Exception:     AttributeError
Message:       'TarContainedFile' object has no attribute 'readable'
```

### Owner

No

I removed my assignment as @huggingface/datasets should be able to answer better than me


Let me have a look...

Thanks for the quick fix @albertvillanova 

Note that the underlying issue is that datasets containing TAR files are not streamable out of the box: they need being iterated with `dl_manager.iter_archive` to avoid performance issues because they access their file content *sequentially* (no random access).

> Note that the underlying issue is that datasets containing TAR files are not streamable out of the box: they need being iterated with `dl_manager.iter_archive` to avoid performance issues because they access their file content _sequentially_ (no random access).

Ah thanks for the clarification! I'll look out for this next time and implement the fix myself :)
Add evaluation metadata to wmt16

Just to confirm: we should add this metadata via GitHub and not Hub PRs for canonical datasets right?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4557). All of your documentation changes will be reflected on that endpoint.

> Just to confirm: we should add this metadata via GitHub and not Hub PRs for canonical datasets right?

yes :)

As discussed with @lewtun, we are closing this PR, because it requires first the task names to be aligned between AutoTrain and datasets.
Dataset Viewer issue for conll2003

### Link

https://huggingface.co/datasets/conll2003/viewer/conll2003/test

### Description

Seems like a cache problem with this config / split:

```
Server error
Status code:   400
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: '/cache/modules/datasets_modules/datasets/conll2003/__init__.py'
```

### Owner

No

Fixed, thanks.
Dataset Viewer issue for xtreme

### Link

https://huggingface.co/datasets/xtreme/viewer/PAN-X.de/test

### Description

There seems to be a problem with the cache of this config / split:

```
Server error
Status code:   400
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: '/cache/modules/datasets_modules/datasets/xtreme/349258adc25bb45e47de193222f95e68a44f7a7ab53c4283b3f007208a11bf7e/xtreme.py'
```

### Owner

No

Fixed, thanks.
Fix WMT dataset loading issue and docs update (Re-opened)

This PR is a fix for #4354 

Changes are made for `wmt14`, `wmt15`, `wmt16`, `wmt17`, `wmt18`, `wmt19` and `wmt_t2t`. And READMEs are updated for the corresponding datasets.

Let me know, if any additional changes are required.

Thanks

_The documentation is not available anymore as the PR was closed or merged._
Stop dropping columns in to_tf_dataset() before we load batches

`to_tf_dataset()` dropped unnecessary columns before loading batches from the dataset, but this is causing problems when using a transform, because the dropped columns might be needed to compute the transform. Since there's no real way to check which columns the transform might need, we skip dropping columns and instead drop keys from the batch after we load it.

cc @amyeroberts and https://github.com/huggingface/notebooks/pull/202

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq Rebasing fixed the test failures, so this should be ready to review now! There's still a failure on Win but it seems unrelated.

Gentle ping @lhoestq ! This is a simple fix (dropping columns after loading a batch from the dataset rather than with `.remove_columns()` to make sure we don't break transforms), and tests are green so we're ready for review!

@lhoestq Test is in!
Tell users to upload on the hub directly

As noted in https://github.com/huggingface/datasets/pull/4534, it is still not clear that it is recommended to add datasets on the Hugging Face Hub directly instead of GitHub, so I updated some docs.

Moreover since users won't be able to get reviews from us on the Hub, I added a paragraph to tell users that they can open a discussion and tag `datasets` maintainers for reviews.

Finally I removed the _previous good reasons_ to add a dataset on GitHub to only keep this one:

> In some rare cases it makes more sense to open a PR on GitHub. For example when you are not the author of the dataset and there is no clear organization / namespace that you can put the dataset under.

Does it sound good to you @albertvillanova @julien-c ?

_The documentation is not available anymore as the PR was closed or merged._

Thanks ! I updated the two remaining files
Perform hidden file check on relative data file path

Fix #4549 

_The documentation is not available anymore as the PR was closed or merged._

I'm aware of this behavior, which is tricky to solve due to fsspec's hidden file handling (see https://github.com/huggingface/datasets/issues/4115#issuecomment-1108819538). I've tested some regex patterns to address this, and they seem to work (will push them on Monday; btw they don't break any of fsspec's tests, so maybe we can contribute this as an enhancement to them). Also, perhaps we should include the files starting with `__`  in the results again (we hadn't had issues with this pattern before). WDYT?

I see. Feel free to merge this one if it's good for you btw :)

> Also, perhaps we should include the files starting with __ in the results again (we hadn't had issues with this pattern before)

The point was mainly to ignore `__pycache__` directories for example. Also also for consistency with the iter_files/iter_archive which are already ignoring them

Very elegant solution! Feel free to merge if the CI is green after adding the tests.

CI failure is unrelated to this PR
imdb source error

## Describe the bug
imdb dataset not loading

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("imdb")
```


## Expected results


## Actual results
```bash
06/23/2022 14:45:18 - INFO - datasets.builder - Dataset not on Hf google storage. Downloading and preparing it from source
06/23/2022 14:46:34 - INFO - datasets.utils.file_utils - HEAD request to http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz timed out, retrying... [1.0]
.....
ConnectionError: Couldn't reach http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz (ConnectTimeout(MaxRetryError("HTTPConnectionPool(host='ai.stanford.edu', port=80): Max retries exceeded with url: /~amaas/data/sentiment/aclImdb_v1.tar.gz (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f2d750cf690>, 'Connection to ai.stanford.edu timed out. (connect timeout=100)'))")))
```
## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Thanks for reporting, @Muhtasham.

Indeed IMDB dataset is not accessible from yesterday, because the data is hosted on the data owners servers at Stanford (http://ai.stanford.edu/) and these are down due to a power outage originated by a fire: https://twitter.com/StanfordAILab/status/1539472302399623170?s=20&t=1HU1hrtaXprtn14U61P55w

As a temporary workaroud, you can load the IMDB dataset with this tweak:
```python
ds = load_dataset("imdb", revision="tmp-fix-imdb")
```
FileNotFoundError when passing a data_file inside a directory starting with double underscores

Bug experienced in the `accelerate` CI: https://github.com/huggingface/accelerate/runs/7016055148?check_suite_focus=true

This is related to https://github.com/huggingface/datasets/pull/4505 and the changes from https://github.com/huggingface/datasets/pull/4412

I have consistently experienced this bug on GitHub actions when bumping to `2.3.2`

We're working on a fix ;)
Metadata.jsonl for Imagefolder is ignored if it's in a parent directory to the splits directories/do not have "{split}_" prefix

If data contains a single `metadata.jsonl` file for several splits, it won't be included in a dataset's `data_files` and therefore ignored. 
This happens when a directory is structured like as follows:
```
train/
  file_1.jpg
  file_2.jpg
test/
  file_3.jpg
  file_4.jpg
metadata.jsonl
```
or like as follows:
```
train_file_1.jpg
train_file_2.jpg
test_file_3.jpg
test_file_4.jpg
metadata.jsonl
```
The same for HF repos.

because it's ignored by the patterns [here](https://github.com/huggingface/datasets/blob/master/src/datasets/data_files.py#L29)

@lhoestq @mariosasko Do you think it's better to add this functionality in `data_files.py` or just specifically in imagefolder/audiofolder code? In `data_files.py` would me more general but I don't know if there are any other cases when that might be needed.
 

I agree it would be nice to support this. It doesn't fit really well in the current data_files.py, where files of each splits are separated in different folder though, maybe we have to modify a bit the logic here. 

One idea would be to extend `get_patterns_in_dataset_repository` and `get_patterns_locally` to additionally check for `metadata.json`, but feel free to comment if you have better ideas (I feel like we're reaching the limits of what the current implementation IMO, so we could think of a different way of resolving the data files if necessary)
[CI] fixing seqeval install in ci by pinning setuptools-scm

The latest setuptools-scm version supported on 3.6 is 6.4.2. However for some reason circleci has version 7, which doesn't work.

I fixed this by pinning the version of setuptools-scm in the circleci job

Fix https://github.com/huggingface/datasets/issues/4544

_The documentation is not available anymore as the PR was closed or merged._
Make DuplicateKeysError more user friendly [For Issue #2556]

# What does this PR do?

## Summary

*DuplicateKeysError error does not provide any information regarding the examples which have the same the key.*

*This information is very helpful for debugging the dataset generator script.*

## Additions
-

## Changes
- Changed `DuplicateKeysError Class` in `src/datasets/keyhash.py` to add current index and duplicate_key_indices to error message.
- Changed `check_duplicate_keys` function in `src/datasets/arrow_writer.py` to find indices of examples with duplicate hash if duplicate keys are found.

## Deletions
-

## To do  : 
- [x] Find way to find and print path `<Path to Dataset>` in Error message 

## Issues Addressed : 

Fixes #2556  

> Nice thanks !
> 
> After your changes feel free to mark this PR as "ready for review" ;)

Marking PR ready for review.

@lhoestq Let me know if there is anything else required or if we are good to go ahead and merge.

_The documentation is not available anymore as the PR was closed or merged._
Fix timestamp conversion from Pandas to Python datetime in streaming mode

Arrow accepts both pd.Timestamp and datetime.datetime objects to create timestamp arrays.
However a timestamp array is always converted to datetime.datetime objects.

This created an inconsistency between streaming in non-streaming. e.g. the `ett` dataset outputs datetime.datetime objects in non-streaming but pd.timestamp in streaming.

I fixed this by always converting pd.Timestamp to datetime.datetime during the example encoding step.
I fixed the same issue for pd.Timedelta as well. Finally I added an extra step of conversion for Series and DataFrame to take this into account in case such data are passed as Series or DataFrame.

Fix https://github.com/huggingface/datasets/issues/4533
Related to https://github.com/huggingface/datasets-server/issues/397

_The documentation is not available anymore as the PR was closed or merged._

CI failures are unrelated to this PR, merging
Avoid splitting by` .py` for the file.

https://github.com/huggingface/datasets/blob/90b3a98065556fc66380cafd780af9b1814b9426/src/datasets/load.py#L272


Hello, 
Thanks you for this library . 

 I was using it and I had one edge case. my home folder name ends with `.py` it is `/home/espoir.py` so anytime I am running the code to load a local module this code here it is failing because after splitting it is trying to save the code to my home directory.


Step to reproduce.

- If you have a home folder which ends with `.py`

- load a module with a local folder 
`qa_dataset = load_dataset("src/data/build_qa_dataset.py")`
it is failed 
A possible workaround would be to use pathlib at the mentioned line

` meta_path = Path(importable_local_file).parent.joinpath("metadata.json")` this can alivate the issue .

Let me what are your thought on this and I can try to fix it by A PR.


Hi @espoirMur, thanks for reporting.

You are right: that code line could be improved and made more generically valid.

On the other hand, I would suggest using `os.path.splitext` instead.

Are you willing to open a PR? :)

I will have a look.. 

This weekend .. 

@albertvillanova  , Can you have a look at #4590. 

Thanks 

#self-assign
Dataset Viewer issue for Pile of Law

### Link

https://huggingface.co/datasets/pile-of-law/pile-of-law

### Description

Hi, I would like to turn off the dataset viewer for our dataset without enabling access requests. To comply with upstream dataset creator requests/licenses, we would like to make sure that the data is not indexed by search engines and so would like to turn off dataset previews. But we do not want to collect user emails because it would violate single blind review, allowing us to deduce potential reviewers' identities. Is there a way that we can turn off the dataset viewer without collecting identity information?

Thanks so much! 

### Owner

Yes

Hi @Breakend, yes – we'll propose a solution today

Thanks so much, I appreciate it!

Thanks so much for adding the docs. I was able to successfully hide the viewer using the 
```
viewer: false
```
flag in the README.md of the dataset. I'm closing the issue because this is resolved. Thanks again!

Awesome! Thanks for confirming. cc @severo 

Just for the record:

- the doc
  
<img width="1430" alt="Capture d’écran 2022-06-27 à 09 29 27" src="https://user-images.githubusercontent.com/1676121/175884089-bca6c0d5-6387-473e-98ca-86a910ede4bd.png">

- the dataset main page

<img width="1134" alt="Capture d’écran 2022-06-27 à 09 29 05" src="https://user-images.githubusercontent.com/1676121/175884152-5f285bf0-3471-45de-927a-e141b00ebb33.png">

- the dataset viewer page

<img width="567" alt="Capture d’écran 2022-06-27 à 09 29 16" src="https://user-images.githubusercontent.com/1676121/175884191-ab6a297b-1c11-417e-bbde-0b7623278a79.png">
Fix WMT dataset loading issue and docs update

This PR is a fix for #4354 

Changes are made for `wmt14`, `wmt15`, `wmt16`, `wmt17`, `wmt18`, `wmt19` and `wmt_t2t`. And READMEs are updated for the corresponding datasets.

As I am on a M1 Mac, I am not able to create a virtual `dev` environment using `pip install -e ".[dev]"`. Issue is with `tensorflow-text` not supported on M1s and there is no supporting repo by Apple or Google. So, if I was needed to perform local testing, I am not able to do that.

Let me know, if any additional changes are required.

Thanks

The PR branch now has some commits unrelated to the changes, probably due to rebasing. Can you please close this PR and open a new one from a new branch? You can use  `git cherry-pick` to preserve the relevant changes:
```bash
git checkout master
git remote add upstream git@github.com:huggingface/datasets.git
git pull --ff-only upstream master
git checkout -b wmt-datasets-fix2
git cherry-pick f2d6c995d5153131168f64fc60fe33a7813739a4 a9fdead5f435aeb88c237600be28eb8d4fde4c55
```

Closing this PR due to unwanted commit changes. Will be opening new PR for the same issue.
Properly raise FileNotFound even if the dataset is private

`tests/test_load.py::test_load_streaming_private_dataset` was failing because the hub now returns 401 when getting the HfApi.dataset_info of a dataset without authentication. `load_dataset` was raising ConnectionError, while it should be FileNoteFoundError since it first checks for local files before checking the Hub.

Moreover when use_auth_token is not set (default is False), we should not pass `token=None` to HfApi.dataset_info, or it will use the local token by default - instead it should use no token. It's currently not possible to ask for no token to be used, so as a workaround I simply set token="no-token"

_The documentation is not available anymore as the PR was closed or merged._
Add `batch_size` parameter when calling `add_faiss_index` and `add_faiss_index_from_external_arrays`

Currently, even though the `batch_size` when adding vectors to the FAISS index can be tweaked in `FaissIndex.add_vectors()`, the function `ArrowDataset.add_faiss_index` doesn't have either the parameter `batch_size` to be propagated to the nested `FaissIndex.add_vectors` function or `*args, **kwargs`, so on, this PR adds the `batch_size` parameter to both `ArrowDataset.add_faiss_index` and `ArrowDataset.add_faiss_index_from_external_arrays`.

This is useful so as to tweak the `batch_size` according to the VM specifications.

Also, I had a doubt while checking the code related to the indices... 

@lhoestq, there's a value in `config.py` named `DATASET_INDICES_FILENAME` which has the arrow extension (which I assume it should be `indices.faiss`, as the Elastic Search indices are not stored in a file, but not sure), and it's just used before actually saving an `ArrowDataset` in disk, but since those indices are never stored AFAIK, is that actually required?

https://github.com/huggingface/datasets/blob/aec86ea4b790ccccc9b2e0376a496728b1c914cc/src/datasets/config.py#L183

https://github.com/huggingface/datasets/blob/aec86ea4b790ccccc9b2e0376a496728b1c914cc/src/datasets/arrow_dataset.py#L1079-L1092

So should I also remove that?

P.S. I also edited the following code comment which I found misleading as it's not actually storing the indices.

https://github.com/huggingface/datasets/blob/8ddc4bbeb1e2bd307b21f5d21f884649aa2bf640/src/datasets/arrow_dataset.py#L1122

_The documentation is not available anymore as the PR was closed or merged._

> @lhoestq, there's a value in config.py named DATASET_INDICES_FILENAME which has the arrow extension (which I assume it should be indices.faiss, as the Elastic Search indices are not stored in a file, but not sure), and it's just used before actually saving an ArrowDataset in disk, but since those indices are never stored AFAIK, is that actually required?

The arrow file is used to store an indices mapping (when you shuffle the dataset for example) - not for a faiss index ;)

Ok cool thanks a lot for the explanation @lhoestq I was not sure about that :+1: I'll also add it there as you suggested!

CI failures are unrelated to this PR and fixed on master, merging
Add `tldr_news` dataset

This PR aims at adding support for a news dataset: `tldr news`.

This dataset is based on the daily [tldr tech newsletter](https://tldr.tech/newsletter) and contains a `headline` as well as a `content` for every piece of news contained in a newsletter.

Hey @lhoestq, 
Sorry for opening a PR, I was following the guide [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md)! Thanks for the review anyway, I will follow the instructions you sent 😃 

Thanks, we will update the guide ;)
Timestamp not returned as datetime objects in streaming mode

As reported in (internal) https://github.com/huggingface/datasets-server/issues/397

```python
>>> from datasets import load_dataset
>>> dataset = load_dataset("ett", name="h2", split="test", streaming=True)
>>> d = next(iter(dataset))
>>> d['start']
Timestamp('2016-07-01 00:00:00')
```

while loading in non-streaming mode it returns `datetime.datetime(2016, 7, 1, 0, 0)`
Add Video feature

The following adds a `Video` feature for encoding/decoding videos on the fly from in memory bytes. It uses my own `encoded-video` library which is basically `pytorchvideo`'s encoded video but with all the `torch` specific stuff stripped out. Because of that, and because the tool I used under the hood is not very mature, I leave this as a draft idea that we can use to build off of.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4532). All of your documentation changes will be reflected on that endpoint.

@nateraw do you have any plans to continue this pr? Or should I write a custom loader script to use my video dataset in the hub?

@fcakyon I think we still want this feature in here, but my solution here isn't the right one, I'm afraid. Using my (very hacky) library is not the right move. Let's move to an issue to discuss the feature/workarounds for now.
Dataset Viewer issue for CSV datasets

### Link

https://huggingface.co/datasets/scikit-learn/breast-cancer-wisconsin

### Description

I'm populating CSV datasets [here](https://huggingface.co/scikit-learn) but the viewer is not enabled and it looks for a dataset loading script, the datasets aren't on queue as well. 

You can replicate the problem by simply uploading any CSV dataset.

### Owner

Yes

this should now be fixed

Confirmed, it's fixed now. Thanks for reporting, and thanks @coyotte508 for fixing it

<img width="1123" alt="Capture d’écran 2022-06-21 à 10 28 05" src="https://user-images.githubusercontent.com/1676121/174753833-1b453a5a-6a90-4717-bca1-1b5fc6b75e4a.png">
Ecoset

## Adding a Dataset
- **Name:** *Ecoset*
- **Description:** *https://www.kietzmannlab.org/ecoset/*
- **Paper:** *https://doi.org/10.1073/pnas.2011417118*
- **Data:** *https://codeocean.com/capsule/9570390/tree/v1*
- **Motivation:**

**Ecoset** was created as a clean and ecologically valid alternative to **Imagenet**.

It is a large image recognition dataset, similar to Imagenet in size and structure. However, the authors of ecoset claim several improvements over Imagenet, like:
- more ecologically valid classes (e.g. not over-focussed on distinguishing different dog breeds)
- less NSFW content
- 'pre-packed image recognition models' that come with the dataset and can be used for validation of other models.


I am working for one of the authors of the paper with the aim of bringing Ecoset to huggingface datasets. Therefore I can work on this issue personally, but could use some help from devs and experienced users if the dataset is of interest to them. I phrased some of my questions on [discuss.huggingface](https://discuss.huggingface.co/t/handling-large-image-datasets/19373).


Hi! Very cool dataset! I answered your questions on the forum. Also, feel free to comment `#self-assign` on this issue to self-assign it.

The dataset lives on the Hub [here](https://huggingface.co/datasets/kietzmannlab/ecoset), so I'm closing this issue.

Hey There, thanks for closing 🤗 

Forgot the issue existed, so I didn't close it after implementing the downloader :)