Add Conceptual 12M

Looks like your dummy_data.zip file is not in the right location ;)
datasets/datasets/conceptual_12m/dummy/default/0.0.0/dummy_data.zip
->
datasets/conceptual_12m/dummy/default/0.0.0/dummy_data.zip


Add Visual Genome

_The documentation is not available anymore as the PR was closed or merged._

Hum there seems to be some issues with tasks in test:
 - some tasks don't fit anything in `tasks.json`. Do I remove them in `task_categories`?
 - some tasks should exist, typically `visual-question-answering` (https://github.com/huggingface/datasets/blame/9f2ff14673cac1f1ad56d80221a793f5938b68c7/src/datasets/utils/resources/tasks.json#L195) yet the exception is failing on me. I'm guessing it's because my `master` is not up-to-date. However this means that the testing only tests my branch instead of the one merged with master?

 cc @mariosasko @lhoestq

> some tasks don't fit anything in tasks.json. Do I remove them in task_categories?

You can keep them, but add `other-` as a prefix to those tasks to make the CI ignore it

> some tasks should exist, typically visual-question-answering (https://github.com/huggingface/datasets/blame/9f2ff14673cac1f1ad56d80221a793f5938b68c7/src/datasets/utils/resources/tasks.json#L195) yet the exception is failing on me. I'm guessing it's because my master is not up-to-date. However this means that the testing only tests my branch instead of the one merged with master?

Feel free to merge upstream/master into your branch ;)

EDIT: actually I just noticed you've already done this, thanks !

After offline discussions: will keep that image essentially it's necessary as I have a mapping that creates a mapping between url and local path (images are downloaded via a zip file) and dummy data needs to store that dummy image. The issue is when I read an annotation, I get a url, compute the local path, and basically I assume the local path exists since I've extracted all the images ... This isn't true if dummy data doesn't have all the images, so instead I've added a script that "fixes" the dummy data after using the CLI, it essentially adds the dummy image in the zip corresponding to the url.


RGBA images not showing

dataset-viewer

Related to the dataset viewer on huggingface.co

Thanks for reporting. It's a known issue, and we hope to fix it soon.

## Dataset viewer issue for ceyda/smithsonian_butterflies_transparent

[**Link:** *link to the dataset viewer page*](https://huggingface.co/datasets/ceyda/smithsonian_butterflies_transparent)

![image](https://user-images.githubusercontent.com/15624271/163117683-e91edb28-41bf-43d9-b371-5c62e14f40c9.png)

Am I the one who added this dataset ? Yes

👉  More of a general issue of 'RGBA' png images not being supported
(the dataset itself is just for the huggan sprint and not that important, consider it just an example)


Add `TruthfulQA` dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4159). All of your documentation changes will be reflected on that endpoint.


Add AUC ROC Metric


Fix formatting in BLEU metric card

Fix #4148


Adding STSb-TR dataset

Semantic Textual Similarity benchmark Turkish (STSb-TR) dataset introduced in our paper [Semantic Similarity Based Evaluation for Abstractive News Summarization](https://aclanthology.org/2021.gem-1.3.pdf) added.


Make HANS dataset streamable

Fix #4133


Generate tasks.json taxonomy from `huggingface_hub`

Ok recomputed the json file, this should be ready to review now! @lhoestq

Note: the generated JSON from `hf/hub-docs` can be found in the output of a GitHub Action run on that repo, for instance in https://github.com/huggingface/hub-docs/runs/6006686983?check_suite_focus=true

(click on "Run export-tasks script")

Should we not add the tasks with hideInDatasets?

yes, probably true – i'll change that in a PR in `hub-docs`


Adding Text-based NP Enrichment (TNE) dataset

Hey @lhoestq, can you please have a look? 🙏

Great, thanks again @lhoestq! I think we're good to go now

Added the [TNE](https://github.com/yanaiela/TNE) dataset to the library


ArrayND error in pyarrow 5

Where do we bump the required pyarrow version? Any inputs on how I fix this issue?

We need to bump it in `setup.py` as well as update some CI job to use pyarrow 6 instead of 5 in `.circleci/config.yaml` and `.github/workflows/benchmarks.yaml`

As found in https://github.com/huggingface/datasets/pull/3903, The ArrayND features fail on pyarrow 5:
```python
import pyarrow as pa
from datasets import Array2D
from datasets.table import cast_array_to_feature

arr = pa.array([[[0]]])
feature_type = Array2D(shape=(1, 1), dtype="int64")
cast_array_to_feature(arr, feature_type)
```
raises
```python
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-04610f9fa78c> in <module>
----> 1 cast_array_to_feature(pa.array([[[0]]]), Array2D(shape=(1, 1), dtype="int32"))

~/Desktop/hf/datasets/src/datasets/table.py in wrapper(array, *args, **kwargs)
   1672             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1673         else:
-> 1674             return func(array, *args, **kwargs)
   1675
   1676     return wrapper

~/Desktop/hf/datasets/src/datasets/table.py in cast_array_to_feature(array, feature, allow_number_to_str)
   1806         return array_cast(array, get_nested_type(feature), allow_number_to_str=allow_number_to_str)
   1807     elif not isinstance(feature, (Sequence, dict, list, tuple)):
-> 1808         return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
   1809     raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
   1810

~/Desktop/hf/datasets/src/datasets/table.py in wrapper(array, *args, **kwargs)
   1672             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1673         else:
-> 1674             return func(array, *args, **kwargs)
   1675
   1676     return wrapper

~/Desktop/hf/datasets/src/datasets/table.py in array_cast(array, pa_type, allow_number_to_str)
   1705         array = array.storage
   1706     if isinstance(pa_type, pa.ExtensionType):
-> 1707         return pa_type.wrap_array(array)
   1708     elif pa.types.is_struct(array.type):
   1709         if pa.types.is_struct(pa_type) and (

AttributeError: 'Array2DExtensionType' object has no attribute 'wrap_array'
```

The thing is that `cast_array_to_feature` is called when writing an Arrow file, so creating an Arrow dataset using any ArrayND type currently fails.

`wrap_array` has been added in pyarrow 6, so we can either bump the required pyarrow version or fix this for pyarrow 5


Add missing label for emotion description


Inconsistent splits generation for datasets without loading script (packaged dataset puts everything into a single split)

bug

Something isn't working

## Describe the bug
Splits for dataset loaders without scripts are prepared inconsistently. I think it might be confusing for users.

## Steps to reproduce the bug
* If you load a packaged datasets from Hub, it infers splits from directory structure / filenames (check out the data [here](https://huggingface.co/datasets/nateraw/test-imagefolder-dataset)):
```python
ds = load_dataset("nateraw/test-imagefolder-dataset")
print(ds)
### Output:
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 6
    })
    test: Dataset({
        features: ['image', 'label'],
        num_rows: 4
    })
})
```
* If you do the same from locally stored data specifying only directory path you'll get the same:
```python
ds = load_dataset("/path/to/local/data/test-imagefolder-dataset")
print(ds)
### Output:
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 6
    })
    test: Dataset({
        features: ['image', 'label'],
        num_rows: 4
    })
})
```
* However, if you explicitely specify package name (like `imagefolder`, `csv`, `json`), all the data is put into a single split:
```python
ds = load_dataset("imagefolder", data_dir="/path/to/local/data/test-imagefolder-dataset")
print(ds)
### Output:
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 10
    })
})
```

## Expected results
For `load_dataset("imagefolder", data_dir="/path/to/local/data/test-imagefolder-dataset")` I expect the same output as of the two first options.


load_dataset for winoground returning decoding error

Something isn't working

I thought I had fixed it with this after some helpful hints from @severo
```python
import datasets
token = 'hf_XXXXX'
dataset = datasets.load_dataset(
    'facebook/winoground',
    name='facebook--winoground',
    split='train',
    streaming=True,
    use_auth_token=token,
)
```
but I found out that wasn't the case
```python
[x for x in dataset]
...
ClientResponseError: 401, message='Unauthorized', url=URL('https://huggingface.co/datasets/facebook/winoground/resolve/a86a60456fbbd242e9a744199071a6bd3e7fd9de/examples.jsonl')
```

Hi ! This dataset structure (image + labels in a JSON file) is not supported yet, though we're adding support for this in  in #4069

The following structure will be supported soon:
```
metadata.json
images/
     image0.png
     image1.png
    ...
```
Where `metadata.json` is a JSON Lines file with labels or other metadata, and each line must have a "file_name" field with the name of the image file.

For the moment are only supported:
- JSON files only
- image files only

Since this dataset is a mix of the two, at the moment it fails trying to read the images as JSON.

Therefore to be able to load this dataset we need to wait for the new structure to be supported (very soon ^^), or add a dataset script in the repository that reads both the JSON and the images cc @TristanThrush

We'll also investigate the issue with the streaming download manager in https://github.com/huggingface/datasets/issues/4139 ;) thanks for reporting

In the meantime, anyone can always download the images.zip and examples.jsonl files directly from huggingface.co - let me know if anyone has issues with that.

I mirrored the files at https://huggingface.co/datasets/facebook/winoground in a folder on my local machine `winground`
and when I tried
```python
import datasets
ds = datasets.load_from_disk('./winoground')
```
I get the following error
```python
--------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Input In [2], in <cell line: 1>()
----> 1 ds = datasets.load_from_disk('./winoground')

File ~/.local/lib/python3.8/site-packages/datasets/load.py:1759, in load_from_disk(dataset_path, fs, keep_in_memory)
   1757     return DatasetDict.load_from_disk(dataset_path, fs, keep_in_memory=keep_in_memory)
   1758 else:
-> 1759     raise FileNotFoundError(
   1760         f"Directory {dataset_path} is neither a dataset directory nor a dataset dict directory."
   1761     )

FileNotFoundError: Directory ./winoground is neither a dataset directory nor a dataset dict directory.
```
so still some work to be done on the backend imo.

Note that `load_from_disk` is the function that reloads an Arrow dataset saved with `my_dataset.save_to_disk`.

Once we do support images with metadata you'll be able to use `load_dataset("facebook/winoground")` directly (or `load_dataset("./winoground")` of you've cloned the winoground repository locally).

Apologies for the delay. I added a custom dataset loading script for winoground. It should work now, with an auth token:

`examples = load_dataset('facebook/winoground', use_auth_token=<your auth token>)`

Let me know if there are any issues

Adding the dataset loading script definitely didn't take as long as I thought it would 😅

## Describe the bug
I am trying to use datasets to load winoground and I'm getting a JSON decoding error.

## Steps to reproduce the bug
```python
from datasets import load_dataset
token = 'hf_XXXXX' # my HF access token
datasets = load_dataset('facebook/winoground', use_auth_token=token)
```

## Expected results
I downloaded images.zip and examples.jsonl manually. I was expecting to have some trouble decoding json so I didn't use jsonlines but instead was able to get a complete set of 400 examples by doing
```python
import json

with open('examples.jsonl', 'r') as f:
    examples = f.read().split('\n')

# Thinking this would error if the JSON is not utf-8 encoded
json_data = [json.loads(x) for x in examples]
print(json_data[-1])
```
and I see
```python
{'caption_0': 'someone is overdoing it',
 'caption_1': 'someone is doing it over',
 'collapsed_tag': 'Relation',
 'id': 399,
 'image_0': 'ex_399_img_0',
 'image_1': 'ex_399_img_1',
 'num_main_preds': 1,
 'secondary_tag': 'Morpheme-Level',
 'tag': 'Scope, Preposition'}

```
so I'm not sure what's going on here honestly. The file `examples.jsonl` doesn't have non-UTF-8 encoded text.

## Actual results
During the split operation after downloading, datasets encounters an error in the JSON ([trace](https://gist.github.com/odellus/e55d390ca203386bf551f38e0c63a46b) abbreviated for brevity).
```
datasets/packaged_modules/json/json.py:144 in Json._generate_tables(self, files)
...
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.4
- Platform: Linux-5.13.0-39-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 7.0.0


fix confusing bleu metric example

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
I would like to see the example in "Metric Card for BLEU" changed.
The 0th element in the predictions list is not closed in square brackets, and the 1st list is missing a comma.
The BLEU score are calculated correctly, but it is difficult to understand, so it would be helpful if you could correct this.
```
>> predictions = [
...     ["hello", "there", "general", "kenobi",       # <- no closing square bracket.
...     ["foo", "bar" "foobar"]                              # <- no comma between "bar" and "foobar"
... ]
>>> references = [
...     [["hello", "there", "general", "kenobi"]],
...     [["foo", "bar", "foobar"]]
... ]
>>> bleu = datasets.load_metric("bleu")
>>> results = bleu.compute(predictions=predictions, references=references)
>>> print(results)
{'bleu': 0.6370964381207871, ...
```

**Describe the solution you'd like**
```
>> predictions = [
...     ["hello", "there", "general", "kenobi",       # <- no closing square bracket.
...     ["foo", "bar" "foobar"]                              # <- no comma between "bar" and "foobar"
... ]
# and
>>> print(results)
{'bleu':1.0, ...
```


Adjust path to datasets tutorial in How-To

_The documentation is not available anymore as the PR was closed or merged._

The link in the How-To overview page to the Datasets tutorials is currently broken. This is just a small adjustment to make it match the format used in https://github.com/huggingface/datasets/blob/master/docs/source/tutorial.md.

(Edit to add: The link in the PR deployment (https://moon-ci-docs.huggingface.co/docs/datasets/pr_4147/en/how_to) is also broken since it's actually hardcoded to `master` and not dynamic to the branch name, but other links seem to behave similarly.)


SAMSum dataset viewer not working

bug

Something isn't working

https://huggingface.co/datasets/samsum

```
Status code:   400
Exception:     ValueError
Message:       Cannot seek streaming HTTP file
```

Currently, only the datasets that can be streamed support the dataset viewer. Maybe @lhoestq @albertvillanova or @mariosasko could give more details about why the dataset cannot be streamed.

It looks like the host (https://arxiv.org) doesn't allow HTTP Range requests, which is what we use to stream data.

This can be fix if we host the data ourselves, which is ok since the dataset is under CC BY-NC-ND 4.0

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No


Redirect TIMIT download from LDC

CI is failing because some tags are outdated, but they're fixed in #4067

We may do a release pretty soon (today ?), let me know if it's fine to include it in the new release

LDC data is protected under US copyright laws and under various legal agreements between the Linguistic Data Consortium/the University of Pennsylvania and data providers which prohibit redistribution of that data by anyone other than LDC. Similarly, LDC's membership agreements, non-member user agreement and various corpus-specific license agreements specifically state that users cannot publish, retransmit, disclose, copy, reproduce or redistribute LDC databases to others outside their organizations.

LDC explicitly asked us to remove the download script for the TIMIT dataset. In this PR I remove all means to download the dataset, and redirect users to download the data from https://catalog.ldc.upenn.edu/LDC93S1


Fix splits in local packaged modules, local datasets without script and hub datasets without script

Thanks !
I'm in favor of this change, even though it's a breaking change:

if you had a dataset
```
data/
  train.csv
  test.csv
```

then running this code would now return both train and test splits:
```python
load_dataset("csv", data_dir="data/")
```
whereas right now it returns only a train split with the data from both CSV files.

In my opinion it's ok do do this breaking change because:
- it makes this behavior consistent with `load_dataset("path/to/data")` that also returns both splits: data_files resolution must be the same
- I don't expect too many affected users (unless people really wanted to group train and test images in the train split on purpose ?) compared to the many new users to come (especially with #4069 )
- this usage will become more and more common as we add packaged builder and imagefolder/audiofolder usage grows, so it may be better to do this change early

Let me know if you think this is acceptable @mariosasko @albertvillanova or not, and if you think we need to first have a warning for some time before switching to this new behavior

Also, if people really want to put train and test, say, images in a single train split they could do
`load_dataset("imagefolder", data_files={"train": "/path/to/data/**})`. Probably (arguably :)), if this is a more counterintuitive case, then it should require manual files specification, not a default one (in which we expect that users do want to infer splits from filenames / dir structure but currently they have to pass smth like `{"train": "/path/to/data/train*", "test": "/path/to/data/test*"}` explicitly as `data_files`)

I also like this change, and  I don't think we even need a warning during the transition period, considering I've been asked several times since the release of `imagefolder` why splits are not correctly inferred if the directory structure is as follows:
```
data_dir
    train
        label_a
            0.jpg
            ...
        label_b
            0.jpg
            ...
    test
        label_a
            0.jpg
            ...
        label_b
            0.jpg
            ...
```

Cool ! Feel free to add a test (maybe something similar to `test_PackagedDatasetModuleFactory_with_data_dir` but with a data_dir that contains several splits) and mark this PR as ready for review then @polinaeterna :)

@lhoestq @mariosasko do you think it's a good idea to do the same with `HubDatasetModuleFactoryWithoutScript` and `LocalDatasetModuleFactoryWithoutScript` (see the latest change). If we agree on the current change, doing
```python
ds = load_dataset("polinaeterna/jsonl_test", data_dir="data/")
```
on dataset with the following structure:
```
train.jsonl
test.jsonl
data/
   train.jsonl
   test.jsonl
```
will result in having two splits from files under `data/` dir in specified repo, while master version returns a single train split.
The same would be for local dataset without script if doing smth like:
```python
ds = load_dataset("/home/polina/workspace/repos/jsonl_test", data_dir="/home/polina/workspace/repos/jsonl_test/data")
```
(though I'm not sure I understand this use case :D)
Let me know if you think we should preserve the same logic for all factories or if I should roll back this change.

@lhoestq to test passing subdirectory (`base_path`) to data_files functions and methods, I extended the temporary test directory with data so that it contains subdirectory. Because of that the number of files in this directory increased, so I had to change some numbers and patterns to account for this change - [907ddf0](https://github.com/huggingface/datasets/pull/4144/commits/907ddf09d3afece5afbae18675c859d6e453f2bf)

Do you think it's ok? Another option is to create another tmp dir and do all the checks inside it.

fixes #4150

I suggest to infer splits structure from files when `data_dir` is passed with `get_patterns_locally`, analogous to what's done in `LocalDatasetModuleFactoryWithoutScript` with `self.path`, instead of generating files with `data_dir/**` patterns and putting them all into a single default (train) split.

I would also suggest to align `HubDatasetModuleFactoryWithoutScript` and `LocalDatasetModuleFactoryWithoutScript` with this logic (remove `data_files = os.path.join(data_dir, "**")`). It's not reflected in the current code now as I'd like to discuss it cause I might be unaware of some use cases. @lhoestq @mariosasko @albertvillanova  WDYT?


Unable to download `Wikepedia` 20220301.en version

Something isn't working

Hi! We've recently updated the Wikipedia script, so these changes are only available on master and can be fetched as follows:
```python
dataset_wikipedia = load_dataset("wikipedia", "20220301.en", revision="master")
```

Hi, how can I load the previous "20200501.en" version of wikipedia which had been downloaded to the default path? Thanks!

## Describe the bug

Unable to download `Wikepedia` dataset, 20220301.en version

## Steps to reproduce the bug
```python
!pip install apache_beam mwparserfromhell
dataset_wikipedia = load_dataset("wikipedia", "20220301.en")
```

## Actual results
```
ValueError: BuilderConfig 20220301.en not found.
Available: ['20200501.aa', '20200501.ab', '20200501.ace', '20200501.ady', '20200501.af', '20200501.ak', '20200501.als', '20200501.am', '20200501.an', '20200501.ang', '20200501.ar', '20200501.arc', '20200501.arz', '20200501.as', '20200501.ast', '20200501.atj', '20200501.av', '20200501.ay', '20200501.az', '20200501.azb', '20200501.ba', '20200501.bar', '20200501.bat-smg', '20200501.bcl', '20200501.be', '20200501.be-x-old', '20200501.bg', '20200501.bh', '20200501.bi', '20200501.bjn', '20200501.bm', '20200501.bn', '20200501.bo', '20200501.bpy', '20200501.br', '20200501.bs', '20200501.bug', '20200501.bxr', '20200501.ca', '20200501.cbk-zam', '20200501.cdo', '20200501.ce', '20200501.ceb', '20200501.ch', '20200501.cho', '20200501.chr', '20200501.chy', '20200501.ckb', '20200501.co', '20200501.cr', '20200501.crh', '20200501.cs', '20200501.csb', '20200501.cu', '20200501.cv', '20200501.cy', '20200501.da', '20200501.de', '20200501.din', '20200501.diq', '20200501.dsb', '20200501.dty', '20200501.dv', '20200501.dz', '20200501.ee', '20200501.el', '20200501.eml', '20200501.en', '20200501.eo', '20200501.es', '20200501.et', '20200501.eu', '20200501.ext', '20200501.fa', '20200501.ff', '20200501.fi', '20200501.fiu-vro', '20200501.fj', '20200501.fo', '20200501.fr', '20200501.frp', '20200501.frr', '20200501.fur', '20200501.fy', '20200501.ga', '20200501.gag', '20200501.gan', '20200501.gd', '20200501.gl', '20200501.glk', '20200501.gn', '20200501.gom', '20200501.gor', '20200501.got', '20200501.gu', '20200501.gv', '20200501.ha', '20200501.hak', '20200501.haw', '20200501.he', '20200501.hi', '20200501.hif', '20200501.ho', '20200501.hr', '20200501.hsb', '20200501.ht', '20200501.hu', '20200501.hy', '20200501.ia', '20200501.id', '20200501.ie', '20200501.ig', '20200501.ii', '20200501.ik', '20200501.ilo', '20200501.inh', '20200501.io', '20200501.is', '20200501.it', '20200501.iu', '20200501.ja', '20200501.jam', '20200501.jbo', '20200501.jv', '20200501.ka', '20200501.kaa', '20200501.kab', '20200501.kbd', '20200501.kbp', '20200501.kg', '20200501.ki', '20200501.kj', '20200501.kk', '20200501.kl', '20200501.km', '20200501.kn', '20200501.ko', '20200501.koi', '20200501.krc', '20200501.ks', '20200501.ksh', '20200501.ku', '20200501.kv', '20200501.kw', '20200501.ky', '20200501.la', '20200501.lad', '20200501.lb', '20200501.lbe', '20200501.lez', '20200501.lfn', '20200501.lg', '20200501.li', '20200501.lij', '20200501.lmo', '20200501.ln', '20200501.lo', '20200501.lrc', '20200501.lt', '20200501.ltg', '20200501.lv', '20200501.mai', '20200501.map-bms', '20200501.mdf', '20200501.mg', '20200501.mh', '20200501.mhr', '20200501.mi', '20200501.min', '20200501.mk', '20200501.ml', '20200501.mn', '20200501.mr', '20200501.mrj', '20200501.ms', '20200501.mt', '20200501.mus', '20200501.mwl', '20200501.my', '20200501.myv', '20200501.mzn', '20200501.na', '20200501.nah', '20200501.nap', '20200501.nds', '20200501.nds-nl', '20200501.ne', '20200501.new', '20200501.ng', '20200501.nl', '20200501.nn', '20200501.no', '20200501.nov', '20200501.nrm', '20200501.nso', '20200501.nv', '20200501.ny', '20200501.oc', '20200501.olo', '20200501.om', '20200501.or', '20200501.os', '20200501.pa', '20200501.pag', '20200501.pam', '20200501.pap', '20200501.pcd', '20200501.pdc', '20200501.pfl', '20200501.pi', '20200501.pih', '20200501.pl', '20200501.pms', '20200501.pnb', '20200501.pnt', '20200501.ps', '20200501.pt', '20200501.qu', '20200501.rm', '20200501.rmy', '20200501.rn', '20200501.ro', '20200501.roa-rup', '20200501.roa-tara', '20200501.ru', '20200501.rue', '20200501.rw', '20200501.sa', '20200501.sah', '20200501.sat', '20200501.sc', '20200501.scn', '20200501.sco', '20200501.sd', '20200501.se', '20200501.sg', '20200501.sh', '20200501.si', '20200501.simple', '20200501.sk', '20200501.sl', '20200501.sm', '20200501.sn', '20200501.so', '20200501.sq', '20200501.sr', '20200501.srn', '20200501.ss', '20200501.st', '20200501.stq', '20200501.su', '20200501.sv', '20200501.sw', '20200501.szl', '20200501.ta', '20200501.tcy', '20200501.te', '20200501.tet', '20200501.tg', '20200501.th', '20200501.ti', '20200501.tk', '20200501.tl', '20200501.tn', '20200501.to', '20200501.tpi', '20200501.tr', '20200501.ts', '20200501.tt', '20200501.tum', '20200501.tw', '20200501.ty', '20200501.tyv', '20200501.udm', '20200501.ug', '20200501.uk', '20200501.ur', '20200501.uz', '20200501.ve', '20200501.vec', '20200501.vep', '20200501.vi', '20200501.vls', '20200501.vo', '20200501.wa', '20200501.war', '20200501.wo', '20200501.wuu', '20200501.xal', '20200501.xh', '20200501.xmf', '20200501.yi', '20200501.yo', '20200501.za', '20200501.zea', '20200501.zh', '20200501.zh-classical', '20200501.zh-min-nan', '20200501.zh-yue', '20200501.zu']
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Ubuntu
- Python version: 3.6
- PyArrow version: 6.0.1


Add ObjectFolder 2.0 dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** ObjectFolder 2.0
- **Description:** ObjectFolder 2.0 is a dataset of 1,000 objects in the form of implicit representations. It contains 1,000 Object Files each containing the complete multisensory profile for an object instance.
- **Paper:** [*link to the dataset paper if available*](https://arxiv.org/abs/2204.02389)
- **Data:** https://github.com/rhgao/ObjectFolder

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Why is the dataset not visible under the dataset preview section?


Error loading arxiv data set

bug

Something isn't working

Hi! I think this error may be related to using an older version of the library. I was able to load the dataset without any issues using the latest version of `datasets`.  Can you upgrade to the latest version of `datasets` and try again? :)

Hi! As @stevhliu suggested, to fix the issue, update the lib to the newest version with:
```
pip install -U datasets
```
and download the dataset as follows:
```python
from datasets import load_dataset
dset =  load_dataset('scientific_papers', 'arxiv', download_mode="force_redownload")
```

Thanks for the quick response! It works now. The problem is that I used nlp. load_dataset instead of datasets. load_dataset.

## Describe the bug
A clear and concise description of what the bug is.

I met the error below when loading arxiv dataset via `nlp.load_dataset('scientific_papers', 'arxiv',)`.
```
Traceback (most recent call last):
  File "scripts/summarization.py", line 354, in <module>
    main(args)
  File "scripts/summarization.py", line 306, in main
    model.hf_datasets = nlp.load_dataset('scientific_papers', 'arxiv')
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/load.py", line 549, in load_dataset
    download_config=download_config, download_mode=download_mode, ignore_verifications=ignore_verifications,
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/builder.py", line 463, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/builder.py", line 522, in _download_and_prepare
    self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/utils/info_utils.py", line 38, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
nlp.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?id=1b3rmCSIoh6VhD4HKWjI4HOW-cSwcwbeC&export=download', 'https://drive.google.com/uc?id=1lvsqvsFi3W-pE1SqNZI0s8NR9rC1tsja&export=download']
```

I then tried to ignore verification steps by `ignore_verifications=True` and there is another error.

```
Traceback (most recent call last):
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/builder.py", line 537, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/builder.py", line 810, in _prepare_split
    for key, record in utils.tqdm(generator, unit=" examples", total=split_info.num_examples, leave=False):
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/datasets/scientific_papers/9e4f2cfe3d8494e9f34a84ce49c3214605b4b52a3d8eb199104430d04c52cc12/scientific_papers.py", line 108, in _generate_examples
    with open(path, encoding="utf-8") as f:
NotADirectoryError: [Errno 20] Not a directory: '/home/username/.cache/huggingface/datasets/downloads/c0deae7af7d9c87f25dfadf621f7126f708d7dcac6d353c7564883084a000076/arxiv-dataset/train.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scripts/summarization.py", line 354, in <module>
    main(args)
  File "scripts/summarization.py", line 306, in main
    model.hf_datasets = nlp.load_dataset('scientific_papers', 'arxiv', ignore_verifications=True)
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/load.py", line 549, in load_dataset
    download_config=download_config, download_mode=download_mode, ignore_verifications=ignore_verifications,
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/builder.py", line 463, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/builder.py", line 539, in _download_and_prepare
    raise OSError("Cannot find data file. " + (self.manual_download_instructions or ""))
OSError: Cannot find data file.
```

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:


Dataset viewer issue for Winoground

related (same dataset): https://github.com/huggingface/datasets/issues/4149. But the issue is different. Looking at it

I thought this issue was related to the error I was seeing, but upon consideration I'd think the dataset viewer would return a 500 (unable to create the split like me) or a 404 (unable to load split b/c it was never created) error if it was having the issue I was seeing in #4149. 401 message makes it look like dataset viewer isn't passing through the identity of the user who has signed the licensing agreement when making the request to GET [examples.jsonl](https://huggingface.co/datasets/facebook/winoground/resolve/a86a60456fbbd242e9a744199071a6bd3e7fd9de/examples.jsonl).

Pinging @SBrandeis, as it seems related to gated datasets and access tokens.

To replicate:

```python
>>> import datasets
>>> dataset= datasets.load_dataset('facebook/winoground', name='facebook--winoground', split='train', use_auth_token="hf_app_...", streaming=True)
>>> next(iter(dataset))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 497, in __iter__
    for key, example in self._iter():
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 494, in _iter
    yield from ex_iterable
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 87, in __iter__
    yield from self.generate_examples_fn(**self.kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 439, in wrapper
    for key, table in generate_tables_fn(**kwargs):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/packaged_modules/json/json.py", line 85, in _generate_tables
    for file_idx, file in enumerate(files):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/utils/streaming_download_manager.py", line 679, in __iter__
    yield from self.generator(*self.args, **self.kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/utils/streaming_download_manager.py", line 731, in _iter_from_urlpaths
    for dirpath, _, filenames in xwalk(urlpath, use_auth_token=use_auth_token):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/utils/streaming_download_manager.py", line 623, in xwalk
    for dirpath, dirnames, filenames in fs.walk(main_hop):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 372, in walk
    listing = self.ls(path, detail=True, **kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/asyn.py", line 85, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/asyn.py", line 65, in sync
    raise return_result
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 196, in _ls
    out = await self._ls_real(url, detail=detail, **kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 150, in _ls_real
    self._raise_not_found_for_status(r, url)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 208, in _raise_not_found_for_status
    response.raise_for_status()
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://huggingface.co/datasets/facebook/winoground/resolve/a86a60456fbbd242e9a744199071a6bd3e7fd9de/examples.jsonl')
```

*edited to fix `use_token` -> `use_auth_token`, thx @odellus*

~~Using your command to replicate and changing `use_token` to `use_auth_token` fixes the problem I was seeing in #4149.~~
Nevermind it gave me an iterator to a method returning the same 401s. Changing `use_token` to `use_auth_token` does not fix the issue.

After investigation with @severo , we found a potential culprit: https://github.com/huggingface/datasets/blob/3cd0a009a43f9f174056d70bfa2ca32216181926/src/datasets/utils/streaming_download_manager.py#L610-L624

The streaming manager does not seem to pass `use_auth_token` to `fsspec` when streaming and not iterating content of a zip archive

cc @albertvillanova @lhoestq

I was able to reproduce it on a private dataset, let me work on a fix

Hey @lhoestq, Thanks for working on a fix! Any plans to merge #4173 into master?

Thanks for the heads up, I still need to fix some tests that are failing in the CI before merging ;)

The fix has been merged, we'll do a new release soon, and update the dataset viewer

## Dataset viewer issue for 'Winoground'

**Link:** [*link to the dataset viewer page*](https://huggingface.co/datasets/facebook/winoground/viewer/facebook--winoground/train)

*short description of the issue*
Getting 401, message='Unauthorized'
The dataset is subject to authorization, but I can access the files from the interface, so I assume I'm granted to access it. I'd assume the permission somehow doesn't propagate to the dataset viewer tool.

Am I the one who added this dataset ? No


Incorrect Russian filenames encoding after extraction by datasets.DownloadManager.download_and_extract()

To reproduce:

```python
>>> import datasets
>>> datasets.get_dataset_split_names('MalakhovIlya/RuREBus', config_name='raw_txt')
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 280, in get_dataset_config_info
    for split_generator in builder._split_generators(
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/MalakhovIlya--RuREBus/21046f5f1a0cf91187d68c30918d78d934ec7113ec435e146776d4f28f12c4ed/RuREBus.py", line 101, in _split_generators
    decode_file_names(folder)
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/MalakhovIlya--RuREBus/21046f5f1a0cf91187d68c30918d78d934ec7113ec435e146776d4f28f12c4ed/RuREBus.py", line 26, in decode_file_names
    for root, dirs, files in os.walk(folder, topdown=False):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/streaming.py", line 66, in wrapper
    return function(*args, use_auth_token=use_auth_token, **kwargs)
TypeError: xwalk() got an unexpected keyword argument 'topdown'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 323, in get_dataset_split_names
    info = get_dataset_config_info(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 285, in get_dataset_config_info
    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

It's not related to the dataset viewer. Maybe @albertvillanova or @lhoestq could help more on this issue.

Hi! This issue stems from the fact that `xwalk`, which is a streamable version of `os.walk`, doesn't support the `topdown` param due to `fsspec`'s `walk` also not supporting it, so fixing this issue could be tricky.

@MalakhovIlyaPavlovich You can avoid the error by tweaking your data processing and not using this param. (and `Path.rename`, which also cannot be streamed)

@mariosasko thank you for your reply. I couldn't reproduce error showed by @severo either on Ubuntu 20.04.3 LTS, Windows 10 and Google Colab environments. But trying to avoid using os.walk(topdown=False) and Path.rename(), In _split_generators I replaced
```
def decode_file_names(folder):
    for root, dirs, files in os.walk(folder, topdown=False):
        root = Path(root)
        for file in files:
            old_name = root / Path(file)
            new_name = root / Path(
                file.encode('cp437').decode('cp866'))
            old_name.rename(new_name)
        for dir in dirs:
            old_name = root / Path(dir)
            new_name = root / Path(dir.encode('cp437').decode('cp866'))
            old_name.rename(new_name)

folder = dl_manager.download_and_extract(self._RAW_TXT_URLS)['raw_txt']
decode_file_names(folder)
```
by
```
def extract(zip_file_path):
    p = Path(zip_file_path)
    dest_dir = str(p.parent / 'extracted' / p.stem)
    os.makedirs(dest_dir, exist_ok=True)
    with zipfile.ZipFile(zip_file_path) as archive:
        for file_info in tqdm(archive.infolist(), desc='Extracting'):
            filename = file_info.filename.encode('cp437').decode('cp866')
            target = os.path.join(dest_dir, *filename.split('/'))
            os.makedirs(os.path.dirname(target), exist_ok=True)
            if not file_info.is_dir():
                with archive.open(file_info) as source, open(target, 'wb') as dest:
                    shutil.copyfileobj(source, dest)
    return dest_dir

zip_file = dl_manager.download(self._RAW_TXT_URLS)['raw_txt']
if not is_url(zip_file):
    folder = extract(zip_file)
else:
    folder = None
```
and now everything works well except data viewer for "raw_txt" subset: dataset preview on hub shows "No data.". As far as I understand dl_manager.download returns original URL when we are calling datasets.get_dataset_split_names and my suspicions are that dataset viewer can do smth similar. I couldn't find information about how it works. I would be very grateful, if you could tell me how to fix this)

This is what I get when I try to stream the `raw_txt` subset:
```python
>>> dset = load_dataset("MalakhovIlya/RuREBus", "raw_txt", split="raw_txt", streaming=True)
>>> next(iter(dset))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
```
So there is a bug in your script.

streaming=True helped me to find solution. I fixed
```
def extract(zip_file_path):
    p = Path(zip_file_path)
    dest_dir = str(p.parent / 'extracted' / p.stem)
    os.makedirs(dest_dir, exist_ok=True)
    with zipfile.ZipFile(zip_file_path) as archive:
        for file_info in tqdm(archive.infolist(), desc='Extracting'):
            filename = file_info.filename.encode('cp437').decode('cp866')
            target = os.path.join(dest_dir, *filename.split('/'))
            os.makedirs(os.path.dirname(target), exist_ok=True)
            if not file_info.is_dir():
                with archive.open(file_info) as source, open(target, 'wb') as dest:
                    shutil.copyfileobj(source, dest)
    return dest_dir

zip_file = dl_manager.download(self._RAW_TXT_URLS)['raw_txt']
folder = extract(zip_file)
```
by
```
folder = dl_manager.download_and_extract(self._RAW_TXT_URLS)['raw_txt']
path = os.path.join(folder, 'MED_txt/unparsed_txt')
for root, dirs, files in os.walk(path):
    decoded_root_name = Path(root).name.encode('cp437').decode('cp866')
```
@mariosasko thank you for your help :)

## Dataset viewer issue for 'MalakhovIlya/RuREBus'

**Link:** https://huggingface.co/datasets/MalakhovIlya/RuREBus

**Description**
Using os.walk(topdown=False) in DatasetBuilder causes following error:
Status code:   400
Exception:     TypeError
Message:       xwalk() got an unexpected keyword argument 'topdown'
Couldn't find where "xwalk" come from. How can I fix this?

Am I the one who added this dataset ? Yes


Add single dataset citations for TweetEval

_The documentation is not available anymore as the PR was closed or merged._

The `test_dataset_cards` method is failing with the error:

```
if error_messages:
>           raise ValueError("\n".join(error_messages))
E           ValueError: The following issues have been found in the dataset cards:
E           YAML tags:
E           The following typing errors are found: {'annotations_creators': "(Expected `typing.List` with length > 0. Found value of type: `<class 'list'>`, with length: 0.\n)\nOR\n(Expected `typing.Dict` with length > 0. Found value of type: `<class 'list'>`, with length: 0.\n)"}
```

Adding `found` as annotation creators.

This PR adds single data citations as per request of the original creators of the TweetEval dataset.

This is a recent email from the creator:

> Could I ask you a favor? Would you be able to add at the end of the README the citations of the single datasets as well? You can just copy our readme maybe? https://github.com/cardiffnlp/tweeteval#citing-tweeteval
(just to be sure that the creator of the single datasets also get credits when tweeteval is used)

Please let me know if this looks okay or if any changes are needed.

Thanks,
Gunjan


Support streaming xtreme dataset for PAN-X config.


ELI5 supporting documents

question

Further information is requested

Hi ! Please post your question on the [forum](https://discuss.huggingface.co/), more people will be able to help you there ;)

if i am using dense search to create supporting documents for eli5 how much time it will take bcz i read somewhere that it takes about 18 hrs??


HANS dataset preview broken

The dataset cannot be loaded, be it in normal or streaming mode.

```python
>>> import datasets
>>> dataset=datasets.load_dataset("hans", split="train", streaming=True)
>>> next(iter(dataset))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 497, in __iter__
    for key, example in self._iter():
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 494, in _iter
    yield from ex_iterable
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 87, in __iter__
    yield from self.generate_examples_fn(**self.kwargs)
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/hans/1bbcb735c482acd54f2e118074b59cfd2bf5f7a5a285d4d540d1e632216672ac/hans.py", line 121, in _generate_examples
    for idx, line in enumerate(open(filepath, "rb")):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 1595, in __next__
    out = self.readline()
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 1592, in readline
    return self.readuntil(b"\n")
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 1581, in readuntil
    self.seek(start + found + len(char))
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 676, in seek
    raise ValueError("Cannot seek streaming HTTP file")
ValueError: Cannot seek streaming HTTP file
>>> dataset=datasets.load_dataset("hans", split="train", streaming=False)
Downloading and preparing dataset hans/plain_text (download: 29.51 MiB, generated: 30.34 MiB, post-processed: Unknown size, total: 59.85 MiB) to /home/slesage/.cache/huggingface/datasets/hans/plain_text/1.0.0/1bbcb735c482acd54f2e118074b59cfd2bf5f7a5a285d4d540d1e632216672ac...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/load.py", line 1687, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 605, in download_and_prepare
    self._download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 1104, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 694, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 1087, in _prepare_split
    for key, record in logging.tqdm(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/tqdm/std.py", line 1180, in __iter__
    for obj in iterable:
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/hans/1bbcb735c482acd54f2e118074b59cfd2bf5f7a5a285d4d540d1e632216672ac/hans.py", line 121, in _generate_examples
    for idx, line in enumerate(open(filepath, "rb")):
ValueError: readline of closed file
```

Hi! I've opened a PR that should make this dataset stremable. You can test it as follows:
```python
from datasets import load_dataset
dset = load_dataset("hans", split="train", streaming=True, revision="49decd29839c792ecc24ac88f861cbdec30c1c40")
```

@severo The current script doesn't throw an error in normal mode (only in streaming mode) on my local machine or in Colab. Can you update your installation of `datasets` and see if that fixes the issue?

Thanks for this. It works well, thanks! The dataset viewer is using https://github.com/huggingface/datasets/releases/tag/2.0.0, I'm eager to upgrade to 2.0.1 😉

## Dataset viewer issue for '*hans*'

**Link:** [https://huggingface.co/datasets/hans](https://huggingface.co/datasets/hans)

HANS dataset preview is broken with error 400

Am I the one who added this dataset ? No


Support streaming xtreme dataset for PAWS-X config.


Support streaming xtreme dataset for udpos config.


Add SBU Captions Photo Dataset


dataset metadata for reproducibility

enhancement

New feature or request

When pulling a dataset from the hub, it would be useful to have some metadata about the specific dataset and version that is used. The metadata could then be passed to the `Trainer` which could then be saved to a model card. This is useful for people who run many experiments on different versions (commits/branches) of the same dataset.

The dataset could have a list of “source datasets” metadata and ignore what happens to them before arriving in the Trainer (i.e. ignore mapping, filtering, etc.).

Here is a basic representation (made by @lhoestq )
```python
>>> from datasets import load_dataset
>>>
>>> my_dataset = load_dataset(...)["train"]
>>> my_dataset = my_dataset.map(...)
>>>
>>> my_dataset.sources
[HFHubDataset(repo_id=..., revision=..., arguments={...})]
```


More robust `cast_to_python_objects` in `TypedSequence`

Adds a fallback to run an expensive version of `cast_to_python_objects` which exhaustively checks entire lists to avoid the `ArrowInvalid: Could not convert` error in `TypedSequence`. Currently, this error can happen in situations where only some images are decoded in `map`, in which case `cast_to_python_objects` fails to recognize that it needs to cast `PIL.Image` objects if they are not at the beginning of the sequence and stops after the first image dictionary (e.g., if `data` is `[{"bytes": None, "path": "some path"}, PIL.Image(), ...]`)

Fix #4124


Add configs with processed data in medical_dialog dataset

There exist processed data files that do not require parsing the raw data files (which can take long time).

Fix #4122.


dataset viewer issue for common_voice

Yes, it's a known issue, and we expect to fix it soon.

Fixed.

<img width="1393" alt="Capture d’écran 2022-04-25 à 15 42 05" src="https://user-images.githubusercontent.com/1676121/165101176-d729d85b-efff-45a8-bad1-b69223edba5f.png">

## Dataset viewer issue for 'common_voice'

**Link:** https://huggingface.co/datasets/common_voice

Server Error
Status code:   400
Exception:     TypeError
Message:       __init__() got an unexpected keyword argument 'audio_column'

Am I the one who added this dataset ? No


BIG-bench

> It looks like the CI is failing on windows because our windows CI is unable to clone the bigbench repository (maybe it has to do with filenames that are longer than 256 characters, which windows don't like). Could the smaller installation of bigbench via pip solve this issue ?
> Otherwise we can see how to remove this limitation in our windows CI.

I'm not sure.
If it's git's fault that it can't handle the long filenames, it will possibly be resolved by the pip install. If it's an issue with windows not liking long filenames after it's installed, then it will not be resolved.
I don't have a windows computer to try it on, but I might be able to tweek this PR and do an experiment to find out.
We're waiting for a quota increase for the pip install (https://github.com/pypa/pypi-support/issues/1782). It's been pending for 2-3 weeks, and I don't have an estimate for when it will be resolved.

>Regarding the dummy data zip files, I think we can just keep datasets/bigbench/dummy/abstract_narrative_understanding/1.0.0/dummy_data.zip and remove all the other ones. We just require to have at least one dummy_data.zip file.

Sounds great. I will trim that down.

Do you know what are the other tests dependencies that have conflicts with bigbench ? I can try to split the CI to end up with a compatible list of test dependencies

Hi @lhoestq,

I haven't played with eliminating requirements form the test dependencies, and I've been trying to resolve this by modifying the bigbench repo to become compatible.
In the original bigbench repo, the version requirements were strict, and specifically it had a datasets==1.17.0 requirement which was causing trouble.
I'm working on PR https://github.com/google/BIG-bench/pull/766 to get some more flexible versions that might be compatible with the test dependencies in  HF/datasets.
We're somewhat flexible in modifying these version numbers if we can figure out what the exact conflict is.

I've spent some time experimenting with different versions, but I don't have a very efficient way of doing this debugging on my work computer (which for some reason doesn't produce the same sets of errors running python 3.9 instead of 3.6 or 3.7 in the tests).
It currently fails at
> The conflict is caused by:
>    bert-score 0.3.6 depends on matplotlib
>    big-bench 0.0.1 depends on matplotlib<4.0 and >=3.5.1

which doesn't seem like it can be the real issue.

If you have any advice for how to resolve these conflicts, that would be greatly appreciated!

Hi again @lhoestq,
After some more or less random guessing of conflicting packages, I've managed to find a configuration that seems to be compatible with HF/datasets.

The errors went away after removing version limits on matplotlib and scipy, and loosening numpy from 1.19 -> 1.17 in the bigbench requirements.

I might do some more tweaking to see if it lets me set some minimal limits on matplotlib and scipy, but I think we at least can move forward.

The WIN tests are still failing, now because of

> Did not find path entry C:\tools\miniconda3\bin
>C:\tools\miniconda3\envs\py37\python.exe: No module named pytest

I have no way of debugging this locally, and unless there's some way to get more verbose logs, I don't know why it's not finding pytest. Would you be able to take a quick look?

Update: Actually, I see it's still failing because of the long filenames. So perhaps the pytest error is just because the previous steps failed.

One more update on the WIN errors.
I think all the long filenames are in files in the github repo that does not need to be included.
We will try to remove them .

Hi ! The remaining error seems to be a `UnicodeDecodeError` from `setup.py`. I think you can fix your setup.py:
```diff
-     with open(os.path.join(os.path.dirname(__file__), fname)) as f:
+     with open(os.path.join(os.path.dirname(__file__), fname), encoding="utf-8") as f:
```
Indeed on windows, when you `open` a file it doesn't always use "utf-8" by default

Hi @lhoestq,
The dependency issues seems to now be resolved 🎉

Now, the WIN tests are failing at
> ERROR tests/test_arrow_dataset.py::test_dummy_dataset_serialize_s3 - botocore...
> ERROR tests/test_dataset_dict.py::test_dummy_dataset_serialize_s3 - botocore...

Is this testing the dummy dataset that's added in bigbench? If so, I might need some help getting the right format in.

The error message I'm seeing is
> raise EndpointConnectionError(endpoint_url=request.url, error=e)
> E           botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://127.0.0.1:5555/test"

Which seems unrelated, but perhaps the real issue is somewhere I'm not seeing?

Woohoo awesome !

Let me check the CI error

Can you try to re-run the CI, just in case CircleCI messed up ?

Hi @lhoestq,
Rerunning did not seem to solve the problem.
The `test_dummy_dataset_serialize_s3` error still seems to remain.

Hi again @lhoestq,
I'm not sure if this is informative or not in terms of debugging, but I deleted the dummy data and the errors for windows still fail and the others still pass.
Do you have any idea what could be causing this error on windows?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4125). All of your documentation changes will be reflected on that endpoint.

Now the last question: let's have the dataset under`google/bigbench` @andersjohanandreassen ?

I think it would be nicer, this way you and anyone in your team can update the dataset card whevener you want without going through a github PR. You just need to join the https://huggingface.co/google page using your google email :)

This PR adds all BIG-bench json tasks to huggingface/datasets.


Image decoding often fails when  transforming Image datasets

bug

Something isn't working

A quick hack I have found is that we can call the image first before running the transforms and it makes sure the image is decoded before being passed on.

For this I just needed to add `example['img'] = example['img']` to the top of my `generate_flipped_data` function, defined above, so that image decode in invoked.

After this minor change this function works:
```python
def generate_flipped_data(example, p=0.5):
    """
    A Dataset mapping functions that transforms some of the image up-side-down.
    If the probability value (p) is 0.5 approximately half the images will be flipped upside-down
    Args:
        example: An example from the dataset containing a Python dictionary with "img" and "is_flipped" key-value pair
        p: probability of flipping the image up-side-down, Default 0.5

    Returns:
        example: A Dataset object

    """
    example['img'] = example['img']  # <<< This is the only change
    if rng.random() > p:  # the flip the image and set is_flipped column to 1
        example['img'] = example['img'].transpose(
            1)  # ImageOps.flip(example['img'])  #example['img'].transpose(Image.FLIP_TOP_BOTTOM)
        example['is_flipped'] = 1

    return example
```

Hi @RafayAK, thanks for reporting.

Current implementation of the Image feature performs the decoding only if the "img" field is accessed by the mapped function.

In your original `generate_flipped_data` function:
- it only accesses the "img" field (and thus performs decoding) if `rng.random() > p`;
- on the other hand, for the cases where `rng.random() <= p`, the "img" field is not accessed and thus no decoding is performed for those examples

By adding the code line `example['img'] = example['img']`, you make sure the "img" field is accessed in all cases, and the decoding is done for all examples.

Also note that there is a little bug in your implementation: `p` is not the probability of flipping, but the probability of not-flipping; the larger is `p`, the smaller is the probability of flipping.

Some refactoring (fixing also `p`):
```python
def generate_flipped_data(example, p=0.5):
    """
    A Dataset mapping functions that transforms some of the image up-side-down.
    If the probability value (p) is 0.5 approximately half the images will be flipped upside-down.

    Args:
        example: An example from the dataset containing a Python dictionary with "img" and "is_flipped" key-value pair
        p: probability of flipping the image up-side-down, Default 0.5

    Returns:
        example: A Dataset object

    """
    do_flip = rng.random() < p  # Note the "<" sign here instead of ">"
    example['img'] = example['img'].transpose(1) if do_flip else example['img']  # Note "img" is always accessed
    example['is_flipped'] = 1 if do_flip else 0
    return example

@albertvillanova Thanks for letting me know this is intended behavior. The docs are severely lacking on this, if I hadn't posted this here I would have never found out how I'm actually supposed to modify images in a Dataset object.

@albertvillanova Secondly if you check the error message it shows that around 1999 images were successfully created, I'm pretty sure some of them were also flipped during the process. Back to my main contention, sometimes the decoding takes place other times it fails.

I suppose to run `map` on any dataset all the examples should be invoked even if on some of them we end up doing nothing, is that right?

Hi @RafayAK! I've opened a PR with the fix, which adds a fallback to reattempt casting to PyArrow format with a more robust (but more expensive) procedure if the first attempt fails. Feel free to test it by installing `datasets` from the PR branch with the following command:
```
pip install git+https://github.com/huggingface/datasets.git@fix-4124
```

@mariosasko I'll try this right away and report back.

@mariosasko Thanks a lot for looking into this, now the `map` function at least behaves as one would expect a function to behave.

Looking forward to exploring Hugging Face more and even contributing 😃.

```bash
 $ conda list | grep datasets
datasets                  2.0.1.dev0               pypi_0    pypi

```

```python
def preprocess_data(dataset):
    """
    Helper funtion to pre-process HuggingFace Cifar-100 Dataset to remove fine_label and coarse_label columns and
    add is_flipped column
    Args:
        dataset: HuggingFace CIFAR-100 Dataset Object

    Returns:
        new_dataset: A Dataset object with "img" and "is_flipped" columns only

    """
    # remove fine_label and coarse_label columns
    new_dataset = dataset.remove_columns(['fine_label', 'coarse_label'])
    # add the column for is_flipped
    new_dataset = new_dataset.add_column(name="is_flipped", column=np.zeros((len(new_dataset)), dtype=np.uint8))

    return new_dataset

def generate_flipped_data(example, p=0.5):
    """
    A Dataset mapping functions that transforms some of the image up-side-down.
    If the probability value (p) is 0.5 approximately half the images will be flipped upside-down
    Args:
        example: An example from the dataset containing a Python dictionary with "img" and "is_flipped" key-value pair
        p: probability of flipping the image up-side-down, Default 0.5

    Returns:
        example: A Dataset object

    """
    # example['img'] = example['img']
    if rng.random() > p:  # the flip the image and set is_flipped column to 1
        example['img'] = example['img'].transpose(
            1)  # ImageOps.flip(example['img'])  #example['img'].transpose(Image.FLIP_TOP_BOTTOM)
        example['is_flipped'] = 1

    return example

my_test = preprocess_data(test_dataset)
my_test = my_test.map(generate_flipped_data)
```

The output now show the function was applied successfully:
``` bash
/home/rafay/anaconda3/envs/pytorch_new/bin/python /home/rafay/Documents/you_only_live_once/upside_down_detector/create_dataset.py
Downloading builder script: 5.61kB [00:00, 3.16MB/s]
Downloading metadata: 4.21kB [00:00, 2.56MB/s]
Reusing dataset cifar100 (/home/rafay/.cache/huggingface/datasets/cifar100/cifar100/1.0.0/f365c8b725c23e8f0f8d725c3641234d9331cd2f62919d1381d1baa5b3ba3142)
Reusing dataset cifar100 (/home/rafay/.cache/huggingface/datasets/cifar100/cifar100/1.0.0/f365c8b725c23e8f0f8d725c3641234d9331cd2f62919d1381d1baa5b3ba3142)
100%|██████████| 10000/10000 [00:01<00:00, 5149.15ex/s]
```

## Describe the bug
When transforming/modifying images in an image dataset using the `map` function the PIL images often fail to decode in time for the image transforms, causing errors.

Using a debugger it is easy to see what the problem is, the Image decode invocation does not take place and the resulting image passed around is still raw bytes:
```
[{'bytes': b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00 \x00\x00\x00 \x08\x02\x00\x00\x00\xfc\x18\xed\xa3\x00\x00\x08\x02IDATx\x9cEVIs[\xc7\x11\xeemf\xde\x82\x8d\x80\x08\x89"\xb5V\\\xb6\x94(\xe5\x9f\x90\xca5\x7f$\xa7T\xe5\x9f&9\xd9\x8a\\.\xdb\xa4$J\xa4\x00\x02x\xc0{\xb3t\xe7\x00\xca\x99\xd3\\f\xba\xba\xbf\xa5?|\xfa\xf4\xa2\xeb\xba\xedv\xa3f^\xf8\xd5\x0bY\xb6\x10\xb3\xaaDq\xcd\x83\x87\xdf5\xf3gZ\x1a\x04\x0f\xa0fp\xfa\xe0\xd4\x07?\x9dN\xc4\xb1\x99\xfd\xf2\xcb/\x97\x97\x97H\xa2\xaaf\x16\x82\xaf\xeb\xca{\xbf\xd9l.\xdf\x7f\xfa\xcb_\xff&\x88\x08\x00\x80H\xc0\x80@.;\x0f\x8c@#v\xe3\xe5\xfc\xd1\x9f\xee6q\xbf\xdf\xa6\x14\'\x93\xf1\xc3\xe5\xe3\xd1x\x14c\x8c1\xa5\x1c\x9dsM\xd3\xb4\xed\x08\x89SJ)\xa5\xedv\xbb^\xafNO\x97D\x84Hf ....
```

## Steps to reproduce the bug
```python
from datasets import load_dataset, Dataset
import numpy as np
# seeded NumPy random number generator for reprodducinble results.
rng = np.random.default_rng(seed=0)

test_dataset = load_dataset('cifar100', split="test")

def preprocess_data(dataset):
    """
    Helper function to pre-process HuggingFace Cifar-100 Dataset to remove fine_label and coarse_label columns and
    add is_flipped column
    Args:
        dataset: HuggingFace CIFAR-100 Dataset Object

    Returns:
        new_dataset: A Dataset object with "img" and "is_flipped" columns only

    """
    # remove fine_label and coarse_label columns
    new_dataset = dataset.remove_columns(['fine_label', 'coarse_label'])
    # add the column for is_flipped
    new_dataset = new_dataset.add_column(name="is_flipped", column=np.zeros((len(new_dataset)), dtype=np.uint8))

    return new_dataset

def generate_flipped_data(example, p=0.5):
    """
    A Dataset mapping function that transforms some of the images up-side-down.
    If the probability value (p) is 0.5 approximately half the images will be flipped upside-down
    Args:
        example: An example from the dataset containing a Python dictionary with "img" and "is_flipped" key-value pair
        p: the probability of flipping the image up-side-down, Default 0.5

    Returns:
        example: A Dataset object

    """
    # example['img'] = example['img']
    if rng.random() > p:  # the flip the image and set is_flipped column to 1
        example['img'] = example['img'].transpose(
            1)  # ImageOps.flip(example['img'])  #example['img'].transpose(Image.FLIP_TOP_BOTTOM)
        example['is_flipped'] = 1

    return example

my_test = preprocess_data(test_dataset)
my_test = my_test.map(generate_flipped_data)

```

## Expected results
The dataset should be transformed without problems.

## Actual results
```
/home/rafay/anaconda3/envs/pytorch_new/bin/python /home/rafay/Documents/you_only_live_once/upside_down_detector/create_dataset.py
Reusing dataset cifar100 (/home/rafay/.cache/huggingface/datasets/cifar100/cifar100/1.0.0/f365c8b725c23e8f0f8d725c3641234d9331cd2f62919d1381d1baa5b3ba3142)
Reusing dataset cifar100 (/home/rafay/.cache/huggingface/datasets/cifar100/cifar100/1.0.0/f365c8b725c23e8f0f8d725c3641234d9331cd2f62919d1381d1baa5b3ba3142)
 20%|█▉        | 1999/10000 [00:00<00:01, 5560.44ex/s]
Traceback (most recent call last):
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2326, in _map_single
    writer.write(example)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 441, in write
    self.write_examples_on_file()
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 399, in write_examples_on_file
    self.write_batch(batch_examples=batch_examples)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 492, in write_batch
    arrays.append(pa.array(typed_sequence))
  File "pyarrow/array.pxi", line 230, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 185, in __arrow_array__
    out = pa.array(cast_to_python_objects(data, only_1d_for_numpy=True))
  File "pyarrow/array.pxi", line 316, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB size=32x32 at 0x7F56AEE61DE0> with type Image: did not recognize Python value type when inferring an Arrow data type

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rafay/Documents/you_only_live_once/upside_down_detector/create_dataset.py", line 55, in <module>
    my_test = my_test.map(generate_flipped_data)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 1953, in map
    return self._map_single(
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 519, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 486, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/fingerprint.py", line 458, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2360, in _map_single
    writer.finalize()
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 522, in finalize
    self.write_examples_on_file()
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 399, in write_examples_on_file
    self.write_batch(batch_examples=batch_examples)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 492, in write_batch
    arrays.append(pa.array(typed_sequence))
  File "pyarrow/array.pxi", line 230, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 185, in __arrow_array__
    out = pa.array(cast_to_python_objects(data, only_1d_for_numpy=True))
  File "pyarrow/array.pxi", line 316, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB size=32x32 at 0x7F56AEE61DE0> with type Image: did not recognize Python value type when inferring an Arrow data type

Process finished with exit code 1
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Linux(Fedora 35)
- Python version: 3.10
- PyArrow version: 7.0.0


Building C4 takes forever

Something isn't working

Hi @StellaAthena, thanks for reporting.

Please note, that our `datasets` library performs several operations in order to load a dataset, among them:
- it downloads all the required files: for C4 "en", 378.69 GB of JSON GZIPped files
- it parses their content to generate the dataset
- it caches the dataset in an Arrow file: for C4 "en", this file size is 1.87 TB
- it memory-maps the Arrow file

If it suits your use case, you might load this dataset in streaming mode:
- no Arrow file is generated
- you can iterate over elements immediately (no need to wait to download all the entire files)

```python
In [45]: from datasets import load_dataset
    ...: ds = load_dataset("c4", "en", split="train", streaming=True)
    ...: for item in ds:
    ...:     print(item)
    ...:     break
    ...:
{'text': 'Beginners BBQ Class Taking Place in Missoula!\nDo you want to get better at making delicious BBQ? You will have the opportunity, put this on your calendar now. Thursday, September 22nd join World Class BBQ Champion, Tony Balay from Lonestar Smoke Rangers. He will be teaching a beginner level class for everyone who wants to get better with their culinary skills.\nHe will teach you everything you need to know to compete in a KCBS BBQ competition, including techniques, recipes, timelines, meat selection and trimming, plus smoker and fire information.\nThe cost to be in the class is $35 per person, and for spectators it is free. Included in the cost will be either a t-shirt or apron and you will be tasting samples of each meat that is prepared.', 'timestamp': '2019-04-25T12:57:54Z', 'url': 'https://klyq.com/beginners-bbq-class-taking-place-in-missoula/'}
```
I hope this is useful for your use case.

## Describe the bug
C4-en is a 300 GB dataset. However, when I try to download it through the hub it takes over _six hours_ to generate the train/test split from the downloaded files. This is an absurd amount of time and an unnecessary waste of resources.

## Steps to reproduce the bug
```python
c4 = datasets.load("c4", "en")
```

## Expected results
I would like to be able to download pre-split data.

## Environment info
- `datasets` version: 2.0.0
- Platform: Linux-5.13.0-35-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


medical_dialog zh has very slow _generate_examples

Something isn't working

Hi @nbroad1881, thanks for reporting.

Let me have a look to try to improve its performance.

Thanks @nbroad1881  for reporting! I don't recall it taking so long. I will also have a look at this.
@albertvillanova please let me know if I am doing something unnecessary or time consuming.

Hi @nbroad1881 and @vrindaprabhu,

As a workaround for the performance of the parsing of the raw data files (this could be addressed in a subsequent PR), I have found that there are also processed data files, that do not require parsing. I have added these as new configurations `processed.en` and `processed.zh`:
```python
ds = load_dataset("medical_dialog", "processed.zh")
```

## Describe the bug
After downloading the files from Google Drive, `load_dataset("medical_dialog", "zh", data_dir="./")` takes an unreasonable amount of time. Generating the train/test split for 33% of the dataset takes over 4.5 hours.

## Steps to reproduce the bug
The easiest way I've found to download files from Google Drive is to use `gdown` and use Google Colab because the download speeds will be very high due to the fact that they are both in Google Cloud.

```python
file_ids = [
        "1AnKxGEuzjeQsDHHqL3NqI_aplq2hVL_E",
        "1tt7weAT1SZknzRFyLXOT2fizceUUVRXX",
        "1A64VBbsQ_z8wZ2LDox586JIyyO6mIwWc",
        "1AKntx-ECnrxjB07B6BlVZcFRS4YPTB-J",
        "1xUk8AAua_x27bHUr-vNoAuhEAjTxOvsu",
        "1ezKTfe7BgqVN5o-8Vdtr9iAF0IueCSjP",
        "1tA7bSOxR1RRNqZst8cShzhuNHnayUf7c",
        "1pA3bCFA5nZDhsQutqsJcH3d712giFb0S",
        "1pTLFMdN1A3ro-KYghk4w4sMz6aGaMOdU",
        "1dUSnG0nUPq9TEQyHd6ZWvaxO0OpxVjXD",
        "1UfCH05nuWiIPbDZxQzHHGAHyMh8dmPQH",
]
for i in file_ids:
    url = f"https://drive.google.com/uc?id={i}"
    !gdown $url

from datasets import load_dataset

ds = load_dataset("medical_dialog", "zh", data_dir="./")
```

## Expected results
Faster load time

## Actual results
`Generating train split: 33%: 625519/1921127 [4:31:03<31:39:20, 11.37 examples/s]`

## Environment info
- `datasets` version: 2.0.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

@vrindaprabhu , could you take a look at this since you implemented it? I think the `_generate_examples` function might need to be rewritten


datasets.load_metric can not load a local metirc

Something isn't working

## Describe the bug
No matter how I hard try to tell load_metric that I want to load a local metric file, it still continues to fetch things on the Internet. And unfortunately it says 'ConnectionError: Couldn't reach'. However I can download this file without connectionerror and tell load_metric its local directory. And it comes back where it begins...

## Steps to reproduce the bug
```python
metric = load_metric(path=r'C:\Users\Gare\PycharmProjects\Gare\blue\bleu.py')
    ConnectionError: Couldn't reach https://github.com/tensorflow/nmt/raw/master/nmt/scripts/bleu.py

metric = load_metric(path='bleu')
    ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.12.1/metrics/bleu/bleu.py

metric = load_metric(path='./blue/bleu.py')
    ConnectionError: Couldn't reach https://github.com/tensorflow/nmt/raw/master/nmt/scripts/bleu.py
```

## Expected results
I do read the docs [here](https://huggingface.co/docs/datasets/package_reference/loading_methods#datasets.load_metric). There are no other parameters that help function to distinguish from local and online file but path. As what I code above, it should load from local.

## Actual results

> metric = load_metric(path=r'C:\Users\Gare\PycharmProjects\Gare\blue\bleu.py')

> ~\AppData\Local\Temp\ipykernel_19636\1855752034.py in <module>
----> 1 metric = load_metric(path=r'C:\Users\Gare\PycharmProjects\Gare\blue\bleu.py')
D:\Program Files\Anaconda\envs\Gare\lib\site-packages\datasets\load.py in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, script_version, **metric_init_kwargs)
    817         if data_files is None and data_dir is not None:
    818             data_files = os.path.join(data_dir, "**")
--> 819
    820         self.name = name
    821         self.revision = revision
D:\Program Files\Anaconda\envs\Gare\lib\site-packages\datasets\load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, return_associated_base_path, data_files, **download_kwargs)
    639         self,
    640         path: str,
--> 641         download_config: Optional[DownloadConfig] = None,
    642         download_mode: Optional[DownloadMode] = None,
    643         dynamic_modules_path: Optional[str] = None,
D:\Program Files\Anaconda\envs\Gare\lib\site-packages\datasets\utils\file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
    297             token = hf_api.HfFolder.get_token()
    298         if token:
--> 299             headers["authorization"] = f"Bearer {token}"
    300     return headers
    301
D:\Program Files\Anaconda\envs\Gare\lib\site-packages\datasets\utils\file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)
    604             def _resumable_file_manager():
    605                 with open(incomplete_path, "a+b") as f:
--> 606                     yield f
    607
    608             temp_file_manager = _resumable_file_manager
ConnectionError: Couldn't reach https://github.com/tensorflow/nmt/raw/master/nmt/scripts/bleu.py

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Windows-10-10.0.22000-SP0
- Python version: 3.7.13
- PyArrow version: 7.0.0
- Pandas version: 1.3.4

Any advice would be appreciated.


Representing dictionaries (json) objects as features

enhancement

New feature or request

In the process of adding a new dataset to the hub, I stumbled upon the inability to represent dictionaries that contain different key names, unknown in advance (and may differ between samples), original asked in the [forum](https://discuss.huggingface.co/t/representing-nested-dictionary-with-different-keys/16442).

For instance:

```
sample1 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
}}
sample2 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
  "c": {"id": 2, "text": "text3"},
}}
sample3 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
  "c": {"id": 2, "text": "text3"},
  "d": {"id": 3, "text": "text4"},
}}
```

the `nps` field cannot be represented as a Feature while maintaining its original structure.
@lhoestq suggested to add JSON as a new feature type, which will solve this problem.

It seems like an alternative solution would be to change the original data format, which isn't an optimal solution in my case. Moreover, JSON is a common structure, that will likely to be useful in future datasets as well.


Hotfix failing CI tests on Windows

_The documentation is not available anymore as the PR was closed or merged._

This PR makes a hotfix for our CI Windows tests: https://app.circleci.com/pipelines/github/huggingface/datasets/11092/workflows/9cfdb1dd-0fec-4fe0-8122-5f533192ebdc/jobs/67414

Fix #4118

I guess this issue is related to this PR:
- huggingface/huggingface_hub#815


Failing CI tests on Windows

bug

Something isn't working

## Describe the bug
Our CI Windows tests are failing from yesterday: https://app.circleci.com/pipelines/github/huggingface/datasets/11092/workflows/9cfdb1dd-0fec-4fe0-8122-5f533192ebdc/jobs/67414


Something isn't working

Hi @arymbe, thanks for reporting.

Unfortunately, I'm not able to reproduce your problem.

Could you please write the complete stack trace? That way we will be able to see which package originates the exception.

Hello, thank you for your fast replied. this is the complete error that I got

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

Input In [27], in <module>
----> 1 from datasets import load_dataset

venv/lib/python3.8/site-packages/datasets/__init__.py:39, in <module>
     37 from .arrow_dataset import Dataset, concatenate_datasets
     38 from .arrow_reader import ReadInstruction
---> 39 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
     40 from .combine import interleave_datasets
     41 from .dataset_dict import DatasetDict, IterableDatasetDict

venv/lib/python3.8/site-packages/datasets/builder.py:40, in <module>
     32 from .arrow_reader import (
     33     HF_GCP_BASE_URL,
     34     ArrowReader,
   (...)
     37     ReadInstruction,
     38 )
     39 from .arrow_writer import ArrowWriter, BeamWriter
---> 40 from .data_files import DataFilesDict, sanitize_patterns
     41 from .dataset_dict import DatasetDict, IterableDatasetDict
     42 from .features import Features

venv/lib/python3.8/site-packages/datasets/data_files.py:297, in <module>
    292     except FileNotFoundError:
    293         raise FileNotFoundError(f"The directory at {base_path} doesn't contain any data file") from None
    296 def _resolve_single_pattern_in_dataset_repository(
--> 297     dataset_info: huggingface_hub.hf_api.DatasetInfo,
    298     pattern: str,
    299     allowed_extensions: Optional[list] = None,
    300 ) -> List[PurePath]:
    301     data_files_ignore = FILES_TO_IGNORE
    302     fs = HfFileSystem(repo_info=dataset_info)

AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

This is weird... It is long ago that the package `huggingface_hub` has a submodule called `hf_api`.

Maybe you have a problem with your installed `huggingface_hub`...

Could you please try to update it?
```shell
pip install -U huggingface_hub
```

Yap, I've updated several times. Then, I've tried numeral combination of datasets and huggingface_hub versions. However, I think your point is right that there is a problem with my huggingface_hub installation. I'll try another way to find the solution. I'll update it later when I get the solution. Thank you :)

I'm sorry I can't reproduce your problem.

Maybe you could try to create a new Python virtual environment and install all dependencies there from scratch. You can use either:
- Python venv: https://docs.python.org/3/library/venv.html
- or conda venv (if you are using conda): https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

## Describe the bug
Could you help me please. I got this following error.

AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

## Steps to reproduce the bug
when I imported the datasets

# Sample code to reproduce the bug
from datasets import list_datasets, load_dataset, list_metrics, load_metric

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: macOS-12.3-x86_64-i386-64bit
- Python version: 3.8.9
- PyArrow version: 7.0.0
- Pandas version: 1.3.5
- Huggingface-hub: 0.5.0
- Transformers: 4.18.0

Thank you in advance.


Pretty print dataset info files

maybe just do it from now on no? (i.e. not for existing `dataset_infos.json` files)

> maybe just do it from now on no? (i.e. not for existing dataset_infos.json files)

Yes, or do this only for datasets created with `push_to_hub` to (always) keep the GH datasets small?

I reverted the change to avoid the size increase and added the `pretty_print` flag, which pretty-prints the JSON, and that flag is only True for datasets created with `push_to_hub`.

Adds indentation to the `dataset_infos.json` file when saving for nicer diffs.

(suggested by @julien-c)

This PR also updates the info files of the GH datasets. Note that this change adds more than **10 MB** to the repo size (the total file size before the change: 29.672298 MB, after: 41.666475 MB), so I'm not sure this change is a good idea.

`src/datasets/info.py` is the only relevant file for reviewers.


ImageFolder add option to ignore some folders like '.ipynb_checkpoints'

enhancement

New feature or request

Maybe it would be nice to ignore private dirs like this one (ones starting with `.`) by default.

CC @mariosasko

Maybe we can add a `ignore_hidden_files` flag to the builder configs of our packaged loaders (to be consistent across all of them), wdyt @lhoestq @albertvillanova?

I think they should always ignore them actually ! Not sure if adding a flag would be helpful

@lhoestq But what if the user explicitly requests those files via regex?

`glob.glob` ignores hidden files (files starting with ".") by default unless they are explicitly requested, but fsspec's `glob` doesn't follow this behavior, which is probably a bug, so maybe we can raise an issue or open a PR in their repo?

> @lhoestq But what if the user explicitly requests those files via regex?

Usually hidden files are meant to be ignored. If they are data files, they must be placed outside a hidden directory in the first place right ? I think it's more sensible to explain this than adding a flag.

> glob.glob ignores hidden files (files starting with ".") by default unless they are explicitly requested, but fsspec's glob doesn't follow this behavior, which is probably a bug, so maybe we can raise an issue or open a PR in their repo?

After globbing using `fsspec`, we already ignore files that start with a `.` in `_resolve_single_pattern_locally` and `_resolve_single_pattern_in_dataset_repository`, I guess we can just account for parent directories as well ?

We could open an issue on `fsspec` but I think they won't change this since it's an important breaking change for them.

**Is your feature request related to a problem? Please describe.**
I sometimes like to peek at the dataset images from jupyterlab. thus '.ipynb_checkpoints' folder appears where my dataset is and (just realized) leads to accidental duplicate image additions. I think this is an easy enough thing to miss especially if the dataset is very large.

**Describe the solution you'd like**
maybe have an option `ignore` or something .gitignore style
`dataset = load_dataset("imagefolder", data_dir="./data/original", ignore="regex?")`

**Describe alternatives you've considered**
Could filter out manually


Allow downloading just some columns of a dataset

New feature or request

In the general case you can’t always reduce the quantity of data to download, since you can’t parse CSV or JSON data without downloading the whole files right ? ^^ However we could explore this case-by-case I guess

Actually for csv pandas has `usecols` which allows loading a subset of columns in a more efficient way afaik, but yes, you're right this might be more complex than I thought.

**Is your feature request related to a problem? Please describe.**
Some people are interested in doing label analysis of a CV dataset without downloading all the images. Downloading the whole dataset does not always makes sense for this kind of use case

**Describe the solution you'd like**
Be able to just download some columns of a dataset, such as doing
```python
load_dataset("huggan/wikiart",columns=["artist", "genre"])
```

Although this might make things a bit complicated in terms of local caching of datasets.


Multiprocessing with FileLock fails in python 3.9

bug

Something isn't working

On python 3.9, this code hangs:
```python
from multiprocessing import Pool
from filelock import FileLock

def run(i):
    print(f"got the lock in multi process [{i}]")

with FileLock("tmp.lock"):
    with Pool(2) as pool:
        pool.map(run, range(2))

```

This is because the subprocesses try to acquire the lock from the main process for some reason. This is not the case in older versions of python.

This can cause many issues in python 3.9. In particular, we use multiprocessing to fetch data files when you load a dataset (as long as there are >16 data files). Therefore `imagefolder` hangs, and I expect any dataset that needs to download >16 files to hang as well.

Let's see if we can fix this and have a CI that runs on 3.9.

cc @mariosasko @julien-c


ImageFolder with Grayscale images dataset

Hi! Replacing:
```python
transformed_dataset = dataset.with_transform(transforms)
transformed_dataset.set_format(type="torch", device="cuda")
```

with:
```python
def transform_func(examples):
    examples["image"] = [transforms(img).to("cuda") for img in examples["image"]]
    return examples

transformed_dataset = dataset.with_transform(transform_func)
```
should fix the issue. `datasets` doesn't support chaining of transforms (you can think of `set_format`/`with_format` as a predefined transform func for `set_transform`/`with_transforms`), so the last transform (in your case, `set_format`) takes precedence over the previous ones (in your case `with_format`). And the PyTorch formatter is not supported by the Image feature, hence the error (adding support for that is on our short-term roadmap).

Ok thanks a lot for the code snippet!

I love the way `datasets` is easy to use but it made it really long to pre-process all the images (400.000 in my case) before training anything. `ImageFolder` from pytorch is faster in my case but force me to have the images on my local machine.

I don't know how to speed up the process without switching to `ImageFolder` :smile:

You can pass `ignore_verifications=True` in `load_dataset` to skip checksum verification, which takes a lot of time if the number of files is large. We will consider making this the default behavior.

Hi, I'm facing a problem with a grayscale images dataset I have uploaded [here](https://huggingface.co/datasets/ChainYo/rvl-cdip) (RVL-CDIP)

I'm getting an error while I want to use images for training a model with PyTorch DataLoader. Here is the full traceback:

```bash
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1765, in __getitem__
    return self._getitem(
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1750, in _getitem
    formatted_output = format_table(
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 532, in format_table
    return formatter(pa_table, query_type=query_type)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 281, in __call__
    return self.format_row(pa_table)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/torch_formatter.py", line 58, in format_row
    return self.recursive_tensorize(row)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/torch_formatter.py", line 54, in recursive_tensorize
    return map_nested(self._recursive_tensorize, data_struct, map_list=False)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 314, in map_nested
    mapped = [
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 315, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 267, in _single_map_nested
    return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 267, in <dictcomp>
    return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 251, in _single_map_nested
    return function(data_struct)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/torch_formatter.py", line 51, in _recursive_tensorize
    return self._tensorize(data_struct)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/torch_formatter.py", line 38, in _tensorize
    if np.issubdtype(value.dtype, np.integer):
AttributeError: 'bytes' object has no attribute 'dtype'
```

I don't really understand why the image is still a bytes object while I used transformations on it. Here the code I used to upload the dataset (and it worked well):

```python
train_dataset = load_dataset("imagefolder", data_dir="data/train")
train_dataset = train_dataset["train"]
test_dataset = load_dataset("imagefolder", data_dir="data/test")
test_dataset = test_dataset["train"]
val_dataset = load_dataset("imagefolder", data_dir="data/val")
val_dataset = val_dataset["train"]

dataset = DatasetDict({
    "train": train_dataset,
    "val": val_dataset,
    "test": test_dataset
})
dataset.push_to_hub("ChainYo/rvl-cdip")
```

Now here is the code I am using to get the dataset and prepare it for training:

```python
img_size = 512
batch_size = 128
normalize = [(0.5), (0.5)]
data_dir = "ChainYo/rvl-cdip"

dataset = load_dataset(data_dir, split="train")

transforms = transforms.Compose([
        transforms.Resize(img_size),
        transforms.CenterCrop(img_size),
        transforms.ToTensor(),
        transforms.Normalize(*normalize)
])

transformed_dataset = dataset.with_transform(transforms)
transformed_dataset.set_format(type="torch", device="cuda")

train_dataloader = torch.utils.data.DataLoader(
    transformed_dataset, batch_size=batch_size, shuffle=True, num_workers=4, pin_memory=True
)
```

But this get me the error above. I don't understand why it's doing this kind of weird thing?
Do I need to map something on the dataset? Something like this:

```python
labels = dataset.features["label"].names
num_labels = dataset.features["label"].num_classes

def preprocess_data(examples):
    images = [ex.convert("RGB") for ex in examples["image"]]
    labels = [ex for ex in examples["label"]]
    return {"images": images, "labels": labels}

features = Features({
    "images": Image(decode=True, id=None),
    "labels": ClassLabel(num_classes=num_labels, names=labels)
})

decoded_dataset = dataset.map(preprocess_data, remove_columns=dataset.column_names, features=features, batched=True, batch_size=100)
```


Update security policy

_The documentation is not available anymore as the PR was closed or merged._


Matthews Correlation Metric Card


Add Spearmanr Metric Card

changes made! @lhoestq let me know what you think

The CI fail is unrelated to this PR and fixed on master, feel free to merge :)