konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
FileNotFoundError when trying to load the downloaded dataset with `load_dataset(..., streaming=True)`

### Describe the bug

Hi, I'm trying to load the dataset `togethercomputer/RedPajama-Data-1T-Sample` with `load_dataset` in streaming mode, i.e., `streaming=True`, but `FileNotFoundError` occurs.

### Steps to reproduce the bug

I've downloaded the dataset and save it to the cache dir in advance. My hope is loading the files in offline environment and without taking too much hours to prepross the entire data before running into the training process.
So I try the following code to load the files streamingly
```py
dataset = load_dataset('togethercomputer/RedPajama-Data-1T-Sample', streaming=True)
print(next(iter(dataset['train'])))
```
Sadly, it raises the following:
```
FileNotFoundError: [Errno 2] No such file or directory: 'CURRENT_CODE_PATH/arxiv_sample.jsonl'
```
I've noticed that the dataset can be properly found in the begining
```
Using the latest cached version of the module from /root/.cache/huggingface/modules/datasets_modules/datasets/togethercomputer--RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 (last modified on Sat Oct 21 20:12:57 2023) since it couldn't be found locally at togethercomputer/RedPajama-Data-1T-Sample., or remotely on the Hugging Face Hub.
```
But it seems that the paths couldn't be properly parsed when loading iteratively.
 
How should I fix this error. I've tried specifying `data_files` or `data_dir` as `.../arxiv_sample.jsonl` but none of them works.

Thanks.

### Expected behavior

Properly load the dataset.

### Environment info

`datasets==2.14.5`

You can clone the `togethercomputer/RedPajama-Data-1T-Sample` repo and load the dataset with `load_dataset("path/to/cloned_repo")` to use it offline.

@mariosasko Thank you for your kind reply! I'll try it as a workaround.
Does that mean that currently it's not supported to simply load with a short name?

It is, but manually downloading repo files to the cache can easily lead to failure (the HF cache is not meant to be modified by a user besides deleting the files 🙂), as in your case. Hence, the clone + `load_dataset("path/to/cloned_repo")` workflow should be used instead.
Conversion to Arrow fails due to wrong type heuristic

### Describe the bug

I have a list of dictionaries with valid/JSON-serializable values. 

One key is the denominator for a paragraph. In 99.9% of cases its a number, but there are some occurences of '1a', '2b' and so on.

If trying to convert this list to a dataset with `Dataset.from_list()`, I always get
`ArrowInvalid: Could not convert '1' with type str: tried to convert to int64`, presumably because pyarrow tries to convert the keys to integers.

Is there any way to circumvent this and fix dtypes? I didn't find anything in the documentation.

### Steps to reproduce the bug

* create a list of dicts with one key being a string of an integer for the first few thousand occurences and try to convert to dataset.


### Expected behavior

There shouldn't be an error (e.g. some flag to turn off automatic str to numeric conversion).

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.0-84-generic-x86_64-with-glibc2.35
- Python version: 3.9.18
- Huggingface_hub version: 0.17.3
- PyArrow version: 13.0.0
- Pandas version: 2.1.1

Unlike Pandas, Arrow is strict with types, so converting the problematic strings to ints (or ints to strings) to ensure all the values have the same type is the only fix. 

JSON support has been requested in Arrow [here](https://github.com/apache/arrow/issues/32538), but I don't expect this to be implemented soon. 

Also, this type could be represented with the Arrow Union type. However, due to low usage, the Union type has limited support in the Arrow ecosystem (e.g., IIRC Parquet still does not support it). So, we should probably wait a bit more before adding support for it in `datasets`

> Unlike Pandas, Arrow is strict with types, so converting the problematic strings to ints (or ints to strings) to ensure all the values have the same type is the only fix.
> 
> JSON support has been requested in Arrow [here](https://github.com/apache/arrow/issues/32538), but I don't expect this to be implemented soon.
> 
> Also, this type could be represented with the Arrow Union type. However, due to low usage, the Union type has limited support in the Arrow ecosystem (e.g., IIRC Parquet still does not support it). So, we should probably wait a bit more before adding support for it in `datasets`

Ok many thanks, I was able to mitigate the problem by manually checking and converting all problematic fields now.
Loading dataset from large GCS bucket very slow since 2.14

### Describe the bug

Since updating to >2.14 we have very slow access to our parquet files on GCS when loading a dataset (>30 min vs 3s). Our GCS bucket has many objects and resolving globs is very slow. I could track down the problem to this change:
https://github.com/huggingface/datasets/blame/bade7af74437347a760830466eb74f7a8ce0d799/src/datasets/data_files.py#L348 
The underlying implementation with gcsfs is really slow. Could you go back to the old way if we are simply giving the parquet files and no glob pattern?

Thank you.




### Steps to reproduce the bug

Load a dataset from a GCS bucket that has many files.

### Expected behavior

Used to be fast (3s) in 2.13

### Environment info

datasets==2.14.5

I've also encountered this issue recently and want to ask if this has been seen.

@albertvillanova for visibility - I'm not sure who the right person is to tag, but I noticed you were active recently so perhaps you can direct this to the right person.

Thanks!
Dataset slice splits can't load training and validation at the same time

### Describe the bug

According to the [documentation](https://huggingface.co/docs/datasets/v2.14.5/loading#slice-splits) is should be possible to run the following command:

`train_test_ds = datasets.load_dataset("bookcorpus", split="train+test")`

to load the train and test sets from the dataset. 

However executing the equivalent code:

`speech_commands_v1 = load_dataset("superb", "ks", split="train+test")`

only yields the following output:

> Dataset({
>     features: ['file', 'audio', 'label'],
>     num_rows: 54175
> })

Where loading the dataset without the split argument yields:

> DatasetDict({
>     train: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 51094
>     })
>     validation: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 6798
>     })
>     test: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 3081
>     })
> })

Thus, the API seems to be broken in this regard. 

This is a bit annoying since I want to be able to use the split argument with `split="train[:10%]+test[:10%]"` to have smaller dataset to work with when validating my model is working correctly. 

### Steps to reproduce the bug

`speech_commands_v1 = load_dataset("superb", "ks", split="train+test")`

### Expected behavior

> DatasetDict({
>     train: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 51094
>     })
>     test: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 3081
>     })
> })

### Environment info

```
import datasets
print(datasets.__version__)
```
> 2.14.5

```
import sys
print(sys.version)
```
> 3.9.17 (main, Jul  5 2023, 20:41:20) 
> [GCC 11.2.0]

The expression "train+test" concatenates the splits.

The individual splits as separate datasets can be obtained as follows:
```python
train_ds, test_ds = load_dataset("<dataset_name>", split=["train", "test"])
train_10pct_ds, test_10pct_ds = load_dataset("<dataset_name>", split=["train[:10%]", "test[:%10]"])
```
sentiment140 dataset unavailable

### Describe the bug

loading the dataset using load_dataset("sentiment140") returns the following error
ConnectionError: Couldn't reach http://cs.stanford.edu/people/alecmgo/trainingandtestdata.zip (error 403)

### Steps to reproduce the bug

Run the following code (version should not matter).

```
from datasets import load_dataset
data = load_dataset("sentiment140")
```

### Expected behavior

The dataset should be loaded just like any other.
The main issue is that it is no longer hosted by stanford. It is still available from a [Google Drive Link](https://docs.google.com/file/d/0B04GJPshIjmPRnZManQwWEdTZjg/edit).

### Environment info

- `datasets` version: 2.14.5
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.8
- Huggingface_hub version: 0.17.3    
- PyArrow version: 13.0.0
- Pandas version: 2.1.1

Thanks for reporting. We are investigating the issue.

We have opened an issue in the corresponding Hub dataset: https://huggingface.co/datasets/sentiment140/discussions/3

Let's continue the discussion there.
Hub datasets with CSV metadata raise ArrowInvalid: JSON parse error: Invalid value. in row 0

When trying to load a Hub dataset that contains a CSV metadata file, it raises an `ArrowInvalid` error:
```
E   pyarrow.lib.ArrowInvalid: JSON parse error: Invalid value. in row 0

pyarrow/error.pxi:100: ArrowInvalid
```

See: https://huggingface.co/datasets/lukarape/public_small_papers/discussions/1
Add return_file_name in load_dataset

Proposition to fix #5806.

Added an optional parameter `return_file_name` in the dataset builder config. When set to `True`, the function will include the file name corresponding to the sample in the returned output.

There is a difference between arrow-based and folder-based datasets to return the file name:
- for arrow-based: a column is concatenated after the table is cast.
- for folder-based: `dataset.info.features` has the entry `file_name` and the original file name is passed to the `sample_metadata` dictionary. 

The difference in behavior might be a concern, also I do not know whether the `file_name` should return the original file path or the downloaded one for folder-based datasets.

I added some tests for the datasets that already had a test file.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6310). All of your documentation changes will be reflected on that endpoint.

> Thanks for the change !
> 
> Since `return` in python often refers to what is actually returned by the function (here `load_dataset`), I think we can use another word for the parameter. Maybe name it `with_file_names`?
> 
> cc @mariosasko in case you have an opinion

I changed the argument name to your suggestion, I agree that it should be less confusing :)

> Thanks! I've left some comments.
> 
> @lhoestq WDYT about returning a data file's name (the last part) instead of the full path? This way we could have the same values in the streaming and the non-streaming mode. (In the non-streaming mode, we would also have to iterate over remote files to not output the files' hash (from the HF cache))

Concerning the last part of the file name, do you have suggestions on how to do that? Because it can happen that the files are located in different folders with the same name so I am wondering what would be the way to go.

Hi @lhoestq @mariosasko, I just pushed a version rebased on main I've seen some activity on the issue associated to the PR so I wonder if you are interested in keeping up the process of merging this PR. I still have the same issues as indicated in the previous comment.

The original issue https://github.com/huggingface/datasets/issues/5806 is about returning the file names in order to be able to resume data loading from a checkpoint.

Data loading checkpointing has been implemented recently actually, using `torchdata` `StatefulDataloader`

See the discussion at https://github.com/huggingface/datasets/issues/5454

I'm not sure if it's worth continuing this PR then ?

> The original issue #5806 is about returning the file names in order to be able to resume data loading from a checkpoint.
> 
> Data loading checkpointing has been implemented recently actually, using `torchdata` `StatefulDataloader`
> 
> See the discussion at #5454
> 
> I'm not sure if it's worth continuing this PR then ?

Thanks for your answer, I'll close the PR if now the feature already exists  :).

I need this for downstream filter. People always put important information in the file name.
module 'resource' has no attribute 'error'

### Describe the bug

just run import:
`from datasets import load_dataset`

and then:
```

  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\__init__.py", line 22, in <module>
    from .arrow_dataset import Dataset
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\arrow_dataset.py", line 66, in <module>
    from .arrow_reader import ArrowReader
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\arrow_reader.py", line 30, in <module>
    from .download.download_config import DownloadConfig
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\download\__init__.py", line 10, in <module>
    from .streaming_download_manager import StreamingDownloadManager
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\download\streaming_download_manager.py", line 21, in <module>
    from ..filesystems import COMPRESSION_FILESYSTEMS
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\filesystems\__init__.py", line 8, in <module>
    import fsspec.asyn
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\fsspec\asyn.py", line 157, in <module>
    ResourceEror = resource.error
AttributeError: module 'resource' has no attribute 'error'

Process finished with exit code 1

```


and the error codes are:
```
try:
    import resource
except ImportError:
    resource = None
    ResourceError = OSError
else:
    ResourceEror = resource.error

```
1. miss spelling : "ResourceEror " should be "ResourceErorr"
2. module 'resource' has no attribute 'error'




### Steps to reproduce the bug

only one step:

`from datasets import load_dataset`

### Expected behavior

slove error: module 'resource' has no attribute 'error'

### Environment info

python=3.10

datasets==2.14.5


This (Windows) issue was fixed in `fsspec` in https://github.com/fsspec/filesystem_spec/pull/1275.  So, to avoid the error, update the `fsspec` installation with `pip install -U fsspec`.

> This (Windows) issue was fixed in `fsspec` in [fsspec/filesystem_spec#1275](https://github.com/fsspec/filesystem_spec/pull/1275). So, to avoid the error, update the `fsspec` installation with `pip install -U fsspec`.

after I run `pip install -U fsspec`

it occurs a new error:
```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflict
s.
datasets 2.14.5 requires fsspec[http]<2023.9.0,>=2023.1.0, but you have fsspec 2023.9.2 which is incompatible.

```

The `fsspec<2023.9.0` upper bound will be removed in the next release. The `ResourceError` fix is also present in version 2023.6.0, so use that version in the meantime (`pip install fsspec==2023.6.0`).

> The `fsspec<2023.9.0` upper bound will be removed in the next release. The `ResourceError` fix is also present in version 2023.6.0, so use that version in the meantime (`pip install fsspec==2023.6.0`).

thanks for reply！
Parquet uploads off-by-one naming scheme

### Describe the bug

I noticed this numbering scheme not matching up in a different project and wanted to raise it as an issue for discussion, what is the actual proper way to have these stored?

<img width="425" alt="image" src="https://github.com/huggingface/datasets/assets/1981179/3ffa2144-7c9a-446f-b521-a5e9db71e7ce">

The `-SSSSS-of-NNNNN` seems to be used widely across the codebase. The section that creates the part in my screenshot is here https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L5287
There are also some edits to this section in the single commit branch.

### Steps to reproduce the bug

1. Upload a dataset that requires at least two parquet files in it
2. Observe the naming scheme

### Expected behavior

The couple options here are of course **1. keeping it as is**

**2. Starting the index at 1:**
train-00001-of-00002-{hash}.parquet
train-00002-of-00002-{hash}.parquet

**3. My preferred option** (which would solve my specific issue), dropping the total entirely:
train-00000-{hash}.parquet
train-00001-{hash}.parquet

This also solves an issue that will occur with an `append` variable for `push_to_hub` (see https://github.com/huggingface/datasets/issues/6290) where as you add a new parquet file, you need to rename everything in the repo as well. 

However, I know there are parts of the repo that use 0 as the starting file or may require the total, so raising the question for discussion.


### Environment info

- `datasets` version: 2.14.6.dev0
- Platform: macOS-14.0-arm64-arm-64bit
- Python version: 3.10.12
- Huggingface_hub version: 0.18.0
- PyArrow version: 12.0.1
- Pandas version: 1.5.3

You can find the reasoning behind this naming scheme [here](https://github.com/huggingface/transformers/pull/16343#discussion_r931182168).

This point has been raised several times, so I'd be okay with starting with `00001-` (also to be consistent with the `transformers` sharding), but I'm not sure @lhoestq agrees.

We start at 0 in `datasets` for consistency with Apache Spark, Apache Beam, Dask and others.

Also note `transformers` isn't a good reference on this topic. I talked with the maintainers when they added shards but it was already released this way. Though we found that there is a backward-compatible way in `transformers` to start at 0, but no request from `transformers` users to changes this AFAIK.

not sure it would be a good idea to break the consistency now, IMO

Makes sense to start at 0 for plenty of good reasons so I'm on board.

What about the second part `-of-0000X`? With single commit PR #6269  just getting merged, there was a note about issues with 100+ file edits https://github.com/huggingface/datasets/pull/6269#issuecomment-1755428581.

That would be my last remaining concern in the context of the `push_to_hub(..., append=True)` work to be done, where appending a single file to the full dataset will require renaming every other existing file in the dataset. If it doesn't seem like a big issue for this work then all the better 👍
ArrowWriter/ParquetWriter `write` method does not increase `_num_bytes` and hence datasets not sharding at `max_shard_size`

### Describe the bug

An example from [1], does not work when limiting shards with `max_shard_size`.

Try the following example with low `max_shard_size`, such as:

```python
builder.download_and_prepare(output_dir, storage_options=storage_options, file_format="parquet", max_shard_size="10MB")
```

The reason for this is that, in line [2] `writer._num_bytes > max_shard_size` is never true, because the `write` method of `ArrowWriter` [3] does not increase `self._num_bytes`.

Such that respective Arrow/Parquet shards are only written to file based on the `writer_batch_size` or `config.DEFAULT_MAX_BATCH_SIZE`, but not based on `max_shard_size`.


[1] https://huggingface.co/docs/datasets/filesystems#download-and-prepare-a-dataset-into-a-cloud-storage

[2] https://github.com/huggingface/datasets/blob/3e8d420808718c9a1453a2e7ee3484ca12c9c70d/src/datasets/builder.py#L1677

[3] https://github.com/huggingface/datasets/blob/3e8d420808718c9a1453a2e7ee3484ca12c9c70d/src/datasets/arrow_writer.py#L459

### Steps to reproduce the bug

Get example from: https://huggingface.co/docs/datasets/filesystems#download-and-prepare-a-dataset-into-a-cloud-storage

Call `builder.download_and_prepare` with low `max_shard_size` such as `10MB`, e.g.:
```python
builder.download_and_prepare(output_dir, storage_options=storage_options, file_format="parquet", max_shard_size="10MB")
```

### Expected behavior

Shards should be written based on `max_shard_size` instead of batch size.

### Environment info

```
>>> import datasets
>>> datasets.__version__
'2.14.6.dev0
```

`writer._num_bytes` is updated every `writer_batch_size`-th call to the `write` method (default `writer_batch_size` is 1000 (examples)). You should be able to see the update by passing a smaller `writer_batch_size` to the `load_dataset_builder`.

We could improve this by supporting the string `writer_batch_size` version as we do with `max_shard_size`, and capping `writer_batch_size` to `max_shard_size` in scenarios where the default `writer_batch_size` > `max_shard_size`. 

Thanks, reducing `writer_batch_size` solved my problem :)
Support for newer versions of JAX

### Feature request

Hi,

I like your idea of adapting the datasets library to be usable with JAX. Thank you for that.

However, in your [setup.py](https://github.com/huggingface/datasets/blob/main/setup.py), you enforce old versions of JAX <= 0.3... It is very cumbersome !

What is the rationale for such a limitation ? Can you remove it please ?

Thanks,

### Motivation

This library is unusable with new versions of JAX ?

### Your contribution

Yes.
Choose columns to stream parquet data in streaming mode

Currently passing columns= to load_dataset in streaming mode fails

```
Tried to load parquet data with columns '['link']' with mismatching features '{'caption': Value(dtype='string', id=None), 'image': {'bytes': Value(dtype='binary', id=None), 'path': Value(dtype='null', id=None)}, 'link': Value(dtype='string', id=None), 'message_id': Value(dtype='string', id=None), 'timestamp': Value(dtype='string', id=None)}'
```

similar to https://github.com/huggingface/datasets/issues/6039

reported at https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/3#65259a09617407d4520f4ad9
how to load the image of dtype float32 or float64

_FEATURES = datasets.Features(
    {
        "image": datasets.Image(),
        "text": datasets.Value("string"),
    },
)
The datasets builder seems only support the unit8 data. How to load the float dtype data? 

Hi! Can you provide a code that reproduces the issue?

Also, which version of `datasets` are you using? You can check this by running `python -c "import datasets; print(datasets.__version__)"` inside the env. We added support for "float images" in `datasets 2.9`.
Incremental dataset (e.g. `.push_to_hub(..., append=True)`)

### Feature request

Have the possibility to do `ds.push_to_hub(..., append=True)`.

### Motivation

Requested in this [comment](https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/3#65252597c4edc168202a5eaa) and 
this [comment](https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/4#6524f675c9607bdffb208d8f). Discussed internally on [slack](https://huggingface.slack.com/archives/C02EMARJ65P/p1696950642610639?thread_ts=1690554266.830949&cid=C02EMARJ65P).


### Your contribution

What I suggest to do for parquet datasets is to use `CommitOperationCopy` + `CommitOperationDelete` from `huggingface_hub`:
1. list files
2. copy files from parquet-0001-of-0004 to parquet-0001-of-0005
3. delete files like parquet-0001-of-0004
4. generate + add last parquet file parquet-0005-of-0005

=> make a single commit with all commit operations at once


I think it should be quite straightforward to implement. Happy to review a PR (maybe conflicting with the ongoing "1 commit push_to_hub" PR https://github.com/huggingface/datasets/pull/6269)

Yea I think waiting for #6269 would be best, or branching from it. For reference, this [PR](https://github.com/LAION-AI/Discord-Scrapers/pull/2) is progressing pretty well which will do similar using the hf hub for our LAION dataset bot https://github.com/LAION-AI/Discord-Scrapers/pull/2. 

Is there any update on this?

Is there any update on this?

No update so far on this feature request but for broader context, this announce will help with incremental datasets https://huggingface.co/blog/xethub-joins-hf :)

Still no update? Whats the current recommended way to upload large datasets to the hub? I can't load it all to memory after some limit right

> Still no update? Whats the current recommended way to upload large datasets to the hub? I can't load it all to memory after some limit right

@tolgadur bro, you may set 'data_dir' as a temporary solution like that:

`Dataset.push_to_hub('your_repo_aame', data_dir='data_{index}')`
Dataset.from_pandas with a DataFrame of PIL.Images

Currently type inference doesn't know what to do with a Pandas Series of PIL.Image objects, though it would be nice to get a Dataset with the Image type this way

A duplicate of https://github.com/huggingface/datasets/issues/4796.

We could get this for free by implementing the `Image` feature as an extension type, as shown in [this](https://colab.research.google.com/drive/1Uzm_tXVpGTwbzleDConWcNjacwO1yxE4?usp=sharing) Colab (example with UUIDs).


+1 to this
Calling this line with a df that contains a PIL image (as they are returned from load_dataset)
`ds = Dataset.from_pandas(df)`
Results in this error:
`ArrowInvalid: ('Could not convert <PIL.PngImagePlugin.PngImageFile image mode=RGB size=1024x1024 at 0x2B41F2D70> with type PngImageFile: did not recognize Python value type when inferring an Arrow data type', 'Conversion failed for column image with type object')`

I found something that can be used as solution.

I have the same problem when I've try to load the images from a pamdas dataset

If you have all on a pandas dataset try 
Dataset.from_dict( your_df.reset_index(drop=True).to_dict(orient='list'), split=set_your_split)

And this avoid the error
map() not recognizing "text"

### Describe the bug

The [map() documentation](https://huggingface.co/docs/datasets/v2.14.5/en/package_reference/main_classes#datasets.Dataset.map) reads:
`
ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True)`

I have been trying to reproduce it in my code as:

`tokenizedDataset = dataset.map(lambda x: tokenizer(x['text']), batched=True)`

But it doesn't work as it throws the error:

> KeyError: 'text'

Can you please guide me on how to fix it?



### Steps to reproduce the bug

1. `from datasets import load_dataset

dataset = load_dataset("amazon_reviews_multi")`

2. Then this code: `from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")`
3. The line I quoted above (which I have been trying)

### Expected behavior

As mentioned in the documentation, it should run without any error and map the tokenization on the whole dataset.

### Environment info

Python 3.10.2

There is no "text" column in the `amazon_reviews_multi`, hence the `KeyError`. You can get the column names by running `dataset.column_names`.
Add Belebele multiple-choice machine reading comprehension (MRC) dataset

### Feature request

Belebele is a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. This dataset enables the evaluation of mono- and multi-lingual models in high-, medium-, and low-resource languages. Each question has four multiple-choice answers and is linked to a short passage from the [FLORES-200](https://github.com/facebookresearch/flores/tree/main/flores200) dataset. The human annotation procedure was carefully curated to create questions that discriminate between different levels of generalizable language comprehension and is reinforced by extensive quality checks. While all questions directly relate to the passage, the English dataset on its own proves difficult enough to challenge state-of-the-art language models. Being fully parallel, this dataset enables direct comparison of model performance across all languages. Belebele opens up new avenues for evaluating and analyzing the multilingual abilities of language models and NLP systems.

Please refer to paper for more details, [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://arxiv.org/abs/2308.16884).


## Composition

- 900 questions per language variant
- 488 distinct passages, there are 1-2 associated questions for each.
- For each question, there is 4 multiple-choice answers, exactly 1 of which is correct.
- 122 language/language variants (including English).
- 900 x 122 = 109,800 total questions.


### Motivation

official repo https://github.com/facebookresearch/belebele

### Your contribution

-

This dataset is already available on the Hub: https://huggingface.co/datasets/facebook/belebele.
FileNotFoundError: Couldn't find a module script at /content/paws-x/paws-x.py. Module 'paws-x' doesn't exist on the Hugging Face Hub either.

### Describe the bug

I'm encountering a "FileNotFoundError" while attempting to use the "paws-x" dataset to retrain the DistilRoBERTa-base model. The error message is as follows: 

FileNotFoundError: Couldn't find a module script at /content/paws-x/paws-x.py. Module 'paws-x' doesn't exist on the Hugging Face Hub either.

### Steps to reproduce the bug

https://colab.research.google.com/drive/11xUUFxloClpmqLvDy_Xxfmo3oUzjY5nx#scrollTo=kUn74FigzhHm

### Expected behavior

The the trained model

### Environment info

colab, "paws-x" dataset , DistilRoBERTa-base model

`evaluate.load("paws-x", "es")` throws the error because there is no such metric in the `evaluate` lib.

So, this is unrelated to our lib.
Would like to Contribute a dataset

I have a dataset of 2500 images that can be used for color-blind machine-learning algorithms. Since , there was no dataset available online , I made this dataset myself and would like to contribute this now to community

Hi! The process of contributing a dataset is explained here: https://huggingface.co/docs/datasets/upload_dataset. Also, check https://huggingface.co/docs/datasets/image_dataset for a more detailed explanation of how to share an image dataset.
FileNotFoundError for dataset with multiple builder config

### Describe the bug

When there is only one config and only the dataset name is entered when using datasets.load_dataset(), it works fine. But if I create a second builder_config for my dataset and enter the config name when using datasets.load_dataset(), the following error will happen.

FileNotFoundError: [Errno 2] No such file or directory: 'C:/Users/chenx/.cache/huggingface/datasets/my_dataset/0_shot_multiple_choice/1.0.0/97c3854a012cfd6b045e3be4c864739902af2d818bb9235b047baa94c302e9a2.incomplete/my_dataset-test-00000-00000-of-NNNNN.arrow'

The "XXX.incomplete folder" in the cache folder of my dataset will disappear before "generating test split", which does not happen when config name is not entered and the config name is "default"

C:\Users\chenx\.cache\huggingface\datasets\my_dataset\0_shot_multiple_choice\1.0.0    
The folder that is supposed to remain under the above directory will disappear, and the data generator will not have a place to generate data into. 

### Steps to reproduce the bug

test = load_dataset('my_dataset', '0_shot_multiple_choice')

### Expected behavior

FileNotFoundError: [Errno 2] No such file or directory: 'C:/Users/chenx/.cache/huggingface/datasets/my_dataset/0_shot_multiple_choice/1.0.0/97c3854a012cfd6b045e3be4c864739902af2d818bb9235b047baa94c302e9a2.incomplete/my_dataset-test-00000-00000-of-NNNNN.arrow'

### Environment info

datasets                  2.14.5
python                    3.8.18

Please tell me if the above info is not enough for solving the problem. I will then make my dataset public temporarily so that you can really reproduce the bug. 

Hi! 
Could you share how to solve this problem? 
I faced this same error.
Broken Link to PubMed Abstracts dataset .

### Describe the bug

The link provided for the dataset is broken,
data_files = 
[https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst](url)

The 

### Steps to reproduce the bug

Steps to reproduce:

1) Head over to [https://huggingface.co/learn/nlp-course/chapter5/4?fw=pt#big-data-datasets-to-the-rescue](url)

2) In the Section  "What is the Pile?", you can see a code snippet that contains the broken link.

### Expected behavior

The link should Redirect to the  "PubMed Abstracts dataset" as expected .

### Environment info

.

This has already been reported in the HF Course repo (https://github.com/huggingface/course/issues/623).

@lhoestq @albertvillanova @lewtun I don't think we are allowed to host these data files on the Hub (due to DMCA), which means the only option is to use a different dataset in the course (and to re-record the video 🙂), no?

Keeping the video is maybe fine, we can add a note on youtube to suggest to load a dataset with a different name. Maybe C4 ? And update the code snippets on the website ?

Maybe you want to try it with the PUBMED dataset that I reproduced based on the The [PubMed Abstract GitHub Site](http://github.com/thoppe/The-Pile-PubMed) and uploaded on the HuggingFace:

```
from datasets import load_dataset
pubmed_dataset = load_dataset("hwang2006/PUBMED_title_abstracts_2020_baseline")
pubmed_dataset

#Downloading data: 100%
#7.98G/7.98G [11:47<00:00, 9.68MB/s]
#Generating train split: 17722096/0 [00:36<00:00, 505376.37 examples/s]

#DatasetDict({
#   train: Dataset({
#        features: ['meta', 'text'],
#        num_rows: 17722096
#    })
#})
```

孔令涛说感谢感谢
Duplicate `data_files` when named `<split>/<split>.parquet`

e.g. with `u23429/stock_1_minute_ticker`

```ipython
In [1]: from datasets import *

In [2]: b = load_dataset_builder("u23429/stock_1_minute_ticker")
Downloading readme: 100%|██████████████████████████| 627/627 [00:00<00:00, 246kB/s]

In [3]: b.config.data_files
Out[3]: 
{NamedSplit('train'): ['hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/train/train.parquet',
  'hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/train/train.parquet'],
 NamedSplit('validation'): ['hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/validation/validation.parquet',
  'hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/validation/validation.parquet'],
 NamedSplit('test'): ['hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/test/test.parquet',
  'hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/test/test.parquet']}

```

This bug issue is present in the current `datasets` 2.14.5 and also on `main` even after https://github.com/huggingface/datasets/pull/6244 cc @mariosasko 

Also reported in https://github.com/huggingface/datasets/issues/6259

I think it's best to drop duplicates with a `set`  (as a temporary fix) and improve the patterns when/if https://github.com/fsspec/filesystem_spec/pull/1382 gets merged. @lhoestq Do you have some other ideas?

Alternatively we could just use this no ?

```python
if config.FSSPEC_VERSION < version.parse("2023.9.0"):
    KEYWORDS_IN_PATH_NAME_BASE_PATTERNS = [
        "{keyword}[{sep}/]**",
        "**[{sep}]{keyword}[{sep}/]**",
        "**/{keyword}[{sep}/]**",
    ]
else:
    KEYWORDS_IN_PATH_NAME_BASE_PATTERNS = [
        "{keyword}[{sep}/]**",
        "**/*[{sep}]{keyword}[{sep}/]**",
        "**/*/{keyword}[{sep}/]**",
    ]
```

This way no need to implement sets, which would require a bit of work since we've always considered a list of pattern to be resolved as the concatenated list of resolved files for each pattern (including duplicates)


Arf `"**/*/{keyword}[{sep}/]**"` does return `data/keyword.txt` in latest `fsspec` but not in `glob.glob`

EDIT: actually forgot to set `recursive=True`

Actually `glob.glob` does return it with `recursive=True` ! my bad

Pff just tested and my idea sucks, pattern 1 and 3 obviously give duplicates 

> I think it's best to drop duplicates with a set (as a temporary fix)

I started https://github.com/huggingface/datasets/pull/6278 to use DataFilesSet objects instead of DataFilesList
Use LibYAML with PyYAML if available

PyYAML, the YAML framework used in this library, allows the use of LibYAML to accelerate the methods `load` and `dump`. To use it, a user would need to first install a PyYAML version that uses LibYAML (not available in PyPI; needs to be manually installed). Then, to actually use them, PyYAML suggests importing the LibYAML version of the `Loader` and `Dumper` and falling back to the default ones. This PR implements this change. See [PyYAML docs](https://pyyaml.org/wiki/PyYAMLDocumentation) for more info.

This change was motivated after trying to use any of [the SugarCREPE datasets in the Hub](https://huggingface.co/datasets?search=sugarcrepe) provided by [the org HuggingFaceM4](https://huggingface.co/datasets/HuggingFaceM4). Such datasets save a lot of information (~1MB) in the YAML metadata from the `README.md` file and I noticed this slowed down the data loading process. BTW, I also noticed cache files for it is also slow because it tries to hash an instance of `DatasetInfo`, which in turn has all this metadata.

Also, I changed two list comprehensions into generator expressions to avoid allocating extra memory unnecessarily.

And BTW, there's [an issue in PyYAML suggesting to make this automatic](https://github.com/yaml/pyyaml/issues/437).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6266). All of your documentation changes will be reflected on that endpoint.

On Ubuntu, if `libyaml-dev` is installed, you can install PyYAML 6.0.1 with LibYAML with the following command (as it's automatically detected):

```bash
pip install git+https://github.com/yaml/pyyaml.git@6.0.1
```

Are the failing tests flaky?

We use `huggingface_hub`'s RepoCard API instead of these modules to parse the YAML block (notice the deprecations), so the `huggingface_hub` repo is the right place to suggest these changes.

Personally, I'm not a fan of these changes, as a single non-standard usage of the `ClassLabel` type is not a sufficient reason to merge them. Also, the dataset in question stores data in a single Parquet file, with the features info embedded in its (schema) metadata, which means the YAML parsing can be skipped while preserving the features by directly loading the Parquet file:
```python
from datasets import load_dataset
ds = load_dataset("parquet", data_files="https://huggingface.co/datasets/HuggingFaceM4/SugarCrepe_swap_obj/resolve/main/data/test-00000-of-00001-ca2ae6017a2336d7.parquet")
```

PS: Yes, these tests are flaky. We are working on fixing them.

Oh, I didn't realize they were deprecated. Thanks for the tip on how to work around this issue!

For future reference, the places to change the code in `huggingface_hub` would be:

https://github.com/huggingface/huggingface_hub/blob/89cc69105074f1d071e0471144605f3cdfe1dab3/src/huggingface_hub/repocard.py#L506

https://github.com/huggingface/huggingface_hub/blob/89cc69105074f1d071e0471144605f3cdfe1dab3/src/huggingface_hub/utils/_fixes.py#L34
CI is broken: ImportError: cannot import name 'context' from 'tensorflow.python'

Python 3.10 CI is broken for `test_py310`.

See: https://github.com/huggingface/datasets/actions/runs/6322990957/job/17169678812?pr=6262

```
FAILED tests/test_py_utils.py::TempSeedTest::test_tensorflow - ImportError: cannot import name 'context' from 'tensorflow.python' (/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/tensorflow/python/__init__.py)
```

```
_________________________ TempSeedTest.test_tensorflow _________________________
[gw1] linux -- Python 3.10.13 /opt/hostedtoolcache/Python/3.10.13/x64/bin/python

self = <tests.test_py_utils.TempSeedTest testMethod=test_tensorflow>

    @require_tf
    def test_tensorflow(self):
        import tensorflow as tf
        from tensorflow.keras import layers
    
        model = layers.Dense(2)
    
        def gen_random_output():
            x = tf.random.uniform((1, 3))
            return model(x).numpy()
    
>       with temp_seed(42, set_tensorflow=True):

tests/test_py_utils.py:155: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/contextlib.py:135: in __enter__
    return next(self.gen)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

seed = 42, set_pytorch = False, set_tensorflow = True

    @contextmanager
    def temp_seed(seed: int, set_pytorch=False, set_tensorflow=False):
        """Temporarily set the random seed. This works for python numpy, pytorch and tensorflow."""
        np_state = np.random.get_state()
        np.random.seed(seed)
    
        if set_pytorch and config.TORCH_AVAILABLE:
            import torch
    
            torch_state = torch.random.get_rng_state()
            torch.random.manual_seed(seed)
    
            if torch.cuda.is_available():
                torch_cuda_states = torch.cuda.get_rng_state_all()
                torch.cuda.manual_seed_all(seed)
    
        if set_tensorflow and config.TF_AVAILABLE:
            import tensorflow as tf
>           from tensorflow.python import context as tfpycontext
E           ImportError: cannot import name 'context' from 'tensorflow.python' (/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/tensorflow/python/__init__.py)

/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/datasets/utils/py_utils.py:257: ImportError
```
REUSE_DATASET_IF_EXISTS don't work 

### Describe the bug

I use the following code  to download natural_question dataset. Even though I have completely download it, the next time I run this code, the new download procedure will start and cover the original /data/lxy/NQ

config=datasets.DownloadConfig(resume_download=True,max_retries=100,cache_dir=r'/data/lxy/NQ',download_desc='NQ')

data=datasets.load_dataset('natural_questions',cache_dir=r'/data/lxy/NQ',download_config=config,download_mode=DownloadMode.REUSE_DATASET_IF_EXISTS)

---
Since I don't have apache_beam installed, it throw a exception. After I pip install apache_beam ,the download restart..

![image](https://github.com/huggingface/datasets/assets/88258534/f28ce7fe-29ea-4348-b87f-e69182a8bd41)


### Steps to reproduce the bug

run this two line code

config=datasets.DownloadConfig(resume_download=True,max_retries=100,cache_dir=r'/data/lxy/NQ',download_desc='NQ')

data=datasets.load_dataset('natural_questions',cache_dir=r'/data/lxy/NQ',download_config=config,download_mode=DownloadMode.REUSE_DATASET_IF_EXISTS)

### Expected behavior

Download behavior can be correctly follow DownloadMode

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-3.10.0-1160.88.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.3

Hi! Unfortunately, the current behavior is to delete the downloaded data when this error happens. So, I've opened a PR that removes the problematic import to avoid losing data due to `apache_beam` not being installed (we host the preprocessed version of `natual_questions` on the HF GCS, so requiring `apache_beam` in that case doesn't make sense)

Thanks for your reply. I met another question that I set `export HF_DATASETS_CACHE=/data/lxy/.cache` , but each time I run load_datasets, the datasets module still looking for NQ in the wrong default cache dir '/home/lxy/.cache' 。How to avoid this incorrect behavior. I am sure HF_DATASETS_CACHE was set correctly since I use echo & to check it.
![image](https://github.com/huggingface/datasets/assets/88258534/e7029f27-b9f9-496c-8948-6234ef695646)
by the way I delete the file in '/home/lxy/.cache' since I found there has some kb size file seems useless.

You need to set this variable before the `datasets` import. Then, you can use `import datasets; datasets.config.HF_DATASETS_CACHE` to verify the cache location.
Duplicated Rows When Loading Parquet Files from Root Directory with Subdirectories

### Describe the bug

When parquet files are saved in "train" and "val" subdirectories under a root directory, and datasets are then loaded using `load_dataset("parquet", data_dir="root_directory")`, the resulting dataset has duplicated rows for both the training and validation sets.

### Steps to reproduce the bug

1. Create a root directory, e.g., "testing123".
2. Under "testing123", create two subdirectories: "train" and "val".
3. Create and save a parquet file with 3 unique rows in the "train" subdirectory.
4. Create and save a parquet file with 4 unique rows in the "val" subdirectory.
5. Load the datasets from the root directory using `load_dataset("parquet", data_dir="testing123")`
6. Iterate through the datasets and print the rows

Here's a collab reproducing these steps:

https://colab.research.google.com/drive/11NEdImnQ3OqJlwKSHRMhr7jCBesNdLY4?usp=sharing

### Expected behavior

- Training set should contain 3 unique rows.
- Validation set should contain 4 unique rows.

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.2
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

Thanks for reporting this issue! We should be able to avoid this by making our `glob` patterns more precise. In the meantime, you can load the dataset by directly assigning splits to the data files: 
```python
from datasets import load_dataset
ds = load_dataset("parquet", data_files={"train": "testing123/train/output_train.parquet", "validation": "testing123/val/output_val.parquet"})
```