Kokolipa/github-issues huggingface dataset:
Issue: module 'resource' has no attribute 'error'
State: closed

### Describe the bug

just run import:
`from datasets import load_dataset`

and then:
```

  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\__init__.py", line 22, in <module>
    from .arrow_dataset import Dataset
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\arrow_dataset.py", line 66, in <module>
    from .arrow_reader import ArrowReader
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\arrow_reader.py", line 30, in <module>
    from .download.download_config import DownloadConfig
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\download\__init__.py", line 10, in <module>
    from .streaming_download_manager import StreamingDownloadManager
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\download\streaming_download_manager.py", line 21, in <module>
    from ..filesystems import COMPRESSION_FILESYSTEMS
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\filesystems\__init__.py", line 8, in <module>
    import fsspec.asyn
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\fsspec\asyn.py", line 157, in <module>
    ResourceEror = resource.error
AttributeError: module 'resource' has no attribute 'error'

Process finished with exit code 1

```


and the error codes are:
```
try:
    import resource
except ImportError:
    resource = None
    ResourceError = OSError
else:
    ResourceEror = resource.error

```
1. miss spelling : "ResourceEror " should be "ResourceErorr"
2. module 'resource' has no attribute 'error'




### Steps to reproduce the bug

only one step:

`from datasets import load_dataset`

### Expected behavior

slove error: module 'resource' has no attribute 'error'

### Environment info

python=3.10

datasets==2.14.5


This (Windows) issue was fixed in `fsspec` in https://github.com/fsspec/filesystem_spec/pull/1275.  So, to avoid the error, update the `fsspec` installation with `pip install -U fsspec`.

> This (Windows) issue was fixed in `fsspec` in [fsspec/filesystem_spec#1275](https://github.com/fsspec/filesystem_spec/pull/1275). So, to avoid the error, update the `fsspec` installation with `pip install -U fsspec`.

after I run `pip install -U fsspec`

it occurs a new error:
```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflict
s.
datasets 2.14.5 requires fsspec[http]<2023.9.0,>=2023.1.0, but you have fsspec 2023.9.2 which is incompatible.

```

The `fsspec<2023.9.0` upper bound will be removed in the next release. The `ResourceError` fix is also present in version 2023.6.0, so use that version in the meantime (`pip install fsspec==2023.6.0`).

> The `fsspec<2023.9.0` upper bound will be removed in the next release. The `ResourceError` fix is also present in version 2023.6.0, so use that version in the meantime (`pip install fsspec==2023.6.0`).

thanks for reply！
Issue: Parquet uploads off-by-one naming scheme
State: open

### Describe the bug

I noticed this numbering scheme not matching up in a different project and wanted to raise it as an issue for discussion, what is the actual proper way to have these stored?

<img width="425" alt="image" src="https://github.com/huggingface/datasets/assets/1981179/3ffa2144-7c9a-446f-b521-a5e9db71e7ce">

The `-SSSSS-of-NNNNN` seems to be used widely across the codebase. The section that creates the part in my screenshot is here https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L5287
There are also some edits to this section in the single commit branch.

### Steps to reproduce the bug

1. Upload a dataset that requires at least two parquet files in it
2. Observe the naming scheme

### Expected behavior

The couple options here are of course **1. keeping it as is**

**2. Starting the index at 1:**
train-00001-of-00002-{hash}.parquet
train-00002-of-00002-{hash}.parquet

**3. My preferred option** (which would solve my specific issue), dropping the total entirely:
train-00000-{hash}.parquet
train-00001-{hash}.parquet

This also solves an issue that will occur with an `append` variable for `push_to_hub` (see https://github.com/huggingface/datasets/issues/6290) where as you add a new parquet file, you need to rename everything in the repo as well. 

However, I know there are parts of the repo that use 0 as the starting file or may require the total, so raising the question for discussion.


### Environment info

- `datasets` version: 2.14.6.dev0
- Platform: macOS-14.0-arm64-arm-64bit
- Python version: 3.10.12
- Huggingface_hub version: 0.18.0
- PyArrow version: 12.0.1
- Pandas version: 1.5.3

You can find the reasoning behind this naming scheme [here](https://github.com/huggingface/transformers/pull/16343#discussion_r931182168).

This point has been raised several times, so I'd be okay with starting with `00001-` (also to be consistent with the `transformers` sharding), but I'm not sure @lhoestq agrees.

We start at 0 in `datasets` for consistency with Apache Spark, Apache Beam, Dask and others.

Also note `transformers` isn't a good reference on this topic. I talked with the maintainers when they added shards but it was already released this way. Though we found that there is a backward-compatible way in `transformers` to start at 0, but no request from `transformers` users to changes this AFAIK.

not sure it would be a good idea to break the consistency now, IMO

Makes sense to start at 0 for plenty of good reasons so I'm on board.

What about the second part `-of-0000X`? With single commit PR #6269  just getting merged, there was a note about issues with 100+ file edits https://github.com/huggingface/datasets/pull/6269#issuecomment-1755428581.

That would be my last remaining concern in the context of the `push_to_hub(..., append=True)` work to be done, where appending a single file to the full dataset will require renaming every other existing file in the dataset. If it doesn't seem like a big issue for this work then all the better 👍
Issue: ArrowWriter/ParquetWriter `write` method does not increase `_num_bytes` and hence datasets not sharding at `max_shard_size`
State: closed

### Describe the bug

An example from [1], does not work when limiting shards with `max_shard_size`.

Try the following example with low `max_shard_size`, such as:

```python
builder.download_and_prepare(output_dir, storage_options=storage_options, file_format="parquet", max_shard_size="10MB")
```

The reason for this is that, in line [2] `writer._num_bytes > max_shard_size` is never true, because the `write` method of `ArrowWriter` [3] does not increase `self._num_bytes`.

Such that respective Arrow/Parquet shards are only written to file based on the `writer_batch_size` or `config.DEFAULT_MAX_BATCH_SIZE`, but not based on `max_shard_size`.


[1] https://huggingface.co/docs/datasets/filesystems#download-and-prepare-a-dataset-into-a-cloud-storage

[2] https://github.com/huggingface/datasets/blob/3e8d420808718c9a1453a2e7ee3484ca12c9c70d/src/datasets/builder.py#L1677

[3] https://github.com/huggingface/datasets/blob/3e8d420808718c9a1453a2e7ee3484ca12c9c70d/src/datasets/arrow_writer.py#L459

### Steps to reproduce the bug

Get example from: https://huggingface.co/docs/datasets/filesystems#download-and-prepare-a-dataset-into-a-cloud-storage

Call `builder.download_and_prepare` with low `max_shard_size` such as `10MB`, e.g.:
```python
builder.download_and_prepare(output_dir, storage_options=storage_options, file_format="parquet", max_shard_size="10MB")
```

### Expected behavior

Shards should be written based on `max_shard_size` instead of batch size.

### Environment info

```
>>> import datasets
>>> datasets.__version__
'2.14.6.dev0
```

`writer._num_bytes` is updated every `writer_batch_size`-th call to the `write` method (default `writer_batch_size` is 1000 (examples)). You should be able to see the update by passing a smaller `writer_batch_size` to the `load_dataset_builder`.

We could improve this by supporting the string `writer_batch_size` version as we do with `max_shard_size`, and capping `writer_batch_size` to `max_shard_size` in scenarios where the default `writer_batch_size` > `max_shard_size`. 

Thanks, reducing `writer_batch_size` solved my problem :)
Issue: Support for newer versions of JAX
State: closed
Labels: name: enhancement

### Feature request

Hi,

I like your idea of adapting the datasets library to be usable with JAX. Thank you for that.

However, in your [setup.py](https://github.com/huggingface/datasets/blob/main/setup.py), you enforce old versions of JAX <= 0.3... It is very cumbersome !

What is the rationale for such a limitation ? Can you remove it please ?

Thanks,

### Motivation

This library is unusable with new versions of JAX ?

### Your contribution

Yes.
Issue: Choose columns to stream parquet data in streaming mode
State: closed
Labels: name: bug

Currently passing columns= to load_dataset in streaming mode fails

```
Tried to load parquet data with columns '['link']' with mismatching features '{'caption': Value(dtype='string', id=None), 'image': {'bytes': Value(dtype='binary', id=None), 'path': Value(dtype='null', id=None)}, 'link': Value(dtype='string', id=None), 'message_id': Value(dtype='string', id=None), 'timestamp': Value(dtype='string', id=None)}'
```

similar to https://github.com/huggingface/datasets/issues/6039

reported at https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/3#65259a09617407d4520f4ad9
Issue: how to load the image of dtype float32 or float64
State: open

_FEATURES = datasets.Features(
    {
        "image": datasets.Image(),
        "text": datasets.Value("string"),
    },
)
The datasets builder seems only support the unit8 data. How to load the float dtype data? 

Hi! Can you provide a code that reproduces the issue?

Also, which version of `datasets` are you using? You can check this by running `python -c "import datasets; print(datasets.__version__)"` inside the env. We added support for "float images" in `datasets 2.9`.
Issue: Incremental dataset (e.g. `.push_to_hub(..., append=True)`)
State: open
Labels: name: enhancement

### Feature request

Have the possibility to do `ds.push_to_hub(..., append=True)`.

### Motivation

Requested in this [comment](https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/3#65252597c4edc168202a5eaa) and 
this [comment](https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/4#6524f675c9607bdffb208d8f). Discussed internally on [slack](https://huggingface.slack.com/archives/C02EMARJ65P/p1696950642610639?thread_ts=1690554266.830949&cid=C02EMARJ65P).


### Your contribution

What I suggest to do for parquet datasets is to use `CommitOperationCopy` + `CommitOperationDelete` from `huggingface_hub`:
1. list files
2. copy files from parquet-0001-of-0004 to parquet-0001-of-0005
3. delete files like parquet-0001-of-0004
4. generate + add last parquet file parquet-0005-of-0005

=> make a single commit with all commit operations at once


I think it should be quite straightforward to implement. Happy to review a PR (maybe conflicting with the ongoing "1 commit push_to_hub" PR https://github.com/huggingface/datasets/pull/6269)

Yea I think waiting for #6269 would be best, or branching from it. For reference, this [PR](https://github.com/LAION-AI/Discord-Scrapers/pull/2) is progressing pretty well which will do similar using the hf hub for our LAION dataset bot https://github.com/LAION-AI/Discord-Scrapers/pull/2. 

Is there any update on this?

Is there any update on this?

No update so far on this feature request but for broader context, this announce will help with incremental datasets https://huggingface.co/blog/xethub-joins-hf :)

Still no update? Whats the current recommended way to upload large datasets to the hub? I can't load it all to memory after some limit right

> Still no update? Whats the current recommended way to upload large datasets to the hub? I can't load it all to memory after some limit right

@tolgadur bro, you may set 'data_dir' as a temporary solution like that:

`Dataset.push_to_hub('your_repo_aame', data_dir='data_{index}')`


Related to this feature request: `pyarrow` 21 is out and added content defined chunking for parquet, which enables deduped uploads to Xet.

Therefore we can have a logic in append=True that modifies an existing Parquet file to add more rows without having to reupload the full file.
Issue: Dataset.from_pandas with a DataFrame of PIL.Images
State: open
Labels: name: enhancement

Currently type inference doesn't know what to do with a Pandas Series of PIL.Image objects, though it would be nice to get a Dataset with the Image type this way

A duplicate of https://github.com/huggingface/datasets/issues/4796.

We could get this for free by implementing the `Image` feature as an extension type, as shown in [this](https://colab.research.google.com/drive/1Uzm_tXVpGTwbzleDConWcNjacwO1yxE4?usp=sharing) Colab (example with UUIDs).


+1 to this
Calling this line with a df that contains a PIL image (as they are returned from load_dataset)
`ds = Dataset.from_pandas(df)`
Results in this error:
`ArrowInvalid: ('Could not convert <PIL.PngImagePlugin.PngImageFile image mode=RGB size=1024x1024 at 0x2B41F2D70> with type PngImageFile: did not recognize Python value type when inferring an Arrow data type', 'Conversion failed for column image with type object')`

I found something that can be used as solution.

I have the same problem when I've try to load the images from a pamdas dataset

If you have all on a pandas dataset try 
Dataset.from_dict( your_df.reset_index(drop=True).to_dict(orient='list'), split=set_your_split)

And this avoid the error
Issue: map() not recognizing "text"
State: closed

### Describe the bug

The [map() documentation](https://huggingface.co/docs/datasets/v2.14.5/en/package_reference/main_classes#datasets.Dataset.map) reads:
`
ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True)`

I have been trying to reproduce it in my code as:

`tokenizedDataset = dataset.map(lambda x: tokenizer(x['text']), batched=True)`

But it doesn't work as it throws the error:

> KeyError: 'text'

Can you please guide me on how to fix it?



### Steps to reproduce the bug

1. `from datasets import load_dataset

dataset = load_dataset("amazon_reviews_multi")`

2. Then this code: `from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")`
3. The line I quoted above (which I have been trying)

### Expected behavior

As mentioned in the documentation, it should run without any error and map the tokenization on the whole dataset.

### Environment info

Python 3.10.2

There is no "text" column in the `amazon_reviews_multi`, hence the `KeyError`. You can get the column names by running `dataset.column_names`.
Issue: Add Belebele multiple-choice machine reading comprehension (MRC) dataset
State: closed
Labels: name: enhancement

### Feature request

Belebele is a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. This dataset enables the evaluation of mono- and multi-lingual models in high-, medium-, and low-resource languages. Each question has four multiple-choice answers and is linked to a short passage from the [FLORES-200](https://github.com/facebookresearch/flores/tree/main/flores200) dataset. The human annotation procedure was carefully curated to create questions that discriminate between different levels of generalizable language comprehension and is reinforced by extensive quality checks. While all questions directly relate to the passage, the English dataset on its own proves difficult enough to challenge state-of-the-art language models. Being fully parallel, this dataset enables direct comparison of model performance across all languages. Belebele opens up new avenues for evaluating and analyzing the multilingual abilities of language models and NLP systems.

Please refer to paper for more details, [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://arxiv.org/abs/2308.16884).


## Composition

- 900 questions per language variant
- 488 distinct passages, there are 1-2 associated questions for each.
- For each question, there is 4 multiple-choice answers, exactly 1 of which is correct.
- 122 language/language variants (including English).
- 900 x 122 = 109,800 total questions.


### Motivation

official repo https://github.com/facebookresearch/belebele

### Your contribution

-

This dataset is already available on the Hub: https://huggingface.co/datasets/facebook/belebele.
Issue: FileNotFoundError: Couldn't find a module script at /content/paws-x/paws-x.py. Module 'paws-x' doesn't exist on the Hugging Face Hub either.
State: closed

### Describe the bug

I'm encountering a "FileNotFoundError" while attempting to use the "paws-x" dataset to retrain the DistilRoBERTa-base model. The error message is as follows: 

FileNotFoundError: Couldn't find a module script at /content/paws-x/paws-x.py. Module 'paws-x' doesn't exist on the Hugging Face Hub either.

### Steps to reproduce the bug

https://colab.research.google.com/drive/11xUUFxloClpmqLvDy_Xxfmo3oUzjY5nx#scrollTo=kUn74FigzhHm

### Expected behavior

The the trained model

### Environment info

colab, "paws-x" dataset , DistilRoBERTa-base model

`evaluate.load("paws-x", "es")` throws the error because there is no such metric in the `evaluate` lib.

So, this is unrelated to our lib.
Issue: Would like to Contribute a dataset
State: closed

I have a dataset of 2500 images that can be used for color-blind machine-learning algorithms. Since , there was no dataset available online , I made this dataset myself and would like to contribute this now to community

Hi! The process of contributing a dataset is explained here: https://huggingface.co/docs/datasets/upload_dataset. Also, check https://huggingface.co/docs/datasets/image_dataset for a more detailed explanation of how to share an image dataset.
Issue: FileNotFoundError for dataset with multiple builder config
State: closed

### Describe the bug

When there is only one config and only the dataset name is entered when using datasets.load_dataset(), it works fine. But if I create a second builder_config for my dataset and enter the config name when using datasets.load_dataset(), the following error will happen.

FileNotFoundError: [Errno 2] No such file or directory: 'C:/Users/chenx/.cache/huggingface/datasets/my_dataset/0_shot_multiple_choice/1.0.0/97c3854a012cfd6b045e3be4c864739902af2d818bb9235b047baa94c302e9a2.incomplete/my_dataset-test-00000-00000-of-NNNNN.arrow'

The "XXX.incomplete folder" in the cache folder of my dataset will disappear before "generating test split", which does not happen when config name is not entered and the config name is "default"

C:\Users\chenx\.cache\huggingface\datasets\my_dataset\0_shot_multiple_choice\1.0.0    
The folder that is supposed to remain under the above directory will disappear, and the data generator will not have a place to generate data into. 

### Steps to reproduce the bug

test = load_dataset('my_dataset', '0_shot_multiple_choice')

### Expected behavior

FileNotFoundError: [Errno 2] No such file or directory: 'C:/Users/chenx/.cache/huggingface/datasets/my_dataset/0_shot_multiple_choice/1.0.0/97c3854a012cfd6b045e3be4c864739902af2d818bb9235b047baa94c302e9a2.incomplete/my_dataset-test-00000-00000-of-NNNNN.arrow'

### Environment info

datasets                  2.14.5
python                    3.8.18

Please tell me if the above info is not enough for solving the problem. I will then make my dataset public temporarily so that you can really reproduce the bug. 

Hi! 
Could you share how to solve this problem? 
I faced this same error.
Issue: Broken Link to PubMed Abstracts dataset .
State: open

### Describe the bug

The link provided for the dataset is broken,
data_files = 
[https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst](url)

The 

### Steps to reproduce the bug

Steps to reproduce:

1) Head over to [https://huggingface.co/learn/nlp-course/chapter5/4?fw=pt#big-data-datasets-to-the-rescue](url)

2) In the Section  "What is the Pile?", you can see a code snippet that contains the broken link.

### Expected behavior

The link should Redirect to the  "PubMed Abstracts dataset" as expected .

### Environment info

.

This has already been reported in the HF Course repo (https://github.com/huggingface/course/issues/623).

@lhoestq @albertvillanova @lewtun I don't think we are allowed to host these data files on the Hub (due to DMCA), which means the only option is to use a different dataset in the course (and to re-record the video 🙂), no?

Keeping the video is maybe fine, we can add a note on youtube to suggest to load a dataset with a different name. Maybe C4 ? And update the code snippets on the website ?

Maybe you want to try it with the PUBMED dataset that I reproduced based on the The [PubMed Abstract GitHub Site](http://github.com/thoppe/The-Pile-PubMed) and uploaded on the HuggingFace:

```
from datasets import load_dataset
pubmed_dataset = load_dataset("hwang2006/PUBMED_title_abstracts_2020_baseline")
pubmed_dataset

#Downloading data: 100%
#7.98G/7.98G [11:47<00:00, 9.68MB/s]
#Generating train split: 17722096/0 [00:36<00:00, 505376.37 examples/s]

#DatasetDict({
#   train: Dataset({
#        features: ['meta', 'text'],
#        num_rows: 17722096
#    })
#})
```

孔令涛说感谢感谢
Issue: Duplicate `data_files` when named `<split>/<split>.parquet`
State: closed
Labels: name: bug

e.g. with `u23429/stock_1_minute_ticker`

```ipython
In [1]: from datasets import *

In [2]: b = load_dataset_builder("u23429/stock_1_minute_ticker")
Downloading readme: 100%|██████████████████████████| 627/627 [00:00<00:00, 246kB/s]

In [3]: b.config.data_files
Out[3]: 
{NamedSplit('train'): ['hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/train/train.parquet',
  'hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/train/train.parquet'],
 NamedSplit('validation'): ['hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/validation/validation.parquet',
  'hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/validation/validation.parquet'],
 NamedSplit('test'): ['hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/test/test.parquet',
  'hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/test/test.parquet']}

```

This bug issue is present in the current `datasets` 2.14.5 and also on `main` even after https://github.com/huggingface/datasets/pull/6244 cc @mariosasko 

Also reported in https://github.com/huggingface/datasets/issues/6259

I think it's best to drop duplicates with a `set`  (as a temporary fix) and improve the patterns when/if https://github.com/fsspec/filesystem_spec/pull/1382 gets merged. @lhoestq Do you have some other ideas?

Alternatively we could just use this no ?

```python
if config.FSSPEC_VERSION < version.parse("2023.9.0"):
    KEYWORDS_IN_PATH_NAME_BASE_PATTERNS = [
        "{keyword}[{sep}/]**",
        "**[{sep}]{keyword}[{sep}/]**",
        "**/{keyword}[{sep}/]**",
    ]
else:
    KEYWORDS_IN_PATH_NAME_BASE_PATTERNS = [
        "{keyword}[{sep}/]**",
        "**/*[{sep}]{keyword}[{sep}/]**",
        "**/*/{keyword}[{sep}/]**",
    ]
```

This way no need to implement sets, which would require a bit of work since we've always considered a list of pattern to be resolved as the concatenated list of resolved files for each pattern (including duplicates)


Arf `"**/*/{keyword}[{sep}/]**"` does return `data/keyword.txt` in latest `fsspec` but not in `glob.glob`

EDIT: actually forgot to set `recursive=True`

Actually `glob.glob` does return it with `recursive=True` ! my bad

Pff just tested and my idea sucks, pattern 1 and 3 obviously give duplicates 

> I think it's best to drop duplicates with a set (as a temporary fix)

I started https://github.com/huggingface/datasets/pull/6278 to use DataFilesSet objects instead of DataFilesList
Issue: Overwriting Split overwrites data but not metadata, corrupting dataset
State: closed

### Describe the bug

I want to be able to overwrite/update/delete splits in my dataset. Currently the only way to do is to manually go into the dataset and delete the split. If I try to overwrite programmatically I end up in an error state and (somewhat) corrupting the dataset. Read below.

**Current Behavior**
When I push to an existing split I get this error:
`ValueError: Split complexRoofLocation_01Apr2023_to_31May2023test already present` 
This seems to suggest that the library doesn't support overwriting splits. 

**Potential Bug**
What’s strange is that datasets, despite the operation erroring out with the ValueError above, does, in fact, overwrite the split:

`Pushing dataset shards to the dataset hub: 100% [.....................] 1/1 [00:00<00:00, 55.04it/s]` 
Even though you got an error message and your code fails, your dataset is now changed. That seems like a bug. Either don't change the dataset, or don't throw the error and allow the script to proceed. 

Additional Bug
While it overwrites the split, it doesn’t overwrite the split’s information. Because of this when you pull down the dataset you may end up getting a `NonMatchingSplitsSizesError` if the size of the dataset during the overwrite is different. For example, my original split had 5 rows, but on my overwrite, I only had 4. Then when I try to download the dataset, I get a `NonMatchingSplitsSizesError` because the dataset's data.json states there’s 5 but only 4 exist in the split.

Expected Behavior
This corrupts the dataset rendering it unusable (until you take manual intervention). Either the library should let the overwrite happen (which it does but should also update the metadata) or it shouldn’t do anything.

### Steps to reproduce the bug

[Colab Notebook](https://colab.research.google.com/drive/1bqVkD06Ngs9MQNdSk_ygCG6y1UqXA4pC?usp=sharing)


### Expected behavior

The split should be overwritten and I should be able to use the new version of the dataset without issue. 

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
Issue: CI is broken: ImportError: cannot import name 'context' from 'tensorflow.python'
State: closed
Labels: name: bug

Python 3.10 CI is broken for `test_py310`.

See: https://github.com/huggingface/datasets/actions/runs/6322990957/job/17169678812?pr=6262

```
FAILED tests/test_py_utils.py::TempSeedTest::test_tensorflow - ImportError: cannot import name 'context' from 'tensorflow.python' (/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/tensorflow/python/__init__.py)
```

```
_________________________ TempSeedTest.test_tensorflow _________________________
[gw1] linux -- Python 3.10.13 /opt/hostedtoolcache/Python/3.10.13/x64/bin/python

self = <tests.test_py_utils.TempSeedTest testMethod=test_tensorflow>

    @require_tf
    def test_tensorflow(self):
        import tensorflow as tf
        from tensorflow.keras import layers
    
        model = layers.Dense(2)
    
        def gen_random_output():
            x = tf.random.uniform((1, 3))
            return model(x).numpy()
    
>       with temp_seed(42, set_tensorflow=True):

tests/test_py_utils.py:155: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/contextlib.py:135: in __enter__
    return next(self.gen)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

seed = 42, set_pytorch = False, set_tensorflow = True

    @contextmanager
    def temp_seed(seed: int, set_pytorch=False, set_tensorflow=False):
        """Temporarily set the random seed. This works for python numpy, pytorch and tensorflow."""
        np_state = np.random.get_state()
        np.random.seed(seed)
    
        if set_pytorch and config.TORCH_AVAILABLE:
            import torch
    
            torch_state = torch.random.get_rng_state()
            torch.random.manual_seed(seed)
    
            if torch.cuda.is_available():
                torch_cuda_states = torch.cuda.get_rng_state_all()
                torch.cuda.manual_seed_all(seed)
    
        if set_tensorflow and config.TF_AVAILABLE:
            import tensorflow as tf
>           from tensorflow.python import context as tfpycontext
E           ImportError: cannot import name 'context' from 'tensorflow.python' (/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/tensorflow/python/__init__.py)

/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/datasets/utils/py_utils.py:257: ImportError
```
Issue: REUSE_DATASET_IF_EXISTS don't work 
State: closed

### Describe the bug

I use the following code  to download natural_question dataset. Even though I have completely download it, the next time I run this code, the new download procedure will start and cover the original /data/lxy/NQ

config=datasets.DownloadConfig(resume_download=True,max_retries=100,cache_dir=r'/data/lxy/NQ',download_desc='NQ')

data=datasets.load_dataset('natural_questions',cache_dir=r'/data/lxy/NQ',download_config=config,download_mode=DownloadMode.REUSE_DATASET_IF_EXISTS)

---
Since I don't have apache_beam installed, it throw a exception. After I pip install apache_beam ,the download restart..

![image](https://github.com/huggingface/datasets/assets/88258534/f28ce7fe-29ea-4348-b87f-e69182a8bd41)


### Steps to reproduce the bug

run this two line code

config=datasets.DownloadConfig(resume_download=True,max_retries=100,cache_dir=r'/data/lxy/NQ',download_desc='NQ')

data=datasets.load_dataset('natural_questions',cache_dir=r'/data/lxy/NQ',download_config=config,download_mode=DownloadMode.REUSE_DATASET_IF_EXISTS)

### Expected behavior

Download behavior can be correctly follow DownloadMode

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-3.10.0-1160.88.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.3

Hi! Unfortunately, the current behavior is to delete the downloaded data when this error happens. So, I've opened a PR that removes the problematic import to avoid losing data due to `apache_beam` not being installed (we host the preprocessed version of `natual_questions` on the HF GCS, so requiring `apache_beam` in that case doesn't make sense)

Thanks for your reply. I met another question that I set `export HF_DATASETS_CACHE=/data/lxy/.cache` , but each time I run load_datasets, the datasets module still looking for NQ in the wrong default cache dir '/home/lxy/.cache' 。How to avoid this incorrect behavior. I am sure HF_DATASETS_CACHE was set correctly since I use echo & to check it.
![image](https://github.com/huggingface/datasets/assets/88258534/e7029f27-b9f9-496c-8948-6234ef695646)
by the way I delete the file in '/home/lxy/.cache' since I found there has some kb size file seems useless.

You need to set this variable before the `datasets` import. Then, you can use `import datasets; datasets.config.HF_DATASETS_CACHE` to verify the cache location.
Issue: Duplicated Rows When Loading Parquet Files from Root Directory with Subdirectories
State: closed

### Describe the bug

When parquet files are saved in "train" and "val" subdirectories under a root directory, and datasets are then loaded using `load_dataset("parquet", data_dir="root_directory")`, the resulting dataset has duplicated rows for both the training and validation sets.

### Steps to reproduce the bug

1. Create a root directory, e.g., "testing123".
2. Under "testing123", create two subdirectories: "train" and "val".
3. Create and save a parquet file with 3 unique rows in the "train" subdirectory.
4. Create and save a parquet file with 4 unique rows in the "val" subdirectory.
5. Load the datasets from the root directory using `load_dataset("parquet", data_dir="testing123")`
6. Iterate through the datasets and print the rows

Here's a collab reproducing these steps:

https://colab.research.google.com/drive/11NEdImnQ3OqJlwKSHRMhr7jCBesNdLY4?usp=sharing

### Expected behavior

- Training set should contain 3 unique rows.
- Validation set should contain 4 unique rows.

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.2
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

Thanks for reporting this issue! We should be able to avoid this by making our `glob` patterns more precise. In the meantime, you can load the dataset by directly assigning splits to the data files: 
```python
from datasets import load_dataset
ds = load_dataset("parquet", data_files={"train": "testing123/train/output_train.parquet", "validation": "testing123/val/output_val.parquet"})
```
Issue: HfHubHTTPError - exceeded our hourly quotas for action: commit
State: closed

### Describe the bug

I try to upload a very large dataset of images, and get the following error:
```
File /fsx-multigen/yuvalkirstain/miniconda/envs/pickapic/lib/python3.10/site-packages/huggingface_hub/hf_api.py:2712, in HfApi.create_commit(self, repo_id, operations, commit_message, commit_description, token, repo_type, revision, create_pr, num_threads, parent_commit, run_as_future)
   2710 try:
   2711     commit_resp = get_session().post(url=commit_url, headers=headers, data=data, params=params)
-> 2712     hf_raise_for_status(commit_resp, endpoint_name="commit")
   2713 except RepositoryNotFoundError as e:
   2714     e.append_to_message(_CREATE_COMMIT_NO_REPO_ERROR_MESSAGE)

File /fsx-multigen/yuvalkirstain/miniconda/envs/pickapic/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py:301, in hf_raise_for_status(response, endpoint_name)
    297     raise BadRequestError(message, response=response) from e
    299 # Convert `HTTPError` into a `HfHubHTTPError` to display request information
    300 # as well (request id and/or server error message)
--> 301 raise HfHubHTTPError(str(e), response=response) from e

HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/datasets/yuvalkirstain/pickapic_v2/commit/main (Request ID: Root=1-65112399-12d63f7d7f28bfa40a36a0fd)

You have exceeded our hourly quotas for action: commit. We invite you to retry later.
```

this makes it much less convenient to host large datasets on HF hub.

### Steps to reproduce the bug

Upload a very large dataset of images

### Expected behavior

the upload to work well

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.15.0-1033-aws-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 1.5.3

how is your dataset structured? (file types, how many commits and files are you trying to push, etc)

I succeeded in uploading it after several attempts with an hour gap between each attempt (inconvenient but worked). The final dataset is [here](https://huggingface.co/datasets/yuvalkirstain/pickapic_v2), code and context to the dataset can be found [here](https://github.com/yuvalkirstain/PickScore/).
I can close the issue if this behavior is intended, as most users probably do not need to upload large-scale datasets.

We could fix this by creating a single commit for all the (Parquet) shards in `push_to_hub` instead of one commit per shard, as we currently do. 

@Wauplin Any updates on the 2-step commit process suggested by you that we need to implement this?

> Any updates on the 2-step commit process suggested by you that we need to implement this?

Re-prioritizing this, sorry. Will let you know but probably can be done this week.
Issue: load_dataset() function's cache_dir does not seems to work
State: closed

### Describe the bug

datasets version: 2.14.5

when trying to run the following command
trec = load_dataset('trec', split='train[:1000]', cache_dir='/path/to/my/dir')

I keep getting error saying the command does not have permission to the default cache directory on my macbook pro machine. 

It seems the cache_dir parameter cannot change the dataset saving directory from the default 

what ever explained in the https://huggingface.co/docs/datasets/cache does not seem to work


### Steps to reproduce the bug

datasets version: 2.14.5

when trying to run the following command
trec = load_dataset('trec', split='train[:1000]', cache_dir='/path/to/my/dir')

I keep getting error saying the command does not have permission to the default cache directory on my macbook pro machine. 

It seems the cache_dir parameter cannot change the dataset saving directory from the default 

what ever explained in the https://huggingface.co/docs/datasets/cache does not seem to work

### Expected behavior

the dataset should be saved to the cache_dir points to 

### Environment info

datasets version: 2.14.5
macos X: Ventura 13.4.1 (c)

Can you share the error message?

Also, it would help if you could check whether `huggingface_hub`'s download behaves the same:
```python
from huggingface_hub import snapshot_download
snapshot_download("trec", repo_type="dataset", cache_dir='/path/to/my/dir)
```

In the next major release, we aim to switch to `huggingface_hub` for file download/caching, but we could align the `cache_dir`'s `umask` behavior earlier than this if their solution works for your use case.

@mariosasko, Hey , is there any update on this? 

Yes, I am still facing the same issue.

```python
from huggingface_hub import snapshot_download
snapshot_download("trec", repo_type="dataset", cache_dir='/path/to/my/dir)
```

But this is working for me.

Thanks.

Could you clarify a bit? What if I wanted only 100 samples from train split, how would i change the above? Thanks

I have exactly same issue

@lhoestq Can this be open again? Reading #7200 this doesn't even address not being able to specify `cache_dir`, it just updates the documentation? Their own comment states this:

> In my testing, I could not specify the cache directory even by load_dataset("dataset_name" cache_dir="..."). It might be another issue. I also welcome any advice to solve this issue.

Nothing else in the PR seems to indicate that this is actually resolved.

I've noticed the same issue. Looks like #7499 was fixing it, but from what I see in the comments this setting is for different caches?

Hi ! you can find the information about the `datasets` and `huggingface_hub` cache directories here: https://huggingface.co/docs/datasets/v3.6.0/en/cache#cache-directory
Issue: Dataset.from_generator() cost much more time in vscode debugging mode then running mode
State: closed

### Describe the bug

Hey there,
I’m using Dataset.from_generator() to convert a torch_dataset to the Huggingface Dataset.
However, when I debug my code on vscode, I find that it runs really slow on Dataset.from_generator() which may even 20 times longer then run the script on terminal.

### Steps to reproduce the bug

I write a simple test code :

```python
import os
from functools import partial
from typing import Callable

import torch
import time
from torch.utils.data import Dataset as TorchDataset

from datasets import load_from_disk, Dataset as HFDataset
  
import torch  
from torch.utils.data import Dataset  
  
class SimpleDataset(Dataset):  
    def __init__(self, data):  
        self.data = data  
        self.keys = list(data[0].keys())
      
    def __len__(self):  
        return len(self.data)  
      
    def __getitem__(self, index):  
        sample = self.data[index]  
        return {key: sample[key] for key in self.keys}  
  

def TorchDataset2HuggingfaceDataset(torch_dataset: TorchDataset, cache_dir: str = None
) -> HFDataset:
    
    """
        convert torch dataset to huggingface dataset
    """
    generator : Callable[[], TorchDataset] = lambda: (sample for sample in torch_dataset)   

    return HFDataset.from_generator(generator, cache_dir=cache_dir)

if __name__ == '__main__':
    data = [  
        {'id': 1, 'name': 'Alice'},  
        {'id': 2, 'name': 'Bob'},  
        {'id': 3, 'name': 'Charlie'}  
    ]
    
    torch_dataset = SimpleDataset(data)
    start_time = time.time() 
    huggingface_dataset = TorchDataset2HuggingfaceDataset(torch_dataset)
    end_time = time.time()
    print("time: ", end_time - start_time)
    print(huggingface_dataset)
```

### Expected behavior

this test on my machine report that the running time on terminal is 0.086,
however the running time in debugging mode on vscode is 0.25, which I think is much longer than expected.

I’d like to know is the anything wrong in the code or just because of debugging?
I have traced the code and I find is this func which I get stuck.

```python
def create_config_id(
        self,
        config_kwargs: dict,
        custom_features: Optional[Features] = None,
    ) -> str:
...
# stuck in this line
suffix = Hasher.hash(config_kwargs_to_add_to_suffix)
```



### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-5.11.0-27-generic-x86_64-with-glibc2.31
- Python version: 3.11.3
- Huggingface_hub version: 0.17.2
- PyArrow version: 11.0.0
- Pandas version: 2.0.1

Answered on the forum: https://discuss.huggingface.co/t/dataset-from-generator-cost-much-more-time-in-vscode-debugging-mode-then-running-mode/56005/2
Issue: exif_transpose not done to Image (PIL problem)
State: closed
Labels: name: enhancement

### Feature request

I noticed that some of my images loaded using PIL have some metadata related to exif that can rotate them when loading.
Since the dataset.features.Image uses PIL for loading, the loaded image may be rotated (width and height will be inverted) thus for tasks as object detection and layoutLM this can create some inconsistencies (between input bboxes and input images). 

For now there is no option in datasets.features.Image to specify that. We need to do the following when preparing examples (when preparing images for training, test or inference): 
```
from PIL import Image, ImageOps 
pil = ImageOps.exif_transpose(pil)
```

reference: https://stackoverflow.com/a/63950647/5720150 

Is it possible to add this by default to the datasets.feature.Image ? or to add the option to do the ImageOps.exif_transpose?

Thank you

### Motivation

Prevent having inverted data related to exif metadata that may affect object detection tasks

### Your contribution

Changing in datasets.featrues.Image I can help with that. 

Indeed, it makes sense to do this by default. 

In the meantime, you can use `.with_transform` to transpose the images when accessing them:

```python
import PIL.ImageOps

def exif_transpose_transform(batch):
  batch["image"] = [PIL.ImageOps.exif_transpose(image) for image in batch["image"]]
  return batch

dataset = dataset.with_transform(exif_transpose_transform)
```

This operation sets some `Image` attributes to `None` (`.format`, `.filename`, etc.), causing our tests to fail, so I think we should wait for Datasets 3.0 to make this change. In version 3.0, storing image paths will be replaced by embedding image bytes, so there will be fewer instances where we use the `.filename` attribute.
Issue: Data alteration when loading dataset with unspecified inner sequence length
State: closed

### Describe the bug

When a dataset saved with a specified inner sequence length is loaded without specifying that length, the original data is altered and becomes inconsistent.

### Steps to reproduce the bug

```python
from datasets import Dataset, Features, Value, Sequence, load_dataset

# Repository ID
repo_id = "my_repo_id"

# Define features with a specific length of 3 for each inner sequence
specified_features = Features({"key": Sequence(Sequence(Value("float32"), length=3))})

# Create a dataset with the specified features
data = [
    [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]],
    [[7.0, 8.0, 9.0], [10.0, 11.0, 12.0]],
]
dataset = Dataset.from_dict({"key": data}, features=specified_features)

# Push the dataset to the hub
dataset.push_to_hub(repo_id)

# Define features without specifying the length
unspecified_features = Features({"key": Sequence(Sequence(Value("float32")))})

# Load the dataset from the hub with this new feature definition
dataset = load_dataset(f"qgallouedec/{repo_id}", split="train", features=unspecified_features)

# The obtained data is altered
print(dataset.to_dict())  # {'key': [[[1.0], [2.0]], [[3.0], [4.0]]]}
```


### Expected behavior

```python
print(dataset.to_dict())  # {'key': [[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]], [[7.0, 8.0, 9.0], [10.0, 11.0, 12.0]]]}
```

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-6.2.0-32-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.3

While this issue may seem specific, it led to a silent problem in my workflow that took days to diagnose. If this feature is not intended to be supported, an error should be raised when encountering this configuration to prevent such issues.

Thanks for reporting! This is a MRE:

```python
import pyarrow as pa
from datasets.table import cast_array_to_feature
from datasets import Sequence, Value
data = [
    [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]],
    [[7.0, 8.0, 9.0], [10.0, 11.0, 12.0]],
]
arr = pa.array(data, pa.list_(pa.list_(pa.float32(), 3)))
cast_array_to_feature(arr, Sequence(Sequence(Value("float32"))))
```

I've opened a PR with a fix.
Issue: Dataloader stuck on multiple GPUs
State: closed

### Describe the bug

I am trying to get CLIP to fine-tuning with my code.
When I tried to run it on multiple GPUs using accelerate, I encountered the following phenomenon.

- Validation dataloader stuck in 2nd epoch only on multi-GPU
Specifically, when the "for inputs in valid_loader:" process is finished, it does not proceed to the next step. train_loader process is completed. Also, both train and valid are working correctly in the first epoch.
The accelerate command at that time is as follows.
`accelerate launch --multi_gpu --num_processes=2 {script_name.py} {--arg1} {--arg2} ...`

- This will not happen when single GPU is used.
`CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...`
- Setting num_workers=0 in dataloader did not change the result.

### Steps to reproduce the bug

1. The codes for fine-tuning the regular CLIP were updated for accelerate.
2. Run the code with the accelerate command as `accelerate launch --multi_gpu --num_processes=2 {script_name.py} {--arg1} {--arg2} ...` and the above problem will occur.
3. CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...` , it works fine.

### Expected behavior

It Should end normally as if it was run on a single GPU.

### Environment info

Since `datasets-cli env` did not work, the environment is described below.

- OS: Ubuntu 22.04 with Docker
- Docker: 24.0.5, build ced0996
- Python: 3.10.12
- torch==2.0.1
- accelerate==0.21.0
- transformers==4.33.1

What type of dataset are you using in this script? `torch.utils.data.Dataset` or `datasets.Dataset`? Please share the `datasets` package version if it's the latter. Otherwise, it's better to move this issue to the `accelerate` repo.

Very sorry, I thought I had a repo in `accelerate!`
I will close this issue and repo the issue in the appropriate place.