dalssy/data_del huggingface dataset:
GitHub issue: `load_dataset` fails to load dataset saved by `save_to_disk`
Author: sliedes (NONE)
State: open | Created: 2024-07-01 12:19:19

### Describe the bug

This code fails to load the dataset it just saved:

```python
from datasets import load_dataset
from transformers import AutoTokenizer

MODEL = "google-bert/bert-base-cased"
tokenizer = AutoTokenizer.from_pretrained(MODEL)

dataset = load_dataset("yelp_review_full")

def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenized_datasets = dataset.map(tokenize_function, batched=True)
tokenized_datasets.save_to_disk("dataset")

tokenized_datasets = load_dataset("dataset/")  # raises
```

It raises `ValueError: Couldn't infer the same data file format for all splits. Got {NamedSplit('train'): ('arrow', {}), NamedSplit('test'): ('json', {})}`.

I believe this bug is caused by the [logic that tries to infer dataset format](https://github.com/huggingface/datasets/blob/9af8dd3de7626183a9a9ec8973cebc672d690400/src/datasets/load.py#L556). It counts the most common file extension. However, a small dataset can fit in a single `.arrow` file and have two JSON metadata files, causing the format to be inferred as JSON:

```shell
$ ls -l dataset/test
-rw-r--r-- 1 sliedes sliedes 191498784 Jul  1 13:55 data-00000-of-00001.arrow
-rw-r--r-- 1 sliedes sliedes      1730 Jul  1 13:55 dataset_info.json
-rw-r--r-- 1 sliedes sliedes       249 Jul  1 13:55 state.json
```

### Steps to reproduce the bug

Execute the code above.

### Expected behavior

The dataset is loaded successfully.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-6.9.3-arch1-1-x86_64-with-glibc2.39
- Python version: 3.12.4
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0
GitHub issue: `drop_duplicates` method
Author: MohamedAliRashad (NONE)
State: open | Created: 2024-07-01 09:01:06
Labels: name: duplicate, name: enhancement

### Feature request

`drop_duplicates` method for huggingface datasets (similiar in simplicity to the `pandas` one)

### Motivation

Ease of use

### Your contribution

I don't think i am good enough to help
GitHub issue: add split argument to Generator
Author: piercus (CONTRIBUTOR)
State: closed | Created: 2024-07-01 08:09:25

## Actual

When creating a multi-split dataset using generators like 

```python
datasets.DatasetDict({
  "val": datasets.Dataset.from_generator(
      generator=generator_val,
      features=features
  ),
  "test": datasets.Dataset.from_generator(
      generator=generator_test,
      features=features,
  )
})
```

It displays (for both test and val)

```
Generating train split
```


## Expected

I would like to be able to improve this behavior by doing

```python
datasets.DatasetDict({
  "val": datasets.Dataset.from_generator(
      generator=generator_val,
      features=features,
      split="val"
  ),
  "test": datasets.Dataset.from_generator(
      generator=generator_test,
      features=features,
      split="test"
  )
})
```


It would display

```
Generating val split
```

and 

```
Generating test split
```

## Proposal

Current PR is adding an explicit `split` argument and replace the implicit "train" split in the following classes/function : 
* Generator
* from_generator
* AbstractDatasetInputStream
* GeneratorDatasetInputStream

Please share your feedbacks
GitHub issue: CI is broken for faiss tests on Windows: node down: Not properly terminated
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-07-01 06:40:03
Labels: name: maintenance

Faiss tests on Windows make the CI run indefinitely until maximum execution time (360 minutes) is reached.

See: https://github.com/huggingface/datasets/actions/runs/9712659783
```
test (integration, windows-latest, deps-minimum)
The job running on runner GitHub Actions 60 has exceeded the maximum execution time of 360 minutes.
test (integration, windows-latest, deps-latest)
The job running on runner GitHub Actions 238 has exceeded the maximum execution time of 360 minutes.
```
```
____________________________ tests/test_search.py _____________________________
[gw1] win32 -- Python 3.8.10 C:\hostedtoolcache\windows\Python\3.8.10\x64\python.exe
worker 'gw1' crashed while running 'tests/test_search.py::IndexableDatasetTest::test_add_faiss_index'
____________________________ tests/test_search.py _____________________________
[gw2] win32 -- Python 3.8.10 C:\hostedtoolcache\windows\Python\3.8.10\x64\python.exe
worker 'gw2' crashed while running 'tests/test_search.py::IndexableDatasetTest::test_add_faiss_index'
```
```
tests/test_search.py::IndexableDatasetTest::test_add_faiss_index 
[gw0] node down: Not properly terminated
[gw0] FAILED tests/test_search.py::IndexableDatasetTest::test_add_faiss_index 

replacing crashed worker gw0

tests/test_search.py::IndexableDatasetTest::test_add_faiss_index 
[gw1] node down: Not properly terminated
[gw1] FAILED tests/test_search.py::IndexableDatasetTest::test_add_faiss_index 

replacing crashed worker gw1

tests/test_search.py::IndexableDatasetTest::test_add_faiss_index 
[gw2] node down: Not properly terminated
[gw2] FAILED tests/test_search.py::IndexableDatasetTest::test_add_faiss_index 

replacing crashed worker gw2
```
GitHub issue: Re-enable raising error from huggingface-hub FutureWarning in CI
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-06-28 07:28:32

Re-enable raising error from huggingface-hub FutureWarning in tests, once that the fix in transformers
- https://github.com/huggingface/transformers/pull/31007

was just released yesterday in transformers-4.42.0: https://github.com/huggingface/transformers/releases/tag/v4.42.0


Fix #7010.
GitHub issue: Re-enable raising error from huggingface-hub FutureWarning in CI
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-06-28 07:23:40
Labels: name: maintenance

Re-enable raising error from huggingface-hub FutureWarning in CI, which was disabled by PR:
- #6876

Note that this can only be done once transformers releases the fix:
- https://github.com/huggingface/transformers/pull/31007
GitHub issue: CI is broken after ruff-0.5.0: E721
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-06-28 05:03:28
Labels: name: maintenance

After ruff-0.5.0 release (https://github.com/astral-sh/ruff/releases/tag/0.5.0), our CI is broken due to E721 rule. 

See: https://github.com/huggingface/datasets/actions/runs/9707641618/job/26793170961?pr=6983
> src/datasets/features/features.py:844:12: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
GitHub issue: Datasetbuilder Local Download FileNotFoundError
Author: purefall (NONE)
State: open | Created: 2024-06-25 15:02:34

### Describe the bug

So I was trying to download a dataset and save it as parquet and I follow the [tutorial](https://huggingface.co/docs/datasets/filesystems#download-and-prepare-a-dataset-into-a-cloud-storage) of Huggingface. However, during the excution I face a FileNotFoundError.

I debug the code and it seems there is a bug there:
So first it creates a .incomplete folder and before moving its contents the following code deletes the directory
[Code](https://github.com/huggingface/datasets/blob/98fdc9e78e6d057ca66e58a37f49d6618aab8130/src/datasets/builder.py#L984)
hence as a result I face with: 

``` FileNotFoundError: [Errno 2] No such file or directory: '~/data/Parquet/.incomplete '```

### Steps to reproduce the bug

```
from datasets import load_dataset_builder
from pathlib import Path

parquet_dir = "~/data/Parquet/" 
Path(parquet_dir).mkdir(parents=True, exist_ok=True)
builder = load_dataset_builder(
    "rotten_tomatoes",
)
builder.download_and_prepare(parquet_dir, file_format="parquet")
```

### Expected behavior

Downloads the files and saves as parquet

### Environment info

Ubuntu,
Python 3.10
```
datasets 2.19.1
```
GitHub issue: IterableDataset: Unsupported ScalarType BFloat16
Author: stoical07 (NONE)
State: closed | Created: 2024-06-25 14:43:26

### Describe the bug

`IterableDataset.from_generator` crashes when using BFloat16:

```
  File "/usr/local/lib/python3.11/site-packages/datasets/utils/_dill.py", line 169, in _save_torchTensor
    args = (obj.detach().cpu().numpy(),)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^

TypeError: Got unsupported ScalarType BFloat16
```

### Steps to reproduce the bug

```python
import torch
from datasets import IterableDataset


def demo(x):
    yield {"x": x}


x = torch.tensor([1.], dtype=torch.bfloat16)

dataset = IterableDataset.from_generator(
    demo,
    gen_kwargs=dict(x=x),
)

example = next(iter(dataset))
print(example)
```

### Expected behavior

Code sample should print:
```python
{'x': tensor([1.], dtype=torch.bfloat16)}
```

### Environment info

```
datasets==2.20.0
torch==2.2.2
```
GitHub issue: CI is broken for tests using hf-internal-testing/librispeech_asr_dummy
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-06-25 07:55:44
Labels: name: maintenance

CI is broken: https://github.com/huggingface/datasets/actions/runs/9657882317/job/26637998686?pr=6996
```
FAILED tests/test_inspect.py::test_get_dataset_config_names[hf-internal-testing/librispeech_asr_dummy-expected4] - AssertionError: assert ['clean'] == ['clean', 'other']
  
  Right contains one more item: 'other'
  
  Full diff:
    [
        'clean',
  -     'other',
    ]
FAILED tests/test_inspect.py::test_get_dataset_default_config_name[hf-internal-testing/librispeech_asr_dummy-None] - AssertionError: assert 'clean' is None
```

Note that repository was recently converted to Parquet: https://huggingface.co/datasets/hf-internal-testing/librispeech_asr_dummy/commit/5be91486e11a2d616f4ec5db8d3fd248585ac07a
GitHub issue: Problematic rank after calling `split_dataset_by_node` twice
Author: yzhangcs (CONTRIBUTOR)
State: closed | Created: 2024-06-21 14:25:26

### Describe the bug

I'm trying to split `IterableDataset` by `split_dataset_by_node`.
But when doing split on a already split dataset, the resulting `rank` is greater than `world_size`.

### Steps to reproduce the bug

Here is the minimal code for reproduction:
```py
>>> from datasets import load_dataset
>>> from datasets.distributed import split_dataset_by_node
>>> dataset = load_dataset('fla-hub/slimpajama-test', split='train', streaming=True)      
>>> dataset = split_dataset_by_node(dataset, 1, 32)
>>> dataset._distributed
DistributedConfig(rank=1, world_size=32)
>>> dataset = split_dataset_by_node(dataset, 1, 15)
>>> dataset._distributed                           
DistributedConfig(rank=481, world_size=480)
```
As you can see, the second rank 481 > 480, which is problematic.

### Expected behavior

I think this error comes from this line @lhoestq
https://github.com/huggingface/datasets/blob/a6ccf944e42c1a84de81bf326accab9999b86c90/src/datasets/iterable_dataset.py#L2943-L2944
We may need to obtain the rank first. Then the above code gives
```py
>>> dataset._distributed                           
DistributedConfig(rank=16, world_size=480)
```

### Environment info

datasets==2.20.0
GitHub issue: Convert polars DataFrame back to datasets
Author: ljw20180420 (NONE)
State: closed | Created: 2024-06-19 11:38:48
Labels: name: enhancement

### Feature request

This returns error.
```python
from datasets import Dataset

dsdf = Dataset.from_dict({"x": [[1, 2], [3, 4, 5]], "y": ["a", "b"]})
Dataset.from_polars(dsdf.to_polars())
```
ValueError: Arrow type large_list<item: int64> does not have a datasets dtype equivalent.

### Motivation

When datasets contain Sequence data type, it will be converted to Arrow type large_list. However, the reverse (from large_list to Sequence) does not work.

### Your contribution

No
GitHub issue: Support NumPy 2.0
Author: NeilGirdhar (CONTRIBUTOR)
State: closed | Created: 2024-06-18 23:30:22
Labels: name: enhancement

### Feature request

Support NumPy 2.0.

### Motivation

NumPy introduces the Array API, which bridges the gap between machine learning libraries.  Many clients of HuggingFace are eager to start using the Array API.

Besides that, NumPy 2 provides a cleaner interface than NumPy 1.

### Tasks

NumPy 2.0 was released for testing so that libraries could ensure compatibility [since mid-March](https://github.com/numpy/numpy/issues/24300#issuecomment-1986815755).  What needs to be done for HuggingFace to support Numpy 2?

- [x] Fix use of `array`: https://github.com/huggingface/datasets/pull/6976
- [ ] Remove [NumPy version limit](https://github.com/huggingface/datasets/pull/6975): https://github.com/huggingface/datasets/pull/6991
GitHub issue: How can I load partial parquet files only?
Author: lucasjinreal (NONE)
State: closed | Created: 2024-06-18 15:44:16

I have a HUGE dataset about 14TB, I unable to download all parquet all. I just take about 100 from it.

dataset = load_dataset("xx/", data_files="data/train-001*-of-00314.parquet")

How can I just using 000 - 100 from a 00314 from all partially?

I search whole net didn't found a solution, **this is stupid if they didn't support it, and I swear I wont using stupid parquet any more**
GitHub issue: Fix regression for pandas < 2.0.0 in JSON loader
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-06-18 10:26:34

A regression was introduced for pandas < 2.0.0 in PR:
- #6914

As described in pandas docs, the `dtype_backend` parameter was first added in pandas 2.0.0: https://pandas.pydata.org/docs/reference/api/pandas.read_json.html

This PR fixes the regression by passing (or not) the `dtype_backend` parameter depending on pandas version.

Maybe, in a future 3.0 `datasets` release, we could just require pandas > 2.0.

Reported by:
- #6977
GitHub issue: Ensure compatibility with numpy 2.0.0
Author: KennethEnevoldsen (CONTRIBUTOR)
State: closed | Created: 2024-06-17 11:29:22

Following the conversion guide, copy=False is no longer required and will result in an error: https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword.

The following fix should resolve the issue.

error found during testing on the MTEB repository e.g. [here](https://github.com/embeddings-benchmark/mteb/pull/938)
GitHub issue: Set temporary numpy upper version < 2.0.0 to fix CI
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-06-17 10:36:54

Set temporary numpy upper version < 2.0.0 to fix CI. See: https://github.com/huggingface/datasets/actions/runs/9546031216/job/26308072017
```
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.
```
GitHub issue: IndexError during training with Squad dataset and T5-small model
Author: ramtunguturi36 (NONE)
State: closed | Created: 2024-06-16 07:53:54

### Describe the bug

I am encountering an IndexError while training a T5-small model on the Squad dataset using the transformers and datasets libraries. The error occurs even with a minimal reproducible example, suggesting a potential bug or incompatibility.

### Steps to reproduce the bug

1.Install the required libraries: !pip install transformers datasets
2.Run the following code:
!pip install transformers datasets

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, TrainingArguments, Trainer, DataCollatorWithPadding

# Load a small, publicly available dataset
from datasets import load_dataset
dataset = load_dataset("squad", split="train[:100]") # Use a small subset for testing

# Load a pre-trained model and tokenizer
model_name = "t5-small"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# Define a basic data collator
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

# Define training arguments
training_args = TrainingArguments(
    output_dir="./results",
    per_device_train_batch_size=2,
    num_train_epochs=1,
)

# Create a trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    data_collator=data_collator,
)

# Train the model
trainer.train()

### Expected behavior

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
[<ipython-input-23-f13a4b23c001>](https://localhost:8080/#) in <cell line: 34>()
     32 
     33 # Train the model
---> 34 trainer.train()

10 frames
[/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py](https://localhost:8080/#) in _check_valid_index_key(key, size)
    427     if isinstance(key, int):
    428         if (key < 0 and key + size < 0) or (key >= size):
--> 429             raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
    430         return
    431     elif isinstance(key, slice):

IndexError: Invalid key: 42 is out of bounds for size 0

### Environment info

transformers version:4.41.2
datasets version:1.18.4
Python version:3.10.12
GitHub issue: Use `HF_HUB_OFFLINE` instead of `HF_DATASETS_OFFLINE`
Author: Wauplin (CONTRIBUTOR)
State: closed | Created: 2024-06-13 14:39:40

To use `datasets` offline, one can use the `HF_DATASETS_OFFLINE` environment variable. This PR makes `HF_HUB_OFFLINE` the recommended environment variable for offline training. Goal is to be more consistent with the rest of HF ecosystem and have a single config value to set.

The changes are backward-compatible meaning that:
- `HF_DATASETS_OFFLINE` environment is still taken into account, though not documented
- `datasets.config.HF_DATASETS_OFFLINE` still exists, though it is not used anymore (in favor of `datasets.config.HF_HUB_OFFLINE`)


**Note:** it might break things in downstream libraries if they were monkeypatching `datasets.config.HF_DATASETS_OFFLINE` in their CI tests (for instance). Not much of a problem IMO.
GitHub issue: Method to load Laion400m
Author: humanely (NONE)
State: open | Created: 2024-06-12 16:04:04
Labels: name: enhancement

### Feature request

Large datasets like Laion400m are provided as embeddings. The provided methods in load_dataset are not straightforward for loading embedding files, i.e. img_emb_XX.npy ; XX = 0 to 99


### Motivation

The trial and experimentation is the key pivot of HF. It would be great if HF can load embeddings files s,ealessly.

### Your contribution

I cam write the loader with some help.
GitHub issue: Remove underlines between badges
Author: novialriptide (CONTRIBUTOR)
State: closed | Created: 2024-06-12 14:32:11

## Before:
<img width="935" alt="image" src="https://github.com/huggingface/datasets/assets/35881688/93666e72-059b-4180-9e1d-ff176a3d9dac">

## After:
<img width="956" alt="image" src="https://github.com/huggingface/datasets/assets/35881688/75df7c3e-f473-44f0-a872-eeecf6a85fe2">
GitHub issue: [Streaming] retry on requests errors
Author: lhoestq (MEMBER)
State: closed | Created: 2024-06-10 15:51:56

reported in https://discuss.huggingface.co/t/speeding-up-streaming-of-large-datasets-fineweb/90714/6  when training using a streaming a dataloader

cc @Wauplin it looks like the retries from `hfh` are not always enough. In this PR I let `datasets` do additional retries (that users can configure in `datasets.config`) since I couldn't find an easy way to increase the max_retries for `hfh` users in general.
GitHub issue: Manual downloads should count as downloads
Author: umarbutler (NONE)
State: open | Created: 2024-06-09 04:52:06
Labels: name: enhancement

### Feature request

I would like to request that manual downloads of data files from Hugging Face dataset repositories count as downloads of a dataset. According to the documentation for the Hugging Face Hub, that is currently not the case: https://huggingface.co/docs/hub/en/datasets-download-stats

### Motivation

This would ensure that downloads are accurately reported to end users.

### Your contribution

N/A
GitHub issue: Better error handling in `dataset_module_factory`
Author: Wauplin (CONTRIBUTOR)
State: closed | Created: 2024-06-07 11:24:15

cc @cakiki who reported it on [slack](https://huggingface.slack.com/archives/C039P47V1L5/p1717754405578539) (private link)

This PR updates how errors are handled in `dataset_module_factory` when the  `dataset_info` cannot be accessed:
1. Use multiple `except ... as e` instead of using `isinstance(e, ...)`
2. Always raise `DatasetNotFoundError` with `from e` so that the initial error is explicitly logged in the stacktrace.
3. Differentiate `RepoNotFoundError` / `GatedRepoError` / `RevisionNotFoundError` cases
GitHub issue: My Private Dataset doesn't exist on the Hub or cannot be accessed
Author: wangguan1995 (NONE)
State: closed | Created: 2024-06-06 06:52:19

### Describe the bug


```
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 1852, in dataset_module_factory
    raise DatasetNotFoundError(msg + f" at revision '{revision}'" if revision else msg)
datasets.exceptions.DatasetNotFoundError: Dataset 'xxx' doesn't exist on the Hub or cannot be accessed
>>> dataset = load_dataset("xxxx", token=True)

404 error 404 Client Error. (Request ID: Root=xxxx)

Repository Not Found for url: https://huggingface.co/api/datasets/xxx/xxx.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 2593, in load_dataset
    builder_instance = load_dataset_builder(
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 2265, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 1910, in dataset_module_factory
    raise e1 from None
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 1852, in dataset_module_factory
    raise DatasetNotFoundError(msg + f" at revision '{revision}'" if revision else msg)
datasets.exceptions.DatasetNotFoundError: Dataset 'xxx' doesn't exist on the Hub or cannot be accessed
```

### Steps to reproduce the bug

123

### Expected behavior

123

### Environment info

123
GitHub issue: Fix typos in docs
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-06-05 10:46:47

Fix typos in docs introduced by:
- #6956

Typos:
- `comparisions` => `comparisons`
- two consecutive sentences both ending in colon
- split one sentence into two

Sorry, I did not have time to review that PR.

CC: @lhoestq
GitHub issue: load_dataset() should load all subsets, if no specific subset is specified
Author: windmaple (NONE)
State: closed | Created: 2024-06-04 11:02:33
Labels: name: enhancement

### Feature request

Currently load_dataset() is forcing users to specify a subset. Example

`from datasets import load_dataset
dataset = load_dataset("m-a-p/COIG-CQIA")`

```---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-10-c0cb49385da6>](https://localhost:8080/#) in <cell line: 2>()
      1 from datasets import load_dataset
----> 2 dataset = load_dataset("m-a-p/COIG-CQIA")

3 frames
[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in _create_builder_config(self, config_name, custom_features, **config_kwargs)
    582                     if not config_kwargs:
    583                         example_of_usage = f"load_dataset('{self.dataset_name}', '{self.BUILDER_CONFIGS[0].name}')"
--> 584                         raise ValueError(
    585                             "Config name is missing."
    586                             f"\nPlease pick one among the available configs: {list(self.builder_configs.keys())}"

ValueError: Config name is missing.
Please pick one among the available configs: ['chinese_traditional', 'coig_pc', 'exam', 'finance', 'douban', 'human_value', 'logi_qa', 'ruozhiba', 'segmentfault', 'wiki', 'wikihow', 'xhs', 'zhihu']
Example of usage:
	`load_dataset('coig-cqia', 'chinese_traditional')`
```
This means a dataset cannot contain all the subsets at the same time. I guess one workaround is to manually specify the subset files like in [here](https://huggingface.co/datasets/m-a-p/COIG-CQIA/discussions/1#658698b44bb41498f75c5622), which is clumsy.




### Motivation

Ideally, if not subset is specified, the API should just try to load all subsets. This makes it much easier to handle datasets w/ subsets.

### Your contribution

Not sure since I'm not familiar w/ the lib src.
GitHub issue: `Dataset.with_format` behaves inconsistently with documentation
Author: iansheng (NONE)
State: closed | Created: 2024-06-04 09:18:32
Labels: name: documentation

### Describe the bug

The actual behavior of the interface `Dataset.with_format` is inconsistent with the documentation.
https://huggingface.co/docs/datasets/use_with_pytorch#n-dimensional-arrays
https://huggingface.co/docs/datasets/v2.19.0/en/use_with_tensorflow#n-dimensional-arrays

> If your dataset consists of N-dimensional arrays, you will see that by default they are considered as nested lists.
> In particular, a PyTorch formatted dataset outputs nested lists instead of a single tensor.
> A TensorFlow formatted dataset outputs a RaggedTensor instead of a single tensor.

But I get a single tensor by default, which is  inconsistent with the description.

Actually the current behavior seems more reasonable to me. Therefore, the document needs to be modified.

### Steps to reproduce the bug

```python
>>> from datasets import Dataset
>>> data = [[[1, 2],[3, 4]],[[5, 6],[7, 8]]]
>>> ds = Dataset.from_dict({"data": data})
>>> ds = ds.with_format("torch")
>>> ds[0]
{'data': tensor([[1, 2],
        [3, 4]])}
>>> ds = ds.with_format("tf")
>>> ds[0]
{'data': <tf.Tensor: shape=(2, 2), dtype=int64, numpy=
array([[1, 2],
       [3, 4]])>}
```

### Expected behavior

```python
>>> from datasets import Dataset
>>> data = [[[1, 2],[3, 4]],[[5, 6],[7, 8]]]
>>> ds = Dataset.from_dict({"data": data})
>>> ds = ds.with_format("torch")
>>> ds[0]
{'data': [tensor([1, 2]), tensor([3, 4])]}
>>> ds = ds.with_format("tf")
>>> ds[0]
{'data': <tf.RaggedTensor [[1, 2], [3, 4]]>}
```

### Environment info

datasets==2.19.1
torch==2.1.0
tensorflow==2.13.1
GitHub issue: load_dataset error
Author: frederichen01 (NONE)
State: closed | Created: 2024-06-04 01:24:45

### Describe the bug

Why does the program get stuck when I use load_dataset method, and it still gets stuck after loading for several hours? In fact, my json file is only 21m, and I can load it in one go using open('', 'r').

### Steps to reproduce the bug

1. pip install datasets==2.19.2
2. from datasets import Dataset, DatasetDict, NamedSplit, Split, load_dataset
3. data = load_dataset('json', data_files='train.json')

### Expected behavior

It is able to load my json correctly

### Environment info

datasets==2.19.2
GitHub issue: to_tf_dataset: Visible devices cannot be modified after being initialized
Author: logasja (NONE)
State: open | Created: 2024-06-03 18:10:57

### Describe the bug

When trying to use to_tf_dataset with a custom data_loader collate_fn when I use parallelism I am met with the following error as many times as number of workers there were in ``num_workers``.

  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/multiprocess/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/datasets/utils/tf_utils.py", line 438, in worker_loop
    tf.config.set_visible_devices([], "GPU")  # Make sure workers don't try to allocate GPU memory
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/tensorflow/python/framework/config.py", line 566, in set_visible_devices
    context.context().set_visible_devices(devices, device_type)
  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/tensorflow/python/eager/context.py", line 1737, in set_visible_devices
    raise RuntimeError(
RuntimeError: Visible devices cannot be modified after being initialized

### Steps to reproduce the bug

1. Download a dataset using HuggingFace load_dataset
2. Define a function that transforms the data in some way to be used in the collate_fn argument
3. Provide a ``batch_size`` and ``num_workers`` value in the ``to_tf_dataset`` function
4. Either retrieve directly or use tfds benchmark to test the dataset

``` python
from datasets import load_datasets
import tensorflow_datasets as tfds
from keras_cv.layers import Resizing

def data_loader(examples):
      x = Resizing(examples[0]['image'], 256, 256, crop_to_aspect_ratio=True)
      return {X[0]: x}

ds = load_datasets("logasja/FDF", split="test")

ds = ds.to_tf_dataset(collate_fn=data_loader, batch_size=16, num_workers=2)

tfds.benchmark(ds)
```

### Expected behavior

Use multiple processes to apply transformations from the collate_fn to the tf dataset on the CPU.

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-6.5.0-1023-oracle-x86_64-with-glibc2.35
- Python version: 3.11.8
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0
GitHub issue: FileNotFoundError：error  when loading C4 dataset
Author: W-215 (NONE)
State: closed | Created: 2024-06-03 13:06:33

### Describe the bug

can't load c4 datasets

When I replace the datasets package to 2.12.2 I get raise datasets.utils.info_utils.ExpectedMoreSplits: {'train'}

How can I fix this？

### Steps to reproduce the bug

1.from datasets import load_dataset
2.dataset = load_dataset('allenai/c4', data_files={'validation': 'en/c4-validation.00003-of-00008.json.gz'}, split='validation')
3.   raise FileNotFoundError(
FileNotFoundError: Couldn't find a dataset script at local_path/c4_val/allenai/c4/c4.py or any data file in the same directory. Couldn't find 'allenai/c4' on the Hugging Face Hub either: FileNotFoundError: Unable to find 'hf://datasets/allenai/c4@1588ec454efa1a09f29cd18ddd04fe05fc8653a2/en/c4-validation.00003-of-00008.json.gz' with any supported extension ['.csv', '.tsv', '.json', '.jsonl', '.parquet', '.geoparquet', '.gpq', '.arrow', '.txt', '.tar', '.blp', '.bmp', '.dib', '.bufr', '.cur', '.pcx', '.dcx', '.dds', '.ps', '.eps', '.fit', '.fits', '.fli', '.flc', '.ftc', '.ftu', '.gbr', '.gif', '.grib', '.h5', '.hdf', '.png', '.apng', '.jp2', '.j2k', '.jpc', '.jpf', '.jpx', '.j2c', '.icns', '.ico', '.im', '.iim', '.tif', '.tiff', '.jfif', '.jpe', '.jpg', '.jpeg', '.mpg', '.mpeg', '.msp', '.pcd', '.pxr', '.pbm', '.pgm', '.ppm', '.pnm', '.psd', '.bw', '.rgb', '.rgba', '.sgi', '.ras', '.tga', '.icb', '.vda', '.vst', '.webp', '.wmf', '.emf', '.xbm', '.xpm', '.BLP', '.BMP', '.DIB', '.BUFR', '.CUR', '.PCX', '.DCX', '.DDS', '.PS', '.EPS', '.FIT', '.FITS', '.FLI', '.FLC', '.FTC', '.FTU', '.GBR', '.GIF', '.GRIB', '.H5', '.HDF', '.PNG', '.APNG', '.JP2', '.J2K', '.JPC', '.JPF', '.JPX', '.J2C', '.ICNS', '.ICO', '.IM', '.IIM', '.TIF', '.TIFF', '.JFIF', '.JPE', '.JPG', '.JPEG', '.MPG', '.MPEG', '.MSP', '.PCD', '.PXR', '.PBM', '.PGM', '.PPM', '.PNM', '.PSD', '.BW', '.RGB', '.RGBA', '.SGI', '.RAS', '.TGA', '.ICB', '.VDA', '.VST', '.WEBP', '.WMF', '.EMF', '.XBM', '.XPM', '.aiff', '.au', '.avr', '.caf', '.flac', '.htk', '.svx', '.mat4', '.mat5', '.mpc2k', '.ogg', '.paf', '.pvf', '.raw', '.rf64', '.sd2', '.sds', '.ircam', '.voc', '.w64', '.wav', '.nist', '.wavex', '.wve', '.xi', '.mp3', '.opus', '.AIFF', '.AU', '.AVR', '.CAF', '.FLAC', '.HTK', '.SVX', '.MAT4', '.MAT5', '.MPC2K', '.OGG', '.PAF', '.PVF', '.RAW', '.RF64', '.SD2', '.SDS', '.IRCAM', '.VOC', '.W64', '.WAV', '.NIST', '.WAVEX', '.WVE', '.XI', '.MP3', '.OPUS', '.zip']
### Expected behavior


The data was successfully imported

### Environment info

python version 3.9
datasets version 2.19.2
GitHub issue: Re-enable import sorting disabled by flake8:noqa directive when using ruff linter
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-06-03 06:24:47

Re-enable import sorting that was wrongly disabled by `flake8: noqa` directive after switching to `ruff` linter in datasets-2.10.0 PR:
- #5519

Note that after the linter switch, we wrongly replaced `flake8: noqa` with `ruff: noqa` in datasets-2.17.0 PR:
- #6619

That replacement was wrong because we kept the `isort: skip` directives although they were indeed disabled by `flake8: noqa` first and by `ruff: noqa` afterwards. See for example `__init__.py` file after the linter switch:
- We kept the `flake8: noqa` directive
https://github.com/huggingface/datasets/blob/06ae3f678651bfbb3ca7dd3274ee2f38e0e0237e/src/datasets/__init__.py#L1
- Whereas we also kept the `isort: skip` directives (that were disabled)
https://github.com/huggingface/datasets/blob/06ae3f678651bfbb3ca7dd3274ee2f38e0e0237e/src/datasets/__init__.py#L82-L84

Fix #6942.
GitHub issue: Import sorting is disabled by flake8 noqa directive after switching to ruff linter
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-06-02 09:43:34
Labels: name: maintenance

When we switched to `ruff` linter in PR:
- #5519

import sorting was disabled in all files containing the `# flake8: noqa` directive
- https://github.com/astral-sh/ruff/issues/11679

We should re-enable import sorting on those files.
GitHub issue: Supporting FFCV: Fast Forward Computer Vision
Author: Luciennnnnnn (NONE)
State: open | Created: 2024-06-01 05:34:52
Labels: name: enhancement

### Feature request

Supporting FFCV, https://github.com/libffcv/ffcv

### Motivation

According to the benchmark, FFCV seems to be fastest image loading method.

### Your contribution

no
GitHub issue: Enable Sharding to Equal Sized Shards
Author: yuvalkirstain (NONE)
State: open | Created: 2024-05-31 21:55:50
Labels: name: enhancement

### Feature request

Add an option when sharding a dataset to have all shards the same size. Will be good to provide both an option of duplication, and by truncation.

### Motivation

Currently the behavior of sharding is "If n % i == l, then the first l shards will have length (n // i) + 1, and the remaining shards will have length (n // i).". However, when using FSDP we want the shards to have the same size. This requires the user to manually handle this situation, but it will be nice if we had an option to shard the dataset into equally sized shards. 

### Your contribution

For now just a PR. I can also add code that does what is needed, but probably not efficient.
Shard to equal size by duplication:
```
remainder = len(dataset) % num_shards
num_missing_examples = num_shards - remainder
duplicated = dataset.select(list(range(num_missing_examples)))
dataset = concatenate_datasets([dataset, duplicated])
shard = dataset.shard(num_shards, shard_idx)
```
Or by truncation:
```
shard = dataset.shard(num_shards, shard_idx)
num_examples_per_shard = len(dataset) // num_shards
shard = shard.select(list(range(num_examples_per_shard)))
```
GitHub issue: ExpectedMoreSplits error when using data_dir
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-31 15:08:42
Labels: name: bug

As reported by @regisss, an `ExpectedMoreSplits` error is raised when passing `data_dir`:
```python
from datasets import load_dataset

dataset = load_dataset(
    "lvwerra/stack-exchange-paired",
    split="train",
    cache_dir=None,
    data_dir="data/rl",
)
```
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2609, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1027, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1140, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/info_utils.py", line 92, in verify_splits
    raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))
datasets.utils.info_utils.ExpectedMoreSplits: {'test'}
```
GitHub issue: Fix expected splits when passing data_files or dir
Author: lhoestq (MEMBER)
State: closed | Created: 2024-05-31 11:04:22

reported on slack:

The following code snippet gives an error with v2.19 but not with v2.18:
from datasets import load_dataset
```
dataset = load_dataset(
    "lvwerra/stack-exchange-paired",
    split="train",
    cache_dir=None,
    data_dir="data/rl",
)
```
and the error is:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2609, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1027, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1140, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/info_utils.py", line 92, in verify_splits
    raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))
datasets.utils.info_utils.ExpectedMoreSplits: {'test'}
```
GitHub issue: save_to_disk() freezes when saving on s3 bucket with multiprocessing
Author: ycattan (NONE)
State: open | Created: 2024-05-30 16:48:39

### Describe the bug

I'm trying to save a `Dataset` using the `save_to_disk()` function with:
- `num_proc > 1`
- `dataset_path` being a s3 bucket path e.g. "s3://{bucket_name}/{dataset_folder}/"

The hf progress bar shows up but the saving does not seem to start. 
When using one processor only (`num_proc=1`), everything works fine.
When saving the dataset on local disk (as opposed to s3 bucket) with `num_proc > 1`, everything works fine.

Thank you for your help! :)

### Steps to reproduce the bug

I tried without any storage options:

```
from datasets import load_dataset

sandbox_ds = load_dataset("openai_humaneval")
sandbox_ds["test"].save_to_disk(
    "s3://bucket-name/test_multiprocessing_saving/",
    num_proc=4,
)
```


and with the specific s3fs storage options: 
```
from datasets import load_dataset
from s3fs import S3FileSystem

def get_s3fs():
    return S3FileSystem()

sandbox_ds = load_dataset("openai_humaneval")
sandbox_ds["test"].save_to_disk(
    "s3://bucket-name/test_multiprocessing_saving/",
    num_proc=4,
    storage_options=get_s3fs().storage_options, # also tried: storage_options=S3FileSystem().storage_options
)
```

I'm guessing I might use `storage_options` parameter wrongly, but I didn't find anything online that made it work.

**NB**: Behavior is the same when trying to save the whole `DatasetDict`.

### Expected behavior

Progress bar fills in and saving is carried out.

### Environment info

`datasets==2.18.0`
GitHub issue: Support for pathlib.Path in datasets 2.19.0
Author: lamyiowce (NONE)
State: open | Created: 2024-05-30 12:53:36

### Describe the bug

After the recent update of `datasets`, Dataset.save_to_disk does not accept a pathlib.Path anymore. It was supported in 2.18.0 and previous versions. Is this intentional? Was it supported before only because of a Python dusk-typing miracle?

### Steps to reproduce the bug

```
from datasets import Dataset
import pathlib

path = pathlib.Path("./my_out_path")
Dataset.from_dict(
    {"text": ["hello world"], "label": [777], "split": ["train"]}
.save_to_disk(path)
```
This results in an error when using datasets 2.19:
```
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/Users/jb/scratch/venv/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 1515, in save_to_disk
    fs, _ = url_to_fs(dataset_path, **(storage_options or {}))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jb/scratch/venv/lib/python3.11/site-packages/fsspec/core.py", line 383, in url_to_fs
    chain = _un_chain(url, kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jb/scratch/venv/lib/python3.11/site-packages/fsspec/core.py", line 323, in _un_chain
    if "::" in path
       ^^^^^^^^^^^^
TypeError: argument of type 'PosixPath' is not iterable
```

Converting to str works, however.
```
Dataset.from_dict(
     {"text": ["hello world"], "label": [777], "split": ["train"]}
).save_to_disk(str(path))
```

### Expected behavior

My dataset gets saved to disk without an error.

### Environment info

aiohttp==3.9.5
aiosignal==1.3.1
attrs==23.2.0
certifi==2024.2.2
charset-normalizer==3.3.2
datasets==2.19.0
dill==0.3.8
filelock==3.14.0
frozenlist==1.4.1
fsspec==2024.3.1
huggingface-hub==0.23.2
idna==3.7
multidict==6.0.5
multiprocess==0.70.16
numpy==1.26.4
packaging==24.0
pandas==2.2.2
pyarrow==16.1.0
pyarrow-hotfix==0.6
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
requests==2.32.3
six==1.16.0
tqdm==4.66.4
typing_extensions==4.12.0
tzdata==2024.1
urllib3==2.2.1
xxhash==3.4.1
yarl==1.9.4
GitHub issue: ValueError: Couldn't infer the same data file format for all splits. Got {'train': ('json', {}), 'validation': (None, {})}
Author: CLL112 (NONE)
State: open | Created: 2024-05-29 12:40:05

### Describe the bug

When I run the code en = load_dataset("allenai/c4", "en", streaming=True), I encounter an error: raise ValueError(f"Couldn't infer the same data file format for all splits. Got {split_modules}") ValueError: Couldn't infer the same data file format for all splits. Got {'train': ('json', {}), 'validation': (None, {})}.
However, running dataset = load_dataset('allenai/c4', streaming=True, data_files={'validation': 'en/c4-validation.00003-of-00008.json.gz'}, split='validation') works fine. What is the issue here?

### Steps to reproduce the bug

run code：
import os
os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'
from datasets import load_dataset

en = load_dataset("allenai/c4", "en", streaming=True)

### Expected behavior

Successfully loaded the dataset.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-6.5.0-28-generic-x86_64-with-glibc2.17
- Python version: 3.8.19
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.0.3
- `fsspec` version: 2024.2.0
GitHub issue: Avoid downloading the whole dataset when only README.me has been touched on hub.
Author: zinc75 (NONE)
State: open | Created: 2024-05-29 10:36:06
Labels: name: enhancement

### Feature request

`datasets.load_dataset()` triggers a new download of the **whole dataset** when the README.md file has been touched on huggingface hub, even if data files / parquet files are the exact same.

I think the current behaviour of the load_dataset function is triggered whenever a change of the hash of latest commit on huggingface hub, but  is there a clever way to only download again the dataset **if and only if** data is modified ?  

### Motivation

The current behaviour is a waste of network bandwidth / disk space / research time.

### Your contribution

I don't have time to submit a PR, but I hope a simple solution will emerge from this issue !
GitHub issue: Fix NonMatchingSplitsSizesError/ExpectedMoreSplits when passing data_dir/data_files in no-code Hub datasets
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-28 13:33:38

Fix `NonMatchingSplitsSizesError` or `ExpectedMoreSplits` error for no-code Hub datasets if the user passes:
- `data_dir`
- `data_files`

The proposed solution is to avoid using exported dataset info (from Parquet exports) in these cases.
Additionally, also if the user passes `revision` other than "main" (so that no network requests are made).

This PR fixes a bug introduced by:
- #6714

Fix #6918, fix #6939.
GitHub issue: Caching map result of DatasetDict.
Author: MostHumble (NONE)
State: open | Created: 2024-05-28 09:07:41

Hi!

I'm currenty using the map function to tokenize a somewhat large dataset, so I need to use the cache to save ~25 mins.

Changing num_proc incduces the recomputation of the map, I'm not sure why and if this is excepted behavior? 

here it says, that cached files are loaded sequentially:

https://github.com/huggingface/datasets/blob/bb2664cf540d5ce4b066365e7c8b26e7f1ca4743/src/datasets/arrow_dataset.py#L3005-L3006

it seems like I can pass in a fingerprint, and load it directly:

https://github.com/huggingface/datasets/blob/bb2664cf540d5ce4b066365e7c8b26e7f1ca4743/src/datasets/arrow_dataset.py#L3108-L3125

**Environment Setup:**

- Python 3.11.9
- datasets  2.19.1 conda-forge
- Linux 6.1.83-1.el9.elrepo.x86_64

**MRE**
```python
fixed raw_datasets
fixed tokenize_function

tokenized_datasets = raw_datasets.map(
                        tokenize_function,
                        batched=True,
                        num_proc=9,
                        remove_columns=['text'],
                        load_from_cache_file= True,
                        desc="Running tokenizer on dataset line_by_line",
                    )


tokenized_datasets = raw_datasets.map(
                        tokenize_function,
                        batched=True,
                        num_proc=5,
                        remove_columns=['text'],
                        load_from_cache_file= True,
                        desc="Running tokenizer on dataset line_by_line",
                    )
```
GitHub issue: Export Parquet Tablet Audio-Set is null bytes in Arrow 
Author: anioji (NONE)
State: open | Created: 2024-05-27 14:27:57

### Describe the bug

Exporting the processed audio inside the table with the dataset.to_parquet function, the object pyarrow {bytes: null, path: "Some/Path"}
At the same time, the same dataset uploaded to the hub has bit arrays

![Screenshot from 2024-05-27 19-14-49](https://github.com/huggingface/datasets/assets/140120605/ddfba089-426f-4659-9df4-7a634c948b9e)
![Screenshot from 2024-05-27 19-12-51](https://github.com/huggingface/datasets/assets/140120605/4cf8c0a1-650e-491b-86c8-b475c284a021)


### Steps to reproduce the bug

1.Get dataset from audio and cast it 
2.Export and push dataset
3.It’s scary to be indignant at the difference in the uploaded dataset and the fact that it was saved locally
```py
from datasets import Dataset, Audio 
df = Dataset.from_csv("./datasets.csv")
df = df.cast_column("audio", Audio(16000))
df.to_parquet("./datasets.parquet")
df.push_to_hub(repo_id="************", token="**********************")
```

You can use "try replicate case" for this
[replicate_packet.zip](https://github.com/huggingface/datasets/files/15457114/replicate_packet.zip)


### Expected behavior

Two parquet tables identical in content. It is obvious?

### Environment info

Python 3.11+ (I try did it in 3.12 and got same result )