kardwalker/my_dataset huggingface dataset:
GitHub issue #7196: concatenate_datasets does not preserve shuffling state
State: open | Created: 2024-10-03T14:30:38Z

### Describe the bug

After concatenate datasets on an iterable dataset, the shuffling state is destroyed, similar to #7156 

This means concatenation cant be used for resolving uneven numbers of samples across devices when using iterable datasets in a distributed setting as discussed in #6623 

I also noticed that the number of shards is the same after concatenation, which I found surprising, but I don't understand the internals well enough to know whether this is actually surprising or not

### Steps to reproduce the bug

```python
import datasets
import torch.utils.data


def gen(shards):
    yield {"shards": shards}


def main():
    dataset1 = datasets.IterableDataset.from_generator(
        gen, gen_kwargs={"shards": list(range(25))}  # TODO: how to understand this?
    )
    dataset2 = datasets.IterableDataset.from_generator(
        gen, gen_kwargs={"shards": list(range(25, 50))}  # TODO: how to understand this?
    )
    dataset1 = dataset1.shuffle(buffer_size=1)
    dataset2 = dataset2.shuffle(buffer_size=1)
    print(dataset1.n_shards)
    print(dataset2.n_shards)

    dataset = datasets.concatenate_datasets(
        [dataset1, dataset2]
    )
    print(dataset.n_shards)
    # dataset = dataset1

    dataloader = torch.utils.data.DataLoader(
        dataset,
        batch_size=8,
        num_workers=0,
    )

    for i, batch in enumerate(dataloader):
        print(batch)
    print("\nNew epoch")

    dataset = dataset.set_epoch(1)

    for i, batch in enumerate(dataloader):
        print(batch)


if __name__ == "__main__":
    main()
```

### Expected behavior

Shuffling state should be preserved

### Environment info

Latest datasets
GitHub issue #7193: Support of num_workers (multiprocessing) in map for IterableDataset
State: open | Created: 2024-10-02T18:34:04Z

### Feature request

Currently, IterableDataset doesn't support setting num_worker in .map(), which results in slow processing here. Could we add support for it? As .map() can be run in the batch fashion (e.g., batch_size is default to 1000 in datasets), it seems to be doable for IterableDataset as the regular Dataset.

### Motivation

Improving data processing efficiency

### Your contribution

Testing
GitHub issue #7192: Add repeat() for iterable datasets
State: open | Created: 2024-10-02T17:48:13Z

### Feature request

It would be useful to be able to straightforwardly repeat iterable datasets indefinitely, to provide complete control over starting and ending of iteration to the user.

An IterableDataset.repeat(n) function could do this automatically

### Motivation

This feature was discussed in this issue https://github.com/huggingface/datasets/issues/7147, and would resolve the need to use the hack of interleave datasets with probability 0 as a simple way to achieve this functionality.

An additional benefit might be the simplification of the use of iterable datasets in a distributed setting:
If the user can assume that datasets will repeat indefinitely, then issues around different numbers of samples appearing on different devices (e.g. https://github.com/huggingface/datasets/issues/6437, https://github.com/huggingface/datasets/issues/6594, https://github.com/huggingface/datasets/issues/6623, https://github.com/huggingface/datasets/issues/6719) can potentially be straightforwardly resolved by simply doing:

ids.repeat(None).take(n_samples_per_epoch)


### Your contribution

I'm not familiar enough with the codebase to assess how straightforward this would be to implement.

If it might be very straightforward, I could possibly have a go.
GitHub issue #7191: Solution to issue: #7080 Modified load_dataset function, so that it prompts the user to select a dataset when subdatasets or splits (train, test) are available
State: closed | Created: 2024-10-02T17:02:45Z

# Feel free to give suggestions please..

### This PR is raised because of issue: https://github.com/huggingface/datasets/issues/7080

![image](https://github.com/user-attachments/assets/8fbc604f-f0a5-4a59-a63e-aa4c26442c83)

### This PR gives solution to https://github.com/huggingface/datasets/issues/7080

1. Checking whether the dataset has splits or subdatasets.
2. Printing the available splits/subdatasets.
3. Asking the user to choose which one to load.
4. Loading only the selected dataset based on the user's input.

### Key Changes:

1. Available Splits/Subdatasets: The code checks for available splits/subdatasets using builder_instance.info.splits.keys().
2. User Prompt: If splits are found, it prints them out and prompts the user to select one.
3. Loading Based on User Input: The dataset is loaded based on the user's choice.

This way, the dataset loading function will interactively prompt the user to select which subdataset or split they want to load instead of automatically loading all of them.
GitHub issue #7190: Datasets conflicts with fsspec 2024.9
State: open | Created: 2024-10-02T16:43:46Z

### Describe the bug

Installing both in latest versions are not possible

`pip install "datasets==3.0.1" "fsspec==2024.9.0"`

But using older version of datasets is ok

`pip install "datasets==1.24.4" "fsspec==2024.9.0"`



### Steps to reproduce the bug

`pip install "datasets==3.0.1" "fsspec==2024.9.0"`


### Expected behavior

install both versions.

### Environment info

debian 11.
python 3.10.15
GitHub issue #7189: Audio preview in dataset viewer for audio array data without a path/filename
State: open | Created: 2024-10-02T16:38:38Z

### Feature request

Huggingface has quite a comprehensive set of guides for [audio datasets](https://huggingface.co/docs/datasets/en/audio_dataset). It seems, however, all these guides assume the audio array data to be decoded/inserted into a HF dataset always originates from individual files. The [Audio-dataclass](https://github.com/huggingface/datasets/blob/3.0.1/src/datasets/features/audio.py#L20) appears designed with this assumption in mind. Looking at its source code it returns a dictionary with the keys `path`, `array` and `sampling_rate`. 

However, sometimes users may have different pipelines where they themselves decode the audio array. This feature request has to do with wishing some clarification in guides on whether it is possible, and in such case how users can insert already decoded audio array data into datasets (pandas DataFrame, HF dataset or whatever) that are later saved as parquet, and still get a functioning audio preview in the dataset viewer. 

Do I perhaps need to write a tempfile of my audio array slice to wav and capture the bytes object with `io.BytesIO` and pass that to `Audio()`? 

### Motivation

I'm working with large audio datasets, and my pipeline reads (decodes) audio from larger files, and slices the relevant portions of audio from that larger file based on metadata I have available. 

The pipeline is designed this way to avoid having to store multiple copies of data, and to avoid having to store tens of millions of small files. 

I tried [test-uploading parquet files](https://huggingface.co/datasets/Lauler/riksdagen_test) where I store the audio array data of decoded slices of audio in an `audio` column with a dictionary with the keys `path`, `array` and `sampling_rate`. But I don't know the secret sauce of what the Huggingface Hub expects and requires to be able to display audio previews correctly. 

### Your contribution

I could contribute a tool agnostic guide of creating HF audio datasets directly as parquet to the HF documentation if there is an interest. Provided you help me figure out the secret sauce of what the dataset viewer expects to display the preview correctly.
GitHub issue #7188: Pin multiprocess<0.70.1 to align with dill<0.3.9
State: closed | Created: 2024-10-02T05:40:18Z

Pin multiprocess<0.70.1 to align with dill<0.3.9.

Note that multiprocess-0.70.1 requires dill-0.3.9: https://github.com/uqfoundation/multiprocess/releases/tag/0.70.17

Fix #7186.
GitHub issue #7187: shard_data_sources() got an unexpected keyword argument 'worker_id'
State: open | Created: 2024-10-02T01:26:35Z

### Describe the bug

```
[rank0]:   File "/home/qinghao/miniconda3/envs/doremi/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 238, in __iter__
[rank0]:     for key_example in islice(self.generate_examples_fn(**gen_kwags), shard_example_idx_start, None):
[rank0]:   File "/home/qinghao/miniconda3/envs/doremi/lib/python3.10/site-packages/datasets/packaged_modules/generator/generator.py", line 32, in _generate_examples
[rank0]:     for idx, ex in enumerate(self.config.generator(**gen_kwargs)):
[rank0]:   File "/home/qinghao/workdir/doremi/doremi/dataloader.py", line 337, in take_data_generator
[rank0]:     for ex in ds:
[rank0]:   File "/home/qinghao/miniconda3/envs/doremi/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1791, in __iter__
[rank0]:     yield from self._iter_pytorch()
[rank0]:   File "/home/qinghao/miniconda3/envs/doremi/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1704, in _iter_pytorch
[rank0]:     ex_iterable = ex_iterable.shard_data_sources(worker_id=worker_info.id, num_workers=worker_info.num_workers)
[rank0]: TypeError: UpdatableRandomlyCyclingMultiSourcesExamplesIterable.shard_data_sources() got an unexpected keyword argument 'worker_id'
```

### Steps to reproduce the bug

IterableDataset cannot use

### Expected behavior

can work on datasets==2.10,  but will raise error for later versions.

### Environment info

datasets==3.0.1
GitHub issue #7186: pinning `dill<0.3.9` without pinning `multiprocess` 
State: closed | Created: 2024-10-01T22:29:32Z

### Describe the bug

The [latest `multiprocess` release](https://github.com/uqfoundation/multiprocess/releases/tag/0.70.17) requires `dill>=0.3.9` which causes issues when installing `datasets` without backtracking during package version resolution. Is it possible to add a pin for multiprocess so something like `multiprocess<=0.70.16` so that the `dill` version is compatible?

### Steps to reproduce the bug

NA

### Expected behavior

NA

### Environment info

NA
GitHub issue #7185: CI benchmarks are broken
State: closed | Created: 2024-10-01T08:16:08Z

Since Aug 30, 2024, CI benchmarks are broken: https://github.com/huggingface/datasets/actions/runs/11108421214/job/30861323975
```
{"level":"error","message":"Resource not accessible by integration","name":"HttpError","request":{"body":"{\"body\":\"<details>\\n<summary>Show benchmarks</summary>\\n\\nPyArrow==8.0.0\\n\\n<details>\\n<summary>Show updated benchmarks!</summary>\\n\\n### Benchmark: benchmark_array_xd.json\\n\\n| metric | read_batch_formatted_as_numpy after write_array2d |
...
"headers":{"accept":"application/vnd.github.v3+json","authorization":"token [REDACTED]","content-type":"application/json; charset=utf-8","user-agent":"octokit-rest.js/18.0.0 octokit-core.js/3.6.0 Node.js/16.20.2 (linux; x64)"},"method":"POST","request":{"agent":{"_events":{},"_eventsCount":2,"cache":
...
"response":{"data":{"documentation_url":"https://docs.github.com/rest/issues/comments#create-an-issue-comment","message":"Resource not accessible by integration","status":"403"},
...
"stack":"HttpError: Resource not accessible by integration\n    at /usr/lib/node_modules/@dvcorg/cml/node_modules/@octokit/request/dist-node/index.js:86:21\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Job.doExecute (/usr/lib/node_modules/@dvcorg/cml/node_modules/bottleneck/light.js:405:18)","status":403}
```
GitHub issue #7184: Pin dill<0.3.9 to fix CI
State: closed | Created: 2024-09-30T14:26:25Z

Pin dill<0.3.9 to fix CI for deps-latest.

Note that dill-0.3.9 was released yesterday Sep 29, 2024:
- https://pypi.org/project/dill/0.3.9/
- https://github.com/uqfoundation/dill/releases/tag/0.3.9

Fix #7183.
GitHub issue #7183: CI is broken for deps-latest
State: closed | Created: 2024-09-30T14:02:07Z

See: https://github.com/huggingface/datasets/actions/runs/11106149906/job/30853879890

```
=========================== short test summary info ============================
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_filter_caching_on_disk - AssertionError: Lists differ: [{'fi[44 chars] {'filename': '/tmp/tmp6xcyyjs4/cache-9533fe2601cd3e48.arrow'}] != [{'fi[44 chars] {'filename': '/tmp/tmp6xcyyjs4/cache-e6e0a8b830976289.arrow'}]

First differing element 1:
{'filename': '/tmp/tmp6xcyyjs4/cache-9533fe2601cd3e48.arrow'}
{'filename': '/tmp/tmp6xcyyjs4/cache-e6e0a8b830976289.arrow'}

  [{'filename': '/tmp/tmp6xcyyjs4/dataset0.arrow'},
-  {'filename': '/tmp/tmp6xcyyjs4/cache-9533fe2601cd3e48.arrow'}]
?                                        ^^^^^ --------

+  {'filename': '/tmp/tmp6xcyyjs4/cache-e6e0a8b830976289.arrow'}]
?                                       ++++++++++ ^^  +
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_caching_on_disk - AssertionError: Lists differ: [{'filename': '/tmp/tmp5gxrti_n/cache-e58d327daec8626f.arrow'}] != [{'filename': '/tmp/tmp5gxrti_n/cache-d87234c5763e54a3.arrow'}]

First differing element 0:
{'filename': '/tmp/tmp5gxrti_n/cache-e58d327daec8626f.arrow'}
{'filename': '/tmp/tmp5gxrti_n/cache-d87234c5763e54a3.arrow'}

- [{'filename': '/tmp/tmp5gxrti_n/cache-e58d327daec8626f.arrow'}]
?                                         ^^ -----------

+ [{'filename': '/tmp/tmp5gxrti_n/cache-d87234c5763e54a3.arrow'}]
?                                       +++++++++++  ^^
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_regex - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::RecurseHashTest::test_hash_ignores_line_definition_of_function - AssertionError: '52e56ee04ad92499' != '0a4f75cec280f634'
- 52e56ee04ad92499
+ 0a4f75cec280f634
FAILED tests/test_fingerprint.py::RecurseHashTest::test_hash_ipython_function - AssertionError: 'a6bd2041ca63d6c0' != '517bf36b7eecdef5'
- a6bd2041ca63d6c0
+ 517bf36b7eecdef5
FAILED tests/test_fingerprint.py::HashingTest::test_hash_tiktoken_encoding - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_compiled_module - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_generator - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_tensor - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_set_doesnt_depend_on_order - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_set_stable - NameError: name 'log' is not defined
ERROR tests/test_iterable_dataset.py::test_iterable_dataset_from_file - NameError: name 'log' is not defined
= 11 failed, 2850 passed, 3 skipped, 23 warnings, 1 error in 191.06s (0:03:11) =
```
GitHub issue #7182: Support features in metadata configs
State: closed | Created: 2024-09-30T11:14:53Z

Support features in metadata configs, like:
```
configs:
  - config_name: default
    features:
      - name: id
        dtype: int64
      - name:  name
        dtype: string
      - name: score
        dtype: float64
```

This will allow to avoid inference of data types.

Currently, we allow passing this information in the `dataset_info` (instead of `configs`) field, but this is not intuitive and it is not properly documented.

TODO:
- [ ] Document usage
GitHub issue #7180: Memory leak when wrapping datasets into PyTorch Dataset without explicit deletion
State: closed | Created: 2024-09-28T14:00:47Z

### Describe the bug

I've encountered a memory leak when wrapping the HuggingFace dataset into a PyTorch Dataset. The RAM usage constantly increases during iteration if items are not explicitly deleted after use.








### Steps to reproduce the bug


Steps to reproduce:

Create a PyTorch Dataset wrapper for 'nebula/cc12m':
````
from torch.utils.data import Dataset
from tqdm import tqdm
from datasets import load_dataset
from torchvision import transforms

Image.MAX_IMAGE_PIXELS = None

class CC12M(Dataset):
    def __init__(self, path_or_name='nebula/cc12m', split='train', transform=None, single_caption=True):
        self.raw_dataset = load_dataset(path_or_name)[split]

        if transform is None:
            self.transform = transforms.Compose([
                transforms.Resize((224, 224)),
                transforms.CenterCrop(224),
                transforms.ToTensor(),
                transforms.Normalize(
                    mean=[0.48145466, 0.4578275, 0.40821073],
                    std=[0.26862954, 0.26130258, 0.27577711]
                )
            ])
        else:
            self.transform = transforms.Compose(transform)

        self.single_caption = single_caption
        self.length = len(self.raw_dataset)

    def __len__(self):
        return self.length

    def __getitem__(self, index):
        item = self.raw_dataset[index]
        caption = item['txt']
        with io.BytesIO(item['webp']) as buffer:
            image = Image.open(buffer).convert('RGB')
            if self.transform:
                image = self.transform(image)
        # del item  # Uncomment this line to prevent the memory leak
        return image, caption
````

Iterate through the dataset without the del item line in __getitem__.

Observe RAM usage increasing constantly.

Add del item at the end of __getitem__:

```
def __getitem__(self, index):
    item = self.raw_dataset[index]
    caption = item['txt']
    with io.BytesIO(item['webp']) as buffer:
        image = Image.open(buffer).convert('RGB')
        if self.transform:
            image = self.transform(image)
    del item  # This line prevents the memory leak
    return image, caption
```


Iterate through the dataset again and observe that RAM usage remains stable.



### Expected behavior


Expected behavior:
RAM usage should remain stable during iteration without needing to explicitly delete items.

Actual behavior:
RAM usage constantly increases unless items are explicitly deleted after use

### Environment info

- `datasets` version: 2.21.0
- Platform: Linux-4.18.0-513.5.1.el8_9.x86_64-x86_64-with-glibc2.28
- Python version: 3.12.4
- `huggingface_hub` version: 0.24.6
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.6.1
GitHub issue #7175: [FSTimeoutError] load_dataset
State: closed | Created: 2024-09-26T15:42:29Z

### Describe the bug

When using `load_dataset`to load [HuggingFaceM4/VQAv2](https://huggingface.co/datasets/HuggingFaceM4/VQAv2), I am getting `FSTimeoutError`. 

### Error

```
TimeoutError: 

The above exception was the direct cause of the following exception:

FSTimeoutError                            Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/fsspec/asyn.py](https://klh9mr78js-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240924-060116_RC00_678132060#) in sync(loop, func, timeout, *args, **kwargs)
     99     if isinstance(return_result, asyncio.TimeoutError):
    100         # suppress asyncio.TimeoutError, raise FSTimeoutError
--> 101         raise FSTimeoutError from return_result
    102     elif isinstance(return_result, BaseException):
    103         raise return_result

FSTimeoutError:
```

It usually fails around 5-6 GB.

<img width="847" alt="Screenshot 2024-09-26 at 9 10 19 PM" src="https://github.com/user-attachments/assets/ff91995a-fb55-4de6-8214-94025d6c8470">


### Steps to reproduce the bug

To reproduce it, run this in colab notebook:

```
!pip install -q -U datasets

from datasets import load_dataset
ds = load_dataset('HuggingFaceM4/VQAv2', split="train[:10%]")
```



### Expected behavior

It should download properly.

### Environment info

Using Colab Notebook.
GitHub issue #7172: Add torchdata as a regular test dependency
State: closed | Created: 2024-09-26T07:45:55Z

Add `torchdata` as a regular test dependency.

Note that previously, `torchdata` was installed from their repo and current main branch (0.10.0.dev) requires Python>=3.9.

Also note they made a recent release: 0.8.0 on Jul 31, 2024.

Fix #7171.
GitHub issue #7171: CI is broken: No solution found when resolving dependencies
State: closed | Created: 2024-09-26T07:24:58Z

See: https://github.com/huggingface/datasets/actions/runs/11046967444/job/30687294297
```
Run uv pip install --system -r additional-tests-requirements.txt --no-deps
  × No solution found when resolving dependencies:
  ╰─▶ Because the current Python version (3.8.18) does not satisfy Python>=3.9
      and torchdata==0.10.0a0+1a98f21 depends on Python>=3.9, we can conclude
      that torchdata==0.10.0a0+1a98f21 cannot be used.
      And because only torchdata==0.10.0a0+1a98f21 is available and
      you require torchdata, we can conclude that your requirements are
      unsatisfiable.
Error: Process completed with exit code 1.
```
GitHub issue #7170: Support JSON lines with missing columns
State: closed | Created: 2024-09-25T05:08:15Z

Support JSON lines with missing columns.

Fix #7169.

The implemented test raised:
```
datasets.table.CastError: Couldn't cast
age: int64
to
{'age': Value(dtype='int32', id=None), 'name': Value(dtype='string', id=None)}
because column names don't match
```

Related to:
- #7160 
- #7162
GitHub issue #7167: Error Mapping on sd3, sdxl and upcoming flux controlnet training scripts in diffusers
State: closed | Created: 2024-09-25T01:39:51Z

### Describe the bug

```
Map:   6%|██████                                                                                                   | 8000/138120 [19:27<5:16:36,  6.85 examples/s]
Traceback (most recent call last):
  File "/workspace/diffusers/examples/controlnet/train_controlnet_sd3.py", line 1416, in <module>
    main(args)
  File "/workspace/diffusers/examples/controlnet/train_controlnet_sd3.py", line 1132, in main
    train_dataset = train_dataset.map(compute_embeddings_fn, batched=True, new_fingerprint=new_fingerprint)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/arrow_dataset.py", line 560, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/arrow_dataset.py", line 3035, in map
    for rank, done, content in Dataset._map_single(**dataset_kwargs):
  File "/usr/local/lib/python3.11/dist-packages/datasets/arrow_dataset.py", line 3461, in _map_single
    writer.write_batch(batch)
  File "/usr/local/lib/python3.11/dist-packages/datasets/arrow_writer.py", line 567, in write_batch
    self.write_table(pa_table, writer_batch_size)
  File "/usr/local/lib/python3.11/dist-packages/datasets/arrow_writer.py", line 579, in write_table
    pa_table = pa_table.combine_chunks()
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/table.pxi", line 4387, in pyarrow.lib.Table.combine_chunks
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/usr/local/lib/python3.11/dist-packages/accelerate/commands/launch.py", line 1174, in launch_command
    simple_launcher(args)
  File "/usr/local/lib/python3.11/dist-packages/accelerate/commands/launch.py", line 769, in simple_launcher
  ```

### Steps to reproduce the bug

The dataset has no problem training on sd1.5 controlnet train script



### Expected behavior

Script not randomly erroing with error above

### Environment info

- `datasets` version: 3.0.0
- Platform: Linux-6.5.0-44-generic-x86_64-with-glibc2.35
- Python version: 3.11.9
- `huggingface_hub` version: 0.25.1
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1


training on A100
GitHub issue #7163: Set explicit seed in iterable dataset ddp shuffling example 
State: closed | Created: 2024-09-23T11:34:06Z

### Describe the bug

In the examples section of the iterable dataset docs https://huggingface.co/docs/datasets/en/package_reference/main_classes#datasets.IterableDataset
the ddp example shuffles without seeding

```python
from datasets.distributed import split_dataset_by_node
ids = ds.to_iterable_dataset(num_shards=512)
ids = ids.shuffle(buffer_size=10_000)  # will shuffle the shards order and use a shuffle buffer when you start iterating
ids = split_dataset_by_node(ds, world_size=8, rank=0)  # will keep only 512 / 8 = 64 shards from the shuffled lists of shards when you start iterating
dataloader = torch.utils.data.DataLoader(ids, num_workers=4)  # will assign 64 / 4 = 16 shards from this node's list of shards to each worker when you start iterating
for example in ids:
    pass
```

This code would - I think - raise an error due to the lack of an explicit seed:

https://github.com/huggingface/datasets/blob/2eb4edb97e1a6af2ea62738ec58afbd3812fc66e/src/datasets/iterable_dataset.py#L1707-L1711


### Steps to reproduce the bug

Run example code

### Expected behavior

Add explicit seeding to example code

### Environment info

latest datasets
GitHub issue #7161: JSON lines with empty struct raise ArrowTypeError
State: closed | Created: 2024-09-23T08:48:56Z

JSON lines with empty struct raise ArrowTypeError: struct fields don't match or are in the wrong order

See example: https://huggingface.co/datasets/wikimedia/structured-wikipedia/discussions/5

> ArrowTypeError: struct fields don't match or are in the wrong order: Input fields: struct<> output fields: struct<pov_count: int64, update_count: int64, citation_needed_count: int64>

Related to:
- #7159
GitHub issue #7160: Support JSON lines with missing struct fields
State: closed | Created: 2024-09-23T08:04:09Z

Support JSON lines with missing struct fields.

Fix #7159.

The implemented test raised:
```
TypeError: Couldn't cast array of type
struct<age: int64>
to
{'age': Value(dtype='int32', id=None), 'name': Value(dtype='string', id=None)}
```
GitHub issue #7156: interleave_datasets resets shuffle state
State: open | Created: 2024-09-20T17:57:54Z

### Describe the bug

```
import datasets
import torch.utils.data


def gen(shards):
    yield {"shards": shards}


def main():
    dataset = datasets.IterableDataset.from_generator(
        gen,
        gen_kwargs={'shards': list(range(25))}
    )
    dataset = dataset.shuffle(buffer_size=1)
    dataset = datasets.interleave_datasets(
        [dataset, dataset], probabilities=[1, 0], stopping_strategy="all_exhausted"
    )

    dataloader = torch.utils.data.DataLoader(
        dataset,
        batch_size=8,
        num_workers=8,
    )

    for i, batch in enumerate(dataloader):
        print(batch)
        if i >= 10:
            break


if __name__ == "__main__":
     main()
```

### Steps to reproduce the bug

Run the script, it will output

```
{'shards': [tensor([ 0,  8, 16, 24,  0,  8, 16, 24])]}
{'shards': [tensor([ 1,  9, 17,  1,  9, 17,  1,  9])]}
{'shards': [tensor([ 2, 10, 18,  2, 10, 18,  2, 10])]}
{'shards': [tensor([ 3, 11, 19,  3, 11, 19,  3, 11])]}
{'shards': [tensor([ 4, 12, 20,  4, 12, 20,  4, 12])]}
{'shards': [tensor([ 5, 13, 21,  5, 13, 21,  5, 13])]}
{'shards': [tensor([ 6, 14, 22,  6, 14, 22,  6, 14])]}
{'shards': [tensor([ 7, 15, 23,  7, 15, 23,  7, 15])]}
{'shards': [tensor([ 0,  8, 16, 24,  0,  8, 16, 24])]}
{'shards': [tensor([17,  1,  9, 17,  1,  9, 17,  1])]}
{'shards': [tensor([18,  2, 10, 18,  2, 10, 18,  2])]}
```

### Expected behavior

The shards should be shuffled.

### Environment info

- `datasets` version: 3.0.0
- Platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.25.0
- PyArrow version: 17.0.0
- Pandas version: 2.0.3
- `fsspec` version: 2023.6.0
GitHub issue #7155: Dataset viewer not working! Failure due to more than 32 splits.
State: closed | Created: 2024-09-18T12:43:21Z

Hello guys, 

I have a dataset and I didn't know I couldn't upload more than 32 splits. Now, my dataset viewer is not working. I don't have the dataset locally on my node anymore and recreating would take a week. And I have to publish the dataset coming Monday. I read about the practice, how I can resolve it and avoid this issue in the future. But, at the moment I need a hard fix for two of my datasets. 

And I don't want to mess or change anything and allow everyone in public to see the dataset and interact with it. Can you please help me?

https://huggingface.co/datasets/laion/Wikipedia-X
https://huggingface.co/datasets/laion/Wikipedia-X-Full
GitHub issue #7150: WebDataset loader splits keys differently than WebDataset library
State: closed | Created: 2024-09-16T06:02:47Z

As reported by @ragavsachdeva (see discussion here: https://github.com/huggingface/datasets/pull/7144#issuecomment-2348307792), our webdataset loader is not aligned with the `webdataset` library when splitting keys from filenames.

For example, we get a different key splitting for filename `/some/path/22.0/1.1.png`:
- datasets library: `/some/path/22` and `0/1.1.png`
- webdataset library: `/some/path/22.0/1`, `1.png`

```python
import webdataset as wds

wds.tariterators.base_plus_ext("/some/path/22.0/1.1.png")
# ('/some/path/22.0/1', '1.png')
```
GitHub issue #7149: Datasets Unknown Keyword Argument Error - task_templates
State: closed | Created: 2024-09-13T10:30:57Z

### Describe the bug

Issue

```python
from datasets import load_dataset
examples = load_dataset('facebook/winoground', use_auth_token=<YOUR USER ACCESS TOKEN>)
```

Gives error

```
TypeError: DatasetInfo.__init__() got an unexpected keyword argument 'task_templates'
```

A simple downgrade to lower `datasets v 2.21.0` solves it.




### Steps to reproduce the bug

1. `pip install datsets`
2.
```python
from datasets import load_dataset
examples = load_dataset('facebook/winoground', use_auth_token=<YOUR USER ACCESS TOKEN>)
```



### Expected behavior

Should load the dataset correctly.

### Environment info

- Datasets version  `3.0.0`
- `transformers` version: 4.45.0.dev0
- Platform: Linux-6.8.0-40-generic-x86_64-with-glibc2.35
- Python version: 3.12.4
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.5
- Accelerate version: 0.35.0.dev0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
GitHub issue #7148: Bug: Error when downloading mteb/mtop_domain
State: closed | Created: 2024-09-13T04:09:39Z

### Describe the bug

When downloading the dataset "mteb/mtop_domain", ran into the following error:

```
Traceback (most recent call last):
  File "/share/project/xzy/test/test_download.py", line 3, in <module>
    data = load_dataset("mteb/mtop_domain", "en", trust_remote_code=True)
  File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 2606, in load_dataset
    builder_instance = load_dataset_builder(
  File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 2277, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1923, in dataset_module_factory
    raise e1 from None
  File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1896, in dataset_module_factory
    ).get_module()
  File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1507, in get_module
    local_path = self.download_loading_script()
  File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1467, in download_loading_script
    return cached_path(file_path, download_config=download_config)
  File "/opt/conda/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 211, in cached_path
    output_path = get_from_cache(
  File "/opt/conda/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 689, in get_from_cache
    fsspec_get(
  File "/opt/conda/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 395, in fsspec_get
    fs.get_file(path, temp_file.name, callback=callback)
  File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/hf_file_system.py", line 648, in get_file
    http_get(
  File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 578, in http_get
    raise EnvironmentError(
OSError: Consistency check failed: file should be of size 2191 but has size 2190 ((…)ets/mteb/mtop_domain@main/mtop_domain.py).
We are sorry for the inconvenience. Please retry with `force_download=True`.
If the issue persists, please let us know by opening an issue on https://github.com/huggingface/huggingface_hub.
```
Try to download through HF datasets directly but got the same error as above.
```python
from datasets import load_dataset

data = load_dataset("mteb/mtop_domain", "en")
```

### Steps to reproduce the bug

```python
from datasets import load_dataset

data = load_dataset("mteb/mtop_domain", "en", force_download=True)
```
With and without `force_download=True` both ran into the same error.

### Expected behavior

Should download the dataset successfully.

### Environment info

- datasets version: 2.21.0
- huggingface-hub version: 0.24.6
GitHub issue #7147: IterableDataset strange deadlock
State: closed | Created: 2024-09-12T18:59:33Z

### Describe the bug

```
import datasets
import torch.utils.data


num_shards = 1024


def gen(shards):
    for shard in shards:
        if shard < 25:
            yield {"shard": shard}


def main():
    dataset = datasets.IterableDataset.from_generator(
        gen,
        gen_kwargs={"shards": list(range(num_shards))},
    )
    dataset = dataset.shuffle(buffer_size=1)
    dataset = datasets.interleave_datasets(
        [dataset, dataset], probabilities=[1, 0], stopping_strategy="all_exhausted"
    )
    dataset = dataset.shuffle(buffer_size=1)

    dataloader = torch.utils.data.DataLoader(
        dataset,
        batch_size=8,
        num_workers=8,
    )

    for i, batch in enumerate(dataloader):
        print(batch)
        if i >= 10:
            break
    print()


if __name__ == "__main__":
    for _ in range(100):
        main()
```

### Steps to reproduce the bug

Running the script above, at some point it will freeze.

- Changing `num_shards` from 1024 to 25 avoids the issue
- Commenting out the final shuffle avoids the issue
- Commenting out the interleave_datasets call avoids the issue

As an aside, if you comment out just the final shuffle, the output from interleave_datasets is not shuffled at all even though there's the shuffle before it. So something about that shuffle config is not being propagated to interleave_datasets.

### Expected behavior

The script should not freeze.

### Environment info

- `datasets` version: 3.0.0
- Platform: macOS-14.6.1-arm64-arm-64bit
- Python version: 3.12.5
- `huggingface_hub` version: 0.24.7
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.6.1


I observed this with 2.21.0 initially, then tried upgrading to 3.0.0 and could still repro.
GitHub issue #7144: Fix key error in webdataset
State: closed | Created: 2024-09-11T10:50:17Z

I was running into

```
example[field_name] = {"path": example["__key__"] + "." + field_name, "bytes": example[field_name]}
KeyError: 'png'
```

The issue is that a filename may have multiple "." e.g. `22.05.png`. Changing `split` to `rsplit` fixes it.

Related https://github.com/huggingface/datasets/issues/6880
GitHub issue #7143: Modify add_column() to optionally accept a FeatureType as param
State: closed | Created: 2024-09-08T10:56:57Z

Fix #7142.

**Before (Add + Cast)**:

```
from datasets import load_dataset, Value
ds = load_dataset("rotten_tomatoes", split="test")
lst = [i for i in range(len(ds))]

ds = ds.add_column("new_col", lst)
# Assigns int64 to new_col by default
print(ds.features)

ds = ds.cast_column("new_col", Value(dtype="uint16", id=None))
print(ds.features)
```


**Before (Numpy Workaround)**:

```
from datasets import load_dataset
import numpy as np
ds = load_dataset("rotten_tomatoes", split="test")
lst = [i for i in range(len(ds))]

ds = ds.add_column("new_col", np.array(lst, dtype=np.uint16))
print(ds.features)
```

**After**:
```
from datasets import load_dataset, Value
ds = load_dataset("rotten_tomatoes", split="test")
lst = [i for i in range(len(ds))]
val = Value(dtype="uint16", id=None))
ds = ds.add_column("new_col", lst, feature=val)
print(ds.features)
```
GitHub issue #7142: Specifying datatype when adding a column to a dataset.
State: closed | Created: 2024-09-08T07:34:24Z

### Feature request

There should be a way to specify the datatype of a column in `datasets.add_column()`. 

### Motivation

To specify a custom datatype, we have to use `datasets.add_column()` followed by `datasets.cast_column()` which is slow for large datasets. Another workaround is to pass a `numpy.array()` of desired type to the `datasets.add_column()` function.

IMO this functionality should be natively supported.

https://discuss.huggingface.co/t/add-column-with-a-particular-type-in-datasets/95674



### Your contribution

I can submit a PR for this.
GitHub issue #7141: Older datasets throwing safety errors with 2.21.0
State: closed | Created: 2024-09-06T16:26:30Z

### Describe the bug

The dataset loading was throwing some safety errors for this popular dataset `wmt14`. 

[in]:

```
import datasets

# train_data = datasets.load_dataset("wmt14", "de-en", split="train")
train_data = datasets.load_dataset("wmt14", "de-en", split="train")
val_data = datasets.load_dataset("wmt14", "de-en", split="validation[:10%]")
```


[out]:

```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
[<ipython-input-9-445f0ecc4817>](https://localhost:8080/#) in <cell line: 4>()
      2 
      3 # train_data = datasets.load_dataset("wmt14", "de-en", split="train")
----> 4 train_data = datasets.load_dataset("wmt14", "de-en", split="train")
      5 val_data = datasets.load_dataset("wmt14", "de-en", split="validation[:10%]")

12 frames
[/usr/local/lib/python3.10/dist-packages/huggingface_hub/hf_api.py](https://localhost:8080/#) in __init__(self, **kwargs)
    636         if security is not None:
    637             security = BlobSecurityInfo(
--> 638                 safe=security["safe"], av_scan=security["avScan"], pickle_import_scan=security["pickleImportScan"]
    639             )
    640         self.security = security

KeyError: 'safe'
```


### Steps to reproduce the bug

See above.

### Expected behavior

Dataset properly loaded.

### Environment info

version: 2.21.0
GitHub issue #7139: Use load_dataset to load imagenet-1K But find a empty dataset
State: open | Created: 2024-09-05T15:12:22Z

### Describe the bug

```python
def get_dataset(data_path, train_folder="train", val_folder="val"):
    traindir = os.path.join(data_path, train_folder)
    valdir = os.path.join(data_path, val_folder)

    def transform_val_examples(examples):
        transform = Compose([
            Resize(256),
            CenterCrop(224),
            ToTensor(),
        ])
        examples["image"] = [transform(image.convert("RGB")) for image in examples["image"]]
        return examples

    def transform_train_examples(examples):
        transform = Compose([
            RandomResizedCrop(224),
            RandomHorizontalFlip(),
            ToTensor(),
        ])
        examples["image"] = [transform(image.convert("RGB")) for image in examples["image"]]
        return examples

    # @fengsicheng: This way is very slow for big dataset like ImageNet-1K (but can pass the network problem using local dataset)
    # train_set = load_dataset("imagefolder", data_dir=traindir, num_proc=4)
    # test_set = load_dataset("imagefolder", data_dir=valdir, num_proc=4)

    train_set = load_dataset("imagenet-1K", split="train", trust_remote_code=True)                                                                                                                                                                                                            
    test_set = load_dataset("imagenet-1K", split="test", trust_remote_code=True)

    print(train_set["label"])

    train_set.set_transform(transform_train_examples)
    test_set.set_transform(transform_val_examples)

    return train_set, test_set
```
    above the code, but output of the print is a list of None:
    
<img width="952" alt="image" src="https://github.com/user-attachments/assets/c4e2fdd8-3b8f-481e-8f86-9bbeb49d79fb">


### Steps to reproduce the bug

1. just ran the code 
2. see the print


### Expected behavior

I do not know how to fix this, can anyone provide help or something? It is hurry for me

### Environment info

- `datasets` version: 2.21.0
- Platform: Linux-5.4.0-190-generic-x86_64-with-glibc2.31
- Python version: 3.10.14
- `huggingface_hub` version: 0.24.6
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.6.1
GitHub issue #7138: Cache only changed columns?
State: open | Created: 2024-09-05T12:56:47Z

### Feature request

Cache only the actual changes to the dataset i.e. changed columns.

### Motivation

I realized that caching actually saves the complete dataset again.
This is especially problematic for image datasets if one wants to only change another column e.g. some metadata and then has to save 5 TB again.

### Your contribution

Is this even viable in the current architecture of the package?
I quickly looked into it and it seems it would require significant changes.

I would spend some time looking into this but maybe somebody could help with the feasibility and some plan to implement before spending too much time on it?
GitHub issue #7137: [BUG] dataset_info sequence unexpected behavior in README.md YAML
State: open | Created: 2024-09-05T06:06:06Z

### Describe the bug

When working on `dataset_info` yaml, I find my data column with format `list[dict[str, str]]` cannot be coded correctly.

My data looks like
```
{"answers":[{"text": "ADDRESS", "label": "abc"}]}
```

My `dataset_info` in README.md is:
```
dataset_info:
- config_name: default
  features:
  - name: answers
    sequence:
    - name: text
      dtype: string
    - name: label
      dtype: string
```

**Error log**:
```
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from list<item: struct<text: string, label: string>> to struct using function cast_struct
```

## Potential Reason
After some analysis, it turns out that my yaml config is requiring `dict[str, list[str]]` instead of `list[dict[str, str]]`. It would work if I change my data to
```
{"answers":{"text": ["ADDRESS"], "label": ["abc", "def"]}}
```

These following 2 different `dataset_info` are actually equivalent.

```
dataset_info:
- config_name: default
  features:
  - name: answers
    dtype:
    - name: text
      sequence: string
    - name: label
      sequence: string

dataset_info:
- config_name: default
  features:
  - name: answers
    sequence:
    - name: text
      dtype: string
    - name: label
      dtype: string
```



### Steps to reproduce the bug


```
# README.md
---
dataset_info:
- config_name: default
  features:
  - name: answers
    sequence:
    - name: text
      dtype: string
    - name: label
      dtype: string
configs:
- config_name: default
  default: true
  data_files:
  - split: train
    path:
    - "test.jsonl"
---



# test.jsonl

# expected but not working
{"answers":[{"text": "ADDRESS", "label": "abc"}]}

# unexpected but working
{"answers":{"text": ["ADDRESS"], "label": ["abc", "def"]}}

```

### Expected behavior

```
dataset_info:
- config_name: default
  features:
  - name: answers
    sequence:
    - name: text
      dtype: string
    - name: label
      dtype: string
```
Should work on following data format:

```
{"answers":[{"text":"ADDRESS", "label": "abc"}]}
```

### Environment info

- `datasets` version: 2.21.0
- Platform: macOS-14.6.1-arm64-arm-64bit
- Python version: 3.12.4
- `huggingface_hub` version: 0.24.5
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.6.1
GitHub issue #7136: Do not consume unnecessary memory during sharding
State: open | Created: 2024-09-04T19:26:06Z

When sharding `IterableDataset`s, a temporary list is created that is then indexed. There is no need to create a temporary list of a potentially very large step/world size, with standard `islice` functionality, so we avoid it.

```shell
pytest tests/test_distributed.py -k iterable
```
Runs successfully.
GitHub issue #7135: Bug: Type Mismatch in Dataset Mapping
State: open | Created: 2024-09-03T16:37:01Z

# Issue: Type Mismatch in Dataset Mapping

## Description
There is an issue with the `map` function in the `datasets` library where the mapped output does not reflect the expected type change. After applying a mapping function to convert an integer label to a string, the resulting type remains an integer instead of a string.

## Reproduction Code
Below is a Python script that demonstrates the problem:

```python
from datasets import Dataset

# Original data
data = {
    'text': ['Hello', 'world', 'this', 'is', 'a', 'test'],
    'label': [0, 1, 0, 1, 1, 0]
}

# Creating a Dataset object
dataset = Dataset.from_dict(data)

# Mapping function to convert label to string
def add_one(example):
    example['label'] = str(example['label'])
    return example

# Applying the mapping function
dataset = dataset.map(add_one)

# Iterating over the dataset to show results
for item in dataset:
    print(item)
    print(type(item['label']))
```

## Expected Output
After applying the mapping function, the expected output should have the `label` field as strings:

```plaintext
{'text': 'Hello', 'label': '0'}
<class 'str'>
{'text': 'world', 'label': '1'}
<class 'str'>
{'text': 'this', 'label': '0'}
<class 'str'>
{'text': 'is', 'label': '1'}
<class 'str'>
{'text': 'a', 'label': '1'}
<class 'str'>
{'text': 'test', 'label': '0'}
<class 'str'>
```

## Actual Output
The actual output still shows the `label` field values as integers:

```plaintext
{'text': 'Hello', 'label': 0}
<class 'int'>
{'text': 'world', 'label': 1}
<class 'int'>
{'text': 'this', 'label': 0}
<class 'int'>
{'text': 'is', 'label': 1}
<class 'int'>
{'text': 'a', 'label': 1}
<class 'int'>
{'text': 'test', 'label': 0}
<class 'int'>
```

## Why necessary
In the case of Image process we often need to convert PIL to tensor with same column name.

Thank for every dev who review this issue. 🤗