Kokolipa/github-issues huggingface dataset:
Issue: CI is broken for deps-latest
State: closed

See: https://github.com/huggingface/datasets/actions/runs/11106149906/job/30853879890

```
=========================== short test summary info ============================
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_filter_caching_on_disk - AssertionError: Lists differ: [{'fi[44 chars] {'filename': '/tmp/tmp6xcyyjs4/cache-9533fe2601cd3e48.arrow'}] != [{'fi[44 chars] {'filename': '/tmp/tmp6xcyyjs4/cache-e6e0a8b830976289.arrow'}]

First differing element 1:
{'filename': '/tmp/tmp6xcyyjs4/cache-9533fe2601cd3e48.arrow'}
{'filename': '/tmp/tmp6xcyyjs4/cache-e6e0a8b830976289.arrow'}

  [{'filename': '/tmp/tmp6xcyyjs4/dataset0.arrow'},
-  {'filename': '/tmp/tmp6xcyyjs4/cache-9533fe2601cd3e48.arrow'}]
?                                        ^^^^^ --------

+  {'filename': '/tmp/tmp6xcyyjs4/cache-e6e0a8b830976289.arrow'}]
?                                       ++++++++++ ^^  +
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_caching_on_disk - AssertionError: Lists differ: [{'filename': '/tmp/tmp5gxrti_n/cache-e58d327daec8626f.arrow'}] != [{'filename': '/tmp/tmp5gxrti_n/cache-d87234c5763e54a3.arrow'}]

First differing element 0:
{'filename': '/tmp/tmp5gxrti_n/cache-e58d327daec8626f.arrow'}
{'filename': '/tmp/tmp5gxrti_n/cache-d87234c5763e54a3.arrow'}

- [{'filename': '/tmp/tmp5gxrti_n/cache-e58d327daec8626f.arrow'}]
?                                         ^^ -----------

+ [{'filename': '/tmp/tmp5gxrti_n/cache-d87234c5763e54a3.arrow'}]
?                                       +++++++++++  ^^
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_regex - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::RecurseHashTest::test_hash_ignores_line_definition_of_function - AssertionError: '52e56ee04ad92499' != '0a4f75cec280f634'
- 52e56ee04ad92499
+ 0a4f75cec280f634
FAILED tests/test_fingerprint.py::RecurseHashTest::test_hash_ipython_function - AssertionError: 'a6bd2041ca63d6c0' != '517bf36b7eecdef5'
- a6bd2041ca63d6c0
+ 517bf36b7eecdef5
FAILED tests/test_fingerprint.py::HashingTest::test_hash_tiktoken_encoding - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_compiled_module - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_generator - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_tensor - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_set_doesnt_depend_on_order - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_set_stable - NameError: name 'log' is not defined
ERROR tests/test_iterable_dataset.py::test_iterable_dataset_from_file - NameError: name 'log' is not defined
= 11 failed, 2850 passed, 3 skipped, 23 warnings, 1 error in 191.06s (0:03:11) =
```
Issue: [FSTimeoutError] load_dataset
State: closed

### Describe the bug

When using `load_dataset`to load [HuggingFaceM4/VQAv2](https://huggingface.co/datasets/HuggingFaceM4/VQAv2), I am getting `FSTimeoutError`. 

### Error

```
TimeoutError: 

The above exception was the direct cause of the following exception:

FSTimeoutError                            Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/fsspec/asyn.py](https://klh9mr78js-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240924-060116_RC00_678132060#) in sync(loop, func, timeout, *args, **kwargs)
     99     if isinstance(return_result, asyncio.TimeoutError):
    100         # suppress asyncio.TimeoutError, raise FSTimeoutError
--> 101         raise FSTimeoutError from return_result
    102     elif isinstance(return_result, BaseException):
    103         raise return_result

FSTimeoutError:
```

It usually fails around 5-6 GB.

<img width="847" alt="Screenshot 2024-09-26 at 9 10 19 PM" src="https://github.com/user-attachments/assets/ff91995a-fb55-4de6-8214-94025d6c8470">


### Steps to reproduce the bug

To reproduce it, run this in colab notebook:

```
!pip install -q -U datasets

from datasets import load_dataset
ds = load_dataset('HuggingFaceM4/VQAv2', split="train[:10%]")
```



### Expected behavior

It should download properly.

### Environment info

Using Colab Notebook.

Is this `FSTimeoutError` due to download network issue from remote resource (from where it is being accessed)?

It seems to happen for all datasets, not just a specific one, and especially for versions after 3.0. (3.0.0, 3.0.1 have this problem)

I had the same error on a different dataset, but after downgrading to datasets==2.21.0, the problem was solved.

Same as https://github.com/huggingface/datasets/issues/7164

This dataset is made of a python script that downloads data from elsewhere than HF, so availability depends on the original host. Ultimately it would be nice to host the files of this dataset on HF

in `datasets` <3.0 there were lots of mechanisms that got removed after the decision to make datasets with python loading scripts legacy for security and maintenance reasons (we only do very basic support now)

@lhoestq Thank you for the clarification! Closing the issue.

I'm getting this too, and also at 5 minutes.  But for `CSTR-Edinburgh/vctk`, so it's not just this dataset, it seems to be a timeout that was introduced and needs to be raised.  The progress bar was moving along just fine before the timeout, and I get more or less of it depending on how fast the network is.

You can change the `aiohttp` timeout from 5min to 1h like this:

```python
import datasets, aiohttp
dataset = datasets.load_dataset(
    dataset_name,
    storage_options={'client_kwargs': {'timeout': aiohttp.ClientTimeout(total=3600)}}
)
```

@JonasLoos Solution solved a download timeout error I received when downloading  `"HuggingFaceM4/VQAv2"` 🎉
Issue: CI is broken: No solution found when resolving dependencies
State: closed
Labels: name: bug

See: https://github.com/huggingface/datasets/actions/runs/11046967444/job/30687294297
```
Run uv pip install --system -r additional-tests-requirements.txt --no-deps
  × No solution found when resolving dependencies:
  ╰─▶ Because the current Python version (3.8.18) does not satisfy Python>=3.9
      and torchdata==0.10.0a0+1a98f21 depends on Python>=3.9, we can conclude
      that torchdata==0.10.0a0+1a98f21 cannot be used.
      And because only torchdata==0.10.0a0+1a98f21 is available and
      you require torchdata, we can conclude that your requirements are
      unsatisfiable.
Error: Process completed with exit code 1.
```
Issue: Set explicit seed in iterable dataset ddp shuffling example 
State: closed

### Describe the bug

In the examples section of the iterable dataset docs https://huggingface.co/docs/datasets/en/package_reference/main_classes#datasets.IterableDataset
the ddp example shuffles without seeding

```python
from datasets.distributed import split_dataset_by_node
ids = ds.to_iterable_dataset(num_shards=512)
ids = ids.shuffle(buffer_size=10_000)  # will shuffle the shards order and use a shuffle buffer when you start iterating
ids = split_dataset_by_node(ds, world_size=8, rank=0)  # will keep only 512 / 8 = 64 shards from the shuffled lists of shards when you start iterating
dataloader = torch.utils.data.DataLoader(ids, num_workers=4)  # will assign 64 / 4 = 16 shards from this node's list of shards to each worker when you start iterating
for example in ids:
    pass
```

This code would - I think - raise an error due to the lack of an explicit seed:

https://github.com/huggingface/datasets/blob/2eb4edb97e1a6af2ea62738ec58afbd3812fc66e/src/datasets/iterable_dataset.py#L1707-L1711


### Steps to reproduce the bug

Run example code

### Expected behavior

Add explicit seeding to example code

### Environment info

latest datasets

thanks for reporting !
Issue: JSON lines with empty struct raise ArrowTypeError
State: closed
Labels: name: bug

JSON lines with empty struct raise ArrowTypeError: struct fields don't match or are in the wrong order

See example: https://huggingface.co/datasets/wikimedia/structured-wikipedia/discussions/5

> ArrowTypeError: struct fields don't match or are in the wrong order: Input fields: struct<> output fields: struct<pov_count: int64, update_count: int64, citation_needed_count: int64>

Related to:
- #7159
Issue: JSON lines with missing struct fields raise TypeError: Couldn't cast array
State: closed
Labels: name: bug

JSON lines with missing struct fields raise TypeError: Couldn't cast array of type.

See example: https://huggingface.co/datasets/wikimedia/structured-wikipedia/discussions/5

One would expect that the struct missing fields are added with null values.

Hello,

I have still the same issue when loading the dataset with the new version:
[https://huggingface.co/datasets/wikimedia/structured-wikipedia/discussions/5](https://huggingface.co/datasets/wikimedia/structured-wikipedia/discussions/5)

I have downloaded and unzipped the wikimedia/structured-wikipedia dataset locally but when loading I have the same issue.

```
import datasets

dataset = datasets.load_dataset("/gpfsdsdir/dataset/HuggingFace/wikimedia/structured-wikipedia/20240916.fr")
```
```
TypeError: Couldn't cast array of type
struct<content_url: string, width: int64, height: int64, alternative_text: string>
to
{'content_url': Value(dtype='string', id=None), 'width': Value(dtype='int64', id=None), 'height': Value(dtype='int64', id=None)}

The above exception was the direct cause of the following exception:
```
My version of datasets is 3.0.1
Issue: interleave_datasets resets shuffle state
State: open

### Describe the bug

```
import datasets
import torch.utils.data


def gen(shards):
    yield {"shards": shards}


def main():
    dataset = datasets.IterableDataset.from_generator(
        gen,
        gen_kwargs={'shards': list(range(25))}
    )
    dataset = dataset.shuffle(buffer_size=1)
    dataset = datasets.interleave_datasets(
        [dataset, dataset], probabilities=[1, 0], stopping_strategy="all_exhausted"
    )

    dataloader = torch.utils.data.DataLoader(
        dataset,
        batch_size=8,
        num_workers=8,
    )

    for i, batch in enumerate(dataloader):
        print(batch)
        if i >= 10:
            break


if __name__ == "__main__":
     main()
```

### Steps to reproduce the bug

Run the script, it will output

```
{'shards': [tensor([ 0,  8, 16, 24,  0,  8, 16, 24])]}
{'shards': [tensor([ 1,  9, 17,  1,  9, 17,  1,  9])]}
{'shards': [tensor([ 2, 10, 18,  2, 10, 18,  2, 10])]}
{'shards': [tensor([ 3, 11, 19,  3, 11, 19,  3, 11])]}
{'shards': [tensor([ 4, 12, 20,  4, 12, 20,  4, 12])]}
{'shards': [tensor([ 5, 13, 21,  5, 13, 21,  5, 13])]}
{'shards': [tensor([ 6, 14, 22,  6, 14, 22,  6, 14])]}
{'shards': [tensor([ 7, 15, 23,  7, 15, 23,  7, 15])]}
{'shards': [tensor([ 0,  8, 16, 24,  0,  8, 16, 24])]}
{'shards': [tensor([17,  1,  9, 17,  1,  9, 17,  1])]}
{'shards': [tensor([18,  2, 10, 18,  2, 10, 18,  2])]}
```

### Expected behavior

The shards should be shuffled.

### Environment info

- `datasets` version: 3.0.0
- Platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.25.0
- PyArrow version: 17.0.0
- Pandas version: 2.0.3
- `fsspec` version: 2023.6.0

It also does preserve `split_by_node`, so in the meantime you should call `shuffle` or `split_by_node` AFTER `interleave_datasets` or `concatenate_datasets`
Issue: Dataset viewer not working! Failure due to more than 32 splits.
State: closed

Hello guys, 

I have a dataset and I didn't know I couldn't upload more than 32 splits. Now, my dataset viewer is not working. I don't have the dataset locally on my node anymore and recreating would take a week. And I have to publish the dataset coming Monday. I read about the practice, how I can resolve it and avoid this issue in the future. But, at the moment I need a hard fix for two of my datasets. 

And I don't want to mess or change anything and allow everyone in public to see the dataset and interact with it. Can you please help me?

https://huggingface.co/datasets/laion/Wikipedia-X
https://huggingface.co/datasets/laion/Wikipedia-X-Full

I have fixed it! But I would appreciate a new feature wheere I could iterate over and see what each file looks like.
Issue: WebDataset loader splits keys differently than WebDataset library
State: closed
Labels: name: bug

As reported by @ragavsachdeva (see discussion here: https://github.com/huggingface/datasets/pull/7144#issuecomment-2348307792), our webdataset loader is not aligned with the `webdataset` library when splitting keys from filenames.

For example, we get a different key splitting for filename `/some/path/22.0/1.1.png`:
- datasets library: `/some/path/22` and `0/1.1.png`
- webdataset library: `/some/path/22.0/1`, `1.png`

```python
import webdataset as wds

wds.tariterators.base_plus_ext("/some/path/22.0/1.1.png")
# ('/some/path/22.0/1', '1.png')
```
Issue: Datasets Unknown Keyword Argument Error - task_templates
State: closed

### Describe the bug

Issue

```python
from datasets import load_dataset
examples = load_dataset('facebook/winoground', use_auth_token=<YOUR USER ACCESS TOKEN>)
```

Gives error

```
TypeError: DatasetInfo.__init__() got an unexpected keyword argument 'task_templates'
```

A simple downgrade to lower `datasets v 2.21.0` solves it.




### Steps to reproduce the bug

1. `pip install datsets`
2.
```python
from datasets import load_dataset
examples = load_dataset('facebook/winoground', use_auth_token=<YOUR USER ACCESS TOKEN>)
```



### Expected behavior

Should load the dataset correctly.

### Environment info

- Datasets version  `3.0.0`
- `transformers` version: 4.45.0.dev0
- Platform: Linux-6.8.0-40-generic-x86_64-with-glibc2.35
- Python version: 3.12.4
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.5
- Accelerate version: 0.35.0.dev0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes


Thanks, for reporting.

We have been fixing most Hub datasets to remove the deprecated (and now non-supported) task templates, but we missed the "facebook/winoground".

It is fixed now: https://huggingface.co/datasets/facebook/winoground/discussions/8



Hello @albertvillanova 

I got the same error while loading this dataset: https://huggingface.co/datasets/alaleye/aloresb...

How can I fix it ? 
Thanks

I am getting the same error on the below code, any fix to this ?

```
from datasets import load_dataset

minds = load_dataset("PolyAI/minds14", name="en-AU", split="train")
minds
```
Issue: Specifying datatype when adding a column to a dataset.
State: closed
Labels: name: enhancement

### Feature request

There should be a way to specify the datatype of a column in `datasets.add_column()`. 

### Motivation

To specify a custom datatype, we have to use `datasets.add_column()` followed by `datasets.cast_column()` which is slow for large datasets. Another workaround is to pass a `numpy.array()` of desired type to the `datasets.add_column()` function.

IMO this functionality should be natively supported.

https://discuss.huggingface.co/t/add-column-with-a-particular-type-in-datasets/95674



### Your contribution

I can submit a PR for this.

#self-assign
Issue: Use load_dataset to load imagenet-1K But find a empty dataset
State: open

### Describe the bug

```python
def get_dataset(data_path, train_folder="train", val_folder="val"):
    traindir = os.path.join(data_path, train_folder)
    valdir = os.path.join(data_path, val_folder)

    def transform_val_examples(examples):
        transform = Compose([
            Resize(256),
            CenterCrop(224),
            ToTensor(),
        ])
        examples["image"] = [transform(image.convert("RGB")) for image in examples["image"]]
        return examples

    def transform_train_examples(examples):
        transform = Compose([
            RandomResizedCrop(224),
            RandomHorizontalFlip(),
            ToTensor(),
        ])
        examples["image"] = [transform(image.convert("RGB")) for image in examples["image"]]
        return examples

    # @fengsicheng: This way is very slow for big dataset like ImageNet-1K (but can pass the network problem using local dataset)
    # train_set = load_dataset("imagefolder", data_dir=traindir, num_proc=4)
    # test_set = load_dataset("imagefolder", data_dir=valdir, num_proc=4)

    train_set = load_dataset("imagenet-1K", split="train", trust_remote_code=True)                                                                                                                                                                                                            
    test_set = load_dataset("imagenet-1K", split="test", trust_remote_code=True)

    print(train_set["label"])

    train_set.set_transform(transform_train_examples)
    test_set.set_transform(transform_val_examples)

    return train_set, test_set
```
    above the code, but output of the print is a list of None:
    
<img width="952" alt="image" src="https://github.com/user-attachments/assets/c4e2fdd8-3b8f-481e-8f86-9bbeb49d79fb">


### Steps to reproduce the bug

1. just ran the code 
2. see the print


### Expected behavior

I do not know how to fix this, can anyone provide help or something? It is hurry for me

### Environment info

- `datasets` version: 2.21.0
- Platform: Linux-5.4.0-190-generic-x86_64-with-glibc2.31
- Python version: 3.10.14
- `huggingface_hub` version: 0.24.6
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.6.1

Imagenet-1k  is a gated dataset which means you’ll have to agree to share your contact info to access it. Have you tried this yet? Once you have, you can sign in with your user token (you can find this in your Hugging Face account settings) when prompted by running.

```
huggingface-cli login
train_set  = load_dataset('imagenet-1k', split='train', use_auth_token=True)
``` 

Thanks a lot! It helps me
Issue: Cache only changed columns?
State: open
Labels: name: enhancement

### Feature request

Cache only the actual changes to the dataset i.e. changed columns.

### Motivation

I realized that caching actually saves the complete dataset again.
This is especially problematic for image datasets if one wants to only change another column e.g. some metadata and then has to save 5 TB again.

### Your contribution

Is this even viable in the current architecture of the package?
I quickly looked into it and it seems it would require significant changes.

I would spend some time looking into this but maybe somebody could help with the feasibility and some plan to implement before spending too much time on it?

so I guess a workaround to this is to simply remove all columns except the ones to cache and then add them back with `concatenate_datasets(..., axis=1)`.

yes this is the right workaround. We're keeping the cache like this to make it easier for people to delete intermediate cache files
Issue: Attempting to return a rank 3 grayscale image from dataset.map results in extreme slowdown 
State: open

### Describe the bug

Background: Digital images are often represented as a  (Height, Width, Channel) tensor. This is the same for huggingface datasets that contain images. These images are loaded in Pillow containers which offer, for example, the `.convert` method.

I can convert an image from a (H,W,3) shape to a grayscale (H,W) image and I have no problems with this. But when attempting to return a (H,W,1) shaped matrix from a map function, it never completes and sometimes even results in an OOM from the OS.


I've used various methods to expand a (H,W) shaped array to a (H,W,1) array. But they all resulted in extremely long map operations consuming a lot of CPU and RAM.

### Steps to reproduce the bug

Below is a minimal example using two methods to get the desired output. Both of which don't work
```py
import tensorflow as tf
import datasets
import numpy as np

ds = datasets.load_dataset("project-sloth/captcha-images")
to_gray_pillow = lambda sample: {'image': np.expand_dims(sample['image'].convert("L"), axis=-1)}
ds_gray = ds.map(to_gray_pillow)

# Alternatively
ds = datasets.load_dataset("project-sloth/captcha-images").with_format("tensorflow")
to_gray_tf = lambda sample: {'image': tf.expand_dims(tf.image.rgb_to_grayscale(sample['image']), axis=-1)}
ds_gray = ds.map(to_gray_tf)
```

### Expected behavior

I expect the map operation to complete and return a new dataset containing grayscale images in a (H,W,1) shape.

### Environment info

datasets 2.21.0
python tested with both 3.11 and 3.12
host os : linux
Issue: Inconsistent output in documentation example: `num_classes` not displayed in `ClassLabel` output
State: closed

In the documentation for [ClassLabel](https://huggingface.co/docs/datasets/v2.21.0/en/package_reference/main_classes#datasets.ClassLabel), there is an example of usage with the following code:

````
from datasets import Features
features = Features({'label': ClassLabel(num_classes=3, names=['bad', 'ok', 'good'])})
features
````

which expects to output (as stated in the documentation):

````
{'label': ClassLabel(num_classes=3, names=['bad', 'ok', 'good'], id=None)}
````

but it generates the following

````
{'label': ClassLabel(names=['bad', 'ok', 'good'], id=None)}
````

If my understanding is correct, this happens because although num_classes is used during the init of the object, it is afterward ignored:

https://github.com/huggingface/datasets/blob/be5cff059a2a5b89d7a97bc04739c4919ab8089f/src/datasets/features/features.py#L975

I would like to work on this issue if this is something needed 😄
Issue: Caching shuffles by np.random.Generator results in unintiutive behavior
State: open

### Describe the bug

Create a dataset. Save it to disk. Load from disk. Shuffle, usning a `np.random.Generator`. Iterate. Shuffle again. Iterate. The iterates are different since the supplied np.random.Generator has progressed between the shuffles.

Load dataset from disk again. Shuffle and Iterate. See same result as before. Shuffle and iterate, and this time it does not have the same shuffling as ion previous run.

The motivation is I have a deep learning loop with 

```
for epoch in range(10):
    for batch in dataset.shuffle(generator=generator).iter(batch_size=32):
        .... # do stuff
```
where I want a new shuffling at every epoch. Instead I get the same shuffling.

### Steps to reproduce the bug

Run the code below two times. 

```python
import datasets
import numpy as np

generator = np.random.default_rng(0)
ds = datasets.Dataset.from_dict(mapping={"X":range(1000)})
ds.save_to_disk("tmp")
print("First loop: ", end="")
for _ in range(10):
    print(next(ds.shuffle(generator=generator).iter(batch_size=1))['X'], end=", ")
print("")

print("Second loop: ", end="")
ds = datasets.Dataset.load_from_disk("tmp")
for _ in range(10):
    print(next(ds.shuffle(generator=generator).iter(batch_size=1))['X'], end=", ")
print("")
```

The output is:

```
$ python main.py 
Saving the dataset (1/1 shards): 100%|███████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 495019.95 examples/s]
First loop: 459, 739, 72, 943, 241, 181, 845, 830, 896, 334, 
Second loop: 741, 847, 944, 795, 483, 842, 717, 865, 231, 840,
$ python main.py 
Saving the dataset (1/1 shards): 100%|████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 22243.40 examples/s]
First loop: 459, 739, 72, 943, 241, 181, 845, 830, 896, 334, 
Second loop: 741, 741, 741, 741, 741, 741, 741, 741, 741, 741, 
```

The second loop, on the second run, only spits out "741, 741, 741...." which is *not* the desired output

### Expected behavior

I want the dataset to shuffle at every epoch since I provide it with a generator for shuffling. 

### Environment info

Datasets version 2.21.0
Ubuntu linux.

I first thought this was a mistake of mine, and also posted on stack overflow. https://stackoverflow.com/questions/78913797/iterating-a-huggingface-dataset-from-disk-using-generator-seems-broken-how-to-d 

It seems to me the issue is the caching step in 

https://github.com/huggingface/datasets/blob/be5cff059a2a5b89d7a97bc04739c4919ab8089f/src/datasets/arrow_dataset.py#L4306-L4316

because the shuffle happens after checking the cache, the rng state won't advance if the cache is used. This is VERY confusing. Also not documented.

My proposal is that you remove the API for using a Generator, and only keep the seed-based API since that is functional and cache-compatible.

Second that, its very confusing.
Issue: [interleave_dataset] sample batches from a single source at a time
State: open
Labels: name: enhancement

### Feature request

interleave_dataset and [RandomlyCyclingMultiSourcesExamplesIterable](https://github.com/huggingface/datasets/blob/3813ce846e52824b38e53895810682f0a496a2e3/src/datasets/iterable_dataset.py#L816) enable us to sample data examples from different sources. But can we also sample batches in a similar manner (each batch only contains data from a single source)?


### Motivation

Some recent research [[1](https://blog.salesforceairesearch.com/sfr-embedded-mistral/), [2](https://arxiv.org/pdf/2310.07554)] shows that source homogenous batching can be helpful for contrastive learning. Can we add a function called `RandomlyCyclingMultiSourcesBatchesIterable` to support this functionality?

### Your contribution

I can contribute a PR. But I wonder what the best way is to test its correctness and robustness.
Issue: datasets cannot handle nested json if features is given.
State: closed

### Describe the bug

I have a json named temp.json.
```json
{"ref1": "ABC", "ref2": "DEF", "cuts":[{"cut1": 3, "cut2": 5}]}
```
I want to load it.
```python
ds = datasets.load_dataset('json', data_files="./temp.json", features=datasets.Features({
    'ref1': datasets.Value('string'),
    'ref2': datasets.Value('string'),
    'cuts': datasets.Sequence({
        "cut1": datasets.Value("uint16"),
        "cut2": datasets.Value("uint16")
    })
}))
```
The above code does not work. However, I can load it without giving features.
```python
ds = datasets.load_dataset('json', data_files="./temp.json")
```
Is it possible to load integers as uint16 to save some memory?

### Steps to reproduce the bug

As in the bug description.

### Expected behavior

The data are loaded and integers are uint16.

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.21.0
- Platform: Linux-5.15.0-118-generic-x86_64-with-glibc2.35
- Python version: 3.11.9
- `huggingface_hub` version: 0.24.5
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0

Hi ! `Sequence` has a weird behavior for dictionaries (from tensorflow-datasets), use a regular list instead:

```python
ds = datasets.load_dataset('json', data_files="./temp.json", features=datasets.Features({
    'ref1': datasets.Value('string'),
    'ref2': datasets.Value('string'),
    'cuts': [{
        "cut1": datasets.Value("uint16"),
        "cut2": datasets.Value("uint16")
    }]
}))
```

> Hi ! `Sequence` has a weird behavior for dictionaries (from tensorflow-datasets), use a regular list instead:
> 
> ```python
> ds = datasets.load_dataset('json', data_files="./temp.json", features=datasets.Features({
>     'ref1': datasets.Value('string'),
>     'ref2': datasets.Value('string'),
>     'cuts': [{
>         "cut1": datasets.Value("uint16"),
>         "cut2": datasets.Value("uint16")
>     }]
> }))
> ```
Thank you!


It works.
Issue: Stream dataset does not iterate if the batch size is larger than the dataset size (related to drop_last_batch)
State: closed

### Describe the bug

Hi there,

I use streaming and interleaving to combine multiple datasets saved in jsonl files. The size of dataset can vary (from 100ish to 100k-ish). I use dataset.map() and a big batch size to reduce the IO cost. It was working fine with datasets-2.16.1 but this problem shows up after I upgraded to datasets-2.19.2. With  2.21.0 the problem remains.

Please see the code below to reproduce the problem.

The dataset can iterate correctly if we set either streaming=False or drop_last_batch=False.

I have to use drop_last_batch=True since it's for distributed training.

### Steps to reproduce the bug

```python
# datasets==2.21.0
import datasets
def data_prepare(examples):
    print(examples["sentence1"][0])
    return examples

batch_size = 101
# the size of the dataset is 100
# the dataset iterates correctly if we set either streaming=False or drop_last_batch=False 
dataset = datasets.load_dataset("mteb/biosses-sts", split="test", streaming=True)
dataset = dataset.map(lambda x: data_prepare(x),
                      drop_last_batch=True,
                      batched=True, batch_size=batch_size)
for ex in dataset:
    print(ex)
    pass

```

### Expected behavior

The dataset iterates regardless of the batch size.

### Environment info

- `datasets` version: 2.21.0
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.14
- `huggingface_hub` version: 0.24.5
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.2.0


That's expected behavior, it's also the same in `torch`:

```python
>>> list(DataLoader(list(range(5)), batch_size=10, drop_last=True))
[]
```
Issue: cudf-cu12 24.4.1, ibis-framework 8.0.0 requires pyarrow<15.0.0a0,>=14.0.1,pyarrow<16,>=2 and datasets 2.21.0 requires pyarrow>=15.0.0
State: open

### Describe the bug

!pip install accelerate>=0.16.0 torchvision transformers>=4.25.1 datasets>=2.19.1 ftfy tensorboard Jinja2 peft==0.7.0

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
cudf-cu12 24.4.1 requires pyarrow<15.0.0a0,>=14.0.1, but you have pyarrow 17.0.0 which is incompatible.
ibis-framework 8.0.0 requires pyarrow<16,>=2, but you have pyarrow 17.0.0 which is incompatible.

to solve above error
!pip install pyarrow==14.0.1

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.21.0 requires pyarrow>=15.0.0, but you have pyarrow 14.0.1 which is incompatible.

### Steps to reproduce the bug

!pip install datasets>=2.19.1

### Expected behavior

run without dependency error

### Environment info

Diffusers version: 0.31.0.dev0

Platform: Linux-6.1.85+-x86_64-with-glibc2.35
Running on Google Colab?: Yes
Python version: 3.10.12
PyTorch version (GPU?): 2.3.1+cu121 (True)
Flax version (CPU?/GPU?/TPU?): 0.8.4 (gpu)
Jax version: 0.4.26
JaxLib version: 0.4.26
Huggingface_hub version: 0.23.5
Transformers version: 4.42.4
Accelerate version: 0.32.1
PEFT version: 0.7.0
Bitsandbytes version: not installed
Safetensors version: 0.4.4
xFormers version: not installed
Accelerator: Tesla T4, 15360 MiB
Using GPU in script?:
Using distributed or parallel set-up in script?:

@sayakpaul  please advice 

Hits the same dependency conflict
Issue: CI is broken for numpy-2: Failed to fetch wheel: llvmlite==0.34.0
State: closed

Ci is broken with error `Failed to fetch wheel: llvmlite==0.34.0`: https://github.com/huggingface/datasets/actions/runs/10466825281/job/28984414269
```
Run uv pip install --system "datasets[tests_numpy2] @ ."
Resolved 150 packages in 4.42s
error: Failed to prepare distributions
  Caused by: Failed to fetch wheel: llvmlite==0.34.0
  Caused by: Build backend failed to build wheel through `build_wheel()` with exit status: 1
--- stdout:
running bdist_wheel
/home/runner/.cache/uv/builds-v0/.tmpcyKh8S/bin/python /home/runner/.cache/uv/built-wheels-v3/pypi/llvmlite/0.34.0/wrk1bNwq1gleSiznvrSEZ/llvmlite-0.34.0.tar.gz/ffi/build.py
LLVM version...
--- stderr:
Traceback (most recent call last):
  File "/home/runner/.cache/uv/built-wheels-v3/pypi/llvmlite/0.34.0/wrk1bNwq1gleSiznvrSEZ/llvmlite-0.34.0.tar.gz/ffi/build.py", line 105, in main_posix
    out = subprocess.check_output([llvm_config, '--version'])
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/subprocess.py", line 503, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'llvm-config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/.cache/uv/built-wheels-v3/pypi/llvmlite/0.34.0/wrk1bNwq1gleSiznvrSEZ/llvmlite-0.34.0.tar.gz/ffi/build.py", line 191, in <module>
    main()
  File "/home/runner/.cache/uv/built-wheels-v3/pypi/llvmlite/0.34.0/wrk1bNwq1gleSiznvrSEZ/llvmlite-0.34.0.tar.gz/ffi/build.py", line 181, in main
    main_posix('linux', '.so')
  File "/home/runner/.cache/uv/built-wheels-v3/pypi/llvmlite/0.34.0/wrk1bNwq1gleSiznvrSEZ/llvmlite-0.34.0.tar.gz/ffi/build.py", line 107, in main_posix
    raise RuntimeError("%s failed executing, please point LLVM_CONFIG "
RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config
error: command '/home/runner/.cache/uv/builds-v0/.tmpcyKh8S/bin/python' failed with exit code 1
```

Note that the CI before was using:
- llvmlite: 0.43.0
- numba: 0.60.0

Now it tries to use:
- llvmlite: 0.34.0
- numba: 0.51.2

The issue is because numba-0.60.0 pins numpy<2.1 and `uv` tries to install latest numpy-2.1.0 with an old numba-0.51.0 version (and llvmlite-0.34.0). See discussion in their repo:
- https://github.com/numba/numba/issues/9708

Latest numpy-2.1.0 will be supported by the next numba-0.61.0 release in September.

Note that our CI requires numba with the "audio" extra:
- librosa > numba
Issue: ConnectionError for gated datasets and unauthenticated users
State: closed

Since the Hub returns dataset info for gated datasets and unauthenticated users, there is dead code: https://github.com/huggingface/datasets/blob/98fdc9e78e6d057ca66e58a37f49d6618aab8130/src/datasets/load.py#L1846-L1852

We should remove the dead code and properly handle this case: currently we are raising a `ConnectionError` instead of a `DatasetNotFoundError` (as before).

See:
- https://github.com/huggingface/dataset-viewer/issues/3025
- https://github.com/huggingface/huggingface_hub/issues/2457
Issue: website broken: Create a new dataset repository, doesn't create a new repo in Firefox
State: closed

### Describe the bug

This issue is also reported here:
https://discuss.huggingface.co/t/create-a-new-dataset-repository-broken-page/102644

This page is broken.
https://huggingface.co/new-dataset

I fill in the form with my text, and click `Create Dataset`.
![Screenshot 2024-08-16 at 15 55 37](https://github.com/user-attachments/assets/de16627b-7a55-4bcf-9f0b-a48227aabfe6)

Then the form gets wiped. And no repo got created. No error message visible in the developer console.
![Screenshot 2024-08-16 at 15 56 54](https://github.com/user-attachments/assets/0520164b-431c-40a5-9634-11fd62c4f4c3)

# Idea for improvement

For better UX, if the repo cannot be created, then show an error message, that something went wrong.

# Work around, that works for me

```python
from huggingface_hub import HfApi, HfFolder
repo_id = 'simon-arc-solve-fractal-v3'
api = HfApi()
username = api.whoami()['name']
repo_url = api.create_repo(repo_id=repo_id, exist_ok=True, private=True, repo_type="dataset")
```

### Steps to reproduce the bug

Go https://huggingface.co/new-dataset

Fill in the form.

Click `Create dataset`.

Now the form is cleared. And the page doesn't jump anywhere.

### Expected behavior

The moment the user clicks `Create dataset`, the repo gets created and the page jumps to the created repo.


### Environment info

Firefox 128.0.3 (64-bit)
macOS Sonoma 14.5


I don't reproduce, I was able to create a new repo: https://huggingface.co/datasets/severo/reproduce-datasets-issues-7108. Can you confirm it's still broken?

I have just tried again.

Firefox: The `Create dataset` doesn't work. It has worked in the past. It's my preferred browser.

Chrome: The `Create dataset` works.

It seems to be a Firefox specific issue.

I have updated Firefox 129.0 (64 bit), and now the `Create dataset` is working again in Firefox.

UX: It would be nice with better error messages on HuggingFace.

maybe an issue with the cookie. cc @Wauplin @coyotte508
Issue: load_dataset broken in 2.21.0
State: closed

### Describe the bug

`eval_set = datasets.load_dataset("tatsu-lab/alpaca_eval", "alpaca_eval_gpt4_baseline", trust_remote_code=True)` 
used to work till 2.20.0 but doesn't work in 2.21.0

In 2.20.0:
![Screenshot 2024-08-16 at 3 57 10 PM](https://github.com/user-attachments/assets/0516489b-8187-486d-bee8-88af3381dee9)


in 2.21.0:
![Screenshot 2024-08-16 at 3 57 24 PM](https://github.com/user-attachments/assets/bc257570-f461-41e4-8717-90a69ed7c24f)


### Steps to reproduce the bug

1. Spin up a new google collab
2. `pip install datasets==2.21.0`
3. `import datasets`
4. `eval_set = datasets.load_dataset("tatsu-lab/alpaca_eval", "alpaca_eval_gpt4_baseline", trust_remote_code=True)` 
5. Will throw an error.



### Expected behavior

Try steps 1-5 again but replace datasets version with 2.20.0, it will work

### Environment info



- `datasets` version: 2.21.0
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.23.5
- PyArrow version: 17.0.0
- Pandas version: 2.1.4
- `fsspec` version: 2024.5.0


There seems to be a PR related to the load_dataset path that went into 2.21.0 -- https://github.com/huggingface/datasets/pull/6862/files

Taking a look at it now

+1

Downgrading to 2.20.0 fixed my issue, hopefully helpful for others.

I tried adding a simple test to `test_load.py` with the alpaca eval dataset but the test didn't fail :(. 

So looks like this might have something to do with the environment? 

There was an issue with the script of the "tatsu-lab/alpaca_eval" dataset.

I was fixed with this PR: 
- [Fix FileNotFoundError](https://huggingface.co/datasets/tatsu-lab/alpaca_eval/discussions/2)

It should work now if you retry to load the dataset.
Issue: IterableDataset: cannot resolve features from list of numpy arrays
State: open

### Describe the bug

when resolve features of `IterableDataset`, got `pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values` error.

```
Traceback (most recent call last):
  File "test.py", line 6
    iter_ds = iter_ds._resolve_features()
  File "lib/python3.10/site-packages/datasets/iterable_dataset.py", line 2876, in _resolve_features
    features = _infer_features_from_batch(self.with_format(None)._head())
  File "lib/python3.10/site-packages/datasets/iterable_dataset.py", line 63, in _infer_features_from_batch
    pa_table = pa.Table.from_pydict(batch)
  File "pyarrow/table.pxi", line 1813, in pyarrow.lib._Tabular.from_pydict
  File "pyarrow/table.pxi", line 5339, in pyarrow.lib._from_pydict
  File "pyarrow/array.pxi", line 374, in pyarrow.lib.asarray
  File "pyarrow/array.pxi", line 344, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 42, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
```

### Steps to reproduce the bug

```python
from datasets import Dataset
import numpy as np

# create list of numpy
iter_ds = Dataset.from_dict({'a': [[[1, 2, 3], [1, 2, 3]]]}).to_iterable_dataset().map(lambda x: {'a': [np.array(x['a'])]})
iter_ds = iter_ds._resolve_features()  # errors here
```

### Expected behavior

features can be successfully resolved

### Environment info

- `datasets` version: 2.21.0
- Platform: Linux-5.15.0-94-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.23.4
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0

Assign this issue to me under Hacktoberfest with hacktoberfest label inserted on the issue
Issue: Some of DownloadConfig's properties are always being overridden in load.py
State: open

### Describe the bug

The `extract_compressed_file` and `force_extract` properties of DownloadConfig are always being set to True in the function `dataset_module_factory` in the `load.py` file. This behavior is very annoying because data extracted will just be ignored the next time the dataset is loaded.

See this image below:
![image](https://github.com/user-attachments/assets/9e76ebb7-09b1-4c95-adc8-a959b536f93c)


### Steps to reproduce the bug

1. Have a local dataset that contains archived files (zip, tar.gz, etc)
2. Build a dataset loading script to download and extract these files
3. Run the load_dataset function with a DownloadConfig that specifically set `force_extract` to False
4. The extraction process will start no matter if the archives was extracted previously

### Expected behavior

The extraction process should not run when the archives were previously extracted and `force_extract` is set to False.

### Environment info

datasets==2.20.0
python3.9
Issue: Add Arabic Docs to datasets
State: open
Labels: name: enhancement

### Feature request

Add Arabic Docs to datasets
[Datasets Arabic](https://github.com/AhmedAlmaghz/datasets/blob/main/docs/source/ar/index.mdx)

### Motivation

@AhmedAlmaghz
https://github.com/AhmedAlmaghz/datasets/blob/main/docs/source/ar/index.mdx

### Your contribution

@AhmedAlmaghz
https://github.com/AhmedAlmaghz/datasets/blob/main/docs/source/ar/index.mdx
Issue: The test test_move_script_doesnt_change_hash fails because it runs the 'python' command while the python executable has a different name
State: open

### Describe the bug

Tests should use the same pythin path as they are launched with, which in the case of FreeBSD is /usr/local/bin/python3.11

Failure:
```
                if err_filename is not None:
>                   raise child_exception_type(errno_num, err_msg, err_filename)
E                   FileNotFoundError: [Errno 2] No such file or directory: 'python'
```


### Steps to reproduce the bug

regular test run using PyTest

### Expected behavior

n/a

### Environment info

FreeBSD 14.1
Issue: Unable to create dataset card for Lushootseed language
State: closed
Labels: name: enhancement

### Feature request

While I was creating the dataset which contained all documents from the Lushootseed Wikipedia, the dataset card asked me to enter which language the dataset was in. Since Lushootseed is a critically endangered language, it was not available as one of the options. Is it possible to allow entering languages that aren't available in the options?

### Motivation

I'd like to add more information about my dataset in the dataset card, and the language is one of the most important pieces of information, since the entire dataset is primarily concerned collecting Lushootseed documents.

### Your contribution

I can submit a pull request

Thanks for reporting.

It is weird, because the language entry is in the list. See: https://github.com/huggingface/huggingface.js/blob/98e32f0ed4ee057a596f66a1dec738e5db9643d5/packages/languages/src/languages_iso_639_3.ts#L15186-L15189

I have reported the issue:
- https://github.com/huggingface/huggingface.js/issues/834



As explained in the reported issue above, the problem only appears in the autocomplete field: you can still enter the `lut` language directly in the markdown editor window.
Issue: load_dataset ignores cached datasets and tries to hit HF Hub, resulting in API rate limit errors
State: open

### Describe the bug

I have been running lm-eval-harness a lot which has results in an API rate limit. This seems strange, since all of the data should be cached locally. I have in fact verified this.

### Steps to reproduce the bug

1. Be Me
2. Run `load_dataset("TAUR-Lab/MuSR")`
3. Hit rate limit error
4. Dataset is in .cache/huggingface/datasets
5. ???

### Expected behavior

We should not run into API rate limits if we have cached the dataset

### Environment info

datasets 2.16.0
python 3.10.4

I'm having the same issue - running into rate limits when doing hyperparameter tuning even though the dataset is supposed to be cached. I feel like this behaviour should at the very least be documented, but honestly you should just not be running into rate limits in the first place when the dataset is cached. It even happens when specifying a specific revision for the dataset, in which case AFAIK there should be no reason to be doing API requests if it's already cached (besides maybe a quick hash check but hitting rate limits for that in ~200 requests across 10 hours of use seems a bit ridiculous).
Issue: More easily support streaming local files
State: open
Labels: name: enhancement

### Feature request

Simplify downloading and streaming datasets locally. Specifically, perhaps add an option to `load_dataset(..., streaming="download_first")` or add better support for streaming symlinked or arrow files.

### Motivation

I have downloaded FineWeb-edu locally and currently trying to stream the dataset from the local files. I have both the raw parquet files using `hugginface-cli download --repo-type dataset HuggingFaceFW/fineweb-edu` and the processed arrow files using `load_dataset("HuggingFaceFW/fineweb-edu")`. 

Streaming the files locally does not work well for both file types for two different reasons.

**Arrow files**

When running `load_dataset("arrow", data_files={"train": "~/.cache/huggingface/datasets/HuggingFaceFW___fineweb-edu/default/0.0.0/5b89d1ea9319fe101b3cbdacd89a903aca1d6052/fineweb-edu-train-*.arrow"})` resolving the data files is fast, but because `arrow` is not included in the known [extensions file list](https://github.com/huggingface/datasets/blob/ce4a0c573920607bc6c814605734091b06b860e7/src/datasets/utils/file_utils.py#L738) , all files are opened and scanned to determine the compression type. Adding `arrow` to the known extension types resolves this issue. 

**Parquet files**

When running `load_dataset("arrow", data_files={"train": "~/.cache/huggingface/hub/dataset-HuggingFaceFW___fineweb-edu/snapshots/5b89d1ea9319fe101b3cbdacd89a903aca1d6052/data/CC-MAIN-*/train-*.parquet"})` the paths do not get resolved because the parquet files are symlinked from the blobs (which contain all files in case there are different versions). This occurs because the [pattern matching](https://github.com/huggingface/datasets/blob/ce4a0c573920607bc6c814605734091b06b860e7/src/datasets/data_files.py#L389) checks if the path is a file and does not check for symlinks. Symlinks (at least on my machine) are of type "other". 


### Your contribution

I have created a PR for fixing arrow file streaming and symlinks. However, I have not checked locally if the tests work or new tests need to be added.

IMO, the easiest option would be to add a `streaming=download_first` option, but I'm afraid that exceeds my current knowledge of how the datasets library works. https://github.com/huggingface/datasets/pull/7083