AniketTec/github-issues huggingface dataset:
Title: Use load_dataset to load imagenet-1K But find a empty dataset
State: open

### Describe the bug

```python
def get_dataset(data_path, train_folder="train", val_folder="val"):
    traindir = os.path.join(data_path, train_folder)
    valdir = os.path.join(data_path, val_folder)

    def transform_val_examples(examples):
        transform = Compose([
            Resize(256),
            CenterCrop(224),
            ToTensor(),
        ])
        examples["image"] = [transform(image.convert("RGB")) for image in examples["image"]]
        return examples

    def transform_train_examples(examples):
        transform = Compose([
            RandomResizedCrop(224),
            RandomHorizontalFlip(),
            ToTensor(),
        ])
        examples["image"] = [transform(image.convert("RGB")) for image in examples["image"]]
        return examples

    # @fengsicheng: This way is very slow for big dataset like ImageNet-1K (but can pass the network problem using local dataset)
    # train_set = load_dataset("imagefolder", data_dir=traindir, num_proc=4)
    # test_set = load_dataset("imagefolder", data_dir=valdir, num_proc=4)

    train_set = load_dataset("imagenet-1K", split="train", trust_remote_code=True)                                                                                                                                                                                                            
    test_set = load_dataset("imagenet-1K", split="test", trust_remote_code=True)

    print(train_set["label"])

    train_set.set_transform(transform_train_examples)
    test_set.set_transform(transform_val_examples)

    return train_set, test_set
```
    above the code, but output of the print is a list of None:
    
<img width="952" alt="image" src="https://github.com/user-attachments/assets/c4e2fdd8-3b8f-481e-8f86-9bbeb49d79fb">


### Steps to reproduce the bug

1. just ran the code 
2. see the print


### Expected behavior

I do not know how to fix this, can anyone provide help or something? It is hurry for me

### Environment info

- `datasets` version: 2.21.0
- Platform: Linux-5.4.0-190-generic-x86_64-with-glibc2.31
- Python version: 3.10.14
- `huggingface_hub` version: 0.24.6
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.6.1

Comments:
Imagenet-1k  is a gated dataset which means you’ll have to agree to share your contact info to access it. Have you tried this yet? Once you have, you can sign in with your user token (you can find this in your Hugging Face account settings) when prompted by running.

```
huggingface-cli login
train_set  = load_dataset('imagenet-1k', split='train', use_auth_token=True)
``` 

Thanks a lot! It helps me
Title: Cache only changed columns?
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

Cache only the actual changes to the dataset i.e. changed columns.

### Motivation

I realized that caching actually saves the complete dataset again.
This is especially problematic for image datasets if one wants to only change another column e.g. some metadata and then has to save 5 TB again.

### Your contribution

Is this even viable in the current architecture of the package?
I quickly looked into it and it seems it would require significant changes.

I would spend some time looking into this but maybe somebody could help with the feasibility and some plan to implement before spending too much time on it?

Comments:
so I guess a workaround to this is to simply remove all columns except the ones to cache and then add them back with `concatenate_datasets(..., axis=1)`.

yes this is the right workaround. We're keeping the cache like this to make it easier for people to delete intermediate cache files
Title: Attempting to return a rank 3 grayscale image from dataset.map results in extreme slowdown 
State: open

### Describe the bug

Background: Digital images are often represented as a  (Height, Width, Channel) tensor. This is the same for huggingface datasets that contain images. These images are loaded in Pillow containers which offer, for example, the `.convert` method.

I can convert an image from a (H,W,3) shape to a grayscale (H,W) image and I have no problems with this. But when attempting to return a (H,W,1) shaped matrix from a map function, it never completes and sometimes even results in an OOM from the OS.


I've used various methods to expand a (H,W) shaped array to a (H,W,1) array. But they all resulted in extremely long map operations consuming a lot of CPU and RAM.

### Steps to reproduce the bug

Below is a minimal example using two methods to get the desired output. Both of which don't work
```py
import tensorflow as tf
import datasets
import numpy as np

ds = datasets.load_dataset("project-sloth/captcha-images")
to_gray_pillow = lambda sample: {'image': np.expand_dims(sample['image'].convert("L"), axis=-1)}
ds_gray = ds.map(to_gray_pillow)

# Alternatively
ds = datasets.load_dataset("project-sloth/captcha-images").with_format("tensorflow")
to_gray_tf = lambda sample: {'image': tf.expand_dims(tf.image.rgb_to_grayscale(sample['image']), axis=-1)}
ds_gray = ds.map(to_gray_tf)
```

### Expected behavior

I expect the map operation to complete and return a new dataset containing grayscale images in a (H,W,1) shape.

### Environment info

datasets 2.21.0
python tested with both 3.11 and 3.12
host os : linux

Comments:
Title: remove filecheck to enable symlinks
State: closed

Enables streaming from local symlinks #7083 

@lhoestq

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7133). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

The CI is failing, looks like it breaks imagefolder loading.

I just checked fsspec internals and maybe instead we can detect symlink by checking `islink` and `size` to make sure it's a file
```python
if info["type"] == "file" or (info.get("islink") and info["size"])
```


hmm actually `size` doesn't seem to filter symlinked directories, we need another way

Does fsspec perhaps allow resolving symlinks? Something like https://docs.python.org/3/library/pathlib.html#pathlib.Path.resolve

there is `info["destination"]` in case of a symlink, so maybe


```python
if info["type"] == "file" or (info.get("islink") and info.get("destination") and os.path.isfile(info["destination"]))
```

I've added a fix which works with some temporary test files locally 

`(info["type"] == "file" or (info.get("islink") and os.path.isfile(os.path.realpath(filepath))))`
Title: Fix data file module inference
State: open

I saved a dataset with two splits to disk with `DatasetDict.save_to_disk`. The train is bigger and ended up in 10 shards, whereas the test split only resulted in 1 split.

Now when trying to load the dataset, an error is raised that not all splits have the same data format:

> ValueError: Couldn't infer the same data file format for all splits. Got {NamedSplit('train'): ('arrow', {}), NamedSplit('test'): ('json', {})}

This is not expected because both splits are saved as arrow files.

I did some debugging and found that this is the case because the list of data_files includes a `state.json` file. 

Now this means for train split I get 10 ".arrow" and 1 ".json" file. Since datasets picks based on the most common extension this is correctly inferred as "arrow". In the test split, there is 1 .arrow and 1 .json file. Given the function description:

> It picks the module based on the most common file extension.
    In case of a draw ".parquet" is the favorite, and then alphabetical order.

This is not quite true though, because in a tie the extensions are actually based on reverse-alphabetical order:
```
for (ext, _), _ in sorted(extensions_counter.items(), key=sort_key, *reverse=True*):
```

Which thus leads to the module wrongly inferred as "json", whereas it should be "arrow", matching the train split.

I first thought about adding "state.json" in the list of excluded files for the inference: https://github.com/huggingface/datasets/blob/main/src/datasets/load.py#L513. However, I think from digging into the code it looks like the right thing to do is to exclude it in the list of `data_files` to start with, because it is more of a metadata than a data file.

Comments:
Hi ! datasets saved using `save_to_disk` should be loaded with `load_from_disk` ;)

It is convienient to just pass in a path to a local dataset or one from the hub and use the same function to load it. Is it not possible to get this fix merged in to allow this? 

We can modify `save_to_disk` to write the dataset in a structure supported by the Hub in this case, it's kind of a legacy function anyway
Title: Inconsistent output in documentation example: `num_classes` not displayed in `ClassLabel` output
State: closed

In the documentation for [ClassLabel](https://huggingface.co/docs/datasets/v2.21.0/en/package_reference/main_classes#datasets.ClassLabel), there is an example of usage with the following code:

````
from datasets import Features
features = Features({'label': ClassLabel(num_classes=3, names=['bad', 'ok', 'good'])})
features
````

which expects to output (as stated in the documentation):

````
{'label': ClassLabel(num_classes=3, names=['bad', 'ok', 'good'], id=None)}
````

but it generates the following

````
{'label': ClassLabel(names=['bad', 'ok', 'good'], id=None)}
````

If my understanding is correct, this happens because although num_classes is used during the init of the object, it is afterward ignored:

https://github.com/huggingface/datasets/blob/be5cff059a2a5b89d7a97bc04739c4919ab8089f/src/datasets/features/features.py#L975

I would like to work on this issue if this is something needed 😄


Comments:
Title: Caching shuffles by np.random.Generator results in unintiutive behavior
State: open

### Describe the bug

Create a dataset. Save it to disk. Load from disk. Shuffle, usning a `np.random.Generator`. Iterate. Shuffle again. Iterate. The iterates are different since the supplied np.random.Generator has progressed between the shuffles.

Load dataset from disk again. Shuffle and Iterate. See same result as before. Shuffle and iterate, and this time it does not have the same shuffling as ion previous run.

The motivation is I have a deep learning loop with 

```
for epoch in range(10):
    for batch in dataset.shuffle(generator=generator).iter(batch_size=32):
        .... # do stuff
```
where I want a new shuffling at every epoch. Instead I get the same shuffling.

### Steps to reproduce the bug

Run the code below two times. 

```python
import datasets
import numpy as np

generator = np.random.default_rng(0)
ds = datasets.Dataset.from_dict(mapping={"X":range(1000)})
ds.save_to_disk("tmp")
print("First loop: ", end="")
for _ in range(10):
    print(next(ds.shuffle(generator=generator).iter(batch_size=1))['X'], end=", ")
print("")

print("Second loop: ", end="")
ds = datasets.Dataset.load_from_disk("tmp")
for _ in range(10):
    print(next(ds.shuffle(generator=generator).iter(batch_size=1))['X'], end=", ")
print("")
```

The output is:

```
$ python main.py 
Saving the dataset (1/1 shards): 100%|███████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 495019.95 examples/s]
First loop: 459, 739, 72, 943, 241, 181, 845, 830, 896, 334, 
Second loop: 741, 847, 944, 795, 483, 842, 717, 865, 231, 840,
$ python main.py 
Saving the dataset (1/1 shards): 100%|████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 22243.40 examples/s]
First loop: 459, 739, 72, 943, 241, 181, 845, 830, 896, 334, 
Second loop: 741, 741, 741, 741, 741, 741, 741, 741, 741, 741, 
```

The second loop, on the second run, only spits out "741, 741, 741...." which is *not* the desired output

### Expected behavior

I want the dataset to shuffle at every epoch since I provide it with a generator for shuffling. 

### Environment info

Datasets version 2.21.0
Ubuntu linux.

Comments:
I first thought this was a mistake of mine, and also posted on stack overflow. https://stackoverflow.com/questions/78913797/iterating-a-huggingface-dataset-from-disk-using-generator-seems-broken-how-to-d 

It seems to me the issue is the caching step in 

https://github.com/huggingface/datasets/blob/be5cff059a2a5b89d7a97bc04739c4919ab8089f/src/datasets/arrow_dataset.py#L4306-L4316

because the shuffle happens after checking the cache, the rng state won't advance if the cache is used. This is VERY confusing. Also not documented.

My proposal is that you remove the API for using a Generator, and only keep the seed-based API since that is functional and cache-compatible.

Second that, its very confusing.
Title: [interleave_dataset] sample batches from a single source at a time
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

interleave_dataset and [RandomlyCyclingMultiSourcesExamplesIterable](https://github.com/huggingface/datasets/blob/3813ce846e52824b38e53895810682f0a496a2e3/src/datasets/iterable_dataset.py#L816) enable us to sample data examples from different sources. But can we also sample batches in a similar manner (each batch only contains data from a single source)?


### Motivation

Some recent research [[1](https://blog.salesforceairesearch.com/sfr-embedded-mistral/), [2](https://arxiv.org/pdf/2310.07554)] shows that source homogenous batching can be helpful for contrastive learning. Can we add a function called `RandomlyCyclingMultiSourcesBatchesIterable` to support this functionality?

### Your contribution

I can contribute a PR. But I wonder what the best way is to test its correctness and robustness.

Comments:
Title: datasets cannot handle nested json if features is given.
State: closed

### Describe the bug

I have a json named temp.json.
```json
{"ref1": "ABC", "ref2": "DEF", "cuts":[{"cut1": 3, "cut2": 5}]}
```
I want to load it.
```python
ds = datasets.load_dataset('json', data_files="./temp.json", features=datasets.Features({
    'ref1': datasets.Value('string'),
    'ref2': datasets.Value('string'),
    'cuts': datasets.Sequence({
        "cut1": datasets.Value("uint16"),
        "cut2": datasets.Value("uint16")
    })
}))
```
The above code does not work. However, I can load it without giving features.
```python
ds = datasets.load_dataset('json', data_files="./temp.json")
```
Is it possible to load integers as uint16 to save some memory?

### Steps to reproduce the bug

As in the bug description.

### Expected behavior

The data are loaded and integers are uint16.

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.21.0
- Platform: Linux-5.15.0-118-generic-x86_64-with-glibc2.35
- Python version: 3.11.9
- `huggingface_hub` version: 0.24.5
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0

Comments:
Hi ! `Sequence` has a weird behavior for dictionaries (from tensorflow-datasets), use a regular list instead:

```python
ds = datasets.load_dataset('json', data_files="./temp.json", features=datasets.Features({
    'ref1': datasets.Value('string'),
    'ref2': datasets.Value('string'),
    'cuts': [{
        "cut1": datasets.Value("uint16"),
        "cut2": datasets.Value("uint16")
    }]
}))
```

> Hi ! `Sequence` has a weird behavior for dictionaries (from tensorflow-datasets), use a regular list instead:
> 
> ```python
> ds = datasets.load_dataset('json', data_files="./temp.json", features=datasets.Features({
>     'ref1': datasets.Value('string'),
>     'ref2': datasets.Value('string'),
>     'cuts': [{
>         "cut1": datasets.Value("uint16"),
>         "cut2": datasets.Value("uint16")
>     }]
> }))
> ```
Thank you!


It works.
Title: Stream dataset does not iterate if the batch size is larger than the dataset size (related to drop_last_batch)
State: closed

### Describe the bug

Hi there,

I use streaming and interleaving to combine multiple datasets saved in jsonl files. The size of dataset can vary (from 100ish to 100k-ish). I use dataset.map() and a big batch size to reduce the IO cost. It was working fine with datasets-2.16.1 but this problem shows up after I upgraded to datasets-2.19.2. With  2.21.0 the problem remains.

Please see the code below to reproduce the problem.

The dataset can iterate correctly if we set either streaming=False or drop_last_batch=False.

I have to use drop_last_batch=True since it's for distributed training.

### Steps to reproduce the bug

```python
# datasets==2.21.0
import datasets
def data_prepare(examples):
    print(examples["sentence1"][0])
    return examples

batch_size = 101
# the size of the dataset is 100
# the dataset iterates correctly if we set either streaming=False or drop_last_batch=False 
dataset = datasets.load_dataset("mteb/biosses-sts", split="test", streaming=True)
dataset = dataset.map(lambda x: data_prepare(x),
                      drop_last_batch=True,
                      batched=True, batch_size=batch_size)
for ex in dataset:
    print(ex)
    pass

```

### Expected behavior

The dataset iterates regardless of the batch size.

### Environment info

- `datasets` version: 2.21.0
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.14
- `huggingface_hub` version: 0.24.5
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.2.0


Comments:
That's expected behavior, it's also the same in `torch`:

```python
>>> list(DataLoader(list(range(5)), batch_size=10, drop_last=True))
[]
```
Title: cudf-cu12 24.4.1, ibis-framework 8.0.0 requires pyarrow<15.0.0a0,>=14.0.1,pyarrow<16,>=2 and datasets 2.21.0 requires pyarrow>=15.0.0
State: open

### Describe the bug

!pip install accelerate>=0.16.0 torchvision transformers>=4.25.1 datasets>=2.19.1 ftfy tensorboard Jinja2 peft==0.7.0

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
cudf-cu12 24.4.1 requires pyarrow<15.0.0a0,>=14.0.1, but you have pyarrow 17.0.0 which is incompatible.
ibis-framework 8.0.0 requires pyarrow<16,>=2, but you have pyarrow 17.0.0 which is incompatible.

to solve above error
!pip install pyarrow==14.0.1

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.21.0 requires pyarrow>=15.0.0, but you have pyarrow 14.0.1 which is incompatible.

### Steps to reproduce the bug

!pip install datasets>=2.19.1

### Expected behavior

run without dependency error

### Environment info

Diffusers version: 0.31.0.dev0

Platform: Linux-6.1.85+-x86_64-with-glibc2.35
Running on Google Colab?: Yes
Python version: 3.10.12
PyTorch version (GPU?): 2.3.1+cu121 (True)
Flax version (CPU?/GPU?/TPU?): 0.8.4 (gpu)
Jax version: 0.4.26
JaxLib version: 0.4.26
Huggingface_hub version: 0.23.5
Transformers version: 4.42.4
Accelerate version: 0.32.1
PEFT version: 0.7.0
Bitsandbytes version: not installed
Safetensors version: 0.4.4
xFormers version: not installed
Accelerator: Tesla T4, 15360 MiB
Using GPU in script?:
Using distributed or parallel set-up in script?:

Comments:
@sayakpaul  please advice 

Hits the same dependency conflict
Title: CI is broken for numpy-2: Failed to fetch wheel: llvmlite==0.34.0
State: closed

Ci is broken with error `Failed to fetch wheel: llvmlite==0.34.0`: https://github.com/huggingface/datasets/actions/runs/10466825281/job/28984414269
```
Run uv pip install --system "datasets[tests_numpy2] @ ."
Resolved 150 packages in 4.42s
error: Failed to prepare distributions
  Caused by: Failed to fetch wheel: llvmlite==0.34.0
  Caused by: Build backend failed to build wheel through `build_wheel()` with exit status: 1
--- stdout:
running bdist_wheel
/home/runner/.cache/uv/builds-v0/.tmpcyKh8S/bin/python /home/runner/.cache/uv/built-wheels-v3/pypi/llvmlite/0.34.0/wrk1bNwq1gleSiznvrSEZ/llvmlite-0.34.0.tar.gz/ffi/build.py
LLVM version...
--- stderr:
Traceback (most recent call last):
  File "/home/runner/.cache/uv/built-wheels-v3/pypi/llvmlite/0.34.0/wrk1bNwq1gleSiznvrSEZ/llvmlite-0.34.0.tar.gz/ffi/build.py", line 105, in main_posix
    out = subprocess.check_output([llvm_config, '--version'])
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/subprocess.py", line 503, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'llvm-config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/.cache/uv/built-wheels-v3/pypi/llvmlite/0.34.0/wrk1bNwq1gleSiznvrSEZ/llvmlite-0.34.0.tar.gz/ffi/build.py", line 191, in <module>
    main()
  File "/home/runner/.cache/uv/built-wheels-v3/pypi/llvmlite/0.34.0/wrk1bNwq1gleSiznvrSEZ/llvmlite-0.34.0.tar.gz/ffi/build.py", line 181, in main
    main_posix('linux', '.so')
  File "/home/runner/.cache/uv/built-wheels-v3/pypi/llvmlite/0.34.0/wrk1bNwq1gleSiznvrSEZ/llvmlite-0.34.0.tar.gz/ffi/build.py", line 107, in main_posix
    raise RuntimeError("%s failed executing, please point LLVM_CONFIG "
RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config
error: command '/home/runner/.cache/uv/builds-v0/.tmpcyKh8S/bin/python' failed with exit code 1
```

Comments:
Note that the CI before was using:
- llvmlite: 0.43.0
- numba: 0.60.0

Now it tries to use:
- llvmlite: 0.34.0
- numba: 0.51.2

The issue is because numba-0.60.0 pins numpy<2.1 and `uv` tries to install latest numpy-2.1.0 with an old numba-0.51.0 version (and llvmlite-0.34.0). See discussion in their repo:
- https://github.com/numba/numba/issues/9708

Latest numpy-2.1.0 will be supported by the next numba-0.61.0 release in September.

Note that our CI requires numba with the "audio" extra:
- librosa > numba
Title: ConnectionError for gated datasets and unauthenticated users
State: closed

Since the Hub returns dataset info for gated datasets and unauthenticated users, there is dead code: https://github.com/huggingface/datasets/blob/98fdc9e78e6d057ca66e58a37f49d6618aab8130/src/datasets/load.py#L1846-L1852

We should remove the dead code and properly handle this case: currently we are raising a `ConnectionError` instead of a `DatasetNotFoundError` (as before).

See:
- https://github.com/huggingface/dataset-viewer/issues/3025
- https://github.com/huggingface/huggingface_hub/issues/2457

Comments: