SurAyush/github_issues huggingface dataset:
Issue: Support for identifier-based automated split construction
State: open
Author: alex-hh

### Feature request

As far as I understand, automated construction of splits for hub datasets is currently based on either file names or directory structure ([as described here](https://huggingface.co/docs/datasets/en/repository_structure))

It would seem to be pretty useful to also allow splits to be based on identifiers of individual examples

This could be configured like 
{"split_name": {"column_name": [column values in split]}}

(This in turn requires unique 'index' columns, which could be explicitly supported or just assumed to be defined appropriately by the user).

I guess a potential downside would be that shards would end up spanning different splits - is this something that can be handled somehow? Would this only affect streaming from hub?

### Motivation

The main motivation would be that all data files could be stored in a single directory, and multiple sets of splits could be generated from the same data. This is often useful for large datasets with multiple distinct sets of splits.

This could all be configured via the README.md yaml configs

### Your contribution

May be able to contribute if it seems like a good idea

Comments:
Hi ! You can already configure the README.md to have multiple sets of splits, e.g.

```yaml
configs:
- config_name: my_first_set_of_split
  data_files:
  - split: train
    path: *.csv
- config_name: my_second_set_of_split
  data_files:
  - split: train
    path: train-*.csv
  - split: test
    path: test-*.csv
```

Hi - I had something slightly different in mind:

Currently the yaml splits specified like this only allow specifying which filenames to pass to each split.
But what if I have a situation where I know which individual *training examples* I want to put in each split.

I could build split-specific files, however for large datasets with overlapping (e.g. multiple sets of) splits this could result in significant duplication of data.

I can see that this could actually be very much intended (i.e. to discourage overlapping splits), but wondered whether some support for handling splits based on individual identifiers is something that could be considered. 

This is not supported right now :/ Though you can load the data in two steps like this
```python
from datasets import load_dataset

full_dataset = load_dataset("username/dataset", split="train")
my_first_set_indices = load_dataset("username/dataset", "my_first_set_of_split", split="train")

my_first_set = full_dataset.select(my_first_set_indices["indices"])
```

you can create such a dataset by adapting this code for example
```python
# upload the full dataset
full_dataset.push_to_hub("username/dataset")
# then upload the indices for each set
DatasetDict({
    "train": Dataset.from_dict({"indices": [0, 1, 2, 3]}),
    "test": Dataset.from_dict({"indices": [4, 5]}),
}).push_to_hub("username/dataset", "my_first_set_of_split")
Issue: Concurrent loading in `load_from_disk` - `num_proc` as a param 
State: closed
Author: unography

### Feature request

https://github.com/huggingface/datasets/pull/6464 mentions a `num_proc` param while loading dataset from disk, but can't find that in the documentation and code anywhere

### Motivation

Make loading large datasets from disk faster

### Your contribution

Happy to contribute if given pointers

Comments:
Issue: support for custom feature encoding/decoding
State: closed
Author: alex-hh

Fix for https://github.com/huggingface/datasets/issues/7220 as suggested in discussion, in preference to #7221 

(only concern would be on effect on type checking with custom feature types that aren't covered by FeatureType?)

Comments:
@lhoestq 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7284). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue: Faulty datasets.exceptions.ExpectedMoreSplitsError
State: open
Author: meg-huggingface

### Describe the bug

Trying to download only the 'validation' split of my dataset; instead hit the error `datasets.exceptions.ExpectedMoreSplitsError`.
Appears to be the same undesired behavior as reported in [#6939](https://github.com/huggingface/datasets/issues/6939), but with `data_files`, not `data_dir`.

Here is the Traceback:
```
Traceback (most recent call last):
  File "/home/user/app/app.py", line 12, in <module>
    ds = load_dataset('datacomp/imagenet-1k-random0.0', token=GATED_IMAGENET, data_files={'validation': 'data/val*'}, split='validation', trust_remote_code=True)
  File "/usr/local/lib/python3.10/site-packages/datasets/load.py", line 2154, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/site-packages/datasets/builder.py", line 924, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/site-packages/datasets/builder.py", line 1018, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/usr/local/lib/python3.10/site-packages/datasets/utils/info_utils.py", line 68, in verify_splits
    raise ExpectedMoreSplitsError(str(set(expected_splits) - set(recorded_splits)))
datasets.exceptions.ExpectedMoreSplitsError: {'train', 'test'}
```

Note: I am using the `data_files` argument only because I am trying to specify that I only want the 'validation' split, and the whole dataset will be downloaded even when the `split='validation'` argument is specified, unless you also specify `data_files`, as described here: https://discuss.huggingface.co/t/how-can-i-download-a-specific-split-of-a-dataset/79027


### Steps to reproduce the bug

1. Create a Space with the default blank 'gradio' SDK https://huggingface.co/new-space
2. Create a file 'app.py' that loads a dataset to only extract a 'validation' split:

`ds = load_dataset('datacomp/imagenet-1k-random0.0', token=GATED_IMAGENET, data_files={'validation': 'data/val*'}, split='validation', trust_remote_code=True)`

### Expected behavior

Downloading validation split.

### Environment info

Default environment for creating a new Space. Relevant to this bug, that is:

```
FROM docker.io/library/python:3.10@sha256:fd0fa50d997eb56ce560c6e5ca6a1f5cf8fdff87572a16ac07fb1f5ca01eb608

--> RUN pip install --no-cache-dir pip==22.3.1 && 	pip install --no-cache-dir 	datasets 	"huggingface-hub>=0.19" "hf-transfer>=0.1.4" "protobuf<4" "click<8.1" 
```

Comments:
Issue: File not found error
State: open
Author: MichielBontenbal

### Describe the bug

I get a FileNotFoundError:
<img width="944" alt="image" src="https://github.com/user-attachments/assets/1336bc08-06f6-4682-a3c0-071ff65efa87">


### Steps to reproduce the bug

See screenshot.



### Expected behavior

I want to load one audiofile from the dataset. 

### Environment info

MacOs Intel 14.6.1 (23G93)
Python 3.10.9
Numpy 1.23
Datasets latest version

Comments:
Link to the dataset: https://huggingface.co/datasets/MichielBontenbal/UrbanSounds
Issue: Add filename in error message when ReadError or similar occur
State: open
Author: elisa-aleman

Please update error messages to include relevant information for debugging when loading datasets with `load_dataset()` that may have a few corrupted files. 

Whenever downloading a full dataset, some files might be corrupted (either at the source or from downloading corruption). 

However the errors often only let me know it was a tar file if `tarfile.ReadError` appears on the traceback, and I imagine similarly for other file types. 

This makes it really hard to debug which file is corrupted, and when dealing with very large datasets, it shouldn't be necessary to force download everything again.

Comments:
Hi Elisa, please share the error traceback here, and if you manage to find the location in the `datasets` code where the error occurs, feel free to open a PR to add the necessary logging / improve the error message.

> please share the error traceback

I don't have access to it but it should be during [this exception](https://github.com/huggingface/datasets/blob/2049c00921c59cdeb835137a1c49639cf175af07/src/datasets/builder.py#L1643) which happens during the loading of a dataset. If one of the downloaded files is corrupted, the for loop will not yield correctly, and the error will come from, say, in the case of tar files, [this iterable](https://github.com/huggingface/datasets/blob/2049c00921c59cdeb835137a1c49639cf175af07/src/datasets/utils/file_utils.py#L1293) which has no explicit error handling that leaves clues as to which file has failed.

I only know the case for tar files, but I consider this issue could be happening across different file types too.

I think having a better error handling for this tar iterable would be useful already, maybe a simple try/except in `_iter_from_urlpath` that checks for `tarfile.ReadError` and raises an error with the `urlpath` mentioned in the error ?

I think not just from higher calls like the `_iter_from_urlpath` but directly wherever a file is attempted to be opened would be the best case, as the traceback would simply lead to that.

so maybe there should be better error messages in each dataset builder definition ? e.g. in https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/webdataset/webdataset.py for webdataset TAR archives
Issue: Add link to video dataset
State: closed
Author: NielsRogge

This PR updates https://huggingface.co/docs/datasets/loading to also link to the new video loading docs.

cc @mfarre 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7277). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue: load_dataset
State: open
Author: santiagobp99

### Describe the bug

I am performing two operations I see on a hugging face tutorial (Fine-tune a language model), and I am defining every aspect inside the mapped functions, also some imports of the library because it doesnt identify anything not defined outside that function where the dataset elements are being mapped:

https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb#scrollTo=iaAJy5Hu3l_B

`- lm_datasets = tokenized_datasets.map(
    group_texts,
    batched=True,
    batch_size=batch_size,
    num_proc=4,
)

- tokenized_datasets = datasets.map(tokenize_function, batched=True, num_proc=4, remove_columns=["text"])
def tokenize_function(examples):
    model_checkpoint = 'gpt2'
    from transformers import AutoTokenizer
    tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)
    return tokenizer(examples["text"])`


### Steps to reproduce the bug

Currently handle all the imports inside the function

### Expected behavior

The code must work es expected in the notebook, but currently this is not happening.

https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb#scrollTo=iaAJy5Hu3l_B

### Environment info

print(transformers.__version__)

4.46.1

Comments:
Issue: Raise error for incorrect JSON serialization
State: closed
Author: varadhbhatnagar

Raise error when `lines = False` and `batch_size < Dataset.num_rows` in `Dataset.to_json()`.

Issue: #7037 

Related PRs:
#7039 #7181 

Comments:
PTAL @lhoestq @albertvillanova 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7273). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue: Memory leak when streaming
State: open
Author: Jourdelune

### Describe the bug

I try to use a dataset with streaming=True, the issue I have is that the RAM usage becomes higher and higher until it is no longer sustainable.

I understand that huggingface store data in ram during the streaming, and more worker in dataloader there are, more a lot of shard will be stored in ram, but the issue I have is that the ram usage is not constant. So after each new shard loaded, the ram usage will be higher and higher.

### Steps to reproduce the bug

You can run this code and see you ram usage, after each shard of 255 examples, your ram usage will be extended.
```py
from datasets import load_dataset
from torch.utils.data import DataLoader

dataset = load_dataset("WaveGenAI/dataset", streaming=True)

dataloader = DataLoader(dataset["train"], num_workers=3)

for i, data in enumerate(dataloader):
    print(i, end="\r")
```

### Expected behavior

The Ram usage should be always the same (just 3 shards loaded in the ram).

### Environment info

- `datasets` version: 3.0.1
- Platform: Linux-6.10.5-arch1-1-x86_64-with-glibc2.40
- Python version: 3.12.4
- `huggingface_hub` version: 0.26.0
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1

Comments:
I seem to have encountered the same problem when loading non streaming datasets. load_from_disk. Causing hundreds of GB of memory, but the dataset actually only has 50GB

FYI when streaming parquet data, only one row group per worker is loaded in memory at a time.

Btw for datasets of embeddings you can surely optimize your RAM by reading the data as torch tensors directly instead of the default python lists

```python
from datasets import load_dataset
from torch.utils.data import DataLoader

dataset = load_dataset("WaveGenAI/dataset", streaming=True).with_format("torch")

dataloader = DataLoader(dataset["train"], num_workers=3)
```
Issue: load_from_disk
State: open
Author: ghaith-mq

### Describe the bug

I have data saved with save_to_disk. The data is big (700Gb). When I try loading it, the only option is load_from_disk, and this function copies the data to a tmp directory, causing me to run out of disk space. Is there an alternative solution to that?

### Steps to reproduce the bug

when trying to load data using load_From_disk after being saved using save_to_disk 

### Expected behavior

run out of disk space

### Environment info

lateest version

Comments:
Hello, It's an interesting issue here. I have the same problem, I have a local dataset and I want to push the dataset to the hub but huggingface does a copy of it.

```py
from datasets import load_dataset

dataset = load_dataset("webdataset", data_files="/media/works/data/*.tar") # copy here
dataset.push_to_hub("WaveGenAI/audios2")
```

Edit: I can use HfApi for my use case
Issue: Source installation fails on Macintosh with python 3.10
State: open
Author: mayankagarwals

### Describe the bug

Hi, 

Decord is a dev dependency not maintained since couple years. 
It does not have an ARM package available rendering it uninstallable on non-intel based macs 

Suggestion is to move to eva-decord (https://github.com/georgia-tech-db/eva-decord)  which doesnt have this problem. 

Happy to raise a PR


### Steps to reproduce the bug

Source installation as mentioned in contributinog.md

### Expected behavior

Installation without decord failing to be installed.

### Environment info

python=3.10, M3 Mac 

Comments:
I encountered the same problem on M1, a workaround I did was to simply comment out the dependency:

```python
...
    "zstandard",
    "polars[timezone]>=0.20.0",
    # "decord==0.6.0",
]
```

This worked for me as the adjustments I did to the code do not use the dependency, but I do not know if the same holds for you.

I also do not think it is a good idea to rely on a dependency (I mean decord) that has not been maintained for 2 years, but I saw that even eva-decord hasn't been maintained since last year.

Did you get it to work with eva-decord?
Issue: The dataset viewer should be available soon. Please retry later.
State: closed
Author: viiika

### Describe the bug

After waiting for 2 hours, it still presents ``The dataset viewer should be available soon. Please retry later.''

### Steps to reproduce the bug

dataset link: https://huggingface.co/datasets/BryanW/HI_EDIT

### Expected behavior

Present the dataset viewer.

### Environment info

NA

Comments:
Waiting is all you need. 10 hours later, it works.
Issue: Allow video with disabeld decoding without decord
State: closed
Author: lhoestq

for the viewer, this way it can use Video(decode=False) and doesn't need decord (which causes segfaults)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7262). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue: Cannot load the cache when mapping the dataset
State: open
Author: zhangn77

### Describe the bug

I'm training the flux controlnet. The train_dataset.map() takes long time to finish. However, when I killed one training process and want to restart a new training with the same dataset. I can't reuse the mapped result even I defined the cache dir for the dataset.

with accelerator.main_process_first():
        from datasets.fingerprint import Hasher

        # fingerprint used by the cache for the other processes to load the result
        # details: https://github.com/huggingface/diffusers/pull/4038#discussion_r1266078401
        new_fingerprint = Hasher.hash(args)
        train_dataset = train_dataset.map(
            compute_embeddings_fn, batched=True, new_fingerprint=new_fingerprint, batch_size=10,
        )

### Steps to reproduce the bug

train flux controlnet and start again



### Expected behavior

will not map again

### Environment info

latest diffusers



Comments:
@zhangn77 Hi ，have you solved this problem? I encountered the same issue during training. Could we discuss it?
Issue: cache can't cleaned or disabled
State: open
Author: charliedream1

### Describe the bug

I tried following ways, the cache can't be disabled. 

I got 2T data, but I also got more than 2T cache file. I got pressure on storage. I need to diable cache or cleaned immediately after processed. Following ways are all not working, please give some help!

```python
from datasets import disable_caching
from transformers import AutoTokenizer
disable_caching()

tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_path)
def tokenization_fn(examples):
    column_name = 'text' if 'text' in examples else 'data'
    tokenized_inputs = tokenizer(
        examples[column_name], return_special_tokens_mask=True, truncation=False,
        max_length=tokenizer.model_max_length
    )
    return tokenized_inputs

data = load_dataset('json', data_files=save_local_path, split='train', cache_dir=None)
data.cleanup_cache_files()
updated_dataset = data.map(tokenization_fn, load_from_cache_file=False)
updated_dataset .cleanup_cache_files()
```

### Expected behavior

no cache file generated

### Environment info

Ubuntu 20.04.6 LTS
datasets  3.0.2

Comments:
Hey I have a similar problem and found a workaround using [temporary directories](https://docs.python.org/3/library/tempfile.html):

```python
from tempfile import TemporaryDirectory

with TemporaryDirectory() as cache_dir:
    data = load_dataset('json', data_files=save_local_path, split='train', cache_dir=cache_dir)
```

But I do agree that it would be more intuitive if `datasets` supported this directly. Especially `disable_caching` is confusing, since it basically doesn't disable caching.
Issue: Don't embed videos
State: closed
Author: lhoestq

don't include video bytes when running download_and_prepare(format="parquet")

this also affects push_to_hub which will just upload the local paths of the videos though

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7259). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue: Always set non-null writer batch size
State: closed
Author: lhoestq

bug introduced in #7230, it was preventing the Viewer limit writes to work

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7258). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue: Retry all requests timeouts
State: closed
Author: lhoestq

as reported in https://github.com/huggingface/datasets/issues/6843

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7256). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue: fix decord import
State: closed
Author: lhoestq

delay the import until Video() is instantiated + also import duckdb first (otherwise importing duckdb later causes a segfault)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7255). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue: mismatch for datatypes when providing `Features` with `Array2D` and user specified `dtype` and using with_format("numpy")
State: open
Author: Akhil-CM

### Describe the bug

If the user provides a `Features` type value to `datasets.Dataset` with members having `Array2D` with a value for `dtype`, it is not respected during `with_format("numpy")` which should return a `np.array` with `dtype` that the user provided for `Array2D`. It seems for floats, it will be set to `float32` and for ints it will be set to `int64`

### Steps to reproduce the bug

```python
import numpy as np
import datasets
from datasets import Dataset, Features, Array2D

print(f"datasets version: {datasets.__version__}")

data_info = {
    "arr_float" : "float64",
    "arr_int" : "int32"
}

sample = {key : [np.zeros([4, 5], dtype=dtype)] for key, dtype in data_info.items()}

features = {key : Array2D(shape=(None, 5), dtype=dtype) for key, dtype in data_info.items()}
features = Features(features)

dataset = Dataset.from_dict(sample, features=features)

ds = dataset.with_format("numpy")
for key in features:
    print(f"{key} feature dtype: ", ds.features[key].dtype)
    print(f"{key} dtype:", ds[key].dtype)
```

Output:
```bash
datasets version: 3.0.2
arr_float feature dtype:  float64
arr_float dtype: float32
arr_int feature dtype:  int32
arr_int dtype: int64
```

### Expected behavior

It should return a `np.array` with `dtype` that the user provided for the corresponding member in the `Features` type value

### Environment info

- `datasets` version: 3.0.2
- Platform: Linux-6.11.5-arch1-1-x86_64-with-glibc2.40
- Python version: 3.12.7
- `huggingface_hub` version: 0.26.1
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0

Comments:
It seems that https://github.com/huggingface/datasets/issues/5517 is exactly the same issue.

It was mentioned there that this would be fixed in version 3.x
Issue: Unable to upload a large dataset zip either from command line or UI
State: open
Author: vakyansh

### Describe the bug

Unable to upload a large dataset zip from command line or UI. UI simply says error. I am trying to a upload a tar.gz file of 17GB.

<img width="550" alt="image" src="https://github.com/user-attachments/assets/f9d29024-06c8-49c4-a109-0492cff79d34">
<img width="755" alt="image" src="https://github.com/user-attachments/assets/a8d4acda-7f02-4279-9c2d-b2e0282b4faa">


### Steps to reproduce the bug

Upload a large file 

### Expected behavior

The file should upload without any issue.

### Environment info

None

Comments: