Kokolipa/github-issues huggingface dataset:
Issue: [Bug] Inconsistent behavior of data_files and data_dir in load_dataset method.
State: closed

### Describe the bug

Inconsistent operation of data_files and data_dir in load_dataset method.


### Steps to reproduce the bug

# First
I have three files, named 'train.json', 'val.json', 'test.json'.
Each one has a simple dict `{text:'aaa'}`.
Their path are `/data/train.json`,  `/data/val.json`,  `/data/test.json`
I load dataset with `data_files` argument:
```py
files = [os.path.join('./data',file) for file in os.listdir('./data')]
ds = load_dataset(
            path='json',
            data_files=files,)
```
And I get:
```py
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 3
    })
})
```
However, If I load dataset with `data_dir` argument:
```py
ds = load_dataset(
            path='json',
            data_dir='./data',)
```
And I get:
```py
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 1
    })
    validation: Dataset({
        features: ['text'],
        num_rows: 1
    })
    test: Dataset({
        features: ['text'],
        num_rows: 1
    })
})
```
Two results are not the same. Their behaviors are not equal, even if the statement [here](https://github.com/huggingface/datasets/blob/d0c152a979d91cc34b605c0298aebc650ab7dd27/src/datasets/load.py#L1790) said that their behaviors are equal. 


# Second
If some filename include 'test' while others do not, `load_dataset` only return `test` dataset and others files are **abandoned**.
Given two files named `test.json` and `1.json`
Each one has a simple dict `{text:'aaa'}`.
I load the dataset using:
```py
ds = load_dataset(
            path='json',
            data_dir='./data',)
```
Only `test` is returned, `1.json` is missing:
```py
DatasetDict({
    test: Dataset({
        features: ['text'],
        num_rows: 1
    })
})
```
Things do not change even I manually set `split='train'`

 



### Expected behavior

1. Fix the above bugs.
2. Although the document says that load_dataset method will `Find which file goes into which split (e.g. train/test) based on file and directory names or on the YAML configuration`, I hope I can manually decide whether to do so. Sometimes users may accidentally put a `test` string in the filename but they just want a single `train` dataset. If the number of files in `data_dir` is huge, it's not easy to find out what cause the second situation metioned above.

### Environment info

datasets==3.2.0
Ubuntu18.84

Hi ! `data_files` with a list is equivalent to `data_files={"train": data_files}` with a train test only.

When no split are specified, they are inferred based on file names, and files with no apparent split are ignored

Thanks for your reply!
`files with no apparent split are ignored`. Is there a option that I can choose to ignored it or not as I mention aboved? Thanks!

To include all the files, the best way is to pass `data_files` yourself. There is no option to disable split detection at the moment

Thanks! I hope you guys can consider adding this option in the future. :)
Issue: One or several metadata.jsonl were found, but not in the same directory or in a parent directory of
State: open

### Describe the bug

ImageFolder with metadata.jsonl error. I downloaded liuhaotian/LLaVA-CC3M-Pretrain-595K locally from Hugging Face. According to the tutorial in https://huggingface.co/docs/datasets/image_dataset#image-captioning, only put images.zip and metadata.jsonl containing information in the same folder. However, after loading, an error was reported: One or several metadata.jsonl were found, but not in the same directory or in a parent directory of.

The data in my jsonl file is as follows:

> {"id": "GCC_train_002448550", "file_name": "GCC_train_002448550.jpg", "conversations": [{"from": "human", "value": "<image>\nProvide a brief description of the given image."}, {"from": "gpt", "value": "a view of a city , where the flyover was proposed to reduce the increasing traffic on thursday ."}]}


### Steps to reproduce the bug

from datasets import load_dataset
image = load_dataset("imagefolder",data_dir='data/opensource_data')

### Expected behavior

success

### Environment info

datasets==3.2.0

Hmmm I double checked in the source code and I found a contradiction: in the current implementation the metadata file is ignored if it's not in the same archive as the zip image somehow:

https://github.com/huggingface/datasets/blob/caa705e8bf4bedf1a956f48b545283b2ca14170a/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L352-L353

in the tests suite the metadata file is placed inside the archive:

https://github.com/huggingface/datasets/blob/caa705e8bf4bedf1a956f48b545283b2ca14170a/tests/packaged_modules/test_imagefolder.py#L223-L223

Thanks for reporting this issue, it seems the documentation is wrong and we never implemented the support for zip + metadata outside zip. We might rewrite part of this code soon though to make it more flexible, it can be a good occasion to fix this. In the meantime feel free to open a PR to fix the documentation if you'd like
Issue: Clarify documentation or Create DatasetCard
State: open
Labels: name: enhancement

### Feature request

I noticed that you can use a Model Card instead of a Dataset Card when pushing a dataset to the Hub, but this isn’t clearly mentioned in [the docs.](https://huggingface.co/docs/datasets/dataset_card)

- Update the docs to clarify that a Model Card can work for datasets too.
- It might be worth creating a dedicated DatasetCard module, similar to the ModelCard module, for consistency and better support.

Not sure if this belongs here or on the [Hub repo](https://github.com/huggingface/huggingface_hub), but thought I’d bring it up!

### Motivation

I just spent an hour like on [this issue](https://github.com/huggingface/trl/pull/2491) trying to create a `DatasetCard` for a script.

### Your contribution

might later
Issue: Too many open files: '/root/.cache/huggingface/token'
State: open

### Describe the bug

I ran this code:
```
from datasets import load_dataset
dataset = load_dataset("common-canvas/commoncatalog-cc-by", cache_dir="/datadrive/datasets/cc", num_proc=1000)
```

And got this error.
Before it was some other file though (lie something...incomplete)

runnting
```
ulimit -n 8192
```
did not help at all.

### Steps to reproduce the bug

Run the code i sent

### Expected behavior

Should be no errors 

### Environment info

linux, jupyter lab.
Issue: TypeError: Value.__init__() missing 1 required positional argument: 'dtype'
State: open

### Describe the bug

ds = load_dataset(
    "./xxx.py",
    name="default",
    split="train",
)

The datasets does not support debugging locally anymore...

### Steps to reproduce the bug

```
from datasets import load_dataset

ds = load_dataset(
    "./repo.py",
    name="default",
    split="train",
)

for item in ds:
    print(item)
```

It works fine for "username/repo", but it does not work for "./repo.py" when debugging locally...
Running above code template will report TypeError: Value.__init__() missing 1 required positional argument: 'dtype'

### Expected behavior

fix this bug

### Environment info

python 3.10 datasets==2.21

same error 
```
data = load_dataset('/opt/deepseek_R1_finetune/hf_datasets/openai/gsm8k', 'main')[split] 
```

> same error
> 
> ```
> data = load_dataset('/opt/deepseek_R1_finetune/hf_datasets/openai/gsm8k', 'main')[split] 
> ```

https://github.com/huggingface/open-r1/issues/204  this help me

Solved by delete `dataset_infos.json` file in dataset dir, or you can transfer datasets from Hugginface to Modelscope by [hf-ms-transfer](https://github.com/wa008/hf-ms-transfer), which will solve this problem by default.
Issue: .map() is not caching and ram goes OOM
State: open

### Describe the bug

Im trying to run a fairly simple map that is converting a dataset into numpy arrays. however, it just piles up on memory and doesnt write to disk. Ive tried multiple cache techniques such as specifying the cache dir, setting max mem, +++ but none seem to work. What am I missing here?

### Steps to reproduce the bug

```
from pydub import AudioSegment
import io
import base64
import numpy as np
import os
CACHE_PATH = "/mnt/extdisk/cache" # "/root/.cache/huggingface/"# 
os.environ["HF_HOME"] = CACHE_PATH
import datasets
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# Create a handler for Jupyter notebook
handler = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)

#datasets.config.IN_MEMORY_MAX_SIZE= 1000#*(2**30) #50 gb
print(datasets.config.HF_CACHE_HOME)
print(datasets.config.HF_DATASETS_CACHE)
# Decode the base64 string into bytes
def convert_mp3_to_audio_segment(example):
    """
    example = ds['train'][0]
    """
    try:
        audio_data_bytes = base64.b64decode(example['audio'])
        # Use pydub to load the MP3 audio from the decoded bytes
        audio_segment = AudioSegment.from_file(io.BytesIO(audio_data_bytes), format="mp3")
        # Resample to 24_000
        audio_segment = audio_segment.set_frame_rate(24_000)
        audio = {'sampling_rate' : audio_segment.frame_rate,
        'array' : np.array(audio_segment.get_array_of_samples(), dtype="float")}
        del audio_segment
        duration = len(audio['array']) / audio['sampling_rate']
    except Exception as e:
        logger.warning(f"Failed to convert audio for {example['id']}. Error: {e}")
        audio = {'sampling_rate' : 0,
        'array' : np.array([]), duration : 0}
    return {'audio' : audio, 'duration' : duration}

ds = datasets.load_dataset("NbAiLab/nb_distil_speech_noconcat_stortinget", cache_dir=CACHE_PATH, keep_in_memory=False)

#%%
num_proc=32
ds_processed = (
    ds
    #.select(range(10))
    .map(convert_mp3_to_audio_segment, num_proc=num_proc, desc="Converting mp3 to audio segment") #, cache_file_name=f"{CACHE_PATH}/stortinget_audio" # , cache_file_name="test"
)
```

### Expected behavior

the map should write to disk

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-6.8.0-45-generic-x86_64-with-glibc2.39
- Python version: 3.12.7
- `huggingface_hub` version: 0.26.3
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

I have the same issue - any update on this?
Issue: Remove upper bound for fsspec
State: open

### Describe the bug

As also raised by @cyyever in https://github.com/huggingface/datasets/pull/7296 and @NeilGirdhar in https://github.com/huggingface/datasets/commit/d5468836fe94e8be1ae093397dd43d4a2503b926#commitcomment-140952162 , `datasets` has a problematic version constraint on `fsspec`.

In our case this causes (unnecessary?) troubles due to a race condition bug in that version of the corresponding `gcsfs` plugin, that causes deadlocks: https://github.com/fsspec/gcsfs/pull/643

We just use a version override to ignore the constraint from `datasets`, but imho the version constraint could just be removed in the first place?

The last few PRs bumping the upper bound were basically uneventful:

* https://github.com/huggingface/datasets/pull/7219
* https://github.com/huggingface/datasets/pull/6921
* https://github.com/huggingface/datasets/pull/6747

### Steps to reproduce the bug

-

### Expected behavior

Installing `fsspec>=2024.10.0` along `datasets` should be possible without overwriting constraints.

### Environment info

All recent datasets versions

Unfortunately `fsspec` versioning allows breaking changes across version and there is no way we can keep it without constrains at the moment. It already broke `datasets` once in the past. Maybe one day once `fsspec` decides on a stable and future proof API but I don't think this will happen anytime soon

edit: bumped to 2024.10.0 in https://github.com/huggingface/datasets/pull/7352
Issue: Unexpected cache behaviour using load_dataset
State: closed

### Describe the bug

Following the (Cache management)[https://huggingface.co/docs/datasets/en/cache] docu and previous behaviour from datasets version 2.18.0, one is able to change the cache directory. Previously, all downloaded/extracted/etc files were found in this folder. As i have recently update to the latest version this is not the case anymore. Downloaded files are stored in `~/.cache/huggingface/hub`.
Providing the `cache_dir` argument in `load_dataset` the cache directory is created and there are some files but the bulk is still in `~/.cache/huggingface/hub`.

I believe this could be solved by adding the cache_dir argument [here](https://github.com/huggingface/datasets/blob/fdda5585ab18ea1292547f36c969d12c408ab842/src/datasets/utils/file_utils.py#L188)


### Steps to reproduce the bug

For example using https://huggingface.co/datasets/ashraq/esc50:
```python
from datasets import load_dataset
ds = load_dataset("ashraq/esc50", "default", cache_dir="~/custom/cache/path/esc50")
```


### Expected behavior

I would expect the bulk of files related to the dataset to be stored somewhere in `~/custom/cache/path/esc50`, but it seems they are in `~/.cache/huggingface/hub/datasets--ashraq--esc50`. 

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.14.0-503.15.1.el9_5.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.14
- `huggingface_hub` version: 0.26.5
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.6.1

Hi ! Since `datasets` 3.x,  the `datasets` specific files are in `cache_dir=` and the HF files are cached using `huggingface_hub` and you can set its cache directory using the `HF_HOME` environment variable.

They are independent, for example you can delete the Hub cache (containing downloaded files) but still reload your cached datasets from the `datasets` cache (containing prepared datasets in Arrow format)
Issue: ImportError: cannot import name 'set_caching_enabled' from 'datasets' 
State: open

### Describe the bug

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/local/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/home/Medusa/axolotl/src/axolotl/cli/__init__.py", line 23, in <module>
    from axolotl.train import TrainDatasetMeta
  File "/home/Medusa/axolotl/src/axolotl/train.py", line 23, in <module>
    from axolotl.utils.trainer import setup_trainer
  File "/home/Medusa/axolotl/src/axolotl/utils/trainer.py", line 13, in <module>
    from datasets import set_caching_enabled
ImportError: cannot import name 'set_caching_enabled' from 'datasets' (/usr/local/lib/python3.10/site-packages/datasets/__init__.py)

### Steps to reproduce the bug

1、axolotl
2、accelerate launch -m axolotl.cli.train examples/medusa/qwen_lora_stage1.yml

### Expected behavior

enable datasets

### Environment info

python3.10

pip install datasets==2.18.0

Hi ! I think you need to update axolotl
Issue: Introduce support for PDFs
State: open
Labels: name: enhancement

### Feature request

The idea (discussed in the Discord server with @lhoestq ) is to have a Pdf type like Image/Audio/Video. For example [Video](https://github.com/huggingface/datasets/blob/main/src/datasets/features/video.py) was recently added and contains how to decode a video file encoded in a dictionary like {"path": ..., "bytes": ...} as a VideoReader using decord. We want to do the same with pdf and get a [pypdfium2.PdfDocument](https://pypdfium2.readthedocs.io/en/stable/_modules/pypdfium2/_helpers/document.html#PdfDocument).

### Motivation

In many cases PDFs contain very valuable information beyond text (e.g. images, figures). Support for PDFs would help create datasets where all the information is preserved.

### Your contribution

I can start the implementation of the Pdf type :)

#self-assign

Awesome ! Let me know if you have any question or if I can help :)

cc @AndreaFrancis as well for viz

Other candidates libraries for the Pdf type: PyMuPDF pypdf and pdfplumber

EDIT: Pymupdf looks like a good choice when it comes to maturity + performance + versatility BUT the license is maybe an issue, and pypdf, pypdfium2 or pdfplumber are good options imo

Related to https://github.com/huggingface/datasets/issues/7058

PyMuPDF is AGPL licensed, so we can't use it. I will move forward with [pdfplumber](https://github.com/jsvine/pdfplumber?tab=readme-ov-file#python-library).

Hi both! I have made a pull request with a first basic implementation of the Pdf feature. I followed closely what I saw on the Video and Image features. It is my first time contributing so any comments are very welcomed. I think it would be useful to outline together what additional things we can implement (e.g. enabling parsing of the pdf). Thanks :)
Issue: Cannot create a dataset with relative audio path
State: open

### Describe the bug

Hello! I want to create a dataset of parquet files, with audios stored as separate .mp3 files. However, it says "No such file or directory" (see the reproducing code).

### Steps to reproduce the bug

Creating a dataset
```
from pathlib import Path
from datasets import Dataset, load_dataset, Audio

Path('my_dataset/audio').mkdir(parents=True, exist_ok=True)
Path('my_dataset/audio/file.mp3').touch(exist_ok=True)
Dataset.from_list(
    [{'audio': {'path': 'audio/file.mp3'}}]
).to_parquet('my_dataset/data.parquet')
```

Result:
```
# my_dataset
# ├── audio
# │   └── file.mp3
# └── data.parquet
```

Trying to load the dataset
```
dataset = (
    load_dataset('my_dataset', split='train')
    .cast_column('audio', Audio(sampling_rate=16_000))
)
dataset[0]

>>> FileNotFoundError: [Errno 2] No such file or directory: 'audio/file.mp3'
```

### Expected behavior

I expect the dataset to load correctly.

I've found 2 workarounds, but they are not very good:
1. I can specify an absolute path to the audio, however, when I move the folder or upload to HF it will stop working.
2. I can set `'path': 'file.mp3'`, and load with `load_dataset('my_dataset', data_dir='audio')` - it seems to work, but does this mean that anyone from Hugging Face who wants to use this dataset should also pass the `data_dir` argument, otherwise it won't work?

### Environment info

datasets 3.1.0, Ubuntu 24.04.1

Hello ! when you `cast_column` you need the paths to be absolute paths or relative paths to your working directory, not the original dataset directory.

Though I'd recommend structuring your dataset as an AudioFolder which automatically links a metadata.jsonl or csv to the audio files via relative paths **within** the dataset repository: https://huggingface.co/docs/datasets/v3.2.0/en/audio_load#audiofolder

@lhoestq thank you, but there are two problems with using AudioFolder:
1. It is said that AudioFolder requires metadata.csv. However, my datset is too large and contains nested and np.ndarray fields, so I can't use csv.
2. It is said that I need to load the dataset with `load_dataset("audiofolder", ...)`. However, if possible, I want my dataset to be loaded as usual with `load_dataset(dataset_name)` after I upload if to HF.

You can use metadata.jsonl if you have nested data :)

And actually if you have a dataset structured as an AudioFolder then `load_dataset(dataset_name)` does work after uploading to HF

I have created an audio dataset. In my repo, I have explained the steps and structure. An example dataset is also available in the repo. https://github.com/pr0mila/ParquetToHuggingFace
Issue: How to get the original dataset name with username?
State: open
Labels: name: enhancement

### Feature request

The issue is related to ray data https://github.com/ray-project/ray/issues/49008 which it requires to check if the dataset is the original one just after `load_dataset` and parquet files are already available on hf hub.

The solution used now is to get the dataset name, config and split, then `load_dataset` again and check the fingerprint. But it's unable to get the correct dataset name if it contains username. So how to get the dataset name with username prefix, or is there another way to query if a dataset is the original one with parquet available?

@lhoestq 

### Motivation

https://github.com/ray-project/ray/issues/49008

### Your contribution

Would like to fix that.

Hi ! why not pass the dataset id to Ray and let it check the parquet files ? Or pass the parquet files lists directly ?

I'm not sure why ray design an API like this to accept a `Dataset` object, so they need to verify the `Dataset` is the original one and use the `DatasetInfo` to query the huggingface hub. I'll advise the ray data team to use dataset id instead of dataset for the `HuggingFaceDatasource` API.
Issue: Enable the Audio Feature to decode / read with an offset + duration
State: open
Labels: name: enhancement

### Feature request

For most large speech dataset, we do not wish to generate hundreds of millions of small audio samples. Instead, it is quite common to provide larger audio files with frame offset (soundfile start and stop arguments). We should be able to pass these arguments to Audio() (column ID corresponding in the dataset row).

### Motivation

I am currently generating a fairly big dataset to .parquet(). Unfortunately, it does not work because all existing functions load the whole .wav file corresponding to the row. All my attempts at bypassing this failed. We should be able to put in the Table only the bytes corresponding to what soundfile reads with an offset (and subset of the audio file). 

### Your contribution

I can totally test whatever code on my large dataset creation script. 

Hi ! What about having audio + start + duration columns and enable something like this ?

```python
for example in ds:
    array = example["audio"].read(start=example["start"], frames=example["duration"])
```

Hi @lhoestq, this would work with a file-based dataset but would be terrible for a sharded one as it would duplicate the large audio file many times. Also, very long audio files are not embedded very well in the parquet file, even with large_binary(). It crashed a few times for me until I switched to one sample == one file :-(
Issue: Creating new dataset from list loses information. (Audio Information Lost - either Datatype or Values).
State: open

### Describe the bug

When creating a dataset from a list of datapoints, information is lost of the individual items.

Specifically, when creating a dataset from a list of datapoints (from another dataset). Either the datatype is lost or the values are lost. See examples below. 

-> What is the best way to create a dataset from a list of datapoints?

---
e.g.:
**When running this code:**
```python
from datasets import load_dataset, Dataset
commonvoice_data = load_dataset("mozilla-foundation/common_voice_17_0", "it", split="test", streaming=True)
datapoint = next(iter(commonvoice_data))
out = [datapoint]
new_data = Dataset.from_list(out) #this loses datatype information
new_data2= Dataset.from_list(out,features=commonvoice_data.features) #this loses value information
```

**We get the following**:
---
1. `datapoint`: (the original datapoint)
```
'audio': {'path': 'it_test_0/common_voice_it_23606167.mp3', 'array': array([0.00000000e+00, 0.00000000e+00, 0.00000000e+00, ...,
       2.21619011e-05, 2.72628222e-05, 0.00000000e+00]), 'sampling_rate': 48000}
 ```
Original Dataset Features:
```
>>> commonvoice_data.features
'audio': Audio(sampling_rate=48000, mono=True, decode=True, id=None)
```
 - Here we see column "audio", has the proper values (both `path` & and `array`) and has the correct datatype (Audio).

 
 ----
 2. new_data[0]:
```
# Cannot be printed (as it prints the entire array).
```
New Dataset 1 Features:
```
>>> new_data.features
'audio': {'array': Sequence(feature=Value(dtype='float64', id=None), length=-1, id=None), 'path': Value(dtype='string', id=None), 'sampling_rate': Value(dtype='int64', id=None)}
```
 - Here we see that the column "audio", has the correct values, but is not the Audio datatype anymore.

---
3. new_data2[0]:
```
'audio': {'path': None, 'array': array([0., 0., 0., ..., 0., 0., 0.]), 'sampling_rate': 48000},
```
New Dataset 2 Features:
```
>>> new_data2.features
'audio': Audio(sampling_rate=48000, mono=True, decode=True, id=None),
```
 - Here we see that the column "audio", has the correct datatype, but all the array & path values were lost!



### Steps to reproduce the bug

## Run:
```python
from datasets import load_dataset, Dataset
commonvoice_data = load_dataset("mozilla-foundation/common_voice_17_0", "it", split="test", streaming=True)
datapoint = next(iter(commonvoice_data))
out = [datapoint]
new_data = Dataset.from_list(out) #this loses datatype information
new_data2= Dataset.from_list(out,features=commonvoice_data.features) #this loses value information
```





### Expected behavior

## Expected:
```datapoint == new_data[0]```

AND

```datapoint == new_data2[0]```

### Environment info

- `datasets` version: 3.1.0
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.26.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1
Issue:  Build Documentation Test Fails Due to "Bad Credentials" Error
State: open

### Describe the bug

The `Build documentation / build / build_main_documentation (push)` job is consistently failing during the "Syncing repository" step. The error occurs when attempting to determine the default branch name, resulting in "Bad credentials" errors.

### Steps to reproduce the bug

1. Trigger the `build_main_documentation` job.
2. Observe the logs during the "Syncing repository" step.

### Expected behavior

The workflow should be able to retrieve the default branch name without encountering credential issues.

### Environment info

```plaintext
Syncing repository: huggingface/notebooks
Getting Git version info
Temporarily overriding HOME='/home/runner/work/_temp/00e62748-9940-4a4f-bbbc-eb2cda6d7ed6' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
/usr/bin/git config --global --add safe.directory /home/runner/work/datasets/datasets/notebooks
Initializing the repository
Disabling automatic garbage collection
Setting up auth
Determining the default branch
  Retrieving the default branch name
  Bad credentials - https://docs.github.com/rest
  Waiting 20 seconds before trying again
  Retrieving the default branch name
  Bad credentials - https://docs.github.com/rest
  Waiting 19 seconds before trying again
  Retrieving the default branch name
  Error: Bad credentials - https://docs.github.com/rest
```

how were you able to fix this please?

> how were you able to fix this please?

I was not able to fix this.
Issue: DataFilesNotFoundError for datasets LM1B
State: closed

### Describe the bug

Cannot load the dataset https://huggingface.co/datasets/billion-word-benchmark/lm1b

### Steps to reproduce the bug

`dataset = datasets.load_dataset('lm1b', split=split)`

### Expected behavior

`Traceback (most recent call last):
  File "/home/hml/projects/DeepLearning/Generative_model/Diffusion-BERT/word_freq.py", line 13, in <module>
    train_data = DiffusionLoader(tokenizer=tokenizer).my_load(task_name='lm1b', splits=['train'])[0]
  File "/home/hml/projects/DeepLearning/Generative_model/Diffusion-BERT/dataloader.py", line 20, in my_load
    return [self._load(task_name, name) for name in splits]
  File "/home/hml/projects/DeepLearning/Generative_model/Diffusion-BERT/dataloader.py", line 20, in <listcomp>
    return [self._load(task_name, name) for name in splits]
  File "/home/hml/projects/DeepLearning/Generative_model/Diffusion-BERT/dataloader.py", line 13, in _load
    dataset = datasets.load_dataset('lm1b', split=split)
  File "/home/hml/.conda/envs/DB/lib/python3.10/site-packages/datasets/load.py", line 2594, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/hml/.conda/envs/DB/lib/python3.10/site-packages/datasets/load.py", line 2266, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/hml/.conda/envs/DB/lib/python3.10/site-packages/datasets/load.py", line 1827, in dataset_module_factory
    ).get_module()
  File "/home/hml/.conda/envs/DB/lib/python3.10/site-packages/datasets/load.py", line 1040, in get_module
    module_name, default_builder_kwargs = infer_module_for_data_files(
  File "/home/hml/.conda/envs/DB/lib/python3.10/site-packages/datasets/load.py", line 598, in infer_module_for_data_files
    raise DataFilesNotFoundError("No (supported) data files found" + (f" in {path}" if path else ""))
datasets.exceptions.DataFilesNotFoundError: No (supported) data files found in lm1b`

### Environment info

datasets: 2.20.0

Hi ! Can you try with a more recent version of `datasets` ? Also you might need to pass trust_remote_code=True since it's a script based dataset
Issue: Efficient Image Augmentation in Hugging Face Datasets
State: open

### Describe the bug

 I'm using the Hugging Face datasets library to load images in batch and would like to apply a torchvision transform to solve the inconsistent image sizes in the dataset and apply some on the fly image augmentation. I can just think about using the collate_fn, but seems quite inefficient. 
 
 I'm new to the Hugging Face datasets library, I didn't find nothing in the documentation or the issues here on github.

Is there an existing way to add image transformations directly to the dataset loading pipeline? 

### Steps to reproduce the bug

from datasets import load_dataset
from torch.utils.data import DataLoader

```python
def collate_fn(batch):
    images = [item['image'] for item in batch]
    texts = [item['text'] for item in batch]
    return {
        'images': images,
        'texts': texts
    }

dataset = load_dataset("Yuki20/pokemon_caption", split="train")
dataloader = DataLoader(dataset, batch_size=4, collate_fn=collate_fn)

# Output shows varying image sizes:
# [(1280, 1280), (431, 431), (789, 789), (769, 769)]
```

### Expected behavior

I'm looking for a way to resize images on-the-fly when loading the dataset, similar to PyTorch's Dataset.__getitem__ functionality. This would be more efficient than handling resizing in the collate_fn.


### Environment info

- `datasets` version: 3.1.0
- Platform: Linux-6.5.0-41-generic-x86_64-with-glibc2.35
- Python version: 3.11.10
- `huggingface_hub` version: 0.26.2
- PyArrow version: 18.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
Issue: wrong return type for `IterableDataset.shard()`
State: closed

### Describe the bug

`IterableDataset.shard()` has the wrong typing for its return as `"Dataset"`. It should be `"IterableDataset"`. Makes my IDE unhappy.

### Steps to reproduce the bug

look at [the source code](https://github.com/huggingface/datasets/blob/main/src/datasets/iterable_dataset.py#L2668)?

### Expected behavior

Correct return type as `"IterableDataset"`

### Environment info

datasets==3.1.0

Oops my bad ! thanks for reporting
Issue: Why return_tensors='pt' doesn't work？
State: open

### Describe the bug

I tried to add input_ids to dataset with map(), and I used the return_tensors='pt', but why I got the callback with the type of List？
![image](https://github.com/user-attachments/assets/ab046e20-2174-4e91-9cd6-4a296a43e83c)


### Steps to reproduce the bug

![image](https://github.com/user-attachments/assets/5d504d4c-22c7-4742-99a1-9cab78739b17)

### Expected behavior

Sorry for this silly question, I'm noob on using this tool. But I think it should return a tensor value as I have used the protocol？
When I tokenize only one sentence using tokenized_input=tokenizer(input, return_tensors='pt' )，it does return in tensor type. Why doesn't it work in map()？

### Environment info

transformers>=4.41.2,<=4.45.0
datasets>=2.16.0,<=2.21.0
accelerate>=0.30.1,<=0.34.2
peft>=0.11.1,<=0.12.0
trl>=0.8.6,<=0.9.6
gradio>=4.0.0
pandas>=2.0.0
scipy
einops
sentencepiece
tiktoken
protobuf
uvicorn
pydantic
fastapi
sse-starlette
matplotlib>=3.7.0
fire
packaging
pyyaml
numpy<2.0.0


Hi ! `datasets` uses Arrow as storage backend which is agnostic to deep learning frameworks like torch. If you want to get torch tensors back, you need to do `dataset = dataset.with_format("torch")`

> Hi ! `datasets` uses Arrow as storage backend which is agnostic to deep learning frameworks like torch. If you want to get torch tensors back, you need to do `dataset = dataset.with_format("torch")`

It does work! Thanks for your suggestion!
Issue: Support for identifier-based automated split construction
State: open
Labels: name: enhancement

### Feature request

As far as I understand, automated construction of splits for hub datasets is currently based on either file names or directory structure ([as described here](https://huggingface.co/docs/datasets/en/repository_structure))

It would seem to be pretty useful to also allow splits to be based on identifiers of individual examples

This could be configured like 
{"split_name": {"column_name": [column values in split]}}

(This in turn requires unique 'index' columns, which could be explicitly supported or just assumed to be defined appropriately by the user).

I guess a potential downside would be that shards would end up spanning different splits - is this something that can be handled somehow? Would this only affect streaming from hub?

### Motivation

The main motivation would be that all data files could be stored in a single directory, and multiple sets of splits could be generated from the same data. This is often useful for large datasets with multiple distinct sets of splits.

This could all be configured via the README.md yaml configs

### Your contribution

May be able to contribute if it seems like a good idea

Hi ! You can already configure the README.md to have multiple sets of splits, e.g.

```yaml
configs:
- config_name: my_first_set_of_split
  data_files:
  - split: train
    path: *.csv
- config_name: my_second_set_of_split
  data_files:
  - split: train
    path: train-*.csv
  - split: test
    path: test-*.csv
```

Hi - I had something slightly different in mind:

Currently the yaml splits specified like this only allow specifying which filenames to pass to each split.
But what if I have a situation where I know which individual *training examples* I want to put in each split.

I could build split-specific files, however for large datasets with overlapping (e.g. multiple sets of) splits this could result in significant duplication of data.

I can see that this could actually be very much intended (i.e. to discourage overlapping splits), but wondered whether some support for handling splits based on individual identifiers is something that could be considered. 

This is not supported right now :/ Though you can load the data in two steps like this
```python
from datasets import load_dataset

full_dataset = load_dataset("username/dataset", split="train")
my_first_set_indices = load_dataset("username/dataset", "my_first_set_of_split", split="train")

my_first_set = full_dataset.select(my_first_set_indices["indices"])
```

you can create such a dataset by adapting this code for example
```python
# upload the full dataset
full_dataset.push_to_hub("username/dataset")
# then upload the indices for each set
DatasetDict({
    "train": Dataset.from_dict({"indices": [0, 1, 2, 3]}),
    "test": Dataset.from_dict({"indices": [4, 5]}),
}).push_to_hub("username/dataset", "my_first_set_of_split")
Issue: Concurrent loading in `load_from_disk` - `num_proc` as a param 
State: closed
Labels: name: enhancement

### Feature request

https://github.com/huggingface/datasets/pull/6464 mentions a `num_proc` param while loading dataset from disk, but can't find that in the documentation and code anywhere

### Motivation

Make loading large datasets from disk faster

### Your contribution

Happy to contribute if given pointers
Issue: Faulty datasets.exceptions.ExpectedMoreSplitsError
State: open

### Describe the bug

Trying to download only the 'validation' split of my dataset; instead hit the error `datasets.exceptions.ExpectedMoreSplitsError`.
Appears to be the same undesired behavior as reported in [#6939](https://github.com/huggingface/datasets/issues/6939), but with `data_files`, not `data_dir`.

Here is the Traceback:
```
Traceback (most recent call last):
  File "/home/user/app/app.py", line 12, in <module>
    ds = load_dataset('datacomp/imagenet-1k-random0.0', token=GATED_IMAGENET, data_files={'validation': 'data/val*'}, split='validation', trust_remote_code=True)
  File "/usr/local/lib/python3.10/site-packages/datasets/load.py", line 2154, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/site-packages/datasets/builder.py", line 924, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/site-packages/datasets/builder.py", line 1018, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/usr/local/lib/python3.10/site-packages/datasets/utils/info_utils.py", line 68, in verify_splits
    raise ExpectedMoreSplitsError(str(set(expected_splits) - set(recorded_splits)))
datasets.exceptions.ExpectedMoreSplitsError: {'train', 'test'}
```

Note: I am using the `data_files` argument only because I am trying to specify that I only want the 'validation' split, and the whole dataset will be downloaded even when the `split='validation'` argument is specified, unless you also specify `data_files`, as described here: https://discuss.huggingface.co/t/how-can-i-download-a-specific-split-of-a-dataset/79027


### Steps to reproduce the bug

1. Create a Space with the default blank 'gradio' SDK https://huggingface.co/new-space
2. Create a file 'app.py' that loads a dataset to only extract a 'validation' split:

`ds = load_dataset('datacomp/imagenet-1k-random0.0', token=GATED_IMAGENET, data_files={'validation': 'data/val*'}, split='validation', trust_remote_code=True)`

### Expected behavior

Downloading validation split.

### Environment info

Default environment for creating a new Space. Relevant to this bug, that is:

```
FROM docker.io/library/python:3.10@sha256:fd0fa50d997eb56ce560c6e5ca6a1f5cf8fdff87572a16ac07fb1f5ca01eb608

--> RUN pip install --no-cache-dir pip==22.3.1 && 	pip install --no-cache-dir 	datasets 	"huggingface-hub>=0.19" "hf-transfer>=0.1.4" "protobuf<4" "click<8.1" 
```
Issue: File not found error
State: open

### Describe the bug

I get a FileNotFoundError:
<img width="944" alt="image" src="https://github.com/user-attachments/assets/1336bc08-06f6-4682-a3c0-071ff65efa87">


### Steps to reproduce the bug

See screenshot.



### Expected behavior

I want to load one audiofile from the dataset. 

### Environment info

MacOs Intel 14.6.1 (23G93)
Python 3.10.9
Numpy 1.23
Datasets latest version

Link to the dataset: https://huggingface.co/datasets/MichielBontenbal/UrbanSounds
Issue: Add filename in error message when ReadError or similar occur
State: open

Please update error messages to include relevant information for debugging when loading datasets with `load_dataset()` that may have a few corrupted files. 

Whenever downloading a full dataset, some files might be corrupted (either at the source or from downloading corruption). 

However the errors often only let me know it was a tar file if `tarfile.ReadError` appears on the traceback, and I imagine similarly for other file types. 

This makes it really hard to debug which file is corrupted, and when dealing with very large datasets, it shouldn't be necessary to force download everything again.

Hi Elisa, please share the error traceback here, and if you manage to find the location in the `datasets` code where the error occurs, feel free to open a PR to add the necessary logging / improve the error message.

> please share the error traceback

I don't have access to it but it should be during [this exception](https://github.com/huggingface/datasets/blob/2049c00921c59cdeb835137a1c49639cf175af07/src/datasets/builder.py#L1643) which happens during the loading of a dataset. If one of the downloaded files is corrupted, the for loop will not yield correctly, and the error will come from, say, in the case of tar files, [this iterable](https://github.com/huggingface/datasets/blob/2049c00921c59cdeb835137a1c49639cf175af07/src/datasets/utils/file_utils.py#L1293) which has no explicit error handling that leaves clues as to which file has failed.

I only know the case for tar files, but I consider this issue could be happening across different file types too.

I think having a better error handling for this tar iterable would be useful already, maybe a simple try/except in `_iter_from_urlpath` that checks for `tarfile.ReadError` and raises an error with the `urlpath` mentioned in the error ?

I think not just from higher calls like the `_iter_from_urlpath` but directly wherever a file is attempted to be opened would be the best case, as the traceback would simply lead to that.

so maybe there should be better error messages in each dataset builder definition ? e.g. in https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/webdataset/webdataset.py for webdataset TAR archives
Issue: load_dataset
State: open

### Describe the bug

I am performing two operations I see on a hugging face tutorial (Fine-tune a language model), and I am defining every aspect inside the mapped functions, also some imports of the library because it doesnt identify anything not defined outside that function where the dataset elements are being mapped:

https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb#scrollTo=iaAJy5Hu3l_B

`- lm_datasets = tokenized_datasets.map(
    group_texts,
    batched=True,
    batch_size=batch_size,
    num_proc=4,
)

- tokenized_datasets = datasets.map(tokenize_function, batched=True, num_proc=4, remove_columns=["text"])
def tokenize_function(examples):
    model_checkpoint = 'gpt2'
    from transformers import AutoTokenizer
    tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)
    return tokenizer(examples["text"])`


### Steps to reproduce the bug

Currently handle all the imports inside the function

### Expected behavior

The code must work es expected in the notebook, but currently this is not happening.

https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb#scrollTo=iaAJy5Hu3l_B

### Environment info

print(transformers.__version__)

4.46.1
Issue: load_from_disk
State: open

### Describe the bug

I have data saved with save_to_disk. The data is big (700Gb). When I try loading it, the only option is load_from_disk, and this function copies the data to a tmp directory, causing me to run out of disk space. Is there an alternative solution to that?

### Steps to reproduce the bug

when trying to load data using load_From_disk after being saved using save_to_disk 

### Expected behavior

run out of disk space

### Environment info

lateest version

Hello, It's an interesting issue here. I have the same problem, I have a local dataset and I want to push the dataset to the hub but huggingface does a copy of it.

```py
from datasets import load_dataset

dataset = load_dataset("webdataset", data_files="/media/works/data/*.tar") # copy here
dataset.push_to_hub("WaveGenAI/audios2")
```

Edit: I can use HfApi for my use case


Is there any update on this issue? I found the same behavior too.
My datasets version is `2.13.2`

Updating to the newest version of datasets lib resolved the issue.
Issue: Source installation fails on Macintosh with python 3.10
State: open

### Describe the bug

Hi, 

Decord is a dev dependency not maintained since couple years. 
It does not have an ARM package available rendering it uninstallable on non-intel based macs 

Suggestion is to move to eva-decord (https://github.com/georgia-tech-db/eva-decord)  which doesnt have this problem. 

Happy to raise a PR


### Steps to reproduce the bug

Source installation as mentioned in contributinog.md

### Expected behavior

Installation without decord failing to be installed.

### Environment info

python=3.10, M3 Mac 

I encountered the same problem on M1, a workaround I did was to simply comment out the dependency:

```python
...
    "zstandard",
    "polars[timezone]>=0.20.0",
    # "decord==0.6.0",
]
```

This worked for me as the adjustments I did to the code do not use the dependency, but I do not know if the same holds for you.

I also do not think it is a good idea to rely on a dependency (I mean decord) that has not been maintained for 2 years, but I saw that even eva-decord hasn't been maintained since last year.

Did you get it to work with eva-decord?
Issue: The dataset viewer should be available soon. Please retry later.
State: closed

### Describe the bug

After waiting for 2 hours, it still presents ``The dataset viewer should be available soon. Please retry later.''

### Steps to reproduce the bug

dataset link: https://huggingface.co/datasets/BryanW/HI_EDIT

### Expected behavior

Present the dataset viewer.

### Environment info

NA

Waiting is all you need. 10 hours later, it works.
Issue: Cannot load the cache when mapping the dataset
State: open

### Describe the bug

I'm training the flux controlnet. The train_dataset.map() takes long time to finish. However, when I killed one training process and want to restart a new training with the same dataset. I can't reuse the mapped result even I defined the cache dir for the dataset.

with accelerator.main_process_first():
        from datasets.fingerprint import Hasher

        # fingerprint used by the cache for the other processes to load the result
        # details: https://github.com/huggingface/diffusers/pull/4038#discussion_r1266078401
        new_fingerprint = Hasher.hash(args)
        train_dataset = train_dataset.map(
            compute_embeddings_fn, batched=True, new_fingerprint=new_fingerprint, batch_size=10,
        )

### Steps to reproduce the bug

train flux controlnet and start again



### Expected behavior

will not map again

### Environment info

latest diffusers



@zhangn77 Hi ，have you solved this problem? I encountered the same issue during training. Could we discuss it?

I also encountered the same problem, why is that？
Issue: cache can't cleaned or disabled
State: open

### Describe the bug

I tried following ways, the cache can't be disabled. 

I got 2T data, but I also got more than 2T cache file. I got pressure on storage. I need to diable cache or cleaned immediately after processed. Following ways are all not working, please give some help!

```python
from datasets import disable_caching
from transformers import AutoTokenizer
disable_caching()

tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_path)
def tokenization_fn(examples):
    column_name = 'text' if 'text' in examples else 'data'
    tokenized_inputs = tokenizer(
        examples[column_name], return_special_tokens_mask=True, truncation=False,
        max_length=tokenizer.model_max_length
    )
    return tokenized_inputs

data = load_dataset('json', data_files=save_local_path, split='train', cache_dir=None)
data.cleanup_cache_files()
updated_dataset = data.map(tokenization_fn, load_from_cache_file=False)
updated_dataset .cleanup_cache_files()
```

### Expected behavior

no cache file generated

### Environment info

Ubuntu 20.04.6 LTS
datasets  3.0.2

Hey I have a similar problem and found a workaround using [temporary directories](https://docs.python.org/3/library/tempfile.html):

```python
from tempfile import TemporaryDirectory

with TemporaryDirectory() as cache_dir:
    data = load_dataset('json', data_files=save_local_path, split='train', cache_dir=cache_dir)
```

But I do agree that it would be more intuitive if `datasets` supported this directly. Especially `disable_caching` is confusing, since it basically doesn't disable caching.