ahmedfarazsyk/datasets-github-issues huggingface dataset:
Different behaviour of IterableDataset.map vs Dataset.map with remove_columns
Author: vttrifonov
State: closed

### Describe the bug

The following code

```python
import datasets as hf
ds1 = hf.Dataset.from_list([{'i': i} for i in [0,1]])
#ds1 = ds1.to_iterable_dataset()
ds2 = ds1.map(
    lambda i: {'i': i+1},
    input_columns = ['i'],
    remove_columns = ['i']
)   
list(ds2)
```

produces

```python
[{'i': 1}, {'i': 2}]
```

as expected. If the line that converts `ds1` to iterable is uncommented so that the `ds2` is a map of an `IterableDataset`, the result is 

```python
[{},{}]
```

I expected the output to be the same as before. It seems that in the second case the removed column is not added back into the output.

The issue seems to be [here](https://github.com/huggingface/datasets/blob/6c6a82a573f946c4a81069f56446caed15cee9c2/src/datasets/iterable_dataset.py#L1093): the columns are removed after the mapping which is not what we want (or what the [documentation says](https://github.com/huggingface/datasets/blob/6c6a82a573f946c4a81069f56446caed15cee9c2/src/datasets/iterable_dataset.py#L2370)) because we want the columns removed from the transformed example but then added if the map produced them.

This is `datasets==3.2.0` and `python==3.10`



### Steps to reproduce the bug

see above

### Expected behavior

see above

### Environment info

see above

Comments: Good catch ! Do you think you can open a PR to fix this issue ?
[Bug] Inconsistent behavior of data_files and data_dir in load_dataset method.
Author: JasonCZH4
State: closed

### Describe the bug

Inconsistent operation of data_files and data_dir in load_dataset method.


### Steps to reproduce the bug

# First
I have three files, named 'train.json', 'val.json', 'test.json'.
Each one has a simple dict `{text:'aaa'}`.
Their path are `/data/train.json`,  `/data/val.json`,  `/data/test.json`
I load dataset with `data_files` argument:
```py
files = [os.path.join('./data',file) for file in os.listdir('./data')]
ds = load_dataset(
            path='json',
            data_files=files,)
```
And I get:
```py
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 3
    })
})
```
However, If I load dataset with `data_dir` argument:
```py
ds = load_dataset(
            path='json',
            data_dir='./data',)
```
And I get:
```py
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 1
    })
    validation: Dataset({
        features: ['text'],
        num_rows: 1
    })
    test: Dataset({
        features: ['text'],
        num_rows: 1
    })
})
```
Two results are not the same. Their behaviors are not equal, even if the statement [here](https://github.com/huggingface/datasets/blob/d0c152a979d91cc34b605c0298aebc650ab7dd27/src/datasets/load.py#L1790) said that their behaviors are equal. 


# Second
If some filename include 'test' while others do not, `load_dataset` only return `test` dataset and others files are **abandoned**.
Given two files named `test.json` and `1.json`
Each one has a simple dict `{text:'aaa'}`.
I load the dataset using:
```py
ds = load_dataset(
            path='json',
            data_dir='./data',)
```
Only `test` is returned, `1.json` is missing:
```py
DatasetDict({
    test: Dataset({
        features: ['text'],
        num_rows: 1
    })
})
```
Things do not change even I manually set `split='train'`

 



### Expected behavior

1. Fix the above bugs.
2. Although the document says that load_dataset method will `Find which file goes into which split (e.g. train/test) based on file and directory names or on the YAML configuration`, I hope I can manually decide whether to do so. Sometimes users may accidentally put a `test` string in the filename but they just want a single `train` dataset. If the number of files in `data_dir` is huge, it's not easy to find out what cause the second situation metioned above.

### Environment info

datasets==3.2.0
Ubuntu18.84

Comments: Hi ! `data_files` with a list is equivalent to `data_files={"train": data_files}` with a train test only.

When no split are specified, they are inferred based on file names, and files with no apparent split are ignored

Thanks for your reply!
`files with no apparent split are ignored`. Is there a option that I can choose to ignored it or not as I mention aboved? Thanks!

To include all the files, the best way is to pass `data_files` yourself. There is no option to disable split detection at the moment

Thanks! I hope you guys can consider adding this option in the future. :)
One or several metadata.jsonl were found, but not in the same directory or in a parent directory of
Author: mst272
State: open

### Describe the bug

ImageFolder with metadata.jsonl error. I downloaded liuhaotian/LLaVA-CC3M-Pretrain-595K locally from Hugging Face. According to the tutorial in https://huggingface.co/docs/datasets/image_dataset#image-captioning, only put images.zip and metadata.jsonl containing information in the same folder. However, after loading, an error was reported: One or several metadata.jsonl were found, but not in the same directory or in a parent directory of.

The data in my jsonl file is as follows:

> {"id": "GCC_train_002448550", "file_name": "GCC_train_002448550.jpg", "conversations": [{"from": "human", "value": "<image>\nProvide a brief description of the given image."}, {"from": "gpt", "value": "a view of a city , where the flyover was proposed to reduce the increasing traffic on thursday ."}]}


### Steps to reproduce the bug

from datasets import load_dataset
image = load_dataset("imagefolder",data_dir='data/opensource_data')

### Expected behavior

success

### Environment info

datasets==3.2.0

Comments: Hmmm I double checked in the source code and I found a contradiction: in the current implementation the metadata file is ignored if it's not in the same archive as the zip image somehow:

https://github.com/huggingface/datasets/blob/caa705e8bf4bedf1a956f48b545283b2ca14170a/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L352-L353

in the tests suite the metadata file is placed inside the archive:

https://github.com/huggingface/datasets/blob/caa705e8bf4bedf1a956f48b545283b2ca14170a/tests/packaged_modules/test_imagefolder.py#L223-L223

Thanks for reporting this issue, it seems the documentation is wrong and we never implemented the support for zip + metadata outside zip. We might rewrite part of this code soon though to make it more flexible, it can be a good occasion to fix this. In the meantime feel free to open a PR to fix the documentation if you'd like
Clarify documentation or Create DatasetCard
Author: August-murr
State: open
Labels: name: enhancement

### Feature request

I noticed that you can use a Model Card instead of a Dataset Card when pushing a dataset to the Hub, but this isn’t clearly mentioned in [the docs.](https://huggingface.co/docs/datasets/dataset_card)

- Update the docs to clarify that a Model Card can work for datasets too.
- It might be worth creating a dedicated DatasetCard module, similar to the ModelCard module, for consistency and better support.

Not sure if this belongs here or on the [Hub repo](https://github.com/huggingface/huggingface_hub), but thought I’d bring it up!

### Motivation

I just spent an hour like on [this issue](https://github.com/huggingface/trl/pull/2491) trying to create a `DatasetCard` for a script.

### Your contribution

might later
Too many open files: '/root/.cache/huggingface/token'
Author: kopyl
State: open

### Describe the bug

I ran this code:
```
from datasets import load_dataset
dataset = load_dataset("common-canvas/commoncatalog-cc-by", cache_dir="/datadrive/datasets/cc", num_proc=1000)
```

And got this error.
Before it was some other file though (lie something...incomplete)

runnting
```
ulimit -n 8192
```
did not help at all.

### Steps to reproduce the bug

Run the code i sent

### Expected behavior

Should be no errors 

### Environment info

linux, jupyter lab.
TypeError: Value.__init__() missing 1 required positional argument: 'dtype'
Author: ghost
State: open

### Describe the bug

ds = load_dataset(
    "./xxx.py",
    name="default",
    split="train",
)

The datasets does not support debugging locally anymore...

### Steps to reproduce the bug

```
from datasets import load_dataset

ds = load_dataset(
    "./repo.py",
    name="default",
    split="train",
)

for item in ds:
    print(item)
```

It works fine for "username/repo", but it does not work for "./repo.py" when debugging locally...
Running above code template will report TypeError: Value.__init__() missing 1 required positional argument: 'dtype'

### Expected behavior

fix this bug

### Environment info

python 3.10 datasets==2.21

Comments: same error 
```
data = load_dataset('/opt/deepseek_R1_finetune/hf_datasets/openai/gsm8k', 'main')[split] 
```

> same error
> 
> ```
> data = load_dataset('/opt/deepseek_R1_finetune/hf_datasets/openai/gsm8k', 'main')[split] 
> ```

https://github.com/huggingface/open-r1/issues/204  this help me

Solved by delete `dataset_infos.json` file in dataset dir, or you can transfer datasets from Hugginface to Modelscope by [hf-ms-transfer](https://github.com/wa008/hf-ms-transfer), which will solve this problem by default.
.map() is not caching and ram goes OOM
Author: simeneide
State: open

### Describe the bug

Im trying to run a fairly simple map that is converting a dataset into numpy arrays. however, it just piles up on memory and doesnt write to disk. Ive tried multiple cache techniques such as specifying the cache dir, setting max mem, +++ but none seem to work. What am I missing here?

### Steps to reproduce the bug

```
from pydub import AudioSegment
import io
import base64
import numpy as np
import os
CACHE_PATH = "/mnt/extdisk/cache" # "/root/.cache/huggingface/"# 
os.environ["HF_HOME"] = CACHE_PATH
import datasets
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# Create a handler for Jupyter notebook
handler = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)

#datasets.config.IN_MEMORY_MAX_SIZE= 1000#*(2**30) #50 gb
print(datasets.config.HF_CACHE_HOME)
print(datasets.config.HF_DATASETS_CACHE)
# Decode the base64 string into bytes
def convert_mp3_to_audio_segment(example):
    """
    example = ds['train'][0]
    """
    try:
        audio_data_bytes = base64.b64decode(example['audio'])
        # Use pydub to load the MP3 audio from the decoded bytes
        audio_segment = AudioSegment.from_file(io.BytesIO(audio_data_bytes), format="mp3")
        # Resample to 24_000
        audio_segment = audio_segment.set_frame_rate(24_000)
        audio = {'sampling_rate' : audio_segment.frame_rate,
        'array' : np.array(audio_segment.get_array_of_samples(), dtype="float")}
        del audio_segment
        duration = len(audio['array']) / audio['sampling_rate']
    except Exception as e:
        logger.warning(f"Failed to convert audio for {example['id']}. Error: {e}")
        audio = {'sampling_rate' : 0,
        'array' : np.array([]), duration : 0}
    return {'audio' : audio, 'duration' : duration}

ds = datasets.load_dataset("NbAiLab/nb_distil_speech_noconcat_stortinget", cache_dir=CACHE_PATH, keep_in_memory=False)

#%%
num_proc=32
ds_processed = (
    ds
    #.select(range(10))
    .map(convert_mp3_to_audio_segment, num_proc=num_proc, desc="Converting mp3 to audio segment") #, cache_file_name=f"{CACHE_PATH}/stortinget_audio" # , cache_file_name="test"
)
```

### Expected behavior

the map should write to disk

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-6.8.0-45-generic-x86_64-with-glibc2.39
- Python version: 3.12.7
- `huggingface_hub` version: 0.26.3
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

Comments: I have the same issue - any update on this?
Remove upper bound for fsspec
Author: fellhorn
State: open

### Describe the bug

As also raised by @cyyever in https://github.com/huggingface/datasets/pull/7296 and @NeilGirdhar in https://github.com/huggingface/datasets/commit/d5468836fe94e8be1ae093397dd43d4a2503b926#commitcomment-140952162 , `datasets` has a problematic version constraint on `fsspec`.

In our case this causes (unnecessary?) troubles due to a race condition bug in that version of the corresponding `gcsfs` plugin, that causes deadlocks: https://github.com/fsspec/gcsfs/pull/643

We just use a version override to ignore the constraint from `datasets`, but imho the version constraint could just be removed in the first place?

The last few PRs bumping the upper bound were basically uneventful:

* https://github.com/huggingface/datasets/pull/7219
* https://github.com/huggingface/datasets/pull/6921
* https://github.com/huggingface/datasets/pull/6747

### Steps to reproduce the bug

-

### Expected behavior

Installing `fsspec>=2024.10.0` along `datasets` should be possible without overwriting constraints.

### Environment info

All recent datasets versions

Comments: Unfortunately `fsspec` versioning allows breaking changes across version and there is no way we can keep it without constrains at the moment. It already broke `datasets` once in the past. Maybe one day once `fsspec` decides on a stable and future proof API but I don't think this will happen anytime soon

edit: bumped to 2024.10.0 in https://github.com/huggingface/datasets/pull/7352
Unexpected cache behaviour using load_dataset
Author: Moritz-Wirth
State: closed

### Describe the bug

Following the (Cache management)[https://huggingface.co/docs/datasets/en/cache] docu and previous behaviour from datasets version 2.18.0, one is able to change the cache directory. Previously, all downloaded/extracted/etc files were found in this folder. As i have recently update to the latest version this is not the case anymore. Downloaded files are stored in `~/.cache/huggingface/hub`.
Providing the `cache_dir` argument in `load_dataset` the cache directory is created and there are some files but the bulk is still in `~/.cache/huggingface/hub`.

I believe this could be solved by adding the cache_dir argument [here](https://github.com/huggingface/datasets/blob/fdda5585ab18ea1292547f36c969d12c408ab842/src/datasets/utils/file_utils.py#L188)


### Steps to reproduce the bug

For example using https://huggingface.co/datasets/ashraq/esc50:
```python
from datasets import load_dataset
ds = load_dataset("ashraq/esc50", "default", cache_dir="~/custom/cache/path/esc50")
```


### Expected behavior

I would expect the bulk of files related to the dataset to be stored somewhere in `~/custom/cache/path/esc50`, but it seems they are in `~/.cache/huggingface/hub/datasets--ashraq--esc50`. 

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.14.0-503.15.1.el9_5.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.14
- `huggingface_hub` version: 0.26.5
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.6.1

Comments: Hi ! Since `datasets` 3.x,  the `datasets` specific files are in `cache_dir=` and the HF files are cached using `huggingface_hub` and you can set its cache directory using the `HF_HOME` environment variable.

They are independent, for example you can delete the Hub cache (containing downloaded files) but still reload your cached datasets from the `datasets` cache (containing prepared datasets in Arrow format)
ImportError: cannot import name 'set_caching_enabled' from 'datasets' 
Author: sankexin
State: open

### Describe the bug

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/local/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/home/Medusa/axolotl/src/axolotl/cli/__init__.py", line 23, in <module>
    from axolotl.train import TrainDatasetMeta
  File "/home/Medusa/axolotl/src/axolotl/train.py", line 23, in <module>
    from axolotl.utils.trainer import setup_trainer
  File "/home/Medusa/axolotl/src/axolotl/utils/trainer.py", line 13, in <module>
    from datasets import set_caching_enabled
ImportError: cannot import name 'set_caching_enabled' from 'datasets' (/usr/local/lib/python3.10/site-packages/datasets/__init__.py)

### Steps to reproduce the bug

1、axolotl
2、accelerate launch -m axolotl.cli.train examples/medusa/qwen_lora_stage1.yml

### Expected behavior

enable datasets

### Environment info

python3.10

Comments: pip install datasets==2.18.0

Hi ! I think you need to update axolotl
Introduce support for PDFs
Author: yabramuvdi
State: open
Labels: name: enhancement

### Feature request

The idea (discussed in the Discord server with @lhoestq ) is to have a Pdf type like Image/Audio/Video. For example [Video](https://github.com/huggingface/datasets/blob/main/src/datasets/features/video.py) was recently added and contains how to decode a video file encoded in a dictionary like {"path": ..., "bytes": ...} as a VideoReader using decord. We want to do the same with pdf and get a [pypdfium2.PdfDocument](https://pypdfium2.readthedocs.io/en/stable/_modules/pypdfium2/_helpers/document.html#PdfDocument).

### Motivation

In many cases PDFs contain very valuable information beyond text (e.g. images, figures). Support for PDFs would help create datasets where all the information is preserved.

### Your contribution

I can start the implementation of the Pdf type :)

Comments: #self-assign

Awesome ! Let me know if you have any question or if I can help :)

cc @AndreaFrancis as well for viz

Other candidates libraries for the Pdf type: PyMuPDF pypdf and pdfplumber

EDIT: Pymupdf looks like a good choice when it comes to maturity + performance + versatility BUT the license is maybe an issue, and pypdf, pypdfium2 or pdfplumber are good options imo

Related to https://github.com/huggingface/datasets/issues/7058

PyMuPDF is AGPL licensed, so we can't use it. I will move forward with [pdfplumber](https://github.com/jsvine/pdfplumber?tab=readme-ov-file#python-library).

Hi both! I have made a pull request with a first basic implementation of the Pdf feature. I followed closely what I saw on the Video and Image features. It is my first time contributing so any comments are very welcomed. I think it would be useful to outline together what additional things we can implement (e.g. enabling parsing of the pdf). Thanks :)
Cannot create a dataset with relative audio path
Author: sedol1339
State: open

### Describe the bug

Hello! I want to create a dataset of parquet files, with audios stored as separate .mp3 files. However, it says "No such file or directory" (see the reproducing code).

### Steps to reproduce the bug

Creating a dataset
```
from pathlib import Path
from datasets import Dataset, load_dataset, Audio

Path('my_dataset/audio').mkdir(parents=True, exist_ok=True)
Path('my_dataset/audio/file.mp3').touch(exist_ok=True)
Dataset.from_list(
    [{'audio': {'path': 'audio/file.mp3'}}]
).to_parquet('my_dataset/data.parquet')
```

Result:
```
# my_dataset
# ├── audio
# │   └── file.mp3
# └── data.parquet
```

Trying to load the dataset
```
dataset = (
    load_dataset('my_dataset', split='train')
    .cast_column('audio', Audio(sampling_rate=16_000))
)
dataset[0]

>>> FileNotFoundError: [Errno 2] No such file or directory: 'audio/file.mp3'
```

### Expected behavior

I expect the dataset to load correctly.

I've found 2 workarounds, but they are not very good:
1. I can specify an absolute path to the audio, however, when I move the folder or upload to HF it will stop working.
2. I can set `'path': 'file.mp3'`, and load with `load_dataset('my_dataset', data_dir='audio')` - it seems to work, but does this mean that anyone from Hugging Face who wants to use this dataset should also pass the `data_dir` argument, otherwise it won't work?

### Environment info

datasets 3.1.0, Ubuntu 24.04.1

Comments: Hello ! when you `cast_column` you need the paths to be absolute paths or relative paths to your working directory, not the original dataset directory.

Though I'd recommend structuring your dataset as an AudioFolder which automatically links a metadata.jsonl or csv to the audio files via relative paths **within** the dataset repository: https://huggingface.co/docs/datasets/v3.2.0/en/audio_load#audiofolder

@lhoestq thank you, but there are two problems with using AudioFolder:
1. It is said that AudioFolder requires metadata.csv. However, my datset is too large and contains nested and np.ndarray fields, so I can't use csv.
2. It is said that I need to load the dataset with `load_dataset("audiofolder", ...)`. However, if possible, I want my dataset to be loaded as usual with `load_dataset(dataset_name)` after I upload if to HF.

You can use metadata.jsonl if you have nested data :)

And actually if you have a dataset structured as an AudioFolder then `load_dataset(dataset_name)` does work after uploading to HF

I have created an audio dataset. In my repo, I have explained the steps and structure. An example dataset is also available in the repo. https://github.com/pr0mila/ParquetToHuggingFace
How to get the original dataset name with username?
Author: npuichigo
State: open
Labels: name: enhancement

### Feature request

The issue is related to ray data https://github.com/ray-project/ray/issues/49008 which it requires to check if the dataset is the original one just after `load_dataset` and parquet files are already available on hf hub.

The solution used now is to get the dataset name, config and split, then `load_dataset` again and check the fingerprint. But it's unable to get the correct dataset name if it contains username. So how to get the dataset name with username prefix, or is there another way to query if a dataset is the original one with parquet available?

@lhoestq 

### Motivation

https://github.com/ray-project/ray/issues/49008

### Your contribution

Would like to fix that.

Comments: Hi ! why not pass the dataset id to Ray and let it check the parquet files ? Or pass the parquet files lists directly ?

I'm not sure why ray design an API like this to accept a `Dataset` object, so they need to verify the `Dataset` is the original one and use the `DatasetInfo` to query the huggingface hub. I'll advise the ray data team to use dataset id instead of dataset for the `HuggingFaceDatasource` API.
Enable the Audio Feature to decode / read with an offset + duration
Author: TParcollet
State: open
Labels: name: enhancement

### Feature request

For most large speech dataset, we do not wish to generate hundreds of millions of small audio samples. Instead, it is quite common to provide larger audio files with frame offset (soundfile start and stop arguments). We should be able to pass these arguments to Audio() (column ID corresponding in the dataset row).

### Motivation

I am currently generating a fairly big dataset to .parquet(). Unfortunately, it does not work because all existing functions load the whole .wav file corresponding to the row. All my attempts at bypassing this failed. We should be able to put in the Table only the bytes corresponding to what soundfile reads with an offset (and subset of the audio file). 

### Your contribution

I can totally test whatever code on my large dataset creation script. 

Comments: Hi ! What about having audio + start + duration columns and enable something like this ?

```python
for example in ds:
    array = example["audio"].read(start=example["start"], frames=example["duration"])
```

Hi @lhoestq, this would work with a file-based dataset but would be terrible for a sharded one as it would duplicate the large audio file many times. Also, very long audio files are not embedded very well in the parquet file, even with large_binary(). It crashed a few times for me until I switched to one sample == one file :-(