josteece/github-issues huggingface dataset:
Support downloading specific splits in `load_dataset`
Opened by CloseChoice (open)

This is PR builds on top of #7706 to revive the unfinished #6832 but isn't just cleaning up things, here are some important changes:
 - `download_mode="FORCE_REDOWNLOAD"` is interpreted as always creating a clean slate, that means that even if we already did:
 ```python
load_dataset("<name>")
load_dataset("<name>", split="train", download_mode="force_redownload")
```
This makes sure that only the train dataset is available after executing both. This was different in the original PR, which proposed that train and test would be available. 
- `download_mode="REUSE_DATASET_IF_EXISTS"` is interpreted as only ever adding new data, never redownloading OR deleting other splits. This was different in the original PR, where  
 ```python
load_dataset("<name>", split="test")
load_dataset("<name>", split="train")
```
resulted in only the train data being available, which I deem very unintuitive and probably not what users want. Also I argue that this is just the first step to a more user friendly partial loading when specifying percentages (or maybe even single instances) via the ReadInstructions, and then doing
 ```python
load_dataset("<name>", split="test[:10%]")
load_dataset("<name>", split="test[10%:]")
```
should result IMO in the whole dataset being cached locally without redownloads.

Furthermore this PR fixes a couple issues with the previous PR, e.g. a [missing comparison](https://github.com/huggingface/datasets/pull/7706/files#diff-f933ce41f71c6c0d1ce658e27de62cbe0b45d777e9e68056dd012ac3eb9324f7R877) and adding tests for the proposed changes in behaviour, which would both fail on @ArjunJagdale's original PR. 

Todo:
 - [ ] update docs?

Future outlook (just my opinions and up for debate):
As mentioned before, I would see this as just a step towards the feature of partial percentage loading (though how the API should behave in that case is not entirely clear for me now) and maybe we could introduce another `download_mode="FORCE_REDOWNLOAD_SPLIT"`, which makes sure that even if a split is specified, only the referenced split is redownloaded and everything else left unchanged, this would then allow users more granular control over what they want to redownload. 
@lhoestq very curious to get your opinion on this.

Comments:
@CloseChoice This looks great! You're absolutely right about the missing comparison - that's a critical bug I missed.
Missing transcript column when loading a local dataset with "audiofolder"
Opened by gweltou (closed)

### Describe the bug

My local dataset is not properly loaded when using `load_dataset("audiofolder", data_dir="my_dataset")` with a `jsonl` metadata file.
Only the `audio` column is read while the `transcript` column is not.

The last tested `datasets` version where the behavior was still correct is 2.18.0.

### Steps to reproduce the bug

Dataset directory structure:
```
my_dataset/
  - data/
    - test/
      - 54db8760de3cfbff3c8a36a36b4d0f77_00390.0_04583.0.mp3
      - 54db8760de3cfbff3c8a36a36b4d0f77_04583.0_05730.0.mp3
      - ...
  - metadata.jsonl
```

`metadata.jsonl` file content:
```
{"file_name": "data/test/54db8760de3cfbff3c8a36a36b4d0f77_00390.0_04583.0.mp3", "transcript": "Ata tudoù penaos e tro ar bed ?"}
{"file_name": "data/test/54db8760de3cfbff3c8a36a36b4d0f77_04583.0_05730.0.mp3", "transcript": "Ur gwir blijadur eo adkavout ac'hanoc'h hiziv."}
...
```
```python3
my_dataset = load_dataset("audiofolder", data_dir="my_dataset")

print(my_dataset)

'''
DatasetDict({
    test: Dataset({
        features: ['audio'],
        num_rows: 347
    })
})
'''

print(my_dataset['test'][0])

'''
{'audio': <datasets.features._torchcodec.AudioDecoder object at 0x75ffcd172510>}
'''
```

### Expected behavior

Being able to access the `transcript` column in the loaded dataset.

### Environment info

- `datasets` version: 4.4.1
- Platform: Linux-6.5.0-45-generic-x86_64-with-glibc2.39
- Python version: 3.13.9
- `huggingface_hub` version: 1.1.2
- PyArrow version: 22.0.0
- Pandas version: 2.3.3
- `fsspec` version: 2025.10.0

Note: same issue with `datasets` v3.6.0

Comments:
First bad commit 5c8869f8c36dbc8c8d423030b7b7c4fd64f8c729

EDIT: This is not a bug or a regression. It was a breaking change introduced in the commit I mentioned and was also documented in there. The docs state how to handle this now, see https://huggingface.co/docs/datasets/main/en/audio_load#audiofolder-with-metadata

or simply, move your metadata into the splits folder and update the paths, in your case this would look like this:
```bash
my_dataset/
  - data/
    - test/
      - 54db8760de3cfbff3c8a36a36b4d0f77_00390.0_04583.0.mp3
      - 54db8760de3cfbff3c8a36a36b4d0f77_04583.0_05730.0.mp3
      - metadata.jsonl
```

and the pahts in the jsonl should be relative to the metadata.json:
```bash
{"file_name": "54db8760de3cfbff3c8a36a36b4d0f77_00390.0_04583.0.mp3", "transcript": "Ata tudoù penaos e tro ar bed ?"}
{"file_name": "54db8760de3cfbff3c8a36a36b4d0f77_04583.0_05730.0.mp3", "transcript": "Ur gwir blijadur eo adkavout ac'hanoc'h hiziv."}
...
```

So I think this can be closed.

Thank you for your quick answer !
I'm sorry I missed that in the documentation.
Everything works fine again after following your recommendations.
I'm closing the issue.
[Distributed] split_dataset_by_node() gives the same number of examples for each node
Opened by lhoestq (open)

this works:

```python
import torch.distributed as dist
from datasets import IterableDataset
from datasets.distributed import split_dataset_by_node
from collections import Counter

def g(shards):
    for shard in shards:
        # shards don't have the same length
        num_examples = 3 + shard 
        for i in range(num_examples):
            yield {"shard": f"{shard=}", "i": i}

if __name__ == "__main__":
    dist.init_process_group(backend="gloo")
    rank, world_size = dist.get_rank(), dist.get_world_size()
    num_shards = 6
    ds = IterableDataset.from_generator(g, gen_kwargs={"shards": list(range(num_shards))})
    ds = split_dataset_by_node(ds, rank=rank, world_size=world_size)
    # Check that each rank has the same number of examples
    # and show the number of examples per shard and per rank
    counter = Counter(ds["shard"])
    print(f"# {rank=}\ttotal={counter.total()}\t{counter}", flush=True)

    # torchrun --nproc_per_node 2 script.py
    # rank=0        total=16        Counter({'shard=4': 7, 'shard=2': 5, 'shard=0': 4})
    # rank=1        total=16        Counter({'shard=3': 6, 'shard=5': 6, 'shard=1': 4})

```

TODO: make it work with DataLoader (communicate with main process to know when the node runs out of data ?)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7854). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Making this work with multiple workers could create a lot of communication for not a lot of benefits, considering you can simply use `Join()` to let nodes shutdown when they run out of data while the other nodes continue training: https://docs.pytorch.org/docs/stable/distributed.algorithms.join.html
Fix embed storage nifti
Opened by CloseChoice (closed)

Fixes #7852 

Adds `embed_storage` function and allows gzipped files to be loaded correctly from local storage.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7853). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Problems with NifTI
Opened by CloseChoice (closed)

### Describe the bug

There are currently 2 problems with the new NifTI feature:
1. dealing with zipped files, this is mentioned and explained [here](https://github.com/huggingface/datasets/pull/7815#issuecomment-3496199503)
2. when uploading via the `niftifolder` feature, the resulting parquet only contains relative paths to the nifti files:

```bash
table['nifti']
<pyarrow.lib.ChunkedArray object at 0x798245d37d60>
[
  -- is_valid: all not null
  -- child 0 type: binary
    [
      null,
      null,
      null,
      null,
      null,
      null
    ]
  -- child 1 type: string
    [
      "/home/tobias/programming/github/datasets/nifti_extracted/T1.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2-interleaved.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2_-interleaved.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2_.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/fieldmap.nii"
    ]
]
```
instead of containing bytes. The code is copy pasted from PDF, so I wonder what is going wrong here.

### Steps to reproduce the bug

see the linked comment

### Expected behavior

downloading should work as smoothly as for pdf

### Environment info

- `datasets` version: 4.4.2.dev0
- Platform: Linux-6.14.0-33-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.35.3
- PyArrow version: 21.0.0
- Pandas version: 2.3.3
- `fsspec` version: 2025.9.0


Comments:
> 2. when uploading via the niftifolder feature, the resulting parquet only contains relative paths to the nifti files:

what did you use to upload the dataset ? iirc push_to_hub() does upload the bytes as well, but to_parquet() doesn't

> > 2. when uploading via the niftifolder feature, the resulting parquet only contains relative paths to the nifti files:
> 
> what did you use to upload the dataset ? iirc push_to_hub() does upload the bytes as well, but to_parquet() doesn't

I used `push_to_hub` but the problem is that the nifti feature does not have an `embed_storage` function
Transform with columns parameter triggers on non-specified column access
Opened by mr-brobot (closed)

### Describe the bug

Iterating over a [`Column`](https://github.com/huggingface/datasets/blob/8b1bd4ec1cc9e9ce022f749abb6485ef984ae7c0/src/datasets/arrow_dataset.py#L633-L692) iterates through the parent [`Dataset`](https://github.com/huggingface/datasets/blob/8b1bd4ec1cc9e9ce022f749abb6485ef984ae7c0/src/datasets/arrow_dataset.py#L695) and applies all formatting/transforms on each row, regardless of which column is being accessed. This causes an error when transforms depend on columns not present in the projection.

### Steps to reproduce the bug

### Load a dataset with multiple columns

```python
ds = load_dataset("mrbrobot/isic-2024", split="train")
```

### Define a transform that specifies an input column

```python
def image_transform(batch):
    batch["image"] = batch["image"]  # KeyError when batch doesn't contain "image"
    return batch

# apply transform only to image column
ds = ds.with_format("torch")
ds = ds.with_transform(image_transform, columns=["image"], output_all_columns=True)
```

### Iterate over non-specified column

```python
# iterate over a different column, triggers the transform on each row, but batch doesn't contain "image"
for t in ds["target"]:  # KeyError: 'image'
    print(t)
```

### Expected behavior

If a user iterates over `ds["target"]` and the transform specifies `columns=["image"]`, the transform should be skipped.

### Environment info

`datasets`: 4.2.0
Python: 3.12.12
Linux: Debian 11.11

Comments:
DOC: `mode` parameter on pdf and video features unused
Opened by CloseChoice (closed)

Following up on https://github.com/huggingface/datasets/pull/7840 I asked claude code to check for undocumented parameters for other features and it found:
- mode parameter on video is documented but unused: https://github.com/huggingface/datasets/blob/main/src/datasets/features/video.py#L48-L49
- the same goes for the mode parameter on the pdf feature: https://github.com/huggingface/datasets/blob/main/src/datasets/features/pdf.py#L47-L48

I assume checking if these modes can be supported and otherwise removing them is the way to go here.


Comments:
They seem to be artefacts from a copy-paste of the Image feature ^^' we should remove them
Add num channels to audio
Opened by CloseChoice (closed)

Fixes #7837

We currently have the [mono attribute for Audio documented](https://github.com/huggingface/datasets/blob/41c05299348a499807432ab476e1cdc4143c8772/src/datasets/features/audio.py#L52C1-L54C22) but not used anywhere resulting in confusion for users. Since torchcodec does not know this attribute I suggest using `num_channels` (currently supported `None` (leave unchanged), mono: `1`, stereo: `2`).

I could also add a mono attribute but found that to be more confusing for developers and would restrict us if at any point in the future more than 2 channels are supported.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7840). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
mono parameter to the Audio feature is missing
Opened by ernestum (closed)

According to the docs, there is a "mono" parameter to the Audio feature, which turns any stereo into mono. In practice the signal is not touched and the mono parameter, even though documented, does not exist.


https://github.com/huggingface/datasets/blob/41c05299348a499807432ab476e1cdc4143c8772/src/datasets/features/audio.py#L52C1-L54C22

Comments:
Hey,  we removed the misleading passage in the docstring and enabled support for `num_channels` as torchcodec does

thanks!
Add DICOM support
Opened by CloseChoice (open)

supports #7804
Add support for the dicom file format. 

This PR follows PR #7815 and PR #7325 closely.
Remarkable differences:
I made sure that we can load all of pydicom's test data, and encountered the `force=True` parameter that we explicitly support here. This allows to trying to load corrupted dicom files, we explicitly test this!

There is one dataset with all of dicom's test data on huggingface which can be loaded using this branch with the following script:
```python
from datasets import load_dataset
from datasets import Features, ClassLabel
from datasets.features import Dicom

features = Features({
    "dicom": Dicom(force=True),  # necessary to be able to load one corrupted file
    "label": ClassLabel(num_classes=2)
})

ds = load_dataset("TobiasPitters/dicom-sample-dataset",
                  features=features)

error_count = 0

for idx, item in enumerate(ds["test"]):
    dicom = item["dicom"]

    try:
        print(f"Type: {type(dicom)}")
        if hasattr(dicom, 'PatientID'):
            print(f"PatientID: {dicom.PatientID}")
        if hasattr(dicom, 'StudyInstanceUID'):
            print(f"StudyInstanceUID: {dicom.StudyInstanceUID}")
        if hasattr(dicom, 'Modality'):
            print(f"Modality: {dicom.Modality}")
    except Exception as e:
        error_count += 1
        print(e)

print(f"Finished processing with {error_count} errors.")
```

todo: 
- [x] add docs (will do so soon)

Comments:
Awesome ! For the docs should we rename https://huggingface.co/docs/datasets/nifti_dataset to medical_imaging_dataset and have both DICOM and NIfTI together or have separate pages in you opinion ?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7835). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Awesome ! For the docs should we rename https://huggingface.co/docs/datasets/nifti_dataset to medical_imaging_dataset and have both DICOM and NIfTI together or have separate pages in you opinion ?

Makes sense, is more intuitive for the user and the pages as proposed in this branch have a lot of overlap. I would then structure it in such a way to write some brief things about medical imaging, then introduce the formats (so basically concatenating the two pages together and removing duplicates).

Pls don't merge currently, since we'll need an `embed_storage` function in here as well. See
https://github.com/huggingface/datasets/pull/7815#issuecomment-3494094692 and the following conversation
Audio.cast_column() or Audio.decode_example() causes Colab kernel crash (std::bad_alloc)
Opened by rachidio (open)

### Describe the bug

When using the huggingface datasets.Audio feature to decode a local or remote (public HF dataset) audio file inside Google Colab, the notebook kernel crashes with std::bad_alloc (C++ memory allocation failure).
The crash happens even with a minimal code example and valid .wav file that can be read successfully using soundfile.

Here is a sample Collab notebook to reproduce the problem.
https://colab.research.google.com/drive/1nnb-GC5748Tux3xcYRussCGp2x-zM9Id?usp=sharing

code sample:
```
...
audio_dataset = audio_dataset.cast_column("audio", Audio(sampling_rate=16000))

# Accessing the first element crashes the Colab kernel
print(audio_dataset[0]["audio"])
```

Error log 
```
WARNING what(): std::bad_alloc
terminate called after throwing an instance of 'std::bad_alloc'
```

Environment

Platform: Google Colab (Python 3.12.12)
datasets Version: 4.3.0
soundfile Version: 0.13.1
torchaudio Version: 2.8.0+cu126

Thanks in advance to help me on this error I get approx two weeks now after it was working before.

Regards

### Steps to reproduce the bug

https://colab.research.google.com/drive/1nnb-GC5748Tux3xcYRussCGp2x-zM9Id?usp=sharing

### Expected behavior

Loading the audio and decode it.
It should safely return:

{
  "path": "path/filaname.wav",
  "array": np.ndarray([...]),
  "sampling_rate": 16000
}

### Environment info

Environment

Platform: Google Colab (Python 3.12.12)
datasets Version: 4.3.0
soundfile Version: 0.13.1
torchaudio Version: 2.8.0+cu126

Comments:
Hi ! `datasets` v4 uses `torchcodec` for audio decoding (previous versions were using `soundfile`). What is your `torchcodec` version ? Can you try other versions of `torchcodec` and see if it works ?

When I install `datasets` with `pip install datasets[audio]` it install this version of `torchcodec`:
```
Name: torchcodec
Version: 0.8.1
```
Can you please point to a working version of `torchcodec`?

Thanks for your help

I believe you simply need to make sure the torchcodec and torch versions work together. Here is how to fix it:

```python
!pip install -U torchcodec torch
```

I am also encountering this same issue when i run `print(ug_court["train"][0])` to view the features of the first row of my audio data

the problem still goes on to when i force training with seeing these features

Thank you @lhoestq I've reinstalled the packages an the error is gone.
My new versions are:
```
Name: torch
Version: 2.8.0
---
Name: torchaudio
Version: 2.8.0
---
Name: torchcodec
Version: 0.8.1
```

Regards

mine too has worked
Fix: Properly render [!TIP] block in stream.shuffle documentation
Opened by art-test-stack (closed)

The second line starting with the bracket doesn't properly render on huggingface/docs. Added "> " to address it.

In the client documentation, the markdown 'TIP' paragraph in docs/stream#shuffle is not well executed, not as the other in the same page / while markdown is correctly considering it.

Documentation:
https://huggingface.co/docs/datasets/v4.3.0/en/stream#shuffle:~:text=%5B!TIP%5D%5BIterableDataset.shuffle()%5D(/docs/datasets/v4.3.0/en/package_reference/main_classes%23datasets.IterableDataset.shuffle)%20will%20also%20shuffle%20the%20order%20of%20the%20shards%20if%20the%20dataset%20is%20sharded%20into%20multiple%20files.

Github source:
https://github.com/huggingface/datasets/blob/main/docs/source/stream.mdx#:~:text=Casting%20only%20works%20if%20the%20original%20feature%20type%20and%20new%20feature%20type%20are%20compatible.%20For%20example%2C%20you%20can%20cast%20a%20column%20with%20the%20feature%20type%20Value(%27int32%27)%20to%20Value(%27bool%27)%20if%20the%20original%20column%20only%20contains%20ones%20and%20zeros.

Comments:
[DOCS][minor] TIPS paragraph not compiled in docs/stream
Opened by art-test-stack (closed)

In the client documentation, the markdown 'TIP' paragraph for paragraph in docs/stream#shuffle is not well executed — not as the other in the same page / while markdown is correctly considering it.

Documentation:
https://huggingface.co/docs/datasets/v4.3.0/en/stream#shuffle:~:text=%5B!TIP%5D%5BIterableDataset.shuffle()%5D(/docs/datasets/v4.3.0/en/package_reference/main_classes%23datasets.IterableDataset.shuffle)%20will%20also%20shuffle%20the%20order%20of%20the%20shards%20if%20the%20dataset%20is%20sharded%20into%20multiple%20files.

Github source:
https://github.com/huggingface/datasets/blob/main/docs/source/stream.mdx#:~:text=Casting%20only%20works%20if%20the%20original%20feature%20type%20and%20new%20feature%20type%20are%20compatible.%20For%20example%2C%20you%20can%20cast%20a%20column%20with%20the%20feature%20type%20Value(%27int32%27)%20to%20Value(%27bool%27)%20if%20the%20original%20column%20only%20contains%20ones%20and%20zeros.

Comments:
Memory leak / Large memory usage with num_workers = 0 and numerous dataset within DatasetDict
Opened by raphaelsty (open)

### Describe the bug

Hi team, first off, I love the datasets library! 🥰

I'm encountering a potential memory leak / increasing memory usage when training a model on a very large DatasetDict.

Setup: I have a DatasetDict containing 362 distinct datasets, which sum up to ~2.8 billion rows.

Training Task: I'm performing contrastive learning with SentenceTransformer and Accelerate on a single node with 4 H100, which requires me to sample from only one dataset at a time.

Training Loop: At each training step, I sample ~16,000 examples from a single dataset, and then switch to a different dataset for the next step. I iterate through all 362 datasets this way. 

Problem: The process's memory usage continuously increases over time, eventually causing a stale status where GPUs would stop working. It seems memory from previously sampled datasets isn't being released. I've set num_workers=0 for all experiments.

Chart 1: Standard DatasetDict The memory usage grows steadily until it make the training stale (RSS memory) <img width="773" height="719" alt="Image" src="https://github.com/user-attachments/assets/6606bef5-1153-4f2d-bf08-82da249d6e8d" />

Chart 2: IterableDatasetDict I also tried  to use IterableDatasetDict and IterableDataset. The memory curve is "smoother," but the result is the same: it grows indefinitely and the training become stale. <img width="339" height="705" alt="Image" src="https://github.com/user-attachments/assets/ee90c1a1-6c3b-4135-9edc-90955cb1695a" />

Any feedback or guidance on how to manage this memory would be greatly appreciated!

### Steps to reproduce the bug

WIP, I'll add some code that manage to reproduce this error, but not straightforward. 

### Expected behavior

The memory usage should remain relatively constant or plateau after a few steps. Memory used for sampling one dataset should be released before or during the sampling of the next dataset.

### Environment info

Python: 3.12
Datasets: 4.3.0
SentenceTransformers: 5.1.1

Comments:
Thanks for the report, this is possibly related #7722 and #7694.

Could you pls provide steps to reproduce this?

To overcome this issue right now I did simply reduce the size of the dataset and ended up running a for loop (my training has now a constant learning rate schedule). From what I understood, and I don't know if it's possible, the solution would be to tell the backend of `datasets` to leave x% of the memory free  (including memory mapping). Can't release the data right now but I will and then allow to reproduce this issue. But it will involve to have some free TB of disk

@raphaelsty thanks for coming back to this. I assume you are running in streaming mode? That should prevent these errors but it looks like more people than just you have this problem, so a clearly reproducing example (including data + code) is highly appreciated.

This could be related to this issue: https://github.com/huggingface/datasets/issues/4883 in which we discussed how RSS and memory mapping works and depends on the OS and disk type.