AniketTec/github-issues huggingface dataset:
Title: Support downloading specific splits in `load_dataset`
State: open

This is PR builds on top of #7706 to revive the unfinished #6832 but isn't just cleaning up things, here are some important changes:
 - `download_mode="FORCE_REDOWNLOAD"` is interpreted as always creating a clean slate, that means that even if we already did:
 ```python
load_dataset("<name>")
load_dataset("<name>", split="train", download_mode="force_redownload")
```
This makes sure that only the train dataset is available after executing both. This was different in the original PR, which proposed that train and test would be available. 
- `download_mode="REUSE_DATASET_IF_EXISTS"` is interpreted as only ever adding new data, never redownloading OR deleting other splits. This was different in the original PR, where  
 ```python
load_dataset("<name>", split="test")
load_dataset("<name>", split="train")
```
resulted in only the train data being available, which I deem very unintuitive and probably not what users want. Also I argue that this is just the first step to a more user friendly partial loading when specifying percentages (or maybe even single instances) via the ReadInstructions, and then doing
 ```python
load_dataset("<name>", split="test[:10%]")
load_dataset("<name>", split="test[10%:]")
```
should result IMO in the whole dataset being cached locally without redownloads.

Furthermore this PR fixes a couple issues with the previous PR, e.g. a [missing comparison](https://github.com/huggingface/datasets/pull/7706/files#diff-f933ce41f71c6c0d1ce658e27de62cbe0b45d777e9e68056dd012ac3eb9324f7R877) and adding tests for the proposed changes in behaviour, which would both fail on @ArjunJagdale's original PR. 

Todo:
 - [ ] update docs?

Future outlook (just my opinions and up for debate):
As mentioned before, I would see this as just a step towards the feature of partial percentage loading (though how the API should behave in that case is not entirely clear for me now) and maybe we could introduce another `download_mode="FORCE_REDOWNLOAD_SPLIT"`, which makes sure that even if a split is specified, only the referenced split is redownloaded and everything else left unchanged, this would then allow users more granular control over what they want to redownload. 
@lhoestq very curious to get your opinion on this.

Comments:
@CloseChoice This looks great! You're absolutely right about the missing comparison - that's a critical bug I missed. 

@lhoestq just resolved the conflicts inflicted by the latest changes. Thought it might be good giving this a shot now before more changes mess with this PR.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7858). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Missing transcript column when loading a local dataset with "audiofolder"
State: closed

### Describe the bug

My local dataset is not properly loaded when using `load_dataset("audiofolder", data_dir="my_dataset")` with a `jsonl` metadata file.
Only the `audio` column is read while the `transcript` column is not.

The last tested `datasets` version where the behavior was still correct is 2.18.0.

### Steps to reproduce the bug

Dataset directory structure:
```
my_dataset/
  - data/
    - test/
      - 54db8760de3cfbff3c8a36a36b4d0f77_00390.0_04583.0.mp3
      - 54db8760de3cfbff3c8a36a36b4d0f77_04583.0_05730.0.mp3
      - ...
  - metadata.jsonl
```

`metadata.jsonl` file content:
```
{"file_name": "data/test/54db8760de3cfbff3c8a36a36b4d0f77_00390.0_04583.0.mp3", "transcript": "Ata tudoù penaos e tro ar bed ?"}
{"file_name": "data/test/54db8760de3cfbff3c8a36a36b4d0f77_04583.0_05730.0.mp3", "transcript": "Ur gwir blijadur eo adkavout ac'hanoc'h hiziv."}
...
```
```python3
my_dataset = load_dataset("audiofolder", data_dir="my_dataset")

print(my_dataset)

'''
DatasetDict({
    test: Dataset({
        features: ['audio'],
        num_rows: 347
    })
})
'''

print(my_dataset['test'][0])

'''
{'audio': <datasets.features._torchcodec.AudioDecoder object at 0x75ffcd172510>}
'''
```

### Expected behavior

Being able to access the `transcript` column in the loaded dataset.

### Environment info

- `datasets` version: 4.4.1
- Platform: Linux-6.5.0-45-generic-x86_64-with-glibc2.39
- Python version: 3.13.9
- `huggingface_hub` version: 1.1.2
- PyArrow version: 22.0.0
- Pandas version: 2.3.3
- `fsspec` version: 2025.10.0

Note: same issue with `datasets` v3.6.0

Comments:
First bad commit 5c8869f8c36dbc8c8d423030b7b7c4fd64f8c729

EDIT: This is not a bug or a regression. It was a breaking change introduced in the commit I mentioned and was also documented in there. The docs state how to handle this now, see https://huggingface.co/docs/datasets/main/en/audio_load#audiofolder-with-metadata

or simply, move your metadata into the splits folder and update the paths, in your case this would look like this:
```bash
my_dataset/
  - data/
    - test/
      - 54db8760de3cfbff3c8a36a36b4d0f77_00390.0_04583.0.mp3
      - 54db8760de3cfbff3c8a36a36b4d0f77_04583.0_05730.0.mp3
      - metadata.jsonl
```

and the pahts in the jsonl should be relative to the metadata.json:
```bash
{"file_name": "54db8760de3cfbff3c8a36a36b4d0f77_00390.0_04583.0.mp3", "transcript": "Ata tudoù penaos e tro ar bed ?"}
{"file_name": "54db8760de3cfbff3c8a36a36b4d0f77_04583.0_05730.0.mp3", "transcript": "Ur gwir blijadur eo adkavout ac'hanoc'h hiziv."}
...
```

So I think this can be closed.

Thank you for your quick answer !
I'm sorry I missed that in the documentation.
Everything works fine again after following your recommendations.
I'm closing the issue.
Title: [Distributed] split_dataset_by_node() gives the same number of examples for each node
State: open

this works:

```python
import torch.distributed as dist
from datasets import IterableDataset
from datasets.distributed import split_dataset_by_node
from collections import Counter

def g(shards):
    for shard in shards:
        # shards don't have the same length
        num_examples = 3 + shard 
        for i in range(num_examples):
            yield {"shard": f"{shard=}", "i": i}

if __name__ == "__main__":
    dist.init_process_group(backend="gloo")
    rank, world_size = dist.get_rank(), dist.get_world_size()
    num_shards = 6
    ds = IterableDataset.from_generator(g, gen_kwargs={"shards": list(range(num_shards))})
    ds = split_dataset_by_node(ds, rank=rank, world_size=world_size)
    # Check that each rank has the same number of examples
    # and show the number of examples per shard and per rank
    counter = Counter(ds["shard"])
    print(f"# {rank=}\ttotal={counter.total()}\t{counter}", flush=True)

    # torchrun --nproc_per_node 2 script.py
    # rank=0        total=16        Counter({'shard=4': 7, 'shard=2': 5, 'shard=0': 4})
    # rank=1        total=16        Counter({'shard=3': 6, 'shard=5': 6, 'shard=1': 4})

```

TODO: make it work with DataLoader (communicate with main process to know when the node runs out of data ?)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7854). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Making this work with multiple workers could create a lot of communication for not a lot of benefits, considering you can simply use `Join()` to let nodes shutdown when they run out of data while the other nodes continue training: https://docs.pytorch.org/docs/stable/distributed.algorithms.join.html
Title: Fix embed storage nifti
State: closed

Fixes #7852 

Adds `embed_storage` function and allows gzipped files to be loaded correctly from local storage.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7853). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Problems with NifTI
State: closed

### Describe the bug

There are currently 2 problems with the new NifTI feature:
1. dealing with zipped files, this is mentioned and explained [here](https://github.com/huggingface/datasets/pull/7815#issuecomment-3496199503)
2. when uploading via the `niftifolder` feature, the resulting parquet only contains relative paths to the nifti files:

```bash
table['nifti']
<pyarrow.lib.ChunkedArray object at 0x798245d37d60>
[
  -- is_valid: all not null
  -- child 0 type: binary
    [
      null,
      null,
      null,
      null,
      null,
      null
    ]
  -- child 1 type: string
    [
      "/home/tobias/programming/github/datasets/nifti_extracted/T1.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2-interleaved.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2_-interleaved.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2_.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/fieldmap.nii"
    ]
]
```
instead of containing bytes. The code is copy pasted from PDF, so I wonder what is going wrong here.

### Steps to reproduce the bug

see the linked comment

### Expected behavior

downloading should work as smoothly as for pdf

### Environment info

- `datasets` version: 4.4.2.dev0
- Platform: Linux-6.14.0-33-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.35.3
- PyArrow version: 21.0.0
- Pandas version: 2.3.3
- `fsspec` version: 2025.9.0


Comments:
> 2. when uploading via the niftifolder feature, the resulting parquet only contains relative paths to the nifti files:

what did you use to upload the dataset ? iirc push_to_hub() does upload the bytes as well, but to_parquet() doesn't

> > 2. when uploading via the niftifolder feature, the resulting parquet only contains relative paths to the nifti files:
> 
> what did you use to upload the dataset ? iirc push_to_hub() does upload the bytes as well, but to_parquet() doesn't

I used `push_to_hub` but the problem is that the nifti feature does not have an `embed_storage` function
Title: Add fasta support
State: open

This PR adds support for FASTA files conversion to Parquet. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7851). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

A few comments:

- Have you tried using this with longer sequences? @UriNeri developed something similar internally and used it with viral genomes. He got some Parquet errors due to genomes not fitting in a `utf8` column. This was fixed by using `large_utf8`.
- If you're only using it to read FASTA files, I think that having BioPython as a dependency is overkill. The library is very large and the FASTA parser isn't particularly fast. I have an example of a fast parser with no external references [here](https://gist.github.com/apcamargo/d039aa04a2cbbcbb14e2d34a0963b862) (this is actually based on [`readfq.py`](https://github.com/lh3/readfq/blob/master/readfq.py), with a couple of extra functions that might not be useful in the context of this PR)

> * If you're only using it to read FASTA files, I think that having BioPython as a dependency is overkill. The library is very large and the FASTA parser isn't particularly fast. I have an example of a fast parser with no external references [here](https://gist.github.com/apcamargo/d039aa04a2cbbcbb14e2d34a0963b862) (this is actually based on [`readfq.py`](https://github.com/lh3/readfq/blob/master/readfq.py), with a couple of extra functions that might not be useful in the context of this PR)

What @apcamargo said, plus FWIW in **our approach** (so might not be relevant here) we use polars (with custom fasta io parser) or polars-bio (that has a `scan_fasta` function) and we foudn out that the page size sometimes need to be adjusted:
```
envs/default/lib/python3.9/site-packages/polars/lazyframe/frame.py:2422, in LazyFrame.collect(self, type_coercion, predicate_pushdown, projection_pushdown, simplify_expression, slice_pushdown, comm_subplan_elim, comm_subexpr_elim, cluster_with_columns, collapse_joins, no_optimization, engine, background, optimizations, **_kwargs)
   2420 # Only for testing purposes
   2421 callback = _kwargs.get("post_opt_callback", callback)
-> 2422 return wrap_df(ldf.collect(engine, callback))
ComputeError: parquet: File out of specification: A page can only contain i32::MAX uncompressed bytes. This one contains 4544943557
```

Which in polars can be solved with:
```
df.write_parquet(
    "test1.patquet",
   compression="zstd",
    row_group_size=10_000,  #  smaller row groups
     data_page_size=1024*1024  # 1MB page size
)
```
Title: Transform with columns parameter triggers on non-specified column access
State: closed

### Describe the bug

Iterating over a [`Column`](https://github.com/huggingface/datasets/blob/8b1bd4ec1cc9e9ce022f749abb6485ef984ae7c0/src/datasets/arrow_dataset.py#L633-L692) iterates through the parent [`Dataset`](https://github.com/huggingface/datasets/blob/8b1bd4ec1cc9e9ce022f749abb6485ef984ae7c0/src/datasets/arrow_dataset.py#L695) and applies all formatting/transforms on each row, regardless of which column is being accessed. This causes an error when transforms depend on columns not present in the projection.

### Steps to reproduce the bug

### Load a dataset with multiple columns

```python
ds = load_dataset("mrbrobot/isic-2024", split="train")
```

### Define a transform that specifies an input column

```python
def image_transform(batch):
    batch["image"] = batch["image"]  # KeyError when batch doesn't contain "image"
    return batch

# apply transform only to image column
ds = ds.with_format("torch")
ds = ds.with_transform(image_transform, columns=["image"], output_all_columns=True)
```

### Iterate over non-specified column

```python
# iterate over a different column, triggers the transform on each row, but batch doesn't contain "image"
for t in ds["target"]:  # KeyError: 'image'
    print(t)
```

### Expected behavior

If a user iterates over `ds["target"]` and the transform specifies `columns=["image"]`, the transform should be skipped.

### Environment info

`datasets`: 4.2.0
Python: 3.12.12
Linux: Debian 11.11

Comments:
Title: DOC: `mode` parameter on pdf and video features unused
State: closed

Following up on https://github.com/huggingface/datasets/pull/7840 I asked claude code to check for undocumented parameters for other features and it found:
- mode parameter on video is documented but unused: https://github.com/huggingface/datasets/blob/main/src/datasets/features/video.py#L48-L49
- the same goes for the mode parameter on the pdf feature: https://github.com/huggingface/datasets/blob/main/src/datasets/features/pdf.py#L47-L48

I assume checking if these modes can be supported and otherwise removing them is the way to go here.


Comments:
They seem to be artefacts from a copy-paste of the Image feature ^^' we should remove them
Title: Add num channels to audio
State: closed

Fixes #7837

We currently have the [mono attribute for Audio documented](https://github.com/huggingface/datasets/blob/41c05299348a499807432ab476e1cdc4143c8772/src/datasets/features/audio.py#L52C1-L54C22) but not used anywhere resulting in confusion for users. Since torchcodec does not know this attribute I suggest using `num_channels` (currently supported `None` (leave unchanged), mono: `1`, stereo: `2`).

I could also add a mono attribute but found that to be more confusing for developers and would restrict us if at any point in the future more than 2 channels are supported.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7840). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: mono parameter to the Audio feature is missing
State: closed

According to the docs, there is a "mono" parameter to the Audio feature, which turns any stereo into mono. In practice the signal is not touched and the mono parameter, even though documented, does not exist.


https://github.com/huggingface/datasets/blob/41c05299348a499807432ab476e1cdc4143c8772/src/datasets/features/audio.py#L52C1-L54C22

Comments:
Hey,  we removed the misleading passage in the docstring and enabled support for `num_channels` as torchcodec does

thanks!
Title: Add DICOM support
State: open

supports #7804
Add support for the dicom file format. 

This PR follows PR #7815 and PR #7325 closely.
Remarkable differences:
I made sure that we can load all of pydicom's test data, and encountered the `force=True` parameter that we explicitly support here. This allows to trying to load corrupted dicom files, we explicitly test this!

There is one dataset with all of dicom's test data on huggingface which can be loaded using this branch with the following script:
```python
from datasets import load_dataset
from datasets import Features, ClassLabel
from datasets.features import Dicom

features = Features({
    "dicom": Dicom(force=True),  # necessary to be able to load one corrupted file
    "label": ClassLabel(num_classes=2)
})

ds = load_dataset("TobiasPitters/dicom-sample-dataset",
                  features=features)

error_count = 0

for idx, item in enumerate(ds["test"]):
    dicom = item["dicom"]

    try:
        print(f"Type: {type(dicom)}")
        if hasattr(dicom, 'PatientID'):
            print(f"PatientID: {dicom.PatientID}")
        if hasattr(dicom, 'StudyInstanceUID'):
            print(f"StudyInstanceUID: {dicom.StudyInstanceUID}")
        if hasattr(dicom, 'Modality'):
            print(f"Modality: {dicom.Modality}")
    except Exception as e:
        error_count += 1
        print(e)

print(f"Finished processing with {error_count} errors.")
```

todo: 
- [x] add docs (will do so soon)

Comments:
Awesome ! For the docs should we rename https://huggingface.co/docs/datasets/nifti_dataset to medical_imaging_dataset and have both DICOM and NIfTI together or have separate pages in you opinion ?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7835). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Awesome ! For the docs should we rename https://huggingface.co/docs/datasets/nifti_dataset to medical_imaging_dataset and have both DICOM and NIfTI together or have separate pages in you opinion ?

Makes sense, is more intuitive for the user and the pages as proposed in this branch have a lot of overlap. I would then structure it in such a way to write some brief things about medical imaging, then introduce the formats (so basically concatenating the two pages together and removing duplicates).

Pls don't merge currently, since we'll need an `embed_storage` function in here as well. See
https://github.com/huggingface/datasets/pull/7815#issuecomment-3494094692 and the following conversation

@lhoestq, this is ready for a first round of review.
Title: Fix: Properly render [!TIP] block in stream.shuffle documentation
State: closed

The second line starting with the bracket doesn't properly render on huggingface/docs. Added "> " to address it.

In the client documentation, the markdown 'TIP' paragraph in docs/stream#shuffle is not well executed, not as the other in the same page / while markdown is correctly considering it.

Documentation:
https://huggingface.co/docs/datasets/v4.3.0/en/stream#shuffle:~:text=%5B!TIP%5D%5BIterableDataset.shuffle()%5D(/docs/datasets/v4.3.0/en/package_reference/main_classes%23datasets.IterableDataset.shuffle)%20will%20also%20shuffle%20the%20order%20of%20the%20shards%20if%20the%20dataset%20is%20sharded%20into%20multiple%20files.

Github source:
https://github.com/huggingface/datasets/blob/main/docs/source/stream.mdx#:~:text=Casting%20only%20works%20if%20the%20original%20feature%20type%20and%20new%20feature%20type%20are%20compatible.%20For%20example%2C%20you%20can%20cast%20a%20column%20with%20the%20feature%20type%20Value(%27int32%27)%20to%20Value(%27bool%27)%20if%20the%20original%20column%20only%20contains%20ones%20and%20zeros.

Comments:
Title: [DOCS][minor] TIPS paragraph not compiled in docs/stream
State: closed

In the client documentation, the markdown 'TIP' paragraph for paragraph in docs/stream#shuffle is not well executed — not as the other in the same page / while markdown is correctly considering it.

Documentation:
https://huggingface.co/docs/datasets/v4.3.0/en/stream#shuffle:~:text=%5B!TIP%5D%5BIterableDataset.shuffle()%5D(/docs/datasets/v4.3.0/en/package_reference/main_classes%23datasets.IterableDataset.shuffle)%20will%20also%20shuffle%20the%20order%20of%20the%20shards%20if%20the%20dataset%20is%20sharded%20into%20multiple%20files.

Github source:
https://github.com/huggingface/datasets/blob/main/docs/source/stream.mdx#:~:text=Casting%20only%20works%20if%20the%20original%20feature%20type%20and%20new%20feature%20type%20are%20compatible.%20For%20example%2C%20you%20can%20cast%20a%20column%20with%20the%20feature%20type%20Value(%27int32%27)%20to%20Value(%27bool%27)%20if%20the%20original%20column%20only%20contains%20ones%20and%20zeros.

Comments: