Minimartzz/github-issues huggingface dataset:
work around `tempfile` silently ignoring `TMPDIR` if the dir doesn't exist
State: open

This should help a lot of users running into `No space left on device` while using `datasets`. Normally the issue is is that `/tmp` is too small and the user needs to use another path, which they would normally set as `export TMPDIR=/some/big/storage`

However, the `tempfile` facility that `datasets` and `pyarrow` use is somewhat broken. If the path doesn't exist it'd ignore it and fall back to using `/tmp`. Watch this:

```
$ export TMPDIR='/tmp/username' 

$ python -c "\
import os
import tempfile
print(os.environ['TMPDIR'])
print(tempfile.gettempdir())"
/tmp/username
/tmp
```

Now let's ensure the path exists:
```
$ export TMPDIR='/tmp/username' 
$ mkdir -p $TMPDIR
$ python -c "\
import os
import tempfile
print(os.environ['TMPDIR'])
print(tempfile.gettempdir())"
/tmp/username
/tmp/username
```

So I recommend `datasets` do either of the 2:
1. assert if `$TMPDIR` dir doesn't exist, telling the user to create it
2. auto-create it

The reason for (1) is that I don't know why `tempdir` doesn't auto-create the dir - perhaps some security implication? I will let you guys make the decision, but the key is not to let things silently fall through and the user puzzling why no matter what they do they can't break past `No space left on device` while using `datasets`

Thank you.

I found this via https://stackoverflow.com/questions/37229398/python-tempfile-gettempdir-does-not-respect-tmpdir while trying to help a colleague to solve this exact issue.

Comments:
Hi! Just created a Pull Request (#7890) to try to fix this using your suggestions. I hope it helps!
test: add verification for HuggingFaceM4/InterleavedWebDocuments
State: open

Adds an integration test for the `HuggingFaceM4/InterleavedWebDocuments` dataset.

- Gracefully skips if the dataset is not yet available on the Hub
- Checks basic loading and structure once it becomes available

Closes #7394

First-time contributor to `datasets` — really excited about this! Happy to make any adjustments needed. 🙂

Comments:
Nifti visualization support
State: closed

closes #7870 

leverage Papaya to visualize nifti images. For this I created a Wrapper class for `nibabel.nifti1.Nifti1Image` that provides the same interface but exposes an additional `_repr_html_` method, which is needed to visualize the image in jupyter (didn't test in colab, but that should work equivalently).

Code to test (execute in a notebook):
```python
from datasets import load_dataset

ds = load_dataset("TobiasPitters/nifti-nitest-extracted",
              split="train")
image = ds[1]

image
```

Here a small video, not the most exciting scan though:
https://github.com/user-attachments/assets/1cca5f01-6fd2-48ef-a4d7-a92c1259c224

Am open to good ways to test this.

EDIT: papaya also supports dicom, didn't test it yet though

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7874). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I tested in Colab and it works perfectly :) now I want to add `_repr_html_` everywhere xD

Re: testing, I think it's fine to test manually such features
Visualization for Medical Imaging Datasets
State: closed

This is a followup to: https://github.com/huggingface/datasets/pull/7815. 

I checked the possibilities to visualize the nifti (and potentially dicom), and here's what I found:
 - https://github.com/aces/brainbrowser, AGPL3 license, last commit 3 months ago, latest (github) release from 2017. It's available on jsdelivr: https://www.jsdelivr.com/package/npm/brainbrowser (but that is from 2015!)
 - https://github.com/rii-mango/Papaya, custom but BSD-style license that would require datasets to list the conditions in their readme somewhere, last commit June 2024. I looked into this library and it looks mature and good enough for our use case, but just working on it for a short time I wasn't able to get this to work, but am sure we could get this working, would probably require some JS on datasets' end. Available on jsdelivr as well: https://www.jsdelivr.com/package/npm/papaya-viewer. Seems like it's frequently loaded.
- https://github.com/hanayik/niivue, BSD3 license, last commit May 26, 2021. Archived. Doesn't look like an option.

I think  the only real option for us Papaya, but there is also the risk that we'll end up with an unmaintained package after a while, since development seems to be slow or even halted. 

I think conceptually we would need to figure out how we can build a good solution for visualizing Medical Image data. On shap, we have a separate javascript folder in which we render visualizations, this could be a blueprint but will require a bundler, etc. Alternatively one could go with a naive approach to just write some html code in a python string and load the package via jsdelivr.


@lhoestq thoughts?

Comments:
It would be amazing to be able to show the Papaya UI in google colab / jupyter notebook. IIRC both allow serving javascript via nbextensions that we can surely use in HTML() objects.

Alternatively we could also start with a simple approach and dump the medical image data as a video file that goes through the slices, so we don't need javascript.
Why does dataset merge fail when tools have different parameters?
State: open

Hi, I have a question about SFT (Supervised Fine-tuning) for an agent model.

Suppose I want to fine-tune an agent model that may receive two different tools: tool1 and tool2. These tools have different parameters and types in their schema definitions.

When I try to merge datasets containing different tool definitions, I get the following error:

TypeError: Couldn't cast array of type
struct<refundFee: struct<description: string, type: string>, ... , servicerId: struct<description: string, type: string>>
to
{
  'refundFee': {'description': Value(dtype='string'), 'type': Value(dtype='string')},
  ...
  'templateId': {'description': Value(dtype='string'), 'type': Value(dtype='string')}
}
From my understanding, the merge fails because the tools column's nested structure is different across datasets — e.g., one struct contains an extra field servicerId while the other does not. This causes HuggingFace Datasets (and its underlying Apache Arrow schema) to reject the merge.

My question is: why is it designed this way?

Is this strict schema matching a hard requirement of the library?
Is there a recommended way to merge datasets with different tool schemas (different parameters and types)?
For an agent model supporting multiple tools, what's the best practice for preparing/merging training data without losing flexibility?
Any guidance or design rationale would be greatly appreciated. Thanks!

Comments:
Hi @hitszxs,
 This is indeed by design,

The `datasets` library is built on top of [Apache Arrow](https://arrow.apache.org/), which uses a **columnar storage format** with strict schema requirements. When you try to concatenate/merge datasets, the library checks if features can be aligned using the [`_check_if_features_can_be_aligned`](https://github.com/huggingface/datasets/blob/main/src/datasets/features/features.py#L2297-L2316) function.

Two datasets can be merged if:
1. Columns with the same name have the **same type**, OR
2. One of them has `Value("null")` (representing missing data)

For struct types (nested dictionaries like your tool schemas), **all fields must match exactly**. This ensures type safety and efficient columnar storage.

## Workarounds for Your Use Case
 Store tools as JSON strings

Instead of using nested struct types, store the tool definitions as JSON strings
Performance Issue: save_to_disk() 200-1200% slower due to unconditional flatten_indices()
State: open

## 🐛 Bug Description

The `save_to_disk()` method unconditionally calls `flatten_indices()` when `_indices` is not None, causing severe performance degradation for datasets processed with filtering, shuffling, or multiprocessed mapping operations.

**Root cause**: This line rebuilds the entire dataset unnecessarily:
```python
dataset = self.flatten_indices() if self._indices is not None else self
```

## 📊 Performance Impact

| Dataset Size | Operation | Save Time | Slowdown |
|-------------|-----------|-----------|----------|
| 100K | Baseline (no indices) | 0.027s | - |
| 100K | Filtered (with indices) | 0.146s | **+431%** |
| 100K | Shuffled (with indices) | 0.332s | **+1107%** |
| 250K | Shuffled (with indices) | 0.849s | **+1202%** |

## 🔄 Reproduction

```python
from datasets import Dataset
import time

# Create dataset
dataset = Dataset.from_dict({'text': [f'sample {i}' for i in range(100000)]})

# Baseline save (no indices)
start = time.time()
dataset.save_to_disk('baseline')
baseline_time = time.time() - start

# Filtered save (creates indices)
filtered = dataset.filter(lambda x: True)
start = time.time()
filtered.save_to_disk('filtered')
filtered_time = time.time() - start

print(f"Baseline: {baseline_time:.3f}s")
print(f"Filtered: {filtered_time:.3f}s") 
print(f"Slowdown: {(filtered_time/baseline_time-1)*100:.1f}%")
```

**Expected output**: Filtered dataset is 400-1000% slower than baseline

## 💡 Proposed Solution

Add optional parameter to control flattening:

```python
def save_to_disk(self, dataset_path, flatten_indices=True):
    dataset = self.flatten_indices() if (self._indices is not None and flatten_indices) else self
    # ... rest of save logic
```

**Benefits**:
- ✅ Immediate performance improvement for users who don't need flattening
- ✅ Backwards compatible (default behavior unchanged)  
- ✅ Simple implementation

## 🌍 Environment

- **datasets version**: 2.x
- **Python**: 3.10+
- **OS**: Linux/macOS/Windows

## 📈 Impact

This affects **most ML preprocessing workflows** that filter/shuffle datasets before saving. Performance degradation scales exponentially with dataset size, making it a critical bottleneck for production systems.

## 🔗 Additional Resources

We have comprehensive test scripts demonstrating this across multiple scenarios if needed for further investigation.

Comments:
fix some broken links
State: closed

would be cool to automate finding those broken links as i think there might be many of them @lhoestq @albertvillanova 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7859). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Support downloading specific splits in `load_dataset`
State: open

This is PR builds on top of #7706 to revive the unfinished #6832 but isn't just cleaning up things, here are some important changes:
 - `download_mode="FORCE_REDOWNLOAD"` is interpreted as always creating a clean slate, that means that even if we already did:
 ```python
load_dataset("<name>")
load_dataset("<name>", split="train", download_mode="force_redownload")
```
This makes sure that only the train dataset is available after executing both. This was different in the original PR, which proposed that train and test would be available. 
- `download_mode="REUSE_DATASET_IF_EXISTS"` is interpreted as only ever adding new data, never redownloading OR deleting other splits. This was different in the original PR, where  
 ```python
load_dataset("<name>", split="test")
load_dataset("<name>", split="train")
```
resulted in only the train data being available, which I deem very unintuitive and probably not what users want. Also I argue that this is just the first step to a more user friendly partial loading when specifying percentages (or maybe even single instances) via the ReadInstructions, and then doing
 ```python
load_dataset("<name>", split="test[:10%]")
load_dataset("<name>", split="test[10%:]")
```
should result IMO in the whole dataset being cached locally without redownloads.

Furthermore this PR fixes a couple issues with the previous PR, e.g. a [missing comparison](https://github.com/huggingface/datasets/pull/7706/files#diff-f933ce41f71c6c0d1ce658e27de62cbe0b45d777e9e68056dd012ac3eb9324f7R877) and adding tests for the proposed changes in behaviour, which would both fail on @ArjunJagdale's original PR. 

Todo:
 - [ ] update docs?

Future outlook (just my opinions and up for debate):
As mentioned before, I would see this as just a step towards the feature of partial percentage loading (though how the API should behave in that case is not entirely clear for me now) and maybe we could introduce another `download_mode="FORCE_REDOWNLOAD_SPLIT"`, which makes sure that even if a split is specified, only the referenced split is redownloaded and everything else left unchanged, this would then allow users more granular control over what they want to redownload. 
@lhoestq very curious to get your opinion on this.

Comments:
@CloseChoice This looks great! You're absolutely right about the missing comparison - that's a critical bug I missed. 

@lhoestq just resolved the conflicts inflicted by the latest changes. Thought it might be good giving this a shot now before more changes mess with this PR.
Missing transcript column when loading a local dataset with "audiofolder"
State: closed

### Describe the bug

My local dataset is not properly loaded when using `load_dataset("audiofolder", data_dir="my_dataset")` with a `jsonl` metadata file.
Only the `audio` column is read while the `transcript` column is not.

The last tested `datasets` version where the behavior was still correct is 2.18.0.

### Steps to reproduce the bug

Dataset directory structure:
```
my_dataset/
  - data/
    - test/
      - 54db8760de3cfbff3c8a36a36b4d0f77_00390.0_04583.0.mp3
      - 54db8760de3cfbff3c8a36a36b4d0f77_04583.0_05730.0.mp3
      - ...
  - metadata.jsonl
```

`metadata.jsonl` file content:
```
{"file_name": "data/test/54db8760de3cfbff3c8a36a36b4d0f77_00390.0_04583.0.mp3", "transcript": "Ata tudoù penaos e tro ar bed ?"}
{"file_name": "data/test/54db8760de3cfbff3c8a36a36b4d0f77_04583.0_05730.0.mp3", "transcript": "Ur gwir blijadur eo adkavout ac'hanoc'h hiziv."}
...
```
```python3
my_dataset = load_dataset("audiofolder", data_dir="my_dataset")

print(my_dataset)

'''
DatasetDict({
    test: Dataset({
        features: ['audio'],
        num_rows: 347
    })
})
'''

print(my_dataset['test'][0])

'''
{'audio': <datasets.features._torchcodec.AudioDecoder object at 0x75ffcd172510>}
'''
```

### Expected behavior

Being able to access the `transcript` column in the loaded dataset.

### Environment info

- `datasets` version: 4.4.1
- Platform: Linux-6.5.0-45-generic-x86_64-with-glibc2.39
- Python version: 3.13.9
- `huggingface_hub` version: 1.1.2
- PyArrow version: 22.0.0
- Pandas version: 2.3.3
- `fsspec` version: 2025.10.0

Note: same issue with `datasets` v3.6.0

Comments:
First bad commit 5c8869f8c36dbc8c8d423030b7b7c4fd64f8c729

EDIT: This is not a bug or a regression. It was a breaking change introduced in the commit I mentioned and was also documented in there. The docs state how to handle this now, see https://huggingface.co/docs/datasets/main/en/audio_load#audiofolder-with-metadata

or simply, move your metadata into the splits folder and update the paths, in your case this would look like this:
```bash
my_dataset/
  - data/
    - test/
      - 54db8760de3cfbff3c8a36a36b4d0f77_00390.0_04583.0.mp3
      - 54db8760de3cfbff3c8a36a36b4d0f77_04583.0_05730.0.mp3
      - metadata.jsonl
```

and the pahts in the jsonl should be relative to the metadata.json:
```bash
{"file_name": "54db8760de3cfbff3c8a36a36b4d0f77_00390.0_04583.0.mp3", "transcript": "Ata tudoù penaos e tro ar bed ?"}
{"file_name": "54db8760de3cfbff3c8a36a36b4d0f77_04583.0_05730.0.mp3", "transcript": "Ur gwir blijadur eo adkavout ac'hanoc'h hiziv."}
...
```

So I think this can be closed.

Thank you for your quick answer !
I'm sorry I missed that in the documentation.
Everything works fine again after following your recommendations.
I'm closing the issue.
[Distributed] split_dataset_by_node() gives the same number of examples for each node
State: open

this works:

```python
import torch.distributed as dist
from datasets import IterableDataset
from datasets.distributed import split_dataset_by_node
from collections import Counter

def g(shards):
    for shard in shards:
        # shards don't have the same length
        num_examples = 3 + shard 
        for i in range(num_examples):
            yield {"shard": f"{shard=}", "i": i}

if __name__ == "__main__":
    dist.init_process_group(backend="gloo")
    rank, world_size = dist.get_rank(), dist.get_world_size()
    num_shards = 6
    ds = IterableDataset.from_generator(g, gen_kwargs={"shards": list(range(num_shards))})
    ds = split_dataset_by_node(ds, rank=rank, world_size=world_size)
    # Check that each rank has the same number of examples
    # and show the number of examples per shard and per rank
    counter = Counter(ds["shard"])
    print(f"# {rank=}\ttotal={counter.total()}\t{counter}", flush=True)

    # torchrun --nproc_per_node 2 script.py
    # rank=0        total=16        Counter({'shard=4': 7, 'shard=2': 5, 'shard=0': 4})
    # rank=1        total=16        Counter({'shard=3': 6, 'shard=5': 6, 'shard=1': 4})

```

TODO: make it work with DataLoader (communicate with main process to know when the node runs out of data ?)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7854). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Making this work with multiple workers could create a lot of communication for not a lot of benefits, considering you can simply use `Join()` to let nodes shutdown when they run out of data while the other nodes continue training: https://docs.pytorch.org/docs/stable/distributed.algorithms.join.html
Fix embed storage nifti
State: closed

Fixes #7852 

Adds `embed_storage` function and allows gzipped files to be loaded correctly from local storage.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7853). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Problems with NifTI
State: closed

### Describe the bug

There are currently 2 problems with the new NifTI feature:
1. dealing with zipped files, this is mentioned and explained [here](https://github.com/huggingface/datasets/pull/7815#issuecomment-3496199503)
2. when uploading via the `niftifolder` feature, the resulting parquet only contains relative paths to the nifti files:

```bash
table['nifti']
<pyarrow.lib.ChunkedArray object at 0x798245d37d60>
[
  -- is_valid: all not null
  -- child 0 type: binary
    [
      null,
      null,
      null,
      null,
      null,
      null
    ]
  -- child 1 type: string
    [
      "/home/tobias/programming/github/datasets/nifti_extracted/T1.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2-interleaved.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2_-interleaved.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2_.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/fieldmap.nii"
    ]
]
```
instead of containing bytes. The code is copy pasted from PDF, so I wonder what is going wrong here.

### Steps to reproduce the bug

see the linked comment

### Expected behavior

downloading should work as smoothly as for pdf

### Environment info

- `datasets` version: 4.4.2.dev0
- Platform: Linux-6.14.0-33-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.35.3
- PyArrow version: 21.0.0
- Pandas version: 2.3.3
- `fsspec` version: 2025.9.0


Comments:
> 2. when uploading via the niftifolder feature, the resulting parquet only contains relative paths to the nifti files:

what did you use to upload the dataset ? iirc push_to_hub() does upload the bytes as well, but to_parquet() doesn't

> > 2. when uploading via the niftifolder feature, the resulting parquet only contains relative paths to the nifti files:
> 
> what did you use to upload the dataset ? iirc push_to_hub() does upload the bytes as well, but to_parquet() doesn't

I used `push_to_hub` but the problem is that the nifti feature does not have an `embed_storage` function
Add fasta support
State: open

This PR adds support for FASTA files conversion to Parquet. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7851). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

A few comments:

- Have you tried using this with longer sequences? @UriNeri developed something similar internally and used it with viral genomes. He got some Parquet errors due to genomes not fitting in a `utf8` column. This was fixed by using `large_utf8`.
- If you're only using it to read FASTA files, I think that having BioPython as a dependency is overkill. The library is very large and the FASTA parser isn't particularly fast. I have an example of a fast parser with no external references [here](https://gist.github.com/apcamargo/d039aa04a2cbbcbb14e2d34a0963b862) (this is actually based on [`readfq.py`](https://github.com/lh3/readfq/blob/master/readfq.py), with a couple of extra functions that might not be useful in the context of this PR)

> * If you're only using it to read FASTA files, I think that having BioPython as a dependency is overkill. The library is very large and the FASTA parser isn't particularly fast. I have an example of a fast parser with no external references [here](https://gist.github.com/apcamargo/d039aa04a2cbbcbb14e2d34a0963b862) (this is actually based on [`readfq.py`](https://github.com/lh3/readfq/blob/master/readfq.py), with a couple of extra functions that might not be useful in the context of this PR)

What @apcamargo said, plus FWIW in **our approach** (so might not be relevant here) we use polars (with custom fasta io parser) or polars-bio (that has a `scan_fasta` function) and we foudn out that the page size sometimes need to be adjusted:
```
envs/default/lib/python3.9/site-packages/polars/lazyframe/frame.py:2422, in LazyFrame.collect(self, type_coercion, predicate_pushdown, projection_pushdown, simplify_expression, slice_pushdown, comm_subplan_elim, comm_subexpr_elim, cluster_with_columns, collapse_joins, no_optimization, engine, background, optimizations, **_kwargs)
   2420 # Only for testing purposes
   2421 callback = _kwargs.get("post_opt_callback", callback)
-> 2422 return wrap_df(ldf.collect(engine, callback))
ComputeError: parquet: File out of specification: A page can only contain i32::MAX uncompressed bytes. This one contains 4544943557
```

Which in polars can be solved with:
```
df.write_parquet(
    "test1.patquet",
   compression="zstd",
    row_group_size=10_000,  #  smaller row groups
     data_page_size=1024*1024  # 1MB page size
)
```