AniketTec/github-issues huggingface dataset:
Title: fix(tests): stabilize flaky Hub LFS integration test
State: open

## Problem

`test_push_dataset_dict_to_hub_overwrite_files` intermittently fails with:
```
BadRequestError: LFS pointer pointed to a file that does not exist
```

This has been causing the `deps-latest` integration tests to fail on main (visible in recent CI runs). I ran into this while working on the BIDS loader PR and dug into the root cause.

## Root Cause

Two race conditions in the test:

1. **LFS propagation timing** - Rapid successive `push_to_hub` calls don't wait for Hub to fully propagate LFS objects between pushes
2. **Repo name reuse** - The second test scenario reused the same repo name from scenario 1, creating a race between deletion and recreation

## Solution

- Add `_wait_for_repo_ready()` helper that polls `list_repo_files` to ensure the repo is consistent before subsequent operations
- Use a unique repo name (`ds_name_2`) for the second scenario, eliminating the delete/create race entirely

## Testing

All 4 integration test variants now pass:
- ✅ `ubuntu-latest, deps-latest` (was failing)
- ✅ `ubuntu-latest, deps-minimum`
- ✅ `windows-latest, deps-latest` (was failing)
- ✅ `windows-latest, deps-minimum`

Validated on fork: https://github.com/The-Obstacle-Is-The-Way/datasets/pull/4

## Related

- #7600 (push_to_hub concurrency)
- #6392 (push_to_hub connection robustness)

cc @lhoestq - small fix but should help CI reliability

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7889). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Maybe `push_to_hub()` can retry on error 400 when committing to the Hub instead ? This way we make sure `push_to_hub()` works without having to add the waiting step

Thanks for the pointer @lhoestq! Updated to add LFS 400 retry handling directly in `push_to_hub()` across all implementations (Dataset, DatasetDict, IterableDataset, IterableDatasetDict). Reverted the test-side waits.
Title: Add type overloads to load_dataset for better static type inference
State: closed

Fixes #7883

This PR adds @overload decorators to load_dataset() to help type checkers like Pylance and mypy correctly infer the return type based on the split and streaming parameters.

Changes:
- Added typing imports (Literal, overload) to load.py
- Added 4 @overload signatures that map argument combinations to specific return types:
  * split=None, streaming=False -> DatasetDict
  * split specified, streaming=False -> Dataset
  * split=None, streaming=True -> IterableDatasetDict
  * split specified, streaming=True -> IterableDataset

This resolves the Pylance error where to_csv() was not recognized on Dataset objects returned by load_dataset(..., split='train'), since the type checker previously saw the return type as a Union that included types without to_csv().

No runtime behavior changes - this is purely a static typing improvement.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7888). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: fix(nifti): enable lazy loading for Nifti1ImageWrapper
State: closed

## Summary
- **Single-line fix**: Change `dataobj=nifti_image.get_fdata()` → `dataobj=nifti_image.dataobj`
- Preserves nibabel's `ArrayProxy` for true lazy loading instead of eagerly loading entire NIfTI files into memory
- Improves error handling: corrupted files now fail at access time with clear context instead of silently during decode

## Problem
The `Nifti1ImageWrapper.__init__` was calling `get_fdata()` which immediately loads the entire image into memory. For large 4D fMRI files (often 1-2GB), this causes:
1. **Memory issues** - Full data loaded during decode, not on demand
2. **Poor error handling** - Corrupted files crash at access time with unclear error messages (e.g., `EOFError` with no file path)
3. **No graceful recovery** - Entire dataset iteration fails on one bad file

## Solution
Use `nifti_image.dataobj` which preserves the underlying `ArrayProxy`, deferring actual I/O to `get_fdata()` calls.

## Test Plan
- [x] Added `test_nifti_lazy_loading` to verify `ArrayProxy` is preserved
- [x] All 22 existing NIfTI tests pass
- [x] End-to-end tested with real OpenNeuro data (ds000102)
- [x] CodeRabbit approved: "Switch to dataobj correctly restores nibabel's lazy loading semantics... This looks solid"

## Related
- Discovered while testing BIDS loader PR #7886
- Complements the NIfTI + NiiVue viewer work from #7878 and #7874

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7887). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: feat(bids): Add BIDS dataset loader for neuroimaging data
State: open

## Summary

Adds native BIDS (Brain Imaging Data Structure) dataset loading support using PyBIDS, enabling `load_dataset('bids', data_dir='/path/to/bids')` workflow for neuroimaging researchers.

**Contributes to #7804** (Support scientific data formats) - BIDS is a widely-used standard for organizing neuroimaging data built on NIfTI files.

## Changes

### Core Implementation
- `src/datasets/packaged_modules/bids/bids.py` - GeneratorBasedBuilder implementation
- `src/datasets/packaged_modules/bids/__init__.py` - Module exports
- `src/datasets/packaged_modules/__init__.py` - Registration with module registry
- `src/datasets/config.py` - `PYBIDS_AVAILABLE` config flag
- `setup.py` - Optional `pybids>=0.21.0` + nibabel dependency

### Features
- Automatic BIDS structure validation
- Subject/session/datatype filtering via config
- JSON sidecar metadata extraction
- NIfTI file decoding via existing Nifti feature

### Documentation & Tests
- `docs/source/bids_dataset.mdx` - User guide with examples
- `tests/packaged_modules/test_bids.py` - Unit tests (4 tests)

## Usage

```python
from datasets import load_dataset

# Load entire BIDS dataset
ds = load_dataset('bids', data_dir='/path/to/bids_dataset')

# Filter by subject/session
ds = load_dataset('bids', 
    data_dir='/path/to/bids_dataset',
    subjects=['01', '02'],
    sessions=['baseline']
)

# Access samples
sample = ds['train'][0]
print(sample['subject'])      # '01'
print(sample['nifti'].shape)  # (176, 256, 256)
print(sample['metadata'])     # JSON sidecar data
```

## Test plan

- [x] All 4 unit tests pass (`pytest tests/packaged_modules/test_bids.py`)
- [x] `make quality` passes (ruff check)
- [x] End-to-end tested with real OpenNeuro data (ds000102)

## Context

This PR is part of the neuroimaging initiative discussed with @TobiasPitters. Follows the BIDS 1.10.1 specification and leverages the existing Nifti feature for NIfTI file handling.

Related PRs:
- #7874 (Nifti visualization support)
- #7878 (Replace papaya with niivue)

Comments:
Title: Fix 7846: add_column and add_item erroneously(?) require new_fingerprint parameter 
State: closed

Summary of the change:

Made new_fingerprint optional (Optional[str] = None) in Dataset.add_column and Dataset.add_item
Added a simple test to verify both methods work without providing a fingerprint

Why this change is safe:

The Dataset constructor already handles fingerprint=None by generating a new fingerprint automatically
No internal logic is broken — if a user provides a fingerprint, it’s still used as before
The change only affects the function signature, making it more user-friendly without changing any functionality

Comments:
@lhoestq can you help review this pull request?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7884). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Data.to_csv() cannot be recognized by pylance
State: closed

### Describe the bug

Hi, everyone ! I am a beginner with datasets. 
I am testing reading multiple CSV files from a zip archive. The result of reading the dataset shows success, and it can ultimately be correctly saved to CSV.

Intermediate results:
```
Generating train split: 62973 examples [00:00, 175939.01 examples/s]
DatasetDict({
    train: Dataset({
        features: ['交易时间\t', '收支方向\t', '业务（产品）种类\t', '交易金额\t', '币种\t', '时点余额\t', '对手方名称\t', '对方机构名称\t', ' 对方钱包ID/账号\t', '交易对手名称\t', '交易对手编号\t', '交易流水号\t', '摘要\t', '附言\t', '备注\t', '用途\t', '客户流水号\t'],
        num_rows: 62973
    })
})
```
However, Pylance gives me the following error:
```
Cannot access attribute "to_csv" for class "DatasetDict"
Attribute "to_csv" is unknownPylance[reportAttributeAccessIssue](https://github.com/microsoft/pylance-release/blob/main/docs/diagnostics/reportAttributeAccessIssue.md)```

Cannot access attribute "to_csv" for class "IterableDatasetDict"
  Attribute "to_csv" is unknownPylance[reportAttributeAccessIssue](https://github.com/microsoft/pylance-release/blob/main/docs/diagnostics/reportAttributeAccessIssue.md)

(method) to_csv: Unknown | ((path_or_buf: datasets.utils.typing.PathLike | BinaryIO, batch_size: int | None = None, num_proc: int | None = None, storage_options: dict[Unknown, Unknown] | None = None, **to_csv_kwargs: Unknown) -> int) | ((path_or_buf: datasets.utils.typing.PathLike | BinaryIO, batch_size: int | None = None, storage_options: dict[Unknown, Unknown] | None = None, **to_csv_kwargs: Unknown) -> int)
```
I ignored the error and continued executing to get the correct result:
```
Dataset({
    features: ['交易时间\t', '收支方向\t', '业务（产品）种类\t', '交易金额\t', '币种\t', '时点余额\t', '对手方名称\t', '对方机构名称\t', '对方 钱包ID/账号\t', '交易对手名称\t', '交易对手编号\t', '交易流水号\t', '摘要\t', '附言\t', '备注\t', '用途\t', '客户流水号\t'],
    num_rows: 62973
})
```
Since the data volume is small, I manually merged the CSV files, and the final result is consistent with what the program saved.


looks like :

<img width="1264" height="150" alt="Image" src="https://github.com/user-attachments/assets/743540d7-ad8c-4531-ae7e-de71a5243a32" />


### Steps to reproduce the bug

this is my code.
```
from datasets import load_dataset

def main():
    url = "data/test.zip"
    data_files = {"train": url}
    dataset = load_dataset("csv", data_files=data_files,split="train", encoding="gbk", skiprows=2)
    # print(dataset)
    dataset.to_csv("data/test.csv")


if __name__ == "__main__":
    main()
```

### Expected behavior

I want to know why this happens. Is there something wrong with my code?

### Environment info

OS: Windows 11    **upgrade from** OS: Windows_NT x64 10.0.22631
Editor: 
VS Code  Version: 1.106.2 (user setup)
"datasets"    version = "4.4.1"


Comments:
Title: Inconsistent loading of LFS-hosted files in epfml/FineWeb-HQ dataset
State: open

### Describe the bug

Some files in the `epfml/FineWeb-HQ` dataset fail to load via the Hugging Face `datasets` library.  

- xet-hosted  files load fine
- LFS-hosted files sometimes fail

Example:  
- Fails: https://huggingface.co/datasets/epfml/FineWeb-HQ/blob/main/data/CC-MAIN-2024-26/000_00003.parquet  
- Works: https://huggingface.co/datasets/epfml/FineWeb-HQ/blob/main/data/CC-MAIN-2024-42/000_00027.parquet  

Discussion: https://huggingface.co/datasets/epfml/FineWeb-HQ/discussions/2  

### Steps to reproduce the bug

```python
from datasets import load_dataset

ds = load_dataset(
    "epfml/FineWeb-HQ",
    data_files="data/CC-MAIN-2024-26/000_00003.parquet",
)
```

Error message:
```
HfHubHTTPError: 403 Forbidden: None.
Cannot access content at: https://cdn-lfs-us-1.hf.co/repos/...
Make sure your token has the correct permissions.
...
<Error><Code>AccessDenied</Code><Message>Access Denied</Message></Error>
```

### Expected behavior

It should load the dataset for all files. 

### Environment info

- python 3.10
- datasets 4.4.1

Comments:
we are investigating the issue, I'll post updates here

It should be all good now, sorry for the inconvenience.

We found an error that happened during the migration of this repository from Git LFS to Xet and fixed the issue
Title: Spurious label column created when audiofolder/imagefolder directories match split names
State: open

## Describe the bug

When using `audiofolder` or `imagefolder` with directories for **splits** (train/test) rather than class labels, a spurious `label` column is incorrectly created.

**Example:** https://huggingface.co/datasets/datasets-examples/doc-audio-4
```
from datasets import load_dataset
ds = load_dataset("datasets-examples/doc-audio-4")
print(ds["train"].features)
```
Shows 'label' column with ClassLabel(names=['test', 'train']) - incorrect!## Root cause

In `folder_based_builder.py`, the `labels` set is accumulated across ALL splits (line 77). When directories are `train/` and `test/`:
- `labels = {"train", "test"}` → `len(labels) > 1` → `add_labels = True`
- Spurious label column is created with split names as class labels

## Expected behavior

No `label` column should be added when directory names match split names.

## Proposed fix

Skip label inference when inferred labels match split names.

cc @lhoestq 

Comments:
Title: work around `tempfile` silently ignoring `TMPDIR` if the dir doesn't exist
State: closed

This should help a lot of users running into `No space left on device` while using `datasets`. Normally the issue is is that `/tmp` is too small and the user needs to use another path, which they would normally set as `export TMPDIR=/some/big/storage`

However, the `tempfile` facility that `datasets` and `pyarrow` use is somewhat broken. If the path doesn't exist it'd ignore it and fall back to using `/tmp`. Watch this:

```
$ export TMPDIR='/tmp/username' 

$ python -c "\
import os
import tempfile
print(os.environ['TMPDIR'])
print(tempfile.gettempdir())"
/tmp/username
/tmp
```

Now let's ensure the path exists:
```
$ export TMPDIR='/tmp/username' 
$ mkdir -p $TMPDIR
$ python -c "\
import os
import tempfile
print(os.environ['TMPDIR'])
print(tempfile.gettempdir())"
/tmp/username
/tmp/username
```

So I recommend `datasets` do either of the 2:
1. assert if `$TMPDIR` dir doesn't exist, telling the user to create it
2. auto-create it

The reason for (1) is that I don't know why `tempdir` doesn't auto-create the dir - perhaps some security implication? I will let you guys make the decision, but the key is not to let things silently fall through and the user puzzling why no matter what they do they can't break past `No space left on device` while using `datasets`

Thank you.

I found this via https://stackoverflow.com/questions/37229398/python-tempfile-gettempdir-does-not-respect-tmpdir while trying to help a colleague to solve this exact issue.

Comments:
Hi! Just created a Pull Request (#7890) to try to fix this using your suggestions. I hope it helps!
Title: test: add verification for HuggingFaceM4/InterleavedWebDocuments
State: open

Adds an integration test for the `HuggingFaceM4/InterleavedWebDocuments` dataset.

- Gracefully skips if the dataset is not yet available on the Hub
- Checks basic loading and structure once it becomes available

Closes #7394

First-time contributor to `datasets` — really excited about this! Happy to make any adjustments needed. 🙂

Comments:
Title: Nifti visualization support
State: closed

closes #7870 

leverage Papaya to visualize nifti images. For this I created a Wrapper class for `nibabel.nifti1.Nifti1Image` that provides the same interface but exposes an additional `_repr_html_` method, which is needed to visualize the image in jupyter (didn't test in colab, but that should work equivalently).

Code to test (execute in a notebook):
```python
from datasets import load_dataset

ds = load_dataset("TobiasPitters/nifti-nitest-extracted",
              split="train")
image = ds[1]

image
```

Here a small video, not the most exciting scan though:
https://github.com/user-attachments/assets/1cca5f01-6fd2-48ef-a4d7-a92c1259c224

Am open to good ways to test this.

EDIT: papaya also supports dicom, didn't test it yet though

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7874). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I tested in Colab and it works perfectly :) now I want to add `_repr_html_` everywhere xD

Re: testing, I think it's fine to test manually such features
Title: Visualization for Medical Imaging Datasets
State: closed

This is a followup to: https://github.com/huggingface/datasets/pull/7815. 

I checked the possibilities to visualize the nifti (and potentially dicom), and here's what I found:
 - https://github.com/aces/brainbrowser, AGPL3 license, last commit 3 months ago, latest (github) release from 2017. It's available on jsdelivr: https://www.jsdelivr.com/package/npm/brainbrowser (but that is from 2015!)
 - https://github.com/rii-mango/Papaya, custom but BSD-style license that would require datasets to list the conditions in their readme somewhere, last commit June 2024. I looked into this library and it looks mature and good enough for our use case, but just working on it for a short time I wasn't able to get this to work, but am sure we could get this working, would probably require some JS on datasets' end. Available on jsdelivr as well: https://www.jsdelivr.com/package/npm/papaya-viewer. Seems like it's frequently loaded.
- https://github.com/hanayik/niivue, BSD3 license, last commit May 26, 2021. Archived. Doesn't look like an option.

I think  the only real option for us Papaya, but there is also the risk that we'll end up with an unmaintained package after a while, since development seems to be slow or even halted. 

I think conceptually we would need to figure out how we can build a good solution for visualizing Medical Image data. On shap, we have a separate javascript folder in which we render visualizations, this could be a blueprint but will require a bundler, etc. Alternatively one could go with a naive approach to just write some html code in a python string and load the package via jsdelivr.


@lhoestq thoughts?

Comments:
It would be amazing to be able to show the Papaya UI in google colab / jupyter notebook. IIRC both allow serving javascript via nbextensions that we can surely use in HTML() objects.

Alternatively we could also start with a simple approach and dump the medical image data as a video file that goes through the slices, so we don't need javascript.
Title: Why does dataset merge fail when tools have different parameters?
State: open

Hi, I have a question about SFT (Supervised Fine-tuning) for an agent model.

Suppose I want to fine-tune an agent model that may receive two different tools: tool1 and tool2. These tools have different parameters and types in their schema definitions.

When I try to merge datasets containing different tool definitions, I get the following error:

TypeError: Couldn't cast array of type
struct<refundFee: struct<description: string, type: string>, ... , servicerId: struct<description: string, type: string>>
to
{
  'refundFee': {'description': Value(dtype='string'), 'type': Value(dtype='string')},
  ...
  'templateId': {'description': Value(dtype='string'), 'type': Value(dtype='string')}
}
From my understanding, the merge fails because the tools column's nested structure is different across datasets — e.g., one struct contains an extra field servicerId while the other does not. This causes HuggingFace Datasets (and its underlying Apache Arrow schema) to reject the merge.

My question is: why is it designed this way?

Is this strict schema matching a hard requirement of the library?
Is there a recommended way to merge datasets with different tool schemas (different parameters and types)?
For an agent model supporting multiple tools, what's the best practice for preparing/merging training data without losing flexibility?
Any guidance or design rationale would be greatly appreciated. Thanks!

Comments:
Hi @hitszxs,
 This is indeed by design,

The `datasets` library is built on top of [Apache Arrow](https://arrow.apache.org/), which uses a **columnar storage format** with strict schema requirements. When you try to concatenate/merge datasets, the library checks if features can be aligned using the [`_check_if_features_can_be_aligned`](https://github.com/huggingface/datasets/blob/main/src/datasets/features/features.py#L2297-L2316) function.

Two datasets can be merged if:
1. Columns with the same name have the **same type**, OR
2. One of them has `Value("null")` (representing missing data)

For struct types (nested dictionaries like your tool schemas), **all fields must match exactly**. This ensures type safety and efficient columnar storage.

## Workarounds for Your Use Case
 Store tools as JSON strings

Instead of using nested struct types, store the tool definitions as JSON strings
Title: Performance Issue: save_to_disk() 200-1200% slower due to unconditional flatten_indices()
State: open

## 🐛 Bug Description

The `save_to_disk()` method unconditionally calls `flatten_indices()` when `_indices` is not None, causing severe performance degradation for datasets processed with filtering, shuffling, or multiprocessed mapping operations.

**Root cause**: This line rebuilds the entire dataset unnecessarily:
```python
dataset = self.flatten_indices() if self._indices is not None else self
```

## 📊 Performance Impact

| Dataset Size | Operation | Save Time | Slowdown |
|-------------|-----------|-----------|----------|
| 100K | Baseline (no indices) | 0.027s | - |
| 100K | Filtered (with indices) | 0.146s | **+431%** |
| 100K | Shuffled (with indices) | 0.332s | **+1107%** |
| 250K | Shuffled (with indices) | 0.849s | **+1202%** |

## 🔄 Reproduction

```python
from datasets import Dataset
import time

# Create dataset
dataset = Dataset.from_dict({'text': [f'sample {i}' for i in range(100000)]})

# Baseline save (no indices)
start = time.time()
dataset.save_to_disk('baseline')
baseline_time = time.time() - start

# Filtered save (creates indices)
filtered = dataset.filter(lambda x: True)
start = time.time()
filtered.save_to_disk('filtered')
filtered_time = time.time() - start

print(f"Baseline: {baseline_time:.3f}s")
print(f"Filtered: {filtered_time:.3f}s") 
print(f"Slowdown: {(filtered_time/baseline_time-1)*100:.1f}%")
```

**Expected output**: Filtered dataset is 400-1000% slower than baseline

## 💡 Proposed Solution

Add optional parameter to control flattening:

```python
def save_to_disk(self, dataset_path, flatten_indices=True):
    dataset = self.flatten_indices() if (self._indices is not None and flatten_indices) else self
    # ... rest of save logic
```

**Benefits**:
- ✅ Immediate performance improvement for users who don't need flattening
- ✅ Backwards compatible (default behavior unchanged)  
- ✅ Simple implementation

## 🌍 Environment

- **datasets version**: 2.x
- **Python**: 3.10+
- **OS**: Linux/macOS/Windows

## 📈 Impact

This affects **most ML preprocessing workflows** that filter/shuffle datasets before saving. Performance degradation scales exponentially with dataset size, making it a critical bottleneck for production systems.

## 🔗 Additional Resources

We have comprehensive test scripts demonstrating this across multiple scenarios if needed for further investigation.

Comments:
Title: fix some broken links
State: closed

would be cool to automate finding those broken links as i think there might be many of them @lhoestq @albertvillanova 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7859). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.