NyssaW/github-issues huggingface dataset:
Add lance format support

Add lance format as one of the `packaged_modules`.

```py
import datasets

ds = datasets.load_dataset("org/lance_repo", split="train")

# Or

ds = datasets.load_dataset("./local/data.lance")
```

Comments:
Mentioned https://github.com/huggingface/datasets/issues/7863 as well

@pdames for vis

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7913). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Cool ! I notice the current implementation doesn't support streaming because of the symlink hack.

I believe you can do something like this instead:

```python
def _generate_tables(self, paths: list[str]):
    for path in paths:
        ds = lance.dataset(path)
        for frag_idx, fragment in enumerate(ds.get_fragments()):
            for batch_idx, batch in enumerate(
                fragment.to_batches(columns=self.config.columns, batch_size=self.config.batch_size)
            ):
                table = pa.Table.from_batches([batch])
                table = self._cast_table(table)
                yield Key(frag_idx, batch_idx), table
```

note that path can be a local one, but also a `hf://` URI

@lhoestq Take another look? 

I took the liberty to make a few changes :)

Now I believe we should be good:
- both local and streaming work fine
- both dataset and single files work fine
- all files are properly downloaded now than all files and metadata files are included in config.data_files
- sharding is supported:
  - dataset: one shard = one fragment
  - single files: one shard = one file 
- streaming dataset resuming works fine thanks to Key()
- the two hacks are visible and with TODOs to remove them when possible
  1. remove the revision in HF uris since only "main" is supported
  2. write proper _version/* files since lance doesn't work if they are symlinks

I think this PR is ready, just let me know what you think before we merge 🚀 

The next steps are:
- open a PR in this repository to document Lance support in `datasets`
- open a PR in https://github.com/huggingface/hub-docs to add `pylance` to the list of integrated library on HF, and have some documentation on how to use it with datasets on HF (here is an example [PR](https://github.com/huggingface/hub-docs/pull/1892))
- open a PR in https://github.com/huggingface/huggingface.js to add Lance as a supported dataset library on the HF website (here is an example [PR](https://github.com/huggingface/huggingface.js/pull/1870))

Feel free to start some drafts (I noticed there are great examples in your HF account now !), I'll be happy to review :)

And once Lance is available in huggingface.js and docs are ready we'll be ready to enable the Dataset Viewer and Lance code snippets on HF !
fix low but large example indexerror

Fixes #7911.

This PR simply implements the approach outlined in the corresponding issue, that if we have large examples, the number of shards should never be more than the number of samples. This is an absolute edge case, but can happen for image data.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7912). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Support cast_kwargs in cast_columns

### Feature request

expose `cast(**cast_kwargs)` to `cast_column()`

https://github.com/huggingface/datasets/blob/0feb65dd8733191dd2d1e74215b422fc5939a56a/src/datasets/arrow_dataset.py#L2205

### Motivation

`cast_column()` wraps `cast()` function without exposing any `cast()` args. For large multi-modal datasets, e.g.

```py
# a dataset with list[{"bytes"}: b'', ...], much more than one image
load_dataset("MLLM-CL/VTCBench").cast_column("images", List(Image(decode=False)))
```

This would fail due to #6206, #7167, where the default value `1000` for batch size in `cast()` is too large and causes `pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays`.


https://github.com/huggingface/datasets/blob/0feb65dd8733191dd2d1e74215b422fc5939a56a/src/datasets/arrow_dataset.py#L2164-L2205 

### Your contribution

#7910 

Comments:
Don't save original_shard_lengths by default for backward compat

following #7897 

but let users enable it with `datasets.config.SAVE_ORIGINAL_SHARD_LENGTHS = True`

this is useful for the Dataset Viewer to know where each row comes from after converting to parquet

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7906). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Unbounded network usage when opening Data Studio

### Describe the bug

Opening the Data Studio tab on a dataset page triggers continuous and unbounded network traffic. This issue occurs across multiple browsers and continues even without user interaction.



### Steps to reproduce the bug

https://huggingface.co/datasets/slone/nllb-200-10M-sample/viewer

### Expected behavior

Data Studio should load a limited, finite amount of data and stop further network activity unless explicitly requested by the user.

### Environment info

- OS: Windows 10
- Browsers: Chrome, Firefox, Edge
- Device: Desktop
- Network: Standard broadband connection


Comments:
cc @cfahlgren1

Thanks for reporting! Looking into this!

This should be fixed. Thank you for your patience! 🤗
Request: Review pending neuroimaging PRs (#7886 BIDS loader, #7887 lazy loading)

## Summary

I'm building production neuroimaging pipelines that depend on `datasets` and would benefit greatly from two pending PRs being reviewed/merged.

## Pending PRs

| PR | Description | Status | Open Since |
|----|-------------|--------|------------|
| [#7886](https://github.com/huggingface/datasets/pull/7886) | BIDS dataset loader | Open | Nov 29 |
| [#7887](https://github.com/huggingface/datasets/pull/7887) | Lazy loading for NIfTI | Open | Nov 29 |

## Use Case

The neuroimaging community uses the BIDS (Brain Imaging Data Structure) standard for organizing MRI/fMRI data. These PRs would enable:

1. **#7886**: `load_dataset('bids', data_dir='/path/to/bids')` - Load local BIDS directories directly
2. **#7887**: Memory-efficient NIfTI handling (single 4D fMRI file can be 1-2GB)

## Current Workaround

Without these, users must either:
- Upload to Hub first, then consume (works but slow iteration)
- Hand-roll BIDS parsing (duplicates effort)

## Request

Could a maintainer review these PRs? Happy to address any feedback. The BIDS loader has tests passing and was end-to-end tested with real OpenNeuro data.

Thank you for the great work on `Nifti()` support - these PRs build on that foundation.

## Related

- Contributes to #7804 (Support scientific data formats)
- Built on @TobiasPitters's Nifti feature work

Comments:
Hi ! sure I'll be happy to take a look, sorry for the delay :)
The child process retrieves the dataset directly from the main process instead of executing `memory_mapped_arrow_table_from_file`.

### Feature request

The child process retrieves the dataset directly from the main process instead of executing `memory_mapped_arrow_table_from_file`.

### Motivation

Because my local disk space is insufficient, I can only store a dataset on a remote Ceph server and process it using datasets.
I used the data-juicer[https://github.com/datajuicer/data-juicer] framework as an outer layer which uses datasets, but it doesn't support streaming datasets. I then encountered a problem: for each load, map, and filter operation, I had to wait for a large number of child processes to execute `memory_mapped_arrow_table_from_file`. Since the actual file was on the remote Ceph server, this operation was limited by network I/O.
I don't know if it's a problem with my usage or if this is how datasets are currently designed.However, I think that if the instances obtained after datasets.load_datasets are directly passed to the child process instead of re-executing `memory_mapped_arrow_table_from_file`, it might solve my problem.Or datasets already support this capability, but I just didn't know it？

### Your contribution

。。。

Comments:
Memory mapping is actually the way for processes to share memory efficiently and without copy. It is efficient when on are using a local disk, and it's discouraged to use it on remote disk for the reasons you observed.

What you can do instead is save the dataset as Parquet on your remote storage (or on Hugging Face Datasets which offers fast uploads thanks to Xet), and then your can reload it in streaming mode. Streaming mode is ideal to use a dataset that is hosted in a remote storage
Add inspect_ai eval logs support

Support for .eval log files from inspect_ai

They are actually ZIP files according to the source code at https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/log/_log.py

Unfortunately their format can't be converted to Parquet, so I had to JSON-encode all the nested values

```python
ds = load_dataset("dvilasuero/kimi-bfcl")
```

this will enable the Viewer for datasets like https://huggingface.co/datasets/dvilasuero/kimi-bfcl

original tweet for context: https://x.com/dvilasuero/status/1996936988176343220?s=20

cc @dvsrepo @julien-c @davanstrien 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7899). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Very cool! 

Any reason not to directly use inspect for loading/converting from the binary format to JSON? https://inspect.aisi.org.uk/reference/inspect_ai.log.html#convert_eval_logs 

The format is simple enough to not have to rely on an additional dependency :)
Save input shard lengths

will be useful for the Viewer, to know what (original) shard each row belongs to

cc @cfahlgren1 

next step is use it in Dataset Viewer and expose an API that returns the file containing the row at rowId

(took the opportunity to remove unusued code)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7897). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: use temp files in push_to_hub to prevent OOM on large datasets

## Summary

Fixes memory accumulation in `_push_parquet_shards_to_hub_single` that causes OOM when uploading large datasets with many shards.

**Root cause**: The current implementation stores ALL parquet shard bytes in memory via `BytesIO`, accumulating in the `additions` list. For N shards of ~300MB each, this requires N × 300MB RAM.

**Fix**: Write parquet to temp file instead of `BytesIO`, pass file path to `CommitOperationAdd`. Delete temp file after `preupload_lfs_files` completes (for LFS uploads only - regular uploads need the file until `create_commit`).

## Changes

- Replace `BytesIO` with `tempfile.NamedTemporaryFile` in `_push_parquet_shards_to_hub_single`
- Use file path in `CommitOperationAdd.path_or_fileobj` instead of bytes
- Delete temp file after upload (only for LFS mode - regular uploads keep file for `create_commit`)
- Add `try...finally` for safe cleanup even on errors
- Remove unused `BytesIO` import

## Test plan

- [x] Added `tests/test_push_to_hub_memory.py` with 4 tests:
  - `test_push_to_hub_uses_file_path_not_bytes_in_commit_operation`
  - `test_push_to_hub_cleans_up_temp_files_for_lfs_uploads`
  - `test_push_to_hub_keeps_temp_files_for_regular_uploads`
  - `test_push_to_hub_uploaded_size_still_calculated`
- [x] All tests pass locally
- [x] ruff check passes

## Context

This was discovered while uploading a 270GB neuroimaging dataset (ARC) with 902 shards. The process was killed by OOM after accumulating ~270GB in the `additions` list.

Fixes #7893
Related: #5990, #7400

Comments:
Closing this PR.

After further investigation (prompted by @lhoestq's feedback on #7893), I discovered that `free_memory=True` does work correctly - memory doesn't actually accumulate in the `additions` list as I originally thought.

I tested standard `push_to_hub()` with 902 shards and it processed 625 shards (69%) without OOM issues. The crash we were experiencing was actually **#7894** (`embed_table_storage` crash on `Sequence()` types), not memory accumulation.

I've closed #7893 as invalid. This PR is no longer needed.

Apologies for the noise - lesson learned about testing before filing.
encode nifti correctly when uploading lazily

When trying to upload nifti datasets lazily I got the error:
```python
from pathlib import Path
from datasets import load_dataset

nifti_dir = Path("<local_path>")

dataset = load_dataset(
        "niftifolder",
        data_dir=str(nifti_dir.absolute()),
        streaming=True,
)

dataset.push_to_hub(repo_id="TobiasPitters/test-nifti-papaya-testdata")
```


```python
pyarrow.lib.ArrowInvalid: Could not convert <datasets.features.nifti.Nifti1ImageWrapper object at 0x77633407af90> with type Nifti1ImageWrapper: did not recognize Python value type when inferring an Arrow data type
```

This PR fixes that by encoding the Nifti1ImageWrappers correctly to bytes. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7892). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix(tests): stabilize flaky Hub LFS integration test

## Problem

`test_push_dataset_dict_to_hub_overwrite_files` intermittently fails with:
```
BadRequestError: LFS pointer pointed to a file that does not exist
```

This has been causing the `deps-latest` integration tests to fail on main (visible in recent CI runs). I ran into this while working on the BIDS loader PR and dug into the root cause.

## Root Cause

Two race conditions in the test:

1. **LFS propagation timing** - Rapid successive `push_to_hub` calls don't wait for Hub to fully propagate LFS objects between pushes
2. **Repo name reuse** - The second test scenario reused the same repo name from scenario 1, creating a race between deletion and recreation

## Solution

- Add `_wait_for_repo_ready()` helper that polls `list_repo_files` to ensure the repo is consistent before subsequent operations
- Use a unique repo name (`ds_name_2`) for the second scenario, eliminating the delete/create race entirely

## Testing

All 4 integration test variants now pass:
- ✅ `ubuntu-latest, deps-latest` (was failing)
- ✅ `ubuntu-latest, deps-minimum`
- ✅ `windows-latest, deps-latest` (was failing)
- ✅ `windows-latest, deps-minimum`

Validated on fork: https://github.com/The-Obstacle-Is-The-Way/datasets/pull/4

## Related

- #7600 (push_to_hub concurrency)
- #6392 (push_to_hub connection robustness)

cc @lhoestq - small fix but should help CI reliability

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7889). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Maybe `push_to_hub()` can retry on error 400 when committing to the Hub instead ? This way we make sure `push_to_hub()` works without having to add the waiting step

Thanks for the pointer @lhoestq! Updated to add LFS 400 retry handling directly in `push_to_hub()` across all implementations (Dataset, DatasetDict, IterableDataset, IterableDatasetDict). Reverted the test-side waits.
Add type overloads to load_dataset for better static type inference

Fixes #7883

This PR adds @overload decorators to load_dataset() to help type checkers like Pylance and mypy correctly infer the return type based on the split and streaming parameters.

Changes:
- Added typing imports (Literal, overload) to load.py
- Added 4 @overload signatures that map argument combinations to specific return types:
  * split=None, streaming=False -> DatasetDict
  * split specified, streaming=False -> Dataset
  * split=None, streaming=True -> IterableDatasetDict
  * split specified, streaming=True -> IterableDataset

This resolves the Pylance error where to_csv() was not recognized on Dataset objects returned by load_dataset(..., split='train'), since the type checker previously saw the return type as a Union that included types without to_csv().

No runtime behavior changes - this is purely a static typing improvement.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7888). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix(nifti): enable lazy loading for Nifti1ImageWrapper

## Summary
- **Single-line fix**: Change `dataobj=nifti_image.get_fdata()` → `dataobj=nifti_image.dataobj`
- Preserves nibabel's `ArrayProxy` for true lazy loading instead of eagerly loading entire NIfTI files into memory
- Improves error handling: corrupted files now fail at access time with clear context instead of silently during decode

## Problem
The `Nifti1ImageWrapper.__init__` was calling `get_fdata()` which immediately loads the entire image into memory. For large 4D fMRI files (often 1-2GB), this causes:
1. **Memory issues** - Full data loaded during decode, not on demand
2. **Poor error handling** - Corrupted files crash at access time with unclear error messages (e.g., `EOFError` with no file path)
3. **No graceful recovery** - Entire dataset iteration fails on one bad file

## Solution
Use `nifti_image.dataobj` which preserves the underlying `ArrayProxy`, deferring actual I/O to `get_fdata()` calls.

## Test Plan
- [x] Added `test_nifti_lazy_loading` to verify `ArrayProxy` is preserved
- [x] All 22 existing NIfTI tests pass
- [x] End-to-end tested with real OpenNeuro data (ds000102)
- [x] CodeRabbit approved: "Switch to dataobj correctly restores nibabel's lazy loading semantics... This looks solid"

## Related
- Discovered while testing BIDS loader PR #7886
- Complements the NIfTI + NiiVue viewer work from #7878 and #7874

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7887). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
feat(bids): Add BIDS dataset loader for neuroimaging data

## Summary

Adds native BIDS (Brain Imaging Data Structure) dataset loading support using PyBIDS, enabling `load_dataset('bids', data_dir='/path/to/bids')` workflow for neuroimaging researchers.

**Contributes to #7804** (Support scientific data formats) - BIDS is a widely-used standard for organizing neuroimaging data built on NIfTI files.

## Changes

### Core Implementation
- `src/datasets/packaged_modules/bids/bids.py` - GeneratorBasedBuilder implementation
- `src/datasets/packaged_modules/bids/__init__.py` - Module exports
- `src/datasets/packaged_modules/__init__.py` - Registration with module registry
- `src/datasets/config.py` - `PYBIDS_AVAILABLE` config flag
- `setup.py` - Optional `pybids>=0.21.0` + nibabel dependency

### Features
- Automatic BIDS structure validation
- Subject/session/datatype filtering via config
- JSON sidecar metadata extraction
- NIfTI file decoding via existing Nifti feature

### Documentation & Tests
- `docs/source/bids_dataset.mdx` - User guide with examples
- `tests/packaged_modules/test_bids.py` - Unit tests (4 tests)

## Usage

```python
from datasets import load_dataset

# Load entire BIDS dataset
ds = load_dataset('bids', data_dir='/path/to/bids_dataset')

# Filter by subject/session
ds = load_dataset('bids', 
    data_dir='/path/to/bids_dataset',
    subjects=['01', '02'],
    sessions=['baseline']
)

# Access samples
sample = ds['train'][0]
print(sample['subject'])      # '01'
print(sample['nifti'].shape)  # (176, 256, 256)
print(sample['metadata'])     # JSON sidecar data
```

## Test plan

- [x] All 4 unit tests pass (`pytest tests/packaged_modules/test_bids.py`)
- [x] `make quality` passes (ruff check)
- [x] End-to-end tested with real OpenNeuro data (ds000102)

## Context

This PR is part of the neuroimaging initiative discussed with @TobiasPitters. Follows the BIDS 1.10.1 specification and leverages the existing Nifti feature for NIfTI file handling.

Related PRs:
- #7874 (Nifti visualization support)
- #7878 (Replace papaya with niivue)

Comments:
Fix 7846: add_column and add_item erroneously(?) require new_fingerprint parameter 

Summary of the change:

Made new_fingerprint optional (Optional[str] = None) in Dataset.add_column and Dataset.add_item
Added a simple test to verify both methods work without providing a fingerprint

Why this change is safe:

The Dataset constructor already handles fingerprint=None by generating a new fingerprint automatically
No internal logic is broken — if a user provides a fingerprint, it’s still used as before
The change only affects the function signature, making it more user-friendly without changing any functionality

Comments:
@lhoestq can you help review this pull request?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7884). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Data.to_csv() cannot be recognized by pylance

### Describe the bug

Hi, everyone ! I am a beginner with datasets. 
I am testing reading multiple CSV files from a zip archive. The result of reading the dataset shows success, and it can ultimately be correctly saved to CSV.

Intermediate results:
```
Generating train split: 62973 examples [00:00, 175939.01 examples/s]
DatasetDict({
    train: Dataset({
        features: ['交易时间\t', '收支方向\t', '业务（产品）种类\t', '交易金额\t', '币种\t', '时点余额\t', '对手方名称\t', '对方机构名称\t', ' 对方钱包ID/账号\t', '交易对手名称\t', '交易对手编号\t', '交易流水号\t', '摘要\t', '附言\t', '备注\t', '用途\t', '客户流水号\t'],
        num_rows: 62973
    })
})
```
However, Pylance gives me the following error:
```
Cannot access attribute "to_csv" for class "DatasetDict"
Attribute "to_csv" is unknownPylance[reportAttributeAccessIssue](https://github.com/microsoft/pylance-release/blob/main/docs/diagnostics/reportAttributeAccessIssue.md)```

Cannot access attribute "to_csv" for class "IterableDatasetDict"
  Attribute "to_csv" is unknownPylance[reportAttributeAccessIssue](https://github.com/microsoft/pylance-release/blob/main/docs/diagnostics/reportAttributeAccessIssue.md)

(method) to_csv: Unknown | ((path_or_buf: datasets.utils.typing.PathLike | BinaryIO, batch_size: int | None = None, num_proc: int | None = None, storage_options: dict[Unknown, Unknown] | None = None, **to_csv_kwargs: Unknown) -> int) | ((path_or_buf: datasets.utils.typing.PathLike | BinaryIO, batch_size: int | None = None, storage_options: dict[Unknown, Unknown] | None = None, **to_csv_kwargs: Unknown) -> int)
```
I ignored the error and continued executing to get the correct result:
```
Dataset({
    features: ['交易时间\t', '收支方向\t', '业务（产品）种类\t', '交易金额\t', '币种\t', '时点余额\t', '对手方名称\t', '对方机构名称\t', '对方 钱包ID/账号\t', '交易对手名称\t', '交易对手编号\t', '交易流水号\t', '摘要\t', '附言\t', '备注\t', '用途\t', '客户流水号\t'],
    num_rows: 62973
})
```
Since the data volume is small, I manually merged the CSV files, and the final result is consistent with what the program saved.


looks like :

<img width="1264" height="150" alt="Image" src="https://github.com/user-attachments/assets/743540d7-ad8c-4531-ae7e-de71a5243a32" />


### Steps to reproduce the bug

this is my code.
```
from datasets import load_dataset

def main():
    url = "data/test.zip"
    data_files = {"train": url}
    dataset = load_dataset("csv", data_files=data_files,split="train", encoding="gbk", skiprows=2)
    # print(dataset)
    dataset.to_csv("data/test.csv")


if __name__ == "__main__":
    main()
```

### Expected behavior

I want to know why this happens. Is there something wrong with my code?

### Environment info

OS: Windows 11    **upgrade from** OS: Windows_NT x64 10.0.22631
Editor: 
VS Code  Version: 1.106.2 (user setup)
"datasets"    version = "4.4.1"


Comments:
Inconsistent loading of LFS-hosted files in epfml/FineWeb-HQ dataset

### Describe the bug

Some files in the `epfml/FineWeb-HQ` dataset fail to load via the Hugging Face `datasets` library.  

- xet-hosted  files load fine
- LFS-hosted files sometimes fail

Example:  
- Fails: https://huggingface.co/datasets/epfml/FineWeb-HQ/blob/main/data/CC-MAIN-2024-26/000_00003.parquet  
- Works: https://huggingface.co/datasets/epfml/FineWeb-HQ/blob/main/data/CC-MAIN-2024-42/000_00027.parquet  

Discussion: https://huggingface.co/datasets/epfml/FineWeb-HQ/discussions/2  

### Steps to reproduce the bug

```python
from datasets import load_dataset

ds = load_dataset(
    "epfml/FineWeb-HQ",
    data_files="data/CC-MAIN-2024-26/000_00003.parquet",
)
```

Error message:
```
HfHubHTTPError: 403 Forbidden: None.
Cannot access content at: https://cdn-lfs-us-1.hf.co/repos/...
Make sure your token has the correct permissions.
...
<Error><Code>AccessDenied</Code><Message>Access Denied</Message></Error>
```

### Expected behavior

It should load the dataset for all files. 

### Environment info

- python 3.10
- datasets 4.4.1

Comments:
we are investigating the issue, I'll post updates here

It should be all good now, sorry for the inconvenience.

We found an error that happened during the migration of this repository from Git LFS to Xet and fixed the issue
Spurious label column created when audiofolder/imagefolder directories match split names

## Describe the bug

When using `audiofolder` or `imagefolder` with directories for **splits** (train/test) rather than class labels, a spurious `label` column is incorrectly created.

**Example:** https://huggingface.co/datasets/datasets-examples/doc-audio-4
```
from datasets import load_dataset
ds = load_dataset("datasets-examples/doc-audio-4")
print(ds["train"].features)
```
Shows 'label' column with ClassLabel(names=['test', 'train']) - incorrect!## Root cause

In `folder_based_builder.py`, the `labels` set is accumulated across ALL splits (line 77). When directories are `train/` and `test/`:
- `labels = {"train", "test"}` → `len(labels) > 1` → `add_labels = True`
- Spurious label column is created with split names as class labels

## Expected behavior

No `label` column should be added when directory names match split names.

## Proposed fix

Skip label inference when inferred labels match split names.

cc @lhoestq 

Comments:
work around `tempfile` silently ignoring `TMPDIR` if the dir doesn't exist

This should help a lot of users running into `No space left on device` while using `datasets`. Normally the issue is is that `/tmp` is too small and the user needs to use another path, which they would normally set as `export TMPDIR=/some/big/storage`

However, the `tempfile` facility that `datasets` and `pyarrow` use is somewhat broken. If the path doesn't exist it'd ignore it and fall back to using `/tmp`. Watch this:

```
$ export TMPDIR='/tmp/username' 

$ python -c "\
import os
import tempfile
print(os.environ['TMPDIR'])
print(tempfile.gettempdir())"
/tmp/username
/tmp
```

Now let's ensure the path exists:
```
$ export TMPDIR='/tmp/username' 
$ mkdir -p $TMPDIR
$ python -c "\
import os
import tempfile
print(os.environ['TMPDIR'])
print(tempfile.gettempdir())"
/tmp/username
/tmp/username
```

So I recommend `datasets` do either of the 2:
1. assert if `$TMPDIR` dir doesn't exist, telling the user to create it
2. auto-create it

The reason for (1) is that I don't know why `tempdir` doesn't auto-create the dir - perhaps some security implication? I will let you guys make the decision, but the key is not to let things silently fall through and the user puzzling why no matter what they do they can't break past `No space left on device` while using `datasets`

Thank you.

I found this via https://stackoverflow.com/questions/37229398/python-tempfile-gettempdir-does-not-respect-tmpdir while trying to help a colleague to solve this exact issue.

Comments:
Hi! Just created a Pull Request (#7890) to try to fix this using your suggestions. I hope it helps!
test: add verification for HuggingFaceM4/InterleavedWebDocuments

Adds an integration test for the `HuggingFaceM4/InterleavedWebDocuments` dataset.

- Gracefully skips if the dataset is not yet available on the Hub
- Checks basic loading and structure once it becomes available

Closes #7394

First-time contributor to `datasets` — really excited about this! Happy to make any adjustments needed. 🙂

Comments:
Nifti visualization support

closes #7870 

leverage Papaya to visualize nifti images. For this I created a Wrapper class for `nibabel.nifti1.Nifti1Image` that provides the same interface but exposes an additional `_repr_html_` method, which is needed to visualize the image in jupyter (didn't test in colab, but that should work equivalently).

Code to test (execute in a notebook):
```python
from datasets import load_dataset

ds = load_dataset("TobiasPitters/nifti-nitest-extracted",
              split="train")
image = ds[1]

image
```

Here a small video, not the most exciting scan though:
https://github.com/user-attachments/assets/1cca5f01-6fd2-48ef-a4d7-a92c1259c224

Am open to good ways to test this.

EDIT: papaya also supports dicom, didn't test it yet though

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7874). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I tested in Colab and it works perfectly :) now I want to add `_repr_html_` everywhere xD

Re: testing, I think it's fine to test manually such features
Visualization for Medical Imaging Datasets

This is a followup to: https://github.com/huggingface/datasets/pull/7815. 

I checked the possibilities to visualize the nifti (and potentially dicom), and here's what I found:
 - https://github.com/aces/brainbrowser, AGPL3 license, last commit 3 months ago, latest (github) release from 2017. It's available on jsdelivr: https://www.jsdelivr.com/package/npm/brainbrowser (but that is from 2015!)
 - https://github.com/rii-mango/Papaya, custom but BSD-style license that would require datasets to list the conditions in their readme somewhere, last commit June 2024. I looked into this library and it looks mature and good enough for our use case, but just working on it for a short time I wasn't able to get this to work, but am sure we could get this working, would probably require some JS on datasets' end. Available on jsdelivr as well: https://www.jsdelivr.com/package/npm/papaya-viewer. Seems like it's frequently loaded.
- https://github.com/hanayik/niivue, BSD3 license, last commit May 26, 2021. Archived. Doesn't look like an option.

I think  the only real option for us Papaya, but there is also the risk that we'll end up with an unmaintained package after a while, since development seems to be slow or even halted. 

I think conceptually we would need to figure out how we can build a good solution for visualizing Medical Image data. On shap, we have a separate javascript folder in which we render visualizations, this could be a blueprint but will require a bundler, etc. Alternatively one could go with a naive approach to just write some html code in a python string and load the package via jsdelivr.


@lhoestq thoughts?

Comments:
It would be amazing to be able to show the Papaya UI in google colab / jupyter notebook. IIRC both allow serving javascript via nbextensions that we can surely use in HTML() objects.

Alternatively we could also start with a simple approach and dump the medical image data as a video file that goes through the slices, so we don't need javascript.
Performance Issue: save_to_disk() 200-1200% slower due to unconditional flatten_indices()

## 🐛 Bug Description

The `save_to_disk()` method unconditionally calls `flatten_indices()` when `_indices` is not None, causing severe performance degradation for datasets processed with filtering, shuffling, or multiprocessed mapping operations.

**Root cause**: This line rebuilds the entire dataset unnecessarily:
```python
dataset = self.flatten_indices() if self._indices is not None else self
```

## 📊 Performance Impact

| Dataset Size | Operation | Save Time | Slowdown |
|-------------|-----------|-----------|----------|
| 100K | Baseline (no indices) | 0.027s | - |
| 100K | Filtered (with indices) | 0.146s | **+431%** |
| 100K | Shuffled (with indices) | 0.332s | **+1107%** |
| 250K | Shuffled (with indices) | 0.849s | **+1202%** |

## 🔄 Reproduction

```python
from datasets import Dataset
import time

# Create dataset
dataset = Dataset.from_dict({'text': [f'sample {i}' for i in range(100000)]})

# Baseline save (no indices)
start = time.time()
dataset.save_to_disk('baseline')
baseline_time = time.time() - start

# Filtered save (creates indices)
filtered = dataset.filter(lambda x: True)
start = time.time()
filtered.save_to_disk('filtered')
filtered_time = time.time() - start

print(f"Baseline: {baseline_time:.3f}s")
print(f"Filtered: {filtered_time:.3f}s") 
print(f"Slowdown: {(filtered_time/baseline_time-1)*100:.1f}%")
```

**Expected output**: Filtered dataset is 400-1000% slower than baseline

## 💡 Proposed Solution

Add optional parameter to control flattening:

```python
def save_to_disk(self, dataset_path, flatten_indices=True):
    dataset = self.flatten_indices() if (self._indices is not None and flatten_indices) else self
    # ... rest of save logic
```

**Benefits**:
- ✅ Immediate performance improvement for users who don't need flattening
- ✅ Backwards compatible (default behavior unchanged)  
- ✅ Simple implementation

## 🌍 Environment

- **datasets version**: 2.x
- **Python**: 3.10+
- **OS**: Linux/macOS/Windows

## 📈 Impact

This affects **most ML preprocessing workflows** that filter/shuffle datasets before saving. Performance degradation scales exponentially with dataset size, making it a critical bottleneck for production systems.

## 🔗 Additional Resources

We have comprehensive test scripts demonstrating this across multiple scenarios if needed for further investigation.

Comments:
Support downloading specific splits in `load_dataset`

This is PR builds on top of #7706 to revive the unfinished #6832 but isn't just cleaning up things, here are some important changes:
 - `download_mode="FORCE_REDOWNLOAD"` is interpreted as always creating a clean slate, that means that even if we already did:
 ```python
load_dataset("<name>")
load_dataset("<name>", split="train", download_mode="force_redownload")
```
This makes sure that only the train dataset is available after executing both. This was different in the original PR, which proposed that train and test would be available. 
- `download_mode="REUSE_DATASET_IF_EXISTS"` is interpreted as only ever adding new data, never redownloading OR deleting other splits. This was different in the original PR, where  
 ```python
load_dataset("<name>", split="test")
load_dataset("<name>", split="train")
```
resulted in only the train data being available, which I deem very unintuitive and probably not what users want. Also I argue that this is just the first step to a more user friendly partial loading when specifying percentages (or maybe even single instances) via the ReadInstructions, and then doing
 ```python
load_dataset("<name>", split="test[:10%]")
load_dataset("<name>", split="test[10%:]")
```
should result IMO in the whole dataset being cached locally without redownloads.

Furthermore this PR fixes a couple issues with the previous PR, e.g. a [missing comparison](https://github.com/huggingface/datasets/pull/7706/files#diff-f933ce41f71c6c0d1ce658e27de62cbe0b45d777e9e68056dd012ac3eb9324f7R877) and adding tests for the proposed changes in behaviour, which would both fail on @ArjunJagdale's original PR. 

Todo:
 - [ ] update docs?

Future outlook (just my opinions and up for debate):
As mentioned before, I would see this as just a step towards the feature of partial percentage loading (though how the API should behave in that case is not entirely clear for me now) and maybe we could introduce another `download_mode="FORCE_REDOWNLOAD_SPLIT"`, which makes sure that even if a split is specified, only the referenced split is redownloaded and everything else left unchanged, this would then allow users more granular control over what they want to redownload. 
@lhoestq very curious to get your opinion on this.

Comments:
@CloseChoice This looks great! You're absolutely right about the missing comparison - that's a critical bug I missed. 

@lhoestq just resolved the conflicts inflicted by the latest changes. Thought it might be good giving this a shot now before more changes mess with this PR.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7858). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Missing transcript column when loading a local dataset with "audiofolder"

### Describe the bug

My local dataset is not properly loaded when using `load_dataset("audiofolder", data_dir="my_dataset")` with a `jsonl` metadata file.
Only the `audio` column is read while the `transcript` column is not.

The last tested `datasets` version where the behavior was still correct is 2.18.0.

### Steps to reproduce the bug

Dataset directory structure:
```
my_dataset/
  - data/
    - test/
      - 54db8760de3cfbff3c8a36a36b4d0f77_00390.0_04583.0.mp3
      - 54db8760de3cfbff3c8a36a36b4d0f77_04583.0_05730.0.mp3
      - ...
  - metadata.jsonl
```

`metadata.jsonl` file content:
```
{"file_name": "data/test/54db8760de3cfbff3c8a36a36b4d0f77_00390.0_04583.0.mp3", "transcript": "Ata tudoù penaos e tro ar bed ?"}
{"file_name": "data/test/54db8760de3cfbff3c8a36a36b4d0f77_04583.0_05730.0.mp3", "transcript": "Ur gwir blijadur eo adkavout ac'hanoc'h hiziv."}
...
```
```python3
my_dataset = load_dataset("audiofolder", data_dir="my_dataset")

print(my_dataset)

'''
DatasetDict({
    test: Dataset({
        features: ['audio'],
        num_rows: 347
    })
})
'''

print(my_dataset['test'][0])

'''
{'audio': <datasets.features._torchcodec.AudioDecoder object at 0x75ffcd172510>}
'''
```

### Expected behavior

Being able to access the `transcript` column in the loaded dataset.

### Environment info

- `datasets` version: 4.4.1
- Platform: Linux-6.5.0-45-generic-x86_64-with-glibc2.39
- Python version: 3.13.9
- `huggingface_hub` version: 1.1.2
- PyArrow version: 22.0.0
- Pandas version: 2.3.3
- `fsspec` version: 2025.10.0

Note: same issue with `datasets` v3.6.0

Comments:
First bad commit 5c8869f8c36dbc8c8d423030b7b7c4fd64f8c729

EDIT: This is not a bug or a regression. It was a breaking change introduced in the commit I mentioned and was also documented in there. The docs state how to handle this now, see https://huggingface.co/docs/datasets/main/en/audio_load#audiofolder-with-metadata

or simply, move your metadata into the splits folder and update the paths, in your case this would look like this:
```bash
my_dataset/
  - data/
    - test/
      - 54db8760de3cfbff3c8a36a36b4d0f77_00390.0_04583.0.mp3
      - 54db8760de3cfbff3c8a36a36b4d0f77_04583.0_05730.0.mp3
      - metadata.jsonl
```

and the pahts in the jsonl should be relative to the metadata.json:
```bash
{"file_name": "54db8760de3cfbff3c8a36a36b4d0f77_00390.0_04583.0.mp3", "transcript": "Ata tudoù penaos e tro ar bed ?"}
{"file_name": "54db8760de3cfbff3c8a36a36b4d0f77_04583.0_05730.0.mp3", "transcript": "Ur gwir blijadur eo adkavout ac'hanoc'h hiziv."}
...
```

So I think this can be closed.

Thank you for your quick answer !
I'm sorry I missed that in the documentation.
Everything works fine again after following your recommendations.
I'm closing the issue.
[Distributed] split_dataset_by_node() gives the same number of examples for each node

this works:

```python
import torch.distributed as dist
from datasets import IterableDataset
from datasets.distributed import split_dataset_by_node
from collections import Counter

def g(shards):
    for shard in shards:
        # shards don't have the same length
        num_examples = 3 + shard 
        for i in range(num_examples):
            yield {"shard": f"{shard=}", "i": i}

if __name__ == "__main__":
    dist.init_process_group(backend="gloo")
    rank, world_size = dist.get_rank(), dist.get_world_size()
    num_shards = 6
    ds = IterableDataset.from_generator(g, gen_kwargs={"shards": list(range(num_shards))})
    ds = split_dataset_by_node(ds, rank=rank, world_size=world_size)
    # Check that each rank has the same number of examples
    # and show the number of examples per shard and per rank
    counter = Counter(ds["shard"])
    print(f"# {rank=}\ttotal={counter.total()}\t{counter}", flush=True)

    # torchrun --nproc_per_node 2 script.py
    # rank=0        total=16        Counter({'shard=4': 7, 'shard=2': 5, 'shard=0': 4})
    # rank=1        total=16        Counter({'shard=3': 6, 'shard=5': 6, 'shard=1': 4})

```

TODO: make it work with DataLoader (communicate with main process to know when the node runs out of data ?)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7854). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Making this work with multiple workers could create a lot of communication for not a lot of benefits, considering you can simply use `Join()` to let nodes shutdown when they run out of data while the other nodes continue training: https://docs.pytorch.org/docs/stable/distributed.algorithms.join.html
Problems with NifTI

### Describe the bug

There are currently 2 problems with the new NifTI feature:
1. dealing with zipped files, this is mentioned and explained [here](https://github.com/huggingface/datasets/pull/7815#issuecomment-3496199503)
2. when uploading via the `niftifolder` feature, the resulting parquet only contains relative paths to the nifti files:

```bash
table['nifti']
<pyarrow.lib.ChunkedArray object at 0x798245d37d60>
[
  -- is_valid: all not null
  -- child 0 type: binary
    [
      null,
      null,
      null,
      null,
      null,
      null
    ]
  -- child 1 type: string
    [
      "/home/tobias/programming/github/datasets/nifti_extracted/T1.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2-interleaved.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2_-interleaved.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2_.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/fieldmap.nii"
    ]
]
```
instead of containing bytes. The code is copy pasted from PDF, so I wonder what is going wrong here.

### Steps to reproduce the bug

see the linked comment

### Expected behavior

downloading should work as smoothly as for pdf

### Environment info

- `datasets` version: 4.4.2.dev0
- Platform: Linux-6.14.0-33-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.35.3
- PyArrow version: 21.0.0
- Pandas version: 2.3.3
- `fsspec` version: 2025.9.0


Comments:
> 2. when uploading via the niftifolder feature, the resulting parquet only contains relative paths to the nifti files:

what did you use to upload the dataset ? iirc push_to_hub() does upload the bytes as well, but to_parquet() doesn't

> > 2. when uploading via the niftifolder feature, the resulting parquet only contains relative paths to the nifti files:
> 
> what did you use to upload the dataset ? iirc push_to_hub() does upload the bytes as well, but to_parquet() doesn't

I used `push_to_hub` but the problem is that the nifti feature does not have an `embed_storage` function
Add fasta support

This PR adds support for FASTA files conversion to Parquet. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7851). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

A few comments:

- Have you tried using this with longer sequences? @UriNeri developed something similar internally and used it with viral genomes. He got some Parquet errors due to genomes not fitting in a `utf8` column. This was fixed by using `large_utf8`.
- If you're only using it to read FASTA files, I think that having BioPython as a dependency is overkill. The library is very large and the FASTA parser isn't particularly fast. I have an example of a fast parser with no external references [here](https://gist.github.com/apcamargo/d039aa04a2cbbcbb14e2d34a0963b862) (this is actually based on [`readfq.py`](https://github.com/lh3/readfq/blob/master/readfq.py), with a couple of extra functions that might not be useful in the context of this PR)

> * If you're only using it to read FASTA files, I think that having BioPython as a dependency is overkill. The library is very large and the FASTA parser isn't particularly fast. I have an example of a fast parser with no external references [here](https://gist.github.com/apcamargo/d039aa04a2cbbcbb14e2d34a0963b862) (this is actually based on [`readfq.py`](https://github.com/lh3/readfq/blob/master/readfq.py), with a couple of extra functions that might not be useful in the context of this PR)

What @apcamargo said, plus FWIW in **our approach** (so might not be relevant here) we use polars (with custom fasta io parser) or polars-bio (that has a `scan_fasta` function) and we foudn out that the page size sometimes need to be adjusted:
```
envs/default/lib/python3.9/site-packages/polars/lazyframe/frame.py:2422, in LazyFrame.collect(self, type_coercion, predicate_pushdown, projection_pushdown, simplify_expression, slice_pushdown, comm_subplan_elim, comm_subexpr_elim, cluster_with_columns, collapse_joins, no_optimization, engine, background, optimizations, **_kwargs)
   2420 # Only for testing purposes
   2421 callback = _kwargs.get("post_opt_callback", callback)
-> 2422 return wrap_df(ldf.collect(engine, callback))
ComputeError: parquet: File out of specification: A page can only contain i32::MAX uncompressed bytes. This one contains 4544943557
```

Which in polars can be solved with:
```
df.write_parquet(
    "test1.patquet",
   compression="zstd",
    row_group_size=10_000,  #  smaller row groups
     data_page_size=1024*1024  # 1MB page size
)
```
Transform with columns parameter triggers on non-specified column access

### Describe the bug

Iterating over a [`Column`](https://github.com/huggingface/datasets/blob/8b1bd4ec1cc9e9ce022f749abb6485ef984ae7c0/src/datasets/arrow_dataset.py#L633-L692) iterates through the parent [`Dataset`](https://github.com/huggingface/datasets/blob/8b1bd4ec1cc9e9ce022f749abb6485ef984ae7c0/src/datasets/arrow_dataset.py#L695) and applies all formatting/transforms on each row, regardless of which column is being accessed. This causes an error when transforms depend on columns not present in the projection.

### Steps to reproduce the bug

### Load a dataset with multiple columns

```python
ds = load_dataset("mrbrobot/isic-2024", split="train")
```

### Define a transform that specifies an input column

```python
def image_transform(batch):
    batch["image"] = batch["image"]  # KeyError when batch doesn't contain "image"
    return batch

# apply transform only to image column
ds = ds.with_format("torch")
ds = ds.with_transform(image_transform, columns=["image"], output_all_columns=True)
```

### Iterate over non-specified column

```python
# iterate over a different column, triggers the transform on each row, but batch doesn't contain "image"
for t in ds["target"]:  # KeyError: 'image'
    print(t)
```

### Expected behavior

If a user iterates over `ds["target"]` and the transform specifies `columns=["image"]`, the transform should be skipped.

### Environment info

`datasets`: 4.2.0
Python: 3.12.12
Linux: Debian 11.11

Comments: