Minimartzz/github-issues huggingface dataset:
Request: Review pending neuroimaging PRs (#7886 BIDS loader, #7887 lazy loading)
State: open

## Summary

I'm building production neuroimaging pipelines that depend on `datasets` and would benefit greatly from two pending PRs being reviewed/merged.

## Pending PRs

| PR | Description | Status | Open Since |
|----|-------------|--------|------------|
| [#7886](https://github.com/huggingface/datasets/pull/7886) | BIDS dataset loader | Open | Nov 29 |
| [#7887](https://github.com/huggingface/datasets/pull/7887) | Lazy loading for NIfTI | Open | Nov 29 |

## Use Case

The neuroimaging community uses the BIDS (Brain Imaging Data Structure) standard for organizing MRI/fMRI data. These PRs would enable:

1. **#7886**: `load_dataset('bids', data_dir='/path/to/bids')` - Load local BIDS directories directly
2. **#7887**: Memory-efficient NIfTI handling (single 4D fMRI file can be 1-2GB)

## Current Workaround

Without these, users must either:
- Upload to Hub first, then consume (works but slow iteration)
- Hand-roll BIDS parsing (duplicates effort)

## Request

Could a maintainer review these PRs? Happy to address any feedback. The BIDS loader has tests passing and was end-to-end tested with real OpenNeuro data.

Thank you for the great work on `Nifti()` support - these PRs build on that foundation.

## Related

- Contributes to #7804 (Support scientific data formats)
- Built on @TobiasPitters's Nifti feature work

Comments:
The child process retrieves the dataset directly from the main process instead of executing `memory_mapped_arrow_table_from_file`.
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

The child process retrieves the dataset directly from the main process instead of executing `memory_mapped_arrow_table_from_file`.

### Motivation

Because my local disk space is insufficient, I can only store a dataset on a remote Ceph server and process it using datasets.
I used the data-juicer[https://github.com/datajuicer/data-juicer] framework as an outer layer which uses datasets, but it doesn't support streaming datasets. I then encountered a problem: for each load, map, and filter operation, I had to wait for a large number of child processes to execute `memory_mapped_arrow_table_from_file`. Since the actual file was on the remote Ceph server, this operation was limited by network I/O.
I don't know if it's a problem with my usage or if this is how datasets are currently designed.However, I think that if the instances obtained after datasets.load_datasets are directly passed to the child process instead of re-executing `memory_mapped_arrow_table_from_file`, it might solve my problem.Or datasets already support this capability, but I just didn't know it？

### Your contribution

。。。

Comments:
ShuffledDataSourcesArrowExamplesIterable cannot properly resume from checkpoint
State: open

### Describe the bug

ShuffledDataSourcesArrowExamplesIterable cannot properly resume from checkpoint

### Steps to reproduce the bug

1. The reproducible code is as follows：

```
from datasets import Dataset, concatenate_datasets, interleave_datasets
ds = Dataset.from_dict({"a": range(12)}).to_iterable_dataset(num_shards=1)
ds = ds.shuffle(seed=42)

for idx, example in enumerate(ds):
    print(example)
    if idx == 2: #The checkpoint can be loaded correctly only when idx <= 1.
        state_dict = ds.state_dict()
        print("checkpoint")
        break

print("state_dict: ",state_dict)

ds.load_state_dict(state_dict)
print(f"restart from checkpoint")
for example in ds:
    print(example)
```
2. The error message is as follows:
```
{'a': 0}
{'a': 7}
{'a': 6}
checkpoint
state_dict:  {'examples_iterable': {'examples_iterable': {'examples_iterable': {'shard_idx': 1, 'shard_example_idx': 0, 'type': 'ShuffledDataSourcesArrowExamplesIterable'}, 'previous_state': {'shard_idx': 0, 'shard_example_idx': 0, 'type': 'ShuffledDataSourcesArrowExamplesIterable'}, 'batch_idx': 12, 'num_chunks_since_previous_state': 12, 'cropped_chunk_length': 0, 'type': 'RebatchedArrowExamplesIterable'}, 'previous_state': {'examples_iterable': {'shard_idx': 1, 'shard_example_idx': 0, 'type': 'ShuffledDataSourcesArrowExamplesIterable'}, 'previous_state': {'shard_idx': 0, 'shard_example_idx': 0, 'type': 'ShuffledDataSourcesArrowExamplesIterable'}, 'batch_idx': 12, 'num_chunks_since_previous_state': 12, 'cropped_chunk_length': 0, 'type': 'RebatchedArrowExamplesIterable'}, 'batch_idx': 3, 'num_chunks_since_previous_state': 2, 'cropped_chunk_length': 0, 'type': 'RebatchedArrowExamplesIterable'}, 'epoch': 0}
restart from checkpoint
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
```
### Expected behavior

I want a correct resume from any checkpoint, but currently the checkpoint can only be loaded correctly when idx <= 1.

### Environment info

datasets Version: 4.4.1

@lhoestq 

Comments:
Add inspect_ai eval logs support
State: closed

Support for .eval log files from inspect_ai

They are actually ZIP files according to the source code at https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/log/_log.py

Unfortunately their format can't be converted to Parquet, so I had to JSON-encode all the nested values

```python
ds = load_dataset("dvilasuero/kimi-bfcl")
```

this will enable the Viewer for datasets like https://huggingface.co/datasets/dvilasuero/kimi-bfcl

original tweet for context: https://x.com/dvilasuero/status/1996936988176343220?s=20

cc @dvsrepo @julien-c @davanstrien 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7899). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Very cool! 

Any reason not to directly use inspect for loading/converting from the binary format to JSON? https://inspect.aisi.org.uk/reference/inspect_ai.log.html#convert_eval_logs 

The format is simple enough to not have to rely on an additional dependency :)
Save input shard lengths
State: closed

will be useful for the Viewer, to know what (original) shard each row belongs to

cc @cfahlgren1 

next step is use it in Dataset Viewer and expose an API that returns the file containing the row at rowId

(took the opportunity to remove unusued code)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7897). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: force contiguous copy for sliced list arrays in embed_array_storage
State: open

## Summary

Fixes SIGKILL crash in `embed_array_storage` when processing sliced/sharded datasets with nested types like `Sequence(Nifti())` or `Sequence(Image())`.

**Root cause**: When `ds.shard()` or `ds.select()` creates a sliced view, `array.values` on a sliced `ListArray` returns values with internal offset references. For nested types, PyArrow's C++ layer can crash (SIGKILL, exit code 137) when materializing these sliced nested structs.

**Fix**: Force a contiguous copy via `pa.concat_arrays([array])` when the array has a non-zero offset before processing list/large_list arrays.

## Changes

- Add offset check in `embed_array_storage` for list/large_list arrays
- Force contiguous copy when `array.offset > 0` to break internal references
- Add regression tests for sliced arrays with Image, Nifti, and LargeList types

## Test plan

- [x] Added `tests/features/test_embed_storage_sliced.py` with 3 tests:
  - `test_embed_array_storage_sliced_list_image`
  - `test_embed_array_storage_sliced_list_nifti`
  - `test_embed_array_storage_sliced_large_list`
- [x] All tests verify `embedded.offset == 0` (contiguous result)
- [x] All tests pass locally
- [x] ruff check passes

## Context

This was discovered while uploading a 270GB neuroimaging dataset (ARC) with `Sequence(Nifti())` columns. The process crashed with SIGKILL (no Python traceback) when `embed_table_storage` was called on sharded data.

Workaround that confirmed the fix: pandas round-trip (`shard.to_pandas()` → `Dataset.from_pandas()`) which forces a contiguous copy.

Fixes #7894
Related: #6686, #7852, #6790

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7896). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: use temp files in push_to_hub to prevent OOM on large datasets
State: closed

## Summary

Fixes memory accumulation in `_push_parquet_shards_to_hub_single` that causes OOM when uploading large datasets with many shards.

**Root cause**: The current implementation stores ALL parquet shard bytes in memory via `BytesIO`, accumulating in the `additions` list. For N shards of ~300MB each, this requires N × 300MB RAM.

**Fix**: Write parquet to temp file instead of `BytesIO`, pass file path to `CommitOperationAdd`. Delete temp file after `preupload_lfs_files` completes (for LFS uploads only - regular uploads need the file until `create_commit`).

## Changes

- Replace `BytesIO` with `tempfile.NamedTemporaryFile` in `_push_parquet_shards_to_hub_single`
- Use file path in `CommitOperationAdd.path_or_fileobj` instead of bytes
- Delete temp file after upload (only for LFS mode - regular uploads keep file for `create_commit`)
- Add `try...finally` for safe cleanup even on errors
- Remove unused `BytesIO` import

## Test plan

- [x] Added `tests/test_push_to_hub_memory.py` with 4 tests:
  - `test_push_to_hub_uses_file_path_not_bytes_in_commit_operation`
  - `test_push_to_hub_cleans_up_temp_files_for_lfs_uploads`
  - `test_push_to_hub_keeps_temp_files_for_regular_uploads`
  - `test_push_to_hub_uploaded_size_still_calculated`
- [x] All tests pass locally
- [x] ruff check passes

## Context

This was discovered while uploading a 270GB neuroimaging dataset (ARC) with 902 shards. The process was killed by OOM after accumulating ~270GB in the `additions` list.

Fixes #7893
Related: #5990, #7400

Comments:
Closing this PR.

After further investigation (prompted by @lhoestq's feedback on #7893), I discovered that `free_memory=True` does work correctly - memory doesn't actually accumulate in the `additions` list as I originally thought.

I tested standard `push_to_hub()` with 902 shards and it processed 625 shards (69%) without OOM issues. The crash we were experiencing was actually **#7894** (`embed_table_storage` crash on `Sequence()` types), not memory accumulation.

I've closed #7893 as invalid. This PR is no longer needed.

Apologies for the noise - lesson learned about testing before filing.
encode nifti correctly when uploading lazily
State: open

When trying to upload nifti datasets lazily I got the error:
```python
from pathlib import Path
from datasets import load_dataset

nifti_dir = Path("<local_path>")

dataset = load_dataset(
        "niftifolder",
        data_dir=str(nifti_dir.absolute()),
        streaming=True,
)

dataset.push_to_hub(repo_id="TobiasPitters/test-nifti-papaya-testdata")
```


```python
pyarrow.lib.ArrowInvalid: Could not convert <datasets.features.nifti.Nifti1ImageWrapper object at 0x77633407af90> with type Nifti1ImageWrapper: did not recognize Python value type when inferring an Arrow data type
```

This PR fixes that by encoding the Nifti1ImageWrappers correctly to bytes. 

Comments:
fix(tests): stabilize flaky Hub LFS integration test
State: open

## Problem

`test_push_dataset_dict_to_hub_overwrite_files` intermittently fails with:
```
BadRequestError: LFS pointer pointed to a file that does not exist
```

This has been causing the `deps-latest` integration tests to fail on main (visible in recent CI runs). I ran into this while working on the BIDS loader PR and dug into the root cause.

## Root Cause

Two race conditions in the test:

1. **LFS propagation timing** - Rapid successive `push_to_hub` calls don't wait for Hub to fully propagate LFS objects between pushes
2. **Repo name reuse** - The second test scenario reused the same repo name from scenario 1, creating a race between deletion and recreation

## Solution

- Add `_wait_for_repo_ready()` helper that polls `list_repo_files` to ensure the repo is consistent before subsequent operations
- Use a unique repo name (`ds_name_2`) for the second scenario, eliminating the delete/create race entirely

## Testing

All 4 integration test variants now pass:
- ✅ `ubuntu-latest, deps-latest` (was failing)
- ✅ `ubuntu-latest, deps-minimum`
- ✅ `windows-latest, deps-latest` (was failing)
- ✅ `windows-latest, deps-minimum`

Validated on fork: https://github.com/The-Obstacle-Is-The-Way/datasets/pull/4

## Related

- #7600 (push_to_hub concurrency)
- #6392 (push_to_hub connection robustness)

cc @lhoestq - small fix but should help CI reliability

Comments:
Add type overloads to load_dataset for better static type inference
State: closed

Fixes #7883

This PR adds @overload decorators to load_dataset() to help type checkers like Pylance and mypy correctly infer the return type based on the split and streaming parameters.

Changes:
- Added typing imports (Literal, overload) to load.py
- Added 4 @overload signatures that map argument combinations to specific return types:
  * split=None, streaming=False -> DatasetDict
  * split specified, streaming=False -> Dataset
  * split=None, streaming=True -> IterableDatasetDict
  * split specified, streaming=True -> IterableDataset

This resolves the Pylance error where to_csv() was not recognized on Dataset objects returned by load_dataset(..., split='train'), since the type checker previously saw the return type as a Union that included types without to_csv().

No runtime behavior changes - this is purely a static typing improvement.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7888). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix(nifti): enable lazy loading for Nifti1ImageWrapper
State: open

## Summary
- **Single-line fix**: Change `dataobj=nifti_image.get_fdata()` → `dataobj=nifti_image.dataobj`
- Preserves nibabel's `ArrayProxy` for true lazy loading instead of eagerly loading entire NIfTI files into memory
- Improves error handling: corrupted files now fail at access time with clear context instead of silently during decode

## Problem
The `Nifti1ImageWrapper.__init__` was calling `get_fdata()` which immediately loads the entire image into memory. For large 4D fMRI files (often 1-2GB), this causes:
1. **Memory issues** - Full data loaded during decode, not on demand
2. **Poor error handling** - Corrupted files crash at access time with unclear error messages (e.g., `EOFError` with no file path)
3. **No graceful recovery** - Entire dataset iteration fails on one bad file

## Solution
Use `nifti_image.dataobj` which preserves the underlying `ArrayProxy`, deferring actual I/O to `get_fdata()` calls.

## Test Plan
- [x] Added `test_nifti_lazy_loading` to verify `ArrayProxy` is preserved
- [x] All 22 existing NIfTI tests pass
- [x] End-to-end tested with real OpenNeuro data (ds000102)
- [x] CodeRabbit approved: "Switch to dataobj correctly restores nibabel's lazy loading semantics... This looks solid"

## Related
- Discovered while testing BIDS loader PR #7886
- Complements the NIfTI + NiiVue viewer work from #7878 and #7874

Comments:
feat(bids): Add BIDS dataset loader for neuroimaging data
State: open

## Summary

Adds native BIDS (Brain Imaging Data Structure) dataset loading support using PyBIDS, enabling `load_dataset('bids', data_dir='/path/to/bids')` workflow for neuroimaging researchers.

**Contributes to #7804** (Support scientific data formats) - BIDS is a widely-used standard for organizing neuroimaging data built on NIfTI files.

## Changes

### Core Implementation
- `src/datasets/packaged_modules/bids/bids.py` - GeneratorBasedBuilder implementation
- `src/datasets/packaged_modules/bids/__init__.py` - Module exports
- `src/datasets/packaged_modules/__init__.py` - Registration with module registry
- `src/datasets/config.py` - `PYBIDS_AVAILABLE` config flag
- `setup.py` - Optional `pybids>=0.21.0` + nibabel dependency

### Features
- Automatic BIDS structure validation
- Subject/session/datatype filtering via config
- JSON sidecar metadata extraction
- NIfTI file decoding via existing Nifti feature

### Documentation & Tests
- `docs/source/bids_dataset.mdx` - User guide with examples
- `tests/packaged_modules/test_bids.py` - Unit tests (4 tests)

## Usage

```python
from datasets import load_dataset

# Load entire BIDS dataset
ds = load_dataset('bids', data_dir='/path/to/bids_dataset')

# Filter by subject/session
ds = load_dataset('bids', 
    data_dir='/path/to/bids_dataset',
    subjects=['01', '02'],
    sessions=['baseline']
)

# Access samples
sample = ds['train'][0]
print(sample['subject'])      # '01'
print(sample['nifti'].shape)  # (176, 256, 256)
print(sample['metadata'])     # JSON sidecar data
```

## Test plan

- [x] All 4 unit tests pass (`pytest tests/packaged_modules/test_bids.py`)
- [x] `make quality` passes (ruff check)
- [x] End-to-end tested with real OpenNeuro data (ds000102)

## Context

This PR is part of the neuroimaging initiative discussed with @TobiasPitters. Follows the BIDS 1.10.1 specification and leverages the existing Nifti feature for NIfTI file handling.

Related PRs:
- #7874 (Nifti visualization support)
- #7878 (Replace papaya with niivue)

Comments:
Fix 7846: add_column and add_item erroneously(?) require new_fingerprint parameter 
State: closed

Summary of the change:

Made new_fingerprint optional (Optional[str] = None) in Dataset.add_column and Dataset.add_item
Added a simple test to verify both methods work without providing a fingerprint

Why this change is safe:

The Dataset constructor already handles fingerprint=None by generating a new fingerprint automatically
No internal logic is broken — if a user provides a fingerprint, it’s still used as before
The change only affects the function signature, making it more user-friendly without changing any functionality

Comments:
@lhoestq can you help review this pull request?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7884). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Data.to_csv() cannot be recognized by pylance
State: closed

### Describe the bug

Hi, everyone ! I am a beginner with datasets. 
I am testing reading multiple CSV files from a zip archive. The result of reading the dataset shows success, and it can ultimately be correctly saved to CSV.

Intermediate results:
```
Generating train split: 62973 examples [00:00, 175939.01 examples/s]
DatasetDict({
    train: Dataset({
        features: ['交易时间\t', '收支方向\t', '业务（产品）种类\t', '交易金额\t', '币种\t', '时点余额\t', '对手方名称\t', '对方机构名称\t', ' 对方钱包ID/账号\t', '交易对手名称\t', '交易对手编号\t', '交易流水号\t', '摘要\t', '附言\t', '备注\t', '用途\t', '客户流水号\t'],
        num_rows: 62973
    })
})
```
However, Pylance gives me the following error:
```
Cannot access attribute "to_csv" for class "DatasetDict"
Attribute "to_csv" is unknownPylance[reportAttributeAccessIssue](https://github.com/microsoft/pylance-release/blob/main/docs/diagnostics/reportAttributeAccessIssue.md)```

Cannot access attribute "to_csv" for class "IterableDatasetDict"
  Attribute "to_csv" is unknownPylance[reportAttributeAccessIssue](https://github.com/microsoft/pylance-release/blob/main/docs/diagnostics/reportAttributeAccessIssue.md)

(method) to_csv: Unknown | ((path_or_buf: datasets.utils.typing.PathLike | BinaryIO, batch_size: int | None = None, num_proc: int | None = None, storage_options: dict[Unknown, Unknown] | None = None, **to_csv_kwargs: Unknown) -> int) | ((path_or_buf: datasets.utils.typing.PathLike | BinaryIO, batch_size: int | None = None, storage_options: dict[Unknown, Unknown] | None = None, **to_csv_kwargs: Unknown) -> int)
```
I ignored the error and continued executing to get the correct result:
```
Dataset({
    features: ['交易时间\t', '收支方向\t', '业务（产品）种类\t', '交易金额\t', '币种\t', '时点余额\t', '对手方名称\t', '对方机构名称\t', '对方 钱包ID/账号\t', '交易对手名称\t', '交易对手编号\t', '交易流水号\t', '摘要\t', '附言\t', '备注\t', '用途\t', '客户流水号\t'],
    num_rows: 62973
})
```
Since the data volume is small, I manually merged the CSV files, and the final result is consistent with what the program saved.


looks like :

<img width="1264" height="150" alt="Image" src="https://github.com/user-attachments/assets/743540d7-ad8c-4531-ae7e-de71a5243a32" />


### Steps to reproduce the bug

this is my code.
```
from datasets import load_dataset

def main():
    url = "data/test.zip"
    data_files = {"train": url}
    dataset = load_dataset("csv", data_files=data_files,split="train", encoding="gbk", skiprows=2)
    # print(dataset)
    dataset.to_csv("data/test.csv")


if __name__ == "__main__":
    main()
```

### Expected behavior

I want to know why this happens. Is there something wrong with my code?

### Environment info

OS: Windows 11    **upgrade from** OS: Windows_NT x64 10.0.22631
Editor: 
VS Code  Version: 1.106.2 (user setup)
"datasets"    version = "4.4.1"


Comments:
Inconsistent loading of LFS-hosted files in epfml/FineWeb-HQ dataset
State: open

### Describe the bug

Some files in the `epfml/FineWeb-HQ` dataset fail to load via the Hugging Face `datasets` library.  

- xet-hosted  files load fine
- LFS-hosted files sometimes fail

Example:  
- Fails: https://huggingface.co/datasets/epfml/FineWeb-HQ/blob/main/data/CC-MAIN-2024-26/000_00003.parquet  
- Works: https://huggingface.co/datasets/epfml/FineWeb-HQ/blob/main/data/CC-MAIN-2024-42/000_00027.parquet  

Discussion: https://huggingface.co/datasets/epfml/FineWeb-HQ/discussions/2  

### Steps to reproduce the bug

```python
from datasets import load_dataset

ds = load_dataset(
    "epfml/FineWeb-HQ",
    data_files="data/CC-MAIN-2024-26/000_00003.parquet",
)
```

Error message:
```
HfHubHTTPError: 403 Forbidden: None.
Cannot access content at: https://cdn-lfs-us-1.hf.co/repos/...
Make sure your token has the correct permissions.
...
<Error><Code>AccessDenied</Code><Message>Access Denied</Message></Error>
```

### Expected behavior

It should load the dataset for all files. 

### Environment info

- python 3.10
- datasets 4.4.1

Comments:
Spurious label column created when audiofolder/imagefolder directories match split names
State: open

## Describe the bug

When using `audiofolder` or `imagefolder` with directories for **splits** (train/test) rather than class labels, a spurious `label` column is incorrectly created.

**Example:** https://huggingface.co/datasets/datasets-examples/doc-audio-4
```
from datasets import load_dataset
ds = load_dataset("datasets-examples/doc-audio-4")
print(ds["train"].features)
```
Shows 'label' column with ClassLabel(names=['test', 'train']) - incorrect!## Root cause

In `folder_based_builder.py`, the `labels` set is accumulated across ALL splits (line 77). When directories are `train/` and `test/`:
- `labels = {"train", "test"}` → `len(labels) > 1` → `add_labels = True`
- Spurious label column is created with split names as class labels

## Expected behavior

No `label` column should be added when directory names match split names.

## Proposed fix

Skip label inference when inferred labels match split names.

cc @lhoestq 

Comments: