noamaanMulla-03/datasets-issues huggingface dataset:
test: add verification for HuggingFaceM4/InterleavedWebDocuments

Adds an integration test for the `HuggingFaceM4/InterleavedWebDocuments` dataset.

- Gracefully skips if the dataset is not yet available on the Hub
- Checks basic loading and structure once it becomes available

Closes #7394

First-time contributor to `datasets` — really excited about this! Happy to make any adjustments needed. 🙂
Spurious label column created when audiofolder/imagefolder directories match split names

## Describe the bug

When using `audiofolder` or `imagefolder` with directories for **splits** (train/test) rather than class labels, a spurious `label` column is incorrectly created.

**Example:** https://huggingface.co/datasets/datasets-examples/doc-audio-4
```
from datasets import load_dataset
ds = load_dataset("datasets-examples/doc-audio-4")
print(ds["train"].features)
```
Shows 'label' column with ClassLabel(names=['test', 'train']) - incorrect!## Root cause

In `folder_based_builder.py`, the `labels` set is accumulated across ALL splits (line 77). When directories are `train/` and `test/`:
- `labels = {"train", "test"}` → `len(labels) > 1` → `add_labels = True`
- Spurious label column is created with split names as class labels

## Expected behavior

No `label` column should be added when directory names match split names.

## Proposed fix

Skip label inference when inferred labels match split names.

cc @lhoestq
Add DICOM support

supports #7804
Add support for the dicom file format. 

This PR follows PR #7815 and PR #7325 closely.
Remarkable differences:
I made sure that we can load all of pydicom's test data, and encountered the `force=True` parameter that we explicitly support here. This allows to trying to load corrupted dicom files, we explicitly test this!

There is one dataset with all of dicom's test data on huggingface which can be loaded using this branch with the following script:
```python
from datasets import load_dataset
from datasets import Features, ClassLabel
from datasets.features import Dicom

features = Features({
    "dicom": Dicom(force=True),  # necessary to be able to load one corrupted file
    "label": ClassLabel(num_classes=2)
})

ds = load_dataset("TobiasPitters/dicom-sample-dataset",
                  features=features)

error_count = 0

for idx, item in enumerate(ds["test"]):
    dicom = item["dicom"]

    try:
        print(f"Type: {type(dicom)}")
        if hasattr(dicom, 'PatientID'):
            print(f"PatientID: {dicom.PatientID}")
        if hasattr(dicom, 'StudyInstanceUID'):
            print(f"StudyInstanceUID: {dicom.StudyInstanceUID}")
        if hasattr(dicom, 'Modality'):
            print(f"Modality: {dicom.Modality}")
    except Exception as e:
        error_count += 1
        print(e)

print(f"Finished processing with {error_count} errors.")
```

todo: 
- [x] add docs (will do so soon)


Awesome ! For the docs should we rename https://huggingface.co/docs/datasets/nifti_dataset to medical_imaging_dataset and have both DICOM and NIfTI together or have separate pages in you opinion ?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7835). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Awesome ! For the docs should we rename https://huggingface.co/docs/datasets/nifti_dataset to medical_imaging_dataset and have both DICOM and NIfTI together or have separate pages in you opinion ?

Makes sense, is more intuitive for the user and the pages as proposed in this branch have a lot of overlap. I would then structure it in such a way to write some brief things about medical imaging, then introduce the formats (so basically concatenating the two pages together and removing duplicates).

Pls don't merge currently, since we'll need an `embed_storage` function in here as well. See
https://github.com/huggingface/datasets/pull/7815#issuecomment-3494094692 and the following conversation

@lhoestq, this is ready for a first round of review.
Add a data type for labeled images (image segmentation)

It might be a mix of Image and ClassLabel, and the color palette might be generated automatically.

---

### Example

every pixel in the images of the annotation column (in https://huggingface.co/datasets/scene_parse_150) has a value that gives its class, and the dataset itself is associated with a color palette (eg https://github.com/open-mmlab/mmsegmentation/blob/98a353b674c6052d319e7de4e5bcd65d670fcf84/mmseg/datasets/ade.py#L47) that maps every class with a color.

So we might want to render the image as a colored image instead of a black and white one.

<img width="785" alt="156741519-fbae6844-2606-4c28-837e-279d83d00865" src="https://user-images.githubusercontent.com/1676121/157005263-7058c584-2b70-465a-ad94-8a982f726cf4.png">

See https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/core/features/labeled_image.py for reference in Tensorflow

Labels: name: enhancement

💯 would be cool to have, but imo after #10769
Fix 7846: add_column and add_item erroneously(?) require new_fingerprint parameter 

Summary of the change:

Made new_fingerprint optional (Optional[str] = None) in Dataset.add_column and Dataset.add_item
Added a simple test to verify both methods work without providing a fingerprint

Why this change is safe:

The Dataset constructor already handles fingerprint=None by generating a new fingerprint automatically
No internal logic is broken — if a user provides a fingerprint, it’s still used as before
The change only affects the function signature, making it more user-friendly without changing any functionality


@lhoestq can you help review this pull request?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7884). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Save input shard lengths

will be useful for the Viewer, to know what (original) shard each row belongs to

cc @cfahlgren1 

next step is use it in Dataset Viewer and expose an API that returns the file containing the row at rowId

(took the opportunity to remove unusued code)


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7897). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: use temp files in push_to_hub to prevent OOM on large datasets

## Summary

Fixes memory accumulation in `_push_parquet_shards_to_hub_single` that causes OOM when uploading large datasets with many shards.

**Root cause**: The current implementation stores ALL parquet shard bytes in memory via `BytesIO`, accumulating in the `additions` list. For N shards of ~300MB each, this requires N × 300MB RAM.

**Fix**: Write parquet to temp file instead of `BytesIO`, pass file path to `CommitOperationAdd`. Delete temp file after `preupload_lfs_files` completes (for LFS uploads only - regular uploads need the file until `create_commit`).

## Changes

- Replace `BytesIO` with `tempfile.NamedTemporaryFile` in `_push_parquet_shards_to_hub_single`
- Use file path in `CommitOperationAdd.path_or_fileobj` instead of bytes
- Delete temp file after upload (only for LFS mode - regular uploads keep file for `create_commit`)
- Add `try...finally` for safe cleanup even on errors
- Remove unused `BytesIO` import

## Test plan

- [x] Added `tests/test_push_to_hub_memory.py` with 4 tests:
  - `test_push_to_hub_uses_file_path_not_bytes_in_commit_operation`
  - `test_push_to_hub_cleans_up_temp_files_for_lfs_uploads`
  - `test_push_to_hub_keeps_temp_files_for_regular_uploads`
  - `test_push_to_hub_uploaded_size_still_calculated`
- [x] All tests pass locally
- [x] ruff check passes

## Context

This was discovered while uploading a 270GB neuroimaging dataset (ARC) with 902 shards. The process was killed by OOM after accumulating ~270GB in the `additions` list.

Fixes #7893
Related: #5990, #7400


Closing this PR.

After further investigation (prompted by @lhoestq's feedback on #7893), I discovered that `free_memory=True` does work correctly - memory doesn't actually accumulate in the `additions` list as I originally thought.

I tested standard `push_to_hub()` with 902 shards and it processed 625 shards (69%) without OOM issues. The crash we were experiencing was actually **#7894** (`embed_table_storage` crash on `Sequence()` types), not memory accumulation.

I've closed #7893 as invalid. This PR is no longer needed.

Apologies for the noise - lesson learned about testing before filing.
Add type overloads to load_dataset for better static type inference

Fixes #7883

This PR adds @overload decorators to load_dataset() to help type checkers like Pylance and mypy correctly infer the return type based on the split and streaming parameters.

Changes:
- Added typing imports (Literal, overload) to load.py
- Added 4 @overload signatures that map argument combinations to specific return types:
  * split=None, streaming=False -> DatasetDict
  * split specified, streaming=False -> Dataset
  * split=None, streaming=True -> IterableDatasetDict
  * split specified, streaming=True -> IterableDataset

This resolves the Pylance error where to_csv() was not recognized on Dataset objects returned by load_dataset(..., split='train'), since the type checker previously saw the return type as a Union that included types without to_csv().

No runtime behavior changes - this is purely a static typing improvement.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7888). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Data.to_csv() cannot be recognized by pylance

### Describe the bug

Hi, everyone ! I am a beginner with datasets. 
I am testing reading multiple CSV files from a zip archive. The result of reading the dataset shows success, and it can ultimately be correctly saved to CSV.

Intermediate results:
```
Generating train split: 62973 examples [00:00, 175939.01 examples/s]
DatasetDict({
    train: Dataset({
        features: ['交易时间\t', '收支方向\t', '业务（产品）种类\t', '交易金额\t', '币种\t', '时点余额\t', '对手方名称\t', '对方机构名称\t', ' 对方钱包ID/账号\t', '交易对手名称\t', '交易对手编号\t', '交易流水号\t', '摘要\t', '附言\t', '备注\t', '用途\t', '客户流水号\t'],
        num_rows: 62973
    })
})
```
However, Pylance gives me the following error:
```
Cannot access attribute "to_csv" for class "DatasetDict"
Attribute "to_csv" is unknownPylance[reportAttributeAccessIssue](https://github.com/microsoft/pylance-release/blob/main/docs/diagnostics/reportAttributeAccessIssue.md)```

Cannot access attribute "to_csv" for class "IterableDatasetDict"
  Attribute "to_csv" is unknownPylance[reportAttributeAccessIssue](https://github.com/microsoft/pylance-release/blob/main/docs/diagnostics/reportAttributeAccessIssue.md)

(method) to_csv: Unknown | ((path_or_buf: datasets.utils.typing.PathLike | BinaryIO, batch_size: int | None = None, num_proc: int | None = None, storage_options: dict[Unknown, Unknown] | None = None, **to_csv_kwargs: Unknown) -> int) | ((path_or_buf: datasets.utils.typing.PathLike | BinaryIO, batch_size: int | None = None, storage_options: dict[Unknown, Unknown] | None = None, **to_csv_kwargs: Unknown) -> int)
```
I ignored the error and continued executing to get the correct result:
```
Dataset({
    features: ['交易时间\t', '收支方向\t', '业务（产品）种类\t', '交易金额\t', '币种\t', '时点余额\t', '对手方名称\t', '对方机构名称\t', '对方 钱包ID/账号\t', '交易对手名称\t', '交易对手编号\t', '交易流水号\t', '摘要\t', '附言\t', '备注\t', '用途\t', '客户流水号\t'],
    num_rows: 62973
})
```
Since the data volume is small, I manually merged the CSV files, and the final result is consistent with what the program saved.


looks like :

<img width="1264" height="150" alt="Image" src="https://github.com/user-attachments/assets/743540d7-ad8c-4531-ae7e-de71a5243a32" />


### Steps to reproduce the bug

this is my code.
```
from datasets import load_dataset

def main():
    url = "data/test.zip"
    data_files = {"train": url}
    dataset = load_dataset("csv", data_files=data_files,split="train", encoding="gbk", skiprows=2)
    # print(dataset)
    dataset.to_csv("data/test.csv")


if __name__ == "__main__":
    main()
```

### Expected behavior

I want to know why this happens. Is there something wrong with my code?

### Environment info

OS: Windows 11    **upgrade from** OS: Windows_NT x64 10.0.22631
Editor: 
VS Code  Version: 1.106.2 (user setup)
"datasets"    version = "4.4.1"
Add inspect_ai eval logs support

Support for .eval log files from inspect_ai

They are actually ZIP files according to the source code at https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/log/_log.py

Unfortunately their format can't be converted to Parquet, so I had to JSON-encode all the nested values

```python
ds = load_dataset("dvilasuero/kimi-bfcl")
```

this will enable the Viewer for datasets like https://huggingface.co/datasets/dvilasuero/kimi-bfcl

original tweet for context: https://x.com/dvilasuero/status/1996936988176343220?s=20

cc @dvsrepo @julien-c @davanstrien 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7899). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Very cool! 

Any reason not to directly use inspect for loading/converting from the binary format to JSON? https://inspect.aisi.org.uk/reference/inspect_ai.log.html#convert_eval_logs 

The format is simple enough to not have to rely on an additional dependency :)
Fix ArrowInvalid for large variables

Fixes #7821

In addition to the solution proposed in the issue, I encountered we also need to support 64bit types when writing using the `RecordBatchStreamWriter`. 

Not sure if we want to create such large objects in the CI, but this is the only way to test that the issue is fixed, therefore I added the unit test.


@lhoestq : from what I see in the CI logs ( tests/test_arrow_dataset.py::test_map_int32_overflow 
/opt/hostedtoolcache/Python/3.9.23/x64/lib/python3.9/site-packages/multiprocess/resource_tracker.py:219: UserWarning: resource_tracker: There appear to be 24 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '), see [here](https://github.com/huggingface/datasets/actions/runs/18720027796/job/53591652242#step:13:6530), the test crashes probably since it consumes too much memory, it works locally for me though. 
  
  Shall we keep the test, remove ti, or skip it, or is there a specific pytest flag you have for slow/memory-consuming tests I can add, so that it gets skipped automatically on the CI but might be run locally?
  
  EDIT: I added the `high_memory` marker (see [here](https://github.com/huggingface/datasets/pull/7825/files#diff-c480dc36aca482db963c43ee66ba21705111f62f826352c96558ef5ed1619791R4789)), so this will not be run in the CI, which is not ideal but I guess the way if we want to keep the test
Request: Review pending neuroimaging PRs (#7886 BIDS loader, #7887 lazy loading)

## Summary

I'm building production neuroimaging pipelines that depend on `datasets` and would benefit greatly from two pending PRs being reviewed/merged.

## Pending PRs

| PR | Description | Status | Open Since |
|----|-------------|--------|------------|
| [#7886](https://github.com/huggingface/datasets/pull/7886) | BIDS dataset loader | Open | Nov 29 |
| [#7887](https://github.com/huggingface/datasets/pull/7887) | Lazy loading for NIfTI | Open | Nov 29 |

## Use Case

The neuroimaging community uses the BIDS (Brain Imaging Data Structure) standard for organizing MRI/fMRI data. These PRs would enable:

1. **#7886**: `load_dataset('bids', data_dir='/path/to/bids')` - Load local BIDS directories directly
2. **#7887**: Memory-efficient NIfTI handling (single 4D fMRI file can be 1-2GB)

## Current Workaround

Without these, users must either:
- Upload to Hub first, then consume (works but slow iteration)
- Hand-roll BIDS parsing (duplicates effort)

## Request

Could a maintainer review these PRs? Happy to address any feedback. The BIDS loader has tests passing and was end-to-end tested with real OpenNeuro data.

Thank you for the great work on `Nifti()` support - these PRs build on that foundation.

## Related

- Contributes to #7804 (Support scientific data formats)
- Built on @TobiasPitters's Nifti feature work


Hi ! sure I'll be happy to take a look, sorry for the delay :)
The child process retrieves the dataset directly from the main process instead of executing `memory_mapped_arrow_table_from_file`.

### Feature request

The child process retrieves the dataset directly from the main process instead of executing `memory_mapped_arrow_table_from_file`.

### Motivation

Because my local disk space is insufficient, I can only store a dataset on a remote Ceph server and process it using datasets.
I used the data-juicer[https://github.com/datajuicer/data-juicer] framework as an outer layer which uses datasets, but it doesn't support streaming datasets. I then encountered a problem: for each load, map, and filter operation, I had to wait for a large number of child processes to execute `memory_mapped_arrow_table_from_file`. Since the actual file was on the remote Ceph server, this operation was limited by network I/O.
I don't know if it's a problem with my usage or if this is how datasets are currently designed.However, I think that if the instances obtained after datasets.load_datasets are directly passed to the child process instead of re-executing `memory_mapped_arrow_table_from_file`, it might solve my problem.Or datasets already support this capability, but I just didn't know it？

### Your contribution

。。。

Labels: name: enhancement

Memory mapping is actually the way for processes to share memory efficiently and without copy. It is efficient when on are using a local disk, and it's discouraged to use it on remote disk for the reasons you observed.

What you can do instead is save the dataset as Parquet on your remote storage (or on Hugging Face Datasets which offers fast uploads thanks to Xet), and then your can reload it in streaming mode. Streaming mode is ideal to use a dataset that is hosted in a remote storage
work around `tempfile` silently ignoring `TMPDIR` if the dir doesn't exist

This should help a lot of users running into `No space left on device` while using `datasets`. Normally the issue is is that `/tmp` is too small and the user needs to use another path, which they would normally set as `export TMPDIR=/some/big/storage`

However, the `tempfile` facility that `datasets` and `pyarrow` use is somewhat broken. If the path doesn't exist it'd ignore it and fall back to using `/tmp`. Watch this:

```
$ export TMPDIR='/tmp/username' 

$ python -c "\
import os
import tempfile
print(os.environ['TMPDIR'])
print(tempfile.gettempdir())"
/tmp/username
/tmp
```

Now let's ensure the path exists:
```
$ export TMPDIR='/tmp/username' 
$ mkdir -p $TMPDIR
$ python -c "\
import os
import tempfile
print(os.environ['TMPDIR'])
print(tempfile.gettempdir())"
/tmp/username
/tmp/username
```

So I recommend `datasets` do either of the 2:
1. assert if `$TMPDIR` dir doesn't exist, telling the user to create it
2. auto-create it

The reason for (1) is that I don't know why `tempdir` doesn't auto-create the dir - perhaps some security implication? I will let you guys make the decision, but the key is not to let things silently fall through and the user puzzling why no matter what they do they can't break past `No space left on device` while using `datasets`

Thank you.

I found this via https://stackoverflow.com/questions/37229398/python-tempfile-gettempdir-does-not-respect-tmpdir while trying to help a colleague to solve this exact issue.


Hi! Just created a Pull Request (#7890) to try to fix this using your suggestions. I hope it helps!
Inconsistent loading of LFS-hosted files in epfml/FineWeb-HQ dataset

### Describe the bug

Some files in the `epfml/FineWeb-HQ` dataset fail to load via the Hugging Face `datasets` library.  

- xet-hosted  files load fine
- LFS-hosted files sometimes fail

Example:  
- Fails: https://huggingface.co/datasets/epfml/FineWeb-HQ/blob/main/data/CC-MAIN-2024-26/000_00003.parquet  
- Works: https://huggingface.co/datasets/epfml/FineWeb-HQ/blob/main/data/CC-MAIN-2024-42/000_00027.parquet  

Discussion: https://huggingface.co/datasets/epfml/FineWeb-HQ/discussions/2  

### Steps to reproduce the bug

```python
from datasets import load_dataset

ds = load_dataset(
    "epfml/FineWeb-HQ",
    data_files="data/CC-MAIN-2024-26/000_00003.parquet",
)
```

Error message:
```
HfHubHTTPError: 403 Forbidden: None.
Cannot access content at: https://cdn-lfs-us-1.hf.co/repos/...
Make sure your token has the correct permissions.
...
<Error><Code>AccessDenied</Code><Message>Access Denied</Message></Error>
```

### Expected behavior

It should load the dataset for all files. 

### Environment info

- python 3.10
- datasets 4.4.1


we are investigating the issue, I'll post updates here

It should be all good now, sorry for the inconvenience.

We found an error that happened during the migration of this repository from Git LFS to Xet and fixed the issue
encode nifti correctly when uploading lazily

When trying to upload nifti datasets lazily I got the error:
```python
from pathlib import Path
from datasets import load_dataset

nifti_dir = Path("<local_path>")

dataset = load_dataset(
        "niftifolder",
        data_dir=str(nifti_dir.absolute()),
        streaming=True,
)

dataset.push_to_hub(repo_id="TobiasPitters/test-nifti-papaya-testdata")
```


```python
pyarrow.lib.ArrowInvalid: Could not convert <datasets.features.nifti.Nifti1ImageWrapper object at 0x77633407af90> with type Nifti1ImageWrapper: did not recognize Python value type when inferring an Arrow data type
```

This PR fixes that by encoding the Nifti1ImageWrappers correctly to bytes. 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7892). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix(nifti): enable lazy loading for Nifti1ImageWrapper

## Summary
- **Single-line fix**: Change `dataobj=nifti_image.get_fdata()` → `dataobj=nifti_image.dataobj`
- Preserves nibabel's `ArrayProxy` for true lazy loading instead of eagerly loading entire NIfTI files into memory
- Improves error handling: corrupted files now fail at access time with clear context instead of silently during decode

## Problem
The `Nifti1ImageWrapper.__init__` was calling `get_fdata()` which immediately loads the entire image into memory. For large 4D fMRI files (often 1-2GB), this causes:
1. **Memory issues** - Full data loaded during decode, not on demand
2. **Poor error handling** - Corrupted files crash at access time with unclear error messages (e.g., `EOFError` with no file path)
3. **No graceful recovery** - Entire dataset iteration fails on one bad file

## Solution
Use `nifti_image.dataobj` which preserves the underlying `ArrayProxy`, deferring actual I/O to `get_fdata()` calls.

## Test Plan
- [x] Added `test_nifti_lazy_loading` to verify `ArrayProxy` is preserved
- [x] All 22 existing NIfTI tests pass
- [x] End-to-end tested with real OpenNeuro data (ds000102)
- [x] CodeRabbit approved: "Switch to dataobj correctly restores nibabel's lazy loading semantics... This looks solid"

## Related
- Discovered while testing BIDS loader PR #7886
- Complements the NIfTI + NiiVue viewer work from #7878 and #7874


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7887). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.