noamaanMulla-03/datasets-issues huggingface dataset:
`set_format` resets features

### Describe the bug

Calling `dataset.set_format('...')` resets the informative features such as `Array2D` etc. back into `List(List(Value))` and so on.

### Steps to reproduce the bug

```py
>>> dataset.features
{'lld': Array2D(shape=(None, 26), dtype='float32')}

>>> dataset.set_format('torch')
>>> dataset.features
{'lld': List(List(Value('float32')))}
```

### Expected behavior

The feature information is not lost during `set_format`, and the feature remains `Array2D` with the correct shape and dtype in the above example.

### Environment info


- `datasets` version: 4.8.2
- Platform: Linux-6.8.0-78-generic-x86_64-with-glibc2.39
- Python version: 3.13.11
- `huggingface_hub` version: 0.36.2
- PyArrow version: 23.0.1
- Pandas version: 2.3.3
- `fsspec` version: 2026.2.0



couldn't replicate with: 

``` bash
Platform: macos
python: 3.10.9
datasets: 4.8.4
huggingface_hub: 0.36.2
pyarrow: 23.0.1
pandas: 2.3.3
fsspec: 2026.2.0

```


``` python
import datasets
from datasets import Array2D, Dataset
import numpy as np

# Create a dataset with Array2D feature
data = {"lld": [np.zeros((5, 26), dtype="float32").tolist() for _ in range(10)]}

features = datasets.Features({"lld": Array2D(shape=(None, 26), dtype="float32")})
dataset = Dataset.from_dict(data, features=features)

print("Before set_format:")
print(dataset.features)

dataset.set_format("torch")

print("\nAfter set_format('torch'):")
print(dataset.features)

```

Hi! Triaging older issues — I couldn't reproduce this on current `main`.

```python
>>> from datasets import Dataset, Features, Array2D
>>> import numpy as np
>>> features = Features({"lld": Array2D(shape=(5, 26), dtype="float32")})
>>> ds = Dataset.from_dict(
...     {"lld": [np.zeros((5, 26), dtype=np.float32).tolist()] * 2},
...     features=features,
... )
>>> ds.features
{'lld': Array2D(shape=(5, 26), dtype='float32')}
>>> ds.set_format("torch"); ds.features
{'lld': Array2D(shape=(5, 26), dtype='float32')}
>>> ds.set_format("numpy"); ds.features
{'lld': Array2D(shape=(5, 26), dtype='float32')}
>>> ds.set_format(None); ds.features
{'lld': Array2D(shape=(5, 26), dtype='float32')}
```

`Array2D` is preserved across `torch` / `numpy` / `None` set_format calls. Tested on `datasets==4.8.6.dev0` with CPU `torch==2.11.0+cpu`. The issue was opened against 4.8.2, so it looks like one of the 4.8.x releases fixed it.

Would you mind closing this out? If there's a code path that still collapses `Array2D` to `List(List(Value))` after `set_format`, point me at the exact dataset/snippet (the original issue body cuts off after the dtype dump) and I'll dig further.

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; the snippet above was run end-to-end on `main`.)

If two people couldn't reproduce with `4.8.4` and `4.8.6` then I guess it doesn't exist anymore, closing
`ClassLabel` docs: Correct value for unknown labels

This small change fixes the documentation to to be compliant with what happens in `encode_example`.
https://github.com/huggingface/datasets/blob/e71b0b19d79c7531f9b9bea7c09916b5f6157f42/src/datasets/features/features.py#L1126-L1129


@lhoestq Can you take a look at this PR? It seems to me that docs is incorrect at that point.
Potential conflicting type checks and dead code in `/src/datasets/table.py`

When statically analyzing and manually reviewing the code, I noticed a potential logic conflicting in `/src/datasets/table.py` as follows:

```
def to_blocks(table: Union[pa.Table, Table]) -> list[list[TableBlock]]:
      if isinstance(table, pa.Table):
          return [[InMemoryTable(table)]]
      elif isinstance(table, ConcatenationTable):  # dead code
          return copy.deepcopy(table.blocks)
      else:
          return [[table]]
```

Within the function, the condition `isinstance(table, ConcatenationTable)` at line 4 will never be True because the previous condition `isinstance(table, pa.Table)` at line 2 would have already caught all instances of `ConcatenationTable` (since `ConcatenationTable` is a subtype of `pa.Table`). This creates a logical conflict in the type checking flow.

Please verify if this logic is intentional or it is an issue warranting a refactoring or fixing.


ConcatenationTable is a subclass of datasets.table.Table but not pa.Table, so it should be fine

Hi! Triaging stale issues — I don't think this one is actually dead code; the premise is off.

The claim is that `isinstance(table, ConcatenationTable)` at line 4 is unreachable because the line-2 `isinstance(table, pa.Table)` would have already matched, since `ConcatenationTable` is described as "a subtype of `pa.Table`".

But `ConcatenationTable` is **not** a subclass of `pa.Table`: `Table` (the base) wraps a `pa.Table` via composition, not inheritance (see `class Table(IndexedTableMixin)` and the `self.table = table` in `__init__`). I verified on current `main`:

```python
>>> from datasets.table import ConcatenationTable, InMemoryTable, Table
>>> import pyarrow as pa
>>> issubclass(Table, pa.Table)
False
>>> issubclass(ConcatenationTable, pa.Table)
False
>>> issubclass(ConcatenationTable, Table)
True
```

So the `elif isinstance(table, ConcatenationTable):` branch is reachable for `ConcatenationTable` instances (and the trailing `else` covers other `Table` subclasses like `InMemoryTable` / `MemoryMappedTable`). The `to_blocks` dispatch is correct as written.

Would you mind closing this out? If you saw the `pa.Table` branch swallowing a `ConcatenationTable` in some other repro, point me at it and I'll take another look.

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; the `issubclass` checks above were run against current `main`.)
fix parquet reshard

.reshard() would not do anything and return the same iterable dataset for parquet, now it returns the resharded iterable dataset - sharded per row group instead of per file


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8193). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Support explicit harness field in agent traces

Hi!

This small PR allows an optional `harness` field to be parsed.

When present, the harness field takes priority over type-value heuristics.

This allows agents to self-identify even when they contain property names that collide with an existing harness ("session", "system", etc.)


For a concrete example, [https://huggingface.co/datasets/jedisct1/agent-traces-swival](https://huggingface.co/datasets/jedisct1/agent-traces-swival) currently shows `claude_code` because it shares field names with Claude Code, despite an explicit `harness` field identifying the harness as not Claude Code.


> For a concrete example, https://huggingface.co/datasets/jedisct1/agent-traces-swival currently shows `claude_code` because it shares field names with Claude Code, despite an explicit `harness` field identifying the harness as not Claude Code.

is there any parsing issues from the actual traces, or more about just the name of the harness?

@cfahlgren1  No parsing issues. This is just about showing the correct harness name.

Hi @cfahlgren1 @lhoestq,

This PR was automatically closed because the branch was deleted.

Would it still make sense to display the current harness name when one is provided?

Thank you!
Fix spurious label column when folder builders see split-named directories

Closes #7880.

## What

`folder_based_builder._split_generators` allocates a single `labels: set[str]` outside the per-split loop and `analyze(...)` mutates it across every split. When the user passes:

```python
load_dataset("audiofolder", data_files={"train": "train/**", "test": "test/**"})
```

and the actual files live directly under `train/` and `test/` (no per-class subdirs), the parent-dir-name accumulator ends up with `labels = {"train", "test"}`. The `add_labels = len(labels) > 1 and len(path_depths) == 1` heuristic then flips True and the loader silently adds a `ClassLabel(names=['test', 'train'])` column. Worse, `add_labels` is recomputed per split inside the loop while features get set once after, so the first split's examples are emitted with `label=None` — which downstream `int2str(None)` then trips over.

The fix adds a single check: if the inferred label set is a subset of the user-declared split names, treat it as splits-not-classes and skip label inference. `drop_labels=False` (the explicit opt-in to label inference) is preserved.

## How is this tested?

New fixture `data_files_with_split_dirs_no_metadata` builds `train/` + `test/` dirs without class subdirs. Parametrized test:

- default → no `label` column
- `drop_labels=True` → no `label` column
- `drop_labels=False` → ClassLabel still produced with `names=['test', 'train']` (opt-in preserved)

The test fails on `main` with both the spurious column and the `int2str(None)` crash on `drop_labels=False` (mis-emitted label=None). It passes with this PR.

Full `tests/packaged_modules/test_folder_based_builder.py` (49 tests), `test_imagefolder.py` (36), `test_videofolder.py` (8), and `test_data_files.py` (224) all pass. `make quality` + `make style` clean.

`test_audiofolder.py` has 11 pre-existing `torchcodec`/`libavutil` failures on `main` that are unrelated to this change.
fix: replace list with Sequence in public API type hints 

Fixes #5354

Replaced `list` with `Sequence` in public API function signatures across:
- `arrow_dataset.py` — from_csv, from_json, from_parquet, from_text
- `iterable_dataset.py` — from_csv, from_json, from_parquet, from_text  
- `combine.py` — interleave_datasets, concatenate_datasets
- `dataset_dict.py` — remove_columns, select_columns

`Sequence` is covariant and more flexible than `list`, which resolves 
the mypy type checking errors reported in the issue.
Replace AutoFeatureExtractor with AutoImageProcessor in image preprocessing docs

## Summary

The `docs/source/use_dataset.mdx` documentation uses `AutoFeatureExtractor` from `transformers` in its image preprocessing example. For vision models, `AutoImageProcessor` is now the recommended API and better reflects current `transformers` conventions.

## Problem

In the [[Apply data augmentations](https://huggingface.co/docs/datasets/use_dataset#apply-data-augmentations)] section of the documentation, the image augmentation example imports and uses `AutoFeatureExtractor`:

```python
from transformers import AutoFeatureExtractor
from datasets import load_dataset, Image

feature_extractor = AutoFeatureExtractor.from_pretrained("google/vit-base-patch16-224-in21k")
dataset = load_dataset("AI-Lab-Makerere/beans", split="train")
```

## Fix

Replace `AutoFeatureExtractor` with `AutoImageProcessor` in the code snippet:

```python
from transformers import AutoImageProcessor
from datasets import load_dataset, Image

image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k")
dataset = load_dataset("AI-Lab-Makerere/beans", split="train")
```

This fix is consistent with the direction taken in the `transformers` library itself — see [[PR #20111](https://github.com/huggingface/transformers/pull/20111)] and [[PR #20501](https://github.com/huggingface/transformers/pull/20501)], where vision feature extractor references were systematically replaced with image processors across the `transformers` codebase and documentation.

> **Note:** the variable name `feature_extractor` can be kept as-is for a minimal diff, but renaming it to `image_processor` better reflects the updated class name and matches current `transformers` conventions.

## References

- [`AutoImageProcessor` docs](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoImageProcessor)
- [[PR #20111](https://github.com/huggingface/transformers/pull/20111)](https://github.com/huggingface/transformers/pull/20111) — Introduced `AutoImageProcessor` as the dedicated class for vision models
- [[PR #20501](https://github.com/huggingface/transformers/pull/20501)](https://github.com/huggingface/transformers/pull/20501) — Replaced all vision feature extractor references with image processors throughout the `transformers` documentation. The `datasets` docs were not in scope for that PR, making this fix a natural follow-up.


I'd like to work on this. I'll open a PR shortly.
Please enable Private Vulnerability Reporting

Hi datasets team,

I have a security finding in datasets that I would like to disclose privately. GitHub Private Vulnerability Reporting is currently disabled on this repo.

Could you turn it on? It's at Settings > Security > Code security and analysis > Private vulnerability reporting. Once enabled I will file the full report through that channel so a GHSA + CVE can be assigned cleanly.

If you'd prefer email instead, the address listed in your SECURITY.md (if any) works too — let me know and I'll send the writeup that way.

Thanks,
0xBassia (https://github.com/0xBassia)
fix pathlib.Path support in save_to_disk and load_from_disk

`url_to_fs` from fsspec only accepts strings, but `save_to_disk` and `load_from_disk` in `arrow_dataset.py`, `dataset_dict.py`, and `load.py` pass the user-supplied path directly without converting it, causing a crash when a `pathlib.Path` is provided. Added `str()` conversion at each `url_to_fs` call site so all path-like objects are handled correctly. To verify, call `dataset.save_to_disk(Path("my_dir"))` and `Dataset.load_from_disk(Path("my_dir"))` with a `pathlib.Path`.

Fixes #6829
audio folder builder cannot detect custom split name

### Describe the bug

when using audio folder builder (`load_dataset("audiofolder", data_dir="/path/to/folder")`), it cannot detect custom split name other than train/validation/test

### Steps to reproduce the bug

i have the following folder structure
```
my_dataset/
├── train/
│   ├── lorem.wav
│   ├── …
│   └── metadata.csv
├── test/
│   ├── ipsum.wav
│   ├── …
│   └── metadata.csv
├── validation/
│   ├── dolor.wav
│   ├── …
│   └── metadata.csv
└── custom/
    ├── sit.wav
    ├── …
    └── metadata.csv
```
using `ds = load_dataset("audiofolder", data_dir="/path/to/my_dataset")`

### Expected behavior

i got `ds` with only 3 splits train/validation/test, whenever i rename train/validation/test folder it also disappear if i re-create `ds`

### Environment info

- `datasets` version: 3.5.0
- Platform: Windows-11-10.0.26100-SP0
- Python version: 3.12.8
- `huggingface_hub` version: 0.30.2
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0


Hi! Triaging older issues — reproduced your behavior, but I think this is by design rather than a bug.

The folder-based builders (`audiofolder`/`imagefolder`/`videofolder`) auto-detect splits using a fixed keyword list (`train`, `validation`/`valid`/`val`/`dev`, `test`/`testing`/`eval`/`evaluation`) — see `SPLIT_KEYWORDS` in `src/datasets/data_files.py`. A directory called `custom/` won't match any of those keywords, so it's not picked up by the default pattern resolver.

The supported workaround for arbitrary split names is to pass `data_files=` explicitly:

```python
from datasets import load_dataset

ds = load_dataset(
    "audiofolder",
    data_files={
        "train": "/path/to/my_dataset/train/*",
        "validation": "/path/to/my_dataset/validation/*",
        "test": "/path/to/my_dataset/test/*",
        "custom": "/path/to/my_dataset/custom/*",
    },
)
```

This works the same way for all `*folder` builders.

Would you mind closing this out? If you'd like the docs (`docs/source/audio_dataset.mdx`, `image_dataset.mdx`, `video_dataset.mdx`) to call out the auto-detection keyword set and the `data_files=` escape hatch more explicitly, drop a 👍 and I'll send a docs PR.

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; I reproduced the symptom locally with the equivalent `imagefolder` setup on current `main`.)

ok that's fine for me, pls update the related docs
Fix duplicate keyword arguments

## Summary

This PR fixes a duplicate keyword argument collision in `load_dataset_builder()` when the same parameter is passed through both `builder_kwargs` and `config_kwargs`.

Previously, calling `load_dataset()` or `load_dataset_builder()` with arguments such as `base_path` could raise:

```python
TypeError: type object got multiple values for keyword argument "base_path"
```

This occurred because `base_path` could exist simultaneously in:

* explicitly passed constructor arguments,
* `builder_kwargs`,
* and `config_kwargs`.

## Root Cause

`load_dataset_builder()` already pops several known constructor arguments from `builder_kwargs` before forwarding them to the builder constructor:

```python
data_dir = builder_kwargs.pop("data_dir", data_dir)
data_files = builder_kwargs.pop("data_files", data_files)
config_name = builder_kwargs.pop("config_name", ...)
dataset_name = builder_kwargs.pop("dataset_name", None)
```

However, `base_path` was not handled similarly, which allowed it to be forwarded multiple times during builder initialization.

## Changes

* Added handling for `base_path` by removing it from `builder_kwargs` before constructor expansion.
* Refactored builder initialization kwargs into a dedicated `builder_kwargs_for_init` dictionary for clearer separation of explicitly managed constructor arguments.
* Preserved the intended precedence behavior where user-provided `config_kwargs` override module-provided defaults.

### Added logic

```python
base_path = builder_kwargs.pop("base_path", None)
```

and conditionally forwarded it through the explicit initialization kwargs.

## Tests

Added a regression test to ensure `load_dataset_builder()` accepts `base_path` without raising duplicate keyword argument errors.

Test coverage includes:

* successful builder initialization with `base_path`
* prevention of duplicate kwarg collisions

## Example

Previously failing call:

```python
from datasets import load_dataset

load_dataset("rotten_tomatoes", base_path="./sample_data")
```

This now initializes successfully without raising `TypeError`.

Fixes #4910
Use multiple input shards for shuffle buffer

```python
ds = load_dataset(..., streaming=True)
ds = ds.shuffle(seed=42)
# or configure local buffer shuffling manually, default is:
ds = ds.shuffle(seed=42, buffer_size=1000, max_buffer_input_shards=10)
```

before👎:
<img width="1061" height="529" alt="image" src="https://github.com/user-attachments/assets/dd85a11b-72fc-474c-8faa-33eda779a4fb" />

after✨:
<img width="1057" height="523" alt="image" src="https://github.com/user-attachments/assets/77acc8d3-61d4-4e57-bc87-43fcd3c2e8f3" />

toy example comparison

```python
from datasets import IterableDataset

ds = IterableDataset.from_dict({"i": range(123_456_789)}, num_shards=1024)
ds = ds.shuffle(seed=42)

print("Cold start ids:")
print(list(ds.take(10)["i"]))
print("Nominal regime ids:")
print(list(ds.skip(10_000).take(10)["i"]))
```

before👎:

```
Cold start ids:
[6148853, 6149537, 6149418, 6149202, 6149197, 6149622, 6148849, 6149461, 6148965, 6148858]
Nominal regime ids:
[6149537, 6149418, 6149202, 6149197, 6149622, 6148849, 6149461, 6148965, 6148858, 6149290]
```

after✨:

```
Cold start ids:
[7836668, 9283505, 95847927, 482299, 9283471, 482341, 112003312, 59920157, 43764666, 95847871]
Nominal regime ids:
[9283505, 95847927, 482299, 9283471, 482341, 112003312, 59920157, 43764666, 95847871, 16758448]
```


related to https://github.com/huggingface/datasets/issues/8015

Note: `ds.state_dict()` and `ds.load_state_dict()` are still supported for this improved shuffling :) enabling dataset checkpointing

Note 2: it uses threads to fetch the first examples in parallel from the input shards


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8194). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: pass token_per_repo_id to embed_table_storage when pushing shards

## Summary

- Fixes #6348
- When `push_to_hub` is called with a `token`, the per-repo token map (`token_per_repo_id`) is built correctly earlier in the function — but it was never forwarded to `embed_table_storage` during the shard-map step.
- This caused a one-line fix: wrap `embed_table_storage` with `functools.partial` so `token_per_repo_id` is passed through.

## Root cause

In `Dataset.push_to_hub` (around line 5896 of `src/datasets/arrow_dataset.py`), the shard is mapped with:

```python
# Before
shard.map(embed_table_storage, ...)

# After
shard.map(partial(embed_table_storage, token_per_repo_id=token_per_repo_id), ...)
```

Without this, any dataset with embedded storage (e.g. `Image`, `Audio`) pushed to a private/gated repo would fail to authenticate when reading the referenced files.

## Test plan

- [ ] Existing unit tests pass (`pytest tests/test_arrow_dataset.py -x -q`)
- [ ] Manual smoke-test: push a dataset with `Image` column to a private Hub repo using a token


token_per_repo_id is only useful for iterable datasets
fix(webdataset): when loading data in WebDataset format using load_datasets during multi-matchines training.

## Description
Fixes #8201 

**The Bug:**
During distributed training with `streaming=True`, if a user provides fewer `.tar` files than the number of distributed ranks, some ranks receive an empty list of files. This caused an `IndexError: list index out of range` in `webdataset.py` because the code assumed `tar_paths[0]` always existed for feature inference. 

**The Fix:**
1. Updated `_split_generators` to scan across splits and capture the `first_tar_path`. Feature inference is now safely skipped for ranks that receive zero files.
2. Updated `_generate_examples` to gracefully handle cases where `self.info.features` is `None` (due to the skipped inference), defaulting to an empty dictionary to prevent downstream `AttributeError` crashes.

## Testing
- [x] Verified logic locally.
- [x] Ran `pytest tests/packaged_modules/test_webdataset.py` (Note: `test_audio_webdataset` skips/fails locally on Windows due to missing FFmpeg `libtorchcodec` dependencies, but core logic tests pass). 
- [x] Formatted code using `ruff`.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8203). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.