NyssaW/github-issues huggingface dataset:
`to_json()` serializes `Json()` feature as escaped string instead of native JSON object

### Describe the bug

When a dataset column uses the `Json()` feature type, calling `to_json()` writes the value as an escaped JSON string (e.g., `"{\"cat_a\":1,\"cat_b\":2}"`) instead of a native JSON object (e.g., `{"cat_a":1,"cat_b":2}`).

This causes two problems:

1. The output JSON file is not interoperable with other tools (pandas, jq, etc.) — they see a string, not an object.
2. Round-trip is broken: `load_dataset("json", ...)` without specifying `features=` infers the column as `Value('string')` instead of `Json()` or even a struct, because the file contains escaped strings.

In contrast, when the same dict data is stored as a struct type (auto-inferred), `to_json()` correctly writes it as a native JSON object.


### Steps to reproduce the bug

```
from datasets import Dataset, Features, Json, load_dataset

# Create dataset with Json() feature
data = [
    {"scores": {"cat_a": 1, "cat_b": 2}},
    {"scores": {"cat_x": 3, "cat_y": 4, "cat_z": 5}},
    {"scores": {"cat_a": 6, "cat_m": 7}},
]
ds = Dataset.from_list(data, features=Features({"scores": Json()}))

print("In-memory features:", ds.features)
# {'scores': Json(decode=True)}

print("In-memory data:", ds[0])
# {'scores': {'cat_a': 1, 'cat_b': 2}}

# Save to JSON
ds.to_json("test_json_feature.json")
```


1. Inspect the saved file `test_json_feature.json`:

```javascript
{"scores":"{\"cat_a\":1,\"cat_b\":2}"}
{"scores":"{\"cat_x\":3,\"cat_y\":4,\"cat_z\":5}"}
{"scores":"{\"cat_a\":6,\"cat_m\":7}"}
```

The `scores` field is a __escaped string__, not a native JSON object.

2. Reload without specifying features:

```python
ds_loaded = load_dataset("json", data_files="test_json_feature.json", split="train")
print("Loaded features:", ds_loaded.features)
# {'scores': Value('string')}

print("Loaded data:", ds_loaded[0])
# {'scores': '{"cat_a":1,"cat_b":2}'}  — it's a string, not a dict!
```

3. For comparison, struct type writes correctly:

```python
ds_struct = Dataset.from_list(data)
ds_struct.to_json("test_struct_feature.json")
```

`test_struct_feature.json` contains the correct native JSON:

```javascript
{"scores":{"cat_a":1,"cat_b":2,"cat_m":null,"cat_x":null,...}}
```


### Expected behavior

`to_json()` should serialize `Json()` columns as __native JSON objects__ (decoded form), not as escaped strings. The output file should look like:

```javascript
{"scores":{"cat_a":1,"cat_b":2}}
{"scores":{"cat_x":3,"cat_y":4,"cat_z":5}}
{"scores":{"cat_a":6,"cat_m":7}}
```

This would:

- Be consistent with how struct-type dict columns are serialized
- Produce interoperable JSON files readable by other tools
- Enable better round-trip behavior (at minimum auto-inferred as struct on reload)


### Environment info

- `datasets` version: 4.8.4

- Platform: Linux (Amazon Linux 2023, kernel 6.1)

- Python version: 3.13.7


Comments:
Hi! I ran into this issue as well, I think the root cause was that under `_batch_json()` of `JsonDatasetWriter` it was directly exporting the encoded string without decoding it first. I will try to add a PR with a small fix. 

#self-assign
fix: preserve features in IterableDataset.add_column (fixes #5752)

## Summary

Fixes #5752.

`IterableDataset.add_column()` lost the dataset's feature schema on every call because it delegated to `map()` without passing the `features` argument. `map()` unconditionally writes `info.features = features`, so the default `None` silently overwrote the existing schema.

**Root cause** (`iterable_dataset.py:3459`):
```python
info = self.info.copy()
info.features = features  # None when add_column() doesn't pass features
```

**Fix**: infer the new column's Arrow type via the existing `_infer_features_from_batch` helper, merge it with the existing features, and pass the result to `map()`.

```python
new_features = None
if self._info.features is not None:
    column_features = _infer_features_from_batch({name: list(column)})
    new_features = Features({**self._info.features, **column_features})
return self.map(..., features=new_features)
```

If the dataset had no features to begin with (`_info.features is None`), behaviour is unchanged — `map()` receives `None` as before.

## Test plan

- [ ] Reproduce the original issue: create a streaming dataset with known features, call `.add_column()`, assert `.features` is not None and contains the new column
- [ ] Verify existing `test_iterable_dataset.py` tests still pass
- [ ] Smoke-test with numpy array column input (not just list)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8115). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
🔒 Pin GitHub Actions to commit SHAs

## 🔒 Pin GitHub Actions to commit SHAs

This PR pins all GitHub Actions to their exact commit SHA instead of mutable tags or branch names.

**Why?**
Pinning to a SHA prevents supply chain attacks where a tag (e.g. `v4`) could be moved to point to malicious code.

### Changes

| Workflow | Action | Avant | Après | SHA |
|---|---|---|---|---|
| `trufflehog.yml` | `actions/checkout` | `v5` | `v6.0.2` | `de0fac2e4500…` |
| `trufflehog.yml` | `trufflesecurity/trufflehog` | `main` | `main` | `6bd2d14f7a4b…` |
| `ci.yml` | `actions/checkout` | `v4` | `v6.0.2` | `de0fac2e4500…` |
| `ci.yml` | `actions/setup-python` | `v5` | `v5` | `a26af69be951…` |
| `ci.yml` | `actions/checkout` | `v4` | `v6.0.2` | `de0fac2e4500…` |
| `ci.yml` | `actions/setup-python` | `v5` | `v5` | `a26af69be951…` |
| `ci.yml` | `conda-incubator/setup-miniconda` | `v2` | `v2` | `9f54435e0e72…` |
| `ci.yml` | `actions/checkout` | `v4` | `v6.0.2` | `de0fac2e4500…` |
| `ci.yml` | `actions/setup-python` | `v5` | `v5` | `a26af69be951…` |
| `ci.yml` | `conda-incubator/setup-miniconda` | `v2` | `v2` | `9f54435e0e72…` |
| `ci.yml` | `actions/checkout` | `v4` | `v6.0.2` | `de0fac2e4500…` |
| `ci.yml` | `actions/setup-python` | `v5` | `v5` | `a26af69be951…` |
| `ci.yml` | `conda-incubator/setup-miniconda` | `v2` | `v2` | `9f54435e0e72…` |
| `build_documentation.yml` | `huggingface/doc-builder/.github/workflows/build_main_documentation.yml` | `main` | `main` | `90b4ee2c10b8…` |
| `build_pr_documentation.yml` | `huggingface/doc-builder/.github/workflows/build_pr_documentation.yml` | `main` | `main` | `90b4ee2c10b8…` |
| `upload_pr_documentation.yml` | `huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml` | `main` | `main` | `90b4ee2c10b8…` |

> 🤖 Generated by `/github-actions-audit` — [security/pin-actions-to-sha]


Closes huggingface/tracking-issues#91

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8114). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Parse agent traces

Supports claude_code, codex and pi

JSON Lines files are parsed so one file = one row

The resulting row has three fields: harness, session_id, traces, and file_name with features:

```python
AGENT_TRACES_FEATURES = datasets.Features(
    {
        "harness": datasets.Value("string"),
        "session_id": datasets.Value("string"),
        "traces": datasets.List(datasets.Json()),
        "file_name": datasets.Value("string"),
    }
)
```
cc @cfahlgren1 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8113). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: replace List with Sequence in type hints for public API functions (#5354)

## Summary

Fixes #5354

Replaces `list[X]` with `Sequence[X]` in function argument type annotations 
across the public API. `List` is invariant while `Sequence` is covariant, 
which means passing a `list` to a parameter typed as `List` causes mypy errors, 
even though lists are valid sequences.

## Changes

- `src/datasets/arrow_dataset.py` — `path_or_paths`, `columns`, `input_columns`, `remove_columns`
- `src/datasets/iterable_dataset.py` — same parameters across equivalent methods  
- `src/datasets/combine.py` — `datasets` and `probabilities` parameters
- `src/datasets/dataset_dict.py` — `column_names` and `columns` parameters

## What was NOT changed

- Docstrings (separate concern)
- Internal variables and function bodies
- Return types
- Non-public/internal helpers

## Notes

PR #7953 addressed only `from_parquet` in `arrow_dataset.py` and had merge 
conflicts. This PR covers the broader public API consistently.

Comments:
Updated — public Dataset.map(), Dataset.filter(), IterableDataset.map() and IterableDataset.filter() signatures now also use Sequence[str] consistently. Thanks for the pointer.

Note: PR #8067 by @biefan also addresses path_or_paths in arrow_dataset.py. This PR covers a broader scope across 4 files and additional parameters (columns, input_columns, remove_columns, map(), filter()). Happy to coordinate with @biefan or adjust scope based on maintainer preference.
add sample_by="document" support for jsonl files

With more agentic traces (_ie Claude Code / Codex / Pi etc_) being published to the Hub as JSONL files, it would be nice to easily support loading / viewing them in Dataset Viewer. 

Datasets like https://huggingface.co/datasets/cfahlgren1/agent-sessions contain one JSONL file per session. Without sample_by, each line becomes a separate row, losing the file-level grouping. With sample_by="document", each file is preserved as a single row:

```python
  ds = load_dataset("cfahlgren1/agent-sessions", sample_by="document")
  # Each row is one session file
```

  Also configurable via YAML metadata:

```yaml
  ---
  configs:
    - config_name: default
      data_files: "*.jsonl"
      sample_by: "document"
  ---
```


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8111). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

would the auto-detection of agent harness (claude vs. codex vs. opencode etc) and other metadata, live in datasets in your opinion, @lhoestq? or in dataset-server, or in the frontend?

> would the auto-detection of agent harness (claude vs. codex vs. opencode etc) and other metadata, live in datasets in your opinion, @lhoestq? or in dataset-server, or in the frontend?

would be nice personally if we could have it live in datasets-server so we can do less parsing in frontend and add some sort of detection of trace datasets 

<img width="833" height="177" alt="image" src="https://github.com/user-attachments/assets/ab76e78e-14d4-417b-8322-44de2de9d415" />


closing this in favor of https://github.com/huggingface/datasets/pull/8113
`set_format` resets features

### Describe the bug

Calling `dataset.set_format('...')` resets the informative features such as `Array2D` etc. back into `List(List(Value))` and so on.

### Steps to reproduce the bug

```py
>>> dataset.features
{'lld': Array2D(shape=(None, 26), dtype='float32')}

>>> dataset.set_format('torch')
>>> dataset.features
{'lld': List(List(Value('float32')))}
```

### Expected behavior

The feature information is not lost during `set_format`, and the feature remains `Array2D` with the correct shape and dtype in the above example.

### Environment info


- `datasets` version: 4.8.2
- Platform: Linux-6.8.0-78-generic-x86_64-with-glibc2.39
- Python version: 3.13.11
- `huggingface_hub` version: 0.36.2
- PyArrow version: 23.0.1
- Pandas version: 2.3.3
- `fsspec` version: 2026.2.0


Comments:
couldn't replicate with: 

``` bash
Platform: macos
python: 3.10.9
datasets: 4.8.4
huggingface_hub: 0.36.2
pyarrow: 23.0.1
pandas: 2.3.3
fsspec: 2026.2.0

```


``` python
import datasets
from datasets import Array2D, Dataset
import numpy as np

# Create a dataset with Array2D feature
data = {"lld": [np.zeros((5, 26), dtype="float32").tolist() for _ in range(10)]}

features = datasets.Features({"lld": Array2D(shape=(None, 26), dtype="float32")})
dataset = Dataset.from_dict(data, features=features)

print("Before set_format:")
print(dataset.features)

dataset.set_format("torch")

print("\nAfter set_format('torch'):")
print(dataset.features)

```
docs: fix typos, grammar, and branding consistency across docs and source

This commit addresses several documentation quality issues found across the repository — fixing typos, grammar errors, brand name inconsistencies, and adding modern tooling references for new contributors.

## Changes

### README.md
- Fix duplicate word: "frameworks frameworks" → "frameworks"
- Standardize brand name: "HuggingFace Datasets Hub" → "Hugging Face Datasets Hub"
- Add `uv` installation section for faster environment setup

### CONTRIBUTING.md
- Add `uv pip install -e ".[dev]"` as an alternative setup command
- Fix grammar: "To do, go" → "To do so, go"
- Fix punctuation: trailing space before period in pre-commit note (`again .` → `again.`)
- Standardize brand name: "HuggingFace [code of conduct]" → "Hugging Face [code of conduct]"

### docs/source/stream.mdx
- Fix article usage: "a [`IterableDataset`]" → "an [`IterableDataset`]" (vowel sound rule)
- Fix code comment: "shuffles the shards order and use" → "uses" (subject-verb agreement)
- Fix phrase: "as soon one of the dataset runs out" → "as soon as one of the datasets runs out"
- Fix pluralization: "every samples in every dataset" → "every sample in every dataset"
- Fix abbreviation punctuation: "i.e the" → "i.e. the"

### docs/source/quickstart.mdx
- Standardize brand name: "a HuggingFace [`~datasets.Dataset`]" → "a Hugging Face [`~datasets.Dataset`]" (3 occurrences)

### docs/README.md
- Standardize copyright notice: "The HuggingFace Team" → "The Hugging Face Team"

### notebooks/README.md
- Standardize copyright notice: "The HuggingFace Team" → "The Hugging Face Team"

### src/datasets/iterable_dataset.py
- Fix typo in `map()` docstring: "simulatenous" → "simultaneous"
- Fix typo in `filter()` docstring: "simulatenous" → "simultaneous"
- Add return type hint to `identity_func`: `(x)` → `(x: Any) -> Any`
- Add return type hint to `_rename_columns_fn`: missing `-> dict` return type

Comments:
fix regression when loading JSON with one file = one object

was failing for datasets with JSON files that are not JSON Lines and that contain one single multi-line JSON object like https://huggingface.co/datasets/evaleval/EEE_datastore

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8086). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add reshard support for JSON Lines and CSV file formats

This PR extends the IterableDataset.reshard() functionality to support JSON Lines and CSV file formats, building on the foundation laid in #7992.

### Summary
- JSON Lines : Resharding splits files by line boundaries (only when num_shards is specified)
- CSV : Resharding splits files by line boundaries (only when num_shards is specified)
- Parquet : Already supported via row groups (enhanced to support num_shards parameter)


### Usage
```python
from datasets import load_dataset

# JSON Lines
ds = load_dataset("json", data_files=["data.jsonl"], streaming=True)
resharded_ds = ds['train'].reshard(num_shards=100)

# CSV
ds = load_dataset("csv", data_files=["data.csv"], streaming=True)
resharded_ds = ds['train'].reshard(num_shards=100)

# Parquet
ds = load_dataset("parquet", data_files=["data.parquet"], streaming=True)
resharded_ds = ds['train'].reshard(num_shards=100)
```

Comments:
@lhoestq , hi, lhoestq, could you please help review this PR? Thanks!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8084). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Cool ! I left a few comments. Mostly on CSV but this also applies to JSON since the logic is almost the same. Let me know what you think :)

@lhoestq Hi, lhoestq, I've made the updates accordingly, could you please take another look? 😊

Hi @lhoestq , just a gentle ping on this. I've re-verified the byte-offset logic for CSV/JSONL and confirmed it handles large files efficiently. Let me know if you have any concerns or would like me to adjust the implementation further. Looking forward to your feedback!
Fix split_dataset_by_node step

fix https://github.com/huggingface/datasets/issues/8079

I also took the opportunity to improve the arrow logic to not have to do an unnecessary python -> arrow -> python in FormattedExamplesIterable when streaming webdatasets

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8081). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix docstring of Json.cast_storage

Fix docstring of `Json.cast_storage`.

This PR makes a minor change to the `cast_storage` method in the `Json` feature, updating its return type and documentation to better reflect its intended functionality.

* Updated the `cast_storage` method to return a `pa.JsonArray` instead of a `pa.Int64Array`, and revised the docstring to accurately describe the return type.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8080). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Inconsistent loading of LFS-hosted files in Dangindev/viet-cultural-vqa dataset

### Describe the bug

Describe the bug
Some files in the Dangindev/viet-cultural-vqa dataset fail to load via the Hugging Face datasets library.

xet-hosted files load fine
LFS-hosted files sometimes fail

Example:

Fails: https://huggingface.co/datasets/Dangindev/viet-cultural-vqa/blob/main/images/am_thuc/banh_chung_Tet/000004.jpg
Works: https://huggingface.co/datasets/Dangindev/viet-cultural-vqa/blob/main/images/am_thuc/banh_bao/000001.jpg



### Steps to reproduce the bug

Steps to reproduce the bug

`from datasets import load_dataset

ds = load_dataset(
    "Dangindev/viet-cultural-vqa",
    data_files="images/am_thuc/banh_chung_Tet/000001.jpg",
)`

Or I can't access in my browser: 

<img width="1338" height="955" alt="Image" src="https://github.com/user-attachments/assets/e4c24c81-ca54-4405-af3b-83f76158da9b" />

### Expected behavior

I can download this dataset

### Environment info

Python or access through browser

Comments:
I also have been stuck in same problem 

This should be fixed, thanks for reporting ! let us know if you ever encounter other issues in the future

It's okay. 

Good job, thanks your support!
`.batch()` error on formatted datasets

The `.batch()` method currently assumes the input (batch) is always a dictionary, which causes errors when it isn't. This can happen with formatted datasets, since formats like `"pyarrow"`, `"pandas"` (only affects `IterableDataset`), and `"polars"` return tables/dataframes instead of dictionaries.

For example:
```python
from datasets import IterableDataset, Dataset
list(IterableDataset.from_dict({"a": [1, 2, 3, 4]}).with_format("pyarrow").batch(2))
# AttributeError: 'pyarrow.lib.Table' object has no attribute 'items'
```

Ideally, the result should be the same whether the format is applied before or after batching, i.e., the following should hold for all the format types:
```python
assert list(IterableDataset.from_dict({"a": [1, 2, 3, 4]}).with_format(format_type).batch(2)) == list(IterableDataset.from_dict({"a": [1, 2, 3, 4]}).batch(2).with_format(format_type))
assert list(Dataset.from_dict({"a": [1, 2, 3, 4]}).with_format(format_type).batch(2)) == list(Dataset.from_dict({"a": [1, 2, 3, 4]}).batch(2).with_format(format_type))
```

Comments:
Hi ! Good catch :) Since table-formatted iterable datasets have an Arrow path (i.e. they have `.iter_arrow()`) I guess a simple fix would be to make `.batch()` make the dataset arrow-formatted and use a `batch_arrow_fn` in that case (instead of the current `_batch_fn` that expects dictionaries). It would also be better performance-wise. How does that sound ?

Yes, and a similar approach can be used for the `Dataset` case: temporarily switching to the `"arrow"` format, running the batched `.map()` (which should also be much faster since it avoids Python ↔ Arrow conversions), and then restoring the original format.
fix null filling in missing jsonl columns

fix regression from #8027 reported [internally](https://huggingface.slack.com/archives/C04BP5S7858/p1773661170461759) cc @gary149 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8069). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
typing: accept sequences for Dataset file loaders

## Summary
- update `Dataset.from_csv`, `from_json`, `from_parquet`, and `from_text` type hints from `list[PathLike]` to `Sequence[PathLike]`
- normalize non-string sequences to lists before passing paths to readers
- add path-type tests to cover tuple input for all four loaders

## Why
`list` is invariant for static type checkers, so `list[str]` can be rejected against `list[PathLike]` even though `str` itself is valid path input. `Sequence[PathLike]` is covariant and better matches real usage.

## Validation
- `uv run --python 3.11 --with-editable . --with pytest --with setuptools --with absl-py -m pytest tests/test_arrow_dataset.py::test_dataset_from_csv_path_type tests/test_arrow_dataset.py::test_dataset_from_json_path_type tests/test_arrow_dataset.py::test_dataset_from_parquet_path_type tests/test_arrow_dataset.py::test_dataset_from_text_path_type -q`
- `uv run --python 3.11 --with ruff --with setuptools ruff check src/datasets/arrow_dataset.py tests/test_arrow_dataset.py`

Fixes #5354


Comments:
fix: avoid duplicate kwargs collisions in load_dataset_builder

## Summary
- avoid duplicate keyword collisions when `config_kwargs` overlap with module-provided `builder_kwargs` in `load_dataset_builder`
- this fixes cases like passing `base_path=...` where `base_path` is already present in `builder_kwargs`
- add a regression test using a local dataset directory to ensure overriding kwargs no longer raises

## Reproduction
Before this change:
```python
from datasets import load_dataset_builder
load_dataset_builder("/path/to/local/data", base_path="/path/to/local/data")
# TypeError: ... got multiple values for keyword argument 'base_path'
```

## Validation
- `uv run --python 3.11 --with-editable . --with pytest --with setuptools -m pytest tests/test_load.py::test_load_dataset_builder_config_kwargs_can_override_builder_kwargs tests/test_load.py::test_load_dataset_builder_for_absolute_data_dir -q`
- `uv run --python 3.11 --with ruff --with setuptools ruff check src/datasets/load.py tests/test_load.py`

Fixes #4910


Comments:
I believe this was fixed in https://github.com/huggingface/datasets/pull/8068
ARM audio support broken

### Describe the bug

`datasets` 4.0 moved to using `torchcodec` for audio datasets, which doesn't support ARM hardware under Linux and Windows


### Steps to reproduce the bug

try: 
* Try to install `torchcodec` under Linux ARM64

or:
* Try to install `torchcodec` under Windows ARM64

or, on either of those platforms, using datasets >= 4.0:
* `dataset = load_dataset("garak-llm/audio_achilles_heel")`


### Expected behavior

dataset will load as expected

### Environment info

ARM environment

Python 3.13+

`datasets>=4.0`

Using `torchaudio`

Comments:
See also https://github.com/NVIDIA/garak/issues/1622#issuecomment-3937720381

Hi @leondz , thanks for the detailed bug report — the description and reproduction steps are really clear.

I’d love to take a shot at investigating and potentially fixing this issue. Before I start digging into the codebase, I just wanted to check if there are any additional details, context, or pointers that might help get started (e.g., specific parts of the codebase related to this behavior).

Also, if there are any edge cases or constraints I should be aware of while reproducing the issue, that would be really helpful.

Thanks again for the thorough report!


torchcodec works on ARM, see the related discussion on https://github.com/meta-pytorch/torchcodec/issues/569

I see it's missing pure-CPU wheels though

Could you see with the torchcodec community what's missing for you to be able to use torchcodec ?

Thanks, also pursuing there. I want to check: in the interim, is is intended for `datasets`  be broken for audio under aarch64/woa64 when installed with `pip`, until `torchcodec` supply appropriate wheels?

If the question is: do we plan to implement a temporary workaround in the meantime, the answer is unfortunately no.
This would be a big effort to fix a temporary situation, while using older versions of `datasets` is actually a simpler workaround.

If older versions of `datasets` are not working for your case and you really have no other option, I'd be happy to review PRs that offer patches on older versions and I can make a patch release.
Read (and write) from HF Storage Buckets: load raw data, process and save to Dataset Repos

TODO:
- [x] load_dataset("csv", data_files=["hf://buckets/..."])
- [x] load_dataset("buckets/...")
- [x] Dataset.to_csv("hf://buckets/..."), to_json, to_parquet, etc.
- [x] DatasetDict.push_to_hub("buckets/...")
- [x] Dataset.push_to_hub("buckets/...")
- [x] DatasetDict.push_to_hub("buckets/...")
- [x] IterableDataset.push_to_hub("buckets/...")
- [x] IterableDatasetDict.push_to_hub("buckets/...")
- [x] tests
- [x] docs

e.g.

raw data in bucket to dataset repo

```python
from datasets import load_dataset

ds = load_dataset("json", data_files=["hf://buckets/..."])
ds = ds.map(...).filter(...)
ds.push_to_hub("username/dataset_name")
```


(also taking the opportunity to factorize the code a bit)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8064). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix formatted iter arrow double yield

I (+Claude) noticed this when reading the code. I did not encounter this bug (I think) so I can't say much more about it. Perhaps the second iteration will immediately stop so the bug is harmless, but to be safe I think a return there is safer.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8063). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Improve error message for invalid data_files pattern format

This PR improves the error message raised in sanitize_patterns()
when an invalid data_files entry is provided.

The new message clearly explains the expected dictionary format:
{'split': <split_name>, 'path': <path_or_list_of_paths>}

This helps users debug incorrect dataset configurations more easily.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8060). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix(arrow_dataset): initialise writer on first non-None map result, not on i==0

## Summary

`Dataset.map` crashes with `AttributeError: 'NoneType' object has no attribute 'write'` when the map function returns `None` for the first N examples and a dict for later ones (issue #7990).

## Root Cause

The `ArrowWriter` was initialised with a guard tied to the example index:

```python
# Non-batched path (line ~3736)
if i == 0:
    buf_writer, writer, tmp_file = init_buffer_and_writer()

# Batched path (line ~3762)
if i and i[0] == 0:
    buf_writer, writer, tmp_file = init_buffer_and_writer()
```

`update_data` is set lazily via `prepare_outputs()`: when the first example returns `None`, `update_data = False` and no writer is created. When example `i=2` returns a dict, `update_data` flips to `True`, but since `i != 0` the writer initialisation is skipped. The subsequent `writer.write(example)` call then crashes because `writer is None`.

## Fix

Replace the index-based guards with `if writer is None`:

```python
# Non-batched path
if writer is None:
    buf_writer, writer, tmp_file = init_buffer_and_writer()

# Batched path
if writer is None:
    buf_writer, writer, tmp_file = init_buffer_and_writer()
```

The writer is now created on the first example/batch that actually produces output, regardless of its position.

## Tests

Added `test_map_writer_initialized_when_first_examples_return_none` to `tests/test_arrow_dataset.py`: applies a map function that returns `None` for indices 0–1 and a transformed dict for the rest, verifying the call completes without error and produces correct output.

Fixes #7990


Comments:
fix(iterable_dataset): preserve features when chaining filter() on typed IterableDataset

## Summary

Chaining two `.filter()` calls on an `IterableDataset` with explicit features raises:

```
TypeError: 'NoneType' object is not a mapping
```

on the second `.filter()` call (issue #8037).

## Root Cause

After the first `.filter()`, the internal `ex_iterable` becomes _typed_ (`is_typed=True`) because `FilteredExamplesIterable` appends a boolean mask column. In `IterableDataset.filter()`, the `FormattedExamplesIterable` wrapper was constructed with:

```python
# Before (buggy)
features=None if ex_iterable.is_typed else self._info.features,
```

This set `FormattedExamplesIterable._features = None`. Then in the second `.filter()`, `FilteredExamplesIterable.__init__` tried to do:

```python
Features({**ex_iterable.features, self.mask_column_name: Value("bool")})
# ex_iterable.features == None  →  TypeError
```

## Fix

Forward the existing features when the iterable is already typed:

```python
# After (fixed)
features=ex_iterable.features if ex_iterable.is_typed else self._info.features,
```

One line change in `src/datasets/iterable_dataset.py`.

## Tests

Added `test_iterable_dataset_filter_chaining_does_not_raise` to `tests/test_iterable_dataset.py`: creates a typed `IterableDataset`, applies two chained `.filter()` calls, and asserts the result is correct without error.

Fixes #8037


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8053). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix IterableDataset features being set to None on chained filter (Fixes #8037)

This PR implements the solution suggested in #8037.

It updates the features argument in IterableDataset.filter() to preserve existing features when ex_iterable.is_typed is True. This prevents the TypeError: 'NoneType' object is not a mapping from being raised when chaining multiple .filter() calls.

Comments:
Thanks for the PR ! The original issue poster actually opened the same PR https://github.com/huggingface/datasets/pull/8053 but with additional tests, so I took the liberty of merging theirs - I hope you don't mind
Support 3D Datasets

### Feature request

In `datasets` library we have options to load `image` and `audio` datasets which can be viewed in the Dataset Viewer.

LIKE:
```bash
from datasets import load_dataset, Image
from datasets import load_dataset, Audio
```
> https://huggingface.co/docs/datasets/en/image_load & https://huggingface.co/docs/datasets/en/audio_process

---

I guess there's a need for supporting 3D datasets as well for *storing* & *loading* 3D object files like (`.glb, .ply, .obj` etc...) maybe with something like:

```bash
from datasets import load_dataset, Mesh # or `_3D` or whatever is naming convention
```

### Motivation

Multi-Modal datasets are rapidly increasing with modalites like 3D, Video etc, `datasets` already support experimental `Video` import. 

I have a dataset where I've stored everything as dict so it atleast look balanced across all those 4 columns and 4 modalites It have (Text, Image, Audio, 3D Mesh) but it don't look that good since RAW binary bytes are visible. And Needs to be decoded for that specific field containing the binary data. 

### Your contribution

Perhaps I could have made a PR for this, but since things might not get much attention here, I shouldn't...

Contributors can try to support it so the the huggingface server is not at heavy duty of loading preview images for each mesh in the dataset viewer that can crash. So a cool svg Icon saying 3D or Mesh inplace of preview can be good. OR they can still try to render an image from those 3D data and use it in preview place just like Image/Video.

Edit:

SO Working on PR for this, gonna push soon...

PUSHED THE PR: #8055 


Comments:
fix: handle nested null types in feature alignment for multi-proc map

When using `Dataset.map()` with `num_proc > 1`, shards containing only empty lists get their features inferred as `List(Value('null'))`, which is incompatible with `List(Struct({...}))` from shards with actual data.

The existing `_check_if_features_can_be_aligned` and `_align_features` only handled top-level `Value('null')` compatibility. This commit adds a `_is_null_feature()` helper that recursively checks for null types inside container features (`List, LargeList, Sequence`), and uses it in both alignment functions.

Fixes #8046

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8047). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`Dataset.map()` with `num_proc > 1` fails when some shards produce `List(Value('null'))` (empty lists)

### Describe the bug

When using `Dataset.map()` with `num_proc > 1`, if the map function produces a list column where some shards contain only empty lists `[]` and other shards contain non-empty lists of structs, the feature type inference is inconsistent across shards:

- Shards with only empty lists → `List(Value('null'))`
- Shards with non-empty lists → `List(Struct({...}))`

During shard concatenation, `_check_if_features_can_be_aligned` raises a `ValueError` because it only handles top-level `Value("null")` compatibility but does **not** handle nested null types like `List(Value('null'))`.

Setting `num_proc=1` works correctly because all data is processed in a single shard.

### Steps to reproduce the bug

```python
import datasets
print(datasets.__version__)  # 4.6.1

data = {
    "id": [f"item_{i}" for i in range(8)],
    "labels": [
        [{"type": "A", "word": "x", "start": 0, "end": 5}],  # only first item has data
        [], [], [], [], [], [], []  # rest are empty
    ]
}
ds = datasets.Dataset.from_dict(data)

# Works fine with num_proc=1
result = ds.map(lambda x: {"label": x["labels"]}, num_proc=1)
print(result.features)  # OK

# Fails with num_proc=2
result = ds.map(lambda x: {"label": x["labels"]}, num_proc=2)
# ValueError: The features can't be aligned because the key label of features
# ... has unexpected type - List(Value('null'))
# (expected either List({'end': Value('int64'), ...}) or Value("null").
```


### Expected behavior

`List(Value('null'))` should be treated as compatible with any `List(...)` type during feature alignment, similar to how `Value('null')` is already treated as compatible with any other type.

### Environment info

- `datasets` version: 4.6.1 (also reproduced on 4.1.1)
- Python version: 3.12.0
- OS: Linux

Comments:

### Root cause

In `src/datasets/features/features.py`, both `_check_if_features_can_be_aligned` and `_align_features` use:

```python
isinstance(v, Value) and v.dtype == "null"
```

to check for null compatibility. However, `List(Value('null'))` is **not** an instance of `Value` — it's an instance of `Sequence` (via `List`). So this check misses any container type wrapping a null, such as `List(Value('null'))`, `Sequence(Value('null'))`, or `LargeList(Value('null'))`.

### Suggested fix

Add a recursive null check function and use it in place of the current top-level-only check:

```python
def _is_null_feature(feature) -> bool:
    """Recursively check if a feature is a null type."""
    if isinstance(feature, Value) and feature.dtype == "null":
        return True
    if isinstance(feature, (Sequence, LargeList)) and hasattr(feature, "feature"):
        return _is_null_feature(feature.feature)
    return False
```

Then replace all occurrences of `isinstance(v, Value) and v.dtype == "null"` with `_is_null_feature(v)` in both `_check_if_features_can_be_aligned` and `_align_features`.
Write IterableDataset to parquet incrementally instead of materializing entire shard in memory

`IterableDataset.to_parquet()` materialized the entire shard into memory via
`pa.concat_tables(list(...))` before writing. For datasets with large shards and multiprocessing (push_to_hub with num_proc > 0) the memory requirements can be really large.

This PR writes parquet incrementally by iterating Arrow batches and writing each
directly to `pq.ParquetWriter`, reducing the memory requirement from shard size to something around row group size (with lots of caveats).

**Note**

A thing to note is that this code path is only executed when `features` are set on the `IterableDataset`, in f.ex. `map` or the dataset is `cast` before `push_to_hub`.

Changes:
- Extend `ParquetDatasetWriter` to accept both `Dataset` and `IterableDataset`.
- `IterableDataset.to_parquet()` now delegates directly to `ParquetDatasetWriter`
- Use `pq.ParquetWriter` as a context manager for proper cleanup on errors
- Falls back to materialization only when `features` is `None` (schema required upfront)

That being said, I'm not entirely convinced that extending the ParquetDatasetWriter to support IterableDataset is the correct approach or even if there are other footguns related to CDC or other stuff.

Comments:
I've tested this change along with the changes from my other PRs (https://github.com/HaukurPall/datasets/tree/all-fixes) and this change **dramatically** reduces the memory requirement per process.

Context:

I'm working with audio datasets, the audio is embedded in the parquet. When loading a 1GB shard with embedded mp3 audio, decoding the audio and writing it back as wav, the shard size increases to around 9GB. I run this processing with num_proc > 1, so the total memory required is large. With this change, the memory required per process drops from ~25GB to ~4GB for this use-case.

@lhoestq any feedback?
Fix silent data loss in push_to_hub when num_proc > num_shards

## Summary

When `num_proc` exceeds the number of output shards in `Dataset.push_to_hub`, workers
without assigned output shards silently drop their data. The metadata still reports the
correct dataset length (using `len(self[split])`), masking the data loss.

This fix caps `num_jobs` at `num_shards` in `_push_parquet_shards_to_hub`, matching
what the streaming path already does.

## Reproduction

1. Have a dataset with a small number of shards (e.g. 2)
2. Call `push_to_hub` with `num_proc > num_shards` (e.g. 6)
3. Only a fraction of the samples are actually written to the hub

Comments:
Add features parameter to IterableDatasetDict.map

IterableDataset.map accepts a features parameter to declare the output schema, but IterableDatasetDict.map did not expose it. This meant users of IterableDatasetDict had no way to preserve feature metadata through map operations.

Comments:
I'm going to close this PR as it depends on correct `features` handling in `IterableDataset`, which is not done correctly at the moment.
Fix schema enforcement in streaming _convert_to_arrow

When the iter_arrow chain is broken (e.g. after .map() without with_format("arrow")), _convert_to_arrow falls back to pa.Table.from_pylist() which infers types from values. If early batches contain None or empty lists, Arrow infers null/list<null> types that conflict with later batches containing real data, causing ArrowInvalid schema mismatch errors.

Add an optional features parameter to _convert_to_arrow that applies cast_table_to_features to each produced table, mirroring what ArrowWriter.write_table does in the map-style path.

Comments:
I'm going to close this PR as I have a more complete branch which fixes feature handling in the streaming case. The `features` handling in the streaming case is btw completely broken at the moment, it just happens to work in some cases. I will add more test cases to show it in a later PR.
Typo in iterable_dataset.py

### Describe the bug

`streaming=True**kwargs,` should be `streaming=True, **kwargs,`
https://github.com/huggingface/datasets/blob/81027be09d5cd9f06a6d64ef1a8a3e9ebd0f86fd/src/datasets/iterable_dataset.py#L2950

### Steps to reproduce the bug

```
from Datasets import Dataset
IterableDataset.from_csv("file.csv")
```

### Expected behavior

Load the csv file as an iterable dataset.

### Environment info

- `datasets` version: 4.6.1
- Platform: Linux-6.14.0-37-generic-x86_64-with-glibc2.39
- Python version: 3.13.11
- `huggingface_hub` version: 1.5.0
- PyArrow version: 23.0.1
- Pandas version: 3.0.1
- `fsspec` version: 2026.2.0

Comments:
IterableDataset.filter() chaining fails due to features being lost when is_typed=True

### Describe the bug

Chaining multiple .filter() calls on an IterableDataset raises a TypeError: 'NoneType' object is not a mapping on the second .filter() call.

In `IterableDataset.filter()`, the internal `ex_iterable` is wrapped with FormattedExamplesIterable:

```python
ex_iterable = FormattedExamplesIterable(
    ex_iterable,
    formatting=self._formatting,
    features=None if ex_iterable.is_typed else self._info.features,
    token_per_repo_id=self._token_per_repo_id,
)
```

After the first .filter(), the resulting ex_iterable becomes typed (is_typed=True) because a ===MASK=== column is added. On the second .filter(), since is_typed=True, features=None is passed to FormattedExamplesIterable. This causes .features to return None, and FilteredExamplesIterable.__init__ fails when trying to unpack it.

The fix should preserve the existing features when is_typed=True, e.g.:

```python
features=ex_iterable.features if ex_iterable.is_typed else self._info.features
```

### Steps to reproduce the bug

```python
from datasets import load_dataset

ds = load_dataset("bigcode/the-stack-v2", "Scala", split="train", streaming=True)
ds = ds.filter(lambda row: row["src_encoding"] == "UTF-8")
ds = ds.filter(lambda row: row["branch_name"] == "refs/heads/master")  # TypeError here
```

## Error traceback
```
File ".../datasets/iterable_dataset.py", line 2975, in filter
    ex_iterable = FilteredExamplesIterable(
File ".../datasets/iterable_dataset.py", line 1606, in __init__
    features = Features({**ex_iterable.features, self.mask_column_name: Value("bool")})
TypeError: 'NoneType' object is not a mapping
```

### Expected behavior

The error should not raised in this context.

### Environment info

datasets: 4.5.0

Comments:
perf: use deque for async map task/index queues

## Problem

`arrow_dataset.py` and `iterable_dataset.py` drain `indices` and `tasks` lists front-to-back via `.pop(0)` during parallel async map operations. With `MAX_NUM_RUNNING_ASYNC_MAP_FUNCTIONS_IN_PARALLEL` items in flight, each `.pop(0)` is **O(n)** making the drain loop **O(n²)**.

## Solution

Switch both `indices` and `tasks` to `collections.deque` with `.popleft()` for **O(1)** front removal.

## Changes

- `src/datasets/arrow_dataset.py`: Import `deque`, type `tasks` and `indices` as deque, use `.popleft()`
- `src/datasets/iterable_dataset.py`: Import `deque`, type `tasks`, `indices`, and `_owned_loops_and_tasks` entry as deque, use `.popleft()`

## Testing

- Syntax verified via `ast.parse()` on both files

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8033). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Friendly ping — CI is green and this is ready for review. Happy to address any feedback. Thanks!
Fix torchcodec audio decoding to respect 'num_channels'

Fixes torchcodec audio decoding when `num_channels` is set on `Audio`.

Before this change, `AudioDecoder["array"]` reduced multi-channel audio to mono by averaging channels, so the requested channel behavior was not respected.

With this PR:
- multi-channel decoded arrays are preserved by default;
- mono output is returned only when `num_channels == 1` is explicitly requested.

#### Previous behavior
`ds_stereo = ds.cast_column("audio", Audio(num_channels={...}))`  - None, 1, 2

Original file: '(16000, 2)' - stereo ✓  
'Audio(num_channels=None)': '(16000,)' - mono ✗  
'Audio(num_channels=2)': '(16000,)' - mono ✗  
'Audio(num_channels=1)': '(16000,)' - mono ✓

#### New behavior
'num_channels=None' preserves the original number of channels from the source file.
'num_channels=2' preserves/converts to stereo output with shape '(2, num_samples)'.
'num_channels=1' downmixes to mono with shape '(num_samples,)'.

#### Results
**Original file shape** (via soundfile): (16000, 2)
HF datasets shape with num_channels=None: (2, 16000)
HF datasets shape with num_channels=1: (16000,)
HF datasets shape with num_channels=2: (2, 16000)


Fixes #8005.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8028). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add `Json()` type

`Json()` type is needed when some fields don't have fixed types, e.g. "tools" and "tool_calls" types in conversation + tool calling datasets like [dataclaw](https://huggingface.co/datasets?other=dataclaw) datasets. Cc @peteromallet and @woctordho @Nanbeige for viz

Examples of supported tool-calling / dataclaw datasets:
```python
ds = load_dataset("peteromallet/dataclaw-peteromallet")  #  happens to not need Json() since tool types are fixed
ds = load_dataset("woctordho/dataclaw")
ds = load_dataset("Nanbeige/ToolMind")
```

The `Json()` type is auto-applied when loading JSON files when mixed types are found. The `Json()`type is set to the list containing objects with mixed types, to end up with a `List(Json())` type for "tools" and "tool_calls" columns.

It is also possible to define a Dataset with a `Json()` type like this:

```python
>>> from datasets import load_dataset, Dataset, Features, Json, List
>>> example = {"a": [{"key": 0}, {"another-key": "another-type"}]}
>>> features = Features({"a": List(Json())})
>>> ds = Dataset.from_list([example], features=features)
>>> ds[0]
{'a': [{'key': 0}, {'another-key': 'another-type'}]}
```

close https://github.com/huggingface/datasets/issues/7869
close https://github.com/huggingface/datasets/issues/4120
close https://github.com/huggingface/datasets/issues/5827
close https://github.com/huggingface/datasets/issues/7418
related to https://github.com/huggingface/datasets/issues/7092
related to https://github.com/huggingface/datasets/issues/6162
related to https://github.com/huggingface/datasets/issues/2799
related to https://huggingface.co/datasets/PatoFlamejanteTV/LocalLLaMA/discussions/1

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8027). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: resolve base_path conflict between builder_kwargs and config_kwargs

## What does this PR fix?

This PR fixes a bug where passing `base_path` to `load_dataset_builder()` or `load_dataset()` causes a TypeError because both `builder_kwargs` and `config_kwargs` can contain the same key.

## Problem

When users call:
```python
from datasets import load_dataset
load_dataset("rotten_tomatoes", base_path="./sample_data")
```

They get:
```
TypeError: type object got multiple values for keyword argument "base_path"
```

This happens because `base_path` can exist in both `builder_kwargs` (from the dataset module) and `config_kwargs` (from user input), causing a conflict when both are unpacked.

## Solution

- Pop `base_path` from `builder_kwargs` before merging with `config_kwargs`
- Explicitly pass `base_path` to the builder constructor

## Testing

The fix should resolve the issue demonstrated in #4910:
```python
from datasets import load_dataset
load_dataset("rotten_tomatoes", base_path="./sample_data")  # Now works!
```

Fixes #4910

Comments: