noamaanMulla-03/datasets-issues huggingface dataset:
`to_json()` serializes `Json()` feature as escaped string instead of native JSON object

### Describe the bug

When a dataset column uses the `Json()` feature type, calling `to_json()` writes the value as an escaped JSON string (e.g., `"{\"cat_a\":1,\"cat_b\":2}"`) instead of a native JSON object (e.g., `{"cat_a":1,"cat_b":2}`).

This causes two problems:

1. The output JSON file is not interoperable with other tools (pandas, jq, etc.) — they see a string, not an object.
2. Round-trip is broken: `load_dataset("json", ...)` without specifying `features=` infers the column as `Value('string')` instead of `Json()` or even a struct, because the file contains escaped strings.

In contrast, when the same dict data is stored as a struct type (auto-inferred), `to_json()` correctly writes it as a native JSON object.


### Steps to reproduce the bug

```
from datasets import Dataset, Features, Json, load_dataset

# Create dataset with Json() feature
data = [
    {"scores": {"cat_a": 1, "cat_b": 2}},
    {"scores": {"cat_x": 3, "cat_y": 4, "cat_z": 5}},
    {"scores": {"cat_a": 6, "cat_m": 7}},
]
ds = Dataset.from_list(data, features=Features({"scores": Json()}))

print("In-memory features:", ds.features)
# {'scores': Json(decode=True)}

print("In-memory data:", ds[0])
# {'scores': {'cat_a': 1, 'cat_b': 2}}

# Save to JSON
ds.to_json("test_json_feature.json")
```


1. Inspect the saved file `test_json_feature.json`:

```javascript
{"scores":"{\"cat_a\":1,\"cat_b\":2}"}
{"scores":"{\"cat_x\":3,\"cat_y\":4,\"cat_z\":5}"}
{"scores":"{\"cat_a\":6,\"cat_m\":7}"}
```

The `scores` field is a __escaped string__, not a native JSON object.

2. Reload without specifying features:

```python
ds_loaded = load_dataset("json", data_files="test_json_feature.json", split="train")
print("Loaded features:", ds_loaded.features)
# {'scores': Value('string')}

print("Loaded data:", ds_loaded[0])
# {'scores': '{"cat_a":1,"cat_b":2}'}  — it's a string, not a dict!
```

3. For comparison, struct type writes correctly:

```python
ds_struct = Dataset.from_list(data)
ds_struct.to_json("test_struct_feature.json")
```

`test_struct_feature.json` contains the correct native JSON:

```javascript
{"scores":{"cat_a":1,"cat_b":2,"cat_m":null,"cat_x":null,...}}
```


### Expected behavior

`to_json()` should serialize `Json()` columns as __native JSON objects__ (decoded form), not as escaped strings. The output file should look like:

```javascript
{"scores":{"cat_a":1,"cat_b":2}}
{"scores":{"cat_x":3,"cat_y":4,"cat_z":5}}
{"scores":{"cat_a":6,"cat_m":7}}
```

This would:

- Be consistent with how struct-type dict columns are serialized
- Produce interoperable JSON files readable by other tools
- Enable better round-trip behavior (at minimum auto-inferred as struct on reload)


### Environment info

- `datasets` version: 4.8.4

- Platform: Linux (Amazon Linux 2023, kernel 6.1)

- Python version: 3.13.7



Hi! I ran into this issue as well, I think the root cause was that under `_batch_json()` of `JsonDatasetWriter` it was directly exporting the encoded string without decoding it first. I will try to add a PR with a small fix. 

#self-assign
Fix batching for table-formatted datasets

## Summary
Fix #8075

Fix `.batch()` on table-formatted `Dataset` and `IterableDataset`.

Before this change, calling `.batch()` on datasets formatted as `pyarrow`, `pandas`, or `polars` could fail because the batching path assumed dict-like inputs. This updates batching to use an Arrow-based path for table-style formats, so batching works regardless of whether the table format is applied before or after `.batch()`. These two forms now behave equivalently:

```python
dataset.with_format(format_type).batch(n)
```

and

```python
dataset.batch(n).with_format(format_type)
```

for `pyarrow`, `pandas`, and `polars`.

## What changed

- added a shared `_batch_arrow_table` helper to build batched Arrow tables
- updated `Dataset.batch()` to route table-formatted batching through an Arrow `.map(...)` path
- updated `IterableDataset.batch()` to do the same and then restore the original table format
- added regression tests for `pyarrow`, `pandas`, and `polars` on both `Dataset` and `IterableDataset`

## Tests

- `python -m pytest tests/test_arrow_dataset.py -k "test_dataset_batch or test_dataset_batch_with_table_format or test_dataset_batch_with_polars_format" -q`
- `python -m pytest tests/test_iterable_dataset.py -k "test_iterable_dataset_batch or test_iterable_dataset_batch_with_table_format or test_iterable_dataset_batch_with_polars_format" -q`
- `python -m ruff check src/datasets/arrow_dataset.py src/datasets/iterable_dataset.py src/datasets/table.py tests/test_arrow_dataset.py tests/test_iterable_dataset.py`


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8126). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`.batch()` error on formatted datasets

The `.batch()` method currently assumes the input (batch) is always a dictionary, which causes errors when it isn't. This can happen with formatted datasets, since formats like `"pyarrow"`, `"pandas"` (only affects `IterableDataset`), and `"polars"` return tables/dataframes instead of dictionaries.

For example:
```python
from datasets import IterableDataset, Dataset
list(IterableDataset.from_dict({"a": [1, 2, 3, 4]}).with_format("pyarrow").batch(2))
# AttributeError: 'pyarrow.lib.Table' object has no attribute 'items'
```

Ideally, the result should be the same whether the format is applied before or after batching, i.e., the following should hold for all the format types:
```python
assert list(IterableDataset.from_dict({"a": [1, 2, 3, 4]}).with_format(format_type).batch(2)) == list(IterableDataset.from_dict({"a": [1, 2, 3, 4]}).batch(2).with_format(format_type))
assert list(Dataset.from_dict({"a": [1, 2, 3, 4]}).with_format(format_type).batch(2)) == list(Dataset.from_dict({"a": [1, 2, 3, 4]}).batch(2).with_format(format_type))
```


Hi ! Good catch :) Since table-formatted iterable datasets have an Arrow path (i.e. they have `.iter_arrow()`) I guess a simple fix would be to make `.batch()` make the dataset arrow-formatted and use a `batch_arrow_fn` in that case (instead of the current `_batch_fn` that expects dictionaries). It would also be better performance-wise. How does that sound ?

Yes, and a similar approach can be used for the `Dataset` case: temporarily switching to the `"arrow"` format, running the batched `.map()` (which should also be much faster since it avoids Python ↔ Arrow conversions), and then restoring the original format.
`load_dataset()` hangs when `hf_xet` is enabled

### Describe the bug

Hi! I am getting a peculiar download issue on the latest versions of `datasets` and `hfh` where datasets hang indefinitely:

```py
  from datasets import load_dataset
  load_dataset("openai/gsm8k", "main", split="test")
```

Note that:

  - streaming=True works.
  - Direct Hub access works.
  - The issue disappears when `HF_HUB_DISABLE_XET=1` is set.

  This looks like the parquet download/materialization path is getting stuck specifically
  when the Xet backend is active. Setting HF_HUB_DISABLE_XET=1 is a working workaround.

### Steps to reproduce the bug

First install the latest versions of the libs:

```sh
pip install -U datasets huggingface_hub
```

Then run

```sh
python -c 'from datasets import load_dataset; print(load_dataset("openai/gsm8k", "main", split="test"))'
```

Note that the following works:

```sh
HF_HUB_DISABLE_XET=1 python -c 'from datasets import load_dataset; print(load_dataset("openai/gsm8k", "main", split="test"))'
```

### Expected behavior

The dataset downloads without hanging

### Environment info

  Environment:

  - datasets==4.8.4
  - huggingface_hub==1.10.1
  - hf_xet installed


Apologies, seems to have been a stale cache issue. Nuking the datasets cache fixed the issue
Fix Audio.cast_storage for CSV-loaded large_string columns

## Summary
This PR fixes `Audio.cast_storage()` so that audio path columns loaded as `large_string` can be cast to `Audio()` correctly.

## Motivation
`Dataset.from_dict({"audio": [...]})` produces a `string` column and `cast_column("audio", Audio())` works, but `load_dataset("csv", ...)` produces a `large_string` column, which currently fails during the cast to the Audio storage type.

## Changes made
- Added handling for `pa.large_string()` in `Audio.cast_storage()`
- Cast large string storage to `pa.string()` before building the Audio struct storage
- Added a regression test covering the CSV-loaded audio path case

## Testing
- [x] Reproduced the issue locally
- [x] Ran the targeted regression test for the CSV-loaded `large_string` case
- [x] Ran `tests/features/test_audio.py` in the local environment

Closes #7970


Thanks for the investigation and the fix ! This makes me think that maybe load_dataset("csv") should return "string" types instead of "large_string" to be consistent with load_dataset("json") as well as the Dataset.from_xxx methods

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8121). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the review and the suggestions.

I’ll update the PR to:

* add the same `large_string` handling for `Video` and `Pdf`
* update the docstrings to mention `pa.large_string()`
* switch the audio test to use the existing `audio_path` fixture
* fix the code quality check failure


I have updated the PR to:

* add the same `large_string` handling for `Video` and `Pdf`
* update the docstrings to mention `pa.large_string()`
* keep the `Audio` fix
* add matching regression tests for `Audio`, `Video`, and `Pdf`

I also fixed the code quality issue and reran the targeted feature tests locally.


Great ! It looks like you didn't commit the change to switch the audio test to use the existing audio_path fixture and to update the Audio docstring, could you add your changes before we merge ?

Thanks - I updated the audio regression test to use the existing audio fixture available in the test suite and added the missing `Audio` docstring update for `pa.large_string()`.
fix: mutable default argument in `_visit_with_path`

Replace `visit_path: _VisitPath = []` with `Optional[_VisitPath] = None` to avoid the mutable default argument anti-pattern ([B006](https://docs.astral.sh/ruff/rules/mutable-argument-default/)).

While the current callers always use `visit_path + [k]` (creating new lists), the shared default list is a latent bug that would surface if any future code path mutates `visit_path` in place (e.g. via `.append()`).

The fix initializes `visit_path = []` inside the function body when `None` is passed.
progress bar optional manual control

Fix: https://github.com/huggingface/datasets/issues/7939

Summary

The default behavior for the progress bar has been adjusted based on dataset size. It remains enabled for small datasets and is automatically disabled for larger datasets to improve performance. Users retain full control and can explicitly enable or disable the progress bar using the progress_bar parameter.

Changes

Modified the logic to disable the progress bar when the number of files exceeds 16.


Hi @lhoestq ,
Can you please trigger the CI CD pipeline ?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8021). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq I have applied the required Ruff formatting changes in the` load.py` file.
Could you please trigger the CI/CD pipeline again?

HI @lhoestq ,
Just a gentle reminder to please trigger the CI/CD pipeline for this PR..!!

HI @julien-c ,
can you please trigger the the CI/CD pipeline for this PR.

Hi @albertvillanova ,
Could you please trigger the CI/CD pipeline?, i can see it is pending from long time 
Thanks 

I don't think we want to add more arguments to this method.
Progress bars can already be disabled like this

```python
from datasets.utils import disable_progress_bars

disable_progress_bars()
```

Hey @lhoestq ,
My intention was to provide more fine-grained control at the method level, particularly for multiprocessing scenarios where multiple progress bars can clutter the logs. This would give users the flexibility to explicitly enable or disable the progress bar as needed,which aligns with the concern raised in the issue https://github.com/huggingface/datasets/issues/7939
use full repo-relative file path instead of filename for agent traces

We want to include full path of the file so it's easier to fetch the full trace from repo

  - Store the full repo-relative path in `file_name` instead of just the basename
  - Extracts the path from the `hf://` origin URL set by the download manager

  Before: `file_name = "614b8d50-097f-4e36-aee0-1fdd053335e7.jsonl"`
  After:  `file_name = "sessions/claude/614b8d50-097f-4e36-aee0-1fdd053335e7.jsonl"`


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8133). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

thanks ! let me push some adjustments, and yea let's use file_path instead
Don't include files list DatasetInfo (and remove old stuff)

this will fix the viewer when it saves the dataset info in mongo, for datasets with a lot of files this can make the mongo doc too big for mongo

I also removed old deprecated stuff linked to files list like the old concept of "post-processing" and cleaned the checksum recording logic


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8128). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix warning in ci

otherwise it raises an issue from transformers smh

```
../../.pyenv/versions/3.12.2/envs/hf-datasets/lib/python3.12/site-packages/transformers/__init__.py:843: in _getattr
    return getattr(importlib.import_module(target, __name__), new_name)
../../.pyenv/versions/3.12.2/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1387: in _gcd_import
    ???
<frozen importlib._bootstrap>:1360: in _find_and_load
    ???
<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:935: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:995: in exec_module
    ???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    #                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
    #           This file was automatically generated from src/transformers/models/aria/modular_aria.py.
    #               Do NOT edit this file manually as any edits will be overwritten by the generation of
    #             the file from the modular. If any change should be done, please apply the change to the
    #                          modular_aria.py file directly. One of our CI enforces this.
    #                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
    # Copyright 2024 The Rhymes-AI Teams Authors and The HuggingFace Inc. team. All rights reserved.
    #
    # Licensed under the Apache License, Version 2.0 (the "License");
    # you may not use this file except in compliance with the License.
    # You may obtain a copy of the License at
    #
    #     http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.
    import torch
>   from torchvision.transforms.v2 import functional as tvF
E   ModuleNotFoundError: No module named 'torchvision'
```


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8140). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
chore: bump doc-builder SHA for main doc build workflow

Bump the pinned doc-builder SHA so that main documentation builds also sync to the HF bucket (dual-write).



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8138). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Closing — not needed right now, active libs already on @main.
`load_dataset` `revision` param not respected when fetching from cache

### Describe the bug

`datasets.load_dataset` `revision` semantics are a bit inconsistent when the dataset is not found on the huggingface hub. When fetching the latest cached version of the dataset, the `revision` argument is ignored, so long as any cached versions of the dataset already exist in the HF cache.

### Steps to reproduce the bug

```python
import datasets
datasets.load_dataset(
    "sentientfutures/ahb",
    "dimensions",
    split="train",
    revision="main"
)
# would expect some error to raise here
datasets.load_dataset(
    "sentientfutures/ahb",
    "dimensions",
    split="train",
    revision="invalid_revision"
)
```



### Expected behavior

On the second call to `datasets.load_dataset` in the 'steps to reproduce the bug' example, expect something like:

```sh
raise DatasetNotFoundError(
datasets.exceptions.DatasetNotFoundError: Revision 'invalid_revision' doesn't exist for dataset 'sentientfutures/ahb' on the Hub.
```

### Environment info

- `datasets` version: 4.4.1
- Platform: Linux-6.2.0-39-generic-x86_64-with-glibc2.37
- Python version: 3.12.12
- `huggingface_hub` version: 0.36.0
- PyArrow version: 22.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2025.9.0


This might be better placed as a feature request not a bug, since the logging `Using the latest cached version of the dataset since sentientfutures/ahb couldn't be found on the Hugging Face Hub` is clear.

https://github.com/huggingface/datasets/pull/7929 This only solves the case of invalid revisions. Fetching a specific revision from the cache would be more work but I think this is a good start and solves issues like https://github.com/UKGovernmentBEIS/inspect_evals/pull/834#issuecomment-3704689637

#self-assign
fix(Dataset.map): initialize writer on first non-None output instead of first index

## What does this PR do?

Fixes #7990

## The Bug

When a map function returns `None` for the first examples and a dict for
later ones, `Dataset.map()` crashes with:

    AttributeError: 'NoneType' object has no attribute 'write'

This happens because the ArrowWriter was initialized only when processing
index 0. If index 0 returns `None`, `update_data` is False at that point,
so the writer is never created. When a later example returns a dict,
`update_data` becomes True but the writer is still None.

## The Fix

Replace index-based checks with `writer is None`:

**Before:**
if i == 0:          # non-batched
if i and i[0] == 0: # batched
    buf_writer, writer, tmp_file = init_buffer_and_writer()

**After:**
if writer is None:  # both cases
    buf_writer, writer, tmp_file = init_buffer_and_writer()

## Reproduction

```python
from datasets import Dataset

ds = Dataset.from_dict({"x": [1, 2, 3]})

def fn(example, idx):
    if idx < 2:
        return None
    return {"x": [example["x"] * 10]}

list(ds.map(fn, with_indices=True))  # Previously crashed, now works

Tests
Non-batched: first examples return None ✅
Batched: first batch returns None ✅
All examples return None → dataset unchanged ✅
Normal map with no None returns ✅
tag openclaw sessions via cwd in pi agent traces

Tag OpenClaw datasets from Pi agent sessions. Will add a PR to add icon for it in moon too

cc: @merveenoyan 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8143). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: tolerate chmod PermissionError on flat-permission filesystems

## Summary

Fixes #8125.

`Dataset.map()` (and the indices writer in `Dataset.flatten_indices` / `Dataset.select` path) calls `os.chmod(cache_file, 0o666 & ~umask)` on the freshly moved cache file. On flat-permission filesystems — GCS FUSE, S3 FUSE mounts, and similar object-storage-backed mounts — `chmod` raises `PermissionError` (a subclass of `OSError`). The cache file itself is already written successfully at that point, but the unguarded chmod takes down the whole `.map()` call with a traceback like:

```
  File "src/datasets/arrow_dataset.py", line 3747, in _map_single
    os.chmod(cache_file_name, 0o666 & ~umask)
PermissionError: [Errno 1] Operation not permitted: '/mnt/data/cache_55b1844ebd9dffff_00000_of_00001.arrow'
```

## Fix

Wrap each of the two `os.chmod` call sites in `src/datasets/arrow_dataset.py` with `try/except OSError`, logging the failure at debug level. The permission update stays best-effort: sane `0o666 & ~umask` permissions on POSIX filesystems that support it, silent graceful degradation on flat-permission mounts where there's nothing to set.

No new API surface, no env-var toggle — the issue body suggested an opt-out env var, but catching `OSError` is narrower and doesn't require users to know about a new knob. If a future caller actually needs to detect the degraded mode, they can observe the debug log.

## Tests

New regression test `test_map_survives_chmod_permission_error` in `tests/test_arrow_dataset.py`:

- Patches `datasets.arrow_dataset.os.chmod` to raise `PermissionError` for paths inside the pytest `tmp_path` (so pytest's own housekeeping is unaffected)
- Runs `Dataset.from_dict({\"a\": [1, 2, 3]}).map(lambda x: {\"b\": x[\"a\"] * 2}, cache_file_name=...)`
- Asserts the mapped values are correct (`[2, 4, 6]`), the cache file exists on disk, and the chmod guard was actually exercised

Verified the test fails on `main` without the fix (propagates `PermissionError`) and passes with it.

## Scope notes

- Two chmod sites changed in `arrow_dataset.py` — both follow the identical `shutil.move → umask → chmod` pattern
- No behavior change on POSIX filesystems that accept chmod
- Issue author asked about an env-var opt-out; this goes narrower (catch-the-error) because the chmod is genuinely best-effort and failure is recoverable

## Checklist

- [x] Added a regression test in `tests/`
- [x] `ruff check` and `ruff format --check` clean on touched files
- [x] Related `test_map_caching*` tests still pass
- [x] Scope isolated to one bug
Parse agent traces

Supports claude_code, codex and pi

JSON Lines files are parsed so one file = one row

The resulting row has three fields: harness, session_id, traces, and file_name with features:

```python
AGENT_TRACES_FEATURES = datasets.Features(
    {
        "harness": datasets.Value("string"),
        "session_id": datasets.Value("string"),
        "traces": datasets.List(datasets.Json()),
        "file_name": datasets.Value("string"),
    }
)
```
cc @cfahlgren1 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8113). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.