NyssaW/github-issues huggingface dataset:
fix(image): guard exif_transpose against corrupted EXIF rational tags

## Summary

When an `IterableDataset` contains images with corrupted EXIF metadata (e.g. a TIFF rational tag whose denominator is zero), `PIL.ImageOps.exif_transpose` raises `ZeroDivisionError` inside `Image.decode_example`. Because this exception escapes the internal generator frame, the streaming iterator dies immediately — calling `next()` again raises `StopIteration` rather than advancing to the next sample. This silently drops an unknown number of training examples and makes the data pipeline non-resumable without a full restart from sample 0, causing silent data repetition.

The fix wraps `PIL.ImageOps.exif_transpose` in a `try/except Exception` block. When PIL raises (for any malformed-EXIF reason), `decode_example` emits a `UserWarning` naming the affected file and returns the image without orientation correction. The exception is no longer allowed to escape `decode_example`, so the streaming iterator remains alive and iteration continues normally from the next sample.

A new test (`test_image_decode_example_with_corrupted_exif_orientation_tag`) mocks `PIL.ImageOps.exif_transpose` to raise `ZeroDivisionError` and asserts that (a) the returned image is the correct size/mode without orientation rotation and (b) a `UserWarning` containing "EXIF" is emitted.

## Issue

Fixes #8165

## Local verification

```
$ cd /tmp/datasets
$ ruff check src/datasets/features/image.py tests/features/test_image.py
All checks passed!
ruff exit: 0

$ python -m pytest tests/features/test_image.py::test_image_decode_example \
    tests/features/test_image.py::test_image_decode_example_with_exif_orientation_tag \
    tests/features/test_image.py::test_image_decode_example_with_corrupted_exif_orientation_tag \
    -v
============================= test session starts ==============================
platform linux -- Python 3.11.15, pytest-9.0.3, pluggy-1.6.0
plugins: datadir-1.8.0, anyio-4.13.0
collected 3 items

tests/features/test_image.py::test_image_decode_example PASSED
tests/features/test_image.py::test_image_decode_example_with_exif_orientation_tag PASSED
tests/features/test_image.py::test_image_decode_example_with_corrupted_exif_orientation_tag PASSED

============================== 3 passed in 0.12s ==============================
=== LOCAL_TEST_PASSED ===
```

## Risk

The `except Exception` catch is intentionally broad to cover any PIL internal error from `exif_transpose` (not just `ZeroDivisionError`), since future Pillow versions may raise different exceptions for similarly malformed data. The worst-case regression is that a genuine PIL bug in `exif_transpose` would now be silently swallowed — but this is mitigated by the `UserWarning`, which lets users notice and investigate affected samples. The happy path (well-formed EXIF) is unchanged and remains outside the try/except.


Comments:
Add batch(by_column=...)

Will be useful for robotics dataset to batch samples by episode cc @pkooij 

example of usage:

```python
from datasets import Dataset

ds = Dataset.from_dict({"episode": [0] * 10 + [1] * 10, "frame": list(range(10)) * 2})
# ds = ds.to_iterable_dataset()
ds = ds.batch(by_column="episode")
for x in ds:
    print(x)
# {'episode': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'frame': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]}
# {'episode': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'frame': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]}
```

It's implemented using efficient Arrow functions for substantial speed up for Dataset and for Parquet IterableDataset.
It also supports lossless state_dict() / load_state_dict().

It works by accumulating Arrow data and grouping the batches together using `pyarrow.ListArray` in an arrow map() function.

Multiprocessing is not supported because it could split batches in two or more when distributing shards to processes (batches can overlap multiple shards). This is fine IMO since multiprocessing is only for Dataset.batch() and since the operation is unlikely to be CPU bound thanks to Arrow functions. Though this could be useful for very large datasets and clusters.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8172). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix(map): fix progress bar exceeding total when load_from_cache_file=False

## Summary

When `load_from_cache_file=False`, the progress bar in `Dataset.map()` displayed a count that exceeded the dataset size (e.g. "800 examples" for a 400-row dataset). 

The bug: `pbar_initial` was computed using `len(existing_cache_files)`, which counts cache files on disk regardless of whether they would be used. With `load_from_cache_file=False`, those files exist but are ignored, yet the bar was initialized to the full dataset size as if all work was already done. The actual processing then added on top of that, pushing the counter past the total and causing tqdm to drop the percentage display entirely.

Fix: use `num_shards - len(unprocessed_kwargs_per_job)` instead, which is 0 when `load_from_cache_file=False`, so the bar starts and counts correctly.

## Issue

Fixes #8167

## Local verification

`py -3.14 -m pytest tests/test_arrow_dataset.py -k "test_map_load_from_cache_file_false_progress_bar_starts_at_zero" -v`

2 passed


## Risk

Low risk, a one-line change to how `pbar_initial` is computed. No behaviour changes outside of the progress bar display.




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8170). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Streaming dataset hangs consistently

### Describe the bug

When loading a dataset with `streaming=True`, some background process prevents the script from ever returning (or at least for a very long time).

This did not happen with `huggingface-hub < 1`.

### Steps to reproduce the bug

from datasets import load_dataset

ds = load_dataset("IRIIS-RESEARCH/Nepali-Text-Corpus", split="train", streaming=True)
print(next(iter(ds)))


### Expected behavior

Should clean up as soon as iterable dataset goes out of scope

### Environment info

- `datasets` version: 4.8.5
- Platform: macOS-26.3.1-arm64-arm-64bit
- Python version: 3.11.13
- `huggingface_hub` version: 1.13.0
- PyArrow version: 24.0.0
- Pandas version: 3.0.2
- `fsspec` version: 2026.2.0

Comments:
+1 also encountered
docs(map): clarify that fn_kwargs yields a stable cache fingerprint vs closures

## Summary

When `Dataset.map()` or `Dataset.filter()` receives a closure that closes over an object containing non-deterministic state — for example a UUID, a random seed, or any mutable attribute of a class instance — `dill` serialises the **entire** captured object into the fingerprint.  Two calls that are logically identical but use different object instances therefore produce different fingerprints, and the computed cache is never reused.  This happens silently: no warning is emitted and the re-computation cost can be severe for large datasets.

Passing the same values through `fn_kwargs` (or `functools.partial`) avoids the problem because only the explicit, deterministic values are hashed — not the surrounding object.  The fix is already available to users today; what was missing was documentation explaining *why* the behaviour occurs and *how* to avoid it.

This PR adds a focused **Tip** block to the `fn_kwargs` parameter in `Dataset.map()` with side-by-side code examples (unstable closure vs stable `fn_kwargs`) and a back-reference in `Dataset.filter()`.  It also adds a new test (`test_fn_kwargs_produces_stable_fingerprint_vs_closure`) that pins the described behaviour and acts as a regression guard.

## Issue

Fixes #7986

## Local verification

```bash
# Linter
ruff check src/datasets/arrow_dataset.py tests/test_fingerprint.py
# All checks passed!

# New test
pytest tests/test_fingerprint.py::test_fn_kwargs_produces_stable_fingerprint_vs_closure -xvs
```

```
============================= test session starts ==============================
platform linux -- Python 3.11.15, pytest-9.0.3, pluggy-1.6.0
collecting ... collected 1 item

tests/test_fingerprint.py::test_fn_kwargs_produces_stable_fingerprint_vs_closure PASSED

============================== 1 passed in 0.06s ===============================
```

```bash
# Full fingerprint test suite
pytest tests/test_fingerprint.py -x -q
# 23 passed, 9 skipped in 3.14s
```

=== LOCAL_TEST_PASSED ===

## Risk

Changes are documentation-only in the library source (`arrow_dataset.py` docstrings) and a new test in `test_fingerprint.py`.  No behaviour is modified; the test asserts facts that have always been true about the `Hasher` API.  The only risk is that the Tip wording turns out to be misleading or incomplete — happy to revise based on reviewer feedback.

Comments:
[BUG] Abnormal progress bar in dataset.map when load_from_cache_file=False

### Describe the bug

datasets                                 4.8.5

```python
from datasets import load_dataset, load_from_disk

data = load_from_disk('pretrain_cached_dataset/train')

def dummy_map(example):
    return example

dataset = data.map(dummy_map)
dataset = data.map(dummy_map, load_from_cache_file=True)   # works fine
dataset = data.map(dummy_map, load_from_cache_file=True, num_proc=4)   # works fine
dataset = data.map(dummy_map, load_from_cache_file=False)  # does not work

```

<img width="935" height="124" alt="Image" src="https://github.com/user-attachments/assets/2325a7f3-80e1-46bd-a8fb-eb503b7e5df3" />


<img width="954" height="58" alt="Image" src="https://github.com/user-attachments/assets/ff5f49ad-bf96-4a20-891e-bfa8018115ea" />

### Steps to reproduce the bug

-

### Expected behavior

-

### Environment info

-

Comments:
https://github.com/huggingface/datasets/blob/4e3bf52ff436d841e61211dfa096b5c47197ae36/src/datasets/arrow_dataset.py#L3593-L3598

Hi, can you provide more details on the issue ?

<img width="340" height="26" alt="Image" src="https://github.com/user-attachments/assets/0dca6f47-4a8d-4871-a908-f58df856ed72" />

This exceeds the maximum value.

#self-assign

Hey, I've opened a PR for this: #8170 

thanks
Fix storage_options lookup for streaming Lance datasets

In `Lance._split_generators`, the `storage_options` lookup used `files[0].split("://", 0)[0] + "://"`, which has two bugs that compound: `split("://", 0)` does not split at all (maxsplit=0), and the `storage_options` dict is keyed by the bare scheme (`"hf"`), not `"hf://"`. The lookup therefore always returned `None`, dropping the HF token before it reached `lance.dataset` and causing 401s for streaming reads of private Lance repos (and the dataset-viewer worker path via `get_dataset_split_names`).

Use `split("://", 1)[0]` to extract the scheme, matching the key written in `DownloadConfig` and the pattern used by `sql.py:60`.

Fixes #8164.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8166). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

CI failure is unrelated and is due to https://github.com/pandas-dev/pandas/issues/65462, merging
add agent trace prompt, sent_at, count fields

- add agent trace metadata columns: `prompt`, `sent_at`, `num_user_messages`, and `num_tool_calls`.
- extract `prompt` / `sent_at` from the first real user message for Claude Code, Pi/OpenClaw, and Codex traces.



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8163). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Include all test files in sdist via MANIFEST.in

Fixes #8159.

## What's wrong

The sdist on PyPI for `datasets==4.8.5` only ships **37 of the 95** files under `tests/`. Without an explicit `MANIFEST.in`, setuptools' default sdist behavior includes only top-level `tests/test_*.py` files and drops everything else.

Files missing from the current sdist:

- `tests/conftest.py`
- `tests/_test_patching.py`
- everything under `tests/commands/`, `tests/features/`, `tests/fixtures/`, `tests/io/`, `tests/packaged_modules/`, `tests/distributed_scripts/`
- the audio / image / PDF / JSON test-data files under `tests/features/data/` and `tests/io/data/`

Anyone installing from sdist (distros, mirrors, air-gapped builds) cannot run the test suite — `pytest` fails to even collect because `tests/conftest.py` is missing the `pytest_plugins` configuration that pulls in `tests.fixtures.*`.

## Fix

Add a minimal `MANIFEST.in` that grafts the whole `tests/` directory and excludes bytecode.

```
graft tests
global-exclude __pycache__
global-exclude *.py[cod]
```

## Verification

Built the sdist locally on `main` before/after the change and compared:

```
$ python -m build --sdist
```

| | Before | After |
|---|---|---|
| Files under `tests/` in sdist | 37 | 95 |
| `tests/conftest.py` | missing | included |
| `tests/_test_patching.py` | missing | included |
| `tests/fixtures/fsspec.py` | missing | included |
| `tests/features/test_audio.py` | missing | included |
| `tests/io/test_csv.py` | missing | included |
| `tests/packaged_modules/test_arrow.py` | missing | included |
| `tests/features/data/test_audio_16000.mp3` | missing | included |
| `__pycache__` files | 0 | 0 |

All 95 files under `tests/` in the working tree are now in the sdist; nothing else is added.

Comments:
Pass library_name/version to HfApi in dataset push and delete paths

Adds `library_name="datasets"` and `library_version=__version__` to 12 bare `HfApi(endpoint=..., token=...)` instantiations across `hub.py`, `arrow_dataset.py`, `iterable_dataset.py`, and `dataset_dict.py`. Commits made via `push_to_hub` / `delete_from_hub` currently report a User-Agent of `unknown/None; hf_hub/X.Y.Z; ...`; with this change they report as `datasets/X.Y.Z; ...`.

This completes a convention the repo already follows in `load.py` and `utils/file_utils.py`. Motivated by some recent UA-attribution work on the Hub side (happy to share more context on Slack).

Opened as draft to let CI verify nothing breaks before requesting review.

## Test plan
- [ ] CI green
- [ ] Optional manual smoke: `Dataset.from_dict({"x": [1]}).push_to_hub(...)` against a staging repo, confirm commit UA reports `datasets/<version>`.

cc @Wauplin to check this makes sense in your opinion! 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8161). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
tests/conftest.py, tests/_test_patching.py, tests/fixtures/fsspec.py and other files are missing from the PYPI distribution

### Describe the bug

Version: 4.8.5


### Steps to reproduce the bug

n/a

### Expected behavior

n/a

### Environment info

n/a

Comments:
Hey, ran into something similar a while back when we were packaging a custom dataset loader for a client at v4.7.x. Turns out, PyPI distributions sometimes skip test files and fixtures like `conftest.py` since they're not critical for runtime—honestly, it's a packaging config thing in `setup.py` or `pyproject.toml`. We ended up manually pulling the source from GitHub for testing purposes when we needed those files. Stick to the repo if you’re debugging or running tests locally.
Fix revision-aware cache lookup in load_dataset

Fixes #7928

When `load_dataset()` falls back to the local cache, for example when offline or when the Hub is unreachable, it ignores the `revision` argument and simply returns the most recently modified cached version. This can lead to unexpected behavior when a specific revision is requested.

This happens because `CachedDatasetModuleFactory` neither accepted nor used the `revision` parameter, and `dataset_module_factory` did not pass it along either.

In this PR, I’ve updated the cache handling so that a specific revision can be respected when loading from cache.

In particular, SHA-based revisions are now matched against cached commit directories. If a matching commit is found, the corresponding cached version is returned. If not, a `DatasetNotFoundError` is raised, listing the available cached SHAs.

For branch or tag names such as `main` or `v1.0`, I was not able to reliably resolve them without access to the Hub. In such cases, a `DatasetNotFoundError` is raised with a clear message suggesting the use of a commit SHA instead.

The existing behavior is preserved when `revision=None`, meaning the latest cached version is still used by default.

Also, I fixed the inner `except Exception` block in `dataset_module_factory`. Previously, it would swallow `DatasetNotFoundError` raised by the cache logic and fall back to the outer Hub-related error. Now, the correct error is propagated.

Test I did:

- [x] Verified that a full SHA present in cache returns the correct version
- [x] Verified that a short SHA prefix resolves correctly when unique
- [x] Confirmed that missing SHAs raise `DatasetNotFoundError` with available SHAs listed
- [x] Confirmed that branch and tag names raise an error with an offline hint
- [x] Ensured that `revision=None` continues to return the latest cached version
- [x] Confirmed that `test_load_dataset_invalid_revision_with_cache` (from PR #7929) still passes

Comments:
Fix tilde expansion for local data_files paths

Summary
- expand `~` before classifying local `data_files` patterns
- add a regression test covering `resolve_pattern(os.path.join("~", ...), ...)`

Impact
- `load_dataset(..., data_files="~/path/to/file.parquet")` currently treats `~` as a literal relative directory under the working tree instead of the user's home directory
- this brings `data_files` in line with the existing `data_dir` handling and fixes issue #6212

Testing
- `python -m pytest tests/test_data_files.py`
- `make quality`
- manual repro: `load_dataset("parquet", data_files="~/.codex-tmp-datasets-tilde-repro/sample.parquet")`

Fixes #6212

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8153). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add Apache Iceberg format support

 # Add Apache Iceberg format support 

## Motivation

  Apache Iceberg is the most widely adopted open table format for data lakes, supported by Databricks,
  Snowflake, AWS Glue, Dremio, and others. A large amount of ML training data lives in Iceberg tables.
  Currently, users must manually export Iceberg data to Parquet before loading it into HuggingFace Datasets —
  this PR removes that friction.

fix this (https://github.com/huggingface/datasets/issues/7863)

##  Usage

  Users pass a pre-configured pyiceberg Catalog object and a table identifier:

  from pyiceberg.catalog.sql import SqlCatalog
  from datasets import load_dataset

  catalog = SqlCatalog("my_catalog", uri="sqlite:///catalog.db", warehouse="/tmp/warehouse")

## Basic loading
  ds = load_dataset("iceberg", catalog=catalog, table="db.my_table")

## Column selection + row filtering (predicate pushdown)
  ds = load_dataset("iceberg", catalog=catalog, table="db.my_table",
                    columns=["text", "label"],
                    filters=[("label", ">", 0)])

## Multiple splits from different tables
  ds = load_dataset("iceberg", catalog=catalog,
                    table={"train": "db.train", "test": "db.test"})

## Time travel via snapshot_id
  ds = load_dataset("iceberg", catalog=catalog, table="db.my_table",
                    snapshot_id=7051729674881785648)

## Streaming
  ds = load_dataset("iceberg", catalog=catalog, table="db.my_table", streaming=True)

  Works with any pyiceberg-supported catalog backend (REST, Hive, Glue, SQL, etc.) — the builder is agnostic
  to how the catalog is configured.

 ## Design decisions

  - Catalog object passed in, not constructed internally. Iceberg catalog configuration varies widely across
  backends (REST, Hive, Glue, SQL each have different auth/connection params). Rather than re-implementing a
  "catalog factory" inside the builder, users bring their own catalog — similar to how the sql builder accepts
   an existing SQLAlchemy connection. This keeps the builder simple and forward-compatible with new catalog
  types.
  - No _EXTENSION_TO_MODULE registration. Unlike file-based formats (Parquet, Lance, CSV), Iceberg tables are
  addressed via catalog + table identifier, not file extensions. Users must specify "iceberg" explicitly as
  the path argument.
  - create_config_id override for fingerprinting. Catalog objects (containing SQLAlchemy engines, connection
  pools, etc.) are not picklable by dill. The override replaces the catalog with a stable string
  representation ("{ClassName}_{name}") before hashing.
  - _CountableBuilderMixin for fast row counting. Uses scan.plan_files() metadata to count rows without
  reading data files.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8148). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix iterable map resume state


Fixes: https://github.com/huggingface/datasets/issues/8145
## Root Cause
  `FormattedExamplesIterable.__iter__` ignored `force_convert_to_python` and used Arrow batch iteration, advancing the
  underlying state by a full Arrow chunk before yielding Python examples.

  ## Fix
  Only use Arrow batch iteration when `force_convert_to_python` is false:
  `if self.ex_iterable.iter_arrow and not self.force_convert_to_python:`

  ## Tests
  - [x] Verified the issue reproduction now resumes at `[200, 201, 202, 203, 204]`.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8147). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I took the liberty of adjusting the fix to still use the iter_arrow in `FormattedExamplesIterable.__iter__`, I hope you don't mind. Let's see if the CI is green now

hey @lhoestq I checked the source of the error for the failing integration test. Seems like all 5 failed checks stemmed from a 401 unauthorized error and does not seem to be related to the code changes. Is it possible for you to briefly check? Thank you so much!
fix: tolerate chmod PermissionError on flat-permission filesystems

## Summary

Fixes #8125.

`Dataset.map()` (and the indices writer in `Dataset.flatten_indices` / `Dataset.select` path) calls `os.chmod(cache_file, 0o666 & ~umask)` on the freshly moved cache file. On flat-permission filesystems — GCS FUSE, S3 FUSE mounts, and similar object-storage-backed mounts — `chmod` raises `PermissionError` (a subclass of `OSError`). The cache file itself is already written successfully at that point, but the unguarded chmod takes down the whole `.map()` call with a traceback like:

```
  File "src/datasets/arrow_dataset.py", line 3747, in _map_single
    os.chmod(cache_file_name, 0o666 & ~umask)
PermissionError: [Errno 1] Operation not permitted: '/mnt/data/cache_55b1844ebd9dffff_00000_of_00001.arrow'
```

## Fix

Wrap each of the two `os.chmod` call sites in `src/datasets/arrow_dataset.py` with `try/except OSError`, logging the failure at debug level. The permission update stays best-effort: sane `0o666 & ~umask` permissions on POSIX filesystems that support it, silent graceful degradation on flat-permission mounts where there's nothing to set.

No new API surface, no env-var toggle — the issue body suggested an opt-out env var, but catching `OSError` is narrower and doesn't require users to know about a new knob. If a future caller actually needs to detect the degraded mode, they can observe the debug log.

## Tests

New regression test `test_map_survives_chmod_permission_error` in `tests/test_arrow_dataset.py`:

- Patches `datasets.arrow_dataset.os.chmod` to raise `PermissionError` for paths inside the pytest `tmp_path` (so pytest's own housekeeping is unaffected)
- Runs `Dataset.from_dict({\"a\": [1, 2, 3]}).map(lambda x: {\"b\": x[\"a\"] * 2}, cache_file_name=...)`
- Asserts the mapped values are correct (`[2, 4, 6]`), the cache file exists on disk, and the chmod guard was actually exercised

Verified the test fails on `main` without the fix (propagates `PermissionError`) and passes with it.

## Scope notes

- Two chmod sites changed in `arrow_dataset.py` — both follow the identical `shutil.move → umask → chmod` pattern
- No behavior change on POSIX filesystems that accept chmod
- Issue author asked about an env-var opt-out; this goes narrower (catch-the-error) because the chmod is genuinely best-effort and failure is recoverable

## Checklist

- [x] Added a regression test in `tests/`
- [x] `ruff check` and `ruff format --check` clean on touched files
- [x] Related `test_map_caching*` tests still pass
- [x] Scope isolated to one bug

Comments:
`.map()` on a streaming IterableDataset silently skips examples after load_state_dict

### Describe the bug

After `ds.map(fn)` on a streaming IterableDataset, saving state_dict() and loading it into a fresh dataset does not resume at the correct position. It jumps forward by up to one Arrow read-chunk (~1000 rows).

### Steps to reproduce the bug

```python
from datasets import Dataset

ds1 = Dataset.from_dict({"id": list(range(5000))}).to_iterable_dataset(num_shards=1).map(
    lambda x: {**x, "tag": "X"}
)
ds2 = Dataset.from_dict({"id": list(range(5000))}).to_iterable_dataset(num_shards=1).map(
    lambda x: {**x, "tag": "X"}
)

it = iter(ds1)
for _ in range(200):
    next(it)
state = ds1.state_dict()
expected = [next(it)["id"] for _ in range(5)]

ds2.load_state_dict(state)
it2 = iter(ds2)
actual = [next(it2)["id"] for _ in range(5)]

print(f"expected={expected}")
print(f"actual  ={actual}")
assert expected == actual, f"resume skipped forward by {actual[0] - expected[0]} rows"
```

```
expected=[200, 201, 202, 203, 204]
actual  =[1000, 1001, 1002, 1003, 1004]
AssertionError: resume skipped forward by 800 rows
```

### Expected behavior

Expected behaviour:
```
expected=[200, 201, 202, 203, 204]
actual  =[200, 201, 202, 203, 204]
```

### Environment info

```
- `datasets` version: 4.8.4
- Platform: Linux-6.8.0-85-generic-x86_64-with-glibc2.39
- Python version: 3.12.12
- `huggingface_hub` version: 1.11.0
- PyArrow version: 23.0.1
- Pandas version: 3.0.2
- `fsspec` version: 2026.2.0
```

Comments:
I think this line here is causing issues: 
https://github.com/huggingface/datasets/blob/2724a653e7582077be647d3fdfe8cd1fac27b85f/src/datasets/iterable_dataset.py#L2225
It ignored force_convert_to_python, so it consumed a full Arrow chunk.
I think it can be fixed if you add `self.force_convert_to_python` to the condition it forces formatting to iterate row-by-row as Python examples instead of using Arrow batches, which prevents state tracking from jumping ahead by a whole Arrow chunk.
tag openclaw sessions via cwd in pi agent traces

Tag OpenClaw datasets from Pi agent sessions. Will add a PR to add icon for it in moon too

cc: @merveenoyan 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8143). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: replace list with Sequence in public API type hints 

Fixes #5354

Replaced `list` with `Sequence` in public API function signatures across:
- `arrow_dataset.py` — from_csv, from_json, from_parquet, from_text
- `iterable_dataset.py` — from_csv, from_json, from_parquet, from_text  
- `combine.py` — interleave_datasets, concatenate_datasets
- `dataset_dict.py` — remove_columns, select_columns

`Sequence` is covariant and more flexible than `list`, which resolves 
the mypy type checking errors reported in the issue.

Comments:
fix warning in ci

otherwise it raises an issue from transformers smh

```
../../.pyenv/versions/3.12.2/envs/hf-datasets/lib/python3.12/site-packages/transformers/__init__.py:843: in _getattr
    return getattr(importlib.import_module(target, __name__), new_name)
../../.pyenv/versions/3.12.2/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1387: in _gcd_import
    ???
<frozen importlib._bootstrap>:1360: in _find_and_load
    ???
<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:935: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:995: in exec_module
    ???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    #                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
    #           This file was automatically generated from src/transformers/models/aria/modular_aria.py.
    #               Do NOT edit this file manually as any edits will be overwritten by the generation of
    #             the file from the modular. If any change should be done, please apply the change to the
    #                          modular_aria.py file directly. One of our CI enforces this.
    #                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
    # Copyright 2024 The Rhymes-AI Teams Authors and The HuggingFace Inc. team. All rights reserved.
    #
    # Licensed under the Apache License, Version 2.0 (the "License");
    # you may not use this file except in compliance with the License.
    # You may obtain a copy of the License at
    #
    #     http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.
    import torch
>   from torchvision.transforms.v2 import functional as tvF
E   ModuleNotFoundError: No module named 'torchvision'
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8140). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
chore: bump doc-builder SHA for main doc build workflow

Bump the pinned doc-builder SHA so that main documentation builds also sync to the HF bucket (dual-write).


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8138). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Closing — not needed right now, active libs already on @main.
Fix: decode JSON type before to_list or to_dict is called

**Motivation**:
There is a change in the approach to decoding JSON types, added in https://github.com/huggingface/datasets/commit/d560b58e1f1ec96591a95a1780ee118f5eea6c74. Since 4.7.0, to_list() returns raw JSON strings for columns stored as Json(), while direct access via getitem returns dicts.

In versions prior to 4.7.0, to_list() returned dicts.

**Added context**:
- Closes https://github.com/huggingface/datasets/issues/8131
- On it, the maintainer suggested taking a similar approach to https://github.com/huggingface/datasets/pull/8122

This PR contains:
- An update to the to_list and to_dict functions for the arrow dataset inspired by a (and using the utils developed for) [similar fix](https://github.com/huggingface/datasets/pull/8122) to another problem caused by the addition of the JSON type.
- Unit tests specifying the intended outcome (which was how dictionaries were processed or handled prior to v4.7.0).

**Additional questions/considerations**:
- This PR restore behaviour prior to the breaking change introduced by the addition of the JSON datatype in v4.7.0. However, does mean we now decode one data type (JSON) and not others. We could decide to decode elements of the list in the same way _get_item does but this would be a big change (i.e memory concerns with images and audio etc etc)
- I implemented the PR with a lazy import - just checking that's in line with the repo's design/ vibe?

Comments:
@lhoestq Could this be reviewed and merged if it's a suitable solution?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8137). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

`datasets` 4.8.5 is out and includes the fix ! :)
use full repo-relative file path instead of filename for agent traces

We want to include full path of the file so it's easier to fetch the full trace from repo

  - Store the full repo-relative path in `file_name` instead of just the basename
  - Extracts the path from the `hf://` origin URL set by the download manager

  Before: `file_name = "614b8d50-097f-4e36-aee0-1fdd053335e7.jsonl"`
  After:  `file_name = "sessions/claude/614b8d50-097f-4e36-aee0-1fdd053335e7.jsonl"`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8133). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

thanks ! let me push some adjustments, and yea let's use file_path instead
fix(Dataset.map): initialize writer on first non-None output instead of first index

## What does this PR do?

Fixes #7990

## The Bug

When a map function returns `None` for the first examples and a dict for
later ones, `Dataset.map()` crashes with:

    AttributeError: 'NoneType' object has no attribute 'write'

This happens because the ArrowWriter was initialized only when processing
index 0. If index 0 returns `None`, `update_data` is False at that point,
so the writer is never created. When a later example returns a dict,
`update_data` becomes True but the writer is still None.

## The Fix

Replace index-based checks with `writer is None`:

**Before:**
if i == 0:          # non-batched
if i and i[0] == 0: # batched
    buf_writer, writer, tmp_file = init_buffer_and_writer()

**After:**
if writer is None:  # both cases
    buf_writer, writer, tmp_file = init_buffer_and_writer()

## Reproduction

```python
from datasets import Dataset

ds = Dataset.from_dict({"x": [1, 2, 3]})

def fn(example, idx):
    if idx < 2:
        return None
    return {"x": [example["x"] * 10]}

list(ds.map(fn, with_indices=True))  # Previously crashed, now works

Tests
Non-batched: first examples return None ✅
Batched: first batch returns None ✅
All examples return None → dataset unchanged ✅
Normal map with no None returns ✅




Comments:
fix: mutable default argument in `_visit_with_path`

Replace `visit_path: _VisitPath = []` with `Optional[_VisitPath] = None` to avoid the mutable default argument anti-pattern ([B006](https://docs.astral.sh/ruff/rules/mutable-argument-default/)).

While the current callers always use `visit_path + [k]` (creating new lists), the shared default list is a latent bug that would surface if any future code path mutates `visit_path` in place (e.g. via `.append()`).

The fix initializes `visit_path = []` inside the function body when `None` is passed.

Comments:
`load_dataset()` hangs when `hf_xet` is enabled

### Describe the bug

Hi! I am getting a peculiar download issue on the latest versions of `datasets` and `hfh` where datasets hang indefinitely:

```py
  from datasets import load_dataset
  load_dataset("openai/gsm8k", "main", split="test")
```

Note that:

  - streaming=True works.
  - Direct Hub access works.
  - The issue disappears when `HF_HUB_DISABLE_XET=1` is set.

  This looks like the parquet download/materialization path is getting stuck specifically
  when the Xet backend is active. Setting HF_HUB_DISABLE_XET=1 is a working workaround.

### Steps to reproduce the bug

First install the latest versions of the libs:

```sh
pip install -U datasets huggingface_hub
```

Then run

```sh
python -c 'from datasets import load_dataset; print(load_dataset("openai/gsm8k", "main", split="test"))'
```

Note that the following works:

```sh
HF_HUB_DISABLE_XET=1 python -c 'from datasets import load_dataset; print(load_dataset("openai/gsm8k", "main", split="test"))'
```

### Expected behavior

The dataset downloads without hanging

### Environment info

  Environment:

  - datasets==4.8.4
  - huggingface_hub==1.10.1
  - hf_xet installed

Comments:
Apologies, seems to have been a stale cache issue. Nuking the datasets cache fixed the issue
Don't include files list DatasetInfo (and remove old stuff)

this will fix the viewer when it saves the dataset info in mongo, for datasets with a lot of files this can make the mongo doc too big for mongo

I also removed old deprecated stuff linked to files list like the old concept of "post-processing" and cleaned the checksum recording logic

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8128). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Support explicit harness field in agent traces

Hi!

This small PR allows an optional `harness` field to be parsed.

When present, the harness field takes priority over type-value heuristics.

This allows agents to self-identify even when they contain property names that collide with an existing harness ("session", "system", etc.)

Comments:
For a concrete example, [https://huggingface.co/datasets/jedisct1/agent-traces-swival](https://huggingface.co/datasets/jedisct1/agent-traces-swival) currently shows `claude_code` because it shares field names with Claude Code, despite an explicit `harness` field identifying the harness as not Claude Code.


> For a concrete example, https://huggingface.co/datasets/jedisct1/agent-traces-swival currently shows `claude_code` because it shares field names with Claude Code, despite an explicit `harness` field identifying the harness as not Claude Code.

is there any parsing issues from the actual traces, or more about just the name of the harness?

@cfahlgren1  No parsing issues. This is just about showing the correct harness name.
Fix batching for table-formatted datasets

## Summary
Fix #8075

Fix `.batch()` on table-formatted `Dataset` and `IterableDataset`.

Before this change, calling `.batch()` on datasets formatted as `pyarrow`, `pandas`, or `polars` could fail because the batching path assumed dict-like inputs. This updates batching to use an Arrow-based path for table-style formats, so batching works regardless of whether the table format is applied before or after `.batch()`. These two forms now behave equivalently:

```python
dataset.with_format(format_type).batch(n)
```

and

```python
dataset.batch(n).with_format(format_type)
```

for `pyarrow`, `pandas`, and `polars`.

## What changed

- added a shared `_batch_arrow_table` helper to build batched Arrow tables
- updated `Dataset.batch()` to route table-formatted batching through an Arrow `.map(...)` path
- updated `IterableDataset.batch()` to do the same and then restore the original table format
- added regression tests for `pyarrow`, `pandas`, and `polars` on both `Dataset` and `IterableDataset`

## Tests

- `python -m pytest tests/test_arrow_dataset.py -k "test_dataset_batch or test_dataset_batch_with_table_format or test_dataset_batch_with_polars_format" -q`
- `python -m pytest tests/test_iterable_dataset.py -k "test_iterable_dataset_batch or test_iterable_dataset_batch_with_table_format or test_iterable_dataset_batch_with_polars_format" -q`
- `python -m ruff check src/datasets/arrow_dataset.py src/datasets/iterable_dataset.py src/datasets/table.py tests/test_arrow_dataset.py tests/test_iterable_dataset.py`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8126). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Feature request: being able to disable chmod for flat permission filesystems

### Feature request

For flat permission file systems, `os.chmod` simply fails when trying to cache datasets - in some places such as: https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L4043-L4048 for arrow datasets. 
This seems to be optional to obtained desired behavior as simply commenting it out makes our workload work as expected. Hence the feature request would be to see if it would be possible to expose an API to disable `os.chmod` operations when users asks to do so (e.g. through an env variable?)

### Motivation

Being able to save properly datasets on flat permission filesystems

### Your contribution

Can test the feature if implemented

Comments:
Context: the operation fails on GCS FUSE mounts, and likely will on similar s3-backed file systems.

  File "/home/user/.venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3747, in _map_single                                                                                                                      
    os.chmod(cache_file_name, 0o666 & ~umask)                                                                                                                                                                                                    
PermissionError: [Errno 1] Operation not permitted: '/mnt/data/cache_ 55b1844ebd9dffff_00000_of_00001.arrow'
fix: decode Json() values before calling DataFrame.to_json() (#8116)

# Summary
fixes (https://github.com/huggingface/datasets/issues/8116)
Basically the issue is that if a dataset column uses Json() as a feature type, calling to_json() on that dataset causes a raw json escaped string to be exported, without decoding it first. 
### root cause
`to_json` exports feature values in their stored form by default, which means that the raw json escaped string would be exported
### changes made
I tried fixing this minimally. before calling pandas `to_json()`, the writer now decodes the batch with `PandasFeaturesDecoder`. That makes Json() values serialize as native JSON. 
### test plan
- [x] reproduced this error locally
- [x] verified the fix with direct JsonDatasetWriter smoke tests and by confirming it reuses the existing PandasFeaturesDecoder path correctly.

Comments:
Just committed the fix, please let me know if there are any other issues

Thanks for the quick fix!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8122). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix Audio.cast_storage for CSV-loaded large_string columns

## Summary
This PR fixes `Audio.cast_storage()` so that audio path columns loaded as `large_string` can be cast to `Audio()` correctly.

## Motivation
`Dataset.from_dict({"audio": [...]})` produces a `string` column and `cast_column("audio", Audio())` works, but `load_dataset("csv", ...)` produces a `large_string` column, which currently fails during the cast to the Audio storage type.

## Changes made
- Added handling for `pa.large_string()` in `Audio.cast_storage()`
- Cast large string storage to `pa.string()` before building the Audio struct storage
- Added a regression test covering the CSV-loaded audio path case

## Testing
- [x] Reproduced the issue locally
- [x] Ran the targeted regression test for the CSV-loaded `large_string` case
- [x] Ran `tests/features/test_audio.py` in the local environment

Closes #7970

Comments:
Thanks for the investigation and the fix ! This makes me think that maybe load_dataset("csv") should return "string" types instead of "large_string" to be consistent with load_dataset("json") as well as the Dataset.from_xxx methods

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8121). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the review and the suggestions.

I’ll update the PR to:

* add the same `large_string` handling for `Video` and `Pdf`
* update the docstrings to mention `pa.large_string()`
* switch the audio test to use the existing `audio_path` fixture
* fix the code quality check failure


I have updated the PR to:

* add the same `large_string` handling for `Video` and `Pdf`
* update the docstrings to mention `pa.large_string()`
* keep the `Audio` fix
* add matching regression tests for `Audio`, `Video`, and `Pdf`

I also fixed the code quality issue and reran the targeted feature tests locally.


Great ! It looks like you didn't commit the change to switch the audio test to use the existing audio_path fixture and to update the Audio docstring, could you add your changes before we merge ?

Thanks - I updated the audio regression test to use the existing audio fixture available in the test suite and added the missing `Audio` docstring update for `pa.large_string()`.
Patch fields truncated — missing trailing newlines

Description: Multiple entries in the dataset have patches (test.patch and fix.patch) and f2p_patch fields with truncated content. The trailing blank lines that are part of valid unified diff format are stripped, causing git apply to fail with "corrupt patch" errors.

Affected instances (confirmed): atlanhq_atlan-python_pr37, app-sre_qontract-reconcile_pr300, arviz-devs_preliz_pr388, arviz-devs_preliz_pr635, app-sre_qontract-reconcile_pr1856

How to reproduce: Compare the patch field with the GitHub PR diff:
bash
curl -L https://github.com/arviz-devs/preliz/pull/635.diff > github.patch
GitHub version ends with trailing blank line, dataset version doesn't
git apply fails on the dataset version


Suggested fix: Ensure patch content preserves trailing whitespace/newlines from the original GitHub diff. The current processing appears to strip trailing lines before storing in the dataset.

Comments:
Hi, I'd like to work on this issue. I'll try to reproduce this issue and see where the newline is being stripped.

Hi @hetao-git

I was able to reproduce this issue, I used the instance "atlanhq_atlan-python_pr37" from the AweAI-Team/Scale-SWE dataset. I observed that applying the GitHub patch (pull/37.diff) on the correct base commit (7463d13b525011571703899ee4e668a9e313feac) works successfully, but the dataset provided patches fail.

- patch -> corrupt patch at line 1245
- f2p_patch -> corrupt patch at line 439

Comparing the dataset patch with the GitHub diff shows differences like truncated and altered content which is likely causing the corruption.

This is suggesting that the issue is originating from the dataset generation or preprocessing, rather than in the datasets library.