noamaanMulla-03/datasets-issues huggingface dataset:
Add dtype="list" to List / Sequence / LargeList features (#8002)

Fixes #8002.

## Summary

Most feature types expose a descriptive `dtype` string:

- `Value("int64").dtype == "int64"`
- `ClassLabel(...).dtype == "int64"`
- `Array2D(...).dtype == "<element-dtype>"`
- `Image().dtype == "PIL.Image.Image"`
- `Audio().dtype == "dict"`

But `List`, `LargeList` and `Sequence(non-dict)` (which actually constructs a `List` under the hood) had no `dtype` attribute. Downstream code that branches on `feature.dtype` therefore crashes — e.g. transformers' `run_classification.py:442`:

```python
if raw_datasets["train"].features["label"].dtype == "list":
    ...
```

raises `AttributeError: 'List' object has no attribute 'dtype'`.

This PR adds `dtype: ClassVar[str] = "list"` to `List` and `LargeList`. `Sequence(dict)` intentionally returns a regular `dict` (TFDS-compat layer), so it stays unchanged.

## Reproduce BEFORE/AFTER yourself (copy-paste)

```bash
# --- BEFORE: origin/main, expect AttributeError ---
git fetch origin main && git checkout origin/main
python - <<'PY'
from datasets import ClassLabel, Sequence, Value
features = {"text": Value("string"), "label": Sequence(ClassLabel(names=["No", "Yes"]))}
try:
    print(features["label"].dtype)
    print("NO ERROR (unexpected)")
except AttributeError as e:
    print(f"REPRO: {e}")  # Expected: 'List' object has no attribute 'dtype'
PY

# --- AFTER: this branch ---
git fetch origin pull/<PR>/head:fix && git checkout fix
python - <<'PY'
from datasets import ClassLabel, Sequence, List, Value
from datasets.features.features import LargeList
print(Sequence(ClassLabel(names=["No", "Yes"])).dtype)  # Expected: list
print(List(Value("int64")).dtype)                       # Expected: list
print(LargeList(Value("int64")).dtype)                  # Expected: list
PY
```

## What I ran locally

```
pytest tests/features/test_features.py -k "feature or List or Sequence"
# 307 passed, 142 skipped, 1 failed (pre-existing PIL import error in test_dataset_cast_to_image_features_polars, unrelated)
```

The new regression test `test_list_sequence_largelist_dtype` fails on `origin/main` with the exact `AttributeError` from the issue, and passes on this branch.

## Edge cases covered

| Input | `.dtype` |
|---|---|
| `List(Value("int64"))` | `"list"` |
| `LargeList(Value("int64"))` | `"list"` |
| `Sequence(ClassLabel(...))` | `"list"` (actually a `List` instance) |
| `Sequence({"a": Value("int64")})` | unchanged — still a regular `dict`, no `dtype` (this is the TFDS-compat path and would be a bigger semantic change) |

## Notes

- Implemented as `ClassVar[str] = "list"` so the dataclass machinery doesn't treat it as an init field.
- No public API change beyond gaining the new attribute.

---

🤖 Disclosure: Authored with assistance from Claude (Anthropic), reviewed and tested by me. The reproducer was run on `origin/main` (fails) and this branch (passes); the regression test was run both ways to confirm it gates the bug.
docs: make `Dataset.map` batched example self-contained (#7703)

Fixes #7703.

## Summary

The batched-map example in `Dataset.map`, `DatasetDict.map` and `docs/README.md` referenced `tokenizer` without defining or importing it:

```python
>>> ds = ds.map(lambda example: tokenizer(example["text"]), batched=True)
```

Users copy-pasting the snippet hit `NameError: name 'tokenizer' is not defined`, contradicting the project's documentation principle that examples should "work as expected" when copied directly (`docs/README.md` itself states this on the next paragraph).

## Fix

Replace with a self-contained lambda that demonstrates `batched=True` without dragging in a `transformers` dependency:

```python
>>> ds = ds.map(lambda batch: {"text": [t.upper() for t in batch["text"]]}, batched=True)
```

Three files updated, all in sync (the same line was duplicated):
- `src/datasets/arrow_dataset.py` (`Dataset.map` docstring)
- `src/datasets/dataset_dict.py` (`DatasetDict.map` docstring)
- `docs/README.md` (the meta-example used in the contributor guide)

## Reproduce BEFORE/AFTER yourself (copy-paste)

```bash
# --- BEFORE: origin/main, expect NameError ---
git fetch origin main && git checkout origin/main
python - <<'PY'
from datasets import load_dataset
ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
def add_prefix(example):
    example["text"] = "Review: " + example["text"]
    return example
ds = ds.map(add_prefix)
# Copy-pasted from the docstring:
ds = ds.map(lambda example: tokenizer(example["text"]), batched=True)  # NameError
PY

# --- AFTER: this branch ---
git fetch origin pull/<PR>/head:fix && git checkout fix
python - <<'PY'
from datasets import load_dataset
ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
def add_prefix(example):
    example["text"] = "Review: " + example["text"]
    return example
ds = ds.map(add_prefix)
ds = ds.map(lambda batch: {"text": [t.upper() for t in batch["text"]]}, batched=True)
print(ds[0]["text"])  # REVIEW: COMPASSIONATELY ...
PY
```

## What I ran locally

The new lambda was executed against a small `Dataset.from_dict` to confirm it works end-to-end. `ruff check` and `ruff format --check` are green on the changed files.

---

🤖 Disclosure: Authored with assistance from Claude (Anthropic), reviewed and tested by me.
docs: fix broken WebDataset documentation link in audio/video/image dataset pages (#7699)

Fixes #7699.

## Summary

The link to the WebDataset documentation in the audio, video and image dataset docs pages points to `https://webdataset.github.io/webdataset`, which returns **HTTP 404**.

The WebDataset library now lives at `https://github.com/webdataset/webdataset`, which is the canonical documentation entry point (its README links to the getting-started guide and the API docs).

Three identical occurrences of the broken link are updated:
- `docs/source/audio_dataset.mdx`
- `docs/source/video_dataset.mdx`
- `docs/source/image_dataset.mdx`

## Reproduce BEFORE/AFTER yourself (copy-paste)

```bash
# --- BEFORE: origin/main ---
curl -sI https://webdataset.github.io/webdataset | head -1
# Expected: HTTP/2 404

# --- AFTER: this branch (no code to run, but the new link resolves) ---
curl -sI https://github.com/webdataset/webdataset | head -1
# Expected: HTTP/2 200
```

The hosted docs at https://huggingface.co/docs/datasets/main/en/video_dataset#webdataset will pick up the new URL after the next docs build.

---

🤖 Disclosure: Authored with assistance from Claude (Anthropic), reviewed by me.
Better error for sliced splits in streaming mode (`train[:10%]`, `train[:N]`) (#7721)

Fixes #7721.

## Summary

```python
load_dataset("user/dataset", split="train[0%:10%]", streaming=True)
```

raised the generic

```
ValueError: Bad split: train[0%:10%]. Available splits: ['train']
```

which makes it look like the slice syntax is malformed. The actual reason is that streaming mode doesn't know the dataset size until iteration, so percentages (`[:10%]`) and row indices (`[:90000]`) can't be translated to a slice ahead of time.

## Fix

When the split string is of the form `<known-split>[<slice>]`, raise a specific message that points at `.skip(...)` / `.take(...)` on the resulting `IterableDataset` (or `streaming=False`) as the streaming-friendly alternative. Other unrecognized splits still get the original message.

```
ValueError: Sliced splits like 'train[0%:10%]' (e.g. 'train[:10%]', 'train[:90000]') are not supported
with streaming=True because the dataset size is not known ahead of iteration.
Use streaming=True with split='train' and apply .skip(...) / .take(...) on the resulting IterableDataset
instead, or load with streaming=False.
```

## Reproduce BEFORE/AFTER yourself (copy-paste)

```bash
# --- BEFORE: origin/main, generic error ---
git fetch origin main && git checkout origin/main
python - <<'PY'
import json, tempfile
from datasets import load_dataset
d = tempfile.mkdtemp()
with open(d + "/data.json", "w") as f: json.dump([{"x": i} for i in range(100)], f)
try:
    load_dataset("json", data_files=[d + "/data.json"], split="train[:10%]", streaming=True)
except ValueError as e:
    print(f"BEFORE: {e}")  # Expected: "Bad split: train[:10%]. Available splits: ['train']"
PY

# --- AFTER: this branch, specific guidance ---
git fetch origin pull/<PR>/head:fix && git checkout fix
python - <<'PY'
import json, tempfile
from datasets import load_dataset
d = tempfile.mkdtemp()
with open(d + "/data.json", "w") as f: json.dump([{"x": i} for i in range(100)], f)
try:
    load_dataset("json", data_files=[d + "/data.json"], split="train[:10%]", streaming=True)
except ValueError as e:
    print(f"AFTER: {e}")  # Expected: "Sliced splits like 'train[:10%]' ... use .skip(...)/.take(...) ..."
PY
```

## What I ran locally

```
pytest tests/test_builder.py -k test_builder_as_streaming_dataset
# 6 passed
```

The four new parametrized regression cases (`train[0%:10%]`, `train[:90000]`, `train[10%]`, `train[10:20]`) fail on `origin/main` and pass on this branch. The unknown-split case (`split="nope"`) still produces the original error message — verified by a separate test.

## Notes

- This is a **diagnostic** change. The actual support for slice/percentage syntax in streaming would require knowing the dataset size, which is a bigger feature; the issue specifically asks for a clearer error.
- No public API change.

---

🤖 Disclosure: Authored with assistance from Claude (Anthropic), reviewed and tested by me.
docs: clarify `num_proc` semantics in `Dataset.batch` / `Dataset.filter` (#7700)

Fixes #7700.

## Summary

`Dataset.batch.num_proc` and `Dataset.filter.num_proc` had a one-liner docstring (\"Max number of processes when generating cache. Already cached shards are loaded sequentially.\") that left readers wondering whether `None` means "no multiprocessing" or "pick a number".

Both methods simply delegate to `Dataset.map`, which already has the precise semantics:

- `None` or `0` → no multiprocessing, run in the main process
- `> 1` → one or multiple worker processes process data in parallel

This PR mirrors that wording on `batch` and `filter` so the three sibling docstrings agree, matching the actual code path (`map` already normalizes `num_proc == 0` to `None` before dispatching, and `batch`/`filter` go through `map`).

## Reproduce BEFORE/AFTER yourself (copy-paste)

```bash
git fetch origin pull/<PR>/head:fix
diff <(git show origin/main:src/datasets/arrow_dataset.py | sed -n '/def batch/,/def map/p' | grep -A4 num_proc | head -8) \
     <(git show fix:src/datasets/arrow_dataset.py | sed -n '/def batch/,/def map/p' | grep -A4 num_proc | head -8)
```

## What I ran locally

- `ruff check` and `ruff format --check` are green on the changed file.
- No code change, so no behavior tests.

## Notes

- No code change, only docstring text on `Dataset.batch` and `Dataset.filter`.
- The actual `num_proc == 0` → `None` normalization is already done inside `Dataset.map`; the docstring just makes that visible to readers.

---

🤖 Disclosure: Authored with assistance from Claude (Anthropic), reviewed by me.
Load and save from/to disk no longer accept pathlib.Path

Reported by @vttrifonov at https://github.com/huggingface/datasets/pull/6704#issuecomment-2071168296:
> This change is breaking in 
> https://github.com/huggingface/datasets/blob/f96e74d5c633cd5435dd526adb4a74631eb05c43/src/datasets/arrow_dataset.py#L1515
> when the input is `pathlib.Path`. The issue is that `url_to_fs` expects a `str` and cannot deal with `Path`. `get_fs_token_paths` converts to `str` so it is not a problem

This change was introduced in:
- #6704 

Labels: name: bug

Opened a PR for this: #8004

Hi! Triaging older issues — I think this one has shipped and can be closed.

I tried to reproduce on the current `main` branch with the snippet below and both code paths now work:

```python
from pathlib import Path
import tempfile
from datasets import Dataset

ds = Dataset.from_dict({"x": [1, 2, 3]})
with tempfile.TemporaryDirectory() as d:
    p = Path(d) / "mydataset"
    ds.save_to_disk(p)              # → "save_to_disk(Path): OK"
    ds.save_to_disk(str(p) + "_str")  # → "save_to_disk(str): OK"
```

Tested against `datasets==4.8.6.dev0` (current `main`).

Would you mind closing this out? If something specific is still missing — a different `save_to_disk` / `load_from_disk` overload that still rejects `pathlib.Path` — point me at it and I'll dig further.

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; the reproducer above was run against `main` end-to-end.)
Audio mono argument no longer supported, despite class documentation

### Describe the bug

Either update the documentation, or re-introduce the flag (and corresponding logic to convert the audio to mono)

### Steps to reproduce the bug

Audio(sampling_rate=16000, mono=True) raises the error 
TypeError: Audio.__init__() got an unexpected keyword argument 'mono'

However, in the class documentation, is says:
Args:
        sampling_rate (`int`, *optional*):
            Target sampling rate. If `None`, the native sampling rate is used.
        mono (`bool`, defaults to `True`):
            Whether to convert the audio signal to mono by averaging samples across
            channels.
        [...]

### Expected behavior

The above call should either work, or the documentation within the Audio class should be updated

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-5.15.0-124-generic-x86_64-with-glibc2.35
- Python version: 3.12.11
- `huggingface_hub` version: 0.34.4
- PyArrow version: 21.0.0
- Pandas version: 2.3.2
- `fsspec` version: 2025.3.0


I want to solve this problem can you please assign it to me
and also can you please guide whether the mono parameter is required to be re-added or the documentation needs an update?

Hi! Triaging older issues — I think the docstring half of this is already resolved.

On current `main` (`datasets==4.8.6.dev0`), the `Audio` feature's `__init__` accepts `sampling_rate`, `num_channels`, `decode`, `stream_index`, `id` — and the docstring documents `num_channels` (not `mono`) with the explanation that `1` = mono, `2` = stereo, `None` = preserve source. There is no `mono` argument anywhere in `src/datasets/features/audio.py` and no leftover `mono` mention in the docstring.

I verified by grepping `src/datasets/features/audio.py` for `mono`: only one match remains, and it's the descriptive prose in the `num_channels` arg doc, not a parameter. So the doc-vs-code mismatch reported here is gone.

Would you mind closing this out? If `Audio(num_channels=1)` is still misbehaving for you in a way the docstring doesn't predict, that would be a separate bug worth a fresh issue (and there's already a related one for stereo at #8005, fwiw).

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; the grep above was run against current `main`.)
Reject `num_shards > len(dataset)` in `Dataset.shard` (#7443)

Fixes #7443.

## Summary

`Dataset.shard(num_shards=N, index=i)` with `N > len(dataset)`:
- silently produced empty shards (`contiguous=True`)
- crashed with `IndexError: Index ... out of range for dataset of size ...` deep in `.select` (`contiguous=False`)

`push_to_hub` / `save_to_disk` then surfaced this confusing IndexError after partial work was already done.

Raise an informative `ValueError` upfront. Empty datasets are exempt so pipeline code that legitimately calls `shard(1, 0)` on an empty result still works.

## Reproduce BEFORE/AFTER

```bash
# --- BEFORE: origin/main, IndexError ---
git fetch origin main && git checkout origin/main
python - <<'PY'
from datasets import Dataset
ds = Dataset.from_dict({"x": [1, 2, 3]})
ds.shard(num_shards=10, index=5)
PY

# --- AFTER: this branch, clear ValueError ---
# ValueError: num_shards (10) must be smaller than or equal to the number of rows in the dataset (3). ...
```

## What I ran

```
pytest tests/test_arrow_dataset.py -k test_shard
# 4 passed
```

The new `test_shard_rejects_num_shards_greater_than_len` covers `contiguous=True`/`False`, the boundary `num_shards == len`, and the empty-dataset exemption. Fails on `origin/main`, passes on this branch.

---

🤖 Disclosure: Authored with assistance from Claude (Anthropic), tested locally.
Preserve `info.features` across `IterableDataset.map(remove_columns=...)` (#7568)

Fixes #7568.

## Summary

`IterableDataset.map(remove_columns=[...])` (without an explicit `features=` and without a user-provided `function`) overwrites `info.features` with `None`, so `IterableDataset.column_names` (which reads `info.features.keys()`) silently becomes `None`. The same regression bubbles up through `IterableDatasetDict.map`.

## Fix

When the caller didn't pass `function` (the identity map case), carry over the existing schema minus the dropped columns. For arbitrary-`function` calls the resulting schema is unknown ahead of iteration, so we preserve the prior conservative `info.features = None` behavior to avoid lying about the columns (this is what `test_map_async` and friends rely on).

```python
function_was_none = function is None
if function is None:
    function = identity_func
...
info.features = features
if features is None and function_was_none and remove_columns is not None and self.info.features is not None:
    removed = {remove_columns} if isinstance(remove_columns, str) else set(remove_columns)
    info.features = Features({k: v for k, v in self.info.features.items() if k not in removed})
```

## Reproduce BEFORE/AFTER

```bash
# --- BEFORE: origin/main, column_names becomes None ---
git fetch origin main && git checkout origin/main
python - <<'PY'
from datasets import IterableDataset, IterableDatasetDict, Features, Value
ds = IterableDataset.from_generator(
    lambda: ({"text": f"hi {i}", "foo": i} for i in range(3)),
    features=Features({"text": Value("string"), "foo": Value("int64")}),
)
dd = IterableDatasetDict({"train": ds})
print(dd.map(remove_columns=["foo"])["train"].column_names)  # Expected: None
PY

# --- AFTER: this branch ---
# Expected: ['text']
```

## What I ran

```
pytest tests/test_iterable_dataset.py -k "map"
# 78 passed
```

The new `test_iterable_dataset_map_remove_columns_preserves_features` covers identity-map + `remove_columns`, the `remove_columns="foo"` (single string) variant, and the explicit-function case (which still gets `features=None`). Fails on `origin/main`, passes on this branch.

## Notes

- Doesn't address the broader case where the user passes a function that doesn't change the schema — that requires runtime inference which the iterable code already avoids by design.

---

🤖 Disclosure: Authored with assistance from Claude (Anthropic), tested locally.
Escape glob chars in `base_path` so directory paths with `[]` work (#7468)

Fixes #7468.

## Summary

`load_dataset(..., data_dir="/path/[D_DATA]/foo")` — or any path with `[`, `]`, `*`, `?` in the literal directory name — failed: `fs.glob` interpreted the bracketed segment as a glob character class, walked a much larger tree (often the whole disk on Windows) and returned no files. Users saw an `EmptyDatasetError` after a long hang.

## Fix

Pass the `base_path` through `glob.escape` before joining it with the user's pattern so the directory portion is treated literally. The pattern part keeps its glob semantics.

```python
if is_relative_path(pattern):
    if is_local_path(base_path):
        base_path = glob_escape(base_path)
    pattern = xjoin(base_path, pattern)
```

## Reproduce BEFORE/AFTER

```bash
# --- BEFORE: origin/main, EmptyDatasetError after a long hang ---
git fetch origin main && git checkout origin/main
python - <<'PY'
import os, tempfile, json
from datasets import load_dataset
tmp = tempfile.mkdtemp()
weird = os.path.join(tmp, "[D_DATA]", "test")
os.makedirs(weird)
with open(os.path.join(weird, "data.json"), "w") as f: json.dump([{"x": 1}], f)
load_dataset("json", data_dir=weird, split="train")  # EmptyDatasetError
PY

# --- AFTER: this branch — loads in <1s ---
```

## What I ran

```
pytest tests/test_data_files.py
# 224 + the new params passed
```

The new `test_resolve_pattern_locally_base_path_with_glob_chars` covers `[`/`]` in the directory name with both `*` and `*.txt` patterns. Fails on `origin/main`, passes on this branch.

---

🤖 Disclosure: Authored with assistance from Claude (Anthropic), tested locally end-to-end.
`datasets.disable_caching` doesn't work

When I use `Dataset.from_generator(my_gen)` to load my dataset, it simply skips my changes to the generator function.
I tried `datasets.disable_caching`, but it doesn't work!


cc

Yes I have the same issue.  It's a confusingly named function.  See [here](https://github.com/huggingface/datasets/blob/main/src/datasets/fingerprint.py#L115-L130)

```
...
If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
    More precisely, if the caching is disabled:
    - cache files are always recreated
    - cache files are written to a temporary directory that is deleted when session closes
    - cache files are named using a random hash instead of the dataset fingerprint
```

Also, unfortunately the member variable `ds.cache_files` is not populated either.

I'll let you know if I find a solution.

Hi! Triaging older issues — I think this is a docs/UX clarification rather than a fix, and the issue can be closed once the doc is improved.

`datasets.disable_caching()` does work, but the name is misleading: per the docstring of `disable_caching` in `src/datasets/fingerprint.py`, it disables **reusing** existing cache files for transforms; cache files are still written, but to a temp dir with a random hash:

```text
If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
More precisely, if the caching is disabled:
- cache files are always recreated
- cache files are written to a temporary directory that is deleted when session closes
- cache files are named using a random hash instead of the dataset fingerprint
```

So when you call `Dataset.from_generator(my_gen)`, the change you made to `my_gen` wasn't picked up because the **fingerprint** hashed differently in your two runs (so the cache miss should already trigger recompute) — or, conversely, hashed identically (because dill / closure capture coincided). `disable_caching` doesn't influence which path you hit there because `from_generator` decides at the loader level, not the transform level.

For the specific "I changed my generator, force re-run" use case, a more reliable lever is one of:

- pass `download_mode="force_redownload"` (or a new `cache_dir=`) to invalidate the fingerprint
- pass an explicit `new_fingerprint=...` per call
- bump a `gen_kwargs` entry your generator captures, so the hash changes deterministically

Closing-nudge: would you mind closing this if the above explains your symptom? Happy to open a docs PR if you think the `disable_caching` docstring should call out that it does **not** force-recompute `from_generator` outputs.

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; the docstring above is verbatim from `src/datasets/fingerprint.py:115-130` on current `main`.)