noamaanMulla-03/datasets-issues huggingface dataset:
[bug] resume from ckpt skips samples if .map is applied

### Describe the bug

resume from ckpt skips samples if .map is applied

Maybe related: https://github.com/huggingface/datasets/issues/7538

### Steps to reproduce the bug

```python
from datasets import Dataset
from datasets.distributed import split_dataset_by_node


# Create dataset with map transformation
def create_dataset():
    ds = Dataset.from_dict({"id": list(range(100))})
    ds = ds.to_iterable_dataset(num_shards=4)
    ds = ds.map(lambda x: x) #comment it out to get desired behavior
    ds = split_dataset_by_node(ds, rank=0, world_size=2)
    return ds


ds = create_dataset()

# Iterate and save checkpoint after 10 samples
it = iter(ds)
for idx, sample in enumerate(it):
    if idx == 9:  # Checkpoint after 10 samples
        checkpoint = ds.state_dict()
        print(f"Checkpoint saved at sample: {sample['id']}")
        break

# Continue with original iterator
original_next_samples = []
for idx, sample in enumerate(it):
    original_next_samples.append(sample["id"])
    if idx >= 4:
        break

# Resume from checkpoint
ds_new = create_dataset()
ds_new.load_state_dict(checkpoint)

# Get samples from resumed iterator
it_new = iter(ds_new)
resumed_next_samples = []
for idx, sample in enumerate(it_new):
    resumed_next_samples.append(sample["id"])
    if idx >= 4:
        break

print(f"\nExpected next samples: {original_next_samples}")
print(f"Actual next samples:   {resumed_next_samples}")
print(
    f"\n❌ BUG: {resumed_next_samples[0] - original_next_samples[0]} samples were skipped!"
)

```

With map
```
Checkpoint saved at sample: 9

Expected next samples: [10, 11, 12, 13, 14]
Actual next samples:   [50, 51, 52, 53, 54]

❌ BUG: 40 samples were skipped!
```

### Expected behavior

without map
```
Expected next samples: [10, 11, 12, 13, 14]
Actual next samples:   [10, 11, 12, 13, 14]

❌ BUG: 0 samples were skipped!
```

### Environment info

datasets == 3.6.0


Thanks for reporting this — it looks like a separate but related bug to #7538, which involved sample loss when resuming an `IterableDataset` wrapped in `FormattedExamplesIterable`. That was resolved in #7553 by re-batching the iterable to track offset correctly.

In this case, the issue seems to arise specifically from applying `.map()` before sharding and checkpointing. That wraps the iterable in `MappedExamplesIterable`, which may not preserve or propagate `shard_example_idx` correctly across `.state_dict()` and `.load_state_dict()` calls.

You can see that without `.map()`, resume works fine — but with `.map()`, it jumps from sample 9 to 50, skipping the rest of the shard.

I'll dig deeper into how `MappedExamplesIterable` manages offsets and whether it supports proper checkpoint resumption. If not, we might need a fix similar to the one in #7553, or a wrapper to preserve resume metadata.

Happy to help fix it!


Let me know if a dedicated test case is required — happy to add one!

When will this be fixed ?
cast columns to Image/Audio/Video with `storage_options`

### Feature request

Allow `storage_options` to be passed in
1. `cast` related operations (e.g., `cast_columns, cast`)
2. `info` related reading (e.g., `from_dict, from_pandas, from_polars`) together with `info.features`
```python3
import datasets

image_path = "s3://bucket/sample.png"
dataset = datasets.Dataset.from_dict({"image_path": [image_path]})

# dataset = dataset.cast_column("image_path", datasets.Image())  # now works without `storage_options`

# expected behavior
dataset = dataset.cast_column("image_path", datasets.Image(), storage_options={"anon": True})
```

### Motivation

I'm using my own registered fsspec filesystem (s3 with customized local cache support). I need to pass cache folder paths `cache_dirs: list[str]` to the filesystem when I read the remote images (cast from file_paths).

### Your contribution

Could help with a PR at weekends

Labels: name: enhancement

A

1

1

Ok

> ### Feature request
> Allow `storage_options` to be passed in
> 
> 1. `cast` related operations (e.g., `cast_columns, cast`)
> 2. `info` related reading (e.g., `from_dict, from_pandas, from_polars`) together with `info.features`
> 
> import datasets
> 
> image_path = "s3://bucket/sample.png"
> dataset = datasets.Dataset.from_dict({"image_path": [image_path]})
> 
> # dataset = dataset.cast_column("image_path", datasets.Image())  # now works without `storage_options`
> 
> # expected behavior
> dataset = dataset.cast_column("image_path", datasets.Image(), storage_options={"anon": True})
> ### Motivation
> I'm using my own registered fsspec filesystem (s3 with customized local cache support). I need to pass cache folder paths `cache_dirs: list[str]` to the filesystem when I read the remote images (cast from file_paths).
> 
> ### Your contribution
> Could help with a PR at weekends



>

#7910
fix: Dataset.map writer initialization when first examples return None

Fixes #7990

## Summary

When Dataset.map is called and the first examples processed return None, the writer is never properly initialized, causing a ValueError.

## Changes

- Modified _map_single to initialize the writer early if the first batch returns empty results
- Ensures writer is set before the first call to writer.write_batch

## Test Plan

- Added test case that reproduces the bug
- Verified the fix resolves the issue
- Existing tests still pass


Closing in favor of #7996 which addresses the same issue with the same fix.
The `Sequence` class in features do not have "dtype"

### Describe the bug

**I'm not sure if this is a bug.**

I see that a `FeatureType` object contains an attribute called `self.dtype` that is not covered when this feature is a `Sequence` or a `List`.

When I try to run a multilabel classification with this example script from the transformers library:

https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_classification.py#L442

I get this error on the linked line:

```shell
AttributeError: 'List' object has no attribute 'dtype'. Did you mean: '_type'?
```

Looking at the check that the script is attempting to perform, could we perhaps add a `self.dtype="list"` attribute for this `FeatureType` 's: `Sequence`, `List`, etc.?

### Steps to reproduce the bug

For example, this code works for me:

```python
from datasets import ClassLabel, Features, Sequence, Value
features = {'text': Value('string'), 'label': ClassLabel(names=['No', 'Yes'])}
print(features["text"].dtype)
print(features["label"].dtype)
```
```output
'string'
'int64'
```

and this code does not work for me:

```python
from datasets import ClassLabel, Features, Sequence, Value
features = {'text': Value('string'), 'label': Sequence(ClassLabel(names=['No', 'Yes']))}
print(features["label"].dtype) # it could be equal to "list"?
```
```output
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'List' object has no attribute 'dtype'. Did you mean: '_type'?
```

### Expected behavior

The attribute `dtype` equal to `"list"` when using objects of type `Sequence`. 

```python
from datasets import ClassLabel, Features, Sequence, Value
features = {'text': Value('string'), 'label': Sequence(ClassLabel(names=['No', 'Yes']))}
print(features["label"].dtype)
```
```output
'list'
```

### Environment info

I have installed `datasets==4.5.0`.
fix save_to_disk/load_from_disk with pathlib.Path input

Since #6704, `save_to_disk` and `load_from_disk` use `fsspec.core.url_to_fs` which expects a `str`, but both methods accept `PathLike` (which includes `pathlib.Path`). Passing a `Path` object raises a `TypeError` because `url_to_fs` can't handle it.

Fixed by converting the path argument with `os.fspath()` before handing it off to `url_to_fs`. This affects all five call sites across `Dataset`, `DatasetDict`, and the standalone `load_from_disk` function.

Fixes #6829
Too many dataloader workers: 4 (max is dataset.num_shards=3). Stopping 1 dataloader workers.

Hi !
I’m working on training with a large-scale dataset (100+ Parquet files) using lazy loading, and I’m struggling to understand/optimize the num_shards setting— in the lerobot repo: streaming_datasets.py:
```
from datasets import load_dataset
self.hf_dataset: datasets.IterableDataset = load_dataset(
            self.repo_id if not self.streaming_from_local else str(self.root),
            split="train",
            streaming=self.streaming,
            data_files="data/*/*.parquet",
            revision=self.revision,
        )

self.num_shards = min(self.hf_dataset.num_shards, max_num_shards)
```

```
dataloader = torch.utils.data.DataLoader(
            datasets[sub_idx],
            num_workers=datasets[sub_idx].num_shards, #cfg.num_workers,
            batch_size=cfg.batch_size,
            shuffle=shuffle and not cfg.dataset.streaming,
            sampler=sampler,
            collate_fn=FlowerDataCollator(),
            pin_memory=device.type == "cuda",
            drop_last=True,
            prefetch_factor=2 if cfg.num_workers > 0 else None,
            )
```

What exactly does hf_dataset.**num_shards** represent? Is it safe to manually override/edit num_shards?

My batch loading is slower than expected (2-3s per batch) despite num_worker cannot be bigger with warning: Too many dataloader workers: 4 (max is dataset.num_shards=3). Stopping 1 dataloader workers. 

Even use num_workers=datasets[sub_idx].num_shards, the waring is still exist! (my num_worker is 4 and hf_dataset.num_shards is 100+, so the datasets[sub_idx].num_shards=4)
Why does the "too many workers" warning persist even when num_workers equals dataset.num_shards—and how do I fix this?

Thanks so much for any insights or help with this! Really appreciate your time and expertise 😊




Hi, thanks for the clear question and code snippet!

From my understanding, **hf_dataset.num_shards** represents the number of actual iterable partitions that the streaming dataset can be read from in parallel. This is not the same as the number of underlying Parquet files—many files can be grouped into a much smaller number of shards internally.

The warning seems to persist because **datasets[sub_idx]** is a sliced or sharded view of the original dataset, and its effective num_shards can be smaller than expected after this step. So even if **num_workers** is set equal to **datasets[sub_idx].num_shards**, additional internal sharding can still result in fewer usable shards, which triggers the warning.

Because of this, manually overriding **num_shards** does not appear to increase real parallelism and may just create extra workers that stay idle. For streaming datasets, it seems safer to let the dataset control sharding and to cap the number of workers to the effective shard count (or even use fewer workers and adjust batch size instead), since performance is often I/O-bound.

Hope this helps clarify what’s going on!
Pass datasets-cli additional args as kwargs to DatasetBuilder in `run_beam.py`

Hi,

Following this <https://discuss.huggingface.co/t/how-to-preprocess-a-wikipedia-dataset-using-dataflowrunner/41991/3>, here is a simple PR to pass any additional args to datasets-cli as kwargs in the DatasetBuilder in `run_beam.py`.

I also took the liberty to add missing setup steps to the `beam.mdx` docs in order to help everyone.

@lhoestq 


Sorry, I was way too long.
Bump fsspec upper bound to 2026.2.0 (fixes #7994)

Fixes #7994 and e.g. “Bumps fsspec upper bound so the latest version can be used; CI will validate compatibility.”


Apologize for the ping @lhoestq but I wanted to know whether this is something you could look into? And more generally whether you have a policy to regularly bump fsspec? Thanks 🙏

thanks for opening a PR, I'm triggering the CI and I'll merge if it's all green :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7995). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Bump fsspec upper bound constraint

Would it be possible to bump fsspec upper bound to the latest (2026.2.0)?

I saw you had some API compat issues in the past (https://github.com/huggingface/datasets/issues/7326) and I understand the need for an upper bound.

But I wonder if you think your CI and tests are a good proxy to catch fsspec API breakage? If that's the case then triggering dataset CI with the latest version of fsspec should tell us whether it's all good right?

Happy to open a PR if needed.
Fix: make environment variable naming consistent (issue #7998)

Fix: https://github.com/huggingface/datasets/issues/7998

**Summary**

Addressed the inconsistent environment variable name HF_DATASETS_DISABLE_PROGRESS_BARS used for toggling progress bars, particularly in docstrings and warning messages.

**Changes**

Updated function docstrings to use the correct environment variable name

Corrected warning messages in tqdm.py to ensure consistency


Hi @lhoestq ,
I’ve addressed the inconsistent environment variable name HF_DATASETS_DISABLE_PROGRESS_BARS in docstrings and warning messages. Please review the changes and let me know if there’s anything else I should update to complete this issue fix.
[doc] Inconsistant ENV VAR Name for Progress Bar Toggle

Code uses env var name `HF_DATASETS_DISABLE_PROGRESS_BARS`.

https://github.com/huggingface/datasets/blob/025593f2f0722f31fc136e0ae45da4ff44d4416a/src/datasets/config.py#L221-L226

Docstrings and warnings report env var name `HF_DATASETS_DISABLE_PROGRESS_BAR` without the ending `S`.

https://github.com/huggingface/datasets/blob/025593f2f0722f31fc136e0ae45da4ff44d4416a/src/datasets/utils/tqdm.py#L61-L73

https://github.com/huggingface/datasets/blob/025593f2f0722f31fc136e0ae45da4ff44d4416a/src/datasets/utils/tqdm.py#L78-L90

https://github.com/huggingface/datasets/blob/025593f2f0722f31fc136e0ae45da4ff44d4416a/src/datasets/utils/tqdm.py#L95-L100

This affects doc webpages as well, e.g., see https://huggingface.co/docs/datasets/en/package_reference/utilities#datasets.enable_progress_bars.
fix: condition for file_name feature check

## Description
Accept `Value("large_string")` in addition to `Value("string")` for the `file_name` (or `*_file_name`) metadata key in folder-based builders (e.g. `audiofolder`).

## Problem
When loading a dataset with `load_dataset("audiofolder", data_dir=...)` and a `metadata.csv` file, the metadata schema is inferred via pandas to Arrow. For string columns, Arrow often uses the `large_string` type, so `Features.from_arrow_schema()` yields `Value("large_string")` for the `file_name` column. The builder only checks for `Value("string")`, so validation fails with:

`ValueError: file_name or *_file_name must be present as dictionary key (with type string) in metadata files`

Users then have to convert metadata to JSONL (which produces Arrow `string`) or work around the loader.

## Solution
Treat `Value("large_string")` as valid for the `file_name` / `*_file_name` key, since it is still a string type and behaves the same for resolution of audio/image paths. This matches the fact that when metadata is loaded from CSV (via pandas and then converted to Arrow), string columns are often inferred as Arrow `large_string`, so accepting it keeps CSV metadata working without format changes.


Hi @lhoestq, have proposed a fix for [#8010](https://github.com/huggingface/datasets/issues/8010)
audiofolder / folder-based loader rejects metadata.csv when file_name column is inferred as large_string

### Describe the bug

When loading a dataset with load_dataset("audiofolder", data_dir=...) and a metadata.csv (with a file_name column), loading can fail with:

`ValueError: file_name or *_file_name must be present as dictionary key (with type string) in metadata files`

even when the CSV clearly has a file_name column with string values.

**Cause**

In folder_based_builder.py, the metadata schema is inferred from the CSV via pandas to Arrow. For string columns, Arrow often uses the large_string type. The check only accepts Value("string") and does not accept Value("large_string"), so validation fails even though the column is a valid string column.


### Steps to reproduce the bug
1. Create a folder with:
   - `metadata.csv` with columns: `file_name`, `transcript`, `duration` (and audio `.wav` files referenced by `file_name`).
2. Run:
```
   from datasets import load_dataset
   ds = load_dataset("audiofolder", data_dir="/path/to/folder")
```

### Expected behavior

Metadata CSV with a `file_name` column (inferred as either `string` or `large_string`) should be accepted, since both represent string data.

### Environment info


- `datasets` version: 4.5.0
- Platform: Linux-6.8.0-100-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.36.0
- PyArrow version: 23.0.0
- Pandas version: 3.0.0
- `fsspec` version: 2025.10.0


Proposing a PR for this [here](https://github.com/huggingface/datasets/pull/8011)

PS: I didn't find a contributing documentation. Please recommend any changes in issue or PR if any. Thank you!
HC3 dataset fails to load with datasets>=3 due to legacy script (HC3.py)

### Describe the bug

It looks like the current documentation suggests loading the dataset like this:

`load_dataset("Hello-SimpleAI/HC3", split="train")
`

But with datasets>=3.x, this raises:

RuntimeError: Dataset scripts are no longer supported, but found HC3.py


The dataset loads correctly if we specify the parquet export branch:

```
load_dataset(
    "Hello-SimpleAI/HC3",
    revision="refs/convert/parquet",
    split="train"
)

```

Since dataset scripts are no longer supported in newer versions of datasets, it might be better to update the documentation to reflect the parquet-based loading method .

Just wanted to flag this so others don’t run into the same confusion.

### Steps to reproduce the bug

### Steps to reproduce

Using `datasets>=3.x`:

```python
from datasets import load_dataset

ds = load_dataset("Hello-SimpleAI/HC3", split="train")
```

### Error

```
RuntimeError: Dataset scripts are no longer supported, but found HC3.py
```

This appears to happen because the repository still contains a legacy loading script (`HC3.py`), and recent versions of `datasets` no longer support dataset scripts.

### Working alternative

The dataset loads correctly when specifying the parquet export revision:

```python
from datasets import load_dataset

ds = load_dataset(
    "Hello-SimpleAI/HC3",
    revision="refs/convert/parquet",
    split="train"
)

print(ds)
```

This successfully downloads and loads the dataset.


### Expected behavior

### Expected behavior

The dataset should load successfully using the example shown in the documentation:

```python
load_dataset("Hello-SimpleAI/HC3", split="train")
```

without requiring a specific `revision` parameter.



### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-6.6.105+-x86_64-with-glibc2.35
- Python version: 3.12.12
- `huggingface_hub` version: 1.4.1
- PyArrow version: 18.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2025.3.0


This is not a bug. This is intended behaviour. The only ways to resolve this are
1. use the old `datasets` versions
2. load the dataset and upload it to your own repo without a script
3. raise an issue in the dataset and wait for the owner to update it.
Flexible Loader

### Feature request

Can we have a utility function that will use `load_from_disk` when given the local path and `load_dataset` if given an HF dataset?

It can be something as simple as this one:

```
def load_hf_dataset(path_or_name):
    if os.path.exists(path_or_name):
        return load_from_disk(path_or_name)
    else:
        return load_dataset(path_or_name)
```

### Motivation

This can be done inside the user codebase, too, but in my experience, it becomes repetitive code.

### Your contribution

I can open a pull request.

Labels: name: enhancement

Ideally `save_to_disk` should save in a format compatible with load_dataset, wdyt ?

> Ideally `save_to_disk` should save in a format compatible with load_dataset, wdyt ?

That would be perfect if not at least a flexible loader.

@lhoestq For now, you can use this small utility library: [nanoml](https://pypi.org/project/nanoml/)
```python
from nanoml.data import load_dataset_flexible
```

I actively develop and maintain this utility library. Open to contributors. Please open issues, PR, or feature requests.
feat: add return_file_name support to CSV packaged builder

## Summary

- Adds an optional `return_file_name: bool = False` parameter to `CsvConfig`
- When `return_file_name=True`, a `file_name` column is appended to every batch in `_generate_tables`, containing the basename of the source CSV file for each row
- Default is `False`, preserving full backward compatibility

## Motivation

Part of #5806. Extends the `file_name` feature (already implemented for the JSON packaged builder in #7948) to the CSV packaged builder. This enables use cases such as resuming training from checkpoints by identifying which data shards have already been consumed.

## Changes

- `src/datasets/packaged_modules/csv/csv.py`: Add `return_file_name: bool = False` field to `CsvConfig`; in `_generate_tables`, append a `file_name` column when the flag is `True`
- `tests/packaged_modules/test_csv.py`: Add three tests covering default behavior (no column), enabled behavior (column present), and correct column values

## Test plan

- [ ] `test_csv_no_file_name_by_default` — verifies `file_name` column is absent by default
- [ ] `test_csv_return_file_name_enabled` — verifies `file_name` column is present when `return_file_name=True`
- [ ] `test_csv_file_name_values` — verifies each row's `file_name` value equals the source file basename

All three tests pass locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Remove legacy Sphinx roles from docstrings

## Summary

Replace all remaining `:obj:`, `:class:`, `:func:`, and `:meth:` Sphinx cross-reference roles with plain backtick references, aligning with the project's current documentation conventions.

**14 files, 71 replacements. No logic changes.**

## Context

As noted in #5324, the codebase had a mix of old Sphinx syntax and the modern backtick-only format used by HuggingFace's doc builder. @stevhliu [mentioned](https://github.com/huggingface/datasets/issues/5324#issuecomment-1525157369) that the user-facing APIs had been cleaned up, but old syntax was still lingering in non-public APIs. This PR catches the remaining occurrences.

## What changed

| Old syntax | New syntax | Count |
|---|---|---|
| `:obj:\`bool\`` | `` `bool` `` | 18 |
| `:class:\`DatasetInfo\`` | `` `DatasetInfo` `` | 15 |
| `:func:\`fsspec.open\`` | `` `fsspec.open` `` | 9 |
| `:meth:\`...\`` | `` `...` `` | 2 |
| `:obj:\`~pathlib.Path\`` | `` `Path` `` | 8 |
| `:class:\`~download.DownloadConfig\`` | `` `DownloadConfig` `` | 3 |
| *other* | — | 16 |

Tilde references (`~module.Class`) are resolved to their short name (`Class`), matching the existing convention in the codebase.

## Files touched

```
src/datasets/arrow_dataset.py
src/datasets/arrow_writer.py
src/datasets/dataset_dict.py
src/datasets/features/features.py
src/datasets/filesystems/compression.py
src/datasets/fingerprint.py
src/datasets/formatting/formatting.py
src/datasets/inspect.py
src/datasets/iterable_dataset.py
src/datasets/load.py
src/datasets/streaming.py
src/datasets/table.py
src/datasets/utils/deprecation_utils.py
src/datasets/utils/file_utils.py
```

Relates to #5324