NyssaW/github-issues huggingface dataset:
progress bar optional manual control

Fix: https://github.com/huggingface/datasets/issues/7939

Summary

The default behavior for the progress bar has been adjusted based on dataset size. It remains enabled for small datasets and is automatically disabled for larger datasets to improve performance. Users retain full control and can explicitly enable or disable the progress bar using the progress_bar parameter.

Changes

Modified the logic to disable the progress bar when the number of files exceeds 16.

Comments:
Hi @lhoestq ,
Can you please trigger the CI CD pipeline ?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8021). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq I have applied the required Ruff formatting changes in the` load.py` file.
Could you please trigger the CI/CD pipeline again?

HI @lhoestq ,
Just a gentle reminder to please trigger the CI/CD pipeline for this PR..!!

HI @julien-c ,
can you please trigger the the CI/CD pipeline for this PR.

Hi @albertvillanova ,
Could you please trigger the CI/CD pipeline?, i can see it is pending from long time 
Thanks 

I don't think we want to add more arguments to this method.
Progress bars can already be disabled like this

```python
from datasets.utils import disable_progress_bars

disable_progress_bars()
```

Hey @lhoestq ,
My intention was to provide more fine-grained control at the method level, particularly for multiprocessing scenarios where multiple progress bars can clutter the logs. This would give users the flexibility to explicitly enable or disable the progress bar as needed,which aligns with the concern raised in the issue https://github.com/huggingface/datasets/issues/7939
feat: add return_file_name support to CSV packaged builder

## Summary

- Adds an optional `return_file_name: bool = False` parameter to `CsvConfig`
- When `return_file_name=True`, a `file_name` column is appended to every batch in `_generate_tables`, containing the basename of the source CSV file for each row
- Default is `False`, preserving full backward compatibility

## Motivation

Part of #5806. Extends the `file_name` feature (already implemented for the JSON packaged builder in #7948) to the CSV packaged builder. This enables use cases such as resuming training from checkpoints by identifying which data shards have already been consumed.

## Changes

- `src/datasets/packaged_modules/csv/csv.py`: Add `return_file_name: bool = False` field to `CsvConfig`; in `_generate_tables`, append a `file_name` column when the flag is `True`
- `tests/packaged_modules/test_csv.py`: Add three tests covering default behavior (no column), enabled behavior (column present), and correct column values

## Test plan

- [ ] `test_csv_no_file_name_by_default` — verifies `file_name` column is absent by default
- [ ] `test_csv_return_file_name_enabled` — verifies `file_name` column is present when `return_file_name=True`
- [ ] `test_csv_file_name_values` — verifies each row's `file_name` value equals the source file basename

All three tests pass locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Comments:
Improve error message for deprecated dataset scripts with migration guidance

Summary
This PR improves the RuntimeError message raised when deprecated local dataset scripts are detected.
Previously, the error message:
    "Dataset scripts are no longer supported"
did not provide actionable migration guidance.
Improvements
- Clarifies the new architecture
- Provides explicit migration steps
- Includes link to official documentation
- Adds a focused unit test to validate error clarity
This should reduce confusion for contributors migrating legacy dataset scripts.

Comments:
Hi team! I’m just checking in on this PR to see if there is any additional information or documentation I can provide to help with the review process. Thanks for your time!
Remove legacy Sphinx roles from docstrings

## Summary

Replace all remaining `:obj:`, `:class:`, `:func:`, and `:meth:` Sphinx cross-reference roles with plain backtick references, aligning with the project's current documentation conventions.

**14 files, 71 replacements. No logic changes.**

## Context

As noted in #5324, the codebase had a mix of old Sphinx syntax and the modern backtick-only format used by HuggingFace's doc builder. @stevhliu [mentioned](https://github.com/huggingface/datasets/issues/5324#issuecomment-1525157369) that the user-facing APIs had been cleaned up, but old syntax was still lingering in non-public APIs. This PR catches the remaining occurrences.

## What changed

| Old syntax | New syntax | Count |
|---|---|---|
| `:obj:\`bool\`` | `` `bool` `` | 18 |
| `:class:\`DatasetInfo\`` | `` `DatasetInfo` `` | 15 |
| `:func:\`fsspec.open\`` | `` `fsspec.open` `` | 9 |
| `:meth:\`...\`` | `` `...` `` | 2 |
| `:obj:\`~pathlib.Path\`` | `` `Path` `` | 8 |
| `:class:\`~download.DownloadConfig\`` | `` `DownloadConfig` `` | 3 |
| *other* | — | 16 |

Tilde references (`~module.Class`) are resolved to their short name (`Class`), matching the existing convention in the codebase.

## Files touched

```
src/datasets/arrow_dataset.py
src/datasets/arrow_writer.py
src/datasets/dataset_dict.py
src/datasets/features/features.py
src/datasets/filesystems/compression.py
src/datasets/fingerprint.py
src/datasets/formatting/formatting.py
src/datasets/inspect.py
src/datasets/iterable_dataset.py
src/datasets/load.py
src/datasets/streaming.py
src/datasets/table.py
src/datasets/utils/deprecation_utils.py
src/datasets/utils/file_utils.py
```

Relates to #5324

Comments:
Speed up local 'get_data_patterns' by avoiding repeated recursive scans

This PR speeds up `get_data_patterns` for local paths.

### Problem

For `load_dataset("imagefolder", data_dir=...)`, `get_data_patterns` was repeatedly scanning the same local directory tree for many split patterns (`train`, `test`, etc.).  
With large folders, this became very slow. This has also been reported before in earlier performance discussions/issues.

### Change

In `get_data_patterns` (local paths only):

- scan files once (`resolve_pattern("**", ...)`)
- then match split patterns in memory

Remote paths keep the old behavior.

No API changes.

### My Env
- Mac mini (M4), 16 GB RAM
- Python 3.13
- Datasets version: 4.5.0

### Benchmarks

`imagefolder` with local `.jpg` files

`data_dir_only`:
- 10k: `4.35s -> 1.40s` (3.10x)
- 100k: `33.48s -> 9.77s` (3.43x)
- 300k: `160.20s -> 35.81s` (4.47x)
- 1M: `1877.70s -> 164.17s` (11.44x) :fire:

`explicit_data_files`:
- 10k: `0.75s -> 0.79s`
- 100k: `7.23s -> 7.29s`
- 300k: `25.44s -> 24.26s`
- 1M: `115.85s -> 112.66s`

As expected, the improvement is on `data_dir_only` (auto pattern detection path).
Memory usage did not show a consistent regression in these runs and stayed within normal run-to-run variance for this benchmark setup.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8014). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> lgtm ! can you run `make style` to fix the CI before we merge ?

done. All good, no warnings.
HC3 dataset fails to load with datasets>=3 due to legacy script (HC3.py)

### Describe the bug

It looks like the current documentation suggests loading the dataset like this:

`load_dataset("Hello-SimpleAI/HC3", split="train")
`

But with datasets>=3.x, this raises:

RuntimeError: Dataset scripts are no longer supported, but found HC3.py


The dataset loads correctly if we specify the parquet export branch:

```
load_dataset(
    "Hello-SimpleAI/HC3",
    revision="refs/convert/parquet",
    split="train"
)

```

Since dataset scripts are no longer supported in newer versions of datasets, it might be better to update the documentation to reflect the parquet-based loading method .

Just wanted to flag this so others don’t run into the same confusion.

### Steps to reproduce the bug

### Steps to reproduce

Using `datasets>=3.x`:

```python
from datasets import load_dataset

ds = load_dataset("Hello-SimpleAI/HC3", split="train")
```

### Error

```
RuntimeError: Dataset scripts are no longer supported, but found HC3.py
```

This appears to happen because the repository still contains a legacy loading script (`HC3.py`), and recent versions of `datasets` no longer support dataset scripts.

### Working alternative

The dataset loads correctly when specifying the parquet export revision:

```python
from datasets import load_dataset

ds = load_dataset(
    "Hello-SimpleAI/HC3",
    revision="refs/convert/parquet",
    split="train"
)

print(ds)
```

This successfully downloads and loads the dataset.


### Expected behavior

### Expected behavior

The dataset should load successfully using the example shown in the documentation:

```python
load_dataset("Hello-SimpleAI/HC3", split="train")
```

without requiring a specific `revision` parameter.



### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-6.6.105+-x86_64-with-glibc2.35
- Python version: 3.12.12
- `huggingface_hub` version: 1.4.1
- PyArrow version: 18.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2025.3.0

Comments:
This is not a bug. This is intended behaviour. The only ways to resolve this are
1. use the old `datasets` versions
2. load the dataset and upload it to your own repo without a script
3. raise an issue in the dataset and wait for the owner to update it.
fix: condition for file_name feature check

## Description
Accept `Value("large_string")` in addition to `Value("string")` for the `file_name` (or `*_file_name`) metadata key in folder-based builders (e.g. `audiofolder`).

## Problem
When loading a dataset with `load_dataset("audiofolder", data_dir=...)` and a `metadata.csv` file, the metadata schema is inferred via pandas to Arrow. For string columns, Arrow often uses the `large_string` type, so `Features.from_arrow_schema()` yields `Value("large_string")` for the `file_name` column. The builder only checks for `Value("string")`, so validation fails with:

`ValueError: file_name or *_file_name must be present as dictionary key (with type string) in metadata files`

Users then have to convert metadata to JSONL (which produces Arrow `string`) or work around the loader.

## Solution
Treat `Value("large_string")` as valid for the `file_name` / `*_file_name` key, since it is still a string type and behaves the same for resolution of audio/image paths. This matches the fact that when metadata is loaded from CSV (via pandas and then converted to Arrow), string columns are often inferred as Arrow `large_string`, so accepting it keeps CSV metadata working without format changes.

Comments:
Hi @lhoestq, have proposed a fix for [#8010](https://github.com/huggingface/datasets/issues/8010)
audiofolder / folder-based loader rejects metadata.csv when file_name column is inferred as large_string

### Describe the bug

When loading a dataset with load_dataset("audiofolder", data_dir=...) and a metadata.csv (with a file_name column), loading can fail with:

`ValueError: file_name or *_file_name must be present as dictionary key (with type string) in metadata files`

even when the CSV clearly has a file_name column with string values.

**Cause**

In folder_based_builder.py, the metadata schema is inferred from the CSV via pandas to Arrow. For string columns, Arrow often uses the large_string type. The check only accepts Value("string") and does not accept Value("large_string"), so validation fails even though the column is a valid string column.


### Steps to reproduce the bug
1. Create a folder with:
   - `metadata.csv` with columns: `file_name`, `transcript`, `duration` (and audio `.wav` files referenced by `file_name`).
2. Run:
```
   from datasets import load_dataset
   ds = load_dataset("audiofolder", data_dir="/path/to/folder")
```

### Expected behavior

Metadata CSV with a `file_name` column (inferred as either `string` or `large_string`) should be accepted, since both represent string data.

### Environment info


- `datasets` version: 4.5.0
- Platform: Linux-6.8.0-100-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.36.0
- PyArrow version: 23.0.0
- Pandas version: 3.0.0
- `fsspec` version: 2025.10.0

Comments:
Proposing a PR for this [here](https://github.com/huggingface/datasets/pull/8011)

PS: I didn't find a contributing documentation. Please recommend any changes in issue or PR if any. Thank you!
Add option for loading audio with video

### Describe the bug

Currently, `torchcodec` don't allow extracting `Audio` from `Video` https://github.com/meta-pytorch/torchcodec/issues/1158, https://github.com/meta-pytorch/torchcodec/issues/1190, but when I upload videos with audio to hub using `videofolder`, then this is not possible to retrieve audio from it. Probably `VideoDecoder` can be extended with `audio` parameter to retrieve this information

### Steps to reproduce the bug

```python
from datasets import load_dataset

test_ds = load_dataset("videofolder", data_dir="/path/to/video")  
# uploaded version 
# test_ds = load_dataset("Samoed/testds")

test_ds["train"][1]["video"]
# <torchcodec.decoders._video_decoder.VideoDecoder
```

### Expected behavior

Some option to retrieve audio information.

### Environment info

```
datasets==4.5.0
```

Comments:
Hi,
I’m interested in working on this issue.

From what I understand, audio decoding would involve an AudioDecoder, and previous discussions have suggested handling it separately. I’d like to explore whether integrating audio extraction in relation to the current video decoding workflow would be appropriate, or if maintaining a separate decoder is the preferred design.

Any guidance would be greatly appreciated.

Thanks!

Hi! I’m also interested in contributing to this issue.

If it would be helpful, I could try implementing a small prototype that exposes audio from the video object (for example via a `get_audio()` method), assuming the underlying decoder provides access to the audio stream.

Would you be open to a draft PR exploring this approach?


> assuming the underlying decoder provides access to the audio stream.

Torccodec.VideoDecoder doesn't support audio in video directly. To use audio from video you need to process mp4 directly and extract audio from it

Thanks for the clarification!

If Torchcodec's VideoDecoder doesn't expose audio streams, then extracting audio directly from the video file (e.g. mp4) makes sense.

Would the preferred approach be to add a helper method in the video feature that reads the file path and extracts audio separately (for example via ffmpeg or a similar backend), or should this be handled as a separate audio feature in the dataset instead?

If that direction makes sense, I’d be interested in trying a small prototype PR.
fix save_to_disk/load_from_disk with pathlib.Path input

Since #6704, `save_to_disk` and `load_from_disk` use `fsspec.core.url_to_fs` which expects a `str`, but both methods accept `PathLike` (which includes `pathlib.Path`). Passing a `Path` object raises a `TypeError` because `url_to_fs` can't handle it.

Fixed by converting the path argument with `os.fspath()` before handing it off to `url_to_fs`. This affects all five call sites across `Dataset`, `DatasetDict`, and the standalone `load_from_disk` function.

Fixes #6829

Comments:
The `Sequence` class in features do not have "dtype"

### Describe the bug

**I'm not sure if this is a bug.**

I see that a `FeatureType` object contains an attribute called `self.dtype` that is not covered when this feature is a `Sequence` or a `List`.

When I try to run a multilabel classification with this example script from the transformers library:

https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_classification.py#L442

I get this error on the linked line:

```shell
AttributeError: 'List' object has no attribute 'dtype'. Did you mean: '_type'?
```

Looking at the check that the script is attempting to perform, could we perhaps add a `self.dtype="list"` attribute for this `FeatureType` 's: `Sequence`, `List`, etc.?

### Steps to reproduce the bug

For example, this code works for me:

```python
from datasets import ClassLabel, Features, Sequence, Value
features = {'text': Value('string'), 'label': ClassLabel(names=['No', 'Yes'])}
print(features["text"].dtype)
print(features["label"].dtype)
```
```output
'string'
'int64'
```

and this code does not work for me:

```python
from datasets import ClassLabel, Features, Sequence, Value
features = {'text': Value('string'), 'label': Sequence(ClassLabel(names=['No', 'Yes']))}
print(features["label"].dtype) # it could be equal to "list"?
```
```output
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'List' object has no attribute 'dtype'. Did you mean: '_type'?
```

### Expected behavior

The attribute `dtype` equal to `"list"` when using objects of type `Sequence`. 

```python
from datasets import ClassLabel, Features, Sequence, Value
features = {'text': Value('string'), 'label': Sequence(ClassLabel(names=['No', 'Yes']))}
print(features["label"].dtype)
```
```output
'list'
```

### Environment info

I have installed `datasets==4.5.0`.

Comments:
Fix: make environment variable naming consistent (issue #7998)

Fix: https://github.com/huggingface/datasets/issues/7998

**Summary**

Addressed the inconsistent environment variable name HF_DATASETS_DISABLE_PROGRESS_BARS used for toggling progress bars, particularly in docstrings and warning messages.

**Changes**

Updated function docstrings to use the correct environment variable name

Corrected warning messages in tqdm.py to ensure consistency

Comments:
Hi @lhoestq ,
I’ve addressed the inconsistent environment variable name HF_DATASETS_DISABLE_PROGRESS_BARS in docstrings and warning messages. Please review the changes and let me know if there’s anything else I should update to complete this issue fix.
Too many dataloader workers: 4 (max is dataset.num_shards=3). Stopping 1 dataloader workers.

Hi !
I’m working on training with a large-scale dataset (100+ Parquet files) using lazy loading, and I’m struggling to understand/optimize the num_shards setting— in the lerobot repo: streaming_datasets.py:
```
from datasets import load_dataset
self.hf_dataset: datasets.IterableDataset = load_dataset(
            self.repo_id if not self.streaming_from_local else str(self.root),
            split="train",
            streaming=self.streaming,
            data_files="data/*/*.parquet",
            revision=self.revision,
        )

self.num_shards = min(self.hf_dataset.num_shards, max_num_shards)
```

```
dataloader = torch.utils.data.DataLoader(
            datasets[sub_idx],
            num_workers=datasets[sub_idx].num_shards, #cfg.num_workers,
            batch_size=cfg.batch_size,
            shuffle=shuffle and not cfg.dataset.streaming,
            sampler=sampler,
            collate_fn=FlowerDataCollator(),
            pin_memory=device.type == "cuda",
            drop_last=True,
            prefetch_factor=2 if cfg.num_workers > 0 else None,
            )
```

What exactly does hf_dataset.**num_shards** represent? Is it safe to manually override/edit num_shards?

My batch loading is slower than expected (2-3s per batch) despite num_worker cannot be bigger with warning: Too many dataloader workers: 4 (max is dataset.num_shards=3). Stopping 1 dataloader workers. 

Even use num_workers=datasets[sub_idx].num_shards, the waring is still exist! (my num_worker is 4 and hf_dataset.num_shards is 100+, so the datasets[sub_idx].num_shards=4)
Why does the "too many workers" warning persist even when num_workers equals dataset.num_shards—and how do I fix this?

Thanks so much for any insights or help with this! Really appreciate your time and expertise 😊



Comments:
Hi, thanks for the clear question and code snippet!

From my understanding, **hf_dataset.num_shards** represents the number of actual iterable partitions that the streaming dataset can be read from in parallel. This is not the same as the number of underlying Parquet files—many files can be grouped into a much smaller number of shards internally.

The warning seems to persist because **datasets[sub_idx]** is a sliced or sharded view of the original dataset, and its effective num_shards can be smaller than expected after this step. So even if **num_workers** is set equal to **datasets[sub_idx].num_shards**, additional internal sharding can still result in fewer usable shards, which triggers the warning.

Because of this, manually overriding **num_shards** does not appear to increase real parallelism and may just create extra workers that stay idle. For streaming datasets, it seems safer to let the dataset control sharding and to cap the number of workers to the effective shard count (or even use fewer workers and adjust batch size instead), since performance is often I/O-bound.

Hope this helps clarify what’s going on!
[doc] Inconsistant ENV VAR Name for Progress Bar Toggle

Code uses env var name `HF_DATASETS_DISABLE_PROGRESS_BARS`.

https://github.com/huggingface/datasets/blob/025593f2f0722f31fc136e0ae45da4ff44d4416a/src/datasets/config.py#L221-L226

Docstrings and warnings report env var name `HF_DATASETS_DISABLE_PROGRESS_BAR` without the ending `S`.

https://github.com/huggingface/datasets/blob/025593f2f0722f31fc136e0ae45da4ff44d4416a/src/datasets/utils/tqdm.py#L61-L73

https://github.com/huggingface/datasets/blob/025593f2f0722f31fc136e0ae45da4ff44d4416a/src/datasets/utils/tqdm.py#L78-L90

https://github.com/huggingface/datasets/blob/025593f2f0722f31fc136e0ae45da4ff44d4416a/src/datasets/utils/tqdm.py#L95-L100

This affects doc webpages as well, e.g., see https://huggingface.co/docs/datasets/en/package_reference/utilities#datasets.enable_progress_bars.

Comments:
fix: Dataset.map writer initialization when first examples return None

Fixes #7990

## Summary

When Dataset.map is called and the first examples processed return None, the writer is never properly initialized, causing a ValueError.

## Changes

- Modified _map_single to initialize the writer early if the first batch returns empty results
- Ensures writer is set before the first call to writer.write_batch

## Test Plan

- Added test case that reproduces the bug
- Verified the fix resolves the issue
- Existing tests still pass

Comments:
Closing in favor of #7996 which addresses the same issue with the same fix.
Fix Dataset.map writer initialization when early examples return None

## Summary

Fixes #7990 

This PR fixes a bug in `Dataset.map()` where the writer initialization was incorrectly tied to the index being 0, causing crashes when the map function returns `None` for the first few examples and later returns a dict.

### Changes

- **Non-batched mode** (line 3676): Changed from `if i == 0:` to `if writer is None:`
- **Batched mode** (line 3701): Changed from `if i and i[0] == 0:` to `if writer is None:`

### Why This Fix Works

The original code assumed that `update_data` would always be determined by the time the first example (i=0) was processed. However, `update_data` is set lazily after processing each example - it becomes `True` when the function first returns a non-None value.

If a function returns `None` for early examples and a dict for later ones:
1. At i=0, the function returns `None`, so `update_data` remains `None`  
2. Writer is NOT initialized (because we're not updating data)
3. At i=2, the function returns a dict, so `update_data` becomes `True`
4. **Old code**: Tries to use `writer` (still None) because i != 0 → crash
5. **New code**: Checks `if writer is None` and initializes it → works correctly

### Test Plan

The fix can be verified with this minimal test case from the issue:

```python
from datasets import Dataset

ds = Dataset.from_dict({"x": [1, 2, 3]})

def fn(example, idx):
    if idx < 2:
        return None
    return {"x": [example["x"] * 10]}

# Should work without errors
result = list(ds.map(fn, with_indices=True))
print(result)  # [{'x': 1}, {'x': 2}, {'x': [30]}]
```

**Before this fix**: Crashes with `AttributeError: 'NoneType' object has no attribute 'write'`  
**After this fix**: Works correctly

### Related

This fix ensures the writer is initialized the first time a non-None value is returned, regardless of which example index that occurs at. This makes the code more robust to different map function behaviors.

Comments:
Friendly ping - any chance someone could take a look at this when they get a chance? Happy to make any changes if needed.

ping - this fixes a real edge case with Dataset.map when early examples return None. would love a review

friendly bump - happy to make any changes if needed!
Bump fsspec upper bound to 2026.2.0 (fixes #7994)

Fixes #7994 and e.g. “Bumps fsspec upper bound so the latest version can be used; CI will validate compatibility.”

Comments:
Apologize for the ping @lhoestq but I wanted to know whether this is something you could look into? And more generally whether you have a policy to regularly bump fsspec? Thanks 🙏

thanks for opening a PR, I'm triggering the CI and I'll merge if it's all green :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7995). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Bump fsspec upper bound constraint

Would it be possible to bump fsspec upper bound to the latest (2026.2.0)?

I saw you had some API compat issues in the past (https://github.com/huggingface/datasets/issues/7326) and I understand the need for an upper bound.

But I wonder if you think your CI and tests are a good proxy to catch fsspec API breakage? If that's the case then triggering dataset CI with the latest version of fsspec should tell us whether it's all good right?

Happy to open a PR if needed.

Comments:
:sparkles: Add 'SparseCsv' builder and 'sparse_collate_fn' for efficient high-dimensional sparse data loading

This PR introduces a new dataset builder, SparseCsv, designed to handle "wide" tabular datasets (e.g., 100k+ columns common in transcriptomics, sparse NLP features, or recommender systems) that are typically too large to load into memory as dense Arrow tables.

It also adds a utility function, `sparse_collate_fn`, to seamlessly convert these sparse examples into `torch.sparse` or `scipy.sparse` matrices during training.

This PR should fix #7377 

Comments:
Add `IterableDataset.reshard()`

To increase the number of shards of a dataset, you can use [`IterableDataset.reshard`]:

```py
>>> dataset
IterableDataset({
    features: ['label', 'title', 'content'],
    num_shards: 4
})
>>> dataset.reshard()
IterableDataset({
    features: ['label', 'title', 'content'],
    num_shards: 3600
})
```

The resharding mechanism depends on the dataset file format.
For example for Parquet, it reshards using row groups instead of having one file per shard.

We can implement other formats later (e.g. JSON Lines, CSV can be split by recovering line boundaries from arbitrary locations)

Other details:
* fixed concatenate after shuffling, now it correctly shuffles the shards: close https://github.com/huggingface/datasets/issues/7196
* fixed interleave after split_by_node: close https://github.com/huggingface/datasets/issues/7868
* changed a bit how the effective seed works in multi node/worker/epoch situations

related to https://github.com/huggingface/datasets/issues/7917

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7992). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
list(api.list_datasets()) giving jsondecode error

i am using the python api wrapper to list all datasets available on hugging face. This is for research, and i need all datasets to determine which % have language tags and other related questions requiring the total list. However, the following code that worked a few months ago:

```
from huggingface_hub import HfApi
api = HfApi(token=token)
datasets = list(api.list_datasets())
```

now gives a JSONDecodeError when reaching 2000 results. My understanding is that this is a pagination issue, as there is no cursor exposed to the API and so it doesnt know where to read from after the limit. 

Is there any way to find all datasets available and combine them into a list?



Comments:
Dataset.map crashes when first examples return None and later examples return dict — writer not initialized

### Describe the bug

I detected a serious [bug from datasets/arrow_dataset.py](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L3676)
---

**Description of the bug**
`Dataset.map` crashes with `writer is None` when the map function returns `None` for the first few examples and a dictionary (or `pa.Table` / DataFrame) for later examples. This happens because the internal writer is initialized only when `i == 0` (or `i[0] == 0` in batched mode), but `update_data` is determined lazily after processing the first example/batch.

**Steps to reproduce**

```python
from datasets import Dataset

ds = Dataset.from_dict({"x": [1, 2, 3]})

def fn(example, idx):
    if idx < 2:
        return None
    return {"x": [example["x"] * 10]}

list(ds.map(fn, with_indices=True))
```

**Expected behavior**

* The function should work regardless of when `update_data` becomes `True`.
* Writer should be initialized the first time a non-`None` return occurs, not tied to the first index.

**Environment info**

* `datasets` version: <insert your version>
* Python version: 3.12
* OS: <insert your OS>

**Suggested fix**
Replace `if i == 0` / `if i[0] == 0` checks with `if writer is None` when initializing the writer.

---

### Steps to reproduce the bug

Here's a ready-to-use version you can paste into that section:

---

### Steps to reproduce the bug

```python
from datasets import Dataset

# Create a minimal dataset
ds = Dataset.from_dict({"x": [1, 2, 3]})

# Define a map function that returns None for first examples, dict later
def fn(example, idx):
    if idx < 2:
        return None
    return {"x": [example["x"] * 10]}

# Apply map with indices
list(ds.map(fn, with_indices=True))
```

**Expected:** function executes without errors.
**Observed:** crashes with `AttributeError: 'NoneType' object has no attribute 'write'` because the internal writer is not initialized when the first non-None return happens after i > 0.

---

This is minimal and clearly demonstrates the exact failure condition (`None` early, `dict` later).

### Expected behavior

---

**Expected behavior**
The `Dataset.map` function should handle map functions that return `None` for some examples and a dictionary (or `pa.Table` / DataFrame) for later examples. In this case, the internal writer should be initialized when the first non-`None` value is returned, so that the dataset can be updated without crashing. The code should run successfully for all examples and return the updated dataset.

---

### Environment info

- python3.12
- datasets==3.6.0 [but the latest version still has this problem]
- transformers==4.55.2


Comments:
map() expects a function that always return dict or that always return none
Remove pre-release workaround in CI for `transformers v5` and `huggingface_hub v1`

This PR removes workaround for pre-release `transformers v5.*` / `huggingface_hub v1.*` in `test_py314_future` job since they are now officially released.

cc @Wauplin just for viz since you introduced this in https://github.com/huggingface/datasets/pull/7783.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7989). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix index out of bound error with original_shard_lengths.

I have gotten the following error

```
    original_shard_lengths[original_shard_id] += 1
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
IndexError: list index out of range
```

Not sure what causes it, but this fixes the error. This may not be the proper fix for the root cause though.

Comments:
this was fixed in https://github.com/huggingface/datasets/pull/8023, sorry I didn't see your PR and took the liberty of fixing this bug as part of this improvement for `from_generator`
Remove unused data files optims

this fixes module inference when there are many metadata files

e.g. the lance dataset at https://huggingface.co/datasets/davanstrien/encyclopaedia-britannica-lance has > 200 metadata files

those optims are not used anymore, they come from a time we were dealing with slow data files iterators instead of lists

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7985). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix unstable tokenizer fingerprinting (enables map cache reuse)

Fix unstable dataset fingerprinting when hashing `PreTrainedTokenizerFast`.

Some tokenizers backed by `tokenizers.Tokenizer` mutate runtime settings (padding/truncation) when called, which can change the serialized state and make dataset fingerprints unstable. That prevents `.map(load_from_cache_file=True)` from reusing cache files.

Fix: when hashing, temporarily disable backend padding/truncation so runtime settings don’t affect the fingerprint, then restore the original settings.

Includes a regression test showing `Hasher.hash(tokenizer)` stays stable after calling the tokenizer.

Comments:
Hi! It looks like the GitHub Actions check suites for this PR are in `action_required` (no workflows actually ran). This is usually due to fork workflow approval.

Could a maintainer please approve/run the workflows so CI can execute? Happy to address anything CI flags once it runs.

Quick ping on this one. Happy to update anything if you want changes.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7982). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @itazap and @ArthurZucker for viz: this is the trick we use to hash a tokenizer

this way the hash doesn't change if the state of the tokenizer changed
Use temp files in push_to_hub to save memory

write parquet data to temp files on disk prior to upload to save memory

this is enabled for for datasets loaded using streaming=True/False

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7979). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

And they are never deleted?

Just to be clear @lhoestq, by not deleting the files on successful shard push they will accumulate and eventually fill the disk. By accumulating files, stream processing is effectively broken. Why stream files if you end up with all of them on disk?

Is there no review process in place for PRs? Even an automated one would have gotten this.

arf what an oversight, thanks for reporting I'll do a patch release asap

4.6.1 is released with a fix !
Fix 4910 kwargs

Fix #4910 : Merge duplicate kwargs in `load_dataset_builder()`

Problem: load_dataset("dataset", base_path="./data")` gives TypeError("multiple values for keyword 'base_path')

Fix: {**builder_kwargs, **config_kwargs} to user kwargs override dataset defaults

Repro : python

Before: TypeError

load_dataset("rotten_tomatoes", base_path="./sample_data")

After: Works 



Comments:
Updated get_dataset_config_names returning default in offline mode 

When a dataset is cached and accessed in offline mode, get_dataset_config_names was returning default instead of the actual cached config names. This happened because CachedDatasetModuleFactory.get_module returned a DatasetModule without builder_configs_parameters, causing the fallback to default in get_dataset_config_names.                                                                                             
                                                                                                                                                   
The fix reads config_name from each dataset_info file in the cache directory and includes them as builder_configs_parameters in the returned DatasetModule. Invalid or missing dataset_info.json files are handled.

**Testing:**                                                                                                                                                                                                                                                                                         
1. Download a dataset in online mode so it gets cached                                                                                           
2. Switch to offline mode and call get_dataset_config_names                                                                                      
3. Verify it returns the cached config names instead of ['default']                                                                              
                                                                                                                                                   
**Example:**                                                                                                                                                                                                                                                                                         

- HF_DATASETS_OFFLINE=0 HF_HOME="/tmp/hftemp" python -c "import datasets; datasets.load_dataset('cais/mmlu', 'all')"                                                                                                                                                                                
- HF_DATASETS_OFFLINE=1 HF_HOME="/tmp/hftemp" python -c "import datasets; print(datasets.get_dataset_config_names('cais/mmlu'))"                                                                                                                                                                    
- -> Expected output: ['all'] 

Fixes https://github.com/huggingface/datasets/issues/7947 

Comments:
Docs: add Dataset.from_dict example

Docs: add a minimal `Dataset.from_dict` example.

This helps new users discover the most direct way to build a small dataset from in-memory Python data.

Docs-only change.

Comments:
Quick ping on this one. Happy to update anything if you want changes.

Hi @stevhliu, quick follow-up on this docs-only PR adding a minimal `Dataset.from_dict` example. Happy to adjust wording or placement if you prefer a different format.
Fix duplicate kwargs in load_dataset_builder

Avoid passing duplicate keyword arguments to `load_dataset_builder`.

Some module factories provide values in `builder_kwargs` (e.g. `base_path`), and users can also pass the same keys via `config_kwargs`, which raises:
`TypeError: ... got multiple values for keyword argument ...`.

Fix: if `config_kwargs` is provided, drop overlapping keys from `builder_kwargs` (keep the user-provided values).

Includes a regression test.

Comments:
Quick ping on this one. Happy to update anything if you want changes.

Hi @lhoestq, quick follow-up on this small regression fix for duplicate builder kwargs. Happy to adjust the approach or test if you prefer a different direction.

Hi ! this issue which was reported here https://github.com/huggingface/datasets/pull/8066 was fixed in https://github.com/huggingface/datasets/pull/8068
Fix resolve_pattern for local symlinked files

Fix `resolve_pattern` for *local symlinked files*.

Problem: on the local `file://` filesystem, `fsspec` can report symlinks as `type=="other"` and omit the `islink` flag, so symlinked files are skipped.

Fix: when `protocol=="file"`, treat `os.path.islink(filepath)` as a link candidate and include it if it resolves to a regular file.

Includes a regression test in `tests/test_data_files.py`.

Comments:
Hi! It looks like the GitHub Actions check suites for this PR are in `action_required` (no workflows actually ran). This is usually due to fork workflow approval.

Could a maintainer please approve/run the workflows so CI can execute? Happy to address anything CI flags once it runs.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7973). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Quick ping on this one. Happy to update anything if you want changes.

I can rerun or fix CI items right away if you want.

Hi @lhoestq, quick follow-up. This PR still looks blocked on maintainer workflow approval, so CI has not run yet. Could you approve/run CI when convenient? Happy to fix anything it finds.

sorry for the delay, which version of fsspec is impacted by the missing `islink` field ?
feat: implement iter_arrow for skip, take and step iterables

This commit optimizes streaming operations by implementing [_iter_arrow](cci:1://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:377:4-391:57) for [SkipExamplesIterable](cci:2://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:1798:0-1892:42), [TakeExamplesIterable](cci:2://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:1944:0-2048:42), and [StepExamplesIterable](cci:2://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:667:0-743:42).

### Key Changes:
- **Fast Batch Processing**: Enabled batch-level slicing for `.skip(n)` and `.take(n)` on streaming datasets, bypassing slow row-by-row iteration.
- **Optimized Sharding**: Updated [StepExamplesIterable](cci:2://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:667:0-743:42) (used in distributed training) to use Arrow's `.take()` to extract multiple records from a batch simultaneously.
- **State Preservation**: Reinforced [_init_state_dict](cci:1://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:505:4-514:31) and [load_state_dict](cci:1://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:2382:4-2430:46) to support flawless checkpointing and resumption while using Arrow iteration.

### Performance Impact:
Users will experience significant performance gains when skipping or taking examples in streaming mode. By staying in the "Arrow path" and avoiding Python dictionary conversions, data loading overhead is drastically reduced, especially for large-scale training jobs.

### Testing:
Integrated 6 new unit tests into [tests/test_iterable_dataset.py](cci:7://file:///c:/Users/ASUS/Desktop/datasets/tests/test_iterable_dataset.py:0:0-0:0) to verify:
- Functional correctness for [skip](cci:1://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:3149:4-3191:9), [take](cci:1://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:3236:4-3271:9), and `step` using Arrow iteration.
- Reliable state checkpointing and resumption after partial iteration.

Comments:
push_to_hub() for videos

possible now that row group sizes are auto-determined based on the content size after https://github.com/huggingface/datasets/pull/7589

Videos are uploaded as PLAIN in Parquet to make sure they can be seeked remotely and with random access to frames in https://github.com/huggingface/datasets/pull/7976

In the future it could be cool to have the same behavior as when videos are separate files, i.e. lazily load them instead of downloading them completely in streaming mode.

Right now there is this discrepency:
- `load_dataset("username/my-folder-of-videos", streaming=True)` -> videos are lazy loaded one by one when iterating, and only actually downloaded when accessing frames in `torchcodec`
- `load_dataset("username/my-video-dataset-in-parquet", streaming=True)` -> videos are downloaded one by one when iterating, even if no frame is accessed in `torchcodec`

close https://github.com/huggingface/datasets/issues/7493

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7971). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Count examples in lance

```python
In [1]: from datasets import load_dataset_builder, StreamingDownloadManager

In [2]: b = load_dataset_builder("lance-format/openvid-lance")
Resolving data files: 100%|█| 240/240 [00:00<00:00, 42675.64it/s

In [3]: b.count_examples(StreamingDownloadManager())
Out[3]: {'train': 937957}
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7969). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Potential conflicting type checks and dead code in `/src/datasets/table.py`

When statically analyzing and manually reviewing the code, I noticed a potential logic conflicting in `/src/datasets/table.py` as follows:

```
def to_blocks(table: Union[pa.Table, Table]) -> list[list[TableBlock]]:
      if isinstance(table, pa.Table):
          return [[InMemoryTable(table)]]
      elif isinstance(table, ConcatenationTable):  # dead code
          return copy.deepcopy(table.blocks)
      else:
          return [[table]]
```

Within the function, the condition `isinstance(table, ConcatenationTable)` at line 4 will never be True because the previous condition `isinstance(table, pa.Table)` at line 2 would have already caught all instances of `ConcatenationTable` (since `ConcatenationTable` is a subtype of `pa.Table`). This creates a logical conflict in the type checking flow.

Please verify if this logic is intentional or it is an issue warranting a refactoring or fixing.

Comments:
ConcatenationTable is a subclass of datasets.table.Table but not pa.Table, so it should be fine
Infer types from lance blobs

Ex: infer Video() type in https://huggingface.co/datasets/lance-format/openvid-lance and Image() type in https://huggingface.co/datasets/lance-format/laion-1m

```python
from datasets import load_dataset

ds = load_dataset("lance-format/laion-1m", streaming=True, split="train")
print(ds.feature["image"])
# Image()

ds = load_dataset("lance-format/openvid-lance", streaming=True, split="train")
print(ds.feature["video_blob"])
# Video()
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7966). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.