AniketTec/github-issues huggingface dataset:
Title: Typo in iterable_dataset.py
State: open

### Describe the bug

`streaming=True**kwargs,` should be `streaming=True, **kwargs,`
https://github.com/huggingface/datasets/blob/81027be09d5cd9f06a6d64ef1a8a3e9ebd0f86fd/src/datasets/iterable_dataset.py#L2950

### Steps to reproduce the bug

```
from Datasets import Dataset
IterableDataset.from_csv("file.csv")
```

### Expected behavior

Load the csv file as an iterable dataset.

### Environment info

- `datasets` version: 4.6.1
- Platform: Linux-6.14.0-37-generic-x86_64-with-glibc2.39
- Python version: 3.13.11
- `huggingface_hub` version: 1.5.0
- PyArrow version: 23.0.1
- Pandas version: 3.0.1
- `fsspec` version: 2026.2.0

Comments:
Title: IterableDataset.filter() chaining fails due to features being lost when is_typed=True
State: open

### Describe the bug

Chaining multiple .filter() calls on an IterableDataset raises a TypeError: 'NoneType' object is not a mapping on the second .filter() call.

In `IterableDataset.filter()`, the internal `ex_iterable` is wrapped with FormattedExamplesIterable:

```python
ex_iterable = FormattedExamplesIterable(
    ex_iterable,
    formatting=self._formatting,
    features=None if ex_iterable.is_typed else self._info.features,
    token_per_repo_id=self._token_per_repo_id,
)
```

After the first .filter(), the resulting ex_iterable becomes typed (is_typed=True) because a ===MASK=== column is added. On the second .filter(), since is_typed=True, features=None is passed to FormattedExamplesIterable. This causes .features to return None, and FilteredExamplesIterable.__init__ fails when trying to unpack it.

The fix should preserve the existing features when is_typed=True, e.g.:

```python
features=ex_iterable.features if ex_iterable.is_typed else self._info.features
```

### Steps to reproduce the bug

```python
from datasets import load_dataset

ds = load_dataset("bigcode/the-stack-v2", "Scala", split="train", streaming=True)
ds = ds.filter(lambda row: row["src_encoding"] == "UTF-8")
ds = ds.filter(lambda row: row["branch_name"] == "refs/heads/master")  # TypeError here
```

## Error traceback
```
File ".../datasets/iterable_dataset.py", line 2975, in filter
    ex_iterable = FilteredExamplesIterable(
File ".../datasets/iterable_dataset.py", line 1606, in __init__
    features = Features({**ex_iterable.features, self.mask_column_name: Value("bool")})
TypeError: 'NoneType' object is not a mapping
```

### Expected behavior

The error should not raised in this context.

### Environment info

datasets: 4.5.0

Comments:
Title: perf: use deque for async map task/index queues
State: open

## Problem

`arrow_dataset.py` and `iterable_dataset.py` drain `indices` and `tasks` lists front-to-back via `.pop(0)` during parallel async map operations. With `MAX_NUM_RUNNING_ASYNC_MAP_FUNCTIONS_IN_PARALLEL` items in flight, each `.pop(0)` is **O(n)** making the drain loop **O(n²)**.

## Solution

Switch both `indices` and `tasks` to `collections.deque` with `.popleft()` for **O(1)** front removal.

## Changes

- `src/datasets/arrow_dataset.py`: Import `deque`, type `tasks` and `indices` as deque, use `.popleft()`
- `src/datasets/iterable_dataset.py`: Import `deque`, type `tasks`, `indices`, and `_owned_loops_and_tasks` entry as deque, use `.popleft()`

## Testing

- Syntax verified via `ast.parse()` on both files

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8033). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Friendly ping — CI is green and this is ready for review. Happy to address any feedback. Thanks!
Title: Remove tmp file in push to hub
State: closed

reported in https://github.com/huggingface/datasets/pull/7979#issuecomment-3973374484

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8030). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Fix torchcodec audio decoding to respect 'num_channels'
State: open

Fixes torchcodec audio decoding when `num_channels` is set on `Audio`.

Before this change, `AudioDecoder["array"]` reduced multi-channel audio to mono by averaging channels, so the requested channel behavior was not respected.

With this PR:
- multi-channel decoded arrays are preserved by default;
- mono output is returned only when `num_channels == 1` is explicitly requested.

#### Previous behavior
`ds_stereo = ds.cast_column("audio", Audio(num_channels={...}))`  - None, 1, 2

Original file: '(16000, 2)' - stereo ✓  
'Audio(num_channels=None)': '(16000,)' - mono ✗  
'Audio(num_channels=2)': '(16000,)' - mono ✗  
'Audio(num_channels=1)': '(16000,)' - mono ✓

#### New behavior
'num_channels=None' preserves the original number of channels from the source file.
'num_channels=2' preserves/converts to stereo output with shape '(2, num_samples)'.
'num_channels=1' downmixes to mono with shape '(num_samples,)'.

#### Results
**Original file shape** (via soundfile): (16000, 2)
HF datasets shape with num_channels=None: (2, 16000)
HF datasets shape with num_channels=1: (16000,)
HF datasets shape with num_channels=2: (2, 16000)


Fixes #8005.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8028). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Add `Json()` type
State: open

`Json()` type is needed when some fields don't have fixed types, e.g. "tools" and "tool_calls" types in conversation + tool calling datasets like [dataclaw](https://huggingface.co/datasets?other=dataclaw) datasets. Cc @peteromallet and @woctordho @Nanbeige for viz

Examples of supported tool-calling / dataclaw datasets:
```python
ds = load_dataset("peteromallet/dataclaw-peteromallet")  #  happens to not need Json() since tool types are fixed
ds = load_dataset("woctordho/dataclaw")
ds = load_dataset("Nanbeige/ToolMind")
```

The `Json()` type is auto-applied when loading JSON files when mixed types are found. The `Json()`type is set to the list containing objects with mixed types, to end up with a `List(Json())` type for "tools" and "tool_calls" columns.

It is also possible to define a Dataset with a `Json()` type like this:

```python
>>> from datasets import load_dataset, Dataset, Features, Json, List
>>> example = {"a": [{"key": 0}, {"another-key": "another-type"}]}
>>> features = Features({"a": List(Json())})
>>> ds = Dataset.from_list([example], features=features)
>>> ds[0]
{'a': [{'key': 0}, {'another-key': 'another-type'}]}
```

close https://github.com/huggingface/datasets/issues/7869
close https://github.com/huggingface/datasets/issues/4120
close https://github.com/huggingface/datasets/issues/5827
related to https://github.com/huggingface/datasets/issues/7092
related to https://github.com/huggingface/datasets/issues/6162
related to https://huggingface.co/datasets/PatoFlamejanteTV/LocalLLaMA/discussions/1

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8027). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: fix: resolve base_path conflict between builder_kwargs and config_kwargs
State: closed

## What does this PR fix?

This PR fixes a bug where passing `base_path` to `load_dataset_builder()` or `load_dataset()` causes a TypeError because both `builder_kwargs` and `config_kwargs` can contain the same key.

## Problem

When users call:
```python
from datasets import load_dataset
load_dataset("rotten_tomatoes", base_path="./sample_data")
```

They get:
```
TypeError: type object got multiple values for keyword argument "base_path"
```

This happens because `base_path` can exist in both `builder_kwargs` (from the dataset module) and `config_kwargs` (from user input), causing a conflict when both are unpacked.

## Solution

- Pop `base_path` from `builder_kwargs` before merging with `config_kwargs`
- Explicitly pass `base_path` to the builder constructor

## Testing

The fix should resolve the issue demonstrated in #4910:
```python
from datasets import load_dataset
load_dataset("rotten_tomatoes", base_path="./sample_data")  # Now works!
```

Fixes #4910

Comments:
Title: progress bar optional manual control
State: open

Fix: https://github.com/huggingface/datasets/issues/7939

Summary

The default behavior for the progress bar has been adjusted based on dataset size. It remains enabled for small datasets and is automatically disabled for larger datasets to improve performance. Users retain full control and can explicitly enable or disable the progress bar using the progress_bar parameter.

Changes

Modified the logic to disable the progress bar when the number of files exceeds 16.

Comments:
Hi @lhoestq ,
Can you please trigger the CI CD pipeline ?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8021). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq I have applied the required Ruff formatting changes in the` load.py` file.
Could you please trigger the CI/CD pipeline again?

HI @lhoestq ,
Just a gentle reminder to please trigger the CI/CD pipeline for this PR..!!
Title: feat: add return_file_name support to CSV packaged builder
State: open

## Summary

- Adds an optional `return_file_name: bool = False` parameter to `CsvConfig`
- When `return_file_name=True`, a `file_name` column is appended to every batch in `_generate_tables`, containing the basename of the source CSV file for each row
- Default is `False`, preserving full backward compatibility

## Motivation

Part of #5806. Extends the `file_name` feature (already implemented for the JSON packaged builder in #7948) to the CSV packaged builder. This enables use cases such as resuming training from checkpoints by identifying which data shards have already been consumed.

## Changes

- `src/datasets/packaged_modules/csv/csv.py`: Add `return_file_name: bool = False` field to `CsvConfig`; in `_generate_tables`, append a `file_name` column when the flag is `True`
- `tests/packaged_modules/test_csv.py`: Add three tests covering default behavior (no column), enabled behavior (column present), and correct column values

## Test plan

- [ ] `test_csv_no_file_name_by_default` — verifies `file_name` column is absent by default
- [ ] `test_csv_return_file_name_enabled` — verifies `file_name` column is present when `return_file_name=True`
- [ ] `test_csv_file_name_values` — verifies each row's `file_name` value equals the source file basename

All three tests pass locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Comments:
Title: Improve error message for deprecated dataset scripts with migration guidance
State: open

Summary
This PR improves the RuntimeError message raised when deprecated local dataset scripts are detected.
Previously, the error message:
    "Dataset scripts are no longer supported"
did not provide actionable migration guidance.
Improvements
- Clarifies the new architecture
- Provides explicit migration steps
- Includes link to official documentation
- Adds a focused unit test to validate error clarity
This should reduce confusion for contributors migrating legacy dataset scripts.

Comments:
Hi team! I’m just checking in on this PR to see if there is any additional information or documentation I can provide to help with the review process. Thanks for your time!
Title: Remove legacy Sphinx roles from docstrings
State: open

## Summary

Replace all remaining `:obj:`, `:class:`, `:func:`, and `:meth:` Sphinx cross-reference roles with plain backtick references, aligning with the project's current documentation conventions.

**14 files, 71 replacements. No logic changes.**

## Context

As noted in #5324, the codebase had a mix of old Sphinx syntax and the modern backtick-only format used by HuggingFace's doc builder. @stevhliu [mentioned](https://github.com/huggingface/datasets/issues/5324#issuecomment-1525157369) that the user-facing APIs had been cleaned up, but old syntax was still lingering in non-public APIs. This PR catches the remaining occurrences.

## What changed

| Old syntax | New syntax | Count |
|---|---|---|
| `:obj:\`bool\`` | `` `bool` `` | 18 |
| `:class:\`DatasetInfo\`` | `` `DatasetInfo` `` | 15 |
| `:func:\`fsspec.open\`` | `` `fsspec.open` `` | 9 |
| `:meth:\`...\`` | `` `...` `` | 2 |
| `:obj:\`~pathlib.Path\`` | `` `Path` `` | 8 |
| `:class:\`~download.DownloadConfig\`` | `` `DownloadConfig` `` | 3 |
| *other* | — | 16 |

Tilde references (`~module.Class`) are resolved to their short name (`Class`), matching the existing convention in the codebase.

## Files touched

```
src/datasets/arrow_dataset.py
src/datasets/arrow_writer.py
src/datasets/dataset_dict.py
src/datasets/features/features.py
src/datasets/filesystems/compression.py
src/datasets/fingerprint.py
src/datasets/formatting/formatting.py
src/datasets/inspect.py
src/datasets/iterable_dataset.py
src/datasets/load.py
src/datasets/streaming.py
src/datasets/table.py
src/datasets/utils/deprecation_utils.py
src/datasets/utils/file_utils.py
```

Relates to #5324

Comments:
Title: Speed up local 'get_data_patterns' by avoiding repeated recursive scans
State: open

This PR speeds up `get_data_patterns` for local paths.

### Problem

For `load_dataset("imagefolder", data_dir=...)`, `get_data_patterns` was repeatedly scanning the same local directory tree for many split patterns (`train`, `test`, etc.).  
With large folders, this became very slow. This has also been reported before in earlier performance discussions/issues.

### Change

In `get_data_patterns` (local paths only):

- scan files once (`resolve_pattern("**", ...)`)
- then match split patterns in memory

Remote paths keep the old behavior.

No API changes.

### My Env
- Mac mini (M4), 16 GB RAM
- Python 3.13
- Datasets version: 4.5.0

### Benchmarks

`imagefolder` with local `.jpg` files

`data_dir_only`:
- 10k: `4.35s -> 1.40s` (3.10x)
- 100k: `33.48s -> 9.77s` (3.43x)
- 300k: `160.20s -> 35.81s` (4.47x)
- 1M: `1877.70s -> 164.17s` (11.44x) :fire:

`explicit_data_files`:
- 10k: `0.75s -> 0.79s`
- 100k: `7.23s -> 7.29s`
- 300k: `25.44s -> 24.26s`
- 1M: `115.85s -> 112.66s`

As expected, the improvement is on `data_dir_only` (auto pattern detection path).
Memory usage did not show a consistent regression in these runs and stayed within normal run-to-run variance for this benchmark setup.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8014). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> lgtm ! can you run `make style` to fix the CI before we merge ?

done. All good, no warnings.
Title: HC3 dataset fails to load with datasets>=3 due to legacy script (HC3.py)
State: closed

### Describe the bug

It looks like the current documentation suggests loading the dataset like this:

`load_dataset("Hello-SimpleAI/HC3", split="train")
`

But with datasets>=3.x, this raises:

RuntimeError: Dataset scripts are no longer supported, but found HC3.py


The dataset loads correctly if we specify the parquet export branch:

```
load_dataset(
    "Hello-SimpleAI/HC3",
    revision="refs/convert/parquet",
    split="train"
)

```

Since dataset scripts are no longer supported in newer versions of datasets, it might be better to update the documentation to reflect the parquet-based loading method .

Just wanted to flag this so others don’t run into the same confusion.

### Steps to reproduce the bug

### Steps to reproduce

Using `datasets>=3.x`:

```python
from datasets import load_dataset

ds = load_dataset("Hello-SimpleAI/HC3", split="train")
```

### Error

```
RuntimeError: Dataset scripts are no longer supported, but found HC3.py
```

This appears to happen because the repository still contains a legacy loading script (`HC3.py`), and recent versions of `datasets` no longer support dataset scripts.

### Working alternative

The dataset loads correctly when specifying the parquet export revision:

```python
from datasets import load_dataset

ds = load_dataset(
    "Hello-SimpleAI/HC3",
    revision="refs/convert/parquet",
    split="train"
)

print(ds)
```

This successfully downloads and loads the dataset.


### Expected behavior

### Expected behavior

The dataset should load successfully using the example shown in the documentation:

```python
load_dataset("Hello-SimpleAI/HC3", split="train")
```

without requiring a specific `revision` parameter.



### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-6.6.105+-x86_64-with-glibc2.35
- Python version: 3.12.12
- `huggingface_hub` version: 1.4.1
- PyArrow version: 18.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2025.3.0

Comments:
This is not a bug. This is intended behaviour. The only ways to resolve this are
1. use the old `datasets` versions
2. load the dataset and upload it to your own repo without a script
3. raise an issue in the dataset and wait for the owner to update it.
Title: fix: condition for file_name feature check
State: closed

## Description
Accept `Value("large_string")` in addition to `Value("string")` for the `file_name` (or `*_file_name`) metadata key in folder-based builders (e.g. `audiofolder`).

## Problem
When loading a dataset with `load_dataset("audiofolder", data_dir=...)` and a `metadata.csv` file, the metadata schema is inferred via pandas to Arrow. For string columns, Arrow often uses the `large_string` type, so `Features.from_arrow_schema()` yields `Value("large_string")` for the `file_name` column. The builder only checks for `Value("string")`, so validation fails with:

`ValueError: file_name or *_file_name must be present as dictionary key (with type string) in metadata files`

Users then have to convert metadata to JSONL (which produces Arrow `string`) or work around the loader.

## Solution
Treat `Value("large_string")` as valid for the `file_name` / `*_file_name` key, since it is still a string type and behaves the same for resolution of audio/image paths. This matches the fact that when metadata is loaded from CSV (via pandas and then converted to Arrow), string columns are often inferred as Arrow `large_string`, so accepting it keeps CSV metadata working without format changes.

Comments:
Hi @lhoestq, have proposed a fix for [#8010](https://github.com/huggingface/datasets/issues/8010)
Title: audiofolder / folder-based loader rejects metadata.csv when file_name column is inferred as large_string
State: closed

### Describe the bug

When loading a dataset with load_dataset("audiofolder", data_dir=...) and a metadata.csv (with a file_name column), loading can fail with:

`ValueError: file_name or *_file_name must be present as dictionary key (with type string) in metadata files`

even when the CSV clearly has a file_name column with string values.

**Cause**

In folder_based_builder.py, the metadata schema is inferred from the CSV via pandas to Arrow. For string columns, Arrow often uses the large_string type. The check only accepts Value("string") and does not accept Value("large_string"), so validation fails even though the column is a valid string column.


### Steps to reproduce the bug
1. Create a folder with:
   - `metadata.csv` with columns: `file_name`, `transcript`, `duration` (and audio `.wav` files referenced by `file_name`).
2. Run:
```
   from datasets import load_dataset
   ds = load_dataset("audiofolder", data_dir="/path/to/folder")
```

### Expected behavior

Metadata CSV with a `file_name` column (inferred as either `string` or `large_string`) should be accepted, since both represent string data.

### Environment info


- `datasets` version: 4.5.0
- Platform: Linux-6.8.0-100-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.36.0
- PyArrow version: 23.0.0
- Pandas version: 3.0.0
- `fsspec` version: 2025.10.0

Comments:
Proposing a PR for this [here](https://github.com/huggingface/datasets/pull/8011)

PS: I didn't find a contributing documentation. Please recommend any changes in issue or PR if any. Thank you!
Title: Add option for loading audio with video
State: open

### Describe the bug

Currently, `torchcodec` don't allow extracting `Audio` from `Video` https://github.com/meta-pytorch/torchcodec/issues/1158, but when I upload videos with audio to hub using `videofolder`, then this is not possible to retrieve audio from it. Probably `VideoDecoder` can be extended with `audio` parameter to retrieve this information

### Steps to reproduce the bug

```python
from datasets import load_dataset

test_ds = load_dataset("videofolder", data_dir="/path/to/video")  
# uploaded version 
# test_ds = load_dataset("Samoed/testds")

test_ds["train"][1]["video"]
# <torchcodec.decoders._video_decoder.VideoDecoder
```

### Expected behavior

Some option to retrieve audio information.

### Environment info

```
datasets==4.5.0
```

Comments:
Title: fix save_to_disk/load_from_disk with pathlib.Path input
State: open

Since #6704, `save_to_disk` and `load_from_disk` use `fsspec.core.url_to_fs` which expects a `str`, but both methods accept `PathLike` (which includes `pathlib.Path`). Passing a `Path` object raises a `TypeError` because `url_to_fs` can't handle it.

Fixed by converting the path argument with `os.fspath()` before handing it off to `url_to_fs`. This affects all five call sites across `Dataset`, `DatasetDict`, and the standalone `load_from_disk` function.

Fixes #6829

Comments:
Title: The `Sequence` class in features do not have "dtype"
State: open

### Describe the bug

**I'm not sure if this is a bug.**

I see that a `FeatureType` object contains an attribute called `self.dtype` that is not covered when this feature is a `Sequence` or a `List`.

When I try to run a multilabel classification with this example script from the transformers library:

https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_classification.py#L442

I get this error on the linked line:

```shell
AttributeError: 'List' object has no attribute 'dtype'. Did you mean: '_type'?
```

Looking at the check that the script is attempting to perform, could we perhaps add a `self.dtype="list"` attribute for this `FeatureType` 's: `Sequence`, `List`, etc.?

### Steps to reproduce the bug

For example, this code works for me:

```python
from datasets import ClassLabel, Features, Sequence, Value
features = {'text': Value('string'), 'label': ClassLabel(names=['No', 'Yes'])}
print(features["text"].dtype)
print(features["label"].dtype)
```
```output
'string'
'int64'
```

and this code does not work for me:

```python
from datasets import ClassLabel, Features, Sequence, Value
features = {'text': Value('string'), 'label': Sequence(ClassLabel(names=['No', 'Yes']))}
print(features["label"].dtype) # it could be equal to "list"?
```
```output
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'List' object has no attribute 'dtype'. Did you mean: '_type'?
```

### Expected behavior

The attribute `dtype` equal to `"list"` when using objects of type `Sequence`. 

```python
from datasets import ClassLabel, Features, Sequence, Value
features = {'text': Value('string'), 'label': Sequence(ClassLabel(names=['No', 'Yes']))}
print(features["label"].dtype)
```
```output
'list'
```

### Environment info

I have installed `datasets==4.5.0`.

Comments: