konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
Question about iterable dataset and streaming

In the doc, I found the following example: https://github.com/huggingface/datasets/blob/611f5a592359ebac6f858f515c776aa7d99838b2/docs/source/stream.mdx?plain=1#L65-L78

I am confused, 
1. If we have already loaded the dataset, why doing `to_iterable_dataset`?  Does it go through the dataset faster than map-style dataset?
2. `load_dataset(streaming=True)` is useful for huge dataset, but the speed is slow. How to make it comparable to `to_iterable_dataset` without loading the whole dataset into RAM?
metric glue breaks with 4.0.0

### Describe the bug

worked fine with 3.6.0, and with 4.0.0 `eval_metric = metric.compute()` in HF Accelerate breaks.

The code that fails is:
https://huggingface.co/spaces/evaluate-metric/glue/blob/v0.4.0/glue.py#L84

```
def simple_accuracy(preds, labels):
    print(preds, labels)
    print(f"{preds==labels}")
    return float((preds == labels).mean())
```
data:
```
Column([1, 0, 0, 1, 1]) Column([1, 0, 0, 1, 0])
False
```

```
[rank0]:     return float((preds == labels).mean())
[rank0]:                  ^^^^^^^^^^^^^^^^^^^^^^
[rank0]: AttributeError: 'bool' object has no attribute 'mean'
```

Some behavior has changed in this new major release of `datasets` and requires updating HF accelerate and perhaps the glue metric code, all belong to HF.

### Environment info

datasets=4.0.0

I released `evaluate` 0.4.5 yesterday to fix the issue - sorry for the inconvenience:

```
pip install -U evaluate
```

Thanks so much, @lhoestq!
To support decoding audio data, please install 'torchcodec'.


In the latest version of datasets==4.0.0, i cannot print the audio data on the Colab notebook. But it works on the 3.6.0 version.

!pip install -q -U datasets huggingface_hub fsspec

from datasets import load_dataset

downloaded_dataset = load_dataset("ymoslem/MediaSpeech", "tr", split="train")
print(downloaded_dataset["audio"][0])

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
[/tmp/ipython-input-4-90623240.py](https://localhost:8080/#) in <cell line: 0>()
----> 1 downloaded_dataset["audio"][0]

10 frames
[/usr/local/lib/python3.11/dist-packages/datasets/features/audio.py](https://localhost:8080/#) in decode_example(self, value, token_per_repo_id)
    170             from ._torchcodec import AudioDecoder
    171         else:
--> 172             raise ImportError("To support decoding audio data, please install 'torchcodec'.")
    173 
    174         if not self.decode:

ImportError: To support decoding audio data, please install 'torchcodec'. 

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-6.1.123+-x86_64-with-glibc2.35
- Python version: 3.11.13
- `huggingface_hub` version: 0.33.2
- PyArrow version: 18.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2025.3.0

Hi ! yes you should `!pip install -U datasets[audio]` to have the required dependencies.

`datasets` 4.0 now relies on `torchcodec` for audio decoding. The `torchcodec` AudioDecoder enables streaming from HF and also allows to decode ranges of audio
Toxicity fails with datasets 4.0.0

### Describe the bug

With the latest 4.0.0 release, huggingface toxicity evaluation module fails with error: `ValueError: text input must be of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).`

### Steps to reproduce the bug

Repro:
```
>>> toxicity.compute(predictions=["This is a response"])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/evaluate/module.py", line 467, in compute
    output = self._compute(**inputs, **compute_kwargs)
  File "/Users/serena.ruan/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-measurement--toxicity/2390290fa0bf6d78480143547c6b08f3d4f8805b249df8c7a8e80d0ce8e3778b/toxicity.py", line 135, in _compute
    scores = toxicity(predictions, self.toxic_classifier, toxic_label)
  File "/Users/serena.ruan/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-measurement--toxicity/2390290fa0bf6d78480143547c6b08f3d4f8805b249df8c7a8e80d0ce8e3778b/toxicity.py", line 103, in toxicity
    for pred_toxic in toxic_classifier(preds):
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/pipelines/text_classification.py", line 159, in __call__
    result = super().__call__(*inputs, **kwargs)
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1431, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1437, in run_single
    model_inputs = self.preprocess(inputs, **preprocess_params)
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/pipelines/text_classification.py", line 183, in preprocess
    return self.tokenizer(inputs, return_tensors=return_tensors, **tokenizer_kwargs)
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2867, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2927, in _call_one
    raise ValueError(
ValueError: text input must be of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).
```

### Expected behavior

This works before 4.0.0 release

### Environment info

- `datasets` version: 4.0.0
- Platform: macOS-15.5-arm64-arm-64bit
- Python version: 3.10.16
- `huggingface_hub` version: 0.33.0
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Hi ! You can fix this by upgrading `evaluate`:

```
pip install -U evaluate
```

Thanks, verified evaluate 0.4.5 works!
Fix double sequence

```python
>>> Features({"a": Sequence(Sequence({"c": Value("int64")}))})
{'a': List({'c': List(Value('int64'))})}
```

instead of `{'a': {'c': List(List(Value('int64')))}}`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7672). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Mapping function not working if the first example is returned as None

### Describe the bug

https://github.com/huggingface/datasets/blob/8a19de052e3d79f79cea26821454bbcf0e9dcd68/src/datasets/arrow_dataset.py#L3652C29-L3652C37

Here we can see the writer is initialized on `i==0`. However, there can be cases where in the user mapping function, the first example is filtered out (length constraints, etc).

In this case, the writer would be a `None` type and the code will report `NoneType has no write function`.

A simple fix is available, simply change line 3652 from `if i == 0:` to `if writer is None:`

### Steps to reproduce the bug

Prepare a dataset

have this function

```
import datasets


def make_map_fn(split, max_prompt_tokens=3):
    def process_fn(example, idx):

        question = example['question']
        reasoning_steps = example['reasoning_steps']
        label = example['label']

        answer_format = ""
        for i in range(len(reasoning_steps)):
            system_message = "Dummy"

        all_steps_formatted = []

        content = f"""Dummy"""

        prompt = [
            {"role": "system", "content": system_message},
            {"role": "user", "content": content},
        ]
        tokenized = tokenizer.apply_chat_template(prompt, return_tensors="pt", truncation=False)
        if tokenized.shape[1] > max_prompt_tokens:
            return None  # skip overly long examples

        data = {
            "dummy": "dummy"
        }

        return data

    return process_fn

...
# load your dataset
...
train = train.map(function=make_map_fn('train'), with_indices=True)
```

### Expected behavior

The dataset mapping shall behave even when the first example is filtered out.

### Environment info

I am using `datasets==3.6.0` but I have observed this issue in the github repo too: https://github.com/huggingface/datasets/blob/8a19de052e3d79f79cea26821454bbcf0e9dcd68/src/datasets/arrow_dataset.py#L3652C29-L3652C37

Hi, map() always expect an output.

If you wish to filter examples, you should use filter(), in your case it could be something like this:

```python
ds = ds.map(my_processing_function).filter(ignore_long_prompts)
```

Realized this! Thanks a lot, I will close this issue then.
How can I add my custom data to huggingface datasets

I want to add my custom dataset in huggingface dataset. Please guide me how to achieve that.

Hey @xiagod 

The easiest way to add your custom data to Hugging Face Datasets is to use the built-in load_dataset function with your local files. Some examples include:

CSV files:
from datasets import load_dataset
dataset = load_dataset("csv", data_files="my_file.csv")

JSON or JSONL files:
from datasets import load_dataset
dataset = load_dataset("json", data_files="my_file.json")


Images stored in folders (e.g. data/train/cat/, data/train/dog/):
from datasets import load_dataset
dataset = load_dataset("imagefolder", data_dir="/path/to/pokemon")


These methods let you quickly create a custom dataset without needing to write a full script.

More information can be found in Hugging Face's tutorial "Create a dataset" or "Load" documentation here: 

https://huggingface.co/docs/datasets/create_dataset 

https://huggingface.co/docs/datasets/loading#local-and-remote-files



If you want to submit your dataset to the Hugging Face Datasets GitHub repo so others can load it follow this guide: 

https://huggingface.co/docs/datasets/upload_dataset
Broken EXIF crash the whole program

### Describe the bug

When parsing this image in the ImageNet1K dataset, the `datasets` crashs whole training process just because unable to parse an invalid EXIF tag.
![Image](https://github.com/user-attachments/assets/3c840203-ac8c-41a0-9cf7-45f64488037d)

### Steps to reproduce the bug

Use the `datasets.Image.decode_example` method to decode the aforementioned image could reproduce the bug.
The decoding function will throw an unhandled exception at the `image.getexif()` method call due to invalid utf-8 stream in EXIF tags.
```
File lib/python3.12/site-packages/datasets/features/image.py:188, in Image.decode_example(self, value, token_per_repo_id)
    186     image = PIL.Image.open(BytesIO(bytes_))
    187 image.load()  # to avoid "Too many open files" errors
--> 188 if image.getexif().get(PIL.Image.ExifTags.Base.Orientation) is not None:
    189     image = PIL.ImageOps.exif_transpose(image)
    190 if self.mode and self.mode != image.mode:

File lib/python3.12/site-packages/PIL/Image.py:1542, in Image.getexif(self)
   1540 xmp_tags = self.info.get("XML:com.adobe.xmp")
   1541 if not xmp_tags and (xmp_tags := self.info.get("xmp")):
-> 1542     xmp_tags = xmp_tags.decode("utf-8")
   1543 if xmp_tags:
   1544     match = re.search(r'tiff:Orientation(="|>)([0-9])', xmp_tags)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 4312: invalid start byte
```

### Expected behavior

The invalid EXIF tag should simply be ignored or issue a warning message, instead of crash the whole program at once.

### Environment info

- `datasets` version: 3.6.0
- Platform: Linux-6.5.0-18-generic-x86_64-with-glibc2.35
- Python version: 3.12.11
- `huggingface_hub` version: 0.33.0
- PyArrow version: 20.0.0
- Pandas version: 2.3.0
- `fsspec` version: 2025.3.0

There are other discussions about error handling for images decoding here : https://github.com/huggingface/datasets/issues/7632 https://github.com/huggingface/datasets/issues/7612

and a PR here: https://github.com/huggingface/datasets/pull/7638 (would love your input on the proposed solution !)
Function load_dataset() misinterprets string field content as part of dataset schema when dealing with `.jsonl` files

### Describe the bug

When loading a `.jsonl` file using `load_dataset("json", data_files="data.jsonl", split="train")`, the function misinterprets the content of a string field  as if it were part of the dataset schema. 

In my case there is a field `body:` with a string value 
```
"### Describe the bug (...) ,action: string, datetime: timestamp[s], author: string, (...) Pandas version: 1.3.4"
```
As a result, I got an exception 
```
"TypeError: Couldn't cast array of type timestamp[s] to null". 
```
Full stack-trace in the attached file below.

I also attach a minimized dataset (data.json, a single entry) that reproduces the error.

**Observations**(on the minimal example): 
- if I remove _all fields before_ `body`, a different error appears,
- if I remove _all fields after_ `body`, yet another error appears,
- if `body` is _the only field_, the error disappears.

So this might be one complex bug or several edge cases interacting. I haven’t dug deeper.

Also changing the file extension to `.json` or `.txt` avoids the problem. This suggests **a possible workaround** for the general case: convert `.jsonl` to `.json`. Though I haven’t verified correctness of that workaround yet.

Anyway my understanding is that `load_dataset` with first argument set to "json" should properly handle `.jsonl` files. Correct me if I'm wrong.

[stack_trace.txt](https://github.com/user-attachments/files/21004153/stack_trace.txt)
[data.json](https://github.com/user-attachments/files/21004164/data.json)

P.S.
I discovered this while going through the HuggingFace tutorial. Specifically [this part](https://huggingface.co/learn/llm-course/chapter5/5?fw=pt).I will try to inform the tutorial team about this issue, as it can be a showstopper for young 🤗 adepts.

### Steps to reproduce the bug

1. Download attached [data.json](https://github.com/user-attachments/files/21004164/data.json) file.
2. Run the following code which should work correctly:
```
from datasets import load_dataset
load_dataset("json", data_files="data.json", split="train")
```
3. Change extension of the `data` file to `.jsonl` and run:
```
from datasets import load_dataset
load_dataset("json", data_files="data.jsonl", split="train")
```

This will trigger an error like the one in the attached [stack_trace.txt](https://github.com/user-attachments/files/21004153/stack_trace.txt).

One can also try removing fields before the `body` field and after it. These actions give different errors.


### Expected behavior

Parsing data in `.jsonl` format should yield the same result as parsing the same data in `.json` format. In any case, the content of a string field should never be interpreted as part of the dataset schema.

### Environment info

datasets version: _3.6.0_
pyarrow version: _20.0.0_
Python version: _3.11.9_
platform version: _macOS-15.5-arm64-arm-64bit_

Somehow I created the issue twice🙈 This one is an exact duplicate of #7664.
Custom metadata filenames

example: https://huggingface.co/datasets/lhoestq/overlapping-subsets-imagefolder/tree/main

To make multiple subsets for an imagefolder (one metadata file per subset), e.g.

```yaml
configs:
  - config_name: default
    metadata_filenames:
      - metadata.csv
  - config_name: other
    metadata_filenames:
      - metadata2.csv
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7663). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
AttributeError: type object 'tqdm' has no attribute '_lock'

### Describe the bug

`AttributeError: type object 'tqdm' has no attribute '_lock'`

It occurs when I'm trying to load datasets in thread pool.

Issue https://github.com/huggingface/datasets/issues/6066 and PR https://github.com/huggingface/datasets/pull/6067 https://github.com/huggingface/datasets/pull/6068 tried to fix this. 


### Steps to reproduce the bug

Will have to try several times to reproduce the error because it is concerned with threads.
 
1. save some datasets for test
```pythonfrom datasets import Dataset, DatasetDict
import os

os.makedirs("test_dataset_shards", exist_ok=True)

for i in range(10):
    data = Dataset.from_dict({"text": [f"example {j}" for j in range(1000000)]})
    data = DatasetDict({'train': data})
    data.save_to_disk(f"test_dataset_shards/shard_{i}")
```

2. load them in a thread pool 
```python
from datasets import load_from_disk
from tqdm import tqdm
from concurrent.futures import ThreadPoolExecutor, as_completed
import glob

datas = glob.glob('test_dataset_shards/shard_*')
with ThreadPoolExecutor(max_workers=10) as pool:
    futures = [pool.submit(load_from_disk, it) for it in datas]
datas = []
for future in tqdm(as_completed(futures), total=len(futures)):
    datas.append(future.result())
```

### Expected behavior

no exception raised

### Environment info

datasets==2.19.0
python==3.10

Deleting a class (**not instance**) attribute might be invalid in this case, which is `tqdm` doing in `ensure_lock`.

```python
from tqdm import tqdm as old_tqdm

class tqdm1(old_tqdm):
     def __delattr__(self, attr):
        try:
            super().__delattr__(attr)
        except AttributeError:
            if attr != '_lock':
                print(attr)
                raise

class Meta(type):
    def __delattr__(cls, name):
        if name == "_lock":
            return  
        return super().__delattr__(name)
    
class tqdm2(old_tqdm, metaclass=Meta):
    pass

del tqdm2._lock
del tqdm1._lock # error
```

https://github.com/huggingface/datasets/blob/e71b0b19d79c7531f9b9bea7c09916b5f6157f42/src/datasets/utils/tqdm.py#L104-L122

A cheaper option (seems to work in my case):  
```python
from datasets import tqdm as hf_tqdm
hf_tqdm.set_lock(hf_tqdm.get_lock())
```
Fix: Prevent loss of info.features and column_names in IterableDatasetDict.map when features is None

This PR fixes a bug where calling `IterableDatasetDict.map()` or `IterableDataset.map()` with the default `features=None` argument would overwrite the existing `info.features` attribute with `None`. This, in turn, caused the resulting dataset to lose its schema, breaking downstream usage of attributes like `column_names`.

Why
Previously, the code would always set `info.features = features`, even if `features` was `None`. When mapping with removal of columns or other transformations, this led to the destruction of the schema and caused failures in code that relied on the dataset schema being present.

How
We now update `info.features` only if `features` is not `None`. This preserves the original schema unless the user explicitly provides a new one.

Reference
Fixes #7568

Hi!
I haven’t included a test for this change, as the fix is quite small and targeted.
Please let me know if you’d like a test for this case or if you’d prefer to handle it during review.
Thanks!

we can't know in advance the `features` after map() (it transforms the data !), so you can reuse the `features` from `info.features`

I'll the patch as suggested — `info.features = features` or `self.info.features` — to ensure schema preservation while keeping the logic simple and explicit. WDYT?


info.features should be None in the general case, and replaced by the user's `features` if it's passed explicitly with `map(..., features=...)`

https://github.com/huggingface/datasets/issues/7568 is not an issue we can fix

> info.features should be None in the general case, and replaced by the user's `features` if it's passed explicitly with `map(..., features=...)`
> 
> #7568 is not an issue we can fix

Thanks for the clarification! Totally makes sense now — I understand that features=None is the expected behavior post-map() unless explicitly passed, and that preserving old schema by default could lead to incorrect assumptions.
Closing this one — appreciate the feedback as always
fix(iterable): ensure MappedExamplesIterable supports state_dict for resume

Fixes #7630

### Problem
When calling `.map()` on an `IterableDataset`, resuming from a checkpoint skips a large number of samples. This is because `MappedExamplesIterable` did not implement `state_dict()` or `load_state_dict()`, so checkpointing was not properly delegated to the underlying iterable.

### What This PR Does
This patch adds:
```python
def state_dict(self):
    return self.ex_iterable.state_dict()

def load_state_dict(self, state):
    self.ex_iterable.load_state_dict(state)
```

to MappedExamplesIterable, so the wrapped base iterable's state can be saved and restored as expected.

Result
Using .map() no longer causes sample skipping after checkpoint resume.

Let me know if a dedicated test case is required — happy to add one!
fix(load): strip deprecated use_auth_token from config_kwargs

Fixes #7504

This PR resolves a compatibility error when loading datasets via `load_dataset()` using outdated arguments like `use_auth_token`.

**What was happening:**
Users passing `use_auth_token` in `load_dataset(..., use_auth_token=...)` encountered a `ValueError`: BuilderConfig ParquetConfig(...) doesn't have a 'use_auth_token' key.

**Why:**
`use_auth_token` has been deprecated and removed from config definitions (replaced by `token`), but the `load_dataset()` function still forwarded it via `**config_kwargs` to BuilderConfigs, leading to unrecognized key errors.

**Fix:**
We now intercept and strip `use_auth_token` from `config_kwargs` inside `load_dataset`, replacing it with a warning:
```python
if "use_auth_token" in config_kwargs:
    logger.warning("The 'use_auth_token' argument is deprecated. Please use 'token' instead.")
    config_kwargs.pop("use_auth_token")

```
This ensures legacy compatibility while guiding users to switch to the token argument.

Let me know if you'd prefer a deprecation error instead of a warning. Thanks!
feat(load): fallback to `load_from_disk()` when loading a saved dataset directory

### Related Issue

Fixes #7503  
Partially addresses #5044 by allowing `load_dataset()` to auto-detect and gracefully delegate to `load_from_disk()` for locally saved datasets.

---

### What does this PR do?

This PR introduces a minimal fallback mechanism in `load_dataset()` that detects when the provided `path` points to a dataset saved using `save_to_disk()`, and automatically redirects to `load_from_disk()`.

#### 🐛 Before (unexpected metadata-only rows):

```python
ds = load_dataset("/path/to/saved_dataset")
# → returns rows with only internal metadata (_data_files, _fingerprint, etc.)
````

#### ✅ After (graceful fallback):

```python
ds = load_dataset("/path/to/saved_dataset")
# → logs a warning and internally switches to load_from_disk()
```

---

### Why is this useful?

* Prevents confusion when reloading local datasets saved via `save_to_disk()`.
* Enables smoother compatibility with frameworks (e.g., TRL, `lighteval`) that rely on `load_dataset()` calls.
* Fully backward-compatible — hub-based loading, custom builders, and streaming remain untouched.
Add columns support to JSON loader for selective key filtering

Fixes #7594 
This PR adds support for filtering specific columns when loading datasets from .json or .jsonl files — similar to how the columns=... argument works for Parquet.

As suggested, support for the `columns=...` argument (previously available for Parquet) has now been extended to **JSON and JSONL** loading via `load_dataset(...)`. You can now load only specific keys/columns and skip the rest — which should help in cases where some fields are unclean, inconsistent, or just unnecessary.

### Example:

```python
from datasets import load_dataset

dataset = load_dataset("json", data_files="your_data.jsonl", columns=["id", "title"])
print(dataset["train"].column_names)
# Output: ['id', 'title']
```

### Summary of changes:

* Added `columns: Optional[List[str]]` to `JsonConfig`
* Updated `_generate_tables()` to filter selected columns
* Forwarded `columns` argument from `load_dataset()` to the config
* Added test for validation(should be fine!)

Let me know if you'd like the same to be added for CSV or others as a follow-up — happy to help.

I need this feature right now. It would be great if it could automatically fill in None for non-existent keys instead of reporting an error.

> I need this feature right now. It would be great if it could automatically fill in None for non-existent keys instead of reporting an error.

Hi @aihao2000, Just to confirm — I have done the changes you asked for!
If you pass columns=["key1", "key2", "optional_key"] to load_dataset(..., columns=...), and any of those keys are missing from the input JSON objects, the loader will automatically fill those columns with None values, instead of raising an error.
Enable parallel shard upload in push_to_hub() using num_proc

Fixes #7591

### Add num_proc support to `push_to_hub()` for parallel shard upload

This PR adds support for parallel upload of dataset shards via the `num_proc` argument in `Dataset.push_to_hub()`.

📌 While the `num_proc` parameter was already present in the `push_to_hub()` signature and correctly passed to `_push_parquet_shards_to_hub()`, it was not being used to parallelize the upload.

🔧 This PR updates the internal `_push_parquet_shards_to_hub()` function to:

- Use `multiprocessing.Pool` and `iflatmap_unordered()` for concurrent shard upload when `num_proc > 1`
- Preserve original serial upload behavior if `num_proc` is `None` or ≤ 1
- Keep tqdm progress and commit behavior unchanged

Let me know if any test coverage or further changes are needed!


it was already added in https://github.com/huggingface/datasets/pull/7606 actually ^^'

Oh sure sure, Closing this one as redundant.
Fix misleading add_column() usage example in docstring

Fixes #7611

This PR fixes the usage example in the Dataset.add_column() docstring, which previously implied that add_column() modifies the dataset in-place.

Why:
The method returns a new dataset with the additional column, and users must assign the result to a variable to preserve the change.

This should make the behavior clearer for users.
@lhoestq @davanstrien 

I believe there are other occurences of cases like this, like select_columns, select, filter, shard and flatten, could you also fix the docstring for them as well before we merge ?

Done! @lhoestq! I've updated the docstring examples for the following methods to clarify that they return new datasets instead of modifying in-place:

- `select_columns`
- `select`
- `filter`
- `shard`
- `flatten`


Also, any suggestions on what kind of issues I should work on next? I tried looking on my own, but I’d be happy if you could assign me something — I’ll do my best!
loading mozilla-foundation--common_voice_11_0 fails

### Describe the bug

Hello everyone,

i am trying to load `mozilla-foundation--common_voice_11_0` and it fails. Reproducer 

```
import datasets
datasets.load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", streaming=True, trust_remote_code=True)
```
and it fails with
```
File ~/opt/envs/.../lib/python3.10/site-packages/datasets/utils/file_utils.py:827, in _add_retries_to_file_obj_read_method.<locals>.read_with_retries(*args, **kwargs)
    825 for retry in range(1, max_retries + 1):
    826     try:
--> 827         out = read(*args, **kwargs)
    828         break
    829     except (
    830         _AiohttpClientError,
    831         asyncio.TimeoutError,
    832         requests.exceptions.ConnectionError,
    833         requests.exceptions.Timeout,
    834     ) as err:

File /usr/lib/python3.10/codecs.py:322, in BufferedIncrementalDecoder.decode(self, input, final)
    319 def decode(self, input, final=False):
    320     # decode input (taking the buffer into account)
    321     data = self.buffer + input
--> 322     (result, consumed) = self._buffer_decode(data, self.errors, final)
    323     # keep undecoded input until the next call
    324     self.buffer = data[consumed:]

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
```
When i remove streaming then everything is good but i need `streaming=True`

### Steps to reproduce the bug

```
import datasets
datasets.load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", streaming=True, trust_remote_code=True)
```

### Expected behavior

Expected that it will download dataset

### Environment info

datasets==3.6.0 
python3.10
on all platforms linux/win/mac

@claude Could you please address this issue

kinda related: https://github.com/huggingface/datasets/issues/7675
Introduce subset_name as an alias of config_name

### Feature request

Add support for `subset_name` as an alias for `config_name` in the datasets library and related tools (such as loading scripts, documentation, and metadata).

### Motivation

The Hugging Face Hub dataset viewer displays a column named **"Subset"**, which refers to what is currently technically called config_name in the datasets library. This inconsistency has caused confusion for many users, especially those unfamiliar with the internal terminology.

I have repeatedly received questions from users trying to understand what "config" means, and why it doesn’t match what they see as "subset" on the Hub. Renaming everything to `subset_name` might be too disruptive, but introducing subset_name as a clear alias for config_name could significantly improve user experience without breaking backward compatibility.

This change would:
- Align terminology across the Hub UI and datasets codebase
- Reduce user confusion, especially for newcomers
- Make documentation and examples more intuitive


I second this! When you come from the Hub, the intuitive question is "how do I set the subset name", and it's not easily answered from the docs: `subset_name` would answer this directly.

I've submitted PR [#7657](https://github.com/huggingface/datasets/pull/7657) to introduce subset_name as a user-facing alias for name in load_dataset, keeping terminology consistent with the Hub UI (“Subset”). It’s fully backward-compatible and includes a conflict check.

Let me know if you'd like me to include tests as part of the PR — happy to add them if needed!

The main usage is as a positional argument anyway, so I wouldn't necessarily agree that we need an alias (with the risk of confusing users). But happy to have more mentions in the docs of syntaxes like `load_dataset("dataset_name", "subset_name")`

> The main usage is as a positional argument anyway, so I wouldn't necessarily agree that we need an alias (with the risk of confusing users). But happy to have more mentions in the docs of syntaxes like `load_dataset("dataset_name", "subset_name")`

Thanks @lhoestq, totally fair point — especially with positional usage being the norm. I’m happy to align with the team’s direction here. If you'd prefer, I can update this PR to shift the focus to documentation/examples (e.g., showing "subset_name" as the second arg).
"open" in globals()["__builtins__"], an error occurs: "TypeError: argument of type 'module' is not iterable"

When I run the following code, an error occurs: "TypeError: argument of type 'module' is not iterable"
```python
print("open" in globals()["__builtins__"])
```
Traceback (most recent call last):
  File "./main.py", line 2, in <module>
    print("open" in globals()["__builtins__"])
          ^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'module' is not iterable

But this code runs fine in datasets, I don't understand why
[src/datasets/utils/patching.py#L96](https://github.com/huggingface/datasets/blob/3.6.0/src/datasets/utils/patching.py#L96)

@kuanyan9527 Your query is indeed valid. Following could be its reasoning:

Quoting from https://stackoverflow.com/a/11181607:
"By default, when in the `__main__` module,` __builtins__` is the built-in module `__builtin__` (note: no 's'); when in any other module, `__builtins__` is an alias for the dictionary of the `__builtin__` module itself."

Can you confirm if you are running the snippet `print("open" in globals()["__builtins__"])` in the default? In that case, as expected, `__builtins__` is a module which is causing the error. But in the codebase, the class `patch_submodule`, is primarily used in the second circumstance, where it acts as a dictionary. Hence causing the code to function successfully.

Hope this helps.

@kuanyan9527 Are there any more queries in this regards, else please feel free to close the issue.
Thank you.

Your answer is very important to me,thanks.

I encountered this error when running datasets with pypy,
`TypeError: argument of type 'module' is not iterable` in [src/datasets/utils/patching.py#L96](https://github.com/huggingface/datasets/blob/3.6.0/src/datasets/utils/patching.py#L96)
by modifying `globals()["__builtins__"]` to `builtins.__dict__`, importing via `import builtins`.
Can this be applied to the community?
Replace Sequence by List

Sequence is just a utility that we need to keep for backward compatibility. And `[ ]` was used instead but doesn't allow passing the length of the list.

This PR removes most mentions of Sequence and usage of `[ ]` and defines a proper List type instead.

before: `Sequence(Value("int64"))` or `[Value("int64")]`
now: `List(Value("int64"))`

This PR conserves full backward compatibility. And it's a good occasion with the release of 4.0.0

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7634). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Graceful Error Handling for cast_column("image", Image(decode=True)) in Hugging Face Datasets

### Feature request

Currently, when using dataset.cast_column("image", Image(decode=True)), the pipeline throws an error and halts if any image in the dataset is invalid or corrupted (e.g., truncated files, incorrect formats, unreachable URLs). This behavior disrupts large-scale processing where a few faulty samples are common.
reference : https://discuss.huggingface.co/t/handle-errors-when-loading-images-404-corrupted-etc/50318/5
                  https://discuss.huggingface.co/t/handling-non-existing-url-in-image-dataset-while-cast-column/69185
                  
Proposed Feature

Introduce a mechanism (e.g., a continue_on_error=True flag or global error handling mode) in Image(decode=True) that:

Skips invalid images and sets them as None, or

Logs the error but allows the rest of the dataset to be processed without interruption.


Example Usage

from datasets import load_dataset, Image

dataset = load_dataset("my_dataset")
dataset = dataset.cast_column("image", Image(decode=True, continue_on_error=True))

Benefits

Ensures robust large-scale image dataset processing.

Improves developer productivity by avoiding custom retry/error-handling code.

Aligns with best practices in dataset preprocessing pipelines that tolerate minor data corruption.


Potential Implementation Options

Internally wrap the decoding in a try/except block.

Return None or a placeholder on failure.

Optionally allow custom error callbacks or logging.

### Motivation

Robustness: Large-scale image datasets often contain a small fraction of corrupt files or unreachable URLs. Halting on the first error forces users to write custom workarounds or preprocess externally.

Simplicity: A built-in flag removes boilerplate try/except logic around every decode step.

Performance: Skipping invalid samples inline is more efficient than a two-pass approach (filter then decode).

### Your contribution


1. API Change
Extend datasets.features.Image(decode=True) to accept continue_on_error: bool = False.

2. Behavior
If continue_on_error=False (default), maintain current behavior: any decode error raises an exception.
If continue_on_error=True, wrap decode logic in try/except:
On success: store the decoded image.
On failure: log a warning (e.g., via logging.warning) and set the field to None (or a sentinel value).


3. Optional Enhancements
Allow a callback hook:
Image(decode=True, continue_on_error=True, on_error=lambda idx, url, exc: ...)
Emit metrics or counts of skipped images.

Hi! This is now handled in PR #7638

Thank you for implementing the suggestion it would be great help in our use case.
Pass user-agent from DownloadConfig into fsspec storage_options

Fixes part of issue #6046

### Problem
The `user-agent` defined in `DownloadConfig` was not passed down to fsspec-based filesystems like `HfFileSystem`, which prevents proper identification/tracking of client requests.

### Solution
Added support for injecting the `user-agent` into `storage_options["headers"]` within `_prepare_single_hop_path_and_storage_options()` based on the `protocol`.

Now, when using `hf://`, `http://`, or `https://`, the custom user-agent is passed automatically.

### Code Location
Modified:
- `src/datasets/utils/file_utils.py`

Used `get_datasets_user_agent(...)` to ensure proper formatting and fallback logic.

- This PR assumes that `HfFileSystem` in `huggingface_hub` supports receiving `headers` in `storage_options`. If not, a follow-up PR can be opened to add this support to `HfFileSystem.__init__`.
- No test was added for this since it’s a config passthrough. If needed, I’d be happy to add one.
[bug] resume from ckpt skips samples if .map is applied

### Describe the bug

resume from ckpt skips samples if .map is applied

Maybe related: https://github.com/huggingface/datasets/issues/7538

### Steps to reproduce the bug

```python
from datasets import Dataset
from datasets.distributed import split_dataset_by_node


# Create dataset with map transformation
def create_dataset():
    ds = Dataset.from_dict({"id": list(range(100))})
    ds = ds.to_iterable_dataset(num_shards=4)
    ds = ds.map(lambda x: x) #comment it out to get desired behavior
    ds = split_dataset_by_node(ds, rank=0, world_size=2)
    return ds


ds = create_dataset()

# Iterate and save checkpoint after 10 samples
it = iter(ds)
for idx, sample in enumerate(it):
    if idx == 9:  # Checkpoint after 10 samples
        checkpoint = ds.state_dict()
        print(f"Checkpoint saved at sample: {sample['id']}")
        break

# Continue with original iterator
original_next_samples = []
for idx, sample in enumerate(it):
    original_next_samples.append(sample["id"])
    if idx >= 4:
        break

# Resume from checkpoint
ds_new = create_dataset()
ds_new.load_state_dict(checkpoint)

# Get samples from resumed iterator
it_new = iter(ds_new)
resumed_next_samples = []
for idx, sample in enumerate(it_new):
    resumed_next_samples.append(sample["id"])
    if idx >= 4:
        break

print(f"\nExpected next samples: {original_next_samples}")
print(f"Actual next samples:   {resumed_next_samples}")
print(
    f"\n❌ BUG: {resumed_next_samples[0] - original_next_samples[0]} samples were skipped!"
)

```

With map
```
Checkpoint saved at sample: 9

Expected next samples: [10, 11, 12, 13, 14]
Actual next samples:   [50, 51, 52, 53, 54]

❌ BUG: 40 samples were skipped!
```

### Expected behavior

without map
```
Expected next samples: [10, 11, 12, 13, 14]
Actual next samples:   [10, 11, 12, 13, 14]

❌ BUG: 0 samples were skipped!
```

### Environment info

datasets == 3.6.0

Thanks for reporting this — it looks like a separate but related bug to #7538, which involved sample loss when resuming an `IterableDataset` wrapped in `FormattedExamplesIterable`. That was resolved in #7553 by re-batching the iterable to track offset correctly.

In this case, the issue seems to arise specifically from applying `.map()` before sharding and checkpointing. That wraps the iterable in `MappedExamplesIterable`, which may not preserve or propagate `shard_example_idx` correctly across `.state_dict()` and `.load_state_dict()` calls.

You can see that without `.map()`, resume works fine — but with `.map()`, it jumps from sample 9 to 50, skipping the rest of the shard.

I'll dig deeper into how `MappedExamplesIterable` manages offsets and whether it supports proper checkpoint resumption. If not, we might need a fix similar to the one in #7553, or a wrapper to preserve resume metadata.

Happy to help fix it!


Let me know if a dedicated test case is required — happy to add one!
Creating a HF Dataset from lakeFS with S3 storage takes too much time!

Hi,

I’m new to HF dataset and I tried to create datasets based on data versioned in **lakeFS** _(**MinIO** S3 bucket as storage backend)_

Here I’m using ±30000 PIL image from MNIST data however it is taking around 12min to execute, which is a lot!

From what I understand, it is loading the images into cache then building the dataset.
– Please find bellow the execution screenshot –

Is there a way to optimize this or am I doing something wrong?

Thanks!

![Image](https://github.com/user-attachments/assets/c79257c8-f023-42a9-9e6f-0898b3ea93fe)

### > Update

The bottleneck, from what I understand, was making one network request per file

For 30k images, this meant 30k separate GET requests to the MinIO server through the S3 API, and that was killing the performance

Using webDataset to transform the large number of files to few .tar files and passing “webdataset” instead of “imagefolder” to the load_dataset function worked perfectly (took only ~11s)
feat: Add h5folder dataset loader for HDF5 support

### Related Issue
Closes #3113

### What does this PR do?

This PR introduces a new dataset loader module called **`h5folder`** to support loading datasets stored in **HDF5 (.h5)** format.

It allows users to do:

```python
from datasets import load_dataset
dataset = load_dataset("h5folder", data_dir="path/to/")
````

### 🧩 Design Overview

* Implemented inside `datasets/packaged_modules/h5folder/h5folder.py`
* Based on the `GeneratorBasedBuilder` API
* Uses `h5py` to read HDF5 files and yield examples
* Expects datasets such as `id`, `data`, and `label` inside `data.h5`
* Converts numpy arrays to Python types before yielding

### 🧪 Example `.h5` Structure (for local testing)

```python
import h5py
import numpy as np

with h5py.File("data.h5", "w") as f:
    f.create_dataset("id", data=np.arange(100))
    f.create_dataset("data", data=np.random.randn(100, 10))
    f.create_dataset("label", data=np.random.randint(0, 2, size=100))
```

### ✅ Testing

- The loader logic follows the structure of existing modules like `imagefolder`
- Will rely on Hugging Face CI to validate integration
- Manually testing planned once merged or during feedback

### 📁 Files Added

* `datasets/src/datasets/packaged_modules/h5folder/h5folder.py`

### 📌 Component(s) Affected

* `area/datasets`
* `area/load`

### 📦 Release Note Classification

* `rn/feature` – Adds support for loading `.h5` datasets via `load_dataset("h5folder", ...)`

---

Let me know if any changes or improvements are needed — happy to iterate. Thanks for reviewing!


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7625). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I guess test failed cause import os, import h5py, and import datasets lines are not alphabetically sorted, or not grouped properly.

![image](https://github.com/user-attachments/assets/ab73f8f9-da50-4ba8-9b2d-b7c30fce94f5)


This commit was accidental - `[Merge branch 'main' into patch-4]`. The 
`[chore: fix import order in h5folder.py to satisfy linter]` should solve the import order issue.
#Dataset Make "image" column appear first in dataset preview UI

Hi!

#Dataset

I’m currently uploading a dataset that includes an `"image"` column (PNG files), along with some metadata columns. The dataset is loaded from a .jsonl file. My goal is to have the "image" column appear as the first column in the dataset card preview UI on the :hugs: Hub.

However, at the moment, the `"image"` column is not the first—in fact, it appears last, which is not ideal for the presentation I’d like to achieve.

I have a couple of questions:

Is there a way to force the dataset card to display the `"image"` column first?
Is there currently any way to control or influence the column order in the dataset preview UI?
Does the order of keys in the .jsonl file or the features argument affect the display order?
Thanks again for your time and help! :blush:

Hi ! It should follow the same order as the order of the keys in the metadata file

Hi! Thank you for your answer. 

As you said it, I I forced every key in every JSON to have an order using `collections. OrderedDict` in Python. Now, it works!

TY
fix: raise error in FolderBasedBuilder when data_dir and data_files are missing

### Related Issues/PRs

Fixes #6152

---

### What changes are proposed in this pull request?

This PR adds a dedicated validation check in the `_info()` method of the `FolderBasedBuilder` class to ensure that users provide either `data_dir` or `data_files` when loading folder-based datasets (such as `audiofolder`, `imagefolder`, etc.).

---

### Why this change?

Previously, when calling:

```python
load_dataset("audiofolder")
````

without specifying `data_dir` or `data_files`, the loader would silently fallback to the **current working directory**, leading to:

* Long loading times
* Unexpected behavior (e.g., scanning unrelated files)

This behavior was discussed in issue #6152. As suggested by maintainers, the fix has now been implemented directly inside the `FolderBasedBuilder._info()` method — keeping the logic localized to the specific builder instead of a generic loader function.

---

### How is this PR tested?

* ✅ Manually tested by calling `load_dataset("audiofolder")` with no `data_dir` or `data_files` → a `ValueError` is now raised early.
* ✅ Existing functionality (with valid input) remains unaffected.

---

### Does this PR require documentation update?

* [x] No

---

### Release Notes

#### Is this a user-facing change?

* [x] Yes

> Folder-based datasets now raise an explicit error if neither `data_dir` nor `data_files` are specified, preventing unintended fallback to the current working directory.

---

#### What component(s) does this PR affect?

* [x] `area/datasets`
* [x] `area/load`

---

<a name="release-note-category"></a>

#### How should the PR be classified?

* [x] `rn/bug-fix` - A user-facing bug fix

---

#### Should this be included in the next patch release?

* [x] Yes

@lhoestq Moved the logic to FolderBasedBuilder._info() as discussed in previous PR (#7618). Let me know if anything else is needed — happy to update!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7623). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Guard against duplicate builder_kwargs/config_kwargs in load_dataset_…

…builder (#4910 )

### What does this PR do?

Fixes edge case in `load_dataset_builder` by raising a `TypeError` if the same key exists in both `builder_kwargs` and `config_kwargs`.

### Implementation details

- Added a guard clause in `load_dataset_builder` to detect duplicate keys between `builder_kwargs` and `config_kwargs`
 
- Wrote a unit test in `tests/test_load_duplicate_keys.py` to verify the exception is raised correctly 

### Fixes

Closes #4910 

### Reviewers

@zach-huggingface
@SunMarc 

Would appreciate your review if you have time - thanks!

Hi folks, this PR fixes the duplicate-kwargs edge case and includes a unit test. Would love a review when you have a moment!

@zach-huggingface
@SunMarc