josteece/github-issues huggingface dataset:
Introduce subset_name as an alias of config_name
Opened by albertvillanova (open)

### Feature request

Add support for `subset_name` as an alias for `config_name` in the datasets library and related tools (such as loading scripts, documentation, and metadata).

### Motivation

The Hugging Face Hub dataset viewer displays a column named **"Subset"**, which refers to what is currently technically called config_name in the datasets library. This inconsistency has caused confusion for many users, especially those unfamiliar with the internal terminology.

I have repeatedly received questions from users trying to understand what "config" means, and why it doesn’t match what they see as "subset" on the Hub. Renaming everything to `subset_name` might be too disruptive, but introducing subset_name as a clear alias for config_name could significantly improve user experience without breaking backward compatibility.

This change would:
- Align terminology across the Hub UI and datasets codebase
- Reduce user confusion, especially for newcomers
- Make documentation and examples more intuitive


Comments:
I second this! When you come from the Hub, the intuitive question is "how do I set the subset name", and it's not easily answered from the docs: `subset_name` would answer this directly.

I've submitted PR [#7657](https://github.com/huggingface/datasets/pull/7657) to introduce subset_name as a user-facing alias for name in load_dataset, keeping terminology consistent with the Hub UI (“Subset”). It’s fully backward-compatible and includes a conflict check.

Let me know if you'd like me to include tests as part of the PR — happy to add them if needed!

The main usage is as a positional argument anyway, so I wouldn't necessarily agree that we need an alias (with the risk of confusing users). But happy to have more mentions in the docs of syntaxes like `load_dataset("dataset_name", "subset_name")`

> The main usage is as a positional argument anyway, so I wouldn't necessarily agree that we need an alias (with the risk of confusing users). But happy to have more mentions in the docs of syntaxes like `load_dataset("dataset_name", "subset_name")`

Thanks @lhoestq, totally fair point — especially with positional usage being the norm. I’m happy to align with the team’s direction here. If you'd prefer, I can update this PR to shift the focus to documentation/examples (e.g., showing "subset_name" as the second arg).
"open" in globals()["__builtins__"], an error occurs: "TypeError: argument of type 'module' is not iterable"
Opened by kuanyan9527 (open)

When I run the following code, an error occurs: "TypeError: argument of type 'module' is not iterable"
```python
print("open" in globals()["__builtins__"])
```
Traceback (most recent call last):
  File "./main.py", line 2, in <module>
    print("open" in globals()["__builtins__"])
          ^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'module' is not iterable

But this code runs fine in datasets, I don't understand why
[src/datasets/utils/patching.py#L96](https://github.com/huggingface/datasets/blob/3.6.0/src/datasets/utils/patching.py#L96)

Comments:
@kuanyan9527 Your query is indeed valid. Following could be its reasoning:

Quoting from https://stackoverflow.com/a/11181607:
"By default, when in the `__main__` module,` __builtins__` is the built-in module `__builtin__` (note: no 's'); when in any other module, `__builtins__` is an alias for the dictionary of the `__builtin__` module itself."

Can you confirm if you are running the snippet `print("open" in globals()["__builtins__"])` in the default? In that case, as expected, `__builtins__` is a module which is causing the error. But in the codebase, the class `patch_submodule`, is primarily used in the second circumstance, where it acts as a dictionary. Hence causing the code to function successfully.

Hope this helps.

@kuanyan9527 Are there any more queries in this regards, else please feel free to close the issue.
Thank you.

Your answer is very important to me,thanks.

I encountered this error when running datasets with pypy,
`TypeError: argument of type 'module' is not iterable` in [src/datasets/utils/patching.py#L96](https://github.com/huggingface/datasets/blob/3.6.0/src/datasets/utils/patching.py#L96)
by modifying `globals()["__builtins__"]` to `builtins.__dict__`, importing via `import builtins`.
Can this be applied to the community?
Fix: Preserve float columns in JSON loader when values are integer-like (e.g. 0.0, 1.0)
Opened by ArjunJagdale (open)

This PR fixes a bug in the JSON loader where columns containing float values like `[0.0, 1.0, 2.0]` were being implicitly coerced to `int`, due to pandas or Arrow type inference.

This caused issues downstream in statistics computation (e.g., dataset-viewer) where such columns were incorrectly labeled as `"int"` instead of `"float"`.

### 🔍 What was happening:
When the JSON loader falls back to `pandas_read_json()` (after `pa.read_json()` fails), pandas/Arrow can coerce float values to integers if all values are integer-like (e.g., `0.0 == 0`).

### ✅ What this PR does:
- Adds a check in the fallback path of `_generate_tables()`
- Ensures that columns made entirely of floats are preserved as `"float64"` even if they are integer-like (e.g. `0.0`, `1.0`)
- This prevents loss of float semantics when creating the Arrow table

### 🧪 Reproducible Example:
```json
[{"col": 0.0}, {"col": 1.0}, {"col": 2.0}]
````

Previously loaded as:

* `int`

Now correctly loaded as:

* `float`

Fixes #6937


Comments:
Replace Sequence by List
Opened by lhoestq (closed)

Sequence is just a utility that we need to keep for backward compatibility. And `[ ]` was used instead but doesn't allow passing the length of the list.

This PR removes most mentions of Sequence and usage of `[ ]` and defines a proper List type instead.

before: `Sequence(Value("int64"))` or `[Value("int64")]`
now: `List(Value("int64"))`

This PR conserves full backward compatibility. And it's a good occasion with the release of 4.0.0

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7634). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Proposal: Small Tamil Discourse Coherence Dataset.
Opened by bikkiNitSrinagar (open)

I’m a beginner from NIT Srinagar proposing a dataset of 50 Tamil text pairs for discourse coherence (coherent/incoherent labels) to support NLP research in low-resource languages.
- Size: 50 samples
- Format: CSV with columns (text1, text2, label)
- Use case: Training NLP models for coherence
I’ll use GitHub’s web editor and Google Colab. Please confirm if this fits.

Comments:
Graceful Error Handling for cast_column("image", Image(decode=True)) in Hugging Face Datasets
Opened by ganiket19 (open)

### Feature request

Currently, when using dataset.cast_column("image", Image(decode=True)), the pipeline throws an error and halts if any image in the dataset is invalid or corrupted (e.g., truncated files, incorrect formats, unreachable URLs). This behavior disrupts large-scale processing where a few faulty samples are common.
reference : https://discuss.huggingface.co/t/handle-errors-when-loading-images-404-corrupted-etc/50318/5
                  https://discuss.huggingface.co/t/handling-non-existing-url-in-image-dataset-while-cast-column/69185
                  
Proposed Feature

Introduce a mechanism (e.g., a continue_on_error=True flag or global error handling mode) in Image(decode=True) that:

Skips invalid images and sets them as None, or

Logs the error but allows the rest of the dataset to be processed without interruption.


Example Usage

from datasets import load_dataset, Image

dataset = load_dataset("my_dataset")
dataset = dataset.cast_column("image", Image(decode=True, continue_on_error=True))

Benefits

Ensures robust large-scale image dataset processing.

Improves developer productivity by avoiding custom retry/error-handling code.

Aligns with best practices in dataset preprocessing pipelines that tolerate minor data corruption.


Potential Implementation Options

Internally wrap the decoding in a try/except block.

Return None or a placeholder on failure.

Optionally allow custom error callbacks or logging.

### Motivation

Robustness: Large-scale image datasets often contain a small fraction of corrupt files or unreachable URLs. Halting on the first error forces users to write custom workarounds or preprocess externally.

Simplicity: A built-in flag removes boilerplate try/except logic around every decode step.

Performance: Skipping invalid samples inline is more efficient than a two-pass approach (filter then decode).

### Your contribution


1. API Change
Extend datasets.features.Image(decode=True) to accept continue_on_error: bool = False.

2. Behavior
If continue_on_error=False (default), maintain current behavior: any decode error raises an exception.
If continue_on_error=True, wrap decode logic in try/except:
On success: store the decoded image.
On failure: log a warning (e.g., via logging.warning) and set the field to None (or a sentinel value).


3. Optional Enhancements
Allow a callback hook:
Image(decode=True, continue_on_error=True, on_error=lambda idx, url, exc: ...)
Emit metrics or counts of skipped images.

Comments:
Hi! This is now handled in PR #7638

Thank you for implementing the suggestion it would be great help in our use case.
Pass user-agent from DownloadConfig into fsspec storage_options
Opened by ArjunJagdale (open)

Fixes part of issue #6046

### Problem
The `user-agent` defined in `DownloadConfig` was not passed down to fsspec-based filesystems like `HfFileSystem`, which prevents proper identification/tracking of client requests.

### Solution
Added support for injecting the `user-agent` into `storage_options["headers"]` within `_prepare_single_hop_path_and_storage_options()` based on the `protocol`.

Now, when using `hf://`, `http://`, or `https://`, the custom user-agent is passed automatically.

### Code Location
Modified:
- `src/datasets/utils/file_utils.py`

Used `get_datasets_user_agent(...)` to ensure proper formatting and fallback logic.

Comments:
- This PR assumes that `HfFileSystem` in `huggingface_hub` supports receiving `headers` in `storage_options`. If not, a follow-up PR can be opened to add this support to `HfFileSystem.__init__`.
- No test was added for this since it’s a config passthrough. If needed, I’d be happy to add one.
[bug] resume from ckpt skips samples if .map is applied
Opened by felipemello1 (open)

### Describe the bug

resume from ckpt skips samples if .map is applied

Maybe related: https://github.com/huggingface/datasets/issues/7538

### Steps to reproduce the bug

```python
from datasets import Dataset
from datasets.distributed import split_dataset_by_node


# Create dataset with map transformation
def create_dataset():
    ds = Dataset.from_dict({"id": list(range(100))})
    ds = ds.to_iterable_dataset(num_shards=4)
    ds = ds.map(lambda x: x) #comment it out to get desired behavior
    ds = split_dataset_by_node(ds, rank=0, world_size=2)
    return ds


ds = create_dataset()

# Iterate and save checkpoint after 10 samples
it = iter(ds)
for idx, sample in enumerate(it):
    if idx == 9:  # Checkpoint after 10 samples
        checkpoint = ds.state_dict()
        print(f"Checkpoint saved at sample: {sample['id']}")
        break

# Continue with original iterator
original_next_samples = []
for idx, sample in enumerate(it):
    original_next_samples.append(sample["id"])
    if idx >= 4:
        break

# Resume from checkpoint
ds_new = create_dataset()
ds_new.load_state_dict(checkpoint)

# Get samples from resumed iterator
it_new = iter(ds_new)
resumed_next_samples = []
for idx, sample in enumerate(it_new):
    resumed_next_samples.append(sample["id"])
    if idx >= 4:
        break

print(f"\nExpected next samples: {original_next_samples}")
print(f"Actual next samples:   {resumed_next_samples}")
print(
    f"\n❌ BUG: {resumed_next_samples[0] - original_next_samples[0]} samples were skipped!"
)

```

With map
```
Checkpoint saved at sample: 9

Expected next samples: [10, 11, 12, 13, 14]
Actual next samples:   [50, 51, 52, 53, 54]

❌ BUG: 40 samples were skipped!
```

### Expected behavior

without map
```
Expected next samples: [10, 11, 12, 13, 14]
Actual next samples:   [10, 11, 12, 13, 14]

❌ BUG: 0 samples were skipped!
```

### Environment info

datasets == 3.6.0

Comments:
Thanks for reporting this — it looks like a separate but related bug to #7538, which involved sample loss when resuming an `IterableDataset` wrapped in `FormattedExamplesIterable`. That was resolved in #7553 by re-batching the iterable to track offset correctly.

In this case, the issue seems to arise specifically from applying `.map()` before sharding and checkpointing. That wraps the iterable in `MappedExamplesIterable`, which may not preserve or propagate `shard_example_idx` correctly across `.state_dict()` and `.load_state_dict()` calls.

You can see that without `.map()`, resume works fine — but with `.map()`, it jumps from sample 9 to 50, skipping the rest of the shard.

I'll dig deeper into how `MappedExamplesIterable` manages offsets and whether it supports proper checkpoint resumption. If not, we might need a fix similar to the one in #7553, or a wrapper to preserve resume metadata.

Happy to help fix it!


Let me know if a dedicated test case is required — happy to add one!
Add test for `as_iterable_dataset()` method in DatasetBuilder
Opened by ArjunJagdale (open)

This PR adds a test for the new `as_iterable_dataset()` method introduced in PR #7628.

The test:
- Loads a builder using `load_dataset_builder("c4", "en")`
- Runs `download_and_prepare()`
- Streams examples using `builder.as_iterable_dataset(split="train[:100]")`
- Verifies streamed examples contain the "text" field

This ensures that the builder correctly streams data from cached Arrow files.


Comments:
Add `as_iterable_dataset()` method to DatasetBuilder for streaming from cached Arrow files
Opened by ArjunJagdale (open)

This PR implements `builder.as_iterable_dataset(split=...)` as discussed in #5481.

It allows users to load an `IterableDataset` directly from cached Arrow files (using ArrowReader and ArrowExamplesIterable), without loading the full dataset into memory.

This is useful for large-scale training scenarios where memory is constrained. A test has also been added in `test_builder.py`.

Related to: #5481


Comments:
Creating a HF Dataset from lakeFS with S3 storage takes too much time!
Opened by Thunderhead-exe (closed)

Hi,

I’m new to HF dataset and I tried to create datasets based on data versioned in **lakeFS** _(**MinIO** S3 bucket as storage backend)_

Here I’m using ±30000 PIL image from MNIST data however it is taking around 12min to execute, which is a lot!

From what I understand, it is loading the images into cache then building the dataset.
– Please find bellow the execution screenshot –

Is there a way to optimize this or am I doing something wrong?

Thanks!

![Image](https://github.com/user-attachments/assets/c79257c8-f023-42a9-9e6f-0898b3ea93fe)

Comments:
### > Update

The bottleneck, from what I understand, was making one network request per file

For 30k images, this meant 30k separate GET requests to the MinIO server through the S3 API, and that was killing the performance

Using webDataset to transform the large number of files to few .tar files and passing “webdataset” instead of “imagefolder” to the load_dataset function worked perfectly (took only ~11s)
feat(map): reuse unchanged columns when input_columns specified to reduce disk usage (#6013)
Opened by ArjunJagdale (closed)

## Summary

This PR addresses [#6013](https://github.com/huggingface/datasets/issues/6013) by reusing unchanged columns from the original dataset in the `map()` method when `input_columns` is specified.

## What’s Implemented

- Injected logic at the end of `Dataset.map()` to:
  - Identify untouched columns not in `input_columns` or `remove_columns`
  - Select those columns from the original dataset
  - Concatenate them with the transformed result using `pyarrow.concat_tables`

## Example Behavior

```python
ds = Dataset.from_dict({"a": [1, 2], "b": [3, 4]})
ds2 = ds.map(lambda x: {"c": x["a"] + 10}, input_columns=["a"], remove_columns=["a"])
print(ds2.column_names)  # Output: ['b', 'c']
````

Column `b` is reused from the original dataset.

## Notes

* This keeps disk usage and caching minimal by avoiding full dataset duplication.
* Only triggered when `input_columns` is set.

---

cc @lhoestq @mariosasko for review 🙂


Comments:
feat: Add h5folder dataset loader for HDF5 support
Opened by ArjunJagdale (open)

### Related Issue
Closes #3113

### What does this PR do?

This PR introduces a new dataset loader module called **`h5folder`** to support loading datasets stored in **HDF5 (.h5)** format.

It allows users to do:

```python
from datasets import load_dataset
dataset = load_dataset("h5folder", data_dir="path/to/")
````

### 🧩 Design Overview

* Implemented inside `datasets/packaged_modules/h5folder/h5folder.py`
* Based on the `GeneratorBasedBuilder` API
* Uses `h5py` to read HDF5 files and yield examples
* Expects datasets such as `id`, `data`, and `label` inside `data.h5`
* Converts numpy arrays to Python types before yielding

### 🧪 Example `.h5` Structure (for local testing)

```python
import h5py
import numpy as np

with h5py.File("data.h5", "w") as f:
    f.create_dataset("id", data=np.arange(100))
    f.create_dataset("data", data=np.random.randn(100, 10))
    f.create_dataset("label", data=np.random.randint(0, 2, size=100))
```

### ✅ Testing

- The loader logic follows the structure of existing modules like `imagefolder`
- Will rely on Hugging Face CI to validate integration
- Manually testing planned once merged or during feedback

### 📁 Files Added

* `datasets/src/datasets/packaged_modules/h5folder/h5folder.py`

### 📌 Component(s) Affected

* `area/datasets`
* `area/load`

### 📦 Release Note Classification

* `rn/feature` – Adds support for loading `.h5` datasets via `load_dataset("h5folder", ...)`

---

Let me know if any changes or improvements are needed — happy to iterate. Thanks for reviewing!


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7625). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I guess test failed cause import os, import h5py, and import datasets lines are not alphabetically sorted, or not grouped properly.

![image](https://github.com/user-attachments/assets/ab73f8f9-da50-4ba8-9b2d-b7c30fce94f5)


This commit was accidental - `[Merge branch 'main' into patch-4]`. The 
`[chore: fix import order in h5folder.py to satisfy linter]` should solve the import order issue.
#Dataset Make "image" column appear first in dataset preview UI
Opened by jcerveto (closed)

Hi!

#Dataset

I’m currently uploading a dataset that includes an `"image"` column (PNG files), along with some metadata columns. The dataset is loaded from a .jsonl file. My goal is to have the "image" column appear as the first column in the dataset card preview UI on the :hugs: Hub.

However, at the moment, the `"image"` column is not the first—in fact, it appears last, which is not ideal for the presentation I’d like to achieve.

I have a couple of questions:

Is there a way to force the dataset card to display the `"image"` column first?
Is there currently any way to control or influence the column order in the dataset preview UI?
Does the order of keys in the .jsonl file or the features argument affect the display order?
Thanks again for your time and help! :blush:

Comments:
Hi ! It should follow the same order as the order of the keys in the metadata file

Hi! Thank you for your answer. 

As you said it, I I forced every key in every JSON to have an order using `collections. OrderedDict` in Python. Now, it works!

TY
fix: raise error in FolderBasedBuilder when data_dir and data_files are missing
Opened by ArjunJagdale (closed)

### Related Issues/PRs

Fixes #6152

---

### What changes are proposed in this pull request?

This PR adds a dedicated validation check in the `_info()` method of the `FolderBasedBuilder` class to ensure that users provide either `data_dir` or `data_files` when loading folder-based datasets (such as `audiofolder`, `imagefolder`, etc.).

---

### Why this change?

Previously, when calling:

```python
load_dataset("audiofolder")
````

without specifying `data_dir` or `data_files`, the loader would silently fallback to the **current working directory**, leading to:

* Long loading times
* Unexpected behavior (e.g., scanning unrelated files)

This behavior was discussed in issue #6152. As suggested by maintainers, the fix has now been implemented directly inside the `FolderBasedBuilder._info()` method — keeping the logic localized to the specific builder instead of a generic loader function.

---

### How is this PR tested?

* ✅ Manually tested by calling `load_dataset("audiofolder")` with no `data_dir` or `data_files` → a `ValueError` is now raised early.
* ✅ Existing functionality (with valid input) remains unaffected.

---

### Does this PR require documentation update?

* [x] No

---

### Release Notes

#### Is this a user-facing change?

* [x] Yes

> Folder-based datasets now raise an explicit error if neither `data_dir` nor `data_files` are specified, preventing unintended fallback to the current working directory.

---

#### What component(s) does this PR affect?

* [x] `area/datasets`
* [x] `area/load`

---

<a name="release-note-category"></a>

#### How should the PR be classified?

* [x] `rn/bug-fix` - A user-facing bug fix

---

#### Should this be included in the next patch release?

* [x] Yes

Comments:
@lhoestq Moved the logic to FolderBasedBuilder._info() as discussed in previous PR (#7618). Let me know if anything else is needed — happy to update!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7623). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Guard against duplicate builder_kwargs/config_kwargs in load_dataset_…
Opened by Shohail-Ismail (closed)

…builder (#4910 )

### What does this PR do?

Fixes edge case in `load_dataset_builder` by raising a `TypeError` if the same key exists in both `builder_kwargs` and `config_kwargs`.

### Implementation details

- Added a guard clause in `load_dataset_builder` to detect duplicate keys between `builder_kwargs` and `config_kwargs`
 
- Wrote a unit test in `tests/test_load_duplicate_keys.py` to verify the exception is raised correctly 

### Fixes

Closes #4910 

### Reviewers

@zach-huggingface
@SunMarc 

Would appreciate your review if you have time - thanks!

Comments:
Hi folks, this PR fixes the duplicate-kwargs edge case and includes a unit test. Would love a review when you have a moment!

@zach-huggingface
@SunMarc
Fixes in docs
Opened by lhoestq (closed)

before release 4.0

(I also did minor improvements to `features` to not show their `id=None` in their `__repr__()`)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7620). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.