helmo/github-issues huggingface dataset:
Provide an option of robust dataset iterator with error handling
State: open | Author: wwwjn (NONE)
Labels: name: enhancement

### Feature request

Adding an option to skip corrupted data samples when __iter__ is called. Currently the datasets behavior is throwing errors if the data sample if corrupted and let user aware and handle the data corruption. When I tried to try-catch the error at user level, the iterator will raise StopIteration when I called next() again. 

The way I try to do error handling is: (This doesn't work, unfortunately)

```
        # Load the dataset with streaming enabled
        dataset = load_dataset(
            "pixparse/cc12m-wds", split="train", streaming=True
        )
        # Get an iterator from the dataset
        iterator = iter(dataset)

        while True:
            try:
                # Try to get the next example
                example = next(iterator)
                
                # Try to access and process the image
                image = example["jpg"]
                pil_image = Image.fromarray(np.array(image))
                pil_image.verify()  # Verify it's a valid image file

            except StopIteration:  # Code path 1
                print("\nStopIteration was raised! Reach the end of dataset")
                raise StopIteration

            except Exception as e:  # Code path 2
                errors += 1
                print("Error! Skip this sample")
                cotinue
            else:
                successful += 1
```

This is because the `IterableDataset` already throws an error (reaches Code path 2). And if I continue call next(), it will hit Code path 1. This is because the inner iterator of `IterableDataset`([code](https://github.com/huggingface/datasets/blob/89bd1f971402acb62805ef110bc1059c38b1c8c6/src/datasets/iterable_dataset.py#L2242)) as been stopped, so calling next() on it will raise StopIteration. 

So I can not skip the corrupted data sample in this way. Would also love to hear any suggestions about creating a robust dataloader.

Thanks for your help in advance!


### Motivation

## Public dataset corruption might be common
A lot of users would use public dataset, and the public dataset might contains some corrupted data, especially for dataset with image / video etc.  I totally understand it's dataset owner and user's responsibility to ensure the data integrity / run data cleaning or preprocessing, but it would be easier for developers who would use the dataset

## Use cases
For example, a robust dataloader would be easy for users who want to try quick tests on different dataset, and chose one dataset which fits their needs. So user could use IterableDataloader with `stream=True` to use the dataset easily without downloading and removing corrupted data samples from the dataset.


### Your contribution

The error handling might not trivial and might need more careful design.
Code example for dataset.add_column() does not reflect correct way to use function
State: open | Author: shaily99 (NONE)

https://github.com/huggingface/datasets/blame/38d4d0e11e22fdbc4acf373d2421d25abeb43439/src/datasets/arrow_dataset.py#L5925C10-L5925C10

The example seems to suggest that dataset.add_column() can add column inplace, however, this is wrong -- it cannot. It returns a new dataset with the column added to it.
i cant confirm email
State: open | Author: lykamspam (NONE)

### Describe the bug

This is dificult, I cant confirm email because I'm not get any email!
I cant post forum because I cant confirm email!
I can send help desk because... no exist on web page.

paragraph 44

### Steps to reproduce the bug

rthjrtrt

### Expected behavior

ewtgfwetgf

### Environment info

sdgfswdegfwe
Update `_dill.py` to use `co_linetable` for Python 3.10+ in place of `co_lnotab`
State: open | Author: qgallouedec (MEMBER)

Not 100% about this one, but it seems to be recommended.

```
/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/datasets/utils/_dill.py:385: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
```
Tests pass locally. And the warning is gone with this change.

https://peps.python.org/pep-0626/#backwards-compatibility

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7609). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

not 100% sure either, I tried removing unnecessary checks - let me know if they sound good to you otherwise I'll revert

I can't reproduce the warning anymore... 🤦🏻‍♂️
Tests typing and fixes for push_to_hub
State: closed | Author: lhoestq (MEMBER)

todo:
- [x] fix TestPushToHub.test_push_dataset_dict_to_hub_iterable_num_proc

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7608). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add `num_proc=` to `.push_to_hub()` (Dataset and IterableDataset)
State: closed | Author: lhoestq (MEMBER)


Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7606). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Make `push_to_hub` atomic (#7600)
State: open | Author: sharvil (NONE)


Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7605). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi ! unfortunately we can't allow atomic commits for commits with hundreds of files additions (HF would time out)

Maybe an alternative would be to retry if there was a commit in between ? this could be the default behavior as well

Thanks for taking a look – much appreciated!

I've verified that commits with up to 20,000 files don't time out and the commit time scales linearly with the number of operations enqueued. It took just under 2 minutes to complete (successfully) the 20k file commit.

The fundamental issue I'm trying to tackle here is dataset corruption: getting into a state where a dataset on the hub cannot be used when downloaded. Non-atomic commits won't get us there, I think. If, for example, 3 of 5 commits complete and the machine/process calling `push_to_hub` has a network, hardware, or other failure that prevents it from completing the rest of the commits (even with retries) we'll now have some pointer files pointing to the new data and others pointing to the old data => corrupted. While this may seem like an unlikely scenario, it's a regular occurrence at scale.

If you still feel strongly that atomic commits are not the right way to go, I can either set it to not be the default or remove it entirely from this PR.

As for retries, it's a good idea. In a non-atomic world, the logic gets more complicated:
- keep an explicit queue of pending add/delete operations
- chunkwise pop from queue and commit with `parent_commit` set to previous chunked commit hash
- if `create_commit` fails:
  - re-fetch README and set `parent_commit` to latest hash for `revision`
  - re-generate dataset card content
  - swap old `CommitOperationAdd` with new one for README in the pending queue
- resume chunkwise committing from the queue as above

Entirely doable, but more involved than I signed up for with this PR.

Just to clarify – setting the `parent_commit` can be separated from making the commit atomic (which is what I'm suggesting by either atomic commits not the default or removing it from this PR). It's crucial to set the parent commit to avoid the read-modify-write race condition on the README schema.
Docs and more methods for IterableDataset: push_to_hub, to_parquet...
State: closed | Author: lhoestq (MEMBER)

to_csv, to_json, to_sql, to_pandas, to_polars, to_dict, to_list

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7604). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Enhance error handling and input validation across multiple modules
State: open | Author: mohiuddin-khan-shiam (NONE)

This PR improves the robustness and user experience by:

1. **Audio Module**:
   - Added clear error messages when required fields ('path' or 'bytes') are missing in audio encoding

2. **DatasetDict**:
   - Enhanced key access error messages to show available splits when an invalid key is accessed

3. **NonMutableDict**:
   - Added input validation for the update() method to ensure proper mapping types

4. **Arrow Reader**:
   - Improved error messages for small dataset percentage splits with suggestions for alternatives

5. **FaissIndex**:
   - Strengthened input validation with descriptive error messages
   - Added proper type checking and shape validation for search queries

These changes make the code more maintainable and user-friendly by providing actionable feedback when issues arise.
`push_to_hub` is not concurrency safe (dataset schema corruption)
State: open | Author: sharvil (NONE)

### Describe the bug

Concurrent processes modifying and pushing a dataset can overwrite each others' dataset card, leaving the dataset unusable.

Consider this scenario:
- we have an Arrow dataset
- there are `N` configs of the dataset
- there are `N` independent processes operating on each of the individual configs (e.g. adding a column, `new_col`)
- each process calls `push_to_hub` on their particular config when they're done processing
- all calls to `push_to_hub` succeed
- the `README.md` now has some configs with `new_col` added and some with `new_col` missing

Any attempt to load a config (using `load_dataset`) where `new_col` is missing will fail because of a schema mismatch between `README.md` and the Arrow files. Fixing the dataset requires updating `README.md` by hand with the correct schema for the affected config. In effect, `push_to_hub` is doing a `git push --force` (I found this behavior quite surprising).

We have hit this issue every time we run processing jobs over our datasets and have to fix corrupted schemas by hand.

Reading through the code, it seems that specifying a [`parent_commit`](https://github.com/huggingface/huggingface_hub/blob/v0.32.4/src/huggingface_hub/hf_api.py#L4587) hash around here https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L5794 would get us to a normal, non-forced git push, and avoid schema corruption. I'm not familiar enough with the code to know how to determine the commit hash from which the in-memory dataset card was loaded.

### Steps to reproduce the bug

See above.

### Expected behavior

Concurrent edits to disjoint configs of a dataset should never corrupt the dataset schema.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.15.0-118-generic-x86_64-with-glibc2.35
- Python version: 3.10.14
- `huggingface_hub` version: 0.30.2
- PyArrow version: 19.0.1
- Pandas version: 2.2.2
- `fsspec` version: 2023.9.0

Comments: @lhoestq can you please take a look? I've submitted a PR that fixes this issue. Thanks.

Thanks for the ping ! As I said in https://github.com/huggingface/datasets/pull/7605 there is maybe a more general approach using retries :)
My already working dataset (when uploaded few months ago) now is ignoring metadata.jsonl
State: open | Author: JuanCarlosMartinezSevilla (NONE)

### Describe the bug

Hi everyone, I uploaded my dataset https://huggingface.co/datasets/PRAIG/SMB a few months ago while I was waiting for a conference acceptance response. Without modifying anything in the dataset repository now the Dataset viewer is not rendering the metadata.jsonl annotations, neither it is being downloaded when using load_dataset. Can you please help? Thank you in advance.

### Steps to reproduce the bug

from datasets import load_dataset
ds = load_dataset("PRAIG/SMB")
ds = ds["train"]

### Expected behavior

 It is expected to have all the metadata available in the jsonl file. Fields like: "score_id", "original_width", "original_height", "regions"... among others.

### Environment info

datasets==3.6.0, python 3.13.3 (but he problem is already in the huggingface dataset page)

Comments: Maybe its been a recent update, but i can manage to load the metadata.jsonl separately from the images with:

```
metadata = load_dataset("PRAIG/SMB", split="train", data_files=["*.jsonl"])
images = load_dataset("PRAIG/SMB", split="train")
```
Do you know it this is an expected behaviour? This makes my dataset viewer to only load the images without the labeling of metadata.jsonl.

Thanks

Hi ! this is because we now expect the metadata file to be inside the directory named after the split "train" (this way each split can have its own metadata and can be loaded independently)

You can fix that by configuring it explicitly in the dataset's README.md header:

```yaml
configs:
- config_name: default
  data_files:
  - split: train
    path:
    - "train/**/*.png"
    - "metadata.jsonl"
```

(or by moving the metadata.jsonl in train/ but in this case you also have to modify the content of the JSONL to fix the relative paths to the images)
Download datasets from a private hub in 2025
State: open | Author: DanielSchuhmacher (NONE)
Labels: name: enhancement

### Feature request

In the context of a private hub deployment, customers would like to use load_dataset() to load datasets from their hub, not from the public hub. This doesn't seem to be configurable at the moment and it would be nice to add this feature.

The obvious workaround is to clone the repo first and then load it from local storage, but this adds an extra step. It'd be great to have the same experience regardless of where the hub is hosted.

This issue was raised before here: https://github.com/huggingface/datasets/issues/3679
@juliensimon

### Motivation

none

### Your contribution

none

Comments: Hi ! First, and in the general case, Hugging Face does offer to host private datasets, and with a subscription you can even choose the region in which the repositories are hosted (US, EU)

Then if you happen to have a private deployment, you can set the HF_ENDPOINT environment variable (same as in https://github.com/huggingface/transformers/issues/38634)

Thank you @lhoestq. Works as described!
Add `IterableDataset.push_to_hub()`
State: closed | Author: lhoestq (MEMBER)

Basic implementation, which writes one shard per input dataset shard.
This is to be improved later.

Close https://github.com/huggingface/datasets/issues/5665

PS: for image/audio datasets structured as actual image/audio files (not parquet), you can sometimes speed it up with `ds.decode(num_threads=...).push_to_hub(...)`

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7595). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add option to ignore keys/columns when loading a dataset from jsonl(or any other data format)
State: open | Author: avishaiElmakies (NONE)
Labels: name: enhancement

### Feature request

Hi, I would like the option to ignore keys/columns when loading a dataset from files (e.g. jsonl).

### Motivation

I am working on a dataset which is built on jsonl. It seems the dataset is unclean and a column has different types in each row. I can't clean this or remove the column (It is not my data and it is too big for me to clean and save on my own hardware). 
I would like the option to just ignore this column when using `load_dataset`, since i don't need it.
I tried to look if this is already possible but couldn't find a solution. if there is I would love some help. If it is not currently possible, I would love this feature

### Your contribution

I don't think I can help this time, unfortunately.

Comments: Good point, I'd be in favor of having the `columns` argument in `JsonConfig` (and the others) to align with `ParquetConfig` to let users choose which columns to load and ignore the rest

Is it possible to ignore columns when using parquet? 

Yes, you can pass `columns=...` to load_dataset to select which columns to load, and it is passed to `ParquetConfig` :)

Ok, i didn't know that. 
Anyway, it would be good to add this to others
Fix broken link to albumentations
State: closed | Author: ternaus (CONTRIBUTOR)

A few months back I rewrote all docs at [https://albumentations.ai/docs](https://albumentations.ai/docs), and some pages changed their links.

In this PR fixed link to the most recent doc in Albumentations about bounding boxes and it's format. 

Fix a few typos in the doc as well.

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7593). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq ping
Remove scripts altogether
State: closed | Author: lhoestq (MEMBER)

TODO:
- [x] remplace fixtures based on script with no-script fixtures
- [x] windaube

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7592). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
feat: use content defined chunking 
State: open | Author: kszucs (COLLABORATOR)

WIP:
- [x] set the parameters in `io.parquet.ParquetDatasetReader`
- [x] set the parameters in `arrow_writer.ParquetWriter`

It requires a new pyarrow pin ">=21.0.0" which is not yet released.

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7589). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
ValueError: Invalid pattern: '**' can only be an entire path component [Colab]
State: closed | Author: wkambale (NONE)

### Describe the bug

I have a dataset on HF [here](https://huggingface.co/datasets/kambale/luganda-english-parallel-corpus) that i've previously used to train a translation model [here](https://huggingface.co/kambale/pearl-11m-translate).

now i changed a few hyperparameters to increase number of tokens for the model, increase Transformer layers, and all

however, when i try to load the dataset, this error keeps coming up.. i have tried everything.. i have re-written the code a hundred times, and this keep coming up

### Steps to reproduce the bug

Imports:

```bash
!pip install datasets huggingface_hub fsspec
```

Python code:

```python
from datasets import load_dataset

HF_DATASET_NAME = "kambale/luganda-english-parallel-corpus"

# Load the dataset
try:
    if not HF_DATASET_NAME or HF_DATASET_NAME == "YOUR_HF_DATASET_NAME":
        raise ValueError(
            "Please provide a valid Hugging Face dataset name."
        )

    dataset = load_dataset(HF_DATASET_NAME)

# Omitted code as the error happens on the line above 

except ValueError as ve:
    print(f"Configuration Error: {ve}")
    raise
except Exception as e:
    print(f"An error occurred while loading the dataset '{HF_DATASET_NAME}': {e}")
    raise e
```

now, i have tried going through this [issue](https://github.com/huggingface/datasets/issues/6737) and nothing helps 

### Expected behavior

loading the dataset successfully and perform splits (train, test, validation)

### Environment info

from the imports, i do not install specific versions of these libraries, so the latest or available version is installed

* `datasets` version: latest
* `Platform`: Google Colab
* `Hardware`: NVIDIA A100 GPU
* `Python` version: latest
* `huggingface_hub` version: latest
* `fsspec` version: latest

Comments: Could you please run the following code snippet in your environment and share the exact output? This will help check for any compatibility issues within the env itself. 

```
import datasets
import huggingface_hub
import fsspec

print("datasets version:", datasets.__version__)
print("huggingface_hub version:", huggingface_hub.__version__)
print("fsspec version:", fsspec.__version__)
```

```bash
datasets version: 2.14.4
huggingface_hub version: 0.31.4
fsspec version: 2025.3.2
```

Version 2.14.4 is not the latest version available, in fact it is from August 08, 2023 (you can check here: https://pypi.org/project/datasets/#history)

Use pip install datasets==3.6.0 to install a more recent version (from May 7, 2025)

I also had the same problem with Colab, after updating to the latest version it was solved.

I hope it helps

thank you @CleitonOERocha. it sure did help.

updating `datasets` to v3.6.0 and keeping `fsspec` on v2025.3.2 eliminates the issue.

Very helpful, thank you!
load_dataset splits typing
State: closed | Author: lhoestq (MEMBER)

close https://github.com/huggingface/datasets/issues/7583

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7587). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Avoid multiple default config names
State: closed | Author: albertvillanova (MEMBER)

Fix duplicating default config names.

Currently, when calling `push_to_hub(set_default=True` with 2 different config names, both are set as default. 

Moreover, this will generate an error next time we try to push another default config name, raised by `MetadataConfigs.get_default_config_name`:
https://github.com/huggingface/datasets/blob/da1db8a5b89fc0badaa0f571b36e122e52ae8c61/src/datasets/arrow_dataset.py#L5757
https://github.com/huggingface/datasets/blob/da1db8a5b89fc0badaa0f571b36e122e52ae8c61/src/datasets/utils/metadata.py#L186-L188

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7585). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add LMDB format support
State: open | Author: trotsky1997 (NONE)
Labels: name: enhancement

### Feature request

Add LMDB format support for large memory-mapping files

### Motivation

Add LMDB format support for large memory-mapping files

### Your contribution

I'm trying to add it

Comments: Hi ! Can you explain what's your use case ? Is it about converting LMDB to Dataset objects (i.e. converting to Arrow) ?
load_dataset type stubs reject List[str] for split parameter, but runtime supports it
State: closed | Author: hierr (NONE)

### Describe the bug

The [load_dataset](https://huggingface.co/docs/datasets/v3.6.0/en/package_reference/loading_methods#datasets.load_dataset) method accepts a `List[str]` as the split parameter at runtime, however, the current type stubs restrict the split parameter to `Union[str, Split, None]`. This causes type checkers like Pylance to raise `reportArgumentType` errors when passing a list of strings, even though it works as intended at runtime.

### Steps to reproduce the bug

1. Use load_dataset with multiple splits e.g.:
```
from datasets import load_dataset

ds_train, ds_val, ds_test = load_dataset(
    "Silly-Machine/TuPyE-Dataset",
    "binary",
    split=["train[:75%]", "train[75%:]", "test"]
)
```
2. Observe that code executes correctly at runtime and Pylance raises `Argument of type "List[str]" cannot be assigned to parameter "split" of type "str | Split | None"`

### Expected behavior

The type stubs for [load_dataset](https://huggingface.co/docs/datasets/v3.6.0/en/package_reference/loading_methods#datasets.load_dataset) should accept `Union[str, Split, List[str], None]` or more specific overloads for the split parameter to correctly represent runtime behavior. 

### Environment info

- `datasets` version: 3.6.0
- Platform: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39
- Python version: 3.12.7
- `huggingface_hub` version: 0.32.0
- PyArrow version: 20.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2025.3.0