noamaanMulla-03/datasets-issues huggingface dataset:
Make `push_to_hub` atomic (#7600)



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7605). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi ! unfortunately we can't allow atomic commits for commits with hundreds of files additions (HF would time out)

Maybe an alternative would be to retry if there was a commit in between ? this could be the default behavior as well

Thanks for taking a look – much appreciated!

I've verified that commits with up to 20,000 files don't time out and the commit time scales linearly with the number of operations enqueued. It took just under 2 minutes to complete (successfully) the 20k file commit.

The fundamental issue I'm trying to tackle here is dataset corruption: getting into a state where a dataset on the hub cannot be used when downloaded. Non-atomic commits won't get us there, I think. If, for example, 3 of 5 commits complete and the machine/process calling `push_to_hub` has a network, hardware, or other failure that prevents it from completing the rest of the commits (even with retries) we'll now have some pointer files pointing to the new data and others pointing to the old data => corrupted. While this may seem like an unlikely scenario, it's a regular occurrence at scale.

If you still feel strongly that atomic commits are not the right way to go, I can either set it to not be the default or remove it entirely from this PR.

As for retries, it's a good idea. In a non-atomic world, the logic gets more complicated:
- keep an explicit queue of pending add/delete operations
- chunkwise pop from queue and commit with `parent_commit` set to previous chunked commit hash
- if `create_commit` fails:
  - re-fetch README and set `parent_commit` to latest hash for `revision`
  - re-generate dataset card content
  - swap old `CommitOperationAdd` with new one for README in the pending queue
- resume chunkwise committing from the queue as above

Entirely doable, but more involved than I signed up for with this PR.

Just to clarify – setting the `parent_commit` can be separated from making the commit atomic (which is what I'm suggesting by either atomic commits not the default or removing it from this PR). It's crucial to set the parent commit to avoid the read-modify-write race condition on the README schema.
Fix: Preserve float columns in JSON loader when values are integer-like (e.g. 0.0, 1.0)

This PR fixes a bug in the JSON loader where columns containing float values like `[0.0, 1.0, 2.0]` were being implicitly coerced to `int`, due to pandas or Arrow type inference.

This caused issues downstream in statistics computation (e.g., dataset-viewer) where such columns were incorrectly labeled as `"int"` instead of `"float"`.

### 🔍 What was happening:
When the JSON loader falls back to `pandas_read_json()` (after `pa.read_json()` fails), pandas/Arrow can coerce float values to integers if all values are integer-like (e.g., `0.0 == 0`).

### ✅ What this PR does:
- Adds a check in the fallback path of `_generate_tables()`
- Ensures that columns made entirely of floats are preserved as `"float64"` even if they are integer-like (e.g. `0.0`, `1.0`)
- This prevents loss of float semantics when creating the Arrow table

### 🧪 Reproducible Example:
```json
[{"col": 0.0}, {"col": 1.0}, {"col": 2.0}]
````

Previously loaded as:

* `int`

Now correctly loaded as:

* `float`

Fixes #6937
Replace Sequence by List

Sequence is just a utility that we need to keep for backward compatibility. And `[ ]` was used instead but doesn't allow passing the length of the list.

This PR removes most mentions of Sequence and usage of `[ ]` and defines a proper List type instead.

before: `Sequence(Value("int64"))` or `[Value("int64")]`
now: `List(Value("int64"))`

This PR conserves full backward compatibility. And it's a good occasion with the release of 4.0.0


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7634). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
feat: Add h5folder dataset loader for HDF5 support

### Related Issue
Closes #3113

### What does this PR do?

This PR introduces a new dataset loader module called **`h5folder`** to support loading datasets stored in **HDF5 (.h5)** format.

It allows users to do:

```python
from datasets import load_dataset
dataset = load_dataset("h5folder", data_dir="path/to/")
````

### 🧩 Design Overview

* Implemented inside `datasets/packaged_modules/h5folder/h5folder.py`
* Based on the `GeneratorBasedBuilder` API
* Uses `h5py` to read HDF5 files and yield examples
* Expects datasets such as `id`, `data`, and `label` inside `data.h5`
* Converts numpy arrays to Python types before yielding

### 🧪 Example `.h5` Structure (for local testing)

```python
import h5py
import numpy as np

with h5py.File("data.h5", "w") as f:
    f.create_dataset("id", data=np.arange(100))
    f.create_dataset("data", data=np.random.randn(100, 10))
    f.create_dataset("label", data=np.random.randint(0, 2, size=100))
```

### ✅ Testing

- The loader logic follows the structure of existing modules like `imagefolder`
- Will rely on Hugging Face CI to validate integration
- Manually testing planned once merged or during feedback

### 📁 Files Added

* `datasets/src/datasets/packaged_modules/h5folder/h5folder.py`

### 📌 Component(s) Affected

* `area/datasets`
* `area/load`

### 📦 Release Note Classification

* `rn/feature` – Adds support for loading `.h5` datasets via `load_dataset("h5folder", ...)`

---

Let me know if any changes or improvements are needed — happy to iterate. Thanks for reviewing!



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7625). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I guess test failed cause import os, import h5py, and import datasets lines are not alphabetically sorted, or not grouped properly.

![image](https://github.com/user-attachments/assets/ab73f8f9-da50-4ba8-9b2d-b7c30fce94f5)


This commit was accidental - `[Merge branch 'main' into patch-4]`. The 
`[chore: fix import order in h5folder.py to satisfy linter]` should solve the import order issue.
One subset per file in repo ?

Right now we consider all the files of a dataset to be the same data, e.g.
```
single_subset_dataset/
├── train0.jsonl
├── train1.jsonl
└── train2.jsonl
```
but in cases like this, each file is actually a different subset of the dataset and should be loaded separately
```
many_subsets_dataset/
├── animals.jsonl
├── trees.jsonl
└── metadata.jsonl
```

It would be nice to detect those subsets automatically using a simple heuristic. For example we can group files together if their paths names are the same except some digits ?


Hi @lhoestq! I’ve opened a PR that addresses this issue
OverflowError: value too large to convert to int32_t 

### Describe the bug

![image](https://github.com/huggingface/datasets/assets/47747764/f58044fb-ddda-48b6-ba68-7bbfef781630)


### Steps to reproduce the bug

just loading datasets 

### Expected behavior

how can I fix it

### Environment info

pip install /mnt/cluster/zhangfan/study_info/LLaMA-Factory/peft-0.6.0-py3-none-any.whl
pip install huggingface_hub-0.19.4-py3-none-any.whl tokenizers-0.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl transformers-4.36.1-py3-none-any.whl pyarrow_hotfix-0.6-py3-none-any.whl datasets-2.15.0-py3-none-any.whl tyro-0.5.18-py3-none-any.whl trl-0.7.4-py3-none-any.whl

done


+1
`load_dataset` defaults to json file format for datasets with 1 shard

### Describe the bug

I currently have multiple datasets (train+validation) saved as 50MB shards. For one dataset the validation pair is small enough to fit into a single shard and this apparently causes problems when loading the dataset. I created the datasets using a DatasetDict, saved them as 50MB arrow files for streaming and then load each dataset. I have no problem loading any of the other datasets with more than 1 arrow file/shard. 

The error indicates the training set got loaded in arrow format (correct) and the validation set in json (incorrect). This seems to be because some of the metadata files are considered as dataset files. 
```
Error loading /nfs/dataset_pt-uk: Couldn't infer the same data file format for all splits. Got {NamedSplit('train'): ('arrow', {}), NamedSplit('validation'): ('json', {})} 
```

![Image](https://github.com/user-attachments/assets/f6e7596a-dd53-46a9-9a23-4e9cac2ac049)

Concretely, there is a mismatch between the metadata created by the `DatasetDict.save_to_file` and the builder for `datasets.load_dataset`:

https://github.com/huggingface/datasets/blob/e71b0b19d79c7531f9b9bea7c09916b5f6157f42/src/datasets/data_files.py#L107

The `folder_based_builder` lists all files and with 1 arrow file the json files (that are actually metadata) are in the majority.
https://github.com/huggingface/datasets/blob/e71b0b19d79c7531f9b9bea7c09916b5f6157f42/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L58

### Steps to reproduce the bug

Create a dataset with metadata and 1 arrow file in validation and multiple arrow files in the training set, following the above description. In my case, I saved the files via:

```python
        dataset = DatasetDict({
            'train': train_dataset,
            'validation': val_dataset
        })
        
        dataset.save_to_disk(output_path, max_shard_size="50MB")
```

### Expected behavior

The dataset would get loaded.

### Environment info

- `datasets` version: 3.6.0
- Platform: Linux-6.14.0-22-generic-x86_64-with-glibc2.41
- Python version: 3.12.7
- `huggingface_hub` version: 0.31.1
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1
i cant confirm email

### Describe the bug

This is dificult, I cant confirm email because I'm not get any email!
I cant post forum because I cant confirm email!
I can send help desk because... no exist on web page.

paragraph 44

### Steps to reproduce the bug

rthjrtrt

### Expected behavior

ewtgfwetgf

### Environment info

sdgfswdegfwe


Will you please clarify the issue by some screenshots or more in-depth explanation?

![Image](https://github.com/user-attachments/assets/ebe58239-72ef-43f6-a849-35736878fbf3)
This is clarify answer. I have not received a letter.

**The graphic at the top shows how I don't get any letter. Can you show in a clear way how you don't get a letter from me?**
Enable splits during streaming the dataset

## Describe the Problem

I'd like to stream only a specific percentage or part of the dataset. 
I want to do splitting when I'm streaming dataset as well.

## Solution
Enabling splits when `streaming = True` as well.
`e.g. dataset = load_dataset('dataset', split='train[:100]', streaming = True)`

## Alternatives
Below is the alternative of doing it.
`dataset = load_dataset("dataset", split='train', streaming = True).take(100)`



Labels: name: enhancement

For the range splits by percentage over streaming datasets, I used a simple approach [here](https://github.com/huggingface/transformers/pull/39286/files#diff-fb91934a0658db99f30d8c52d92f41d6ab83210134e7f21af8baac6ee65f548fR228) which can be reused to provide things like `[:25%]`, built-in and internally within `datasets`. 

However, it only works for integer percentages like `10%`, `15%`, etc but not for floating point ranges like `10.5%` or `15.1%`. I want to believe that's fine, because streaming datasets are normally large, thus very rarely one might need to split with floating point precision. There seem to be solutions for this, but they may add a lot of unnecessary complications.

Another caveat is that it does not split sequentially, I mean a split of `[:10%]` for example, does not mean that we will get the "first" 10% of the dataset rows! Instead, it gives us every first 10 rows of each bunch of 100 rows as we iterate over the dataset. A short look into the function linked above gives you the idea. 

Additionally, there might be other properties and behaviours in range splits over non-streaming datasets that won't work in the above proposed approach. Nevertheless, we can document these limitations and still provide it, as having some sort of range split is better than not having it at all! 🤗
Code example for dataset.add_column() does not reflect correct way to use function

https://github.com/huggingface/datasets/blame/38d4d0e11e22fdbc4acf373d2421d25abeb43439/src/datasets/arrow_dataset.py#L5925C10-L5925C10

The example seems to suggest that dataset.add_column() can add column inplace, however, this is wrong -- it cannot. It returns a new dataset with the column added to it.


Hi @shaily99 

Thanks for pointing this out — you're absolutely right!

The current example in the docstring for add_column() implies in-place modification, which is misleading since add_column() actually returns a new dataset.

#self-assign
feat: add subset_name as alias for name in load_dataset

fixes #7637

This PR introduces subset_name as a user-facing alias for the name (previously `config_name`) argument in load_dataset. It aligns terminology with the Hugging Face Hub UI (which shows “Subset”), reducing confusion for new users.

Supports `subset_name` in `load_dataset()`

Adds `.subset_name` property to DatasetBuilder

Maintains full backward compatibility

Raises clear error if name and `subset_name` conflict
Auto-merge option for `convert-to-parquet`

### Feature request

Add a command-line option, e.g. `--auto-merge-pull-request` that enables automatic merging of the commits created by the `convert-to-parquet` tool.

### Motivation

Large datasets may result in dozens of PRs due to the splitting mechanism. Each of these has to be manually accepted via the website.

### Your contribution

Happy to look into submitting a PR if this is of interest to maintainers.

Labels: name: enhancement

Alternatively, there could be an option to switch from submitting PRs to just committing changes directly to `main`.

Why not, I'd be in favor of `--merge-pull-request` to call `HfApi().merge_pull_request()` at the end of the conversion :) feel free to open a PR if you'd like

#self-assign

Closing since convert to parquet has been removed... https://github.com/huggingface/datasets/pull/7592#issuecomment-3073053138
Add `--merge-pull-request` option for `convert_to_parquet`

Closes #7527 

Note that this implementation **will only merge the last PR in the case that they get split up by `push_to_hub`**. See https://github.com/huggingface/datasets/discussions/7555 for more details.


This is ready for a review, happy to make any changes. The main question for maintainers is how this should interact with #7555. If my suggestion there is accepted, this PR can be kept as is. If not, more changes are required to merge all the PR parts.

Closing since convert to parquet has been removed... https://github.com/huggingface/datasets/pull/7592#issuecomment-3073053138
No Samsum dataset

### Describe the bug

https://huggingface.co/datasets/Samsung/samsum dataset not found error 404

Originated from https://github.com/meta-llama/llama-cookbook/issues/948

### Steps to reproduce the bug

go to website https://huggingface.co/datasets/Samsung/samsum
see the error

also downloading it with python throws 
```
Couldn't find 'Samsung/samsum' on the Hugging Face Hub either: FileNotFoundError: Samsung/samsum@f00baf5a7d4abfec6820415493bcb52c587788e6/samsum.py (repository not found)
```

### Expected behavior

Dataset exists

### Environment info

```
- `datasets` version: 3.2.0
- Platform: macOS-15.4.1-arm64-arm-64bit
- Python version: 3.12.2
- `huggingface_hub` version: 0.26.5
- PyArrow version: 16.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
```


According to the following https://huggingface.co/posts/seawolf2357/424129432408590, as of now the dataset seems to be inaccessible.

@IgorKasianenko, would https://huggingface.co/datasets/knkarthick/samsum suffice for your purpose?


Thanks @SP1029 for the update!
That will work for now, using it as replacement. Is there a officially recommended way to maintain the CC licensed dataset under the organization account? 
Feel free to close this issue

> Is there an officially recommended way to maintain a CC-licensed dataset under an organizational account?

@IgorKasianenko, apologies, this is not my area of expertise.

> Please feel free to close this issue.

I have limited access and may not be able to do that. Since you opened it, you would be able to close it.

dataset_samsum = load_dataset("knkarthick/samsum")

is working
Error type: ArrowInvalid Details: Failed to parse string: '[254,254]' as a scalar of type int32

### Describe the bug

I am getting the following error while I am trying to upload the CSV sheet to train a model. My CSV sheet content is exactly same as shown in the example CSV file in the Auto Train page. Attaching screenshot of error for reference. I have also tried converting the index of the answer that are integer into string by placing inverted commas and also without inverted commas. 
Can anyone please help me out? 
FYI : I am using Chrome browser.

Error type: ArrowInvalid
Details: Failed to parse string: '[254,254]' as a scalar of type int32

![Screenshot 2023-08-19 165827](https://github.com/huggingface/datasets/assets/90616801/95fad96e-7dce-4bb5-9f83-9f1659a32891)


### Steps to reproduce the bug

Kindly let me know how to fix this?

### Expected behavior

Kindly let me know how to fix this?

### Environment info

Kindly let me know how to fix this?


Answered on the forum [here](https://discuss.huggingface.co/t/error-type-arrowinvalid-details-failed-to-parse-string-254-254-as-a-scalar-of-type-int32/51323).

it seems that no answer in this link
NotImplementedError when using streaming=True in Google Colab environment

### Describe the bug

When attempting to load a large dataset (like tiiuae/falcon-refinedweb or allenai/c4) using streaming=True in a standard Google Colab notebook, the process fails with a NotImplementedError: Loading a streaming dataset cached in a LocalFileSystem is not supported yet. This issue persists even after upgrading datasets and huggingface_hub and restarting the session.

### Steps to reproduce the bug

Open a new Google Colab notebook.

(Optional but recommended) Run !pip install --upgrade datasets huggingface_hub and restart the runtime.

Run the following code:

Python

from datasets import load_dataset

try:
    print("Attempting to load a stream...")
    streaming_dataset = load_dataset('tiiuae/falcon-refinedweb', streaming=True)
    print("Success!")
except Exception as e:
    print(e)

### Expected behavior

The load_dataset command should return a StreamingDataset object without raising an error, allowing iteration over the dataset.

Actual Behavior
The code fails and prints the following error traceback:

[PASTE THE FULL ERROR TRACEBACK HERE]
(Note: Copy the entire error message you received, from Traceback... to the final error line, and paste it in this section.)

### Environment info

Platform: Google Colab

datasets version: [Run !pip show datasets in Colab and paste the version here]

huggingface_hub version: [Run !pip show huggingface_hub and paste the version here]

Python version: [Run !python --version and paste the version here]


Hi, @Aniket17200, try upgrading datasets using '!pip install -U datasets'. I hope this will resolve your issue.

Thank you @tanuj-rai, it's working great
Allow dataset row indexing with np.int types (#7423)

@lhoestq 
Proposed fix for #7423. Added a couple simple tests as requested. I had some test failures related to Java and pyspark even when installing with dev but these don't seem to be related to the changes here and fail for me even on clean main.

The typeerror raised when using the wrong type is: "Wrong key type: '{key}' of type '{type(key)}'. Expected one of int, slice, range, str or Iterable." I think that is fine. But I could modify the int part to something more generic (although I'm not sure what) if wanted.


+1

@lhoestq can you take a look at this?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7438). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you
Support all negative values in ClassLabel

We usually use -1 to represent a missing label, but we should also support any negative values (some users use -100 for example). This is a regression from `datasets` 2.3

Fix https://github.com/huggingface/datasets/issues/4508


_The documentation is not available anymore as the PR was closed or merged._

Thanks for this fix! I'm not sure what the release timeline is, but FYI #4508 is a breaking issue for transformer token classification using Trainer and PyTorch. PyTorch defaults to -100 as the ignored label for [negative log loss](https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html?highlight=nllloss#torch.nn.NLLLoss), so switching labels to -1 leads to index errors using Trainer defaults.

As a workaround, I'm using master branch directly (`pip install git+https://github.com/huggingface/datasets.git@master` for anyone who needs to do the same) until this gets released.

The new release `2.4` fixes the issue, feel free to update `datasets` :) 
```
pip install -U datasets
```

@lhoestq I hope it's OK to ping you here. I've noticed that `encode_example` does only work with -1. I already created #7645 to fix the documentation, but then I stumbled across your original changes to the docs text in this PR.

I am talking about this part in `ClassLabel -> encode_example`:

https://github.com/huggingface/datasets/blob/e71b0b19d79c7531f9b9bea7c09916b5f6157f42/src/datasets/features/features.py#L1126-L1129
xopen: invalid start byte for streaming dataset with trust_remote_code=True

### Describe the bug

I am trying to load YODAS2 dataset with datasets==3.6.0
```
from datasets import load_dataset
next(iter(load_dataset('espnet/yodas2', name='ru000', split='train', streaming=True, trust_remote_code=True)))
```
And get `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 1: invalid start byte`

The cause of the error is the following:

```
from datasets.utils.file_utils import xopen

filepath = 'https://huggingface.co/datasets/espnet/yodas2/resolve/c9674490249665d658f527e2684848377108d82c/data/ru000/text/00000000.json'
xopen(filepath, 'r').read()
>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 1: invalid start byte
```

And the cause of this is the following:

```
import fsspec

fsspec.open(
    'hf://datasets/espnet/yodas2@c9674490249665d658f527e2684848377108d82c/data/ru000/text/00000000.json',
    mode='r',
    hf={'token': None, 'endpoint': 'https://huggingface.co'},
).open().read()
>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 1: invalid start byte
```

Is it true that streaming=True loading is not supported anymore for trust_remote_code=True, even with datasets==3.6.0? This breaks backward compatibility.

### Steps to reproduce the bug

```
from datasets import load_dataset
next(iter(load_dataset('espnet/yodas2', name='ru000', split='train', streaming=True)))
```

### Expected behavior

No errors expected

### Environment info

datasets==3.6.0, ubuntu 24.04


Hi ! it would be cool to convert this dataset to Parquet. This will make it work for `datasets>=4.0`, enable the Dataset Viewer and make it more reliable to load/stream (currently it uses a loading script in python and those are known for having issues sometimes)

using `datasets==3.6.0`, here is the command to convert it and open a Pull Request:

```
datasets-cli convert_to_parquet espnet/yodas2 --trust_remote_code
```

Though it's likely that the `UnicodeDecodeError` comes from the loading script. If the script has a bug, it must be fixed to be able to convert the dataset without errors