konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
fix: raise error when folder-based datasets are loaded without data_dir or data_files


### Related Issues/PRs

<!-- Uncomment 'Resolve' if this PR can close the linked items. -->
<!-- Resolve --> #6152

---

### What changes are proposed in this pull request?

This PR adds an early validation step for folder-based datasets (like `audiofolder`) to prevent silent fallback behavior.

**Before this fix**:
- When `data_dir` or `data_files` were not provided, the loader defaulted to the current working directory.
- This caused unexpected behavior like:
  - Long loading times
  - Scanning unintended local files

**Now**:
- If both `data_dir` and `data_files` are missing, a `ValueError` is raised early with a helpful message.

---

### How is this PR tested?

- [x] Manual test via `load_dataset("audiofolder")` with missing `data_dir`
- [ ] Existing unit tests (should not break any)
- [ ] New tests (if needed, maintainers can guide)

---

### Does this PR require documentation update?

- [x] No. You can skip the rest of this section.

---

### Release Notes

#### Is this a user-facing change?

- [x] Yes. Give a description of this change to be included in the release notes for users.

> Adds early error handling for folder-based datasets when neither `data_dir` nor `data_files` is specified, avoiding unintended resolution to the current directory.

#### What component(s), interfaces, languages, and integrations does this PR affect?

Components:
- [x] `area/datasets`
- [x] `area/load`

---

<a name="release-note-category"></a>

#### How should the PR be classified in the release notes? Choose one:

- [x] `rn/bug-fix` - A user-facing bug fix worth mentioning in the release notes

---

#### Should this PR be included in the next patch release?

- [x] Yes (this PR will be cherry-picked and included in the next patch release)


Great ! Since this logic is specific to one builder class maybe this check can be in the class definition ? I think you can put it in FolderBasedBuilder's `_info()` method.
Unwanted column padding in nested lists of dicts

```python
from datasets import Dataset

dataset = Dataset.from_dict({
    "messages": [
        [
            {"a": "...",},
            {"b": "...",},
        ],
    ]
})
print(dataset[0])
```

What I get:
```
{'messages': [{'a': '...', 'b': None}, {'a': None, 'b': '...'}]}
```

What I want:

```
{'messages': [{'a': '...'}, {'b': '...'}]}
```

Is there an easy way to automatically remove these auto-filled null/none values?

If not, I probably need a recursive none exclusion function, don't I?

Datasets 3.6.0

Answer from @lhoestq:

> No
> This is because Arrow and Parquet a columnar format: they require a fixed type for each column. So if you have nested dicts, each item should have the same subfields

The way around I found is the handle it after sampling with this function:

```python
def remove_padding(example):
    if isinstance(example, list):
        return [remove_padding(value) if isinstance(value, (dict, list)) else value for value in example]
    elif isinstance(example, Mapping):
        return {
            key: remove_padding(value) if isinstance(value, (dict, list)) else value
            for key, value in example.items()
            if value is not None
        }
    else:
        raise TypeError("Input must be a list or a dictionary.")

# Example:
example = next(iter(dataset))
example = remove_padding(example)
```
Lazy column

Same as https://github.com/huggingface/datasets/pull/7564 but for `Dataset`, cc @TopCoder2K FYI

e.g. `ds[col]` now returns a lazy Column instead of a list

This way calling `ds[col][idx]` only loads the required data in memory

(bonus: also supports subfields access with `ds[col][subcol][idx]`)

the breaking change will be for the next major release, which also includes removal of dataset scripts support

close https://github.com/huggingface/datasets/issues/4180

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7614). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Code example for dataset.add_column() does not reflect correct way to use function

https://github.com/huggingface/datasets/blame/38d4d0e11e22fdbc4acf373d2421d25abeb43439/src/datasets/arrow_dataset.py#L5925C10-L5925C10

The example seems to suggest that dataset.add_column() can add column inplace, however, this is wrong -- it cannot. It returns a new dataset with the column added to it.

Hi @shaily99 

Thanks for pointing this out — you're absolutely right!

The current example in the docstring for add_column() implies in-place modification, which is misleading since add_column() actually returns a new dataset.

#self-assign
i cant confirm email

### Describe the bug

This is dificult, I cant confirm email because I'm not get any email!
I cant post forum because I cant confirm email!
I can send help desk because... no exist on web page.

paragraph 44

### Steps to reproduce the bug

rthjrtrt

### Expected behavior

ewtgfwetgf

### Environment info

sdgfswdegfwe

Will you please clarify the issue by some screenshots or more in-depth explanation?

![Image](https://github.com/user-attachments/assets/ebe58239-72ef-43f6-a849-35736878fbf3)
This is clarify answer. I have not received a letter.

**The graphic at the top shows how I don't get any letter. Can you show in a clear way how you don't get a letter from me?**
Update `_dill.py` to use `co_linetable` for Python 3.10+ in place of `co_lnotab`

Not 100% about this one, but it seems to be recommended.

```
/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/datasets/utils/_dill.py:385: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
```
Tests pass locally. And the warning is gone with this change.

https://peps.python.org/pep-0626/#backwards-compatibility

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7609). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

not 100% sure either, I tried removing unnecessary checks - let me know if they sound good to you otherwise I'll revert

I can't reproduce the warning anymore... 🤦🏻‍♂️


Ah now I can reproduce!, and I can confirm that the warning is gone when you apply the change in this PR
Tests typing and fixes for push_to_hub

todo:
- [x] fix TestPushToHub.test_push_dataset_dict_to_hub_iterable_num_proc

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7608). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Make `push_to_hub` atomic (#7600)


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7605). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi ! unfortunately we can't allow atomic commits for commits with hundreds of files additions (HF would time out)

Maybe an alternative would be to retry if there was a commit in between ? this could be the default behavior as well

Thanks for taking a look – much appreciated!

I've verified that commits with up to 20,000 files don't time out and the commit time scales linearly with the number of operations enqueued. It took just under 2 minutes to complete (successfully) the 20k file commit.

The fundamental issue I'm trying to tackle here is dataset corruption: getting into a state where a dataset on the hub cannot be used when downloaded. Non-atomic commits won't get us there, I think. If, for example, 3 of 5 commits complete and the machine/process calling `push_to_hub` has a network, hardware, or other failure that prevents it from completing the rest of the commits (even with retries) we'll now have some pointer files pointing to the new data and others pointing to the old data => corrupted. While this may seem like an unlikely scenario, it's a regular occurrence at scale.

If you still feel strongly that atomic commits are not the right way to go, I can either set it to not be the default or remove it entirely from this PR.

As for retries, it's a good idea. In a non-atomic world, the logic gets more complicated:
- keep an explicit queue of pending add/delete operations
- chunkwise pop from queue and commit with `parent_commit` set to previous chunked commit hash
- if `create_commit` fails:
  - re-fetch README and set `parent_commit` to latest hash for `revision`
  - re-generate dataset card content
  - swap old `CommitOperationAdd` with new one for README in the pending queue
- resume chunkwise committing from the queue as above

Entirely doable, but more involved than I signed up for with this PR.

Just to clarify – setting the `parent_commit` can be separated from making the commit atomic (which is what I'm suggesting by either atomic commits not the default or removing it from this PR). It's crucial to set the parent commit to avoid the read-modify-write race condition on the README schema.
Docs and more methods for IterableDataset: push_to_hub, to_parquet...

to_csv, to_json, to_sql, to_pandas, to_polars, to_dict, to_list

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7604). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`push_to_hub` is not concurrency safe (dataset schema corruption)

### Describe the bug

Concurrent processes modifying and pushing a dataset can overwrite each others' dataset card, leaving the dataset unusable.

Consider this scenario:
- we have an Arrow dataset
- there are `N` configs of the dataset
- there are `N` independent processes operating on each of the individual configs (e.g. adding a column, `new_col`)
- each process calls `push_to_hub` on their particular config when they're done processing
- all calls to `push_to_hub` succeed
- the `README.md` now has some configs with `new_col` added and some with `new_col` missing

Any attempt to load a config (using `load_dataset`) where `new_col` is missing will fail because of a schema mismatch between `README.md` and the Arrow files. Fixing the dataset requires updating `README.md` by hand with the correct schema for the affected config. In effect, `push_to_hub` is doing a `git push --force` (I found this behavior quite surprising).

We have hit this issue every time we run processing jobs over our datasets and have to fix corrupted schemas by hand.

Reading through the code, it seems that specifying a [`parent_commit`](https://github.com/huggingface/huggingface_hub/blob/v0.32.4/src/huggingface_hub/hf_api.py#L4587) hash around here https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L5794 would get us to a normal, non-forced git push, and avoid schema corruption. I'm not familiar enough with the code to know how to determine the commit hash from which the in-memory dataset card was loaded.

### Steps to reproduce the bug

See above.

### Expected behavior

Concurrent edits to disjoint configs of a dataset should never corrupt the dataset schema.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.15.0-118-generic-x86_64-with-glibc2.35
- Python version: 3.10.14
- `huggingface_hub` version: 0.30.2
- PyArrow version: 19.0.1
- Pandas version: 2.2.2
- `fsspec` version: 2023.9.0

@lhoestq can you please take a look? I've submitted a PR that fixes this issue. Thanks.

Thanks for the ping ! As I said in https://github.com/huggingface/datasets/pull/7605 there is maybe a more general approach using retries :)

Dropping this due to inactivity; we've implemented push_to_hub outside of HF datasets that's concurrency safe. Feel free to use the code I provided as a starting point if there's still interest in addressing this issue.
My already working dataset (when uploaded few months ago) now is ignoring metadata.jsonl

### Describe the bug

Hi everyone, I uploaded my dataset https://huggingface.co/datasets/PRAIG/SMB a few months ago while I was waiting for a conference acceptance response. Without modifying anything in the dataset repository now the Dataset viewer is not rendering the metadata.jsonl annotations, neither it is being downloaded when using load_dataset. Can you please help? Thank you in advance.

### Steps to reproduce the bug

from datasets import load_dataset
ds = load_dataset("PRAIG/SMB")
ds = ds["train"]

### Expected behavior

 It is expected to have all the metadata available in the jsonl file. Fields like: "score_id", "original_width", "original_height", "regions"... among others.

### Environment info

datasets==3.6.0, python 3.13.3 (but he problem is already in the huggingface dataset page)

Maybe its been a recent update, but i can manage to load the metadata.jsonl separately from the images with:

```
metadata = load_dataset("PRAIG/SMB", split="train", data_files=["*.jsonl"])
images = load_dataset("PRAIG/SMB", split="train")
```
Do you know it this is an expected behaviour? This makes my dataset viewer to only load the images without the labeling of metadata.jsonl.

Thanks

Hi ! this is because we now expect the metadata file to be inside the directory named after the split "train" (this way each split can have its own metadata and can be loaded independently)

You can fix that by configuring it explicitly in the dataset's README.md header:

```yaml
configs:
- config_name: default
  data_files:
  - split: train
    path:
    - "train/**/*.png"
    - "metadata.jsonl"
```

(or by moving the metadata.jsonl in train/ but in this case you also have to modify the content of the JSONL to fix the relative paths to the images)

Thank you very much, dataset viewer is already working as expected!!
Download datasets from a private hub in 2025

### Feature request

In the context of a private hub deployment, customers would like to use load_dataset() to load datasets from their hub, not from the public hub. This doesn't seem to be configurable at the moment and it would be nice to add this feature.

The obvious workaround is to clone the repo first and then load it from local storage, but this adds an extra step. It'd be great to have the same experience regardless of where the hub is hosted.

This issue was raised before here: https://github.com/huggingface/datasets/issues/3679
@juliensimon

### Motivation

none

### Your contribution

none

Hi ! First, and in the general case, Hugging Face does offer to host private datasets, and with a subscription you can even choose the region in which the repositories are hosted (US, EU)

Then if you happen to have a private deployment, you can set the HF_ENDPOINT environment variable (same as in https://github.com/huggingface/transformers/issues/38634)

Thank you @lhoestq. Works as described!
Add `IterableDataset.push_to_hub()`

Basic implementation, which writes one shard per input dataset shard.
This is to be improved later.

Close https://github.com/huggingface/datasets/issues/5665

PS: for image/audio datasets structured as actual image/audio files (not parquet), you can sometimes speed it up with `ds.decode(num_threads=...).push_to_hub(...)`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7595). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add option to ignore keys/columns when loading a dataset from jsonl(or any other data format)

### Feature request

Hi, I would like the option to ignore keys/columns when loading a dataset from files (e.g. jsonl).

### Motivation

I am working on a dataset which is built on jsonl. It seems the dataset is unclean and a column has different types in each row. I can't clean this or remove the column (It is not my data and it is too big for me to clean and save on my own hardware). 
I would like the option to just ignore this column when using `load_dataset`, since i don't need it.
I tried to look if this is already possible but couldn't find a solution. if there is I would love some help. If it is not currently possible, I would love this feature

### Your contribution

I don't think I can help this time, unfortunately.

Good point, I'd be in favor of having the `columns` argument in `JsonConfig` (and the others) to align with `ParquetConfig` to let users choose which columns to load and ignore the rest

Is it possible to ignore columns when using parquet? 

Yes, you can pass `columns=...` to load_dataset to select which columns to load, and it is passed to `ParquetConfig` :)

Ok, i didn't know that. 
Anyway, it would be good to add this to others

Hi @lhoestq 

I'd like to take this up!

As you suggested, I’ll extend the support for the columns parameter (currently used in ParquetConfig) to JsonConfig as well. This will allow users to selectively load specific keys/columns from .jsonl (or .json) files and ignore the rest — solving the type inconsistency issues in unclean datasets.

Hi @avishaiElmakies and @lhoestq 

Just wanted to let you know that this is now implemented in #7594
As suggested, support for the `columns=...` argument (previously available for Parquet) has now been extended to **JSON and JSONL** loading via `load_dataset(...)`. You can now load only specific keys/columns and skip the rest — which should help in cases where some fields are unclean, inconsistent, or just unnecessary.

### ✅ Example:

```python
from datasets import load_dataset

dataset = load_dataset("json", data_files="your_data.jsonl", columns=["id", "title"])
print(dataset["train"].column_names)
# Output: ['id', 'title']
```

### 🔧 Summary of changes:

* Added `columns: Optional[List[str]]` to `JsonConfig`
* Updated `_generate_tables()` to filter selected columns
* Forwarded `columns` argument from `load_dataset()` to the config
* Added test case to validate behavior

Let me know if you'd like the same to be added for CSV or others as a follow-up — happy to help.

@ArjunJagdale this looks great! Thanks!
I believe that every format that is supported by `datasets` should probably have this feature since it is very useful and will streamline the api (people will know that they can just use `columns` to select the columns they want, and it will  not be dependent on the data format) 

Thanks @avishaiElmakies — totally agree, making `columns=...` support consistent across all formats would be really helpful for users.
Fix broken link to albumentations

A few months back I rewrote all docs at [https://albumentations.ai/docs](https://albumentations.ai/docs), and some pages changed their links.

In this PR fixed link to the most recent doc in Albumentations about bounding boxes and it's format. 

Fix a few typos in the doc as well.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7593). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq ping
Add num_proc parameter to push_to_hub

### Feature request

A number of processes parameter to the dataset.push_to_hub method

### Motivation

Shards are currently uploaded serially which makes it slow for many shards, uploading can be done in parallel and much faster


Hi @SwayStar123 

I'd be interested in taking this up. I plan to add a `num_proc` parameter to `push_to_hub()` and use parallel uploads for shards using `concurrent.futures`. Will explore whether `ThreadPoolExecutor` or `ProcessPoolExecutor` is more suitable based on current implementation. Let me know if that sounds good!


Just a quick update — `push_to_hub()` already had the `num_proc` argument in its signature and was correctly passing it internally to `_push_parquet_shards_to_hub()`.

The actual change required was inside `_push_parquet_shards_to_hub()` to enable parallel shard uploads using `multiprocessing` when `num_proc > 1`.

@lhoestq @SwayStar123 

> Hi @SwayStar123 
> 
> I'd be interested in taking this up. I plan to add a `num_proc` parameter to `push_to_hub()` and use parallel uploads for shards using `concurrent.futures`. Will explore whether `ThreadPoolExecutor` or `ProcessPoolExecutor` is more suitable based on current implementation. Let me know if that sounds good!
> 

Hey thanks for working on it. But I'm not a hf dev so I don't know the best way to do it.
feat: use content defined chunking 

WIP:
- [x] set the parameters in `io.parquet.ParquetDatasetReader`
- [x] set the parameters in `arrow_writer.ParquetWriter`

It requires a new pyarrow pin ">=21.0.0" which is not yet released.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7589). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Need to set `DEFAULT_MAX_BATCH_SIZE = 1024 * 1024`
ValueError: Invalid pattern: '**' can only be an entire path component [Colab]

### Describe the bug

I have a dataset on HF [here](https://huggingface.co/datasets/kambale/luganda-english-parallel-corpus) that i've previously used to train a translation model [here](https://huggingface.co/kambale/pearl-11m-translate).

now i changed a few hyperparameters to increase number of tokens for the model, increase Transformer layers, and all

however, when i try to load the dataset, this error keeps coming up.. i have tried everything.. i have re-written the code a hundred times, and this keep coming up

### Steps to reproduce the bug

Imports:

```bash
!pip install datasets huggingface_hub fsspec
```

Python code:

```python
from datasets import load_dataset

HF_DATASET_NAME = "kambale/luganda-english-parallel-corpus"

# Load the dataset
try:
    if not HF_DATASET_NAME or HF_DATASET_NAME == "YOUR_HF_DATASET_NAME":
        raise ValueError(
            "Please provide a valid Hugging Face dataset name."
        )

    dataset = load_dataset(HF_DATASET_NAME)

# Omitted code as the error happens on the line above 

except ValueError as ve:
    print(f"Configuration Error: {ve}")
    raise
except Exception as e:
    print(f"An error occurred while loading the dataset '{HF_DATASET_NAME}': {e}")
    raise e
```

now, i have tried going through this [issue](https://github.com/huggingface/datasets/issues/6737) and nothing helps 

### Expected behavior

loading the dataset successfully and perform splits (train, test, validation)

### Environment info

from the imports, i do not install specific versions of these libraries, so the latest or available version is installed

* `datasets` version: latest
* `Platform`: Google Colab
* `Hardware`: NVIDIA A100 GPU
* `Python` version: latest
* `huggingface_hub` version: latest
* `fsspec` version: latest

Could you please run the following code snippet in your environment and share the exact output? This will help check for any compatibility issues within the env itself. 

```
import datasets
import huggingface_hub
import fsspec

print("datasets version:", datasets.__version__)
print("huggingface_hub version:", huggingface_hub.__version__)
print("fsspec version:", fsspec.__version__)
```

```bash
datasets version: 2.14.4
huggingface_hub version: 0.31.4
fsspec version: 2025.3.2
```

Version 2.14.4 is not the latest version available, in fact it is from August 08, 2023 (you can check here: https://pypi.org/project/datasets/#history)

Use pip install datasets==3.6.0 to install a more recent version (from May 7, 2025)

I also had the same problem with Colab, after updating to the latest version it was solved.

I hope it helps

thank you @CleitonOERocha. it sure did help.

updating `datasets` to v3.6.0 and keeping `fsspec` on v2025.3.2 eliminates the issue.

Very helpful, thank you!
Avoid multiple default config names

Fix duplicating default config names.

Currently, when calling `push_to_hub(set_default=True` with 2 different config names, both are set as default. 

Moreover, this will generate an error next time we try to push another default config name, raised by `MetadataConfigs.get_default_config_name`:
https://github.com/huggingface/datasets/blob/da1db8a5b89fc0badaa0f571b36e122e52ae8c61/src/datasets/arrow_dataset.py#L5757
https://github.com/huggingface/datasets/blob/da1db8a5b89fc0badaa0f571b36e122e52ae8c61/src/datasets/utils/metadata.py#L186-L188

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7585). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Requesting a specific split (eg: test) still downloads all (train, test, val) data when streaming=False.

### Describe the bug

When using load_dataset() from the datasets library (in load.py), specifying a particular split (e.g., split="train") still results in downloading data for all splits when streaming=False. This happens during the builder_instance.download_and_prepare() call.
This behavior leads to unnecessary bandwidth usage and longer download times, especially for large datasets, even if the user only intends to use a single split.

### Steps to reproduce the bug

dataset_name = "skbose/indian-english-nptel-v0"
dataset = load_dataset(dataset_name, token=hf_token, split="test")


### Expected behavior

Optimize the download logic so that only the required split is downloaded when streaming=False when a specific split is provided.

### Environment info

Dataset: skbose/indian-english-nptel-v0
Platform: M1 Apple Silicon
Python verison: 3.12.9
datasets>=3.5.0

Hi ! There was a PR open to improve this: https://github.com/huggingface/datasets/pull/6832 
but it hasn't been continued so far.

It would be a cool improvement though !
arrow_schema is not compatible with list

### Describe the bug

```
import datasets
f = datasets.Features({'x': list[datasets.Value(dtype='int32')]})
f.arrow_schema

Traceback (most recent call last):
  File "datasets/features/features.py", line 1826, in arrow_schema
    return pa.schema(self.type).with_metadata({"huggingface": json.dumps(hf_metadata)})
                     ^^^^^^^^^
  File "datasets/features/features.py", line 1815, in type
    return get_nested_type(self)
           ^^^^^^^^^^^^^^^^^^^^^
  File "datasets/features/features.py", line 1252, in get_nested_type
    return pa.struct(
           ^^^^^^^^^^
  File "pyarrow/types.pxi", line 5406, in pyarrow.lib.struct
  File "pyarrow/types.pxi", line 3890, in pyarrow.lib.field
  File "pyarrow/types.pxi", line 5918, in pyarrow.lib.ensure_type
TypeError: DataType expected, got <class 'list'>
```

The following works
```
f = datasets.Features({'x': datasets.LargeList(datasets.Value(dtype='int32'))})
```

### Expected behavior

according to https://github.com/huggingface/datasets/blob/458f45a22c3cc9aea5f442f6f519333dcfeae9b9/src/datasets/features/features.py#L1765 python list should be a valid type specification for features

### Environment info

- `datasets` version: 3.5.1
- Platform: macOS-15.5-arm64-arm-64bit
- Python version: 3.12.9
- `huggingface_hub` version: 0.30.2
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Thanks for reporting, I'll look into it

Actually it looks like you just forgot parenthesis:

```diff
- f = datasets.Features({'x': list[datasets.Value(dtype='int32')]})
+ f = datasets.Features({'x': list([datasets.Value(dtype='int32')])})
```

or simply using the `[ ]` syntax:

```python
f = datasets.Features({'x':[datasets.Value(dtype='int32')]})
```

I'm closing this issue if you don't mind

Ah is that what the syntax is? I don't think I was able to find an actual example of it so I assumed it was in the same way that you specify types eg. `list[int]`. This is good to know, thanks.
Fix regex library warnings

# PR Summary
This small PR resolves the regex library warnings showing starting Python3.11:
```python
DeprecationWarning: 'count' is passed as positional argument
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7576). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Missing multilingual directions in IWSLT2017 dataset's processing script

### Describe the bug

Hi, 

Upon using `iwslt2017.py` in `IWSLT/iwslt2017` on the Hub for loading the datasets, I am unable to obtain the datasets for the language pairs `de-it`, `de-ro`, `de-nl`, `it-de`, `nl-de`, and `ro-de` using it. These 6 pairs do not show up when using `get_dataset_config_names()` to obtain the list of all the configs present in `IWSLT/iwslt2017`. This should not be the case since as mentioned in their original paper (please see https://aclanthology.org/2017.iwslt-1.1.pdf), the authors specify that "_this year we proposed the multilingual translation between any pair of languages from {Dutch, English, German, Italian, Romanian}..._" and because these datasets are indeed present in `data/2017-01-trnmted/texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.zip`.

Best Regards,
Anand

### Steps to reproduce the bug

Check the output of `get_dataset_config_names("IWSLT/iwslt2017", trust_remote_code=True)`: only 24 language pairs are present and the following 6 config names are absent: `iwslt2017-de-it`, `iwslt2017-de-ro`, `iwslt2017-de-nl`, `iwslt2017-it-de`, `iwslt2017-nl-de`, and `iwslt2017-ro-de`.

### Expected behavior

The aforementioned 6 language pairs should also be present and hence, all these 6 language pairs' IWSLT2017 datasets must also be available for further use. 

I would suggest removing `de` from the `BI_LANGUAGES` list and moving it over to the `MULTI_LANGUAGES` list instead in `iwslt2017.py` to account for all the 6 missing language pairs (the same `de-en` dataset is present in both `data/2017-01-trnmted/texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.zip` and `data/2017-01-trnted/texts/de/en/de-en.zip` but the `de-ro`, `de-nl`, `it-de`, `nl-de`, and `ro-de` datasets are only present in  `data/2017-01-trnmted/texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.zip`: so, its unclear why the following comment: _`# XXX: Artificially removed DE from here, as it also exists within bilingual data`_ has been added as `L71` in `iwslt2017.py`). The `README.md` file in `IWSLT/iwslt2017`must then be re-created using `datasets-cli test path/to/iwslt2017.py --save_info --all_configs` to pass all split size verification checks for the 6 new language pairs which were previously non-existent. 

### Environment info

- `datasets` version: 3.5.0
- Platform: Linux-6.8.0-56-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.30.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

I have opened 2 PRs on the Hub: `https://huggingface.co/datasets/IWSLT/iwslt2017/discussions/7` and `https://huggingface.co/datasets/IWSLT/iwslt2017/discussions/8` to resolve this issue

cool ! I pinged the owners of the dataset on HF to merge your PRs :)
No Samsum dataset

### Describe the bug

https://huggingface.co/datasets/Samsung/samsum dataset not found error 404

Originated from https://github.com/meta-llama/llama-cookbook/issues/948

### Steps to reproduce the bug

go to website https://huggingface.co/datasets/Samsung/samsum
see the error

also downloading it with python throws 
```
Couldn't find 'Samsung/samsum' on the Hugging Face Hub either: FileNotFoundError: Samsung/samsum@f00baf5a7d4abfec6820415493bcb52c587788e6/samsum.py (repository not found)
```

### Expected behavior

Dataset exists

### Environment info

```
- `datasets` version: 3.2.0
- Platform: macOS-15.4.1-arm64-arm-64bit
- Python version: 3.12.2
- `huggingface_hub` version: 0.26.5
- PyArrow version: 16.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
```

According to the following https://huggingface.co/posts/seawolf2357/424129432408590, as of now the dataset seems to be inaccessible.

@IgorKasianenko, would https://huggingface.co/datasets/knkarthick/samsum suffice for your purpose?


Thanks @SP1029 for the update!
That will work for now, using it as replacement. Is there a officially recommended way to maintain the CC licensed dataset under the organization account? 
Feel free to close this issue

> Is there an officially recommended way to maintain a CC-licensed dataset under an organizational account?

@IgorKasianenko, apologies, this is not my area of expertise.

> Please feel free to close this issue.

I have limited access and may not be able to do that. Since you opened it, you would be able to close it.
add check if repo exists for dataset uploading

Currently, I'm reuploading datasets for [`MTEB`](https://github.com/embeddings-benchmark/mteb/). Some of them have many splits (more than 20), and I'm encountering the error:
`Too many requests for https://huggingface.co/datasets/repo/create`.

It seems that this issue occurs because the dataset tries to recreate itself every time a split is uploaded. To resolve this, I've added a check to ensure that if the dataset already exists, it won't attempt to recreate it.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7565). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq Can you review, please? I don't think that errors in CI are related to my changes
Implementation of iteration over values of a column in an IterableDataset object

Refers to [this issue](https://github.com/huggingface/datasets/issues/7381).

A couple of questions:
1. I've noticed two strange things: 1) "Around 80% of the final dataset is made of the `en_dataset`" in https://huggingface.co/docs/datasets/stream, 2) "Click on "Pull request" to send your to the project maintainers" in https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md Are `en_dataset` and "your [???]" typos? If so, I can fix them in this PR.
2. Should I update https://huggingface.co/docs/datasets/stream or https://huggingface.co/docs/datasets/access#iterabledataset to include the new feature?

Great ! and chained indexing was easy indeed, thanks :)

regarding your questions:

> I've noticed two strange things: 1) "Around 80% of the final dataset is made of the en_dataset" in https://huggingface.co/docs/datasets/stream, 2) "Click on "Pull request" to send your to the project maintainers" in https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md Are en_dataset and "your [???]" typos? If so, I can fix them in this PR.

Oh good catch, both should be fixed indeed. Feel free to open a new PR for those docs fixes

> Should I update https://huggingface.co/docs/datasets/stream or https://huggingface.co/docs/datasets/access#iterabledataset to include the new feature?

Yep good idea, I think in both places, since /stream is supposed to be exhaustive, and /access already mentions accessing a specific column for `Dataset`

@lhoestq, thank you for the answers!

> Yep good idea, I think in both places, since /stream is supposed to be exhaustive, and /access already mentions accessing a specific column for Dataset

👍, I'll try to add something.

By the way, do you have any ideas about why the CI pipelines have failed? Essentially, I've already encountered these problems [here](https://github.com/huggingface/datasets/issues/7381#issuecomment-2863421974).
I think `check_code_quality` has failed due to the usage of `pre-commit`. The problem seems to be the old version of the ruff hook. I've tried `v0.11.8` (the one that was installed with `pip install -e ".[quality]"`) and `pre-commit` seems to work like `make style` now. However, I don't have any ideas about `pyav` since I don't know what it is...

I've updated /stream and /access, please check the style and clarity. By the way, I would like to add `IterableDataset.skip` near `IterableDataset.take` to mimic [slicing](https://huggingface.co/docs/datasets/access/#slicing). What do you think?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7564). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix regression

reported in https://github.com/huggingface/datasets/pull/7557 (I just reorganized the condition)

wanted to apply this change to the original PR but github didn't let me apply it directly - merging this one instead

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7558). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
check for empty _formatting

Fixes a regression from #7553 breaking shuffling of iterable datasets 
<img width="884" alt="Screenshot 2025-05-07 at 9 16 52 AM" src="https://github.com/user-attachments/assets/d2f43c5f-4092-4efe-ac31-a32cbd025fe3" />


Thanks for reporting and for the fix ! I tried to reorganize the condition in your PR but didn't get the right permission so. I ended up merging https://github.com/huggingface/datasets/pull/7558 directly so I can make a release today - I hope you don't mind
Add `--merge-pull-request` option for `convert_to_parquet`

Closes #7527 

Note that this implementation **will only merge the last PR in the case that they get split up by `push_to_hub`**. See https://github.com/huggingface/datasets/discussions/7555 for more details.

This is ready for a review, happy to make any changes. The main question for maintainers is how this should interact with #7555. If my suggestion there is accepted, this PR can be kept as is. If not, more changes are required to merge all the PR parts.
datasets downloads and generates all splits, even though a single split is requested (for dataset with loading script)

### Describe the bug

`datasets` downloads and generates all splits, even though a single split is requested.  [This](https://huggingface.co/datasets/jordiae/exebench) is the dataset in question.  It uses a loading script.  I am not 100% sure that this is a bug, because maybe with loading scripts `datasets` must actually process all the splits?  But I thought loading scripts were designed to avoid this.

### Steps to reproduce the bug

See [this notebook](https://colab.research.google.com/drive/14kcXp_hgcdj-kIzK0bCG6taE-CLZPVvq?usp=sharing)

Or:
```python
from datasets import load_dataset
dataset = load_dataset('jordiae/exebench', split='test_synth', trust_remote_code=True)
```

### Expected behavior

I expected only the `test_synth` split to be downloaded and processed.

### Environment info

- `datasets` version: 3.5.1
- Platform: Linux-6.1.123+-x86_64-with-glibc2.35
- Python version: 3.11.12
- `huggingface_hub` version: 0.30.2
- PyArrow version: 18.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2025.3.0

Hi ! there has been some effort on allowing to download only a subset of splits in https://github.com/huggingface/datasets/pull/6832 but no one has been continuing this work so far. This would be a welcomed contribution though

Also note that loading script are often unoptimized, and we recommend using datasets in standard formats like Parquet instead.

Btw there is a CLI tool to convert a loading script to parquet:

```
datasets-cli convert_to_parquet <dataset-name> --trust_remote_code
```

Closing in favor of #6832
Rebatch arrow iterables before formatted iterable

close https://github.com/huggingface/datasets/issues/7538 and https://github.com/huggingface/datasets/issues/7475

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7553). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq Our CI found an issue with this changeset causing a regression with shuffling iterable datasets 
<img width="884" alt="Screenshot 2025-05-07 at 9 16 52 AM" src="https://github.com/user-attachments/assets/bf7d9c7e-cc14-47da-8da6-d1a345992d7c" />
Enable xet in push to hub

follows https://github.com/huggingface/huggingface_hub/pull/3035

related to https://github.com/huggingface/datasets/issues/7526

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7552). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue with offline mode and partial dataset cached

### Describe the bug

Hi, 

a issue related to #4760 here when loading a single file from a dataset, unable to access it in offline mode afterwards

### Steps to reproduce the bug

```python
import os
# os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["HF_TOKEN"] = "xxxxxxxxxxxxxx"

import datasets

dataset_name = "uonlp/CulturaX"
data_files = "fr/fr_part_00038.parquet"

ds = datasets.load_dataset(dataset_name, split='train', data_files=data_files)
print(f"Dataset loaded   : {ds}")
```
Once the file has been cached, I rerun with the HF_HUB_OFFLINE activated an get this error : 
```
ValueError: Couldn't find cache for uonlp/CulturaX for config 'default-1e725f978350254e'
Available configs in the cache: ['default-2935e8cdcc21c613']
```

### Expected behavior

Should be able to access the previously cached files

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.4.0-215-generic-x86_64-with-glibc2.31
- Python version: 3.12.0
- `huggingface_hub` version: 0.27.0
- PyArrow version: 19.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1


It seems the problem comes from builder.py / create_config_id()

On the first call, when the cache is empty we have
```
config_kwargs = {'data_files': {'train': ['hf://datasets/uonlp/CulturaX@6a8734bc69fefcbb7735f4f9250f43e4cd7a442e/fr/fr_part_00038.parquet']}}
```
leading to config_id beeing 'default-2935e8cdcc21c613'

then, on the second call, 
```
config_kwargs = {'data_files': 'fr/fr_part_00038.parquet'}
```
thus explaining why the hash is not the same, despite having the same parameter when calling load_dataset : data_files="fr/fr_part_00038.parquet"

Same behavior with version 3.5.1

Same issue when loading `google/IndicGenBench_flores_in` with `dataset==2.21.0` and `dataset==3.6.0` .




> It seems the problem comes from builder.py / create_config_id()
> 
> On the first call, when the cache is empty we have
> 
> ```
> config_kwargs = {'data_files': {'train': ['hf://datasets/uonlp/CulturaX@6a8734bc69fefcbb7735f4f9250f43e4cd7a442e/fr/fr_part_00038.parquet']}}
> ```
> 
> leading to config_id beeing 'default-2935e8cdcc21c613'
> 
> then, on the second call,
> 
> ```
> config_kwargs = {'data_files': 'fr/fr_part_00038.parquet'}
> ```
> 
> thus explaining why the hash is not the same, despite having the same parameter when calling load_dataset : data_files="fr/fr_part_00038.parquet"


I have identified that the issue indeed lies in the `data_files` within `config_kwargs`. 
The format and prefix of `data_files` differ depending on whether `HF_HUB_OFFLINE` is set, leading to different final `config_id` values. 
When I use other datasets without passing the `data_files` parameter, this issue does not occur.

A possible solution might be to standardize the formatting of `data_files` within the `create_config_id` function.
Avoid global umask for setting file mode.

This PR updates the method for setting the permissions on `cache_path` after calling `shutil.move`. The call to `shutil.move` may not preserve permissions if the source and destination are on different filesystems. Reading and resetting umask can cause race conditions, so directly read what permissions were set for the `temp_file` instead.

This fixes https://github.com/huggingface/datasets/issues/7536.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7547). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Large memory use when loading large datasets to a ZFS pool

### Describe the bug

When I load large parquet based datasets from the hub like `MLCommons/peoples_speech` using `load_dataset`, all my memory (500GB) is used and isn't released after loading, meaning that the process is terminated by the kernel if I try to load an additional dataset. This makes it impossible to train models using multiple large datasets.

### Steps to reproduce the bug

`uv run --with datasets==3.5.1 python`

```python
from datasets import load_dataset
load_dataset('MLCommons/peoples_speech', 'clean')
load_dataset('mozilla-foundation/common_voice_17_0', 'en')
```

### Expected behavior

I would expect that a lot less than 500GB of RAM would be required to load the dataset, or at least that the RAM usage would be cleared as soon as the dataset is loaded (and thus reside as a memory mapped file) such that other datasets can be loaded.

### Environment info

I am currently using the latest datasets==3.5.1 but I have had the same problem with multiple other versions.

Hi ! datasets are memory mapped from disk, so they don't fill out your RAM. Not sure what's the source of your memory issue.

What kind of system are you using ? and what kind of disk ?

Well, the fact of the matter is that my RAM is getting filled out by running the given example, as shown in [this video](https://streamable.com/usb0ql).

My system is a GPU server running Ubuntu. The disk is a SATA SSD attached to the server using a backplane. It is formatted with ZFS, mounted in /cache, and my HF_HOME is set to /cache/hf

I really need this fixed, so I am more than willing to test out various suggestions you might have, or write a PR if we can figure out what is going on.

I'm not super familiar with ZFS, but it looks like it loads the data in memory when the files are memory mapped, which is an issue.

Maybe it's a caching mechanism ? Since `datasets` accesses every memory mapped file to read a small part (the metadata of the arrow record batches), maybe ZFS brings the whole files in memory for quicker subsequent reads. This is an antipattern when it comes to lazy loading datasets of that size though

This is the answer.

I tried changing my HF_HOME to an NFS share, and no RAM is then consumed loading the dataset.

I will try to see if I can find a way to configure the ZFS pool to not cache the files (disabling the ARC/primary cache didn't work), and if I do write the solution in this issue. If I can't I guess I have to reformat my cache drive.
Networked Pull Through Cache

### Feature request

Introduce a HF_DATASET_CACHE_NETWORK_LOCATION configuration (e.g. an environment variable) together with a companion network cache service.

Enable a three-tier cache lookup for datasets:

1. Local on-disk cache
2. Configurable network cache proxy
3. Official Hugging Face Hub

### Motivation

- Distributed training & ephemeral jobs: In high-performance or containerized clusters, relying solely on a local disk cache either becomes a streaming bottleneck or incurs a heavy cold-start penalty as each job must re-download datasets.
- Traffic & cost reduction: A pull-through network cache lets multiple consumers share a common cache layer, reducing duplicate downloads from the Hub and lowering egress costs.
- Better streaming adoption: By offloading repeat dataset pulls to a locally managed cache proxy, streaming workloads can achieve higher throughput and more predictable latency.
- Proven pattern: Similar proxy-cache solutions (e.g. Harbor’s Proxy Cache for Docker images) have demonstrated reliability and performance at scale: https://goharbor.io/docs/2.1.0/administration/configure-proxy-cache/

### Your contribution

I’m happy to draft the initial PR for adding HF_DATASET_CACHE_NETWORK_LOCATION support in datasets and sketch out a minimal cache-service prototype.

I have limited bandwidth so I would be looking for collaborators if anyone else is interested.
Add try_original_type to DatasetDict.map

This PR resolves #7472 for DatasetDict 

The previously merged PR #7483 added `try_original_type` to ArrowDataset, but DatasetDict misses `try_original_type`

Cc: @lhoestq 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7544). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sure! I just committed the changes

@lhoestq 
Let me know if there are other things to do before merge or other places to add `try_original_type` argument