noamaanMulla-03/datasets-issues huggingface dataset:
Feature request: IterableDataset.push_to_hub

### Feature request

It'd be great to have a lazy push to hub, similar to the lazy loading we have with `IterableDataset`.

Suppose you'd like to filter [LAION](https://huggingface.co/datasets/laion/laion400m) based on certain conditions, but as LAION doesn't fit into your disk, you'd like to leverage streaming:
```
from datasets import load_dataset

dataset = load_dataset("laion/laion400m", streaming=True, split="train")
```
Then you could filter the dataset based on certain conditions:
```
filtered_dataset = dataset.filter(lambda example: example['HEIGHT'] > 400)
```

In order to persist this dataset and push it back to the hub, one currently needs to first load the entire filtered dataset on disk and then push:

```
from datasets import Dataset

Dataset.from_generator(filtered_dataset.__iter__).push_to_hub(...)
```
It would be great if we can instead lazy push to the data to the hub (basically stream the data to the hub), not being limited by our disk size:
```
filtered_dataset.push_to_hub("my-filtered-dataset")
```

### Motivation

This feature would be very useful for people that want to filter huge datasets without having to load the entire dataset or a filtered version thereof on their local disk.

### Your contribution

Happy to test out a PR :)

Labels: name: enhancement

+1

+1

+1, should be possible now? :) https://huggingface.co/blog/xethub-joins-hf

Haha we're working hard to integrate Xet in the HF back-end, it will enable cool use cases :)

Anyway about `IterableDataset.push_to_hub`, I'd be happy to to provide guidance and answer questions if anyone wants to start a first simple implementation of this

+1

+1

+1

+1

Currently running into this when filtering [Common Corpus](https://huggingface.co/datasets/PleIAs/common_corpus) for Dutch entries.

Extra points for somehow making it resumable on error. 11 TB is a lot of data to stream on a home connection without encountering any sort of errors along the way.

If it can help, IterableDataset does implement `.state_dict()` and `.load_state_dict()` that you can use to resume a stream already

+1

+1

+1

Just added a first implementation for `IterableDataset.push_to_hub()` :)

I'll do a new release soon, in the meantime feel free to install `datasets` from source to try it out !
Enhance error handling and input validation across multiple modules

This PR improves the robustness and user experience by:

1. **Audio Module**:
   - Added clear error messages when required fields ('path' or 'bytes') are missing in audio encoding

2. **DatasetDict**:
   - Enhanced key access error messages to show available splits when an invalid key is accessed

3. **NonMutableDict**:
   - Added input validation for the update() method to ensure proper mapping types

4. **Arrow Reader**:
   - Improved error messages for small dataset percentage splits with suggestions for alternatives

5. **FaissIndex**:
   - Strengthened input validation with descriptive error messages
   - Added proper type checking and shape validation for search queries

These changes make the code more maintainable and user-friendly by providing actionable feedback when issues arise.
Docs and more methods for IterableDataset: push_to_hub, to_parquet...

to_csv, to_json, to_sql, to_pandas, to_polars, to_dict, to_list


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7604). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Tests typing and fixes for push_to_hub

todo:
- [x] fix TestPushToHub.test_push_dataset_dict_to_hub_iterable_num_proc


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7608). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Download datasets from a private hub in 2025

### Feature request

In the context of a private hub deployment, customers would like to use load_dataset() to load datasets from their hub, not from the public hub. This doesn't seem to be configurable at the moment and it would be nice to add this feature.

The obvious workaround is to clone the repo first and then load it from local storage, but this adds an extra step. It'd be great to have the same experience regardless of where the hub is hosted.

This issue was raised before here: https://github.com/huggingface/datasets/issues/3679
@juliensimon

### Motivation

none

### Your contribution

none

Labels: name: enhancement

Hi ! First, and in the general case, Hugging Face does offer to host private datasets, and with a subscription you can even choose the region in which the repositories are hosted (US, EU)

Then if you happen to have a private deployment, you can set the HF_ENDPOINT environment variable (same as in https://github.com/huggingface/transformers/issues/38634)

Thank you @lhoestq. Works as described!
fix: raise error when folder-based datasets are loaded without data_dir or data_files


### Related Issues/PRs

<!-- Uncomment 'Resolve' if this PR can close the linked items. -->
<!-- Resolve --> #6152

---

### What changes are proposed in this pull request?

This PR adds an early validation step for folder-based datasets (like `audiofolder`) to prevent silent fallback behavior.

**Before this fix**:
- When `data_dir` or `data_files` were not provided, the loader defaulted to the current working directory.
- This caused unexpected behavior like:
  - Long loading times
  - Scanning unintended local files

**Now**:
- If both `data_dir` and `data_files` are missing, a `ValueError` is raised early with a helpful message.

---

### How is this PR tested?

- [x] Manual test via `load_dataset("audiofolder")` with missing `data_dir`
- [ ] Existing unit tests (should not break any)
- [ ] New tests (if needed, maintainers can guide)

---

### Does this PR require documentation update?

- [x] No. You can skip the rest of this section.

---

### Release Notes

#### Is this a user-facing change?

- [x] Yes. Give a description of this change to be included in the release notes for users.

> Adds early error handling for folder-based datasets when neither `data_dir` nor `data_files` is specified, avoiding unintended resolution to the current directory.

#### What component(s), interfaces, languages, and integrations does this PR affect?

Components:
- [x] `area/datasets`
- [x] `area/load`

---

<a name="release-note-category"></a>

#### How should the PR be classified in the release notes? Choose one:

- [x] `rn/bug-fix` - A user-facing bug fix worth mentioning in the release notes

---

#### Should this PR be included in the next patch release?

- [x] Yes (this PR will be cherry-picked and included in the next patch release)



Great ! Since this logic is specific to one builder class maybe this check can be in the class definition ? I think you can put it in FolderBasedBuilder's `_info()` method.
Update `_dill.py` to use `co_linetable` for Python 3.10+ in place of `co_lnotab`

Not 100% about this one, but it seems to be recommended.

```
/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/datasets/utils/_dill.py:385: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
```
Tests pass locally. And the warning is gone with this change.

https://peps.python.org/pep-0626/#backwards-compatibility


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7609). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

not 100% sure either, I tried removing unnecessary checks - let me know if they sound good to you otherwise I'll revert

I can't reproduce the warning anymore... 🤦🏻‍♂️


Ah now I can reproduce!, and I can confirm that the warning is gone when you apply the change in this PR
My already working dataset (when uploaded few months ago) now is ignoring metadata.jsonl

### Describe the bug

Hi everyone, I uploaded my dataset https://huggingface.co/datasets/PRAIG/SMB a few months ago while I was waiting for a conference acceptance response. Without modifying anything in the dataset repository now the Dataset viewer is not rendering the metadata.jsonl annotations, neither it is being downloaded when using load_dataset. Can you please help? Thank you in advance.

### Steps to reproduce the bug

from datasets import load_dataset
ds = load_dataset("PRAIG/SMB")
ds = ds["train"]

### Expected behavior

 It is expected to have all the metadata available in the jsonl file. Fields like: "score_id", "original_width", "original_height", "regions"... among others.

### Environment info

datasets==3.6.0, python 3.13.3 (but he problem is already in the huggingface dataset page)


Maybe its been a recent update, but i can manage to load the metadata.jsonl separately from the images with:

```
metadata = load_dataset("PRAIG/SMB", split="train", data_files=["*.jsonl"])
images = load_dataset("PRAIG/SMB", split="train")
```
Do you know it this is an expected behaviour? This makes my dataset viewer to only load the images without the labeling of metadata.jsonl.

Thanks

Hi ! this is because we now expect the metadata file to be inside the directory named after the split "train" (this way each split can have its own metadata and can be loaded independently)

You can fix that by configuring it explicitly in the dataset's README.md header:

```yaml
configs:
- config_name: default
  data_files:
  - split: train
    path:
    - "train/**/*.png"
    - "metadata.jsonl"
```

(or by moving the metadata.jsonl in train/ but in this case you also have to modify the content of the JSONL to fix the relative paths to the images)

Thank you very much, dataset viewer is already working as expected!!
Lazy column

Same as https://github.com/huggingface/datasets/pull/7564 but for `Dataset`, cc @TopCoder2K FYI

e.g. `ds[col]` now returns a lazy Column instead of a list

This way calling `ds[col][idx]` only loads the required data in memory

(bonus: also supports subfields access with `ds[col][subcol][idx]`)

the breaking change will be for the next major release, which also includes removal of dataset scripts support

close https://github.com/huggingface/datasets/issues/4180


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7614). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: raise error in FolderBasedBuilder when data_dir and data_files are missing

### Related Issues/PRs

Fixes #6152

---

### What changes are proposed in this pull request?

This PR adds a dedicated validation check in the `_info()` method of the `FolderBasedBuilder` class to ensure that users provide either `data_dir` or `data_files` when loading folder-based datasets (such as `audiofolder`, `imagefolder`, etc.).

---

### Why this change?

Previously, when calling:

```python
load_dataset("audiofolder")
````

without specifying `data_dir` or `data_files`, the loader would silently fallback to the **current working directory**, leading to:

* Long loading times
* Unexpected behavior (e.g., scanning unrelated files)

This behavior was discussed in issue #6152. As suggested by maintainers, the fix has now been implemented directly inside the `FolderBasedBuilder._info()` method — keeping the logic localized to the specific builder instead of a generic loader function.

---

### How is this PR tested?

* ✅ Manually tested by calling `load_dataset("audiofolder")` with no `data_dir` or `data_files` → a `ValueError` is now raised early.
* ✅ Existing functionality (with valid input) remains unaffected.

---

### Does this PR require documentation update?

* [x] No

---

### Release Notes

#### Is this a user-facing change?

* [x] Yes

> Folder-based datasets now raise an explicit error if neither `data_dir` nor `data_files` are specified, preventing unintended fallback to the current working directory.

---

#### What component(s) does this PR affect?

* [x] `area/datasets`
* [x] `area/load`

---

<a name="release-note-category"></a>

#### How should the PR be classified?

* [x] `rn/bug-fix` - A user-facing bug fix

---

#### Should this be included in the next patch release?

* [x] Yes


@lhoestq Moved the logic to FolderBasedBuilder._info() as discussed in previous PR (#7618). Let me know if anything else is needed — happy to update!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7623). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
FolderBase Dataset automatically resolves under current directory when data_dir is not specified

### Describe the bug

FolderBase Dataset automatically resolves under current directory when data_dir is not specified.

For example:
```
load_dataset("audiofolder")
```
takes long time to resolve and collect data_files from current directory. But I think it should reach out to this line for error handling https://github.com/huggingface/datasets/blob/cb8c5de5145c7e7eee65391cb7f4d92f0d565d62/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L58-L59

### Steps to reproduce the bug

```
load_dataset("audiofolder")
```

### Expected behavior

Error report

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.15.0-78-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3

Labels: name: good first issue

@lhoestq 

Makes sense, I guess this can be fixed in the load_dataset_builder method.
It concerns every packaged builder I think (see values in `_PACKAGED_DATASETS_MODULES`)

I think the behavior is related to these lines, which short circuited the error handling.
https://github.com/huggingface/datasets/blob/664a1cb72ea1e6ef7c47e671e2686ca4a35e8d63/src/datasets/load.py#L946-L952

So should data_dir be checked here or still delegating to actual `DatasetModule`? In that case, how to properly set `data_files` here.

This is location in PackagedDatasetModuleFactory.get_module seems the be the right place to check if at least data_dir or data_files are passed

@mariosasko can you please assign this issue to me,I want to work on this

#self-assign

@mariosasko is this issue still open? i would love to kickstart my journey to open source with this issue!
Regards
zutarich

@zutarich It is unless @debrupf2946 is working on it.

#self-assign

I am working and will open a pull request soon @Etelis 


@mariosasko can i take this up? 

#self-assign

Yes, feel free to work on this :)

i think its working as expected . Heres the log i get for the same line -

![image](https://github.com/huggingface/datasets/assets/63234112/8a857ec5-8dd0-4b01-b3a7-7c93444b9558)


#self-assign

Hey! I saw that PR #7314 fixed the issue when no matching data files are found, which is great. But I think the builder still defaults to the current directory when `data_dir` isn't specified, which could lead to unexpected behavior.

If no one’s actively working on that part, I’d be happy to pick it up and propose a fix. Let me know if that works!


Cool, feel free to ping me if you open a PR @aniyagnik :)

Hey! I noticed this is still open and unresolved. I’d like to take this up and submit a PR to prevent resolving to the current directory when data_dir is not specified!!!

Hi @aniyagnik @npuichigo @sahillihas @lhoestq @mariosasko,
I've opened a PR to address the issue discussed in #6152 by adding an early validation check for folder-based datasets. Would appreciate your review whenever convenient. Thanks!

https://github.com/huggingface/datasets/pull/7618
[Tests] Ensure `datasets` supports renamed repositories

On https://hf.co/datasets you can rename a dataset (or sometimes move it to another user/org). The website handles redirections correctly and AFAIK `datasets` does as well.

However it would be nice to have an integration test to make sure we don't break support for renamed datasets.

To implement this we can use the /api/repos/move endpoint on hub-ci to rename/move a repo (it is documented at https://huggingface.co/docs/hub/api)

Labels: name: good second issue

You could also switch to using `huggingface_hub` more directly, where such a guarantee is already tested =)

cc @Wauplin 

Hi! @lhoestq, I'd love to work on this issue as part of my ongoing contributions to `datasets`.

I'm planning to implement an integration test that ensures `load_dataset()` works even after a dataset repo is renamed or moved. I can work locally and interact with the `/api/repos/move` endpoint, using either `huggingface_hub` or `requests`, and will also run the test suite to validate behavior end-to-end.

Before I begin, I have two quick questions:
1. Is there a recommended test dataset or sandbox repo that I should use for safely renaming/moving during tests?
2. Do I need special permissions or tokens to use `hub-ci` or to move repos between users/orgs?

Once confirmed, I’ll proceed with writing and validating the test locally and open a PR soon after. Thank you!
[FR] `map` should reuse unchanged columns from the previous dataset to avoid disk usage

### Feature request

Currently adding a new column with `map` will cause all the data in the dataset to be duplicated and stored/cached on the disk again. It should reuse unchanged columns. 

### Motivation

This allows having datasets with different columns but sharing some basic columns. Currently, these datasets would become too expensive to store and one would need some kind of on-the-fly join; which also doesn't seem implemented.

### Your contribution

_

Labels: name: enhancement

name: good second issue

You can use the `remove_columns` parameter in `map` to avoid duplicating the columns (and save disk space) and then concatenate the original dataset with the map result:
```python
from datasets import concatenate_datasets
# dummy example
ds_new = ds.map(lambda x: {"new_col": x["col"] + 2}, remove_columns=ds.column_names)
ds_combined = concatenate_datasets([ds, ds_new], axis=1)
```

Doing this automatically is hard to implement efficiently unless we know ahead of time which existing columns will be modified by a `map` transform. We have this info when `input_columns` are specified, so I think this is the only case we can optimize.

Hi @mariosasko 👋 I’d like to start working on this issue.
Add `as_iterable_dataset()` method to DatasetBuilder for streaming from cached Arrow files

This PR implements `builder.as_iterable_dataset(split=...)` as discussed in #5481.

It allows users to load an `IterableDataset` directly from cached Arrow files (using ArrowReader and ArrowExamplesIterable), without loading the full dataset into memory.

This is useful for large-scale training scenarios where memory is constrained. A test has also been added in `test_builder.py`.

Related to: #5481
Add test for `as_iterable_dataset()` method in DatasetBuilder

This PR adds a test for the new `as_iterable_dataset()` method introduced in PR #7628.

The test:
- Loads a builder using `load_dataset_builder("c4", "en")`
- Runs `download_and_prepare()`
- Streams examples using `builder.as_iterable_dataset(split="train[:100]")`
- Verifies streamed examples contain the "text" field

This ensures that the builder correctly streams data from cached Arrow files.
#Dataset Make "image" column appear first in dataset preview UI

Hi!

#Dataset

I’m currently uploading a dataset that includes an `"image"` column (PNG files), along with some metadata columns. The dataset is loaded from a .jsonl file. My goal is to have the "image" column appear as the first column in the dataset card preview UI on the :hugs: Hub.

However, at the moment, the `"image"` column is not the first—in fact, it appears last, which is not ideal for the presentation I’d like to achieve.

I have a couple of questions:

Is there a way to force the dataset card to display the `"image"` column first?
Is there currently any way to control or influence the column order in the dataset preview UI?
Does the order of keys in the .jsonl file or the features argument affect the display order?
Thanks again for your time and help! :blush:


Hi ! It should follow the same order as the order of the keys in the metadata file

Hi! Thank you for your answer. 

As you said it, I I forced every key in every JSON to have an order using `collections. OrderedDict` in Python. Now, it works!

TY
Pass user-agent from DownloadConfig into fsspec storage_options

Fixes part of issue #6046

### Problem
The `user-agent` defined in `DownloadConfig` was not passed down to fsspec-based filesystems like `HfFileSystem`, which prevents proper identification/tracking of client requests.

### Solution
Added support for injecting the `user-agent` into `storage_options["headers"]` within `_prepare_single_hop_path_and_storage_options()` based on the `protocol`.

Now, when using `hf://`, `http://`, or `https://`, the custom user-agent is passed automatically.

### Code Location
Modified:
- `src/datasets/utils/file_utils.py`

Used `get_datasets_user_agent(...)` to ensure proper formatting and fallback logic.


- This PR assumes that `HfFileSystem` in `huggingface_hub` supports receiving `headers` in `storage_options`. If not, a follow-up PR can be opened to add this support to `HfFileSystem.__init__`.
- No test was added for this since it’s a config passthrough. If needed, I’d be happy to add one.
Loading just one particular split is not possible for imagenet-1k

### Describe the bug

I'd expect the following code to download just the validation split but instead I get all data on my disk (train, test and validation splits)

`
from datasets import load_dataset

dataset = load_dataset("imagenet-1k", split="validation", trust_remote_code=True)
`

Is it expected to work like that? 

### Steps to reproduce the bug

1. Install the required libraries (python, datasets, huggingface_hub)
2. Login using huggingface cli
2. Run the code in the description

### Expected behavior

Just a single (validation) split should be downloaded.

### Environment info

python: 3.12.2
datasets: 2.18.0
huggingface_hub: 0.22.2


+1

+1
Creating a HF Dataset from lakeFS with S3 storage takes too much time!

Hi,

I’m new to HF dataset and I tried to create datasets based on data versioned in **lakeFS** _(**MinIO** S3 bucket as storage backend)_

Here I’m using ±30000 PIL image from MNIST data however it is taking around 12min to execute, which is a lot!

From what I understand, it is loading the images into cache then building the dataset.
– Please find bellow the execution screenshot –

Is there a way to optimize this or am I doing something wrong?

Thanks!

![Image](https://github.com/user-attachments/assets/c79257c8-f023-42a9-9e6f-0898b3ea93fe)


### > Update

The bottleneck, from what I understand, was making one network request per file

For 30k images, this meant 30k separate GET requests to the MinIO server through the S3 API, and that was killing the performance

Using webDataset to transform the large number of files to few .tar files and passing “webdataset” instead of “imagefolder” to the load_dataset function worked perfectly (took only ~11s)
Proposal: Small Tamil Discourse Coherence Dataset.

I’m a beginner from NIT Srinagar proposing a dataset of 50 Tamil text pairs for discourse coherence (coherent/incoherent labels) to support NLP research in low-resource languages.
- Size: 50 samples
- Format: CSV with columns (text1, text2, label)
- Use case: Training NLP models for coherence
I’ll use GitHub’s web editor and Google Colab. Please confirm if this fits.