Minimartzz/github-issues huggingface dataset:
Download datasets from a private hub in 2025
State: closed
Labels: description: New feature or request
name: enhancement

### Feature request

In the context of a private hub deployment, customers would like to use load_dataset() to load datasets from their hub, not from the public hub. This doesn't seem to be configurable at the moment and it would be nice to add this feature.

The obvious workaround is to clone the repo first and then load it from local storage, but this adds an extra step. It'd be great to have the same experience regardless of where the hub is hosted.

This issue was raised before here: https://github.com/huggingface/datasets/issues/3679
@juliensimon

### Motivation

none

### Your contribution

none

Comments:
Hi ! First, and in the general case, Hugging Face does offer to host private datasets, and with a subscription you can even choose the region in which the repositories are hosted (US, EU)

Then if you happen to have a private deployment, you can set the HF_ENDPOINT environment variable (same as in https://github.com/huggingface/transformers/issues/38634)

Thank you @lhoestq. Works as described!
Add `IterableDataset.push_to_hub()`
State: closed

Basic implementation, which writes one shard per input dataset shard.
This is to be improved later.

Close https://github.com/huggingface/datasets/issues/5665

PS: for image/audio datasets structured as actual image/audio files (not parquet), you can sometimes speed it up with `ds.decode(num_threads=...).push_to_hub(...)`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7595). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add option to ignore keys/columns when loading a dataset from jsonl(or any other data format)
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

Hi, I would like the option to ignore keys/columns when loading a dataset from files (e.g. jsonl).

### Motivation

I am working on a dataset which is built on jsonl. It seems the dataset is unclean and a column has different types in each row. I can't clean this or remove the column (It is not my data and it is too big for me to clean and save on my own hardware). 
I would like the option to just ignore this column when using `load_dataset`, since i don't need it.
I tried to look if this is already possible but couldn't find a solution. if there is I would love some help. If it is not currently possible, I would love this feature

### Your contribution

I don't think I can help this time, unfortunately.

Comments:
Good point, I'd be in favor of having the `columns` argument in `JsonConfig` (and the others) to align with `ParquetConfig` to let users choose which columns to load and ignore the rest

Is it possible to ignore columns when using parquet? 

Yes, you can pass `columns=...` to load_dataset to select which columns to load, and it is passed to `ParquetConfig` :)

Ok, i didn't know that. 
Anyway, it would be good to add this to others

Hi @lhoestq 

I'd like to take this up!

As you suggested, I’ll extend the support for the columns parameter (currently used in ParquetConfig) to JsonConfig as well. This will allow users to selectively load specific keys/columns from .jsonl (or .json) files and ignore the rest — solving the type inconsistency issues in unclean datasets.

Hi @avishaiElmakies and @lhoestq 

Just wanted to let you know that this is now implemented in #7594
As suggested, support for the `columns=...` argument (previously available for Parquet) has now been extended to **JSON and JSONL** loading via `load_dataset(...)`. You can now load only specific keys/columns and skip the rest — which should help in cases where some fields are unclean, inconsistent, or just unnecessary.

### ✅ Example:

```python
from datasets import load_dataset

dataset = load_dataset("json", data_files="your_data.jsonl", columns=["id", "title"])
print(dataset["train"].column_names)
# Output: ['id', 'title']
```

### 🔧 Summary of changes:

* Added `columns: Optional[List[str]]` to `JsonConfig`
* Updated `_generate_tables()` to filter selected columns
* Forwarded `columns` argument from `load_dataset()` to the config
* Added test case to validate behavior

Let me know if you'd like the same to be added for CSV or others as a follow-up — happy to help.

@ArjunJagdale this looks great! Thanks!
I believe that every format that is supported by `datasets` should probably have this feature since it is very useful and will streamline the api (people will know that they can just use `columns` to select the columns they want, and it will  not be dependent on the data format) 

Thanks @avishaiElmakies — totally agree, making `columns=...` support consistent across all formats would be really helpful for users.

#Codex Fix

#Codex Fix
Fix broken link to albumentations
State: closed

A few months back I rewrote all docs at [https://albumentations.ai/docs](https://albumentations.ai/docs), and some pages changed their links.

In this PR fixed link to the most recent doc in Albumentations about bounding boxes and it's format. 

Fix a few typos in the doc as well.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7593). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq ping
Remove scripts altogether
State: closed

TODO:
- [x] remplace fixtures based on script with no-script fixtures
- [x] windaube

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7592). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @lhoestq,
I wanted to ask
are you planning to stop supporting dataset builds using `GeneratorBasedBuilder`?

If so, could you share the reason why?

We stopped supporting dataset scripts altogether, whether they are based on GeneratorBasedBuilder or any other builder. This means you can't `load_dataset()` a dataset script anymore. We did this mostly for security reasons which is blocking for many users and also impossible to build upon (e.g. the for the Dataset Viewer on HF)

Ah, so only the `trust_remote_code` feature of `load_dataset` is deprecated, and

```python
from datasets import load_dataset_builder
  
builder = load_dataset_builder('cornell-movie-review-data/rotten_tomatoes')  
builder.download_and_prepare()  
```

we can still load data using `load_dataset_builder` and `download_and_prepare`, right?
That's a relief. I thought the removal of `trust_remote_code` in `load_dataset` meant `GeneratorBasedBuilder` was being deprecated too, haha.
Got it, thanks for the clarification!


Can you give an example on how to upgrade from using `trust_remote_code`? I used to load_dataset from a script generating my training data in a streaming way.

For guys who dislike this change +1

> Can you give an example on how to upgrade from using trust_remote_code? I used to load_dataset from a script generating my training data in a streaming way.

here is a command to convert a script based dataset on HF to a Parquet dataset:

```
uv run --with "datasets==3.6.0" datasets-cli convert_to_parquet <username/dataset-name> --trust_remote_code
```

And if you want to define a streaming pipeline, feel free to use the `IterableDataset` API instead:

```python
from datasets import load_dataset

ds = load_dataset(..., streaming=True)  # source data
ds = ds.map(...)  # preprocessing
ds = ds.filter(...)  # remove bad samples

# For training
# data_loader = DataLoader(ds, ...)

# Saving the dataset
# ds.push_to_hub(...)
```
Add num_proc parameter to push_to_hub
State: closed
Labels: description: New feature or request
name: enhancement

### Feature request

A number of processes parameter to the dataset.push_to_hub method

### Motivation

Shards are currently uploaded serially which makes it slow for many shards, uploading can be done in parallel and much faster


Comments:
Hi @SwayStar123 

I'd be interested in taking this up. I plan to add a `num_proc` parameter to `push_to_hub()` and use parallel uploads for shards using `concurrent.futures`. Will explore whether `ThreadPoolExecutor` or `ProcessPoolExecutor` is more suitable based on current implementation. Let me know if that sounds good!


Just a quick update — `push_to_hub()` already had the `num_proc` argument in its signature and was correctly passing it internally to `_push_parquet_shards_to_hub()`.

The actual change required was inside `_push_parquet_shards_to_hub()` to enable parallel shard uploads using `multiprocessing` when `num_proc > 1`.

@lhoestq @SwayStar123 

> Hi @SwayStar123 
> 
> I'd be interested in taking this up. I plan to add a `num_proc` parameter to `push_to_hub()` and use parallel uploads for shards using `concurrent.futures`. Will explore whether `ThreadPoolExecutor` or `ProcessPoolExecutor` is more suitable based on current implementation. Let me know if that sounds good!
> 

Hey thanks for working on it. But I'm not a hf dev so I don't know the best way to do it.

done in https://github.com/huggingface/datasets/pull/7606
ValueError: Invalid pattern: '**' can only be an entire path component [Colab]
State: closed

### Describe the bug

I have a dataset on HF [here](https://huggingface.co/datasets/kambale/luganda-english-parallel-corpus) that i've previously used to train a translation model [here](https://huggingface.co/kambale/pearl-11m-translate).

now i changed a few hyperparameters to increase number of tokens for the model, increase Transformer layers, and all

however, when i try to load the dataset, this error keeps coming up.. i have tried everything.. i have re-written the code a hundred times, and this keep coming up

### Steps to reproduce the bug

Imports:

```bash
!pip install datasets huggingface_hub fsspec
```

Python code:

```python
from datasets import load_dataset

HF_DATASET_NAME = "kambale/luganda-english-parallel-corpus"

# Load the dataset
try:
    if not HF_DATASET_NAME or HF_DATASET_NAME == "YOUR_HF_DATASET_NAME":
        raise ValueError(
            "Please provide a valid Hugging Face dataset name."
        )

    dataset = load_dataset(HF_DATASET_NAME)

# Omitted code as the error happens on the line above 

except ValueError as ve:
    print(f"Configuration Error: {ve}")
    raise
except Exception as e:
    print(f"An error occurred while loading the dataset '{HF_DATASET_NAME}': {e}")
    raise e
```

now, i have tried going through this [issue](https://github.com/huggingface/datasets/issues/6737) and nothing helps 

### Expected behavior

loading the dataset successfully and perform splits (train, test, validation)

### Environment info

from the imports, i do not install specific versions of these libraries, so the latest or available version is installed

* `datasets` version: latest
* `Platform`: Google Colab
* `Hardware`: NVIDIA A100 GPU
* `Python` version: latest
* `huggingface_hub` version: latest
* `fsspec` version: latest

Comments:
Could you please run the following code snippet in your environment and share the exact output? This will help check for any compatibility issues within the env itself. 

```
import datasets
import huggingface_hub
import fsspec

print("datasets version:", datasets.__version__)
print("huggingface_hub version:", huggingface_hub.__version__)
print("fsspec version:", fsspec.__version__)
```

```bash
datasets version: 2.14.4
huggingface_hub version: 0.31.4
fsspec version: 2025.3.2
```

Version 2.14.4 is not the latest version available, in fact it is from August 08, 2023 (you can check here: https://pypi.org/project/datasets/#history)

Use pip install datasets==3.6.0 to install a more recent version (from May 7, 2025)

I also had the same problem with Colab, after updating to the latest version it was solved.

I hope it helps

thank you @CleitonOERocha. it sure did help.

updating `datasets` to v3.6.0 and keeping `fsspec` on v2025.3.2 eliminates the issue.

Very helpful, thank you!
Avoid multiple default config names
State: closed

Fix duplicating default config names.

Currently, when calling `push_to_hub(set_default=True` with 2 different config names, both are set as default. 

Moreover, this will generate an error next time we try to push another default config name, raised by `MetadataConfigs.get_default_config_name`:
https://github.com/huggingface/datasets/blob/da1db8a5b89fc0badaa0f571b36e122e52ae8c61/src/datasets/arrow_dataset.py#L5757
https://github.com/huggingface/datasets/blob/da1db8a5b89fc0badaa0f571b36e122e52ae8c61/src/datasets/utils/metadata.py#L186-L188

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7585). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add LMDB format support
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

Add LMDB format support for large memory-mapping files

### Motivation

Add LMDB format support for large memory-mapping files

### Your contribution

I'm trying to add it

Comments:
Hi ! Can you explain what's your use case ? Is it about converting LMDB to Dataset objects (i.e. converting to Arrow) ?
load_dataset type stubs reject List[str] for split parameter, but runtime supports it
State: closed

### Describe the bug

The [load_dataset](https://huggingface.co/docs/datasets/v3.6.0/en/package_reference/loading_methods#datasets.load_dataset) method accepts a `List[str]` as the split parameter at runtime, however, the current type stubs restrict the split parameter to `Union[str, Split, None]`. This causes type checkers like Pylance to raise `reportArgumentType` errors when passing a list of strings, even though it works as intended at runtime.

### Steps to reproduce the bug

1. Use load_dataset with multiple splits e.g.:
```
from datasets import load_dataset

ds_train, ds_val, ds_test = load_dataset(
    "Silly-Machine/TuPyE-Dataset",
    "binary",
    split=["train[:75%]", "train[75%:]", "test"]
)
```
2. Observe that code executes correctly at runtime and Pylance raises `Argument of type "List[str]" cannot be assigned to parameter "split" of type "str | Split | None"`

### Expected behavior

The type stubs for [load_dataset](https://huggingface.co/docs/datasets/v3.6.0/en/package_reference/loading_methods#datasets.load_dataset) should accept `Union[str, Split, List[str], None]` or more specific overloads for the split parameter to correctly represent runtime behavior. 

### Environment info

- `datasets` version: 3.6.0
- Platform: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39
- Python version: 3.12.7
- `huggingface_hub` version: 0.32.0
- PyArrow version: 20.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2025.3.0

Comments:
Requesting a specific split (eg: test) still downloads all (train, test, val) data when streaming=False.
State: open

### Describe the bug

When using load_dataset() from the datasets library (in load.py), specifying a particular split (e.g., split="train") still results in downloading data for all splits when streaming=False. This happens during the builder_instance.download_and_prepare() call.
This behavior leads to unnecessary bandwidth usage and longer download times, especially for large datasets, even if the user only intends to use a single split.

### Steps to reproduce the bug

dataset_name = "skbose/indian-english-nptel-v0"
dataset = load_dataset(dataset_name, token=hf_token, split="test")


### Expected behavior

Optimize the download logic so that only the required split is downloaded when streaming=False when a specific split is provided.

### Environment info

Dataset: skbose/indian-english-nptel-v0
Platform: M1 Apple Silicon
Python verison: 3.12.9
datasets>=3.5.0

Comments:
Hi ! There was a PR open to improve this: https://github.com/huggingface/datasets/pull/6832 
but it hasn't been continued so far.

It would be a cool improvement though !

Been having this problem with datasets and dataloader for a while.
arrow_schema is not compatible with list
State: closed

### Describe the bug

```
import datasets
f = datasets.Features({'x': list[datasets.Value(dtype='int32')]})
f.arrow_schema

Traceback (most recent call last):
  File "datasets/features/features.py", line 1826, in arrow_schema
    return pa.schema(self.type).with_metadata({"huggingface": json.dumps(hf_metadata)})
                     ^^^^^^^^^
  File "datasets/features/features.py", line 1815, in type
    return get_nested_type(self)
           ^^^^^^^^^^^^^^^^^^^^^
  File "datasets/features/features.py", line 1252, in get_nested_type
    return pa.struct(
           ^^^^^^^^^^
  File "pyarrow/types.pxi", line 5406, in pyarrow.lib.struct
  File "pyarrow/types.pxi", line 3890, in pyarrow.lib.field
  File "pyarrow/types.pxi", line 5918, in pyarrow.lib.ensure_type
TypeError: DataType expected, got <class 'list'>
```

The following works
```
f = datasets.Features({'x': datasets.LargeList(datasets.Value(dtype='int32'))})
```

### Expected behavior

according to https://github.com/huggingface/datasets/blob/458f45a22c3cc9aea5f442f6f519333dcfeae9b9/src/datasets/features/features.py#L1765 python list should be a valid type specification for features

### Environment info

- `datasets` version: 3.5.1
- Platform: macOS-15.5-arm64-arm-64bit
- Python version: 3.12.9
- `huggingface_hub` version: 0.30.2
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Comments:
Thanks for reporting, I'll look into it

Actually it looks like you just forgot parenthesis:

```diff
- f = datasets.Features({'x': list[datasets.Value(dtype='int32')]})
+ f = datasets.Features({'x': list([datasets.Value(dtype='int32')])})
```

or simply using the `[ ]` syntax:

```python
f = datasets.Features({'x':[datasets.Value(dtype='int32')]})
```

I'm closing this issue if you don't mind

Ah is that what the syntax is? I don't think I was able to find an actual example of it so I assumed it was in the same way that you specify types eg. `list[int]`. This is good to know, thanks.
Fix regex library warnings
State: closed

# PR Summary
This small PR resolves the regex library warnings showing starting Python3.11:
```python
DeprecationWarning: 'count' is passed as positional argument
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7576). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Missing multilingual directions in IWSLT2017 dataset's processing script
State: open

### Describe the bug

Hi, 

Upon using `iwslt2017.py` in `IWSLT/iwslt2017` on the Hub for loading the datasets, I am unable to obtain the datasets for the language pairs `de-it`, `de-ro`, `de-nl`, `it-de`, `nl-de`, and `ro-de` using it. These 6 pairs do not show up when using `get_dataset_config_names()` to obtain the list of all the configs present in `IWSLT/iwslt2017`. This should not be the case since as mentioned in their original paper (please see https://aclanthology.org/2017.iwslt-1.1.pdf), the authors specify that "_this year we proposed the multilingual translation between any pair of languages from {Dutch, English, German, Italian, Romanian}..._" and because these datasets are indeed present in `data/2017-01-trnmted/texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.zip`.

Best Regards,
Anand

### Steps to reproduce the bug

Check the output of `get_dataset_config_names("IWSLT/iwslt2017", trust_remote_code=True)`: only 24 language pairs are present and the following 6 config names are absent: `iwslt2017-de-it`, `iwslt2017-de-ro`, `iwslt2017-de-nl`, `iwslt2017-it-de`, `iwslt2017-nl-de`, and `iwslt2017-ro-de`.

### Expected behavior

The aforementioned 6 language pairs should also be present and hence, all these 6 language pairs' IWSLT2017 datasets must also be available for further use. 

I would suggest removing `de` from the `BI_LANGUAGES` list and moving it over to the `MULTI_LANGUAGES` list instead in `iwslt2017.py` to account for all the 6 missing language pairs (the same `de-en` dataset is present in both `data/2017-01-trnmted/texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.zip` and `data/2017-01-trnted/texts/de/en/de-en.zip` but the `de-ro`, `de-nl`, `it-de`, `nl-de`, and `ro-de` datasets are only present in  `data/2017-01-trnmted/texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.zip`: so, its unclear why the following comment: _`# XXX: Artificially removed DE from here, as it also exists within bilingual data`_ has been added as `L71` in `iwslt2017.py`). The `README.md` file in `IWSLT/iwslt2017`must then be re-created using `datasets-cli test path/to/iwslt2017.py --save_info --all_configs` to pass all split size verification checks for the 6 new language pairs which were previously non-existent. 

### Environment info

- `datasets` version: 3.5.0
- Platform: Linux-6.8.0-56-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.30.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Comments:
I have opened 2 PRs on the Hub: `https://huggingface.co/datasets/IWSLT/iwslt2017/discussions/7` and `https://huggingface.co/datasets/IWSLT/iwslt2017/discussions/8` to resolve this issue

cool ! I pinged the owners of the dataset on HF to merge your PRs :)
No Samsum dataset
State: closed

### Describe the bug

https://huggingface.co/datasets/Samsung/samsum dataset not found error 404

Originated from https://github.com/meta-llama/llama-cookbook/issues/948

### Steps to reproduce the bug

go to website https://huggingface.co/datasets/Samsung/samsum
see the error

also downloading it with python throws 
```
Couldn't find 'Samsung/samsum' on the Hugging Face Hub either: FileNotFoundError: Samsung/samsum@f00baf5a7d4abfec6820415493bcb52c587788e6/samsum.py (repository not found)
```

### Expected behavior

Dataset exists

### Environment info

```
- `datasets` version: 3.2.0
- Platform: macOS-15.4.1-arm64-arm-64bit
- Python version: 3.12.2
- `huggingface_hub` version: 0.26.5
- PyArrow version: 16.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
```

Comments:
According to the following https://huggingface.co/posts/seawolf2357/424129432408590, as of now the dataset seems to be inaccessible.

@IgorKasianenko, would https://huggingface.co/datasets/knkarthick/samsum suffice for your purpose?


Thanks @SP1029 for the update!
That will work for now, using it as replacement. Is there a officially recommended way to maintain the CC licensed dataset under the organization account? 
Feel free to close this issue

> Is there an officially recommended way to maintain a CC-licensed dataset under an organizational account?

@IgorKasianenko, apologies, this is not my area of expertise.

> Please feel free to close this issue.

I have limited access and may not be able to do that. Since you opened it, you would be able to close it.

dataset_samsum = load_dataset("knkarthick/samsum")

is working
add check if repo exists for dataset uploading
State: closed

Currently, I'm reuploading datasets for [`MTEB`](https://github.com/embeddings-benchmark/mteb/). Some of them have many splits (more than 20), and I'm encountering the error:
`Too many requests for https://huggingface.co/datasets/repo/create`.

It seems that this issue occurs because the dataset tries to recreate itself every time a split is uploaded. To resolve this, I've added a check to ensure that if the dataset already exists, it won't attempt to recreate it.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7565). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq Can you review, please? I don't think that errors in CI are related to my changes

Similar implementation done in https://github.com/huggingface/datasets/pull/7708