noamaanMulla-03/datasets-issues huggingface dataset:
`map` with any joblib backend

We recently enabled the (experimental) parallel backend switch for data download and extraction but not for `map` yet.

Right now we're using our `iflatmap_unordered` implementation for multiprocessing that uses a shared Queue to gather progress updates from the subprocesses and show a progress bar in the main process.

If a Queue implementation that would work on any joblib backend by leveraging the filesystem that is shared among workers, we can have `iflatmap_unordered` for joblib and therefore a `map` with any joblib backend with a progress bar !

Note that the Queue doesn't need to be that optimized though since we can choose a small frequency for progress updates (like 1 update per second).

Labels: name: enhancement

#self-assign

Hi @lhoestq 👋🏼

I’d like to work on this!

Planning to support progress tracking with `map()` using any joblib backend (like "loky") by replacing the Queue-based approach in `iflatmap_unordered` with a file-based progress tracking mechanism (e.g. shared temp file with periodic updates).

This would allow the progress bar to work even in backends where inter-process Queues aren't supported. Let me know if this sounds good — I’ll get started!


I think ideally it should have a general solution, since some joblib backends don't have a shared filesystem
Add num_proc parameter to push_to_hub

### Feature request

A number of processes parameter to the dataset.push_to_hub method

### Motivation

Shards are currently uploaded serially which makes it slow for many shards, uploading can be done in parallel and much faster


Labels: name: enhancement

Hi @SwayStar123 

I'd be interested in taking this up. I plan to add a `num_proc` parameter to `push_to_hub()` and use parallel uploads for shards using `concurrent.futures`. Will explore whether `ThreadPoolExecutor` or `ProcessPoolExecutor` is more suitable based on current implementation. Let me know if that sounds good!


Just a quick update — `push_to_hub()` already had the `num_proc` argument in its signature and was correctly passing it internally to `_push_parquet_shards_to_hub()`.

The actual change required was inside `_push_parquet_shards_to_hub()` to enable parallel shard uploads using `multiprocessing` when `num_proc > 1`.

@lhoestq @SwayStar123 

> Hi @SwayStar123 
> 
> I'd be interested in taking this up. I plan to add a `num_proc` parameter to `push_to_hub()` and use parallel uploads for shards using `concurrent.futures`. Will explore whether `ThreadPoolExecutor` or `ProcessPoolExecutor` is more suitable based on current implementation. Let me know if that sounds good!
> 

Hey thanks for working on it. But I'm not a hf dev so I don't know the best way to do it.

done in https://github.com/huggingface/datasets/pull/7606
Remove scripts altogether

TODO:
- [x] remplace fixtures based on script with no-script fixtures
- [x] windaube


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7592). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @lhoestq,
I wanted to ask
are you planning to stop supporting dataset builds using `GeneratorBasedBuilder`?

If so, could you share the reason why?

We stopped supporting dataset scripts altogether, whether they are based on GeneratorBasedBuilder or any other builder. This means you can't `load_dataset()` a dataset script anymore. We did this mostly for security reasons which is blocking for many users and also impossible to build upon (e.g. the for the Dataset Viewer on HF)

Ah, so only the `trust_remote_code` feature of `load_dataset` is deprecated, and

```python
from datasets import load_dataset_builder
  
builder = load_dataset_builder('cornell-movie-review-data/rotten_tomatoes')  
builder.download_and_prepare()  
```

we can still load data using `load_dataset_builder` and `download_and_prepare`, right?
That's a relief. I thought the removal of `trust_remote_code` in `load_dataset` meant `GeneratorBasedBuilder` was being deprecated too, haha.
Got it, thanks for the clarification!


Can you give an example on how to upgrade from using `trust_remote_code`? I used to load_dataset from a script generating my training data in a streaming way.

For guys who dislike this change +1

> Can you give an example on how to upgrade from using trust_remote_code? I used to load_dataset from a script generating my training data in a streaming way.

here is a command to convert a script based dataset on HF to a Parquet dataset:

```
uv run --with "datasets==3.6.0" datasets-cli convert_to_parquet <username/dataset-name> --trust_remote_code
```

And if you want to define a streaming pipeline, feel free to use the `IterableDataset` API instead:

```python
from datasets import load_dataset

ds = load_dataset(..., streaming=True)  # source data
ds = ds.map(...)  # preprocessing
ds = ds.filter(...)  # remove bad samples

# For training
# data_loader = DataLoader(ds, ...)

# Saving the dataset
# ds.push_to_hub(...)
```
docs: Add column overwrite example to batch mapping guide

This PR adds a complementary example showing the **column-overwriting** pattern, which is both more direct and more flexible for many transformations.

### Proposed Change

The original `remove_columns` example remains untouched. Below it, this PR introduces an alternative approach that overwrites an existing column during batch mapping.

This teaches users a core `.map()` capability for in-place transformations without extra intermediate steps.

**New Example:**

> ```python
> >>> from datasets import Dataset
> >>> dataset = Dataset.from_dict({"a": [0, 1, 2]})
> # Overwrite "a" directly to duplicate each value
> >>> duplicated_dataset = dataset.map(
> ...     lambda batch: {"a": [x for x in batch["a"] for _ in range(2)]},
> ...     batched=True
> ... )
> >>> duplicated_dataset
> Dataset({
>     features: ['a'],
>     num_rows: 6
> })
> >>> duplicated_dataset["a"]
> [0, 0, 1, 1, 2, 2]
> ```



Hi @lhoestq, just a gentle follow-up on this PR.
Add columns support to JSON loader for selective key filtering

Fixes #7594 
This PR adds support for filtering specific columns when loading datasets from .json or .jsonl files — similar to how the columns=... argument works for Parquet.

As suggested, support for the `columns=...` argument (previously available for Parquet) has now been extended to **JSON and JSONL** loading via `load_dataset(...)`. You can now load only specific keys/columns and skip the rest — which should help in cases where some fields are unclean, inconsistent, or just unnecessary.

### Example:

```python
from datasets import load_dataset

dataset = load_dataset("json", data_files="your_data.jsonl", columns=["id", "title"])
print(dataset["train"].column_names)
# Output: ['id', 'title']
```

### Summary of changes:

* Added `columns: Optional[List[str]]` to `JsonConfig`
* Updated `_generate_tables()` to filter selected columns
* Forwarded `columns` argument from `load_dataset()` to the config
* Added test for validation(should be fine!)

Let me know if you'd like the same to be added for CSV or others as a follow-up — happy to help.


I need this feature right now. It would be great if it could automatically fill in None for non-existent keys instead of reporting an error.

> I need this feature right now. It would be great if it could automatically fill in None for non-existent keys instead of reporting an error.

Hi @aihao2000, Just to confirm — I have done the changes you asked for!
If you pass columns=["key1", "key2", "optional_key"] to load_dataset(..., columns=...), and any of those keys are missing from the input JSON objects, the loader will automatically fill those columns with None values, instead of raising an error.

Hi! any update on this PR?
Add wikipedia-2023-redirects dataset

Title: Add wikipedia-2023-redirects dataset (redirect resolution + pageviews)

Summary
- New dataset loader: wikipedia_2023_redirects
- Canonical Wikipedia pages enriched with:
  - redirects (aliases pointing to the page)
  - 2023 pageviews (aggregated)
- Streaming support; robust parsing; license notes included
- Tests with tiny dummy data (XML + TSVs); covers streaming

Motivation
RAG/retrieval often benefits from:
- Query expansion via redirect aliases
- Popularity prior via pageviews
This loader offers a practical, maintenance-light way to access canonical pages alongside their redirect aliases and 2023 pageview totals.

Features
- id: string
- title: string
- url: string
- text: string
- redirects: list[string]
- pageviews_2023: int32
- timestamp: string

Licensing
- Wikipedia text: CC BY-SA 3.0 (attribution and share-alike apply)
- Pageviews: public domain
The PR docs mention both, and the module docstring cites sources.

Notes
- The URLs in _get_urls_for_config are wired to dummy files for tests. In production, these would point to Wikimedia dumps:
  - XML page dumps: https://dumps.wikimedia.org/
  - Pageviews: https://dumps.wikimedia.org/other/pageviews/
- The schema is intentionally simple and stable. Pageview aggregation is per-title sum across 2023.

Testing
- make style && make quality
- pytest -q tests/test_dataset_wikipedia_2023_redirects.py

Example
```python
from datasets import load_dataset
ds = load_dataset("wikipedia_2023_redirects", split="train")
print(ds[0]["title"], ds[0]["redirects"][:5], ds[0]["pageviews_2023"])
```

Acknowledgements
- Wikipedia/Wikimedia Foundation for the source data
- Hugging Face Datasets for the dataset infrastructure



you should host this dataset on HF with `ds.push_to_hub()` ! we stopped using dataset scripts some time ago
module 'pyarrow' has no attribute 'PyExtensionType'

### Describe the bug

When importing certain libraries, users will encounter the following error which can be traced back to the datasets library.

module 'pyarrow' has no attribute 'PyExtensionType'.

Example issue: https://github.com/explodinggradients/ragas/issues/2170

The issue occurs due to the following. I will proceed to submit a PR with the below fix:

**Issue Reason**
The issue is that PyArrow version 21.0.0 doesn’t have PyExtensionType. This was changed in newer versions of PyArrow. The

PyExtensionType class was renamed to ExtensionType in PyArrow 13.0.0 and later versions.


** Issue Solution**
Making the following changes to the following lib files should temporarily resolve the issue.

I will submit a PR to the dataets library in the meantime.

env_name/lib/python3.10/site-packages/datasets/features/features.py:

```
> 521            self.shape = tuple(shape)
       522            self.value_type = dtype
       523            self.storage_dtype = self._generate_dtype(self.value_type)
       524 -          pa.PyExtensionType.__init__(self, self.storage_dtype)
       524 +          pa.ExtensionType.__init__(self, self.storage_dtype)
       525
       526        def __reduce__(self):
       527            return self.__class__, (
```

Updated venv_name/lib/python3.10/site-packages/datasets/features/features.py:
```      
 510        _type: str = field(default=“Array5D”, init=False, repr=False)
       511
       512
       513 -  class _ArrayXDExtensionType(pa.PyExtensionType):
       513 +  class _ArrayXDExtensionType(pa.ExtensionType):
       514        ndims: Optional[int] = None
       515
       516        def __init__(self, shape: tuple, dtype: str):
```

### Steps to reproduce the bug

Ragas version: 0.3.1
Python version: 3.11

**Code to Reproduce**

_**In notebook:**_

!pip install ragas
from ragas import evaluate


### Expected behavior

The required package installs without issue.

### Environment info

In Jupyter Notebook.
venv


Just checked out the files and thishad already been addressed

For others who find this issue: 

`pip install --upgrade "datasets>=2.20.0"` 

from https://github.com/explodinggradients/ragas/issues/2170#issuecomment-3204393672 can fix it.
Comment/feature request: Huggingface 502s from GHA

This is no longer a pressing issue, but for completeness I am reporting that in August 26th, GET requests to `https://datasets-server.huggingface.co/info\?dataset\=livebench/math` were returning 502s when invoked from [github actions](https://github.com/UKGovernmentBEIS/inspect_evals/actions/runs/17241892475/job/48921123754) (that link will expire eventually, [here are the logs](https://github.com/user-attachments/files/22233578/logs_44225296943.zip)). 

When invoked from actions, it appeared to be consistently failing for ~6 hours. However, these 502s never occurred when the request was invoked from my local machine in that same time period.

I suspect that this is related to how the requests are routed with github actions versus locally. 

Its not clear to me if the request even reached huggingface servers or if its the github proxy that stopped it from going through, but I wanted to report it nonetheless in case this is helpful information. I'm curious if huggingface can do anything on their end to confirm cause. 

And a feature request for if this happens in the future (assuming huggingface has visibilty on it):  A "datasets status" page highlighting if 502s occur for  specific individual datasets could be useful for people debugging on the other end of this!
Replacement of "Sequence" feature with "List" breaks backward compatibility

PR #7634 replaced the Sequence feature with List in 4.0.0, so datasets saved with version 4.0.0 with that feature cannot be loaded by earlier versions. There is no clear option in 4.0.0 to use the legacy feature type to preserve backward compatibility.

Why is this a problem? I have a complex preprocessing and training pipeline dependent on 3.6.0; we manage a very large number of separate datasets that get concatenated during training. If just one of those datasets is saved with 4.0.0, they become unusable, and we have no way of "fixing" them. I can load them in 4.0.0 but I can't re-save with the legacy feature type, and I can't load it in 3.6.0 for obvious reasons.

Perhaps I'm missing something here, since the PR says that backward compatibility is preserved; if so, it's not obvious to me how.


Backward compatibility here means 4.0.0 can load datasets saved with older versions.

You will need 4.0.0 to load datasets saved with 4.0.0
Dill version update

### Describe the bug

Why the datasets is not updating the dill ? 
Just want to know if I update the dill version in dill what will be the repucssion. 
For now in multiplaces I have to update the library like process requirequire dill 0.4.0 so why not datasets. 
Adding a pr too. 

### Steps to reproduce the bug

.

### Expected behavior

.

### Environment info

.


#7752 

related: #7510
Mosaic Streaming (MDS) Support

### Feature request

I'm a huge fan of the current HF Datasets `webdataset` integration (especially the built-in streaming support). However, I'd love to upload some robotics and multimodal datasets I've processed for use with [Mosaic Streaming](https://docs.mosaicml.com/projects/streaming/en/stable/), specifically their [MDS Format](https://docs.mosaicml.com/projects/streaming/en/stable/fundamentals/dataset_format.html#mds). 

Because the shard files have similar semantics to WebDataset, I'm hoping that adding such support won't be too much trouble?

### Motivation

One of the downsides with WebDataset is a lack of out-of-the-box determinism (especially for large-scale training and reproducibility), easy job resumption, and the ability to quickly debug / visualize individual examples. 

Mosaic Streaming provides a [great interface for this out of the box](https://docs.mosaicml.com/projects/streaming/en/stable/#key-features), so I'd love to see it supported in HF Datasets.

### Your contribution

Happy to help test things / provide example data. Can potentially submit a PR if maintainers could point me to the necessary WebDataset logic / steps for adding a new streaming format!

Labels: name: enhancement

Hi ! that would be great :) Though note that `datasets` doesn't implement format-specific resuming when streaming, so in general I think it's better if users can use the mosaic-streaming library to read their MDS datasets. I wonder if they support `hf://` paths though...

Anyway for those interested, the code for WebDataset is a single file here: https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/webdataset/webdataset.py.

It implements `_split_generators` that downloads files and returns the lists of splits (train/validation/test) and `_split_generators` to generate examples (dicts) from the downloaded files. Streaming is automatically supported by making download steps lazy and by extending `open()` to work with remote URLs.

edit: we could try with datasets like https://huggingface.co/datasets/jhu-clsp/mmBERT-pretrain-p2-fineweb2-remaining
add check if repo exists for dataset uploading

Currently, I'm reuploading datasets for [`MTEB`](https://github.com/embeddings-benchmark/mteb/). Some of them have many splits (more than 20), and I'm encountering the error:
`Too many requests for https://huggingface.co/datasets/repo/create`.

It seems that this issue occurs because the dataset tries to recreate itself every time a split is uploaded. To resolve this, I've added a check to ensure that if the dataset already exists, it won't attempt to recreate it.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7565). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq Can you review, please? I don't think that errors in CI are related to my changes

Similar implementation done in https://github.com/huggingface/datasets/pull/7708
Parquet: use data page v2 for efficient page pruning

This is needed to enable page pruning with DataFusion, which will be useful for the Dataset Viewer.

Indeed page pruning with DataFusion allows to download only certain pages of a row group, reducing the I/O required to read just a few rows.

But while data page v1 generally works, it's not easy with DataFusion to do page pruning on datasets with nested data. This is because rows can span multiple pages in v1, contrary to v2.

cc @severo for viz


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7762). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Closing this since it looks like the page offset index is enough :)
add support for pyarrow string view in features



@lhoestq who do you think would be the best to have a look at this? Any pointers would be appreciated, thanks!

Hi ! what's the rationale for supporting string view ? I'm afraid it can complexify the typing logic without much value

Hi @lhoestq ! I mainly want to be able to create features by using `Features.from_arrow_schema(dataset_schema)` on an arrow dataset with string view columns, currently there is no easy way to do this, and string_view is becoming an increasingly common data type for string columns in arrow. Thanks for having a look!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7718). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix: Correct float feature generation in `generate_examples`

This PR fixes a bug in the `generate_examples` function where `datasets.Value` features with a `float` dtype were incorrectly generated using `np.random.randint`. This resulted in integer values being cast to float, which is not representative of true floating-point data.

**Key changes include:**
1.  Added explicit handling for float features using `np.random.rand` to generate continuous values.
2.  Introduced fail-fast type checks for unsupported dtypes to improve robustness.
3.  Added validation for sequence features to ensure `seq_shapes` is provided.


### Before Fix
Float features were generated incorrectly as integers cast to float:

```text
Example 0:
int_feature: 0
float_feature: 9.0  <-- Incorrect: An integer disguised as a float
string_feature: The small grey turtle was surprisingly fast...
seq_feature: [0.3048 0.4291 0.4283]
```
### After Fix
Float features are now correctly generated as continuous numbers in the range [0, 1):
```text

Example 0:
int_feature: 0
float_feature: 0.0183  <-- Correct: A true random float
string_feature: The small grey turtle was surprisingly fast...
seq_feature: [0.9237 0.7972 0.8526]
Bump dill to 0.4.0

This bumps `dill` to 0.3.9 and closes #7510 

It turns out the only thing required to make the tests pass was to extend the version checks to include 0.3.9.


Have you tried to run `pytest tests/test_fingerprint.py` ? It seems dill 0.3.9 breaks a lot of tests

```
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_regex - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_tokenizer - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_tokenizer_with_cache - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::RecurseHashTest::test_hash_ignores_line_definition_of_function - AssertionError: 'c48ebfacf8768f50' != '27e49d047c02c83b'
FAILED tests/test_fingerprint.py::RecurseHashTest::test_hash_ipython_function - AssertionError: '65edc6b6d425a8e9' != '9f364fe298fb286a'
FAILED tests/test_fingerprint.py::HashingTest::test_hash_tiktoken_encoding - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_compiled_module - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_generator - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_tensor - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_set_doesnt_depend_on_order - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_set_stable - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::test_move_script_doesnt_change_hash - AssertionError: assert b'93072ca404a697db\n' == b'cf89a7e497a97e32\n'
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7763). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @lhoestq! Yes, I did. It's not really `dill` that breaks things. Rather the shims that `datasets` has in place did not include the next version. 

FYI: I also tested it with `dill-0.4.0` and the changes would need to be analogous, but I wanted to be conservative in this PR. 

The NameError is fixed in your PR since it defines the right `log()` function for 0.3.9.

But I'm less sure about the AssertionError that may be related to deterministic hashing or ipython/shell function hashing. We would need to solve these

EDIT: ah actually it does ! cool ! let me update the branch and re-run the CI