Minimartzz/github-issues huggingface dataset:
cast columns to Image/Audio/Video with `storage_options`
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

Allow `storage_options` to be passed in
1. `cast` related operations (e.g., `cast_columns, cast`)
2. `info` related reading (e.g., `from_dict, from_pandas, from_polars`) together with `info.features`
```python3
import datasets

image_path = "s3://bucket/sample.png"
dataset = datasets.Dataset.from_dict({"image_path": [image_path]})

# dataset = dataset.cast_column("image_path", datasets.Image())  # now works without `storage_options`

# expected behavior
dataset = dataset.cast_column("image_path", datasets.Image(), storage_options={"anon": True})
```

### Motivation

I'm using my own registered fsspec filesystem (s3 with customized local cache support). I need to pass cache folder paths `cache_dirs: list[str]` to the filesystem when I read the remote images (cast from file_paths).

### Your contribution

Could help with a PR at weekends

Comments:
A

1

1

Ok

> ### Feature request
> Allow `storage_options` to be passed in
> 
> 1. `cast` related operations (e.g., `cast_columns, cast`)
> 2. `info` related reading (e.g., `from_dict, from_pandas, from_polars`) together with `info.features`
> 
> import datasets
> 
> image_path = "s3://bucket/sample.png"
> dataset = datasets.Dataset.from_dict({"image_path": [image_path]})
> 
> # dataset = dataset.cast_column("image_path", datasets.Image())  # now works without `storage_options`
> 
> # expected behavior
> dataset = dataset.cast_column("image_path", datasets.Image(), storage_options={"anon": True})
> ### Motivation
> I'm using my own registered fsspec filesystem (s3 with customized local cache support). I need to pass cache folder paths `cache_dirs: list[str]` to the filesystem when I read the remote images (cast from file_paths).
> 
> ### Your contribution
> Could help with a PR at weekends



>
polars dataset cannot cast column to Image/Audio/Video
State: closed

### Describe the bug

`from_polars` dataset cannot cast column to Image/Audio/Video, while it works on `from_pandas` and `from_dict`

### Steps to reproduce the bug

```python3
import datasets
import pandas as pd
import polars as pl

image_path = "./sample.png"

# polars
df = pl.DataFrame({"image_path": [image_path]})
dataset = datasets.Dataset.from_polars(df)
dataset = dataset.cast_column("image_path", datasets.Image())

# # raises Error
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from large_string to struct using function cast_struct


# pandas 
df = pd.DataFrame({"image_path": [image_path]})
dataset = datasets.Dataset.from_pandas(df)
dataset = dataset.cast_column("image_path", datasets.Image())

# # pass
{'image_path': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=338x277 at 0x7FBA719D4050>}


# dict
dataset = datasets.Dataset.from_dict({"image_path": [image_path]})
dataset = dataset.cast_column("image_path", datasets.Image())

# # pass
{'image_path': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=338x277 at 0x7FBA719D4050>}
```


### Expected behavior

`from_polars` case shouldn't raise error and have the same outputs as `from_pandas` and `from_dict`

### Environment info

```
# Name                    Version                   Build  Channel
datasets                  4.0.0                    pypi_0    pypi
pandas                    2.3.1                    pypi_0    pypi
polars                    1.32.3                   pypi_0    pypi
```

Comments:
I fixed this with a combination of `to_dict` and `from_dict`:

```py
datasets.Dataset.from_dict(df.to_dict(as_series=False))
```

@samuelstevens Yeah, I'm using similar workaround as well. But it would be ideal if we can avoid the copy.
Bump dill to 0.4.0
State: closed

This bumps `dill` to 0.3.9 and closes #7510 

It turns out the only thing required to make the tests pass was to extend the version checks to include 0.3.9.

Comments:
Have you tried to run `pytest tests/test_fingerprint.py` ? It seems dill 0.3.9 breaks a lot of tests

```
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_regex - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_tokenizer - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_tokenizer_with_cache - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::RecurseHashTest::test_hash_ignores_line_definition_of_function - AssertionError: 'c48ebfacf8768f50' != '27e49d047c02c83b'
FAILED tests/test_fingerprint.py::RecurseHashTest::test_hash_ipython_function - AssertionError: '65edc6b6d425a8e9' != '9f364fe298fb286a'
FAILED tests/test_fingerprint.py::HashingTest::test_hash_tiktoken_encoding - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_compiled_module - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_generator - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_tensor - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_set_doesnt_depend_on_order - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_set_stable - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::test_move_script_doesnt_change_hash - AssertionError: assert b'93072ca404a697db\n' == b'cf89a7e497a97e32\n'
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7763). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @lhoestq! Yes, I did. It's not really `dill` that breaks things. Rather the shims that `datasets` has in place did not include the next version. 

FYI: I also tested it with `dill-0.4.0` and the changes would need to be analogous, but I wanted to be conservative in this PR. 

The NameError is fixed in your PR since it defines the right `log()` function for 0.3.9.

But I'm less sure about the AssertionError that may be related to deterministic hashing or ipython/shell function hashing. We would need to solve these

EDIT: ah actually it does ! cool ! let me update the branch and re-run the CI
Parquet: use data page v2 for efficient page pruning
State: closed

This is needed to enable page pruning with DataFusion, which will be useful for the Dataset Viewer.

Indeed page pruning with DataFusion allows to download only certain pages of a row group, reducing the I/O required to read just a few rows.

But while data page v1 generally works, it's not easy with DataFusion to do page pruning on datasets with nested data. This is because rows can span multiple pages in v1, contrary to v2.

cc @severo for viz

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7762). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Closing this since it looks like the page offset index is enough :)
Comment/feature request: Huggingface 502s from GHA
State: open

This is no longer a pressing issue, but for completeness I am reporting that in August 26th, GET requests to `https://datasets-server.huggingface.co/info\?dataset\=livebench/math` were returning 502s when invoked from [github actions](https://github.com/UKGovernmentBEIS/inspect_evals/actions/runs/17241892475/job/48921123754) (that link will expire eventually, [here are the logs](https://github.com/user-attachments/files/22233578/logs_44225296943.zip)). 

When invoked from actions, it appeared to be consistently failing for ~6 hours. However, these 502s never occurred when the request was invoked from my local machine in that same time period.

I suspect that this is related to how the requests are routed with github actions versus locally. 

Its not clear to me if the request even reached huggingface servers or if its the github proxy that stopped it from going through, but I wanted to report it nonetheless in case this is helpful information. I'm curious if huggingface can do anything on their end to confirm cause. 

And a feature request for if this happens in the future (assuming huggingface has visibilty on it):  A "datasets status" page highlighting if 502s occur for  specific individual datasets could be useful for people debugging on the other end of this!




Comments:
Option for Anonymous Dataset link
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

Allow for anonymized viewing of datasets. For instance, something similar to [Anonymous GitHub](https://anonymous.4open.science/).

### Motivation

We generally publish our data through Hugging Face. This has worked out very well as it's both our repository and archive (thanks to the DOI feature!). However, we have an increasing challenge when it comes to sharing our datasets for paper (both conference and journal) submissions. Due to the need to share data anonymously, we can't use the Hugging Face URLs, but datasets tend to be too large for inclusion as a zip. Being able to have an anonymous link would be great since we can't be double-publishing the data.

### Your contribution

Sorry, I don't have a contribution to make to the implementation of this. Perhaps it would be possible to work off the [Anonymous GitHub](https://github.com/tdurieux/anonymous_github) code to generate something analogous with pointers to the data still on Hugging Face's servers (instead of the duplication of data required for the GitHub version)?

Comments:
Add support for `.conll` file format in datasets
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

I’d like to request native support in the Hugging Face datasets library for reading .conll files (CoNLL format). This format is widely used in NLP tasks, especially for Named Entity Recognition (NER), POS tagging, and other token classification problems.

Right now `.conll` datasets need to be manually parsed or preprocessed before being loaded into datasets. Having built in support would save time and make workflows smoother for researchers and practitioners.

I propose - 
Add a conll dataset builder or file parser to datasets that can:
- Read `.conll` files with customizable delimiters (space, tab).
- Handle sentence/document boundaries (typically indicated by empty lines).
- Support common CoNLL variants (e.g., CoNLL-2000 chunking, CoNLL-2003 NER).
- Output a dataset where each example contains:
- tokens: list of strings
- tags (or similar): list of labels aligned with tokens

Given a .conll snippet like:
```
EU NNP B-ORG
rejects VBZ O
German JJ B-MISC
call NN O
. . O
```


The dataset should load as:
```
{
  "tokens": ["EU", "rejects", "German", "call", "."],
  "tags": ["B-ORG", "O", "B-MISC", "O", "O"]
}
```




### Motivation

- CoNLL files are a standard benchmark format in NLP (e.g., CoNLL-2003, CoNLL-2000).
- Many users train NER or sequence labeling models (like BERT for token classification) directly on `.conll` 
- Right now you have to write your own parsing scripts. Built in support would unify this process and would be much more convenient 

### Your contribution

I’d be happy to contribute by implementing this feature. My plan is to- 
- Add a new dataset script (conll.py) to handle .conll files.
- Implement parsing logic that supports sentence/document boundaries and token-label alignment.
- Write unit tests with small `.conll` examples to ensure correctness.
- Add documentation and usage examples so new users can easily load `.conll` datasets.

This would be my first open source contribution, so I’ll follow the `CONTRIBUTING.md` guidelines closely and adjust based on feedback from the maintainers.

Comments:
That would be cool ! feel free to ping me if I can help reviewing a PR
datasets.map(f, num_proc=N) hangs with N>1 when run on import
State: open

### Describe the bug

If you `import` a module that runs `datasets.map(f, num_proc=N)` at the top-level, Python hangs.


### Steps to reproduce the bug

1. Create a file that runs datasets.map at the top-level:

```bash
cat <<EOF > import_me.py
import datasets

the_dataset = datasets.load_dataset("openai/openai_humaneval")
the_dataset = the_dataset.map(lambda item: item, num_proc=2)
EOF
```

2. Start Python REPL:

```bash
uv run --python 3.12.3 --with "datasets==4.0.0"  python3
Python 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
```

3. Import the file:

```python
import import_me
````

Observe hang.

### Expected behavior

Ideally would not hang, or would fallback to num_proc=1 with a warning.

### Environment info


- `datasets` version: 4.0.0
- Platform: Linux-6.14.0-29-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.34.4
- PyArrow version: 21.0.0
- Pandas version: 2.3.2
- `fsspec` version: 2025.3.0


Comments:
Fix: Update Dill Version in Setup py
State: closed

Currently the DIll version is less than 3.9 and now major libraries like Multiprocess, gepa requires Dill version as 0.4.0 and this is making a conflict in installation. So added this small PR to update the DIll. 

Comments:
https://github.com/huggingface/datasets/issues/7751

same as https://github.com/huggingface/datasets/pull/7763: some tests need to be fixed to support 0.4.0

fixed by #7763
Dill version update
State: open

### Describe the bug

Why the datasets is not updating the dill ? 
Just want to know if I update the dill version in dill what will be the repucssion. 
For now in multiplaces I have to update the library like process requirequire dill 0.4.0 so why not datasets. 
Adding a pr too. 

### Steps to reproduce the bug

.

### Expected behavior

.

### Environment info

.

Comments:
#7752 

related: #7510
Fix typo in error message for cache directory deletion
State: closed

This PR fixes a small typo in an error message in `src/datasets/fingerprint.py`:

https://github.com/huggingface/datasets/blob/910fab20606893f69b4fccac5fcc883dddf5a14d/src/datasets/fingerprint.py#L63

```diff
- occured
+ occurred
```

Comments:
Add wikipedia-2023-redirects dataset
State: open

Title: Add wikipedia-2023-redirects dataset (redirect resolution + pageviews)

Summary
- New dataset loader: wikipedia_2023_redirects
- Canonical Wikipedia pages enriched with:
  - redirects (aliases pointing to the page)
  - 2023 pageviews (aggregated)
- Streaming support; robust parsing; license notes included
- Tests with tiny dummy data (XML + TSVs); covers streaming

Motivation
RAG/retrieval often benefits from:
- Query expansion via redirect aliases
- Popularity prior via pageviews
This loader offers a practical, maintenance-light way to access canonical pages alongside their redirect aliases and 2023 pageview totals.

Features
- id: string
- title: string
- url: string
- text: string
- redirects: list[string]
- pageviews_2023: int32
- timestamp: string

Licensing
- Wikipedia text: CC BY-SA 3.0 (attribution and share-alike apply)
- Pageviews: public domain
The PR docs mention both, and the module docstring cites sources.

Notes
- The URLs in _get_urls_for_config are wired to dummy files for tests. In production, these would point to Wikimedia dumps:
  - XML page dumps: https://dumps.wikimedia.org/
  - Pageviews: https://dumps.wikimedia.org/other/pageviews/
- The schema is intentionally simple and stable. Pageview aggregation is per-title sum across 2023.

Testing
- make style && make quality
- pytest -q tests/test_dataset_wikipedia_2023_redirects.py

Example
```python
from datasets import load_dataset
ds = load_dataset("wikipedia_2023_redirects", split="train")
print(ds[0]["title"], ds[0]["redirects"][:5], ds[0]["pageviews_2023"])
```

Acknowledgements
- Wikipedia/Wikimedia Foundation for the source data
- Hugging Face Datasets for the dataset infrastructure


Comments:
you should host this dataset on HF with `ds.push_to_hub()` ! we stopped using dataset scripts some time ago
Fix: Canonical 'multi_news' dataset is broken and should be updated to a Parquet version
State: open

Hi, 

The canonical `multi_news` dataset is currently broken and fails to load. This is because it points to the [alexfabri/multi_news](https://huggingface.co/datasets/alexfabbri/multi_news) repository, which contains a legacy loading script (`multi_news.py`) that requires the now-removed `trust_remote_code` parameter.

The original maintainer's GitHub and Hugging Face repositories appear to be inactive, so a community-led fix is needed.

I have created a working fix by converting the dataset to the modern Parquet format, which does not require a loading script. The fixed version is available here and loads correctly:

**[Awesome075/multi_news_parquet](https://huggingface.co/datasets/Awesome075/multi_news_parquet)**

Could the maintainers please guide me or themselves update the official `multi_news` dataset to use this working Parquet version? This would involve updating the canonical pointer for "multi_news" to resolve to the new repository.

This action would fix the dataset for all users and ensure its continued availability.

Thank you!

Comments:
@sayakpaul @a-r-r-o-w could you verify this issue then i can contribute to solve this issue!😊
Audio mono argument no longer supported, despite class documentation
State: open

### Describe the bug

Either update the documentation, or re-introduce the flag (and corresponding logic to convert the audio to mono)

### Steps to reproduce the bug

Audio(sampling_rate=16000, mono=True) raises the error 
TypeError: Audio.__init__() got an unexpected keyword argument 'mono'

However, in the class documentation, is says:
Args:
        sampling_rate (`int`, *optional*):
            Target sampling rate. If `None`, the native sampling rate is used.
        mono (`bool`, defaults to `True`):
            Whether to convert the audio signal to mono by averaging samples across
            channels.
        [...]

### Expected behavior

The above call should either work, or the documentation within the Audio class should be updated

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-5.15.0-124-generic-x86_64-with-glibc2.35
- Python version: 3.12.11
- `huggingface_hub` version: 0.34.4
- PyArrow version: 21.0.0
- Pandas version: 2.3.2
- `fsspec` version: 2025.3.0

Comments:
I want to solve this problem can you please assign it to me
and also can you please guide whether the mono parameter is required to be re-added or the documentation needs an update?
Refactor HDF5 and preserve tree structure
State: closed

Closes #7741. Followup to #7690

- Recursive parsing and feature inference, to preserve the tree structure of the file. Note this means we now visit all links in the file. It also means we have to call` combine_chunks` on any large non-root datasets.
- Support for `complex64` (two `float32`s, used to be converted to two `float64`s)
- Support for ndim complex, compound, more field types for compound (due to reusing the main parser, compound types are treated like groups)
- Cleaned up varlen support
- Always do feature inference and always cast to features (used to cast to schema)
- Updated tests to use `load_dataset` instead of internal APIs
- Removed `columns` in config. Have to give Features (i.e., must specify types) if filtering

Comments:
@lhoestq this is ready for you now!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7743). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
module 'pyarrow' has no attribute 'PyExtensionType'
State: open

### Describe the bug

When importing certain libraries, users will encounter the following error which can be traced back to the datasets library.

module 'pyarrow' has no attribute 'PyExtensionType'.

Example issue: https://github.com/explodinggradients/ragas/issues/2170

The issue occurs due to the following. I will proceed to submit a PR with the below fix:

**Issue Reason**
The issue is that PyArrow version 21.0.0 doesn’t have PyExtensionType. This was changed in newer versions of PyArrow. The

PyExtensionType class was renamed to ExtensionType in PyArrow 13.0.0 and later versions.


** Issue Solution**
Making the following changes to the following lib files should temporarily resolve the issue.

I will submit a PR to the dataets library in the meantime.

env_name/lib/python3.10/site-packages/datasets/features/features.py:

```
> 521            self.shape = tuple(shape)
       522            self.value_type = dtype
       523            self.storage_dtype = self._generate_dtype(self.value_type)
       524 -          pa.PyExtensionType.__init__(self, self.storage_dtype)
       524 +          pa.ExtensionType.__init__(self, self.storage_dtype)
       525
       526        def __reduce__(self):
       527            return self.__class__, (
```

Updated venv_name/lib/python3.10/site-packages/datasets/features/features.py:
```      
 510        _type: str = field(default=“Array5D”, init=False, repr=False)
       511
       512
       513 -  class _ArrayXDExtensionType(pa.PyExtensionType):
       513 +  class _ArrayXDExtensionType(pa.ExtensionType):
       514        ndims: Optional[int] = None
       515
       516        def __init__(self, shape: tuple, dtype: str):
```

### Steps to reproduce the bug

Ragas version: 0.3.1
Python version: 3.11

**Code to Reproduce**

_**In notebook:**_

!pip install ragas
from ragas import evaluate


### Expected behavior

The required package installs without issue.

### Environment info

In Jupyter Notebook.
venv

Comments:
Just checked out the files and thishad already been addressed

For others who find this issue: 

`pip install --upgrade "datasets>=2.20.0"` 

from https://github.com/explodinggradients/ragas/issues/2170#issuecomment-3204393672 can fix it.
Preserve tree structure when loading HDF5
State: closed
Labels: description: New feature or request
name: enhancement

### Feature request

https://github.com/huggingface/datasets/pull/7740#discussion_r2285605374

### Motivation

`datasets` has the `Features` class for representing nested features. HDF5 files have groups of datasets which are nested, though in #7690 the keys are flattened. We should preserve that structure for the user.

### Your contribution

I'll open a PR (#7743)

Comments:
Document HDF5 support
State: closed

I think these are at least the main places where we should put content. Ideally it is not just repeated in the final version


ref #7690 


- [x] Wait for #7743 to land

Comments:
@lhoestq any guidance on what else to add/feedback on what is there now? It seems a bit minimal, but I don't think it's worth doing an entire page on HDF5?
Replacement of "Sequence" feature with "List" breaks backward compatibility
State: open

PR #7634 replaced the Sequence feature with List in 4.0.0, so datasets saved with version 4.0.0 with that feature cannot be loaded by earlier versions. There is no clear option in 4.0.0 to use the legacy feature type to preserve backward compatibility.

Why is this a problem? I have a complex preprocessing and training pipeline dependent on 3.6.0; we manage a very large number of separate datasets that get concatenated during training. If just one of those datasets is saved with 4.0.0, they become unusable, and we have no way of "fixing" them. I can load them in 4.0.0 but I can't re-save with the legacy feature type, and I can't load it in 3.6.0 for obvious reasons.

Perhaps I'm missing something here, since the PR says that backward compatibility is preserved; if so, it's not obvious to me how.

Comments:
Backward compatibility here means 4.0.0 can load datasets saved with older versions.

You will need 4.0.0 to load datasets saved with 4.0.0