ahmedfarazsyk/datasets-github-issues huggingface dataset:
Comment/feature request: Huggingface 502s from GHA
Author: Scott-Simmons
State: open

This is no longer a pressing issue, but for completeness I am reporting that in August 26th, GET requests to `https://datasets-server.huggingface.co/info\?dataset\=livebench/math` were returning 502s when invoked from [github actions](https://github.com/UKGovernmentBEIS/inspect_evals/actions/runs/17241892475/job/48921123754) (that link will expire eventually, [here are the logs](https://github.com/user-attachments/files/22233578/logs_44225296943.zip)). 

When invoked from actions, it appeared to be consistently failing for ~6 hours. However, these 502s never occurred when the request was invoked from my local machine in that same time period.

I suspect that this is related to how the requests are routed with github actions versus locally. 

Its not clear to me if the request even reached huggingface servers or if its the github proxy that stopped it from going through, but I wanted to report it nonetheless in case this is helpful information. I'm curious if huggingface can do anything on their end to confirm cause. 

And a feature request for if this happens in the future (assuming huggingface has visibilty on it):  A "datasets status" page highlighting if 502s occur for  specific individual datasets could be useful for people debugging on the other end of this!
Option for Anonymous Dataset link
Author: egrace479
State: open
Labels: name: enhancement

### Feature request

Allow for anonymized viewing of datasets. For instance, something similar to [Anonymous GitHub](https://anonymous.4open.science/).

### Motivation

We generally publish our data through Hugging Face. This has worked out very well as it's both our repository and archive (thanks to the DOI feature!). However, we have an increasing challenge when it comes to sharing our datasets for paper (both conference and journal) submissions. Due to the need to share data anonymously, we can't use the Hugging Face URLs, but datasets tend to be too large for inclusion as a zip. Being able to have an anonymous link would be great since we can't be double-publishing the data.

### Your contribution

Sorry, I don't have a contribution to make to the implementation of this. Perhaps it would be possible to work off the [Anonymous GitHub](https://github.com/tdurieux/anonymous_github) code to generate something analogous with pointers to the data still on Hugging Face's servers (instead of the duplication of data required for the GitHub version)?
Add support for `.conll` file format in datasets
Author: namesarnav
State: open
Labels: name: enhancement

### Feature request

I’d like to request native support in the Hugging Face datasets library for reading .conll files (CoNLL format). This format is widely used in NLP tasks, especially for Named Entity Recognition (NER), POS tagging, and other token classification problems.

Right now `.conll` datasets need to be manually parsed or preprocessed before being loaded into datasets. Having built in support would save time and make workflows smoother for researchers and practitioners.

I propose - 
Add a conll dataset builder or file parser to datasets that can:
- Read `.conll` files with customizable delimiters (space, tab).
- Handle sentence/document boundaries (typically indicated by empty lines).
- Support common CoNLL variants (e.g., CoNLL-2000 chunking, CoNLL-2003 NER).
- Output a dataset where each example contains:
- tokens: list of strings
- tags (or similar): list of labels aligned with tokens

Given a .conll snippet like:
```
EU NNP B-ORG
rejects VBZ O
German JJ B-MISC
call NN O
. . O
```


The dataset should load as:
```
{
  "tokens": ["EU", "rejects", "German", "call", "."],
  "tags": ["B-ORG", "O", "B-MISC", "O", "O"]
}
```




### Motivation

- CoNLL files are a standard benchmark format in NLP (e.g., CoNLL-2003, CoNLL-2000).
- Many users train NER or sequence labeling models (like BERT for token classification) directly on `.conll` 
- Right now you have to write your own parsing scripts. Built in support would unify this process and would be much more convenient 

### Your contribution

I’d be happy to contribute by implementing this feature. My plan is to- 
- Add a new dataset script (conll.py) to handle .conll files.
- Implement parsing logic that supports sentence/document boundaries and token-label alignment.
- Write unit tests with small `.conll` examples to ensure correctness.
- Add documentation and usage examples so new users can easily load `.conll` datasets.

This would be my first open source contribution, so I’ll follow the `CONTRIBUTING.md` guidelines closely and adjust based on feedback from the maintainers.

Comments: That would be cool ! feel free to ping me if I can help reviewing a PR
datasets.map(f, num_proc=N) hangs with N>1 when run on import
Author: arjunguha
State: open

### Describe the bug

If you `import` a module that runs `datasets.map(f, num_proc=N)` at the top-level, Python hangs.


### Steps to reproduce the bug

1. Create a file that runs datasets.map at the top-level:

```bash
cat <<EOF > import_me.py
import datasets

the_dataset = datasets.load_dataset("openai/openai_humaneval")
the_dataset = the_dataset.map(lambda item: item, num_proc=2)
EOF
```

2. Start Python REPL:

```bash
uv run --python 3.12.3 --with "datasets==4.0.0"  python3
Python 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
```

3. Import the file:

```python
import import_me
````

Observe hang.

### Expected behavior

Ideally would not hang, or would fallback to num_proc=1 with a warning.

### Environment info


- `datasets` version: 4.0.0
- Platform: Linux-6.14.0-29-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.34.4
- PyArrow version: 21.0.0
- Pandas version: 2.3.2
- `fsspec` version: 2025.3.0
Dill version update
Author: Navanit-git
State: open

### Describe the bug

Why the datasets is not updating the dill ? 
Just want to know if I update the dill version in dill what will be the repucssion. 
For now in multiplaces I have to update the library like process requirequire dill 0.4.0 so why not datasets. 
Adding a pr too. 

### Steps to reproduce the bug

.

### Expected behavior

.

### Environment info

.

Comments: #7752 

related: #7510
Fix: Canonical 'multi_news' dataset is broken and should be updated to a Parquet version
Author: Awesome075
State: open

Hi, 

The canonical `multi_news` dataset is currently broken and fails to load. This is because it points to the [alexfabri/multi_news](https://huggingface.co/datasets/alexfabbri/multi_news) repository, which contains a legacy loading script (`multi_news.py`) that requires the now-removed `trust_remote_code` parameter.

The original maintainer's GitHub and Hugging Face repositories appear to be inactive, so a community-led fix is needed.

I have created a working fix by converting the dataset to the modern Parquet format, which does not require a loading script. The fixed version is available here and loads correctly:

**[Awesome075/multi_news_parquet](https://huggingface.co/datasets/Awesome075/multi_news_parquet)**

Could the maintainers please guide me or themselves update the official `multi_news` dataset to use this working Parquet version? This would involve updating the canonical pointer for "multi_news" to resolve to the new repository.

This action would fix the dataset for all users and ensure its continued availability.

Thank you!

Comments: @sayakpaul @a-r-r-o-w could you verify this issue then i can contribute to solve this issue!😊
Audio mono argument no longer supported, despite class documentation
Author: jheitz
State: open

### Describe the bug

Either update the documentation, or re-introduce the flag (and corresponding logic to convert the audio to mono)

### Steps to reproduce the bug

Audio(sampling_rate=16000, mono=True) raises the error 
TypeError: Audio.__init__() got an unexpected keyword argument 'mono'

However, in the class documentation, is says:
Args:
        sampling_rate (`int`, *optional*):
            Target sampling rate. If `None`, the native sampling rate is used.
        mono (`bool`, defaults to `True`):
            Whether to convert the audio signal to mono by averaging samples across
            channels.
        [...]

### Expected behavior

The above call should either work, or the documentation within the Audio class should be updated

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-5.15.0-124-generic-x86_64-with-glibc2.35
- Python version: 3.12.11
- `huggingface_hub` version: 0.34.4
- PyArrow version: 21.0.0
- Pandas version: 2.3.2
- `fsspec` version: 2025.3.0

Comments: I want to solve this problem can you please assign it to me
and also can you please guide whether the mono parameter is required to be re-added or the documentation needs an update?
module 'pyarrow' has no attribute 'PyExtensionType'
Author: mnedelko
State: open

### Describe the bug

When importing certain libraries, users will encounter the following error which can be traced back to the datasets library.

module 'pyarrow' has no attribute 'PyExtensionType'.

Example issue: https://github.com/explodinggradients/ragas/issues/2170

The issue occurs due to the following. I will proceed to submit a PR with the below fix:

**Issue Reason**
The issue is that PyArrow version 21.0.0 doesn’t have PyExtensionType. This was changed in newer versions of PyArrow. The

PyExtensionType class was renamed to ExtensionType in PyArrow 13.0.0 and later versions.


** Issue Solution**
Making the following changes to the following lib files should temporarily resolve the issue.

I will submit a PR to the dataets library in the meantime.

env_name/lib/python3.10/site-packages/datasets/features/features.py:

```
> 521            self.shape = tuple(shape)
       522            self.value_type = dtype
       523            self.storage_dtype = self._generate_dtype(self.value_type)
       524 -          pa.PyExtensionType.__init__(self, self.storage_dtype)
       524 +          pa.ExtensionType.__init__(self, self.storage_dtype)
       525
       526        def __reduce__(self):
       527            return self.__class__, (
```

Updated venv_name/lib/python3.10/site-packages/datasets/features/features.py:
```      
 510        _type: str = field(default=“Array5D”, init=False, repr=False)
       511
       512
       513 -  class _ArrayXDExtensionType(pa.PyExtensionType):
       513 +  class _ArrayXDExtensionType(pa.ExtensionType):
       514        ndims: Optional[int] = None
       515
       516        def __init__(self, shape: tuple, dtype: str):
```

### Steps to reproduce the bug

Ragas version: 0.3.1
Python version: 3.11

**Code to Reproduce**

_**In notebook:**_

!pip install ragas
from ragas import evaluate


### Expected behavior

The required package installs without issue.

### Environment info

In Jupyter Notebook.
venv

Comments: Just checked out the files and thishad already been addressed

For others who find this issue: 

`pip install --upgrade "datasets>=2.20.0"` 

from https://github.com/explodinggradients/ragas/issues/2170#issuecomment-3204393672 can fix it.
Preserve tree structure when loading HDF5
Author: klamike
State: closed
Labels: name: enhancement

### Feature request

https://github.com/huggingface/datasets/pull/7740#discussion_r2285605374

### Motivation

`datasets` has the `Features` class for representing nested features. HDF5 files have groups of datasets which are nested, though in #7690 the keys are flattened. We should preserve that structure for the user.

### Your contribution

I'll open a PR (#7743)
Replacement of "Sequence" feature with "List" breaks backward compatibility
Author: evmaki
State: open

PR #7634 replaced the Sequence feature with List in 4.0.0, so datasets saved with version 4.0.0 with that feature cannot be loaded by earlier versions. There is no clear option in 4.0.0 to use the legacy feature type to preserve backward compatibility.

Why is this a problem? I have a complex preprocessing and training pipeline dependent on 3.6.0; we manage a very large number of separate datasets that get concatenated during training. If just one of those datasets is saved with 4.0.0, they become unusable, and we have no way of "fixing" them. I can load them in 4.0.0 but I can't re-save with the legacy feature type, and I can't load it in 3.6.0 for obvious reasons.

Perhaps I'm missing something here, since the PR says that backward compatibility is preserved; if so, it's not obvious to me how.

Comments: Backward compatibility here means 4.0.0 can load datasets saved with older versions.

You will need 4.0.0 to load datasets saved with 4.0.0
Allow saving multi-dimensional ndarray with dynamic shapes
Author: ryan-minato
State: open
Labels: name: enhancement

### Feature request

I propose adding a dedicated feature to the datasets library that allows for the efficient storage and retrieval of multi-dimensional ndarray with dynamic shapes. Similar to how Image columns handle variable-sized images, this feature would provide a structured way to store array data where the dimensions are not fixed.

A possible implementation could be a new Array or Tensor feature type that stores the data in a structured format, for example, 

```python
{
  "shape": (5, 224, 224), 
  "dtype": "uint8", 
  "data": [...]
}
```

This would allow the datasets library to handle heterogeneous array sizes within a single column without requiring a fixed shape definition in the feature schema.

### Motivation

I am currently trying to upload data from astronomical telescopes, specifically FITS files, to the Hugging Face Hub. This type of data is very similar to images but often has more than three dimensions. For example, data from the SDSS project contains five channels (u, g, r, i, z), and the pixel values can exceed 255, making the Pillow based Image feature unsuitable.

The current datasets library requires a fixed shape to be defined in the feature schema for multi-dimensional arrays, which is a major roadblock. This prevents me from saving my data, as the dimensions of the arrays can vary across different FITS files.

https://github.com/huggingface/datasets/blob/985c9bee6bfc345787a8b9dd316e1d4f3b930503/src/datasets/features/features.py#L613-L614

A feature that supports dynamic shapes would be incredibly beneficial for the astronomy community and other fields dealing with similar high-dimensional, variable-sized data (e.g., medical imaging, scientific simulations).

### Your contribution

I am willing to create a PR to help implement this feature if the proposal is accepted.

Comments: I agree this would be super valuable.

It looks like this was discussed a few years ago in https://github.com/huggingface/datasets/issues/5272#issuecomment-1550200824 but there were some issues. Those PRs are merged now and it looks like Arrow [officially supports](https://arrow.apache.org/docs/format/CanonicalExtensions.html#variable-shape-tensor) this so it's a good time to re-evaluate!

Happy to help with this, maybe we can think of adding a new type `Tensor` (instead of Array2D, 3D etc. which imply a fixed number of dims - we can keep them for backward compat anyways) that uses VariableShapeTensor (or FixedShapeTensor if the shape is provided maybe ? happy to discuss this)
Add the possibility of a backend for audio decoding
Author: intexcor
State: open
Labels: name: enhancement

### Feature request

Add the possibility of a backend for audio decoding. Before version 4.0.0, soundfile was used, and now torchcodec is used, but the problem is that torchcodec requires ffmpeg, which is problematic to install on the same colab. Therefore, I suggest adding a decoder selection when loading the dataset.

### Motivation

I use a service for training models in which ffmpeg cannot be installed.

### Your contribution

I use a service for training models in which ffmpeg cannot be installed.

Comments: is there a work around im stuck

never mind just downgraded
OSError: libcudart.so.11.0: cannot open shared object file: No such file or directory
Author: SaleemMalikAI
State: open

> Hi  is there any solution for that eror i try to install this one 
pip install torch==1.12.1+cpu torchaudio==0.12.1+cpu -f https://download.pytorch.org/whl/torch_stable.html  
this is working fine but tell me how to install pytorch version that is fit for gpu 

Comments: Is this related to the "datasets" library? @SaleemMalikAI
NonMatchingSplitsSizesError and ExpectedMoreSplitsError
Author: efsotr
State: open

### Describe the bug

When loading dataset, the info specified by `data_files` did not overwrite the original info.


### Steps to reproduce the bug

```python
from datasets import load_dataset

traindata = load_dataset(
            "allenai/c4",
            "en",
            data_files={"train": "en/c4-train.00000-of-01024.json.gz", 
                        "validation": "en/c4-validation.00000-of-00008.json.gz"},
        )
```


```log
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=828589180707, num_examples=364868892, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='train', num_bytes=809262831, num_examples=356317, shard_lengths=[223006, 133311], dataset_name='c4')}, {'expected': SplitInfo(name='validation', num_bytes=825767266, num_examples=364608, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='validation', num_bytes=102199431, num_examples=45576, shard_lengths=None, dataset_name='c4')}]
```

```python
from datasets import load_dataset

traindata = load_dataset(
            "allenai/c4",
            "en",
            data_files={"train": "en/c4-train.00000-of-01024.json.gz"},
            split="train"
        )
```

```log
ExpectedMoreSplitsError: {'validation'}
```

### Expected behavior

No error

### Environment info

datasets 4.0.0

Comments: To load just one shard without errors, you should use data_files directly with split set to "train", but don’t specify "allenai/c4", since that points to the full dataset with all shards.

Instead, do this:
```
from datasets import load_dataset
from datasets import load_dataset

# Load only one shard of C4
traindata = load_dataset(
    "json",   # <-- use "json" since you’re directly passing JSON files
    data_files={"train": "https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-train.00000-of-01024.json.gz"},
    split="train"
)

print(traindata)
```
If you want both train and validation but only a subset of shards, do:
```
traindata = load_dataset(
    "json",
    data_files={
        "train": "https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-train.00000-of-01024.json.gz",
        "validation": "https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-validation.00000-of-00008.json.gz"
    }
)

print(traindata)
```

I just want to load a few files from allenai/c4.
If I do not specify allenai/c4, where will the files be loaded from?

My apologies, I’ve modified my previous answer.
You just need to specify the full path, for example:

https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-train.00000-of-01024.json.gz

<img width="1843" height="633" alt="Image" src="https://github.com/user-attachments/assets/b2922958-9d87-4b62-a00e-c5ca02e31c27" />

I hope this updated answer is helpful.
config paths that start with ./ are not valid as hf:// accessed repos, but are valid when accessed locally
Author: doctorpangloss
State: open

### Describe the bug

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - images/xyz/*.jpg
```
will correctly download but

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - ./images/xyz/*.jpg
```
will error with `FileNotFoundError` due to improper url joining. `load_dataset` on the same directory locally works fine.

### Steps to reproduce the bug

1. create a README.md with the front matter of the form

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - ./images/xyz/*.jpg
```

2. `touch ./images/xyz/1.jpg`
3. Observe this directory loads with `load_dataset("filesystem_path", "some_config")` correctly.
4. Observe exceptions when you load this with `load_dataset("repoid/filesystem_path", "some_config")`

### Expected behavior

`./` prefix should be interpreted correctly

### Environment info

datasets 4.0.0
datasets 3.4.0

reproduce
Don't remove `trust_remote_code` arg!!!
Author: autosquid
State: open
Labels: name: enhancement

### Feature request

defaulting it to False is nice balance. we need manully setting it to True in certain scenarios!

Add `trust_remote_code` arg back please!

### Motivation

defaulting it to False is nice balance. we need manully setting it to True in certain scenarios!


### Your contribution

defaulting it to False is nice balance. we need manully setting it to True in certain scenarios!
Out of memory even though using load_dataset(..., streaming=True)
Author: padmalcom
State: open

### Describe the bug

I am iterating over a large dataset that I load using streaming=True to avoid running out of memory. Unfortunately, I am observing that memory usage increases over time and I'm finally running in an oom.

### Steps to reproduce the bug

```
  ds = load_dataset("openslr/librispeech_asr", split="train.clean.360", streaming=True)
  for i,sample in enumerate(tqdm(ds)):
    target_file = os.path.join(NSFW_TARGET_FOLDER, f'audio{i}.wav')
    try:
      sf.write(target_file, sample['audio']['array'], samplerate=sample['audio']['sampling_rate'])
    except Exception as e:
      print(f"Could not write audio {i} in ds: {e}")
```


### Expected behavior

I'd expect to have a small memory footprint and memory being freed after each iteration of the for loop. Instead the memory usage is increasing. I tried to remove the logic to write the sound file and just print the sample but the issue remains the same.

### Environment info

Python 3.12.11
Ubuntu 24
datasets 4.0.0 and 3.6.0
Bad split error message when using percentages
Author: padmalcom
State: open

### Describe the bug

Hi, I'm trying to download a dataset. To not load the entire dataset in memory, I split it as described [here](https://huggingface.co/docs/datasets/v4.0.0/loading#slice-splits) in 10% steps.

When doing so, the library returns this error:
    raise ValueError(f"Bad split: {split}. Available splits: {list(splits_generators)}")
ValueError: Bad split: train[0%:10%]. Available splits: ['train']

Edit: Same happens with a split like _train[:90000]_
### Steps to reproduce the bug

```
  for split in range(10):
    split_str = f"train[{split*10}%:{(split+1)*10}%]"
    print(f"Processing split {split_str}...")
    ds = load_dataset("user/dataset", split=split_str, streaming=True)
```


### Expected behavior

I'd expect the library to split my dataset in 10% steps.

### Environment info

python 3.12.11
ubuntu 24
dataset 4.0.0

Comments: I'd like to work on this: add clearer validation/messages for percent-based splits + tests

The most basic example is this code:
`load_dataset("openslr/librispeech_asr", split="train[10%:20%]")`

This results in this ValueError:
```
    raise ValueError(f'Unknown split "{split}". Should be one of {list(name2len)}.')
ValueError: Unknown split "train". Should be one of ['test.clean', 'test.other', 'train.clean.100', 'train.clean.360', 'train.other.500', 'validation.clean', 'validation.other'].
```