Sulak2020/github-issues-multirepo-datasets-Sulakshana huggingface dataset:
Title: module 'pyarrow' has no attribute 'PyExtensionType'
State: open

### Describe the bug

When importing certain libraries, users will encounter the following error which can be traced back to the datasets library.

module 'pyarrow' has no attribute 'PyExtensionType'.

Example issue: https://github.com/explodinggradients/ragas/issues/2170

The issue occurs due to the following. I will proceed to submit a PR with the below fix:

**Issue Reason**
The issue is that PyArrow version 21.0.0 doesn’t have PyExtensionType. This was changed in newer versions of PyArrow. The

PyExtensionType class was renamed to ExtensionType in PyArrow 13.0.0 and later versions.


** Issue Solution**
Making the following changes to the following lib files should temporarily resolve the issue.

I will submit a PR to the dataets library in the meantime.

env_name/lib/python3.10/site-packages/datasets/features/features.py:

```
> 521            self.shape = tuple(shape)
       522            self.value_type = dtype
       523            self.storage_dtype = self._generate_dtype(self.value_type)
       524 -          pa.PyExtensionType.__init__(self, self.storage_dtype)
       524 +          pa.ExtensionType.__init__(self, self.storage_dtype)
       525
       526        def __reduce__(self):
       527            return self.__class__, (
```

Updated venv_name/lib/python3.10/site-packages/datasets/features/features.py:
```      
 510        _type: str = field(default=“Array5D”, init=False, repr=False)
       511
       512
       513 -  class _ArrayXDExtensionType(pa.PyExtensionType):
       513 +  class _ArrayXDExtensionType(pa.ExtensionType):
       514        ndims: Optional[int] = None
       515
       516        def __init__(self, shape: tuple, dtype: str):
```

### Steps to reproduce the bug

Ragas version: 0.3.1
Python version: 3.11

**Code to Reproduce**

_**In notebook:**_

!pip install ragas
from ragas import evaluate


### Expected behavior

The required package installs without issue.

### Environment info

In Jupyter Notebook.
venv

Comments: Just checked out the files and thishad already been addressedFor others who find this issue: 

`pip install --upgrade "datasets>=2.20.0"` 

from https://github.com/explodinggradients/ragas/issues/2170#issuecomment-3204393672 can fix it.
Title: Preserve tree structure when loading HDF5
State: closed
Labels: name: enhancement
description: New feature or request

### Feature request

https://github.com/huggingface/datasets/pull/7740#discussion_r2285605374

### Motivation

`datasets` has the `Features` class for representing nested features. HDF5 files have groups of datasets which are nested, though in #7690 the keys are flattened. We should preserve that structure for the user.

### Your contribution

I'll open a PR (#7743)
Title: Document HDF5 support
State: closed

I think these are at least the main places where we should put content. Ideally it is not just repeated in the final version


ref #7690 


- [x] Wait for #7743 to land

Comments: @lhoestq any guidance on what else to add/feedback on what is there now? It seems a bit minimal, but I don't think it's worth doing an entire page on HDF5?
Title: Replacement of "Sequence" feature with "List" breaks backward compatibility
State: open

PR #7634 replaced the Sequence feature with List in 4.0.0, so datasets saved with version 4.0.0 with that feature cannot be loaded by earlier versions. There is no clear option in 4.0.0 to use the legacy feature type to preserve backward compatibility.

Why is this a problem? I have a complex preprocessing and training pipeline dependent on 3.6.0; we manage a very large number of separate datasets that get concatenated during training. If just one of those datasets is saved with 4.0.0, they become unusable, and we have no way of "fixing" them. I can load them in 4.0.0 but I can't re-save with the legacy feature type, and I can't load it in 3.6.0 for obvious reasons.

Perhaps I'm missing something here, since the PR says that backward compatibility is preserved; if so, it's not obvious to me how.

Comments: Backward compatibility here means 4.0.0 can load datasets saved with older versions.

You will need 4.0.0 to load datasets saved with 4.0.0
Title: Allow saving multi-dimensional ndarray with dynamic shapes
State: open
Labels: name: enhancement
description: New feature or request

### Feature request

I propose adding a dedicated feature to the datasets library that allows for the efficient storage and retrieval of multi-dimensional ndarray with dynamic shapes. Similar to how Image columns handle variable-sized images, this feature would provide a structured way to store array data where the dimensions are not fixed.

A possible implementation could be a new Array or Tensor feature type that stores the data in a structured format, for example, 

```python
{
  "shape": (5, 224, 224), 
  "dtype": "uint8", 
  "data": [...]
}
```

This would allow the datasets library to handle heterogeneous array sizes within a single column without requiring a fixed shape definition in the feature schema.

### Motivation

I am currently trying to upload data from astronomical telescopes, specifically FITS files, to the Hugging Face Hub. This type of data is very similar to images but often has more than three dimensions. For example, data from the SDSS project contains five channels (u, g, r, i, z), and the pixel values can exceed 255, making the Pillow based Image feature unsuitable.

The current datasets library requires a fixed shape to be defined in the feature schema for multi-dimensional arrays, which is a major roadblock. This prevents me from saving my data, as the dimensions of the arrays can vary across different FITS files.

https://github.com/huggingface/datasets/blob/985c9bee6bfc345787a8b9dd316e1d4f3b930503/src/datasets/features/features.py#L613-L614

A feature that supports dynamic shapes would be incredibly beneficial for the astronomy community and other fields dealing with similar high-dimensional, variable-sized data (e.g., medical imaging, scientific simulations).

### Your contribution

I am willing to create a PR to help implement this feature if the proposal is accepted.

Comments: I agree this would be super valuable.

It looks like this was discussed a few years ago in https://github.com/huggingface/datasets/issues/5272#issuecomment-1550200824 but there were some issues. Those PRs are merged now and it looks like Arrow [officially supports](https://arrow.apache.org/docs/format/CanonicalExtensions.html#variable-shape-tensor) this so it's a good time to re-evaluate!Happy to help with this, maybe we can think of adding a new type `Tensor` (instead of Array2D, 3D etc. which imply a fixed number of dims - we can keep them for backward compat anyways) that uses VariableShapeTensor (or FixedShapeTensor if the shape is provided maybe ? happy to discuss this)
Title: docs: Add column overwrite example to batch mapping guide
State: closed

This PR adds a complementary example showing the **column-overwriting** pattern, which is both more direct and more flexible for many transformations.

### Proposed Change

The original `remove_columns` example remains untouched. Below it, this PR introduces an alternative approach that overwrites an existing column during batch mapping.

This teaches users a core `.map()` capability for in-place transformations without extra intermediate steps.

**New Example:**

> ```python
> >>> from datasets import Dataset
> >>> dataset = Dataset.from_dict({"a": [0, 1, 2]})
> # Overwrite "a" directly to duplicate each value
> >>> duplicated_dataset = dataset.map(
> ...     lambda batch: {"a": [x for x in batch["a"] for _ in range(2)]},
> ...     batched=True
> ... )
> >>> duplicated_dataset
> Dataset({
>     features: ['a'],
>     num_rows: 6
> })
> >>> duplicated_dataset["a"]
> [0, 0, 1, 1, 2, 2]
> ```


Comments: Hi @lhoestq, just a gentle follow-up on this PR.
Title: Fixing __getitem__ of datasets which behaves inconsistent to documentation when setting _format_type to None
State: closed

Setting _format_type to None, should return plain python object but as of 4.0.0 returns Column. This fails in libs such as sentencetransformers (such as in generation of hard negatives) where plain python is expected.

Comments: this breaking change is actually expected, happy to help with a fix in sentencetransformers to account for thisThank you for the context. I thought this was a mismatch do the documentation. Good to know it was intentional. No worries, can add a PR to sentence transformers.
Title: Add the possibility of a backend for audio decoding
State: open
Labels: name: enhancement
description: New feature or request

### Feature request

Add the possibility of a backend for audio decoding. Before version 4.0.0, soundfile was used, and now torchcodec is used, but the problem is that torchcodec requires ffmpeg, which is problematic to install on the same colab. Therefore, I suggest adding a decoder selection when loading the dataset.

### Motivation

I use a service for training models in which ffmpeg cannot be installed.

### Your contribution

I use a service for training models in which ffmpeg cannot be installed.

Comments: is there a work around im stucknever mind just downgraded
Title: OSError: libcudart.so.11.0: cannot open shared object file: No such file or directory
State: open

> Hi  is there any solution for that eror i try to install this one 
pip install torch==1.12.1+cpu torchaudio==0.12.1+cpu -f https://download.pytorch.org/whl/torch_stable.html  
this is working fine but tell me how to install pytorch version that is fit for gpu 

Comments: Is this related to the "datasets" library? @SaleemMalikAI
Title: NonMatchingSplitsSizesError and ExpectedMoreSplitsError
State: open

### Describe the bug

When loading dataset, the info specified by `data_files` did not overwrite the original info.


### Steps to reproduce the bug

```python
from datasets import load_dataset

traindata = load_dataset(
            "allenai/c4",
            "en",
            data_files={"train": "en/c4-train.00000-of-01024.json.gz", 
                        "validation": "en/c4-validation.00000-of-00008.json.gz"},
        )
```


```log
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=828589180707, num_examples=364868892, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='train', num_bytes=809262831, num_examples=356317, shard_lengths=[223006, 133311], dataset_name='c4')}, {'expected': SplitInfo(name='validation', num_bytes=825767266, num_examples=364608, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='validation', num_bytes=102199431, num_examples=45576, shard_lengths=None, dataset_name='c4')}]
```

```python
from datasets import load_dataset

traindata = load_dataset(
            "allenai/c4",
            "en",
            data_files={"train": "en/c4-train.00000-of-01024.json.gz"},
            split="train"
        )
```

```log
ExpectedMoreSplitsError: {'validation'}
```

### Expected behavior

No error

### Environment info

datasets 4.0.0

Comments: To load just one shard without errors, you should use data_files directly with split set to "train", but don’t specify "allenai/c4", since that points to the full dataset with all shards.

Instead, do this:
```
from datasets import load_dataset
from datasets import load_dataset

# Load only one shard of C4
traindata = load_dataset(
    "json",   # <-- use "json" since you’re directly passing JSON files
    data_files={"train": "https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-train.00000-of-01024.json.gz"},
    split="train"
)

print(traindata)
```
If you want both train and validation but only a subset of shards, do:
```
traindata = load_dataset(
    "json",
    data_files={
        "train": "https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-train.00000-of-01024.json.gz",
        "validation": "https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-validation.00000-of-00008.json.gz"
    }
)

print(traindata)
```I just want to load a few files from allenai/c4.
If I do not specify allenai/c4, where will the files be loaded from?My apologies, I’ve modified my previous answer.
You just need to specify the full path, for example:

https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-train.00000-of-01024.json.gz

<img width="1843" height="633" alt="Image" src="https://github.com/user-attachments/assets/b2922958-9d87-4b62-a00e-c5ca02e31c27" />

I hope this updated answer is helpful.
Title: config paths that start with ./ are not valid as hf:// accessed repos, but are valid when accessed locally
State: open

### Describe the bug

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - images/xyz/*.jpg
```
will correctly download but

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - ./images/xyz/*.jpg
```
will error with `FileNotFoundError` due to improper url joining. `load_dataset` on the same directory locally works fine.

### Steps to reproduce the bug

1. create a README.md with the front matter of the form

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - ./images/xyz/*.jpg
```

2. `touch ./images/xyz/1.jpg`
3. Observe this directory loads with `load_dataset("filesystem_path", "some_config")` correctly.
4. Observe exceptions when you load this with `load_dataset("repoid/filesystem_path", "some_config")`

### Expected behavior

`./` prefix should be interpreted correctly

### Environment info

datasets 4.0.0
datasets 3.4.0

reproduce
Title: fix(webdataset): don't .lower() field_name
State: closed

This fixes cases where keys have upper case identifiers

Comments: fixes: https://github.com/huggingface/datasets/issues/7732The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7726). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.CI failures are unrelated, merging :)
Title: Don't remove `trust_remote_code` arg!!!
State: open
Labels: name: enhancement
description: New feature or request

### Feature request

defaulting it to False is nice balance. we need manully setting it to True in certain scenarios!

Add `trust_remote_code` arg back please!

### Motivation

defaulting it to False is nice balance. we need manully setting it to True in certain scenarios!


### Your contribution

defaulting it to False is nice balance. we need manully setting it to True in certain scenarios!
Title: Out of memory even though using load_dataset(..., streaming=True)
State: open

### Describe the bug

I am iterating over a large dataset that I load using streaming=True to avoid running out of memory. Unfortunately, I am observing that memory usage increases over time and I'm finally running in an oom.

### Steps to reproduce the bug

```
  ds = load_dataset("openslr/librispeech_asr", split="train.clean.360", streaming=True)
  for i,sample in enumerate(tqdm(ds)):
    target_file = os.path.join(NSFW_TARGET_FOLDER, f'audio{i}.wav')
    try:
      sf.write(target_file, sample['audio']['array'], samplerate=sample['audio']['sampling_rate'])
    except Exception as e:
      print(f"Could not write audio {i} in ds: {e}")
```


### Expected behavior

I'd expect to have a small memory footprint and memory being freed after each iteration of the for loop. Instead the memory usage is increasing. I tried to remove the logic to write the sound file and just print the sample but the issue remains the same.

### Environment info

Python 3.12.11
Ubuntu 24
datasets 4.0.0 and 3.6.0
Title: Bad split error message when using percentages
State: open

### Describe the bug

Hi, I'm trying to download a dataset. To not load the entire dataset in memory, I split it as described [here](https://huggingface.co/docs/datasets/v4.0.0/loading#slice-splits) in 10% steps.

When doing so, the library returns this error:
    raise ValueError(f"Bad split: {split}. Available splits: {list(splits_generators)}")
ValueError: Bad split: train[0%:10%]. Available splits: ['train']

Edit: Same happens with a split like _train[:90000]_
### Steps to reproduce the bug

```
  for split in range(10):
    split_str = f"train[{split*10}%:{(split+1)*10}%]"
    print(f"Processing split {split_str}...")
    ds = load_dataset("user/dataset", split=split_str, streaming=True)
```


### Expected behavior

I'd expect the library to split my dataset in 10% steps.

### Environment info

python 3.12.11
ubuntu 24
dataset 4.0.0

Comments: I'd like to work on this: add clearer validation/messages for percent-based splits + testsThe most basic example is this code:
`load_dataset("openslr/librispeech_asr", split="train[10%:20%]")`

This results in this ValueError:
```
    raise ValueError(f'Unknown split "{split}". Should be one of {list(name2len)}.')
ValueError: Unknown split "train". Should be one of ['test.clean', 'test.other', 'train.clean.100', 'train.clean.360', 'train.other.500', 'validation.clean', 'validation.other'].
```
Title: Datasets 4.0 map function causing column not found
State: open

### Describe the bug

Column returned after mapping is not found in new instance of the dataset.

### Steps to reproduce the bug

Code for reproduction. After running get_total_audio_length, it is errored out due to `data` not having `duration`
```
def compute_duration(x):
    return {"duration": len(x["audio"]["array"]) / x["audio"]["sampling_rate"]}


def get_total_audio_length(dataset):
    data = dataset.map(compute_duration, num_proc=NUM_PROC)
    print(data)
    durations=data["duration"]
    total_seconds = sum(durations)
    return total_seconds
```

### Expected behavior

New datasets.Dataset instance should have new columns attached.

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-5.4.0-124-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.33.2
- PyArrow version: 20.0.0
- Pandas version: 2.3.0
- `fsspec` version: 2023.12.2

Comments: Hi, I tried to reproduce this issue on the latest `main` branch but it seems to be working correctly now. My test script (which creates a dummy dataset and applies the `.map()` function) successfully creates and accesses the new column without a `KeyError`.

It's possible this was fixed by a recent commit. The maintainers might want to consider closing this issue.Hi, have you tried on a large dataset (200GB+) perhaps? I will try my best to do a rerun with main branch when I have the time.I ran it on a small dataset, maybe that’s why I didn’t hit the issue. If it still shows up on your side with the latest main, let me know. I can try it on a bigger set too.
Title: Specify dataset columns types in typehint
State: open
Labels: name: enhancement
description: New feature or request

### Feature request

Make dataset optionaly generic to datasets usage with type annotations like it was done in `torch.Dataloader` https://github.com/pytorch/pytorch/blob/134179474539648ba7dee1317959529fbd0e7f89/torch/utils/data/dataloader.py#L131

### Motivation

In MTEB we're using a lot of datasets objects, but they're a bit poor in typehints. E.g. we can specify this for dataloder

```python
from typing import TypedDict
from torch.utils.data import DataLoader

class CorpusInput(TypedDict):
    title: list[str]
    body: list[str]


class QueryInput(TypedDict):
    query: list[str]
    instruction: list[str]

def queries_loader() -> DataLoader[QueryInput]:
    ...

def corpus_loader() -> DataLoader[CorpusInput]:
   ...
```
But for datasets we can only specify columns in type in comments 
```python
from datasets import Dataset

QueryDataset = Dataset
"""Query dataset should have `query` and `instructions` columns as `str` """
```

### Your contribution

I can create draft implementation
Title: add support for pyarrow string view in features
State: closed


Comments: @lhoestq who do you think would be the best to have a look at this? Any pointers would be appreciated, thanks!Hi ! what's the rationale for supporting string view ? I'm afraid it can complexify the typing logic without much valueHi @lhoestq ! I mainly want to be able to create features by using `Features.from_arrow_schema(dataset_schema)` on an arrow dataset with string view columns, currently there is no easy way to do this, and string_view is becoming an increasingly common data type for string columns in arrow. Thanks for having a look!The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7718). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Cached dataset is not used when explicitly passing the cache_dir parameter
State: open

### Describe the bug

Hi, we are pre-downloading a dataset using snapshot_download(). When loading this exact dataset with load_dataset() the cached snapshot is not used. In both calls, I provide the cache_dir parameter.

### Steps to reproduce the bug

```
from datasets import load_dataset, concatenate_datasets
from huggingface_hub import snapshot_download
    
def download_ds(name: str):
    snapshot_download(repo_id=name, repo_type="dataset", cache_dir="G:/Datasets/cache")

def prepare_ds():
    audio_ds = load_dataset("openslr/librispeech_asr", num_proc=4, cache_dir="G:/Datasets/cache")
    print(sfw_ds.features)

if __name__ == '__main__':
    download_ds("openslr/librispeech_asr")
    prepare_ds()
```

### Expected behavior

I'd expect that the cached version of the dataset is used. Instead, the same dataset is downloaded again to the default cache directory.

### Environment info

Windows 11
datasets==4.0.0
Python 3.12.11

Comments: Hi, I've investigated this issue and can confirm the bug. Here are my findings:

**1. Reproduction:**
I was able to reproduce the issue on the latest `main` branch. Using the provided code snippet, `snapshot_download` correctly populates the custom `cache_dir`, but `load_dataset` with the same `cache_dir` triggers a full re-download and re-processing of the dataset, ignoring the existing cache.

**2. Investigation:**
I traced the `cache_dir` parameter from `load_dataset` down to the `DatasetBuilder` class in `src/datasets/builder.py`. The root cause seems to be a mismatch between the cache path structure created by `snapshot_download` and the path structure expected by the `DatasetBuilder`.

Specifically, the `_relative_data_dir` method in `DatasetBuilder` constructs a path using `namespace___dataset_name` (with three underscores), while the cache from `snapshot_download` appears to use a `repo_id` based format like `datasets--namespace--dataset_name` (with double hyphens).

**3. Attempted Fix & Result:**
I attempted a fix by modifying the `_relative_data_dir` method to replace the path separator "/" in `self.repo_id` with "--", to align it with the `snapshot_download` structure.

This partially worked: `load_dataset` no longer re-downloads the files. However, it still re-processes them every time (triggering "Generating train split...", etc.) instead of loading the already processed Arrow files from the cache.

This suggests the issue is deeper than just the directory name and might be related to how the builder verifies the integrity or presence of the processed cache files.

I hope these findings are helpful for whoever picks up this issue.
Title: Concurrent IterableDataset push_to_hub
State: closed

Same as https://github.com/huggingface/datasets/pull/7708 but for `IterableDataset`

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7710). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.