noamaanMulla-03/datasets-issues huggingface dataset:
IndexError during training with Squad dataset and T5-small model

### Describe the bug

I am encountering an IndexError while training a T5-small model on the Squad dataset using the transformers and datasets libraries. The error occurs even with a minimal reproducible example, suggesting a potential bug or incompatibility.

### Steps to reproduce the bug

1.Install the required libraries: !pip install transformers datasets
2.Run the following code:
!pip install transformers datasets

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, TrainingArguments, Trainer, DataCollatorWithPadding

# Load a small, publicly available dataset
from datasets import load_dataset
dataset = load_dataset("squad", split="train[:100]") # Use a small subset for testing

# Load a pre-trained model and tokenizer
model_name = "t5-small"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# Define a basic data collator
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

# Define training arguments
training_args = TrainingArguments(
    output_dir="./results",
    per_device_train_batch_size=2,
    num_train_epochs=1,
)

# Create a trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    data_collator=data_collator,
)

# Train the model
trainer.train()

### Expected behavior

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
[<ipython-input-23-f13a4b23c001>](https://localhost:8080/#) in <cell line: 34>()
     32 
     33 # Train the model
---> 34 trainer.train()

10 frames
[/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py](https://localhost:8080/#) in _check_valid_index_key(key, size)
    427     if isinstance(key, int):
    428         if (key < 0 and key + size < 0) or (key >= size):
--> 429             raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
    430         return
    431     elif isinstance(key, slice):

IndexError: Invalid key: 42 is out of bounds for size 0

### Environment info

transformers version:4.41.2
datasets version:1.18.4
Python version:3.10.12



add remove_unused_columns=False to training_args
https://github.com/huggingface/datasets/issues/6535#issuecomment-1874024704

Closing this issue because it was a reported and fixed in transformers.
My Private Dataset doesn't exist on the Hub or cannot be accessed

### Describe the bug


```
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 1852, in dataset_module_factory
    raise DatasetNotFoundError(msg + f" at revision '{revision}'" if revision else msg)
datasets.exceptions.DatasetNotFoundError: Dataset 'xxx' doesn't exist on the Hub or cannot be accessed
>>> dataset = load_dataset("xxxx", token=True)

404 error 404 Client Error. (Request ID: Root=xxxx)

Repository Not Found for url: https://huggingface.co/api/datasets/xxx/xxx.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 2593, in load_dataset
    builder_instance = load_dataset_builder(
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 2265, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 1910, in dataset_module_factory
    raise e1 from None
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 1852, in dataset_module_factory
    raise DatasetNotFoundError(msg + f" at revision '{revision}'" if revision else msg)
datasets.exceptions.DatasetNotFoundError: Dataset 'xxx' doesn't exist on the Hub or cannot be accessed
```

### Steps to reproduce the bug

123

### Expected behavior

123

### Environment info

123


I can load public dataset, but for my private dataset it fails

https://huggingface.co/docs/datasets/upload_dataset

I have checked the API HTTP link. Repository Not Found for url: https://huggingface.co/api/datasets/xxx/xxx.

![image](https://github.com/huggingface/datasets/assets/39621324/4aceef59-0c65-4161-9665-676d25d73225)

It just works fine.

It seems that everything is in a mass huh....

![image](https://github.com/huggingface/datasets/assets/39621324/fb2fe12c-4f0a-4bf6-9656-63ba50347b10)


https://huggingface.co/datasets/rajpurkar/squad/blob/main/squad.py fails again

https://github.com/huggingface/datasets/blob/main/templates/new_dataset_script.py#L81 can not use this, too complex. I just need a def to load my file to a dict

I am facing the same issue. Did you find a fix?

You should authenticate to be able to access private or gated repos: https://huggingface.co/docs/hub/datasets-gated#access-gated-datasets-as-a-user
load_dataset error

### Describe the bug

Why does the program get stuck when I use load_dataset method, and it still gets stuck after loading for several hours? In fact, my json file is only 21m, and I can load it in one go using open('', 'r').

### Steps to reproduce the bug

1. pip install datasets==2.19.2
2. from datasets import Dataset, DatasetDict, NamedSplit, Split, load_dataset
3. data = load_dataset('json', data_files='train.json')

### Expected behavior

It is able to load my json correctly

### Environment info

datasets==2.19.2


Hi, @lion-ops.

In our Continuous Integration we have many tests on loading JSON files and all of them work properly.

Could you please share your "train.json" file, so that we can try to reproduce the issue you have? 

> Hi, @lion-ops.
> 
> In our Continuous Integration we have many tests on loading JSON files and all of them work properly.
> 
> Could you please share your "train.json" file, so that we can try to reproduce the issue you have?

Thank you for your reply. I can load it normally in another server. Is it possible that the disk of my server is a network disk in the LAN, so it will be downloaded from the LAN and get stuck?
IterableDataset `set_epoch` is ignored when DataLoader `persistent_workers=True`

### Describe the bug

When persistent workers are enabled, the epoch that's set via the IterableDataset instance held by the training process is ignored by the workers as they are disconnected across processes.

PyTorch samplers for non-iterable datasets have a mechanism to sync this, datasets.IterableDataset does not.

In my own use of IterableDatasets I usually track the epoch count which crosses process boundaries in a multiprocessing.Value 

### Steps to reproduce the bug

Use a streaming dataset (Iterable) w/ the recommended pattern below and `persistent_workers=True` in the torch DataLoader. 

```
for epoch in range(epochs):
    shuffled_dataset.set_epoch(epoch)
    for example in shuffled_dataset:
        ...
```

### Expected behavior

When the canonical bit of code above is used with `num_workers > 0` and `persistent_workers=True`, the epoch set via `set_epoch()` is propagated to the IterableDataset instances in the worker processes


### Environment info

N/A

Labels: name: bug

name: streaming
Casting list array to fixed size list raises error

When trying to cast a list array to fixed size list, an AttributeError is raised:
> AttributeError: 'pyarrow.lib.FixedSizeListType' object has no attribute 'length'

Steps to reproduce the bug:
```python
import pyarrow as pa
from datasets.table import array_cast

arr = pa.array([[0, 1]])
array_cast(arr, pa.list_(pa.int64(), 2))
```

Stack trace:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-12-6cb90a1d8216> in <module>
      3 
      4 arr = pa.array([[0, 1]])
----> 5 array_cast(arr, pa.list_(pa.int64(), 2))

~/huggingface/datasets/src/datasets/table.py in wrapper(array, *args, **kwargs)
   1802             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1803         else:
-> 1804             return func(array, *args, **kwargs)
   1805 
   1806     return wrapper

~/huggingface/datasets/src/datasets/table.py in array_cast(array, pa_type, allow_primitive_to_str, allow_decimal_to_str)
   1920                 else:
   1921                     array_values = array.values[
-> 1922                         array.offset * pa_type.length : (array.offset + len(array)) * pa_type.length
   1923                     ]
   1924                     return pa.FixedSizeListArray.from_arrays(_c(array_values, pa_type.value_type), pa_type.list_size)

AttributeError: 'pyarrow.lib.FixedSizeListType' object has no attribute 'length'
```

Labels: name: bug
How to load data from lakefs

My dataset is stored on the company's lakefs server. How can I write code to load the dataset? It would be great if I could provide code examples or provide some references



You can create a `pandas` DataFrame following [this](https://lakefs.io/data-version-control/dvc-using-python/) tutorial, and then convert this DataFrame to a `Dataset` with `datasets.Dataset.from_pandas`. For larger datasets (to memory map them), you can use `Dataset.from_generator` with a generator function that reads lakeFS files with `s3fs`.

@mariosasko  hello,
This can achieve and https://huggingface.co/datasets Does the same effect apply to the dataset? For example, downloading while using

There is a blogspot from lakes on this topic: https://lakefs.io/blog/data-version-control-hugging-face-datasets/
Streaming dataset not returning data

### Describe the bug

I'm deciding to post here because I'm still not sure what the issue is, or if I am using IterableDatasets wrongly.
I'm following the guide on here https://huggingface.co/learn/cookbook/en/fine_tuning_code_llm_on_single_gpu pretty much to a tee and have verified that it works when I'm fine-tuning on the provided dataset.

However, I'm doing some data preprocessing steps (filtering out entries), when I try to swap out the dataset for mine, it fails to train. However, I eventually fixed this by simply setting `stream=False` in `load_dataset`.

Coud this be some sort of network / firewall issue I'm facing?

### Steps to reproduce the bug

I made a post with greater description about how I reproduced this problem before I found my workaround: https://discuss.huggingface.co/t/problem-with-custom-iterator-of-streaming-dataset-not-returning-anything/94551

Here is the problematic dataset snippet, which works when streaming=False (and with buffer keyword removed from shuffle)

```
commitpackft = load_dataset(
    "chargoddard/commitpack-ft-instruct", split="train", streaming=True
).filter(lambda example: example["language"] == "Python")
 
 
def form_template(example):
    """Forms a template for each example following the alpaca format for CommitPack"""
    example["content"] = (
        "### Human: " + example["instruction"] + " " + example["input"] + " ### Assistant: " + example["output"]
    )
    return example
 
 
dataset = commitpackft.map(
    form_template,
    remove_columns=["id", "language", "license", "instruction", "input", "output"],
).shuffle(
    seed=42, buffer_size=10000
)  # remove everything since its all inside "content" now
validation_data = dataset.take(4000)
train_data = dataset.skip(4000)
```

The annoying part about this is that it only fails during training and I don't know when it will fail, except that it always fails during evaluation.

### Expected behavior

The expected behavior is that I should be able to get something from the iterator when called instead of getting nothing / stuck in a loop somewhere.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.4.0-121-generic-x86_64-with-glibc2.31
- Python version: 3.11.7
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0
Unable to load a dataset

## Describe the bug
Unable to load a dataset from Huggingface that I have just saved.


## Steps to reproduce the bug
On Google colab
`! pip install datasets `
`from datasets import load_dataset`
`my_path = "wiki_dataset"`
`dataset = load_dataset('wikipedia', "20200501.fr")`
`dataset.save_to_disk(my_path)`
`dataset = load_dataset(my_path)`


## Expected results
Loading the dataset

## Actual results
ValueError: Couldn't cast
_data_files: list<item: struct<filename: string>>
  child 0, item: struct<filename: string>
      child 0, filename: string
_fingerprint: string
_format_columns: null
_format_kwargs: struct<>
_format_type: null
_indexes: struct<>
_output_all_columns: bool
_split: string
to
{'builder_name': Value(dtype='string', id=None), 'citation': Value(dtype='string', id=None), 'config_name': Value(dtype='string', id=None), 'dataset_size': Value(dtype='int64', id=None), 'description': Value(dtype='string', id=None), 'download_checksums': {}, 'download_size': Value(dtype='int64', id=None), 'features': {'title': {'dtype': Value(dtype='string', id=None), 'id': Value(dtype='null', id=None), '_type': Value(dtype='string', id=None)}, 'text': {'dtype': Value(dtype='string', id=None), 'id': Value(dtype='null', id=None), '_type': Value(dtype='string', id=None)}}, 'homepage': Value(dtype='string', id=None), 'license': Value(dtype='string', id=None), 'post_processed': Value(dtype='null', id=None), 'post_processing_size': Value(dtype='null', id=None), 'size_in_bytes': Value(dtype='int64', id=None), 'splits': {'train': {'name': Value(dtype='string', id=None), 'num_bytes': Value(dtype='int64', id=None), 'num_examples': Value(dtype='int64', id=None), 'dataset_name': Value(dtype='string', id=None)}}, 'supervised_keys': Value(dtype='null', id=None), 'task_templates': Value(dtype='null', id=None), 'version': {'version_str': Value(dtype='string', id=None), 'description': Value(dtype='string', id=None), 'major': Value(dtype='int64', id=None), 'minor': Value(dtype='int64', id=None), 'patch': Value(dtype='int64', id=None)}}
because column names don't match

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1



Labels: name: bug

Hi! `load_dataset` is intended to be used to load a canonical dataset (`wikipedia`), a packaged dataset (`csv`, `json`, ...) or a dataset hosted on the Hub. For local datasets saved with `save_to_disk("path/to/dataset")`, use `load_from_disk("path/to/dataset")`.

Maybe we should raise an informative error message in this case...

How should we load a locally self-gathered dataset then?
AttributeError: module 'pyarrow' has no attribute 'PyExtensionType' during import 

I'm using Colab. And suddenly this morning, there is this error. Have a look below!

![screenshot-colab research google com-2021 01 26-08-15-36](https://user-images.githubusercontent.com/45964869/105799890-fdaf3b80-5fae-11eb-8f06-11b65cdccc30.png)



Hi ! I'm not able to reproduce the issue. Can you try restarting your runtime ?

The PyExtensionType is available in pyarrow starting 0.17.1 iirc. If restarting your runtime doesn't fix this, can you try updating pyarrow ?
```
pip install pyarrow --upgrade
```

We should bump up the version test of pyarrow maybe no?

https://github.com/huggingface/datasets/blob/master/src/datasets/__init__.py#L60

Yes indeed.

Also it looks like Pyarrow 3.0.0 got released on pypi 10 hours ago. This might be related to the bug, I'll investigate
EDIT: looks like the 3.0.0 release doesn't have unexpected breaking changes for us, so I don't think the issue comes from that

Maybe colab moved to pyarrow 0.16 by default (instead of 0.14 before)?

Installing datasets installs pyarrow>=0.17.1 so in theory it doesn't matter which version of pyarrow colab has by default (which is currently pyarrow 0.14.1).

Also now the colab runtime refresh the pyarrow version automatically after the update from pip (previously you needed to restart your runtime).

I guess what happened is that Colab didn't refresh pyarrow for some reason, and the AttributeError was raised *before* the pyarrow version check from `datasets` at https://github.com/huggingface/datasets/blob/master/src/datasets/__init__.py#L60

Yes colab doesn’t reload preloaded library unless you restart the instance. Maybe we should move the check on top of the init 

Yes I'll do that :)

I updated the pyarrow version check in #1782

restarting worked for me
> ```

yes it worked for me
load `streaming=True` dataset with downloaded cache

### Describe the bug

We build a dataset which contains several hdf5 files and write a script using `h5py` to generate the dataset. The hdf5 files are large and the processed dataset cache takes more disk space. So we hope to try streaming iterable dataset. Unfortunately, `h5py` can't convert a remote URL into a hdf5 file descriptor. So we use `fsspec` as an interface like below:

```python
def _generate_examples(self, filepath, split):
        for file in filepath:
            with fsspec.open(file, "rb") as fs:
                with h5py.File(fs, "r") as fp:
                    # for event_id in sorted(list(fp.keys())):
                    event_ids = list(fp.keys())
                    ......
```

### Steps to reproduce the bug

The `fsspec` works, but it takes 10+ min to print the first 10 examples, which is even longer than the downloading time. I'm not sure if it just caches the whole hdf5 file and generates the examples.

### Expected behavior

So does the following make sense so far?

1. download the files
```python
dataset = datasets.load('path/to/myscripts', split="train", name="event", trust_remote_code=True)
```

2. load the iterable dataset faster (using the raw file cache at path `.cache/huggingface/datasets/downloads`)
```python
dataset = datasets.load('path/to/myscripts', split="train", name="event", trust_remote_code=True, streaming=true)
```

I made some tests, but the code above can't get the expected result. I'm not sure if this is supported. I also find the issue #6327 . It seemed similar to mine, but I couldn't find a solution.

### Environment info

- `datasets` = 2.18.0
- `h5py` = 3.10.0
- `fsspec` = 2023.10.0


When you pass `streaming=True`, the cache is ignored. The remote data URL is used instead and the data is streamed from the remote server.

Thanks for your reply! So is there any solution to get my expected behavior besides clone the whole repo ? Or could I adjust my script to load the downloaded arrow files and generate the dataset streamingly?
Support NumPy 2.0

### Feature request

Support NumPy 2.0.

### Motivation

NumPy introduces the Array API, which bridges the gap between machine learning libraries.  Many clients of HuggingFace are eager to start using the Array API.

Besides that, NumPy 2 provides a cleaner interface than NumPy 1.

### Tasks

NumPy 2.0 was released for testing so that libraries could ensure compatibility [since mid-March](https://github.com/numpy/numpy/issues/24300#issuecomment-1986815755).  What needs to be done for HuggingFace to support Numpy 2?

- [x] Fix use of `array`: https://github.com/huggingface/datasets/pull/6976
- [ ] Remove [NumPy version limit](https://github.com/huggingface/datasets/pull/6975): https://github.com/huggingface/datasets/pull/6991

Labels: name: enhancement
Register `.zstd` extension for zstd-compressed files

For example, https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0 dataset files have `.zstd` extension which is currently ignored (only `.zst` is registered).


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7032). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@albertvillanova hm I don't know tbh, it's just that "mlfoundations/dclm-baseline-1.0" dataset contains [files](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0/tree/main/global-shard_01_of_10/local-shard_0_of_10) with this extension and these files seem to be valid 

not sure why CI is failing but seems to be unrelated to this pr? can I merge @lhoestq @albertvillanova ?

yes you can merge, the CI failure is unrelated (surely an issue with hub-ci)

ah why not, you could try opening a PR

btw there is a channel with them at (internal) https://app.slack.com/client/T1RCG4490/C079AKTV11P if you want to let them know

@lhoestq, your previous comment was addressed to me or Polina?

@polinaeterna let me know if it is OK for you.

I opened https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0/discussions/7

Should we close this PR then?
Add option to disable progress bar when reading a dataset ("Loading dataset from disk")

### Feature request

Add an option in load_from_disk to disable the progress bar even if the number of files is larger than 16.

### Motivation

I am reading a lot of datasets that it creates lots of logs.

<img width="1432" alt="image" src="https://github.com/huggingface/datasets/assets/57996478/8d4bbf03-6b89-44b6-937c-932f01b4eb2a">


### Your contribution

Seems like an easy fix to make.  I can create a PR if necessary.

Labels: name: enhancement

You can disable progress bars for all of `datasets` with `disable_progress_bars`. [Link](https://huggingface.co/docs/datasets/en/package_reference/utilities#datasets.enable_progress_bars)

So you could do something like:

```python
from datasets import load_from_disk, enable_progress_bars, disable_progress_bars

disable_progress_bars()
# Your code
load_from_disk(....)

enable_progress_bars()
```


Thank you! Closing the issue.
ImportError: numpy.core.multiarray when using `filter`

### Describe the bug

I can't apply the filter method on my dataset. 

### Steps to reproduce the bug

The following snippet generates a bug: 

```python
from datasets import load_dataset 

ami = load_dataset('kamilakesbi/ami', 'ihm')

ami['train'].filter(
    lambda example: example["file_name"] == 'EN2001a'
)
```

I get the following error: 

`ImportError: numpy.core.multiarray failed to import (auto-generated because you didn't call 'numpy.import_array()' after cimporting numpy; use '<void>numpy._import_array' to disable if you are certain you don't need it).`

### Expected behavior

It should work properly! 

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0


Could you please check your `numpy` version?

I got this issue while using numpy version 2.0. 

I solved it by switching back to numpy 1.26.0 :) 

We recently added support for numpy 2.0, but it is not released yet.

Ok I see, thanks! I think we can close this issue for now as switching back to version 1.26.0 solves the problem :)