mengta666/huggingface_github huggingface dataset:
Issue: load_dataset ignores cached datasets and tries to hit HF Hub, resulting in API rate limit errors

### Describe the bug

I have been running lm-eval-harness a lot which has results in an API rate limit. This seems strange, since all of the data should be cached locally. I have in fact verified this.

### Steps to reproduce the bug

1. Be Me
2. Run `load_dataset("TAUR-Lab/MuSR")`
3. Hit rate limit error
4. Dataset is in .cache/huggingface/datasets
5. ???

### Expected behavior

We should not run into API rate limits if we have cached the dataset

### Environment info

datasets 2.16.0
python 3.10.4

Comments:
I'm having the same issue - running into rate limits when doing hyperparameter tuning even though the dataset is supposed to be cached. I feel like this behaviour should at the very least be documented, but honestly you should just not be running into rate limits in the first place when the dataset is cached. It even happens when specifying a specific revision for the dataset, in which case AFAIK there should be no reason to be doing API requests if it's already cached (besides maybe a quick hash check but hitting rate limits for that in ~200 requests across 10 hours of use seems a bit ridiculous).
Issue: Generating train split takes a long time

### Describe the bug

Loading a simple webdataset takes ~45 minutes.

### Steps to reproduce the bug

```
from datasets import load_dataset
dataset = load_dataset("PixArt-alpha/SAM-LLaVA-Captions10M")
```

### Expected behavior

The dataset should load immediately as it does when loaded through a normal indexed WebDataset loader. Generating splits should be optional and there should be a message showing how to disable it. 

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-4.18.0-372.32.1.el8_6.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.14
- `huggingface_hub` version: 0.24.1
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0

Comments:
@alexanderswerdlow 
When no specific split is mentioned, the load_dataset library will load all available splits of the dataset. For example, if a dataset has "train" and "test" splits, the load_dataset function will load both into the DatasetDict object.

![image](https://github.com/user-attachments/assets/379e6f57-7e1b-4cc3-bc36-dae3e878a51c)


The dataset PixArt-alpha/SAM-LLaVA-Captions10M may have been uploaded with different predefined splits (e.g., "train", "test", etc.), and by default, Hugging Face will load all splits unless you specifically request only one.

### If you want to load only a specific split (e.g., only the "train" set), you can specify it in the split parameter like this:
```python
from datasets import load_dataset
dataset = load_dataset("PixArt-alpha/SAM-LLaVA-Captions10M", split="train")
```

### You can also load multiple splits if needed:
```python
dataset = load_dataset("PixArt-alpha/SAM-LLaVA-Captions10M", split=["train", "test"])
```



@alexanderswerdlow, I will now work on this..

## Idea:
Whenever this code has ran:
```python
from datasets import load_dataset
dataset = load_dataset("PixArt-alpha/SAM-LLaVA-Captions10M")
```

It should show all the splits of the datasets, and user has to choose which one should be loaded before generating a split like this,,

![image](https://github.com/user-attachments/assets/8fbc604f-f0a5-4a59-a63e-aa4c26442c83)
Issue: column_names ignored by load_dataset() when loading CSV file

### Describe the bug

load_dataset() ignores the column_names kwarg when loading a CSV file.  Instead, it uses whatever values are on the first line of the file.


### Steps to reproduce the bug

Call `load_dataset` to load data from a CSV file and specify `column_names` kwarg.

### Expected behavior

The resulting dataset should have the specified column names **and** the first line of the file should be considered as data values.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.10.0-30-cloud-amd64-x86_64-with-glibc2.31
- Python version: 3.9.2
- `huggingface_hub` version: 0.24.2
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0

Comments:
I confirm that `column_names` values are not copied to `names` variable because in this case `CsvConfig.__post_init__` is not called: `CsvConfig` is instantiated with default values and afterwards the `config_kwargs` are used to overwrite its attributes.

@luismsgomes in the meantime, you can avoid the bug if you pass `names` instead of `column_names`.
Issue: Convert_to_parquet fails for datasets with multiple configs

If the dataset has multiple configs, when using the `datasets-cli convert_to_parquet` command to avoid issues with the data viewer caused by loading scripts, the conversion process only successfully converts the data corresponding to the first config. When it starts converting the second config, it throws an error:

```
Traceback (most recent call last):
  File "/opt/anaconda3/envs/dl/bin/datasets-cli", line 8, in <module>
    sys.exit(main())
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/datasets/commands/datasets_cli.py", line 41, in main
    service.run()
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/datasets/commands/convert_to_parquet.py", line 83, in run
    dataset.push_to_hub(
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/datasets/dataset_dict.py", line 1713, in push_to_hub
    api.create_branch(repo_id, branch=revision, token=token, repo_type="dataset", exist_ok=True)
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 5503, in create_branch
    hf_raise_for_status(response)
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 358, in hf_raise_for_status
    raise BadRequestError(message, response=response) from e
huggingface_hub.utils._errors.BadRequestError:  (Request ID: Root=1-669fc665-7c2e80d75f4337496ee95402;731fcdc7-0950-4eec-99cf-ce047b8d003f)

Bad request:
Invalid reference for a branch: refs/pr/1
```

Comments:
Many users have encountered the same issue, which has caused inconvenience.

https://discuss.huggingface.co/t/convert-to-parquet-fails-for-datasets-with-multiple-configs/86733


Thanks for reporting.

I will make the code more robust.

I have opened an issue in the huggingface-hub repo:
- https://github.com/huggingface/huggingface_hub/issues/2419

I am opening a PR to avoid calling `create_branch` if the branch already exists.
Issue: Datasets.datafiles resolve_pattern `TypeError: can only concatenate tuple (not "str") to tuple`

### Describe the bug

in data_files.py, line 332, 
`fs,  _, _ = get_fs_token_paths(pattern, storage_options=storage_options)`
If we run the code on AWS, as fs.protocol will be a tuple like: `('file', 'local')`
So,  `isinstance(fs.protocol, str) == False` and
`protocol_prefix = fs.protocol + "://" if fs.protocol != "file" else ""` will raise 
`TypeError: can only concatenate tuple (not "str") to tuple`.

### Steps to reproduce the bug

Steps to reproduce:
1. Run on a cloud server like AWS,
2. `import datasets.data_files as datafile`
3. datafile.resolve_pattern('path/to/dataset', '.')
4. `TypeError: can only concatenate tuple (not "str") to tuple`

### Expected behavior

Should return path of the dataset, with fs.protocol at the beginning

### Environment info

- `datasets` version: 2.14.0
- Platform: Linux-3.10.0-1160.119.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.19
- Huggingface_hub version: 0.23.5
- PyArrow version: 16.1.0
- Pandas version: 1.1.5

Comments:
Hi,

This issue was fixed in `datasets` 2.15.0:
- #6105

You will need to update your `datasets`:
```
pip install -U datasets
```

Duplicate of:
- #6100
Issue: Save nparray as list

### Describe the bug

When I use the `map` function to convert images into features, datasets saves nparray as a list. Some people use the `set_format` function to convert the column back, but doesn't this lose precision?

### Steps to reproduce the bug

the map function
```python
def convert_image_to_features(inst, processor, image_dir):
    image_file = inst["image_url"]
    file = image_file.split("/")[-1]
    image_path = os.path.join(image_dir, file)
    image = Image.open(image_path)
    image = image.convert("RGBA")

    inst["pixel_values"] = processor(images=image, return_tensors="np")["pixel_values"]
    return inst
```

main function
```python
map_fun = partial(
      convert_image_to_features, processor=processor, image_dir=image_dir
  )
ds = ds.map(map_fun, batched=False, num_proc=20)
print(type(ds[0]["pixel_values"])
```

### Expected behavior

(type < list>) 

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-4.19.91-009.ali4000.alios7.x86_64-x86_64-with-glibc2.35
- Python version: 3.11.5
- `huggingface_hub` version: 0.23.4
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0

Comments:
In addition, when I use `set_format ` and index the ds, the following error occurs:
the code
```python
ds.set_format(type="np", colums="pixel_values")
```
error
<img width="918" alt="image" src="https://github.com/user-attachments/assets/b28bbff2-20ea-4d28-ab62-b4ed2d944996">


>  Some people use the set_format function to convert the column back, but doesn't this lose precision?

Under the hood the data is saved in Arrow format using the same precision as your numpy arrays?
By default the Arrow data is read as python lists, but you can indeed read them back as numpy arrays with the same precision

(you can fix your second issue by fixing the typo `colums` -> `columns`)

> (you can fix your second issue by fixing the typo `colums` -> `columns`)

You are right, I was careless. Thank you.

> > Some people use the set_format function to convert the column back, but doesn't this lose precision?
> 
> Under the hood the data is saved in Arrow format using the same precision as your numpy arrays? By default the Arrow data is read as python lists, but you can indeed read them back as numpy arrays with the same precision

Yes, after testing I found that there was no loss of precision. Thanks again for your answer.
Issue: ImportError: numpy.core.multiarray when using `filter`

### Describe the bug

I can't apply the filter method on my dataset. 

### Steps to reproduce the bug

The following snippet generates a bug: 

```python
from datasets import load_dataset 

ami = load_dataset('kamilakesbi/ami', 'ihm')

ami['train'].filter(
    lambda example: example["file_name"] == 'EN2001a'
)
```

I get the following error: 

`ImportError: numpy.core.multiarray failed to import (auto-generated because you didn't call 'numpy.import_array()' after cimporting numpy; use '<void>numpy._import_array' to disable if you are certain you don't need it).`

### Expected behavior

It should work properly! 

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0

Comments:
Could you please check your `numpy` version?

I got this issue while using numpy version 2.0. 

I solved it by switching back to numpy 1.26.0 :) 

We recently added support for numpy 2.0, but it is not released yet.

Ok I see, thanks! I think we can close this issue for now as switching back to version 1.26.0 solves the problem :)
Issue: Save Dataset as Sharded Parquet 

### Feature request

`to_parquet` currently saves the dataset as one massive, monolithic parquet file, rather than as several small parquet files. It should shard large datasets automatically.

### Motivation

This default behavior makes me very sad because a program I ran for 6 hours saved its results using `to_parquet`, putting the entire billion+ row dataset into a 171 GB *single shard parquet file* which pyarrow, apache spark, etc. all cannot work with without completely exhausting the memory of my system. I was previously able to work with larger-than-memory parquet files, but not this one. I *assume* the reason why this is happening is because it is a single shard. Making sharding the default behavior puts datasets in parity with other frameworks, such as spark, which automatically shard when a large dataset is saved as parquet. 

### Your contribution

I could change the logic here https://github.com/huggingface/datasets/blob/bf6f41e94d9b2f1c620cf937a2e85e5754a8b960/src/datasets/io/parquet.py#L109-L158
to use `pyarrow.dataset.write_dataset`, which seems to support sharding, or periodically open new files. We would only shard if the user passed in a path rather than file handle. 

Comments:
To anyone else who finds themselves in this predicament, it's possible to read the parquet file in the same way that datasets writes it, and then manually break it into pieces. Although, you need a couple of magic options (`thrift_*`) to deal with the huge metadata, otherwise pyarrow immediately crashes.
```python
import pyarrow.parquet as pq
import pyarrow as pa

r = pq.ParquetReader()

r.open("./outrageous-file.parquet",thrift_string_size_limit=2**31-1, thrift_container_size_limit=2**31-1)

from more_itertools import chunked
import tqdm

for i,chunk in tqdm.tqdm(enumerate(chunked(range(r.num_row_groups),10000))):
    w = pq.ParquetWriter(f"./chunks.parquet/chunk{i}.parquet",schema=r.schema_arrow)
    for idx in chunk:
        w.write_table(r.read_row_group(idx))
    w.close()
```

You can also use `.shard()` and call `to_parquet()` on each shard in the meantime:

```python
num_shards = 128
output_path_template = "output_dir/{index:05d}.parquet"
for index in range(num_shards):
    shard = ds.shard(index=index, num_shards=num_shards, contiguous=True)
    shard.to_parquet(output_path_template.format(index=index))
```
Issue: `sort` after `filter` unreasonably  slow

### Describe the bug

as the tittle says ...

### Steps to reproduce the bug

`sort` seems to be normal.

```python
from datasets import Dataset
import random

nums = [{"k":random.choice(range(0,1000))} for _ in range(100000)]

ds = Dataset.from_list(nums)

print("start sort")

ds = ds.sort("k")

print("finish sort")
```

but `sort` after `filter` is extremely slow.

```python
from datasets import Dataset
import random

nums = [{"k":random.choice(range(0,1000))} for _ in range(100000)]

ds = Dataset.from_list(nums)

ds = ds.filter(lambda x:x > 100, input_columns="k")

print("start sort")

ds = ds.sort("k")

print("finish sort")
```


### Expected behavior

Is this a bug, or is it a misuse of the `sort` function?

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-3.10.0-1127.19.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.13
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2023.10.0

Comments:
`filter` add an indices mapping on top of the dataset, so `sort` has to gather all the rows that are kept to form a new Arrow table and sort the table. Gathering all the rows can take some time, but is a necessary step. You can try calling `ds = ds.flatten_indices()` before sorting to remove the indices mapping.

> `filter` add an indices mapping on top of the dataset, so `sort` has to gather all the rows that are kept to form a new Arrow table and sort the table. Gathering all the rows can take some time, but is a necessary step. You can try calling `ds = ds.flatten_indices()` before sorting to remove the indices mapping.

This worked, thank you so much.
Issue: load `streaming=True` dataset with downloaded cache

### Describe the bug

We build a dataset which contains several hdf5 files and write a script using `h5py` to generate the dataset. The hdf5 files are large and the processed dataset cache takes more disk space. So we hope to try streaming iterable dataset. Unfortunately, `h5py` can't convert a remote URL into a hdf5 file descriptor. So we use `fsspec` as an interface like below:

```python
def _generate_examples(self, filepath, split):
        for file in filepath:
            with fsspec.open(file, "rb") as fs:
                with h5py.File(fs, "r") as fp:
                    # for event_id in sorted(list(fp.keys())):
                    event_ids = list(fp.keys())
                    ......
```

### Steps to reproduce the bug

The `fsspec` works, but it takes 10+ min to print the first 10 examples, which is even longer than the downloading time. I'm not sure if it just caches the whole hdf5 file and generates the examples.

### Expected behavior

So does the following make sense so far?

1. download the files
```python
dataset = datasets.load('path/to/myscripts', split="train", name="event", trust_remote_code=True)
```

2. load the iterable dataset faster (using the raw file cache at path `.cache/huggingface/datasets/downloads`)
```python
dataset = datasets.load('path/to/myscripts', split="train", name="event", trust_remote_code=True, streaming=true)
```

I made some tests, but the code above can't get the expected result. I'm not sure if this is supported. I also find the issue #6327 . It seemed similar to mine, but I couldn't find a solution.

### Environment info

- `datasets` = 2.18.0
- `h5py` = 3.10.0
- `fsspec` = 2023.10.0

Comments:
When you pass `streaming=True`, the cache is ignored. The remote data URL is used instead and the data is streamed from the remote server.

Thanks for your reply! So is there any solution to get my expected behavior besides clone the whole repo ? Or could I adjust my script to load the downloaded arrow files and generate the dataset streamingly?
Issue: A bug of Dataset.to_json() function

### Describe the bug

When using the Dataset.to_json() function, an unexpected error occurs if the parameter is set to lines=False. The stored data should be in the form of a list, but it actually turns into multiple lists, which causes an error when reading the data again.
The reason is that to_json() writes to the file in several segments based on the batch size. This is not a problem when lines=True, but it is incorrect when lines=False, because writing in several times will produce multiple lists(when len(dataset) > batch_size).

### Steps to reproduce the bug

try this code:
```python
from datasets import load_dataset
import json

train_dataset = load_dataset("Anthropic/hh-rlhf", data_dir="harmless-base")["train"]
output_path = "./harmless-base_hftojs.json"
print(len(train_dataset))
train_dataset.to_json(output_path, lines=False, force_ascii=False, indent=2)

with open(output_path, encoding="utf-8") as f:
    data = json.loads(f.read())

```
it raise error: json.decoder.JSONDecodeError: Extra data: line 4003 column 1 (char 1373709)

Extra square brackets have appeared here:
<img width="265" alt="image" src="https://github.com/huggingface/datasets/assets/26499566/81492332-386d-42e8-88d1-b6d4ae3682cc">


### Expected behavior

The code runs normally.

### Environment info

datasets=2.20.0

Comments:
Thanks for reporting, @LinglingGreat.

I confirm this is a bug.

@albertvillanova I would like to take a shot at this if you aren't working on it currently. Let me know!
Issue: `from_generator` does not allow to specify the split name

### Describe the bug

I'm building train, dev, and test using `from_generator`; however, in all three cases, the logger prints `Generating train split:`
It's not possible to change the split name since it seems to be hardcoded: https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/generator/generator.py

### Steps to reproduce the bug

```
In [1]: from datasets import Dataset

In [2]: def gen():
   ...:     yield {"pokemon": "bulbasaur", "type": "grass"}
   ...: 

In [3]: ds = Dataset.from_generator(gen)
Generating train split: 1 examples [00:00, 133.89 examples/s]
```

### Expected behavior

It should be possible to specify any split name

### Environment info

- `datasets` version: 2.19.2
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.5
- `huggingface_hub` version: 0.23.3
- PyArrow version: 15.0.0
- Pandas version: 2.0.3
- `fsspec` version: 2023.10.0

Comments:
Thanks for reporting, @pminervini.

I agree we should give the option to define the split name.

Indeed, there is a PR that addresses precisely this issue:
- #7015

I am reviewing it.

Booom! thank you guys :)
Issue: Add option to disable progress bar when reading a dataset ("Loading dataset from disk")

### Feature request

Add an option in load_from_disk to disable the progress bar even if the number of files is larger than 16.

### Motivation

I am reading a lot of datasets that it creates lots of logs.

<img width="1432" alt="image" src="https://github.com/huggingface/datasets/assets/57996478/8d4bbf03-6b89-44b6-937c-932f01b4eb2a">


### Your contribution

Seems like an easy fix to make.  I can create a PR if necessary.

Comments:
You can disable progress bars for all of `datasets` with `disable_progress_bars`. [Link](https://huggingface.co/docs/datasets/en/package_reference/utilities#datasets.enable_progress_bars)

So you could do something like:

```python
from datasets import load_from_disk, enable_progress_bars, disable_progress_bars

disable_progress_bars()
# Your code
load_from_disk(....)

enable_progress_bars()
```


Thank you! Closing the issue.
Issue: load_dataset on AWS lambda throws OSError(30, 'Read-only file system') error

### Describe the bug

I'm using AWS lambda to run a python application. I run the `load_dataset` function with cache_dir="/tmp" and is still throws the OSError(30, 'Read-only file system') error. Is even updated all the HF envs to point to /tmp dir but the issue still persists. I can confirm that the I can write to /tmp directory. 

### Steps to reproduce the bug

```python
d = load_dataset(
      path=hugging_face_link,
      split=split,
      token=token,
      cache_dir="/tmp/hugging_face_cache",
 )
```

### Expected behavior

Everything written to the file system as part of the load_datasets function should be in the /tmp directory. 

### Environment info

datasets version: 2.16.1
Platform: Linux-5.10.216-225.855.amzn2.x86_64-x86_64-with-glibc2.26
Python version: 3.11.9
huggingface_hub version: 0.19.4
PyArrow version: 16.1.0
Pandas version: 2.2.2
fsspec version: 2023.10.0

Comments:
hi ! can you share the full stack trace ? this should help locate what files is not written in the cache_dir
Issue: `load_dataset` fails to load dataset saved by `save_to_disk`

### Describe the bug

This code fails to load the dataset it just saved:

```python
from datasets import load_dataset
from transformers import AutoTokenizer

MODEL = "google-bert/bert-base-cased"
tokenizer = AutoTokenizer.from_pretrained(MODEL)

dataset = load_dataset("yelp_review_full")

def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenized_datasets = dataset.map(tokenize_function, batched=True)
tokenized_datasets.save_to_disk("dataset")

tokenized_datasets = load_dataset("dataset/")  # raises
```

It raises `ValueError: Couldn't infer the same data file format for all splits. Got {NamedSplit('train'): ('arrow', {}), NamedSplit('test'): ('json', {})}`.

I believe this bug is caused by the [logic that tries to infer dataset format](https://github.com/huggingface/datasets/blob/9af8dd3de7626183a9a9ec8973cebc672d690400/src/datasets/load.py#L556). It counts the most common file extension. However, a small dataset can fit in a single `.arrow` file and have two JSON metadata files, causing the format to be inferred as JSON:

```shell
$ ls -l dataset/test
-rw-r--r-- 1 sliedes sliedes 191498784 Jul  1 13:55 data-00000-of-00001.arrow
-rw-r--r-- 1 sliedes sliedes      1730 Jul  1 13:55 dataset_info.json
-rw-r--r-- 1 sliedes sliedes       249 Jul  1 13:55 state.json
```

### Steps to reproduce the bug

Execute the code above.

### Expected behavior

The dataset is loaded successfully.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-6.9.3-arch1-1-x86_64-with-glibc2.39
- Python version: 3.12.4
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0


Comments:
In my case the error was:
```
ValueError: You are trying to load a dataset that was saved using `save_to_disk`. Please use `load_from_disk` instead.
```
Did you try `load_from_disk`?

More generally, any reason there is no API consistency between save_to_disk and push_to_hub ? 

Would be nice to be able to save_to_disk and then upload manually to the hub and load_dataset (which works in some situations but not all)...

I have the exact same problem !

`load_from_disk` managed to load the dataset, but the bug with `load_dataset` needs to be fixed. 

any update ? I need some function from load dataset like num_proc, or streaming to optimize ram, ... but got this error
Issue: Datasetbuilder Local Download FileNotFoundError

### Describe the bug

So I was trying to download a dataset and save it as parquet and I follow the [tutorial](https://huggingface.co/docs/datasets/filesystems#download-and-prepare-a-dataset-into-a-cloud-storage) of Huggingface. However, during the excution I face a FileNotFoundError.

I debug the code and it seems there is a bug there:
So first it creates a .incomplete folder and before moving its contents the following code deletes the directory
[Code](https://github.com/huggingface/datasets/blob/98fdc9e78e6d057ca66e58a37f49d6618aab8130/src/datasets/builder.py#L984)
hence as a result I face with: 

``` FileNotFoundError: [Errno 2] No such file or directory: '~/data/Parquet/.incomplete '```

### Steps to reproduce the bug

```
from datasets import load_dataset_builder
from pathlib import Path

parquet_dir = "~/data/Parquet/" 
Path(parquet_dir).mkdir(parents=True, exist_ok=True)
builder = load_dataset_builder(
    "rotten_tomatoes",
)
builder.download_and_prepare(parquet_dir, file_format="parquet")
```

### Expected behavior

Downloads the files and saves as parquet

### Environment info

Ubuntu,
Python 3.10
```
datasets 2.19.1
```

Comments:
Ok it seems the solution is to use the directory string without the trailing "/" which in my case  as: 

`parquet_dir = "~/data/Parquet" `

Still i think this is a weird behavior...
Issue: IterableDataset: Unsupported ScalarType BFloat16

### Describe the bug

`IterableDataset.from_generator` crashes when using BFloat16:

```
  File "/usr/local/lib/python3.11/site-packages/datasets/utils/_dill.py", line 169, in _save_torchTensor
    args = (obj.detach().cpu().numpy(),)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^

TypeError: Got unsupported ScalarType BFloat16
```

### Steps to reproduce the bug

```python
import torch
from datasets import IterableDataset


def demo(x):
    yield {"x": x}


x = torch.tensor([1.], dtype=torch.bfloat16)

dataset = IterableDataset.from_generator(
    demo,
    gen_kwargs=dict(x=x),
)

example = next(iter(dataset))
print(example)
```

### Expected behavior

Code sample should print:
```python
{'x': tensor([1.], dtype=torch.bfloat16)}
```

### Environment info

```
datasets==2.20.0
torch==2.2.2
```

Comments:
@lhoestq Thank you for merging #6607, but unfortunately the issue persists for `IterableDataset` :pensive: 

Hi ! I opened https://github.com/huggingface/datasets/pull/7002 to fix this bug

Amazing, thank you so much @lhoestq! :pray:
Issue: Problematic rank after calling `split_dataset_by_node` twice

### Describe the bug

I'm trying to split `IterableDataset` by `split_dataset_by_node`.
But when doing split on a already split dataset, the resulting `rank` is greater than `world_size`.

### Steps to reproduce the bug

Here is the minimal code for reproduction:
```py
>>> from datasets import load_dataset
>>> from datasets.distributed import split_dataset_by_node
>>> dataset = load_dataset('fla-hub/slimpajama-test', split='train', streaming=True)      
>>> dataset = split_dataset_by_node(dataset, 1, 32)
>>> dataset._distributed
DistributedConfig(rank=1, world_size=32)
>>> dataset = split_dataset_by_node(dataset, 1, 15)
>>> dataset._distributed                           
DistributedConfig(rank=481, world_size=480)
```
As you can see, the second rank 481 > 480, which is problematic.

### Expected behavior

I think this error comes from this line @lhoestq
https://github.com/huggingface/datasets/blob/a6ccf944e42c1a84de81bf326accab9999b86c90/src/datasets/iterable_dataset.py#L2943-L2944
We may need to obtain the rank first. Then the above code gives
```py
>>> dataset._distributed                           
DistributedConfig(rank=16, world_size=480)
```

### Environment info

datasets==2.20.0

Comments:
ah yes good catch ! feel free to open a PR with your suggested fix
Issue: Convert polars DataFrame back to datasets

### Feature request

This returns error.
```python
from datasets import Dataset

dsdf = Dataset.from_dict({"x": [[1, 2], [3, 4, 5]], "y": ["a", "b"]})
Dataset.from_polars(dsdf.to_polars())
```
ValueError: Arrow type large_list<item: int64> does not have a datasets dtype equivalent.

### Motivation

When datasets contain Sequence data type, it will be converted to Arrow type large_list. However, the reverse (from large_list to Sequence) does not work.

### Your contribution

No

Comments:
Hi ! Thanks for reporting :)

We don't support `large_list` yet, though it should be added to `Sequence` IMO (maybe with a parameter `large=True` ?)
Issue: IndexError during training with Squad dataset and T5-small model

### Describe the bug

I am encountering an IndexError while training a T5-small model on the Squad dataset using the transformers and datasets libraries. The error occurs even with a minimal reproducible example, suggesting a potential bug or incompatibility.

### Steps to reproduce the bug

1.Install the required libraries: !pip install transformers datasets
2.Run the following code:
!pip install transformers datasets

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, TrainingArguments, Trainer, DataCollatorWithPadding

# Load a small, publicly available dataset
from datasets import load_dataset
dataset = load_dataset("squad", split="train[:100]") # Use a small subset for testing

# Load a pre-trained model and tokenizer
model_name = "t5-small"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# Define a basic data collator
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

# Define training arguments
training_args = TrainingArguments(
    output_dir="./results",
    per_device_train_batch_size=2,
    num_train_epochs=1,
)

# Create a trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    data_collator=data_collator,
)

# Train the model
trainer.train()

### Expected behavior

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
[<ipython-input-23-f13a4b23c001>](https://localhost:8080/#) in <cell line: 34>()
     32 
     33 # Train the model
---> 34 trainer.train()

10 frames
[/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py](https://localhost:8080/#) in _check_valid_index_key(key, size)
    427     if isinstance(key, int):
    428         if (key < 0 and key + size < 0) or (key >= size):
--> 429             raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
    430         return
    431     elif isinstance(key, slice):

IndexError: Invalid key: 42 is out of bounds for size 0

### Environment info

transformers version:4.41.2
datasets version:1.18.4
Python version:3.10.12


Comments:
add remove_unused_columns=False to training_args
https://github.com/huggingface/datasets/issues/6535#issuecomment-1874024704

Closing this issue because it was a reported and fixed in transformers.
Issue: Manual downloads should count as downloads

### Feature request

I would like to request that manual downloads of data files from Hugging Face dataset repositories count as downloads of a dataset. According to the documentation for the Hugging Face Hub, that is currently not the case: https://huggingface.co/docs/hub/en/datasets-download-stats

### Motivation

This would ensure that downloads are accurately reported to end users.

### Your contribution

N/A

Comments:
We're unlikely to add more features/support for datasets with python loading scripts, which include datasets with manual download. Sorry for the inconvenience
Issue: My Private Dataset doesn't exist on the Hub or cannot be accessed

### Describe the bug


```
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 1852, in dataset_module_factory
    raise DatasetNotFoundError(msg + f" at revision '{revision}'" if revision else msg)
datasets.exceptions.DatasetNotFoundError: Dataset 'xxx' doesn't exist on the Hub or cannot be accessed
>>> dataset = load_dataset("xxxx", token=True)

404 error 404 Client Error. (Request ID: Root=xxxx)

Repository Not Found for url: https://huggingface.co/api/datasets/xxx/xxx.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 2593, in load_dataset
    builder_instance = load_dataset_builder(
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 2265, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 1910, in dataset_module_factory
    raise e1 from None
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 1852, in dataset_module_factory
    raise DatasetNotFoundError(msg + f" at revision '{revision}'" if revision else msg)
datasets.exceptions.DatasetNotFoundError: Dataset 'xxx' doesn't exist on the Hub or cannot be accessed
```

### Steps to reproduce the bug

123

### Expected behavior

123

### Environment info

123

Comments:
I can load public dataset, but for my private dataset it fails

https://huggingface.co/docs/datasets/upload_dataset

I have checked the API HTTP link. Repository Not Found for url: https://huggingface.co/api/datasets/xxx/xxx.

![image](https://github.com/huggingface/datasets/assets/39621324/4aceef59-0c65-4161-9665-676d25d73225)

It just works fine.

It seems that everything is in a mass huh....

![image](https://github.com/huggingface/datasets/assets/39621324/fb2fe12c-4f0a-4bf6-9656-63ba50347b10)


https://huggingface.co/datasets/rajpurkar/squad/blob/main/squad.py fails again

https://github.com/huggingface/datasets/blob/main/templates/new_dataset_script.py#L81 can not use this, too complex. I just need a def to load my file to a dict

I am facing the same issue. Did you find a fix?

You should authenticate to be able to access private or gated repos: https://huggingface.co/docs/hub/datasets-gated#access-gated-datasets-as-a-user
Issue: `Dataset.with_format` behaves inconsistently with documentation

### Describe the bug

The actual behavior of the interface `Dataset.with_format` is inconsistent with the documentation.
https://huggingface.co/docs/datasets/use_with_pytorch#n-dimensional-arrays
https://huggingface.co/docs/datasets/v2.19.0/en/use_with_tensorflow#n-dimensional-arrays

> If your dataset consists of N-dimensional arrays, you will see that by default they are considered as nested lists.
> In particular, a PyTorch formatted dataset outputs nested lists instead of a single tensor.
> A TensorFlow formatted dataset outputs a RaggedTensor instead of a single tensor.

But I get a single tensor by default, which is  inconsistent with the description.

Actually the current behavior seems more reasonable to me. Therefore, the document needs to be modified.

### Steps to reproduce the bug

```python
>>> from datasets import Dataset
>>> data = [[[1, 2],[3, 4]],[[5, 6],[7, 8]]]
>>> ds = Dataset.from_dict({"data": data})
>>> ds = ds.with_format("torch")
>>> ds[0]
{'data': tensor([[1, 2],
        [3, 4]])}
>>> ds = ds.with_format("tf")
>>> ds[0]
{'data': <tf.Tensor: shape=(2, 2), dtype=int64, numpy=
array([[1, 2],
       [3, 4]])>}
```

### Expected behavior

```python
>>> from datasets import Dataset
>>> data = [[[1, 2],[3, 4]],[[5, 6],[7, 8]]]
>>> ds = Dataset.from_dict({"data": data})
>>> ds = ds.with_format("torch")
>>> ds[0]
{'data': [tensor([1, 2]), tensor([3, 4])]}
>>> ds = ds.with_format("tf")
>>> ds[0]
{'data': <tf.RaggedTensor [[1, 2], [3, 4]]>}
```

### Environment info

datasets==2.19.1
torch==2.1.0
tensorflow==2.13.1

Comments:
Hi ! It seems the documentation was outdated in this paragraph

I fixed it here: https://github.com/huggingface/datasets/pull/6956

Fixed.
Issue: load_dataset error

### Describe the bug

Why does the program get stuck when I use load_dataset method, and it still gets stuck after loading for several hours? In fact, my json file is only 21m, and I can load it in one go using open('', 'r').

### Steps to reproduce the bug

1. pip install datasets==2.19.2
2. from datasets import Dataset, DatasetDict, NamedSplit, Split, load_dataset
3. data = load_dataset('json', data_files='train.json')

### Expected behavior

It is able to load my json correctly

### Environment info

datasets==2.19.2

Comments:
Hi, @lion-ops.

In our Continuous Integration we have many tests on loading JSON files and all of them work properly.

Could you please share your "train.json" file, so that we can try to reproduce the issue you have? 

> Hi, @lion-ops.
> 
> In our Continuous Integration we have many tests on loading JSON files and all of them work properly.
> 
> Could you please share your "train.json" file, so that we can try to reproduce the issue you have?

Thank you for your reply. I can load it normally in another server. Is it possible that the disk of my server is a network disk in the LAN, so it will be downloaded from the LAN and get stuck?
Issue: save_to_disk() freezes when saving on s3 bucket with multiprocessing

### Describe the bug

I'm trying to save a `Dataset` using the `save_to_disk()` function with:
- `num_proc > 1`
- `dataset_path` being a s3 bucket path e.g. "s3://{bucket_name}/{dataset_folder}/"

The hf progress bar shows up but the saving does not seem to start. 
When using one processor only (`num_proc=1`), everything works fine.
When saving the dataset on local disk (as opposed to s3 bucket) with `num_proc > 1`, everything works fine.

Thank you for your help! :)

### Steps to reproduce the bug

I tried without any storage options:

```
from datasets import load_dataset

sandbox_ds = load_dataset("openai_humaneval")
sandbox_ds["test"].save_to_disk(
    "s3://bucket-name/test_multiprocessing_saving/",
    num_proc=4,
)
```


and with the specific s3fs storage options: 
```
from datasets import load_dataset
from s3fs import S3FileSystem

def get_s3fs():
    return S3FileSystem()

sandbox_ds = load_dataset("openai_humaneval")
sandbox_ds["test"].save_to_disk(
    "s3://bucket-name/test_multiprocessing_saving/",
    num_proc=4,
    storage_options=get_s3fs().storage_options, # also tried: storage_options=S3FileSystem().storage_options
)
```

I'm guessing I might use `storage_options` parameter wrongly, but I didn't find anything online that made it work.

**NB**: Behavior is the same when trying to save the whole `DatasetDict`.

### Expected behavior

Progress bar fills in and saving is carried out.

### Environment info

`datasets==2.18.0`

Comments:
I got the same issue. Any updates so far for this issue?

Same here. Any updates?

+1, experiencing this as well
Issue: Support for pathlib.Path in datasets 2.19.0

### Describe the bug

After the recent update of `datasets`, Dataset.save_to_disk does not accept a pathlib.Path anymore. It was supported in 2.18.0 and previous versions. Is this intentional? Was it supported before only because of a Python dusk-typing miracle?

### Steps to reproduce the bug

```
from datasets import Dataset
import pathlib

path = pathlib.Path("./my_out_path")
Dataset.from_dict(
    {"text": ["hello world"], "label": [777], "split": ["train"]}
.save_to_disk(path)
```
This results in an error when using datasets 2.19:
```
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/Users/jb/scratch/venv/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 1515, in save_to_disk
    fs, _ = url_to_fs(dataset_path, **(storage_options or {}))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jb/scratch/venv/lib/python3.11/site-packages/fsspec/core.py", line 383, in url_to_fs
    chain = _un_chain(url, kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jb/scratch/venv/lib/python3.11/site-packages/fsspec/core.py", line 323, in _un_chain
    if "::" in path
       ^^^^^^^^^^^^
TypeError: argument of type 'PosixPath' is not iterable
```

Converting to str works, however.
```
Dataset.from_dict(
     {"text": ["hello world"], "label": [777], "split": ["train"]}
).save_to_disk(str(path))
```

### Expected behavior

My dataset gets saved to disk without an error.

### Environment info

aiohttp==3.9.5
aiosignal==1.3.1
attrs==23.2.0
certifi==2024.2.2
charset-normalizer==3.3.2
datasets==2.19.0
dill==0.3.8
filelock==3.14.0
frozenlist==1.4.1
fsspec==2024.3.1
huggingface-hub==0.23.2
idna==3.7
multidict==6.0.5
multiprocess==0.70.16
numpy==1.26.4
packaging==24.0
pandas==2.2.2
pyarrow==16.1.0
pyarrow-hotfix==0.6
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
requests==2.32.3
six==1.16.0
tqdm==4.66.4
typing_extensions==4.12.0
tzdata==2024.1
urllib3==2.2.1
xxhash==3.4.1
yarl==1.9.4

Comments:
+1 I just noticed this when I tried to update `datasets` today.

The same issue, I also get error.
Issue: ValueError: Couldn't infer the same data file format for all splits. Got {'train': ('json', {}), 'validation': (None, {})}

### Describe the bug

When I run the code en = load_dataset("allenai/c4", "en", streaming=True), I encounter an error: raise ValueError(f"Couldn't infer the same data file format for all splits. Got {split_modules}") ValueError: Couldn't infer the same data file format for all splits. Got {'train': ('json', {}), 'validation': (None, {})}.
However, running dataset = load_dataset('allenai/c4', streaming=True, data_files={'validation': 'en/c4-validation.00003-of-00008.json.gz'}, split='validation') works fine. What is the issue here?

### Steps to reproduce the bug

run code：
import os
os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'
from datasets import load_dataset

en = load_dataset("allenai/c4", "en", streaming=True)

### Expected behavior

Successfully loaded the dataset.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-6.5.0-28-generic-x86_64-with-glibc2.17
- Python version: 3.8.19
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.0.3
- `fsspec` version: 2024.2.0


Comments:
How do you solve it ?


> How do you solve it ?

Please check your Python environment and dataset version. I have just resolved the issue, which was caused by a Python environment switching error
Issue: Avoid downloading the whole dataset when only README.me has been touched on hub.

### Feature request

`datasets.load_dataset()` triggers a new download of the **whole dataset** when the README.md file has been touched on huggingface hub, even if data files / parquet files are the exact same.

I think the current behaviour of the load_dataset function is triggered whenever a change of the hash of latest commit on huggingface hub, but  is there a clever way to only download again the dataset **if and only if** data is modified ?  

### Motivation

The current behaviour is a waste of network bandwidth / disk space / research time.

### Your contribution

I don't have time to submit a PR, but I hope a simple solution will emerge from this issue ! 

Comments:
you're right, we're tackling this here: https://github.com/huggingface/dataset-viewer/issues/2757

@severo : great !
Issue: Add MedImg for streaming

### Feature request

Host the MedImg dataset (similar to Imagenet but for biomedical images). 

### Motivation

There is a clear need for biomedical image foundation models and large scale biomedical datasets that are easily streamable. This would be an excellent tool for the biomedical community.

### Your contribution

MedImg can be found [here](https://www.cuilab.cn/medimg/#).

Comments:
@mariosasko, @lhoestq, @albertvillanova
Hello! Can anyone help? or can you guys suggest who can help with this?

Hi ! Feel free to download the dataset and create a `Dataset` object with it.

Then your'll be able to use `push_to_hub()` to upload the dataset to HF in Parquet format and make it streamable :)

> Hi ! Feel free to download the dataset and create a `Dataset` object with it.
> 
> Then your'll be able to use `push_to_hub()` to upload the dataset to HF in Parquet format and make it streamable :)

The dataset is several TB in total, which I do not have the resources to handle.

Hi @lhoestq and @albertvillanova , just following up about this.

for big datasets you can push_to_hub one part at a time (e.g. as different splits) and merge the parts (just a simple modification in the YAML part of the README)

Sure, that makes sense. However, isn't there a size limit to what typical users can push?

Yes there is a limit, simply let us know by email at datasets [at] huggingface.co - this way we can give you a storage grant also help making sure the dataset is all good for people to use it easily

> Yes there is a limit, simply let us know by email at datasets [at] huggingface.co - this way we can give you a storage grant also help making sure the dataset is all good for people to use it easily

Got it, that would be great.