NyssaW/github-issues huggingface dataset:
Incorrect test set labels for RTE and CoLA datasets via load_dataset

### Describe the bug

The test set labels for the RTE and CoLA datasets when loading via datasets load_dataset are all -1.
Edit: It appears this is also the case for every other dataset except for MRPC (stsb, sst2, qqp, mnli (both matched and mismatched), qnli, wnli, ax). Is this intended behavior to safeguard the test set for evaluation purposes?

### Steps to reproduce the bug

!pip install datasets
from datasets import load_dataset

rte_data = load_dataset('glue', 'rte')
cola_data = load_dataset('glue', 'cola')
print(rte_data['test'][0:30]['label'])
print(cola_data['test'][0:30]['label'])

Output:
[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]
[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]

The non-label test data seems to be fine:
e.g. rte_data['test'][1] is:
{'sentence1': "Authorities in Brazil say that more than 200 people are being held hostage in a prison in the country's remote, Amazonian-jungle state of Rondonia.",
 'sentence2': 'Authorities in Brazil hold 200 people as hostage.',
 'label': -1,
 'idx': 1}
Training and validation data are also fine:
e.g. rte_data['train][0] is:
{'sentence1': 'No Weapons of Mass Destruction Found in Iraq Yet.',
 'sentence2': 'Weapons of Mass Destruction Found in Iraq.',
 'label': 1,
 'idx': 0}


### Expected behavior

Expected the labels to be binary 0/1 values; Got all -1s instead

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0

Comments:
As this is a specific issue of the "glue" dataset, I have transferred it to the dataset Discussion page: https://huggingface.co/datasets/glue/discussions/15

Let's continue the discussion there!
Error Pushing to Hub

### Describe the bug

Error when trying to push a dataset in a special format to hub

### Steps to reproduce the bug

```
import datasets
from datasets import Dataset
dataset_dict = {
    "filename": ["apple", "banana"],
    "token": [[[1,2],[3,4]],[[1,2],[3,4]]],
    "label": [0, 1],
}
dataset = Dataset.from_dict(dataset_dict)
dataset = dataset.cast_column("token", datasets.features.features.Array2D(shape=(2, 2),dtype="int16"))

dataset.push_to_hub("SequenceModel/imagenet_val_256")
```
Error:
```
...

ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/tuple'
  in "<unicode string>", line 8, column 16:
            shape: !!python/tuple
                   ^
```

### Expected behavior

Dataset being pushed to hub

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-5.19.0-1022-gcp-x86_64-with-glibc2.35
- Python version: 3.11.5
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0

Comments:
OverflowError: value too large to convert to int32_t 

### Describe the bug

![image](https://github.com/huggingface/datasets/assets/47747764/f58044fb-ddda-48b6-ba68-7bbfef781630)


### Steps to reproduce the bug

just loading datasets 

### Expected behavior

how can I fix it

### Environment info

pip install /mnt/cluster/zhangfan/study_info/LLaMA-Factory/peft-0.6.0-py3-none-any.whl
pip install huggingface_hub-0.19.4-py3-none-any.whl tokenizers-0.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl transformers-4.36.1-py3-none-any.whl pyarrow_hotfix-0.6-py3-none-any.whl datasets-2.15.0-py3-none-any.whl tyro-0.5.18-py3-none-any.whl trl-0.7.4-py3-none-any.whl

done

Comments:
+1
Error when writing a dataset to HF Hub: A commit has happened since. Please refresh and try again.

**Describe the bug**

Getting a `412 Client Error: Precondition Failed` when trying to write a dataset to the HF hub.

```
huggingface_hub.utils._errors.HfHubHTTPError: 412 Client Error: Precondition Failed for url: https://huggingface.co/api/datasets/GLorr/test-dask/commit/main (Request ID: Root=1-657ae26f-3bd92bf861bb254b2cc0826c;50a09ab7-9347-406a-ba49-69f98abee9cc)

A commit has happened since. Please refresh and try again.
```

**Steps to reproduce the bug**
This is a minimal reproducer:
```
import dask.dataframe as dd
import pandas as pd
import random
import os

import huggingface_hub

import datasets

huggingface_hub.login(token=os.getenv("HF_TOKEN"))

data = {"number": [random.randint(0,10) for _ in range(1000)]}
df = pd.DataFrame.from_dict(data)
dataframe = dd.from_pandas(df, npartitions=1)
dataframe = dataframe.repartition(npartitions=3)

schema = datasets.Features({"number": datasets.Value("int64")}).arrow_schema

repo_id = "GLorr/test-dask"
repo_path = f"hf://datasets/{repo_id}"
huggingface_hub.create_repo(repo_id=repo_id, repo_type="dataset", exist_ok=True)
dd.to_parquet(dataframe, path=f"{repo_path}/data", schema=schema)
```

**Expected behavior**
Would expect to write to the hub without any problem.

**Environment info**
```
datasets==2.15.0
huggingface-hub==0.19.4
```


Comments:
I transferred from datasets-server, since the issue is more about `datasets` and the integration with `huggingface_hub`.
Image Data loaded Twice

### Describe the bug

![1702472610561](https://github.com/huggingface/datasets/assets/28867010/4b7ef5e7-32c3-4b73-84cb-5de059caa0b6)
When I learn from https://huggingface.co/docs/datasets/image_load and try to load image data from a folder. I noticed that the image was read twice in the returned data. As you can see in the attached image, there are only four images in the train folder, but reading brings up eight images

### Steps to reproduce the bug

from datasets import Dataset, load_dataset

dataset = load_dataset("imagefolder", data_dir="data/", drop_labels=False)
# print(dataset["train"][0]["image"] == dataset["train"][1]["image"])

print(dataset)
print(dataset["train"]["image"])
print(len(dataset["train"]["image"]))

### Expected behavior

DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 8
    })
})
[<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D1CA8B0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D2452E0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D245310>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D2453A0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D245460>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D245430>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D2454F0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D245550>]
8

### Environment info

- `datasets` version: 2.14.5
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.9.17
- Huggingface_hub version: 0.19.4
- PyArrow version: 13.0.0
- Pandas version: 2.0.3

Comments:
`load_dataset(...,save_infos=True)` not working without loading script

### Describe the bug

It seems that saving a dataset infos back into the card file is not working for datasets without a loading script.

After tracking the problem a bit it looks like saving the infos uses `Builder.get_imported_module_dir()` as its destination directory.
Internally this is a call to `inspect.getfile()` but since the actual builder class used is dynamically created (cf. `datasets.load.configure_builder_class`) this method actually return te path to the parent builder class (e.g. `datasets.packaged_modules.json.JSON`). 



### Steps to reproduce the bug

1. Have a local dataset without any loading script
2. Make sure there are no dataset infos in the README.md
3. Load with `save_infos=True`
4. No change in the dataset README.md
5. A new README.md file is created in the directory of the parent builder class (e.g. for json in `.../site-packages/datasets/packaged_modules/json/README.md`)

### Expected behavior

The dataset README.md should be updated and no file should be created in the python environment.

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.3
- `fsspec` version: 2023.6.0


Comments:
Also, once the README.md exists in the python environment it is used when loading another dataset in the same format (e.g. json) since it always resolves the path to the same directory.
The consequence here is any other dataset won't load because of infos mismatch.
To reproduce this aspect:
1. Do a `load_datasets(...,save_infos=True)` with one dataset without a loading script
2. Try to load another dataset without a loading script in the same format (e.g. json) but with a different schema
load_dataset imageflder for aws s3 path 

### Feature request

I would like to load a dataset from S3 using the imagefolder option 
something like 
`dataset = datasets.load_dataset('imagefolder', data_dir='s3://.../lsun/train/bedroom', fs=S3FileSystem(), streaming=True)  `

### Motivation

no need of data_files

### Your contribution

no experience with this

Comments:
429 Client Error

Hello, I was downloading the following dataset and after 20% of data was downloaded, I started getting error 429. It is not resolved since a few days. How should I resolve it?
Thanks

Dataset:
https://huggingface.co/datasets/cerebras/SlimPajama-627B

Error:
`requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/train/chunk1/example_train_3300.jsonl.zst`


Comments:
Transferring repos as this is a datasets issue 

I'm getting a similar issue even though I've already downloaded the dataset 😅 

```
huggingface_hub.utils._errors.HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/datasets/HuggingFaceM4/WebSight
```
Update builder hash with info

Currently if you change the `dataset_info` of a dataset (e.g. in the YAML part of the README.md), the cache ignores this change.

This is problematic because you want to regenerate a dataset if you change the features or the split sizes for example (e.g. after push_to_hub)

Ideally we should take the resolved files into account as well but this will be for another PR

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6487). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Closing this one in favor of https://github.com/huggingface/datasets/pull/6458/commits/565c294fc12bc547730a023a610ed4f92313d8fb in https://github.com/huggingface/datasets/pull/6458
FileNotFoundError: [Errno 2] No such file or directory: 'nul'

### Describe the bug

it seems that sth wrong with my terrible "bug body" life, When i run this code, "import datasets"
i meet this error FileNotFoundError: [Errno 2] No such file or directory: 'nul'
![image](https://github.com/huggingface/datasets/assets/73683903/3973c120-ebb1-42b7-bede-b9de053e861d)
![image](https://github.com/huggingface/datasets/assets/73683903/0496adff-a7a7-4dcb-929e-ec11ede71f04)


### Steps to reproduce the bug

1.import datasets

### Expected behavior

i just run a single line code and stuct in this bug 

### Environment info

OS: Windows10
Datasets==2.15.0
python=3.10

Comments:
Hi! It seems like the problem is your environment. Maybe this issue can help: https://github.com/pytest-dev/pytest/issues/9519.
using torchrun, save_to_disk suddenly shows SIGTERM

### Describe the bug

When I run my code using the "torchrun" command, when the code reaches the "save_to_disk" part, suddenly I get the following warning and error messages:
Because the dataset is too large, the "save_to_disk" function splits it into 70 parts for saving. However, an error occurs suddenly when it reaches the 14th shard.
WARNING: torch.distributed.elastic.multiprocessing.api: Sending process 2224968 closing signal SIGTERM
ERROR: torch.distributed.elastic.multiprocessing.api: failed (exitcode: -7). traceback: Signal 7 (SIGBUS) received by PID 2224967.

### Steps to reproduce the bug

ds_shard = ds_shard.map(map_fn, *args, **kwargs)
ds_shard.save_to_disk(ds_shard_filepaths[rank])

Saving the dataset (14/70 shards):  20%|██        | 875350/4376702 [00:19<01:53, 30863.15 examples/s]
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 2224968 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -7) local_rank: 0 (pid: 2224967) of binary: /home/bingxing2/home/scx6964/.conda/envs/ariya235/bin/python
Traceback (most recent call last):
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
==========================================================
run.py FAILED
----------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
----------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-12-08_20:09:04
  rank      : 0 (local_rank: 0)
  exitcode  : -7 (pid: 2224967)
  error_file: <N/A>
  traceback : Signal 7 (SIGBUS) received by PID 2224967



### Expected behavior

I hope it can save successfully without any issues, but it seems there is a problem.

### Environment info

`datasets` version: 2.14.6
- Platform: Linux-4.19.90-24.4.v2101.ky10.aarch64-aarch64-with-glibc2.28
- Python version: 3.10.11
- Huggingface_hub version: 0.17.3
- PyArrow version: 14.0.0
- Pandas version: 2.1.2

Comments:
How to load data from lakefs

My dataset is stored on the company's lakefs server. How can I write code to load the dataset? It would be great if I could provide code examples or provide some references


Comments:
You can create a `pandas` DataFrame following [this](https://lakefs.io/data-version-control/dvc-using-python/) tutorial, and then convert this DataFrame to a `Dataset` with `datasets.Dataset.from_pandas`. For larger datasets (to memory map them), you can use `Dataset.from_generator` with a generator function that reads lakeFS files with `s3fs`.

@mariosasko  hello,
This can achieve and https://huggingface.co/datasets Does the same effect apply to the dataset? For example, downloading while using

There is a blogspot from lakes on this topic: https://lakefs.io/blog/data-version-control-hugging-face-datasets/
CI on windows is broken: PermissionError

See: https://github.com/huggingface/datasets/actions/runs/7104781624/job/19340572394
```
FAILED tests/test_load.py::test_loading_from_the_datasets_hub - NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpfcnps56i\\hf-internal-testing___dataset_with_script\\default\\0.0.0\\c240e2be3370bdbd\\dataset_with_script-train.arrow'
```

Comments:
If an image in a dataset is corrupted, we get unescapable error

### Describe the bug

Example discussed in detail here: https://huggingface.co/datasets/sasha/birdsnap/discussions/1

### Steps to reproduce the bug

```
from datasets import load_dataset, VerificationMode
dataset = load_dataset(
    'sasha/birdsnap',
    split="train",
    verification_mode=VerificationMode.ALL_CHECKS,
    streaming=True # I recommend using streaming=True when reproducing, as this dataset is large
)
for idx, row in enumerate(dataset):
    # Iterating to 9287 took 7 minutes for me
    # If you already have the data locally cached and set streaming=False, you see the same error just by with dataset[9287]
    pass
    # error at 9287 OSError: image file is truncated (45 bytes not processed)
    # note that we can't avoid the error using a try/except + continue inside the loop
```

### Expected behavior

Able to escape errors in casting to Image() without killing the whole loop

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-5.15.0-84-generic-x86_64-with-glibc2.31
- Python version: 3.11.5
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.3
- `fsspec` version: 2023.10.0

Comments:
New version release request

### Feature request

Hi!
I am using `datasets` in library `xtuner` and am highly interested in the features introduced since v2.15.0.

To avoid installation from source in our pypi wheels, we are eagerly waiting for the new release. So, Does your team have a new release plan for v2.15.1 and could you please share it with us?

Thanks very much!


### Motivation

.

### Your contribution

.

Comments:
We will publish it soon (we usually do it in intervals of 1-2 months, so probably next week)

Thanks!
Can't align optional features of struct

### Describe the bug

Hello!

I'm currently experiencing an issue where I can't concatenate datasets if an inner field of a Feature is Optional.

I have a column named `speaker`, and this holds some information about a speaker.

```python
@dataclass
class Speaker:
   name: str
   email: Optional[str]
```

If I have two datasets, one happens to have `email` always None, then I get `The features can't be aligned because the key email of features`




### Steps to reproduce the bug

You can run the following script:

```python

ds = Dataset.from_dict({'speaker': [{'name': 'Ben', 'email': None}]})
ds2 = Dataset.from_dict({'speaker': [{'name': 'Fred', 'email': 'abc@aol.com'}]})
concatenate_datasets([ds, ds2])

>>>The features can't be aligned because the key speaker of features {'speaker': {'email': Value(dtype='string', id=None), 'name': Value(dtype='string', id=None)}} has unexpected type - {'email': Value(dtype='string', id=None), 'name': Value(dtype='string', id=None)} (expected either {'email': Value(dtype='null', id=None), 'name': Value(dtype='string', id=None)} or Value("null").

```

### Expected behavior

I think this should work; if two top-level columns were in the same situation it would properly cast to `string`.

```python
ds = Dataset.from_dict({'email': [None, None]})
ds2 = Dataset.from_dict({'email': ['abc@aol.com', 'one@yahoo.com']})
concatenate_datasets([ds, ds2])
>>> # Works!
```

### Environment info

- `datasets` version: 2.15.1.dev0
- Platform: Linux-5.15.0-89-generic-x86_64-with-glibc2.35
- Python version: 3.9.13
- `huggingface_hub` version: 0.19.4
- PyArrow version: 9.0.0
- Pandas version: 1.4.4
- `fsspec` version: 2023.6.0

I would be happy to fix this issue.

Comments:
Friendly bump, I would be happy to work on this issue once I get the go-ahead from the dev team. 

Thanks for the PR!

I'm struggling with this as well and would love to see this PR merged. My case is slightly different, with keys completely missing rather than being `None`:

```
ds = Dataset.from_dict({'speaker': [{'name': 'Ben'}]})
ds2 = Dataset.from_dict({'speaker': [{'name': 'Fred', 'email': 'abc@aol.com'}]})
print(concatenate_datasets([ds, ds2]).features)
print(concatenate_datasets([ds, ds2]).to_dict())
```

I would expect this to work as well because other Dataset functions already handle this situation well. For example, this works just as expected:

```
ds = Dataset.from_dict({'n': [1,2]})
ds_mapped = ds.map(lambda x: {
    'speaker': {'name': 'Ben'} if x['n'] == 1 else {'name': 'Fred', 'email': 'abc@aol.com'}
})
print(ds_mapped)
```

@vova-cyberhaven can you check with the new release if it fixes your issue?
`TypeError`: huggingface_hub.hf_file_system.HfFileSystem.find() got multiple values for keyword argument 'maxdepth'

### Describe the bug

Please see https://github.com/huggingface/huggingface_hub/issues/1872

### Steps to reproduce the bug

Please see https://github.com/huggingface/huggingface_hub/issues/1872

### Expected behavior

Please see https://github.com/huggingface/huggingface_hub/issues/1872

### Environment info

Please see https://github.com/huggingface/huggingface_hub/issues/1872

Comments:
Updating `fsspec>=2023.10.0` did solve the issue.

May be it should be pinned somewhere?

> Maybe this should go in datasets directly... anyways you can easily fix this error by updating datasets>=2.15.1.dev0.

@lhoestq @mariosasko for what I understand this is a bug fixed in `datasets` already, right? No need to do anything in `huggingface_hub`?

I've opened a PR with a fix in `huggingface_hub`: https://github.com/huggingface/huggingface_hub/pull/1875

Thanks! PR is merged and will be shipped in next release of `huggingface_hub`.
Unable to read "marsyas/gtzan" data

Hi, this is my code and the error:
```
from datasets import load_dataset
gtzan = load_dataset("marsyas/gtzan", "all")
```
[error_trace.txt](https://github.com/huggingface/datasets/files/13464397/error_trace.txt)
[audio_yml.txt](https://github.com/huggingface/datasets/files/13464410/audio_yml.txt)
Python 3.11.5
Jupyter Notebook 6.5.4
Windows 10

I'm able to download and work with other datasets, but not this one. For example, both these below work fine:
```
from datasets import load_dataset
dataset = load_dataset("facebook/voxpopuli", "pl", split="train", streaming=True)
minds = load_dataset("PolyAI/minds14", name="en-US", split="train")
```
Thanks for your help

https://huggingface.co/datasets/marsyas/gtzan/tree/main

Comments:
Hi! We've merged a [PR](https://huggingface.co/datasets/marsyas/gtzan/discussions/1) that fixes the script's path logic on Windows.

I have transferred the discussion to the corresponding dataset: https://huggingface.co/datasets/marsyas/gtzan/discussions/2

Let's continue there.

@mariosasko @albertvillanova 

Thank you both very much for the speedy resolution :)
Support multiple image/audio columns in ImageFolder/AudioFolder

### Feature request

Have a metadata.csv file with multiple columns that point to relative image or audio files.

### Motivation

Currently, ImageFolder allows one column, called `file_name`, pointing to relative image files. On the same model, AudioFolder allows one column, called `file_name`, pointing to relative audio files.

But it's not possible to have two image columns, or to have two audio column, or to have one audio column and one image column.

### Your contribution

no specific contribution

Comments:
A duplicate of https://github.com/huggingface/datasets/issues/5760
Trouble loading image folder with additional features -  metadata file ignored

### Describe the bug

Loading image folder with a caption column using `load_dataset(<image_folder_path>)` doesn't load the captions. 

When loading a local image folder with captions using `datasets==2.13.0`
```
from datasets import load_dataset
data = load_dataset(<image_folder_path>)
data.column_names
```
yields 
`{'train': ['image', 'prompt']}`

but when using `datasets==2.15.0`
yeilds 
`{'train': ['image']}`

Putting the images and `metadata.jsonl` file into a nested `train` folder **or** loading with `load_dataset("imagefolder", data_dir=<image_folder_path>)` solves the issue and
yields 
`{'train': ['image', 'prompt']}`

### Steps to reproduce the bug

1. create a folder `<image_folder_path>` that contains images and a metadata file with additional features- e.g. "prompt"
2. run:
```
from datasets import load_dataset
data = load_dataset("<image_folder_path>")
data.column_names
```

### Expected behavior

`{'train': ['image', 'prompt']}`

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0

Comments:
I reproduced too:
- root: metadata file is ignored (https://huggingface.co/datasets/severo/doc-image-3)
- data/ dir: metadata file is ignored (https://huggingface.co/datasets/severo/doc-image-4)
- train/ dir: works (https://huggingface.co/datasets/severo/doc-image-5)
Trouble Loading a Gated Dataset For User with Granted Permission

### Describe the bug

I have granted permissions to several users to access a gated huggingface dataset. The users accepted the invite and when trying to load the dataset using their access token they get 
`FileNotFoundError: Couldn't find a dataset script at .....` . Also when they try to click the url link for the dataset they get a 404 error.

### Steps to reproduce the bug

1. Grant access to gated dataset for specific users 
2. Users accept invitation
3. Users login to hugging face hub using cli login
4. Users run load_dataset

### Expected behavior

Dataset is loaded normally for users who were granted access to the gated dataset.

### Environment info

datasets==2.15.0


Comments:
> Also when they try to click the url link for the dataset they get a 404 error.

This seems to be a Hub error then (cc @SBrandeis)

Could you report this to https://discuss.huggingface.co/c/hub/23, providing the URL of the dataset, or at least if the dataset is public or private?

Thanks for the reply! I've created an issue on the hub's board here: https://discuss.huggingface.co/t/trouble-loading-a-gated-dataset-for-user-with-granted-permission/65565
Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

### Describe the bug

1. I ran dataset mapping with `num_proc=6` in it and got this error:
`RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method`

I can't actually find a way to run multi-GPU dataset mapping. Can you help?

### Steps to reproduce the bug

1. Rund SDXL training with `num_proc=6`: https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py

### Expected behavior

Should work well

### Environment info

6x A100 SXM, Linux

Comments:
[This doc section](https://huggingface.co/docs/datasets/main/en/process#multiprocessing) explains how to modify the script to avoid this error.

@mariosasko thank you very much, i'll check it

@mariosasko no it does not

`Dataset.filter() got an unexpected keyword argument 'with_rank'`
load_dataset does not load all of the data in my input file

### Describe the bug

I have 127 elements in my input dataset.  When I do a len on the dataset after loaded, it is only 124 elements.

### Steps to reproduce the bug

    train_dataset = nlp.load_dataset(data_args.dataset_path, name=data_args.qg_format, split=nlp.Split.TRAIN)
    valid_dataset = nlp.load_dataset(data_args.dataset_path, name=data_args.qg_format, split=nlp.Split.VALIDATION)
    logger.info(len(train_dataset))
    logger.info(len(valid_dataset))

Both train and valid input are 127 items.  However, they both only load 124 items.  The input format is in json.  At the end of the day, I am trying to create .pt files.  

### Expected behavior

I see all 127 elements in my dataset when performing len

### Environment info

Python 3.10.  CentOS operating system.  nlp==0.40, datasets==2.14.5, transformers==4.26.1

Comments:
You should use `datasets.load_dataset` instead of `nlp.load_dataset`, as the `nlp` package is outdated.

If switching to `datasets.load_dataset` doesn't fix the issue, sharing the JSON file (feel free to replace the data with dummy data) would be nice so that we can reproduce it ourselves.
Allow to choose the `writer_batch_size` when using `save_to_disk`

### Feature request

Add an argument in `save_to_disk` regarding batch size, which would be passed to `shard` and other methods.

### Motivation

The `Dataset.save_to_disk` method currently calls `shard` without passing a `writer_batch_size` argument, thus implicitly using the default value (1000). This can result in RAM saturation when using a lot of processes on long text sequences or other modalities, or for specific IO configs.

### Your contribution

I would be glad to submit a PR, as long as it does not imply extensive tests refactoring.

Comments:
We have a config variable that controls the batch size in `save_to_disk`:
```python
import datasets
datasets.config.DEFAULT_MAX_BATCH_SIZE = <smaller_batch_size>
...
ds.save_to_disk(...)
```

Thank you for your answer!

From what I am reading in `https://github.com/huggingface/datasets/blob/2.14.5/src/datasets/arrow_dataset.py`, every function involved (`select`, `shard`, ...) has a default hardcoded batch size of 1000, as such:
```python
def select(
        self,
        indices: Iterable,
        keep_in_memory: bool = False,
        indices_cache_file_name: Optional[str] = None,
        writer_batch_size: Optional[int] = 1000,
        new_fingerprint: Optional[str] = None,
    ) -> "Dataset":
...
```
Then, `ArrowWriter` is instantiated with the specified `writer_batch_size`. In `ArrowWriter`, `writer_batch_size` is set to `datasets.config.DEFAULT_MAX_BATCH_SIZE` if it is `None`(https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_writer.py#L345C14-L345C31). However, in our case, it is already set to 1000 by "parent" methods, so it won't happen.

Nevertheless, due to this: 
```python
def _save_to_disk_single(job_id: int, shard: "Dataset", fpath: str, storage_options: Optional[dict]):
        batch_size = config.DEFAULT_MAX_BATCH_SIZE
...
```
it seems to work. I will use it as such, but it should maybe be added to documentation? And maybe improved in next versions?
User token is printed out!

This line prints user token on command line! Is it safe?

https://github.com/huggingface/datasets/blob/12ebe695b4748c5a26e08b44ed51955f74f5801d/src/datasets/load.py#L2091

Comments:
Indeed, this is not a good practice. I've opened a PR that removes the token value from the (deprecation) warning.
using DownloadManager to download from local filesystem and disable_progress_bar, there will be an exception 

### Describe the bug

i'm using datasets.download.download_manager.DownloadManager to download files like "file:///a/b/c.txt", and i disable_progress_bar() to disable bar. there will be an exception as follows:

`AttributeError: 'function' object has no attribute 'close'
Exception ignored in: <function TqdmCallback.__del__ at 0x7fa8683d84c0>
Traceback (most recent call last):
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/fsspec/callbacks.py", line 233, in __del__
    self.tqdm.close()`

i check your source code in datasets/utils/file_utils.py:348 you define TqdmCallback derive from fsspec.callbacks.TqdmCallback

but in the newest fsspec code [https://github.com/fsspec/filesystem_spec/blob/master/fsspec/callbacks.py](url) , line 146, in this case, _DEFAULT_CALLBACK will take effect, but in line 234, it calls "close()" function which _DEFAULT_CALLBACK don't have such thing.

so i think the class "TqdmCallback" in datasets/utils/file_utils.py may override "__del__" function or report this bug to fsspec.



### Steps to reproduce the bug

as i said

### Expected behavior

no exception

### Environment info

datasets: 2.14.4
python: 3.9
platform: x86_64

Comments:
`IterableDataset` lost but not keep columns when map function adding columns with names in `remove_columns`

### Describe the bug

IterableDataset lost but not keep columns when map function adding columns with names in remove_columns,
Dataset not.

May be related to the code below:
https://github.com/huggingface/datasets/blob/06c3ffb8d068b6307b247164b10f7c7311cefed4/src/datasets/iterable_dataset.py#L750-L756

### Steps to reproduce the bug

```python
dataset: IterableDataset = load_dataset("Anthropic/hh-rlhf", streaming=True, split="train")
column_names = list(next(iter(dataset)).keys())  #  ['chosen', 'rejected']
# map_fn will return dict {"chosen": xxx, "rejected": xxx, "prompt": xxx, "history": xxxx}
dataset = dataset.map(map_fn, batched=True, remove_columns=column_names)
next(iter(dataset))
# output
# {'prompt': 'xxx, 'history': xxx}   
```

```python
# when load_dataset with streaming=False, the column_names are kept: 
dataset: Dataset = load_dataset("Anthropic/hh-rlhf", streaming=False, split="train")
column_names = list(next(iter(dataset)).keys())  #  ['chosen', 'rejected']
# map_fn will return dict {"chosen": xxx, "rejected": xxx, "prompt": xxx, "history": xxxx}
dataset = dataset.map(map_fn, batched=True, remove_columns=column_names)
next(iter(dataset))
# output
# {'prompt': 'xxx, 'history': xxx, "chosen": xxx, "rejected": xxx}   
```


### Expected behavior

IterableDataset keep columns when map function adding columns with names in remove_columns

### Environment info

datasets==2.14.6

Comments:
CI Build PR Documentation is broken: ImportError: cannot import name 'TypeAliasType' from 'typing_extensions'

Our CI Build PR Documentation is broken. See: https://github.com/huggingface/datasets/actions/runs/6799554060/job/18486828777?pr=6390
```
ImportError: cannot import name 'TypeAliasType' from 'typing_extensions'
```

Comments:
Cannot import datasets on google colab (python 3.10.12)

### Describe the bug

I'm trying A full colab demo notebook of zero-shot-distillation from https://github.com/huggingface/transformers/tree/main/examples/research_projects/zero-shot-distillation but i got this type of error when importing  datasets on my google colab (python version is 3.10.12)

![image](https://github.com/huggingface/datasets/assets/15389235/6f7758a2-681d-4436-87d0-5e557838e368)

I found the same problem that have been solved in [#3326 ] but it seem still error on the google colab. I can't try on my local using jupyter notebook because of my laptop resource doesn't fulfill the requirements. 

Please can anyone help me solve this problem. Thank you 😅 

### Steps to reproduce the bug

Error:

```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-8-b6e092f83978>](https://localhost:8080/#) in <cell line: 1>()
----> 1 from datasets import load_dataset
      2 
      3 # Print all the available datasets
      4 from huggingface_hub import list_datasets
      5 print([dataset.id for dataset in list_datasets()])

6 frames
[/usr/lib/python3.10/functools.py](https://localhost:8080/#) in update_wrapper(wrapper, wrapped, assigned, updated)
     59     # Issue #17482: set __wrapped__ last so we don't inadvertently copy it
     60     # from the wrapped function when updating __dict__
---> 61     wrapper.__wrapped__ = wrapped
     62     # Return the wrapper so this can be used as a decorator via partial()
     63     return wrapper

AttributeError: readonly attribute
```

### Expected behavior

Run success on Google Colab (free)

### Environment info

Windows 11 x64, Google Colab free

Comments:
You are most likely using an outdated version of `datasets` in the notebook, which can be verified with the `!datasets-cli env` command. You can run `!pip install -U datasets` to update the installation.

okay, it works! thank you so much! 😄
Safely load datasets by disabling execution of dataset loading script

### Feature request

Is there a way to disable execution of dataset loading script using `load_dataset`? This is a security vulnerability that could lead to arbitrary code execution. 

Any suggested workarounds are welcome as well. 

### Motivation

This is a security vulnerability that could lead to arbitrary code execution. 

### Your contribution

n/a

Comments:
great idea IMO

this could be a `trust_remote_code=True` flag like in transformers. We could also default to loading the Parquet conversion rather than executing code (for dataset repos that have both)

@julien-c that would be great!

We added the `trust_remote_code` argument to `load_dataset()` in `datasets` 2.16:
- in the future users will have to pass trust_remote_code=True to use datasets with a script
- for now we just show a warning when a dataset script is used
- we fallback on the Hugging Face Parquet exports when possible (to keep compatibility with old datasets with scripts)

So feel free to use `trust_remote_code=False` in the meantime to disable loading from dataset loading scripts :)

Passing `trust_remote_code=True` explicitly is now mandatory to load a dataset with a script since https://github.com/huggingface/datasets/pull/6954
TypeError: Cannot convert pyarrow.lib.ChunkedArray to pyarrow.lib.Array

### Describe the bug

Hi, I am preprocessing a large custom dataset with numpy arrays. I am running into this TypeError during writing in a dataset.map() function. I've tried decreasing writer batch size, but this error persists. This error does not occur for smaller datasets. 



Thank you!



### Steps to reproduce the bug

Traceback (most recent call last):
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 1354, in _write_generator_to_queue
    for i, result in enumerate(func(**kwargs)):
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3493, in _map_single
    writer.write_batch(batch)
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/datasets/arrow_writer.py", line 555, in write_batch
    arrays.append(pa.array(typed_sequence))
  File "pyarrow/array.pxi", line 243, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/datasets/arrow_writer.py", line 184, in __arrow_array__
    out = numpy_to_pyarrow_listarray(data)
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/datasets/features/features.py", line 1394, in numpy_to_pyarrow_listarray
    values = pa.ListArray.from_arrays(offsets, values)
  File "pyarrow/array.pxi", line 2004, in pyarrow.lib.ListArray.from_arrays
TypeError: Cannot convert pyarrow.lib.ChunkedArray to pyarrow.lib.Array

### Expected behavior

Type should not be a ChunkedArray

### Environment info

datasets v2.14.5
arrow v1.2.3
pyarrow v12.0.1

Comments:
Seconding encountering this issue.
Raise a different exception for inexisting dataset vs files without known extension

See https://github.com/huggingface/datasets-server/issues/2082#issuecomment-1805716557

We have the same error for:
- https://huggingface.co/datasets/severo/a_dataset_that_does_not_exist: a dataset that does not exist
- https://huggingface.co/datasets/severo/test_files_without_extension: a dataset with files without a known extension

```
>>> import datasets
>>> datasets.get_dataset_config_names('severo/a_dataset_that_does_not_exist')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 351, in get_dataset_config_names
    dataset_module = dataset_module_factory(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1508, in dataset_module_factory
    raise FileNotFoundError(
FileNotFoundError: Couldn't find a dataset script at /home/slesage/hf/datasets-server/services/worker/severo/a_dataset_that_does_not_exist/a_dataset_that_does_not_exist.py or any data file in the same directory. Couldn't find 'severo/a_dataset_that_does_not_exist' on the Hugging Face Hub either: FileNotFoundError: Dataset 'severo/a_dataset_that_does_not_exist' doesn't exist on the Hub. If the repo is private or gated, make sure to log in with `huggingface-cli login`.
>>> datasets.get_dataset_config_names('severo/test_files_without_extension')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 351, in get_dataset_config_names
    dataset_module = dataset_module_factory(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1508, in dataset_module_factory
    raise FileNotFoundError(
FileNotFoundError: Couldn't find a dataset script at /home/slesage/hf/datasets-server/services/worker/severo/test_files_without_extension/test_files_without_extension.py or any data file in the same directory. Couldn't find 'severo/test_files_without_extension' on the Hugging Face Hub either: FileNotFoundError: No (supported) data files or dataset script found in severo/test_files_without_extension.
```

To differentiate, we must parse the error message (only the end is different). We should have a different exception for these two errors.

Comments:
Add ability to set lock type

### Feature request

Allow setting file lock type, maybe from an environment variable

Currently, it only depends on whether fnctl is available:

https://github.com/huggingface/datasets/blob/12ebe695b4748c5a26e08b44ed51955f74f5801d/src/datasets/utils/filelock.py#L463-L470C16

### Motivation

In my environment, flock isn't supported on a network attached drive

### Your contribution

I'll be happy to submit a pr.

Comments:
We've replaced our filelock implementation with the `filelock` package, so their repo is the right place to request this feature.

In the meantime, the following should work: 
```python
import filelock
filelock.FileLock = filelock.SoftFileLock

import datasets
...
```
Webdataset dataset builder

Allow `load_dataset` to support the Webdataset format.

It allows users to download/stream data from local files or from the Hugging Face Hub.

Moreover it will enable the Dataset Viewer for Webdataset datasets on HF.

## Implementation details

- I added a new Webdataset builder
- dataset with TAR files are now read using the Webdataset builder
- Basic decoding from `webdataset` is used by default, except unsafe ones like pickle
- HF authentication support is done with `xopen`

## TODOS

- [x] tests
- [x] docs

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I added an error message if the first examples don't appear to be in webdataset format
```
"The TAR archives of the dataset should be in Webdataset format, "
"but the files in the archive don't share the same prefix or the same types."
```

@mariosasko could you review this ? I think it's fine to have webdataset as an optional dependency for now, then depending on usage and user feedbacks see if it makes sense to have our own implementation or not

I just removed the dependency on `webdataset` @mariosasko :)

took your comments into account, lmk if you see anything else
How to create 3d medical imgae dataset?

### Feature request

I am newer to huggingface, after i look up `datasets` docs, I can't find how to create the dataset contains 3d medical image (ends with '.mhd', '.dcm', '.nii')

### Motivation

help us to upload 3d medical dataset to huggingface!

### Your contribution

I'll submit a PR if I find a way to add this feature

Comments:
How to load existing downloaded dataset ?

Hi @mariosasko @lhoestq  @katielink 

Thanks for your contribution and hard work.

### Feature request

First, I download a dataset as normal by:
```
from datasets import load_dataset

dataset = load_dataset('username/data_name', cache_dir='data')
```

The dataset format in `data` directory will be:
```
-data
  |-data_name
    |-test-00000-of-00001-bf4c733542e35fcb.parquet
    |-train-00000-of-00001-2a1df75c6bce91ab.parquet
```


Then I use SCP to clone this dataset into another machine, and then try:
```
from datasets import load_dataset

dataset = load_dataset('data/data_name')  # load from local path
```

This leads to re-generating training and validation split for each time, and the disk quota will be duplicated occupation.

How can I just load the dataset without generating and saving these splits again?

### Motivation

I do not want to download the same dataset in two machines, scp is much faster and better than HuggingFace API. I hope we can directly load the downloaded datasets (.parquest)

### Your contribution

Please refer to the feature

Comments:
Feel free to use `dataset.save_to_disk(...)`, then scp the directory containing the saved dataset and reload it on your other machine using `dataset = load_from_disk(...)`
Formatting overhead

### Describe the bug

Hi! I very recently noticed that my training time is dominated by batch formatting. Using Lightning's profilers, I located the bottleneck within `datasets.formatting.formatting` and then narrowed it down with `line-profiler`. It turns out that almost all of the overhead is due to creating new instances of `self.python_arrow_extractor`. I admit I'm confused why that could be the case - as far as I can tell there's no complex `__init__` logic to execute.

![image](https://github.com/huggingface/datasets/assets/320321/5e022e0b-0d21-43d0-8e6f-9e641142e96b)

### Steps to reproduce the bug

1. Set up a dataset `ds` with potentially several (4+) columns (not sure if this is necessary, but it did at one point of the investigation make overhead worse)
2. Process it using a custom transform, `ds = ds.with_transform(transform_func)`
3. Decorate this function https://github.com/huggingface/datasets/blob/main/src/datasets/formatting/formatting.py#L512 with `@profile` from https://pypi.org/project/line-profiler/
4. Profile with `$ kernprof -l script_to_profile.py`

### Expected behavior

Batch formatting should have acceptable overhead.

### Environment info

```
datasets=2.14.6
pyarrow=14.0.0
```

Comments:
Ah I think the `line-profiler` log is off-by-one and it is in fact the `extract_batch` method that's taking forever. Will investigate further.

I tracked it down to a quirk of my setup. Apologies.
imagenet-1k downloads over and over

### Describe the bug

What could be causing this?
```
$ python3
Python 3.8.13 (default, Mar 28 2022, 11:38:47) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> load_dataset("imagenet-1k")
Downloading builder script: 100%|██████████| 4.72k/4.72k [00:00<00:00, 7.51MB/s]
Downloading readme: 100%|███████████████████| 85.4k/85.4k [00:00<00:00, 510kB/s]
Downloading extra modules: 100%|████████████| 46.4k/46.4k [00:00<00:00, 300kB/s]
Downloading data: 100%|████████████████████| 29.1G/29.1G [19:36<00:00, 24.8MB/s]
Downloading data: 100%|████████████████████| 29.3G/29.3G [08:38<00:00, 56.5MB/s]
Downloading data: 100%|████████████████████| 29.0G/29.0G [09:26<00:00, 51.2MB/s]
Downloading data: 100%|████████████████████| 29.2G/29.2G [09:38<00:00, 50.6MB/s]
Downloading data: 100%|███████████████████▉| 29.2G/29.2G [09:37<00:00, 44.1MB/s^Downloading data:   0%|                     | 106M/29.1G [00:05<23:49, 20.3MB/s]
```

### Steps to reproduce the bug

See above commands/code

### Expected behavior

imagenet-1k is downloaded

### Environment info

- `datasets` version: 2.14.6
- Platform: Linux-6.2.0-34-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.15.1
- PyArrow version: 14.0.0
- Pandas version: 1.5.2

Comments:
Have you solved this problem?