KRadim/github-issues-minimal huggingface dataset:
Dataset cannot convert too large dictionnary
State: open

### Describe the bug

Hello everyone!

I tried to build a new dataset with the command "dict_valid = datasets.Dataset.from_dict({'input_values': values_array})".
However, I have a very large dataset (~400Go) and it seems that dataset cannot handle this.

Indeed, I can create the dataset until a certain size of my dictionnary, and then I have the error "OverflowError: Python int too large to convert to C long".

Do you know how to solve this problem?
Unfortunately I cannot give a reproductible code because I cannot share a so large file, but you can find the code below (it's a test on only a part of the validation data ~10Go, but it's already the case).

Thank you!

### Steps to reproduce the bug

SAVE_DIR = './data/'
features = h5py.File(SAVE_DIR+'features.hdf5','r')

valid_data = features["validation"]["data/features"]

v_array_values = [np.float32(item[()]) for item in valid_data.values()]
for i in range(len(v_array_values)):
    v_array_values[i] = v_array_values[i].round(decimals=5)

dict_valid = datasets.Dataset.from_dict({'input_values': v_array_values})

### Expected behavior

The code is expected to give me a Huggingface dataset.

### Environment info

python: 3.8.15
numpy: 1.22.3
datasets: 2.3.2
pyarrow: 8.0.0
Custom split names
State: closed
Labels: name: enhancement

### Feature request

Hi,
I participated in multiple NLP tasks where there are more than just train, test, validation splits, there could be multiple validation sets or test sets. But it seems currently only those mentioned three splits supported. It would be nice to have the support for more splits on the hub. (currently i can have more splits when I am loading datasets from urls, but not hub)

### Motivation

Easier access to more splits

### Your contribution

No
load_dataset gives "403" error when using Financial phrasebank
State: open

When I try to load this dataset, I receive the following error: 

ConnectionError: Couldn't reach https://www.researchgate.net/profile/Pekka_Malo/publication/251231364_FinancialPhraseBank-v10/data/0c96051eee4fb1d56e000000/FinancialPhraseBank-v10.zip (error 403)

Has this been seen before? Thanks. The website loads when I try to access it manually.
add kwargs to index search
State: closed

This PR proposes to add kwargs to index search methods.

This is particularly useful for setting the timeout of a query on elasticsearch.

A typical use case would be:
```python
dset.add_elasticsearch_index("filename", es_client=es_client)
scores, examples = dset.get_nearest_examples("filename", "my_name-train_29", request_timeout=60)
```
Allow "jsonl" data type signifier
State: open
Labels: name: enhancement

### Feature request

`load_dataset` currently does not accept `jsonl` as type but only `json`. 

### Motivation

I was working with one of the `run_translation` scripts and used my own datasets (`.jsonl`) as train_dataset. But the default code did not work because

```
FileNotFoundError: Couldn't find a dataset script at jsonl\jsonl.py or any data file in the same directory. Couldn't find 'jsonl' on the Hugging Face Hub either: FileNotFoundError: Dataset 'jsonl' doesn't exist on the Hub. If the repo is private or gated, make sure to log in with `huggingface-cli login`.
```

The reason is because the script has these lines to extract the data type by its extension. Therefore, the derived type is `jsonl` which is not recognized by datasets as the error above shows.

https://github.com/huggingface/transformers/blob/ade26bf9912f69e2110137443e4406d7dbe253e7/examples/pytorch/translation/run_translation.py#L342-L356

I suppose you could argue that this is the script's fault (in which case I'll do a PR over at `transformers`) but it makes sense to me to add `jsonl` as an alias to `json` in `datasets`.


### Your contribution

At the moment I cannot work on this. I think it can be as "easy" as having an alias for json, namely jsonl.
glue datasets returning -1 for test split
State: closed

### Describe the bug

Downloading any dataset from GLUE has -1 as class labels for test split. Train and validation have regular 0/1 class labels.  This is also present in the dataset card online. 

### Steps to reproduce the bug

```
dataset = load_dataset("glue", "sst2")
for d in dataset:
    # prints out -1
    print(d["label"]
```

### Expected behavior

Expected behavior should be 0/1 instead of -1. 

### Environment info

- `datasets` version: 2.4.0
- Platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.17
- Python version: 3.8.16
- PyArrow version: 8.0.0
- Pandas version: 1.5.3
Remove set_access_token usage + fail tests if FutureWarning
State: closed

`set_access_token` is deprecated and will be removed in `huggingface_hub>=0.14`.

This PR removes it from the tests (it was not used in `datasets` source code itself). FYI, it was not needed since `set_access_token` was just setting git credentials and `datasets` doesn't seem to use git anywhere.

In the future, use `set_git_credential` if needed. It is a git-credential-agnostic helper, i.e. you can store your git token in `git-credential-cache`, `git-credential-store`, `osxkeychain`, etc. The legacy `set_access_token` could only set in `git-credential-store` no matter the user preference.

(for context, I found out about this while working on https://github.com/huggingface/huggingface_hub/pull/1381)

---

In addition to this, I have added

```
filterwarnings =
    error::FutureWarning:huggingface_hub*
```

to the `setup.cfg` config file to fail on future warnings from `huggingface_hub`. In `hfh`'s CI we trigger on FutureWarning from any package but it's less robust (any package update leads can lead to a failure). No obligation to keep it like that (I can remove it if you prefer) but I think it's a good idea in order to track future FutureWarnings.

FYI, in `huggingface_hub` tests we use `-Werror::FutureWarning --log-cli-level=INFO -sv --durations=0`
- FutureWarning are processed as error
- verbose mode / INFO logs (and above) are captured for easier debugging in github report
- track each test duration, just to see where we can improve. We have a quite long CI (~10min) so it helped improve that.
Bump pyarrow to 8.0.0
State: closed

Fix those for Pandas 2.0 (tested [here](https://github.com/huggingface/datasets/actions/runs/4346221280/jobs/7592010397) with pandas==2.0.0.rc0):

```python
=========================== short test summary info ============================
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_to_parquet_in_memory - ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.
A suitable version of pyarrow or fastparquet is required for parquet support.
Trying to import the above resulted in these errors:
 - Pandas requires version '7.0.0' or newer of 'pyarrow' (version '6.0.1' currently installed).
 - Missing optional dependency 'fastparquet'. fastparquet is required for parquet support. Use pip or conda to install fastparquet.
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_to_parquet_on_disk - ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.
A suitable version of pyarrow or fastparquet is required for parquet support.
Trying to import the above resulted in these errors:
 - Pandas requires version '7.0.0' or newer of 'pyarrow' (version '6.0.1' currently installed).
 - Missing optional dependency 'fastparquet'. fastparquet is required for parquet support. Use pip or conda to install fastparquet.
===== 2 failed, 2137 passed, 18 skipped, 32 warnings in 212.76s (0:03:32) ======
```

EDIT: also for performance - with 8.0 we can use `.to_reader()`
CI is broken after fsspec-2023.3.0 release
State: closed
Labels: name: bug

As reported by @lhoestq, our CI is broken after `fsspec` 2023.3.0 release:
```
FAILED tests/test_filesystem.py::test_compression_filesystems[Bz2FileSystem] - AssertionError: assert [{'created': ...: False, ...}] == ['file.txt']
  At index 0 diff: {'name': 'file.txt', 'size': 70, 'type': 'file', 'created': 1678175677.1887748, 'islink': False, 'mode': 33188, 'uid': 1001, 'gid': 123, 'mtime': 1678175677.1887748, 'ino': 286957, 'nlink': 1} != 'file.txt'
  Full diff:
    [
  -  'file.txt',
  +  {'created': 1678175677.1887748,
  +   'gid': 123,
  +   'ino': 286957,
  +   'islink': False,
  +   'mode': 33188,
  +   'mtime': 1678175677.1887748,
  +   'name': 'file.txt',
  +   'nlink': 1,
  +   'size': 70,
  +   'type': 'file',
  +   'uid': 1001},
    ]
```
Also:
```
FAILED tests/test_filesystem.py::test_compression_filesystems[GzipFileSystem] - AssertionError: assert [{'created': ...: False, ...}] == ['file.txt']
FAILED tests/test_filesystem.py::test_compression_filesystems[Lz4FileSystem] - AssertionError: assert [{'created': ...: False, ...}] == ['file.txt']
FAILED tests/test_filesystem.py::test_compression_filesystems[XzFileSystem] - AssertionError: assert [{'created': ...: False, ...}] == ['file.txt']
FAILED tests/test_filesystem.py::test_compression_filesystems[ZstdFileSystem] - AssertionError: assert [{'created': ...: False, ...}] == ['file.txt']
===== 5 failed, 2134 passed, 18 skipped, 38 warnings in 157.21s (0:02:37) ======
```

See:
- fsspec/filesystem_spec#1205
IterableDataset.add_column is unable to accept another IterableDataset as a parameter.
State: closed
Labels: name: wontfix

### Describe the bug

`IterableDataset.add_column` occurs an exception when passing another `IterableDataset` as a parameter.
The method seems to accept only eager evaluated values.

https://github.com/huggingface/datasets/blob/35b789e8f6826b6b5a6b48fcc2416c890a1f326a/src/datasets/iterable_dataset.py#L1388-L1391


I wrote codes below to make it.
```py
def add_column(dataset: IterableDataset, name: str, add_dataset: IterableDataset, key: str) -> IterableDataset:
    iter_add_dataset = iter(add_dataset)

    def add_column_fn(example):
        if name in example:
            raise ValueError(f"Error when adding {name}: column {name} is already in the dataset.")
        return {name: next(iter_add_dataset)[key]}

    return dataset.map(add_column_fn)
```

Is there other way to do it? Or is it intended?

### Steps to reproduce the bug

Thie codes below occurs `NotImplementedError`
```py
from datasets import IterableDataset


def gen(num):
    yield {f"col{num}": 1}
    yield {f"col{num}": 2}
    yield {f"col{num}": 3}


ids1 = IterableDataset.from_generator(gen, gen_kwargs={"num": 1})
ids2 = IterableDataset.from_generator(gen, gen_kwargs={"num": 2})
new_ids = ids1.add_column("new_col", ids1)

for row in new_ids:
    print(row)
```

### Expected behavior

`IterableDataset.add_column` is able to task `IterableDataset` and lazy evaluated values as a parameter since IterableDataset is lazy evalued.

### Environment info
- `datasets` version: 2.8.0
- Platform: Linux-3.10.0-1160.36.2.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.7
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
Arrow map type in parquet files unsupported
State: open

### Describe the bug

When I try to load parquet files that were processed with Spark, I get the following issue:

`ValueError: Arrow type map<string, string ('warc_headers')> does not have a datasets dtype equivalent.`

Strangely, loading the dataset with `streaming=True` solves the issue.

### Steps to reproduce the bug

The dataset is private, but this can be reproduced with any dataset that has Arrow maps.

### Expected behavior

Loading the dataset no matter whether streaming is True or not.

### Environment info

- `datasets` version: 2.10.1
- Platform: Linux-5.15.0-1029-gcp-x86_64-with-glibc2.31
- Python version: 3.10.7
- PyArrow version: 8.0.0
- Pandas version: 1.4.2
`load_from_disk` vs `load_dataset` performance.
State: open

### Describe the bug

I have downloaded `openwebtext` (~12GB) and filtered out a small amount of junk (it's still huge). Now, I would like to use this filtered version for future work. It seems I have two choices:
1. Use `load_dataset` each time, relying on the cache mechanism, and re-run my filtering.
2. `save_to_disk` and then use `load_from_disk` to load the filtered version.

The performance of these two approaches is wildly different:
* Using `load_dataset` takes about 20 seconds to load the dataset, and a few seconds to re-filter (thanks to the brilliant filter/map caching)
* Using `load_from_disk` takes 14 minutes! And the second time I tried, the session just crashed (on a machine with 32GB of RAM)

I don't know if you'd call this a bug, but it seems like there shouldn't need to be two methods to load from disk, or that they should  not take such wildly different amounts of time, or that one should not crash. Or maybe that the docs could offer some guidance about when to pick which method and why two methods exist, or just how do most people do it?

Something I couldn't work out from reading the docs was this: can I modify a dataset from the hub, save it (locally) and use `load_dataset` to load it? This [post seemed to suggest that the answer is no](https://discuss.huggingface.co/t/save-and-load-datasets/9260).

### Steps to reproduce the bug

See above

### Expected behavior

Load times should be about the same.

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.10.8
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
audiofolder only creates dataset of 13 rows (files) when the data folder it's reading from has 20,000 mp3 files. 
State: closed

### Describe the bug

x = load_dataset("audiofolder", data_dir="x")
When running this, x is a dataset of 13 rows (files) when it should be 20,000 rows (files) as the data_dir "x" has 20,000 mp3 files. Does anyone know what could possibly cause this (naming convention of mp3 files, etc.)


### Steps to reproduce the bug

x = load_dataset("audiofolder", data_dir="x")


### Expected behavior

x = load_dataset("audiofolder", data_dir="x") should create a dataset of 20,000 rows (files).

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-3.10.0-1160.80.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.16
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
Fix outdated `verification_mode` values
State: closed

~I think it makes sense not to save `dataset_info.json` file to a dataset cache directory when loading dataset with `verification_mode="no_checks"` because otherwise when next time the dataset is loaded **without** `verification_mode="no_checks"`, it will be loaded successfully, despite some values in info might not correspond to the ones in the repo which was the reason for using `verification_mode="no_checks"` first.~

Updated values of `verification_mode` to the current ones in some places ("none" -> "no_checks", "all" -> "all_checks")
Problems with downloading The Pile
State: closed

### Describe the bug

The downloads in the screenshot seem to be interrupted after some time and the last download throws a "Read timed out" error.

![image](https://user-images.githubusercontent.com/11065386/222687870-ec5fcb65-84e8-467d-9593-4ad7bdac4d50.png)

Here are the downloaded files:
![image](https://user-images.githubusercontent.com/11065386/222688200-454c2288-49e5-4682-96e6-1eb69aca0852.png)

They should be all 14GB like here (https://the-eye.eu/public/AI/pile/train/).

Alternatively, can I somehow download the files by myself and use the datasets preparing script?

### Steps to reproduce the bug

dataset = load_dataset('the_pile', split='train', cache_dir='F:\datasets')

### Expected behavior

The files should be downloaded correctly.

### Environment info

- `datasets` version: 2.10.1
- Platform: Windows-10-10.0.22623-SP0
- Python version: 3.10.5
- PyArrow version: 9.0.0
- Pandas version: 1.4.2
Authorization error
State: closed

### Describe the bug

Get `Authorization error` when try to push data into hugginface datasets hub.

### Steps to reproduce the bug

I did all steps in the [tutorial](https://huggingface.co/docs/datasets/share), 

1. `huggingface-cli login` with WRITE token
2. `git lfs install`
3. `git clone https://huggingface.co/datasets/namespace/your_dataset_name`
4.
  ```
  cp /somewhere/data/*.json .
  git lfs track *.json
  git add .gitattributes
  git add *.json
  git commit -m "add json files"
  ```

but when I execute `git push` I got the error:

```
Uploading LFS objects:   0% (0/1), 0 B | 0 B/s, done.
batch response: Authorization error.
error: failed to push some refs to 'https://huggingface.co/datasets/zeusfsx/ukrainian-news'
```

Size of data ~100Gb.  I have five json files - different parts.

### Expected behavior

All my data pushed into hub

### Environment info

- `datasets` version: 2.10.1
- Platform: macOS-13.2.1-arm64-arm-64bit
- Python version: 3.10.10
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
Dataloader getitem not working for DreamboothDatasets
State: closed

### Describe the bug

 Dataloader getitem is not working as before (see example of [DreamboothDatasets](https://github.com/huggingface/peft/blob/main/examples/lora_dreambooth/train_dreambooth.py#L451C14-L529))
moving Datasets to 2.8.0 solved the issue.

### Steps to reproduce the bug

1- using DreamBoothDataset to load some images 
2- error after loading when trying to visualise the images 

### Expected behavior

I was expecting a numpy array of the image

### Environment info

- Platform: Linux-5.10.147+-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 9.0.0
- Pandas version: 1.3.5
in-place dataset update
State: closed
Labels: name: wontfix

### Motivation

For the circumstance that I creat an empty `Dataset` and keep appending new rows into it, I found that it leads to creating a new dataset at each call. It looks quite memory-consuming. I just wonder if there is any more efficient way to do this.

```python
from datasets import Dataset

ds = Dataset.from_list([])

ds.add_item({'a': [1, 2, 3], 'b': 4})
print(ds)
>>> Dataset({
>>>     features: [],
>>>     num_rows: 0
>>> })

ds = ds.add_item({'a': [1, 2, 3], 'b': 4})
print(ds)
>>> Dataset({
>>>     features: ['a', 'b'],
>>>     num_rows: 1
>>> })
```

### Feature request

Call for in-place dataset update functions, that update the existing `Dataset` in place without creating a new copy. The interface is supposed to keep the same style as PyTorch, such as the in-place version of a `function` is named `function_`. For example, the in-pace version of `add_item`, i.e., `add_item_`, immediately updates the `Dataset`.

```python
from datasets import Dataset

ds = Dataset.from_list([])

ds.add_item({'a': [1, 2, 3], 'b': 4})
print(ds)
>>> Dataset({
>>>     features: [],
>>>     num_rows: 0
>>> })

ds.add_item_({'a': [1, 2, 3], 'b': 4})
print(ds)
>>> Dataset({
>>>     features: ['a', 'b'],
>>>     num_rows: 1
>>> })
```

### Related Functions

* `.map`
* `.filter`
* `.add_item`
Revert "pass the dataset features to the IterableDataset.from_generator"
State: closed

This reverts commit b91070b9c09673e2e148eec458036ab6a62ac042 (temporarily)

It hurts iterable dataset performance a lot (e.g. x4 slower because it encodes+decodes images unnecessarily). I think we need to fix this before re-adding it

cc @mariosasko @Hubert-Bonisseur
.sort() is broken when used after .filter(), only in 2.10.0
State: closed
Labels: name: bug

### Describe the bug

Hi, thank you for your support!

It seems like the addition of multiple key sort (#5502) in 2.10.0 broke the `.sort()` method.

After filtering a dataset with `.filter()`, the `.sort()` seems to refer to the query_table index of the previous unfiltered dataset, resulting in an IndexError.

This only happens with the 2.10.0 release.

### Steps to reproduce the bug

```Python
from datasets import load_dataset

# dataset with length of 1104
ds = load_dataset('glue', 'ax')['test']
ds = ds.filter(lambda x: x['idx'] > 1100)
ds.sort('premise')
print('Done')
```

  File "/home/dongkeun/datasets_test/test.py", line 5, in <module>
    ds.sort('premise')
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 528, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/fingerprint.py", line 511, in wrapper
    out = func(dataset, *args, **kwargs)
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3959, in sort
    sort_table = query_table(
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/formatting/formatting.py", line 588, in query_table
    _check_valid_index_key(key, size)
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/formatting/formatting.py", line 537, in _check_valid_index_key
    _check_valid_index_key(max(key), size=size)
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/formatting/formatting.py", line 531, in _check_valid_index_key
    raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 1103 is out of bounds for size 3

### Expected behavior

It should sort the dataset and print "Done". Which it does on 2.9.0.

### Environment info

- `datasets` version: 2.10.0
- Platform: Linux-5.15.0-41-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
Cache is not transportable
State: closed

### Describe the bug

I would like to share cache between two machines (a Windows host machine and a WSL instance).

I run most my code in WSL. I have just run out of space in the virtual drive. Rather than expand the drive size, I plan to move to cache to the host Windows machine, thereby sharing the downloads.

I'm hoping that I can just copy/paste the cache files, but I notice that a lot of the file names start with the path name, e.g. `_home_davidg_.cache_huggingface_datasets_conll2003_default-451...98.lock` where `home/davidg` is where the cache is in WSL. 

This seems to suggest that the cache is not portable/cannot be centralised or shared. Is this the case, or are the files that start with path names not integral to the caching mechanism? Because copying the cache files _seems_ to work, but I'm not filled with confidence that something isn't going to break.

A related issue, when trying to load a dataset that should come from cache (running in WSL, pointing to cache on the Windows host) it seemed to work fine, but it still uses a WSL directory for `.cache\huggingface\modules\datasets_modules`. I see nothing in the docs about this, or how to point it to a different place.

I have asked a related question on the forum: https://discuss.huggingface.co/t/is-datasets-cache-operating-system-agnostic/32656

### Steps to reproduce the bug

View the cache directory in WSL/Windows.

### Expected behavior

Cache can be shared between (virtual) machines and be transportable.

It would be nice to have a simple way to say "Dear Hugging Face packages, please put ALL your cache in `blah/de/blah`" and have all the Hugging Face packages respect that single location.

### Environment info

```
- `datasets` version: 2.9.0
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.10.8
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
- ```
Unable to load coyo700M dataset
State: closed

### Describe the bug

Seeing this error when downloading https://huggingface.co/datasets/kakaobrain/coyo-700m:
```ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.```


Full stack trace

```Downloading and preparing dataset parquet/kakaobrain--coyo-700m to /root/.cache/huggingface/datasets/kakaobrain___parquet/kakaobrain--coyo-700m-ae729692ae3e0073/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec...
Downloading data files: 100%
1/1 [00:00<00:00, 63.35it/s]
Extracting data files: 100%
1/1 [00:00<00:00, 5.00it/s]
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
[/usr/local/lib/python3.8/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1859                 _time = time.time()
-> 1860                 for _, table in generator:
   1861                     if max_shard_size is not None and writer._num_bytes > max_shard_size:

9 frames
ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
[/usr/local/lib/python3.8/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1890             if isinstance(e, SchemaInferenceError) and e.__context__ is not None:
   1891                 e = e.__context__
-> 1892             raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1893 
   1894         yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset```

### Steps to reproduce the bug

```
from datasets import load_dataset
hf_dataset = load_dataset("kakaobrain/coyo-700m")
```

### Expected behavior

The above commands load the dataset successfully. Or handles exception and continue loading the remainder.

### Environment info

colab. any
[DOC] Mistaken docs on set_format
State: closed
Labels: name: good first issue

### Describe the bug

https://huggingface.co/docs/datasets/v2.10.0/en/package_reference/main_classes#datasets.Dataset.set_format
<img width="700" alt="image" src="https://user-images.githubusercontent.com/36224762/221506973-ae2e3991-60a7-4d4e-99f8-965c6eb61e59.png">

While actually running it will result in:
<img width="1094" alt="image" src="https://user-images.githubusercontent.com/36224762/221507032-007dab82-8781-4319-b21a-e6e4d40d97b3.png">


### Steps to reproduce the bug

_

### Expected behavior

_

### Environment info

- `datasets` version: 2.10.0
- Platform: Linux-5.10.147+-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 9.0.0
- Pandas version: 1.3.5
Support cloud storage in load_dataset via fsspec
State: closed

Closes https://github.com/huggingface/datasets/issues/5281

This PR uses fsspec to support datasets on cloud storage (tested manually with GCS). ETags are currently unsupported for cloud storage. In general, a much larger refactor could be done to just use fsspec for all schemes (ftp, http/s, s3, gcs) to unify the interfaces here, but I ultimately opted to leave that out of this PR

I didn't create a GCS filesystem class in `datasets.filesystems` since the S3 one appears to be a wrapper around `s3fs.S3FileSystem` and mainly used to generate docs.
Cannot load `the_pile_openwebtext2`
State: closed

### Describe the bug

I met the same bug mentioned in #3053 which is never fixed. Because several `reddit_scores` are larger than `int8` even `int16`. https://huggingface.co/datasets/the_pile_openwebtext2/blob/main/the_pile_openwebtext2.py#L62

### Steps to reproduce the bug

```python3
from datasets import load_dataset
dataset = load_dataset("the_pile_openwebtext2")
```

### Expected behavior

load as normal.

### Environment info

- `datasets` version: 2.10.0
- Platform: Linux-5.4.143.bsk.7-amd64-x86_64-with-glibc2.31
- Python version: 3.9.2
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
I was getting a similar error `pyarrow.lib.ArrowInvalid: Integer value 528 not in range: -128 to 127` - AFAICT, this is because the type specified for `reddit_scores` is `datasets.Sequence(datasets.Value("int8"))`, but the actual values can be well outside the max range for 8-bit integers.
State: closed

              I was getting a similar error `pyarrow.lib.ArrowInvalid: Integer value 528 not in range: -128 to 127` - AFAICT, this is because the type specified for `reddit_scores` is `datasets.Sequence(datasets.Value("int8"))`, but the actual values can be well outside the max range for 8-bit integers.

I worked around this by downloading the `the_pile_openwebtext2.py` and editing it to use local files and drop reddit scores as a column (not needed for my purposes).

_Originally posted by @tc-wolf in https://github.com/huggingface/datasets/issues/3053#issuecomment-1281392422_
Metadata for each column
State: open
Labels: name: enhancement

### Feature request

Being able to put some metadata for each column as a string or any other type.


### Motivation

I will bring the motivation by an example, lets say we are experimenting with embedding produced by some image encoder network, and we want to iterate through a couple of preprocessing and see which one works better in our downstream task, here as workaround right now what I do is the compute the hash of the preprocessing that the images went through as part of the new columns name, it would be nice to attach some kinda meta data in these scenarios to the each columns. metadata

### Your contribution

Maybe we could map another relational like database as the metadata?
c4 dataset streaming fails with `FileNotFoundError`
State: closed

### Describe the bug

Loading the `c4` dataset in streaming mode with `load_dataset("c4", "en", split="validation", streaming=True)` and then using it fails with a `FileNotFoundException`.

### Steps to reproduce the bug

```python
from datasets import load_dataset

dataset = load_dataset("c4", "en", split="train", streaming=True)
next(iter(dataset))
```

causes a 

```
FileNotFoundError: https://huggingface.co/datasets/allenai/c4/resolve/1ddc917116b730e1859edef32896ec5c16be51d0/en/c4-train.00000-of-01024.json.gz
```

I can download this file manually though e.g. by entering this URL in a browser. 

There is an underlying HTTP 403 status code:

```
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://cdn-lfs.huggingface.co/datasets/allenai/c4/8ef8d75b0e045dec4aa5123a671b4564466b0707086a7ed1ba8721626dfffbc9?response-content-disposition=attachment%3B+filename*%3DUTF-8''c4-train.00000-of-01024.json.gz%3B+filename%3D%22c4-train.00000-of-01024.json.gz%22%3B&response-content-type=application/gzip&Expires=1677483770&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jZG4tbGZzLmh1Z2dpbmdmYWNlLmNvL2RhdGFzZXRzL2FsbGVuYWkvYzQvOGVmOGQ3NWIwZTA0NWRlYzRhYTUxMjNhNjcxYjQ1NjQ0NjZiMDcwNzA4NmE3ZWQxYmE4NzIxNjI2ZGZmZmJjOT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSomcmVzcG9uc2UtY29udGVudC10eXBlPWFwcGxpY2F0aW9uJTJGZ3ppcCIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTY3NzQ4Mzc3MH19fV19&Signature=yjL3UeY72cf2xpnvPvD68eAYOEe2qtaUJV55sB-jnPskBJEMwpMJcBZvg2~GqXZdM3O-GWV-Z3CI~d4u5VCb4YZ-HlmOjr3VBYkvox2EKiXnBIhjMecf2UVUPtxhTa9kBVlWjqu4qKzB9gKXZF2Cwpp5ctLzapEaT2nnqF84RAL-rsqMA3I~M8vWWfivQsbBK63hMfgZqqKMgdWM0iKMaItveDl0ufQ29azMFmsR7qd8V7sU2Z-F1fAeohS8HpN9OOnClW34yi~YJ2AbgZJJBXA~qsylfVA0Qp7Q~yX~q4P8JF1vmJ2BjkiSbGrj3bAXOGugpOVU5msI52DT88yMdA__&Key-Pair-Id=KVTP0A1DKRTAX')
```


### Expected behavior

This should retrieve the first example from the C4 validation set. This worked a few days ago but stopped working now.


### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
Use soundfile for mp3 decoding instead of torchaudio
State: closed

I've removed `torchaudio` completely and switched to use `soundfile` for everything. With the new version of `soundfile` package this should work smoothly because the `libsndfile` C library is bundled, in Linux wheels too. 

Let me know if you think it's too harsh and we should continue to support `torchaudio` decoding.

I decided that we can drop it completely because:
1. it's always something wrong with `torchaudio` (for example recently https://github.com/huggingface/datasets/issues/5488 )
2. the results of mp3 decoding are different depending on `torchaudio` version 
3. `soundfile` is slightly faster then the latest `torchaudio`
4. anyway users can pass any custom decoding function with any library they want if needed (worth putting a snippet in the docs).

cc @sanchit-gandhi @vaibhavad
Datasets 2.10.0 does not reuse the dataset cache
State: closed

### Describe the bug

download_mode="reuse_dataset_if_exists" will always consider that a dataset doesn't exist.

Specifically, upon losing an internet connection trying to load a dataset for a second time in ten seconds, a connection error results, showing a breakpoint of:

```
File ~/jupyterlab/.direnv/python-3.9.6/lib/python3.9/site-packages/datasets/load.py:1174, in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1165 except Exception as e:  # noqa: catch any exception of hf_hub and consider that the dataset doesn't exist
   1166     if isinstance(
   1167         e,
   1168         (
   (...)
   1172         ),
   1173     ):
-> 1174         raise ConnectionError(f"Couldn't reach '{path}' on the Hub ({type(e).__name__})")
   1175     elif "404" in str(e):
   1176         msg = f"Dataset '{path}' doesn't exist on the Hub"

ConnectionError: Couldn't reach 'lsb/tenk' on the Hub (ConnectionError)
```

This has been around since at least v2.0.


### Steps to reproduce the bug

```
from datasets import load_dataset
import numpy as np

tenk = load_dataset("lsb/tenk") # ten thousand integers
print(np.average(tenk['train']['a'])) # prints 4999.5

### now disconnect your internet

tenk_too = load_dataset("lsb/tenk", download_mode="reuse_dataset_if_exists") 
# Raises ConnectionError: Couldn't reach 'lsb/tenk' on the Hub (ConnectionError)
```

### Expected behavior

I expected that I would be able to reuse the dataset I just downloaded.

### Environment info


- `datasets` version: 2.10.0
- Platform: macOS-13.1-arm64-arm-64bit
- Python version: 3.9.6
- PyArrow version: 7.0.0
- Pandas version: 1.5.2
load_dataset fails for JSON in windows
State: closed

### Describe the bug

Steps:
1. Created a dataset in a Linux VM and created a small sample using dataset.to_json() method.
2. Downloaded the JSON file to my local Windows machine for working and saved in say - r"C:\Users\name\file.json"
3. I am reading the file in my local PyCharm - the location of python file is different than the location of the JSON. 
4. When I read using load_dataset("json",args.input_json), it throws and error from builder.py. 
raise InvalidConfigName(
                    f"Bad characters from black list '{invalid_windows_characters}' found in '{self.name}'. "
                    f"They could create issues when creating a directory for this config on Windows filesystem."
6. When I bring the data to the current directory, it works fine. 

### Steps to reproduce the bug

Steps:
1. Created a dataset in a Linux VM and created a small sample using dataset.to_json() method.
2. Downloaded the JSON file to my local Windows machine for working and saved in say - r"C:\Users\name\file.json"
3. I am reading the file in my local PyCharm - the location of python file is different than the location of the JSON. 
4. When I read using load_dataset("json",args.input_json), it throws and error from builder.py. 
raise InvalidConfigName(
                    f"Bad characters from black list '{invalid_windows_characters}' found in '{self.name}'. "
                    f"They could create issues when creating a directory for this config on Windows filesystem."
6. When I bring the data to the current directory, it works fine. 

### Expected behavior

Should be able to read from a path different than current directory in Windows machine.

### Environment info

datasets version: 2.3.1
python version: 3.8
Windows OS
load_dataset gives FileNotFoundError on imagenet-1k if license is not accepted on the hub
State: closed

### Describe the bug

When calling ```load_dataset('imagenet-1k')``` FileNotFoundError is raised, if not logged in and if logged in with huggingface-cli but not having accepted the licence on the hub. There is no error once accepting.

### Steps to reproduce the bug

```
from datasets import load_dataset

imagenet = load_dataset("imagenet-1k", split="train", streaming=True)

FileNotFoundError: Couldn't find a dataset script at /content/imagenet-1k/imagenet-1k.py or any data file in the same directory. Couldn't find 'imagenet-1k' on the Hugging Face Hub either: FileNotFoundError: Dataset 'imagenet-1k' doesn't exist on the Hub
```
tested on a colab notebook.

### Expected behavior

I would expect a specific error indicating that I have to login then accept the dataset licence.

I find this bug very relevant as this code is on a guide on the [Huggingface documentation for Datasets](https://huggingface.co/docs/datasets/about_mapstyle_vs_iterable)

### Environment info

google colab cpu-only instance
dataset.to_iterable_dataset() loses useful info like dataset features
State: closed
Labels: name: enhancement, name: good first issue

### Describe the bug

Hello,
I like the new `to_iterable_dataset` feature but I noticed something that seems to be missing.

When using `to_iterable_dataset` to transform your map style dataset into iterable dataset, you lose valuable metadata like the features. 

These metadata are useful if you want to interleave iterable datasets, cast columns etc.

### Steps to reproduce the bug

```python
dataset = load_dataset("lhoestq/demo1")["train"]
print(dataset.features)
# {'id': Value(dtype='string', id=None), 'package_name': Value(dtype='string', id=None), 'review': Value(dtype='string', id=None), 'date': Value(dtype='string', id=None), 'star': Value(dtype='int64', id=None), 'version_id': Value(dtype='int64', id=None)}
dataset = dataset.to_iterable_dataset()
print(dataset.features)
# None
```

### Expected behavior

Keep the relevant information

### Environment info

datasets==2.10.0
Directly reading parquet files in a s3 bucket from the load_dataset method
State: open
Labels: name: duplicate, name: enhancement

### Feature request

Right now, we have to read the get the parquet file to the local storage. So having ability to read given the bucket directly address would be benificial 

### Motivation

In a production set up, this feature can help us a lot. So we do not need move training datafiles in between storage.  

### Your contribution

I am willing to help if there's anyway.
Add writer_batch_size for ArrowBasedBuilder
State: closed

This way we can control the size of the record_batches/row_groups of arrow/parquet files.

This can be useful for `datasets-server` to keep control of the row groups size which can affect random access performance for audio/image/video datasets

Right now having 1,000 examples per row group cause some image datasets to be pretty slow for random access (e.g. 4seconds for `beans` to get 20 rows)
Update csv.py
State: closed

Removed mangle_dup_cols=True from BuilderConfig.

It triggered following deprecation warning:
/usr/local/lib/python3.8/dist-packages/datasets/download/streaming_download_manager.py:776: FutureWarning: the 'mangle_dupe_cols' keyword is deprecated and will be removed in a future version. Please take steps to stop the use of 'mangle_dupe_cols'
  return pd.read_csv(xopen(filepath_or_buffer, "rb", use_auth_token=use_auth_token), **kwargs)

Further documentation of pandas: https://pandas.pydata.org/docs/whatsnew/v1.4.0.html#mangle-dupe-cols-in-read-csv-no-longer-renames-unique-columns-conflicting-with-target-names

At first sight it seems like this flag is resolved internally, it might need some more research.
Fix map suffix_template
State: closed

#5455 introduced a small bug that lead `map` to ignore the `suffix_template` argument and not put suffixes to cached files in multiprocessing.

I fixed this and also improved a few things:

- regarding logging: "Loading cached processed dataset" is now logged only once even in multiprocessing (it used to be logged `num_proc` times)
- regarding new_fingerprint: I made sure that the returned dataset satisfies `ds._fingerprint==new_fingerprint` if `new_fingerprint` is passed to `map`
Use default audio resampling type
State: closed

...instead of relying on the optional librosa dependency `resampy`.

It was only used for `_decode_non_mp3_file_like` anyway and not for the other ones - removing it fixes consistency between decoding methods (except torchaudio decoding)

Therefore I think it is a better solution than adding `resampy` as a dependency in https://github.com/huggingface/datasets/pull/5554

cc @polinaeterna
Resolve four broken refs in the docs
State: closed

Hello!

## Pull Request overview
* Resolve 4 broken references in the docs

## The problems
Two broken references [here](https://huggingface.co/docs/datasets/package_reference/main_classes#datasets.Dataset.class_encode_column): 
![image](https://user-images.githubusercontent.com/37621491/220056232-366b64dc-33c9-461b-8f82-1ac4aa570280.png)

---

One broken reference [here](https://huggingface.co/docs/datasets/package_reference/main_classes#datasets.Dataset.unique): 
![image](https://user-images.githubusercontent.com/37621491/220057135-2f249d60-c01d-48b5-82bb-5085a7635198.png)

---

One missing reference [here](https://huggingface.co/docs/datasets/v2.9.0/en/package_reference/main_classes#datasets.DatasetDict.class_encode_column):
![image](https://user-images.githubusercontent.com/37621491/220057025-4a8e5556-5041-4ec7-b8d8-ed4fdc266495.png)

- Tom Aarsen
Apply flake8-comprehensions to codebase
State: closed
Labels: name: enhancement

### Feature request

Apply ruff flake8 comprehension checks to codebase. 

### Motivation

This should strictly improve the performance / readability of the codebase by removing unnecessary iteration, function calls, etc. This should generate better Python bytecode which should strictly improve performance.

I already applied this fixes to PyTorch and Sympy with little issue and have opened PRs to diffusers and transformers todo this as well.

### Your contribution

Making a PR.
Add JAX device selection when formatting
State: closed

## What's in this PR?

After exploring for a while the JAX integration in 🤗`datasets`, I found out that, even though JAX prioritizes the TPU and GPU as the default device when available, the `JaxFormatter` doesn't let you specify the device where you want to place the `jax.Array`s in case you don't want to rely on JAX's default array placement.

So on, I've included the `device` param in `JaxFormatter` but there are some things to take into consideration:
* A formatted `Dataset` is copied with `copy.deepcopy` which means that if one adds the param `device` in `JaxFormatter` as a `jaxlib.xla_extension.Device`, it "fails" because that object cannot be serialized (instead of serializing the param adds a random hash instead). That's the reason why I added a function `_map_devices_to_str` to basically create a mapping of strings to `jaxlib.xla_extension.Device`s so that `self.device` is a string and not a `jaxlib.xla_extension.Device`.
* To create a `jax.Array` in a device you need to either create it in the default device and then move it to the desired device with `jax.device_put` or directly create it in the device you want with `jax.default_device()` context manager.
* JAX will create an array by default in `jax.devices()[0]`

More information on JAX device management is available at https://jax.readthedocs.io/en/latest/faq.html#controlling-data-and-computation-placement-on-devices

## What's missing in this PR?

I've tested it both locally in CPU (Mac M2 and Mac M1, as no GPU support for Mac yet), and in GPU and TPU in Google Colab, let me know if you want me to provide you the Notebook for the latter.

But I did not implement any integration test as I wanted to get your feedback first.
Downloaded datasets do not cache at $HF_HOME
State: closed

### Describe the bug

In the huggingface course (https://huggingface.co/course/chapter3/2?fw=pt) it said that if we set HF_HOME, downloaded datasets would be cached at specified address but it does not. downloaded models from checkpoint names are downloaded and cached at HF_HOME but this is not the case for datasets, they are still cached at ~/.cache/huggingface/datasets.

### Steps to reproduce the bug

Run the following code
```
from datasets import load_dataset
raw_datasets = load_dataset("glue", "mrpc")
raw_datasets
```

it downloads and store dataset at ~/.cache/huggingface/datasets

### Expected behavior

to cache dataset at HF_HOME.

### Environment info

python 3.10.6
Kubuntu 22.04
HF_HOME located on a separate partition
the pile datasets url seems to change back
State: closed

### Describe the bug

in #3627, the host url of the pile dataset became `https://mystic.the-eye.eu`. Now the new url is broken, but `https://the-eye.eu` seems to work again.

### Steps to reproduce the bug

```python3
from datasets import load_dataset
dataset = load_dataset("bookcorpusopen")
```
shows
```python3
ConnectionError: Couldn't reach https://mystic.the-eye.eu/public/AI/pile_preliminary_components/books1.tar.gz (ProxyError(MaxRetryError("HTTPSConnectionPool(host='mystic.the-eye.eu', port=443): Max retries exceeded with url: /public/AI/pile_pr
eliminary_components/books1.tar.gz (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 504 Gateway Timeout')))")))
```

### Expected behavior

Downloading as normal.

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.4.143.bsk.7-amd64-x86_64-with-glibc2.31
- Python version: 3.9.2
- PyArrow version: 6.0.1
- Pandas version: 1.5.3
IndexError: invalid index of a 0-dim tensor. Use `tensor.item()` in Python or `tensor.item<T>()` in C++ to convert a 0-dim tensor to a number
State: closed
Labels: name: good first issue

### Describe the bug

When dataset contains a 0-dim tensor, formatting.py raises a following error and fails.
```bash
Traceback (most recent call last):
  File "<path>/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 501, in format_row
    return _unnest(formatted_batch)
  File "<path>/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 137, in _unnest
    return {key: array[0] for key, array in py_dict.items()}
  File "<path>/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 137, in <dictcomp>
    return {key: array[0] for key, array in py_dict.items()}
IndexError: invalid index of a 0-dim tensor. Use `tensor.item()` in Python or `tensor.item<T>()` in C++ to convert a 0-dim tensor to a number
```

### Steps to reproduce the bug

Load whichever dataset and add transform method to add 0-dim tensor. Or create/find a dataset containing 0-dim tensor. E.g.

```python
from datasets import load_dataset
import torch

dataset = load_dataset("lambdalabs/pokemon-blip-captions", split='train')
def t(batch):
    return {"test": torch.tensor(1)}
    
dataset.set_transform(t)
d_0 = dataset[0]
```

### Expected behavior

Extractor will correctly get a row from the dataset, even if it contains 0-dim tensor.

### Environment info

`datasets==2.8.0`, but it looks like it is also applicable to main branch version (as of 16th February)