kardwalker/my_dataset huggingface dataset:
GitHub issue #5639: Parquet file wrongly recognized as zip prevents loading a dataset
State: closed | Created: 2023-03-15T15:20:45Z

### Describe the bug

When trying to `load_dataset_builder` for `HuggingFaceGECLM/StackExchange_Mar2023`, extraction fails, because parquet file [devops-00000-of-00001-22fe902fd8702892.parquet](https://huggingface.co/datasets/HuggingFaceGECLM/StackExchange_Mar2023/resolve/1f8c9a2ab6f7d0f9ae904b8b922e4384592ae1a5/data/devops-00000-of-00001-22fe902fd8702892.parquet) is wrongly identified by python as being a zip not a parquet.
(Full thread on [Slack](https://huggingface.slack.com/archives/C02V51Q3800/p1678890880803599))

### Steps to reproduce the bug

```python
from datasets import load_dataset_builder

ds = load_dataset_builder("HuggingFaceGECLM/StackExchange_Mar2023")
```

### Expected behavior

Loading the file normally.

### Environment info

- `datasets` version: 2.3.2
- Platform: Linux-5.14.0-1058-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3
GitHub issue #5638: xPath to implement all operations for Path
State: closed | Created: 2023-03-15T13:47:11Z

### Feature request

Current xPath implementation is a great extension of Path in order to work with remote objects. However some methods such as `mkdir` are not implemented correctly. It should instead rely on `fsspec` methods, instead of defaulting do `Path` methods which only work locally.

### Motivation

I'm using xPath to interact with remote objects.

### Your contribution

I could try to make a PR. I'm a bit unfamiliar with chaining right now.
GitHub issue #5637: IterableDataset with_format does not support 'device' keyword for jax
State: open | Created: 2023-03-15T11:04:12Z

### Describe the bug

As seen here: https://huggingface.co/docs/datasets/use_with_jax dataset.with_format() supports the keyword 'device', to put data on a specific device when loaded as jax. However, when called on an IterableDataset, I got the error `TypeError: with_format() got an unexpected keyword argument 'device'`

Looking over the code, it seems IterableDataset support only pytorch and no support for jax device keyword? 
https://github.com/huggingface/datasets/blob/fc5c84f36684343bff3e424cb0fd1ac5ecdd66da/src/datasets/iterable_dataset.py#L1029



### Steps to reproduce the bug

1. Load an IterableDataset (tested in streaming mode)
2. Call with_format('jax',device=device)

### Expected behavior

I expect to call `with_format('jax', device=device)`  as per [documentation](https://huggingface.co/docs/datasets/use_with_jax) without error

### Environment info

Tested with installing newest (dev) and also pip release (2.10.1).
- `datasets` version: 2.10.2.dev0
- Platform: Linux-5.15.89+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- Huggingface_hub version: 0.12.1
- PyArrow version: 11.0.0
- Pandas version: 1.3.5
GitHub issue #5635: Pass custom metadata filename to Image/Audio folders
State: open | Created: 2023-03-14T15:08:16Z

This is a quick fix. 
Now it requires to pass data via `data_files` parameters and include a required metadata file there and pass its filename as `metadata_filename` parameter.
For example, with the structure like:
```
data
   images_dir/
      im1.jpg
      im2.jpg
      ...
   metadata_dir/
      meta_file1.jsonl
      meta_file2.jsonl
      ...
```
to load data with `metadata_file1.jsonl` do:
```python
ds = load_dataset("imagefolder", data_files=["data/images_dir/**", "data/metadata_dir/meta_file1.jsonl"], metadata_filename="meta_file1.jsonl")
```

Note that if you have multiple splits, metadata file should be specified in each of them in `data_files`, smth like:
```python
data_files={
"train": ["data/train/**", "data/metadata_dir/meta_file1.jsonl"], 
"test": ["data/train/**", "data/metadata_dir/meta_file1.jsonl"]
}
```
GitHub issue #5632: Dataset cannot convert too large dictionnary
State: open | Created: 2023-03-13T10:14:40Z

### Describe the bug

Hello everyone!

I tried to build a new dataset with the command "dict_valid = datasets.Dataset.from_dict({'input_values': values_array})".
However, I have a very large dataset (~400Go) and it seems that dataset cannot handle this.

Indeed, I can create the dataset until a certain size of my dictionnary, and then I have the error "OverflowError: Python int too large to convert to C long".

Do you know how to solve this problem?
Unfortunately I cannot give a reproductible code because I cannot share a so large file, but you can find the code below (it's a test on only a part of the validation data ~10Go, but it's already the case).

Thank you!

### Steps to reproduce the bug

SAVE_DIR = './data/'
features = h5py.File(SAVE_DIR+'features.hdf5','r')

valid_data = features["validation"]["data/features"]

v_array_values = [np.float32(item[()]) for item in valid_data.values()]
for i in range(len(v_array_values)):
    v_array_values[i] = v_array_values[i].round(decimals=5)

dict_valid = datasets.Dataset.from_dict({'input_values': v_array_values})

### Expected behavior

The code is expected to give me a Huggingface dataset.

### Environment info

python: 3.8.15
numpy: 1.22.3
datasets: 2.3.2
pyarrow: 8.0.0
GitHub issue #5631: Custom split names
State: closed | Created: 2023-03-12T17:21:43Z

### Feature request

Hi,
I participated in multiple NLP tasks where there are more than just train, test, validation splits, there could be multiple validation sets or test sets. But it seems currently only those mentioned three splits supported. It would be nice to have the support for more splits on the hub. (currently i can have more splits when I am loading datasets from urls, but not hub)

### Motivation

Easier access to more splits

### Your contribution

No
GitHub issue #5629: load_dataset gives "403" error when using Financial phrasebank
State: open | Created: 2023-03-11T07:46:39Z

When I try to load this dataset, I receive the following error: 

ConnectionError: Couldn't reach https://www.researchgate.net/profile/Pekka_Malo/publication/251231364_FinancialPhraseBank-v10/data/0c96051eee4fb1d56e000000/FinancialPhraseBank-v10.zip (error 403)

Has this been seen before? Thanks. The website loads when I try to access it manually.
GitHub issue #5628: add kwargs to index search
State: closed | Created: 2023-03-10T21:24:58Z

This PR proposes to add kwargs to index search methods.

This is particularly useful for setting the timeout of a query on elasticsearch.

A typical use case would be:
```python
dset.add_elasticsearch_index("filename", es_client=es_client)
scores, examples = dset.get_nearest_examples("filename", "my_name-train_29", request_timeout=60)
```
GitHub issue #5625: Allow "jsonl" data type signifier
State: open | Created: 2023-03-10T13:21:48Z

### Feature request

`load_dataset` currently does not accept `jsonl` as type but only `json`. 

### Motivation

I was working with one of the `run_translation` scripts and used my own datasets (`.jsonl`) as train_dataset. But the default code did not work because

```
FileNotFoundError: Couldn't find a dataset script at jsonl\jsonl.py or any data file in the same directory. Couldn't find 'jsonl' on the Hugging Face Hub either: FileNotFoundError: Dataset 'jsonl' doesn't exist on the Hub. If the repo is private or gated, make sure to log in with `huggingface-cli login`.
```

The reason is because the script has these lines to extract the data type by its extension. Therefore, the derived type is `jsonl` which is not recognized by datasets as the error above shows.

https://github.com/huggingface/transformers/blob/ade26bf9912f69e2110137443e4406d7dbe253e7/examples/pytorch/translation/run_translation.py#L342-L356

I suppose you could argue that this is the script's fault (in which case I'll do a PR over at `transformers`) but it makes sense to me to add `jsonl` as an alias to `json` in `datasets`.


### Your contribution

At the moment I cannot work on this. I think it can be as "easy" as having an alias for json, namely jsonl.
GitHub issue #5624: glue datasets returning -1 for test split
State: closed | Created: 2023-03-09T14:47:18Z

### Describe the bug

Downloading any dataset from GLUE has -1 as class labels for test split. Train and validation have regular 0/1 class labels.  This is also present in the dataset card online. 

### Steps to reproduce the bug

```
dataset = load_dataset("glue", "sst2")
for d in dataset:
    # prints out -1
    print(d["label"]
```

### Expected behavior

Expected behavior should be 0/1 instead of -1. 

### Environment info

- `datasets` version: 2.4.0
- Platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.17
- Python version: 3.8.16
- PyArrow version: 8.0.0
- Pandas version: 1.5.3
GitHub issue #5623: Remove set_access_token usage + fail tests if FutureWarning
State: closed | Created: 2023-03-09T08:46:01Z

`set_access_token` is deprecated and will be removed in `huggingface_hub>=0.14`.

This PR removes it from the tests (it was not used in `datasets` source code itself). FYI, it was not needed since `set_access_token` was just setting git credentials and `datasets` doesn't seem to use git anywhere.

In the future, use `set_git_credential` if needed. It is a git-credential-agnostic helper, i.e. you can store your git token in `git-credential-cache`, `git-credential-store`, `osxkeychain`, etc. The legacy `set_access_token` could only set in `git-credential-store` no matter the user preference.

(for context, I found out about this while working on https://github.com/huggingface/huggingface_hub/pull/1381)

---

In addition to this, I have added

```
filterwarnings =
    error::FutureWarning:huggingface_hub*
```

to the `setup.cfg` config file to fail on future warnings from `huggingface_hub`. In `hfh`'s CI we trigger on FutureWarning from any package but it's less robust (any package update leads can lead to a failure). No obligation to keep it like that (I can remove it if you prefer) but I think it's a good idea in order to track future FutureWarnings.

FYI, in `huggingface_hub` tests we use `-Werror::FutureWarning --log-cli-level=INFO -sv --durations=0`
- FutureWarning are processed as error
- verbose mode / INFO logs (and above) are captured for easier debugging in github report
- track each test duration, just to see where we can improve. We have a quite long CI (~10min) so it helped improve that.
GitHub issue #5620: Bump pyarrow to 8.0.0
State: closed | Created: 2023-03-07T13:31:53Z

Fix those for Pandas 2.0 (tested [here](https://github.com/huggingface/datasets/actions/runs/4346221280/jobs/7592010397) with pandas==2.0.0.rc0):

```python
=========================== short test summary info ============================
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_to_parquet_in_memory - ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.
A suitable version of pyarrow or fastparquet is required for parquet support.
Trying to import the above resulted in these errors:
 - Pandas requires version '7.0.0' or newer of 'pyarrow' (version '6.0.1' currently installed).
 - Missing optional dependency 'fastparquet'. fastparquet is required for parquet support. Use pip or conda to install fastparquet.
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_to_parquet_on_disk - ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.
A suitable version of pyarrow or fastparquet is required for parquet support.
Trying to import the above resulted in these errors:
 - Pandas requires version '7.0.0' or newer of 'pyarrow' (version '6.0.1' currently installed).
 - Missing optional dependency 'fastparquet'. fastparquet is required for parquet support. Use pip or conda to install fastparquet.
===== 2 failed, 2137 passed, 18 skipped, 32 warnings in 212.76s (0:03:32) ======
```

EDIT: also for performance - with 8.0 we can use `.to_reader()`
GitHub issue #5616: CI is broken after fsspec-2023.3.0 release
State: closed | Created: 2023-03-07T08:06:39Z

As reported by @lhoestq, our CI is broken after `fsspec` 2023.3.0 release:
```
FAILED tests/test_filesystem.py::test_compression_filesystems[Bz2FileSystem] - AssertionError: assert [{'created': ...: False, ...}] == ['file.txt']
  At index 0 diff: {'name': 'file.txt', 'size': 70, 'type': 'file', 'created': 1678175677.1887748, 'islink': False, 'mode': 33188, 'uid': 1001, 'gid': 123, 'mtime': 1678175677.1887748, 'ino': 286957, 'nlink': 1} != 'file.txt'
  Full diff:
    [
  -  'file.txt',
  +  {'created': 1678175677.1887748,
  +   'gid': 123,
  +   'ino': 286957,
  +   'islink': False,
  +   'mode': 33188,
  +   'mtime': 1678175677.1887748,
  +   'name': 'file.txt',
  +   'nlink': 1,
  +   'size': 70,
  +   'type': 'file',
  +   'uid': 1001},
    ]
```
Also:
```
FAILED tests/test_filesystem.py::test_compression_filesystems[GzipFileSystem] - AssertionError: assert [{'created': ...: False, ...}] == ['file.txt']
FAILED tests/test_filesystem.py::test_compression_filesystems[Lz4FileSystem] - AssertionError: assert [{'created': ...: False, ...}] == ['file.txt']
FAILED tests/test_filesystem.py::test_compression_filesystems[XzFileSystem] - AssertionError: assert [{'created': ...: False, ...}] == ['file.txt']
FAILED tests/test_filesystem.py::test_compression_filesystems[ZstdFileSystem] - AssertionError: assert [{'created': ...: False, ...}] == ['file.txt']
===== 5 failed, 2134 passed, 18 skipped, 38 warnings in 157.21s (0:02:37) ======
```

See:
- fsspec/filesystem_spec#1205
GitHub issue #5615: IterableDataset.add_column is unable to accept another IterableDataset as a parameter.
State: closed | Created: 2023-03-07T01:52:00Z

### Describe the bug

`IterableDataset.add_column` occurs an exception when passing another `IterableDataset` as a parameter.
The method seems to accept only eager evaluated values.

https://github.com/huggingface/datasets/blob/35b789e8f6826b6b5a6b48fcc2416c890a1f326a/src/datasets/iterable_dataset.py#L1388-L1391


I wrote codes below to make it.
```py
def add_column(dataset: IterableDataset, name: str, add_dataset: IterableDataset, key: str) -> IterableDataset:
    iter_add_dataset = iter(add_dataset)

    def add_column_fn(example):
        if name in example:
            raise ValueError(f"Error when adding {name}: column {name} is already in the dataset.")
        return {name: next(iter_add_dataset)[key]}

    return dataset.map(add_column_fn)
```

Is there other way to do it? Or is it intended?

### Steps to reproduce the bug

Thie codes below occurs `NotImplementedError`
```py
from datasets import IterableDataset


def gen(num):
    yield {f"col{num}": 1}
    yield {f"col{num}": 2}
    yield {f"col{num}": 3}


ids1 = IterableDataset.from_generator(gen, gen_kwargs={"num": 1})
ids2 = IterableDataset.from_generator(gen, gen_kwargs={"num": 2})
new_ids = ids1.add_column("new_col", ids1)

for row in new_ids:
    print(row)
```

### Expected behavior

`IterableDataset.add_column` is able to task `IterableDataset` and lazy evaluated values as a parameter since IterableDataset is lazy evalued.

### Environment info
- `datasets` version: 2.8.0
- Platform: Linux-3.10.0-1160.36.2.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.7
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
GitHub issue #5612: Arrow map type in parquet files unsupported
State: open | Created: 2023-03-06T12:03:24Z

### Describe the bug

When I try to load parquet files that were processed with Spark, I get the following issue:

`ValueError: Arrow type map<string, string ('warc_headers')> does not have a datasets dtype equivalent.`

Strangely, loading the dataset with `streaming=True` solves the issue.

### Steps to reproduce the bug

The dataset is private, but this can be reproduced with any dataset that has Arrow maps.

### Expected behavior

Loading the dataset no matter whether streaming is True or not.

### Environment info

- `datasets` version: 2.10.1
- Platform: Linux-5.15.0-1029-gcp-x86_64-with-glibc2.31
- Python version: 3.10.7
- PyArrow version: 8.0.0
- Pandas version: 1.4.2
GitHub issue #5609: `load_from_disk` vs `load_dataset` performance.
State: open | Created: 2023-03-05T05:27:15Z

### Describe the bug

I have downloaded `openwebtext` (~12GB) and filtered out a small amount of junk (it's still huge). Now, I would like to use this filtered version for future work. It seems I have two choices:
1. Use `load_dataset` each time, relying on the cache mechanism, and re-run my filtering.
2. `save_to_disk` and then use `load_from_disk` to load the filtered version.

The performance of these two approaches is wildly different:
* Using `load_dataset` takes about 20 seconds to load the dataset, and a few seconds to re-filter (thanks to the brilliant filter/map caching)
* Using `load_from_disk` takes 14 minutes! And the second time I tried, the session just crashed (on a machine with 32GB of RAM)

I don't know if you'd call this a bug, but it seems like there shouldn't need to be two methods to load from disk, or that they should  not take such wildly different amounts of time, or that one should not crash. Or maybe that the docs could offer some guidance about when to pick which method and why two methods exist, or just how do most people do it?

Something I couldn't work out from reading the docs was this: can I modify a dataset from the hub, save it (locally) and use `load_dataset` to load it? This [post seemed to suggest that the answer is no](https://discuss.huggingface.co/t/save-and-load-datasets/9260).

### Steps to reproduce the bug

See above

### Expected behavior

Load times should be about the same.

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.10.8
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
GitHub issue #5608: audiofolder only creates dataset of 13 rows (files) when the data folder it's reading from has 20,000 mp3 files. 
State: closed | Created: 2023-03-05T00:14:45Z

### Describe the bug

x = load_dataset("audiofolder", data_dir="x")
When running this, x is a dataset of 13 rows (files) when it should be 20,000 rows (files) as the data_dir "x" has 20,000 mp3 files. Does anyone know what could possibly cause this (naming convention of mp3 files, etc.)


### Steps to reproduce the bug

x = load_dataset("audiofolder", data_dir="x")


### Expected behavior

x = load_dataset("audiofolder", data_dir="x") should create a dataset of 20,000 rows (files).

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-3.10.0-1160.80.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.16
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
GitHub issue #5607: Fix outdated `verification_mode` values
State: closed | Created: 2023-03-03T19:50:29Z

~I think it makes sense not to save `dataset_info.json` file to a dataset cache directory when loading dataset with `verification_mode="no_checks"` because otherwise when next time the dataset is loaded **without** `verification_mode="no_checks"`, it will be loaded successfully, despite some values in info might not correspond to the ones in the repo which was the reason for using `verification_mode="no_checks"` first.~

Updated values of `verification_mode` to the current ones in some places ("none" -> "no_checks", "all" -> "all_checks")
GitHub issue #5606: Add `Dataset.to_list` to the API
State: closed | Created: 2023-03-03T16:17:10Z

Since there is `Dataset.from_list` in the API, we should also add `Dataset.to_list` to be consistent.
 
Regarding the implementation, we can re-use `Dataset.to_dict`'s code and replace the `to_pydict` calls with `to_pylist`.
GitHub issue #5604: Problems with downloading The Pile
State: closed | Created: 2023-03-03T09:52:08Z

### Describe the bug

The downloads in the screenshot seem to be interrupted after some time and the last download throws a "Read timed out" error.

![image](https://user-images.githubusercontent.com/11065386/222687870-ec5fcb65-84e8-467d-9593-4ad7bdac4d50.png)

Here are the downloaded files:
![image](https://user-images.githubusercontent.com/11065386/222688200-454c2288-49e5-4682-96e6-1eb69aca0852.png)

They should be all 14GB like here (https://the-eye.eu/public/AI/pile/train/).

Alternatively, can I somehow download the files by myself and use the datasets preparing script?

### Steps to reproduce the bug

dataset = load_dataset('the_pile', split='train', cache_dir='F:\datasets')

### Expected behavior

The files should be downloaded correctly.

### Environment info

- `datasets` version: 2.10.1
- Platform: Windows-10-10.0.22623-SP0
- Python version: 3.10.5
- PyArrow version: 9.0.0
- Pandas version: 1.4.2
GitHub issue #5601: Authorization error
State: closed | Created: 2023-03-02T12:08:39Z

### Describe the bug

Get `Authorization error` when try to push data into hugginface datasets hub.

### Steps to reproduce the bug

I did all steps in the [tutorial](https://huggingface.co/docs/datasets/share), 

1. `huggingface-cli login` with WRITE token
2. `git lfs install`
3. `git clone https://huggingface.co/datasets/namespace/your_dataset_name`
4.
  ```
  cp /somewhere/data/*.json .
  git lfs track *.json
  git add .gitattributes
  git add *.json
  git commit -m "add json files"
  ```

but when I execute `git push` I got the error:

```
Uploading LFS objects:   0% (0/1), 0 B | 0 B/s, done.
batch response: Authorization error.
error: failed to push some refs to 'https://huggingface.co/datasets/zeusfsx/ukrainian-news'
```

Size of data ~100Gb.  I have five json files - different parts.

### Expected behavior

All my data pushed into hub

### Environment info

- `datasets` version: 2.10.1
- Platform: macOS-13.2.1-arm64-arm-64bit
- Python version: 3.10.10
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
GitHub issue #5600: Dataloader getitem not working for DreamboothDatasets
State: closed | Created: 2023-03-02T11:00:27Z

### Describe the bug

 Dataloader getitem is not working as before (see example of [DreamboothDatasets](https://github.com/huggingface/peft/blob/main/examples/lora_dreambooth/train_dreambooth.py#L451C14-L529))
moving Datasets to 2.8.0 solved the issue.

### Steps to reproduce the bug

1- using DreamBoothDataset to load some images 
2- error after loading when trying to visualise the images 

### Expected behavior

I was expecting a numpy array of the image

### Environment info

- Platform: Linux-5.10.147+-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 9.0.0
- Pandas version: 1.3.5
GitHub issue #5598: Fix push_to_hub with no dataset_infos
State: closed | Created: 2023-03-01T13:54:06Z

As reported in https://github.com/vijaydwivedi75/lrgb/issues/10, `push_to_hub` fails if the remote repository already exists and has a README.md without `dataset_info` in the YAML tags

cc @clefourrier
GitHub issue #5597: in-place dataset update
State: closed | Created: 2023-03-01T12:58:18Z

### Motivation

For the circumstance that I creat an empty `Dataset` and keep appending new rows into it, I found that it leads to creating a new dataset at each call. It looks quite memory-consuming. I just wonder if there is any more efficient way to do this.

```python
from datasets import Dataset

ds = Dataset.from_list([])

ds.add_item({'a': [1, 2, 3], 'b': 4})
print(ds)
>>> Dataset({
>>>     features: [],
>>>     num_rows: 0
>>> })

ds = ds.add_item({'a': [1, 2, 3], 'b': 4})
print(ds)
>>> Dataset({
>>>     features: ['a', 'b'],
>>>     num_rows: 1
>>> })
```

### Feature request

Call for in-place dataset update functions, that update the existing `Dataset` in place without creating a new copy. The interface is supposed to keep the same style as PyTorch, such as the in-place version of a `function` is named `function_`. For example, the in-pace version of `add_item`, i.e., `add_item_`, immediately updates the `Dataset`.

```python
from datasets import Dataset

ds = Dataset.from_list([])

ds.add_item({'a': [1, 2, 3], 'b': 4})
print(ds)
>>> Dataset({
>>>     features: [],
>>>     num_rows: 0
>>> })

ds.add_item_({'a': [1, 2, 3], 'b': 4})
print(ds)
>>> Dataset({
>>>     features: ['a', 'b'],
>>>     num_rows: 1
>>> })
```

### Related Functions

* `.map`
* `.filter`
* `.add_item`
GitHub issue #5589: Revert "pass the dataset features to the IterableDataset.from_generator"
State: closed | Created: 2023-02-28T17:52:04Z

This reverts commit b91070b9c09673e2e148eec458036ab6a62ac042 (temporarily)

It hurts iterable dataset performance a lot (e.g. x4 slower because it encodes+decodes images unnecessarily). I think we need to fix this before re-adding it

cc @mariosasko @Hubert-Bonisseur
GitHub issue #5588: Flatten dataset on the fly in `save_to_disk`
State: closed | Created: 2023-02-28T15:37:46Z

Flatten a dataset on the fly in `save_to_disk` instead of doing it with `flatten_indices`  to avoid creating an additional cache file.

(this is one of the sub-tasks in https://github.com/huggingface/datasets/issues/5507)
GitHub issue #5586: .sort() is broken when used after .filter(), only in 2.10.0
State: closed | Created: 2023-02-28T12:18:09Z

### Describe the bug

Hi, thank you for your support!

It seems like the addition of multiple key sort (#5502) in 2.10.0 broke the `.sort()` method.

After filtering a dataset with `.filter()`, the `.sort()` seems to refer to the query_table index of the previous unfiltered dataset, resulting in an IndexError.

This only happens with the 2.10.0 release.

### Steps to reproduce the bug

```Python
from datasets import load_dataset

# dataset with length of 1104
ds = load_dataset('glue', 'ax')['test']
ds = ds.filter(lambda x: x['idx'] > 1100)
ds.sort('premise')
print('Done')
```

  File "/home/dongkeun/datasets_test/test.py", line 5, in <module>
    ds.sort('premise')
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 528, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/fingerprint.py", line 511, in wrapper
    out = func(dataset, *args, **kwargs)
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3959, in sort
    sort_table = query_table(
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/formatting/formatting.py", line 588, in query_table
    _check_valid_index_key(key, size)
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/formatting/formatting.py", line 537, in _check_valid_index_key
    _check_valid_index_key(max(key), size=size)
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/formatting/formatting.py", line 531, in _check_valid_index_key
    raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 1103 is out of bounds for size 3

### Expected behavior

It should sort the dataset and print "Done". Which it does on 2.9.0.

### Environment info

- `datasets` version: 2.10.0
- Platform: Linux-5.15.0-41-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
GitHub issue #5585: Cache is not transportable
State: closed | Created: 2023-02-28T00:53:06Z

### Describe the bug

I would like to share cache between two machines (a Windows host machine and a WSL instance).

I run most my code in WSL. I have just run out of space in the virtual drive. Rather than expand the drive size, I plan to move to cache to the host Windows machine, thereby sharing the downloads.

I'm hoping that I can just copy/paste the cache files, but I notice that a lot of the file names start with the path name, e.g. `_home_davidg_.cache_huggingface_datasets_conll2003_default-451...98.lock` where `home/davidg` is where the cache is in WSL. 

This seems to suggest that the cache is not portable/cannot be centralised or shared. Is this the case, or are the files that start with path names not integral to the caching mechanism? Because copying the cache files _seems_ to work, but I'm not filled with confidence that something isn't going to break.

A related issue, when trying to load a dataset that should come from cache (running in WSL, pointing to cache on the Windows host) it seemed to work fine, but it still uses a WSL directory for `.cache\huggingface\modules\datasets_modules`. I see nothing in the docs about this, or how to point it to a different place.

I have asked a related question on the forum: https://discuss.huggingface.co/t/is-datasets-cache-operating-system-agnostic/32656

### Steps to reproduce the bug

View the cache directory in WSL/Windows.

### Expected behavior

Cache can be shared between (virtual) machines and be transportable.

It would be nice to have a simple way to say "Dear Hugging Face packages, please put ALL your cache in `blah/de/blah`" and have all the Hugging Face packages respect that single location.

### Environment info

```
- `datasets` version: 2.9.0
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.10.8
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
- ```
GitHub issue #5584: Unable to load coyo700M dataset
State: closed | Created: 2023-02-27T19:35:03Z

### Describe the bug

Seeing this error when downloading https://huggingface.co/datasets/kakaobrain/coyo-700m:
```ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.```


Full stack trace

```Downloading and preparing dataset parquet/kakaobrain--coyo-700m to /root/.cache/huggingface/datasets/kakaobrain___parquet/kakaobrain--coyo-700m-ae729692ae3e0073/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec...
Downloading data files: 100%
1/1 [00:00<00:00, 63.35it/s]
Extracting data files: 100%
1/1 [00:00<00:00, 5.00it/s]
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
[/usr/local/lib/python3.8/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1859                 _time = time.time()
-> 1860                 for _, table in generator:
   1861                     if max_shard_size is not None and writer._num_bytes > max_shard_size:

9 frames
ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
[/usr/local/lib/python3.8/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1890             if isinstance(e, SchemaInferenceError) and e.__context__ is not None:
   1891                 e = e.__context__
-> 1892             raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1893 
   1894         yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset```

### Steps to reproduce the bug

```
from datasets import load_dataset
hf_dataset = load_dataset("kakaobrain/coyo-700m")
```

### Expected behavior

The above commands load the dataset successfully. Or handles exception and continue loading the remainder.

### Environment info

colab. any
GitHub issue #5581: [DOC] Mistaken docs on set_format
State: closed | Created: 2023-02-27T08:03:09Z

### Describe the bug

https://huggingface.co/docs/datasets/v2.10.0/en/package_reference/main_classes#datasets.Dataset.set_format
<img width="700" alt="image" src="https://user-images.githubusercontent.com/36224762/221506973-ae2e3991-60a7-4d4e-99f8-965c6eb61e59.png">

While actually running it will result in:
<img width="1094" alt="image" src="https://user-images.githubusercontent.com/36224762/221507032-007dab82-8781-4319-b21a-e6e4d40d97b3.png">


### Steps to reproduce the bug

_

### Expected behavior

_

### Environment info

- `datasets` version: 2.10.0
- Platform: Linux-5.10.147+-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 9.0.0
- Pandas version: 1.3.5
GitHub issue #5580: Support cloud storage in load_dataset via fsspec
State: closed | Created: 2023-02-27T04:06:05Z

Closes https://github.com/huggingface/datasets/issues/5281

This PR uses fsspec to support datasets on cloud storage (tested manually with GCS). ETags are currently unsupported for cloud storage. In general, a much larger refactor could be done to just use fsspec for all schemes (ftp, http/s, s3, gcs) to unify the interfaces here, but I ultimately opted to leave that out of this PR

I didn't create a GCS filesystem class in `datasets.filesystems` since the S3 one appears to be a wrapper around `s3fs.S3FileSystem` and mainly used to generate docs.
GitHub issue #5577: Cannot load `the_pile_openwebtext2`
State: closed | Created: 2023-02-24T13:01:48Z

### Describe the bug

I met the same bug mentioned in #3053 which is never fixed. Because several `reddit_scores` are larger than `int8` even `int16`. https://huggingface.co/datasets/the_pile_openwebtext2/blob/main/the_pile_openwebtext2.py#L62

### Steps to reproduce the bug

```python3
from datasets import load_dataset
dataset = load_dataset("the_pile_openwebtext2")
```

### Expected behavior

load as normal.

### Environment info

- `datasets` version: 2.10.0
- Platform: Linux-5.4.143.bsk.7-amd64-x86_64-with-glibc2.31
- Python version: 3.9.2
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
GitHub issue #5576: I was getting a similar error `pyarrow.lib.ArrowInvalid: Integer value 528 not in range: -128 to 127` - AFAICT, this is because the type specified for `reddit_scores` is `datasets.Sequence(datasets.Value("int8"))`, but the actual values can be well outside the max range for 8-bit integers.
State: closed | Created: 2023-02-24T12:57:49Z

              I was getting a similar error `pyarrow.lib.ArrowInvalid: Integer value 528 not in range: -128 to 127` - AFAICT, this is because the type specified for `reddit_scores` is `datasets.Sequence(datasets.Value("int8"))`, but the actual values can be well outside the max range for 8-bit integers.

I worked around this by downloading the `the_pile_openwebtext2.py` and editing it to use local files and drop reddit scores as a column (not needed for my purposes).

_Originally posted by @tc-wolf in https://github.com/huggingface/datasets/issues/3053#issuecomment-1281392422_
GitHub issue #5575: Metadata for each column
State: open | Created: 2023-02-24T10:53:44Z

### Feature request

Being able to put some metadata for each column as a string or any other type.


### Motivation

I will bring the motivation by an example, lets say we are experimenting with embedding produced by some image encoder network, and we want to iterate through a couple of preprocessing and see which one works better in our downstream task, here as workaround right now what I do is the compute the hash of the preprocessing that the images went through as part of the new columns name, it would be nice to attach some kinda meta data in these scenarios to the each columns. metadata

### Your contribution

Maybe we could map another relational like database as the metadata?
GitHub issue #5574: c4 dataset streaming fails with `FileNotFoundError`
State: closed | Created: 2023-02-24T07:57:32Z

### Describe the bug

Loading the `c4` dataset in streaming mode with `load_dataset("c4", "en", split="validation", streaming=True)` and then using it fails with a `FileNotFoundException`.

### Steps to reproduce the bug

```python
from datasets import load_dataset

dataset = load_dataset("c4", "en", split="train", streaming=True)
next(iter(dataset))
```

causes a 

```
FileNotFoundError: https://huggingface.co/datasets/allenai/c4/resolve/1ddc917116b730e1859edef32896ec5c16be51d0/en/c4-train.00000-of-01024.json.gz
```

I can download this file manually though e.g. by entering this URL in a browser. 

There is an underlying HTTP 403 status code:

```
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://cdn-lfs.huggingface.co/datasets/allenai/c4/8ef8d75b0e045dec4aa5123a671b4564466b0707086a7ed1ba8721626dfffbc9?response-content-disposition=attachment%3B+filename*%3DUTF-8''c4-train.00000-of-01024.json.gz%3B+filename%3D%22c4-train.00000-of-01024.json.gz%22%3B&response-content-type=application/gzip&Expires=1677483770&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jZG4tbGZzLmh1Z2dpbmdmYWNlLmNvL2RhdGFzZXRzL2FsbGVuYWkvYzQvOGVmOGQ3NWIwZTA0NWRlYzRhYTUxMjNhNjcxYjQ1NjQ0NjZiMDcwNzA4NmE3ZWQxYmE4NzIxNjI2ZGZmZmJjOT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSomcmVzcG9uc2UtY29udGVudC10eXBlPWFwcGxpY2F0aW9uJTJGZ3ppcCIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTY3NzQ4Mzc3MH19fV19&Signature=yjL3UeY72cf2xpnvPvD68eAYOEe2qtaUJV55sB-jnPskBJEMwpMJcBZvg2~GqXZdM3O-GWV-Z3CI~d4u5VCb4YZ-HlmOjr3VBYkvox2EKiXnBIhjMecf2UVUPtxhTa9kBVlWjqu4qKzB9gKXZF2Cwpp5ctLzapEaT2nnqF84RAL-rsqMA3I~M8vWWfivQsbBK63hMfgZqqKMgdWM0iKMaItveDl0ufQ29azMFmsR7qd8V7sU2Z-F1fAeohS8HpN9OOnClW34yi~YJ2AbgZJJBXA~qsylfVA0Qp7Q~yX~q4P8JF1vmJ2BjkiSbGrj3bAXOGugpOVU5msI52DT88yMdA__&Key-Pair-Id=KVTP0A1DKRTAX')
```


### Expected behavior

This should retrieve the first example from the C4 validation set. This worked a few days ago but stopped working now.


### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
GitHub issue #5573: Use soundfile for mp3 decoding instead of torchaudio
State: closed | Created: 2023-02-23T19:19:44Z

I've removed `torchaudio` completely and switched to use `soundfile` for everything. With the new version of `soundfile` package this should work smoothly because the `libsndfile` C library is bundled, in Linux wheels too. 

Let me know if you think it's too harsh and we should continue to support `torchaudio` decoding.

I decided that we can drop it completely because:
1. it's always something wrong with `torchaudio` (for example recently https://github.com/huggingface/datasets/issues/5488 )
2. the results of mp3 decoding are different depending on `torchaudio` version 
3. `soundfile` is slightly faster then the latest `torchaudio`
4. anyway users can pass any custom decoding function with any library they want if needed (worth putting a snippet in the docs).

cc @sanchit-gandhi @vaibhavad
GitHub issue #5572: Datasets 2.10.0 does not reuse the dataset cache
State: closed | Created: 2023-02-23T17:28:11Z

### Describe the bug

download_mode="reuse_dataset_if_exists" will always consider that a dataset doesn't exist.

Specifically, upon losing an internet connection trying to load a dataset for a second time in ten seconds, a connection error results, showing a breakpoint of:

```
File ~/jupyterlab/.direnv/python-3.9.6/lib/python3.9/site-packages/datasets/load.py:1174, in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1165 except Exception as e:  # noqa: catch any exception of hf_hub and consider that the dataset doesn't exist
   1166     if isinstance(
   1167         e,
   1168         (
   (...)
   1172         ),
   1173     ):
-> 1174         raise ConnectionError(f"Couldn't reach '{path}' on the Hub ({type(e).__name__})")
   1175     elif "404" in str(e):
   1176         msg = f"Dataset '{path}' doesn't exist on the Hub"

ConnectionError: Couldn't reach 'lsb/tenk' on the Hub (ConnectionError)
```

This has been around since at least v2.0.


### Steps to reproduce the bug

```
from datasets import load_dataset
import numpy as np

tenk = load_dataset("lsb/tenk") # ten thousand integers
print(np.average(tenk['train']['a'])) # prints 4999.5

### now disconnect your internet

tenk_too = load_dataset("lsb/tenk", download_mode="reuse_dataset_if_exists") 
# Raises ConnectionError: Couldn't reach 'lsb/tenk' on the Hub (ConnectionError)
```

### Expected behavior

I expected that I would be able to reuse the dataset I just downloaded.

### Environment info


- `datasets` version: 2.10.0
- Platform: macOS-13.1-arm64-arm-64bit
- Python version: 3.9.6
- PyArrow version: 7.0.0
- Pandas version: 1.5.2
GitHub issue #5571: load_dataset fails for JSON in windows
State: closed | Created: 2023-02-23T16:50:11Z

### Describe the bug

Steps:
1. Created a dataset in a Linux VM and created a small sample using dataset.to_json() method.
2. Downloaded the JSON file to my local Windows machine for working and saved in say - r"C:\Users\name\file.json"
3. I am reading the file in my local PyCharm - the location of python file is different than the location of the JSON. 
4. When I read using load_dataset("json",args.input_json), it throws and error from builder.py. 
raise InvalidConfigName(
                    f"Bad characters from black list '{invalid_windows_characters}' found in '{self.name}'. "
                    f"They could create issues when creating a directory for this config on Windows filesystem."
6. When I bring the data to the current directory, it works fine. 

### Steps to reproduce the bug

Steps:
1. Created a dataset in a Linux VM and created a small sample using dataset.to_json() method.
2. Downloaded the JSON file to my local Windows machine for working and saved in say - r"C:\Users\name\file.json"
3. I am reading the file in my local PyCharm - the location of python file is different than the location of the JSON. 
4. When I read using load_dataset("json",args.input_json), it throws and error from builder.py. 
raise InvalidConfigName(
                    f"Bad characters from black list '{invalid_windows_characters}' found in '{self.name}'. "
                    f"They could create issues when creating a directory for this config on Windows filesystem."
6. When I bring the data to the current directory, it works fine. 

### Expected behavior

Should be able to read from a path different than current directory in Windows machine.

### Environment info

datasets version: 2.3.1
python version: 3.8
Windows OS
GitHub issue #5570: load_dataset gives FileNotFoundError on imagenet-1k if license is not accepted on the hub
State: closed | Created: 2023-02-23T16:44:32Z

### Describe the bug

When calling ```load_dataset('imagenet-1k')``` FileNotFoundError is raised, if not logged in and if logged in with huggingface-cli but not having accepted the licence on the hub. There is no error once accepting.

### Steps to reproduce the bug

```
from datasets import load_dataset

imagenet = load_dataset("imagenet-1k", split="train", streaming=True)

FileNotFoundError: Couldn't find a dataset script at /content/imagenet-1k/imagenet-1k.py or any data file in the same directory. Couldn't find 'imagenet-1k' on the Hugging Face Hub either: FileNotFoundError: Dataset 'imagenet-1k' doesn't exist on the Hub
```
tested on a colab notebook.

### Expected behavior

I would expect a specific error indicating that I have to login then accept the dataset licence.

I find this bug very relevant as this code is on a guide on the [Huggingface documentation for Datasets](https://huggingface.co/docs/datasets/about_mapstyle_vs_iterable)

### Environment info

google colab cpu-only instance
GitHub issue #5568: dataset.to_iterable_dataset() loses useful info like dataset features
State: closed | Created: 2023-02-23T13:45:33Z

### Describe the bug

Hello,
I like the new `to_iterable_dataset` feature but I noticed something that seems to be missing.

When using `to_iterable_dataset` to transform your map style dataset into iterable dataset, you lose valuable metadata like the features. 

These metadata are useful if you want to interleave iterable datasets, cast columns etc.

### Steps to reproduce the bug

```python
dataset = load_dataset("lhoestq/demo1")["train"]
print(dataset.features)
# {'id': Value(dtype='string', id=None), 'package_name': Value(dtype='string', id=None), 'review': Value(dtype='string', id=None), 'date': Value(dtype='string', id=None), 'star': Value(dtype='int64', id=None), 'version_id': Value(dtype='int64', id=None)}
dataset = dataset.to_iterable_dataset()
print(dataset.features)
# None
```

### Expected behavior

Keep the relevant information

### Environment info

datasets==2.10.0
GitHub issue #5566: Directly reading parquet files in a s3 bucket from the load_dataset method
State: open | Created: 2023-02-22T22:13:40Z

### Feature request

Right now, we have to read the get the parquet file to the local storage. So having ability to read given the bucket directly address would be benificial 

### Motivation

In a production set up, this feature can help us a lot. So we do not need move training datafiles in between storage.  

### Your contribution

I am willing to help if there's anyway.
GitHub issue #5565: Add writer_batch_size for ArrowBasedBuilder
State: closed | Created: 2023-02-22T15:09:30Z

This way we can control the size of the record_batches/row_groups of arrow/parquet files.

This can be useful for `datasets-server` to keep control of the row groups size which can affect random access performance for audio/image/video datasets

Right now having 1,000 examples per row group cause some image datasets to be pretty slow for random access (e.g. 4seconds for `beans` to get 20 rows)
GitHub issue #5562: Update csv.py
State: closed | Created: 2023-02-22T07:56:10Z

Removed mangle_dup_cols=True from BuilderConfig.

It triggered following deprecation warning:
/usr/local/lib/python3.8/dist-packages/datasets/download/streaming_download_manager.py:776: FutureWarning: the 'mangle_dupe_cols' keyword is deprecated and will be removed in a future version. Please take steps to stop the use of 'mangle_dupe_cols'
  return pd.read_csv(xopen(filepath_or_buffer, "rb", use_auth_token=use_auth_token), **kwargs)

Further documentation of pandas: https://pandas.pydata.org/docs/whatsnew/v1.4.0.html#mangle-dupe-cols-in-read-csv-no-longer-renames-unique-columns-conflicting-with-target-names

At first sight it seems like this flag is resolved internally, it might need some more research.
GitHub issue #5559: Fix map suffix_template
State: closed | Created: 2023-02-21T15:26:26Z

#5455 introduced a small bug that lead `map` to ignore the `suffix_template` argument and not put suffixes to cached files in multiprocessing.

I fixed this and also improved a few things:

- regarding logging: "Loading cached processed dataset" is now logged only once even in multiprocessing (it used to be logged `num_proc` times)
- regarding new_fingerprint: I made sure that the returned dataset satisfies `ds._fingerprint==new_fingerprint` if `new_fingerprint` is passed to `map`
GitHub issue #5556: Use default audio resampling type
State: closed | Created: 2023-02-21T10:45:50Z

...instead of relying on the optional librosa dependency `resampy`.

It was only used for `_decode_non_mp3_file_like` anyway and not for the other ones - removing it fixes consistency between decoding methods (except torchaudio decoding)

Therefore I think it is a better solution than adding `resampy` as a dependency in https://github.com/huggingface/datasets/pull/5554

cc @polinaeterna
GitHub issue #5551: Suggest scikit-learn instead of sklearn
State: closed | Created: 2023-02-20T16:16:57Z

This is kinda unimportant fix but, the suggested `pip install sklearn` does not work.

The current error message if sklearn is not installed:

```
ImportError: To be able to use [dataset name], you need to install the following dependency: sklearn.
Please install it using 'pip install sklearn' for instance.
```