ahmedfarazsyk/datasets-github-issues huggingface dataset:
in-place dataset update
Author: speedcell4
State: closed
Labels: name: wontfix

### Motivation

For the circumstance that I creat an empty `Dataset` and keep appending new rows into it, I found that it leads to creating a new dataset at each call. It looks quite memory-consuming. I just wonder if there is any more efficient way to do this.

```python
from datasets import Dataset

ds = Dataset.from_list([])

ds.add_item({'a': [1, 2, 3], 'b': 4})
print(ds)
>>> Dataset({
>>>     features: [],
>>>     num_rows: 0
>>> })

ds = ds.add_item({'a': [1, 2, 3], 'b': 4})
print(ds)
>>> Dataset({
>>>     features: ['a', 'b'],
>>>     num_rows: 1
>>> })
```

### Feature request

Call for in-place dataset update functions, that update the existing `Dataset` in place without creating a new copy. The interface is supposed to keep the same style as PyTorch, such as the in-place version of a `function` is named `function_`. For example, the in-pace version of `add_item`, i.e., `add_item_`, immediately updates the `Dataset`.

```python
from datasets import Dataset

ds = Dataset.from_list([])

ds.add_item({'a': [1, 2, 3], 'b': 4})
print(ds)
>>> Dataset({
>>>     features: [],
>>>     num_rows: 0
>>> })

ds.add_item_({'a': [1, 2, 3], 'b': 4})
print(ds)
>>> Dataset({
>>>     features: ['a', 'b'],
>>>     num_rows: 1
>>> })
```

### Related Functions

* `.map`
* `.filter`
* `.add_item`

Comments: We won't support in-place modifications since `datasets` is based on the Apache Arrow format which doesn't support in-place modifications.

In your case the old dataset is garbage collected pretty quickly so you won't have memory issues.

Note that datasets loaded from disk (memory mapped) are not loaded in memory, and therefore the new dataset actually use the same buffers as the old one.

Thank you for your detailed reply.

> In your case the old dataset is garbage collected pretty quickly so you won't have memory issues.

I understand this, but it still copies the old dataset to create the new one, is this correct? So maybe it is not memory-consuming, but time-consuming?

Indeed, and because of that it is more efficient to add multiple rows at once instead of one by one, using `concatenate_datasets` for example.
.sort() is broken when used after .filter(), only in 2.10.0
Author: MattYoon
State: closed
Labels: name: bug

### Describe the bug

Hi, thank you for your support!

It seems like the addition of multiple key sort (#5502) in 2.10.0 broke the `.sort()` method.

After filtering a dataset with `.filter()`, the `.sort()` seems to refer to the query_table index of the previous unfiltered dataset, resulting in an IndexError.

This only happens with the 2.10.0 release.

### Steps to reproduce the bug

```Python
from datasets import load_dataset

# dataset with length of 1104
ds = load_dataset('glue', 'ax')['test']
ds = ds.filter(lambda x: x['idx'] > 1100)
ds.sort('premise')
print('Done')
```

  File "/home/dongkeun/datasets_test/test.py", line 5, in <module>
    ds.sort('premise')
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 528, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/fingerprint.py", line 511, in wrapper
    out = func(dataset, *args, **kwargs)
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3959, in sort
    sort_table = query_table(
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/formatting/formatting.py", line 588, in query_table
    _check_valid_index_key(key, size)
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/formatting/formatting.py", line 537, in _check_valid_index_key
    _check_valid_index_key(max(key), size=size)
  File "/home/dongkeun/miniconda3/envs/datasets_test/lib/python3.9/site-packages/datasets/formatting/formatting.py", line 531, in _check_valid_index_key
    raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 1103 is out of bounds for size 3

### Expected behavior

It should sort the dataset and print "Done". Which it does on 2.9.0.

### Environment info

- `datasets` version: 2.10.0
- Platform: Linux-5.15.0-41-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- PyArrow version: 11.0.0
- Pandas version: 1.5.3

Comments: Thanks for reporting and thanks @mariosasko for fixing ! We just did a patch release `2.10.1` with the fix
Cache is not transportable
Author: davidgilbertson
State: closed

### Describe the bug

I would like to share cache between two machines (a Windows host machine and a WSL instance).

I run most my code in WSL. I have just run out of space in the virtual drive. Rather than expand the drive size, I plan to move to cache to the host Windows machine, thereby sharing the downloads.

I'm hoping that I can just copy/paste the cache files, but I notice that a lot of the file names start with the path name, e.g. `_home_davidg_.cache_huggingface_datasets_conll2003_default-451...98.lock` where `home/davidg` is where the cache is in WSL. 

This seems to suggest that the cache is not portable/cannot be centralised or shared. Is this the case, or are the files that start with path names not integral to the caching mechanism? Because copying the cache files _seems_ to work, but I'm not filled with confidence that something isn't going to break.

A related issue, when trying to load a dataset that should come from cache (running in WSL, pointing to cache on the Windows host) it seemed to work fine, but it still uses a WSL directory for `.cache\huggingface\modules\datasets_modules`. I see nothing in the docs about this, or how to point it to a different place.

I have asked a related question on the forum: https://discuss.huggingface.co/t/is-datasets-cache-operating-system-agnostic/32656

### Steps to reproduce the bug

View the cache directory in WSL/Windows.

### Expected behavior

Cache can be shared between (virtual) machines and be transportable.

It would be nice to have a simple way to say "Dear Hugging Face packages, please put ALL your cache in `blah/de/blah`" and have all the Hugging Face packages respect that single location.

### Environment info

```
- `datasets` version: 2.9.0
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.10.8
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
- ```

Comments: Hi ! No the cache is not transportable in general. It will work on a shared filesystem if you use the same python environment, but not across machines/os/environments.

In particular, reloading cached datasets does work, but reloading cached processed datasets (e.g. from `map`) may not work. This is because some hashes used by caching are based on pickle dumps of the function you pass to `map`.

Finally you may copy the cache to another machine, but all the `cached-*.arrow` files are unlikely to be reloaded.

OK good to know. Thanks @lhoestq !
Unable to load coyo700M dataset
Author: manuaero
State: closed

### Describe the bug

Seeing this error when downloading https://huggingface.co/datasets/kakaobrain/coyo-700m:
```ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.```


Full stack trace

```Downloading and preparing dataset parquet/kakaobrain--coyo-700m to /root/.cache/huggingface/datasets/kakaobrain___parquet/kakaobrain--coyo-700m-ae729692ae3e0073/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec...
Downloading data files: 100%
1/1 [00:00<00:00, 63.35it/s]
Extracting data files: 100%
1/1 [00:00<00:00, 5.00it/s]
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
[/usr/local/lib/python3.8/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1859                 _time = time.time()
-> 1860                 for _, table in generator:
   1861                     if max_shard_size is not None and writer._num_bytes > max_shard_size:

9 frames
ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
[/usr/local/lib/python3.8/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1890             if isinstance(e, SchemaInferenceError) and e.__context__ is not None:
   1891                 e = e.__context__
-> 1892             raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1893 
   1894         yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset```

### Steps to reproduce the bug

```
from datasets import load_dataset
hf_dataset = load_dataset("kakaobrain/coyo-700m")
```

### Expected behavior

The above commands load the dataset successfully. Or handles exception and continue loading the remainder.

### Environment info

colab. any

Comments: Hi @manuaero 

Thank you for your interest in the COYO dataset.

Our dataset provides the img-url and alt-text in the form of a parquet, so to utilize the coyo dataset you will need to download it directly.

We provide a [guide](https://github.com/kakaobrain/coyo-dataset/blob/main/download/README.md) to download, so check it out.

Thank you.
[DOC] Mistaken docs on set_format
Author: NightMachinery
State: closed
Labels: name: good first issue

### Describe the bug

https://huggingface.co/docs/datasets/v2.10.0/en/package_reference/main_classes#datasets.Dataset.set_format
<img width="700" alt="image" src="https://user-images.githubusercontent.com/36224762/221506973-ae2e3991-60a7-4d4e-99f8-965c6eb61e59.png">

While actually running it will result in:
<img width="1094" alt="image" src="https://user-images.githubusercontent.com/36224762/221507032-007dab82-8781-4319-b21a-e6e4d40d97b3.png">


### Steps to reproduce the bug

_

### Expected behavior

_

### Environment info

- `datasets` version: 2.10.0
- Platform: Linux-5.10.147+-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 9.0.0
- Pandas version: 1.3.5

Comments: Thanks for reporting!
Cannot load `the_pile_openwebtext2`
Author: wjfwzzc
State: closed

### Describe the bug

I met the same bug mentioned in #3053 which is never fixed. Because several `reddit_scores` are larger than `int8` even `int16`. https://huggingface.co/datasets/the_pile_openwebtext2/blob/main/the_pile_openwebtext2.py#L62

### Steps to reproduce the bug

```python3
from datasets import load_dataset
dataset = load_dataset("the_pile_openwebtext2")
```

### Expected behavior

load as normal.

### Environment info

- `datasets` version: 2.10.0
- Platform: Linux-5.4.143.bsk.7-amd64-x86_64-with-glibc2.31
- Python version: 3.9.2
- PyArrow version: 11.0.0
- Pandas version: 1.5.3

Comments: Hi! I've merged a PR to use `int32` instead of `int8` for `reddit_scores`, so it should work now.
I was getting a similar error `pyarrow.lib.ArrowInvalid: Integer value 528 not in range: -128 to 127` - AFAICT, this is because the type specified for `reddit_scores` is `datasets.Sequence(datasets.Value("int8"))`, but the actual values can be well outside the max range for 8-bit integers.
Author: wjfwzzc
State: closed

              I was getting a similar error `pyarrow.lib.ArrowInvalid: Integer value 528 not in range: -128 to 127` - AFAICT, this is because the type specified for `reddit_scores` is `datasets.Sequence(datasets.Value("int8"))`, but the actual values can be well outside the max range for 8-bit integers.

I worked around this by downloading the `the_pile_openwebtext2.py` and editing it to use local files and drop reddit scores as a column (not needed for my purposes).

_Originally posted by @tc-wolf in https://github.com/huggingface/datasets/issues/3053#issuecomment-1281392422_
            

Comments: Duplicated issue.
Metadata for each column
Author: parsa-ra
State: open
Labels: name: enhancement

### Feature request

Being able to put some metadata for each column as a string or any other type.


### Motivation

I will bring the motivation by an example, lets say we are experimenting with embedding produced by some image encoder network, and we want to iterate through a couple of preprocessing and see which one works better in our downstream task, here as workaround right now what I do is the compute the hash of the preprocessing that the images went through as part of the new columns name, it would be nice to attach some kinda meta data in these scenarios to the each columns. metadata

### Your contribution

Maybe we could map another relational like database as the metadata? 

Comments: Hi! Indeed it would be useful to support this. PyArrow natively supports schema-level and column-level metadata, so implementing this should be straightforward. The API I have in mind would work as follows:
```python
col_feature = Value("string", metadata="Some column-level metadata")

features = Features({"col": col_feature}, metadata="Some schema-level metadata")
```

WDYT?

Sorry for the late reply, 
Yes, I think this is the most straight-forward approach with the things that we already have.



@mariosasko Let me know how I can help.

Hi, is this feature to be implemented in the near future? It would be really nice if that would be the case! 

Hi, I also need this feature for tell my customer if any of the feature is encrypted with a certain key.
Datasets 2.10.0 does not reuse the dataset cache
Author: lsb
State: closed

### Describe the bug

download_mode="reuse_dataset_if_exists" will always consider that a dataset doesn't exist.

Specifically, upon losing an internet connection trying to load a dataset for a second time in ten seconds, a connection error results, showing a breakpoint of:

```
File ~/jupyterlab/.direnv/python-3.9.6/lib/python3.9/site-packages/datasets/load.py:1174, in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1165 except Exception as e:  # noqa: catch any exception of hf_hub and consider that the dataset doesn't exist
   1166     if isinstance(
   1167         e,
   1168         (
   (...)
   1172         ),
   1173     ):
-> 1174         raise ConnectionError(f"Couldn't reach '{path}' on the Hub ({type(e).__name__})")
   1175     elif "404" in str(e):
   1176         msg = f"Dataset '{path}' doesn't exist on the Hub"

ConnectionError: Couldn't reach 'lsb/tenk' on the Hub (ConnectionError)
```

This has been around since at least v2.0.


### Steps to reproduce the bug

```
from datasets import load_dataset
import numpy as np

tenk = load_dataset("lsb/tenk") # ten thousand integers
print(np.average(tenk['train']['a'])) # prints 4999.5

### now disconnect your internet

tenk_too = load_dataset("lsb/tenk", download_mode="reuse_dataset_if_exists") 
# Raises ConnectionError: Couldn't reach 'lsb/tenk' on the Hub (ConnectionError)
```

### Expected behavior

I expected that I would be able to reuse the dataset I just downloaded.

### Environment info


- `datasets` version: 2.10.0
- Platform: macOS-13.1-arm64-arm-64bit
- Python version: 3.9.6
- PyArrow version: 7.0.0
- Pandas version: 1.5.2
load_dataset fails for JSON in windows
Author: abinashsahu
State: closed

### Describe the bug

Steps:
1. Created a dataset in a Linux VM and created a small sample using dataset.to_json() method.
2. Downloaded the JSON file to my local Windows machine for working and saved in say - r"C:\Users\name\file.json"
3. I am reading the file in my local PyCharm - the location of python file is different than the location of the JSON. 
4. When I read using load_dataset("json",args.input_json), it throws and error from builder.py. 
raise InvalidConfigName(
                    f"Bad characters from black list '{invalid_windows_characters}' found in '{self.name}'. "
                    f"They could create issues when creating a directory for this config on Windows filesystem."
6. When I bring the data to the current directory, it works fine. 

### Steps to reproduce the bug

Steps:
1. Created a dataset in a Linux VM and created a small sample using dataset.to_json() method.
2. Downloaded the JSON file to my local Windows machine for working and saved in say - r"C:\Users\name\file.json"
3. I am reading the file in my local PyCharm - the location of python file is different than the location of the JSON. 
4. When I read using load_dataset("json",args.input_json), it throws and error from builder.py. 
raise InvalidConfigName(
                    f"Bad characters from black list '{invalid_windows_characters}' found in '{self.name}'. "
                    f"They could create issues when creating a directory for this config on Windows filesystem."
6. When I bring the data to the current directory, it works fine. 

### Expected behavior

Should be able to read from a path different than current directory in Windows machine.

### Environment info

datasets version: 2.3.1
python version: 3.8
Windows OS

Comments: Hi! 

You need to pass an input json file explicitly as `data_files` to `load_dataset` to avoid this error:
```python
 ds = load_dataset("json", data_files=args.input_json)
```



Thanks it worked!
load_dataset gives FileNotFoundError on imagenet-1k if license is not accepted on the hub
Author: buoi
State: closed

### Describe the bug

When calling ```load_dataset('imagenet-1k')``` FileNotFoundError is raised, if not logged in and if logged in with huggingface-cli but not having accepted the licence on the hub. There is no error once accepting.

### Steps to reproduce the bug

```
from datasets import load_dataset

imagenet = load_dataset("imagenet-1k", split="train", streaming=True)

FileNotFoundError: Couldn't find a dataset script at /content/imagenet-1k/imagenet-1k.py or any data file in the same directory. Couldn't find 'imagenet-1k' on the Hugging Face Hub either: FileNotFoundError: Dataset 'imagenet-1k' doesn't exist on the Hub
```
tested on a colab notebook.

### Expected behavior

I would expect a specific error indicating that I have to login then accept the dataset licence.

I find this bug very relevant as this code is on a guide on the [Huggingface documentation for Datasets](https://huggingface.co/docs/datasets/about_mapstyle_vs_iterable)

### Environment info

google colab cpu-only instance

Comments: Hi, thanks for the feedback! Would it help to add a tip or note saying the dataset is gated and you need to accept the license before downloading it?

The error is now more informative:
```
FileNotFoundError: Couldn't find a dataset script at /content/imagenet-1k/imagenet-1k.py or any data file in the same directory. Couldn't find 'imagenet-1k' on the Hugging Face Hub either: FileNotFoundError: Dataset 'imagenet-1k' doesn't exist on the Hub. If the repo is private or gated, make sure to log in with `huggingface-cli login`.
```
dataset.to_iterable_dataset() loses useful info like dataset features
Author: bruno-hays
State: closed
Labels: name: enhancement

name: good first issue

### Describe the bug

Hello,
I like the new `to_iterable_dataset` feature but I noticed something that seems to be missing.

When using `to_iterable_dataset` to transform your map style dataset into iterable dataset, you lose valuable metadata like the features. 

These metadata are useful if you want to interleave iterable datasets, cast columns etc.

### Steps to reproduce the bug

```python
dataset = load_dataset("lhoestq/demo1")["train"]
print(dataset.features)
# {'id': Value(dtype='string', id=None), 'package_name': Value(dtype='string', id=None), 'review': Value(dtype='string', id=None), 'date': Value(dtype='string', id=None), 'star': Value(dtype='int64', id=None), 'version_id': Value(dtype='int64', id=None)}
dataset = dataset.to_iterable_dataset()
print(dataset.features)
# None
```

### Expected behavior

Keep the relevant information

### Environment info

datasets==2.10.0

Comments: Hi ! Oh good catch. I think the features should be passed to `IterableDataset.from_generator()` in `to_iterable_dataset()` indeed.

Setting this as a good first issue if someone would like to contribute, otherwise we can take care of it :)

#self-assign

seems like the feature parameter is missing from `return IterableDataset.from_generator(Dataset._iter_shards, gen_kwargs={"shards": shards})`  hence it defaults to None.
Directly reading parquet files in a s3 bucket from the load_dataset method
Author: shamanez
State: open
Labels: name: duplicate

name: enhancement

### Feature request

Right now, we have to read the get the parquet file to the local storage. So having ability to read given the bucket directly address would be benificial 

### Motivation

In a production set up, this feature can help us a lot. So we do not need move training datafiles in between storage.  

### Your contribution

I am willing to help if there's anyway. 

Comments: Hi ! I think is in the scope of this other issue: to https://github.com/huggingface/datasets/issues/5281
Apply flake8-comprehensions to codebase
Author: Skylion007
State: closed
Labels: name: enhancement

### Feature request

Apply ruff flake8 comprehension checks to codebase. 

### Motivation

This should strictly improve the performance / readability of the codebase by removing unnecessary iteration, function calls, etc. This should generate better Python bytecode which should strictly improve performance.

I already applied this fixes to PyTorch and Sympy with little issue and have opened PRs to diffusers and transformers todo this as well.

### Your contribution

Making a PR.
Downloaded datasets do not cache at $HF_HOME
Author: ErfanMoosaviMonazzah
State: closed

### Describe the bug

In the huggingface course (https://huggingface.co/course/chapter3/2?fw=pt) it said that if we set HF_HOME, downloaded datasets would be cached at specified address but it does not. downloaded models from checkpoint names are downloaded and cached at HF_HOME but this is not the case for datasets, they are still cached at ~/.cache/huggingface/datasets.

### Steps to reproduce the bug

Run the following code
```
from datasets import load_dataset
raw_datasets = load_dataset("glue", "mrpc")
raw_datasets
```

it downloads and store dataset at ~/.cache/huggingface/datasets

### Expected behavior

to cache dataset at HF_HOME.

### Environment info

python 3.10.6
Kubuntu 22.04
HF_HOME located on a separate partition

Comments: Hi ! Can you make sure you set `HF_HOME` before importing `datasets` ?

Then you can print
```python
print(datasets.config.HF_CACHE_HOME)
print(datasets.config.HF_DATASETS_CACHE)
```