SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: audiofolder only creates dataset of 13 rows (files) when the data folder it's reading from has 20,000 mp3 files. 

Comment:
Hi! I'm sorry, I don't want to reveal my entire dataset, but here's a snippet (all of the mp3 files below are some of the ones not being recognized by audiofolder. Also, for another dataset, audiofolder loaded zero mp3 files because "train" was in the name of one of the mp3 files. 
my_dataset
├── data
│   ├── VHA_Innovation_Stories_-_Day_2-123.mp3
│   ├── VHA_Innovation_Stories_-_Day_2-124.mp3
│   ├── ASSOCIATION_OF_GENERAL_PRACTITIONERS_OF_JAMAICA_NEPHROLOGY_CONFERENCE_-_JULY_3,_2022-93.mp3
│   ├── ASSOCIATION_OF_GENERAL_PRACTITIONERS_OF_JAMAICA_NEPHROLOGY_CONFERENCE_-_JULY_3,_2022-94.mp3
│   ├── ASSOCIATION_OF_GENERAL_PRACTITIONERS_OF_JAMAICA_NEPHROLOGY_CONFERENCE_-_JULY_3,_2022-95.mp3
│   ├── Your_Impact\357\274\232_Neurosurgery_equipment-5.mp3
│   └── Your_Impact\357\274\232_Neurosurgery_equipment-6.mp3
└── metadata.csv

Here's a few of the 13 files recognized by the dataset:
British_Heart_Foundation_-_Your_guide_to_a_Coronary_Angiogram,_a_test_for_heart_disease-1.mp3
British_Heart_Foundation_-_Your_guide_to_a_Coronary_Angiogram,_a_test_for_heart_disease-2.mp3
British_Heart_Foundation_-_Your_guide_to_a_Coronary_Angiogram,_a_test_for_heart_disease-3.mp3
IVP_⧸_IVU_test_Procedure_for_Kidneys_intravenous_pyelogram_-_medical_radiology_X-ray_ivp-1.mp3
IVP_⧸_IVU_test_Procedure_for_Kidneys_intravenous_pyelogram_-_medical_radiology_X-ray_ivp-2.mp3
Issue: Add `Dataset.to_list` to the API

Comment:
Hello, I have an interest in this issue.
Is the `Dataset.to_dict` you are describing correct in the code here?

https://github.com/huggingface/datasets/blob/35b789e8f6826b6b5a6b48fcc2416c890a1f326a/src/datasets/arrow_dataset.py#L4633-L4667
Issue: Problems with downloading The Pile

Comment:
Hi! 


You can specify `download_config=DownloadConfig(resume_download=True))` in `load_dataset` to resume the download when re-running the code after the timeout error:
```python
from datasets import load_dataset, DownloadConfig
dataset = load_dataset('the_pile', split='train', cache_dir='F:\datasets', download_config=DownloadConfig(resume_download=True))
```
Issue: Problems with downloading The Pile

Comment:
@mariosasko , I used your suggestion but its not saving anything , just stops and runs from the same point .
below is the script to download and save on disk .

```
from datasets import load_dataset, DownloadConfig


#load the Pile dataset from Hugging Face Datasets
#dataset = load_dataset('the_pile')
dataset = load_dataset('the_pile', split='train', cache_dir='datasets', download_config=DownloadConfig(resume_download=True))


# save each file in the dataset to disk
for i, example in enumerate(dataset['train']):
    filename = f'pile_file_{i}.json'
    with open(filename, 'w') as f:
        f.write(str(example))

print("Finished saving Pile dataset files to disk.")
```
Issue: Problems with downloading The Pile

Comment:
@mariosasko , it shows nothing in dataset folder

```
 du -sh /mnt/nlp/hugging_face/*
20K     /mnt/nlp/hugging_face/datasets
4.0K    /mnt/nlp/hugging_face/download_pile.py
```
Issue: Problems with downloading The Pile

Comment:
Users with slow internet speed are doomed (4MB/s). The dataset downloads fine at minimum speed 10MB/s.

Also, when the train splits were generated and then I removed the downloads folder to save up disk space, it started redownloading the whole dataset. Is there any way to use the already generated splits instead?
Issue: Problems with downloading The Pile

Comment:
@sentialx @mariosasko , anytime on my above script , am I downloading and saving dataset correctly . Please suggest :)
Issue: Authorization error

Comment:
Hi! 

It's better to report this kind of issue in the `huggingface_hub` repo, so if you still haven't resolved it, I suggest you open an issue there.
Issue: Authorization error

Comment:
Yeah,  I solved it. Problem was in osxkeychain.  When I do `hugginface-cli login` it's add token with default account (username)`hg_user` but my repo contain other username.  When I changed username in keychain -  it works now.
Issue: Dataloader getitem not working for DreamboothDatasets

Comment:
Hi! 

> (see example of DreamboothDatasets)


Could you please provide a link to it?  If you are referring to the example in the `diffusers` repo, your issue is unrelated to `datasets` as that example uses `Dataset` from PyTorch to load data.
Issue: in-place dataset update

Comment:
We won't support in-place modifications since `datasets` is based on the Apache Arrow format which doesn't support in-place modifications.

In your case the old dataset is garbage collected pretty quickly so you won't have memory issues.

Note that datasets loaded from disk (memory mapped) are not loaded in memory, and therefore the new dataset actually use the same buffers as the old one.
Issue: in-place dataset update

Comment:
Thank you for your detailed reply.

> In your case the old dataset is garbage collected pretty quickly so you won't have memory issues.

I understand this, but it still copies the old dataset to create the new one, is this correct? So maybe it is not memory-consuming, but time-consuming?
Issue: in-place dataset update

Comment:
Indeed, and because of that it is more efficient to add multiple rows at once instead of one by one, using `concatenate_datasets` for example.
Issue: [TypeError: Couldn't cast array of type] Can only load a subset of the dataset

Comment:
Apparently some JSON objects have a `"labels"` field. Since this field is not present in every object, you must specify all the fields types in the README.md

EDIT: actually specifying the feature types doesn’t solve the issue, it raises an error because “labels” is missing in the data
Issue: [TypeError: Couldn't cast array of type] Can only load a subset of the dataset

Comment:
We've updated the dataset to remove the extra `labels` field from some files, closing this issue. Thanks!
Issue: [TypeError: Couldn't cast array of type] Can only load a subset of the dataset

Comment:
A similar error occurs in the Pile dataset (EleutherAI/the_pile)

Loading the dataset produces the following error.

```
TypeError: Couldn't cast array of type
struct<file: string, id: string>
to
{'id': Value(dtype='string', id=None)}
```
Issue: Error while downloading the xtreme udpos dataset

Comment:
Hi! I cannot reproduce this error on my machine.

The raised error could mean that one of the downloaded files is corrupted. To verify this is not the case, you can run `load_dataset` as follows:
```python
train_dataset = load_dataset('xtreme', 'udpos.English', split="train", cache_dir=args.cache_dir, download_mode="force_redownload", verification_mode="all_checks")
```
Issue: Error while downloading the xtreme udpos dataset

Comment:
Hi! Apologies for the delayed response! I tried the above and it doesn't solve the issue. Actually, the dataset gets downloaded most times, but sometimes this error occurs (at random afaik). Is it possible that there is a server issue for this particular dataset? I am able to download other datasets using the same code on the same machine with no issues :( I get this error now : 
```
Downloading data:  16%|███████████████▌                                                                                   | 55.9M/355M [04:45<25:25, 196kB/s]
Traceback (most recent call last):
  File "/home/skhanuja/Optimal-Resource-Allocation-for-Multilingual-Finetuning/src/train_al.py", line 1107, in <module>
    main()
  File "/home/skhanuja/Optimal-Resource-Allocation-for-Multilingual-Finetuning/src/train_al.py", line 439, in main
    en_dataset = load_dataset("xtreme", "udpos.English", split="train", download_mode="force_redownload", verification_mode="all_checks")
  File "/home/skhanuja/miniconda3/envs/multilingual_ft/lib/python3.10/site-packages/datasets/load.py", line 1782, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/skhanuja/miniconda3/envs/multilingual_ft/lib/python3.10/site-packages/datasets/builder.py", line 872, in download_and_prepare
    self._download_and_prepare(
  File "/home/skhanuja/miniconda3/envs/multilingual_ft/lib/python3.10/site-packages/datasets/builder.py", line 1649, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/skhanuja/miniconda3/envs/multilingual_ft/lib/python3.10/site-packages/datasets/builder.py", line 949, in _download_and_prepare
    verify_checksums(
  File "/home/skhanuja/miniconda3/envs/multilingual_ft/lib/python3.10/site-packages/datasets/utils/info_utils.py", line 62, in verify_checksums
    raise NonMatchingChecksumError(
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://lindat.mff.cuni.cz/repository/xmlui/bitstream/handle/11234/1-3105/ud-treebanks-v2.5.tgz']
Set `verification_mode='no_checks'` to skip checksums verification and ignore this error
```
Issue: Error while downloading the xtreme udpos dataset

Comment:
If this happens randomly, then this means the data file from the error message is not always downloaded correctly. 

The only solution in this scenario is to download the dataset again by passing `download_mode="force_redownload"` to the `load_dataset` call.
Issue: .sort() is broken when used after .filter(), only in 2.10.0

Comment:
Thanks for reporting and thanks @mariosasko for fixing ! We just did a patch release `2.10.1` with the fix
Issue: Cache is not transportable

Comment:
Hi ! No the cache is not transportable in general. It will work on a shared filesystem if you use the same python environment, but not across machines/os/environments.

In particular, reloading cached datasets does work, but reloading cached processed datasets (e.g. from `map`) may not work. This is because some hashes used by caching are based on pickle dumps of the function you pass to `map`.

Finally you may copy the cache to another machine, but all the `cached-*.arrow` files are unlikely to be reloaded.
Issue: Unable to load coyo700M dataset

Comment:
Hi @manuaero 

Thank you for your interest in the COYO dataset.

Our dataset provides the img-url and alt-text in the form of a parquet, so to utilize the coyo dataset you will need to download it directly.

We provide a [guide](https://github.com/kakaobrain/coyo-dataset/blob/main/download/README.md) to download, so check it out.

Thank you.
Issue: Cannot load `the_pile_openwebtext2`

Comment:
Hi! I've merged a PR to use `int32` instead of `int8` for `reddit_scores`, so it should work now.
Issue: Metadata for each column

Comment:
Hi! Indeed it would be useful to support this. PyArrow natively supports schema-level and column-level metadata, so implementing this should be straightforward. The API I have in mind would work as follows:
```python
col_feature = Value("string", metadata="Some column-level metadata")

features = Features({"col": col_feature}, metadata="Some schema-level metadata")
```

WDYT?
Issue: Metadata for each column

Comment:
Sorry for the late reply, 
Yes, I think this is the most straight-forward approach with the things that we already have.
Issue: c4 dataset streaming fails with `FileNotFoundError`

Comment:
This problem now appears again, this time with an underlying HTTP 502 status code:

```
aiohttp.client_exceptions.ClientResponseError: 502, message='Bad Gateway', url=URL('https://huggingface.co/datasets/allenai/c4/resolve/1ddc917116b730e1859edef32896ec5c16be51d0/en/c4-validation.00002-of-00008.json.gz')
```
Issue: c4 dataset streaming fails with `FileNotFoundError`

Comment:
Re-executing a minute later, the underlying cause is an HTTP 403 status code, as reported yesterday:

```
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://cdn-lfs.huggingface.co/datasets/allenai/c4/4bf6b248b0f910dcde2cdf2118d6369d8208c8f9515ec29ab73e531f380b18e2?response-content-disposition=attachment%3B+filename*%3DUTF-8''c4-validation.00002-of-00008.json.gz%3B+filename%3D%22c4-validation.00002-of-00008.json.gz%22%3B&response-content-type=application/gzip&Expires=1677571273&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jZG4tbGZzLmh1Z2dpbmdmYWNlLmNvL2RhdGFzZXRzL2FsbGVuYWkvYzQvNGJmNmIyNDhiMGY5MTBkY2RlMmNkZjIxMThkNjM2OWQ4MjA4YzhmOTUxNWVjMjlhYjczZTUzMWYzODBiMThlMj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSomcmVzcG9uc2UtY29udGVudC10eXBlPWFwcGxpY2F0aW9uJTJGZ3ppcCIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTY3NzU3MTI3M319fV19&Signature=WW42NOKkLuX~xVB1QfbkqzdvGo2AOXpgbF3PjTXy6iKd~ffilr1N9ScPXfvTXqy5yvdhJg1G0xJy1zYtUjGAL8GEx3Av-0vIhpWMGYTM8XKEU5gYA9qt30oVtNph6TkTYSABrsYTaj-hzQL9WCgyapmjvG69ETMh4wj44r2rcbk4T3j0l6l4u76Gh~lyRSll3aK4qycdUwcyL7FECDu~0W1mJIJwKkCrWHhSpHJSshb-0ElwG71pq4eyQ5g2uxHdK6JbRF7loxUpRQQJ1vlk0EHXdw0wTMaQ9tqHy6xcrQd8Ep0Yvx3tUD8MR0vWOcbQKnL6LwPQByc8tkChlpjnig__&Key-Pair-Id=KVTP0A1DKRTAX')
```
Issue: c4 dataset streaming fails with `FileNotFoundError`

Comment:
> It's been resolved again ;)

I'm experiencing the same issue when trying to load this dataset, `FileNotFoundError: https://huggingface.co/datasets/allenai/c4/resolve/1ddc917116b730e1859edef32896ec5c16be51d0/realnewslike/c4-train.00000-of-00512.json.gz`
Issue: load_dataset fails for JSON in windows

Comment:
Hi! 

You need to pass an input json file explicitly as `data_files` to `load_dataset` to avoid this error:
```python
 ds = load_dataset("json", data_files=args.input_json)
```
Issue: load_dataset gives FileNotFoundError on imagenet-1k if license is not accepted on the hub

Comment:
Hi, thanks for the feedback! Would it help to add a tip or note saying the dataset is gated and you need to accept the license before downloading it?
Issue: load_dataset gives FileNotFoundError on imagenet-1k if license is not accepted on the hub

Comment:
The error is now more informative:
```
FileNotFoundError: Couldn't find a dataset script at /content/imagenet-1k/imagenet-1k.py or any data file in the same directory. Couldn't find 'imagenet-1k' on the Hugging Face Hub either: FileNotFoundError: Dataset 'imagenet-1k' doesn't exist on the Hub. If the repo is private or gated, make sure to log in with `huggingface-cli login`.
```
Issue: dataset.to_iterable_dataset() loses useful info like dataset features

Comment:
Hi ! Oh good catch. I think the features should be passed to `IterableDataset.from_generator()` in `to_iterable_dataset()` indeed.

Setting this as a good first issue if someone would like to contribute, otherwise we can take care of it :)
Issue: dataset.to_iterable_dataset() loses useful info like dataset features

Comment:
seems like the feature parameter is missing from `return IterableDataset.from_generator(Dataset._iter_shards, gen_kwargs={"shards": shards})`  hence it defaults to None.
Issue: `.shuffle` throwing error `ValueError: Protocol not known: parent`

Comment:
Hi ! The indices mapping is written in the same cachedirectory as your dataset.

Can you run this to show your current cache directory ?
```python
print(train_dataset.cache_files)
```
Issue: `.shuffle` throwing error `ValueError: Protocol not known: parent`

Comment:
```
[{'filename': '.../train/dataset.arrow'}, {'filename': '.../train/dataset.arrow'}]
```

These are the actual paths where `.hf` files are stored.
Issue: `.shuffle` throwing error `ValueError: Protocol not known: parent`

Comment:
I'm not aware of any `.hf` file ? What are you referring to ?

Also the error says "Protocol unknown: parent". Is there a chance you may have ended up with a path that contains this string `parent://` ?
Issue: `.shuffle` throwing error `ValueError: Protocol not known: parent`

Comment:
I figured out why the issue was occuring but don't know the long-term fix.
The dataset I was trying to shuffle was loaded from a saved file which had `::` delimiter in filename. When I try with the exact same file without `::` in filename, it works as expected.
Quick fix is to not use colons in filename. But if this is expected behaviour, this should be clearly stated in the documentation.
Thanks for help @lhoestq
Issue: Downloaded datasets do not cache at $HF_HOME

Comment:
Hi ! Can you make sure you set `HF_HOME` before importing `datasets` ?

Then you can print
```python
print(datasets.config.HF_CACHE_HOME)
print(datasets.config.HF_DATASETS_CACHE)
```
Issue: the pile datasets url seems to change back

Comment:
Thanks for reporting, @wjfwzzc.

I am transferring this issue to the corresponding dataset on the Hub: https://huggingface.co/datasets/bookcorpusopen/discussions/1
Issue: the pile datasets url seems to change back

Comment:
Thank you. All fixes are done:
- [x] https://huggingface.co/datasets/bookcorpusopen/discussions/2
- [x] https://huggingface.co/datasets/the_pile/discussions/1
- [x] https://huggingface.co/datasets/the_pile_books3/discussions/1
- [x] https://huggingface.co/datasets/the_pile_openwebtext2/discussions/2
- [x] https://huggingface.co/datasets/the_pile_stack_exchange/discussions/2
Issue: Flattening indices in selected datasets is extremely inefficient

Comment:
Running the script above on the branch https://github.com/huggingface/datasets/pull/5542 results in the expected behaviour:
```
Num chunks for original ds: 1
Original ds save/load
save_to_disk -- RAM memory used: 0.671875 MB -- Total time: 0.255265 s
load_from_disk -- RAM memory used: 42.796875 MB -- Total time: 0.014899 s
Num chunks for original ds after reloading: 5000

Num chunks for selected ds: 1
flatten_indices -- RAM memory used: 42.546875 MB -- Total time: 23.735089 s
Num chunks for selected ds after flattening: 5000

Selected ds save/load
save_to_disk -- RAM memory used: 0.0 MB -- Total time: 0.287112 s
load_from_disk -- RAM memory used: 38.84375 MB -- Total time: 0.014772 s
Num chunks for selected ds after reloading: 5000
```
Issue: IndexError: invalid index of a 0-dim tensor. Use `tensor.item()` in Python or `tensor.item<T>()` in C++ to convert a 0-dim tensor to a number

Comment:
Hi! The `set_transform` does not apply a custom formatting transform on a single example but the entire batch, so the fixed version of your transform would look as follows:
```python
from datasets import load_dataset
import torch

dataset = load_dataset("lambdalabs/pokemon-blip-captions", split='train')
def t(batch):
    return {"test": torch.tensor([1] * len(batch[next(iter(batch))]))}
    
dataset.set_transform(t)
d_0 = dataset[0]
```

Still, the formatter's error message should mention that a dict of **sequences** is expected as the returned value (not just a dict) to make debugging easier.
Issue: IndexError: invalid index of a 0-dim tensor. Use `tensor.item()` in Python or `tensor.item<T>()` in C++ to convert a 0-dim tensor to a number

Comment:
> Hi! The `set_transform` does not apply a custom formatting transform on a single example but the entire batch, so the fixed version of your transform would look as follows:
> 
> ```python
> from datasets import load_dataset
> import torch
> 
> dataset = load_dataset("lambdalabs/pokemon-blip-captions", split='train')
> def t(batch):
>     return {"test": torch.tensor([1] * len(batch[next(iter(batch))]))}
>     
> dataset.set_transform(t)
> d_0 = dataset[0]
> ```
> 
> Still, the formatter's error message should mention that a dict of **sequences** is expected as the returned value (not just a dict) to make debugging easier.

ok, will change it according to suggestion. Thanks for the reply!
Issue: load_dataset in seaborn is not working for me. getting this error.

Comment:
Hi! `seaborn`'s `load_dataset` pulls datasets from [here](https://github.com/mwaskom/seaborn-data) and not from our Hub, so this issue is not related to our library in any way and should be reported in their repo instead.
Issue: Increase speed of data files resolution

Comment:
You were right, if `self.dir_cache` is not None in glob, it is exactly the same as what is returned by find, at least for all the tests we have, and some extended evaluation I did across a random sample of about 1000 datasets. 

Thanks for the nice hints, and let me know if this is not exactly what we want here!

see PR: https://github.com/huggingface/datasets/pull/5704
Issue: Increase speed of data files resolution

Comment:
I think we can make the data files resolution (significantly) faster in 2 steps:

1. `glob` calls `find` (which in turn calls `ls`), so we need `find` to be fast, and this can be achieved by fetching all the entries in a single API call and avoiding calls to `ls`. Implementing this for `HfFileSystem.find` (the one in `huggingface_hub`) is on my TO-DO list.
2. caching the repeated `find` calls in `_get_data_files_patterns` when the `data_files` patterns are not provided in `load_dataset`. To address this, we can introduce a `_resolve_single_pattern` function that would accept a filesystem object and a list of regex patterns to resolve. Then we can wrap this filesystem object in `_get_data_files_patterns` with an object that would cache the find calls before resolving the patterns with `_resolve_single_pattern`. (Feel free to suggest a cleaner implementation)

WDYT?
Issue: Increase speed of data files resolution

Comment:
Good idea :) 

For 2:

That would work ! It's also possible to have a FileSystem with a cache on `.find` and use it inside the resolver passed to `_get_data_files_patterns`. Right now they're pretty simple:

```python
# for remote repositories
resolver = partial(_resolve_single_pattern_in_dataset_repository, dataset_info, base_path=base_path)
# for local
resolver = partial(_resolve_single_pattern_locally, base_path)
```
Issue: Increase speed of data files resolution

Comment:
something like this maybe (with Quentin's reimplementation of `HfFilesystem.find`)?

 ```
 @lru_cache(max_size=None)
 def _find(self, path, maxdepth=None, withdirs=False, detail=False, **kwargs):
```

In any case please let me know if I can help in any way!
Issue: Failure to hash function when using .map() 

Comment:
Hi ! `enc` is not hashable:
```python
import tiktoken
from datasets.fingerprint import Hasher

enc = tiktoken.get_encoding("gpt2")
Hasher.hash(enc)
# raises TypeError: cannot pickle 'builtins.CoreBPE' object
```
It happens because it's not picklable, and because of that it's not possible to cache the result of `map`, hence the warning message.

You can find more details about caching here: https://huggingface.co/docs/datasets/about_cache

You can also provide your own unique hash in `map` if you want, with the `new_fingerprint` argument.
Or disable caching using
```python
import datasets
datasets.disable_caching()
```
Issue: Failure to hash function when using .map() 

Comment:
@lhoestq Thank you for the explanation and advice. Will relay all of this to the repo where this (non)issue arose. 

Great job with huggingface!
Issue: Failure to hash function when using .map() 

Comment:
Just a heads up that when I'm trying to use TikToken along with the a given Dataset `.map()` method, I am still met with the following error :

```
  File "/opt/conda/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/opt/conda/lib/python3.8/pickle.py", line 578, in save
    rv = reduce(self.proto)
TypeError: cannot pickle 'builtins.CoreBPE' object
```

My current environment is running datasets v2.10.0.
Issue: Failure to hash function when using .map() 

Comment:
@lhoestq @edhenry I am on datasets version `'2.12.0'. I see the same `TypeError: cannot pickle 'builtins.CoreBPE' object` that others are seeing.
Issue: Failure to hash function when using .map() 

Comment:
I am able to reproduce this on datasets 2.14.2. The `datasets.disable_caching()` doesn't work around it.

@lhoestq - you might want to reopen this issue. Because of this issue folks won't be able run Karpathy's NanoGPT :(.
Issue: Failure to hash function when using .map() 

Comment:
update： temporarily solved the problem by setting
```
--preprocess_num_workers  1
```

-------------
I  have met the same problem, here is my env:
```
datasets                      2.14.4
transformers                  4.31.0
tiktoken                      0.4.0
torch                         1.13.1
```