SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: load_dataset error: sndfile library not found

Comment:
Hi @i-am-neo,

The audio support is an extra feature of `datasets` and therefore it must be installed as an additional optional dependency:
```shell
pip install datasets[audio]
```
Additionally, for specific MP3 support (which is not the case for AMI dataset, that contains WAV audio files), there is another third-party dependency on `torchaudio`.

You have all the information in our docs: https://huggingface.co/docs/datasets/audio_process#installation
Issue: load_dataset error: sndfile library not found

Comment:
Hi @i-am-neo, thanks again for your detailed report.

Our `datasets` library support for audio relies on a third-party Python library called `librosa`, which is installed when you do:
```shell
pip install datasets[audio]
```

However, the `librosa` library has a dependency on `soundfile`; and `soundfile` depends on a non-Python package called `sndfile`. 

On Linux (which is your case), this must be installed manually using your operating system package manager, for example:
```shell
sudo apt-get install libsndfile1
```

Please, let me know if this works and if so, I will update our docs with all this information.
Issue: load_dataset error: sndfile library not found

Comment:
@albertvillanova thanks, all good.  The key is ```libsndfile1``` - it may help others to note that in your docs.  I had installed libsndfile previously.
Issue: Audio.encode_example() throws an error when writing example from array

Comment:
Thanks @polinaeterna for reporting this issue.

In relation to the decoding of MP3 audio files without torchaudio, I remember Patrick made some tests and these had quite bad performance. That is why he proposed to support MP3 files only with torchaudio. But yes, nice to give an alternative to non-torchaudio users (with a big warning on performance).
Issue: Audio.encode_example() throws an error when writing example from array

Comment:
>  I remember Patrick made some tests and these had quite bad performance. That is why he proposed to support MP3 files only with torchaudio.

Yeah, I know, but as far as I understand, some users just categorically don't want to have torchaudio in their environment. Anyway, it's just a more or less random example, they can use any library they like following the same logic (I'm just not a big expert in decoding utils so if you can give me some presentation / resources about that I would really appreciate it 🤗)
Issue: Streaming dataset + interleave + DataLoader hangs with multiple workers

Comment:
Hi ! Thanks for reporting, could this be related to https://github.com/huggingface/datasets/issues/3950 ?

Currently streaming datasets only works in single process, but we're working on having in work in distributed setups as well :) (EDIT: done)
Issue: Image column is not decoded in map when using with with_transform

Comment:
Hi! This behavior stems from this line: https://github.com/huggingface/datasets/blob/799b817d97590ddc97cbd38d07469403e030de8c/src/datasets/arrow_dataset.py#L1919
Basically, the `Image`/`Audio` columns are decoded only if the `format_type` attribute is `None` (`set_format`/`with_format` and `set_transform`/`with_transform` assign a non-`None` value to it) and the  `input_columns` param is not specified (see https://github.com/huggingface/datasets/issues/3756). We will remove these limitations soon.
Issue: Improve AutomaticSpeechRecognition task template

Comment:
There is an open PR to do that: #3364. I just haven't had time to finish it...
Issue: Improve AutomaticSpeechRecognition task template

Comment:
> There is an open PR to do that: #3364. I just haven't had time to finish it...

😬 thanks...
Issue: Dataset loads indefinitely after modifying default cache path (~/.cache/huggingface)

Comment:
Hi ! I didn't managed to reproduce the issue. When you kill the process, is there any stacktrace that shows at what point in the code python is hanging ?
Issue: Dataset loads indefinitely after modifying default cache path (~/.cache/huggingface)

Comment:
Hi @lhoestq , I've traced the issue back to file locking. It's similar to this thread, using Lustre filesystem as well. https://github.com/huggingface/datasets/issues/329 . In this case the user was able to modify and add -o flock option while mounting and it solved the problem.  
However in other cases such as mine, we do not have the permissions to modify the commands while mounting. I'm still trying to figure out a workaround. Any ideas how can we use a mounted Lustre filesystem with no flock option?
Issue: Dataset loads indefinitely after modifying default cache path (~/.cache/huggingface)

Comment:
Hi @kelvinAI , I've had this issue on our institution's system which uses Lustre (in addition to our compute nodes being siloed off from external network access). The workaround I made for downloading/loading datasets was to set the `$HFHOME` environment variable to a location on the node's local storage (SSD), effectively a location that gets cleared regularly and sometimes gets used for temporary or cached files which is pretty common, e.g. "scratch" storage. Maybe your sysadmins, if you have them, could point you to subdirectories on a node that aren't linked to the Lustre filesystem. After downloading to scratch I found that the transformers, modules, and metrics cached folders were fine to move to my user drives on the Lustre filesystem but cached datasets that had fingerprints still had some issues with filelock, so it would help to use the function `my_dataset.save_to_disk('path/on/lustre_fs')` and static class function `Dataset.load_from_disk('path/on/lustre_fs')`. In rough steps:

1. Initially download to scratch storage with `ds = datasets.load_dataset(dataset_name)`
2. Call `ds.save_to_disk(my_path_on_lustre)` with a path in your user space on the Lustre filesystem
3. Load datasets with `from datasets import Dataset; new_ds = Dataset.load_from_disk(my_path_on_lustre)`

Obviously this hinges on there existing scratch storage on the nodes you're using. Fingers crossed.
Issue: Dataset loads indefinitely after modifying default cache path (~/.cache/huggingface)

Comment:
Hi @jpmcd , thanks for sharing your experience. For my case, the Lustre filesystem (with more storage space) is the scratch storage like the one you've mentioned. We have a local storage for each user but unfortunately there's not enough space in it to 'cache' huge datasets, hence that is why I tried changing HF_HOME to point to the scratch disk with more space and encountered the flock issue. Unfortunately I'm not aware of any viable solution to this for now so I simply fall back to using torch dataset.
Issue: Dataset loads indefinitely after modifying default cache path (~/.cache/huggingface)

Comment:
@jpmcd your comment saved me from pulling my hair out in frustration. Setting `HF_HOME` to a directory that's not on Lustre works like a charm. ✨
Issue: Local and automatic tests fail

Comment:
Hi ! To be able to run the tests, you need to install all the test dependencies and additional ones with
```
pip install -e .[tests]
pip install -r additional-tests-requirements.txt --no-deps
```

In particular, you probably need to `sacrebleu`. It looks like it wasn't able to instantiate `sacrebleu.TER` properly.
Issue: Infinitely attempting lock

Comment:
Hi ! Thanks for reporting. We're using `py-filelock` as our locking mechanism.

Can you try deleting the .lock file mentioned in the logs and try again ? Make sure that no other process is generating the `cnn_dailymail` dataset.

If it doesn't work, could you try to set up a lock using  the latest version of `py-filelock` and see if it works ?

```
pip install filelock
```
here is a code example from the `py-filelock` documentation that you can try:
```python
from filelock import Timeout, FileLock

lock = FileLock("high_ground.txt.lock")
with lock:
    with open("high_ground.txt", "a") as f:
        f.write("You were the chosen one.")
```
Issue: I can't view HFcallback dataset for ASR Space

Comment:
the dataset viewer is working on this dataset. I imagine the issue is that we would expect to be able to listen to the audio files in the `Please Record Your Voice file` column, right?

maybe @lhoestq or @albertvillanova could help

<img width="1019" alt="Capture d’écran 2022-03-24 à 17 36 20" src="https://user-images.githubusercontent.com/1676121/159966006-57dcf8f7-b65f-4200-ac8c-66859318a8bb.png">
Issue: I can't view HFcallback dataset for ASR Space

Comment:
The structure of the dataset is not supported. Only the CSV file is parsed and the audio files are ignored.

We're working on supporting audio datasets with a specific structure in #3963
Issue: ConnectionError  and SSLError

Comment:
Hi ! You can download the `oscar.py` file from this repository at `/datasets/oscar/oscar.py`.

Then you can load the dataset by passing the local path to `oscar.py` to `load_dataset`:
```python
load_dataset("path/to/oscar.py", "unshuffled_deduplicated_it")
```
Issue: ConnectionError  and SSLError

Comment:
it works,but another error occurs.
```
ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/oscar/1.0/unshuffled/deduplicated/it/it_sha256.txt (SSLError(MaxRetryError("HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /datasets.huggingface.co/oscar/1.0/unshuffled/deduplicated/it/it_sha256.txt (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')))")))
```
I can access `https://s3.amazonaws.com/datasets.huggingface.co/oscar/1.0/unshuffled/deduplicated/it/it_sha256.txt` and `https://aws.amazon.com/cn/s3/` directly, so why it reports a SSLError, should I need tomodify the host file？
Issue: ConnectionError  and SSLError

Comment:
you are so wise!
it report [ConnectionError] in python 3.9.7
and works well in python 3.8.12

I need you help again: how can I specify the path for download files?
the data is too large and my C hardware is not enough
Issue: ConnectionError  and SSLError

Comment:
Cool ! And you can specify the path for download files with to the `cache_dir` parameter:
```python
from datasets import load_dataset
dataset = load_dataset('oscar', 'unshuffled_deduplicated_it', cache_dir='path/to/directory')
Issue: ConnectionError  and SSLError

Comment:
It takes me some days to download data completely, Despise sometimes it occurs again, change py version is feasible way to  avoid this ConnectionEror.
parameter `cache_dir` works well, thanks for your kindness again!
Issue: Cannot preview cnn_dailymail dataset

Comment:
I guess the cache got corrupted due to a previous issue with Google Drive service.

The cache should be regenerated, e.g. by passing `download_mode="force_redownload"`.

CC: @severo
Issue: Cannot preview cnn_dailymail dataset

Comment:
Note that the dataset preview uses its own cache, not `datasets`' cache. So `download_mode="force_redownload"` doesn't help. But yes indeed the cache must be refreshed.

The CNN Dailymail dataste is currently hosted on Google Drive, which is an unreliable host and we've had many issues with it. Unless we found another most reliable host for the data, we will keep running into issues from time to time.

At Hugging Face we're not allowed to host the CNN Dailymail data by ourselves AFAIK
Issue: Cannot preview cnn_dailymail dataset

Comment:
I remove the tag dataset-viewer, since it's more an issue with the hosting on Google Drive
Issue: Cannot preview cnn_dailymail dataset

Comment:
Sounds good. I was looking for another host of this dataset but couldn't find any (yet)
Issue: Cannot preview cnn_dailymail dataset

Comment:
It seems like the issue is with the streaming mode, not with the hosting:

```python
>>> import datasets
>>> dataset = datasets.load_dataset('cnn_dailymail', name="3.0.0", split="train", streaming=True, download_mode="force_redownload")
Downloading builder script: 9.35kB [00:00, 10.2MB/s]
Downloading metadata: 9.50kB [00:00, 12.2MB/s]
>>> len(list(dataset))
0
>>> dataset = datasets.load_dataset('cnn_dailymail', name="3.0.0", split="train", streaming=False)
Reusing dataset cnn_dailymail (/home/slesage/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234)
>>> len(dataset)
287113
```

Note, in particular, that the streaming mode is failing silently, returning 0 row while I would have expected an exception instead. The result is that the dataset viewer shows `No data` instead of a detailed error.

<img width="1511" alt="Capture d’écran 2022-04-12 à 11 50 46" src="https://user-images.githubusercontent.com/1676121/162935341-d50f1e73-d053-41d4-917f-e79708a0ca23.png">
Issue: Cannot preview cnn_dailymail dataset

Comment:
Well this is because the host (Google Drive) returns a document that is not the actual data, but an error page
Issue: Cannot preview cnn_dailymail dataset

Comment:
Yes it definitely should ! I don't have the bandwidth to work on this right now though
Issue: Cannot preview cnn_dailymail dataset

Comment:
Indeed, streaming was not supported: tgz archives were not properly iterated.

I've opened a PR to support streaming.

However, keep in mind that Google Drive will keep generating issues from time to time, like 403,...
Issue: Cannot preview 'indonesian-nlp/eli5_id' dataset

Comment:
Hi @cahya-wirawan, thanks for reporting.

Your dataset is working OK in streaming mode:
```python
In [1]: from datasets import load_dataset
   ...: ds = load_dataset("indonesian-nlp/eli5_id", split="train", streaming=True)
   ...: item = next(iter(ds))
   ...: item
Using custom data configuration indonesian-nlp--eli5_id-9fe728a7e760fb7b

Out[1]: 
{'q_id': '1oy5tc',
 'title': 'dalam sepak bola apa gunanya menyia-nyiakan dua permainan pertama dengan terburu-buru - di tengah - bukan permainan terburu-buru biasa saya mendapatkannya',
 'selftext': '',
 'document': '',
 'subreddit': 'explainlikeimfive',
 'answers': {'a_id': ['ccwtgnz', 'ccwtmho', 'ccwt946', 'ccwvj0u'],
  'text': ['Jaga pertahanan tetap jujur, rasakan operan terburu-buru, buka permainan yang lewat. Pelanggaran yang terlalu satu dimensi akan gagal. Dan mereka yang bergegas ke tengah kadang-kadang dapat dibuka lebar-lebar untuk ukuran yard yang besar.',
   'Jika Anda melempar bola sepanjang waktu, maka pertahanan akan beradaptasi untuk selalu menutupi umpan. Dengan melakukan permainan lari sederhana sesekali, Anda memaksa pertahanan untuk tetap dekat dan menjaga dari lari. Terkadang, pelanggaran dapat membuat pertahanan lengah dengan berpura-pura berlari dan membebaskan penerima mereka. Selain itu, Anda tidak perlu mendapatkan yard besar di setiap permainan. Terkadang, paling baik mendapatkan beberapa yard sekaligus. Selama Anda mendapatkan yang pertama, Anda dalam kondisi yang baik.',
   'Dalam kebanyakan kasus, O-Line seharusnya membuat lubang untuk dilalui kembali. Jika Anda menjalankan terlalu banyak permainan ke luar / melempar, pertahanan akan mengejar. Juga, 2 permainan 5 yard memberi Anda satu set down baru.',
   'Saya Anda tidak suka jenis drama itu, tonton CFL. Kami hanya mendapatkan 3 down sehingga Anda tidak bisa menyia-nyiakannya. Lebih banyak lagi yang lewat.'],
  'score': [3, 2, 2, 2]},
 'title_urls': {'url': []},
 'selftext_urls': {'url': []},
 'answers_urls': {'url': []}}
```
Therefore, it should be properly rendered in the previewer. Let me ping @severo to have a look at it.
Issue: Cannot preview 'indonesian-nlp/eli5_id' dataset

Comment:
Thanks @albertvillanova for checking it. Btw, I have another dataset indonesian-nlp/lfqa_id which has the same issue. However, this dataset is still private, is it the reason why the preview doesn't work?
Issue: TypeError: Couldn't cast array of type for JSONLines dataset

Comment:
Hi @lewtun, thanks for reporting.

It seems that our library fails at inferring the dtype of the columns:
- `milestone`
- `performed_via_github_app` 

(and assigns them `null` dtype).
Issue: Load local dataset error

Comment:
Hi! Instead of @nateraw's `image-folder`, I suggest using the newly released `imagefolder` dataset:
```python
>>> from datasets import load_dataset
>>> data_files={'train': ['/ssd/datasets/imagenet/pytorch/train/**'], 'validation': ['/ssd/datasets/imagenet/pytorch/val/**']}
>>> ds = load_dataset('imagefolder', data_files=data_files, cache_dir='./', task='image-classification')
```


Let us know if that resolves the issue.
Issue: Load local dataset error

Comment:
You can pass `ignore_verifications=True` in `load_dataset` to make it fast (to skip checksum verification). I'll add this tip to the docs.
Issue: Load local dataset error

Comment:
> You can pass `ignore_verifications=True` in `load_dataset` to make it fast (to skip checksum verification). I'll add this tip to the docs.

Thanks！It's worked well.
Issue: Load local dataset error

Comment:
> You can pass `ignore_verifications=True` in `load_dataset` to make it fast (to skip checksum verification). I'll add this tip to the docs.

I find current `load_dataset` loads ImageNet still slowly, even add `ignore_verifications=True`.
First loading, it costs about 20 min in my servers.
```
real	19m23.023s
user	21m18.360s
sys	7m59.080s
```

Second reusing, it costs about 15 min in my servers.
```
real	15m20.735s
user	12m22.979s
sys	5m46.960s
```

I think it's too much slow, is there other method to make it faster?
Issue: Load local dataset error

Comment:
And in transformers the [ViT example](https://github.com/huggingface/transformers/blob/main/examples/pytorch/image-classification/run_image_classification.py), could you make some changes ? Like the `collect_fn`
```python
def collate_fn(examples):
    pixel_values = torch.stack([example["pixel_values"] for example in examples])
    labels = torch.tensor([example["labels"] for example in examples])
    return {"pixel_values": pixel_values, "labels": labels}
```
How to know the keys of example?
Issue: Load local dataset error

Comment:
Loading the image files slowly, is it because the multiple processes load files at the same time?
Issue: Load local dataset error

Comment:
Could you please share the output you get after the second loading? Also, feel free to interrupt (`KeyboardInterrupt`) the process while waiting for it to end and share a traceback to show us where the process hangs.   

> And in transformers the [ViT example](https://github.com/huggingface/transformers/blob/main/examples/pytorch/image-classification/run_image_classification.py), could you make some changes ? Like the `collect_fn`
> 
> ```python
> def collate_fn(examples):
>     pixel_values = torch.stack([example["pixel_values"] for example in examples])
>     labels = torch.tensor([example["labels"] for example in examples])
>     return {"pixel_values": pixel_values, "labels": labels}
> ```
> 
> How to know the keys of example?

What do you mean by "could you make some changes".The `ViT` script doesn't remove unused columns by default, so the keys of an example are equal to the columns of the given dataset.
Issue: Load local dataset error

Comment:
> Could you please share the output you get after the second loading? Also, feel free to interrupt (`KeyboardInterrupt`) the process while waiting for it to end and share a traceback to show us where the process hangs.
> 
> > And in transformers the [ViT example](https://github.com/huggingface/transformers/blob/main/examples/pytorch/image-classification/run_image_classification.py), could you make some changes ? Like the `collect_fn`
> > ```python
> > def collate_fn(examples):
> >     pixel_values = torch.stack([example["pixel_values"] for example in examples])
> >     labels = torch.tensor([example["labels"] for example in examples])
> >     return {"pixel_values": pixel_values, "labels": labels}
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > How to know the keys of example?
> 
> What do you mean by "could you make some changes".The `ViT` script doesn't remove unused columns by default, so the keys of an example are equal to the columns of the given dataset.

Thanks for your reply!

1. I did not record the second output, so I run it again.  
```
(merak) txacs@master:/dat/txacs/test$ time python test.py 
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1281167/1281167 [00:02<00:00, 469497.89it/s]
Resolving data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50001/50001 [00:00<00:00, 70123.73it/s]
Using custom data configuration default-baebca6347576b33
Reusing dataset image_folder (./image_folder/default-baebca6347576b33/0.0.0/ee92df8e96c6907f3c851a987be3fd03d4b93b247e727b69a8e23ac94392a091)
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:10<00:00,  5.37s/it]
Loading cached processed dataset at ./image_folder/default-baebca6347576b33/0.0.0/ee92df8e96c6907f3c851a987be3fd03d4b93b247e727b69a8e23ac94392a091/cache-cd3fbdc025e03f8c.arrow
Loading cached processed dataset at ./image_folder/default-baebca6347576b33/0.0.0/ee92df8e96c6907f3c851a987be3fd03d4b93b247e727b69a8e23ac94392a091/cache-b5a9de701bbdbb2b.arrow
DatasetDict({
    train: Dataset({
        features: ['image', 'labels'],
        num_rows: 1281167
    })
    validation: Dataset({
        features: ['image', 'labels'],
        num_rows: 50000
    })
})

real	10m10.413s
user	9m33.195s
sys	2m47.528s
```
Although it cost less time than the last, but still slowly.

2. Sorry, forgive my poor statement. I solved it, updating to new script 'run_image_classification.py'.
Issue: Load local dataset error

Comment:
Thanks for rerunning the code to record the output. Is it the `"Resolving data files"` part on your machine that takes a long time to complete, or is it `"Loading cached processed dataset at ..."˙`? We plan to speed up the latter by splitting bigger Arrow files into smaller ones, but your dataset doesn't seem that big, so not sure if that's the issue.
Issue: Load local dataset error

Comment:
> Thanks for rerunning the code to record the output. Is it the `"Resolving data files"` part on your machine that takes a long time to complete, or is it `"Loading cached processed dataset at ..."˙`? We plan to speed up the latter by splitting bigger Arrow files into smaller ones, but your dataset doesn't seem that big, so not sure if that's the issue.

Sounds good! The main position, which costs long time, is from program starting to `"Resolving data files"`. I hope you can solve it early, thanks!