SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Error loading file audio when downloading the Common Voice dataset directly from the Hub

Comment:
I think the issue comes from the fact that you set `batched=False` while `map_to_pred` still returns a list of strings for "transcription". You can fix it by adding `[0]` at the end of this line to get the string:
```python
batch["transcription"] = processor.batch_decode(gen_tokens, skip_special_tokens=True)[0]
```
Issue: Error loading file audio when downloading the Common Voice dataset directly from the Hub

Comment:
We no longer use `torchaudio` for decoding MP3 files, and the problem with model cards has been addressed, so I'm closing this issue.
Issue: NonMatchingChecksumError on Spider dataset 

Comment:
Hi @kolk, thanks for reporting.

Indeed, Google Drive service recently changed their service and we had to add a fix to our library to cope with that change:
- #3787 

We just made patch release last week: 1.18.4 https://github.com/huggingface/datasets/releases/tag/1.18.4

Please, feel free to update your local `datasets` version, so that you get the fix:
```shell
pip install -U datasets
```
Issue: CONLL2003 Dataset not available

Comment:
Thanks for reporting, @omarespejel.

I'm sorry but I can't reproduce the issue: the loading of the dataset works perfecto for me and I can reach the data URL: https://data.deepai.org/conll2003.zip

Might it be due to a temporary problem in the data owner site (https://data.deepai.org/) that is fixed now?
Could you please try loading the dataset again and tell if the problem persists?
Issue: CONLL2003 Dataset not available

Comment:
I am getting the same issue. I use google colab with CPU.
The code I used is exactly the same as described above.
```
from datasets import load_dataset
dataset = load_dataset("conll2003")
```

The produced error:
![image](https://github.com/huggingface/datasets/assets/9371628/d87f7fb0-ef58-4755-abb5-f8f92c51fe02)

Note: This error is different from what was initially described in this thread. This is because I use CPU. When I use GPU I reproduce the same initial error of the thread.

Moreover, I receive the following warning:
```
WARNING:urllib3.connection:Certificate did not match expected hostname: data.deepai.org. Certificate: {'subject': ((('commonName', '*.b-cdn.net'),),), 'issuer': ((('countryName', 'GB'),), (('stateOrProvinceName', 'Greater Manchester'),), (('localityName', 'Salford'),), (('organizationName', 'Sectigo Limited'),), (('commonName', 'Sectigo RSA Domain Validation Secure Server CA'),)), 'version': 3, 'serialNumber': 'DDED48B13E1EA03983E833AB2C35EF07', 'notBefore': 'Nov  7 00:00:00 2022 GMT', 'notAfter': 'Nov 11 23:59:59 2023 GMT', 'subjectAltName': (('DNS', '*.b-cdn.net'), ('DNS', 'b-cdn.net')), 'OCSP': ('http://ocsp.sectigo.com/',), 'caIssuers': ('http://crt.sectigo.com/SectigoRSADomainValidationSecureServerCA.crt',)}
Downloading and preparing dataset conll2003/conll2003 to /root/.cache/huggingface/datasets/conll2003/conll2003/1.0.0/9a4d16a94f8674ba3466315300359b0acd891b68b6c8743ddf60b9c702adce98...
WARNING:urllib3.connection:Certificate did not match expected hostname: data.deepai.org. Certificate: {'subject': ((('commonName', '*.b-cdn.net'),),), 'issuer': ((('countryName', 'GB'),), (('stateOrProvinceName', 'Greater Manchester'),), (('localityName', 'Salford'),), (('organizationName', 'Sectigo Limited'),), (('commonName', 'Sectigo RSA Domain Validation Secure Server CA'),)), 'version': 3, 'serialNumber': 'DDED48B13E1EA03983E833AB2C35EF07', 'notBefore': 'Nov  7 00:00:00 2022 GMT', 'notAfter': 'Nov 11 23:59:59 2023 GMT', 'subjectAltName': (('DNS', '*.b-cdn.net'), ('DNS', 'b-cdn.net')), 'OCSP': ('http://ocsp.sectigo.com/',), 'caIssuers': ('http://crt.sectigo.com/SectigoRSADomainValidationSecureServerCA.crt',)}
```
Issue: Can't import datasets: partially initialized module 'fsspec' has no attribute 'utils'

Comment:
Update: `"python3 -c "from from datasets import Dataset, DatasetDict"` works, but not if I import without the `python3 -c`
Issue: Can't import datasets: partially initialized module 'fsspec' has no attribute 'utils'

Comment:
Hi @arunasank, thanks for reporting.

It seems that this can be caused because you are using an old version of `fsspec`: the reason why it works if you run `python3` seems to be that `python3` runs in a Python virtual env (with an updated version of `fsspec`); whereas the error arises when you run the import from other Python virtual env (with an old version of `fsspec`).

In order to fix this, you should update `fsspec` from within the "problematic" Python virtual env:
```
pip install -U "fsspec[http]>=2021.05.0"
Issue: Can't import datasets: partially initialized module 'fsspec' has no attribute 'utils'

Comment:
from lightgbm import LGBMModel,LGBMClassifier, plot_importance
after importing lib getting (partially initialized module 'fsspec' has no attribute 'utils' (most likely due to a circular import) error, can help me
Issue: Can't import datasets: partially initialized module 'fsspec' has no attribute 'utils'

Comment:
@deepakmahtha I think you are not using `datasets`: this is the GitHub repository of Hugging Face Datasets.

If you are using `lightgbm`, you should report the issue to their repository instead.

Anyway, we have proposed a possible fix just in a comment above: to update fsspec.
https://github.com/huggingface/datasets/issues/3902#issuecomment-1066517824
Issue: Dataset viewer issue for IndicParaphrase- the preview doesn't show

Comment:
It seems to have been fixed:

<img width="1534" alt="Capture d’écran 2022-04-12 à 14 10 07" src="https://user-images.githubusercontent.com/1676121/162959599-6b7fef7c-8411-4e03-8f00-90040a658079.png">
Issue: Missing google file for `multi_news` dataset

Comment:
`datasets` 1.18.4 fixes the issue when you load the dataset with `load_dataset`.

When loading in streaming mode, the fix is indeed on https://github.com/huggingface/datasets/pull/3843 which will be merged soon :)
Issue: Missing google file for `multi_news` dataset

Comment:
That is. The PR #3843 was just opened a bit later we had made our 1.18.4 patch release...
Once merged, that will fix this issue.
Issue: Missing google file for `multi_news` dataset

Comment:
OK. Should fix the viewer for 50 datasets

<img width="148" alt="Capture d’écran 2022-03-14 à 11 51 02" src="https://user-images.githubusercontent.com/1676121/158157853-6c544a47-2d6d-4ac4-964a-6f10951ec36b.png">
Issue: Cannot load beans dataset (Couldn't reach the dataset)

Comment:
Hi ! A pull request is open to fix the dataset, we'll release a patch soon with a new release of `datasets` :)
Issue: IterableDataset columns and feature types

Comment:
@lhoestq so in order to address what’s not completed in this issue, do you think it makes sense to add a param `features` to `IterableDataset.map` so that the output features right after the `map` are defined there?
Issue: IterableDataset columns and feature types

Comment:
@lhoestq cool then if you agree I can work on that! I’ll also update the docs accordingly once done, thanks!
Issue: IterableDataset columns and feature types

Comment:
I've already started with a PR as a draft @lhoestq, should we also try to look for a way to explicitly request pre-fetching right after a map operation is applied, so that the features are inferred if the user says explicitly so? Thanks!
Issue: IterableDataset columns and feature types

Comment:
> should we also try to look for a way to explicitly request pre-fetching right after a map operation is applied, so that the features are inferred if the user says explicitly so?

Right now one can use `ds = ds._resolve_features()` do to so. It can be used after `map` or `load_dataset` if the features are not known. Maybe we can make this method public ?
Issue: How to use Image folder

Comment:
Hi @INF800,

Please note that the `imagefolder` feature enhancement was just recently merged to our master branch (https://github.com/huggingface/datasets/commit/207be676bffe9d164740a41a883af6125edef135), but has not yet been released.

We are planning to make the 2.0 release of our library in the coming days and then that feature will be available by updating your `datasets` library from PyPI.

In the meantime, you can incorporate that feature if you install our library from our GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```

Then:
```python
In [1]: from datasets import load_dataset
        ds = load_dataset("imagefolder", data_files="https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip", split="train")
Using custom data configuration default-7eb4e80d960deb18
Downloading and preparing dataset image_folder/default to .../.cache/huggingface/datasets/image_folder/default-7eb4e80d960deb18/0.0.0/8de8dc6d68ce3c81cc102b93cc82ede27162b5d30cd003094f935942c8294f60...
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 690.19it/s]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 852.85it/s]
Dataset image_folder downloaded and prepared to .../.cache/huggingface/datasets/image_folder/default-7eb4e80d960deb18/0.0.0/8de8dc6d68ce3c81cc102b93cc82ede27162b5d30cd003094f935942c8294f60. Subsequent calls will reuse this data.

In [2]: ds
Out[2]: 
Dataset({
    features: ['image', 'label'],
    num_rows: 25000
})
```
Issue: How to use Image folder

Comment:
Hey @albertvillanova. Does this load entire dataset in memory? Because I am facing huge trouble with loading very big datasets (OOM errors)
Issue: How to use Image folder

Comment:
Can you provide the error stack trace? The loader only stores the `data_files` dict, which can get big after globbing. Then, the OOM error would mean you don't have enough memory to keep all the paths to the image files. You can circumvent this by generating an archive and loading the dataset from there. Maybe we can optimize the globbing part in our data files resolution at some point, cc @lhoestq for visibility.
Issue: How to use Image folder

Comment:
Hey, memory error is resolved. It was fluke.

But there is another issue. Currently `load_dataset("imagefolder", data_dir="./path/to/train",)` takes only `train` as arg to `split` parameter.

I am creating vaildation dataset using

```
ds_valid = datasets.DatasetDict(valid=load_dataset("imagefolder", data_dir="./path/to/valid",)['train'])
```
Issue: How to use Image folder

Comment:
`data_dir="path/to/folder"` is a shorthand syntax fox `data_files={"train": "path/to/folder/**"}`, so use `data_files` in that case instead:
```python
ds = load_dataset("imagefolder", data_files={"train": "path/to/train/**", "test": "path/to/test/**", "valid": "path/to/valid/**"})
```
Issue: How to use Image folder

Comment:
And there was another issue. I loaded black and white images (jpeg file). Using load dataset. It reads it as PIL jpeg data format. But instead of converting it into 3 channel tensor, input to collator function is coming as a single channel tensor.
Issue: How to use Image folder

Comment:
We don't apply any additional preprocessing on top of `PIL.Image.open(image_file)`, so you need to do the conversion yourself:

```python
def to_rgb(batch):
    batch["image"] = [img.convert("RGB") for img in batch["image"]]
    return batch

ds_rgb = ds.map(to_rgb, batched=True)
```

Please use our Forum for questions of this kind in the future.
Issue: HTTP error 504 Server Error: Gateway Time-out

Comment:
yes but is there any way you could try pushing with `git` command line directly instead of `push_to_hub`?
Issue: HTTP error 504 Server Error: Gateway Time-out

Comment:
Okay. I didnt saved the dataset to my local machine. So, I processed the dataset and pushed it directly to the hub. I think I should try saving those dataset to my local machine by `save_to_disk` and then push it with git command line
Issue: HTTP error 504 Server Error: Gateway Time-out

Comment:
`push_to_hub` is the preferred way of uploading a dataset to the Hub, which can then be reloaded with `load_dataset`. Feel free to try again and see if the server is working as expected now. Maybe we can add a retry mechanism in the meantime to workaround 504 errors.

Regarding `save_to_disk`, this must only be used for local serialization (because it's uncompressed and compatible with memory-mapping). If you upload a dataset saved with `save_to_disk` to the Hub, then to reload it you will have to download/clone the repository locally by yourself and use `load_from_disk`.
Issue: Making the Hub the place for datasets in Portuguese 

Comment:
Hi @omarespejel! I think the philosophy for `datasets` issues is to create concrete issues with proposals to add a specific, individual dataset rather than umbrella issues for things such as datasets for a language, since we could end up with hundreds of issues (one per language). I see NILC - USP has many datasets, I would suggest to either create an issue for their datasets, or even better, we are trying to push to upload datasets as community datasets instead of adding them to the core library as guided in https://huggingface.co/docs/datasets/share. That would have the additional benefit that the dataset would live under the NILC organization.

@lhoestq correct me if I'm wrong please 😄
Issue: big_patent cased version

Comment:
To follow up on this: the cased and uncased versions actually contain different content, and the cased one is easier since it contains a Summary of the Invention in the input.

See the paper describing the issue here:
https://aclanthology.org/2022.gem-1.34/
Issue: big_patent cased version

Comment:
Thanks for proposing the addition of the cased version of this dataset and for pinging again recently.

I have just merged a PR that adds the cased version: https://huggingface.co/datasets/big_patent/discussions/3

The cased version (2.1.2) is the default one:
```python
ds = load_dataset("big_patent", "all")
```

To use the 1.0.0 version (lower cased tokenized words), pass both parameters `codes` and `version`:
```python
ds = load_dataset("big_patent", codes="all", version="1.0.0")
```

Closed by: https://huggingface.co/datasets/big_patent/discussions/3
Issue: Unable to dowload big_patent (FileNotFoundError)

Comment:
Hi @slvcsl, thanks for reporting.

Yesterday we just made a patch release of our `datasets` library that fixes this issue: version 1.18.4.
https://pypi.org/project/datasets/#history

Please, feel free to update `datasets` library to the latest version: 
```shell
pip install -U datasets
```
And then you should force redownload of the data file to update your local cache: 
```python
ds = load_dataset("big_patent", "g", split="validation", download_mode="force_redownload")
```
- Note that before the fix, you just downloaded and cached the Google Drive virus scan warning page, instead of the data file

This issue was already reported 
- #3784

and its root cause is a change in the Google Drive service. See:
- #3786 

We already fixed it. See:
- #3787
Issue: Order of dataset changes due to glob.glob.

Comment:
I agree using `glob.glob` alone is bad practice because it's not deterministic. Using `sorted` is a nice solution.

Note that the `xglob` function you are referring to in the `streaming_download_manager.py` code just extends `glob.glob` for URLs - we don't change its behavior. That's why it has no `sorted()`
Issue: Bad error message when loading private dataset

Comment:
We raise the error “ FileNotFoundError: can’t find the dataset” mainly to follow best practice in security (otherwise users could be able to guess what private repositories users/orgs may have)

We can indeed reformulate this and add the "If this is a private repository,..." part !
Issue: load only England English dataset from common voice english dataset

Comment:
Hi @amanjaiswal777,

First note that the dataset you are trying to load is deprecated: it was the Common Voice dataset release as of Dec 2020.

Currently, Common Voice dataset releases are directly hosted on the Hub, under the Mozilla Foundation organization: https://huggingface.co/mozilla-foundation

For example, to get their latest Common Voice relase (8.0):
- Go to the dataset page and request access permission (Mozilla Foundation requires this for people willing to use their datasets): https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0
- Looking at the dataset card, you can check that data instances have, among other fields, the ones you are interested in: "accent", "age",... 
- Then you can load their "en" language dataset as usual, besides passing your authentication token (more info on auth token here: https://huggingface.co/docs/hub/security)
  ```python
  from datasets import load_dataset
  ds_en = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token=True)
  ```
- Finally, you can filter only the data instances you are interested in (more info on `filter` here: https://huggingface.co/docs/datasets/process#select-and-filter):
  ```python
  ds_england_en = ds_en.filter(lambda item: item["accent"] == "England English")
  ```

Feel free to reopen this issue if you need further assistance.
Issue: Load audio dataset error

Comment:
Hi @lemoner20, thanks for reporting.

I'm sorry but I cannot reproduce your problem:
```python
In [1]: from datasets import load_dataset, load_metric, Audio
   ...: raw_datasets = load_dataset("superb", "ks", split="train")
   ...: print(raw_datasets[0]["audio"])
Downloading builder script: 30.2kB [00:00, 13.0MB/s]                                                                                                                                                       
Downloading metadata: 38.0kB [00:00, 16.6MB/s]                                                                                                                                                             
Downloading and preparing dataset superb/ks (download: 1.45 GiB, generated: 9.64 MiB, post-processed: Unknown size, total: 1.46 GiB) to .../.cache/huggingface/datasets/superb/ks/1.9.0/fc1f59e1fa54262dfb42de99c326a806ef7de1263ece177b59359a1a3354a9c9...
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.49G/1.49G [00:37<00:00, 39.3MB/s]
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 71.3M/71.3M [00:01<00:00, 36.1MB/s]
Downloading data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:41<00:00, 20.67s/it]
Extracting data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:28<00:00, 14.24s/it]
Dataset superb downloaded and prepared to .../.cache/huggingface/datasets/superb/ks/1.9.0/fc1f59e1fa54262dfb42de99c326a806ef7de1263ece177b59359a1a3354a9c9. Subsequent calls will reuse this data.
{'path': '.../.cache/huggingface/datasets/downloads/extracted/8571921d3088b48f58f75b2e514815033e1ffbd06aa63fd4603691ac9f1c119f/_background_noise_/doing_the_dishes.wav', 'array': array([ 0.        ,  0.        ,  0.        , ..., -0.00592041,
       -0.00405884, -0.00253296], dtype=float32), 'sampling_rate': 16000}
``` 

Which version of `datasets` are you using? Could you please fill in the environment info requested in the bug report template? You can run the command `datasets-cli env` and copy-and-paste its output below
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:
Issue: Load audio dataset error

Comment:
@albertvillanova Thanks for your reply. The environment info below

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-4.19.91-007.ali4000.alios7.x86_64-x86_64-with-debian-buster-sid
- Python version: 3.6.12
- PyArrow version: 6.0.1
Issue: Load audio dataset error

Comment:
Thanks @lemoner20,

I cannot reproduce your issue in datasets version 1.18.3 either.

Maybe redownloading the data file may work if you had already cached this dataset previously. Could you please try passing "force_redownload"?
```python
raw_datasets = load_dataset("superb", "ks", split="train", download_mode="force_redownload")
Issue: Load audio dataset error

Comment:
@albertvillanova, you can actually reproduce the error if you reach the cell `common_voice_train[0]["path"]` of this [notebook](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_Tune_XLSR_Wav2Vec2_on_Turkish_ASR_with_%F0%9F%A4%97_Transformers.ipynb#scrollTo=_0kRndSvqaKk). Error gets solved after updating the versions of the libraries used in there.
Issue: Load audio dataset error

Comment:
@jvel07, thanks for reporting and finding a solution.

Maybe we could tell @patrickvonplaten about the version pinning issue in his notebook.
Issue: NonMatchingChecksumError when checksum is None

Comment:
Hi @jxmorris12, thanks for reporting.

The objective of `verify_checksums` is to check that both checksums are equal. Therefore if one is None and the other is non-None, they are not equal, and the function accordingly raises a NonMatchingChecksumError. That behavior is expected.

The question is: how did you generate the expected checksum? Normally, it should not be None. To properly generate it (it is contained in the `dataset_infos.json` file), you should have runned: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md
```shell
datasets-cli test <your-dataset-folder> --save_infos --all_configs
```

On the other hand, you should take into account that the generation of this file is NOT mandatory for personal/community datasets (we only require it for "canonical" datasets, i.e., datasets added to our library GitHub repository: https://github.com/huggingface/datasets/tree/master/datasets). Therefore, other option would be just to delete the `dataset_infos.json` file. If that file is not present, the function `verify_checksums` is not executed.

Finally, you can circumvent the `verify_checksums` function by passing `ignore_verifications=True` to `load_dataset`:
```python
load_dataset(..., ignore_verifications=True)
```
Issue: NonMatchingChecksumError when checksum is None

Comment:
Thanks @albertvillanova!

That's fine. I did run that command when I was adding a new dataset. Maybe because the command crashed in the middle, the checksum wasn't stored properly. I don't know where the bug is happening. But either (i) `verify_checksums` should properly handle this edge case, where the passed checksum is None or (ii) the `datasets-cli test` shouldn't generate a corrupted dataset_infos.json file.

Just a more high-level thing, I was trying to follow the instructions for adding a dataset in the CONTRIBUTING.md, so if running that command isn't even necessary, that should probably be mentioned in the document, right? But that's somewhat of a moot point, since something isn't working quite right internally if I was able to get into this corrupted state in the first place, just by following those instructions.
Issue: NonMatchingChecksumError when checksum is None

Comment:
Hi @jxmorris12,

Definitely, your `dataset_infos.json` was corrupted (and wrongly contains expected None checksum). 

While we further investigate how this can happen and fix it, feel free to delete your `dataset_infos.json` file and recreate it with:
```shell
datasets-cli test <your-dataset-folder> --save_infos --all_configs
```

Also note that `verify_checksum` is working as expected: if it receives a None and and a non-None checksums as input pair, it must raise an exception: they are not equal. That is not a bug.