Checksums didn't match for dataset source
Opened by rafikg (closed)

## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No


Comments:
Same issue with `dataset = load_dataset("dbpedia_14")`
```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbQ2Vic1kxMmZZQ1k']

I think this is a side-effect of #3787. The checksums won't match because the URLs have changed. @rafikg @Y0mingZhang, while this is fixed, maybe you can load the datasets as such:

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]", ignore_verifications=True)`
`dataset = load_dataset("dbpedia_14", ignore_verifications=True)`

This will, most probably, skip the verifications and integrity checks listed [here](https://huggingface.co/docs/datasets/loading_datasets.html#integrity-verifications)

Hi! Installing the `datasets` package from master (`pip install git+https://github.com/huggingface/datasets.git`) and then redownloading the datasets with `download_mode` set to `force_redownload` (e.g. `dataset = load_dataset("dbpedia_14", download_mode="force_redownload")`) should fix the issue.

Hi @rafikg and @Y0mingZhang, thanks for reporting.

Indeed it seems that Google Drive changed their way to access their data files. We have recently handled that change:
- #3787

but it will be accessible to users only in our next release of the `datasets` version.
- Note that our latest release (version 1.18.3) was made before this fix: https://github.com/huggingface/datasets/releases/tag/1.18.3

In the meantime, as @mariosasko explained, you can incorporate this "fix" by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, you should force the redownload of the data (before the fix, you are just downloading/caching the virus scan warning page, instead of the data file):
```shell
data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]", download_mode="force_redownload")

@albertvillanova  by running:
```
pip install git+https://github.com/huggingface/datasets#egg=datasets
data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]", download_mode="force_redownload", ignore_verifications=True)
```

I had a pickle error **UnpicklingError: invalid load key, '<'** in this part of code  both `locally and on google colab`:

```
"""Yields examples."""
with open(filepath, "rb") as f:
         data = pickle.load(f)
for id_, row in enumerate(data.items()):
         yield id_, {"url": row[0], "article": self._process_article(row[1])}
```


This issue impacts many more datasets than the ones mention in this thread.  Can we post # of downloads for each dataset by day (by successes and failures)?  If so, it should be obvious which ones are failing.

I can see this problem too in xcopa, unfortunately installing the latest master (1.18.4.dev0) doesn't work, @albertvillanova .

```
from datasets import load_dataset
dataset = load_dataset("xcopa", "it")
```

Throws

```
in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42 

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/cambridgeltl/xcopa/archive/master.zip']
```

Hi @rafikg, I think that is another different issue. Let me check it... 

I guess maybe you are using a different Python version that the one the dataset owner used to create the pickle file...

@kwchurch the datasets impacted for this specific issue are the ones which are hosted at Google Drive.

@afcruzs-ms I think your issue is a different one, because that dataset is not hosted at Google Drive. Would you mind open another issue for that other problem, please? Thanks! :)

@albertvillanova just to let you know that I tried it locally and on colab and it is the same error

There are many many datasets on HugggingFace that are receiving this checksum error.  Some of these datasets are very popular.  There must be a way to track these errors, or to do regression testing.  We don't want to catch each of these errors on each dataset, one at a time.

@rafikg I am sorry, but I can't reproduce your issue. For me it works OK for all languages. See: https://colab.research.google.com/drive/1yIcLw1it118-TYE3ZlFmV7gJcsF6UCsH?usp=sharing

@kwchurch the PR #3787 fixes this issue (generated by a change in Google Drive service) for ALL datasets with this issue. Once we make our next library release (in a couple of days), the fix will be accessible to all users that update our library from PyPI.

By the way, @rafikg, I discovered the URL for Spanish was wrong. I've created a PR to fix it:
- #3806 

I have the same problem with "wider_face" dataset. It seems that "load_dataset" function can not download the dataset from google drive.


still getting this issue with datasets==2.2.2 for 
dataset_fever_original_dev = load_dataset('fever', "v1.0", split="labelled_dev")
(this one seems to be hosted by aws though)

update: also tried to install from source to get the latest 2.2.3.dev0, but still get the error below (and also force-redownloaded)

update2: Seems like this issues is linked to a change in the links in the specific fever datasets: https://fever.ai/
"28/04/2022
Dataset download URLs have changed
Download URLs for shared task data for FEVER, FEVER2.0 and FEVEROUS have been updated. New URLS begin with https://fever.ai/download/[task name]/[filename]. All resource pages have been updated with the new URLs. Previous dataset URLs may not work and should be updated if you require these in your scripts. "

=> I don't know how to update the links for HF datasets - would be great if someone could update them :) 

```

Downloading and preparing dataset fever/v1.0 (download: 42.78 MiB, generated: 38.39 MiB, post-processed: Unknown size, total: 81.17 MiB) to /root/.cache/huggingface/datasets/fever/v1.0/1.0.0/956b0a9c4b05e126fd956be73e09da5710992b5c85c30f0e5e1c500bc6051d0a...

Downloading data files: 100%
6/6 [00:07<00:00, 1.21s/it]
Downloading data:
278/? [00:00<00:00, 2.34kB/s]
Downloading data:
278/? [00:00<00:00, 1.53kB/s]
Downloading data:
278/? [00:00<00:00, 7.43kB/s]
Downloading data:
278/? [00:00<00:00, 5.54kB/s]
Downloading data:
278/? [00:00<00:00, 6.19kB/s]
Downloading data:
278/? [00:00<00:00, 7.51kB/s]
Extracting data files: 100%
6/6 [00:00<00:00, 108.05it/s]

---------------------------------------------------------------------------

NonMatchingChecksumError                  Traceback (most recent call last)

[<ipython-input-20-92ec5c728ecf>](https://localhost:8080/#) in <module>()
     27 # get labels for fever-nli-dev from original fever - only works for dev
     28 # "(The labels for both dev and test are hidden but you can retrieve the label for dev using the cid and the original FEVER data.)"" https://github.com/easonnie/combine-FEVER-NSMN/blob/master/other_resources/nli_fever.md
---> 29 dataset_fever_original_dev = load_dataset('fever', "v1.0", split="labelled_dev")
     30 df_fever_original_dev = pd.DataFrame(data={"id": dataset_fever_original_dev["id"], "label": dataset_fever_original_dev["label"], "claim": dataset_fever_original_dev["claim"], "evidence_id": dataset_fever_original_dev["evidence_id"]})
     31 df_fever_dev = pd.merge(df_fever_dev, df_fever_original_dev, how="left", left_on="cid", right_on="id")

4 frames

[/usr/local/lib/python3.7/dist-packages/datasets/utils/info_utils.py](https://localhost:8080/#) in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42 

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://s3-eu-west-1.amazonaws.com/fever.public/train.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/shared_task_dev.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/shared_task_dev_public.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/shared_task_test.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/paper_dev.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/paper_test.jsonl']
```


I think this has to be fixed on the google drive side, but you also have to delete the bad stuff from your local cache.  This is not a great design, but it is what it is.

We have fixed the issues with the datasets:
- wider_face: by hosting their data files on the HuggingFace Hub (CC: @HosseynGT)
- fever: by updating to their new data URLs (CC: @MoritzLaurer)

The yelp_review_full datasets has this problem as well and can't be fixed with the suggestion.

This is a super-common failure mode.  We really need to find a better workaround.  My solution was to wait until the owner of the dataset in question did the right thing, and then I had to delete my cached versions of the datasets with the bad checksums.  I don't understand why this happens.  Would it be possible to maintain a copy of the most recent version that was known to work, and roll back to that automatically if the checksums fail?  And if the checksums fail, couldn't the system automatically flush the cached versions with the bad checksums?  It feels like we are blaming the provider of the dataset, when in fact, there are things that the system could do to ease the pain.  Let's take these error messages seriously.  There are too many of them involving too many different datasets.

the [exams](https://huggingface.co/datasets/exams) dataset also has this issue and the provided fix above doesn't work

Same for [DART dataset](https://huggingface.co/datasets/dart):
```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://raw.githubusercontent.com/Yale-LILY/dart/master/data/v1.1.1/dart-v1.1.1-full-train.json', 'https://raw.githubusercontent.com/Yale-LILY/dart/master/data/v1.1.1/dart-v1.1.1-full-dev.json', 'https://raw.githubusercontent.com/Yale-LILY/dart/master/data/v1.1.1/dart-v1.1.1-full-test.json']
```

same for multi_news dataset

- @thesofakillers the issue with `exams` was fixed on 16 Aug by this PR:
  - #4853
- @Aktsvigun the issue with `dart` has been transferred to the Hub: https://huggingface.co/datasets/dart/discussions/1
  - and fixed by PR: https://huggingface.co/datasets/dart/discussions/2
- @Carol-gutianle the issue with `multi_news` have been transferred to the Hub as well: https://huggingface.co/datasets/multi_news/discussions/1
  - not reproducible: maybe you should try to update `datasets`

For information to everybody, we are removing the checksum verifications (that were creating a bad user experience). This will be in place in the following weeks.

auto_gptq is required for real quantization
['/home/sam/Doctorproject/OmniQuant-main/main.py', '--model', '/home/sam/Doctorproject/OmniQuant-main/PATH/TO/LLaMA/llama-7b/', '--epochs', '20', '--output_dir', '/home/sam/Doctorproject/OmniQuant-main/outdir/llama-7b-w3a16/', '--eval_ppl', '--wbits', '3', '--abits', '16', '--lwc', '--net', 'llama-7b', '--aug_loss']
[2024-03-13 17:58:48 root](main.py 262): INFO Namespace(model='/home/sam/Doctorproject/OmniQuant-main/PATH/TO/LLaMA/llama-7b/', cache_dir='./cache', output_dir='/home/sam/Doctorproject/OmniQuant-main/outdir/llama-7b-w3a16/', save_dir=None, resume=None, real_quant=False, calib_dataset='wikitext2', nsamples=128, batch_size=1, seed=2, tasks='', eval_ppl=True, num_fewshot=0, wbits=3, abits=16, group_size=None, alpha=0.5, let_lr=0.005, lwc_lr=0.01, wd=0, epochs=20, let=False, lwc=True, aug_loss=True, symmetric=False, disable_zero_point=False, a_dynamic_method='per_token', w_dynamic_method='per_channel', limit=-1, multigpu=False, deactive_amp=False, attn_implementation='eager', net='llama-7b', act_scales=None, act_shifts=None)
Loading checkpoint shards:   0%|          | 0/33 [00:00<?, ?it/s]/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
Loading checkpoint shards: 100%|██████████| 33/33 [00:11<00:00,  2.98it/s]
vocab size:  32000
[2024-03-13 17:58:59 root](main.py 331): INFO === start quantization ===
get_wikitext2
[2024-03-13 18:02:20 datasets.load](load.py 1586): WARNING Using the latest cached version of the module from /home/sam/.cache/huggingface/modules/datasets_modules/datasets/wikitext/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126 (last modified on Wed Mar 13 16:54:26 2024) since it couldn't be found locally at wikitext, or remotely on the Hugging Face Hub.
Using the latest cached version of the module from /home/sam/.cache/huggingface/modules/datasets_modules/datasets/wikitext/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126 (last modified on Wed Mar 13 16:54:26 2024) since it couldn't be found locally at wikitext, or remotely on the Hugging Face Hub.
Downloading data: 243B [00:00, 877kB/s]
Generating test split:   0%|          | 0/4358 [00:00<?, ? examples/s]
Traceback (most recent call last):
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1742, in _prepare_split_single
    for key, record in generator:
  File "/home/sam/.cache/huggingface/modules/datasets_modules/datasets/wikitext/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/wikitext.py", line 187, in _generate_examples
    with open(data_file, encoding="utf-8") as f:
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/streaming.py", line 75, in wrapper
    return function(*args, download_config=download_config, **kwargs)
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py", line 507, in xopen
    return open(main_hop, mode, *args, **kwargs)
NotADirectoryError: [Errno 20] Not a directory: '/home/sam/.cache/huggingface/datasets/downloads/94be2a7b3fff32ae7379658c8d3821035b666baddad3a06d29b55ab3a4ab3115/wikitext-2-raw/wiki.test.raw'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/sam/Doctorproject/OmniQuant-main/main.py", line 382, in <module>
    main()
  File "/home/sam/Doctorproject/OmniQuant-main/main.py", line 339, in main
    dataloader, _ = get_loaders(
  File "/home/sam/Doctorproject/OmniQuant-main/datautils.py", line 178, in get_loaders
    return get_wikitext2(nsamples, seed, seqlen, model)
  File "/home/sam/Doctorproject/OmniQuant-main/datautils.py", line 37, in get_wikitext2
    traindata = load_dataset(path='wikitext', name='wikitext-2-raw-v1', split='train', download_mode="force_redownload")
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/load.py", line 2598, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1021, in download_and_prepare
    self._download_and_prepare(
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1783, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1116, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1621, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1778, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset


@albertvillanova @Y0mingZhang @kwchurch @HosseynGT @rafikg I tried the solutions you provided above, but none of them worked. Could you please give me some guidance