Cache not being used when loading commonvoice 8.0.0
Opened by clabornd (closed)

### Describe the bug

I have commonvoice 8.0.0 downloaded in `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/b2f8b72f8f30b2e98c41ccf855954d9e35a5fa498c43332df198534ff9797a4a`.  The folder contains all the arrow files etc, and was used as the cached version last time I touched the ec2 instance I'm working on.  Now, with the same command that downloaded it initially:

```
dataset = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token="<mytoken>")
```

it tries to redownload the dataset to `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/05bdc7940b0a336ceeaeef13470c89522c29a8e4494cbeece64fb472a87acb32`

### Steps to reproduce the bug

Steps to reproduce the behavior:

1.  ```dataset = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token="<mytoken>")```
2. dataset is updated by maintainers
3. ```dataset = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token="<mytoken>")```

### Expected behavior

I expect that it uses the already downloaded data in `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/b2f8b72f8f30b2e98c41ccf855954d9e35a5fa498c43332df198534ff9797a4a`.

Not sure what's happening in 2. but if, say it's an issue with the dataset referenced by "mozilla-foundation/common_voice_8_0" being modified by the maintainers, how would I force datasets to point to the original version I downloaded?

EDIT:  It was indeed that the maintainers had updated the dataset (v 8.0.0).  However I still cant load the dataset from disk instead of redownloading, with for example:

```
load_dataset(".cache/huggingface/datasets/downloads/extracted/<hash>/cv-corpus-8.0-2022-01-19/en/", "en")

> ...
> File [~/miniconda3/envs/aa_torch2/lib/python3.10/site-packages/datasets/table.py:1938](.../ python3.10/site-packages/datasets/table.py:1938), in cast_array_to_feature(array, feature, allow_number_to_str)
   1937 elif not isinstance(feature, (Sequence, dict, list, tuple)):
-> 1938     return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
...
   1794         e = e.__context__
-> 1795     raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1797 yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset
```


### Environment info

datasets==2.7.0
python==3.10.8
OS:  AWS Linux

Comments:
You can avoid this by using the `revision` parameter in `load_dataset` to always force downloading a specific commit (if not specified it defaults to HEAD, hence the redownload).

Thanks @mariosasko this works well, looks like I should have read the documentation a bit more carefully.  

It is still a bit confusing which hash I should provide:  passing `revision = c8fd66e85f086e3abb11eeee55b1737a3d1e8487` from https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0/commits/main caused the cached version at `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/b2f8b72f8f30b2e98c41ccf855954d9e35a5fa498c43332df198534ff9797a4a` to be loaded, so I had to know that it was the previous commit unless I've missed something else.