Bug: Can't download TriviaQA with `load_dataset` - custom `cache_dir`
Author: SapirWeissbuch
State: open

Hello,
I'm having issue downloading TriviaQA dataset with `load_dataset`.

## Environment info
- `datasets` version: 1.1.3
- Platform: Linux-4.19.129-aufs-1-x86_64-with-debian-10.1
- Python version: 3.7.3

## The code I'm running:
```python
import datasets
dataset = datasets.load_dataset("trivia_qa", "rc", cache_dir = "./datasets")
```

## The output:
1. Download begins:
```
Downloading and preparing dataset trivia_qa/rc (download: 2.48 GiB, generated: 14.92 GiB, post-processed: Unknown size, total: 17.40 GiB) to /cs/labs/gabis/sapirweissbuch/tr
ivia_qa/rc/1.1.0/e734e28133f4d9a353af322aa52b9f266f6f27cbf2f072690a1694e577546b0d...                                                                                         
Downloading:  17%|███████████████████▉                                                                                                   | 446M/2.67G [00:37<04:45, 7.77MB/s]
```
2. 100% is reached
3. It got stuck here for about an hour, and added additional 30G of data to "./datasets" directory. I killed the process eventually.

A similar issue can be observed in Google Colab:

https://colab.research.google.com/drive/1nn1Lw02GhfGFylzbS2j6yksGjPo7kkN-?usp=sharing

## Expected behaviour:
The dataset "TriviaQA" should be successfully downloaded.


Comments: Hi @SapirWeissbuch,
When you are saying it freezes, at that time it is unzipping the file from the zip file it downloaded. Since it's a very heavy file it'll take some time. It was taking ~11GB after unzipping when it started reading examples for me. Hope that helps!
![Screenshot 2020-12-21 at 23 40 52](https://user-images.githubusercontent.com/19718818/102808355-3b380c00-43e6-11eb-81ab-c31019ae6322.png)


Hi @bhavitvyamalik 
Thanks for the reply!
Actually I let it run for 30 minutes before I killed the process. In this time, 30GB were extracted (much more than 11GB), I checked the size of the destination directory.

What version of Datasets are you using?


I'm using datasets version: 1.1.3. I think you should drop `cache_dir` and use only
`dataset = datasets.load_dataset("trivia_qa", "rc")`

Tried that on colab and it's working there too
![image](https://user-images.githubusercontent.com/19718818/102814269-4db74300-43f0-11eb-8f26-ecfcf4632002.png)


Train, Validation, and Test splits contain 138384, 18669, and 17210 samples respectively. It takes some time to read the samples. Even in your colab notebook it was reading the samples before you killed the process. Let me know if it works now!

Hi, it works on colab but it still doesn't work on my computer, same problem as before - overly large and long extraction process.
I have to use a custom 'cache_dir' because I don't have any space left in my home directory where it is defaulted, maybe this could be the issue?

I tried running this again - More details of the problem:
Code:
```
datasets.load_dataset("trivia_qa", "rc", cache_dir="/path/to/cache")
```

The output:
```
Downloading and preparing dataset trivia_qa/rc (download: 2.48 GiB, generated: 14.92 GiB, post-processed: Unknown size, total: 17.40 GiB) to path/to/cache/trivia_qa/rc/1.1.0/e734e28133f4d9a353af322aa52b9f266f6f27cbf2f072690a1694e577546b0d...                                                                          
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.67G/2.67G [03:38<00:00, 12.2MB/s]

```
The process continues (no progress bar is visible).
I tried `du -sh .` in `path/to/cache`, and the size keeps increasing, reached 35G before I killed the process.

Google Colab with custom `cache_dir` has same issue.
https://colab.research.google.com/drive/1nn1Lw02GhfGFylzbS2j6yksGjPo7kkN-?usp=sharing#scrollTo=2G2O0AeNIXan

1) You can clear the huggingface folder in your `.cache` directory to use default directory for datasets. Speed of extraction and loading of samples depends a lot on your machine's configurations too.

2) I tried on colab `dataset = datasets.load_dataset("trivia_qa", "rc", cache_dir = "./datasets")`. After memory usage reached around 42GB (starting from 32GB used already), the dataset was loaded in the memory. Even Your colab notebook shows 
![image](https://user-images.githubusercontent.com/19718818/102852229-c7c4e780-4443-11eb-91d6-bf21024358a3.png)
which means it's loaded now.

Facing the same issue.
I am able to download datasets without `cache_dir`, however, when I specify the `cache_dir`, the process hangs indefinitely after partial download. 
Tried for `data = load_dataset("cnn_dailymail", "3.0.0")`

Hi @ashutoshml,
I tried this and it worked for me:
`data = load_dataset("cnn_dailymail", "3.0.0", cache_dir="./dummy")`

I'm using datasets==1.8.0. It took around 3-4 mins for dataset to unpack and start loading examples.

Ok. I waited for 20-30 mins, and it still is stuck.
I am using datasets==1.8.0.

Is there anyway to check what is happening? like a` --verbose` flag?

![Screenshot 2021-06-25 at 6 37 43 PM](https://user-images.githubusercontent.com/2375919/123429653-cdfb7280-d5e4-11eb-9fa7-ff295800cc86.png)