OSError: Not enough disk space.
Author: qqaatw
State: closed
Labels: name: bug

## Describe the bug
I'm trying to download `natural_questions` dataset from the Internet, and I've specified the cache_dir which locates in a mounted disk and has enough disk space. However, even though the space is enough, the disk space checking function still reports the space of root `/` disk having no enough space. 

The file system structure is like below. The root `/` has `115G` disk space available, and the `sda1` is mounted to `/mnt`, which has `1.2T` disk space available:
```
/
/mnt/sda1/path/to/args.dataset_cache_dir 
```

## Steps to reproduce the bug
```python
dataset_config = DownloadConfig(
    cache_dir=os.path.abspath(args.dataset_cache_dir),
    resume_download=True,
)
dataset = load_dataset("natural_questions", download_config=dataset_config)
```

## Expected results

Can download the dataset without an error.

## Actual results

The following error raised:
```
OSError: Not enough disk space. Needed: 134.92 GiB (download: 41.97 GiB, generated: 92.95 GiB, post-processed: Unknown size)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Ubuntu 18.04
- Python version: 3.8.10
- PyArrow version:


Comments: Maybe we can change the disk space calculating API from `shutil.disk_usage` to  `os.statvfs` in UNIX-like system, which can provide correct results.
```
statvfs = os.statvfs('path')
avail_space_bytes = statvfs.f_frsize * statvfs.f_bavail
```

Hi @qqaatw, thanks for reporting.

Could you please try:
```python
dataset = load_dataset("natural_questions", cache_dir=os.path.abspath(args.dataset_cache_dir))
```

@albertvillanova it works! Thanks for your suggestion. Is that a bug of `DownloadConfig`?

`DownloadConfig` only sets the location to download the files. On the other hand, `cache_dir` sets the location for both downloading and caching the data. You can find more information here: https://huggingface.co/docs/datasets/loading_datasets.html#cache-directory

I had encountered the same error when running a command `ds = load_dataset('food101')` in a docker container. The error I got: `OSError: Not enough disk space. Needed: 9.43 GiB (download: 4.65 GiB, generated: 4.77 GiB, post-processed: Unknown size)`

In case anyone encountered the same issue, this was my fix:

```sh
# starting the container (mount project directory onto /app, so that the code and data in my project directory are available in the container)
docker run -it --rm -v $(pwd):/app my-demo:latest bash
```


```python
# other code ...
ds = load_dataset('food101', cache_dir="/app/data") # set cache_dir to the absolute path of a directory (e.g. /app/data) that's mounted from the host (MacOS in my case) into the docker container

# this assumes ./data directory exists in your project folder. If not, create it or point it to any other existing directory where you want to store the cache
```

Thanks @albertvillanova for posting the fix above :-) 

import datasets
    datasets.builder.has_sufficient_disk_space = lambda needed_bytes, directory='.': True