Cached dataset overflowing disk space
State: closed | Author: BirgerMoell (CONTRIBUTOR)

I'm training a Swedish Wav2vec2 model on a Linux GPU and having issues that the huggingface cached dataset folder is completely filling up my disk space (I'm training on a dataset of around 500 gb).

The cache folder is 500gb (and now my disk space is full).

Is there a way to toggle caching or set the caching to be stored on a different device (I have another drive with 4 tb that could hold the caching files).

This might not technically be a bug, but I was unsure and I felt that the bug was the closest one.

Traceback (most recent call last):
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/multiprocess/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 186, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/datasets/fingerprint.py", line 397, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1983, in _map_single
    writer.finalize()
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/datasets/arrow_writer.py", line 418, in finalize
    self.pa_writer.close()
  File "pyarrow/ipc.pxi", line 402, in pyarrow.lib._CRecordBatchWriter.close
  File "pyarrow/error.pxi", line 97, in pyarrow.lib.check_status
OSError: [Errno 28] Error writing bytes to file. Detail: [errno 28] No space left on device
"""

The above exception was the direct cause of the following exception:


Comments: Hi! I'm transferring this issue over to `datasets`

I'm using the datasets concatenate dataset to combine the datasets and then train.
train_dataset = concatenate_datasets([dataset1, dataset2, common_voice_train])



Hi @BirgerMoell.

You have several options:
- to set caching to be stored on a different path location, other than the default one (`~/.cache/huggingface/datasets`):
  - either setting the environment variable `HF_DATASETS_CACHE` with the path to the new cache location
  - or by passing it with the parameter `cache_dir` when loading each of the datasets: `dataset = load_dataset(..., cache_dir=your_new_location)`

  You can get all the information in the docs: https://huggingface.co/docs/datasets/loading_datasets.html#cache-directory
- I wouldn't recommend disabling caching, because current implementation generates cache files anyway, although in a temporary directory and they are deleted when the session closes. See details here: https://huggingface.co/docs/datasets/processing.html#enable-or-disable-caching
- You could alternatively load the datasets in streaming mode. This is a new feature which allows loading the datasets without downloading the entire files. More information here: https://huggingface.co/docs/datasets/dataset_streaming.html

Hi @BirgerMoell,

We are planning to add a new feature to datasets, which could be interesting in your case: Add the option to delete temporary files (decompressed files) from the cache directory (see: #2481, #2604).

We will ping you once this feature is implemented, so that the size of your cache directory will be considerably reduced.