Mounting datasets cache fails due to absolute paths.
Opened by charliebudd (closed)

### Describe the bug

Creating a datasets cache and mounting this into, for example, a docker container, renders the data unreadable due to absolute paths written into the cache.

### Steps to reproduce the bug

1. Create a datasets cache by downloading some data
2. Mount the dataset folder into a docker container or remote system.
3. (Edit) Set `HF_HOME` or `HF_DATASET_CACHE` to point to the mounted cache.
4. Attempt to access the data from within the docker container.
5. An error is thrown saying no file exists at \<absolute path to original cache location\>

### Expected behavior

The data is loaded without error

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.4.0-162-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.16.4
- PyArrow version: 13.0.0
- Pandas version: 2.0.3

Comments:
You may be able to make it work by tweaking some environment variables, such as [`HF_HOME`](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/environment_variables#hfhome) or [`HF_DATASETS_CACHE`](https://huggingface.co/docs/datasets/cache#cache-directory).

> You may be able to make it work by tweaking some environment variables, such as [`HF_HOME`](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/environment_variables#hfhome) or [`HF_DATASETS_CACHE`](https://huggingface.co/docs/datasets/cache#cache-directory).

I am already doing this. The problem is that, while this seemingly allows flexibility, the absolute paths written into the cache still have the old cache directory. The paths written into the cache should be relative to the cache location to allow this sort of flexibility. Sorry, I omitted this in the reproduction steps, I have now added it.

I'm unable to reproduce this with the cache
```bash
export HF_CACHE=$PWD/hf_cache
python -c "import datasets; datasets.load_dataset('imdb')"
```
imported inside a dummy container that is built from
```bash
FROM python:3.9

WORKDIR /usr/src/app

RUN pip install datasets

COPY ./hf_cache ./hf_cache

ENV HF_HOME=./hf_cache
ENV HF_DATASETS_OFFLINE=1

CMD ["python"]
```
What do you mean by "absolute paths written into the cache"? Paths inside the HF cache paths are based on hash (hashed URL of the downloaded files, etc.)

@mariosasko Same problem:  the absolute paths written into the cache still have the old cache directory. Like:

{'bytes': None, 'path': 'E:\\work-20240321\\datasets\\downloads\\extracted\\9752883596854dc57e01c74cc3f494b2ba63754dadd9e77f9d1932deddbd2273\\58f33a03-026f-4adc-b69f-b89d16b9f35a.webp'}

When I move this cached directory to another directory, these datasets cannot be used casue path changes. So, the paths written into the cache should be relative to the cache location to allow this sort of flexibility. 

Sorry, the reply on this thread escaped my attention. The problem with @mariosasko's attempted reproduction is the absolute path `./hf_cache`  is the same in the host system and the docker container, so naturally the paths would be correct. Modifying the docker image as below should reproduce the error...

```
FROM python:3.9

WORKDIR /usr/src/app

RUN pip install datasets

COPY ./hf_cache ./my_cache/

ENV HF_HOME=./my_cache/
ENV HF_DATASETS_OFFLINE=1

CMD ["python"]
```

The paths written inside the cache will still have `./hf_cache` prefixing all the paths. If they were relative paths (relative to the top level of the cache) this would be avoided.