The datasets.map function does not load cached dataset after moving python script
Author: hobbitlzy
State: closed
Labels: name: bug

## Describe the bug
The datasets.map function caches the processed data to a certain directory. When the map function is called another time with totally the same parameters, the cached data are supposed to be reloaded instead of re-processing. However, it doesn't reuse cached data sometimes. I use the common data processing in different tasks, the datasets are processed again, the only difference is that I run them in different files.

## Steps to reproduce the bug
Just run the following codes in different .py files.
```python
if __name__ == '__main__':
    from datasets import load_dataset
    from transformers import AutoTokenizer
    raw_datasets = load_dataset("wikitext", "wikitext-2-raw-v1")

    tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")


    def tokenize_function(examples):
        return tokenizer(examples["text"], padding="max_length", truncation=True)


    tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)
```

## Expected results
The map function should reload data in the second or any later runs.

## Actual results
The processing happens in each run.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: linux
- Python version: 3.7.6
- PyArrow version: 3.0.0

This is the first time I report a bug. If there is any problem or confusing description, please let me know 😄.


Comments: This also happened to me on COLAB.
Details:
I ran the `run_mlm.py` in two different notebooks. 
In the first notebook, I do tokenization since I can get 4 CPU cores without any GPUs, and save the cache into a folder which I copy to drive.
In the second notebook, I copy the cache folder from drive and re-run the run_mlm.py script (this time I uncomment the trainer code which happens after the tokenization)

Note: I didn't change anything in the arguments, not even the preprocessing_num_workers
 

Thanks for reporting ! This is indeed a bug, I'm looking into it

#2854 fixed the issue :)

We'll do a new release of `datasets` soon to make the fix available.
In the meantime, feel free to try it out by installing `datasets` from source

If you have other issues or any question, feel free to re-open the issue :)

Hello there, 

Although I don't change any parameter in the map function, I've faced the same issue.

This is the code that I use to map and tokenise my dataset:

```
train_data = train_data.map(
                              lambda samples: self.tokenizer(
                                          text=samples['source'], 
                                          text_target=samples['target'], 
                                          padding='max_length', truncation=True),
                                batched=True, batch_size=128)
```

Hi @TheTahaaa,

Could you please open a new Bug issue with all the information about your local environment? 

> Hi @TheTahaaa,
> 
> Could you please open a new Bug issue with all the information about your local environment?

Hey there,

I managed to solve the problem! Initially, I created the dataset within a function and then tokenised it separately in another part of the code. This approach prevented the tokeniser from accessing the dataset from the cache. So, I moved the tokenisation process into the dataset creation function itself. Now, the .map() function tokenises the data quickly by loading it from the cache!