Map cache with tokenizer
Opened by jonathanasdf (open)

Similar issue to https://github.com/huggingface/datasets/issues/5985, but across different sessions rather than two calls in the same session.

Unlike that issue, explicitly calling tokenizer(my_args) before the map() doesn't help, because the tokenizer was created with a different hash to begin with...

setup
```
from transformers import AutoTokenizer
AutoTokenizer.from_pretrained('bert-base-uncased').save_pretrained("tok")
```

this prints different value each time
```
from transformers import AutoTokenizer
from datasets.utils.py_utils import dumps # Huggingface datasets
print(hash(dumps(AutoTokenizer.from_pretrained("tok"))))
```

Comments:
https://github.com/huggingface/datasets/issues/5147 may be a solution, by passing in the tokenizer in a fn_kwargs and ignoring it in the fingerprint calculations

I have a similar issue. I was using a Jupyter Notebook and every time I call the map function it performs tokenization from scratch again although the cache files of last run still exists. 

I ran with 20 processes and now in the cache folder there are two groups of cached results of tokenized dataset:

```
.rw-r--r-- fad3ew bii_dsc_community 240 MB Sat Aug 26 12:56:46 2023 cache-1982fea76aa54a13_00001_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Sat Aug 26 13:02:08 2023 cache-1982fea76aa54a13_00004_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Sat Aug 26 12:56:40 2023 cache-1982fea76aa54a13_00005_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 241 MB Sat Aug 26 12:50:59 2023 cache-1982fea76aa54a13_00006_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Sat Aug 26 12:57:37 2023 cache-1982fea76aa54a13_00007_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Sat Aug 26 12:57:31 2023 cache-1982fea76aa54a13_00008_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Sat Aug 26 12:59:47 2023 cache-1982fea76aa54a13_00010_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 241 MB Sat Aug 26 12:59:44 2023 cache-1982fea76aa54a13_00011_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 241 MB Sat Aug 26 12:55:24 2023 cache-1982fea76aa54a13_00012_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 241 MB Sat Aug 26 12:56:21 2023 cache-1982fea76aa54a13_00013_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Sat Aug 26 12:57:24 2023 cache-1982fea76aa54a13_00014_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Sat Aug 26 13:00:48 2023 cache-1982fea76aa54a13_00015_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Sat Aug 26 12:56:56 2023 cache-1982fea76aa54a13_00017_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Sat Aug 26 12:56:54 2023 cache-1982fea76aa54a13_00018_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Sat Aug 26 12:57:27 2023 cache-1982fea76aa54a13_00019_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Wed Aug 23 19:15:40 2023 cache-454431f643cdc5e8_00000_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Wed Aug 23 19:16:46 2023 cache-454431f643cdc5e8_00001_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Wed Aug 23 19:14:53 2023 cache-454431f643cdc5e8_00002_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Wed Aug 23 19:13:10 2023 cache-454431f643cdc5e8_00003_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Wed Aug 23 19:13:04 2023 cache-454431f643cdc5e8_00004_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Wed Aug 23 19:16:42 2023 cache-454431f643cdc5e8_00005_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 241 MB Wed Aug 23 19:01:29 2023 cache-454431f643cdc5e8_00006_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Wed Aug 23 19:16:41 2023 cache-454431f643cdc5e8_00007_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Wed Aug 23 19:14:04 2023 cache-454431f643cdc5e8_00008_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Wed Aug 23 19:17:41 2023 cache-454431f643cdc5e8_00009_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Wed Aug 23 19:17:06 2023 cache-454431f643cdc5e8_00010_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 241 MB Wed Aug 23 19:17:16 2023 cache-454431f643cdc5e8_00011_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 241 MB Wed Aug 23 19:15:13 2023 cache-454431f643cdc5e8_00012_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 241 MB Wed Aug 23 19:16:01 2023 cache-454431f643cdc5e8_00013_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Wed Aug 23 19:16:35 2023 cache-454431f643cdc5e8_00014_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Wed Aug 23 19:16:20 2023 cache-454431f643cdc5e8_00015_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Wed Aug 23 19:14:48 2023 cache-454431f643cdc5e8_00016_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Wed Aug 23 18:59:32 2023 cache-454431f643cdc5e8_00017_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Wed Aug 23 19:17:58 2023 cache-454431f643cdc5e8_00018_of_00020.arrow
.rw-r--r-- fad3ew bii_dsc_community 240 MB Wed Aug 23 19:15:25 2023 cache-454431f643cdc5e8_00019_of_00020.arrow
```

can we specify the cache file for map so that it won't redo everything again?

@Luosuu [map](https://huggingface.co/docs/datasets/v2.14.4/en/package_reference/main_classes#datasets.Dataset.map) has cache_file_name parameter

In my case, I do want the cache to detect when the map function changes, so I can't pass a constant cache file name.

Implementing a proper hashing function for the (fast) tokenizers is currently impossible for the reasons mentioned in the referenced issues. So the only alternative to the `cache_file_name` (or `new_fingerprint`) parameter is a custom serializer (e.g., that deserializes the tokenizer from a local save path) defined using `copyreg` or a class that wraps the tokenizer object and has `__reduce__`(`__setstate__`/`__getstate__`)