AutoTokenizer hash value got change after datasets.map
Opened by tshu-w (open)

## Describe the bug
AutoTokenizer hash value got change after datasets.map

## Steps to reproduce the bug
1. trash huggingface datasets cache
2. run the following code:
```python
from transformers import AutoTokenizer, BertTokenizer
from datasets import load_dataset
from datasets.fingerprint import Hasher
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')

def tokenize_function(example):
    return tokenizer(example["sentence1"], example["sentence2"], truncation=True)

raw_datasets = load_dataset("glue", "mrpc")

print(Hasher.hash(tokenize_function))
print(Hasher.hash(tokenizer))

tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)

print(Hasher.hash(tokenize_function))
print(Hasher.hash(tokenizer))
```
got
```
Reusing dataset glue (/home1/wts/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1112.35it/s]
f4976bb4694ebc51
3fca35a1fd4a1251
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00,  6.96ba/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 15.25ba/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  5.81ba/s]
d32837619b7d7d01
5fd925c82edd62b6
```
3. run raw_datasets.map(tokenize_function, batched=True) again and see some dataset are not using cache.

## Expected results
`AutoTokenizer` work like specific Tokenizer (The hash value don't change after map):
```python
from transformers import AutoTokenizer, BertTokenizer
from datasets import load_dataset
from datasets.fingerprint import Hasher
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

def tokenize_function(example):
    return tokenizer(example["sentence1"], example["sentence2"], truncation=True)

raw_datasets = load_dataset("glue", "mrpc")

print(Hasher.hash(tokenize_function))
print(Hasher.hash(tokenizer))

tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)

print(Hasher.hash(tokenize_function))
print(Hasher.hash(tokenizer))
```

```
Reusing dataset glue (/home1/wts/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1091.22it/s]
46d4b31f54153fc7
5b8771afd8d43888
Loading cached processed dataset at /home1/wts/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad/cache-6b07ff82ae9d5c51.arrow
Loading cached processed dataset at /home1/wts/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad/cache-af738a6d84f3864b.arrow
Loading cached processed dataset at /home1/wts/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad/cache-531d2a603ba713c1.arrow
46d4b31f54153fc7
5b8771afd8d43888
```


## Environment info
- `datasets` version: 1.18.0
- Platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyArrow version: 6.0.1


Comments:
This issue was original reported at https://github.com/huggingface/transformers/issues/14931 and It seems like this issue also occur with other AutoClass like AutoFeatureExtractor.

Thanks for moving the issue here !

I wasn't able to reproduce the issue on my env (the hashes stay the same):
```
- `transformers` version: 1.15.0
- `tokenizers` version: 0.10.3
- `datasets` version: 1.18.1
- `dill` version: 0.3.4
- Platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-debian-10.11
- Python version: 3.7.10
- PyArrow version: 6.0.1
```
However I was able to reproduce it on Google Colab (the hashes end up different):
```
- `transformers` version: 1.15.0
- `tokenizers` version: 0.10.3
- `datasets` version: 1.18.1
- `dill` version: 0.3.4
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0
```
I'll investigate why it doesn't work properly on Google Colab :)

I found the issue: the tokenizer has something inside it that changes.

Before the call, `tokenizer._tokenizer.truncation` is None, and after the call it changes to this for some reason:
```
{'max_length': 512, 'strategy': 'longest_first', 'stride': 0}
```

Does anybody know why calling the tokenizer would change its state this way ? cc @Narsil @SaulLu maybe ?

`tokenizer.encode(..)` does not accept argument like max_length, strategy or stride.

In `tokenizers` you have to modify the tokenizer state by setting various `TruncationParams` (and/or `PaddingParams`).
However, since this is modifying the state, you need to mutably borrow the tokenizer (a rust concept). The key principle is that there can ever be only 1 mutable borrow at a time during the span of the tokenizer lifecycle.

Because of this, if `transformers` blindly set  `TruncationParams` and `PaddingParams` on every call, it would cause the tokenizer to crash (or make the various threads accessing it hang, which is not necessarily better).

In order to avoid that, we decided to handle it this way : https://github.com/huggingface/transformers/pull/12550 . 

Which should explain the state of the tokenizer being modified (hence its hash).

Now for a temporary solution, simply encoding once with the tokenizer should give it it's proper hash (since by default the tokenizer doesn't have this state, looks at the first encoding call, and creates it).

We could try and set these 2 dicts at initialization time, but it wouldn't work if a user modified the tokenizer state later
```python
tokenizer = AutoTokenizer.from_pretrained(..)
tokenizer.truncation_side = "left"
# Now we have a difference between `tokenizer._tokenizer.truncation` and `tokenizer.truncation_side`
```
If we wanted to fix it correctly it would mean mapping every assignation to it's proper location on `tokenizer.{padding/truncation}`

I think it's important to note that we cannot guarantee a tokenizer' hash remains the same if *any* of those parameters are modified through the `.map` function.

Edit: Another option would be to override the default __hash__ function, but I don't know if there's  a sound implementation that could fit.

Thanks a lot for the explanation !
I think if we set these 2 dicts at initialization time it would be amazing already

Shall we open an issue in `transformers` to ask for these dictionaries to be set when the tokenizer is instantiated ?

> Edit: Another option would be to override the default hash function, but I don't know if there's a sound implementation that could fit.

In `datasets` we can easily have custom hashing for objects of the other HF libraries if we want. For example we ignore the cache some tokenizers have. However in this specific case it touches parameters that may change the behavior of the tokenizer itself. I'm not sure the logic that determines how a tokenizer behaves should be in `datasets`

A hack we could have in the `datasets` lib would be to call the tokenizer before hashing it in order to set all its parameters correctly - but it sounds a lot like a hack and I'm not sure this can work in the long run

Fully agree with everything you said. 

I think the best course of action is creating an issue in `transformers`. I can start the work on this.
I think the code changes are fairly simple. Making a sound test + not breaking other stuff might be different :D

It should be noted that this problem also occurs in other AutoClasses, such as AutoFeatureExtractor, so I don't think handling it in Datasets is a long-term practice either.

> I think the best course of action is creating an issue in `transformers`. I can start the work on this.

@Narsil Hi, I reopen this issue in `transformers` https://github.com/huggingface/transformers/issues/14931

Here is @Narsil comment from https://github.com/huggingface/transformers/issues/14931#issuecomment-1074981569
> # TL;DR
> Call the function once on a dummy example beforehand will fix it.
> 
> ```python
> tokenizer("Some", "test", truncation=True)
> ```
> 
> # Long answer
> If I remember the last status, it's hard doing anything, since the call itself
> 
> ```python
> tokenizer(example["sentence1"], example["sentence2"], truncation=True)
> ```
> 
> will modify the tokenizer. It's the `truncation=True` that modifies the tokenizer to put it into truncation mode if you will. Calling the tokenizer once with that argument would fix the cache.
> 
> Finding a fix that :
> 
> * Doesn't imply a huge chunk of work on `tokenizers` (with potential loss of performance, and breaking backward compatibility)
> * Doesn't imply `datasets` running a first pass of the loop
> * Doesn't imply `datasets` looking at the map function itself
> * Uses a sound `hash` for this object in `datasets`.
> 
> is IIRC impossible for this use case.
> 
> I can explain a bit more why the first option is not desirable.
> 
> In order to "fix" this for tokenizers, we would need to make `tokenizer(..)` purely without side effects. This means that the "options" of tokenization (like `truncation` and `padding` at least) would have


For me this workaround only works if I don't pass the `num_proc=X` argument to `datasets.map`

Is there an easy solution for setting both num_proc and padding/truncation for fast tokenizer or caching just not a thing in this case?