Flax/Roberta - Tokenizer
Author: peregilk
Pull request: no

## Environment info
transformers version: 4.14.0.dev0
Platform: Linux-5.4.0-1043-gcp-x86_64-with-glibc2.29
Python version: 3.8.10
Flax version (CPU?/GPU?/TPU?): 0.3.4 (cpu)
Jax version: 0.2.17
JaxLib version: 0.1.68

### Who can help
@patrickvonplaten, @patil-suraj, @LysandreJik

## Information
I am reporting this as a very strange behaviour when converting a tokenizer from Flax to PyTorch. My initial tests indicates that it also can mess up the Flax training. In any case I do not think it can be intentional.

##To reproduce
Use the procedure explained [here](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling) to create a RoBERTa BPE tokenizer. In the final step this saves the tokenizer.json. This can be used for training a Flax model.

Then use the following procedure to load and save a tokenizer from pretrained:
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(".")
model.save_pretrained(".")
('./tokenizer_config.json', './special_tokens_map.json', './vocab.json', './merges.txt', './added_tokens.json', './tokenizer.json')
```

Apart from creating the extra files, this also overwrites the original tokenizer.json. This should be an identical file. However it is not. In the setting for the mask-token, it changes the lstrip=false to ltrip=true.
```json
{"id":4,"special":true,"content":"<mask>","single_word":false,"lstrip":false,"rstrip":false,"normalized":false}
{"id":4,"special":true,"content":"<mask>","single_word":false,"lstrip":true,"rstrip":false,"normalized":false}
```

Using the "Flax-version" of tokenizer.json messes up the results in the HuggingFace widget. My initial test also indicates that I am getting better results training the Flax model using the settings from the "RoBERTa-version" of tokenizer.json. Though I have not really been able to verify these results yet.



Comments:
Note that tokenizers are independent of the framework, so flax/pytorch shouldn't make any difference. 

> My initial tests indicates that it also can mess up the Flax training

What do you mean by it can mess up the flax training ?

Is there any difference any tokenization when you use `AutoTokenizer`, could you maybe check for a few text examples if the two tokenizers give different results and if so post the code snippet here? Thanks!

Thanks @patil-suraj.

I think this code snippet below shows pretty well what is going on. This initially reads a tokenizer.json that is created by using the script referred above. In the original json the setting for the ```<mask>``` is ```lstrip```is set to ```False```. This seem to be respected when it is read by AutoTokenizer, however, using ```tokenizer.save_pretrained``` it overrides this value and sets it to ```True```.  

```python
>>> tokenizer = AutoTokenizer.from_pretrained(".")
>>> tokenizer.encode("My <mask>.")
[15447, 225, 4, 18]
>>> tokenizer.tokenize("My <mask>.")
['My', 'Ġ', '<mask>', '.']
>>> tokenizer.save_pretrained(".")
('./tokenizer_config.json', './special_tokens_map.json', './vocab.json', './merges.txt', './added_tokens.json', './tokenizer.json')
>>> tokenizer = AutoTokenizer.from_pretrained(".")
>>> tokenizer.encode("My <mask>.")
[15447, 4, 18]
>>> tokenizer.tokenize("My <mask>.")
['My', ' <mask>', '.']
```

I have tried training two models on this. One using ```lstrip=False``` and the other ```lstrip=False```. Using a 1GB corpus and training for 200k. Loss/MLM accuracy is roughly the same but for some (strange) reason, but I am getting different results on the downstream tasks. For the downstream tasks I am evaluating the pytorch model and here lstrip=True (This also needs to be set to "True" for the HuggingFace widget to work).

As you know, the variance when finetuning on downstream tasks is often huge, and the ```<mask>```-token itself is not used directly in the finetuning. On average I am getting better results on the latter model. I am not super-confident on this finding though. I have not done any test where I initiated from the same weights.


Hi @patil-suraj,
Did you have a chance to verify that my example code snippet shows there is a bug here? Or am I missing something?

@peregilk do you get the same errors when you use `RobertaTokenizer` instead of `AutoTokenizer`? I think what might happen here is the second call is in fact a Fast tokenizer while the first is a slow one

@patrickvonplaten 

It does not look like this is the case. Inspecting the tokenizers, loading with AutoTokenizer always seems to load a fast tokenizer, while loading with RobertaTokenizer always seem to load a slow one. This is valid both before and after saving the tokenizer.

I tried this on the tokenizer from your model:
```bash
wget https://huggingface.co/patrickvonplaten/norwegian-roberta-base/raw/main/tokenizer.json
wget https://huggingface.co/patrickvonplaten/norwegian-roberta-base/raw/main/config.json
```

Then I load them:
```python
from transformers import RobertaTokenizer, AutoTokenizer
tokenizerA = AutoTokenizer.from_pretrained(".") 
tokenizerB = RobertaTokenizer.from_pretrained(".")

# Inspecting tokenizerA shows it is fast and tokenizerB is slow. They do give different results:
tokenizerA.tokenize("My <mask>.")
['My', 'Ġ', '<mask>', '.']
tokenizerB.tokenize("My <mask>.")
['My', ' <mask>', '.']

# Then save any of the tokenizers (Saving tokenizerA and tokenizerB seem to give the same result)
tokenizerA.save_pretrained(".")
```

Now try and load the tokenizer
```python
tokenizerA = AutoTokenizer.from_pretrained(".")
tokenizerB = RobertaTokenizer.from_pretrained(".")

# Inspecting these tokenizers also shows one is fast and the other slow. However, they now give identical results.
tokenizerA.tokenize("My <mask>.")
['My', ' <mask>', '.']
tokenizerB.tokenize("My <mask>.")
['My', '<mask>', '.']
```

The main problem is that it seems like training a model on the tokenizer that has not been loaded and saved, will result in a sub-optimal model. This might be because the training script is using AutoTokenizer when tokenizing (and then finish the training by saving the tokenizer).

Thanks for checking, yes this definitely looks like a bug. I'll leave it on my todo list, but probably will only have time in 2 weeks to look into it. If @SaulLu or @LysandreJik have more time by any chance, feel free to go ahead!

Great!

The workaround here is of course very easy - just load and save the tokenizer before starting any training. However, not doing this might hurt the training. To me it seems like switching back to the "unsaved" tokenizer then improves the model, but not as much as if it was trained from scratch with the "saved" tokenizer. But these things are really hard to evaluate since there always is variance in the benchmarks.

Also correcting a small error in my previous post. I am writing that the last output is **identical** for both tokenizers. If you look very closely this is not correct. TokenizerA gives the token  ' \<mask\>', while tokenizerB gives '\<mask\>' (without the leading space).

There are therefore 3 different ways this can be tokenized: ['My', 'Ġ', '<mask>', '.'], ['My', ' <mask>', '.'] and ['My', '<mask>', '.'].



It looks really weird!

On my side, I know that I won't be able to look at this issue until next week at the earliest.

Any chance anyone can take a look at this? Though the workaround is pretty easy, it did mess up several of our models before we noticed it. It is likely that it also will cause problems for others. 

Looking into it tomorrow

@patrickvonplaten Where you able to reproduce this errors?

Just took a look.

```python
from transformers import RobertaTokenizer, AutoTokenizer
tokenizerA = AutoTokenizer.from_pretrained(".") 
tokenizerB = RobertaTokenizer.from_pretrained(".")

# Inspecting tokenizerA shows it is fast and tokenizerB is slow. They do give different results:
tokenizerA.tokenize("My <mask>.")
['My', 'Ġ', '<mask>', '.']
tokenizerB.tokenize("My <mask>.")
['My', ' <mask>', '.']

# Then save any of the tokenizers (Saving tokenizerA and tokenizerB seem to give the same result)
tokenizerA.save_pretrained(".")
```

doesn't work for me as loading the "non-fast" tokenizer fails with:

```
OSError: Can't load tokenizer for '.'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '.' is the correct path to a directory containing all relevant files for a RobertaTokenizer tokenizer.
```

In general, the tokenizer of this model was trained purely to be used with the fast tokenizer class - e.g. it's not an "official" tokenizer. It doesn't surprise me that much that the tokenizers give different resultls here. @peregilk - I think we should try to just use the fast tokenizer here? 

Nevertheless, the issues does uncover another mismatch between fast and slow architectures here it seems cc @SaulLu 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.