Problem converting slow tokenizer to fast: token out of vocabulary
Author: BramVanroy
Pull request: no
Labels: name: Core: Tokenization

name: Fast Tokenizers

When I try to use a [Dutch RoBERTa model](https://huggingface.co/pdelobelle/robbert-v2-dutch-base/tree/main#how-to-use) as suggested, the library tries to convert the old (slow) tokenizer to the fast one. However, this leads to issues. (I can just keep the slow one, but I need to use the offset and word_ids functionality which is only available in the fast tokenizers.)

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("pdelobelle/robbert-v2-dutch-base")
```

Error trace:

```
 File "C:/dev/python/jasper-tok2vec/main.py", line 10, in main
    tokenizer = AutoTokenizer.from_pretrained("pdelobelle/robbert-v2-dutch-base")
  File "C:\Users\bramv\.virtualenvs\jasper-tok2vec-4kc9ajCV\lib\site-packages\transformers\models\auto\tokenization_auto.py", line 378, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "C:\Users\bramv\.virtualenvs\jasper-tok2vec-4kc9ajCV\lib\site-packages\transformers\tokenization_utils_base.py", line 1804, in from_pretrained
    return cls._from_pretrained(
  File "C:\Users\bramv\.virtualenvs\jasper-tok2vec-4kc9ajCV\lib\site-packages\transformers\tokenization_utils_base.py", line 1877, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "C:\Users\bramv\.virtualenvs\jasper-tok2vec-4kc9ajCV\lib\site-packages\transformers\models\roberta\tokenization_roberta_fast.py", line 160, in __init__
    super().__init__(
  File "C:\Users\bramv\.virtualenvs\jasper-tok2vec-4kc9ajCV\lib\site-packages\transformers\models\gpt2\tokenization_gpt2_fast.py", line 133, in __init__
    super().__init__(
  File "C:\Users\bramv\.virtualenvs\jasper-tok2vec-4kc9ajCV\lib\site-packages\transformers\tokenization_utils_fast.py", line 89, in __init__
    fast_tokenizer = convert_slow_tokenizer(slow_tokenizer)
  File "C:\Users\bramv\.virtualenvs\jasper-tok2vec-4kc9ajCV\lib\site-packages\transformers\convert_slow_tokenizer.py", line 642, in convert_slow_tokenizer
    return converter_class(transformer_tokenizer).converted()
  File "C:\Users\bramv\.virtualenvs\jasper-tok2vec-4kc9ajCV\lib\site-packages\transformers\convert_slow_tokenizer.py", line 262, in converted
    BPE(
Exception: Error while initializing BPE: Token `Ċ` out of vocabulary
```

### Environment

- `transformers` version: 4.1.1
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.2
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)


### Who can help
 
@mfuntowicz

Comments:
That looks like a tough one...we might need help from @n1t0 here

The files for this model can be found [here](https://huggingface.co/pdelobelle/robbert-v2-dutch-base/tree/main).

In `merges.txt` is the merge rule: `ĉ Ċ`
That means that the symbol `Ċ` should be present in `vocab.json`
It is not.
So the error does kind of make sense.

 @iPieter are these the correct files?
In the paper you mention:
> We limited the vocabulary to 40k words,
> which is 10k words less than RobBERT v1, due to
> additional tokens including non-negligible num-
> ber of Unicode tokens that are not used in Dutch.

There are 39982 words in the vocabulary. 
Is it possible that some of the symbols/tokens are missing?




@schelv Thanks for looking into the issue. Those files should be correct, there are indeed 39982 tokens.

I am using the same files internally without any issues on the old tokenizer (i.e. correct behaviour and sensible predictions), for which the files were specifically created. I also looked into this error before, but cannot find an easy fix. There are a few other merge rules that are also conflicting. 

My suspicion is that this stems from the translation of the tokenizer's files from Fairseq to HF. This was done a year ago, so the details might be a bit fuzzy. The problem was that the Fairseq library, where we trained RobBERT, used the vocab and merges file, but also generated an additional file (`dict.txt`) that was used to count the number of occurrences for each token (which is ok, not the issue) and also orders the tokens. 

These new, ordered positions were then used in Fairseq, while HF uses the id's from the vocab.json file. So this means there is an additional dictionary lookup. To fix this behaviour in HF transformers, I created a script to merge the vocab.sjon with the dict.txt. Otherwise, the token id's from unrelated tokens would be used for the embedding layer and the MLM task, giving a garbage output. 

I will investigate this translation step again, but I'm confused by the fact that the behaviour is correct with the slow tokenizer.



Thanks for the quick answer!

Just checking if I understand you correctly:
Fairseq uses token id's that are based on the token occurrence count. This information is stored in dict.txt
The token id's of the original vocab.json were updated with the information from dict.txt? did this create the current vocab.json that is loaded by the transformers library?

If you upload the original files somewhere I can also take a look at them.🙂

Yes, that's exactly right. The conversion script is [here](https://github.com/iPieter/RobBERT/blob/340bd9d87ef362462fccf8f44e7740c7dfd1d865/src/convert_roberta_dict.py) and [this is the unit test](https://github.com/iPieter/RobBERT/blob/340bd9d87ef362462fccf8f44e7740c7dfd1d865/tests/test_convert_roberta_dict.py) that missed this case. The original files are also downloadable from our github release [here](https://github.com/iPieter/RobBERT/releases/tag/v2.0), but you have to download the entire model. 

However, when writing the previous response, I had an insight in what the issue might be. So'll try to debug and report the results here.

_Long report ahead, TLDR: it now works_ 😃 

There might be tokens in the `vocab.json` that was generated by the HuggingFace tokenizer library that are not found by the Fairseq tokenizer, thus they don't occur in the `dict.txt`. After some debugging, I found the these tokens (the special tokens are handled later): 

```python
00:'Á'
01:'÷'
02:'À'
03:'þ'
04:'ø'
05:'ÿ'
06:'ú'
07:'ö'
08:'ĉĊ'
09:'
'
10:'č'
11:'ĠTheCompany'
12:'û'
13:'ü'
14:'<unk>'
15:'ý'
16:'õ'
17:'<pad>'
18:'Ċ'
19:'
'
20:'<mask>'
21:'ù'
```

So these tokens are what is causing the fast tokenizer to complain, since they appear in the `vocab.json` set and not in the `dict.txt` set. Ignoring the special tokens (`<unk>`, `
`, `
` and `<pad>`), this brings the latest vocab id to 39996, not yet 40k. So there is a second bug in my conversion script. 

The second bug has to do with the fact that Fairseq adds 2 custom tokens by default that I didn't remove. That's not a big deal, but they do affect the vocab length, so let's be totally correct and add those two tokens and the mask token (since `robbert-v2-dutch-base` is and MLM model) as well:

<img width="1019" alt="image" src="https://user-images.githubusercontent.com/6965756/109134694-9a9ff880-7756-11eb-9734-f678d6dc8845.png">

Time for a sanity check:
```python
[{'sequence': 'Er staat een boom in mijn tuin.', 'score': 0.16917602717876434, 'token': 2600, 'token_str': ' boom'},
 {'sequence': 'Er staat een bankje in mijn tuin.', 'score': 0.08176644891500473, 'token': 21620, 'token_str': ' bankje'}, 
{'sequence': 'Er staat een schutting in mijn tuin.', 'score': 0.0384209081530571, 'token': 15000, 'token_str': ' schutting'}, 
{'sequence': 'Er staat een vijver in mijn tuin.', 'score': 0.038086555898189545, 'token': 8217, 'token_str': ' vijver'}, 
{'sequence': 'Er staat een plant in mijn tuin.', 'score': 0.03249552100896835, 'token': 2721, 'token_str': ' plant'}]
````

The vocab.json in `robbert-v2-dutch-base` is updated, so this issue can be closed.



Great, thanks a lot for the investigation @iPieter! Now I can use the fast tokenizers in all their glory.

Thanks from me as well! Looking forward to using it!