TokenizerFast.from_file() is stuck when loading a large tokenizer.json with tokens added and "pre-trained" starting from an existing, trained model
Author: PiercarloSlavazza
Pull request: no

## Environment info

- OS: Macos Monterey, v12.2.1, Macbook Pro M1; the issue also occurred when the program has been executed using Ubuntu 20.04 LTS, hosted both on Linode and Paperspace
- `transformers` version: 4.16.0.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.12
- PyTorch version (GPU?): 1.10.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

@SaulLu

## Information

Model I am using (Bert, XLNet ...): Distilbert

The problem arises when using:
* the official example scripts - details below

The tasks I am working on is:
* my own task or dataset - details below

## To reproduce

1. grab the attached tar containing the pair of files `tokenizer_config.json` and `tokenizer.json` causing the issue - [tokenizer_pretrained_w_additional_tokens.tar.gz](https://github.com/huggingface/transformers/files/8102655/tokenizer_pretrained_w_additional_tokens.tar.gz)
2. extract the archive
2. just call `AutoTokenizer.from_pretrained(<folder where the archive has been extracted>)`

## Expected behavior

* What is expected: the tokenizer is loaded in a reasonable amount of time (say 1 min)
* What happens: the code is executed without runtime erros, but after several minutes (> 10min) the program is still running, with a CPU core 100% busy, no log is written to the console (please let me know if there is some switch that can enable debug logs)
* Notes:
    * the last step that I have been able to follow via the PyCharm debugger is line 108 of the in use version of the `tokenization_utils_fast.py` file, that is just `fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)`
        * from that point onwards PyCharm isn't able to "follow the code" - please note that I am NOT proficient in Python
    * the tokenizer json files have been obtained by proceeding in the following way:
        * I started from a [trained model for Italian available on the 🤗 Models Repo](https://huggingface.co/indigo-ai/BERTino)
        * I have added a bunch of additional tokens to the tokenizer (as explained in the answers of [this Stackoverflow question](https://stackoverflow.com/q/71067376/445090)): note that the tokens are not "words", but let's say codes meaningful in my domain, and they amount to 125936
        * I have executed an additional pre-training of the model with the new tokens: the training run successfully for 3 epochs with the default parameters of the [run_mlm.py script](https://raw.githubusercontent.com/huggingface/transformers/master/examples/pytorch/language-modeling/run_mlm.py)
        * Then I moved the trained model with all its "ancillary" files to other machines, and executed the above code (with the ultimate aim to use the model+additional tokens for computing embeddings for the additional tokens)

Final note: it _might_ be ok for tokenizer to take so much time to be loaded, _but_ I would like to know:
* whether if it is actually ok or it is a bug
* if it is bug, what can we do in order to gather further information - or, if the bug is known, where it has been tracked
* if it is NOT a bug:
    * why it's taking so long
    * how much time we might estimate to take to complete the loading
    * what we can do in order to speed up the thing - changes to the code, switch to some other type of tokenizer etc

Comments:
Some more context: I figured out how to a) enable logs b) switch to the "slow" (i.e. Python only) tokenizer - it is as simple as:

```python
AutoTokenizer.from_pretrained(…, use_fast=False, verbose=True)
```

That way:
* its now clear what's happening when the CPU runs at 100% for so long time (I expect that the behaviour of the "fast", Rust-based tokenizer is not dissimilar): it's just building a trie for fast tokens lookup
* I do not know whether this can be done in a more efficient way, but, currently, it's 50' that is running and it added 18675 tokens out of a total of 125936: so, it should take approx. 5 hours to load the tokenizer… How much faster it is supposed to be the "fast" tokenizer?

@SaulLu in conclusion I do think that this is issue is NOT a bug. Do you think is there some way to speed up the whole thing (apart, of course, trimming my vocabulary)? Maybe is there some mean to build the trie just once and then load it from some cache in subsequent executions (be it the slow _or_ the fast tokenizer)?

I add another bit: it's unclear to me why this issue did not arise during the pre-training. Maybe the tokenizer used during the training phase does not use the trie? Sounds a bit weird…

_Maybe_ I found a thing: after the call of `AutoTokenizer.from_pretrained()`, [in this point](https://github.com/huggingface/transformers/blob/f87db5e412f561c089e70ab2b15bfa070c2b07f5/src/transformers/tokenization_utils_base.py#L1964) of `tokenization_utils_base.py`, the code loops through each additional token: so the trie is _re-built from scratch_ for every and each token (see [here](https://github.com/huggingface/transformers/blob/f87db5e412f561c089e70ab2b15bfa070c2b07f5/src/transformers/tokenization_utils.py#L445)) - while, on the contrary, it should have been built just once with _all_ the additional tokens (that it is what I do think happen when doing the pre-training).

@SaulLu do you think I am on the right track? Do you an idea of why the trie is rebuilt for every and each additional token?

I did a "quick&dirty" patch to the code, and it seems to work: I have removed the code from [this line](https://github.com/huggingface/transformers/blob/f87db5e412f561c089e70ab2b15bfa070c2b07f5/src/transformers/tokenization_utils_base.py#L1947) to this [other line](https://github.com/huggingface/transformers/blob/f87db5e412f561c089e70ab2b15bfa070c2b07f5/src/transformers/tokenization_utils_base.py#L1964) (from my understanding, this is a kind of check that the tokenizer status is "compatible" with the addition of new tokens), and replaced it with this line:

```python
tokenizer.add_tokens([token_sorted[0] for token_sorted in added_tok_encoder_sorted])
```

That way all the 125936 tokens are added in just 121 seconds (on the basis of what I have observed during the pre-training, I do think that the "fast" tokenizer would be way faster), and the rest of the code (loading the model and computing the embeddings) seem to run flawlessy (of course I will double check it later on).

@SaulLu does this patch make sense to you too? Do you think is it possible to create a proper patch and apply it to the fast tokenizer as well?  

As a side note, I noticed that the additional tokens are written in the `tokenizer.json` file with a trailing new line character, as in

```json
    {
      "id": 31102,
      "special": false,
      "content": "urnexpltokenslegalref0\n",
      "single_word": false,
      "lstrip": false,
      "rstrip": false,
      "normalized": true
    }
```

@SaulLu  is this an expected (and harmless) behaviour, or a bug (for which, in case, I will open a different issue, of course)?

> @SaulLu does this patch make sense to you too? Do you think is it possible to create a proper patch and apply it to the fast tokenizer as well?

I have filed [an issue to in the github repo of the "fast" tokenizers](https://github.com/huggingface/tokenizers/issues/914) and I working on that now (and found another related pitfall also on the transformers side).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.