New LlamaTokenizer compat issues
Author: Qubitium
Pull request: no

### System Info

Cuda 11.8
Latest git transformers
tokenizers 0.13.3
pytorch 2.0
python 3.9

### Reproduction

Run trained HF llama models based on https://huggingface.co/decapoda-research/llama-7b-hf without error.

Using latest head transformer+tokenizers I am seeing:

1. Very long loading time for LlamaTokenizer with cpu pegged at 100%
2. Incorrect tokenization of trained Llama (7B tested) models that have the Lora adapter applied under old transformer/tokenizer code (from last week).

I see that there are several commits regarding LlamaTokenizer but what is the correct usage now and is this compatible with old models trained on "old" LlamaTokenizer?

For example,  ```https://huggingface.co/decapoda-research/llama-7b-hf``` tokenizer supposedly has bad default values vs original. And the new commits are supposed to resolve this. Does this mean we need to regenerate a HF compatible 7B from original META llama using latest transformer and retrain all over again? 

Just need clarifications on I can move forward using the latest transformer/tokenizer code. 


Comments:
You shouldn't use this checkpoint. They are ignoring all PRs to update the weights/configs/tokenizers to the latest fixes in Transformers (the repo was generated in the middle of the PR adding Llama so was never compatible with Hugging Face). You should indeed re-run the conversion script to be up to date, or use other checkpoints. For instance, I've found [this one](https://huggingface.co/huggyllama/llama-7b) to be fully compatible with Transformers.

Since that repo has never worked with Transformers, I cannot speak for models fine-tuned using it sadly. The tokenizer implementation has been fixed to match the original tokenizer of the researchers.

@sgugger  The new LlamaTokenizerFast (the current default) is taking tremendously amount of time to load versus old tokenizer.  Cpu is pegged at 100% and even on 5900x will take like 90s to load. Is this normal?

It won't happen if you have the fast tokenizer file. The repo I linked in my comment above has it. It's because the conversion from slow to fast tokenizer is very slow for LLaMA. As a workaround, you can also use `LlamaTokenizer` instead of `AutoTokenizer` (which will force using the slow tokenizer).

Sorry to comment on a old closed issue, but this pops out as first result when ddg'ing for "LlamaTokenizer extremely slow", so there are probably many people who will get here.

If I understand correctly, this happens when the tokenizer is in an old format, and it happens because the tokenizer is converted to the new format each time we load it, is that correct? If so, is there any way to persist the converted tokenizer, to use that the next time, instead of converting it again and again?

You can save it, then reload it from the save you did. But note that if you follow the doc and use the conversion script on the weights obtained by Llama, you will get the fast tokenizer file created for you automatically.

Thanks.

For anyone else reaching this page and wondering how to do it,  the method to use is `save_pretrained`, like this:

```
tokenizer.save_pretrained("./models/tokenizer/")
````

I can confirm loading this new version of the tokenizer fixes the slow load for me.