Llama Tokenizer Unexpectedly Producing Unknown Token
Author: rehaanahmad2013
Pull request: no
Labels: name: Core: Tokenization

### System Info

- `transformers` version: 4.29.2
- Platform: Linux-5.19.0-1027-aws-x86_64-with-glibc2.31
- Python version: 3.10.8
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)

### Who can help?

@ArthurZucker @younesbelkada I am trying to use special tokens with the LlamaTokenizer in Transformers 4.31.0 and with certain configurations of input, the tokenizer is returning a token id of 0 corresponding to the unknown token. For example, I have added the special token "<REPR_END>", and if I pass that through the tokenizer to get [1, 32003] which is good. Additionally if I pass the word "inform" through the tokenizer, I get [1, 1871], which is also good.

However, if I pass "<REPR_END>inform" through the tokenizer, I get [1, 32003, 0] which does not make sense. If I try this exact same input in Transformers 4.29.2, I get [1, 32003, 1871] which is correct.

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction
```python 
from transformers.models.llama.tokenization_llama import LlamaTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-13b-hf", use_auth_token=...)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.add_tokens(['<TARGET_BEGIN>', '<TARGET_END>', '<REPR_BEGIN>', '<REPR_END>'], special_tokens=True)

print(tokenizer("<REPR_END>inform")
```

### Expected behavior

I should expect to get the output [1, 32003, 1871] but I do not. I instead get [1, 32003, 0]

Comments:
Hey! 👋🏻  Thanks for providing a reproduction script. I suspect that you do not have `tokenizers` installed, since when I use `use_fast = True` (which is the default if you have tokenizers) the issue is not present. 
Now, this behaviour is expected: 
- Quick fix, use `legacy=True` when initialising the tokenizer: `tokenizer = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-13b-hf", legacy = True)`
- Other quick fix, use the fast tokenizer ( `pip install tokenizers`)

This is a very nice catch otherwise! The issue is that [`in`, `form`]  should be the tokenization of `inform`, but when we use the hack around sentencepiece, we actually just output [`inform`] which is not recognised as a token. Also if you pass `"<REPR_END> inform"` the extra space is automatically strip by default. This is also gonna be fixed. 

cc @Narsil I think I'll implement handling the `add_dummy_prefix = False` parameter. As pointed out somewhere else, our decoding function is also broken for Llama (it add extra spaces). 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.