[LLAMA]: LLAMA tokenizer
Author: YuanLiuuuuuu
Pull request: no

### System Info

- `transformers` version: 4.28.0.dev0
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.13.2
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```py
from transformers import LlamaTokenizer

tokenizer = LlamaTokenizer.from_pretrained(..., add_eos_token=True)
input_text = "Hello, Huggingface"
tokens = tokenizer(input_text)
# token.input_ids = [0, ..., 0]
```

### Expected behavior

at commit id ``7ade6ef7d``

the ``eos_token_id`` and ``bos_token_id`` is 0 and 0, while those from LLAMA official repo released by META is 2 and 1. 

Why there is such kind of distinction?

Comments:
cc @ArthurZucker 

Hey! This is because: 
1. You are not using the correct version of transformers:
2. You are not using the correct checkpoints
Here is the snipper I used to get the expected output:
```python 
>>> from transformers import LlamaTokenizer
>>> tokenizer = LlamaTokenizer.from_pretrained("huggyllama/llama-7b", add_eos_token= True)
>>> tokenizer.decode(tokenizer.encode("Hello", add_special_tokens = True))
'
 Hello
'
```



> Hey! This is because:
> 
> 1. You are not using the correct version of transformers:
> 2. You are not using the correct checkpoints
>    Here is the snipper I used to get the expected output:
> 
> ```python
> >>> from transformers import LlamaTokenizer
> >>> tokenizer = LlamaTokenizer.from_pretrained("huggyllama/llama-7b", add_eos_token= True)
> >>> tokenizer.decode(tokenizer.encode("Hello", add_special_tokens = True))
> '
 Hello
'
> ```

Which version of transformers do you use?

`main` but latest release also has these changes

Faced the same issue. It seems like a mismatch between transformers and llama chkt version.

It appears that in commit c0f99b4d2ec73090595914dde4c16da207e21d73, a major change has been made to llama tokenizer, so you either install an earlier version (commit 9eae4aa57650c1dbe1becd4e0979f6ad1e572ac0 or before), or convert llama weight using the latest commit. 

Or you can just use the tokenizer files from `huggyllama` and save them wherever you want 

> Faced the same issue. It seems like a mismatch between transformers and llama chkt version.
> 
> It appears that in commit [c0f99b4](https://github.com/huggingface/transformers/commit/c0f99b4d2ec73090595914dde4c16da207e21d73), a major change has been made to llama tokenizer, so you either install an earlier version (commit [9eae4aa](https://github.com/huggingface/transformers/commit/9eae4aa57650c1dbe1becd4e0979f6ad1e572ac0) or before), or convert llama weight using the latest commit.

Many thanks! Already fix it.

> Faced the same issue. It seems like a mismatch between transformers and llama chkt version.
> 
> It appears that in commit [c0f99b4](https://github.com/huggingface/transformers/commit/c0f99b4d2ec73090595914dde4c16da207e21d73), a major change has been made to llama tokenizer, so you either install an earlier version (commit [9eae4aa](https://github.com/huggingface/transformers/commit/9eae4aa57650c1dbe1becd4e0979f6ad1e572ac0) or before), or convert llama weight using the latest commit.

You have just saved my life! Debugged for days for this weird problem.

> Or you can just use the tokenizer files from `huggyllama` and save them wherever you want

Hi, how can I convert a llama model trained with older transformers (commit [9eae4aa](https://github.com/huggingface/transformers/commit/9eae4aa57650c1dbe1becd4e0979f6ad1e572ac0) or before) to be compatible with the latest transformer code?