Query on Llama2 Tokenizer Behavior During Causal LM Instruction Tuning
Author: Hannibal046
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-5.15.0-1050-azure-x86_64-with-glibc2.17
- Python version: 3.8.18
- Huggingface_hub version: 0.20.1
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import LlamaTokenizer
tokenizer = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-7b")

prompt = """Because of his beliefs he knew that if he were to kill he would suffer what?

A. shoot
B. commit crime
C. damnation
D. charming
E. think twice

The answer is: """

label = prompt + "C"

print(tokenizer(prompt,return_length=True).length)
print(tokenizer(label, return_length=True).length)
```
![image](https://github.com/huggingface/transformers/assets/38466901/d0ad322c-ecf3-419c-8fdd-616e23649f39)


### Expected behavior

Hello,

I am currently in the process of preparing data for Causal Language Model instruction tuning. As the loss calculation is solely based on the labels, it is imperative for me to accurately determine the length of the prompts in order to exclude them from the loss computation.

Upon examining the code, I observed that the Llama2 tokenizer seems to append a special empty string token at the end of each prompt. This addition results in an unchanged token length irrespective of whether a label is present or not.

Could you please clarify if this behavior is intentional or if it represents a bug? If this is indeed the expected functionality, could you suggest an alternative method or best practice for achieving the correct exclusion of prompt tokens in the loss calculation?

I appreciate your assistance on this matter.

Thank you!

Comments:
This behavior can be changed by the `add_bos_token` attribute. something like `tokenizer = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-7b", add_bos_token = False)` to de-activate the addition of the token. 
ALso `tokenizer(..., add_special_tokens = False)` de activates this as well


Hi, thanks for response! It seems that the problem is not in the special token. Even if I set `add_special_tokens=False`, it would still output the same length.
![image](https://github.com/huggingface/transformers/assets/38466901/f0555dc5-803d-4fc4-906f-58b1cd49737d)

I check the last token of `prompt`, it is an empty string rather than a space in the prompt.
![image](https://github.com/huggingface/transformers/assets/38466901/86637155-1a22-401e-9666-f4d92a77fa81)

So I guess this is the default behavior of Tokenizer?

You should use `tokenize` to check the tokens or `encode`:
```python
In [2]: tokenizer.tokenize(prompt)
Out[2]: 
[...,
 'The',
 '▁answer',
 '▁is',
 ':',
 '▁']

In [3]: tokenizer.tokenize(label)
Out[3]: 
[..., 
 'The',
 '▁answer',
 '▁is',
 ':',
 '▁C']
```
as you can see, `_` or `▁C` is a single token. This is expected. It's just not the same token. 

Thanks! Got it!