[LLaMA3] 'add_bos_token=True, add_eos_token=True' seems not taking effect
Author: kiva12138
Pull request: no
Labels: name: Core: Tokenization

name: Feature request

### System Info

Platform = Windows
PyTorch = 2.3.0
Transformers = 4.41.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
import torch
from transformers import AutoTokenizer

LLaMAPath = '/path/to/llama3-8b'

# The following two yields the same results, all of them contains BOS token and no EOS token
tokenizer = AutoTokenizer.from_pretrained(LLaMAPath, add_bos_token=True, add_eos_token=True)
# tokenizer = AutoTokenizer.from_pretrained(LLaMAPath, add_bos_token=False, add_eos_token=False)

tokenizer.add_special_tokens({"pad_token": "
"}) 
inputs = tokenizer(['hi, how are you today?'], padding=True, return_tensors='pt')
print(inputs)
```

All of the statements above produce `[128000,   6151,     11,   1268,    527,    499,   3432,     30]`

### Expected behavior

I think when using `tokenizer = AutoTokenizer.from_pretrained(LLaMAPath, add_bos_token=True, add_eos_token=True)`, we get `[128000,   6151,     11,   1268,    527,    499,   3432,     30,  128001]`,

when using `tokenizer = AutoTokenizer.from_pretrained(LLaMAPath, add_bos_token=False, add_eos_token=False)`, we get `[6151,     11,   1268,    527,    499,   3432,     30]`,

Comments:
I'm having the same issue. Neither of these change the encodings:
`tokenizer.add_bos_token = False`
`tokenizer.add_eos_token = True`

cc @ArthurZucker 

Hey! This is related to #30607, the tokenizer for Llama3 is a `PreTrainedTokenizerFast`, not the `LLamaTokenizer` or a `LlamaTokenizerFast`. Though it might actually be good to support an easy way to add bos and eos. Currently what you have to do is update the `TemplateProcessor` which is fairly annoying (not beginner friendly). 

That's something which should be handle on the `tokenizers` side 

> Hey! This is related to #30607, the tokenizer for Llama3 is a `PreTrainedTokenizerFast`, not the `LLamaTokenizer` or a `LlamaTokenizerFast`. Though it might actually be good to support an easy way to add bos and eos. Currently what you have to do is update the `TemplateProcessor` which is fairly annoying (not beginner friendly).
> 
> That's something which should be handle on the `tokenizers` side

@ArthurZucker  I think it's called `TemplateProcessing`, not `TemplateProcessor`. For those wondering this is how I used it to get the tokenizer to put the eos token:

```python
bos = "
"
eos = "
"
tokenizer._tokenizer.post_processor = processors.Sequence(
    [
        processors.ByteLevel(trim_offsets=False),
        processors.TemplateProcessing(
            single=f"{bos}:0 $A:0 {eos}:0",
            pair=f"{bos}:0 $A:0 {bos}:1 $B:1 {eos}:1",
            special_tokens=[
                (bos, tokenizer.bos_token_id),
                (eos, tokenizer.eos_token_id),
            ],
        ),
    ]
)
``` 

Now I'm worried that the padding tokens won't get added properly, but that's a different issue...

Padding token is unrelated, it's added if you ask the tokenizer to pad the input! 
And yes, thanks for providing the snippet @eyloncaplan 😉 

In case anyone else is blocked by this issue, I copied code from #31316 into a [function](https://github.com/kddubey/cappr/blob/d4e2f6b65085eba223cc43a3feedf7d2e4f4132e/src/cappr/huggingface/_patch_tokenizer.py#L68) which patches the tokenizer to support dynamically setting `add_bos_token` and `add_eos_token`.

<details>
<summary>Running this script—</summary>

```python
from transformers import AutoTokenizer

model_id = "yujiepan/llama-3.1-tiny-random"

text = "a b"

print("Load plain tokenizer\n")
tokenizer = AutoTokenizer.from_pretrained(model_id)

print("   Default:", tokenizer(text)["input_ids"])

tokenizer.add_eos_token = True
print("   Add EOS:", tokenizer(text)["input_ids"])

print("\nLoad and patch tokenizer\n")
tokenizer2 = AutoTokenizer.from_pretrained(model_id)
force_support(tokenizer2)

tokenizer2.add_eos_token = True
print("  Add EOS:", tokenizer2(text)["input_ids"])

tokenizer2.add_eos_token = False
print("Don't add:", tokenizer2(text)["input_ids"])

```

</details>

—prints:

```
Load plain tokenizer

   Default: [128000, 64, 293]
   Add EOS: [128000, 64, 293]

Load and patch tokenizer

  Add EOS: [128000, 64, 293, 128009]
Don't add: [128000, 64, 293]
```

Hey @kiva12138, sorry for the very slow reply, we're catching up on the tokenization backlog.

Good news: this is fixed on current `main`. The v5 tokenizer refactor wired `add_bos_token`/`add_eos_token` (with a generic `update_post_processor`) into the base fast tokenizer, so `PreTrainedTokenizerFast` (which Llama 3 uses) now honors both flags. 
Running your exact snippet on a Llama-3 tokenizer
- `from_pretrained(..., add_bos_token=True, add_eos_token=True)` yields `[128000, ..., 30, 128001]` and
- `add_bos_token=False, add_eos_token=False` yields `[..., 30]` with no BOS

exactly what you expected; setting `tok.add_eos_token = True` after load works too, so the manual `TemplateProcessing` patch is no longer needed