Inconsistent handling of tokenizer bos_token
Author: fxmarty-amd
Pull request: no
Labels: name: Core: Tokenization

name: bug

name: for_v5?

### System Info

```
- `transformers` version: 4.55.4
- Platform: Linux-6.8.0-78-generic-x86_64-with-glibc2.39
- Python version: 3.12.11
- Huggingface_hub version: 0.34.6
- Safetensors version: 0.6.2
- Accelerate version: 1.10.1
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.8.0+rocm6.4 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: AMD Instinct MI300X
```

### Who can help?

@ArthurZucker @itazap 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi,

I noticed that `tokenizer.add_bos_token` is an inconsistent attribute for various architectures in Transformers. Specifically, some architectures do not have an attribute `tokenizer.add_bos_token`, and they either add or don't add bos_token (e.g. `meta-llama/Llama-3.1-70B-Instruct` and `openai/gpt-oss-20b` below).

Moreover, for the tokenizers that have the attribute, I noticed that the key `add_bos_token` is not always there in the tokenizer_config.json (e.g. `HuggingFaceTB/SmolLM-135M` below).

<img width="2098" height="425" alt="Image" src="https://github.com/user-attachments/assets/5b64bb2b-b298-4988-8199-32ab57b5a048" />

This is an issue as AFAIK we are currently not able to tell from the tokenizer_config.json or from a loaded tokenizer whether a model was trained with bos tokens and whether bos tokens are effectively used by default during inference in Transformers (e.g. llama & gpt-oss inconsistency).

Related: https://github.com/EleutherAI/lm-evaluation-harness/issues/3295

The issue also exists in lighteval: https://github.com/huggingface/lighteval/blob/16318bb4cc736cfb29d72eb31bd19c55d0283565/src/lighteval/models/nanotron/nanotron_model.py#L342-L343

Thank you!

```python
model_ids = [
    "Qwen/Qwen1.5-MoE-A2.7B-Chat",
    "meta-llama/Llama-3.1-70B-Instruct",
    "HuggingFaceTB/SmolLM-135M",
    "Qwen/Qwen2.5-0.5B-Instruct",
    "microsoft/Phi-4-mini-instruct",
    "deepseek-ai/DeepSeek-R1",
    "meta-llama/Llama-2-7b-chat-hf",
    "unsloth/Llama-3.2-1B-Instruct",
    "facebook/opt-125m",
    "openai/gpt-oss-20b",
    "google/gemma-3-270m",
    "RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic",
    "NousResearch/Hermes-3-Llama-3.1-405B"
]

results = {}

for model_id in model_ids:
    results[model_id] = {}

    print("\n-----", model_id)
    tokenizer = AutoTokenizer.from_pretrained(model_id)

    prompt = "Today I am in Paris"

    inp = tokenizer(prompt, return_tensors="pt")

    results[model_id]["tokenizer.add_bos_token"] = getattr(tokenizer, "add_bos_token", "no attribute")
    print("    tokenizer.add_bos_token:", results[model_id]["tokenizer.add_bos_token"])

    results[model_id]["tokenizer.bos_token_id"] = getattr(tokenizer, "bos_token_id", "no attribute")
    print("    tokenizer.bos_token_id:", results[model_id]["tokenizer.bos_token_id"])

    filepath = huggingface_hub.hf_hub_download(model_id, "tokenizer_config.json")

    with open(filepath) as f:
        tokenizer_config = json.load(f)

    is_in_tokenizer_config = "add_bos_token" in tokenizer_config
    val = tokenizer_config.get("add_bos_token", "N/A")

    results[model_id]["'add_bos_token'` key is in tokenizer_config.json"] = is_in_tokenizer_config
    results[model_id]["tokenizer_config.json's add_bos_token value"] = val
    
    print(f"    `'add_bos_token'` key is in tokenizer_config.json: {results[model_id]["'add_bos_token'` key is in tokenizer_config.json"]} (value: {results[model_id]["tokenizer_config.json's add_bos_token value"]})")
    

    
    if hasattr(tokenizer, "bos_token_id") and tokenizer.bos_token_id is not None:
        is_added = (torch.sum(inp["input_ids"] == tokenizer.bos_token_id) > 0).item()
        print("    bos_token_id added by default:", is_added)
    else:
        is_added = False
        print("    bos_token_id added by default: False")
    
    results[model_id]["bos_token_id added by default in sequence"] = is_added
```

### Expected behavior

Consistent way to detect whether bos tokens are used by default during tokenization

Comments:
We do plan on making this available for all tokenizers! 

see #40938 😉 could not share it with you on slack so had to go and create it!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.