[vllm + v5 fix] handle TokenizersBackend fallback properly for v5
Author: itazap
Pull request: yes

## What this PR does

Given the different issues that were noticed by @hmellor on vLLM, we wanted to make sure we did not end up with crazy breaks. We ran a full test suite (code can be found in #44298) and the results showed 22 model converters that had _potential issues_. 

We test what would be the `AutoTokenizer.from_pretrained(model_id)._tokenizer` in v4 vs in v5, reporting diff in the `json` serialization. 

The full report is available here: 
[report.html](https://github.com/user-attachments/files/25740204/report.html)

Here is an explanation for each converter: 

### Important fixes
*  `ReformerConverter/PegasusConverter/T5Converter/MBart50Converter/XLMRobertaConverter/NllbConverter`: this is the biggest "change" we did miss the `precompiled_charsmap` and it is required for very specific whitespace cases. For all of these, we only changed the ones that had XNLI missmatches. The motivation is to have an explicit noramalizer vs something arbitrary. This is to follow the default in: https://github.com/huggingface/transformers/blob/v4.57-release/src/transformers/convert_slow_tokenizer.py#L629-L638 
* PegasusConverter note: `Sequence(pretokenizers=[WhitespaceSplit(), Metaspace(replacement="▁", prepend_scheme=always, split=True)])` the [pegasus-xsum](https://huggingface.co/google/pegasus-xsum/tree/main) model does not have `WhitespaceSplit()`, so this diff wasn't added by default to pegasus as it would break XLNI roundtrip. 
* MBart50Converter: was not usable / could not convert before so that's a big fix. We checked the expected results with v4, that's why the integration tests are changed.
* GemmaConverter: we missed a `Split` + in v5 named tokens (mask_token) are always special. This is a known change for v5 and would require relaxing the constraint that named tokens are special. Will consider this in the future.
* BertConverter: `do_lower_case=True` by default which we missed going to v5.

All of these are unigram models that need the precompiled for super super specific shit.

### Not that important, could affect small stuff
*  `BlenderbotConverter`: it had a `RobertaProcessing` but it should not 
*  `BigBirdConverter`: Should not have the strip normalizer? 

### Mild concerns fixes/ absolutely safe to ignore
*  `Qwen2Converter`: For both the `pre_tokenizer` and the `decoder`, the `trim_offsets` and `use_regex` flag are useless. https://github.com/huggingface/tokenizers/issues/1960 will address this but TLDR: behavior is the same. 
*  `TikTokenConverter`: Same as Qwen2Converter, the `decoder` flags are useless
*  `LlamaConverter`: this IS expected as most of the old llama models had the `legacy` flag with the metaspace issue (link it) prepend normalizer is shit 
*  `CLIPConverter`: the split pattern has `"<\|startoftext\|>|<\|endoftext\|>"` which we added. It does not make a difference as it's part of the special tokens, but we are removing this. 
*  `OpenAIGPTConverter`: behavior was identical but for fairness it needs `BertNormalizer`
*  `MarkupLMConverter`: seems safe to ignore in v4 it did not have RobertaProcessing either
*  `SeamlessM4TConverter`: weird 
*  `RobertaConverter`: default template <sep> is diferent + `add_prefix_space=False`
*  `GPT2Converter`: equivalent diff / safe to ignore.  `pre_tokenizer=ByteLevel(add_prefix_space=False, trim_offsets=True, use_regex=True)` is equivalent to the default Regex: `Sequence(pretokenizers=[Split(pattern=Regex("(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]|\s[\r\n]..."), behavior=Removed, invert=True), ByteLevel(add_prefix_space=False, trim_offsets=True, use_regex=False)])`  link to [tokenizers byte_level ](https://github.com/huggingface/tokenizers/blob/c4e27cfc84d7534702a8c5a7eb2677ebb2ab48a5/tokenizers/src/pre_tokenizers/byte_level.rs#L127) and [default](https://github.com/huggingface/tokenizers/blob/c4e27cfc84d7534702a8c5a7eb2677ebb2ab48a5/tokenizers/src/pre_tokenizers/byte_level.rs#L44) 
* DebertaConverter: equivalent diff / safe to ignore: TemplateProcessing is serialized in a slightly different order.

UPDATE TO: https://github.com/huggingface/transformers/pull/44179/changes (deepseek v2, internlm2)

Models with incorrect tokenizer_class in tokenization_config.json that should use TokenziersBackend

Sentencepiece /Tiktoken models:

-  we don't properly read all the tokenizer parameters from a sentencpeiece `tokenizer.model` file, so we need to extract that info and create a generic fallback that will create a `_tokenizer` object from the 


****

<details>
  <summary>Script to compare roundtrip tokenizer outputs:</summary>

```python 
#!/usr/bin/env python
import argparse
import json
import re

from datasets import load_dataset
from tqdm import tqdm

import transformers
from transformers import AutoTokenizer


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("model_name", type=str)
    parser.add_argument("version_tag", type=str)
    parser.add_argument("--num-samples", type=int, default=1000)
    parser.add_argument("--compare-to", type=str, default=None)
    args = parser.parse_args()

    safe_model = re.sub(r"[^0-9A-Za-z_.\-]+", "_", args.model_name.replace("/", "-").replace(" ", "_"))
    out_path = f"xlni_{safe_model}_{args.version_tag}"

    print(f"Transformers: {transformers.__version__}")
    print(f"Loading tokenizer: {args.model_name} (trust_remote_code=True, use_fast=True)")
    tok = AutoTokenizer.from_pretrained(args.model_name, trust_remote_code=True, use_fast=True)
    print(f"  Loaded {type(tok).__name__}")

    if hasattr(tok, "_tokenizer") and tok._tokenizer is not None:
        with open(f"{out_path}_tokenizer.txt", "w", encoding="utf-8") as f:
            f.write(str(tok._tokenizer))

    results = []

    for lang in ["ar", "bg", "de", "el", "en", "es", "fr", "hi", "ru", "sw", "th", "tr", "ur", "vi", "zh"]:
        ds = load_dataset("facebook/xnli", lang, split="validation")
        limit = min(args.num_samples, len(ds))
        for i, ex in enumerate(tqdm(ds, total=limit, desc=f"XNLI [{lang}]")):
            if i >= limit:
                break
            for field in ("premise", "hypothesis"):
                text = ex.get(field, "")
                if not text:
                    continue
                ids = tok.encode(text, add_special_tokens=False)
                decoded = tok.decode(ids, skip_special_tokens=True)
                results.append(
                    {
                        "language": lang,
                        "index": i,
                        "field": field,
                        "text": text,
                        "ids": ids,
                        "decoded": decoded,
                        "roundtrip_ok": decoded == text,
                    }
                )

    try:
        code_ds = load_dataset("code_search_net", "python", split="train")
        limit = min(args.num_samples, len(code_ds))
        for i, ex in enumerate(tqdm(code_ds, total=limit, desc="Code [code_search_net/python]")):
            if i >= limit:
                break
            for field in ("func_code_string", "func_documentation_string", "code", "docstring"):
                text = ex.get(field)
                if not text:
                    continue
                ids = tok.encode(text, add_special_tokens=False)
                decoded = tok.decode(ids, skip_special_tokens=True)
                results.append(
                    {
                        "language": "code",
                        "source": "code_search_net/python",
                        "index": i,
                        "field": field,
                        "text": text,
                        "ids": ids,
                        "decoded": decoded,
                        "roundtrip_ok": decoded == text,
                    }
                )
    except Exception as e:
        print(f"Could not load code_search_net/python: {e}")

    if args.compare_to:
        with open(args.compare_to, "r", encoding="utf-8") as f:
            prev_payload = json.load(f)
        prev_results = prev_payload.get("results", [])
        n = min(len(prev_results), len(results))
        if n:
            prev_ok = sum(1 for r in prev_results[:n] if r.get("roundtrip_ok"))
            curr_ok = sum(1 for r in results[:n] if r.get("roundtrip_ok"))
            print("\n--- Roundtrip comparison ---")
            print(f"Samples compared: {n}")
            print(f"Prev roundtrip OK: {prev_ok}/{n} ({100 * prev_ok / n:.1f}%)")
            print(f"Curr roundtrip OK: {curr_ok}/{n} ({100 * curr_ok / n:.1f}%)")
            diffs = []
            for i in range(n):
                a = prev_results[i]
                b = results[i]
                if a.get("index") != b.get("index") or a.get("field") != b.get("field"):
                    break
                if a.get("roundtrip_ok") != b.get("roundtrip_ok") or a.get("decoded") != b.get("decoded"):
                    diffs.append((a, b))
            print(f"Changed samples:   {len(diffs)}")
            if diffs:
                print("\nFirst few changed samples:")
                for a, b in diffs[:10]:
                    print(f"  #{a['index']} prev_ok={a['roundtrip_ok']} curr_ok={b['roundtrip_ok']}")
                    prev_dec = a.get("decoded") or ""
                    curr_dec = b.get("decoded") or ""
                    max_len = max(len(prev_dec), len(curr_dec))
                    pos = 0
                    while (
                        pos < max_len
                        and pos < len(prev_dec)
                        and pos < len(curr_dec)
                        and prev_dec[pos] == curr_dec[pos]
                    ):
                        pos += 1
                    radius = 40
                    start = max(0, pos - radius)
                    end = min(max_len, pos + radius)
                    prev_window = prev_dec[start:end]
                    curr_window = curr_dec[start:end]
                    marker = " " * max(0, pos - start) + "^"
                    print(f"    prev_decoded: {prev_window!r}")
                    print(f"    curr_decoded: {curr_window!r}")
                    print(f"    diff_window : {marker}")

    payload = {
        "model_name": args.model_name,
        "model_name_sanitized": safe_model,
        "transformers_version": transformers.__version__,
        "num_samples": args.num_samples,
        "num_results": len(results),
        "results": results,
    }

    with open(out_path, "w", encoding="utf-8") as f:
        json.dump(payload, f, ensure_ascii=False, indent=2)

    print(f"\nSaved {len(results)} roundtrip entries to '{out_path}'.")


if __name__ == "__main__":
    raise SystemExit(main())
```

```
(uv_env) (base) itazaporozhets@Huggingfaces-MacBook-Pro-2 transformers % python scripts/compare_tokenizers.py xlangai/OpenCUA-7B v5 --compare-to xlni_xlangai-OpenCUA-7B_v4_use_fast
/Users/itazaporozhets/Documents/Repos/transformers/uv_env/lib/python3.11/site-packages/torch/cuda/__init__.py:61: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
Transformers: 5.3.0.dev0
Loading tokenizer: xlangai/OpenCUA-7B (trust_remote_code=True, use_fast=True)
Could not extract SentencePiece model from /Users/itazaporozhets/.cache/huggingface/hub/models--xlangai--OpenCUA-7B/snapshots/a2efb7d2b104d477a4a2666a357e79550a28aafc/tiktoken.model using sentencepiece library due to Error parsing message with type 'sentencepiece.ModelProto'. Falling back to TikToken extractor.
  Loaded TokenizersBackend
XNLI languages: [ar] [bg] [de] [el] [en] [es] [fr] [hi] [ru] [sw] [th] [tr] [ur] [vi] [zh]
XNLI: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15000/15000 [00:01<00:00, 8747.38it/s]
Code [code_search_net/python]: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 2027.26it/s]

--- Roundtrip comparison ---
Samples compared: 32000
Prev roundtrip OK: 32000/32000 (100.0%)
Curr roundtrip OK: 32000/32000 (100.0%)
Changed samples:   0

Saved 32000 roundtrip entries to 'xlni_xlangai-OpenCUA-7B_v5'.
(uv_env) (base) itazaporozhets@Huggingfaces-MacBook-Pro-2 transformers % python scripts/compare_tokenizers.py internlm/internlm2-chat-7b v5 --compare-to xlni_internlm-internlm2-chat-7b_v4_use_fast
/Users/itazaporozhets/Documents/Repos/transformers/uv_env/lib/python3.11/site-packages/torch/cuda/__init__.py:61: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
Transformers: 5.3.0.dev0
Loading tokenizer: internlm/internlm2-chat-7b (trust_remote_code=True, use_fast=True)
Unrecognized keys in `rope_parameters` for 'rope_type'='dynamic': {'rope_theta'}
  Loaded TokenizersBackend
XNLI languages: [ar] [bg] [de] [el] [en] [es] [fr] [hi] [ru] [sw] [th] [tr] [ur] [vi] [zh]
XNLI: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15000/15000 [00:01<00:00, 7579.54it/s]
Code [code_search_net/python]: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 1971.22it/s]

--- Roundtrip comparison ---
Samples compared: 32000
Prev roundtrip OK: 32000/32000 (100.0%)
Curr roundtrip OK: 32000/32000 (100.0%)
Changed samples:   0

Saved 32000 roundtrip entries to 'xlni_internlm-internlm2-chat-7b_v5'.

(uv_env) (base) itazaporozhets@Huggingfaces-MacBook-Pro-2 transformers % python scripts/compare_tokenizers.py stepfun-ai/Step-3.5-Flash v5 --compare-to xlni_stepfun-ai-Step-3.5-Flash_v4_use_fast
/Users/itazaporozhets/Documents/Repos/transformers/uv_env/lib/python3.11/site-packages/torch/cuda/__init__.py:61: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
Transformers: 5.3.0.dev0
Loading tokenizer: stepfun-ai/Step-3.5-Flash (trust_remote_code=True, use_fast=True)
  Loaded TokenizersBackend
XNLI languages: [ar] [bg] [de] [el] [en] [es] [fr] [hi] [ru] [sw] [th] [tr] [ur] [vi] [zh]
XNLI: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15000/15000 [00:01<00:00, 8374.95it/s]
Code [code_search_net/python]: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 1728.44it/s]

--- Roundtrip comparison ---
Samples compared: 32000
Prev roundtrip OK: 32000/32000 (100.0%)
Curr roundtrip OK: 32000/32000 (100.0%)
Changed samples:   0

Saved 32000 roundtrip entries to 'xlni_stepfun-ai-Step-3.5-Flash_v5'.
(uv_env) (base) itazaporozhets@Huggingfaces-MacBook-Pro-2 transformers % python scripts/compare_tokenizers.py ai21labs/Jamba-tiny-dev v5 --compare-to xlni_ai21labs-Jamba-tiny-dev_v4_use_fast
/Users/itazaporozhets/Documents/Repos/transformers/uv_env/lib/python3.11/site-packages/torch/cuda/__init__.py:61: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
Transformers: 5.3.0.dev0
Loading tokenizer: ai21labs/Jamba-tiny-dev (trust_remote_code=True, use_fast=True)
  Loaded TokenizersBackend
XNLI languages: [ar] [bg] [de] [el] [en] [es] [fr] [hi] [ru] [sw] [th] [tr] [ur] [vi] [zh]
XNLI: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15000/15000 [00:01<00:00, 10206.83it/s]
Code [code_search_net/python]: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 1798.66it/s]

--- Roundtrip comparison ---
Samples compared: 32000
Prev roundtrip OK: 32000/32000 (100.0%)
Curr roundtrip OK: 32000/32000 (100.0%)
Changed samples:   0

Saved 32000 roundtrip entries to 'xlni_ai21labs-Jamba-tiny-dev_v5'.

(uv_env) (base) itazaporozhets@Huggingfaces-MacBook-Pro-2 transformers % python scripts/compare_tokenizers.py adept/fuyu-8b  v5 --compare-to  xlni_adept-fuyu-8b_v4_use_fast
/Users/itazaporozhets/Documents/Repos/transformers/uv_env/lib/python3.11/site-packages/torch/cuda/__init__.py:61: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
Transformers: 5.3.0.dev0
Loading tokenizer: adept/fuyu-8b (trust_remote_code=True, use_fast=True)
  Loaded TokenizersBackend
XNLI languages: [ar] [bg] [de] [el] [en] [es] [fr] [hi] [ru] [sw] [th] [tr] [ur] [vi] [zh]
XNLI: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15000/15000 [00:02<00:00, 5958.00it/s]
Code [code_search_net/python]: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 1367.24it/s]

--- Roundtrip comparison ---
Samples compared: 32000
Prev roundtrip OK: 32000/32000 (100.0%)
Curr roundtrip OK: 32000/32000 (100.0%)
Changed samples:   0

Saved 32000 roundtrip entries to 'xlni_adept-fuyu-8b_v5'.
(uv_env) (base) itazaporozhets@Huggingfaces-MacBook-Pro-2 transformers % python scripts/compare_tokenizers.py microsoft/Phi-3-mini-4k-instruct v5 --compare-to xlni_microsoft-Phi-3-mini-4k-instruct_v4
/Users/itazaporozhets/Documents/Repos/transformers/uv_env/lib/python3.11/site-packages/torch/cuda/__init__.py:61: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
Transformers: 5.3.0.dev0
Loading tokenizer: microsoft/Phi-3-mini-4k-instruct (trust_remote_code=True, use_fast=True)
config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 967/967 [00:00<00:00, 443kB/s]
configuration_phi3.py: 11.2kB [00:00, 5.54MB/s]
A new version of the following files was downloaded from https://huggingface.co/microsoft/Phi-3-mini-4k-instruct:
- configuration_phi3.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
  Loaded TokenizersBackend
XNLI languages: [ar] [bg] [de] [el] [en] [es] [fr] [hi] [ru] [sw] [th] [tr] [ur] [vi] [zh]
XNLI: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15000/15000 [00:01<00:00, 9145.49it/s]
Code [code_search_net/python]:   0%|                                                                                                                                            | 0/1000 [00:00<?, ?it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (4599 > 4096). Running this sequence through the model will result in indexing errors
Code [code_search_net/python]: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 1624.12it/s]

--- Roundtrip comparison ---
Samples compared: 32000
Prev roundtrip OK: 32000/32000 (100.0%)
Curr roundtrip OK: 32000/32000 (100.0%)
Changed samples:   0

Saved 32000 roundtrip entries to 'xlni_microsoft-Phi-3-mini-4k-instruct_v5'.
(uv_env) (base) itazaporozhets@Huggingfaces-MacBook-Pro-2 transformers % python scripts/compare_tokenizers.py mucai/vip-llava-7b v5 --compare-to xlni_mucai-vip-llava-7b_v4
/Users/itazaporozhets/Documents/Repos/transformers/uv_env/lib/python3.11/site-packages/torch/cuda/__init__.py:61: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
Transformers: 5.3.0.dev0
Loading tokenizer: mucai/vip-llava-7b (trust_remote_code=True, use_fast=True)
config.json: 1.09kB [00:00, 626kB/s]
  Loaded TokenizersBackend
XNLI languages: [ar] [bg] [de] [el] [en] [es] [fr] [hi] [ru] [sw] [th] [tr] [ur] [vi] [zh]
XNLI: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15000/15000 [00:01<00:00, 9549.95it/s]
Code [code_search_net/python]:   0%|                                                                                                                                            | 0/1000 [00:00<?, ?it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (4609 > 2048). Running this sequence through the model will result in indexing errors
Code [code_search_net/python]: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 1680.68it/s]

--- Roundtrip comparison ---
Samples compared: 32000
Prev roundtrip OK: 32000/32000 (100.0%)
Curr roundtrip OK: 32000/32000 (100.0%)
Changed samples:   0

Saved 32000 roundtrip entries to 'xlni_mucai-vip-llava-7b_v5'.
```

</details>

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_44255). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Also before we can merge IMO we need to make sure the auto conversion is foolproof otherwise we are ignoring code but the conversion is still not correct

run-slow: auto, gemma, lasr, llama, siglip2, t5

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/22619348206)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/auto", "models/gemma", "models/lasr", "models/llama", "models/siglip2", "models/t5"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/22619348206)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [b7d01aac](https://github.com/huggingface/transformers/commit/b7d01aacdaaff759eb8518e1115d5beaef045103) | workflow commit (merge commit) |
| PR | [ed4c16b6](https://github.com/huggingface/transformers/commit/ed4c16b6fb8853d1ed5441bfb092762bf7db3f4c) | branch commit (from PR) |
| main | [28d02a31](https://github.com/huggingface/transformers/commit/28d02a31386a4dfd85e2a91f86b7dfcebd44596d) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, bert, blenderbot, gemma, gpt_neox, lasr, llama, mbart50, nllb, openai, pegasus, reformer, siglip2, t5, xlm_roberta