Route byte-level llama tokenizers to TokenizersBackend
Author: subin9
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47017)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47017)
<!-- ci-dashboard-badge:end -->

Model type "llama" spans both SentencePiece (Llama-1/2) and byte-level (Llama-3 / tiktoken) tokenizers under one Hub tokenizer_class (LlamaTokenizerFast). In v5, LlamaTokenizer.__init__ unconditionally installs a Metaspace pre-tokenizer/decoder, which silently drops spaces for byte-level repos (see #45488), e.g. deepseek-ai/DeepSeek-R1-Distill-Llama-*.

The existing MODEL_IDS_TO_TOKENIZERS_BACKEND allowlist only covers specific checkpoints (e.g. the 8B) and misses others (the 70B is still broken). Instead, for the small set of dual-scheme model types, inspect the serialized tokenizer.json: if it declares a ByteLevel pre_tokenizer/decoder, route to TokenizersBackend (which respects tokenizer.json). SentencePiece Llama-1/2 stays on LlamaTokenizer unchanged.

# What does this PR do?

`model_type == "llama"` covers two incompatible tokenizer schemes under a single Hub `tokenizer_class`
(`LlamaTokenizerFast`):

- **Llama-1/2** — SentencePiece (Metaspace, `▁`)
- **Llama-3 and its derivatives** (e.g. `deepseek-ai/DeepSeek-R1-Distill-Llama-*`) — byte-level BPE (GPT-2 / tiktoken, `Ġ`)

In v5, `LlamaTokenizer.__init__` unconditionally installs a `Metaspace` pre-tokenizer/decoder, overwriting whatever
`tokenizer.json` declares. For byte-level repos this silently drops spaces on **both encode and decode**
(`"Hello world"` → `"Helloworld"`), which is a silent accuracy regression — see #45488.

The current mitigation (`MODEL_IDS_TO_TOKENIZERS_BACKEND`, from #46091) is a per-checkpoint allowlist. It fixes
`deepseek-ai/deepseek-r1-distill-llama-8b` but misses others — for example **`DeepSeek-R1-Distill-Llama-70B` is
still broken on `main`** (it is not in the list).

Instead of enumerating checkpoints, this PR adds a small **content-based** rule: for the (rare) set of dual-scheme
model types (`{"llama"}`), inspect the serialized `tokenizer.json`; if it declares a `ByteLevel`
pre-tokenizer/decoder, route to `TokenizersBackend` (which respects `tokenizer.json`). SentencePiece Llama-1/2
(no `ByteLevel` in `tokenizer.json`) keeps using `LlamaTokenizer` unchanged. `LlamaTokenizer.__init__` is not
modified, so there is no risk to the SentencePiece path.

This covers **any** byte-level Llama repo (8B, 70B, future distills) without maintaining a checkpoint allowlist.

Fixes #45488 for Llama-3-derived checkpoints.

## Verification

Round-trip on `"Hello world.\nI'm an AI, so I don't have consciousness."` (transformers `main`):

| repo | tokenizer | before (`main`) | after |
|---|---|---|---|
| `DeepSeek-R1-Distill-Llama-70B` (not in allowlist) | byte-level | `LlamaTokenizer`, roundtrip **False** (`"Helloworld.I'manAI."`) | `TokenizersBackend`, roundtrip **True** |
| `DeepSeek-R1-Distill-Llama-8B` | byte-level | `TokenizersBackend`, True | `TokenizersBackend`, True |
| `NousResearch/Llama-2-7b-hf` | SentencePiece | `LlamaTokenizer`, True | `LlamaTokenizer`, True (unchanged) |

A no-network unit test for the detection helper is added in
`tests/models/auto/test_tokenization_auto.py::AutoTokenizerTest::test_tokenizer_json_is_byte_level`.

Comments:
cc @arthurzucker @itazap

This has been an issue since 5.3.0 I believe, would be great to get this fixed. 
Thank you @subin9 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47017). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the review @itazap! (and thanks @qflen for the redirect 🙏)

Thanks for flagging the redundant work — you're right that `_tokenizer_json_is_byte_level` fetches + fully parses `tokenizer.json`, and then `convert_to_native_format` parses it again right after. I dug into moving this to an existence check like `_has_tekken_tokenizer_file`, and here's where I landed:

**On the pure `tokenizer.model` existence check:** it's cheaper, but it's a heuristic proxy rather than an authoritative signal, so a straight swap has a couple of correctness edges:
- A byte-level Llama-3 repo that happens to ship a root `tokenizer.model` (community re-uploads, `save_pretrained` artifacts) would be misread as SentencePiece → routed to `LlamaTokenizer` → the exact space-dropping bug this PR fixes comes back.
- A SentencePiece Llama-1/2 fine-tune that only kept `tokenizer.json` (no `tokenizer.model`) would be misread as byte-level → behavior change for models that currently work.
- The current check is conservative (only diverts on *positive* byte-level detection); an absence-based check diverts much more broadly.

Note the official `meta-llama/Meta-Llama-3` repos keep `.model` under `original/`, so root existence works for them — but "Llama-3 never ships `tokenizer.model`" isn't reliably true across the ecosystem.

**What I'd propose instead** — a hybrid that keeps the authoritative decision but kills the "parse on every llama load" cost:

```python
if (
    tokenizer_auto_map is None
    and TokenizersBackend is not None
    and config_model_type in _DUAL_SCHEME_MODEL_TYPES
    and not has_file(pretrained_model_name_or_path, "tokenizer.model", **kwargs)  # SP Llama-1/2 short-circuits, no parse
    and _tokenizer_json_is_byte_level(pretrained_model_name_or_path, **kwargs)     # confirm byte-level by content
):
    return TokenizersBackend.from_pretrained(...)
```

So Llama-1/2 loads only do a cheap `has_file` HEAD and never parse, while byte-level routing stays content-authoritative. I'll match the subfolder handling from `_has_tekken_tokenizer_file`.

Separately, I can drop the double-parse by threading the already-parsed json (or resolved path) into `convert_to_native_format` if you think that's worth doing here.

Does the hybrid sound reasonable to you, or would you prefer the simpler existence check and accept the edge cases?


---

@itazap the 7 red checks look unrelated to this PR (which only touches `tokenization_auto.py`):
- `tests/cli/*` → `'HFCliTyperGroup' object has no attribute '_add_completion'`, the known `huggingface_hub` 1.22.0 breakage tracked in #47059.
- `seed_oss::test_tp_forward` → TP vs non-TP numeric flakiness (max diff 0.23).

Both came in with the main merge, not the diff — could you re-run once main is green? Thanks!


Following up on my own proposal above, @itazap — I tested the hybrid before pushing it, and it doesn't hold. The `has_file` short-circuit is an AND-gate, so it skips the content check in exactly the case the content check exists for.

The edge I described above as hypothetical turns out to be real: **`benyamini/DeepSeek-R1-Distill-Llama-8B-AWQ-w4g128`** is byte-level, ships a root `tokenizer.model`, and declares `LlamaTokenizer` — quantization pipelines copy that file over from the SentencePiece checkpoint they were derived from. Round-trip on `"Hello world. I'm an AI."`:

| | resolved class | decode |
|---|---|---|
| `main` | `LlamaTokenizer` | `"Helloworld.I'manAI."` ❌ |
| existence check only | `LlamaTokenizer` | `"Helloworld.I'manAI."` ❌ |
| hybrid (my proposal above) | `LlamaTokenizer` | `"Helloworld.I'manAI."` ❌ |
| this PR as-is | `TokenizersBackend` | ✅ |

I also mis-diagnosed the cost above: it's one Hub round-trip, not the parse. Warm cache on `DeepSeek-R1-Distill-Llama-8B` — `has_file` 197ms, `cached_file` 209ms, `json.load` 12–61ms. So the existence check saves ~5% and can't see the case above, while the hybrid is the slowest of the three (two round-trips). The one genuinely redundant piece is the second parse in `convert_to_native_format` (~60ms); happy to thread the resolved json through if you'd like that gone.

So I'd keep the routing logic as it stands. I've pushed a no-network regression test pinning the case above — it fails on `main` (`'Helloworld' != 'Hello world'`) and passes here. CI is green.


**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47017)
**Latest run:** [29201034798:2](https://github.com/huggingface/transformers/actions/runs/29201034798)
**Result:** `success` | **Jobs:** 15 | **Tests:** 170,766 | **Failures:** 0 | **Duration:** 14h 31m

<!-- ci-dashboard-recap:end -->