LlamaTokenizer in v5 overrides tokenizer.json's ByteLevel pre-tokenizer with Metaspace, silently breaks DeepSeek V3/R1 family
Author: dc3671
Pull request: no

### System info

- `transformers`: 5.3.0
- `tokenizers`: 0.22.2
- Python: 3.12 / Linux

### Who can help?

@ArthurZucker

### Reproduction

Tokenizer-only, ~7 MB download:

```python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V3")
print(repr(tok.decode(tok.encode("hello world", add_special_tokens=False))))
# 'helloworld'  — expected 'hello world'
```

### Expected vs actual

| | v4.x / raw `tokenizers.Tokenizer.from_file` | v5.x `AutoTokenizer` |
|---|---|---|
| round-trip of `"hello world"` | `"hello world"` | `"helloworld"` |
| `backend_tokenizer.pre_tokenizer` | `Sequence([..., ByteLevel])` (from `tokenizer.json`) | `Metaspace(replacement="▁", split=False)` |

### Root cause

DeepSeek's `tokenizer_config.json` sets `"tokenizer_class": "LlamaTokenizerFast"`, but `tokenizer.json` is ByteLevel BPE (GPT-2 style, not SentencePiece). In v5, `LlamaTokenizer.__init__` unconditionally installs a `Metaspace` pre-tokenizer, overwriting whatever `tokenizer.json` declared. Since the vocab has no `▁`-prefixed tokens, the metaspace marker drops and spaces vanish.

Loading the same file via `tokenizers.Tokenizer.from_file(...)` or `PreTrainedTokenizerFast.from_pretrained(...)` works — the bug is specifically in `LlamaTokenizer`'s `__init__`.

### Affected models

All models shipping `tokenizer_class: Llama*Tokenizer` + a ByteLevel `tokenizer.json`. Confirmed broken on all 5 DeepSeek V3/R1 upstream repos:
`DeepSeek-V3`, `DeepSeek-V3.1`, `DeepSeek-V3.2-Exp`, `DeepSeek-R1`, `DeepSeek-R1-0528`.

Note: `LlamaTokenizer is LlamaTokenizerFast` in v5, so either class name in `tokenizer_config.json` triggers the bug.

### Downstream impact

Catastrophic silent accuracy loss on any HF-based inference stack. For example, GSM8K on DeepSeek-V3-Lite drops ~63.7% → ~26.1% purely from this tokenizer regression.

### Suggested fix

In `LlamaTokenizer.__init__`, only install the default `Metaspace` pre-tokenizer when `tokenizer.json` is absent (the "train from scratch" path). When loading an existing `tokenizer.json`, trust its `pre_tokenizer`.

### Workaround

```python
from transformers import PreTrainedTokenizerFast
tok = PreTrainedTokenizerFast.from_pretrained("deepseek-ai/DeepSeek-V3")
```


Comments:
cc @arthurzucker @itazap

hey @dc3671 it is fixed in the latest version! We load from `TokenziersBackend` (aka `PreTrainedTokenizerFast`) for deepseek models: https://github.com/huggingface/transformers/blob/4f302caf4c8186c0352cf82ed1306c786928ee41/src/transformers/models/auto/tokenization_auto.py#L350-L354


@itazap thanks~ maybe you can link the PR here and close the issue.

https://github.com/huggingface/transformers/pull/44801 🤗 

Hello，the problem persists even after I updated the transformers using the method below. 
pip install git+https://github.com/huggingface/transformers.git@main

@myym0 can you provide a reproducer please?

This appears to be broken again after https://github.com/huggingface/transformers/pull/45078

Repro in sglang: https://github.com/sgl-project/sglang/issues/25315

Other transformers issue for this same bug: #45812 #45920

```python
Testing model: deepseek-ai/DeepSeek-V3
Transformers version: 5.8.0.dev0
'hello world'
```

v3 is working on main

@itazap Does HF Transformers have any CI to cover this? It feels that this issue has showed up many times...

@trevor-m could you check if the main branch is okay now? thanks!

Sorry for the hassle! ☹️ 

yep we test v3 in `tests/models/deepseek_v3/test_modeling_deepseek_v3.py` and `deepseek-ai/DeepSeek-R1` in `tests/test_tokenizers_backend_mixin.py`. However, some checkpoints like `https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B` incorrectly map to Llama. So we have to add a workaround or fix the config (opened a hub PR here: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B/discussions/36) and workaround PR here https://github.com/huggingface/transformers/pull/45936) 

> [@myym0](https://github.com/myym0) can you provide a reproducer please?

Sorry for the late reply. I updated transformers to the latest version (5.8.1), but the issue still occurs.

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V3")
print(repr(tok.decode(tok.encode("hello world", add_special_tokens=False))))
# 'helloworld'  — expected 'hello world'

I tested the main branch today (https://github.com/huggingface/transformers/commit/b797f0d4c74ded2e368bbc971c4aaca9004318cc) and it appears to be fixed: 
```
python3 -c "from transformers import AutoTokenizer; tok = AutoTokenizer.from_pretrained('deepseek-ai/DeepSeek-R1-0528'); print(repr(tok.decode(tok.encode('hello world', add_special_tokens=False))))"
'hello world'
```

I tested a few more versions:
5.6.0: bad
5.6.1: bad
5.6.2: bad
5.7.0: good
5.8.0: good
5.8.1: good

@myym0 Not sure why we got different results


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

*nudge* This is still causing tokenizer issues (and thus breaking conversion in `llama.cpp`, forcing us to downgrade to 4.x) with various models.

sorry for the wait ! https://github.com/huggingface/transformers/pull/46091 will be merged ASAP to fix, will comment soon when it is

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Confirming this also affects **DeepSeek-R1-Distill-Llama-70B** (both the original `deepseek-ai/DeepSeek-R1-Distill-Llama-70B` and the FP8-dynamic quantized variant `neuralmagic/DeepSeek-R1-Distill-Llama-70B-FP8-dynamic`), served through vLLM 0.25.0.

**Environment**
- Platform: Linux aarch64 (NVIDIA DGX Spark, GB10)
- transformers: 5.13.0
- tokenizers: 0.22.2 (also reproduced on 0.22.0 and 0.22.1)
- vLLM: 0.25.0

**Minimal repro (bypassing vLLM entirely, same pattern as the original report):**
```python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained('deepseek-ai/DeepSeek-R1-Distill-Llama-70B')
ids = tok.encode('Ciao! Come stai oggi?')
print(repr(tok.decode(ids)))
# Expected: 'Ciao! Come stai oggi?'
# Actual:   'Ciao!Comestaioggi?'
```

`convert_ids_to_tokens` confirms the byte-level BPE tokens themselves are correct (`Ġ` markers present as expected), so tokenization/encoding is fine — the bug is isolated to the decode step, consistent with the root cause described above.

**Downstream symptom via vLLM serving:** rather than spaces disappearing outright, the OpenAI-compatible `/v1/chat/completions` endpoint (both streaming and non-streaming) returns the raw byte-level markers literally in the `content`/`reasoning` fields instead of either a space or nothing, e.g.:
```
"content": "\n\nOkay,ĠtheĠuserĠgreetedĠme..."
```
i.e. `Ġ`/`Ċ` leak through as literal characters rather than being silently dropped. Possibly vLLM's incremental detokenizer partially compensates for the broken decoder in a way that surfaces the raw markers instead of collapsing them — but the root cause traces back to the same `LlamaTokenizer.__init__` Metaspace override described in this issue. Downgrading `tokenizers` (0.22.0/0.22.1/0.22.2) did not help, which lines up with this being a `transformers` v5 issue rather than a `tokenizers` version regression.

Ruled out before landing here: vLLM's V2 model runner, `--tokenizer-mode slow`, pointing `--tokenizer` explicitly at the non-quantized source repo, the reasoning parser, and streaming vs. non-streaming — all show the identical symptom, and the `tokenizer.json`'s `decoder`/`pre_tokenizer` fields are standard/intact (`ByteLevel`, correctly configured), which matches this issue's finding that the file itself isn't the problem.

Happy to test a patched build if one becomes available — this is blocking precision-sensitive use (financial/legal document processing) where we can't rely on a client-side workaround indefinitely.

fix has been merged into main!