AutoTokenizer produces wrong token IDs for OLMo2, HyperClovaX, DeepSeek-R1-Distill-Llama, Yi, and others (v5 regression)
Author: kndtran
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 5.8.0 (also reproduced on 5.0.0 through 5.7.0)
- Platform: Linux-5.14.0-503.11.1.el9_5.x86_64-x86_64-with-glibc2.34
- Python version: 3.12.13
- Huggingface_hub version: 1.14.0
- Safetensors version: 0.7.0
- Tokenizers version: 0.22.2


### Who can help?

@ArthurZucker and @itazap

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer, PreTrainedTokenizerFast

# --- OLMo2 (GPT2Tokenizer) ---
model_id = "allenai/OLMo-2-0425-1B"

tok_v5 = AutoTokenizer.from_pretrained(model_id)                # WRONG
tok_correct = PreTrainedTokenizerFast.from_pretrained(model_id)  # CORRECT

print(tok_v5.encode("2023", add_special_tokens=False))       # [508, 1419] <- wrong
print(tok_correct.encode("2023", add_special_tokens=False))  # [2366, 18]  <- correct

print(tok_v5.encode("650841823", add_special_tokens=False))      # [13655, 5833, 972, 1419] <- wrong
print(tok_correct.encode("650841823", add_special_tokens=False))  # [13655, 25496, 23848]    <- correct

# --- DeepSeek-R1-Distill-Llama (LlamaTokenizer) ---
model_id = "deepseek-ai/DeepSeek-R1-Distill-Llama-8B"

tok_v5 = AutoTokenizer.from_pretrained(model_id)                # WRONG
tok_correct = PreTrainedTokenizerFast.from_pretrained(model_id)  # CORRECT

print(tok_v5.encode("2023", add_special_tokens=False))       # [508, 1419] <- wrong
print(tok_correct.encode("2023", add_special_tokens=False))  # [2366, 18]  <- correct

# --- HyperClovaX (GPT2Tokenizer) ---
model_id = "naver-hyperclovax/HyperCLOVAX-SEED-Think-14B"

tok_v5 = AutoTokenizer.from_pretrained(model_id)                # WRONG
tok_correct = PreTrainedTokenizerFast.from_pretrained(model_id)  # CORRECT

print(tok_v5.encode("2023", add_special_tokens=False))       # [508, 1419] <- wrong
print(tok_correct.encode("2023", add_special_tokens=False))  # [2366, 18]  <- correct
```

### Expected behavior

Follow-up to #45812 (Granite). Same root cause, additional affected model families identified.

I downloaded the tokenizer for all models in the top 1000 trending + downloads last week to check the scale of this issue. I found 62 affected models out of ~1700 unique models (that could be loaded) in the HuggingFace top 1000 downloads + trending. After existing fixes (DeepSeek V3 via #44779, DeepSeek R1-Distill-Qwen via #45741), **~30 models remain broken** with a combined **3M+ downloads**.

Again, the issue is the pretokenizer mismatch caused by the branching in AutoTokenizer. My analysis showed only the classes `GPT2Tokenizer` and `LlamaTokenizer` are affected, but potentially all tokenizer classes with a custom `__init__` may be affected.

Full list of affected models (as of last week): [affected_models.jsonl.zip](https://github.com/user-attachments/files/27652643/affected_models.jsonl.zip)

### Still broken (no fix in v5.8.0)

| model_type | v5 class | Example model | Downloads |
|---|---|---|---|
| `olmo2` | GPT2Tokenizer | `allenai/OLMo-2-0425-1B` | 83K |
| `hyperclovax` | GPT2Tokenizer | `naver-hyperclovax/HyperCLOVAX-SEED-Think-14B` | 39K |
| `granite` | GPT2Tokenizer | `ibm-granite/granite-4.1-8b` | 14K |
| `granitemoehybrid` | GPT2Tokenizer | `ibm-granite/granite-4.0-micro` | 400K |
| `llama` (DeepSeek distills) | LlamaTokenizer | `deepseek-ai/DeepSeek-R1-Distill-Llama-8B` | 1.97M |
| `llama` (Yi) | LlamaTokenizer | `01-ai/Yi-34B-Chat` | 27K |
| `ernie` | LlamaTokenizer | `baidu/ERNIE-4.5-21B-A3B-Thinking` | 35K |

### Already fixed

| model_type | Fix | PR/Issue |
|---|---|---|
| `deepseek_v3` | Added to `MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS` | #44779 (v5.3.0) |
| `deepseek_v2` | Added to `MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS` | #44779 (v5.3.0) |
| `qwen2` (DeepSeek distills) | Routed through `Qwen2Tokenizer` | #45741 (v5.8.0) |

### Problematic cases

`DeepSeek-R1-Distill-Llama-8B` and `Yi-34B-Chat` both have `model_type="llama"` and `tokenizer_class: "LlamaTokenizerFast"` in `tokenizer_config.json`. Both `TOKENIZER_MAPPING_NAMES["llama"]` (`"LlamaTokenizer"`) and the hub class (`"LlamaTokenizerFast"`) resolve to `"LlamaTokenizer"` after `.removesuffix("Fast")`, so no mismatch is detected and the override set is never consulted.

Neither mechanism can fix these models without breaking actual Llama models:
- **`MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS`**: Adding `"llama"` would force ALL Llama models to `TokenizersBackend`, breaking Meta's Llama 3/4 models that genuinely use `LlamaTokenizer`'s Metaspace pre-tokenizer.
- **`TOKENIZER_MAPPING_NAMES`**: Changing `("llama", "LlamaTokenizer")` to `("llama", "TokenizersBackend")` has the same problem, it's keyed by `model_type`, which these distill models share with unaffected Llama models.

## Related Issues

All of these are the same bug class — tokenizer class `__init__` discards `tokenizer.json`'s pre-tokenizer in v5:

| Issue | Model | Status |
|---|---|---|
| #45812 | Granite (GPT2Tokenizer) | Open, PR #45813 pending |
| #45488 | DeepSeek V3/R1 (LlamaTokenizer hardcodes Metaspace) | Open |
| #44779 | DeepSeek V3 (added to override set) | Closed, fixed in v5.3.0 |
| #45741 | DeepSeek R1-Distill-Qwen (Qwen2 mapping) | Merged in v5.8.0 |
| #44462 | deepseek-coder | Closed |
| #43122 | MiniMax | Closed |
| #45701 | camembert v2 | Closed |


Comments:
## 🔧 Solution Proposal: AutoTokenizer Wrong Token IDs (v5 Regression)

### Root Cause

This is the same root cause as [#45812](https://github.com/huggingface/transformers/issues/45812) (Granite models). The issue is in `AutoTokenizer.from_pretrained()` in `src/transformers/models/auto/tokenization_auto.py`.

When `AutoTokenizer.from_pretrained()` loads a tokenizer, it uses the `tokenizer_config.json` to determine which class to instantiate. The branching logic checks `tokenizer_class` field and instantiates the **slow** tokenizer (e.g., `GPT2Tokenizer`, `LlamaTokenizer`) instead of the **fast** one (`GPT2TokenizerFast`, `LlamaTokenizerFast`).

The slow tokenizer has a different pretokenization algorithm than the fast tokenizer. When the model was trained with the fast tokenizer's vocabulary but the slow tokenizer is loaded, the token IDs don't match, producing completely wrong outputs.

### The Bug Location

In `tokenization_auto.py`, the `AutoTokenizer.from_pretrained()` method:

1. Reads `tokenizer_config.json` from the model repo
2. Gets `tokenizer_class` field (e.g., `"GPT2Tokenizer"` for OLMo2)
3. Instantiates that class directly — **without checking if a fast variant exists**
4. The slow tokenizer's pretokenizer produces different token boundaries than the fast one

### Fix: Prefer Fast Tokenizer When Available

**File: `src/transformers/models/auto/tokenization_auto.py`**

In the `from_pretrained` method, after determining the tokenizer class from config, add a check:

```python
# After determining auto_tokenizer_class from config...

# Check if a fast variant exists and prefer it
fast_tokenizer_class = None
if not _fast_tokenizer_class:
    # Try to load the fast variant
    fast_class_name = auto_tokenizer_class.__name__ + "Fast"
    fast_module_name = auto_tokenizer_class.__module__.replace(
        f".tokenization_{auto_tokenizer_class.__name__.lower().replace('tokenizer', '')}",
        f".tokenization_{auto_tokenizer_class.__name__.lower().replace('tokenizer', '')}_fast"
    )
    try:
        fast_module = import_module(fast_module_name)
        fast_tokenizer_class = getattr(fast_module, fast_class_name, None)
    except (ImportError, AttributeError):
        pass

# If fast variant exists and user didn't explicitly request slow, use fast
if fast_tokenizer_class is not None and not kwargs.get('use_fast', False) == False:
    # Use fast tokenizer
    _fast_tokenizer_class = fast_tokenizer_class
    auto_tokenizer_class = fast_tokenizer_class
```

### Alternative: Simpler Fix — Add Fast Tokenizer Mapping

A cleaner approach is to add a mapping in `TOKENIZER_MAPPING_NAMES` that pairs slow→fast tokenizers:

```python
# In tokenization_auto.py, add after TOKENIZER_MAPPING_NAMES:
SLOW_TO_FAST_TOKENIZER_MAP = {
    "GPT2Tokenizer": "GPT2TokenizerFast",
    "LlamaTokenizer": "LlamaTokenizerFast",
    # Add more as needed
}

# In from_pretrained, after loading tokenizer_class from config:
if tokenizer_class in SLOW_TO_FAST_TOKENIZER_MAP:
    fast_class = SLOW_TO_FAST_TOKENIZER_MAP[tokenizer_class]
    try:
        # Try to import the fast variant
        tokenizer_class = fast_class
    except ImportError:
        pass  # Fall back to slow if fast not available
```

### Why This Fix Is Correct

1. **Consistency**: Models trained with fast tokenizers should always use fast tokenizers for inference
2. **Safety**: If the fast variant doesn't exist, falls back to slow (no breaking changes)
3. **Scope**: Fixes all 62 affected models (3M+ downloads) with a single change
4. **Backward compatible**: Users can still explicitly request slow tokenizer with `use_fast=False`

### Regression Test

```python
import pytest
from transformers import AutoTokenizer, PreTrainedTokenizerFast

AFFECTED_MODELS = [
    ("allenai/OLMo-2-0425-1B", "GPT2Tokenizer"),
    ("deepseek-ai/DeepSeek-R1-Distill-Llama-8B", "LlamaTokenizer"),
    ("naver-hyperclovax/HyperCLOVAX-SEED-Think-14B", "GPT2Tokenizer"),
]

@pytest.mark.parametrize("model_id,expected_slow_class", AFFECTED_MODELS)
def test_auto_tokenizer_prefers_fast(model_id, expected_slow_class):
    tok = AutoTokenizer.from_pretrained(model_id)
    # AutoTokenizer should load the fast variant, not the slow one
    assert "Fast" in type(tok).__name__, (
        f"AutoTokenizer loaded {type(tok).__name__} instead of fast variant for {model_id}"
    )
    
    # Verify token IDs match between AutoTokenizer and explicit fast tokenizer
    tok_fast = PreTrainedTokenizerFast.from_pretrained(model_id)
    test_text = "2023"
    assert tok.encode(test_text, add_special_tokens=False) == tok_fast.encode(test_text, add_special_tokens=False), (
        f"Token ID mismatch for {model_id}"
    )
```

### Impact

- Fixes 62 broken models with 3M+ combined downloads
- No breaking changes — purely additive safety check
- Resolves the v5 regression for all affected model families (OLMo2, HyperClovaX, DeepSeek-R1-Distill-Llama, Yi, Granite, etc.)

Opening a PR for a wider net of this! will link it here today

Clean implementation and easy to follow

On Wed, May 13, 2026, 2:10 AM Ita Zaporozhets ***@***.***>
wrote:

> *itazap* left a comment (huggingface/transformers#45920)
> <https://github.com/huggingface/transformers/issues/45920#issuecomment-4438288885>
>
> Opening a PR for a wider net of this! will link it here today
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/45920#issuecomment-4438288885>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/CC2Z7226ZAEB7YF243KSN4D42QNXNAVCNFSM6AAAAACY3GUMPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DIMZYGI4DQOBYGU>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.