Loading local non-Mistral tokenizer incorrectly trigger fix_mistral_regex warning.
Author: tohz
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.57.3
- Platform: Linux-6.8.0-87-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 0.36.0
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.9.1+cu128 (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Loading local non-Mistral tokenizer from a directory that **also contains model files** will incorrectly trigger `fix_mistral_regex` warning.

### Code
```python3
from transformers import AutoTokenizer
from transformers import AutoModelForSequenceClassification


def load_tokenizer(model_id):
    tokenizer = AutoTokenizer.from_pretrained(model_id)

    return tokenizer


def load_model(model_path):
    model = AutoModelForSequenceClassification.from_pretrained(
        model_path, device_map="auto"
    )

    return model


model_id = "BAAI/bge-reranker-v2-m3"

output_dir1 = "tmp/output-1"

tok = load_tokenizer(model_id)
tok.save_pretrained(output_dir1)

# output_dir1 contains only tokenizer files.

# Following line DOES NOT trigger fix_mistral_regex=True warning.
tok2 = load_tokenizer(output_dir1)

print(tok2)

print("-----")

output_dir2 = "tmp/output-2"

tok3 = load_tokenizer(model_id)
tok3.save_pretrained(output_dir2)

model = load_model(model_id)
model.save_pretrained(output_dir2)

# output_dir2 contains both tokenizer and model files.

# Following line DOES trigger fix_mistral_regex=True warning.
tok4 = load_tokenizer(output_dir2)

print(tok4)

```

### Output

```
XLMRobertaTokenizerFast(name_or_path='tmp/output-1', vocab_size=250002, model_max_length=8192, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '
', 'eos_token': '
', 'unk_token': '<unk>', 'sep_token': '
', 'pad_token': '<pad>', 'cls_token': '
', 'mask_token': '<mask>'}, clean_up_tokenization_spaces=True, added_tokens_decoder={
	0: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	1: AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	2: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	3: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	250001: AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=False, special=True),
}
)
-----
The tokenizer you are loading from 'tmp/output-2' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
XLMRobertaTokenizerFast(name_or_path='tmp/output-2', vocab_size=250002, model_max_length=8192, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '
', 'eos_token': '
', 'unk_token': '<unk>', 'sep_token': '
', 'pad_token': '<pad>', 'cls_token': '
', 'mask_token': '<mask>'}, clean_up_tokenization_spaces=True, added_tokens_decoder={
	0: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	1: AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	2: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	3: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	250001: AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=False, special=True),
}
)
```


### Expected behavior

The `fix_mistral_regex` warning should not be triggered for non-Mistral tokenizers.

Comments:
cc @itazap @arthurzucker

Hey @tohz 👋 

the issue is essentially due to this check here https://github.com/huggingface/transformers/blob/47b0e478f324b54f177ea7998a0791870fdd0324/src/transformers/tokenization_utils_base.py#L2475
This should be changed if we patch it. v5rc0 should work.

The warning is triggered because you save the tokenizer on v 4.57.3 which is not handled within the version checks. 

If i use transformers==4.57.3，should i  change tokenization_utils_base to  >= version.parse("4.57.3”) ？ 

`version.parse(transformers_version) < version.parse("5.0.0")` would make more sense in this case (in case we have any more patches for v4).

I’m fine-tuning **InternVL3_5-1B** (which uses a Qwen3-based LLM) with `transformers==4.57.3` and encountered the following warning:

> *The tokenizer you are loading from '...' has an incorrect regex pattern [...] You should set `fix_mistral_regex=True` to fix this issue.*

To investigate whether this actually affects tokenization, I wrote a script that compares the `encode`/`decode` behavior with and without `fix_mistral_regex=True` using a representative caption text (e.g., detaile image descriptions containing contractions, possessives, and punctuation).

Surprisingly, **the token IDs, decoded text, and cross-compatibility are identical in both cases**—no difference was observed in:
- Token count
- Token sequence
- Round-trip decode fidelity (`decode(encode(text)) == text`)


**My questions:**
- Should I still explicitly set `fix_mistral_regex=True` just to suppress the warning?
- Is it safe to **pre-save a fixed tokenizer** using:
```
# fix_tokenizer_once_and_forever.py
from transformers import AutoTokenizer
from pathlib import Path
import os

model_path = "/data/code/haobang.geng/models/InternVL3_5-1B"
fix_model_path = "/data/code/haobang.geng/models/InternVL3_5-1B-fixed"

print("=" * 80)
print("比较 fix_mistral_regex=True 和 False 时的 encode/decode 结果")
print("=" * 80)

test_text = "Photograph of a woman with long dark brown hair, partially visible from the neck down. She is wearing a tight, white Adidas short-sleeve T-shirt, which is slightly sheer, revealing the outline of her large breasts beneath. Her right hand, with red-painted nails, is placed over her left breast, partially covering it. The Adidas logo is visible on the upper left side of the shirt. The background is a plain, light-colored wall. The image focuses on her chest and hand, with soft lighting highlighting the textures of the fabric and her skin. The overall tone is intimate and casual."

print(f"\n测试文本长度: {len(test_text)} 字符")
print(f"测试文本预览: {test_text[:100]}...\n")

# 加载不带 fix_mistral_regex 的 tokenizer（默认行为）
print("1. 加载 tokenizer (fix_mistral_regex=False/默认)...")
tokenizer_no_fix = AutoTokenizer.from_pretrained(
    model_path,
    trust_remote_code=True,
    # 不设置 fix_mistral_regex，使用默认值
)

# 加载带 fix_mistral_regex 的 tokenizer
print("2. 加载 tokenizer (fix_mistral_regex=True)...")
tokenizer_with_fix = AutoTokenizer.from_pretrained(
    model_path,
    trust_remote_code=True,
    fix_mistral_regex=True,  # 关键！修复 regex 问题
)

print("\n" + "=" * 80)
print("开始比较 encode 结果...")
print("=" * 80)

# Encode 比较
no_fix_input_ids = tokenizer_no_fix.encode(test_text)
with_fix_input_ids = tokenizer_with_fix.encode(test_text)

print(f"\n不带 fix_mistral_regex 的 encode 结果:")
print(f"  Token 数量: {len(no_fix_input_ids)}")
print(f"  前20个 tokens: {no_fix_input_ids[:20]}")
print(f"  后20个 tokens: {no_fix_input_ids[-20:]}")

print(f"\n带 fix_mistral_regex=True 的 encode 结果:")
print(f"  Token 数量: {len(with_fix_input_ids)}")
print(f"  前20个 tokens: {with_fix_input_ids[:20]}")
print(f"  后20个 tokens: {with_fix_input_ids[-20:]}")

# 比较 encode 结果
encode_same = no_fix_input_ids == with_fix_input_ids
print(f"\n✓ Encode 结果是否完全一致: {encode_same}")

if not encode_same:
    print("\n⚠️  Encode 结果不一致！差异分析:")
    if len(no_fix_input_ids) != len(with_fix_input_ids):
        print(f"  - Token 数量不同: {len(no_fix_input_ids)} vs {len(with_fix_input_ids)}")
    else:
        diff_indices = [i for i, (a, b) in enumerate(zip(no_fix_input_ids, with_fix_input_ids)) if a != b]
        print(f"  - 有 {len(diff_indices)} 个位置的 token 不同")
        if len(diff_indices) <= 10:
            print(f"  - 不同位置: {diff_indices}")
            for idx in diff_indices[:5]:  # 只显示前5个
                print(f"    位置 {idx}: {no_fix_input_ids[idx]} vs {with_fix_input_ids[idx]}")

print("\n" + "=" * 80)
print("开始比较 decode 结果...")
print("=" * 80)

# Decode 比较
no_fix_decoded = tokenizer_no_fix.decode(no_fix_input_ids)
with_fix_decoded = tokenizer_with_fix.decode(with_fix_input_ids)

print(f"\n不带 fix_mistral_regex 的 decode 结果:")
print(f"  长度: {len(no_fix_decoded)} 字符")
print(f"  内容预览: {no_fix_decoded[:150]}...")

print(f"\n带 fix_mistral_regex=True 的 decode 结果:")
print(f"  长度: {len(with_fix_decoded)} 字符")
print(f"  内容预览: {with_fix_decoded[:150]}...")

# 比较 decode 结果
decode_same = no_fix_decoded == with_fix_decoded
print(f"\n✓ Decode 结果是否完全一致: {decode_same}")

if not decode_same:
    print("\n⚠️  Decode 结果不一致！")
    print(f"  原始文本长度: {len(test_text)}")
    print(f"  不带 fix 解码长度: {len(no_fix_decoded)}")
    print(f"  带 fix 解码长度: {len(with_fix_decoded)}")
    
    # 找出第一个不同的字符位置
    min_len = min(len(no_fix_decoded), len(with_fix_decoded))
    for i in range(min_len):
        if no_fix_decoded[i] != with_fix_decoded[i]:
            print(f"\n  第一个不同位置: {i}")
            print(f"  不带 fix: '{no_fix_decoded[max(0, i-20):i+20]}'")
            print(f"  带 fix:   '{with_fix_decoded[max(0, i-20):i+20]}'")
            break
    else:
        if len(no_fix_decoded) != len(with_fix_decoded):
            print(f"\n  长度不同，但前 {min_len} 个字符相同")

# 交叉 decode 测试（用对方的 encode 结果 decode）
print("\n" + "=" * 80)
print("交叉 decode 测试（用对方的 encode 结果 decode）...")
print("=" * 80)

cross_decode_1 = tokenizer_no_fix.decode(with_fix_input_ids)
cross_decode_2 = tokenizer_with_fix.decode(no_fix_input_ids)

print(f"\n用 fix=True 的 encode 结果，用 fix=False 的 tokenizer decode:")
print(f"  结果: {cross_decode_1[:150]}...")
print(f"  是否等于原始文本: {cross_decode_1 == test_text}")

print(f"\n用 fix=False 的 encode 结果，用 fix=True 的 tokenizer decode:")
print(f"  结果: {cross_decode_2[:150]}...")
print(f"  是否等于原始文本: {cross_decode_2 == test_text}")

# 最终结论
print("\n" + "=" * 80)
print("最终结论")
print("=" * 80)
print(f"1. Encode 结果一致: {encode_same}")
print(f"2. Decode 结果一致: {decode_same}")
print(f"3. 原始文本 = 不带 fix 的 decode: {test_text == no_fix_decoded}")
print(f"4. 原始文本 = 带 fix 的 decode: {test_text == with_fix_decoded}")

if encode_same and decode_same:
    print("\n✅ 结论: fix_mistral_regex=True 和 False 的 encode/decode 结果完全一致")
    print("   可能不需要使用 fix_mistral_regex=True")
else:
    print("\n⚠️  结论: fix_mistral_regex=True 和 False 的 encode/decode 结果不一致")
    if test_text == with_fix_decoded and test_text != no_fix_decoded:
        print("   ✓ 使用 fix_mistral_regex=True 可以正确还原原始文本")
        print("   ✗ 不使用 fix_mistral_regex 会导致文本还原错误")
    elif test_text == no_fix_decoded and test_text != with_fix_decoded:
        print("   ✗ 使用 fix_mistral_regex=True 反而导致文本还原错误")
        print("   ✓ 不使用 fix_mistral_regex 可以正确还原原始文本")
    else:
        print("   ⚠️  两种情况都无法完全还原原始文本")

print("\n" + "=" * 80)

  ```
- Given that InternVL is multimodal and not a pure Mistral model, is this warning perhaps a **false positive** triggered by overly broad detection logic in `transformers`?