Different tokenization with same tokenizer from 4.57.3 to 5.0
Author: awni
Pull request: no
Labels: name: Fast Tokenizers

name: bug

### System Info

Moving from transformers 4.57.3 to 5.0+ introduces a different and seemingly incorrect tokenization when using the same tokenizer.

I believe the new version is incorrect because when using it, we get bad results (the model starts to introduce unexpected artifacts in the response).

### Who can help?

@ArthurZucker 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Run the following with the two versions and compare the tokenized prompt.

```python
from transformers import AutoTokenizer                                         
                                                                               
tokenizer = AutoTokenizer.from_pretrained("mlx-community/MiniMax-M2.1-4bit")   
messages = [                                                                   
    {                                                                          
        "role": "system",                                                      
        "content": '"You are opencode, an interactive CLI tool that helps users with software engineering tasks. Use the instructions below and the tools available to you to assist the user.\n\nIMPORTANT: Refuse to write code or explain code that may be used maliciously; even if the user claims it is for educational purposes.'
    }
]
    
prompt = tokenizer.apply_chat_template(                                        
    messages, add_generation_prompt=True, tokenize=True, return_dict=False     
)
print(prompt) 
```

### Expected behavior

They should be the same.

Comments:
```python
In [12]: print(tokenizer.convert_ids_to_tokens(prompt))
[']~!b[', ']~b]', 'system', 'Ċ', '"You', 'Ġare', 'Ġopen', 'code', ',', 'Ġan', 'Ġinteractive', 'ĠCLI', 'Ġtool', 'Ġthat', 'Ġhelps', 'Ġusers', 'Ġwith', 'Ġsoftware', 'Ġengineering', 'Ġtasks', '.', 'ĠUse', 'Ġthe', 'Ġinstructions', 'Ġbelow', 'Ġand', 'Ġthe', 'Ġtools', 'Ġavailable', 'Ġto', 'Ġyou', 'Ġto', 'Ġassist', 'Ġthe', 'Ġuser', '.ĊĊ', 'IMPORTANT', ':', 'ĠRef', 'use', 'Ġto', 'Ġwrite', 'Ġcode', 'Ġor', 'Ġexplain', 'Ġcode', 'Ġthat', 'Ġmay', 'Ġbe', 'Ġused', 'Ġmalicious', 'ly', ';', 'Ġeven', 'Ġif', 'Ġthe', 'Ġuser', 'Ġclaims', 'Ġit', 'Ġis', 'Ġfor', 'Ġeducational', 'Ġpurposes', '.', '[e~[', 'Ċ', ']~b]', 'ai', 'Ċ', '<think>', 'Ċ']

In [13]: print(tokenizer.convert_ids_to_tokens(b))
[']~!b[', ']~b]', 'system', 'Ċ', '"', 'You', 'Ġare', 'Ġopen', 'code', ',', 'Ġan', 'Ġinteractive', 'ĠCLI', 'Ġtool', 'Ġthat', 'Ġhelps', 'Ġusers', 'Ġwith', 'Ġsoftware', 'Ġengineering', 'Ġtasks', '.', 'ĠUse', 'Ġthe', 'Ġinstructions', 'Ġbelow', 'Ġand', 'Ġthe', 'Ġtools', 'Ġavailable', 'Ġto', 'Ġyou', 'Ġto', 'Ġassist', 'Ġthe', 'Ġuser', '.', 'Ċ', 'Ċ', 'IMPORTANT', ':', 'ĠRef', 'use', 'Ġto', 'Ġwrite', 'Ġcode', 'Ġor', 'Ġexplain', 'Ġcode', 'Ġthat', 'Ġmay', 'Ġbe', 'Ġused', 'Ġmalicious', 'ly', ';', 'Ġeven', 'Ġif', 'Ġthe', 'Ġuser', 'Ġclaims', 'Ġit', 'Ġis', 'Ġfor', 'Ġeducational', 'Ġpurposes', '.', '[e~[', 'Ċ', ']~b]', 'ai', 'Ċ', '<think>', 'Ċ']
```
I think what helps more is if you show token to token comparison. 
In that case it seems `'"You'` is now split into `'"', 'You'`. 

Usually this means `pre_tokenizer` / `normalizer` issues. 

```python
# v4.57.3
In [17]: tokenizer._tokenizer.pre_tokenizer
Out[17]: Sequence(pretokenizers=[Split(pattern=Regex("[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+"), behavior=Removed, invert=True), ByteLevel(add_prefix_space=False, trim_offsets=True, use_regex=False)])

In [18]: tokenizer._tokenizer.normalizer
Out[18]: NFC()
```
vs
```python
# main
In [2]: tokenizer._tokenizer.normalizer

In [3]: tokenizer._tokenizer.pre_tokenizer
Out[3]: ByteLevel(add_prefix_space=False, trim_offsets=True, use_regex=True)
```

So I can reproduce the issue. 
If you just do:
```python
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("mlx-community/MiniMax-M2.1-4bit")
```

We realize the `tokenizer.json` is absolutely correct, and the issue is that the class they used is `GPT2Tokenizer` when it should be `PreTrainedTokenizerFast` in the `tokenizer_config.json`.... 

cc @itazap I am afraid MANY other repos have this bug




https://huggingface.co/mlx-community/MiniMax-M2.1-4bit/discussions/3 is the solution

#42894 will also get the fix, in this cas the minimax was mapped to GPT2, it should not