Incorrect tokenization `tokenizers` for escaped strings / Mismatch with `mistral_common`
Author: patrickvonplaten
Pull request: no
Labels: name: bug

### System Info

```
In [3]: mistral_common.__version__
Out[3]: '1.8.6'
```

```
In [4]: import transformers; transformers.__version__
Out[4]: '5.0.0.dev0'
```

```
In [5]: import tokenizers; tokenizers.__version__
Out[5]: '0.22.1'
```

### Who can help?

@ArthurZucker @itazap 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```py
from transformers import AutoTokenizer
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_common.protocol.instruct.request import ChatCompletionRequest

req = ChatCompletionRequest(messages=[
    {'role': 'system', 'content': ''},
    {'role': 'user', 'content': 'hey'},
    {'role': 'assistant', 'content': 'ju\x16'},
    {'role': 'user', 'content': 'hey'},
])

tokenizer_orig = MistralTokenizer.from_hf_hub("mistralai/Ministral-3-3B-Instruct-2512")
tokenizer_hf = AutoTokenizer.from_pretrained("mistralai/Ministral-3-3B-Instruct-2512")

orig_tokens = tokenizer_orig.encode_chat_completion(req).tokens
orig_text = tokenizer_orig.encode_chat_completion(req).text

print("Expected")
print(orig_text)
print(orig_tokens)

hf_tokens = tokenizer_hf.apply_chat_template(req.to_openai()["messages"])
hf_text = tokenizer_hf.convert_ids_to_tokens(hf_tokens)

print("HF")
print(hf_tokens)
print(hf_text)
```

gives:

```
Expected

[SYSTEM_PROMPT][/SYSTEM_PROMPT]
hey
ju

hey

[1, 17, 18, 3, 74058, 4, 5517, 1022, 2, 3, 74058, 4]
HF
[1, 17, 18, 3, 74058, 4, 5517, 1022, 1032, 2, 3, 74058, 4]
['
', '[SYSTEM_PROMPT]', '[/SYSTEM_PROMPT]', '
', 'hey', '
', 'ju', 'Ė', 'Ġ', '
', '
', 'hey', '
']
```

As you can see the token `1032` should not be there. I'm not sure exactly what is happening and it could very well be that the behavior of `tokenizers` makes sense here. 

**However**, this is a mismatch with `mistral_common` which means that any such tokenization will give slightly different token ids leading to slightly incorrect results since all Mistral models are trained with `mistral_common`.

This is especially important for "long-log" parsing tasks that often have escaped strings.

It's def an edge case, but would still be very nice to fix.

### Expected behavior

Align encoding.

Comments:
I think it boils down to `mistral_common` stripping invalid / unsafe bytes when applying chat completion rules, where as `tokenizers` treats unknown byte-sequences as valid byte-level tokens.

Ah okay I'll check it can probably be mitigated with `byte_fallback=False`! Let me check

```python
tokenizer_hf._tokenizer.decoder = decoders.Sequence([decoders.BPEDecoder(), decoders.ByteLevel(add_prefix_space=True, trim_offsets=True, use_regex=True), decoders.Fuse()])
```
makes the decoded text the same and I have the same input tokens
```python
Expected

[SYSTEM_PROMPT][/SYSTEM_PROMPT]
hey
ju

hey

[1, 17, 18, 3, 74058, 4, 5517, 1022, 2, 3, 74058, 4]
HF
[1, 17, 18, 3, 74058, 4, 5517, 1022, 2, 3, 74058, 4]

[SYSTEM_PROMPT][/SYSTEM_PROMPT]
hey
ju

hey

```

I am running this on #42563  but it should not change much 

This is fixed so closing