Repository: huggingface/transformers
Pull request: Fix: use `TokenizersBackend` for Olmo3 to preserve custom `pre_tokenizer`

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

Fixes a bug where `GPT2Tokenizer.__init__` unconditionally overwrites the `pre_tokenizer` with a generic `ByteLevel`, discarding the custom `Sequence` pre_tokenizer defined in the model's `tokenizer.json`.

This causes incorrect tokenization of consecutive newlines for Olmo3 models (e.g., `\n\nhello` is split into separate tokens `['Ċ', 'Ċ', 'hello']` instead of being kept together as `['ĊĊ', 'hello']`). This subtle tokenization difference can have a significant impact on downstream tasks, especially for instruction-tuned models that rely on a chat template. For example, if the chat template appends `\n\n` after the assistant message tag to trigger the model's response, the correct tokenization is a single `ĊĊ` token -- which is what the model saw during pre-training. With the current (broken) `pre_tokenizer`, inference produces two separate `Ċ` tokens instead. Since the sequence `[Ċ, Ċ]` was never seen in that position during pre-training, the model may produce degraded or unexpected outputs at generation time.

Switching the mapping from `GPT2Tokenizer` to `TokenizersBackend` loads `tokenizer.json` directly, preserving the original `Split` + `ByteLevel(use_regex=False)` sequence.

**Affected models:** `allenai/Olmo-3-7B-Instruct` and other Olmo3-family models.

**Regression introduced in:** v5.0.0rc1

### Reproduction

```python
from transformers import AutoTokenizer
import json

model_id = "allenai/Olmo-3-7B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Check the pre_tokenizer type
backend_json = json.loads(tokenizer.backend_tokenizer.to_str())
print(json.dumps(backend_json["pre_tokenizer"], indent=2))
# Before fix: {"type": "ByteLevel", ...}
# After fix:  {"type": "Sequence", "pretokenizers": [{"type": "Split", ...}, {"type": "ByteLevel", ...}]}

# Tokenization of "\n\nhello"
pre_tokens = tokenizer._tokenizer.pre_tokenizer.pre_tokenize_str("\n\nhello")
split_words = [word for word, _ in pre_tokens]
print(f"Pre-tokens: {split_words}")
# Before fix (wrong): ['Ċ', 'Ċ', 'hello']  — newlines split individually
# After fix (correct): ['ĊĊ', 'hello']     — newlines grouped per the Split regex

tokens = tokenizer.tokenize("\n\nhello")
print(f"Tokens: {tokens}")
# Before fix (wrong): ['Ċ', 'Ċ', 'hello']
# After fix (correct): ['ĊĊ', 'hello']
```

@ArthurZucker 

Changed files:
filename: src/transformers/models/auto/tokenization_auto.py
patch: @@ -233,7 +233,7 @@
         ("nystromformer", "AlbertTokenizer" if is_tokenizers_available() else None),
         ("olmo", "GPTNeoXTokenizer" if is_tokenizers_available() else None),
         ("olmo2", "GPTNeoXTokenizer" if is_tokenizers_available() else None),
-        ("olmo3", "GPT2Tokenizer" if is_tokenizers_available() else None),
+        ("olmo3", "TokenizersBackend" if is_tokenizers_available() else None),
         ("olmoe", "GPTNeoXTokenizer" if is_tokenizers_available() else None),
         ("omdet-turbo", "CLIPTokenizer" if is_tokenizers_available() else None),
         ("oneformer", "CLIPTokenizer" if is_tokenizers_available() else None),