Repository: huggingface/transformers
Pull request: fix(testing): Fix LayoutXLM tokenization test and LightOnOCR SDPA flash test failures on main CI

### What does this PR do?

The following failing tests were identified and fixed in this PR:

→ **LayoutXLM:** [This PR (rm slow tokenizers)](https://github.com/huggingface/transformers/pull/40936) changed [models/auto/tokenization_auto.py](https://github.com/huggingface/transformers/blob/05c0e1d390/src/transformers/models/auto/tokenization_auto.py) to map configs to single tokenizer classes instead of (slow, fast) tuples. It did touch `tests/test_tokenization_common.py`; having said that, it didn't update the `merge_model_tokenizer_mappings` fn so the [0] and [1] subscripts fail now.
→ **LightOnOCR:** [This commit (Add LightOnOCR model implementation)](https://github.com/huggingface/transformers/commit/3d276453a2) added the LightOnOCR model which internally uses [Pixtral's vision encoder](https://github.com/huggingface/transformers/blob/main/src/transformers/models/lighton_ocr/configuration_lighton_ocr.py#L80-L94). The skip list for `test_sdpa_can_dispatch_on_flash` already included "pixtral" ([added on Jun 27, 2025](https://github.com/huggingface/transformers/commit/ccf2ca162e)), but LightOnOCR was not added to the same list despite having identical behavior (internally generates a block attention mask incompatible with Flash Attention), this should fix that :)

cc: @Rocketknight1

→ **After:** Both tests correctly skip gracefully (ref screenshots). `merge_model_tokenizer_mappings` should handle V5's single-class tokenizer mapping, and `lighton_ocr` should be skipped in `test_sdpa_can_dispatch_on_flash` alongside pixtral!

**[CI Failures:](https://github.com/huggingface/transformers/actions/runs/21973341645)**

<img width="800" height="220" alt="Screenshot 2026-02-13 214801" src="https://github.com/user-attachments/assets/069747d6-7f66-4ef5-afd8-dab815490d5e" /><br>

**Before the fix (feel free to cross-check; these errors are reproducible):**

<img width="800" height="500" alt="before" src="https://github.com/user-attachments/assets/3645c593-2fcf-4fcb-9726-6c21dcfc0de9" />

**After the fix (feel free to cross-check):**

<img width="800" height="500" alt="after" src="https://github.com/user-attachments/assets/805dac11-43b9-42f0-a386-936343a26829" /><br>

### Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you fix any necessary existing tests?

Changed files:
filename: tests/test_modeling_common.py
patch: @@ -3599,6 +3599,7 @@ def test_sdpa_can_dispatch_on_flash(self):
                 "jamba",
                 "kosmos-2",
                 "mllama",
+                "lighton_ocr",
                 "pixtral",
                 "sam",
                 "sam_hq",

filename: tests/test_tokenization_common.py
patch: @@ -25,7 +25,7 @@
 from collections import OrderedDict
 from itertools import takewhile
 from pathlib import Path
-from typing import TYPE_CHECKING, Any, Union
+from typing import TYPE_CHECKING, Any
 
 from parameterized import parameterized
 
@@ -134,9 +134,9 @@ def filter_roberta_detectors(_, pretrained_name: str):
 
 def merge_model_tokenizer_mappings(
     model_mapping: dict["PretrainedConfig", "PreTrainedModel"],
-    tokenizer_mapping: dict["PretrainedConfig", tuple["PreTrainedTokenizer", "TokenizersBackend"]],
+    tokenizer_mapping: dict["PretrainedConfig", "TokenizersBackend"],
 ) -> dict[
-    Union["PreTrainedTokenizer", "TokenizersBackend"],
+    "TokenizersBackend",
     tuple["PretrainedConfig", "PreTrainedModel"],
 ]:
     configurations = list(model_mapping.keys())
@@ -145,15 +145,12 @@ def merge_model_tokenizer_mappings(
     for configuration in configurations:
         if configuration in model_mapping and configuration in tokenizer_mapping:
             model = model_mapping[configuration]
-            tokenizer = tokenizer_mapping[configuration][0]
-            tokenizer_fast = tokenizer_mapping[configuration][1]
+            tokenizer = tokenizer_mapping[configuration]
 
             if tokenizer is not None:
-                if configuration.__name__.startswith(tokenizer.__name__.replace("Tokenizer", "")):
+                name = tokenizer.__name__.replace("TokenizerFast", "").replace("Tokenizer", "")
+                if configuration.__name__.startswith(name):
                     model_tokenizer_mapping.update({tokenizer: (configuration, model)})
-            if tokenizer_fast is not None:
-                if configuration.__name__.startswith(tokenizer_fast.__name__.replace("TokenizerFast", "")):
-                    model_tokenizer_mapping.update({tokenizer_fast: (configuration, model)})
 
     return model_tokenizer_mapping