Repository: huggingface/transformers
Pull request: fix(models): Fix LayoutLMv2 NER crash and broken batched truncation/padding

### What does this PR do?

The following issues were identified and fixed in this PR:

→ The NER/token classification issue and the downstream bug uncovered in the batched preprocessing use case with `LayoutLMv2Tokenizer`.
→ **Reasoning:** The NER use case makes it apparent that the error is hit at [this line](https://github.com/harshaljanjani/transformers/blob/main/src/transformers/models/layoutlmv2/tokenization_layoutlmv2.py#L661) in [LayoutLMv2Tokenizer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/layoutlmv2/tokenization_layoutlmv2.py#L112), which is missing `self.only_label_first_subword`. `PretrainedTokenizerBase` doesn't create `self.only_label_first_subword` either, so this must be added along with the [other custom attributes](https://github.com/harshaljanjani/transformers/blob/main/src/transformers/models/layoutlmv2/tokenization_layoutlmv2.py#L220-L223), and this directly resolved the NER use case as shown in the screenshot.
→ For the second fix; [any padding="max_length" or truncation=True call without an explicit max_length arg compares self.model_max_length > LARGE_INTEGER (1e20)](https://github.com/harshaljanjani/transformers/blob/main/src/transformers/tokenization_utils_base.py#L2348-L2359), which in this case evaluates to `True` ([since model_max_length falls back to VERY_LARGE_INTEGER](https://github.com/harshaljanjani/transformers/blob/main/src/transformers/tokenization_utils_base.py#L129)), and both get translated into no-ops. Sequences in the batch are of different lengths and can't be tensorized, and the misleading `ValueError` tells the user to add `padding=True` and `truncation=True`, but they already did? The fix restores `model_max_length=512`. I confirmed both the [base](https://huggingface.co/microsoft/layoutlmv2-base-uncased/blob/main/config.json) and [large](https://huggingface.co/microsoft/layoutlmv2-large-uncased/blob/main/config.json) model configs have `max_position_embeddings=512`, so 512 as a default is correct, and followed the same pattern as [MarianTokenizer](https://github.com/harshaljanjani/transformers/blob/main/src/transformers/models/marian/tokenization_marian.py#L118) and [TapasTokenizer](https://github.com/harshaljanjani/transformers/blob/main/src/transformers/models/tapas/tokenization_tapas.py#L256) :)

Originally removed in #42894, just wanted to double-check if this was intentional; happy to adjust this fix if I’ve missed something :)

Fixes #44186.

**Before both fixes applied:**

<img width="500" height="500" alt="4" src="https://github.com/user-attachments/assets/fab48b55-d38c-4507-a5a9-6afd09fcdaa9" /><br>

**Attribute fix resolves NER; batched use case still fails (feel free to cross-check; the errors are reproducible):**

<img width="500" height="700" alt="5" src="https://github.com/user-attachments/assets/19e2bfd7-0c0d-4ade-8763-eb4d48872a8a" /><br>

**After both fixes applied; NER + batched use case work (feel free to cross-check):**

<img width="500" height="700" alt="6" src="https://github.com/user-attachments/assets/05c41d74-1c1b-4b0d-8e3c-bf56457291a2" /><br>

### Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you fix any necessary existing tests?

Changed files:
filename: src/transformers/models/layoutlmv2/tokenization_layoutlmv2.py
patch: @@ -175,6 +175,7 @@ def __init__(
         only_label_first_subword=True,
         tokenize_chinese_chars=True,
         strip_accents=None,
+        model_max_length=512,
         **kwargs,
     ):
         self.do_lower_case = do_lower_case
@@ -214,13 +215,15 @@ def __init__(
             only_label_first_subword=only_label_first_subword,
             tokenize_chinese_chars=tokenize_chinese_chars,
             strip_accents=strip_accents,
+            model_max_length=model_max_length,
             **kwargs,
         )
 
         self.cls_token_box = cls_token_box
         self.sep_token_box = sep_token_box
         self.pad_token_box = pad_token_box
         self.pad_token_label = pad_token_label
+        self.only_label_first_subword = only_label_first_subword
 
         # Now set post_processor with actual token IDs
         cls = str(self.cls_token)