Repository: huggingface/transformers
Pull request: fix(testing): Fix PaliGemma 2 and PaddleOCR-VL test failures on main

### What does this PR do?

The following failing tests were identified and fixed in this PR:

→ **PaliGemma 2:** The [PaliGemma 1 test class](https://github.com/huggingface/transformers/blob/main/tests/models/paligemma/test_modeling_paligemma.py#L192) contains `additional_model_inputs = ["token_type_ids"]` (needed since `"token_type_ids": torch.zeros_like(input_ids)` is present in `inputs_dict`), which is missing from the PaliGemma 2 test class, even though [token_type_ids is assigned there as well](https://github.com/huggingface/transformers/blob/main/tests/models/paligemma2/test_modeling_paligemma2.py#L161); causes `ValueError`
→ **PaddleOCR-VL:** The [base offload tests](https://github.com/huggingface/transformers/blob/main/tests/test_modeling_common.py#L2951) expect the model to split across GPU and CPU. Since the PR ["Do not use accelerate hooks if the device_map has only 1 device"](https://github.com/huggingface/transformers/pull/43019), [hf_device_map is only set inside this path](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L4146-L4148). At the 70% split, the GPU budget (11.6MB) cannot fit `embed_tokens` (13.2MB), so [get_balanced_memory](https://github.com/huggingface/transformers/blob/main/src/transformers/integrations/accelerate.py#L352) places everything on the CPU and we end up with a single-device map. I think this also happens to be the same limitation handled in the [Qwen2-VL tests](https://github.com/huggingface/transformers/blob/main/tests/models/qwen2_vl/test_modeling_qwen2_vl.py#L366-L376) (albeit with a vaguely listed reason); but PaddleOCR-VL was added later in 8c84144bfc without the corresponding skips. I found the limitation by instrumenting the code as follows:

<img width="800" height="700" alt="1" src="https://github.com/user-attachments/assets/45dfe9c8-ccc2-48ca-8bb9-265d8b21f118" />

cc: @Rocketknight1

[**CI Failures**](https://github.com/huggingface/transformers/actions/runs/23126592523)

**Before the fix (feel free to cross-check; these errors are reproducible):**

<img width="800" height="700" alt="3" src="https://github.com/user-attachments/assets/fc752355-b2d2-4c32-86b8-2e8bfa547782" />

**After the fix (feel free to cross-check):**

<img width="800" height="700" alt="2" src="https://github.com/user-attachments/assets/f18daec6-cfec-4888-88be-85235e72a5bc" />

### Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you fix any necessary existing tests?

Changed files:
filename: tests/models/paddleocr_vl/test_modeling_paddleocr_vl.py
patch: @@ -184,6 +184,24 @@ def setUp(self):
     def test_config(self):
         self.config_tester.run_common_tests()
 
+    @unittest.skip(
+        reason="embed_tokens is ~80% of test model size, exceeding the 70% GPU budget so device_map puts everything on CPU"
+    )
+    def test_cpu_offload(self):
+        pass
+
+    @unittest.skip(
+        reason="embed_tokens is ~80% of test model size, exceeding the 70% GPU budget so device_map puts everything on CPU"
+    )
+    def test_disk_offload_bin(self):
+        pass
+
+    @unittest.skip(
+        reason="embed_tokens is ~80% of test model size, exceeding the 70% GPU budget so device_map puts everything on CPU"
+    )
+    def test_disk_offload_safetensors(self):
+        pass
+
     def test_mismatching_num_image_tokens(self):
         """
         Tests that an explicit error is thrown when the number of image tokens

filename: tests/models/paligemma2/test_modeling_paligemma2.py
patch: @@ -173,6 +173,7 @@ class PaliGemma2ForConditionalGenerationModelTest(ModelTesterMixin, GenerationTe
     pipeline_model_mapping = {"image-text-to-text": PaliGemmaForConditionalGeneration}
 
     _is_composite = True
+    additional_model_inputs = ["token_type_ids"]
 
     def setUp(self):
         self.model_tester = PaliGemma2VisionText2TextModelTester(self)