Repository: huggingface/transformers
Pull request: Ensure same `dtype` for subconfig when `_from_config`

# What does this PR do?

Fixes "auto" dtype when the model is initialized `from_config`

It was already fixed for `from_pretrained` in https://github.com/huggingface/transformers/pull/42990 but vLLM creates models with `AutoModel._from_config` which caused the same dtype mismatch error

Added a test as well. cc @hmellor 

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Touches core model initialization and could affect dtype override behavior for any model with `sub_configs`, though the change is narrow (dtype propagation) and covered by a targeted regression test.
> 
> **Overview**
> Fixes dtype mismatches when instantiating nested/compound models via `PreTrainedModel._from_config` by **forcing the chosen `dtype` onto all `config.sub_configs`** (mirroring `from_pretrained` behavior), ensuring modules don’t initialize with conflicting dtypes (notably for `dtype="auto"` paths used by vLLM/AutoModel).
> 
> Adds a regression test around `LlavaForConditionalGeneration._from_config` verifying that a parent `config.dtype` overrides differing sub-config dtypes and results in a consistent dtype across language/vision components and the top-level model.
> 
> <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit 5a895f48329d797c49634f83b31b1c8287c38022. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Changed files:
filename: src/transformers/modeling_utils.py
patch: @@ -1454,6 +1454,12 @@ def _from_config(cls, config, **kwargs):
         if isinstance(dtype, str):
             dtype = getattr(torch, dtype)
 
+        # Set the same `dtype` on all subconfigs to avoid dtype mismatch. When "auto" dtype
+        # with nested models, we can't dispatch different dtype per backbone module
+        for sub_config_key in config.sub_configs:
+            if (sub_config := getattr(config, sub_config_key)) is not None:
+                sub_config.dtype = dtype
+
         # If passing `attn_implementation` as kwargs, respect it (it will be applied recursively on subconfigs)
         if "attn_implementation" in kwargs:
             config._attn_implementation = kwargs.pop("attn_implementation")

filename: tests/utils/test_modeling_utils.py
patch: @@ -652,6 +652,15 @@ def test_model_from_config_dtype_composite(self):
                 TINY_LLAVA, dtype={"text_config": "float32", "vision_config": "int64", "": "float16"}
             )
 
+        # Check that `from_config` also works and uses the same dtype for all modules
+        config = AutoConfig.from_pretrained(TINY_LLAVA)
+        config.text_config.dtype = torch.float16
+        config.dtype = torch.float32
+        model = LlavaForConditionalGeneration._from_config(config)
+        self.assertEqual(model.model.language_model.dtype, torch.float32)
+        self.assertEqual(model.model.vision_tower.dtype, torch.float32)
+        self.assertEqual(model.dtype, torch.float32)
+
     def test_model_from_pretrained_dtype(self):
         # test that the model can be instantiated with dtype of either
         # 1. explicit from_pretrained's dtype argument