`dtype` is silently ignored when loading composite (multimodal) checkpoints through AutoModelForCausalLM (affects all Qwen3.5 repos)
Author: qflen
Pull request: no

### System Info

- `transformers` version: 5.10.0.dev0 (main @ effde20942); also reproduced on the v5.9.0 release
- Platform: macOS-26.1-arm64-arm-64bit (Apple Silicon)
- Python version: 3.11.15
- PyTorch version (GPU?): 2.11.0 (False; MPS available, repro runs on CPU)
- Huggingface_hub version: 1.14.0
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0

### Who can help?

@Cyrilvallez @zucchini-nlp

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Reproduction

```python
import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.float32)
print(next(model.parameters()).dtype)  # torch.bfloat16, expected torch.float32
```

Same result with `dtype="float32"`, with or without `device_map`, and with `dtype=torch.float64`. The concrete class honors the same argument, so the bug is specific to the Auto path:

```python
from transformers import Qwen3_5ForCausalLM

model = Qwen3_5ForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.float32)
print(next(model.parameters()).dtype)  # torch.float32, correct
```

Fully offline reproduction (no Hub access, also a regression-test candidate):

```python
import torch
from transformers import AutoModelForCausalLM, Qwen3_5Config, Qwen3_5ForConditionalGeneration

config = Qwen3_5Config(
    text_config=dict(
        vocab_size=64, hidden_size=32, intermediate_size=64, num_hidden_layers=2,
        num_attention_heads=2, num_key_value_heads=1, head_dim=16,
        linear_conv_kernel_dim=2, linear_key_head_dim=8, linear_value_head_dim=8,
        linear_num_key_heads=1, linear_num_value_heads=2, max_position_embeddings=128,
        layer_types=["linear_attention", "full_attention"],
        rope_parameters={"rope_type": "default", "rope_theta": 10000.0,
                         "partial_rotary_factor": 1.0, "mrope_section": [3, 3, 2]},
    ),
    vision_config=dict(depth=2, hidden_size=32, out_hidden_size=32, intermediate_size=64,
                       num_heads=2, patch_size=4, temporal_patch_size=2, spatial_merge_size=1),
)
Qwen3_5ForConditionalGeneration(config).to(torch.bfloat16).save_pretrained("/tmp/tiny_composite")

model = AutoModelForCausalLM.from_pretrained("/tmp/tiny_composite", dtype=torch.float32)
print(type(model).__name__, next(model.parameters()).dtype)
# Qwen3_5ForCausalLM torch.bfloat16, expected torch.float32
```

### Root cause (line numbers as of effde20942)

1. `_BaseAutoModelClass.from_pretrained` pops `dtype` from kwargs only when it is the string `"auto"` (`src/transformers/models/auto/auto_factory.py:330-333`). A concrete `torch.dtype` or dtype string stays in kwargs.
2. `AutoConfig.from_pretrained(..., return_unused_kwargs=True, **kwargs)` (`auto_factory.py:338`) then absorbs `dtype` into the loaded composite config's top level via standard config-kwarg handling, and it is no longer in the returned unused kwargs:

   ```python
   from transformers import AutoConfig
   cfg, unused = AutoConfig.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.float32, return_unused_kwargs=True)
   print(type(cfg).__name__, cfg.dtype, cfg.text_config.dtype, "dtype" in unused)
   # Qwen3_5Config torch.float32 torch.bfloat16 False
   ```

3. Because the resolved model class is text-only, `model_class.config_class` matches the composite's `text_config`, so the config is swapped: `config = config.get_text_config()` (`auto_factory.py:394-398`, same pattern in `from_config` around lines 240-244). The top-level `dtype`, which now holds the user's value, is dropped here. Note this block already propagates `quantization_config` from the parent config for exactly this reason (#45494), but not `dtype`.
4. `model_class.from_pretrained(...)` is then called with no `dtype` in kwargs, so it falls back to its default `"auto"`, and `_get_dtype` (`src/transformers/modeling_utils.py:837-838`) resolves it from the extracted sub-config (`text_config.dtype: bfloat16` in all Qwen3.5 checkpoints) or, when that is unset, from the checkpoint weights. The user's request is silently gone.

### Impact

- All `Qwen/Qwen3.5-*` repos ship `architectures: ["Qwen3_5ForConditionalGeneration"]` (MoE: `Qwen3_5MoeForConditionalGeneration`) with `text_config.dtype: bfloat16` (verified on 0.8B/2B/4B/9B/122B-A10B), so every `AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-*", dtype=...)` ignores the requested dtype.
- The failure is silent: no warning, and users believe they are running fp32/fp16. As the offline repro shows, it does not even require `dtype` in the sub-config json (it then falls back to the weights dtype), so it affects any composite checkpoint loaded through an Auto class whose resolved class's `config_class` equals `sub_configs["text_config"]`.
- Concrete fallout: in #46190 the reporter's float32 control run silently executed in bfloat16, which made expected bf16 rounding look like a large correctness bug in GatedDeltaNet.

### Expected behavior

`dtype=torch.float32` through the Auto classes should produce fp32 parameters, identical to loading through the concrete class.

### Possible fixes

1. Mirror #45494: in both `get_text_config()` extraction sites, carry the parent config's `dtype` over to the extracted text config when it is not None.
2. Alternatively, pop a user-provided `dtype` before the `AutoConfig.from_pretrained` call (extending the existing `"auto"` special case at `auto_factory.py:330-333`) and pass it explicitly to `model_class.from_pretrained`, so the user value never round-trips through the config object.

I am happy to open a PR with either approach plus an offline regression test based on the snippet above, lmk.

Comments:
Hi, I'd like to take a stab at this. Planning to go with Option 1 (propagating dtype at the get_text_config() extraction sites, mirroring #45494). Will open a PR shortly.

Hi @Achyuthan-S, please read last sentence of issue body ~

Relevant: 
https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md
https://github.com/huggingface/transformers/blob/main/.ai/AGENTS.md#coordination-before-coding

Apologies for jumping ahead without coordinating — I missed the 'lmk' at the end. I've already implemented the fix (Option 1, mirroring #45494) with a passing regression test. Happy to share the patch here for review before opening a PR, or defer to you if you'd prefer to take it. 

Ohhh, loading a multimodal model as `CausalLM`, nice 🫠  (never liked this idea lol)

To start with, we recently deprecated different `dtype` per backbone as it can lead to unexpected dtype mismatches. So when loading the whole VLM, we'd propagate `fp32` in model init step and raise a deprecation warning. Loading a backbone-only is the problem here

IMO the second option makes more sense, so we keep the config's values by default and overwrite it only when user provides a different `dtype` @qflen ping me when you get a PR up :)