Gemma 3 series model default loading behavior of dtype
Author: HuangBugWei
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.51.0
- Platform: Linux-5.4.17-2136.339.5.el8uek.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.30.1
- Safetensors version: 0.5.2
- Accelerate version: 1.6.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: fp16
        - use_cpu: False
        - debug: False
        - num_processes: 1
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
- DeepSpeed version: 0.16.3
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: yes
- Using GPU in script?: yes
- GPU type: NVIDIA GeForce RTX 3090

### Who can help?

@gante @zucchini-nlp 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
# pip install accelerate

from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from PIL import Image
import requests
import torch

model_id = "google/gemma-3-12b-it"

model = Gemma3ForConditionalGeneration.from_pretrained(
    model_id, device_map="auto",
    # torch_dtype=torch.bfloat16
).eval()

processor = AutoProcessor.from_pretrained(model_id)

print(model.dtype)
# torch.float32
```

### Expected behavior

Hello,

When I attempted to load the Gemma 3 12B model, I ran into an OOM error. After investigation, I found that the model was being loaded in FP32 by default.

I know explicitly setting the `torch_dtype` allows users to choose the precision that best suits their needs and provides a clearer understanding of their code's behavior.

However, the official example code for Gemma 3 12B does not explicitly specify the torch_dtype argument in the from_pretrained function. And according to the official documentation,
https://github.com/huggingface/transformers/blob/794fde7b1c3d041519fc28ea3e1461b0cfcad4e7/src/transformers/modeling_utils.py#L3838-L3860

Based on this documentation, it seems that if `torch_dtype` is not explicitly set, the library should first check for a `torch_dtype` entry in the model's `config.json`. And `config.json` of Gemma 3 is BF16, thus, the default loading behavior of FP32 is a bit surprising.

Could you please confirm whether the current default behavior of Gemma 3 12B is intended, or if I'm misunderstanding something. 

Thank you for your time and attention to this matter.

Comments:
@HuangBugWei hey! When `torch_dtype` is not set at load time, we use torch's default dtype which is fp32. If you want to use the config's value, you need to set `torch_dtype="auto"`



@zucchini-nlp 
Got it, my mistake for not noticing that detail. Thanks for the reply!

Great, closing as resolved!