torch_dtype is actually used now?
Author: dakinggg
Pull request: no
Labels: name: bug

### System Info

different transformers versions. see description

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Previously (v4.46.3, didn't check all versions), `torch_dtype` in the config was ignored, meaning that model weights would get loaded in fp32 by default (correct behavior for training). On latest transformers version (v4.49.0), it seems it is now used, and so the weights get loaded with whatever is in the checkpoint. Was this change intentional? I previously recall seeing somewhere in the code that you weren't going to make the change to actually use torch_dtype until v5, and I didn't see anything in release notes at a glance, although maybe I missed it.

```
In [1]: import transformers

In [2]: llama1bcfg = transformers.AutoConfig.from_pretrained('meta-llama/Llama-3.2-1B-Instruct')

In [3]: llama1b = transformers.AutoModelForCausalLM.from_config(llama1bcfg)

In [4]: next(llama1b.parameters()).dtype
Out[4]: torch.bfloat16
```

### Expected behavior

Not actually sure, would like to confirm what you expect now.

Comments:
Yep, that was cause by one of PRs to enable different `dtype` setting for each backbone in multimodal models. Indeed a breaking change, but I opted to not revert it since I expect this is an intuitive behavior

@ArthurZucker WDYT? If we revert, we either lose dtypes for multimodals or have to manually pass `dtype` when init backbone in each model file  

FWIW I think its a pretty big change to make silently, with no warnings, no notice in the release, etc. Especially because of this comment that was (still is actually) in the code: https://github.com/huggingface/transformers/blob/052e652d6d53c2b26ffde87e039b723949a53493/src/transformers/modeling_utils.py#L2795-L2796

Also it seems to be different when using different construction APIs

```
In [1]: import transformers

In [2]: llama1bcfg = transformers.AutoConfig.from_pretrained('meta-llama/Llama-3.2-1B-Instruct')

In [3]: llama1bcfg.torch_dtype
Out[3]: torch.bfloat16

In [4]: llama1b_frompt = transformers.AutoModelForCausalLM.from_pretrained('meta-llama/Llama-3.2-1B-Instruct')

In [5]: next(llama1b_frompt.parameters()).dtype
Out[5]: torch.float32

In [6]: llama1b_fromcfg = transformers.AutoModelForCausalLM.from_config(llama1bcfg)

In [7]: next(llama1b_fromcfg.parameters()).dtype
Out[7]: torch.bfloat16
```

Sorry about that, completely missed the notification... it is a big breaking change that should not have made it to the release

As we did not have much report, we are just gonna update the release notes and the code there ! Thanks a lot @dakinggg 

FWIW, I'd be a bit worried that people just hadn't noticed, because sometimes this will error (e.g. you have different layers of a model initialized differently, and some part of the training stack doesn't like having different dtypes for different params), but if you don't have unit tests to check this, and don't encounter any error cases, you will just accidentally. use bf16 weights instead of fp32 (likely fine if you are just doing inference, and not fine if you are trianing)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.