from_pretrained torch_dtype DO NOT affect model buffers
Author: Chandler-Bing
Pull request: no

### System Info

pass

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I recently encountered a problem with inconsistent model output. After debugging, I find why.

If I use` torch_dtype=torch.bfloat16` in the `from_pretrained `method, it only affects the model parameters, but has no effect on the model buffer. The buffer is still `float32`.
But in the configuration of trainer ` --bf16_full_eval=True `,   [model.to(bfloat16)](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3692) had been used, so that all parameters and buffers of the model become bfloat16.
And that's why I see difference logits of same pretrained models

**So the question is,** **is `from_pretrained torch_dtype `designed to only focus on the parameters and not the buffer? Or is this a bug?**

### Expected behavior

I'm not sure what  behavior should be, but it will be nice to see some docs about it 

Comments:
Hey! 
To answer that question I would need a simple reproducer, as well as the output of `transformers-cli env` to see which version you are running this one. 
AFAIK, buffers should we affected. I recently specifically wanted to keep certain buffers in a precision and it did not work

> Hey! To answer that question I would need a simple reproducer, as well as the output of `transformers-cli env` to see which version you are running this one. AFAIK, buffers should we affected. I recently specifically wanted to keep certain buffers in a precision and it did not work

@ArthurZucker 
sure, you could get the code here, [colab](https://colab.research.google.com/drive/1PqFZsFGi8E31mTf1sSidLvKrdHlLWHr_#scrollTo=ZiozDjj1ExIV)
```
from  transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
        'Qwen/Qwen1.5-0.5B',torch_dtype=torch.bfloat16)

for name,buffer in model.named_buffers():
    if buffer.dtype==torch.float32:
        print(name,buffer.dtype)
```
the model is randomly chosen to test. 
![image](https://github.com/huggingface/transformers/assets/29994840/6ad665ee-095d-40ab-86ae-f95db4fcf462)

seems just this buffer `rotary_emb.inv_freq` still` torch.float32`

Actually we want the inv freq to be in float 32 to keep as much precision as possible. 
<img width="669" alt="image" src="https://github.com/huggingface/transformers/assets/48595927/7a834a6a-1644-4153-be67-f167853885f7">
because in the code we specify the dtype, the default torch dtype is not used

```python3
import torch.nn as nn
class myModule(nn.Module):
  def __init__(self):
    super().__init__()
    self.register_buffer("freq", torch.ones(12,12, dtype=torch.float32))
torch.set_default_dtype(torch.bfloat16)
module = myModule()
module.freq.dtype
```
vs
```python3
import torch.nn as nn
class myModule(nn.Module):
  def __init__(self):
    super().__init__()
    self.register_buffer("freq", torch.ones(12,12))
torch.set_default_dtype(torch.bfloat16)
module = myModule()
module.freq.dtype
``` 

Closing this as it's affecting buffers that don't specify a dtype, but not others.

> Actually we want the inv freq to be in float 32 to keep as much precision as possible. <img alt="image" width="669" src="https://private-user-images.githubusercontent.com/48595927/330699622-7a834a6a-1644-4153-be67-f167853885f7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUyNzYyNjIsIm5iZiI6MTcyNTI3NTk2MiwicGF0aCI6Ii80ODU5NTkyNy8zMzA2OTk2MjItN2E4MzRhNmEtMTY0NC00MTUzLWJlNjctZjE2Nzg1Mzg4NWY3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAyVDExMTkyMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTA1ZjIzYzhmN2I2NjczN2UwNmNkZTQ2YTg1M2IxZWZhMDljMzg2NjkxYWZiNjdkZDI5YTc3ZWFkODQ1MzIyNzMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Wty1q_tebU--jgLGsAdIRASUnEo8T8FoZvpnJ-QReNM"> because in the code we specify the dtype, the default torch dtype is not used
> 
> ```python
> import torch.nn as nn
> class myModule(nn.Module):
>   def __init__(self):
>     super().__init__()
>     self.register_buffer("freq", torch.ones(12,12, dtype=torch.float32))
> torch.set_default_dtype(torch.bfloat16)
> module = myModule()
> module.freq.dtype
> ```
> 
> vs
> 
> ```python
> import torch.nn as nn
> class myModule(nn.Module):
>   def __init__(self):
>     super().__init__()
>     self.register_buffer("freq", torch.ones(12,12))
> torch.set_default_dtype(torch.bfloat16)
> module = myModule()
> module.freq.dtype
> ```

which py files needs to be edited in order NOT to have this forced float32?

i suggest gives option in from_pretrained function to allow buffer has same dtype as torch_dtype.
since 2023, new PCs have npu that can handle bfloat16
amd zen4 laptop and desktop also has avx512 bfloat16 capability

Depending on the modeling, you can just change `_keep_in_fp32_modules` (From here: https://github.com/huggingface/transformers/blob/0c8021c5d566bcc6897ba3712412414ab47f5e44/src/transformers/models/t5/modeling_t5.py#L785-L797)