load and train with bf16，saved torch_dtype is float32
Author: gyangfan
Pull request: no

### System Info

- `transformers` version: 4.39.3
- Platform: Linux-3.10.0-1160.102.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.1
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.2
- Accelerate version: 0.29.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.2+cu121 (True)

load a bf16 model  then train with --bf16 and log show [**Using auto half precision backend**],  the final saved model parameter torch_dtype in config.json is **float32**
1、why saved float32？
2、how do i load the trained model？if load with float32，Can I get a better output?

Comments:
Hi @gyangfan, thanks for raising an issue! 

It might be useful to look at the [usage tips for other models](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/llama2#usage-tips) which were trained with bf16. I'm surprised that the `torch_dtype` in the config if `float32`, it should be the same as the model's weights e.g. [like here for llama 2](https://huggingface.co/meta-llama/Llama-2-7b-hf/blob/01c7f73d771dfac7d292323805ebc428287df4f9/config.json#L21). 

Without a script showing how you're creating, training and loading the model, it's not possible to know exactly how the weights and saving/loading is being handled. 

cc @younesbelkada who knows more about the weight handling 

Hi @gyangfan 
Thanks for the issue! 
Note `--bf16` will not load the model in bf16, it will just activate the bf16 mixed precision training, in that case it is recommended to keep the model weights in fp32 for more stable training. If you want to load the model in bf16 you need to manually pass `torch_dtype=torch.bfloat16` in `from_pretrained`

Hi @younesbelkada， thank fot the relpy！
That's exactly what I did，to pass `torch_dtype=torch.bfloat16` in `from_pretrained` to load the model in bf16. If i want to train the model with mixed precision，as you say，fp32 is more stable for training，so do I start training with `model = model.float() `to upcast the model to fp32?  This is something I didn't see in the `Trainer` process，It seems to be that I load with bf16, that is train with pure bf16，Is this compatible with bf16 mixed precision. 

Please correct me if my understanding is wrong.

Hi @gyangfan 
Thanks for getting back ! 
Per my understanding:
- training with bf16 mixed precision = loading the model without `torch_dtype=xxx` to load it in fp32 + activate the bf16 mixed precision training with `bf16=True` flag in `TrainingArguments`
- training in pure bf16 = loading the model in bf16. You will need less memory than the first option but training might be less stable than the first option 
Let me know if this makes sense (I am also not 100% sure here) - cc @pacman100 @muellerzr in case I missed something here

Hi @younesbelkada 
Thanks for your explanation！
I shouldn't use the `torch_dtype` saved in `config.json` if i want to finetune the pretrained llm model.
The  `torch_dtype` provided in `config.json` maybe not indicate how it was trained and saved，for example `torch_dtype=torch.bfloat16` may show it was trained with  mixed precision with bf16，but  not trained with pure bf16 and not saved the weights in bf16，it just helpful when i load the model with `from_pretrained` and `torch_dtype="auto"`


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @younesbelkada 
Thank you for the explanations from you and your colleagues!

At first, I hope to summarize the background.
1. When using `from_pretrained` without setting `torch_dtype=xxx`, the model will be load in fp32 regardless of dtype in the model weight files (such bf16 for mistral model files)
2. Without `torch_dtype=xxx`, set `bf16=True` or `fp16=True` in training args means mixed precision training with bf16/fp16. In this case, there are both 16-bit and 32-bit version model weights according to the doc [fp16 in Methods and tools for efficient training on a single GPU transformers v4.46](https://huggingface.co/docs/transformers/perf_train_gpu_one#fp16)

Then, I have some questions.
1. When I load a model with `torch_dtype=torch.float16 or bfloat16` and train it with trainer following hf code examples, is the model trained in pure fp16/bf16? According to [#24819 (comment)](https://github.com/huggingface/transformers/issues/24819#issuecomment-1635759400), is `--fp16/bf16` fully ignored? Both `fp16/bf16=True` and `fp16/bf16=False` are ok and I will get a full half precision train?
2. Is pure half precision training still unstable currently? Is there any new technology to solve this problem? Or it is not able to be solved? I wonder whether any popular LLM is pre-trained in pure half precision?
3. If I want to load the model in fp16/bf16 and train with lora in fp16/bf16, what should I do? Use `torch_dtype=torch.float16` and `--fp16`? In this case, which is the lora adapter is trained in? Pure half precision or mixed precision? Sorry, I don't want to use qlora and load the model in 8-bit int. But loading a 7B model in fp32 costs 28GB mem. That is impossible for gpus like 3090. So, I want to load it in half precision.

+1 @shelfofclub 
Did you find any answers?