`save_pretrained` changes `model.config.torch_dtype` and breaks the following forward pass
Author: GuilhermeFreire
Pull request: no

### System Info

- `transformers` version: 4.42.3
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.14
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.32.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: yes
- GPU type: NVIDIA H100 80GB HBM3

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Just load a model with unsloth's `FastLanguageModel` and try to do a forward pass after a save step.

Minimal example to reproduce:

```python
from unsloth import FastLanguageModel
from datasets import Dataset
from trl import SFTConfig, SFTTrainer
import os

model, tokenizer = FastLanguageModel.from_pretrained(
        model_name="meta-llama/Meta-Llama-3-8B-Instruct",
        dtype=None,  # This will set it to bfloat16 on Ampere+ arch GPUs
        load_in_4bit=False,
        token=os.getenv("HF_ACCESS_TOKEN"),
    )

dataset = Dataset.from_list([{"text": "Hello everyone"}, {"text": "Hello everyone 2"}, {"text": "Hello everyone 3"}]*500)

trainer = SFTTrainer(
        model=model,
        tokenizer=tokenizer,
        train_dataset=dataset,
        args=SFTConfig(
            dataset_text_field="text",
            per_device_train_batch_size=1,
            max_steps=50,
            learning_rate=1e-5,
            bf16=True,
            logging_steps=1,
            save_strategy="steps",
            save_steps=3,
            save_total_limit=5,
            optim="adamw_8bit",
            output_dir="temp_model",
        ),
    )

trainer.train()
```

Error:
```
Traceback (most recent call last):
  File "/home/paperspace/Documents/slingshot-ml/unsloth_save/min_crash.py", line 38, in <module>
    trainer.train()
  File "<string>", line 121, in train
  File "<string>", line 358, in _fast_inner_training_loop
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/transformers/trainer.py", line 3307, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/transformers/trainer.py", line 3338, in compute_loss
    outputs = model(**inputs)
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/accelerate/utils/operations.py", line 819, in forward
    return model_forward(*args, **kwargs)
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/accelerate/utils/operations.py", line 807, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/unsloth/models/llama.py", line 814, in _CausalLM_fast_forward
    outputs = self.model(
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/unsloth/models/llama.py", line 541, in LlamaModel_fast_forward
    inputs_embeds = inputs_embeds.to(self.config.torch_dtype)
RuntimeError: Invalid device string: 'bfloat16'
Traceback (most recent call last):
  File "/home/paperspace/Documents/slingshot-ml/unsloth_save/min_crash.py", line 38, in <module>
    trainer.train()
  File "<string>", line 121, in train
  File "<string>", line 358, in _fast_inner_training_loop
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/transformers/trainer.py", line 3307, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/transformers/trainer.py", line 3338, in compute_loss
    outputs = model(**inputs)
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/accelerate/utils/operations.py", line 819, in forward
    return model_forward(*args, **kwargs)
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/accelerate/utils/operations.py", line 807, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/unsloth/models/llama.py", line 814, in _CausalLM_fast_forward
    outputs = self.model(
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/paperspace/miniconda3/envs/unsloth-test/lib/python3.10/site-packages/unsloth/models/llama.py", line 541, in LlamaModel_fast_forward
    inputs_embeds = inputs_embeds.to(self.config.torch_dtype)
RuntimeError: Invalid device string: 'bfloat16'
```

### Expected behavior

I'd expect the model to be able to save a checkpoint and keep training  like normal.

The problem comes from this line:

https://github.com/huggingface/transformers/blob/350aed70765cac5acfa7a4cfac4d1e6e2dc0b19b/src/transformers/modeling_utils.py#L2454

It was introduced 3 years ago in commit 7682e97702e4317231b3afe92359de384dba1e20

It changes the `torch_dtype` of the model for some reason and that subsequently breaks the next forward pass as the type is now wrong. [This is the line in the unsloth repo when it tries to cast to the now invalid `torch_dtype` (for some reason github preview won't work)](https://github.com/unslothai/unsloth/blob/f1dc078c266b914d8fe652c893ed94f3b35919c8/unsloth/models/llama.py#L561).

Why is this casting to string necessary in the first place? Maybe revert it back after saving the config?
I'm happy to create a PR with a fix, but I feel like I don't have enough context on why this is happening.

Comments:
Hi @GuilhermeFreire, thanks for raising an issue! 

The dtype is cast to a string in the config for serialization purposes in the json file. The decision to save as `"float16"` instead of `"torch.float16"` is [discussed in the PR description](https://github.com/huggingface/transformers/pull/12316). It's essentially an arbitrary choice relating to how we represent types for different frameworks tf/pt/jax. 

The `"torch_dtype"` field in the config indicates the dtype of the weights when they are saved out. However, it isn't used when reloading a model. When we load in weights, we always load in in fp32, unless the `torch_dtype` argument is explicitly passed into the model's `from_pretrained` call i.e. `AutoModel.from_pretrained(checkpoint, torch_dtype=torch.float16)`. The reason for this is to ensure the model is compatible to load across all devices and environments e.g. on CPU regardless of how the model was saved out. 

Unfortunately, we can't control how other libraries use our config values or our code. We obviously want to make sure we're as compatible as possible and make efforts to avoid things like breaking changes. In this case, we haven't changed how `torch_dtype` has been represented in the configs, and as it's been done like this for more than three years, changing it would break many more things than it would fix. 

The simplest solution would be to open a PR in unsloth s.t. the dtype is inferred from its string representation in the transformers config. 

On our side, maybe we could make this clearer? [It's documented in our config docs](https://huggingface.co/docs/transformers/v4.42.0/en/main_classes/configuration), but open to suggestions of other places we could highlight. 

That makes a lot of sense, thanks! I'll create an issue / PR on unsloth, it makes more sense for it to be there. Also thanks for the clarification on the casting to string, I totally missed the PR and looked only at the final commit 😅