torch_dtype='auto' is not working when using AutoModel.from_pretrained(...)
Author: Cyrilvallez
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: macOS-12.2.1-x86_64-i386-64bit
- Python version: 3.11.3
- Huggingface_hub version: 0.15.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@younesbelkada @ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The `torch_dtype='auto'` argument is not forwarded correctly when using `AutoModelForCausalLM(model_name, torch_dtype='auto')`. For example, the opt model 'facebook/opt-125m' has `torch_dtype: float16` in its config file but the following happens:

```python
from transformers import AutoModelForCausalLM, OPTForCausalLM

model = OPTForCausalLM.from_pretrained('facebook/opt-125m', torch_dtype='auto')
model.dtype
# CORRECT dtype
>>> torch.float16

model = AutoModelForCausalLM.from_pretrained('facebook/opt-125m', torch_dtype='auto')
model.dtype
# INCORRECT dtype 
>>> torch.float32
```

### Expected behavior

Both outputs should be `torch.float16`  as in the config file specification.

 From what I've looked into, this comes from 
```python
if kwargs_copy.get("torch_dtype", None) == "auto":
    _ = kwargs_copy.pop("torch_dtype")
```

in `transformers.models.auto.auto_factory.py`, line 441. The additional kwarg specifying dtype is poped and the dtype is only inferred by the dtype argument of the config file, which is then not given explicitly (only implicitly in the config) to `PretrainedModel.from_pretrained(model_name, config=config,...)`, which does not use it if the explicit `torch_dtype` argument is not provided.

I would be happy to help solve the issue if needed. 

Also, I find it strange that
```python
from transformers import AutoConfig, AutoModelForCausalLM

config = AutoConfig.from_pretrained('facebook/opt-125m')
config.torch_dtype
>>> torch.float16

model = AutoModelForCausalLM.from_pretrained('facebook/opt-125m', config=config)
model.dtype
>>> torch.float32
```

i.e. `AutoModelForCausalLM.from_pretrained(...)` does not respect the dtype of the config as I was saying before (this is the reason of the previous bug). But maybe this is to avoid errors when model configs specify `torch_dtype: bfloat16` and users try to instantiate on the cpu?

Anyway, when specifying 'auto', i.e. `AutoModelForCausalLM.from_pretrained(...torch_dtype='auto')`, I think it is absolutely necessary for the model to be instantiated with the dtype specified on the config file if any, even if it may break code for `torch.bfloat16` models, because users using this feature are aware of what they are doing.

Comments:
Pinging @younesbelkada as I think he looked into this previously! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Still needs to be addressed

Are you sure? Running:
```py
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained('facebook/opt-125m', torch_dtype='auto')
model.dtype
```
gives me `float16`.

In general, note that we do not recomment using `torch_dtype="auto"` but setting the `torch_dtype` you want yourself, as users of the Hub never set the dtype in the config to a value they thought about (this is just an indication of the dtype they used before pushing the model on the Hub). One dtype might work on their hardware without working on yours.

As for your other question of why `AutoModelForCausalLM.from_pretrained(...)` does not automatically respects the dtype of the config, we just follow PyTorch convention here: if you instantiate a model without specifying a dtype, then load a state dict into it, the resulting model will be in float32 (the default floating point format) whatever the type of the state dict. The dtype of the original model is preserved. Wanting to load in a specific dtype should be specifically indicated by the user, as float32 (the default dtype) is the only dtype that works on all kinds of hardware (float16 fails on the CPU, bflaot16 on older GPUs etc.).

Ha yes, it seems that the latest version (v4.31.0) solved the issue! Sorry about that, and thanks for the additional explanation!