Transformers does not work well with `with torch.device("meta"):`
Author: fxmarty-amd
Pull request: no
Labels: name: bug

### System Info

```
- `transformers` version: 4.49.0
- Platform: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.14
- Huggingface_hub version: 0.28.1
- Safetensors version: 0.4.4
- Accelerate version: 1.2.1
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.6.0+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
```

### Who can help?

@ArthurZucker @SunMarc 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi, when loading a model on meta device many warnings "copying from a non-meta parameter in the checkpoint to a meta parameter in the current model" are printed.

Reproduction:

```python
from transformers import AutoModelForCausalLM
import torch

with torch.device("meta"):
    original_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m")
```

gives a lot of warnings:
```
/home/fxmarty/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py:2397: UserWarning: for model.decoder.embed_tokens.weight: copying from a non-meta parameter in the checkpoint to a meta parameter in the current model, which is a no-op. (Did you mean to pass `assign=True` to assign items in the state dictionary to their corresponding key in the module instead of copying them in place?)
  warnings.warn(
/home/fxmarty/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py:2397: UserWarning: for model.decoder.embed_positions.weight: copying from a non-meta parameter in the checkpoint to a meta parameter in the current model, which is a no-op. (Did you mean to pass `assign=True` to assign items in the state dictionary to their corresponding key in the module instead of copying them in place?)
  warnings.warn(
/home/fxmarty/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py:2397: UserWarning: for model.decoder.final_layer_norm.weight: copying from a non-meta parameter in the checkpoint to a meta parameter in the current model, which is a no-op. (Did you mean to pass `assign=True` to assign items in the state dictionary to their corresponding key in the module instead of copying them in place?)
  warnings.warn(
/home/fxmarty/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py:2397: UserWarning: for model.decoder.final_layer_norm.bias: copying from a non-meta parameter in the checkpoint to a meta parameter in the current model, which is a no-op. (Did you mean to pass `assign=True` to assign items in the state dictionary to their corresponding key in the module instead of copying them in place?)
  warnings.warn(
/home/fxmarty/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py:2397: UserWarning: for model.decoder.layers.0.self_attn.k_proj.weight: copying from a non-meta parameter in the checkpoint to a meta parameter in the current model, which is a no-op. (Did you mean to pass `assign=True` to assign items in the state dictionary to their corresponding key in the module instead of copying them in place?)
  warnings.warn(
/home/fxmarty/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py:2397: UserWarning: for model.decoder.layers.0.self_attn.k_proj.bias: copying from a non-meta parameter in the checkpoint to a meta parameter in the current model, which is a no-op. (Did you mean to pass `assign=True` to assign items in the state dictionary to their corresponding key in the module instead of copying them in place?)
  warnings.warn(
/home/fxmarty/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py:2397: UserWarning: for model.decoder.layers.0.self_attn.v_proj.weight: copying from a non-meta parameter in the checkpoint to a meta parameter in the current model, which is a no-op. (Did you mean to pass `assign=True` to assign items in the state dictionary to their corresponding key in the module instead of copying them in place?)
  warnings.warn(
...
```

I don't recall it being the case previously, maybe something changed in `from_pretrained` or torch, that results in these warnings?

### Expected behavior

No warnings, params on meta device

Comments:
cc @SunMarc @muellerzr since this crosses over with accelerate a bit, but also @fxmarty-amd the errors don't surprise me, since `from_pretrained` loads weight data into the model, and meta tensors don't actually hold weight data! Maybe you could avoid the errors by initializing the same model architecture without weight loading like this:

```python
import torch
from transformers import AutoConfig, AutoModelForCausalLM

config = AutoConfig.from_pretrained("facebook/opt-125m")
with torch.device("meta"):
    model = AutoModelForCausalLM.from_config(config)
```

> I don't recall it being the case previously, maybe something changed in `from_pretrained` or torch, that results in these warnings?

This is due to the torch if I recall correctly. I can check what can be done on our side but since you don't need the weights, I think @Rocketknight1 solution will be more appropriate.

Thank you this makes sense, I'll try to use `from_config` whenever possible (although external libraries relying on `from_pretrained` may not expose `from_config`).