Method `from_pretrained` does not respect the mapping of tied tensors in `safetensors` checkpoint
Author: daskol
Pull request: no
Labels: name: bug

### System Info

The issue exists for `transformers>=4.54.0<=4.56.1` for sure (tested).

```bash
$ transformers env
- `transformers` version: 4.56.1
- Platform: Linux-6.16.7-arch1-1-x86_64-with-glibc2.42
- Python version: 3.13.7
- Huggingface_hub version: 0.35.0
- Safetensors version: 0.5.3
- Accelerate version: 1.10.1
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.8.0 (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.10.5 (cpu)
- Jax version: 0.7.0
- JaxLib version: 0.7.0
- Using distributed or parallel set-up in script?: <fill in>
```

### Who can help?

@CyrilVallez Method `AutoModel.from_pretrained` populates model weights from a checkpoint in `safetensors` format then it ties model weights in a way independent of checkpoint metadata and overwrite loaded weights on the real device with fake weights on `meta`. Here is relevant piece of code.

https://github.com/huggingface/transformers/blob/91393fe4cc3266a05bc0d129e34ff5f761bb46e2/src/transformers/modeling_utils.py#L5164-L5194

### Details

The `transformers` library loads weights first and then ties input embeddings to output embeddings as follows (see [transformers/modeling_utils.py#L3236][1]).

```python
def _tie_or_clone_weights(self, output_embeddings, input_embeddings):
    if self.config.torchscript:
        ...
    else:
        output_embeddings.weight = input_embeddings.weight
```

However, the choice what keys of state dict to load depends on `transformers` implementation. Specifically, it makes decision based on available keys in state dict loaded from a checkpoint. This can bring to the situation when

1. it loads output embeddings but not input ones;
2. then it ties input embeddings to output ones.

As the result, both the input and output embeddings lives on `meta` device! In other words, no weights are loaded for bot input and output embeddings.

### Reproduction

The issue can be reproduced with a `convert.py` script from `safetensors` repo (see [bindings/python/convert.py#L216][2]). The routine `convert_file` sorts all keys alphabetically and takes the first item in the list. For example, it keeps `model.lm_head.weight` rather than `model.model.decoder.embed_tokens.weight` in case of OPT model.

```python
>>> from convert import convert_file
>>> convert_file('facebook/opt-125m/pytorch_model.bin', 'facebook/opt-125m/model.safetensors', [])
>>> from transformers import AutoModelForCausalLM
>>> model = AutoModelForCausalLM.from_pretrained('facebook/opt-125m')
>>> model.to('cpu')
NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.
```

[1]: https://github.com/huggingface/transformers/blob/v4.56.1/src/transformers/modeling_utils.py#L3236
[2]: https://github.com/huggingface/safetensors/blob/v0.6.2/bindings/python/convert.py#L216

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The issue can be reproduced with a `convert.py` script from `safetensors` repo (see [bindings/python/convert.py#L216][2]). The routine `convert_file` sorts all keys alphabetically and takes the first item in the list. For example, it keeps `model.lm_head.weight` rather than `model.model.decoder.embed_tokens.weight` in case of OPT model.

```python
>>> from convert import convert_file
>>> convert_file('facebook/opt-125m/pytorch_model.bin', 'facebook/opt-125m/model.safetensors', [])
>>> from transformers import AutoModelForCausalLM
>>> model = AutoModelForCausalLM.from_pretrained('facebook/opt-125m')
>>> model.to('cpu')
NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.
```

[1]: https://github.com/huggingface/transformers/blob/v4.56.1/src/transformers/modeling_utils.py#L3236
[2]: https://github.com/huggingface/safetensors/blob/v0.6.2/bindings/python/convert.py#L216

### Expected behavior

- All weights in state dict are restored from a checkpoint.
- No weights remains on `meta` device.

Comments:
Hey! It's the same issue as https://github.com/huggingface/transformers/issues/38870! Basically, without enough informations, `safetensors` saves the wrong key: for itself and in general it's not a problem, but `transformers` excepts one given key and not the other, so there it becomes a problem. We merged a PR on the automatic converter on the hub here https://huggingface.co/spaces/safetensors/convert/commit/447ad805b4eaf25e2570ad2940b23510a9b0e713 to save the correct keys, and I will upstream the change directly to safetensors converter, which you looked at!

That's great! Thank you.

I'd like to draw your attention to `transformers` rather than `safetensors`. I spent sometime on debugging the issue and found out that `modeling_utils.py` tries to tie multiple times (twice if I'm correct) but entire `AutoModel` facility ties weights merely three times (+ on weight initialization in `post_init`, surely).

From this perspective, `safetensors` and `convert.py` does their job well. They save tensors and put into metadata information about weights tied in the original model. However, `AutoModel` facility can't properly tie weights despite all required information stored in metadata. Having read `modeling_utils.py`, I understand the difficulties and corner cases of loading model on proper devices,  it is indeed difficult. Despite that I think, a proper place to fix weight tying is not `safetensors` but rather `AutoModel` facility in `transformers`.

Being able to tie weights from any key in Transformers directly would indeed be very nice, and something I thought about before. However, due to the historical (unfortunately bad) choice of using a single key instead of checking the existence of any of the key and tying accordingly, it is now quite hard to change. I'd like to allow it at some point, but it will require some work and I can't say for sure when this will be done (it's not high priority)