Can't save checkpoint with shared tensors
Author: AndrewRyanChama
Pull request: no

### System Info

Name: transformers
Version: 4.39.3



### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am trying to use a huggingface trained model in an external tool (in this case exllamav2) but it is not possible due to the issue saving the tensors

In this example I simply load the model and save it
```
model = AutoModelForCausalLM.from_pretrained('Qwen/Qwen1.5-0.5B')
model.save_pretrained("testmodel")
```

In the model.safetensors, there is no way to load lm_head.weights, while in the original Qwen https://huggingface.co/Qwen/Qwen1.5-0.5B/tree/main there is lm_head.weights

This is causing other librarires to fail when trying to load the saved checkpoint, even when they can load the original checkpoint.

### Expected behavior

safetensors saved by huggingface transformers should match the original tensors in the qwen repo

Comments:
Hi @AndrewRyanChama, thanks for opening this issue! 

I'm able to do the following:

```py
In [1]: from transformers import AutoModelForCausalLM

In [2]: model_0 = AutoModelForCausalLM.from_pretrained('Qwen/Qwen1.5-0.5B')
   ...: model_0.save_pretrained("testmodel")

In [3]: model_1 = AutoModelForCausalLM.from_pretrained('testmodel')

In [4]: (model_0.lm_head.weight == model_1.lm_head.weight).all()
Out[4]: tensor(True)
```

Could you give some more information about the errors being encountered?

One thing to note is that because the `lm_head` [is tied](https://github.com/huggingface/transformers/blob/caa5c65db1f4db617cdac2ad667ba62edf94dd98/src/transformers/models/qwen2/modeling_qwen2.py#L1091), the weights are only saved once in the safetensors file 



Yes, the safetensors file is able to be read within huggingface without any issue.  The concern is that, as you said, the weights are only saved once in the safetensors file.

The problem is that external libraries that consume this safetensors file with an independent implementation expect to be able to retrieve each tensor individually from the safetensors file.  After this recent change to the huggingface functionality, this is causing breakages because it can't find the lm_head tensor.

While it might be the case that the weights are tied in the huggingface implementation, this is not necessary to know at inference time as doing inference doesn't care if the input and head tensors are different or the same.  But by hiding these tensors, it's giving an additional modeling constraint on any consumer of the safetensors file.


On the original safetensors published by the Qwen authors, this issue is not present.  Reading the safetensors file give an entry for each tensor:

```
with safe_open("Qwen/Qwen1.5-0.5B/model.safetensors", framework="pt", device=0) as f:
    print(f.keys()) # lm_head.weights is included in this list

with safe_open("testmodel/model.safetensors", framework="pt", device=0) as f:
    print(f.keys()) # lm_head.weights is missing from the list
```
This is a breaking change because now *less* information is available from the safetensors file, and it must be provided externally.


@AndrewRyanChama I see. 

The tricky thing here is that there's ultimately a coupling for models which are integrated into transformers between what's on the checkpoint's repo and the transformers library. This doesn't just apply to the weights, but for other files such as the tokenizer files or the model config. 

I'm not sure it make sense to have extra weights unnecessarily saved out if they're tied. Perhaps which weights are tied could be part of the safetensors metadata cc @Narsil? 

What makes this difficult is that the tying behaviour is currently defined in the modeling code e.g. which [weights to tie here](https://github.com/huggingface/transformers/blob/766810153b3a701e99229229c3899e370be7b7a1/src/transformers/models/cohere/modeling_cohere.py#L1036) and [from the config](https://github.com/huggingface/transformers/blob/766810153b3a701e99229229c3899e370be7b7a1/src/transformers/models/cohere/modeling_cohere.py#L1045), which of course could change, and is configurable by the user. 



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.