MobileLLM safetensors seem to be missing model.embed_tokens.weight
Author: avishaiElmakies
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.46.2
- Platform: Linux-6.6.20-aufs-1-x86_64-with-glibc2.36
- Python version: 3.11.2
- Huggingface_hub version: 0.26.2
- Safetensors version: 0.4.5
- Accelerate version: 1.1.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: no
- GPU type: NVIDIA RTX A5000

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`mobilellm = AutoModelForCausalLM.from_pretrained("facebook/MobileLLM-125M",trust_remote_code=True)`

will output `Some weights of MobileLLMForCausalLM were not initialized from the model checkpoint at facebook/MobileLLM-125M and are newly initialized: ['model.embed_tokens.weight']`

and the weights will be random. when using `use_safetensor=False`. everything seems to work as expected.

### Expected behavior

using safetensors should work the same as when not using them.

Comments:
Hi 👋
Am able to reproduce this, checking this!

can you please provide the code snippet where you are not seeing any error (without using safe tensors)  @avishaiElmakies 

There should be a single "error" about lm_head.weight, since the model uses weight tieing for the embeeding and output layer. Both safetensors and normal loading does this. 

the problem is that when using safetensors the embedding layer seems to be missing which causes problems with both the embedding layer and the output layer. 

maybe I should have been more clear about that in the bug report (sorry about that). 

Hey @avishaiElmakies 

No worries. 

I have fixed the issue and raised a PR here https://huggingface.co/facebook/MobileLLM-125M/discussions/7
The solution and how to test the solution is also mentioned in the same PR. 

You can download the updated safetensors and your code will work as expected :) 




@mayankagarwals thank you! 

350M model also has this problem...

This might be a bug in the safetensors conversion for that model. Have you tried loading the model without safetensors, saving it with safetensors locally using `save_pretrained()`, and then reloading your new safetensors checkpoint? If that fixes the bug, then the issue is in the model repos - you should submit a PR there to fix it!

Added PR for same @NTDXYG https://huggingface.co/facebook/MobileLLM-350M/discussions/4

Can you folks help me by tagging maintainers on these PRS to bring it to notice of the maintainers 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.