Some weights of the model checkpoint at /models/DeepSeek-V3_bf16 were not used when initializing DeepseekV3ForCausalLM
Author: Godlovecui
Pull request: no
Labels: name: bug

### System Info

I use  AutoModelForCausalLM.from_pretrained to load DeepSeek_V3, it raises below warning:

![Image](https://github.com/user-attachments/assets/19b42d79-ed03-4f56-8b8b-765c8d96d32e)
then I print the model state keys, it only has 60 layers, however, the deepseek v3 weight actual has 61 layers, the last layer is missing.

![Image](https://github.com/user-attachments/assets/0400c975-5dfc-4cd2-b4d2-213ee7980b3e)

How to fix it? Thank you~



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

model = AutoModelForCausalLM.from_pretrained(
    "path/to/deepseek_v3_bf16,
    device_map="cpu",
    torch_dtype="auto",
    trust_remote_code=True,
)
print(model.state_dict().keys())

### Expected behavior

 AutoModelForCausalLM.from_pretrained could load deepseek v3 correctly.

Comments:
@Godlovecui hey!

Seems like there is a mismatch between the actual state dict keys and the model's configuratino in DeepSeek. Can you open an issue in the hub repo (https://huggingface.co/deepseek-ai/DeepSeek-V3/tree/main), since the error is related to model weights?

@zucchini-nlp 
ok, Thank you for your reply. The issue is opened here:
https://huggingface.co/deepseek-ai/DeepSeek-V3/discussions/62

bor, there is something wrong betweeen model.safetensors.index.json and config.json. In model.safetensors.index.json, there are 62 layers from layer0 to layer61. But in config.json, the num_hidden_layers is 61.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I think this is because the Main Model of DeepSeek v3 is designed to contain 61 hidden layers (`"num_hidden_layers": 61`) with an additional hidden layer appended as `model.layers.61` for Multi-Token Prediction (MTP) Modules (`"num_nextn_predict_layers": 1`). 

However, it seems that the MTP module functionality is not yet released to open-source as the `num_nextn_predict_layers` setting is not referenced anywhere in `modeling_deepseek.py`, suggesting the absence of MTP logic. 

Therefore, this is causing the mismatch between the layer number in `model.safetensors.index.json` (0 ~ 61) with the number of hidden layers actually loaded (0 ~ 60) as mentioned by @Silencezong 

-------------------------------
As the DeepSeek v3 report paper explained, 
```
Our MTP strategy mainly aims to improve the performance of the main
model, so during inference, we can directly discard the MTP modules and the main model can
function independently and normally.
```
which seems to be how the DeepSeek model is released to open-source: with the MTP modules only used during training and discarded for inference.