DeepSeek V3 Support
Author: casper-hansen
Pull request: no
Labels: name: New model

### Model description

#### Transformer model

DeepSeek V3 is a Transformer model that utilizes Mixture of Experts (similar to Qwen2 MoE) and Multi-head Latent Attention (MLA).

![image](https://github.com/user-attachments/assets/351e5e4b-63c5-47c0-888d-109c90a78549)

#### Multi-token Prediction

The model is able to predict multiple tokens sequentially at each step through the MTP modules. The first token is generated by the causal LM which feeds the output token into what I would describe as a "Transformer head" to generate additional tokens for the current step. DeepSeek notes in their release that *"MTP support is currently under active development within the community, and we welcome your contributions and feedback."* (i.e. code for this is not released).

![image](https://github.com/user-attachments/assets/35bce43f-9b7a-4a95-9062-35ae072e9771)


### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Transformers Code: https://huggingface.co/deepseek-ai/DeepSeek-V3
GitHub Code (minimal implementation): https://github.com/deepseek-ai/DeepSeek-V3/tree/main/inference
Paper: https://github.com/deepseek-ai/DeepSeek-V3/blob/main/DeepSeek_V3.pdf

Comments:
Historical perspective on Deepseek. Deepseek v2 support was never added by Deepseek team. There is a community v2 PR that never went out of draft phase. 

https://github.com/huggingface/transformers/pull/31976

Unless added by oss community or hf, history shows deepseek will not proactively add hf support as their priority is sglang, lmdeploy and others.

Lets hope someone or hf pickup this ball on this as this is not a simple model to support.

@ArthurZucker 

The DeepSeek v3 code is mostly available already though. They put an MIT license on the code in their repository. So a PR mostly needs a multi-token prediction implementation. 

https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py

> The DeepSeek v3 code is mostly available already though. They put an MIT license on the code in their repository. So a PR mostly needs a multi-token prediction implementation.
> 
> https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py

This isn't quite correct right? DeepSeek-V2/V2.5 never got natively implemented as per https://github.com/huggingface/transformers/pull/31976 and there is modeling code for V2 as well. https://huggingface.co/deepseek-ai/DeepSeek-V2/blob/main/modeling_deepseek.py

Looking at the [ADD_NEW_MODEL_PROPOSAL_TEMPLATE.md](https://github.com/huggingface/transformers/blob/main/templates/adding_a_new_model/ADD_NEW_MODEL_PROPOSAL_TEMPLATE.md) from the REPO I'm willing to try my hand at attempting a native DeepSeek-V3 impelemntation but since that README was last updated 10 months ago is the mentor system still valid and is this the up to date way to go about it?

cc @Cyrilvallez on the last comment above regarding guiding model integrations ^

> cc @Cyrilvallez on the last comment above regarding guiding model integrations ^

Thanks! Would love to do this as per protocol as I am dying to use transformers tools on this model!

https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py#L439
what does this mean? It seems that this code can only run inference
![image](https://github.com/user-attachments/assets/f488d7f6-f6af-416d-a5fe-147ac41a56e4)


Bumping for @Cyrilvallez advisement. We, not really me @fairydreaming, skipped transformers functionally for llama.cpp. But this would be very appreciated to have for those who want to try like exl2 or AWQ

Hey @Nottlespike! Very nice that you want to tackle this! Sorry for the late answer, I was still on vacation. Regarding model integrations rules, you can also check  [here](https://huggingface.co/docs/transformers/en/add_new_model) and [modular rules](https://huggingface.co/docs/transformers/main/modular_transformers).  
What you want to do is to isolate the small individual changes from your model (deepseek), and existing models in the library (e.g. mixtral, qwen2 moe, deepseek v1...), then create a modular file based on those differences. You can check e.g. this [past model integration](https://github.com/huggingface/transformers/pull/33823) for an example of model addition with modular.  
The [modeling code](https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py) already on the hub should provide a strong starting point.  
Let me know if you have any questions 🤗

Hi, just a note that I recently [traced some logits](https://huggingface.co/datasets/baffo32/llm_logits/tree/main) from the two versions of this model, and the two implementations of the architecture appear to deterministically select different experts. I have not looked at why.

EDIT:
I need to make clear that (1) **it is normal for independent model implementations to behave slightly differently**. This is because pytorch has nondeterministic behavior (accuracy-compromising module optimizations and random number generation) that can change outputs when things are reorganized.  And that (2) I meant to post this nitty-gritty comment on the PR thread, not the issue thread. Sorry for that.
I make these comments because my browser is reporting that the PR thread was flagged as a dangerous phishing attempt. Unless somebody has looked more closely and has any reason to believe that is true, this PR looked very forthright and normal for this repository, to me, an honest and strong attempt to contribute.