[WIP] add deepseek-v3
Author: bzantium
Pull request: yes
Labels: name: New model

# What does this PR do?

This PR adds the codes for the DeepSeekV3.
code relies heavily on [original remote code](https://huggingface.co/deepseek-ai/DeepSeek-V3).

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

resolved: #35425 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case: #35425 
- [] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

to: @ArthurZucker

Comments:
Hi @bzantium, this looks great so far! We'll need added tests for the model + a green CI, and then feel free to ping me to assign a reviewer, or if you have any problems with the port.

@bzantium Thanks for the amazing work! I was wondering if you were able to train V3 with FSDP? If so how many GPUs did you need? Thanks!

One big thing would be `TP` support, the `base_tp_plan` would probably need to be updated to make sure each mlp's gat up down have the correct order, unless the direct usage of dist remove this need

This is great work and I'm looking forward to try it out. For multi-token prediction, is this planned to be implemented in this PR via the `num_nextn_predict_layers` attribute in the config?

Thanks for the comments in detail; following your comments, I revised code quite a lot and fixed some mismatch between original code and this PR. I checked the outputs from both are the same. I think now I can add test codes. For `TP` support, I think they can be applied only for `mlp` layer but not for `self_attn` because they have functions like `split` on the `hidden_dim`. I added as following:
```
    base_model_tp_plan = {
        "layers.*.gate_proj": "colwise",
        "layers.*.up_proj": "colwise",
        "layers.*.down_proj": "rowwise",
    }
```
to: @ArthurZucker 

> @bzantium Thanks for the amazing work! I was wondering if you were able to train V3 with FSDP? If so how many GPUs did you need? Thanks!

I did not try training yet, since this PR only supports for inference currently. I plan to add trainable code afterwards.

> This is great work and I'm looking forward to try it out. For multi-token prediction, is this planned to be implemented in this PR via the `num_nextn_predict_layers` attribute in the config?

thanks for attention, I don't have plan to implement multi-token prediction for this time since there is no additional parameters for additional layer which is required for multi-token prediction. 

> > This is great work and I'm looking forward to try it out. For multi-token prediction, is this planned to be implemented in this PR via the `num_nextn_predict_layers` attribute in the config?
> 
> thanks for attention, I don't have plan to implement multi-token prediction for this time since there is no additional parameters for additional layer which is required for multi-token prediction.

The weights are provided. DeepSeek provided a [neat description](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/README_WEIGHTS.md#2-multi-token-prediction-mtp-modules) of this. Since `num_nextn_predict_layers=1` and `num_hidden_layers=61`, the MTP module is layer 61.

![image](https://github.com/user-attachments/assets/8977b429-8b55-4dae-9655-47ffc8640fdb)


> > > This is great work and I'm looking forward to try it out. For multi-token prediction, is this planned to be implemented in this PR via the `num_nextn_predict_layers` attribute in the config?
> > 
> > 
> > thanks for attention, I don't have plan to implement multi-token prediction for this time since there is no additional parameters for additional layer which is required for multi-token prediction.
> 
> The weights are provided. DeepSeek provided a [neat description](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/README_WEIGHTS.md#2-multi-token-prediction-mtp-modules) of this. Since `num_nextn_predict_layers=1` and `num_hidden_layers=61`, the MTP module is layer 61.
> 
> ![image](https://private-user-images.githubusercontent.com/27340033/408114906-8977b429-8b55-4dae-9655-47ffc8640fdb.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzgyMzk3OTYsIm5iZiI6MTczODIzOTQ5NiwicGF0aCI6Ii8yNzM0MDAzMy80MDgxMTQ5MDYtODk3N2I0MjktOGI1NS00ZGFlLTk2NTUtNDdmZmM4NjQwZmRiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAxMzAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMTMwVDEyMTgxNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTI1YTRkMTZjOGRjZDhkYzg4OGExZDdmNWRlOGNjNjg2M2Y4ZWI3MWE1OWI0ZDgzYzlmYzAxN2EwZTAyOGExOTcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.gYyjZtGa72Twm6_YM5ZPeKk9Siy796mUjUfO6fjQxWU)

I missed this! thanks for the reference, let me check how to deal with it.

Do you want me to jump on the PR and help you merge this faster @bzantium ? 🤗 

> Do you want me to jump on the PR and help you merge this faster @bzantium ? 🤗

Of course! As you commented, it looks like there's still a lot to work left. (code optimization, training code, testing code and multi-token prediction if possible) 
to: @ArthurZucker 

thanks for refactoring! I more optimize `moe_infer` and make it trainable so rename it to `moe`. 
also, I revised `load_hook` to properly work as intended. At last, I rolled back to use Llama instead of Mixtral since there exists not to match each other.
to: @ArthurZucker 

Nice, put it a bit on pause talked to @NouamaneTazi about how we would go about "training" this, as TP would be a little bit hard (you said so as well). We need to enable the layer norm to use TP 

> Nice, put it a bit on pause talked to @NouamaneTazi about how we would go about "training" this, as TP would be a little bit hard (you said so as well). We need to enable the layer norm to use TP

Great, I think applying expert parallel instead of tensor parallel would be better in this case if possible. (may need to larger change than TP for `transformers` to apply generally). Original code provided direct expert parallelism implementation for moe.
ref: https://pytorch.org/blog/training-moes/

I found that `_register_load_state_dict_pre_hook` is not executed when using `tp_plan=auto`. so I use `reshape_for_rope` instead of `load_hook` like following:
```python
cos, sin = position_embeddings
q_rot, k_rot = self.reshape_for_rope(q_rot, k_rot)
q_rot, k_rot = apply_rotary_pos_emb(q_rot, k_rot, cos, sin)
...

def reshape_for_rope(self, q: torch.Tensor, k: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
    b, h, s, d = q.shape
    q = q.view(b, h, s, d // 2, 2).transpose(-1, -2).reshape(b, h, s, d)
    b, h, s, d = k.shape
    k = k.view(b, h, s, d // 2, 2).transpose(-1, -2).reshape(b, h, s, d)
    return q, k
```

In addition, following the `TP` [vllm reference](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/deepseek_v3.py), I revised Attention forward and add more modules to `base_model_tp_plan`.
```python
base_model_tp_plan = {
    "layers.*.gate_proj": "colwise",
    "layers.*.up_proj": "colwise",
    "layers.*.down_proj": "rowwise",
    "layers.*.self_attn.q_b_proj": "colwise",
    "layers.*.self_attn.kv_b_proj": "colwise",
    "layers.*.self_attn.o_proj": "rowwise",
}
```
to: @ArthurZucker 

Yep sounds good! I'll be a bit slow today! 
We should enforce the use of the load hook, otherwise rotating for rope is gonna be a real issue in terms of performances! 

A few things for the todo-list: 
- Layer-norm probably needs a special layer replacement, if I am not mistaken
- The load balancing function for now is the one we usually use, I need to check on the paper if it's the same or not
- add `megablock` as a soft dependency to try and use kernels ? 
- make sure we support fp8 (cc @SunMarc) ! 

I need to run a bit of maintenance today so don't worry, but I'll take a look whenever I can here! 

I fixed it by using both `_register_load_state_dict_pre_hook` and `_register_state_dict_hook`. I got the same results with/without `TP`.
```python
class DeepseekV3Model(LlamaModel):
    def __init__(self, config):
        super().__init__(config)
        self._register_load_state_dict_pre_hook(self.load_pre_hook)
        self._register_state_dict_hook(self.load_hook)
        self.post_init()

    def load_pre_hook(self, state_dict, prefix, *args):
        """
        Weights have to be permuted for correct rope formulation. We can't do this in the weights
        as every other framework already uses the `Llama` original function (which is copyrighted btw).
        And I am not even sure it's better.... anyways end of my rant
        """

        def permute_for_rope(input_tensor):
            """
            When you go from the complex ROPE formulation to sin and cos one, you need
            to permute the query and key weights (to avoid doing it on the fly)
            """
            n_heads, dim1, dim2 = input_tensor.shape[0], input_tensor.shape[1], input_tensor.shape[2]
            input_tensor = input_tensor.reshape(n_heads * dim1, dim2)
            input_tensor = input_tensor.view(n_heads, dim1 // 2, 2, dim2)
            input_tensor = input_tensor.transpose(1, 2).reshape(n_heads, dim1, dim2)
            return input_tensor

        def permute_layer_for_rope(key, num_heads, head_dim, rope_dim):
            weight = state_dict[key]
            weight = weight.view(num_heads, head_dim, -1)
            weight_rot = weight[:, -rope_dim:]
            weight_rot = permute_for_rope(weight_rot)
            weight[:, -rope_dim:] = weight_rot
            weight = weight.view(-1, weight.shape[-1])
            state_dict[key] = weight

        for k in state_dict:
            if "q_b_proj." in k:
                permute_layer_for_rope(
                    k,
                    num_heads=self.config.num_attention_heads,
                    head_dim=self.config.qk_head_dim,
                    rope_dim=self.config.qk_rope_head_dim,
                )
            if "kv_a_proj_with_mqa." in k:
                permute_layer_for_rope(
                    k,
                    num_heads=1,
                    head_dim=self.config.kv_lora_rank + self.config.qk_rope_head_dim,
                    rope_dim=self.config.qk_rope_head_dim,
                )

    def load_hook(self, module, state_dict, prefix, *args):
        self.load_pre_hook(state_dict, prefix, *args)
```
@ArthurZucker 

Yep looks alright. 
For testing quickly we can create a tiny model (this will help in our CI) and I'll be able to download it as well ! 

I am currently using this model (5.35B) for testing, same config but with 6 layers and less experts. I will upload much smaller model later if you need.

https://huggingface.co/bzantium/tiny-deepseek-v3

to: @ArthurZucker 

Perfect!

Sorry to jump in here. But I read the DeepSeek-V2 paper in detail, and noted that the implementation they published, which you seem to be using here, is really quite inefficient, and somewhat at odds to what they write in their paper. This also applies to https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py.

To explain:
The key idea in the paper is to use low rank approximations to Q, K, V tensors. This reduces KV cache size. But it also leads to more efficient computations, even in training.

To this end, you have to avoid computing the Q, K, V tensors, but instead work in the lower ranks. This is quite possible (I worked it out), but is not properly detailed in their paper.

What is real funny: This trick of extending K and Q by an extra part for RoPE encoding: the only reason this is done in the paper is to allow not to have to compute Q, K explicitly. If you compute them explicitly (as is done here), you can just as well apply RoPE encoding to them!

There is a way to make this work so that the role of K and V is played by a single tensor of shape `(batch_size, 1, q_len, kv_lora_rank + qk_rope_head_dim)`. There is no `num_attention_heads` here. And single tensor, because "V" is really just a part of "K". The role of Q is played by a tensor of shape `(batch_size, num_attention_heads, q_len, kv_lora_rank + qk_rope_head_dim)`, and dot products are computed between these.

When I says "plays the role of K, ...", I mean that you call `torch.nn.functional.scaled_dot_product_attention` with these, instead of the original Q, K, V.

Ahhhhh very nice. 
Indeed I usually base stuff off code rather than papers, would be nice to get an optimized version 

thanks for the great comment! Could you suggest code block to apply? Following what you mention, attention could be applied to compressed latent instead of QKV, but I cannot directly imagine how it is possible. for Cache, I see what you meant; we need to cache latent KV not original K,V.
to: @mseeger 

Talked to @zzhhjjj and seems like if we go for the optimized version we won't be able to have TP (Tensor Parallel)

What is TP? The optimized version I am referring to is just a rewriting of the maths.

I am working for a company, and have asked whether I am allowed to contribute this. If they give me green light, I am happy to do it. I think HuggingFace should have the best code possible.

It does not have to delay this PR here. If this goes in first, I can still open a second PR, which would provide (1) an alternative to the Attention class, and also some code to convert from the current weights to the new ones. The new formulation uses somewhat different weights, because the *_b tensors are combined into one (more or less).

BTW: This is all I have, I did not look at the MoE part or anything else. I also did not run any comparisons.

As for the cache: You only have to store what is called `C_KV` and `K_R` in the DeepSeek-V2 paper. You return the concat of these two as K tensor, and C_KV alone as V tensor.

I agree with @bzantium that me jumping in should not delay this PR, also because frankly I need to get approval from myu employer. So please do go ahead. I'd then make my contribution afterwards. However, I'd appreciate help then, especially with doing testing.

> I agree with @bzantium that me jumping in should not delay this PR, also because frankly I need to get approval from myu employer. So please do go ahead. I'd then make my contribution afterwards. However, I'd appreciate help then, especially with doing testing.

If you can implement testing codes, it would be helpful greatly!

Sure @bzantium , I'd do that. I'd implement tests which ensure that the computations give the same result. I'll also provide code to convert the weights.

Hello, I more or less obtained permission to contribute, and I am working on this now.

I looked at it more carefully, and here is the story:

- Your code `DeepseekV3Attention`, equivalent to what has been open sourced by Deepseek, is good for training
- But for inference, it has major issues. First and foremost, the whole point about MLA (as opposed to just using standard MHA) is that the KV cache is much smaller. But this is not the case here.
- What I'll contribute fixes that, so the KV cache will be as small as intended for MLA, and the computations for inference will also be better. However, for training, you still want to use the original code here, which is slightly faster.
- This means that when creating the model, the user should decide whether it is used for training / fine-tuning, or for inference. I believe this is already part of the config anyway.

An alternative would be a bit more complex. @bzantium , @ArthurZucker , any opinion?

Could also use a single `DeepseekV3Attention`, which does something different, depending on `self.training`. If `True`, it would be the current code. If `False` (so inference), it would be the code I'll contribute, and the weights are determined from the original weights.

The advantage of this would be everything is behind the scenes, the user would not notice anything.

@mseeger I just sent you invitation as collaborator. I think that's great idea (splitting train/inference). You can commit to feature/#35425 branch or give suggestion.

@bzantium Thanks. I'll send you a pull request into this branch once I am ready.

For now, I'll try to make this a single class. One drawback of this is internal complexity (but easy to use to the outside). Another one is that it needs a bit more memory.

As I said, the alternative is the user decides up front which version they want.

FP8 support for deepseek have landed https://github.com/huggingface/transformers/pull/36026

@bzantium I finished coding the advanced MLA, am just writing a few tests. I'll send a PR against your branch in the next days

Sure, we could just as well do quantisation on the KV cache content on top, but maybe leave this for another PR.

@mseeger Awesome! I will check when you send PR :)

OK @bzantium , I sent a PR into your feature branch, so you can see how it all works. It also has some tests.
Let me know what you think.

Then, we'd need to decide how to best progress with working on this together.

I'll review this asap!

@mseeger yes, need to discuss better way to work together on this PR. another PR on the branch disperses attention. 
@ArthurZucker please check: https://github.com/bzantium/transformers/pull/1

Running this with the big model! 

@bzantium if you have time to add the tiny logit tests would be awesome!

Cool to see this happening. One question I have: Is `modular_deepseek_v3.py` already up-2-date? It seems to me that `DeepseekV3Model` is not using the `DeepseekV3DecoderLayer` and `DeepseekV3RotaryEmbedding`. My best guess you first try and get everything running for `modelling_deepseek_v3.py`, and worry about this modular later?

I see them both being used! 

@ArthurZucker I add test for deepseek-v3 but it fails on some tests but I cannot find why. could you help?

Yeah for sure!

I actually run in a few issue with the load hooks on TP=16, 2 nodes, chasing this a bit as well 

@ArthurZucker  If you find why, let me know, will check too. still looking into implementation.

Hello @bzantium , I noted a few things about your code that are probably wrong. I'll send a PR against your branch shortly.


OK, I sent a PR against the branch with some fixes.

One reason for the remaining failures could be that the head size of K and V tensors is different than normal, because you have these additional `qk_rope_head_dim` entries. May mean one has to generalize some of the common tests.

Based on the test logs, I found two reasons:
1) as @mseeger said, some tests are not compatible for multi latent attention (checking head_dim) so I skipped some tests (maybe need to skip more).
2) because of load_pre_hook, load and save tensors become different so we need to add save_hook or remove load_hook.

Ah interesting, that is indeed an issue (load / save) to be careful about. 

Okay, I'll give it another shot in a bit!  

> Left some comments with fixes

Thanks for the comments! you can give suggestions directly like below! (not just text)
click Add a suggestion button after dragging where to fix, and replace original code with your code. 

<img width="640" alt="image" src="https://github.com/user-attachments/assets/a8817b3e-1410-48f5-80f0-e2baf46640fc" />

Also, you better give a suggestion on `modular_deepseek_v3.py` because `modeling_deepseek_v3.py` is automatically generated using modular file.

I found more reasons to fail:
```
1 failed because `AssertionError: nan not found in [0.0, 1.0] ` -> Parameter layers.2.mlp.gate.weight of model <class 'transformers.models.deepseek_v3.modeling_deepseek_v3.DeepseekV3Model'> seems not properly initialized
   1 failed because `AssertionError: False is not true ` -> model.layers.2.mlp.experts.0.gate_proj.weight in DeepseekV3ForCausalLM has no gradient!
   2 failed because `AssertionError: DeepseekV3Model: Tensor layers.2.mlp.gate.weight` -> Tensor-likes are not close!
   2 failed because `AssertionError: False is not true ` -> model.layers.2.mlp.experts.4.gate_proj.weight in DeepseekV3ForCausalLM has no gradient!
```

1. first one is because weight is initialized with `torch.empty`.
`self.weight = nn.Parameter(torch.empty((self.n_routed_experts, config.hidden_size)))`

2. "has no gradient" problem is maybe because of topk selection which is key of the moe. I think this is mainly because how I implement `get_topk_indices` for `router`.

Is there something I can help with?

Sorry have been working on #36335  to give us the tools to run the model as it was very very slow just loading the full checkpoint! 

This is a smaller, trained model using the DeepSeek V3 architecture. In BF16, not FP8. Might be helpful :) 
https://huggingface.co/moonshotai/Moonlight-16B-A3B-Instruct

Hi @ArthurZucker , thanks for the efforts! Do we have any plans for this PR? We are checking whether we could fine-tune deepseek-r1 through huggingface trainer. 

@bzantium , are we blocked on some particular errors?

Just a heads up. Recent new OS releases by DeepSeek seem pretty cryptic, mostly CUDA code for some specific GPUs (who can blame them?). But IMHO, this reinforces the need to have their basic architecture in Hugging Face, I mean in `src/transformers/models/`, so the community can experiment with it, on any GPU they like. Such a generic implementation could still call the DeepSeek code if the hardware happens to be the one they support, but this to me would come later.

Also, I'd be keen to contribute proper inference code, once this is in here.

@mseeger @ArthurZucker Thanks for the information! We look forward to trying it once the PR is closed😁

Completely agree with @mseeger ! 
My hands are taken by loads of other releases happening on the side, but I made sure to refactor TP to work well. 
I don't really know the status of the PR TBH but before we merge we need someone to run the full model making sure generations are correct! That is one of the blockers I think! 

@bzantium do you think it is ready (last message had still some failures) I would not focus on CI tests but more on the integration tests with dummy model but important features (MLA etc) 

I was focused on passing the CI tests but it's a quite bottleneck for now. As you said it would be better to successfully run generation first and merge it if it is okay. I will work on this then and share soon!
to: @ArthurZucker  

Yep, the CI should normally help but you can skip stuff if it's blocking !

@ArthurZucker @bzantium is this PR ready to be merged? Is there anything blocking? Also can we load fp8 weights with this PR? 

I cannot help with running the full model.

Anyway, once this is in, I do have follow-up PRs, which taken together make sure that multi-head latent attention is properly used during generation. As it stands, KV cache size is just as large as with a normal model (Llama, say), so you may as well just not do it. The DeepSeek models have been trained with MLA for the explicit reason that inference becomes more efficient.

- https://github.com/huggingface/transformers/pull/36218: This is required to exploit the reduced shape of key, value in MHA computations. It stands on its own
- New PR with the code that does MLA different for inference and leads to much smaller KV caches

FYI I've been working on some fp8 dequantize support
https://github.com/huggingface/transformers/compare/main...kylesayrs:transformers:kylesayrs/support-fp8-decompression#diff-faf0f03d7fcee1138b40be5d19010f2e143296983071d3cb31f5de3bc4f02b46R434-R444

Pushed a few things, only checkpoint issues:
 ```
Some weights of the model checkpoint at deepseek-r1 were not used when initializing DeepseekV3ForCausalLM: model.layers.61.enorm.weight
        model.layers.61.shared_head.norm.weight
        model.layers.61.eh_proj.weight
        model.layers.61.embed_tokens.weight
        model.layers.61.shared_head.head.weight
        model.layers.61.hnorm.weight
```
which should be easy to fix. 
TP works with multi-node but FP8Linear layers are breaking the forward pass. Still need this to go through!

Hi! Thanks for the contribution. I just wonder if your code has been tested working in training? asking as we found out that `e_score_correction_bias` makes training hanging during back propagation using deepspeed. 

Since `e_score_correction_bias` does not contribute to the loss, also wonder what is the rationale behind to remove `@torch.no_grad` on top of `get_topk_indices` method in GateMoe layer

Not tested for training yet gimme a bit of time :) 

```python
["<｜begin▁of▁sentence｜>Hey how are you doing? This is\nDr. Suresh Shenvi. I'm a double\nboard-certified plastic surgeon here\nin Hawaii. So today's topic is going to\nbe on the differences between silicone\nand saline implants. So this is probably\n"]
``` 
here are my generation hahah need to debug a but pushing latest

Okay output is correct, model thinks! Will merge this soon! 

Okay our HUB has small issues but these are the only tests left to pass:
```python
FAILED tests/models/deepseek_v3/test_modeling_deepseek_v3.py::DeepseekV3ModelTest::test_can_use_safetensors - AssertionError: DeepseekV3ForCausalLM: Tensor model.layers.4.mlp.gate.weight: Tensor-likes are not close!
FAILED tests/models/deepseek_v3/test_modeling_deepseek_v3.py::DeepseekV3ModelTest::test_feed_forward_chunking - AssertionError: Tensor-likes are not close!
FAILED tests/models/deepseek_v3/test_modeling_deepseek_v3.py::DeepseekV3ModelTest::test_from_pretrained_no_checkpoint - AssertionError: False is not true
FAILED tests/models/deepseek_v3/test_modeling_deepseek_v3.py::DeepseekV3ModelTest::test_initialization - AssertionError: -0.0012460040161386132 not found in [0.0, 1.0] : Parameter layers.2.mlp.gate.weight of model <class 'transformers.models.deepseek_v3.modeling_deepseek_v3.DeepseekV3Model'> seems not properly initialized
FAILED tests/models/deepseek_v3/test_modeling_deepseek_v3.py::DeepseekV3ModelTest::test_left_padding_compatibility - AssertionError: Tensor-likes are not close!
FAILED tests/models/deepseek_v3/test_modeling_deepseek_v3.py::DeepseekV3ModelTest::test_load_save_without_tied_weights - AssertionError: DeepseekV3Model: Tensor layers.4.mlp.gate.weight: Tensor-likes are not close!
[DeepseekV3ForCausalLM] SUBFAIL tests/models/deepseek_v3/test_modeling_deepseek_v3.py::DeepseekV3ModelTest::test_training_gradient_checkpointing - AssertionError: False is not true : model.layers.2.mlp.experts.4.gate_proj.weight in DeepseekV3ForCausalLM has no gradient!
[DeepseekV3ForCausalLM] SUBFAIL tests/models/deepseek_v3/test_modeling_deepseek_v3.py::DeepseekV3ModelTest::test_training_gradient_checkpointing_use_reentrant - AssertionError: False is not true : model.layers.2.mlp.experts.2.gate_proj.weight in DeepseekV3ForCausalLM has no gradient!
[DeepseekV3ForCausalLM] SUBFAIL tests/models/deepseek_v3/test_modeling_deepseek_v3.py::DeepseekV3ModelTest::test_training_gradient_checkpointing_use_reentrant_false - AssertionError: False is not true : model.layers.2.mlp.experts.2.gate_proj.weight in DeepseekV3ForCausalLM has no gradient!
```

All tests pass locally! Will merge, as I said super happy to have contributions to improve! 

sorry for not updating code from my side for a long time and thanks for your contributions! I was quite busy to take care of my baby 😢. Happy to see all tests are passed!

Thanks a lot for you help @bzantium ! 🤗 kudos to you and @mseeger !

Oups sorry 

It's totally okay, great work!

Hi everyone, thanks a lot for contributions! Does that mean we have support in HF to load the checkpoints in FP8 directly? or we need convert it firstly in bf16 using DeepSeek's script?

It load fp8 firectly!

Conversion is needed to run bf16 tho, or PR to support on the fly de-compression of the FP8 !


@mseeger sorry for late merge, now you can create issue and give PR for efficient inference!