Add DeepSeek V2 Model into Transformers
Author: VladOS95-cyber
Pull request: yes
Labels: name: New model

# What does this PR do?

DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models.

This PR add DeepSeekV2 into the Transformers library. There is a new thing in transformers called modular, which adds new models by creating a modeling_modelname.py file. Since DeepSeekV2 could reuse some Llama arch parts, it serves as an ideal use case for this modular approach.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. Link: https://github.com/huggingface/transformers/issues/35317
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@ArthurZucker @Rocketknight1

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @arthurzucker let me know if you want me to take the first review here!

Hi @Rocketknight1! The main structure as well as most code of the model is done, except unit tests which I add later. Just one thing that really bothers me is that model produces always the same token. I don't know how many times I checked everything compared with original implementation but from my point of view, everything is absolutely logically the same, I compared hidden_states from every logical layer as well, it was identical. But anyway, model produces gubberish. I keep investigating of course, but I am really confused. Maybe is there anything I really missed?
For now, I check everything on deepseek-ai/DeepSeek-V2-Lite.

hi @VladOS95-cyber, that's a very odd bug! Try comparing the output logits for an input sentence rather than just the hidden states - if those are identical, then I don't understand how this model could generate gibberish but the original model could generate correct text.

Hey @Rocketknight1! I found an issue, just one line was accidentally removed in forward prop of Decoder layer.. I think there was a mistake in between commits and I did not notice it as it was removed after I compared states. I fixed that and model works as expected! So, now, I am going to work on unit tests and some small refactoring in names, types and so on.

Hi @Rocketknight1! I added unit and integration tests and did small refactoring. So, this PR is completely ready for review.

hi @VladOS95-cyber, take a look in the CI! There are still some test failures. You may also need to add some classes to the documentation - look at the specific errors in "check_repository_consistency" and check out other model addition PRs to see where the class autodoc lines should go

Hi @Rocketknight1! All tests are green

Hi @ArthurZucker @Rocketknight1! I resolved all comments I could or left mine in response.

Thanks! before merging the PR we need to make sure the modeling code works! (big and small) do you have access to enough compute to do that or should we do it ? 


> Thanks! before merging the PR we need to make sure the modeling code works! (big and small) do you have access to enough compute to do that or should we do it ?

Hi @ArthurZucker! Unfortunately I don't, I've already spent everything on testing during development..

Hi @ArthurZucker! Do I have to do anything else at this moment? 

Nope I got caught up in the Llama4 sprint, waiting for nodes and will test it out when I can! 

> Nope I got caught up in the Llama4 sprint, waiting for nodes and will test it out when I can!

Sure, got it, thank you for the answer!

cc @Cyrilvallez as I will be off for two weeks!

Hi @Cyrilvallez! What's the next steps will be here?

Hey! Sorry for the delay! Could you make sure to rebase to fix the conflicts? The init system has been greatly simplified in https://github.com/huggingface/transformers/pull/36827. That will help me test the model with the TP plan, as we indeed need to make sure it works!

Hi @Cyrilvallez! I have just merged everything and fixed conflicts. 

Hi @Cyrilvallez! Do you have any news so far or you have been occupied by smth else?

Hi @ArthurZucker and @Cyrilvallez! I have not heard anything from you quite a while, tell me please, we are still interested in this implementation or it is already obsolete?

We are ! Super sorry 😢 

Hi @ArthurZucker! Thank you for your comments, sry I was in vacation, I'll try to take a look asap in the nearest days

Hey @ArthurZucker! I resolved your comments and pushed new changes. Just regarding unit tests I did not understand the point. 

Hi @Cyrilvallez! I resolved all your comments, please, take a look again. 

Hi @Cyrilvallez! Is anything else missed or should I do?

Hi @Cyrilvallez! I resolved your comments except two and I just provided explanation why it is done this way. Please, take a look.

run slow: deepseek_v2

run-slow: deepseek_v2

This comment contains run-slow, running the specified jobs: 

models: ['models/deepseek_v2']
quantizations: [] ...

> Allright, LGTM!! Thanks a lot for the efforts, and for bearing with us 🤗 Great work!

Thank you for support along the way!

@VladOS95-cyber as you can see, some slow tests are still failing and should be taken care of 🤗

For IntegrationTests resulting in OOM, deepseek-ai/DeepSeek-V2-Lite is a bit too large, please run it with some kind of quantization to make it smaller for the test runner

> @VladOS95-cyber as you can see, some slow tests are still failing and should be taken care of 🤗

From pytest/custom-tests, yeah, I take a look. Torch tests are failed for a different model though. 

> For IntegrationTests resulting in OOM, deepseek-ai/DeepSeek-V2-Lite is a bit too large, please run it with some kind of quantization to make it smaller for the test runner

@Cyrilvallez  But quantization has never been implemented for V2 right? I mean GGUF one, or is there another workaround to do that easily, at least for tests?

All our quantization methods should work out of the box, you can use e.g. fp8 https://huggingface.co/docs/transformers/en/quantization/finegrained_fp8

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, deepseek_v2

Hey @Cyrilvallez! I made some fixes, let's try to trigger tests again please

run-slow: deepseek_v2

This comment contains run-slow, running the specified jobs: 

models: ['models/deepseek_v2']
quantizations: [] ...

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, deepseek_v2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, deepseek_v2

Looks like fp8 will not run on older hardware, my bad I forgot. Let's use bit and bytes instead then. Sorry about that

run-slow: deepseek_v2

This comment contains run-slow, running the specified jobs: 

models: ['models/deepseek_v2']
quantizations: [] ...

> Looks like fp8 will not run on older hardware, my bad I forgot. Let's use bit and bytes instead then. Sorry about that

Ok, sure, so I will remove fp8 quantization then and try to apply another logic for quantization

Just another quantiozation method instead: https://huggingface.co/docs/transformers/en/quantization/bitsandbytes

@Cyrilvallez I just tried to use `BitsAndBytesConfig(load_in_8bit=True)` for quantization and model produces very different output compared with normal usage. But I ran it on cpu, so i suppose it should be way better on gpu, but I cannot test that unfortunately, but we should not expect a big difference in quantized/not quantized version, right?With not quantized model all tests are green. Should I push it anyway and we can run tests again or?

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, deepseek_v2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, deepseek_v2

run-slow: deepseek_v2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, deepseek_v2

This comment contains run-slow, running the specified jobs: 

models: ['models/deepseek_v2']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36400). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, deepseek_v2

run-slow: deepseek_v2

This comment contains run-slow, running the specified jobs: 

models: ['models/deepseek_v2']
quantizations: [] ...

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, deepseek_v2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, deepseek_v2

Hi @Cyrilvallez! Thank you so much for taking care of tests, really appreciate that and thank you for your support! I am very glad that it is completed!

@VladOS95-cyber really nice work!
I just wanted to ask where can I find the HF compatible weights? Or is it supposed to work with the original weights from this official collection: [collections/deepseek-ai/deepseek-v2](https://huggingface.co/collections/deepseek-ai/deepseek-v2-669a1c8b8f2dbc203fbd7746)

> @VladOS95-cyber really nice work! I just wanted to ask where can I find the HF compatible weights? Or is it supposed to work with the original weights from this official collection: [collections/deepseek-ai/deepseek-v2](https://huggingface.co/collections/deepseek-ai/deepseek-v2-669a1c8b8f2dbc203fbd7746)

Hi, thank you! It is supposed to work with original weights from official DeepSeek HF repo.