Add deepseek 3.2 exp
Author: ArthurZucker
Pull request: yes
Labels: name: New model

# What does this PR do?

```python
from transformers import FineGrainedFP8Config, AutoModelForCausalLM, AutoTokenizer
import torch 


model_name = "deepseek-ai/DeepSeek-V3.2"
quantization_config = FineGrainedFP8Config(
    modules_to_not_convert=["model.layers.*.mlp.gate.*", "*.self_attn.indexer.weights_proj.*"],
    weight_block_size = (128, 128),

)
quantized_model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto", quantization_config=quantization_config)

tokenizer = AutoTokenizer.from_pretrained(model_name)
input_text = "What are we having for dinner?"
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda:0")

output = quantized_model.generate(**input_ids, max_new_tokens=10)
print(tokenizer.decode(output[0], skip_special_tokens=True))

```


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41251). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hello, thanks for your support for deepseek v3.2! I wonder when this PR will be ready?

Working on it! Hoping by next week 🤗 


wow this got old! 

The submitted code is currently unusable and does not support the Deepseek-v3.2 official version. Is this PR still being updated?

> The submitted code is currently unusable and does not support the Deepseek-v3.2 official version. Is this PR still being updated?

https://github.com/yunkchen/transformers/tree/v4.57.3_add_dpskv32

> > The submitted code is currently unusable and does not support the Deepseek-v3.2 official version. Is this PR still being updated?
> 
> https://github.com/yunkchen/transformers/tree/v4.57.3_add_dpskv32

There is stillDeepseekV32Attention a bug when using LLMC to quantify the model:[rank0]: Traceback (most recent call last):
[rank0]:   File "/mnt/hcufs/env_scripts/xyf_test/llmc/llmc/__main__.py", line 248, in <module>
[rank0]:     main(config)
[rank0]:   File "/mnt/hcufs/env_scripts/xyf_test/llmc/llmc/__main__.py", line 27, in main
[rank0]:     model = MODEL_REGISTRY[config.model.type](config)
[rank0]:   File "/mnt/hcufs/env_scripts/xyf_test/llmc/llmc/models/deepseekv3.py", line 20, in __init__
[rank0]:     super().__init__(config, device_map, use_cache)
[rank0]:   File "/mnt/hcufs/env_scripts/xyf_test/llmc/llmc/models/base_model.py", line 40, in __init__
[rank0]:     self.build_model()
[rank0]:   File "/mnt/hcufs/env_scripts/xyf_test/llmc/llmc/models/deepseekv3.py", line 40, in build_model
[rank0]:     self.model = AutoModelForCausalLM.from_pretrained(
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 604, in from_pretrained
[rank0]:     return model_class.from_pretrained(
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py", line 277, in _wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4971, in from_pretrained
[rank0]:     model = cls(config, *model_args, **model_kwargs)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/transformers/models/deepseek_v32/modeling_deepseek_v32.py", line 555, in __init__
[rank0]:     self.model = DeepseekV32Model(config)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/transformers/models/deepseek_v32/modeling_deepseek_v32.py", line 477, in __init__
[rank0]:     [DeepseekV32DecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/transformers/models/deepseek_v32/modeling_deepseek_v32.py", line 477, in <listcomp>
[rank0]:     [DeepseekV32DecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/transformers/models/deepseek_v32/modeling_deepseek_v32.py", line 404, in __init__
[rank0]:     self.self_attn = DeepseekV32Attention(config=config, layer_idx=layer_idx)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/transformers/models/deepseek_v32/modeling_deepseek_v32.py", line 316, in __init__
[rank0]:     self.softmax_scale = self.qk_head_dim**-0.5
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1928, in __getattr__
[rank0]:     raise AttributeError(
[rank0]: AttributeError: 'DeepseekV32Attention' object has no attribute 'qk_head_dim'

> https://github.com/yunkchen/transformers/tree/v4.57.3_add_dpskv32

New commit pushed, sorry.

> > https://github.com/yunkchen/transformers/tree/v4.57.3_add_dpskv32
> 
> New commit pushed, sorry.

A bug still occurred when running the LLMC quantization model:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/mnt/hcufs/env_scripts/xyf_test/llmc/llmc/__main__.py", line 248, in <module>
[rank0]:     main(config)
[rank0]:   File "/mnt/hcufs/env_scripts/xyf_test/llmc/llmc/__main__.py", line 55, in main
[rank0]:     model.collect_first_block_input(calib_data, padding_mask)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/mnt/hcufs/env_scripts/xyf_test/llmc/llmc/models/base_model.py", line 246, in collect_first_block_input
[rank0]:     self.model(**data)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper
[rank0]:     output = func(self, *args, **kwargs)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/transformers/models/deepseek_v32/modeling_deepseek_v32.py", line 526, in forward
[rank0]:     outputs: BaseModelOutputWithPast = self.model(
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]: TypeError: check_model_inputs.<locals>.wrapped_fn() got an unexpected keyword argument 'input_ids'

> > https://github.com/yunkchen/transformers/tree/v4.57.3_add_dpskv32
> 
> New commit pushed, sorry.

I'm seeing problems in this branch with rope factor/beta_fast/beta_slow values not being floats. Is this an oversight?

> > The submitted code is currently unusable and does not support the Deepseek-v3.2 official version. Is this PR still being updated?
> 
> https://github.com/yunkchen/transformers/tree/v4.57.3_add_dpskv32

Isn't this implementation still O(L^2) since it just masks full attention to the indexer's topk?

@ArthurZucker Is this ready to merge? I'd really love to experiment with some DeepSeek 3.2 Speciale fine tunes.

@ArthurZucker @yunkchen 
hello， Has there been any progress?

@ArthurZucker Happy holidays, checking in again :)

Can we get this merged please?

Hey! Thanks, just got back from holidays, we shipped https://github.com/huggingface/transformers/blob/57278c904c5158999d31a0db8bfcd63360c37b48 but now I should be able to get back! sorry for the delay everyone v5 needed a slow down in model addition to support all the new features, especially default FP8 weight support!

Thanks @ArthurZucker, do you have an ETA? Getting this in would be massively helpful to me and the community. Happy to help however I can.

Hi @ArthurZucker, just checking in to see if there are any updates on this? There is a lot of interest for this change, so we’re excited to see it move forward! :) Thanks!

Hey all! Yeah I am mega underwater! shipping v5 took / takes longer than we thought, but its still planned. Hopefully for end of week! really sorry its very vague but yeah should be alright 

UPDATE: #44054 is gonna take a bit of time so I'll finish this one first

Hi @ArthurZucker, Checking if there is any new updates. Look forward to the DeepSeek3.2 feature!

+1 @ArthurZucker, getting this in would be extremely helpful

@ArthurZucker Are we still blocked on #44054? I believe that GLM-5 added DSA support that's already merged into HF transformers.

yeah i am sorry 

glm <=> deepseek 

Looking forward to the merge asap

Thank you so much for your continue work on this @ArthurZucker. Is it ready to merge?

@michaelroyzen Just asked for a review, will be today I think! 

Follow up PR will be with the flash_mla interface 


run-slow: auto, deepseek_v3, deepseek_v32, glm4_moe_lite, glm_moe_dsa, longcat_flash, mistral4, youtu

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27287063571)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/auto", "models/deepseek_v3", "models/deepseek_v32", "models/glm4_moe_lite", "models/glm_moe_dsa", "models/longcat_flash", "models/mistral4", "models/youtu"]
quantizations: []

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, deepseek_v3, deepseek_v32, glm4_moe_lite, glm_moe_dsa, longcat_flash, mistral4, youtu