feat(deepseek_v3): Add grouped GEMM kernel for faster MoE computation
Author: bzantium
Pull request: yes
Labels: name: Mixture of Experts

# What does this PR do?
This PR introduces an optimization that significantly accelerates the Mixture-of-Experts (MoE) layer computations in the `DeepseekV3` model by integrating the `grouped_gemm` library. This enhances performance for both training and inference.

The original MoE implementation processed expert networks sequentially using a Python loop, which created a performance bottleneck due to high GPU kernel launch overhead. This PR addresses that issue with the following key changes:

  * **🚀 Grouped GEMM Kernel Integration**: A new `grouped_forward` operational path replaces the iterative Python loop over experts with a single, high-performance kernel call from the `grouped_gemm` library. This minimizes GPU overhead and maximizes throughput.

  * **🧩 Expert Module Fusing**: To efficiently leverage the `grouped_gemm` kernel, this PR implements a `fuse_experts()` utility and a `GroupedDeepseekV3MLP` module. These tools combine the weights of multiple experts into a single, contiguous tensor, which is a prerequisite for the kernel.

  * **⚙️ Configuration and Usability**: A `use_grouped_gemm` flag has been added to `DeepseekV3Config` to enable this optimization.
    **Important**: If you set `use_grouped_gemm=True` directly when loading a model (e.g., in `.from_pretrained()`), you must provide a `state_dict` where the expert weights have already been fused. For standard checkpoints, the recommended workflow is to load the model normally and then call the `model.fuse_experts()` method.

  * **⚠️ Dependency Handling**: The model now gracefully handles the optional dependency. If `use_grouped_gemm` is enabled but the library isn't found, it raises a clear `ImportError` with installation instructions.

-----

## **How to use**

1.  **Install the required library**:

    ```bash
    pip install git+https://github.com/fanshiqing/grouped_gemm@main
    ```

2.  **Load a standard model and fuse the experts after loading** (Recommended method):

    ```python
    from transformers import AutoModelForCausalLM

    # Load a standard checkpoint from the Hub
    model = AutoModelForCausalLM.from_pretrained(
        "moonshotai/Moonlight-16B-A3B-Instruct",
        device_map="cuda:0",
        # Do not set use_grouped_gemm=True here
    )

    # Fuse the experts in-place
    model.fuse_experts()

    # The model is now optimized and ready for faster inference or training
    ```

For checkpoints that have already been saved in the fused format, you can load them directly by setting `use_grouped_gemm=True` in the `.from_pretrained()` call.

Fixes #40582


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@ArthurZucker @Rocketknight1 

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: deepseek_v3, dots1, glm4_moe, glm4v_moe

Great to see progress on this! Previously a common concern is how to support PEFT/LoRA, such as in https://github.com/huggingface/transformers/pull/40016 . I've written something that may help: https://github.com/woct0rdho/transformers-qwen3-moe-fused

@woct0rdho Thanks to share great work! I will check this out.
@ArthurZucker @Rocketknight1 please review this and let me know what I should do more to integrate this kind of works.

Will have a look thanks for the PR, glad to see you here again! 🤗 

Thank you for the review and the clear direction! I appreciate you pointing me to #40132. The plan to isolate the expert class first and then use on-the-fly weight conversion makes a lot of sense.
to: @ArthurZucker 

Awesome, first pr is close to being merged!

Waiting for support！

Waiting for support as well, would really appreciate MoE LoRA capability!

Will all moe models benefit from this? Even trainer?

Thanks for many attention to this feature! I will start working on it since #40132 have successfully merged.

Since https://github.com/huggingface/transformers/pull/42697 is merged, I will close this PR.