Title: TransMLA: Migrating GQA Models to MLA with Full DeepSeek Compatibility and Speedup
Venue: NeurIPS 2025
Accepted: yes
Keywords: Deepseek

MLA

RoPE

PCA

LLaMA

Qwen

GQA

MHA
Abstract: Modern large-language models often face communication bottlenecks on current hardware rather than computational limitations. 
*Multi-head latent attention (MLA)* addresses this by compressing the key-value cache using low-rank matrices, while the Absorb operation prevents the KV cache from reverting to its original size, significantly boosting both training and inference speed.
Despite the success of DeepSeek V2/V3/R1, most model providers have heavily invested in optimizing GQA-based models and, therefore, lack strong incentives to retrain MLA-based models from scratch.
This paper demonstrates that MLA provides superior expressive power compared to GQA with the same KV cache overhead, thereby offering a rationale for transitioning from GQA to MLA.
In addition, we introduce TransMLA, a framework that seamlessly converts any GQA-based pre-trained model (e.g., LLaMA, Qwen, Gemma, Mistral/Mixtral) into an MLA-based model. 
For the first time, our method enables *direct conversion of these models into a format compatible with DeepSeek's codebase*, allowing them to fully leverage the existing, highly-optimized support for the DeepSeek architecture within inference engines like vLLM and SGlang.
By compressing 93\% of the KV cache in LLaMA-2-7B, we achieve a **10x speedup** with an 8K context length while maintaining meaningful output. 
Moreover, the model requires only **6B tokens** for fine-tuning to recover comparable performance across multiple benchmarks.
TransMLA provides a practical path for migrating GQA-based models to the MLA structure, and when combined with DeepSeek’s advanced optimizations—such as FP8 quantization and Multi-Token Prediction—further inference acceleration can be achieved.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents TransMLA, a framework for converting GQA-based language models (e.g., LLaMA, Qwen) into the MLA (Multi-head Latent Attention) format used by DeepSeek models. TransMLA enables off-the-shelf migration of pre-trained GQA models into the MLA structure, thereby inheriting the benefits of DeepSeek’s inference-time acceleration (e.g., vLLM, SGlang, FP8). The authors propose several innovations to support this conversion: (1) RoRoPE, a PCA-based method to concentrate RoPE positional information into a small number of dimensions, (2) FreqFold, which exploits frequency similarity in RoPE to further reduce redundancy, and (3) Balanced Key-Value PCA, which equalizes key and value norms to enable better joint compression. Extensive experiments on smolLM and LLaMA-2 demonstrate that TransMLA achieves substantial compression (up to 93%) and speedup (up to 10.6×) while preserving model quality with minimal fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nStrong practical relevance: This work addresses a highly relevant engineering challenge to migrate existing GQA models to DeepSeek’s efficient MLA format without retraining from scratch.\n\nWell-motivated and well-engineered: The techniques such as RoRoPE and Balanced KV are practical, intuitive, and empirically effective.\n\nThorough empirical evaluation: The paper includes both performance recovery metrics and inference latency benchmarks across multiple hardware setups, providing a well-rounded view of the benefits.\n\nSubstantial performance gain: At similar compression rates, TransMLA clearly outperforms MHA2MLA both in accuracy and speed, especially under high compression ratios.\n\nCompatibility with ecosystem: Seamless integration with DeepSeek’s toolchain is a strong point, making this work immediately usable for real-world deployment.\n\n**Weaknesses**\n\nBalanced Key-Value PCA is heuristic: While intuitive and empirically validated, the balancing method lack theoretical guarantees or principled formulation (e.g., via optimization objectives).\n\nCompression calibration requires data: The PCA-based compression still relies on a calibration dataset for collecting activations, which may not be trivial in privacy-sensitive or black-box model settings.\n\nClarity in exposition: Some parts, such as RoPE slicing and rotation math (Section 4.2), are quite dense and could benefit from visual aids or step-by-step walkthroughs.\n\nQUESTIONS:\nCan the authors release the converted MLA models (e.g., LLaMA2 7B)? This would greatly benefit the community and validate the plug-and-play nature of TransMLA.\n\nHow sensitive is Balanced KV to the choice of scaling constant? Could this be learned or optimized per-layer?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose TransMLA, a framework that converts GQA/MHA modules from existing LLM models (such as Llama) into MLA modules that fit DeepSeek's frameworks. To adopt the RoPE from GQA/MHA to the decoupled RoPE in DeepSeek's MLA, the authors propose RoRoPE and FreqFold to concentrate the RoPE information to the first key/query head. Besides, a balanced key-value method is proposed for model compression.  The proposed method is shown to be a better conversion than existing works (such as MHA2MLA) and could greatly save the memory cost of the KV cache.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The proposed method achieves much better performance than MHA2MLA, especially when there is no additional training. This means the proposed TransMLA is a better approximation to existing MHA/GQA and can be used as a starting point for future research \n\nWeakness:\n- Although the proposed TransMLA achieves good performance, it requires a calibration dataset for PCA and the effect of the calibration dataset is not studied. For example, what is the effect of the size of the calibration dataset? Also, what if we choose another dataset as the calibration dataset?\n- The description of the proposed method is not very clear, and there are some typos (please refer to Questions)\n- There is a lack of ablation studies. For example, what would be the commonsense reasoning performance if we don't enable all the components (e.g., RoRoPE without FreqFold)\n- The authors missed some other related works in the literature (please refer to Questions)\n- The authors tested TransMLA on SmolLM-1.7B and Llama2-7B, but there is no experiment on Qwen/Mistral (which is claimed in the abstract)\n- The proposed method is not compared to other KV cache compression methods such as KVquant and Palu \n- The proposed method is only tested on common-sense reasoning tasks. It would be good to test on some long-context tasks (such as LongBench) since MLA's advantage is mainly on long context lengths\n\nQUESTIONS:\n1. Overall I think Figure 1 is relatively clear to me but they are not well-explained in Section 4. \n - For example, it is not known how $q_{rope}$, $q_{nope}$, $k_{rope}$, $k_{nope}$ are defined in TransMLA.\n - It is not known how the RoPE is removed for $q_{nope}$ and  $k_{nope}$ after applying the PCA rotation matrix. \n - Also, it is not know how the matrices in MLA are exactly initialized (e.g., $W_{dq}$, $W_{qr}$, $W_{uq}$, $W_{dkv}$, $W_{uk}$, $W_{uv}$)\n\n2. The sum sign seems to be missing in Equation (29)\n\n3. It is a little bit not clear to me how this claim \"MLA consistently exhibits greater expressive capacity than GQA given the same KV cache size\" helps in this paper. When setting the same KV cache size, MLA seems clearly stronger than GQA as there are more parameters. It would be great to know if there are any study/theoriem comparing the expressive capacity of GQA and MLA given similar amount of parameters. \n\n4. There seems to be another related work named X-EcoMLA[1]. How is the proposed TransMLA compared to this work?\n\n5. I am wondering how the authors benefit from the DeepSeek optimizations for the inference speed measurement. Is the FlashMLA adopted in this experiment? It would be beneficial to add more details for the inference optimization.\n\n[1] Li, Guihong, et al. \"X-ecomla: Upcycling pre-trained attention into mla for efficient and extreme kv compression.\" arXiv preprint arXiv:2503.11132 (2025).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe TransMLA framework aims to seamlessly convert existing GQA (Group-Query Attention) pre-trained models into MLA (Multi-head Latent Attention) models. TransMLA uses RoRoPE (Rotary Position Embedding Rotation) and FreqFold: to concentrate the rotational position encoding (RoPE) information into a small number of dimensions (especially the first attention head), thereby removing RoPE from the remaining dimensions to support DeepSeek's unique Absorb operation. It is observed that the norm of the key after removing RoPE is significantly larger than the value. The authors balance the norms of the two by scaling to improve the compression performance of the joint low-rank decomposition. TransMLA achieves significant KV cache compression and inference acceleration.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nIt provides an immediate and feasible path for existing models to utilize DeepSeek's optimization ecosystem, which has strong practical value.\n\nImpressive results were achieved in KV cache compression and inference acceleration. For LLaMA-2-7B, it achieved 93% KV cache compression and 10.6x inference acceleration. \n\nWeakness:\nThe paper does not make significant improvements to MLA, and the innovations proposed are not the source of major performance improvements (such as kvache compression, speed improvement, parameter pruning, etc.) For example, the paper admits that its \"Balance KV technology itself is relatively trivial\", but includes it as an \"innovation\" point as the main contribution.\n\nThe method is overly dependent on a specific ecosystem. All the \"innovations\" in the paper - RoRoPE, FreqFold and even support for Absorb operations - are to achieve compatibility with DeepSeek's unique architecture, but do not discuss the performance differences, advantages and disadvantages of MHA and MLA, and whether there are some improvement solutions. This high degree of coupling has led to doubts about the universality of its methodology, making it impossible to become a general attention mechanism conversion framework with broad scientific value.\n\nQUESTIONS:\nIs MLA really better than GQA? The paper only proves the phenomenon of equal KV Cache, but does it have an advantage under equal parameter values ​​or headdim? Or is GQA good enough for us? There are many ways to solve KVCache, not necessarily MLA. \n\nIf MLA is indeed theoretically superior, and DeepSeek actively open-sources its optimization solution, then why does the GQA model still dominate and require such a complex conversion?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a comprehensive method for converting models based on MHA, as well as simple variants such as GQA, into MLA-compatible architectures, aiming to achieve enhanced expressiveness and overall efficiency. The proposed approach first performs Key fusion, followed by tailored handling of RoPE-related Query and Key components to better adapt them to the specialized RoPE mechanisms in MLA, namely RoRoPE and FreqFold. Finally, a joint PCA-based dimensionality reduction is applied to the Key and Value matrices, guided by the distributional characteristics of their weights. The result is a model that can be effectively adapted to MLA. The authors validate their approach by conducting conversion experiments on models of two different scales, followed by fine-tuning on a small amount of tokens, demonstrating the effectiveness of their method in restoring model capabilities. Furthermore, ablation studies are provided to underscore the necessity of the proposed RoPE handling and weight-based dimensionality reduction strategies.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The authors design an effective training pipeline that enables the transformation of non-native MLA models into their MLA-compatible counterparts. The empirical results validate the method's effectiveness. In particular, the RoPE adaptation and KV compression techniques offer valuable insights for researchers working on MLA conversion and optimization.\n\n2. The manuscript presents results across different compression scales. Notably, at lower compression ratios, TransMLA is able to retain strong performance even without additional training.\n\n3. The authors employ well-designed visualizations and empirical studies to convincingly demonstrate the necessity and effectiveness of their algorithmic components.\n\n**Weaknesses**\n\n1. A major limitation of the manuscript lies in the choice of baseline models. Both SmolLM-1.7B and LLaMA-2-7B are based on MHA, which, according to prior work and empirical observations, tends to be particularly amenable to KV compression. In contrast, more recent and mainstream models often adopt GQA or similar attention variants, which are known to present greater challenges for compression, especially in the KV cache. As such, the experimental results presented may have limited relevance to current model architectures.\n\n2. The capability restoration experiment for LLaMA-2-7B uses the same training data as that used for SmolLM-1.7B. However, LLaMA-2-7B was trained in the early 2023 using a different dataset. The data used for post-conversion fine-tuning may thus be of higher quality than the original pretraining corpus, implying that the observed gains could result not from genuine restoration, but from additional learning. This undermines the interpretability of the experimental results as a strict recovery study.\n\nQUESTIONS:\nCould the authors provide updated results using more recent models, and align the recovery training data with the original pretraining data to enhance the credibility and relevance of the experimental section?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}