TyphoonMLA: A Mixed Naive-Absorb MLA Kernel For Shared Prefix
Authors: Ahmet Caner Yüzügüler
Ahmet Çelik
Jiawei Zhuang
Lukas Cavigelli
Venue: ICLR 2026 Poster (2026)
Keywords: Large language models
inference
multi-head latent attention
shared prefix
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: A novel MLA kernel that integrates naive and absorb implementations to fully exploit data reuse available in attention calculations as a result of the shared prefix

Abstract: Multi-Head Latent Attention (MLA) is a recent attention mechanism adopted in state-of-the-art LLMs such as DeepSeek-v3 and Kimi K2. Thanks to its novel formulation, MLA allows two functionally equivalent but computationally distinct kernel implementations: naive and absorb. While the naive kernels (e.g., FlashAttention) are typically preferred in training and prefill for their computational efficiency, existing decoding kernels (e.g., FlashMLA) rely on the absorb method to minimize HBM bandwidth usage. However, the compute-bound nature of the absorb implementations prohibits performance benefits from data reuse opportunities in attention calculations, such as shared prefixes. In this work, we introduce TyphoonMLA, a hybrid approach that combines naive and absorb formulations to harness the strengths of both. TyphoonMLA effectively leverages the shared prefix by applying the naive formulation to the compute-bound parts of attention calculations, while reducing the bandwidth requirements for non-shared parts by using the absorb formulation. As a result, TyphoonMLA improves the throughput of attention calculations in MLA architectures by up to 3× and 3.24× on NPU and GPUs, and boosts end-to-end throughput by up to 1.48× in tokens per second, with only a 3\% overhead in HBM size.

Published as a conference paper at ICLR 2026
TYPHOONMLA: A MIXED NAIVE-ABSORE MLA KERNEL FOR SHARED PREFIX
Ahmet Caner Yüzügiüerr, Ahmet Çelik, Jiawei Zhuang & Lukas Cavigelli
Huawei
Switzerland
ahmet. celik, zhuangjiawei, lukas. luwas.cavigelle.............. wwweellieeeaweecoom com
ABSTRACT
Multi-Head Latent Attention (MLA) is recent attention mechanism adopted in state-of-the -f--oof-ttt-rn--- art LLMs such as DeepSeek-v3 and Kimi K2. Thanks to its novel for- mulation, MLA allows two functionally equivalent but computationally distinct kernel implementations: naive and absorb. While the naive kernels (c.g.. FlashAt- tention) are typically preferred in training and prefill for their computational ef- ficiency, existing decoding kernels (e.g., FlashMLA) rely on the absorb method to minimize HBM bandwidth usage. However, the compute-boune nature of the absorb implementations prohibits performance benefits from data reuse opportu- nities in attention calculations, such as shared prefixes. this work, we introduce TyphoonMLA, hybrid approach that combines naive and absorb formulations to harness the strengths of both. TyphoonMLA effectively leverages the shared prefix by applying the naive formulation the compute-bound parts of attention calculations, while reducing the bandwidth requirements for non-shared parts by using the absorb formulation. As a result, TyphoonMLA improves the throughput of attention calculations in MLA architectures by to 3x and 3,24x on NPU and GPUs, and boosts end-to-end throughput by up 1.48x in tokens per second, with only a 3% overhead in HBM size.
INTRODUCTION
Large Language Models (LLMs) have been widely adopted in various application domains, ranging from chat assistants (OpenAI, 2023) to coding agents (Chen et al., 2021), thanks to their unprece- dented language processing and reasoning capabilities. However, their substantial computational requirements result in slow and inefficient inference, which undermines the user experience and increases operational costs. Therefore, addressing these computational challenges is essential for enabling broader adoption and ensuring their sustainable deployment.
To improve the efficiency of LLM inference, a new attention architecture called Multi-Head Latent Attention (MLA) has recently been introduced (DeepSeek-Al al., 2024a). In this architecture, the key and value tensors that hold contextual information from previous tokens, i.c., the K KV-cache, are stored in a low-rank latent space. which helps eliminate memory bandwidth bottlenecks in attention layers. MLA's flexibility to merge the latent-space projection layers with matrix multiplication operations in attention calculations (i.e., absorption trick) makes possible to implement MLA in two functionally equivalent but computationally distinct ways, namely, naive and absorb.
In training and prefill, where performance typically limited by the compute capacity (e.g., the total throughput of Tensor cores in GPUs), the compute-efficient naive implementation is preferred. In decode, where the HBM bandwidth dictates the performance, the memory-efficient absorb imple- mentation is used. As result, MLA architectures utilize computational resources more effectively both in training and inference, making them a more efficient alternative to other attention archi- tectures, such Multi-Head Attention (MHA) (Vaswani et al., 2017) 7) Grouped-Query Attention (GQA) (Ainslie et al., 2023). Consequently, MLA serves as the backbone of various state-of-the-an LLMs, such as DeepSeek-vä. (DeepSeck-AL et al., 2024b) and Kimi K2 (Bai et al., 2025).
Another promising direction to improve the efficiency of attention kernels is prefix sharing (Juravsky al., 2024). In various application scenarios, portion of the K KV-cache is shared across multiple

---

Published as a conference paper at ICLR 2026
queries, creating data reuse opportunity that can help alleviate memory bottlenecks. For instance, many of the inference services today employ a system prompt. which is shared among all user queries to enhance the safety and quality of generated responses. Both official documentation and recent leaks through prompt-injection attacks have revealed that system prompts used in popular inference services reach lengths of tens of thousands of tokens (e.g.. Claude-4 has system prompt of 26k tokens (Johnson, 2025)). Furthermore, parallel reasoning techniques, such as Tree-of-thought (ToT) (Yao et al., 2023) and Graph-of-thought (GoT) (Besta et al., 2024), as well as the speculative decoding techniques (Wang et al., 2025a), often result in multiple queries attending to the same part of the KV-cache, boosting the potential for data reuse in attention computation.
To effectively benefit from this potential data reuse in attention calculations, recent studies have proposed several MHA and GQA kernel implementations (Yao et al., 2025; Pan et al., 2025; Wang et al., 2025b). In these designs, the shared parts of the KV-cache are read only once from HBM and reused across multiple queries, reducing HBM accesses and mitigating memory bandwidth bot- tlenecks. Although these techniques improve the efficiency and performance of MHA and GQA kernels, they are not directly applicable to MLA, since the performance a typical MLA kernel implementation is limited by computation rather than memory bandwidth. As result, current MLA kernels fail to fully exploit the data reuse opportunities available in attention computation.
In this paper, propose novel MLA method that effectively leverages data reuse in the shared parts of the KV-cache to improve the efficiency and performance of attention computation. Our key observation that leads to the proposed solution lies in the fact that, while the absorb implementa- tion is preferred in memory-bound regions, the naive implementation becomes more efficient in the compute-bound regions when the KV-cache is shared across multiple queries, floating-point operations than the absorb implementation. Building on this insight, our proposed MLA method merges the naive and absorb implementations: it uses the naive formulation in the shared parts of the KV-ccche to exploit its computational efficiency and the absorb formulation in the non-shared parts of the K V-cachee to benefit from its memory bandwidth efficiency.
it requires fewer
We demonstrate through extensive experiments on GPUs and NPUs that TyphoonMLA substantially improves resource utilization and offers a speedup of up to 3.2x. Moreover, the proposed method compatible with other optimization techniques, such as PagedAttention (Kwon et al., 2023) and Radix (Zheng et al., 2024), and it supports existing parallelization strategies, such as tensor and sequence parallelism. Therefore. TyphoonMLA can be effortlessly integrated into popular infer- ence frameworks, such as vLLM and SGLang. Furthermore, the proposed method is mathematically equivalent to standard MIA implementations; thus, it does not cause any accuracy degradation and requires neither training nor fine-tuning.
In short, this paper makes the following contributions:
We introduce TyphoonMLA, a novel MLA inference method that achieves higher com- putational efficiency and resource utilization attention calculations. To the best of our knowledge, this is the first work that combines both naive and absorb implementations in MLA calculations.
We provide a detailed analysis that shows that TyphoonMLA requires fewer floating point operations in compute-bound regions and consumes less HBM bandwidth in the memory- bound regions than existing MLA kernels.
We conduct a series of experiments on NPUs and GPUs, demonstrating that TyphoonMI.A improves attention throughput by up to 3.2x for DeepSeek-v3 and Kimi K2, with only 3% overhead in memory footprint.
Our code is open-sourced and publicly available', Generative AI tools were used to edit and refine this paper to improve the clarity and quality of writing.
2

---

Published as a conference paper at ICLR 2026
Nalve
TyphoonMILA
KV-cadre
Fissnttantion
Nesttowe Stand
in Idtart spece
Syatnes Proeege
Profi
prggectoon
Caalee
Castee
User Provegt
User Prorpp
Ueer Proomet
Cache
Cache
User Prowept
FIARS Plorrs
Wics
Cache
Profil
Cashre
Cacte
Absorb
RIMES Piores
Mice
Fiet
PRU
Wa
Cacche
Eriitt
Cuotfe
RMMS
IoPE
F045 Namm
Cachee
Figure 1: The naive formulation of MLA. b) The absorb formulation of MLA. c) The prefill and decode stages TyphoonMLA.
BACKGROUND
In this section, we first provide background on MIA and describe its two implementation variants: naive and absorb. We then discuss prefix sharing in LLM inference and its potential for improving the efficiency of self-attention kernels in MLA layers.
2.1 MULTI-HEAD LATENT ATTENTION
MLA is an attention mechanism introduced by DeepSeek-v2 and adopted by state-of-the-ar LLMs such DeepSeek-v3 (DeepSeek-AI al., 2024b), Kimi K2 (Bai et al., 2025), and Long- Cat (Meituan et al., 2025). Unlike standard attention mechanisms, MLA employs low-rank key- value joint compression mechanism reduce KV-cache size and bandwidth requirements during ference. Moreover, its novel positional encoding enables the key-value projections to be rearranged in way that leads two computationally distinct implementations, namely naive and absorb.
The naive implementation of MLA keeps the KV-cache in an uncompressed form. Fig. I(a) illustrates the components of MLA in naive implementation. Therein, after passing through the rotational embedding and RMS normalization layers, the key and value tensors are decompressed using the up- up-projection matrix, denoted as W WKVb in the figure. As result, the and caches consist of multiple attention heads, and the self-attention calculation becomes equivalent to the standard MHA formulation.
Naive:
Absorb: The absorb implementation of MLA utilizes the commutative property of matrix multi- plication to reposition the up-scaling matrix. In absorb formulation, as shown in Fig. 1(b), the up- projection matrix is split into two submatrices (denoted WKVD1 and W WxY22) and placed before and after the query (WQs) and output (Wo) projection layers. This formulation allows the KV-eache to remain in compressed form, represented as PE and noPE caches in the figure. However, apply- ing the up- up-projection matrix to the queries further expands their embedding dimensions, requiring more floating-pons operations in the self-attention calculations. Moreover, the resulting formulation is incompatible with existing self-attention kernels (e.g., FlashAttention (Dao et al., 2022)), neces- sitating the development of specialized kernels for MLA layers (c.g., FlashMLA (Jiashi Li, 2025), ThunderMLA (Spector et al., 2025)).
2.2 PREFIX SHARING
In various inference scenarios, large portions of the KV-cache are shared across multiple queries, creating opportunities for data reuse. First, modern inference services often employ system prompts

---

Published as a conference paper at ICLR 2026
to enhance the safety and quality of generated responses. These system prompts include tool age instructions, API call descriptions, and even recent news updates after the model's knowledge cutoff dates, and can consist of tens of thousands of tokens (Johnson, 2025). Second, recent LLM reasoning techniques, such as Chain-of- Thouught with self-consistency (Wang et al., 2023), Tree- of-Thought (Yao et al., 2023), and Traph-offTThougee (Besta et al., 2024), reformulate inference as a tree or graph search, in which multiple branches run in parallel while sharing a common prefix. Finally, certain speculative decoding techniques (Wang et al., 2025a) require validating multiple candidate tokens in parallel, which share long sequence of past tokens. In all these cases, multiple queries attend overlapping regions of the KV-cache, offering a significant data reuse opportunity to improve the efficiency and performance of attention computations.
PROPOSED METHOD
In the previous section, discussed two distinct ways of implementing MLA. In this section, provide details about TyphoonMLA, which combines both implementations.
3.1 TYPHOONMLA
As previously discussed, the shared prefix introduces data reuse in the shared parts of the attention calculations. Building on this insight, TyphoonMIAA partitions the attention calculation into low- and high-arithmetic-nntenn
components and applies absorb and naive formulations, respectively. Fig. 1(c) illustrates how TyphoonMLA works.
Prefill: In the prefill stage, a tree-like query structure formed from the incoming user requests and shared prefix (e.g., system prompt). The queries processed by the LLM using prefix- aware naive kernel, producing PE and noPE caches in the low-rank latent space. The portion of the PE and noPE caches that corresponds to the shared prefix is then expanded via the up-projection to form the and caches. Consequently, the shared and non-shared parts of the KV-cache are stored in uncompressed and compressed formats, respectively, enabling the of both naive and absorb formulations. Notably, the up-projection incurs no additional computational overhead, as it already performed by standard naive kernels during the prefill stage.
Decode: In the decode stage, TyphoonMLA first applies the down-projection layers WQ and WKV to the queries, obtaining the q and lv vectors. The q vectors pass through an RMS normaliza- tion layer, followed by the up- pp-projection layer W WQt- The ky vectors are split into two subvectors. one of which passes through a rotational embedding layer, and the other through an RMS normalization. All operations up to this point are common to TyphoonMLA, naive, and absorb implementations.
Unlike naive and absorb implementations, TyphoonMLA performs self-attention calculations using both compressed and uncompressed KV-cache while reusing the shared operations. As described in Algorithm 1, TyphoonMLA takes the output of Wos up-projection layer (Q), shared KV-ccche in uncompressed form (Cx and Cv), non-shared KV-cache in latent form (CN and CR). and KV up up-projection matrices (W KVb1 and W WKV12) as inputs. The input queries Q are first split from their
Algorithm TyphoonMI.A Decode Attention Kernel
Require: Queries after Wos proj. and RoPE layers, Q E R BBxHxD
Require: Shared KV-eache in uncompressed form, Ck BxL, Cv RBBBx Require: Non-shared KV-cache latent space, Cx E I2 Bx CR E R BxbxD Require: KV up-projection matrices, WKVM E R xXDxDD, WKV E DDDxD
I: QN, Qx) +- Q
Split Q from dimension Des into (D., D,
2: QR (QPE(QR)
> Apply positional encoding & Merge after RoPE
3: QK + QN,, Qrl
4: Ox. <- softmax Qx Cx
D Compute naive component
5: QA <- Qu
6: O softmax Q&CT. CRC
Compute absorb component
O + O& Kv12
8: O (O (N,4a)
> Combine partial outputs using LSEs
9: return O
4

---

Published as a conference paper at ICLR 2026
Table 1: Computational analysis of naive, absorb, and TyphoonML.A. B: batch size, Syi query sequence length, Lg: shared context length, Lu: non-shared context length, H: number of heads, Dyk: head dim. for Q and K. D,: head dim for V. Dr: KV LoRA rank, D, noPE head dim, D,: RoPE head dim. TyphoonMLA always requires smaller memory operations than naive (highlighted in red) and fewer
operations (MACs) than absorb (highlighted in blue )
MAC
DEEPSEEK- V3 (x1024)
NAIVE
BS,L, + D.) BS,,,, D.) HS,L, D.) (2, D,)
40 x BL, 40 BL,
ABSORB
136 BL, 136 BL,
TYPHOONMLA
BS,L, (5,,,. D.)
40 X BL. +336 BL..
HBM R/W
DEEPSEEK v3 (x1024)
NAIVE
Le + Dz) + BL..((Dg)) D.)) LD(Di + D,) + (D; + DD)
40 L. + 40 B
ABSORB
0.56 x L, + 0.56 x BLn
TYPHOONMIA
L.HH(Dot + D.) + BI.. (D: + Dr)
10 x I, + 0.56 x BL..
embedding dimension into two tensors, and a positional encoding is applied. Then, similar to the naive implementation, the queries multiplied by the uncompressed K cache, Ck. passed through a softmax function, and multiplied by the cache, Cv. Similar to the absorb implementation, the queries are first up-projected using the W WRVZI matrix. multiplied by the compressed PE and noPE caches (CR and Cn), passed through a softmax function, and multiplied again by the noPE cache. As required by the absorb formulation, the output of the absorb component is then up-projected once again using the WXV22 matrix. Finally, the partial results from these two components are then aggregated using a combine function with the help of the log-sum-exp (LSE) of the softmax denominators (l, and La). TyphoonMLA is mathematically equivalent to both absorb and naive implementations; therefore, it does not require any re-training or fine-tuning.
Fall-back to Absorb:
At small batch sizes, when there is insufficient data reuse, the naive imple- mentation can become less efficient than absorb due to higher memory access costs. To address this, TyphoonMLA automatically switches to an absorb-only kernel whenever the batch size falls below a predefined threshold. By doing so, TyphoonMI.A avoids any performance penalty small batch sizes, ensuring consistently high efficiency across wide range of batch sizes.
Parallelization:
TyphoonMLA is fully compatible with existing parallelization strategies for atten- tion computation. Although the compressed PE and noPE caches have single head dimension, the uncompressed K and caches can be parallelized across attention heads. Additionally, both compressed and uncompressed caches can be easily parallelized across the sequence dimension. As a result, TyphoonMLA seamlessly supports both tensor and sequence parallelism, enabling efficient scaling across multiple NPUs GPUs.
3.2 COMPUTATIONAL ANALYSIS
To quantify the potential performance benefits of the proposed method, we derive the computational requirements of attention calculations terms of the number of multiply-accmmulat operations (MAC) and the size of memory read/write operations (HBM R/W) for TyphoonMLA, as well as the naive and absorb formulations. Table summarizes these derivations with respect to architectural and generation parameters. In the right-most column, we substitute the architectural parameters with those of the DeepSeek-v3 to facilitate direct comparison. In our analysis, we consider only self- attention computations, excluding the projection layers, since self-attention dominates execution time at long sequence lengths and asymptotically approaches the total runtime.
Since the absorb implementation is typically compute-bound, its performance is primarily deter- mined by the number of MACs. In contrast, the naive implementation is generally memory-bound so its efficiency is dominated by the amount of data read from HBM. Therefore, to maximize overall performance, TyphoonMLA aims to achieve lower MACs than absorb in compute-bous bound regions and fewer HBM read and writes than naive in heoud regions.
5

---

Published as a conference paper at ICLR 2026
Table 1 shows that the naive implementation requires reading (40 x L +0 BL,) 1024 words
from HBM. In comparison, TyphoonMLA reads only (40 x L, + 0.56 x BL,) x 1024 words, which is the same for the shared portion but approximately 70x smaller for the non-shared portion.
The absorb implementation, typically compute-bound, requires (136 x BLg + 136 x BL,) X 1024 MAC operations. TyphoonML.A, in contrast, requires only (40 BT, 136 BI,,) x 1024 MAC operations, which is the same for the non-shared portion, but 3,4x smaller in the shared portion. This analysis reveals that TyphoonMLA requires fewer bytes read from HBM than the naive formulation in samorry-bunne regions and fewer MACs than the absorb formulation in compute- bound regions, reducing the overall computational complexity of attention calculations in MLA.
To combine the partial results of the naive and absorb parts, TyphoonMLA uses Combine func- tion, similar to the epilogue stage in Flash Attention (Dao al., 2022). This function performs only vector operations and requires reading BBS, HD, bytes from HBM and performing 2BS, H.SHHH.. D MAC operations. Since the computational complexity of this function is independent of the KV sequence length, which typically ranges from hundreds to thousands, its computational overhead is insignificant compared to the other components of the attention calculations.
For TyphoonMLA to achieve speedup, reading the shared portion of the KV-cache should be faster with the naive implementation than computation with the absorb implementation. This happens only when there is sufficient data reuse, which occurs when the batch size is larger than a threshold. By equating the memory read time of the naive and the computation time of the absorb implementation for the shared part of the K KV-ccche, we identify the batch size threshold, Bo, in terms of architectural parameters (defined above) and hardware specifications, namely throughput T and HBM memory bandwidth M, as follows:
LgH(Dam + Do) L + D.) M
(Dom + D.) T Sy(2D, + D+) M
Ba
(1)
T
When the architectural parameters replaced with those of DeepSeek-v' v3 and the hardware pa- rameters with Ascend NPU (T = 376 TOPS/s, M = 1.8 TB/s), we obtain Bg = 61. This threshold indicates the break-even point, beyond which TyphoonMLA becomes computationally more effi- cient than the absorb implementation, whereas, for batch sizes smaller than Bo TyphoonMLA falls back the absorb kernel to avoid any slowdown.
EXPERIMENTS
Building on the complexity analysis presented in Section 3.2, we now provide empirical evidence to validate the performance benefits of TyphoonMLA. In this section, evaluate the performance of TyphoonMLA and compare it against various baselines on NPUs and GPUs. First, we measure TyphoonMLA's throughput on popular datasets using DeepSeek-v3 and Kimi K2 with different system prompts. Next, we examine its performance breakdown to validate our insights. Finally, we assess TyphoonMLA's memory footprint and compare it against the absorb baseline.
We implemented TyphoonMLA for Ascend NPUs using the CANN toolkit (Huawei, 2024a). For the absorb component of TyphoonMLA, we developed cus- tom kernel that supports paged and variable-length KV-cache using the Ascend CATLASS library (Huawei, 2025). For the naive component of TyphoonMLA, we employed the IonFuusedInerAtteere www.ertttttiittooot..ooo ionScore function from TorchNPU (Huawei, 2024b). For the pro- jection layers and combine logic, we used TorchNPU's e einsum and vector operators.
Experiments on NPUs:
To evaluate the impact of system prompt length on performance, we selected three system prompts with varying lengths, summarized in Table 2. As benchmark datasets, we used the MMLU (Hendrycks al., 2021), GSM8K (Cobbe et al.. 2021), and SimpleQA (Wei et al., 2024), which comprise questions and answers from various topics. We repeat each experiment for batch sizes of 64, 128, 256, 512, and 1024.
To simulate a realistic deployment scenario, we adopted continuous batching with a paged K KV-cache with block size of 128. Each experiment starts by randomly sampling questions from dataset and forming a batch of queries. At the end of each decode iteration, the completed queries are replaced with new questions sampled from the dataset. Each experiment is continued until the entire dataset

---

Published as a conference paper at ICLR 2026
PageddittentiorIA
CAToASS-Aborb
TyphoonMILA
D3epSeek-v3
DeepSeek-v3
D SimpleQA
SimpleQA
Throuhhuus
Prompt
0.1
0.0
Throughput
0.75
Prompt 8
0.50
025
Throughout
0.0
256
1024
64
128
256
512
1024
128
256
1024
128
256
512
1024
128
256
512
1024
128
256
512
1024
Batch size
Figure 2: Benchmark results on Ascend NPUs. Y-axes represent normalized throughput in terms of the number of generated tokens per second per layer. Some data points for baselines are missing as their memory footprint exceeds the HBM capacity.
is processed. The throughput is then calculated by dividing the total number of generated tokens by the total execution time across all decode iterations. We calculate the speedup as the throughput of TyphoonMLA divided by the best of the two baselines.
We run the experiments on an Ascend NPU, equipped with 24 Davinci cores and 64 GB of HBM memory, which provides a throughput 376 TOPS/s in FP16 pre- cision and an HBM bandwidth of 1.8TB/s. We compared TyphoonMLA's throughput with that the baseline ker- nels, namely TorchNPU PagedAttentiinMMAA kernel and a custom-built CATLASS Absorb-only kernel.
Table 2: System prompts used in the ex- periments, taken from (Johnson, 2025).
Name
LLM service
#tokens
Prompt A
Claude-4
26472
Prompt
OpenAl/o3
7069
Fig. presents the normalized throughput of Typhoon- MLA and the baseline methods for the attention layers of DeepSeek-v3 and Kimi K2. TyphoonMLLA consis- tently outperforms both baselines across all datasets, sys-
Prompt C
Grok/Personas
4759
tem prompts, and batch sizes, achieving speedups between 1.2x and 3x. As expected, TyphoonMLA offers the highest speedup with Prompt A, since longer system prompts increase the ratio of shared non-shared portions of the KV-cache. We further observe that the speedup is generally higher for the Kimi K2 than for DeepSeek-v3. This is because the number of attention heads in Kimi K2 is 64, half of DeepSeek-v3, which makes the former's performance more sensitive to data reuse. Overall, these results confirm that TyphoonMI.A effectively exploits the shared prefix and delivers consistent and significant speedups over the existing MIAA kernels.
We also implemented TyphoonMLA using the FlashInfer naive and absorb MLA kernels (Ye et al., 2025) and repeated our NPU experiments on a GPU. Fig. reports the normalized throughput of TyphoonMLA on GPU with PetaFLOPSIA of theoretical throughput in FP16, 3.3 TB/s HBM bandwidth, comparing it against the FlashMLA (Jiashi Li, 2025) and FlashInfer (bbsorb-onl) baselines batch sizes of 64, 128, 256, 512, and 1024. TyPhoonMLA achieves higher throughputs than both baselines, with factors up to 3.24x. Similar to the NPU re- sults, TyphoonMIA accelerates Kimi K2 more than DeepSeek-v3, owing its smaller number of
Experiments on GPUs:

---

Published as a conference paper at ICLR 2026
FlashMLA
Flashinfer
TyphoonMLA
DeepSeek-v3
Seelov3
DeepSeek-v3 SimpleQA
400
Throughout
300
Prompt
200
1000
Throughpus
Prompt
1500
Prompt
Throughpus
1000
64
128
256
512
128
256
512
1024
256
512
1024
64
128
256
512
1024
64
128
236
512
1024
64
128
256
1024
Batch size
Figure 3: Benchmark results on GPU for various batch sizes. Y-axes represent throughput in terms of the number of generated tokens per second per layer.
Table 3:
Token generation rate for DeepSeek-và v3 processing MMLU dataset with a batch size 128
per GPU.
FlashMLA
TyphoonMLA
Attention time (ms)
Total time (ms)
TGR (KToken/s)
Attention time (ms)
Total time (ms)
TGR (KToken/s)
Prompt
99.1
127.2
58.1
86.3
1.48
Prompt B
34.5
62.6
2.04
25.9
54.0
2.37
Prompt C
26.9
55.0
2.33
22.0
50.1
2.56
attention heads. Moreover, it achieves the highest speedups with Prompt A, as has the highest token count. Overall, these results demonstrate that TyphoonMLA generalizes effectively to GPUs, delivering substantial performance gains across diverse hardware platforms.
Latency Breakdown: To demonstrate how TyphoonMLA accelerates attention computations and how its components behave under different batch sizes, we profiled its execution using the CANN toolkit's msprof tool and compared it against the CATLASS absorb-only baseline. Fig. shows the execution time breakdown of TyphoonMLA (bars on the left-hand side) and the CATLASS baseline (bars the right-hand side). In the figure, we denote the naive and absorb parts of attention computation in TyphoonMLA as Stage 1, Artn and Stage 2 Artn, respectively; the up and down- scaling required in the absorb formulation as W WKV.B1-Proj and Wxv.B2-Proj and the combination logic that merges the intermediate results of both stages as Sombinel.SE. We excluded the other linear layers of the attention block, since they are identical across both methods and negligible at long sequence lengths. In this experiment, sequence length of each request to 512, and used the architectural parameters Kimi K2. Batch sizes smaller than 128 omitted, in these cases, TyphoonMLA falls back to the absorb-only implementation, becoming identical to the baseline. As the CATLASS baseline is absorb-only. contains only Stage 2 components.
set the shared prefix length to 4096, the non-shared
The runtime breakdown of TyphoonMLA shows that its performance gains align closely with the estimates from theoretical analysis in Section 3.2. At batch size of 1024, the attention caleu-
8

---

Published as a conference paper at ICLR 2026
lations in the CATLASS baseline take 6.43 ms, while the naive and absorb parts of TyphoonMLA take 1.63 ms and 1.06 ms, respectively. Since the non-shared part of attention is identical between TYPhoonMLA and the baseline, we estimate the execution time of the shared part in the baseline as 6.43 1.06 = 5.37 ms. The ratio between the execution times of the shared parts in the baseline and TYP is therefore
3.3, which matches the reduction in the number of operations between the naive and absorb formulations derived in Section 3.2. These results not only validate our initial insights and theoretical analysis but also demonstrate that TyphoonMLA's hybrid design translates directly into measurable runtime improvements.
End-to-End Speedup: Previous experimental results demonstrated that TyphoonMLA significantly accelerates the attention layers MLA architectures. However, the execution time of the other layers in LLMs (e.g.. MoF,
Latency breakdown
Stage 1,
Stage 2, War.a--ro)
collective communication etc.) would remain unchanged.
Executton time (mo)
Stage 2, Attn
Therefore, now investigate the impact of the proposed method on the end-to-end LLM decoding performance. To that end, estimate the token generation rate (TGR) for DeepSeek-v3 with a batch size of 128 per device on a system with 128 GPUs by measuring attention layers a GPU and using the profiling data for other layers pro- vided by DeepSeck-Al (Deepseek-A) 2025).
Way.az pro) CombineeLS
44
128
256
512
1024
Batch Size
Table 3 reports the estimated TGR and per-iteration exe- cution time for both the attention layers and the full model when using TyphoonMLA, comparing against those with FlashMLA as the baseline. Since attention computation constitutes a substantial portion of overall runtime, the the attention lay-
Figure Latency breakdown of Ty- phoonMLA (bars on the left-hand side) and CATLASS absorb-only baseline (bars the right-hand side) for Kimi K2 architecture. Stage and Stage 2 represent the naive and absorb parts of TyphoonMLA, and have sequence lengths of 4096 and 512, respectively.
speedup achieved by TyphoonMI.A ers translates directly into notable end-to-end throughput gains, reaching up to a 1.48x improvement in tokens per second.
HBM Footprint: Since TyphoonMLA stores the shared portion of the KV-cache in uncompressed form, its memory footprint differs from that of the absorb base- lines. To quantify the impact of TyphoonMIAA on the memory footprint, we analyze TYPhoonMLA''s HBM usage under various deployment settings and compare it against the absorb baseline. We as- sume that the model is distributed across 384 NPUs, in CloudMatrix cluster (Zuo al., 2025), employing full expert parallelism on MoE layers and combination of data, tensor, and sequence parallelism of factors 24, 4, and 4, respectively. Fig. 5 shows the HBM size of DeepSeek-v3 for batch sizes ranging from 4K to 32K and maximum sequence lengths from 32K to 256K, assuming Prompt A as the shared prefix.
At small batch sizes and sequence lengths, HBM usage is dominated by the model weights, which are identical for both TyphoonMLA and the absorb baseline. As batch size and sequence length increase, the KV-cache grows for both methods. However, at large batch sizes and sequence lengths, the memory occupied by the KV-cache that corresponds to the shared prefix becomes negligible compared to the non-shared portion, which is identical for both methods. Consequently, Typhoon- MLA incurs only a minimal HBM overhead, limited to approximately 3% across a wide range of deployment scenarios.
5 RELATED WORK
LLM Serving Acceleration: Various prior work proposes system-level optimizations for LLM serving. Orca (Yu et al., 2022) introduced continuous batching, which allows replacing the com- pleted requests in batch with new ones to improve effective throughput. Inspired by the OS virtual memory concept, PttenAAtentton (Kwon et al., 2023) partitions the KV-cache into memory pages to efficiently handle KV-cache with variable length. TyphoonMLA supports both continuous batching and Paged/Attention; thus, it benefits from the performance gains of such system-level optimiza- tions. Furthermore, to distribute L.M inference across multiple devices, existing systems typically

---

Published as a conference paper at ICLR 2026
Model Size
KV Cache Size
Seq. Length:32K
Seq, Lengthie4K
Seq. Length:128K
Seq. Langth.256K
Batchvas Batche-k
TyphoonMLA
Absorb-
7,80GB
B.55CB
10. 10.04G8
13,03CB
TyohoonMLA-
8.19G8
9.69GB
12.69GB
18 68CB
7.945B
9.44GB
12.4458
18.43GB
TyohoonMLA-
8.48GB
11.48GB
17.476B
29. 29.46GB
Absorb
11.23G8
17.22G8
29.21GB
TychoonMLAA
9.06GB
15. 00GB
27,05GGB
51. 00.00GB
Absorb-
8.81GB
14.81GB
26.800
26.80GB
50.8008
20
40
60
20
40
60
80
20
40
60
80
20
40
HBM size (GB)
HBM size (5B)
HBM size (GB)
HBM size (GB)
Figure 5: HBM footprint comparison for DeepSeek-v3 v3 in FP8 precision for both weights and KV- cache.
employ tensor (Shoeybi al., 2019) and sequence (Jacobs al., 2023) parallelism. TyphoonMLA also seamlessly supports both types of parallelization, enabling deployment at scale.
Efficient Attention Kernels: As attention computation constitutes significant portion of the total execution time of LLM inference, many prior studies focused on developing efficient attention ker- nels. Flash Attention (Dao al., 2022) partitions attention computations into tiles that fit on the on- chip buffers to minimize HBM bandwidth usage. FlashDecoding (Dao, 2024) introduces the Split-K method improve the parallelization across GPU cores. Flashinfer (Ye al., 2025) provides customizable attention template to facilitate the development of custom attention kernels. However, all these techniques are developed for MHA; hence, they are not applicable MLA. FlashMLA (Ji- ashi Li, 2025), FlashMLA-ETAI (Dege et al., 2025), and ThunderMLA (Spector et al., 2025) pro- pose efficient kernel designs for MILA. Unfortunately, these kernels are agnostic to the shared prefix and absorb-only implementation; therefore, they not fully exploit hardware resources.
Prefix Sharing: Several prior works (c.g., SGLang, Hydragen etc.) have proposed techniques to exploit shared prefixes in attention calculations and KV-cache management (Juravsky et al., 2024; Zheng et al., 2024). Other frameworks (e.g..
FastTree etc.) developed special- ized MHA/GQA GPU kernels for tree-structured KV-caches leverage the data reuse introduced by shared prefixes (Ye et al., 2024a; Zhu et al., 2024; Ye et al., 2024b; Yao et al., 2025; Pan et al., 2025; Wang al., 2025b). However, these kernel designs target MHA/GQA architectures and aim reduce HBM bandwidth usage. Consequently, they are not applicable to MILA, whose decode stage typically compute-bound. In contrast, TyphoonMLA addresses this gap by reducing the number of FLOPS required in the compute-bound decode stage of MLA, achieving substantial speedups. Moreover, the mechanisms handle complex rree-structurd KV-caches introduced by the prior work would be equally applicable to both naive and absorb implementations. Therefore, applying these techniques to TyphoonMLA would trivial.
CONCLUSIONS
In this paper, we proposed a novel MLA method, TyphoonMLA, which combines the naive and ab- sorb implementations to effectively exploit shared prefixes in the KV-cache. Our theoretical analysis showed that, in the presence of a shared prefix, TyphoonMI requires fewer FLOPS in compute- bound parts and smaller HBM bandwidth in memory-bound parts of the attention calculations than the existing MLA kernels. Experimental results on NPUs and GPUs show that TyphoonMLA im- proves attention throughput by up to 3.2x DeepSeek-v3 and Kimi K2 models, while incurring only a minimal increase in HBM footprint.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 4895-4901. Association for Computational Linguistics, 2023. doi: 10. 10.18653/V1/2023..1 htt.://122222.......OOO EMN.L- 298. URL https: //wii.//oooo/ooo/ http:////db..ooo1111/33333333 0t/t:///////.111/1//22220 emm.p-mmain 298.
Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao, Tong Gao, Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Hao Hu, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Chao Hong. Yangyang Hu, Zhenxing Hu, Weixiao Huang, Zhiqi Huang. Zihao Huang. Tao Jiang, Zhejun Jiang, Xinyi Jin, Yongsheng Kang. Guokun Lai, Cheng Li, Fang Li, Haoyang Li, Ming Li, Wentao Li, Yanhao Li, Yiwei Li, Zhaowei Li, Zheming Li, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyuan Liu, Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Lijun Lu, Shengling Ma, Xinyu Ma, Yingwei Ma, Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng, Ruoyu Qin, Bowen Qu, Zeyu Shang, Lidong Shi, Shengyuan Shi, Feifan Song, Jianlin Su. Zhengyuan Su, Xinjie Sun, Flood Sung, Heyi Tang. Jiawen Tao, Qifeng Teng, Chensi Wang. Dinglu Wang. Feng Wang, and Haiming Wang. Kimi K2: open agentic intelligence. CoRR, abs/2507 20534, 2025. doi: 10. 10.488550ARXX1.22577.00 205344 URL https: doi. .rrg/ 10. 4.0.48550/40 arrvv 2507 20534.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Giani- nazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyezyk, and Torsten Hoe- fler. Graph of thoughts: Solving elaborate problems with large language models. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), Thirty-Eighth AAAI Conference on Ar- rificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances Artificial Intelli- gence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 17682-17690, AAAI Press, 2024. doi: (0.1609/AAAL 29720. URL attps:///ol: htt.ddiooog////// org/ 0. 609/ aaa v38i1 1 29720.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder. Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar- ian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plap- pert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Pe- ter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CORR, abs/2107.03374, 2021. URL hwt.ttttp///......m http//axivvo/ org/ abs/ 03374.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman, Training verifiers to solve math word problems. CoRR, abs/2110.14168,, 2021. URL
https://aaae org/ als/ /als///11...m 1111..1168....111111111111111111 14168.
Tri Dao. FlashAttention--2: Faster attention with better parallelism and work partitioning. In Inter- national Conference on Learning Representations (ICLR), 2024.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural
11

---

Published as a conference paper at ICLR 2026
Information
Processing Systems 35: Annual Conference Neural Information Process-
ing Systems 2022. NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http:/ /ttt::/ppaper..../ wwspppessiipppp....ee.........se weaps.cciii ctppspeeppppppppppppeepppp wwwes/ wwt.s//ppppeee/e2//2... hash./
Deepseek-AI. Profiling Data in DeepSeek Infra, 2025. URL. htttps://gittieht com/ deepseek-ai/proffe leeaaaaa..
DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi
Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, Hao Zhang, Hanwei Xu, Hao Yang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang. Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang. Meng Li, Míaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen. R. L. Jin, Ruiqi Ge. Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, Tao Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang. Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang. Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Zihan Wang, and et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. CoRR, abs/2405. 0os/2405504434, 2024a. doi: (048850/ARXIV.24005 04434.
URL https: / //w.iii.o/oo
2405.
DeepSeck-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangho Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Ji- aqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song. Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu. Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. Chen, R. L. Jin, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun. Tian Pei, Tianyu Sun, W. L. Xiao, and Wangding Zeng. Deepseek- v3 technical report. CoRR, abs/2412.19437, 2024b. doi: 1044500/RXXI.2422.19937 URL https:///do wtw.oioorg/..... 1..00.885555 48550/ a 2412.19437.
Pengcuo Dege, Qiuming Luo, Rui Mao, and Chang Kong. Flashmla-etap: Efficient transpose at- tention pipeline for accelerating MLA inference on NVIDIA H20 gpus. CoRR, abs/2506.01969, 2025. doi: 10448550/ARXI122506001000 URL https:://d/ /ttttp::/ddd.oog/11.....58 wtt............/........ 25C6. 01969.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- /otet//ommmdddd77511111
view.net, 2021. URL https: /oppnrevieeeeee / net
software re caa///an////aa///aaaa community. Huawei. CANN Community Edition, 2024a. URL
com/en//
Huawei. Ascend Extension for PyTorch, 2024b. URL httpttt://ggteee.co//cccmem pytorch.
Huawei. Ascend CATLASS, 2025. //tptttppp://giieee.oom
12

---

Published as a conference paper at ICLR 2026
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. CoRR, (atp2309.145099 2023. doi: 10. 48550/ARXIV 14509. URL https::///// /doi.di... 220
Shengyu Liu Jiashi Li. Flashmla: Efficient mla decoding kernels. https: com/
deepseek-ai/ ai/aassaaa/aaaa.a.. 2025.
Á. Johnson. System Prompts Leaks, 2025. URL pesstem_prompt__111
Jordan Juravsky, Bradley C. A. Brown, Ryan Ehrlich, Daniel Y. Fu, Christopher Ré, and Aza- lia Mirhoseini. Hydragen: High-throughput LLM inference with shared prefixes. CoRR, abs/2402.05099, 2024. doi: 1044850/ARXI..242200509 URL https://don org/10. 48550/arXiv 2402. 05099.
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Jason Flinn, Margo Seltzer, Peter Druschel, Antoine Kaufmann, and Jonathan Mace (eds.), Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pp. 61 1-626. ACM, 2023. doi: 10.1145/ 3600006.3613165 URL https: //o::///oo/// org 10 00l..1111360000..0 3613165.
Meituan, Bei Li, Bingye Lei, Bo Wang, Bolin Rong, Chao Wang, Chao Zhang, Chen Gao, Chen Zhang, Cheng Sun, et al. Longcat-flash technical report. arXiv preprint arXiv:2509.01322, 2025.
OpenAl. GPT-4 technical report. CoRR, wbtpaaa..////////////. 20233 doi: 1048550ARXXX223000077744 08774.
URL http::ttt/////ctt.m /doi orooorgg//// /00 2tw.00045500///.xi....
Zaifeng Pan, Yitong Ding, Yue Guan, Zheng Wang. Zhongkai Yu, Xulong Tang, Yida Wang, and Yufei Ding. Fasttree: Optimizing attention kernel and runtime for tree-structured Ilm inference. In Eighth Conference on Machine Learning and Systems, 2025.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model par- 2019. URL http: /ttp//awvv.r..aaaaeeeeeo /1999 08053.
allelism. CoRR,
Benjamin Spector, Aaryan Singhal, Dan Fu, and Chris Ré. ThunderMLA: Flashmla, faster and fuseder!, 2025. URL https://hayyssearch stanford. edu/blog/ 2025-03- - --ttddeeuudd.....
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio. Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. cc
5998 5998-6008, 2017. URL https: / wwwp://wcweeeiioo.oeeeooooeeeeee cc/ fbd053c1 c4..3ccc44444440 Abs.aeaaaaaaaaaaaatatttt....... http://wwppppppppppppppea1aa pappr/2017 //ahhhhh/h/////
Jikai Wang, Yi Su, Juntao Li, Qingrong Xia, Zi Ye, Xinyu Duan, Zhefeng Wang. and Min Zhang. Opt-tree: Speculative decoding with adaptive draft tree structure. Trans. Assoc. Compul. Lin- guistics, 13:188-199 2025a. doi: 10.1162/TACL 00733.. URL https: /oii. org/10. 1162/tacc_a_00 a_aaccl___0000 00735.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning language models. In The Eleventh International Conference on Learning Representationss ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: /ttpr//iieeppeeeeeeee.eemm. net forum? 11111P1-IIMME
Zhibin Wang, Rui Ning, Chao Fang. Zhonghui Zhang, Xi Lin, Shaobo Ma, Mo Zhou, Xue Li, Zhongfeng Wang. Chengying Huan, Rong Gu, Kun Yang, Guihai Chen. Sheng Zhong, and Chen Tian. Flashforge: Ultra-efficient prefix-aware attention for LLM decoding. CoRR,
13

---

Published as a conference paper at ICLR 2026
abs/2505.17694, 2025b. doi: 10.48500ARRXIV2550.1178 URL https::/dbl.crg//// 48550/arXiv.2500
Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. CoRR, 010/2411004368 2024. doi: 10.48550/ARXIV.2411 1..0000.. URL https: waoooooor..... 10. 4.445550/axiiii 2411.04368.
Samuel Williams, Andrew Waterman, and David A. Patterson. Roofline: an insightful visual performance model for multicore architectures. Commun. ACM, 52(4):65-76, 2009. doi: 1498785.
N0.1145/1498765.44998 1498785. URL https::: /ttps::/ddo... /tt////...r//11111////111//
Jinwei Yao, Kaiqi Chen, Kexun Zhang, Jiaxuan You, Binhang Yuan, Zeke Wang, and Tao Lin. Deft: Decoding with flash tree-attention for efficient tree-structured LLM inference. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.nel, 2025. URL https:tt... hwtp///ppeeeeeeeeeeeeeeeeeeooeea net http://wwww..ettttteeeeeeeeeeeee forum?id-ucc ptpptiiu---7pppuu9.. F0Qqu...
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: wwwpasapppspppaapeaaee .aaressniisss nips. wttppaperfllleeeaaeeeeeeeeeeeeee wwwps//papeeeeeeeeeaaeeaaa 20///has//hhh////// 27 d11
F4dd7aae aataaaaaeeaeeeaaa0000003333 oat.eeeeeeeeee---taaaa-o--ooooee
html.
Lu Ye, Ze Tao, Yong Huang, and Yang Li. Chunkattention: Efficient self-attention with prefix- aware KV cache and two-phase partition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024. pp. 11608-11620 Association for Computational Linguistics, 2024a. doi: 10. 10.18863//1/200 1/2024 122022AALLLLLN...22 URL https:///dot www.o..org//00 1851116653/// /0w. http:/2222222111111 lngg. 623.
Zihao Ye, Ruihang Lai, Bo-Ru Lu, Chien- Yu Lin, Size Zheng, Lequn Chen, Tianqi Chen, and Luis Ceze. Cascade inference: Memory bandwidth efficient shared prefix batch decoding, 2024b.
Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanic Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, al. Flashinfer: Efficient and customizable attention engine for IIm inference serving. arXiv preprint arXiv:2501.01000 2025.
Gycong-In In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer- beasddd generative models. In Marcos K. Aguil- and Hakim Weatherspoon (eds.), 16th USENIX Symposium
Operating Systems De- sign and Implementation, OSDI 2022, Carlsbad, CA, USA, July 11-13, 2022, pp. 521-538. USENIX Association, 2022. URL https:t///// wwww uswnix.en..... wwaemnixxrrrree//eee httg://onfereene/ssi222. presentation/pe yu.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 /hash/
15, 2024, 2024. URL htt.::/tp/// /paperssppp/ppp..speepee/ nips. cc/paper_filleeeecee Abstract www.eet--ooooeeeen.ehhe.ete
72 222
Lei Zhu, Xinjiang Wang, Wayne Zhang, and Rynson W. H. Lau. Relayattention for efficient large language model serving with long system prompts. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 4945 4957. Association for Computational Linguistics, 2024, doi: 10. 10.18653/VV/2024. 2ON-200 URL https: ptttpa://ol //doi.d//////o .org 0 16553vvlll /2024 agl lon. 270.
14

---

Published as a conference paper at ICLR 2026
Pengfei Zuo, Huimin Lin, Junbo Deng, Nan Zou, Xingkun Yang, Yingyu Diao, Weifeng Gao, Ke Xu, Zhangyu Chen, Shirui Lu, Zhao Qiu, Peiyang Li, Xianyu Chang, Zhengzhong Yu, Fangzheng Miao. Jia Zheng, Ying Li, Yuan Feng, Bei Wang, Zaijian Zong, Mosong Zhou, Wenli Zhou, Houjiang Chen, Xingyu Liao, Yipeng Li, Wenxiao Zhang, Ping Zhu, Yinggang Wang, Chuanjje Xiao, Depeng Liang. Dong Cao, Juncheng Liu, Yongqiang Yang. Xiaolong Bai, Yi Li, Huaguo Xie, Huatao Wu, Zhibin Yu, Lv Chen, Hu Liu, Yujun Ding, Haipei Zhu, Jing Xia, Yi Xiong, Zhou Yu, and Heng Liao. Serving large language models huawei cloudmatrix384, CoRR, abs/2506. 12708 2025. doi: (0.48500/RXIV.250066 12708. URL https: / /oi...i... org/10. 48550/a aariiv 2506. 12708.
15

---

Published as a conference paper at ICLR 2026
DeepSeekv3
Kimi-K
Throughput (QTakens/s)
10
104
Roofline Absorb Roofline
10a
1/4
1/2
1
2
8
16
32
64
128
256
512
1/4
1/2
1
2
8
16
32
64
128
256 512
8atch size
Batch size
Figure 6: Roofline analysis of the naive and absorb implementations for DeepSeek-v3 and Kimi K2 models with a memory bandwidth of 1.8 TB/s and cube throughput 400 TFLOPS/s.
A
APPENDIX
A.1 ROOFLINE ANALYSIS
To analyze the computational characteristics of the naive and absorb implementations in the pres- ence of shared prefix, we plot the roofline model (Williams al., 2009) of an Al accelerator for DeepSeek-v3 and Kimi K2 under both implementations, as shown in Fig.6. The y-axis in the plots represents throughput, calculated the number query tokens processed per second by the MIAA kernel, while the x-axis represents the batch size, which determines the number of operations per byte (i.e., the operational intensity). The roofline model has two regions of operation: a bandwidth- limited region, where performance is limited by memory bandwidth due to insufficient operational intensity, and a compute- compute-bound region, where performance is limited by the computational capacity of processor, such as the total throughput of the cube units in NPUs and GPUs.
As expected, the absorb implementation provides higher throughput at low operational intensities thanks to the reduced memory bandwidth usage resulting from the compressed K VV-ccche stored in latent space. As batch size increases and operational intensity rises, the throughput the absorb implementation does not improve further. For DeepSeek-v3, performance remains flat once the compute-bound region is reached, while for Kimi K2, throughput quickly saturates beyond a batch size of two. Thus, increasing operational intensity does not improve the performance of absorb implementations, as they are typically compute-bound.
In contrast, the naive implementation greatly benefits from the increasing operational intensity and achieves higher throughputs, it is typically memory-bound at small batch sizes due to the un- compressed K KV-ccche. At batch sizes larger than 64, where significant data reuse occurs, the naive implementation achieves up to 3.4x higher throughput than the absorb implementation. thanks to its lower number of required operations. This analysis indicates that, although the absorb imple- mentation is advantageous at low operational intensities, the naive implementation offers superior performance when sufficient operational intensity is present. These observations motivate our pro- posed kernel, which combines both approaches to maximize efficiency and performance across both compute and memory-bound regions of MIA calculations.
A.2 THEORETICAL ANALYSIS OF TYPHOONMIA
In this section, visualize the computational characteristics of the naive and absorb formulations using the computational model presented in Table 1. Fig. 7 depicts the estimated execution time of the absorb implementation, naive implementation, and TyphoonMIA for varying batch sizes.
In the shared context part of the attention calculations, the execution time of the absorb formulation increases linearly with the batch size, since its execution compute- bound. In contrast, the execu- tion time of the naive formulation remains constant until the batch size of 128, since its execution is memory-bound and the memory handwidth usage does not change with the batch size in the shared
16

---

Published as a conference paper at ICLR 2026
Shared Context
Nonshared Context
Full Context
(C, = 4096, Cp 0)
(C2 = O, C. = 128)
(Cg 4096, C, 128)
....
Naive
.....
Nalve
Nalve
0.6
Absorb
1.25
Absorb
Absorb
TyphoonMLA
TyphoonMLA
TyphoonMLA
Execution Time
1.00
1.0
0.50
0.5
0.25
2
4
8
16
32
64
128
256
16 32
64
128
256
1
2
4
8
16
32
64
128
256
Batch Size
Batch Size
Batch Size
Figure 7: Theoretical analysis of Naive, Absorb, and TyphoonMLA.
prefix part of the KV-cache, Therefore, although faster at small batch sizes, the absorb becomes slower than the naive at batch sizes larger than 64. Therefore, TyphoonMLA switches from absorb to naive implementation at the batch size of 64 in order to achieve minimum execution time over the full range of batch sizes.
In the non-shared context part of the attention calculations, the absorb is always faster than naive. Therefore, TyphoonMLA employs the absorb implementation at all batch sizes. As a result of the combination of the stages, TyphoonMLA is identical to the absorb implementation at batch sizes lower or equal to 64, beyond which the computational benefits the naive implementation weighs in and makes TyphoonMLA significantly faster than the absorb baseline.
A.3 SENSITIVITY BATCH SIZE
To analyze how different components TyphoonnMLA benefit from data reuse, we profile Typhoon- MLA and the baseline methods an Ascend NPU across range of batch sizes during the execution of DeepSeek-v3. Fig.8 shows the execution time of the shared part, non-shared part, and the overall attention calculation. Consistent with the complexity analysis in Section 3.2, the absorb baseline outperforms the naive baseline at small batch sizes in the shared part of the attention calculations due the reduced memory bandwidth requirements of the compressed KV-cache in latent space, as shown in F(a. As the batch size increases, the execution time of the absorb baselines grows nearly linearly, while the naive baseline remains about the same, thanks to improved operational intensity. Around batch size of 64, the naive baseline becomes faster than the absorb. To maintain
Shared Prefix
shared
Full
(lerchNPU)
(lorchNPU)
(TorchNPU)
Absort (Catisss)
1.75
Absorb (Catlass)
Absort (Catlass)
Absort (TerchNPU)
Absort: (TencWNPU)
Absort (TbrcWNPU)
Kernel Time
TyphoonMLA
TyphoonMLA
TyphoonMLA
1.25
1.00
0.50
15 52 64
128
256
0.00
125 212
16 54 128 252 Batch Sbee
Batch Size
Batch Sire
(a) Shared part MLA
(b) Non-shared part MLA
(c) Full execution
Figure Performance breakdown of TyphoonMLAA and the baselines on Ascend NPUs for DeepSeek-v3, assuming shared prefix of length 4096 and query length of 128. Execution time of individual components is measured using the CANN toolkit's msprof tool.
17

---

Published as a conference paper at ICLR 2026
optimal performance across all batch sizes, TyphoonMLA switches from the absorb-only only implemen- tation to the mixed naive-absorb kernel once the batch size exceeds this cut-off point.
Fig.8(b) shows the execution time of TyphoonMLA and the baseline methods for the non-shared part of the attention calculations. Since this part does not contain any shared prefix, increasing the batch size does not improve the operational intensity; hence, the absorb implementations remain faster than the naive across all batch sizes. Consequently, TyphoonMLA consistently employs the absorb implementation for the non-shared part. When these twO portions combined, shown in Fig.8(c), TyphoonMLA behaves identically to the absorb baseline up to the cut-off batch size of 64, beyond which TyphoonMLA outperforms the absorb baseline, achieving speedup of up to 2x at the batch size of 512. These results confirm that TyphoonMLA's performance gains primarily arise from exploiting the shared portion of the K V-cache.
18


Reviews:
[{"review_id": "S0lOnIoBlm", "reviewer": "Reviewer_jvoP", "summary": "This paper presents a method that accelerates inference for transformers in the presence of shared prefixes, such as system prompts in modern LLMs. The method, called TyphonMLA, is a hybrid kernel that combines regular attention and MLA from DeepSeek. It consists of using regular attention for shared prefixes and MLA for the rest. Moreover, the kernel falls back to standard MLA at small batch sizes. In practice, the hybridization is carried by merging both attention results via a shared softmax denominator. On NPUs and GPUs, with real long system prompts (Claude 4, GPT o3, Grok), TyphoonMLA reaches up to 3x throughput over standard FlashMLA baselines, with just 3% HBM overhead.", "questions": "Can you describe in more details how TyphonMLA kernel is implemented in a pseudocde?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "There are many things that I liked in this paper and that I believe it will be helpful to others at ICLR:\n\n-  MLA decode today is compute-bound because of absorb, so prefix-sharing kernels from MHA/GQA don't directly help. TyphonMLA fills this gap.\n\n- The idea is very simple but impactful. \n\n- The paper brings concrete computational analysis. For example, Table 1 shows MAC ops and HBM reads for naive, absorb, and TyphoonMLA; and Figure 4 shows a detailed latency breakdown of TyphonMLA across different stages of the kernel. \n\n- Since the idea is quite straightforward and the method is exact (same output as original MLA), TyphonMLA works well with standard inference techniques, such as paged KV. Therefore, it can be easily included into frameworks like vLLM and SGLang.\n\n- Strong results on both NPUs and GPUs, with two real MLA models (DeepSeek-v3, Kimi K2) using multiple real system prompts.", "weaknesses": "A few things to be improved in the paper include: \n\n- Right now Table 1 is overwhelming to read as it contains a lot of symbols. One idea is to organize the table by placing MAC rows first and HBM reads rows later so that methods are more easily comparable. Distributing $L_s$ and $L_n$ for all methods would also improve things. E.g., $BS_q (L_s + L_n) H (2D_l + D_r)$ to $BS_q L_s H(2D_l + D_r) + B S_q L_n H(2D_l + D_r)$.\n\n- The impressive results of 3x speedup happens in shared prefix scenarios, which is not always the case in transformers.\n\n- I believe the bar plots are clean and informative, but they would be much better with error bars / confidence intervals to get an idea of how robust are the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CS0xfodn6T", "reviewer": "Reviewer_tmtF", "summary": "This paper addresses a performance bottleneck in modern Large Language Models (LLMs) that use Multi-Head Latent Attention (MLA), such as DeepSeek-v3. The standard absorb implementation used for decoding is memory-efficient but compute-bound, preventing it from benefiting from data reuse in scenarios with shared prefixes (e.g., system prompts).   \n\nThe authors propose TyphoonMLA, a novel hybrid attention kernel that partitions the computation. It applies the computationally cheaper naive formulation to the shared prefix portion of the Key-Value Cache to reduce floating-point operations (FLOPs), while using the memory-bandwidth-efficient absorb formulation for the non-shared parts. This hybrid approach is a bit-exact, drop-in replacement that requires no model retraining.", "questions": "You mention that the kernel falls back to an absorb-only implementation below a \"predefined threshold\". How is this threshold determined? Is it found empirically for each hardware/model pair? It would be helpful to understand how sensitive the overall performance is to this value. An ablation study on this threshold would add a lot of value.\n\nYou claim compatibility with tree-structured workloads like Tree-of-Thought. However, the method described seems tailored to a simple linear prefix. How would TyphoonMLA handle a more complex KV cache layout, where there might be a root, several shared branches, and then unique leaves?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "While prefix sharing optimization isn't new, the paper is the first to identify and address the specific bottleneck it creates for Multi-Head Latent Attention (MLA) models. \n\nThe computational analysis in Section 3.2 (Table 1) is clear and compelling. It precisely breaks down the HBM read and MAC requirements, mathematically demonstrating why the hybrid approach should be superior by reducing MACs in the compute-bound shared region and HBM reads in the memory-bound non-shared region.\n\nBesides, this is not a theoretical optimization for a toy problem. It directly accelerates models that are at the forefront of LLM development (DeepSeek, Kimi). As system prompts and context lengths grow, this bottleneck becomes increasingly important.", "weaknesses": "My main issue with the paper is that the core idea feels less novel than presented. Splitting attention into shared and non-shared parts is a known trick in the MHA/GQA world—we've seen this before in libraries like FlashInfer with its \"Cascade Inference\" and in Hydragen. The paper doesn't acknowledge this prior art, which makes its contribution seem more like a clever adaptation to MLA's specific compute/memory tradeoffs rather than a brand-new algorithm.\n\nBeyond that, there are a few other points that need tightening up:\n1. The paper is a bit hand-wavy about the \"predefined threshold\" used to switch back to the absorb kernel. It's not clear how this is set or how sensitive the performance is to this magic number.   \n\n2. It claims to handle complex tree-structured prefixes (like in Tree-of-Thought), but doesn't show how. This is a non-trivial problem that other kernels, like FlashForge, were specifically designed to solve.   \n\n3. The NPU results are impressive, but they feel a bit like a black box. Because the implementation relies on proprietary Huawei libraries (CANN/CATLASS), it's hard to judge how portable the approach is. The paper doesn't discuss the underlying principles that would make this work on other NPU architectures", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VI9Dn4rUMz", "reviewer": "Reviewer_UutA", "summary": "This paper introduces TyphoonMLA, a novel hybrid attention kernel designed to accelerate inference with shared prefixes for Large Language Models (LLMs) that utilize the Multi-Head Latent Attention (MLA) architecture, such as DeepSeek-v3 and Kimi K2. The authors identify a key performance bottleneck in existing MLA implementations (e.g., FlashMLA), which exclusively use a compute-bound \"absorb\" formulation to minimize HBM bandwidth. While effective for single sequences, this approach does not take advantage of data reuse opportunities in common shared-prefix scenarios (e.g., system prompts, tree-based decoding).\n\nThe core contribution is a hybrid method that partitions the attention calculation. It applies the compute-efficient \"naive\" formulation to the shared-prefix portion of the KV-cache, where high data reuse mitigates its higher bandwidth cost. Concurrently, it uses the bandwidth-efficient \"absorb\" formulation for the non-shared portions. By combining the outputs of these two parallel paths, TyphoonMLA achieves significant throughput gains—up to 3.24x on GPUs and 3x on NPUs—with a negligible HBM footprint overhead of approximately 3%.", "questions": "1. Table 1 and Section 3.2 do not clearly define what `k` means, for example in $L_s40k + BL_n40k$. Please clarify the meaning of `k`.\n2. Figure 6 analyzes the Roofline for different batch sizes, but Figures 2 and 3 only show results for bsz=256, 512, and 1024. Could you add results for batch sizes 1, 8, 32, and 128? If small batch sizes are not relevant in practice, please provide supporting evidence.\n3. Figure 3 shows only bsz=1024; can you also show results for multiple batch sizes on the GPU, similar to Figure 2 for the NPU?\n4. Can the paper provide end-to-end acceleration comparisons for TTFT and TGR between FlashMLA, FlashInfer, and TyphoonMLA?\n5. Could you elaborate on the system's behavior in a dynamic serving environment with heterogeneous requests? Specifically:  \n   a) How does the system handle load balancing between the naive and absorb paths when non-shared sequence lengths are highly variable within a batch?  \n   b) What are the specific memory management challenges (e.g., fragmentation) when integrating the hybrid KV-cache with continuous batching?\n6. The performance gains are shown for a contiguous shared prefix. Have you considered generalizing this hybrid approach to more complex, non-contiguous sharing patterns (e.g., shared data blocks in a RAG context), or the potential applicability of the core \"hybrid compute/memory-bound\" principle to other architectures like standard MHA/GQA as hardware evolves?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of creating a hybrid kernel that combines the naive and absorb formulations within a single attention computation is simple and insightful. Prior work has explored prefix sharing for standard MHA/GQA, but those mechanisms are typically memory-bound. This paper clearly identifies that MLA's decoding stage is compute-bound, making the problem fundamentally different and requiring a new approach. The solution is an effective combination of existing concepts tailored to the characteristics of the MLA architecture.\n2. The Roofline analysis (Figure 6) clearly motivates the problem and visualizes the performance trade-off between the two formulations. The computational analysis in Table 1 provides a solid mathematical foundation for the expected gains. The experiments are thorough, validating the approach on two different hardware platforms (NVIDIA GPUs, Huawei Ascend NPUs), two state-of-the-art models, and realistic workloads using real-world system prompts of varying lengths. The inclusion of a latency breakdown (Figure 4) is a major strength, as it serves as a microbenchmark that directly validates the core hypothesis—that the speedup mainly comes from accelerating the shared part of the computation.\n3. The paper is clearly written and easy to follow. Figure 1 provides a good visual schematic that contrasts the naive, absorb, and the proposed TyphoonMLA data flows, making the core mechanism immediately understandable. The results are presented clearly and support the paper's claims.\n4. As LLMs are increasingly deployed with long and complex system prompts, and with the rise of reasoning techniques like Tree-of-Thought, optimizing for shared prefixes is becoming more important. The MLA architecture is used by several open-source models, and this work provides a solution that offers substantial, real-world performance improvements. The fact that it requires no model retraining and can be integrated into existing serving frameworks like vLLM increases its practical relevance.", "weaknesses": "1. The paper addresses the MLA kernel performance in shared prefix scenarios, but in Table 2, the system prompts are sampled from Claude-4, OpenAI/o3, and Grok/Personas, which do not use MLA. However, the models tested are DeepSeek-V3 and Kimi K2, which do use MLA. The experimental design could be improved by using system prompts that are more representative of DeepSeek-V3 and Kimi K2.\n2. The paper does not explore how the length of the shared prefix affects acceleration. It is unclear at what length positive gains in TTFT and TGR metrics are achieved.\n3. Section 3.1 describes the \"Fall-back to Absorb\" mechanism, but does not clearly provide the predefined threshold of gbs >= 64 in the main text or experiments; instead, it is discussed in the appendix. It is suggested to clarify this in the main paper in future versions.\n4. The overhead of the \"CombineLSE\" step, which merges the results from the two parallel paths, is reported as small but is not analyzed. The paper would be improved by briefly discussing this step's computational complexity and how it scales with batch size or the number of attention heads, as this is important for understanding limitations in extreme-scale scenarios.\n5. The authors claim the method is numerically identical to standard MLA, which is theoretically sound. However, the paper does not provide empirical evidence, such as a validation table confirming identical outputs (e.g., bit-for-bit) or benchmark results compared to the baseline. Such results would confirm the implementation's correctness.\n6. The paper claims compatibility with systems like vLLM but does not sufficiently discuss the engineering complexity of deploying this method in dynamic, real-world serving environments. Key challenges are not addressed, such as managing a hybrid KV-cache layout within a memory manager like PagedAttention (which could increase fragmentation), or handling load imbalances due to highly variable non-shared sequence lengths within a batch.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)