Title: Multi-head Temporal Latent Attention
Venue: NeurIPS 2025
Accepted: yes
Keywords: Attention

Key-Value cache compression

speech translation

speech recognition

spoken language understand

text summarisation
Abstract: While Transformer self-attention offers strong parallelism, the  Key-Value (KV) cache grows linearly with sequence length and becomes a bottleneck for inference efficiency. Multi-head latent attention was recently developed to compress the KV cache into a low-rank latent space. This paper proposes Multi-head Temporal Latent Attention (MTLA), which further reduces the KV cache size along the temporal dimension, greatly lowering the memory footprint of self-attention inference. MTLA employs a hyper-network to dynamically merge temporally adjacent KV cache vectors. To address the mismatch between the compressed KV cache and processed sequence lengths, a stride-aware causal mask is proposed to ensure efficient parallel training and consistency with inference behaviour. Experiments across tasks, including speech translation, speech recognition, speech understanding and text summarisation, demonstrate that MTLA achieves competitive performance compared to standard Multi-Head Attention (MHA), while greatly improving inference speed and GPU memory usage. For example, on a English-German speech translation task, MTLA achieves a 5.3$\times$ speedup and a reduction in GPU memory usage by a factor of 8.3 compared to MHA, while maintaining translation quality.


Peer reviews:
comments: SUMMARY:
This paper proposes MTLA, a novel method that compresses the KV cache along the temporal dimension. This method is orthogonal to other compression methods along the hidden dimension. Experiments across multiple tasks show that MTLA greatly accelerates inference and reduces GPU memory usage at inference without sacrificing accuracy.

STRENGTHS AND WEAKNESSES:
Strengths:
* This is a very novel method that successfully compress KV cache along the temporal dimension.
* Experiments also show promising results.
* The paper is well written and easy to follow.

Weakness:
* Although the proposed method could be a promising acceleration solution to LLMs, the authors only conduct experiments on relative small settings (d=512). It would be great if authors could add experiments on LM tasks. Even a 1~7B LM model could make this work more siginificant.

QUESTIONS:
* If my understanding to the algorithm is correct, Figure 2 (c) is incorrect. Concretely, the first element of each row (instead of each column) should be q1, q2, q3 and q4, respectively.

LIMITATIONS:
See Weaknesses.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-28

comments: SUMMARY:
This paper optimizes KV cache for attention-based inference by introducing Multi-head Temporal Latent Attention, which compresses KV cache along the temporal dimension via dynamic merging of adjacent tokens and adjusts training-inference methods to reduce overhead.

STRENGTHS AND WEAKNESSES:
Strengths:
1.	The paper presents a clear and well-structured introduction to the method, making it easy to follow the design rationale and model workflow.
2.	It provides a logical review of related work, clearly highlighting the distinctions between the proposed approach and existing methods.
3.	Experiments on speech tasks demonstrate significant inference speedups while maintaining accuracy, showcasing practical efficiency gains.
Weaknesses:
1.	Although temporal merging shows effectiveness in experiments, the paper lacks in-depth analysis of its principles and validity, requiring more theoretical insights.
2.	The tested scenarios focus on long-sequence speech tasks, and the generalizability of the approach across broader applications remains unvalidated.
3.	The compression ratio s is a hyperparameter that requires manual tuning, but the paper does not offer guidance on optimal setting strategies or insights into its impact.

QUESTIONS:
1.	What is the practical or theoretical significance of merging latent embeddings of tokens temporally?
2.	Does the proposed method alter the model training process, and how do training time and parameter space perform?
3.	Could you provide parameter implementations under different s values to further clarify the real effects of temporal merging and offer deeper insights?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to compress the self-attention KV-cache along the temporal dimension, extending the recent Multi-head Latent Attention (MLA) technique to further reduce compute and memory overhead.\n\nSince the proposed approach modifies the KV cache temporal sequence during attention, special care must be taken during training to ensure that it is consistent with the partial decimation in time that must occur during token-by-token sampling during inference.\n\nExperiments on small scale speech and text tasks demonstrate significant memory/speed improvements without hurting performance compared to a vanilla attention baseline and other commonly used attention optimizations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- Straightforward, well motivated approach to reduce self-attention compute/memory requirements, extending recent work on latent attention.\n- Experimental validation of decoder-only transformer across several different tasks.\n- Good results - large inference speedups over latent attention baseline.\n\nWeaknesses\n\n- The main weakness is that the experimental evaluation is on on very small scale models of around 100M parameters, more than an order of magnitude smaller than the smallest typical LLM size (~2-3B parameters).  The author's justify this choice due to limited access to compute, which is totally understandable, but this might limit interest in the work.\n\n- Curious choice of evaluation tasks, focusing primarily on speech tasks.  The proposed technique seems quite generally applicable to any sequence modeling task.  Is there a reason for not including incorporating other NLP tasks, e.g., machine translation (which is supported in fairseq) in the experiments?  It is reasonable to assume that the proposed technique better suited to speech tasks because sequence lengths are longer.  And perhaps speech signals are more naturally compressible in time than tokenized text, making them better suited to the proposed temporal compression technique?  But it would nevertheless be valuable to include some results on some other NLP tasks, even if sequences are shorter, to validate that compressing context does not hurt performance on such tasks.\n\n   - Similarly $s > 2$ is only evaluated on the speech translation task.  It would be valuable to see how MTLA performs on other tasks with more compression., especially XSum since it is the only text-only task considered in the paper.  \n\n- Some design choices not thoroughly justified/evaluated, especially the construction of the hypernetwork that compute the merging weights $w_i$ in Eq. 13.  Are these details important?  Does performance suffer if $w_i = 1/s$ (simple averaging) or some other constant?  If nothing else, some discussion motivating this choice  seems warranted.  Even better would be a quick ablation experiment comparing the proposed technique to a simple baseline.\n\nQUESTIONS:\nSee specific suggestions for improvements listed in Weakness above.\n\nMinor issues/questions not raised above:\n\n - I found Fig 1 to be a bit confusing.  Shouldn't the \"Multi-head Queries\" also be derived from the \"Attention Input\"?  Also, it would be even more clear if the different components were explicitly connected to the notation introduced in Sec 3.\n\n- Sec 4.3, line 184:  Did you mean \"Eq. 9\" instead of \"Eq. 8\"?\n\n- Sec 4.3, line 201: $9d_hl/(2s)$ would be $2.25 d_hl$, not $2.5 d_hl$ if $s=2$.\n\n- Table 5 repeats most of Table 1.  These can probably be consolidated into a single table.\n\n- The final two sentences of the first paragraph of Sec. 5.2 (lines 221-224) are near exact repeats of the preceding two sentences.\n\n- End of Sec 6.3: Why only comment on the significance of one result?  Are the performance improvements of \"Proposed MTLA w/$s=4$\" over the other baselines not significant?\n\nLIMITATIONS:\nYes.  The authors explicitly acknowledge the limitations of the small scale experiments due to limited compute resources available to them.  This is totally reasonable.  However, as described above, it feels like an oversight to focus exclusively on long sequence tasks in the experimental evaluation with no other justification.  Some experimental comparisons on other text tasks (at small scale) would give a more rounded perspective on how broadly applicable MLTA is.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCurrent LLM architectures face severe memory constraints during inference. To address this challenge, the authors introduce a novel technique that compresses the KV-cache along the temporal dimension using a learnable approach. They report over 3× speedups and substantial VRAM savings during inference compared to vanilla multi-head attention.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The authors present Multi-head Temporal Latent Attention (MTLA), a novel self-attention variant that pioneers learnable compression of the KV-cache along the temporal axis while preserving the key-value compression from MLA. This approach opens a promising new research direction for improving inference-time efficiency in Transformer-based models.\n- The research addresses a well-motivated problem, as current LLMs face significant memory bottlenecks due to KV-cache growth. The proposed approach introduces clever technical solutions with sound theoretical explanations.\n- The empirical results are compelling: the method demonstrates consistent performance across diverse tasks (speech translation, ASR, text summarization, and SLU) while matching the accuracy of previous methods. This suggests that the compression effectively removes redundancy from the KV-cache without sacrificing model quality.\n# Weaknesses\n- **Novelty claim**: While the authors claim this is the first work to explore KV-cache compression along the temporal axis, [Nawrot et al. (2024)](https://arxiv.org/abs/2403.09636) have already investigated temporal compression of the KV-cache. Although their approach focuses on retrofitting pre-trained LLMs rather than making fundamental architectural changes, this prior work should be acknowledged.\n- **Limited comparison against baselines**: The paper does not empirically compare against established KV-cache pruning methods such as LazyLLM and SnapKV, despite mentioning them in the related work.\n- **Limited comparison to other orthogonal approaches**: the KV-cache can be compressed with other methods (e.g. quantization). The authors do not compare the speed-ups obtained by MTLA to other orthogonal methods, or a combination of both.\n- **GPU memory usage reporting**: The analysis of GPU memory usage lacks detail regarding measurement methodology and the breakdown between model weights, activations, and KV-cache consumption. Moreover, the average GPU memory usage is reported, whereas a more informative measure would be the peak memory usage.\n- **Benchmarks might not reflect real deployment metrics**: The reported memory usage and inference time metrics heavily depend on implementation details. Since the authors acknowledge not using optimizations like FlashAttention or compiled kernels, the results may not reflect realistic deployment scenarios.\n- **Lack of confidence intervals**: The experimental results lack error bars or confidence intervals, which would strengthen the statistical validity of the comparisons given the stochastic nature of neural network training.\n- **Limited scale**: The core motivation addresses memory bottlenecks that primarily affect large models processing very long sequences. However, the evaluation uses small models with approximately 0.1B parameters, which may not adequately demonstrate the method's benefits at scale.\n- **Lack of evaluation on common tasks**: The evaluation lacks standard LLM benchmarks (language modeling, code generation, reasoning tasks) and crucially omits long-context evaluations (100k+ tokens) where the method's advantages should be most pronounced.\n\nQUESTIONS:\n1. Can this approach be combined with other head reduction techniques such as GQA?\n2. Have you experimented with FP16/BF16 precision for training and inference?\n3. While resource constraints prevent testing on large-scale LLMs, what evidence suggests this approach will scale effectively to larger models? Could you provide scaling law experiments, even if limited to smaller model sizes?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Multi-head Temporal Latent Attention (MTLA), a novel attention mechanism that reduces inference-time memory and improves speed by compressing the Key-Value (KV) cache both in the latent and temporal dimensions. To ensure consistency between training and inference behaviors, the paper proposes a stride-aware causal mask.\nThe evaluation on speech translation, speech recognition, speech understanding, and text summarization shows that MTLA can achieve substantial improvements in efficiency with competitive accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- The paper is well-written and easy to follow.\n- The proposed method shows strong empirical performance on speech translation, speech recognition, speech understanding, and text summarization tasks.\n- The stride-aware causal mask enables efficient parallel training while preserving inference-time attention behavior\n\nWeakness and Questions:\n- Lack of comparison to other token compression works like [1]. \n- Although the paper mentions FlashAttention in the limitations section, it remains unclear how MTLA compares in real-world end-to-end latency. Integrating MTLA with FlashAttention may not be trivial. It will be great for the paper to have such a kernel to show that the proposed attention can achieve wall-clock time speedup.\n- Although the proposed method has been evaluated in various tasks like speech recognition and summarization. I feel that those tasks are not challenging enough. For example, what is the context length for those tasks? Considering that attention only becomes a major bottleneck only when the context length is long enough. It would be great for the paper to also test the proposed methods on more challenging long-context tasks. \n- Another concern about the evaluation is that it remains unclear whether the proposed attention mechanism is effective in long-context LLM scenarios, which represent a major challenge for long-context inference. It would strengthen the paper to justify or demonstrate its effectiveness in such settings.\n\n[1] Zhang, Peitian, et al. \"Long Context Compression with Activation Beacon.\" The Thirteenth International Conference on Learning Representations.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}