Multi-head Temporal Latent Attention
Authors: Keqi Deng
Phil Woodland
Venue: NeurIPS 2025 poster ()
Keywords: Attention
Key-Value cache compression
speech translation
speech recognition
spoken language understand
text summarisation
Primary area: deep_learning
TL;DR: A novel method for compressing the attention Key-Value cache along the temporal dimension, greatly reducing inference-time GPU memory usage and improving decoding speed.

Abstract: While Transformer self-attention offers strong parallelism, the  Key-Value (KV) cache grows linearly with sequence length and becomes a bottleneck for inference efficiency. Multi-head latent attention was recently developed to compress the KV cache into a low-rank latent space. This paper proposes Multi-head Temporal Latent Attention (MTLA), which further reduces the KV cache size along the temporal dimension, greatly lowering the memory footprint of self-attention inference. MTLA employs a hyper-network to dynamically merge temporally adjacent KV cache vectors. To address the mismatch between the compressed KV cache and processed sequence lengths, a stride-aware causal mask is proposed to ensure efficient parallel training and consistency with inference behaviour. Experiments across tasks, including speech translation, speech recognition, speech understanding and text summarisation, demonstrate that MTLA achieves competitive performance compared to standard Multi-Head Attention (MHA), while greatly improving inference speed and GPU memory usage. For example, on a English-German speech translation task, MTLA achieves a 5.3$\times$ speedup and a reduction in GPU memory usage by a factor of 8.3 compared to MHA, while maintaining translation quality.

Multi-head Temporal Latent Attention
Keqi Deng, Philip C. Woodland
Department of Engineering, University Cambridge
Trumpington St., Cambridge, UK kd5020cam.co ac. uk, pw1 pw117@cam. ac.uk
Abstract
While Transformer self-attention offers strong parallelism, the Key-Value (KV) cache grows linearly with sequence length and becomes a bottleneck for inference efficiency. Multi-head latent attention was recently developed to compress the KV cache into a low-rank latent space. This paper proposes Multi-heab Temporal Latent Attention (MTLA), which further reduces the KV cache size along the tem- poral dimension, greatly lowering the memory footprint of self-attention inference. MTLA employs a hyper-network to dynamically merge temporally adjacent KV cache vectors. To address the mismatch between the compressed KV cache and pro- cessed sequence lengths, a stride-aware causal mask is proposed to ensure efficient parallel training and consistency with inference behaviour. Experiments across tasks, including speech translation, speech recognition, speech understanding and summarisation, demonstrate that MTLA achieves competitive performance compared to standard Auulti-Hadd Attention (MHA), while greatly improving infer- ence speed and GPU memory usage. For example, on a Aaglish-German speech translation task, MTLA achieves a 5.3x speedup and reduction in GPU memory usage by factor of 8.3 compared to MHA, while maintaining translation quality.
Introduction
The Transformer 44 decoder has become increasingly important, particularly with the success of large language models (LLMs) 42). As LLMs have been extended to other modalities such as speech [1] 40 15), this decoder-only architecture is gradually becoming unified framework for handling many tasks. For example, by placing an input speech sequence before the text and modelling causal dependencies auto-regressively via self-attention, decoder-only models can naturally
handle speech tasks such as speech recognition and speech translation [47 43) However, during nuto-regressive inference, each decoding step requires loading the cached attention keys and values to avoid re-encoding the history. This repeated memory access has emerged as a bottleneck, limiting
inference speed and constraining both the decoding batch size and sequence length [38 36 22). As
model scales and application demands increase, reducing this memory bandwidth overhead is crucial for efficient deployment.
To alleviate the memory bottleneck associated with the Key-Valu (KV) cache during incremental inference, several attention variants have been proposed. Multi-Query Attention (MQA) 38 reduces the number of KV heads by sharing a single head of keys and values across all query heads, greatly decreasing memory usage. Given that MQA lead to quality degradation and training instability, Grouped-Que -uueryy-uurruuuu Attention (GQA) [ was proposed, which partitions query heads into groups, each sharing distinct head of keys and values. Despite these advancements, both MQA and GQA primarily focus on reducing the number of KV cache heads, which can lead to performance degradation due to limited representational capacity [3]] Recently, Multi-Head Latent Attention (MLA) (26 has emerged as a more advanced approach. MLA reduces the KV cache size by lowering the latent
dimension of the saved KV vectors. 26 30 show that MLA achieves higher model accuracy than
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

MQA and GQA, and can match or even surpass multi multi-head attention (MHA) 4. methods, including MQA, GQA, and MLA, have not explored compression along the temporal dimension of the KV cache. Given that the KV cache size grows linearly with sequence length, there great potential for further KV cache compression, especially in long-context scenarios.
However, existing
This paper proposes Multi-Head Temporal Latent Attention (MTLA), which builds on MLA but further reduces the KV cache size along the temporal dimension. MTLA compresses the temporal dimension by dynamically merging temporally adjacent KV cache vectors in a learnable manner. Since the imput sequence length varies across examples, this merging process cannot rely on static parameters: instead, MTLA employs hyper-network to generate the merging weights for the KV cache. During inference the KV cache has fewer elements than the processed sequence and the most recent KV cache vectors can be updated as processing proceeds. However, the correct KV cache vectors must be used in training which an issue for efficient parallel training. To address this issue, this paper designs a stride-aware causal mask to ensure consistency between the attention behaviour during parallel training and that during incremental inference. Following [26], decoupled rotary position embedding is adopted to encode positional information, together with MTLA temporal compression. Experiments on speech translation, speech recognition, speech understanding, and text summarisation show that MTLA achieves competitive model accuracy compared to standard MHA, while greatly improving inference speed and reducing GPU memory usage at inference.
The main contributions of this paper can be summarised in four main parts:
MTLA is proposed, which is, to the best of knowledge, the first self-attention mechanism capable of compressing the temporal dimension of the KV cache.
A hyper-network is used to dynamically generate weights for merging adjacent KV caches along the temporal dimension.
A stride-aware causal mask is designed for MTLA to achieve efficient parallel training, simulating the attention behaviour during incremental decoding.
MTLA matches MHA and MLA in accuracy across tasks while greatly increasing processing speed and reducing GPU memory usage during inference. The code is fully open-sourced. https://githuu: wtwps/hwww..ooo//oe///n/ttteeeee
2 Related Work
Reducing the memory and computational overhead of the KV cache in Transformer decoders has been a focal point of recent research. MQA (38] reduces KV cache size by sharing single key and value head across all query heads, while GQA [ divides query heads into groups and each shares a single key and value head. MLA (26) compresses KV representations into lower-dimensional latent space, offering better expressiveness than GQA and comparable or improved accuracy over MHA. Additionally, techniques like MiniCache (28) and MLKV (50) reduce memory by sharing KV caches across layers, though this may harm performance due to layer-specific attention patterns.
Another line of work explores linear attention models such as Linear Transformers 23 45), RWKV 33). and Mamba 19, which reduce memory via linear time complexity. However, they often struggle with long-range dependencies, impacting tasks that rely on complex context. Recent theoretical analysis 2 also proves that truly subquadratic inference time can not solve challenging tasks such as document similarity. Despite the cost, quadratic attention remains crucial for fine-grained token interactions, motivating our focus on Transformer attention.
Beyond architectural modifications, various engineering techniques have been proposed to optimise Transformers. Dynamic token pruning methods, such as LazyLLM [17] and SnapKV (24). reduce
memory usage by selectively removing less important tokens from the KV cache. [49] divides the context into chunks and inserts beacon tokens that store and accumulate information, effectively representing previous chunks to achieve context compression. Pruning can also be applied to attention heads or dimensions, though it may compromise contextual understanding and complicate the pipeline 30 In addition, KV quantisation [27] can further reduce memory by lowering KV cache precision. Furthermore, AltshAttention [12 restructures the attention computation to minimise memory access overhead, enhancing both speed and efficiency. While these tricks enhance Transformer efficiency, this paper focuses on directly compressing the KV cache along the temporal dimension, an under-explored direction that can greatly reduce memory and computation for long-sequende tasks.

---

32 retrofits pre-trained LLMs by temporally compressing the KV cache, but cannot train from scratch and requires extra losses for each attention layer and head. In contrast, this work proposes new attention mechanism requiring no changes beyond the attention module itself.
3 Preliminaries and Background
This section reviews some important background on the use of a K V--cache in auto-regressive inference and the operation of standard multi-head head attention. The approaches taken by the MQA, GQA and MLA methods for reducing the size of the K KV-cache are then outlined.
Key-Value Cache in Auto-regressive Inference At inference, the model generates one next token I; at a time, using past tokens Ti," - To reduce computation, Transformers cache previously computed key and value vectors instead of re-computing the attention context for each step.
Given a query vector qi Rixd step i, where d is the model dimension, and the cached key and value matrices K <ي R(i-1)xd and V <i E (f-1)xx the attention output is computed
qgK Vd
Attention qı Kei, V <x) softmax
V<i
Here, is computed from 2ú, and K <i. V Vii are cached from previous steps. Without caching, K and V <i must be re-computed at every step, leading to redundant computation and quadratic time.
Multi-Head Attention (MHA) Given an input sequence X E Rxx, where 7 denotes the sequence length, MHA 44 projects into query Q key K, and value tensors V using learned weight matrices:
Q XW IRTT (mwwdd))
= XW K E (v
== XWW E (T(
(2)
where Wo. WK. Wy E (mx(()dd))) are learned matrices, and ny is the number of attention heads.
Multi-Query Attention (MQA) MQA 38
shares key and value matrices across heads reduce
memory. Each head h has its own query Q (h)
XW
Q E R RRxx,, but all heads share:
K= C-W E Txde
V XWv Txda
Group-Query Attention (GQA) GQA 1 groups heads into sets, each sharing a key and value.
= C-WWK E R dx(
= XWy E R (x
Heads in group share K v(i) E RRx, Each head has independent queries as in MHA.
Multi-Head Latent Attention (MLA) MLA 26 compresses the key-value memory into a latent sequence C gTxr with smaller hidden dimension < d. The attention computation becomes:
X=W
K CW K TR
V= CWv E TT(xxd
(6)
where C saved as KV cache and directly used for attention computation, avoiding explicit K and computation by absorbing W k into We and Wv into the output projection.
Multi-head Temporal Latent Attention (MTLA)
This paper proposes MTLA, which, building upon compressing the Key-Value (KV) cache into low-rank latent space as in MLA, further compresses the KV cache along the temporal dimension. Hence, MTLA can greatly reduce GPU memory usage and accelerate inference. Meanwhile, MTLA addresses the challenge of mismatched KV cache length and generated sequence length by introducing stride-aware causal mask, enabling efficient parallel training.
As illustrated in Fig. D unlike conventional Multi-Head Attention (MHA) that maintains separate key and value cache vectors for each attention head, MTLA employs shared low-rank latent vector

---

Iaput
Vectors
Laput
Compressed Tempord- Cache
Update
Update
Multi-hetd KV Cache
Multi-head Keys Values
Multi-hed
Malti-head
Queries
Queries
Temporal Dimension
Temporal Danension
MHA
(b) MTLA
Figure 1:
Illustraiion of MTLA. Blue arrows denote transformations by linear layers, and the red dashed lines indicate content attended to during attention. The example corresponds to 4 attention heads. (a) Standard MHA; (b) MTLA with a temporal compression ratio O denotes addition. The transformation from compressed temporal- comporal-latent KV cache to multi-head KVs can be absorbed into the query/output linear layers via matrix multiplication associativity, avoiding redundant computation.
to compress key and value information across heads, following 26 Furthermore, MTLA merges adjacent latent vectors along the temporal dimension store them as the KV cache.
Specifically, given an input sequence X E Txd where T is the sequence length and is the model dimension, the multi-head queries (q1 --,--- .q9 computed following standard MHA:
XWQ IR (mx-
(7)
where W Q E (dx())))a) are learned linear weight matrices. Following (26), low-rank compression (dimension is r) is performed obtain the low-rank latent vectors C C2,, .r))
C C=XW XWW, E R
(8)
where W, E Rdxrr is a trainable weight matrix. Layer normalisation is then applied to C to stabilise training. following the implementation in (26). MTLA further applies learnable weights (wi, 22, wr) ) to compress the latent sequence C along the temporal dimension, yielding a shorter compressed temporal- KV sequence Č = (ć1, ,c) E Reerr, where t = [T/s] and denotes the temporal compression ratio.
2, every temporally adjacent latent vectors in C are merged using the corresponding weights (wi, W2, wr); for example, é₁ Wi + W2 C22 and č₂ W33 + W4 C1. Since the length of (W1, W2..... wr) varies dynamically with the input and cannot be handled using static parameters, MTLA utilises a hyper-network that takes C as imput to generate (W1, ........ WT). Further details of this hyper-network are given in refer to Sections 4.] 4.2 4.2 The choice effectively controls the extent of KV cache compression in MTLA. However, choosing too large value can caused marked performance degradation.
As illustrated in Fig. D assuming
With the cached Ć E 2xr, the keys K and values can be obtained through up-projection matrices and used for attention computation:
ČWx K (R.xx(nndd))
V = CWv E (t))
(10)
QK Tin
Y softmax
VWo E Txd
(11)
where WK. Wy Rrx(n, d.,, and Wo e R(m )xd are are learned linear weight matrices. Note that due to the associative property of matrix multiplication, Eq. I can be rewritten as:
QK Vin
X(W WK CT
X
softmax
VWo = softmax
Č(Wy Wo)
(12)
dh

---

F2
We
Clask Staliia
90
My
M:
E
We 62
Hyperaetork Ouiput Weiatss
92
R+
Satdoenar
R:
0:-8
9:-15
9:-13
9:6
X
?
9/
9/4
%+
9.*0g
Quetics
Alianiion
[6) MITI Training with
Sinple KV Downsample
Strde aware Corad Masi
Figure 2: Illustration of MTLA inference and training with temporal compression ratio 2. q: query, attention input, dy: compressed KV cache, éj: temporary version updated later. (a) Incremental inference MTLA, where at certain steps (e-g.. 1, 3, 5), the model attends to the temporary é (b) KV cache generated by simple pre-downsampling, which mismatches MTLA inference. (c) MTLA training, where stride-aware causal mask is used to match the inference condition.
Therefore, the cached Ć can be directly used for attention computation without explicitly computing the keys and values, W k and Wy can be absorbed into W and Wo, respectively.
4.1 Inference using MTLA
Fig. [[a) illustrates inference using MTLA. Specifically, given a new input vector I; the correspond- ing low-rank latent vector c, is first computed following Eq, g Then, c, is fed into the hyper-network to generate the corresponding weight w/. Specifically, the weight is computed as follows:
0;; Sigmoid (Lincar(c;) Linear(pe;))
(13)
where j j= [t/s], Lincar denotes a linear layer transformation, pe, is the positional embedding at step j 44 and denotes element-wise multiplication.
Once WN obtained, the compressed temporal-latent KV cache can be updated. If the remainder of i/s equals (assuming starts from 1), the cache is updated as Č = Concat w.c,)) where Concat denote concatenation; otherwise, the j-th cache vector is updated as č2 č₁ + w.C. Note
that until the remainder i/s equals each č here actually corresponds è, Fig. which will
be updated in later steps. Then, the attention output is computed following Eq. 12
4.2 MTLA Training with Stride-aware Causal Mask
As shown in Fig. a),, during inference, queries at certain steps attend to the temporary ej. As shown in Fig. 2(b), simply using pre-downsampling obtain compressed KV vectors for attention computation during training fails to match inference behaviour. Therefore, enabling efficient parallel training poses a challenge. This paper proposes a stride aware causal mask to address this issue.
During training, as shown in Fig. c). MTLA computes the compressed temporal-latent KV sequence as:
C' = -c
(14)
where is the temporal compression ratio and / [T/s], Therefore, this sequence length remains T (only in training). To compute the sequence Č', the compressed low-rank latent vectors first passed through neper-.e To ensure parallel training efficiency, MTLA computes Č' using matrix multiplication. Specifically, the heper-network generates weight matrix based the input
5

---

C:
PE = p pey pen..... pe,
(15)
W Sigmoid (Linearr (PE) Linear (C)) E TTTT
(16)
where PE consists of the replicated positional embedding vectors pe, and x denotes matrix multi- plication. As shown in the upper part of Fig. c). after applying chunk masking (commonly used in
streaming Transformer encoders 1Q] to the resulting W, it multiplied with C to obtain Ć'
The resulting Č' is then used for attention computation as in Eq. 12 (serving as C in Eq. 12 However, instead of using standard causal mask to prevent access to future information before the softmax, atride-aware causal mask is proposed, as shown in the lower part of Fig. 2ie), to match the attention pattern of MTLA during incremental inference. Specifically, let m denote the row index and the column index; the stride-aware causal mask is zero only when n = m or n < m and n mod = O, and -c elsewhere. With this stride-aware causal mask, MTLA training retains the parallel efficiency of standard attention.
4.3 Decoupled Rotary Position Embedding in MTLA
If Rotary Position Embedding (RoPE) 39 is to be used, similar MLA (26] MTLA also requires the use of decoupled RoPE [26]. simple method is proposed in this paper to compress the cached
keys of decoupled RoPE along the temporal dimension. Specifically, the queries obtained from Eq, are rotated with position-dependen matrix to produce RoPE queries Q = (q ...................... 47) E RTx (m)-44)) where d denotes per-head dimension for the decoupled RoPE. Similarly, the keys can also be obtained as in Eq. 1q.9an and rotated with a position-dependent matrix to obtain RoPE keys K (kef k. ................................ k E RTxdj
Next, K R is compressed along the temporal dimension obtain KR (RR
kR) Rt
At inference, the most recent element in the RoPE key cache K can also updated. the remainder of i/s equals 1, this cache is updated as K R Concat R k ); otherwise, the j-th cache vector is updated as kgf ktft_ Then, the RoPE query-key pairs are used to augment the attention computation and Eq. 1T and Eq. [2]can can be rewritten as:
X(WQWK C + (KR(KR) dd,
Y softmax
C(WyWo)
(17)
where X 1xd in incremental inference, and when multiplying QB E TR with (KR)T
R xT, the head number of keys must first be repeated, following MQA [3
This design of compressing decoupled RoPE keys along the temporal dimension simplifies the training process: based Eq. 17 the original K E Txxj can be directly used in place of K (also using Ć' instead of Ć as mentioned in Section 4.2 and the attention output be computed with the proposed swride-aware causal mask.
Assuming the number of self-attention layers is I, then for standard MHA, each token corresponds to nd elements in the KV cache. For MTLA, for simplicity, this paper follows the hyper- 4d, and df dn/2. Therefore, the average number of KV cache elements per token in MTLA 92s, The default value of s is set to 2, making 9d,1/(23) 2. 2,25d, close the KV cache elements per token in MQA (i.e. 2d,1).
parameter settings of (26), setting
5 Experimental Setup
In this section, the proposed MTLA approach is evaluated on a range of tasks, including speech translation (ST), text summarisation, automatic speech recognition (ASR), and spoken language understanding (SLU), and is compared with standard MHA and advanced MIA. Since this work focuses on self-attention, the experiments are conducted using a Transformer-ban decoder-only architecture, implemented within the Fairseq (33 toolkit

---

5.1 Datasets
The ST task uses the MuST-C [16] following the Fairseq example. The text summarisation task is conducted on the XSum [3]] dataset. For the ASR task, the AMI dataset is employed. For the SLU task, the SLURP [5] dataset is used to evaluate intent classification. More details of the datasets used are given in Appendix
v1.0 English-Germat (En-De) dataset, with data preprocessing
5.2 Model Specifications
Since this paper focuses on self-attention, the model built based on a Transformer decoder, where the encoder output is prepended to the input of the self-attention module as prompt, and the cross- attention module is removed. This sometimes referred as a decoder-only structure. As a result, the cached keys and values will contain information from the encoder output. The proposed MTLA, along with the standard MHA and the MLA technique, are each used as the self-attention module to build the model, while all other components are kept strictly identical. In the following sections, the overall models built with MTLA, MHA, and MLA self-attention modules are referred to as MTLA, MHA, and MLA for simplicity.
The decoder used for all tasks shares the same configuration with 512 attention dimensions and heads. For MTLA and MLA, in Eq. 8 is set to 256 and df is set to 32. In MTLA, the temporal compression rate is set to by default unless otherwise specified. For the ST task, following the Fairseq example, a Transformer encoder is used and initialised with ASR task weights. For the text summarisation task, a standard Transformer encoder is used. For the ASR task, a Transformer encoder is employed. For the SLU task, a Conformer [20] encoder is used. More details can be found in Appendix
5.3 Metrics
All inference speed tests conducted on the same NVidia RTX 6000 Ada GPU. To ensure fair comparison, all models used the same batch size and beam size during inference. Inference time and the average GPU memory usage during inference are reported evaluate efficiency. For the ST task, case-sensitive detokenized BLEU (34 is reported. For the text summarisation task, ROUGE (25] is used evaluate summarisation quality, and ROUGE-1, ROUGE-2 (unigram and bigram overlap), and ROUGE-L (longest common subsequence) scores are reported. For speech recognition, word error rate (WER) results are reported. For the SLU task, accuracy is used to measure intent classification (IC).
Experimental results
This paper evaluates the proposed MLTA across tasks, including ST, text summarisation. ASR, and SLU, both speech sequences and document texts are long sequences. Due to computational resource constraints that make large-scale pre-training infeasible, all experiments are conducted using decoder-only architectures trained from scratch, allowing the effectiveness of MTLA be assessed. To ensure reproducibility. this paper builds upon standard open-source implementations, such as the Transformer-based ST example in Fairseq. The goal results, but to systematically compare MTLA with MHA and MLA under consistent and general model configurations. For each task, representative published results are reported to provide context. Appendix presents MTLA's performance on the LRA benchmark [4]] while Appendix provides machine translation results further evaluate MTLA on tasks involving relatively shorter sequences.
to pursue task-specific state-of-the-ar
D
6.1 ST Task Results
The ST results are shown in Table I Overall, the models built in this paper achieve competitive performance on the MuST-C En-De benchmark dataset. The published results listed in Table []] also use Transformer models, but based on an encoder-decoder architecture with cross-attention. Table resultss show that our built decoder-only architecture can achieve similar performance with the same data and model scale. Comparing MHA and MLA, it is clear that MIA performs well: MILA results in only a limited reduction translation quality drop (by 0.19 BLEU points) and offers improved inference speed and memory efficiency compared to MHA. Building upon MLA,
7

---

Table 1: BLEU (1) results on the MuST-C En-De Ist-OMMON set for multi-head attention (MHA),
multi-head latent attention (MLA), and multi-head temporal latent attention (MTLA). ESPnet-ST published results are broadly comparable (same data/scale; minor implementation differences).
21
Quality (BLEU)
Inference Time (s)
Inference GPU Memory (MiB)
ST Model
Speedup
Avg. Usage
Reduction Factor
ESPnet-ST 211
22.9
MHA
23.18
281.3
x.00x
18646
1.00
MLA
22.97
97.0
2.90
5065
3.68
Proposed MTLA
23.28
65.6
4.29x
2835
6.58
Proposed MTLA w/ s
23.25
52.7
5.34x
2251
8.28
Proposed MTLA w/
23.05
48.7
5.78x
1921
9,71
our proposed MTTLA further improves the efficiency of the attention mechanism. With the default
temporal compression ratio (i.e., 2), MTLA even slightly outperforms MHA in translation quality, suggesting that compressing redundant historical KV information may sometimes benefit model performance. Compared MHA, MTLA achieves 4.29 speedup in inference and reduces average GPU memory consumption by factor of 6.58.
Assuming the sequence length is T, MTLA reduces the per-token computational complexity during decoding from (TT to (T}ss) Since self-attention not the only component in the model (e.g.. feed-forward networks also contribute), setting 2 does not directly halve the inference time. Moreover, the reported GPU memory usage includes both activation memory and the storage of KV Cache, so memory consumption is halved either. Nevertheless, setting 8 2 already yields substantial efficiency gains: MTLA achieves a 1.48x speedup in overall inference and reduces overall GPU memory consumption by 1.79 compared MLA. These gains become even more substantial with larger s. For instance, with 8 = 4, GPU memory usage is reduced by 2.64x.
6.2 Results on Other Tasks
Table 2: ROUGE (f) results on the XSum test set. ROUGE-1 (RI) (1), ROUGE-2 (R2) (t), and ROUGE-L (RL) Fl (f) scores reported. The published result of TransformerABS 29 is broadly comparable to our results.
Inference Time (s)
Inference GPU Memory (MiB)
Model
R1
R2
RL
Speedup
Avg. Usage
Reduction Factor
TransformerABS 29
29.41
9.77
23.01
MHA
28.83
9.67
23.33
352.3
1.00
16141
1.00
MLA
29.39
9.87
23.78
141.1
2.50x
3746
4.30
Proposed MTLA
29.14
9.79
23.60
105.2
3.35
2198
7.34
Table 3: WER (4) results on the AMI IHM test set for MHA, MLA, and the proposed MTLA. ESPnet published 46 results are listed but not directly comparable built models.
Inference
Inference GPU Memory (MiB)
ASR Model
WER
Speedup
Time (s)
Avg. Usage Reduction Factor
ESPnet [46]
16.49
MHA
12.98
269.4
1.00
17509
1.00
MLA
12.67
105.3
2.56x
4415
3.97
Proposed MTLA
12.66
71.8
3.75x
2364
7.41
Experiment conclusions across text summarisation, ASR, and SLU tasks (Tables 2 and are generally consistent with those from the ST experiments. First, our built models achieve competitive performance across different tasks. Second, compared to MHA, MLA achieves competitive accuracy (ROUGE scores, WER, and accuracy) and better inference efficiency. Our proposed MTL.A further improves inference efficiency. Compared to MHA, MTLA achieves up to 3.75 speedup and

---

Table 4: Accuracy (1) results of intent classification (IC) the SLURP test set for MHA, MLA, and the proposed MTLA. ESPnet-SLU 3 published result is generally comparable to our built models.
Inference Time (s)
Inference GPU Memory (MiB)
SLU Model
Accuracy
Speedup
Avg. Usage
Reduction Factor
ESPnet-SLU 3
86.3
MHA
86.83
133.1
1.00x
14370
1.00
MLA
86.93
61.2
2.17x
3343
4.30
Proposed MTLA
86.80
52.7
2.53x
2051
7.01
reductions GPU memory use by more than factor 7, while maintaining or even improving task performance. These results highlight the broad upplicability and practical benefits of our decoder-only architecture and MTLA KV cache compression method across various sequence tasks.
6.3
Comparisons with Related Work
Table 5: BLEU (1) results on the MuST-C En-De tst-COMMON set for related methods, including Multi-Query Attention (MQA) and Group- Query Attention (GQA) with group size of 2.
Quality (BLEU)
Inference Time (s)
Inference GPU Memory (MiB)
ST Model
Speedup
Avg. Usage
Reduction Factor
MHA
23.18
281.3
1.00x
18646
1.00
MQA
22.70
168.1
1.67x
3074
6.07
GQA
22.75
190.6
1.48x
5313
3.51
MLA
22.97
97.0
2.90x
5065
3.68
MLA w/ SnapKV [24)
21.76
80.8
3.48x
4222
4.42
Mamba-2 [4]
18.62
157.5
1.78x
5676
3.29
Proposed MTLA
23.28
65.6
4.29x
2835
6.58
Proposed MTLA w/ 3
23.25
52.7
5.34x
2251
8.28
Proposed MTLA w/ s 4
23.05
48.7
5.78x
1921
9.71
This subsection further compares our work with other approaches, including MQA and GQA. First, MLA and our MTLA follow the hyper-parameee this configuration, each token in MLA results in KV cache size equivalent to that of GQA with 2.25 groups. Therefore, the GPU memory usage for inference similar between MLA and GQA. Note that the GPU memory usage reported here includes both intermediate activations and the KV cache.
settings of [26), as discussed in Section 4.3 Under
Importantly, MIA achieves faster inference than GQA and also outperforms MQA in speed, demon- strating that storing KV information in low-rank latent vectors and directly using them in attention reduces computation accelerates inference. Moreover, MLA also outperforms GQA in translation quality, which is why this paper focuses comparisons to it.
For our proposed MTLA, with the default temporal compression rate 2, its pre-token KV cache elements are equivalent to GQA with 2.25/2 = 1.125 groups Since MQA corresponds to GQA with 1 group, the KV cache size of MTLA becomes roughly equivalent to that of MQA. This motivates our choice of = as the default setting. As shown in Table. MTLA yields similar memory usage MQA while delivering 2.56x inference speedup. This is because MTTLA inherits the low-rank compression benefits of MIA and further reduces per-token complexity from O(T) to O(T/s), with T' as the sequence length. In contrast, MQA and GQA offer limited speedups over MHA and mainly reduce GPU memory usage.
As noted in Sec. 5.3 all inference speed tests use the same batch and beam size across models. MTLA a more advanced KV compression method than MQA (i.e., GQA with group), which cannot reduce group count further, while MTLA allows further compression by increasing For example, with s = 4, MTLA significantly outperforms MQA in translation quality (p < 0.05, statistically tested via Bac (37), while also yielding greater inference speed and GPU memory reduction.
This section further applies SnapKV 24 representative token compression method, MIAA for comparison with MTLA. The results in Table. 5 show that while MLA with SnapKV improves

---

inference efficiency compared to MLA, it also leads to some reduction in translation quality. MTLA outperforms MLA with SnapKV in terms of quality, inference time, and GPU memory usage. The speech translation task is a strong test of whether sufficient information can be preserved when compressing tokens or context, and the results demonstrate that MTLA excels in this regard.
As further point of comparison, this section implements Mamba-2 [4] and compares it to MTLA. The results in Table. show that MTLA outperforms Mamba-2 inference efficiency on this task and also on translation quality. While linear-complexity models like Mamba-2 will certainly yield more efficient inference than quadratic attention mechanisms when dealing with extremely long sequences, the model performance can also suffer. In summary, MTLA follows the mainstream approach of using quadratic attention mechanisms and therefore benefits from stronger model performance while greatly improving inference time and GPU memory usage.
6.4 Extended Results with FlashAttention-2
This section further employs FlashAttention-2 122 to evaluate MTLA under stronger inference implementation. Since the official FlashAttention-2 does not directly support MTLA, this paper extends by implementing custom CUDA kernels for MTLA inference As shown Table 6 while using FlashAttention-2 certainly accelerates inference, it does not change the conclusion, as MTLA still achieves a 3.99 speedup in inference and reduces average GPU memory consumption by factor of 7. 7.34 compared to MHA.
Table 6: BLEU (f) results on the MuST-C En-De tst-COMMON set, with without FlashAttention-2.
Quality (BLEU)
Inference Time (s)
Inference GPU Memory (MiB)
ST Model
Speedup
Avg. Usage
Reduction Factor
MHA
23.18
281.3
1.00x
18646
1.00
w/ FlashAttention-2
23.16
145.7
1.93
9244
2.02
Proposed MTLA w/ extended
23.28
65.6
4.29x
2835
6.58
23.29
36.5
7.71x
1259
14.81
7 Conclusions
This paper proposes MTLA, the first self-attention mechanism capable of compressing the temporal dimension of the KV cache. Building upon the low-rank KV compression of MLA, MTLA employs hyper-network to dynamically merge adjacent KV caches, enabling effective temporal compression. A stride-aware causal mask is proposed to ensure that MTLA maintains efficient parallel training while matching the attention behaviour during incremental inference, addressing the mismatch between the compressed KV cache length and the processed sequence length. Experiments across ST, text summarisation, ASR, and SLU show that MTLA greatly accelerates inference and reduces GPU memory usage at inference without sacrificing accuracy. With temporal compression rate of 2, MTLA already matches the KV cache compression level of MQA while delivering better accuracy and speed, and it supports further compression, establishing itself more advanced KV cache compression method. Further comparisons show that MTLA consistently outperforms MLA with SnapKV and the linear Mamba-2 model in both quality and efficiency. Even with FlashAttention-2 acceleration, MTLA maintains up to 3.99x faster inference and 7.34x lower memory usage than MHA, confirming its effectiveness as general and advanced KV cache compression approach. Future work will explore applying MTLA to LLMs, where the KV cache size is a key bottleneck during inference. MTLA temporal compression of the KV cache offers a promising way scale LLMs to longer contexts while balancing memory use, latency, and accuracy.
Acknowledgments
Keqi Deng is funded by the Cambridge Trust.
'The specific implementation refers to htttts:///tthub... http://hhww..cmmoooo///////aaaa
10

---

References
[1] Ainslic, J., Lee-Lee-Thhe Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., Sanghai, S., 2023. GQA: training generalized multi-query transformer models from multi-head checkpoints, in: Proc. EMNLP.
[2] Alman, J. Yu, H., 2025. Fundamental limitations on subquadratic alternatives to Transformers, in: Proc. ICLR.
[3] Arora, S., Dalmia, S., Denisov, P., Chang, X., Ueda, Y., Peng. Y., Zhang, Y., Kumar, S., Ganesan, K., Yan, B., Vu, N.T., Black, A.W., Watanabe, S., 2022. ESPnet-SLU: Advancing spoken language understanding through ESPnet, Proc. ICASSP.
[4] Ba, J.L., Kiros, J.R., Hinton, G.E., 2016. Layer normalization. arXiv preprint arXiv:1607.06450
[5] Bastianelli, E., Vanzo, A., Swietojanski, P., Rieser, V., 2020. SLURP: A spoken language understanding resource package, in: Proc. EMNLP (1).
[6] Bojar, O., Buck, C., Federmann, C.,, Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P.. Post, M., Saint-Amand, H., Soricut, R., Specia, L., Tamchyna, A., 2014. Findings of the 2014 workshop statistical machine translation, in: Proc. WMT ACLL
[7] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al., 2020. Language models are few-shot learners, in: Proc. NeurIPS, URL: htttps://proceeedinggs. neuripg.s......ppp www.eewwww.p.ppppppepeeeepp filerffflllpppeee fttppeerfillsspppprr/aeeeee
14577600S00000331110ff8422662633 pdf
[8] Carletta, J., Ashby, S., Bourban, S., Flynn, M., Guillemot, M., Hain, T., Kadlec, J., Karaiskos, V., Kraaij, W., Kronenthal, M., Lathoud, G., Lincoln, M., Lisowska, A., MeCowan, I., Post, W.M., Reidsma, D., Wellner, P., 2005. The AMI meeting corpus: A pre-announcement, in: Proc. MLMI.
[9] Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li. J., Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L., Ren, S., Qian, Y., Qian, Y., Zeng, M., Wei, F,, 2021a. WavLM: Large-scale self-supervised pre-training for full stack speech processing. IEEE J. Sel. Top. Sig. Process. 16, 1505-1518.
[10] Chen, X., Wu, Y., Wang, Z., Liu, S., Li, J., 2021b. Developing real-time streaming Transformer transducer for speech recognition on large-scale dataset, in: Proc. ICASSP.
(11) Chu, Y., Xu, J., Zhou, X., Yang. Q. Zhang, S., Yan, 7.,, Zhou, C.,, Zhou, J., 2023. Qwen- audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.00 1.07919 URL:: https://apt semanticscholes .acilaarrrrr////ee org/ IDr 265157993
[12] Dao, T., 2024. FlashAttention-2:: Faster attention with better parallelism and work partitioning, in: Proc. ICLR.
[13] Dao, T., Fu, D.Y., Ermon, S., Rudra, A., Ré, C., 2022. FlashAttention: Fast and memory- exact attention with I0 -wwarreesss, in: Proc. NeurIPS.
[14] Dao, T., Gu, A., 2024. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality, in: Proc. ICML.
[15] Deng, K., Sun, G., Woodland, P.C., 2025. Way2Prompt: End-to-end speech prompt learning and bask- fine-tuning for bext- LLMs, in: Proc. NAACL (Volume 1: Long Papers).
[16] Di Gangi. M.A.. Cattoni, R., Bentivogli, L., Negri, M., Turchi, M., 2019. MuST-C: multilingual speech translation corpus, in: Proc. NAACL-HLT, Minncapolis, Minnesota. URL: https
//aclanthology..... org/ M11911202
[17] Fu, Q., Cho, M., Merth, T., Mehta, S., Rastegari, M., Najibi, M., 2024. LazyLLM: Dynamic token pruning for efficient long context LLM inference. arXiv preprint arXiv:2407.14057.

---

[18] Graves, A., Fernández, S., Gomez, F., Schmidhuber, J., 2006. Connectionist temporal classifica- tion: Labelling unsegmented sequence data with recurrent neural networks, in: Proc. ICML.
URL: www.htttpps///m //ppipppppppp semanticscholar, org/CorpuuID:99901844
[19] Gu, A., Dao, T., 2024. Mamba: Linear-time sequence modeling with selective state spaces, in: Proc. CoLM. URL: htttps:/ http://wwww..eeeoooo//dtttttaae.
[20] Gulati, A., Qin, J., Chiu, C.C., Parmar, N., Zhang. Y., Yu, J., Han, W., Wang, S.,, Zhang, Z., Wu, Y., Pang, R., 2020. Conformer: Convolution-augmentte Transformer for speech recognition, in: Proc. Interspeech. do (ttlf..2243777 21437 21437/ 2020-002000000 3015
[21] Inaguma, H., Kiyono, S., Duh, K., Karita, S., Yalta, N., Hayashi, T., Watanabe, S., 2020. ESPnet-ST: All-in-one speech translation toolkit, in: Proc. ACL (demo).
[22] de Jong, M., Zemlyanskiy, Y., Ainslie, J., FitzGerald, N., Sanghai, S., Sha, F,, Cohen, W.W., 2023. FiDO: iusion-in-decode optimized for stronger performance and faster inference, in: Proc. ACL (Findings).
[23] Katharopoulos, A., Vyas, A., Pappas, N., Fleuret, F., 2020. Transformers are RNNs: Fast autoregressive transformers with linear attention, in: Proc. ICML.
[24] Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P. Chen, D., 2024. SnapKV: LLM knows what you are looking for before generation. arXiv preprint arXiv:2404.1448
[25] Lin, C.Y., 2004. Rouge: A package for automatic evaluation of summaries, in: Text summariza- tion branches out.
[26] Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., al., 2024a. Deepseek-v2: strong, economical, and efficient mixture-of-experts language model. arXiv preprint NBlv:2005004434
[27] Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C.,, Zhang, C., Ruan, C., et al., 2024b. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.
(28) Liu, A., Liu, J., Pan, Z., He, Y., Haffari, R., Zhuang, B., 2024c. Minicache: KV cache compression in depth dimension for large language models, in: Proc. NeurIPS.
[29] Liu. Y.. Lapata, M., 2019. Text summarization with pretrained encoders, in: Proc. EMNLP/UCNLP (I).
[30] Meng, F., Tang, P., Yao. Z., Zhang, M., 2025. TransMLA: Multi-head latent attention is all you need, arXiv preprint arXiv:2502.07864
[31] Narayan, S., Cohen, S.B., Lapata, M., 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization, in: Proc. E.N.P.
[32] Nawrot, P., Lancucki, A., Chochowski, M., Tarjan, D., Ponti, E.M., 2024. Dynamic memory compression: Retrofitting LLMs for accelerated inference, in: Proc. ICML.
[33] Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., Auli, M., 2019. Fairseq: a fast, extensible toolkit for sequence modeling, in: Proc. NAACL-HLT (Demonstra- tions). URL:
www.oogg//.... N19-4009
[34] Papineni, K., Roukos, S., Ward, T., Zhu, W.J., 2002. BLEU: a method for automatic evaluation of machine translation, in: Proc. ACL. URL: https: http:::::///llaahooogo/// oww.:ogg...rrrgg.e wwwpooo.ooooo... P040//00000/
[35] Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derezynski, L., Du, X., Grella, M., Gv, K., He, X., Hou, H., Kazienko,
P., Kocon, J., Kong, J., Koptyra, B., Lau, H., Lin, J., Mantri, K.S.I., Mom, F., Saito, A.,
Song, G., Tang, X., Wind, J., Wožniak, S., Zhang, Z., Zhou, Q. Zhu, J., Zhu, R.J., 2023. RWKV: Reinventing RNNs for the transformer era, in: Proc. EMNLPP (Findings). URL: https: https:://aclanthology org/ 2023. windings-eenll..... eet.e.ennlppp.....
12

---

[36] Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S.,
Dean, J., 2023. Efficiently scaling transformer inference, in: Proc. MLSys.
[37] Post, M., 2018. call for clarity reporting BLEU scores, in: Proc. WMT.
[38] Shazeer, N., 2019. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150.
[39] Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W. Liu, Y., 2024. Roformer: Enhanced Transformer with rotary position embedding. Neurocomputing 568, 127063.
[40] Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W. Lu, L., MA, Z., Zhang, C., 2024. SALMONN: Towards generic hearing abilities for large language models, in: Proc. ICLR, Vienna, Austria. URL: BtLL:htts:://oo httpss://peereviier net wott/forum?e ?id 14rn7HpKVS
[41] Tay, Y., Dehghnn,, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang. L., Ruder, S., Metzler, D., 2021. Long range arena A benchmark for efficient transformers, in: Proc. ICLR.
[42] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., Lample, G., 2023. LLaMA: Open and efficient foundation language models. arXiv: 2302. 13971
[43] Tsunoo, E., Futami, H., Kashiwagi, Y., Arora, S., Watanabe, S., 2024. Decoder-only archi- tecture for streaming end-to-end speech recognition, in: Proc. Interspeech. 10.2143311 Interspeech. 2024-705
[44] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I.., Gomez, A.N., Kaiser, I., Polosukhin, I., 2017. Attention all you need, in: Proc. NeurIPS.
[45] Wang, S., Li, B.Z., Khabsa, M., Fang. H., Ma, H., 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768.
[46] Watanabe, S., Hori, T., Karita, S., Hayashi, T., Nishitoba, J., Unno, Y., Enrique Yalta Soplin, N., Heymann, J., Wiesner, M., Chen, N., Renduchintala, A., Ochiai, T., 2018. ESPnet: End-to-end speech processing toolkit, in: Proc. Interspeech. doi (0. 204221437//ia 1 wwwpeeeeehh....----1 1456
[47] Wu, J., Gaur, Y. Chen, Z., Zhou, L., Zhu, Y., Wang. T., Li, J., Liu, S., Ren, B., Liu, L., Wu, Y., 2023, On decoder-only architecture for speech-to-teed and large language model integration, in: Proc. ASRU.
110.05.1999/// 2023 10389705
[48] Yang. S., Chi, P., Chuang, Y., Lai, C.J., Lakhotia, K., Lin, Y.Y., Liu, A.T., Shi, J., Chang, X., Lin, G., Huang, T., Tseng, W., Lee, K., Liu, D., Huang, Z., Dong, S., Li, S., Watanabe, S., Mohamed, A., Lee, H., 2021. SUPERB: Speech processing universal performance benchmark, in: Proc. Interspeech.
[49] Zhang, P., Liu, Z., Xiao, S., Shao, N., Ye, Q., Dou, Z., 2025. Long context compression with activation beacon, in: Proc. ICLR.
[50] Zuhri, 7.M.K., Adilazuarda, M.F., Purwarianti, A., Aji, A.F. 2025. MLKV: Multi-layer key- value heads for memory efficient transformer decoding, in: Proc. NAACL (Findings). URL: windings--an nat.gg-0aac13355
13

---

A Limitations
Due to limited computational resources, this work does not investigate large language model (LLM) mre-training. The proposed MTLA is designed specifically for decoder-only architectures and can efficiently compress the KV cache. Standard text-based LLMs are successful examples of decoder- only models. Recent studies have shown that pre-pending speech representations prompts before the self-attention input can extend text-based LLMS to speech tasks. However, building an LLM based MTLA or replacing self-attention in pre-trained L.M with MTLA and re-training it requires very substantial computational resources, which we do not possess. As a result, we are unable to construct an LLM based on MTLA to verify its extension to other tasks, such as speech. Instead, we construct decoder-only models and train them from scratch to evaluate MTLA across a range of tasks.
Second, as Transformer- based models have been extensively developed by the community in recent years, there is large amount of related work. It is not feasible for implement and compare all such approaches. In this work, we compare MTLA with the most relevant and representative KV-cache compression methods, including MQA, GQA, and MLA. In addition, we also include comparisons with the typical token compression method SnapK V and the state space model Mamba-2. Further comparisons are only feasible through theoretical discussion,
presented in Section [
This work focuses on long-sequence tasks, particularly speech, due to the naturally long sequence length of speech inputs. We also conduct evaluations text summarisation task. While many addi- tional tasks could be used to further evaluate its effectiveness of MTLA, we leave such investigations for future work. Given the growing dimensionality of modern LLMs and the increasing of long reasoning chains to improve output quality, the MTLA, which compresses the KV cache along both the latent and temporal dimensions, can be particularly valuable.
B Broader impact
Decoder-only architectures based on self-attention have become increasingly popular in recent years, especially in the context of large language models (LLMs). However, due to their high dimensionality and massive number of parameters, LLMS incur expensive inference costs and are heavily dependent on GPUs This problem is further exacerbated by the use of chain-of-thought. which enhances reasoning ability but results in significantly longer output sequences, making inference even more costly. Such inference consumes substantial energy from GPUs. By contrast, our proposed MTLA compresses the Key-Value Cache in both latent and temporal dimensions, greatly improving inference efficiency, which can be of great value to make LLMs more energy- and environmentally sustainable. Therefore, our work has the potential to generate a positive societal impact. We do not know of any negative societal impact.
C Data Set Statistics
The ST task uses the MUST-C 16 v1. English- German (En-De) dataset, with data preprocessing
following the Fairseq example, using 8,000 unigrams as the target language modelling units and fbank features as input. The text summarisation task is conducted on the XSum [3I] dataset, where 30,000 BPE units are used. For the ASR task, the AMI [8] dataset is employed. Due to the challenging nature of the data, fixed WavLM 9 Large features are extracted using the S3PRL [48] toolkit as input. When measuring inference speed, this feature is pre-stored and 100 BPF units are used. For the SLU task, the SLURP [5) dataset is used to evaluate intent classification, with fbank features as input. Following 3. intent classification is performed by jointly predicting the transcription and the intent to achieve better performance. A total of 500 BPE units are used for transcription modelling.
The data set statistics for the datasets used in the experiments are shown in Table 7 The MuST-C [16] v1.0 En-De dataset comprises Baglish--Grmann speech translation data collected from TED Talks. The Augmented Multi-Party Interaction (AMI) Meeting Corpus [8] offers 100 hours of English meeting recordings captured in instrumented rooms, featuring multimodal data such as audio, video, and whiteboard content, with annotations including speech transcriptions and dialogue acts. The Spoken Language Understanding Resource Package (SLURP)) [ dataset is comprehensive English spoken language understanding resource encompassing 18 domains, designed to facilitate tasks like intent
14

---

classification and slot filling, with diverse set of utterances. The XSum [3 dataset consists of BBC news articles from 2010 2017, each paired with a single-sentence abstractive summary. totalling over 226K document-summarry pairs, and is widely used for evaluating summarisation models.
Table 7: Statistics of datasets used in this paper
MuST-C v1.0 En-De
Domain
TED Talk
Train set
train
-Duration
400.0 hours
-German words
3880K
Test sets
dev
Ist-COMMON
-Duration
2.3 hours
4.1 hours
-German words
26K
44K
XSum Dataset
Domain
BBC News Articles
Train set
train
-Documents
204K
-Avg- article length
431 words
-Avg. summary length
23 words
Test sets
dev
Test
-ooumennts
11K
11K
AMI Meeting Corpus
Domain
Meetings train 100.0 hours
Train set
-Duration
-Utterances
108K
Test sets
dev 13K
test
-Utterances
12K
SLURP Dataset
Domain
Human-Computer Interaction (HCI) commands
Train set
train
-Duration
83.7 hours
-Utterances
120K
Test sets
dev
test
Duration
6.9 hours
10.3 hours
UUterrncees
9K
13K
D Hyper-parameter Details and Training
The decoder used for all tasks shares the same configuration: layers, 512 attention dimensions, 2048 feed-forward dimensions, and 8 attention heads. The encoder for all tasks also uses this configuration, except that the number of layers is increased to 12. For MTLA and MLA, r Eq. i set to 256 and aff is set to 32. In MTLA, the linear layers in Eq. 13 and Fq. 16 map the 256-dimensional input to 64-dimensional space. The temporal compression rate s is set to 2 by default unless otherwise specified. Standard RoPE [39 is used in MHA to obtain positional information. Decoupled RoPE is used in MLA, which is also employed in the proposed MTLA along with the temporal compression described in Section
For the ST task, following the Fairseq example, a Transformer encoder (including convolutional layers for 4x downsampling) is used and initialised with ASR task weights. For the text summarisation task, a standard Transformer encoder used. For the ASR task, a Transformer encoder with convolutional layers for 2x downsampling is employed. In addition to the cross-entropy loss, a connectionist temporal classification (CTC) [8] auxiliary loss computed with weight 1. For the SLU task, Conformer [20] encoder is used: beyond the configuration (c.g. 512 attention dimension) used in
15

---

Transformer, its depthwise convolutional layer has kernel size of 31. Before entering the Conformer, a convolutional layer with downsampling applied.
For the ST task, training follows the Fairseq example, using a learning rate of 2e-3, 10,000 warm-up steps, and maximum of 100,000 update steps. Each batch corresponds 320,000 frames of Fbank features, which is approximately 53 minutes of speech. The MHA, MLA, and MTLA models all have 78M parameters. The GQA and MQA models constructed Section 6.3 have 74M parameters. During inference, each batch corresponds to 50,000 frames of Fbank features, and the beam size is set to 50. For the text summarisation task, training uses learning rate of 2e-4, 15,000 warm-up steps, and maximum of 60,000 update steps, Each batch corresponds to 40000 tokens. The MHA, MLA, and MTLA models all have 79M parameters. During inference, each batch corresponds 60,000 tokens, and the beam size is set to 10. For the ASR task, training uses learning rate of 2e-4, 15,000 warm-up steps, and maximum of 10,000 update steps. Each batch corresponds to approximately 16 minutes of speech. The MHA, MLA, and MTLA models all have 67M trainable parameters. During inference, each batch corresponds to 20 minutes of speech, and the beam size is set to 20. For the SLU task, training uses learning rate of 2e-4, 50,000 warm-up steps, and a maximum of 30,000 update steps. Each batch corresponds to 18,000 frames Fbank features. The MHA, MLA, and MTLA models all have 103M parameters. During inference, each batch corresponds to 130,000 frames of Fbank features, and the beam size is set to 10.
Model training was performed on single NVidia RTX 6000 Ada GPU with 48GB of memory. For the ST task, each epoch took about 13 minutes. For the text summarisation, each epoch took about 20 minutes. For the ASR task, each epoch took about 50 minutes. For the SLU task, each epoch took about 15 minutes.
E Results on Long Range Arena
Table 8: Experimental results Long- Long-Range Range Arena benchmark than MTLA are taken from [1]]
41
The published results other
Model
Lis
Text(1)
Retrieval(t)
Image(f)
Pathfinder(1)
Avg(t)
Transformer
36.37
64.27
57.46
42.44
71.40
54.39
Local Attention
15.82
52.98
53.39
41.46
66.63
46.06
Sparse Transformers
17.07
63.58
59.59
44.24
71.71
51.24
Longformer
35.63
62.85
56.89
42.22
69.71
53.46
Linformer
35.70
53.94
52.27
38.56
76.34
51.36
Reformer
37.27
56.10
53.40
38.07
68.50
50.67
Sinkhorn Transformers
33.67
61.20
53.83
41.23
67.45
51.39
Synthesizer
36.99
61.68
54.67
41.61
69.45
52.88
BigBird
36.05
64.02
59.29
40.83
74.87
55.01
Linear Transformer
16.13
65.90
53.09
42.34
75.30
50.55
Performer
18.01
65.40
53.82
42.77
77.05
51.41
Proposed MTLA
40.47
66.99
59.88
48.10
67.55
56.60
This section further evaluates MTLA on the Long-Range Arena (LRA) benchmark [4 The LRA benchmark not included in the main text, it is primarily designed for decoder- only architectures and is therefore more suitable for evaluating encoder self-attention mechanisms. As shown in Table 8 MTLA achieves strong performance on the LRA benchmark compared to other attention mechanisms. These results complement the findings presented in the main text, demonstrating that MTLA consistently performs well across diverse modalities, including text, speech, and vision. The consistent performance across tasks indicates that MTLA effectively captures longerange dependencies while maintaining computational efficiency. It is worth noting that state space models (SSMs) are known to outperform Transformer-baat based attention mechanisms on the LRA benchmark due to their formulation as long convolutions with time-decay dynamics, which are particularly advantageous for tasks with strong positional dependencies. Nevertheless, we include Table because LRA remains challenging and well-established benchmark for attention mechanisms, providing valuable test of MTLA's capability under difficult long-con context conditions.
16

---

Results on Machine Translation
Table 9: Machine translation BLEU (1) results WMT14
6
English-German translation
Model on WMT14 En-De
BLEU
MLA
25.63
Proposed MTLA
25.57
While the focus of this paper is on nong-sequence tasks, text-based translation generally involves much shorter sequences than speech translation. Nevertheless, this section presents MTLA results on the machine translation task using WMT14 English English-German data. As shown in Table MTLA achieves competitive performance compared to MIA, demonstrating that context compression does degrade performance on this task.
G Assets and licenses
The following licenses apply the datasets used in this paper:
CC-BY-NC-ND-4.0:
applies to
MuST-C data.
CC-BY-SA-4.0- httppp/pp..pppoooooo//c/co/cooaa to XSum data.
CC BY 4.0: OTtp:://ppaaa.oo///oooo/////AA10 to AMI data.
CC BY-NC 4.0: [ttpe://ggga..///loooooee//C---- applies to SLURP data.
The following license applies the code and Python package used in this paper:
Apache-2.0: applies to Fairseq httpp////gttttcco//eecebbrrrrrar blob/main/LICENSE)
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions and focus of this paper have been clearly stated. See Abstract and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
This paper has described the proposed method and experimental setup in detail. See Section and Section 5. Further details of the training sets used are given in Appendix C and the detailed experimental setups in Appendix D.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code was submitted as Supplementary Material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https
/ "pipsscc/puullc//uuu /ttp://gguddeeeeeeouuuuumbbbba ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section Appendix C, and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Statistical significance was reported. See Section 6.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
20

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This work conforms woth the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
See Appendix B.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The original papers have been cited and the versions of the used data have been stated clearly. See Section 5. The relevant licenses are listed in Appendix E.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes)
Justification:
The code was submitted as Supplementary Material along with the documenta-
tion.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects. All datasets used are pre-existing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects. All datasets used pre-existing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
23

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This method proposed in this paper can advance the development of decoder- only LLMs; however, the development process does not involve using LLMs in any impor- tant, original, non-standard role. All contributions entirely from the authors.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neet) http://pp...coooofeeeeee/////// for what should or should not be described.
24


Reviews:
[{"review_id": "cKHgpthaRo", "reviewer": "Reviewer_2249", "summary": "This paper introduces Multi-head Temporal Latent Attention (MTLA), a novel attention mechanism that reduces inference-time memory and improves speed by compressing the Key-Value (KV) cache both in the latent and temporal dimensions. To ensure consistency between training and inference behaviors, the paper proposes a stride-aware causal mask.\nThe evaluation on speech translation, speech recognition, speech understanding, and text summarization shows that MTLA can achieve substantial improvements in efficiency with competitive accuracy.", "questions": "See above", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n- The paper is well-written and easy to follow.\n- The proposed method shows strong empirical performance on speech translation, speech recognition, speech understanding, and text summarization tasks.\n- The stride-aware causal mask enables efficient parallel training while preserving inference-time attention behavior\n\nWeakness and Questions:\n- Lack of comparison to other token compression works like [1]. \n- Although the paper mentions FlashAttention in the limitations section, it remains unclear how MTLA compares in real-world end-to-end latency. Integrating MTLA with FlashAttention may not be trivial. It will be great for the paper to have such a kernel to show that the proposed attention can achieve wall-clock time speedup.\n- Although the proposed method has been evaluated in various tasks like speech recognition and summarization. I feel that those tasks are not challenging enough. For example, what is the context length for those tasks? Considering that attention only becomes a major bottleneck only when the context length is long enough. It would be great for the paper to also test the proposed methods on more challenging long-context tasks. \n- Another concern about the evaluation is that it remains unclear whether the proposed attention mechanism is effective in long-context LLM scenarios, which represent a major challenge for long-context inference. It would strengthen the paper to justify or demonstrate its effectiveness in such settings.\n\n[1] Zhang, Peitian, et al. \"Long Context Compression with Activation Beacon.\" The Thirteenth International Conference on Learning Representations.", "was_revised": true, "final_justification": "After several rounds of discussion with authors and also checking other reviewers' comments,  I think my major concern could not be resolved:\n\n- This time-wise attention compression has been widely studied on attention and this work is an extension on MLA -- so the novelty authors claim is it has not been widely studied on MLA, which is a relatively new variant attention. However, I don't see where the challenges on extending all previously studied attention compression /kv compression mechanism here. Therefore, I think method wise, it is fairly incremental.  \n- The evaluation and effectiveness of the methods can't be justified IMO with current evaluation and scale.", "rebuttal": "Thanks for the insightful review: your suggestions will be very helpful for improving the manuscript.\n\nThe concerns raised by the reviewer were mentioned in the Limitations section of the original manuscript, such as the engineering effort required to extend FlashAttention to MTLA and validation on more tasks. We initially considered these as future work because we wanted to keep a clear focus of this paper, specifically on presenting our novel method for temporal KV cache compression and validating it on a set of typical tasks. In fact, we have indeed been actively working on these planned future extensions, such as integrating MTLA with FlashAttention. In light of the reviewer's comments, we have decided to include the results of this new extra work here and will also include it in an appendix of the final version of the paper. We hope that this will address the reviewer's concerns and enable the reviewer to revise the overall score.\n\nBelow, we provide a point-by-point response to the reviewer's comments.\n\n- Weakness 1:\n    > Lack of comparison to other token compression works like [1]. \n    [1] Zhang, Peitian, et al. \"Long Context Compression with Activation Beacon.\" The Thirteenth International Conference on Learning Representations.\n    - Thanks for the suggested reference. As mentioned in the Limitations section, Transformer-based models have been extensively developed by the community over many years, and the number of related techniques precludes an exhaustive comparison. However, the token/context compression work mentioned by the reviewer is indeed relevant and othogonal to our KV cache compression research direction. We will therefore cite and discuss reference [1] in the Related Work section of the final version of the paper.\n\n- Weakness 2: \n    > Although the paper mentions FlashAttention in the limitations section, it remains unclear how MTLA compares in real-world end-to-end latency. Integrating MTLA with FlashAttention may not be trivial. It will be great for the paper to have such a kernel to show that the proposed attention can achieve wall-clock time speedup.\n    - As the reviewer noted, in the Limitations section of the original manuscript we mentioned that extending FlashAttention to MTLA was planned as future work, as it requires substantial engineering effort, while this paper focuses on presenting our novel temporal compression technique. Recently, we have indeed been actively working on extending MTLA, including the integration of FlashAttention. A CUDA kernel based on MTLA has now been implemented. Below, we include some results for MTLA using our extended version of FlashAttention on the same speech translation task presented in Table 1 of the original manuscript.\n        | ST Model  | Quality (BLEU) |Inference Time (s)| Inference GPU Memory (MiB) |\n        | :----------- | :-------: |:-------: | :-------:|\n        |MHA|23.18|281.3|18646|\n        |MHA with FlashAttention|23.16|145.7|9244|\n        |MTLA|23.28|65.6|2835|\n        |MTLA with Extended FlashAttention|23.29|36.5|1259|\n        \n        As the results show, using FlashAttention speeds up computation, but it does not change the experimental conclusions. We will include these results and further detail in the appendix of the final version. We will also open-source our extension to FlashAttention for MTLA.\n        \n- Weakness 3: \n    > Although the proposed method has been evaluated in various tasks like speech recognition and summarization. I feel that those tasks are not challenging enough. For example, what is the context length for those tasks? Considering that attention only becomes a major bottleneck only when the context length is long enough. It would be great for the paper to also test the proposed methods on more challenging long-context tasks.\n    - This paper has already been evaluated on various long-sequence tasks, since the sequences used for speech tasks and document texts are typically long sequences. For example, in the MuST-C speech translation dataset we used, the maximum sequence length exceeds 5K, with an average length of more than 0.6K; in the XSum dataset, the maximum can be beyond 13K, with an average of more than 0.5K. Hence these tasks would already normally be viewed as long-sequence tasks with suitable training data. In these scenarios, effective handling of the context length becomes the primary bottleneck. This is also why using our proposed MTLA leads to large improvements in inference speed and reductions in GPU memory usage compared to other attention mechanisms. On the other hand, extremely long-sequence tasks often rely on LLMs to exhibit emergent capabilities, since the corresponding training data with extremely long sequences is scarce. As we mentioned in the Limitations section, due to limited resources, we are unable to afford to pretrain LLMs with MTLA, which is why we chose these well-established long-sequence tasks in the original mauscript.\n    - However, in order to further address the reviewer’s concerns, we have additionally conducted evaluations on the Long-Range Arena (LRA) benchmark (Tay et al., 2021). LRA is a widely used benchmark for testing efficient Transformer models in long-context scenarios. We did not include this benchmark in the original version of the paper because LRA is not primarily designed for decoder-only architectures and may be more suitable for encoder self-attention.\n    - Below, we present the results of MTLA on the LRA benchmark (with other results taken from the Tay et al., 2021 paper):\n        | Model| Listops（`↑`）| Text（`↑`） |Retrieval（`↑`）| Image（`↑`） |Pathfinder（`↑`） | Avg（`↑`）|\n        | :----------- | :-------: | :-------: |:-------: |:-------: |:-------: | :-------:|\n        |Transformer| 36.37| 64.27| 57.46| 42.44 |71.40|54.39|\n        |Local Attention| 15.82| 52.98| 53.39| 41.46| 66.63|  46.06|\n        |Sparse Transformers| 17.07 |63.58| 59.59| 44.24| 71.71 | 51.24|\n       | Longformer |35.63| 62.85 |56.89 |42.22| 69.71| 53.46|\n      | Linformer| 35.70 |53.94| 52.27| 38.56| 76.34| 51.36|\n       |Reformer| 37.27 |56.10| 53.40| 38.07| 68.50 |50.67|\n       |Sinkhorn Transformers| 33.67 |61.20 |53.83 |41.23 |67.45 | 51.39|\n       |Synthesizer |36.99| 61.68 |54.67| 41.61 |69.45| 52.88|\n       |BigBird| 36.05 |64.02 |59.29| 40.83| 74.87| 55.01|\n       |Linear Transformer| 16.13| 65.90| 53.09| 42.34| 75.30| 50.55|\n       |Performer |18.01| 65.40| 53.82| 42.77| 77.05|51.41|\n       |**MTLA**|40.47|66.99|59.88|48.10|67.55|**56.60**|\n    \n      \"ListOps\" is a 10-way classification task with a sequence length of 2K. \"Text\" is a binary classification task with sequences up to 2,048 tokens. \"Retrieval\" is also a binary classification task, requiring processing of 8K tokens during evaluation. \"Image\" is a 10-way classification task with a sequence length of 1,024. \"Pathfinder\" is a binary image classification task with sequences of length 1,024. For more detail about LRA refer to Tay at al. (2021).\n\n    - The results above show that MTLA achieves strong performance on the LRA benchmark compared to other attention mechanisms. Combined with the experiments in the original paper, MTLA consistently demonstrates strong performance across a range of tasks involving text, speech, and vision.\n      \n    Tay et al. 2021. \"Long Range Arena: A benchmark for efficient transformers.\" In Proc. ICLR.\n\n- Weakness 4: \n    > Another concern about the evaluation is that it remains unclear whether the proposed attention mechanism is effective in long-context LLM scenarios, which represent a major challenge for long-context inference. It would strengthen the paper to justify or demonstrate its effectiveness in such settings.\n    - We agree that it would further strengthen the paper if we could afford to pretrain an LLM based on MTLA. However, as mentioned in the Limitations section, due to limited computational resources, we are unable to afford to do LLM pretraining, which is why we did not explicitly link MTLA to LLMs in the main text of the original manuscript. This is a challenge we will need to address in future, but we believe the experimental validation we currently provide should be acceptable within the scope of an academic paper. We do plan to actively promote the widespread adoption of MTLA and KV cache temporal compression, so scaling to LLMs is definitely a direction we will strive to investigate in future.\n    \n    We hope that our clarifications have effectively addressed your concerns. We noticed that your comments did not raise technical concerns about our proposed method, but rather provided suggestions regarding experimental validation. In the responses above, we have added a substantial number of additional experimental results, which follow your suggestions and provide a broader evaluation of MTLA, demonstrating its effectiveness. We hope you will kindly take into account the new results presented here in your final review scores."}, {"review_id": "Atgxliz7mQ", "reviewer": "Reviewer_xgRJ", "summary": "Current LLM architectures face severe memory constraints during inference. To address this challenge, the authors introduce a novel technique that compresses the KV-cache along the temporal dimension using a learnable approach. They report over 3× speedups and substantial VRAM savings during inference compared to vanilla multi-head attention.", "questions": "1. Can this approach be combined with other head reduction techniques such as GQA?\n2. Have you experimented with FP16/BF16 precision for training and inference?\n3. While resource constraints prevent testing on large-scale LLMs, what evidence suggests this approach will scale effectively to larger models? Could you provide scaling law experiments, even if limited to smaller model sizes?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- The authors present Multi-head Temporal Latent Attention (MTLA), a novel self-attention variant that pioneers learnable compression of the KV-cache along the temporal axis while preserving the key-value compression from MLA. This approach opens a promising new research direction for improving inference-time efficiency in Transformer-based models.\n- The research addresses a well-motivated problem, as current LLMs face significant memory bottlenecks due to KV-cache growth. The proposed approach introduces clever technical solutions with sound theoretical explanations.\n- The empirical results are compelling: the method demonstrates consistent performance across diverse tasks (speech translation, ASR, text summarization, and SLU) while matching the accuracy of previous methods. This suggests that the compression effectively removes redundancy from the KV-cache without sacrificing model quality.\n# Weaknesses\n- **Novelty claim**: While the authors claim this is the first work to explore KV-cache compression along the temporal axis, [Nawrot et al. (2024)](https://arxiv.org/abs/2403.09636) have already investigated temporal compression of the KV-cache. Although their approach focuses on retrofitting pre-trained LLMs rather than making fundamental architectural changes, this prior work should be acknowledged.\n- **Limited comparison against baselines**: The paper does not empirically compare against established KV-cache pruning methods such as LazyLLM and SnapKV, despite mentioning them in the related work.\n- **Limited comparison to other orthogonal approaches**: the KV-cache can be compressed with other methods (e.g. quantization). The authors do not compare the speed-ups obtained by MTLA to other orthogonal methods, or a combination of both.\n- **GPU memory usage reporting**: The analysis of GPU memory usage lacks detail regarding measurement methodology and the breakdown between model weights, activations, and KV-cache consumption. Moreover, the average GPU memory usage is reported, whereas a more informative measure would be the peak memory usage.\n- **Benchmarks might not reflect real deployment metrics**: The reported memory usage and inference time metrics heavily depend on implementation details. Since the authors acknowledge not using optimizations like FlashAttention or compiled kernels, the results may not reflect realistic deployment scenarios.\n- **Lack of confidence intervals**: The experimental results lack error bars or confidence intervals, which would strengthen the statistical validity of the comparisons given the stochastic nature of neural network training.\n- **Limited scale**: The core motivation addresses memory bottlenecks that primarily affect large models processing very long sequences. However, the evaluation uses small models with approximately 0.1B parameters, which may not adequately demonstrate the method's benefits at scale.\n- **Lack of evaluation on common tasks**: The evaluation lacks standard LLM benchmarks (language modeling, code generation, reasoning tasks) and crucially omits long-context evaluations (100k+ tokens) where the method's advantages should be most pronounced.", "was_revised": true, "final_justification": "This work presents MTLA, an extension to Multi-head Latent Attention where compression is also applied on the sequence length dimension, effectively reducing the size of the KV-Cache and improving inference speed.\n\nThe results look promising, and the authors have addressed all the weaknesses. \nWe reckon that the main weakness of the paper, the small scale of the experiments, cannot be easily resolved due to the resource limitations of the authors. Nevertheless, the authors successfully test their method on a variety of tasks, and are planning on releasing the implementation to the public. We believe that the strengths outweigh the weaknesses, and that MTLA opens an exciting area of future research on efficient attention architectures.", "rebuttal": "Thanks for your very detailed review and insightful comments. It's encouraging when the motivation and strengths of our work are clearly understood. We provide a point-by-point response to your review below. \n\n- Weakness 1:\n    > While the authors claim this is the first work to explore KV-cache compression along the temporal axis, Nawrot et al. (2024) have already investigated temporal compression of the KV-cache. Although their approach focuses on retrofitting pre-trained LLMs rather than making fundamental architectural changes, this prior work should be acknowledged.\n    - Thanks for the suggested reference. In the final version, we will cite and discuss Nawrot et al. (2024).\n\n- Weakness 2:\n    > The paper does not empirically compare against established KV-cache pruning methods such as LazyLLM and SnapKV, despite mentioning them in the related work.  \n    > the KV-cache can be compressed with other methods (e.g. quantization). The authors do not compare the speed-ups obtained by MTLA to other orthogonal methods, or a combination of both.\n    - Thanks for the suggesions. As mentioned in the Limitations section, Transformer-based models have been extensively developed by the community over many years, and the number of related techniques precludes an exhaustive comparison. Therefore, we chose to implement and compare only the most relevant and representative KV cache compression method. On the other hand, techniques such as KV cache pruning and quantisation are orthogonal to our approach. Including all of these methods would make our experimental setup very complex and reduce clarity. Moreover, since our method is based on MLA, some of these techniques require further adaptation before they can be applied to MLA and meaningfully compared with our MTLA. We do however investigate one cache pruning method below.\n    - In response to the reviewer's suggestion, we selected a representative KV cache pruning method, SnapKV, adapted it for use with MLA, and compared it with our MTLA. The results on the same speech translation task presented in Table 1 of the original manuscript are shown below.\n        | ST Model  | Quality (BLEU) |Inference Time (s)| Inference GPU Memory (MiB) |\n        | :----------- | :-------: |:-------: | :-------:|\n        |MLA|22.97|97.0|5065|\n        |MLA with SnapKV|21.76|80.8|4222|\n        |MTLA|23.28|65.6|2835|\n        \n        The results show that SnapKV can effectively improve inference efficiency, but it comes with some performance degradation. SnapKV may still be more suitable for LLM scenarios, where there is more redundant information. In contrast, MTLA, as a fundamental attention mechanism, may have broader application potential.\n\n- Weakness 3:\n    > The analysis of GPU memory usage lacks detail regarding measurement methodology and the breakdown between model weights, activations, and KV-cache consumption. Moreover, the average GPU memory usage is reported, whereas a more informative measure would be the peak memory usage.\n    - Thanks for the suggestion. We will include more detailed information in the final version of the paper.\n\n- Weakness 4:\n    > The reported memory usage and inference time metrics heavily depend on implementation details. Since the authors acknowledge not using optimizations like FlashAttention or compiled kernels, the results may not reflect realistic deployment scenarios.\n    - Thanks for pointing this out. In the Limitations section of the original manuscript we mentioned that extending FlashAttention to MTLA was planned as future work, as it requires substantial engineering effort, while this paper focuses on presenting our novel temporal compression technique. Recently, we have indeed been actively working on extending MTLA, including the integration of FlashAttention. A CUDA kernel based on MTLA has now been implemented. Below, we include some results for MTLA using our extended version of FlashAttention on the same speech translation task presented in Table 1 of the original manuscript.\n        | ST Model  | Quality (BLEU) |Inference Time (s)| Inference GPU Memory (MiB) |\n        | :----------- | :-------: |:-------: | :-------:|\n        |MHA|23.18|281.3|18646|\n        |MHA with FlashAttention|23.16|145.7|9244|\n        |MTLA|23.28|65.6|2835|\n        |MTLA with Extended FlashAttention|23.29|36.5|1259|\n        \n        As the results show, using FlashAttention speeds up computation, but it does not change the experimental conclusions. We will include these results and more detail in the appendix of the final version, and we will also open-source our FlashAttention extension for MTLA.\n        \n- Weakness 5:\n    > The experimental results lack error bars or confidence intervals, which would strengthen the statistical validity of the comparisons given the stochastic nature of neural network training.\n    - Due to limited computational resources, conducting multiple training runs with different random seeds would greatly increase the training cost. Therefore, instead of varying the random seed, we chose to vary the task types and validated our method across a range of different tasks, consistently reaching the same conclusion. Additionally, we conducted statistical tests (as shown in Section 6.3) to confirm that the improvements brought by our method are statistically significant. We appreciate the reviewer’s suggestion and will mention this point in the Limitations section of the final version.\n\n- Weakness 6:\n    > The core motivation addresses memory bottlenecks that primarily affect large models processing very long sequences. However, the evaluation uses small models with approximately 0.1B parameters, which may not adequately demonstrate the method's benefits at scale.  \n    > The evaluation lacks standard LLM benchmarks (language modeling, code generation, reasoning tasks) and crucially omits long-context evaluations (100k+ tokens) where the method's advantages should be most pronounced.\n    - We agree that it would further strengthen the paper if we could afford to pretrain an LLM based on MTLA. However, as we mentioned in the Limitations section, due to limited computational resources, we are unable to afford LLM pretraining, which is why we did not explicitly link MTLA to LLMs in the main text of the original manuscript. This is a challenge we want to address in future, but we believe the experimental validation we currently provide is acceptable within the scope of an academic paper. We do plan to actively promote the widespread adoption of MTLA and KV cache temporal compression, so investigating MTLA with LLMs is a clear future direction.\n\n- Question 1:\n    > Can this approach be combined with other head reduction techniques such as GQA?\n    - Yes, the KV cache time compression technique we propose can certainly be combined with other attention mechanisms, including GQA.\n\n- Question 2:\n    > Have you experimented with FP16/BF16 precision for training and inference?\n    - We have tested BF16 training and found that MTLA works well with it. However, in order to ensure a fair comparison across all constructed models, we ultimately did not use BF16 in the experiments reported in the original manuscript. We will open-source all the code and ensure support for both FP16 and BF16. As mentioned above, we have also extended FlashAttention to MTLA, and it supports FP16/BF16.\n\n- Question 3:\n    > While resource constraints prevent testing on large-scale LLMs, what evidence suggests this approach will scale effectively to larger models? Could you provide scaling law experiments, even if limited to smaller model sizes?\n    - In fact, our proposed MTLA, like standard attention, still has quadratic complexity, and is therefore fundamentally similar. Since attention has been widely validated for its scaling capabilities, MTLA can naturally be expected to exhibit the same properties. On the other hand, the set of tasks used in the original manuscript is not suitable for conducting scaling law experiments, as models with a very large number of parameters tend to overfit on them. Large-scale LM pretraining tasks are more appropriate for demonstrating scaling effects, since larger models can better exhibit their advantages in such settings with large amount of data. At present, we are indeed unable to afford such large LLM experiments due to limited computational resources."}, {"review_id": "7izlC33uRz", "reviewer": "Reviewer_yMqX", "summary": "The paper proposes a method to compress the self-attention KV-cache along the temporal dimension, extending the recent Multi-head Latent Attention (MLA) technique to further reduce compute and memory overhead.\n\nSince the proposed approach modifies the KV cache temporal sequence during attention, special care must be taken during training to ensure that it is consistent with the partial decimation in time that must occur during token-by-token sampling during inference.\n\nExperiments on small scale speech and text tasks demonstrate significant memory/speed improvements without hurting performance compared to a vanilla attention baseline and other commonly used attention optimizations.", "questions": "See specific suggestions for improvements listed in Weakness above.\n\nMinor issues/questions not raised above:\n\n - I found Fig 1 to be a bit confusing.  Shouldn't the \"Multi-head Queries\" also be derived from the \"Attention Input\"?  Also, it would be even more clear if the different components were explicitly connected to the notation introduced in Sec 3.\n\n- Sec 4.3, line 184:  Did you mean \"Eq. 9\" instead of \"Eq. 8\"?\n\n- Sec 4.3, line 201: $9d_hl/(2s)$ would be $2.25 d_hl$, not $2.5 d_hl$ if $s=2$.\n\n- Table 5 repeats most of Table 1.  These can probably be consolidated into a single table.\n\n- The final two sentences of the first paragraph of Sec. 5.2 (lines 221-224) are near exact repeats of the preceding two sentences.\n\n- End of Sec 6.3: Why only comment on the significance of one result?  Are the performance improvements of \"Proposed MTLA w/$s=4$\" over the other baselines not significant?", "limitations": "Yes.  The authors explicitly acknowledge the limitations of the small scale experiments due to limited compute resources available to them.  This is totally reasonable.  However, as described above, it feels like an oversight to focus exclusively on long sequence tasks in the experimental evaluation with no other justification.  Some experimental comparisons on other text tasks (at small scale) would give a more rounded perspective on how broadly applicable MLTA is.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- Straightforward, well motivated approach to reduce self-attention compute/memory requirements, extending recent work on latent attention.\n- Experimental validation of decoder-only transformer across several different tasks.\n- Good results - large inference speedups over latent attention baseline.\n\nWeaknesses\n\n- The main weakness is that the experimental evaluation is on on very small scale models of around 100M parameters, more than an order of magnitude smaller than the smallest typical LLM size (~2-3B parameters).  The author's justify this choice due to limited access to compute, which is totally understandable, but this might limit interest in the work.\n\n- Curious choice of evaluation tasks, focusing primarily on speech tasks.  The proposed technique seems quite generally applicable to any sequence modeling task.  Is there a reason for not including incorporating other NLP tasks, e.g., machine translation (which is supported in fairseq) in the experiments?  It is reasonable to assume that the proposed technique better suited to speech tasks because sequence lengths are longer.  And perhaps speech signals are more naturally compressible in time than tokenized text, making them better suited to the proposed temporal compression technique?  But it would nevertheless be valuable to include some results on some other NLP tasks, even if sequences are shorter, to validate that compressing context does not hurt performance on such tasks.\n\n   - Similarly $s > 2$ is only evaluated on the speech translation task.  It would be valuable to see how MTLA performs on other tasks with more compression., especially XSum since it is the only text-only task considered in the paper.  \n\n- Some design choices not thoroughly justified/evaluated, especially the construction of the hypernetwork that compute the merging weights $w_i$ in Eq. 13.  Are these details important?  Does performance suffer if $w_i = 1/s$ (simple averaging) or some other constant?  If nothing else, some discussion motivating this choice  seems warranted.  Even better would be a quick ablation experiment comparing the proposed technique to a simple baseline.", "was_revised": true, "final_justification": "The author's additional experimental validation help to strengthen the minor criticisms of the work.  But the main weakness remains: I expect that most readers will wonder (as did the reviewers) how well the proposed method applies at LLM scale.\n\nIn my opinion the simplicity and demonstrated effectiveness of the method at small scale are intriguing enough to overcome this weakness.  But I also understand if others would disagree, hence my borderline recommendation.", "rebuttal": "Thanks for your very detailed review and insightful comments. It's encouraging when the motivation and strengths of our work are clearly understood. We provide a point-by-point response to your review below. \n\n- Weakness 1:\n    > The main weakness is that the experimental evaluation is on on very small scale models of around 100M parameters, more than an order of magnitude smaller than the smallest typical LLM size (~2-3B parameters). The author's justify this choice due to limited access to compute, which is totally understandable, but this might limit interest in the work.\n     - Thanks for pointing this out. We agree that it would be interesting if we can afford to pretrain an LLM based on MTLA. However, as we mentioned in the Limitations section, due to limited computational resources, we are unable to afford LLM pretraining, which is why we did not explicitly link MTLA to LLMs in the main text of the original manuscript. This is a challenge we will need to address in the future, but we believe the experimental validation we currently provide should be acceptable within the scope of an academic paper. \nWe do plan to actively promote the widespread adoption of MTLA and KV cache temporal compression, so investigating MTLA with LLMs is a clear future direction.\n\n\n\n\n- Weakness 2:\n    > Curious choice of evaluation tasks, focusing primarily on speech tasks. The proposed technique seems quite generally applicable to any sequence modeling task. Is there a reason for not including incorporating other NLP tasks, e.g., machine translation (which is supported in fairseq) in the experiments? It is reasonable to assume that the proposed technique better suited to speech tasks because sequence lengths are longer. And perhaps speech signals are more naturally compressible in time than tokenized text, making them better suited to the proposed temporal compression technique? But it would nevertheless be valuable to include some results on some other NLP tasks, even if sequences are shorter, to validate that compressing context does not hurt performance on such tasks.\n    - This paper has been validated on various speech tasks and text summarisation tasks. As explained in Section 6 of the original manuscript, the focus of this work is on long-sequence tasks, and both speech and document inputs naturally involve long sequences. In contrast, text-based translation tasks generally involve much shorter sequences than speech, and since they are all translation tasks, including them would introduce some unnecessary redundancy. Therefore, we did not include them in the original manuscript. Following the reviewer’s suggestion, we conducted additional experiments on the WMT14 English-German translation task using Fairseq. These results are shown below.\n        | Model on WMT14 En-De | BLEU |\n        | :-----------  | -------:|\n        |MLA|25.63|\n        |MTLA|25.57|\n        \n        The results show that MTLA achieves competitive performance with MLA in machine translation. Furthermore, to further address the reviewer’s concerns, we have now conducted additional validation on more tasks. We additionally conducted evaluations on the Long-Range Arena (LRA) benchmark (Tay et al., 2021).LRA is a widely used benchmark for testing efficient Transformer models in long-context scenarios. We did not include this benchmark in the original version of the paper because LRA is not primarily designed for decoder-only architectures and may be more suitable for encoder self-attention.\n    - Below, we present the results of MTLA on the LRA benchmark (with other results taken from the Tay et al., 2021 paper): \n        | Model| Listops（`↑`）| Text（`↑`） |Retrieval（`↑`）| Image（`↑`） |Pathfinder（`↑`） | Avg（`↑`）|\n        | :----------- | :-------: | :-------: |:-------: |:-------: |:-------: | :-------:|\n        |Transformer| 36.37| 64.27| 57.46| 42.44 |71.40|54.39|\n        |Local Attention| 15.82| 52.98| 53.39| 41.46| 66.63|  46.06|\n        |Sparse Transformers| 17.07 |63.58| 59.59| 44.24| 71.71 | 51.24|\n       | Longformer |35.63| 62.85 |56.89 |42.22| 69.71| 53.46|\n      | Linformer| 35.70 |53.94| 52.27| 38.56| 76.34| 51.36|\n       |Reformer| 37.27 |56.10| 53.40| 38.07| 68.50 |50.67|\n       |Sinkhorn Transformers| 33.67 |61.20 |53.83 |41.23 |67.45 | 51.39|\n       |Synthesizer |36.99| 61.68 |54.67| 41.61 |69.45| 52.88|\n       |BigBird| 36.05 |64.02 |59.29| 40.83| 74.87| 55.01|\n       |Linear Transformer| 16.13| 65.90| 53.09| 42.34| 75.30| 50.55|\n       |Performer |18.01| 65.40| 53.82| 42.77| 77.05|51.41|\n       |**MTLA**|40.47|66.99|59.88|48.1|67.55|**56.60**|\n    \n\n      \"ListOps\" is a 10-way classification task with a sequence length of 2K. \"Text\" is a binary classification task with sequences up to 2,048 tokens. \"Retrieval\" is also a binary classification task, requiring processing of 8K tokens during evaluation. \"Image\" is a 10-way classification task with a sequence length of 1,024. \"Pathfinder\" is a binary image classification task with sequences of length 1,024. For more detail about LRA refer to Tay at al. (2021).\n      \n    - The results above show that MTLA achieves competitive performance on the LRA benchmark compared to other attention mechanisms. Combined with the experiments in the original paper, MTLA has demonstrated consistently strong performance across a range of tasks involving text, speech, and vision.\n      \n    Tay et al. 2021. \"Long range arena: A benchmark for efficient transformers.\" In Proc. ICLR.\n\n- Weakness 3:\n    > Similarly s>2 is only evaluated on the speech translation task. It would be valuable to see how MTLA performs on other tasks with more compression., especially XSum since it is the only text-only task considered in the paper.\n    - In the original manuscript, we set s=2 as the default value for MTLA, as it makes the KV cache size comparable to MQA. Following the reviewer’s suggestion, we include below the results on XSum with s > 2:\n        | Model on XSum | R1 | R2|RL|\n        | :-----------  | :-------:| :-------:| :-------:|\n        |MHA|28.83|9.67|23.33|\n        |MTLA|29.14|9.79|23.60|\n        |MTLA with s=3|28.80|9.55|23.33|\n        |MTLA with s=4|28.49|9.49|23.10|\n    \n    The results above show that with a higher compression rate, i.e., a larger value of \"s\", MTLA can still achieve competitive performance. If the target task involves longer and more redundant information, \"s\" could theoretically be set to a larger value. However, in this paper, we prefer to recommend s = 2 as the default setting.\n\n- Weakness 4:\n    > Some design choices not thoroughly justified/evaluated, especially the construction of the hypernetwork that compute the merging weights w_i\n in Eq. 13. Are these details important? Does performance suffer if w_i = 1/s (simple averaging) or some other constant? If nothing else, some discussion motivating this choice seems warranted. Even better would be a quick ablation experiment comparing the proposed technique to a simple baseline.\n     - Using a hypernetwork to generate the weights w_i allows for dynamically merging temporally adjacent KV cache vectors. Following the reviewer’s suggestion, we conducted an additional ablation study comparing our approach to the suggested baseline of w_i = 1/s. The results are shown below:\n         | ST Model  | Quality (BLEU) |\n        | :----------- |  :-------:|\n        |MTLA with w_i = 1/s|22.90|\n        |MTLA|23.28|\n        \n        The results show that using a hypernetwork to dynamically generate the weights yields better performance, which is also intuitively reasonable.\n\n- Question 1:\n    > I found Fig 1 to be a bit confusing. Shouldn't the \"Multi-head Queries\" also be derived from the \"Attention Input\"? Also, it would be even more clear if the different components were explicitly connected to the notation introduced in Sec 3.\n    - We appreciate such detailed suggestions. The multi-head query indeed originates from the attention input. We could add additional arrows from the attention input to the multi-head query to make this clearer, although doing so would slightly increase the visual complexity of the figure. Thank you for the valuable feedback. We will make some adjustments in the final version to improve clarity.\n\n- Question 2:\n    > Sec 4.3, line 184: Did you mean \"Eq. 9\" instead of \"Eq. 8\"?\n    > Sec 4.3, line 201: 9d_hl/(2s) would be 2.25 d_hl, not 2.5 d_hl if s=2\n    - We appreciate these detailed comments and will address these typos in the final version of the paper.\n\n- Question 3:\n    > Table 5 repeats most of Table 1. These can probably be consolidated into a single table.\n    - The reason we did this is to make reading easier, so that readers do not need to flip back and forth between pages for comparison. We will consider your suggestion for the final version of the paper.\n\n- Question 4:\n    > The final two sentences of the first paragraph of Sec. 5.2 (lines 221-224) are near exact repeats of the preceding two sentences.\n    - Thanks for pointing this out. We will make sure to fix this issue in the final version of the paper.\n\n- Question 5:\n    > End of Sec 6.3: Why only comment on the significance of one result? Are the performance improvements of \"Proposed MTLA w/ s=4\" over the other baselines not significant?\n    - This is because in that paragraph (lines 298–302), we focus on the comparison between MTLA and MQA, as MTLA is comparable to MQA in terms of KV cache size when s=2. The reason we conducted statistical testing was to show that even with s=4, MTLA still achieves better translation quality than MQA.\n\nWe hope that our clarifications have effectively addressed your concerns. We hope you will kindly consider the value and contribution of our work, along with the additional information provided here, in your final review scores."}, {"review_id": "WvR9HGKgja", "reviewer": "Reviewer_6bRo", "summary": "This paper optimizes KV cache for attention-based inference by introducing Multi-head Temporal Latent Attention, which compresses KV cache along the temporal dimension via dynamic merging of adjacent tokens and adjusts training-inference methods to reduce overhead.", "questions": "1.\tWhat is the practical or theoretical significance of merging latent embeddings of tokens temporally?\n2.\tDoes the proposed method alter the model training process, and how do training time and parameter space perform?\n3.\tCould you provide parameter implementations under different s values to further clarify the real effects of temporal merging and offer deeper insights?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.\tThe paper presents a clear and well-structured introduction to the method, making it easy to follow the design rationale and model workflow.\n2.\tIt provides a logical review of related work, clearly highlighting the distinctions between the proposed approach and existing methods.\n3.\tExperiments on speech tasks demonstrate significant inference speedups while maintaining accuracy, showcasing practical efficiency gains.\nWeaknesses:\n1.\tAlthough temporal merging shows effectiveness in experiments, the paper lacks in-depth analysis of its principles and validity, requiring more theoretical insights.\n2.\tThe tested scenarios focus on long-sequence speech tasks, and the generalizability of the approach across broader applications remains unvalidated.\n3.\tThe compression ratio s is a hyperparameter that requires manual tuning, but the paper does not offer guidance on optimal setting strategies or insights into its impact.", "was_revised": true, "final_justification": "I think this work presents a clear idea and validates it experimentally, but the analysis lacks depth, so I'm inclined to keep the rating unchanged.", "rebuttal": "We appreciate the encouraging comments in the summary and strengths! Below, we provide a point-by-point response to the reviewer's comments.\n\n- Weakness 1:\n    > 1. Although temporal merging shows effectiveness in experiments, the paper lacks in-depth analysis of its principles and validity, requiring more theoretical insights. \n    - In the original manuscript, specifically in lines 39 to 42 of the Introduction, we provided our analysis and insights. In particular, previous works such as MLA focus on compressing the feature dimension of the KV cache, and approaches like GQA or MQA reduce the number of attention heads. While these approaches have demonstrated effectiveness, they may face inherent constraints. For example, the number of attention heads cannot be reduced below one. In contrast, as modern models produce increasingly long outputs, the length of the KV cache also becomes significantly larger. This makes compression along the temporal dimension a promising direction. This idea is also theoretically grounded, considering that recurrent neural networks can be seen as a special case of attention where the KV cache always has a length of one. Therefore, our proposed temporal compression technique is both reasonable and well aligned with emerging practical needs. We also revisited the potential of this direction in the experimental section, specifically in Section 6.3, lines 298 to 302.\n\n- Weakness 2:\n    > 2. The tested scenarios focus on long-sequence speech tasks, and the generalizability of the approach across broader applications remains unvalidated.\n    - In the original manuscript, we conducted a series of experiments on speech tasks, as well as a text summarisation task, because both speech and documents involve long-sequence scenarios. \n    - To further address the reviewer’s concerns, we have now conducted additional validation on more tasks. We additionally conducted evaluations on the Long-Range Arena (LRA) benchmark (Tay et al., 2021).LRA is a widely used benchmark for testing efficient Transformer models in long-context scenarios. We did not include this benchmark in the original version of the paper because LRA is not primarily designed for decoder-only architectures and may be more suitable for encoder self-attention. Below, we present the results of MTLA on the LRA benchmark (with other results taken from the Tay et al., 2021 paper):\n        | Model| Listops（`↑`）| Text（`↑`） |Retrieval（`↑`）| Image（`↑`） |Pathfinder（`↑`） | Avg（`↑`）|\n        | :----------- | :-------: | :-------: |:-------: |:-------: |:-------: | -------:|\n        |Transformer| 36.37| 64.27| 57.46| 42.44 |71.40|54.39|\n        |Local Attention| 15.82| 52.98| 53.39| 41.46| 66.63|  46.06|\n        |Sparse Transformers| 17.07 |63.58| 59.59| 44.24| 71.71 | 51.24|\n       | Longformer |35.63| 62.85 |56.89 |42.22| 69.71| 53.46|\n      | Linformer| 35.70 |53.94| 52.27| 38.56| 76.34| 51.36|\n       |Reformer| 37.27 |56.10| 53.40| 38.07| 68.50 |50.67|\n       |Sinkhorn Transformers| 33.67 |61.20 |53.83 |41.23 |67.45 | 51.39|\n       |Synthesizer |36.99| 61.68 |54.67| 41.61 |69.45| 52.88|\n       |BigBird| 36.05 |64.02 |59.29| 40.83| 74.87| 55.01|\n       |Linear Transformer| 16.13| 65.90| 53.09| 42.34| 75.30| 50.55|\n       |Performer |18.01| 65.40| 53.82| 42.77| 77.05|51.41|\n       |**MTLA**|40.47|66.99|59.88|48.1|67.55|**56.60**|\n    \n      \"ListOps\" is a 10-way classification task with a sequence length of 2K. \"Text\" is a binary classification task with sequences up to 2,048 tokens. \"Retrieval\" is also a binary classification task, requiring processing of 8K tokens during evaluation. \"Image\" is a 10-way classification task with a sequence length of 1,024. \"Pathfinder\" is a binary image classification task with sequences of length 1,024. For more detail about LRA refer to Tay at al. (2021).\n      \n    - The results above show that MTLA achieves competitive performance on the LRA benchmark compared to other attention mechanisms. Combined with the experiments in the original paper, MTLA has demonstrated consistently strong performance across a range of tasks involving text, speech, and vision.\n      \n    Tay et al. 2021. \"Long range arena: A benchmark for efficient transformers.\" In Proc. ICLR.\n        \n- Weakness 3:\n    > 3. The compression ratio s is a hyperparameter that requires manual tuning, but the paper does not offer guidance on optimal setting strategies or insights into its impact.\n    - This paper has already recommended setting the default value of compression ratio “s” to 2, since in this case the KV cache size is comparable to that of MQA. We have mentioned this point in several places in the original manuscript (lines 200, 227, 259, 290, and 293). Increasing the value of s results in a higher degree of KV Cache compression, but it may lead to some decrease in model accuracy, as shown in Table 5 of the original manuscript. There is no single optimal value of \"s\" that applies to all situations. The best choice depends on the specific trade-off between performance and efficiency required by the application. For this reason, we recommend setting the default value to 2.\n\n- Question 1:\n    > What is the practical or theoretical significance of merging latent embeddings of tokens temporally?\n    - Our work focuses on compressing the key/value (KV) cache of the Transformer self-attention mechanism along the temporal dimension during inference. In autoregressive generation with Transformers, the KV cache grows linearly with the sequence length and becomes a bottleneck for inference. Each decoding step requires loading the cached attention keys and values to avoid re-encoding the history. This repeated memory access has become a limiting factor, affecting inference speed and restricting both decoding batch size and sequence length. Therefore, compressing the KV cache can effectively reduce this memory bandwidth overhead, which is crucial for efficient deployment. As we discussed in our response to Weakness 1, since modern models generate increasingly longer sequences, compressing the KV cache along the temporal dimension holds significant potential. This direction of research can substantially improve Transformer inference efficiency, and our experimental results support this claim.\n\n- Question 2:\n    > Does the proposed method alter the model training process, and how do training time and parameter space perform?\n    - As described in Section 4.2 of the original manuscript, MTLA is fully parallelisable during training and shares the same computational and memory complexity as standard attention. Therefore, our proposed method does not alter the model training process.\n\n- Question 3:\n    > Could you provide parameter implementations under different s values to further clarify the real effects of temporal merging and offer deeper insights?\n    - As shown in Figures 1 and 2 of the original manuscript, \"s\" is used to control the degree of temporal compression of the KV Cache. Specifically, if s = 2, every two adjacent KV Cache vectors are merged into one vector, and the total length of the KV Cache becomes half of the original; if s = 3, every three adjacent KV Cache vectors are merged into one vector, and the total length of the KV Cache becomes one-third of the original; and so on. Therefore, by adjusting \"s\", the compression level of the KV Cache can be well controlled.\n\nWe hope that our clarifications have effectively addressed your concerns. We hope you will kindly consider the value and contribution of our work in your final evaluation, taking into account our responses to your review."}, {"review_id": "gtKR6PXSsF", "reviewer": "Reviewer_hHHE", "summary": "This paper proposes MTLA, a novel method that compresses the KV cache along the temporal dimension. This method is orthogonal to other compression methods along the hidden dimension. Experiments across multiple tasks show that MTLA greatly accelerates inference and reduces GPU memory usage at inference without sacrificing accuracy.", "questions": "* If my understanding to the algorithm is correct, Figure 2 (c) is incorrect. Concretely, the first element of each row (instead of each column) should be q1, q2, q3 and q4, respectively.", "limitations": "See Weaknesses.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* This is a very novel method that successfully compress KV cache along the temporal dimension.\n* Experiments also show promising results.\n* The paper is well written and easy to follow.\n\nWeakness:\n* Although the proposed method could be a promising acceleration solution to LLMs, the authors only conduct experiments on relative small settings (d=512). It would be great if authors could add experiments on LM tasks. Even a 1~7B LM model could make this work more siginificant.", "was_revised": true, "final_justification": "Overall, this is an inspiring paper that temporally compresses the KV cache. Although the computational resource limits this paper,  I think the empirical study on a relative small dataset and model size has shown promising results. Therefore, I lean to accept this paper.", "rebuttal": "We appreciate the encouraging comments in the summary and strengths! It's encouraging when the motivation and strengths of our work are understood. Below, we provide a point-by-point response to the reviewer's comments.\n\n- Weakness 1:\n    > Although the proposed method could be a promising acceleration solution to LLMs, the authors only conduct experiments on relative small settings (d=512). It would be great if authors could add experiments on LM tasks. Even a 1~7B LM model could make this work more siginificant.\n    - Thanks for the valuable suggestion. We agree that it would be very useful to pretrain an LLM based on MTLA. However, as we mentioned in the Limitations section, we are constrained by computational resources and currently cannot afford the cost of pretraining such a model. Even at the scale of 1-7B parameters, ensuring a fair comparison with other 1-7B parameter LLMs still requires large-scale training data and extensive pretraining effort. This is a challenge we will need to address in future, but we believe the experimental validation we currently provide should be acceptable within the scope of an academic paper. We do plan to actively promote the widespread adoption of MTLA and KV cache temporal compression, so scaling to LLMs is definitely an important future direction.\n\n- Question 1:\n    > If my understanding to the algorithm is correct, Figure 2 \\(c) is incorrect. Concretely, the first element of each row (instead of each column) should be q1, q2, q3 and q4, respectively.\n    - Thanks for pointing this out. We will fix this typo in the final version of the paper."}]

Author rebuttal:

Meta-review: This paper introduces Multi-head Temporal Latent Attention (MTLA), a technique for improving the inference efficiency of Transformer models. MTLA compresses the Key-Value (KV) cache along the temporal dimension by dynamically merging adjacent cache vectors using a hyper-network. This method extends the recently proposed Multi-head Latent Attention (MLA), which compresses the cache along the feature dimension, thus achieving compression in both dimensions. The authors also propose a stride-aware causal mask to ensure consistency between parallel training and incremental inference.

*   Strengths: The paper focuses on the critical and well-motivated problem of KV cache memory growth in autoregressive models. The core idea of learnable temporal compression of the KV cache is an important and timely research direction (xgRJ, hHHE), and the setup building on MLA work is interesting and well-motivated. The paper is well written and easy to follow (hHHE) and the proposed stride-aware casual mask is a simple and effective solution. The method demonstrates good empirical improvements in inference speed and memory reduction across a variety of tasks without sacrificing performance (2249, yMqX).

*   Weaknesses: The two main weaknesses of the paper are (1) having experiments only on very small scale models (only 100M params). Even with limited resources these days this is considered very small scale, and multiple reviewers found it unconvincing since it is unclear how MTLA will perform for larger more practical models (hHHE, xgRJ, yMqX). Also (2) reviewers brought up some concerns about the positioning of the paper and the baselines it is comparing against (​​2249, xgRJ). The paper is motivated and positioned as an extension of MLA, which is logical from the method point of view, but due to the goal of the method which unlike MLA focuses on reducing the length of KV, there are multiple techniques for KV pruning or efficient attention that should be compared against as baselines. During the rebuttal the authors added some comparison to SnapKV and Mamba2. Further comparisons will strengthen the paper.
Decision: Accept (poster)