Reconstructing KV Caches with Cross-Layer Fusion for Enhanced Transformers
Authors: Hongzhan Lin
ZhiqiBai
Xinmiao Zhang
Siran Yang
Jiamang Wang
Yunlong Xu
Jiaheng Liu
Yongchi Zhao
Xiang Li
Yuchi Xu
Wenbo Su
Bo Zheng
Venue: ICLR 2026 Poster (2026)
Keywords: KV Cache
Pretraining
LLM
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: This paper proposes FusedKV and FusedKV-Lite. Both methods reduce KV cache memory by up to 50% while achieving lower perplexity than standard Transformers, offering a memory-efficient, high-performance architectural alternative

Abstract: Transformer decoders have achieved strong results across tasks, but the memory required for the KV cache becomes prohibitive at long sequence lengths.
Although Cross-layer KV Cache sharing (e.g., YOCO, CLA) offers a path to mitigate KV Cache bottleneck, it typically underperforms within-layer methods like GQA.
To understand the root cause, we investigate the information flow of keys and values of the top-layers.
Our preliminary reveals a clear distribution: values are predominantly derived from the bottom layer, while keys draw more information from both bottom and middle layers.
Building upon this, we propose FusedKV, whose top-layer KV caches are a learnable fusion of the most informative ones from the bottom and middle layers. 
This fusion operates directly on post-RoPE keys, preserving relative positional information without the computational cost of re-applying rotary embeddings.
To further improve efficiency, we propose FusedKV-Lite, an cross-layer sharing approach, where top-layer KV caches are directly derived from the bottom-layer values and the middle-layer keys.
Compared to FusedKV, FusedKV-Lite reduces I/O overhead at the cost of a slight increase in perplexity.
In experiments on LLMs ranging from 332M to 4B parameters, our proposed method reduce 50\% cache memory while achieving lower validation perplexity than the standard Transformer decoder, establishing it as a memory-efficient, high-performance architectural alternative. We have made our Triton implementation available.

RECONSTRUCTING KV CACHES WITH CROSS-LAYER FUSION FOR ENHANCED TRANSFORMERS
Hongzhan Lin Zhiqi Bai Xinmiao Zhang', Xiang Li', Siran Yang', Yunlong Xu', Jiaheng Liu', Yongchi Zhao', Jiamang Wang', Yuchi Xu', Wenbo Su', Bo Zheng
Taobbo & Tmall Group of Alibaba 2 GSAL, Renmin University of China
Nanjing University
ABSTRACT
Transformer decoders have achieved strong results across tasks, but the memory required for the KV cache becomes prohibitive at long sequence lengths. Al- though Cross-layer KV Cache sharing (c.g.. YOCO, CLA) offers path to mitigate KV Cache bottleneck, it typically underperforms within-layer methods like GQA. To understand the root cause, we investigate the information flow keys and values of the top-layers. Our preliminary reveals clear distribution: values predominantly derived from the bottom layer, while keys draw more information from both bottom and middle layers. Building upon this, we propose FusedK V, whose top-layer KV caches are a learnable fusion of the most informative ones from the bottom and middle layers. This fusion operates directly on post-RoPE keys, preserving relative positional information without the computational cost of re-applying rotary embeddings. To further improve efficiency, we propose BusedKV-Liit
cross-layer sharing approach, where top-layer KV caches are directly derived from the bottom-layer values and the middle-layer keys. Com- pared to FusedKV, FusedKV-Lite reduces I/O overhead at the cost of slight increase in perplexity. In experiments on LLMs ranging from 332M to 4B pa- rameters, our proposed method reduce 50% cache memory while achieving lower validation perplexity than the standard Transformer decoder, establishing it as a memory-efficient, high-performance architectural alternative. We have made code available
40
200
2354
2X
2X
Trasking Lane
200
20
100
Cache Memory(GB)
Prefilling Latency(s)
Vanilla
GQA
YOCO
Lese
FusedKV
Training Tolos
Figure 1: V and FusedK Lite reduce KV cache and prefilling latency by 2x (left) while also achieving superior pretraining loss on a 1.5B model compared to other methods FusedK converge around 1.26x faster than Vanilla.
INTRODUCTION
Large language models (LLMs) based on the Transformer (Vaswani et al. 2017) decoder have achieved unprecedented success across wide range of tasks and applications. However, practi-
1 Equal contribution Corresponding author https://gttuu.coom/ http://ggghbb..ooomlmmnnhnnnnnnn LwingFuuunllaa///usssakky

---

Layer indox
12-
12-
15
15
3
Layer index
Layer Index
Figure 2: Fusion weight for reconstructing key (left) and value (right) caches in the top layers of 16-layer model. The figure reveals a clear asymmetry in key-value caches.
cal deployment for real-world applications requires LLM inference to achieve low latency and high throughput (Hu et al. 2024 Ge et al., 2024, [ao et al. 2022). This becomes particularly challenging with increasingly long contexts, primarily due the linear memory overhead of the key-value (KV)
caches in autoregressive generation (Liu et al. 2024b Adnan et al. 2024 et al. 2024).
To mitigate this, various methods have been proposed reduce KV caches. Within the layer, Group-
Query Attention (GQA P (Ainslie et al. 2023) and multi--uery Attention (MQA) (Shazeer 2019)
reduce redundancy by sharing
head key and value across multiple query heads. MLA (Liu 2024a) uses low-rank joint compression for key-value caches. Across the layer. CLA (Brandon al. 2024) shares the KV cache between adjacent layers, while Y (Sun et al. 2024) reuses
at
the intermediate layer cache for the top-half layers. While cross-layer sharing techniques also reduce memory overhead, they consistently underperform within-layer methods, which motivates the need for a more effective cross-layer sharing strategy.
In this paper, reveal a previously overlooked asymmetric key-value sharing principle: for top- half layers where values are most effectively reconstructed by the bottom layer, whereas keys draw their more critical information from the bottom and middle layer. Motivated by this, we propose FusedKV, which reconstructs the top-half layer caches by performing dimension-wisg fusion of caches from two highly informative source layers: the bottom and middle layer. This fusion can be applied directly to post-RoPE keys, design that preserves relative positional information without the computational overhead of RoPE recomputation. FusedKV halves the cache size while achieving significantly lower perplexity compared to the full-cached model, shown in Figure I We further provide a more effecient version, FusedKV-LLn where top-half layers' keys are directly reused from the middle layer and values from the bottom layer. Compared FusedKV, FusedK Lite decreases I/O overhead by one-third with only slight degradation in performance.
weighted
Our contributions are threefold: (1) We identify key-value asymmetry in cache reconstruction. (2) Based on this insight, we propose FusedK a memory-efficient architecture, and its more efficient version FusedKV-L.ite Extensive experiments demonstrate that methods reduce KV cache memory by 50% while achieving lower perplexity than standard Transformer decoder. (3) We provide an efficient Triton implementation of FusedKV and a systematic evaluation of KV cache reduction methods, including performance efficiency comparisons and integration of our approach with existing architectures.
METHOD
In this section, we first define the overall framework for cross-layer KV cache sharing. We then introduce two specific methods, FusedK V-Lite and FusedKV, and demonstrate FusedKV's compat- ibility with RoPE. Finally, we provide a complexity analysis of different KV sharing methods.
2 Unless otherwise noted, GQA mentioned in this paper configuration where two query heads share single key-value head.
Note that in the original YOCO paper, the cache-storing layers use efficient attention mechanisms such Sliding Window Attention(SWA). In this paper, unless otherwise stated, we full attention.
2

---

2.1 OVERALL FRAMEWORK DEFINITION
We partition the set of decoder layers, - [l,..., , L), into two disjoint subsets: Storage Layers (Cs), for which the KV caches are explicitly stored in memory during generation and Reconstruc- tion Layers RL)), whose KV caches are not stored but are recomputed on-demand via reconstruc- tion function. For any reconstruction layer LR. its Key, K' R*xd and Value, V Rxxd are generated by a parameterized reconstruction function F; using the KV caches from subset of the storage layers. The formula is as follows:
(K', V*) = (1(K), v')li (11)):
where (i) is a Source Layer Mapping Function that specifies a set of source storage layer indices for reconstruction layer i. The trainable parameters 8; associated with the i-th reconstruction layer could be data-dependent or data-independent.
2.2 RECONSTRUCTION FUNCTION
The architecture of the reconstruction function F determines the complexity and effectiveness of the fusion process. We explore two primary categories for the reconstruction function F, differing in their complexity and parameterization.
Direct Cache Reuse.
The most straightforward approach for reconstruction to directly reuse the KV cache from the source layers without any transformation. In this case, The reconstruction be formulated as:
KK', v = (l)) vo()), where <(t) Ls.
the reconstruction function F, acts as selector, the source layer mapping D(i) typically contains single index, i.e., [+ii] = 1. Recent methods such as YOCO (Sun et al. 2024) and CLA (Brandon
al. 2024) adopt this approach.
YOCO partitions layers into two contiguous blocks: a storage block (Cs = (
and a reconstruction block (Lnz {T./2 2 1,..., L)). All reconstruction layers access the KV cache from the final storage layer, defined by the constant mapping @(i) T./2.
CLA employs an interleaved partitioning scheme, designating odd aud-numeered layers for storage (Cs 31, ..... L 1}) and even-numbered layers for reconstruction (Cn I.)).. Each reconstruction layer i accesses the KV cache from its immediately preceding storage layer, with the mapping ((1) = 1.
Weighted Fusion of Caches. Direct cache reuse may lead to representation collapse shared layers, potentially limiting their layer-specific contributions in forward flowing. more expressive method involves computing the reconstructed KV cache as a weighted linear combination of the caches from multiple source layers. The general form is:
K
Σ JE4())
aij K.
V
Σ bij VVVi
JE
Where the learnable weight ai and byj be scalars (IR), vectors (Rd), matrices (R=xd), and are broadcast to match the shape of the Key and Value caches for the Hadamard product O. In this way, we reconstruct the target cache by applying a feature-wise gating mechanism, selectively re- re-weighting features from each source layer before their aggregation.
2.3 FUSEDKV AND FUSEDK V-LTE
Preliminary experiments. We conduct dense fusion experiment on 1 (B-parameter, 16-layer LLM. The top 8 layers reconstructs its cache from a learnable scalar fusion of all the bottom caches. As shown in Figure 10 of the Appendix, dense fusion attains a lower training loss than the vanilla, indicating that top-layer KV caches can be effectively reconstructed from earlier layers. As depicted in Figure 2 we also find a clear asymmetry between keys and values: value-cache fusion is domi- nated by layers 0-1 for reconstruction layers 10-15, while key-cache weights are more diffuse but concentrate on source layers 6-7. These results indicate that the bottom and middle layers are more informative for reconstructing top-layer caches.
3

---

Self Attention
Self- Attention
Self-Attention
Valuz 2+
Quarry
Vilkaz
Volue Zn
Zin
2n
Peff-Attention
Self-Attention
Self Self-Attention
Querry
Value
Quarry
Quary
Self-Attention
Self-Attention
Self-Attention
Self-Attention
Self-A Seff-Attention
Self-Attention
Query
Vila:
(a) Vanilla
(b) FusedKV-LLtte
FusedKV
Figure 3: Illustration of KV cache strategies. (a) Vanilla: The standard method with a unique KV cache for each layer. (b) FusedKV-Lite: For layers i n, the Key cache is reused from layer and the Value cache from layer 1. (c) FusedKV: For layers > n, the caches are a learnable weighted fusion (denoted by ()) of the caches from layer and layer n.
F Motivated by the observation in dense fusion experiment, we propose a simplified yet effective strategy named FusedKV. As illustrated in Figure 3(). FusedKV reconstructs the cache by computing a learnable weighted fusion of caches from two highly informative source layers: the bottom (layer 1) and the middle (layer n)). The process is formulated as:
K ai,i K + ai K"
i>
(4)
V = b o V b OVV. i>
(5)
FusedKV allows each reconstruction layer synthesize its cache from both the low-level, founda- tional features the initial layer and the more abstract, contextual representations from the final source layer. This approach strikes effective balance between representational power and the memory traffic costs associated with cache fusion.
Fused V-Lite further improve efficiency, we reconstruct the keys by directly reusing the cache from the last source layer and the values from the bottom layer, as illustrated in Figure The reconstruction formulated as:
K K V
i>n.
By reusing only single source key and value cache, FusedKV-Litt avoids the additional I/O over- head from fusion, thereby maintaining efficiency on par with the vanilla model, making it highly efficient for I/O-bound inference scenarios.
2.4 ROPE COMPATIBILITY
In this section, establish that RoPE is compatible with learnable weights when the weights symmetric, and we further show that fusing Post-RoPE key caches preserves relative positional encoding.
Weight vectors must he 2D-diagonal. To ensure that learnalle weights do not disrupt the rela- tive positional encoding property of RoPE (Su et al.] (024),, we analyze the attention score. For a query at position and key position n, the attention score An.j in the j-th 2D subspace, after incorporating learnable weight vector wy [12j, w2j+1 1T. be decomposed as follows (see
4

---

Appendix A.4 for the full derivation):
A₂
W2j
W2i+1
(9m.2 + 2m.2j+18m.24+11 cos((m n)0,)
2
+ (9m.23kin.25t 4m.2)+11h..2 sin (m 12) n))8,)
W22
W2j+1
(9m.2) - qm. cos (m n)
2
kn.22 + fm.22+11kn22) sin((m + m))0,)
Equation 7 reveals that when W2y * W2j+1, the attention score becomes a mixture of a relative position term (dependent preserve RoPE's pure relative position dependency, we must enforce identity within each 2D weight pair, i.e., W2j W2j+1- 1-
n) and absolute position term (dependent m + n). To
With the symmetry constraint established, we show that weighted fusion of multiple RoPE-transformed key vectors maintains relative positional encoding. Let kg = Σin (wi, (b))) be a fused key vector from N different storage layers, where each weight vector w symmetric (whh22 1) The attention score with this fused key is:
Weighted Fusion Preserves RoPE.
P k = Im
Σ Ewwh @ kin)
Σ Ire (wo,, T k
As established by Equation with the symmetry constraint, each term T (w @ k ) in the sum is function of only the content vectors and the relative position combination, q.T kig also upholds this property. This compatibility is practically significant. It allows storage layers to retain their original post-RoPE KV caches, avoiding re-computing RoPE at inference time.
Consequently, their linear
Table Complexity analysis. L denotes the number of model layers, S denotes the sequence length, D denotes the head dimension, and Hg and Hąv represents the number of query heads and key-value heads, respectively.
Method
Prefilling FL.0Ps
Dacoding FLOPS
10.
MILAGQAA
DSS 411
LII, all
YOOO
LSH,DDS 3H -
LR,, s1, 221
ISRT..
218HI..D
Las
DSS 22,0 +212+. +2- +2222
LiL, al. 2H
LSHi.D
2LSHL.DD
DusedKV
LSHI,D2S 22I,D +1l.D+2lle.D+11 211l..++++ +2+3"++)+ 3,,3%, 21
LII, al1, 2ile. D +35 (1:)
LSHi,,D
aLSHI,.D
2.5 INFERENCE
We conduct complexity analysis of different attention mechanisms in Table T The comparison focuses on the computational cost in prefilling and decoding phase. KV cache memory footprint, and KV cache I/O volume. These calculations are confined the attention component, excluding the feed-forward networks. We find that both FusedK V-Lite and FusedK V reduce prefilling FLOPS and cache memory compared the vanilla model. Due its fusion computation, FusedKV incurs slightly higher cache I/O volume compared to the other methods.
We implement a Triton-based attention kernel for proposed FusedKV operator, and benchmark the performance of F and FusedK V-Lite from three aspects: (i) attention throughput, which captures the speed of token generation by the attention kernel, (ii) end-to-end prefill performance, which is quantified by Time to First Token (TTFT), and (iii) end-to-end decoding throughput, which expressed by Time Per Output Token (TPOT).
Attention Throughput. As shown in Figure 4 (left), the throughput of FusedKV kernel is 28.4% lower on average than MHA due to an extra cache I/O. In contrast, FusedKV-Lite, which maintains comparable cache I/O, achieves identical throughput MHA.
5

---

0.8
0.
0.4
0.2
8C
1/v
61c
1kk
1/
6xx
1244
2964
Sequence Length
Sequence Langth
FueesRty
10:41
40.00
Figure 4: Left: The attention throughput of different kernels (The higher is better). Right: Time to First Token (TTFT) of different models, showing the end-to-end prefilling performance (The lower is better). All methods are normalized by the vanilla (MHA) baseline.
1.0
1.0
0.8
0.6-
0.4
0.4
0.2
0.2
32k
Fuenday
Voco
VOCD-50022
Ruceclttii Ie-GQK2
Figure 5: Time Per Output Token (TPOT) performance ratios. The left panel displays the memory- bound scenario, and the right panel displays the compute-bound scenario.
Time to First Token (TTFT), As shown in Figure 4 (right), both FusedKV and Fused exhibit a clear advantage over vanilla implementation on prefilling latency. Specifically, for input sequence length of 8k and beyond. the TTFT is reduced by approximately 50% relative vanilla. This improvement originates from FusedK V and FusedK V-Lite layers, which reuse the KV cache from preceding layers and skip the cache prefilling current layers, thus halving the overall prefill- ing latency for the end-to-end model.
Time Per Output Token (TPOT). We further examine the decoding speed under both memory- bound and compute-boune hrund scenarios. As illustrated in Figure (left), the memory- -ound setting, the additional cache I/O overhead in the FusedKV layers leads to an approximately 1.5x increase TPOT compared to the vanilla implementation. In compute-bound settings, where the baseline uses GQA with H, >> HA.. this cache I/O overhead is effectively hidden by the computational workloads. For evaluation, we adopt GQA with 128 query heads and 2 key-value heads as the baseline. The computational overhead introduced by cache fusion accounts for only 3x Hy 4x 256 of that required by attention operation, which is practically negligible. Consequently, TPOT of FusedKV remains comparable to that of the baseline. In contrast, FusedK V-Lite maintains cache IO nearly identical to that of the vanilla implementation. Therefore, its TPOT is similar to the baseline under both memory-boute and compute-bound scenarios.
3 EXPERIMENTS
3.1 GENERAL SETUP
Architecture and Training Details We introduce three dense language models with 332M, 650M and 1.5B parameters, both following the Qwen3 architecture (Team) 2025). More configuration are detailed in Table 6 All three models share the same configuration: a vocabulary tokens, context length of 8,192, and 16 attention heads. They trained on the Fine Edu dataset (Lozhkov et al. 2024). We use the AdamW optimizer (Loshchilov Hutter 2017) with
128 0,0000
31 0.9. Ba 0.95. The learning rate followed cosine schedule, warming up for 2,000 steps

---

Table 2: Evaluation of various KV cache reduction methods on downstream tasks, highlighting the competitive performance FusedKV V and FusedK V-tiee across 332M, 650M, and 1.5B model sizes.
Model
Cache
Valid
Wiki
MNLI
SCIQ
LAMB- Acc
Hella Swag
ARC-E
ARC-C
MMLU
Avg
4
4
Acc1
332M porams,
Vanilla
2.651
22.85
34.64
84.20
24.49
42.12
59.26
29.69
28.93
43.33
CLA
1/2
2.676
24.62
34,80
76.70
13.51
40.75
54.67
27.65
28.56
39.52
YOCO
2.663
23.31
33.64
80.60
26.61
41.74
59.93
30.29
28.77
43.08
GQA
2.662
23.29
34,99
80.50
25.52
41.72
60.23
31,40
29,45
43,40
FusedKV-Lite
2.639
22.78
34.22
79.60
26.39
42.07
58.67
32,00
29.38
43.
PusedKV
2.642
22.35
33.62
80.30
28.10
42.74
60.98
30.12
29.26
43,59
650M params.
Vanilla
2.483
18.47
34.39
88.10
31.67
49.13
65.36
31,62
48.08
CLA
1/2
2.511
19.60
34.25
86.10
27.94
47.48
65.91
32.94
30.81
46,49
YOCO
2.498
19.21
34.96
86.70
30.74
48.65
64.14
34,39
30.66
47.18
GQA
2.497
19.05
33.11
86.30
30.56
48.62
31.56
47.08
FusedKV-Lite
2.473
18:55
34.24
86,90
32,95
49.53
66.58
37.46
31.88
48,51
FusedKV
2.474
18.09
33.33
86.60
31.03
49.68
65.61
34,39
31,54
47.45
Vanilla
2.241
13.67
35.69
94.70
40,54
60.67
72.55
42.58
35.12
54.55
1.5B params.
CLA
1/2
2.258
14.19
35.00
92.60
39.61
59.04
73.99
42.32
34.83
53.91
YOCO
2.244
13.65
35.30
91.70
40.81
60.09
73.19
42.92
35.35
54.19
GQA
2.245
13.74
34.63
92.40
41.20
60.16
73.91
44,28
35.49
54.58
FusedK
2.225
13.45
36.43
93.70
41.61
60.77
74.79
43.52
36.29
55.30
FusedKV
2.221
13.33
37.43
93.50
43.33
61.88
74.20
44,20
36.18
55.82
2.74
Valldatton Loss
Validation Loss
Valldation Loss
Vanilla
Vanilla
Vanilla
2.56
2.72
Rused
Fusec
2.32
2.54
2.70
2,50
2.68
2.50
2.48
808
1008
1206
1008
1656
1806
2006
1205
1408
1608
1808
2008
2008
2408 2816 3256 Tokens
3636
4058
Tokens
Tokens
Figure 6: Validation loss curves of 332M (left), 650M (center), and 1.5B (right) dense models, where V and V-sedKV-Lite consistently achieves lower validation loss than the Vanilla.
a peak of 3 x 10-4 and decaying to a minimum of 3 x 10-5, A more detailed configuration can be found in Table
Evaluation For model evaluation and comparison, we assess perplexity on a 500M-token valida- tion split randomly sampled from FineWeb-Edu. We also evaluate perplexity on Wiki'eet (Mer- iity et al., (20T6). We also conduct five-shot evaluation across diverse language tasks, including MNLI (Williams et al. 2018), SCIQ (Welbl et al.) (017), LAMBADA (LAMB-Acc) (Paperno al. 2016), Hellaswag (Zellers et al./ 2019), ARC-Easy (Clark et al. 2018). ARC- ERC-Challenge (Clark ct al, All evaluations are based the LM Evaluation Har-
2018 and MMLU (Hendrycks al." 2021
ness
(Gao et al.).
3.2
MAIN RESULTS
As presented in Table 2 our proposed methods, FusedK and FusedK demonstrate strong performance across three model scales. Compared to the full-cache Vanilla baseline and other cache-saving methods, our approaches consistently achieve competitive superior results on both language modeling perplexity and downstream task accuracy.
The effectiveness of our methods is particularly pronounced at the 1.5B scale. FusedKV not only achieves the lowest validation loss (2.221) and WikiText perplexity (13.33) but also, alongside FusedKV-Lite, secures the highest average downstream accuracies (55.82 and 55.30, respectively). Notably, these results substantially outperform the full-cache vanilla baseline (54.55) and include top performance on challenging tasks such ARC-E, MMLU, and HellaSwag.
This trend of strong performance remains consistent across smaller model sizes. At the 332M and 650M scales, FusedK V-Lite and FusedK respectively lead all methods in average downstream
7

---

Table 3: FusedKV outperforms the vanilla on 4B-parameter LLMs in perplexity and downstream
tasks.
Valid
Wiki Text +
LAMB-
Hclla Swag
Avg
Model
MNLI
SCIQ
ARC-E
ARC-C
MMLU
Loss
Acc
Acct
Vanilla
2.002
9.18
37.27
96.00
49,60
68.92
76.43
46.59
38.71
59.07
FusedKV
1.978
8.94
38.52
95,20
50.18
69.94
77.78
48.63
39.83
60.01
curacy, though we note their performance on challenging tasks like MMLU and ARC-C remains close to the chance level. Crucially, when compared to methods with equivalent 50% cache savings like YOCO and GQA, our approaches consistently yield superior accuracy. These comprehensive results, further illustrated by the validation loss curves in Figure 6 highlight the ability of our meth- ods to enhance model performance and in-context learning capabilities. Furthermore, we present results for longecontext scenarios in Appendix A.7
3.3 SCALING LAW EXPERIMENTS
Loss Curves with Various Model Sizes
2.6
Vanilla KV
We compare the loss scaling behavior of the vanilla and FusedKV across increasing model parameters ranging from 332M to 4B. The training configura- tions used scaling experiments are detailed in Ta- ble + of the Appendix, with model sizes and to- ken counts detailed in Table (6] Specifically, the 332M, 650M, 1.5B and 4B models are trained on 200B, 200B, 400B, and 800B tokens, respectively. As illustrated in Figure 7 FusedKV achieves better scaling efficiency, with its loss decreasing more no- tably as the model capacity grows. For 4B-parameter LLMs, FusedKV exhibits better downstream perfor- mance than vanilla, as depicted in Table This demonstrates its ability to retain performance at larger scales, making it particularly promising for page-parameter models.
2.5
2.4
2.3
2.2
2.0
332M
650M
1.5B
48
Model Size
Figure 7: Scaling law curves of FusedKV and the vanilla.
3.4 EXTENSIBILITY AND COMPATIBILITY
We extensively validate the compatibility of FusedKV and FusedKV-Lite across diverse spectrum of efficiency-oriiee oriented designs, including Multi-Heal Latent Attention (MLA) (Liu et al. 2024a), Grouped-QU Quueryyuuuuuuuuuu Attention (GQA) (Ainslie et al. 2023). Mixture-of- -xperrtttsssssss-ooooooo (MoE) (Shazeer et al. 2017), and hybrid structures like Sliding Window Attention (SWA) (Jiang et al. 2023) Empirical results demonstrate that cross-layer fusion mechanism is largely orthogonal to these architectural optimizations, often yielding synergistic benefits in both inference and memory footprint with neg- ligible performance degradation. Full experimental details, configurations, and results are provided in Appendix A.6
3.5
ABLATION STUDY
In this section, we investigate two key aspects of our method: the directionality of the asymmetric KV assignment and the impact of parameterizing the reconstruction with learnable weights. We conduct a more detailed ablation study in Appendix A.8
Directionality of KV asymmetry. We first investigate the impact of the asymmetric assign- ment strategy. As shown in Figure g reversed assignment (FusedKV-Lite-Rew where Ki K², V = V
8) degrades performance substantially compared with our proposed FusedKV. M-setKV-Lite-A exhibits markedly higher validation loss and commensurately lower downstream accuracy. This gap demon- strates that when directly reusing a single-source KV cache, layer- Keys and early-layer Values more informative for reproducing the original behavior.
Lite (K* = K V = VV i > 8). Compared with FusedKV-LLtee
8

---

(%)
22.0
AascktLits-b
NascRt MoiRALitee we-.ttte--mmaae usmmabre
23.5
Fuaodtoy Ule Leamaki
21.0
23.5
25.0
19.5
10.0
18.5
836
1008
1208
1440
1608
1015
2008
508
1008
1205
1408
1028
1805
2008
Tokens
Tokers
Figure 8: Ablation study of KV asymmetry and learnable weights 650M dense models with 200B tokens. We report the perplexity on Wikitext and the five-shot accuracy on Hellaswag for three key FusedK
variants:
MessedV-Lite-Rer and FusedK Leaaeeeeaa-.....................
Impact of learnable weights. We also study the impact of parameterizing the reconstruction with uses learnable vectors to perform channel-
learnable weights. The variant, BasseKKVLLit-Lemmmat wise re-weighting: K ag @ K and V 6 V for layer indices i >> 8. The results in Figure that FusedKV-Lit--Laamade
(green line) improves upon the fixed-weight FusedKV- Lite baseline (blue line). It achieves lower perplexity on both Wikitext and LAMBADA. FusedK Lite- Leeenbeaee also outperforms FusedK V-Lite on Hellaswag in the later stages of training. This demonstrates that allowing the model to adaptively re-weight source Key and Value channels in- creases expressiveness and yields downstream gains.
GRADIENT VISUALIZATION
We visualize the L2 of gradients for the query, key, and value projection matrices different network depths. As depicted in Figure 9 we track these gradient norms throughout the training of a 100M-parametrr model. The results reveal distinct pattern: FusedKV and FusedKV-Lite consistently exhibit markedly larger gradient 1.2 norm compared to baselines, particularly within the shallower layers of the network (e-g., Layer and Layer 5). A larger gradient magnitude implies more substantial parameter updates during backpropagation. Therefore, the pronounced gradients in the shallow layers for FusedKV and FusedKV-Lite suggest that these layers are learning more effectively and rapidly. By facilitating stronger gradient signals to the initial layers, learnable fusion mechanism enables the model to more quickly refine its foundational representations. This suggests that the primary benefit of our fusion mechanism gradient flow is to accelerate learning in the crucial early layers, which form the bedrock for the hierarchical features learned by the deeper parts of the network.
RELATED WORK
5.1 CROSS-LAYER KV CACHE
KV caching is major memory bottleneck in LLM inference (Wu et al.) Liao & Vargas 2024 Memory footprint can be reduced by limiting KV state retention to a subset of layers. YOCO (Sun al. 2024) reuses a global KV cache from an intermediate layer for the model's latter half, roughly halving memory: CLA (Brandon et al. 2024) shares caches across adjacent layers. More aggres- sive cross-layer reuse includes Mixture of Recurrent (Bae et al.) 2025), which stacks recurrent blocks (each comprising multiple transformer layers) so later blocks reuse the first block's 's KV. SVFormer (Zhou et al. 2025), which does not share the key but reuses the bottom layer's values; and LCKV (Wu& & Tu 2024), which caches only the last layer's KV for all earlier layers, with safeguards against cyclic dependencies. However, most approaches often overlook the distinct roles of keys and values: keys drive relevance scoring, while values supply contextual content. Treating KV as a single unit for caching neglects this functional disparity. Moreover, sharing KV states in- discriminately across layers risks undermining each layer's 's specific role in hierarchical processing, potentially introducing irrelevant context and hindering its unique function. These critical aspects remain underexplored.

---

Linear_0
Linear.K
Line
13.03
22.00
22002
22000
2008
20000
200%
1000
19
1008
2008
20000
2000
10000
14000
1200
2000
100
Figure 9: Comparison of gradient L2 norms shows that FusedKV and FusedKV-Lite maintain sig- nificantly stronger gradient flow in shallow layers compared to baselines. This suggests healthier gradient flow that accelerates the convergence of early layers.
5.2 EARLY-FEATURE FUSION
Another line of research leverages early-layer feature fusion to stabilize gradients and enrich global features( (He et al.)
2016
Huang al. 2017 Pagliardini et al.) [022 (Abdullaev & Nguyen 2025 )
ResNet (He et al.] 2016) introduces residual connections that add outputs from previous layers to the current layer to stabilize training. Subsequent approaches further enhance the information flow within this framework. DenseNet (Hnuang et al. (017)) concatenates the outputs of all previous layers, while Denseformer (Pagliardini et al.) (022) and MambaMixer (Behrouz et al. 2024) incorporate learnable weighted coefficients into the residual paths, making the residual representations more expressive. Furthermore, reusing features from earlier layers can help alleviate the over-smoothing problem (Wang al. 22022] Shi et al.] (2022), where representations converge and lose diversity as the depth of the network increases, leading to degraded performance. To mitigate this issue, Neu- TRENO (Nguyen et al. 2023) introduces skip connection that injects a fraction of the bottom-layer output into every subsequent self-attention layer to mitigate over-smo. Similarly, (Abdullaev & Nguyen 2025) enriches the transformer's representational capacity by reusing value-residual in- formation through twice-attention mechanism.
6 CONCLUSION
In this work, we identified a key-value asymmetry principle, revealing that keys and values in up- per layers are best reconstructed from different source layers. Based on this insight, we proposed FusedK V, an efficient architecture that reconstructs the top-half layer caches via a weighted fusion of caches from the bottom and middle layers. We also introduced FusedKV-Lite, more 1/O-efficient variant that uses direct asymmetric sharing. Our extensive experiments demonstrate that F reduces the KV cache by 50% while remarkably achieving lower perplexity than the full-cached baseline. By offering substantial memory savings without trade-off in performance, FusedKV V and FusedKV- V-Lite establish a effective, integrable paradigm for cross-layer cache sharing, paving the way for more efficient deployment of powerful long-context language models.
10

---

7 ETHICS STATEMENT
The authors have read and adhered to the ICLR Code of Ethics. This work utilizes publicly avail- able datasets and does not involve any sensitive personal information or experiments with human subjects. We have considered the potential societal impacts of our research and believe the work does not raise significant ethical concerns. We encourage the responsible use of our methods and findings.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, the source code for our experiments has been submitted as supplementary material. It is accessible via anonymous link and includes scripts to replicate our main results.
REFERENCES
Laziz Abdullaev and Tan Nguyen. Transformer meets twicing: Harnessing unattended residual information. arXiv preprint wxxiv:253.00668 2025.
Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J Nair, Ilya Soloveychik, and Pu- rushotham Kamath. Keyformer: Ky cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems, 6:114-127, 2024.
Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305. 13245, 2023.
Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, et al. sive depths for adaptive token-level computation. arXiv preprint arXiv:2507.10524, 2025.
Learning dynamic recur-
Ali Behrouz, Michele Santacatterina, and Ramin Zabih. Mambamixer: Efficient selective state space models with dual token and channel selection. arXiv preprint 2.t.:24033.1888. 2024.
William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ra- gan Kelly. Reducing transformer key-value cache size with cross-layer attention, 2024. URL https://arxii. stg/abbs/240.11298// 2024.
Li-Wen Chang, Wenlei Bao, Qi Hou, Chengquan Jiang, Ningxin Zheng, Yinmin Zhong, Xuanrun Zhang, Zuquan Song, Chengji Yao, Ziheng Jiang, et al. Flux: Fast software-based communication overlap gpus through kernel fusion. arXiv preprint arXiv:2406.0068 06858, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint
2018.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344-16359 2022.
L. Gao, J Tow, B Abbasi, S Biderman, Black, A DiPofi, Foster, L Golding, Hsu, A Le Noac'h, h, et al. A framework for few-shot language model evaluation, 07 2024. URL htlps://zenodo. org/records/1268600..
Suyu Ge. Xihui Lin, Yunan Zhang, Jiawei Han, and Hao Peng. A little goes long way: Efficient long context training and inference with partial contexts. arXiv preprint arXiv:2410.01485. 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
11

---

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Theng-Ping Hsieh, Simeng Sun. Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's s the real context size of your long-context language models? arXiv preprint arXiv:2404.0665.. 2024.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential small language models with scalable training strategies. arXiv preprint arXiv:2404066390 06395, 2024.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7b, 2023. URL https: //axxiv.rgg/abs/
06825
Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, and Lei Chen. A survey large language model acceleration based kv cache management. arXiv preprint arXiv:2412. 19442, 2024.
Bingli Liao and Danilo Vasconcellos Vargas. Beyond kv caching: Shared attention for efficient llms. arXiv preprint urXiv:2407.12866 2024.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture- of-experts language model. arXiv preprini arXiv:2405. 04434, 2024a.
Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. Minicache: Kv cache compression in depth dimension for large language models. Advances in Neural Infor- mation Processing Systems, 199179997-140011, 2024b.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: a tuning-free asymmetric 2bit quantization for kv cache. In Proceedings of the 41st International Conference on Machine Learning, pp. 32332-32344, 2024c.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.05101 2017.
arXiv preprint
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024. URL httpss:/hhoggiigaa.....oaaaaaeet
HuuggingFaacee//Fieee
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Tam Nguyen, Tan Nguyen, and Richard Baraniuk. Mitigating over-smoothing in transformers via regularized nonlocal functionals, Advances in Neural Information Processing Systems, 36:80233 80256, 2023.
Matteo Pagliardini, Amirkeivan Mohtashami, Francois Fleuret, and Martin Jaggi. Denseformer: Enhancing information flow in transformers via depth weighted averaging. Advances in neural information processing systems, 37:199497-336500 2024.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint arXiv: 1606.0606333 2016.
12

---

Noam Shazeer. Fast transformer decoding: One write-head is all you need, 2019. URL https: /atxiv.org/aab//11.... 02110
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06533 2017.
Han Shi, Jiahui Gao, Hang Xu, Xiaodan Liang, Zhenguo Li, Lingpeng Kong, Stephen Lee, and James T Kwok. Revisiting over-smoothing in bert from the perspective of graph. arXiv preprint arXiv:2202.08625 2022.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568:127063 2024.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations.
Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang. Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. Advances in Neural Information Processing Systems, 37:7339-7361, 2024.
Alimama Al Infra Team and Future Living Lab. Infersim: A lightweight Ilm inference performance simulator. 2025.
com/ http:///..........oo.//a/aaaaaan
Qwen Team. Qwen3 technical report, 2025, UKL/attss///rrrrrrrrrr///////./3
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Bin Wang, Bojun Wang. Changyi Wan, Guanzhe Huang, Hanpeng Hu, Haonan Jia, Hao Nie, Min- gliang Li, Nuo Chen, Siyu Chen, et al. Step-3 is large yet affordable: Model-system co-design for cost-effective decoding. arXiv preprint (ttii::2507.142221 2025.
Peihao Wang, Wenqing Zheng, Tianlong Chen, and Zhangyang Wang. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory practice. arXiv preprint arXiv:2203.05962, 2022.
Johannes Welbi, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprini arXiv: /707006277 2017.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challeen corpus for sentence understanding through inference, 2018. URL http:::/aax.vvoorr//bbb/////////
05426
Haoyi Wu and Kewei Tu. Layer-condensed kv cache for efficiece inference of large language models. arXiv preprint ar 4rXh::2405.10637, 2024.
Songhao Wu, Ang Lv, Xun Zhang, Guojun Yin, Wei Lin, Rui Yan, al. Polarquant: Leveraging polar transformation for key cache quantization and decoding acceleration. In The Thirty-ninth Annual Conference on Neural Information Prucessing Systems.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint wxXiv:/90.007830 2019.
Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang, Fares Obeid, and Zhenzhong Lan. Value residual learn- ing. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). pp. tional Linguistics. ISBN 979-8-89176-251
Vienna, Austria, July 2025. Association for Computa- U / //acl hntho logy. org/2025.
agl woo.on..... 1375/
13

---

A APPENDIX
A.1 LLM USAGE STATEMENT
In the preparation of this manuscript, employ Large Language Models for the purpose of lan- guage editing and refinement. The use the LLM was strictly confined to improving grammatical accuracy, sentence structure, and overall readability of the author-written text. The authors have carefully reviewed and revised all LLM-generated suggestions and assume full responsibility for the entire content and integrity of this paper.
A.2 ARCHITECTURES OF DIFFERENT MODELS
we adopt decoder-only transformer architecture akin to Qwen3 [Team 2025), with model sizes ranging from 332M 4B parameters. The specific architecture of models summarized in Table The hyperparameters for Pretraining is shown in Table 4
Table 4: Hyperparameters for Pretraining.
Dense Model
Optimizer
AdamW
Learning Rate (LR)
3e-4
Minimum LR
3e-5
LR Schedule
cosine
Weight Decay
0.1
81
0.9
82
0.95
Gradient Clipping
1
Batch Size
512
Warmup steps
2,000
Init std
0.02
RoPE 0
10000
Activation
SwiGLU
Training Loss
Vanilla
Dense Fusion
2.30
2.29
Want
2.28
2.27
20000
Steps
Figure 10: Training loss of a 16 layer 1B 1B-paramter model pretarined on 80B tokens.
A.3 TRAINING AND INITIALIZATION STRATEGIES FOR F
In this section, we provide a detailed description of the different training and initialization strate- gies explored for FusedKV. These strategies vary in their architectural implementation and weight initialization schemes, offering different trade-offs between layer-wise dependency and training sim- plicity.
14

---

Table 5: Performance comparison of three initialization methods with 650M-parameter model trained with 200B tokens.
Valid
Wiki Text 4
LAMB- Acc
Hella Swag
Avg
method
MNLI
SCTQ
ARC-E
ARC-C
MMLU
Loss
Acci
Equivalent Initialization
2.475
18.06
32.86
89.40
32.52
49.91
67.80
36.18
32.15
48.69
Normal Initialization
2.478
18.33
32.87
89.10
32,47
49.46
66.79
35.15
31.40
48.18
Iterative
2.474
22.27
18.09
86.00
31.03
49.68
65.61
34.39
31.54
47.45
Table 6: Model architecture for dense models.
332M
650M
1.5B
4B
Model Dimension
1024
1536
2048
2560
FFN Dimension
4096
4096
5632
9728
Attention heads
16
16
16
32
Layers
12
16
24
36
Vocabulary Size
128000
128000
128000
128000
Weight Tying
False
False
False
False
Context Length
8192
8192
8192
8192
Iterative Fusion with Auxiliary Caches. This approach introduces an iterative fusion process for the reconstruction layers (Cx). We auxiliary maintain one-layer caches, which iteratively updated. For the first reconstruction layer (i = n + 1), the process is identical to the standard FusedKV, where its key and value caches computed by fusing the caches from source layers and n:
K an,1 ② K- + an+ln ② K Vat1 bat1,1 & V bnt1,n ④ V
For subsequent reconstruction layers (i 12 1), the key cache K' is formed by fusing the reconstructed key from the previous layer (Ki-1) with the cache from the last storage layer (K"). Similarly, the value cache V is a fusion of the previous layer's value (Vi-1) and the cache from the first storage layer (V'). This is formulated as:
K' = a 2 K + tain 3 K", 8>n+1
V bi V + +b, V i>n+1
For initialization, all learnable d-dimensional weight vectors (a, b) are drawn from standard normal distribution, (O(, 1). This iterative design creates dependency chain across reconstruction layers, allowing for progressively refined representations.
Standard FusedKV with Initialization Variants. This is the standard fusion method, where each reconstruction layer independently computes its KV cache by directly fusing the caches from source layers and n. In this approach, there is no dependency between the reconstruction layers, allowing for simpler, parallel computation during training and inference. For this architecture, we explore two distinct initialization schemes:
Normal Initialization. Each of the d-dimensional weight vectors a,,, ain. ba, bin for every layer is independently sampled from standard normal distribution, (O, 1).
Equivalent Initialization. This scheme initializes the standard FusedKV by recursively computing its weights to match the output of the iterative method at initialization. First, sample set of auxiliary weights from standard normal distribution for all reconstruction layers i >> n:
d/, a/,1 bi, b/, b/,1 1
The final model weights for layer are then deterministically computed as follows:
15

---

For the key weights:
1
i=
01,1
d 3 01-1,1
i>n+1
dntin
i=
ai,
d C a4-1,
+
i>n+l
For the value weights:
b 1111
i=nt1
b
1/i-1 1 + b/,1 i>n+l
i= n +1 1 G b
b 1
i> n+1
This one-time computation the initial values for the model's parameters (a.b), which are subsequently treated as independent and learnable.
We explore three distinct training and initialization strategies for FusedKV, performance compari- son of these strategies presented in Table 5
A.4 DETAILED DERIVATION OF ROPE COMPATIBILITY
In this appendix, provide the full derivation for the attention when tranfforaffo vectors are combined with a learnable weight vector. This derivation supports the claim in the main text that symmetric weights required preserve relative positional encoding.
Without loss of generality, we consider the computation within the j-th 2D subspace of one attention head. The query Qm at position m and key kn position n transformed by RoPE rotation matrices Rmo, and Rae,, respectively. A learnable weight vector wj [W2j W2++ 13 is applied element-wise the transformed key. The attention score An.j computed as:
An qm.. (w) @ kn.))
where
qm.j
Rmo qm.j cos(m),) sin(m)))
cos(me)) cos(mo)))
sin(m0,))
4m,25
9m.22t+1/
9m, cos (mi,)) - 4m.25+1 sin (md,) mm sin(m)))) + qm, 2 cos(mo)))
and
knnj
Bno,kind (n0,))
sin(no;)) cos(n0,))
sinn(nn (n0z)
kn,22+1)
Nin cos(n0,) kin sin()10;)
kin 2n,2 sin(n0,)) + kn,25+1 cos(n0;))
We now expand the dot product:
A₁ 9m,25 cos(m8,) ) 9m.2f sin(m,,) T
W2j (kn.2) cos(no,) - kn,2f+1 sin ((nn)))) W2j+1 (kn,2) sin(n0,) + kn,25+1 cos(n)))
9m,25 sin(m(f) + 9m,2j41 cos(m6,)
(j cos cmm0)) 2m, sin(m);;) w2y (ki,,2 cos(no,) kn, 1 sin())
(4m.25 sin(m0,) + 4m.2 cos(m);)) W2)+1 (kn,2 sin + kn,2 cos(n))))
4m.22 kn (W2y cos( (os(mf,) cos(n)))))))))))))))))))))) + W2j+1 sin (md,) sin(n0;)) +)) +
4m, k (wz sin(me,) sin(n0,) w2i+1 cos (m0,)) cos(n0,))
km. (w2y+1 sin(me), cos(nd,) - W22 cos(m1)) sin(no,)) +
4m,2j4 111.241knn22 (w2z+1 cos cos(mo;) sin(n 22y sin(mol,) cos(n(;))
16

---

Using the identities cos(A) cos(B)
cos(A- toool (A
and sin(A)
cos( B) B)) we regroup the terms.
/2 cos(m),) cos(nd,) + w2s+1 sin(m),, sin(n0,) (inmm)))sin((n))
=
W2j + w2j+1 cos( (m - m))0)) + W2j W2j+1 cos((m + n))0g)
fe peiiiiie
2
2
sin(m00) sin(n0z) + w2j+ (o)()))) cos(n10)
W2j + w2j+1 cos (m - 0) W2j w2j+1
cos( (m + n))0;)
2
2
Similarly, using sin(A) cos( (3s = sin( A (A+BB) ttstn sin (mmm)) (n8)) - 222 cos( (m8, 3) sin(nd,) =
(A+B) and cos(A) sin (3) = sin A B))++ (A A+B)
2
2
W2j+1 + W2j sin (m n))0)² + W2j+1 W2j
sin sin((m + n))
2
12ft1 cos(m))) sin(n0,) w2j sin(m (
w2j+1 + W2j sin (m n))0,) + W2j+1 W2j sin ((m + n))0,)
2
Substituting these back and collecting terms for (m - 1²) and (m + n)) yields the final form:
W22 + W2s+1
A; =
(4m.2) kn + 1 cos((m n))
2
+ + - 4m.22+ )2) sin - 12 ))1
W22
W2j+1
(4m,2) - Qm knn cos cos((m (m n))0,)
2
sineii (4m.2) + 4m.22+1hhn.2 sin((m + n)),,)
This final equation clearly shows that the term multiplied by May way-wait11 depends on the absolute positions through (m + n.) This term vanishes if and only if uy2 = 125+11 leaving only the terms dependent on the relative position m n.
A.5 ANALYSIS OF TRAINING DYNAMICS
Models Exhibit Practical Convergence. To provide comprehensive understanding of the per- formance evolution during training, we evaluate checkpoints of our 1.5B-parameter models every 40B tokens. The results, presented in Figure I1 allow for detailed analysis of convergence trends and the sustained effectiveness of our proposed methods. The learning curves across a majority of the benchmarks demonstrate clear diminishing returns. For instance, the Wikitext perplexity curve, as well as the average score across all tasks, begin to flatten in the later stages of training. The performance improvement between 320B and 400B tokens is considerably smaller than in carlier intervals. This pattern indicates that the models are approaching a state of practical convergence, where relative model capabilities can be compared reliably, even if absolute convergence has not been reached. The performance gap between our methods and the baseline does not diminish over time.
Sustained Advantage at Convergence. To further validate the sustained advantage of our meth- ods under near-converged conditions, we continued training from the 400B token checkpoint for an additional 200B tokens. During this extended phase, we used a fixed learning rate of 3e-5, matching the final rate of the initial training. The results, presented in Figure 12, show that the performance curves for all models have almost completely flattened, indicating that they have reached state of convergence. Crucially, both FusedKV and FusedK maintain a consistent and significant per- formance lead over the Vanilla baseline across all benchmarks. This confirms that the performance gap does not diminish even after extensive training, providing strong evidence that our methods yield fundamentally more capable model, rather than merely accelerating initial convergence.
17

---

WiMTest
MHLI
SCIQ
Tusedit PusedRoALt
GDA
37
Scire
62
Score
122
12
26
662
1250
2333
4030
2000
1032
32
Sc0r
33
16
31
682
1620
2008
2002
4230
030
1600 20.0
4626
692
1682
2430
3202
<5a
Figure 11: Performance evaluation during final training from 400B to 600B tokens. The models exhibit converged performance, as shown by the stable metric scores. Crucially, the performance gap between our proposed methods (FusedKV and V-ste and the other baselines does not diminish, confirming the long-term effectiveness of approach.
Table 7: Configuration parameters for the MLA baseline model.
Parameter
Value
attention.hewt heads
16
q.lora_ran
864
kv.lora_rank
736
gk.nope.head..dim
gk.rope.head.diim
32
v. v.head.din
96
A.6 COMPATIBILITY WITH OTHER ARCHITECTURES
A.6.1 COMPATIBILITY WITH MLA
While our proposed cross-layer fusion method, FusedKV, offers a novel way to reduce KV cache size by reconstructing information, alternative approaches achieve this through fundamental archi- tectural modifications. A prominent example is Multi-Head Latent Attention (Liu et al.) 2024a) which reduces cache dimensionality by design rather than relying on reconstruction across layers. To investigate whether these two architextures are synergistic, we evaluate the compatibility and combined benefits of applying our method on top of MLAA-baeed model.
Experimental Setup. We conduct experiments by applying FusedK to model already equipped with MLA. Our baseline is Table This configuration is designed to halve the cache size compared to a standard Multi-Hess Attention (MHA) model while maintaining the same attention computation cost.
model, configured with the MLA settings detailed in
18

---

WinTTatt
MILI
SCIO
124
$4.0
00.34
Saare
$3.0
$2.5
044
122
33
01.3
34.2
14.07
$4.90
73.0
12.04
$3.4
4828
$200
0008
4000
5933
4230
2208
A
Average
4000
812
40.5
35.
40
400
4828
$250
6000
4.00
5323
5032
6020
4038
6850
2008
63.0
Figure 12: Model performance at convergence (400B to 600B tokens). This extended training run with fixed learning rate shows that the performance advantage of FusedKV and FusedKV-Lite is maintained even after the models have fully converged.
In our combined setup, denoted as MLA+FusedKV, we apply our fusion mechanism to the MLA architecture. Specifically, for the top half of the layers, we reconstruct the compressed key-value rep- resentation (kv-comp ress) and the key's positional embeddings (k-pos -mm_ through learnable fusion of the correspondings from the bottom and middle layers.
It is worth noting that the standard MLA design jointly compresses K and V into single kv compress matrix. This design choice conflicts with the asymmetric sharing mechanism cen- tral to our more aggressive optimization, FusedKV-LLite Adapting MILA to support such asymmetric structures would require non-tn non-trivial architectural modifications, which we leave as a promising di- rection for future work.
Model Performance and Memory Footprint We first compare the performance on perplexity and downstream benchmarks. As shown Table the combination of FusedKV with MLA yields comparable results. MLA+FusedKV achieves a lower perplexity on WikiText compared to the base- line MLA and outperforms on 5 out of 8 downstream benchmarks. This result demonstrates that our cross-layer fusion method provides benefits even when stacked on top of MLA.
As expected, the baseline MLA model itself incurs a performance drop compared to standard MHA due to the information loss inherent in its KV compression scheme. The results indicate that FusedKV can be valuable tool to further compress models like MLA, offering practitioners compelling trade-off between performance and an even smaller memory footprint.
Inference Speed Analysis. A direct empirical measurement of inference speed is heavily depen- dent highly optimized, hardware-spee speeifc kernels. To provide principled analysis without the substantial engineering effort required for integration into production-gras frameworks like vLLM SGLang, estimated the inference speed using the open-source InferSim toolkit Team Lab.
19

---

Table 8: Comparison of performance between MLA and MLA+FusedKV. Memory is shown as a
fraction of the MHA model's cache size. FusedKV demonstrates synergistic gains when applied to MLA, enabling further cache reduction with a minimal performance trade-off.
Cache Mem.
Wiki Text
LAMB-
Hella Swag
Avg
method
MNLI
SCIQ
ARC-E
ARC-C
MMLU
Acc
Acct
MHA
1
18.47
34.39
88.1
31.67
49.13
65.36
36.26
31.62
48.08
FusedVV
1/2
18.09
33.33
86.6
31.03
49.68
65.61
34.39
31.54
47.45
MILA
1/2
21.03
33.81
85.8
27.15
43.62
62.84
32.25
29.77
45.03
Fu.A+Fu
1/4
20.08
32.52
80.9
28.66
45.13
62.92
33.45
29.93
44.79
Table 9: Comparison of TTFT and TPOT between MLA, MLA+FusedKV, and MHA+FusedKV across varying input lengths for a 16-head model (1/O-bound). Our method significantly reduces prefill time (TTFT) with small overhead in generation time (TPOT).
TTFT (ms)
TPOT (ms)
Input/Oniput Length
MILL//LAAFFBBBR MRA-FusedKV)
2k/2k
25.14/20.221 18:21
23.12 27 45 90.36
4k/2k
47.6/31.52 27.18
32.00/ 39.21
134.32
8k/2k
103.07 59.29 50.64
27.52 34,01/
121.68
16k/2k
254.28 134.96 117.46
25.33 /1 31.46
116.16
32k/2k
366.31 331,42
113.36
64k/2k
2288.26 11552.37 1082.
23.76/ 29.62/ 110.08
12SK/2K
8010.93 4014..6 3874.62
23.58 29. 39) 109.32
256k/2k
29774.55 14897.16 14617.86
23.02 28.81/ 108.92
Table 10: Comparison TTFT and TPOT a 48-head model (compute-bound), In this scenario, the I/O overhead of F is hidden by computation, resulting in comparable TPOT to the base- line MLA while retaining the TTFT advantage.
Input/Output Length
TTFT (ms) (MLA/ MLA+F
TPOT (ms) (MLA MMLt=FusedK9)
2k/2k
35.04 / 25.28
51.35 55.35
4k/2k
73.29 /4446
77.99 77.99
8k/2k
180.61 / 198.22
68.29 68.29
16k/2k
516.42/ / 266.33
63.53 63.53
32k/2k
64k/2k
5917.23 2967. .9
60,08 60.08
128k/2k
22147.561 11084.81
59.64 59.64
256k/2k
85563.08 42795.90
58.79/58.77
(025). We analyze two key metrics: Time to First Token (TTFT) for the prefill phase and Time Per Output Token (TPOT) for the decoding phase.
1/O-BOUND SCENARIO. first analyze a 16-head model configuration, which typically repre- sents I/O-bound scenario. As detailed in Table applying FusedKVV significantly improves prefill speed. MLA+FusedKV reduces TTFT to nearly half that of MLA alone, a benefit stemming from the smaller cache size that needs to be written memory. For decoding, MIA+FusedKV incurs minor TPOT increase (approx. 4-6 ms) over MLA. This slight overhead is due to the additional I/O operations required for fetching tensors from multiple layers perform the fusion step during each token's 's generation.
COMPUTE-BOUd BOUND SCENARIO. As established in recent work Wang et al. (2025): Chang et al. 2024), when model's Arithmetic Intensity(Al) exceeds the GPU's compute-to-memory ratio, the latency from I/O communication can be effectively hidden by computation. For MLA, the Al is ap- proximately x heads Wang et al. (2025). To demonstrate this principle, we configured a model with 48 attention heads, creating a compute-bound scenario where the AI of MLA+ FusedKs surpasses NVIDIA H20 GPU's compute- to co-memory ratio. As shown Table I0 in this setting, the additional I/O latency from fusion step is fully overlapped by the increased computation. Consequently, while the prefill (TTFT) benefits remain, the decoding speed (TPOT) becomes identical to that of the standard MIAA model.
20

---

A.6.2 COMPATIBILITY WITH GQA
Beyond MLA, another widely adopted architectural optimization for improving inference efficiency is Grouped-Query Attention (GQA) Ainslie et al. (003)) To investigate synergistic effects with GQA, we conduct experiments on a 650M-parameter model, combining FusedKV with GQA architecture. All training parameters were aligned with those used in Table Our results, presented TT] demonstrate that FusedK V is highly compatible with GQA and their combination yields compelling performance-cost trade-off.
Table 11: Performance comparison demonstrating the synergy between V and GQA. The combined method FusedKV+GQA achieves a 4x cache reduction and substantial speedup, while mitigating the accuracy loss of GQA alone.
Wiki Text
LAMB-
Hella Swag
Avg. Acc²
Cache
method
MNLI
SCIQ
ARC-E
ARC-C
MMLU
TPOT
Acc
Mem.
Vanilla (MHA)
18.47
34.39
88.1
31.67
49.13
65.36
36.26
31.62
48.08
1.000
GQA
19.05
33.05
86.1
30.64
48.58
65.28
34.13
31.51
47.04
0.512
FusedKV
18.09
33.33
86.6
31.03
49.68
65.61
34.39
31.54
47.45
1.408
1/2
FusedKV+GQQ
18.48
34.11
85.3
30.16
49.04
64.87
35.67
30,92
47.15
0.741
1/4
A.6.3
COMPATIBILITY WITH MoE
Experimental Setup. We conducted an experiment on an MoE (Shazeer et al. 2017) model vari- ant. Starting with the 650M-parameter dense model, we replaced its FFN layers with 8-expert, top-2 gating MoE structure. This model was then trained on a corpus of 100B tokens. Due to the limited model scale and training data available for this specific experiment, our analysis focuses on foundational metrics: training loss and Wikitext perplexity.
Performance comparision. The results, presented in Table 12 demonstrate that our meth- ods maintain strong competitive advantage within the MoE framework. Both FusedKV+MoE and
outperform the cross-layer sharing baseline YOCO+MoE. Notably, FusedK MusedKV+MMr Moo achieves the best Wikitext perplexity, while FusedKV-Lite+ secures the lowest training loss.
Table 12: Performance comparison on Mixture-of-Experts (MoE) model. Our methods outperform baseline sharing techniques, demonstrating their applicability sparse architectures.
Method
WikiText
Train Loss,
VanillaMMoE
2.513
GQA+MoE
19.71
2,535
YOCO+MoE
20.17
2.544
Fused Mit+MoB
19.58
FusedKV+MoE
19.50
2.531
A.6.4 COMPATIBILITY WITH SLIDING WINDOW
A prominent hybrid architecture is the integration of Sliding Window Attention (SWA) Jiang et al. (2023) with full attention. In this section, we investigate the adaptability of FusedKV and FusedK Lite to hybrid model that combines both SWA and full attention.
Experimental Setup. To explore this, we construct a hybrid architecture based on 650M- parameter, 16-layer model. The architecture interleaves SWA layers with full attention (FA) layers in repeating pattern:
Window Attention Layers (Lw)): (1,2,3,5, 7,9, 10, 11, 13, 14, 15), using a window size of 512.
Full Attention Layers (Ly): {4, (4,8,17 12, 16).
We design two integration strategies for applying our methods to the top-half layers of the model: (+W): KV cache reconstruction/Sharing is applied only the Sliding Window Attention layers.
21

---

Table 13: Performance comparison on a hybrid SWA architecture. Best result per column is in bold. Our methods show strong compatibility, significantly reducing cache memory while maintaining or improving the average downstream score.
Method
Cache
Wikiffet
MNLI
SCIQ
LAMB-Aw
HellaSwag
ARC-E
ARC-C
MMLU
Average
Baseline (Hybsid)
29.7%
24.04
34.38
87.1
31.88
49.35
65.19
35.41
31.58
47.84
V-sed
25.4%
30.23
Fused Vitite +WF
12.9%
32:40
35.26
89.3
31.26
50.20
66.79
37.03
31.78
48.80
FusedKV +W
25.4%
28.34
33.75
86.4
32.23
50.12
68.14
36.35
31.37
48.34
FasedKV +WF
12.9%
29.63
34,00
83.7
32.49
50.49
68.01
37.20
31.30
48.85
(+WF)) KV cache reconstruction/shariig is applied to both Window and Full Attention layers in the top-half layers. The specific rules for the more comprehensive +WF strategies are defined as follows:
FUSEDK V-LIEE +WF the target layer's type:
This strategy employs direct sharing from distinct source layers based on
For SWA layers: K" = KT V'= v1 y== v4
if > and i E Tw if > 8 and i L
For FA layers: K' = K.
F + +WF This strategy uses learnable fusion, again respecting the layer types:
For SWA layers: K' F(K', K"),
v'= =(V V*), For FA layers: K* (KKT,*) V PT=4, Vo),
if and i Lw
if and i L,
Performance on Hybrid Architecture The results, presented in Table 13 demonstrate that both VusedKV and FusedKV-Lite can be effectively adapted hybrid architectures, yielding not only sig- nificant memory savings but also competitive or even superior performance on downstream tasks. While a slight degradation in WikiText perplexity is observed when applying our methods, the av- erage performance across the eight downstream benchmarks is consistently improved. These results strongly suggest that our cross-layer fusion and asymmetric sharing are not limited to uniform ar- chitectures but is flexible and powerful technique that can be tailored to advanced hybrid models.
A.6.5 COMPATIBILITY WITH OTHER EFFICIENCY TECHNIQUES
A key measure of a new efficiency technique's 's utility is its ability to compose with existing methods. To evaluate the broader applicability of FusedKV V and Fused V-Lite, investigate its compatibility with two prevalent model compression strategies: quantization (Liu et al. 2024c) and pruning (Sun a..).
Interaction with Quantization Quantization reduces memory footprint by representing weights and activations with (ower-precision data types. We explored the orthogonality of our method by combining FusedKV and V-Lite with Kivi (Liu et al. 2024c). The experiments were con- ducted on 1.5B B-B-aarmettrr model, applying 4-bit and b-b quantization to the KV cache.
The results, presented in Table 14 are highly encouraging, When combined with 4-bit quantization, both FusedKV and FusedK Lite maintain their performance with almost no degradation (an aver- age accuracy drop of only 0.2 points). This combination achieves a substantial 4x reduction in the final cache memory footprint on top of the savings from our method itself. This demonstrates that V and FusedK is largely orthogonal to quantization.
Interaction with Pruning Model pruning reduces the number of parameters by removing weights deemed unimportant, which directly translates to lower model storage and potentially faster compu- tation. When combining our methods with Wanda (Sun et al.). we specifically pruned 50% of the Wanda model's parameters using unstructured approach.
As shown in Table 15 a post-hoc application of pruning to our method resulted a noticeable degra- dation in performance across all benchmarks. This suggests that while method can be combined with common pruning methods, a more sophisticated co-design or joint optimization strategy may be required to mitigate the compounded performance loss.
22

---

Table 14: Performance of V and Fused Lite combined with 2-bit and 4-hit quantization.
With 4-bit quantization, performance is maintained while reducing cache memory by 4x.
Method
WikiText,
MNLI
SC1Q
LAMB-Ace
HellaSSwg
ARC-E
ARC-C
MMLU
Average
Cache Mem.
FusedKV (FP16)
13.33
37.43
93.5
43.33
74.20
44.20
36.18
55.82
4-bit Quant.
13.39
37.37
93.4
42.73
61.78
73.86
43.94
36.30
53.63
1/4
2-bit Quant.
15.49
34,89
90.7
25.42
59.10
70.24
40.61
34.11
50.73
1/8
ViLit (FPI6)
4-bit Quant.
13.53
36.41
93.9
40.85
60.70
74.03
43.60
36.36
55.12
1/4
2-hit Quant.
16.33
36.16
91.5
20.05
58.71
68.94
42.58
34.03
50.28
1/8
Table 15: Performance of V and FusedK combined with Wanda (50% unstructured pruning). A naive combination leads to significant performance drop.
Method
WikiTest
MNII
SCIQ
LAMB-Ace
HellaSwag
ARC-E:
ARC-C
MML.U
Average
Model Storage
FusedKV
13.33
37.43
93.5
43.33
61.88
74.30
44,20
36.18
55.82
FusedKV Wanda
32.76
51.09
63.85
48.56
1/2
PusedKV-L.it
13.45
36.43
93.7
41.61
60.77
74.79
43.52
36.29
55.30
FosedK Wanda
21.12
34.57
91.3
27.50
49.59
67.51
36.09
31,51
48.30
1/2
Table 16: Performance comparison the RULER benchmark for L.SB-parameter models with 128k context window. Our methods, FusedKV and FusedK V-Lite, retain a significant portion of the vanilla model'ss long-context performance, far surpassing the competing compression method YOCO.
plopk.1
guivigaiga
Pu.spadd
40:11
91.
75.67
94.1
654
15.1
206
27.0
100
$2.0
2001
21,1
12.5
43:51
76.1
13
918
188
675
283
900
189
200
150
38
Table 17: Ablation study on key and value source indices for FusedK V-Lite. The top layers reuse caches from the specified source layers. The results confirming our design hypothesis of FusedKV. Lite. Best results in each group are in bold.
Method
Valid Loss,
WilaiTsst
MNLI
SCIQ
LAMB-Ace
HellnSwag
ARC-R
ARC-C
MMLUU
Average
Vanilla (MHA)
2.483
18.47
34,39
88.1
31.67
49.13
65.36
36.26
31.62
48.08
Ablaztonn Viluer Source Indox valuelkeya (FtttdKY-Lite)
Layer 8)
value Sicey8
2.489
18.62
34,06
$8.0
31,86
48.82
66.46
37.37
30.99
48.22
value 0ieyy (Symmetric)
2.498
19.21
34.96
36.7
30.74
18.65
61,11
34.39
30.66
47.18
Ablotion Key Source Index /Vatue Source Laper 2)
valuelkeyd (FusedkV-LLite)
vall keys
2.473
34.36
31.72
valuel kkey3
2.479
18.60
34.28
87.1
31.42
49.19
66.16
36.95
30.89
48.00
A.7 LONG CONTEXT PERFORMANCE
The primary benefit of KV cache optimization is most pronounced in narios. To rigorously validate the efficacy of our proposed methods under such conditions, we evaluated FusedKV and FusedKV-Lite models with an extended context window.
inference
Context Window Extension. We extended our 1.5B-parameter models to support a 128k context window. We employ two-stage training strategy. In the first stage, extend the context length to 32k with 50B tokens, and the RoPE base to 50,000. In the second stage, we further extend the context from 32k 128k tokens with an additional 25B tokens, and the RoPE base is 500,000.
Performances. We evaluate models on the RULER benchmark (Hsich et al., 2024). The results, presented in Table 16. demonstrate the effectiveness of our approach in preserving long-range de- pendencies. This stark difference highlights the effectiveness of FusedKV and Fused While some information loss is inevitable with any form of cache compression, our methods demonstrate a significantly more graceful trade-off, retaining critical long-range information that is essential for practical long-context applications.
23

---

A.8 ABLATION STUDY ON ASYMMETRIC KV CACHE SHARING
A core principle of our FusedK method is the hypothesis of asymmetric cache sharing: that the optimal source layers for reconstructing keys and values are different. To systematically validate this hypothesis and quantify the impact of source layer selection, we conducted a detailed ablation study on 650M -5rrmmeeee,, 16-layer model. For this study, the top layers (layers 9-16) do not generate their own KV caches. Instead, they reconstruct them by sourcing from single, earlier layer. We independently vary the source layer index for the key and value caches isolate their respective contributions.
Importance of Low-Level Value Information. The first part of the ablation (varying the value source) reveals clear performance trend: valuelkey8 > value5key8 value8key8. Sourcing the value cache from the earliest layer (Layer 1) consistently yields the best average perfor- mance and validation loss. As the value source layer index increases and approaches the key source index, performance systematically degrades. The fully symmetric configuration which is YOCO, shows the most significant performance drop. This confirms that for reconstruct- ing the value cache, which contains content information, sourcing from foundational, lower-indexed layers is demonstrably superior.
8),
Importance of High-Level Key Information. The second part of the ablation (varying the key source) shows that performance declines as the key source index decreases: valuelkey > valuelkey5 > valuelkey3 This indicates that higher-indexed keys, which have undergone more layers of processing and presumably contain richer semantic and positional context, are more informative for reconstructing the attention mechanism in the top layers. However, it is notewor- thy that the performance degradation from varying the key source index is less pronounced than when the value source is changed. This observation aligns with analysis in Figure which sug- gests that the contribution weights for keys are more diffusely distributed across layers compared to values.
In summary, this ablation study validates that the optimal configuration for KV cache reconstruction indeed asymmetric. The value cache benefits most from low-level information, while the key cache benefits most from high-level information. Our FusedKV-Lite design, which reuses caches from Layer and Layer 8, represents a well-justified and empirically validated trade-off.
24


Reviews:
[{"review_id": "crZahH43Ri", "reviewer": "Reviewer_j1dt", "summary": "This paper proposes FusedKV and FusedKV-Lite as techniques to reduce key-value cache memory in transformer decoder-only models. The key contribution is identifying that top-layer key-value caches can be asymmetrically reconstructed: values are predominantly derived from the bottom layer, while keys benefit from information spanning both bottom and middle layers. FusedKV learns a dimension-wise weighted fusion of caches from these two source layers, operating on post-RoPE keys to preserve relative positional information without recomputation. FusedKV-Lite offers a simpler variant that directly reuses the middle-layer keys and bottom-layer values, eliminating fusion overhead. Across model scales from 332M to 4B parameters, both methods reduce cache memory by 50 percent while achieving lower validation perplexity than standard transformers, establishing an effective cross-layer cache sharing paradigm.", "questions": "- How does this paper's method compare quantitatively against MLA in terms of accuracy, cache memory footprint, and end-to-end inference speed? Since MLA reduces cache size structurally rather than through reconstruction, this comparison would establish whether cross-layer fusion offers advantages over fundamental architectural changes, or whether practitioners should adopt MLA instead. What are the performance-cost tradeoffs?\n- How does this paper's approach interact with token-level or head-level cache reduction? If combined with SpindleKV or head-importance pruning, could further memory savings be achieved without degradation? This clarifies orthogonality versus redundancy with complementary approaches.\n- Does performance degrade significantly on extremely long sequences (>100k tokens) or knowledge-intensive retrieval tasks? Stress testing on long-context QA would strengthen robustness claims across diverse inference scenarios.\n- What is the training overhead compared to vanilla training? How do learnable fusion weights affect gradient flow, training stability, or convergence speed? Reporting training time, memory, and loss curves would clarify practical costs.\n- Can the asymmetric sharing principle generalize to encoder-decoder, mixture-of-experts, or other architectural variants? Current evaluation is limited to dense decoder-only models.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The asymmetric key-value sharing principle is a clear, well-motivated insight grounded in empirical analysis (Figure 2), distinguishing this work from prior indiscriminate cross-layer approaches.\n- Solid experimental validation across multiple model sizes (332M to 4B) with consistent gains over baselines. Validation loss curves (Figure 6) demonstrate stability across training.\n- Practical design: operating on post-RoPE keys elegantly preserves positional information without computational overhead.\n- Comprehensive ablation studies (Figure 8) clearly demonstrate the importance of asymmetry direction and learnable weights.\n- Efficient implementation details provided, including Triton kernels and complexity analysis (Table 1).", "weaknesses": "- Missing fundamental architectural comparison: This paper does not compare against MLA, which reduces cache dimensionality by design rather than reconstructing across layers. For practitioners optimizing cache size, adopting MLA is a more principled solution than reconstructing caches across standard multi-head attention layers. Direct comparison (accuracy, memory, inference speed) would clarify when cross-layer fusion is preferable to architectural redesign -- and particularly the potential of the combination (do the gains stack?). After all, shouldn't we start improving on top of the best known methods?\n- Limited scope of baselines: Missing comparisons to recent token-level or head-level reduction methods such as SpindleKV (2025), Value Residual Learning (2025), and Mixture-of-Recursions (2025), which operate on complementary dimensions.​\n- Evaluation restricted to standard language modeling and GLUE-like benchmarks. The most KV-cache intensive scenarios are missing from the evaluation: a) truly long-sequence tasks (>100k tokens); b) multi-modal or encoder-decoder architectures.\n- No analysis/discussion of training cost or convergence properties relative to vanilla training. Does learnable fusion increase training time or memory significantly?\n- Potential interaction with other efficiency techniques unclear. Can this paper's method be orthogonally combined with quantization, pruning, or token merging for further gains?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cdHWT6eIjb", "reviewer": "Reviewer_K5c9", "summary": "This paper proposes a new KV cache sharing algorithm, FusedKV. By leveraging their findings on key-value reconstructions, they suggested which layers should be connected for sharing kv pairs. This ends up sharing values from bottom layers and keys from middle layers. They tested two variants, with and without learnable weights.", "questions": "See above Weakness parts.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Drawing insights from preliminary experiment (dense fusion) seems a good research direction to build up their own methods.\n- Measure various metrics to compare methods.\n- Validation at scale (larger model sizes or larger training token numbers) seems good.", "weaknesses": "- I feel like FusedKV is not a good approach due to its lower throughput, which comes from its higher KV IO, and marginal quality differences over FusedKV-Lite.\n- Proposed method is too heuristic. Although I personally like heuristic, simple yet effective method, I'm not sure about the generalizability of this proposed method. Using the first and middle layers as the source layer can be not optimal for other architectures (or with other modality). It seems like well-tailored, specific sharing strategy for this scenario.\n- In Figure 9, why do you think dense fusion outperforms vanilla models? Don't they (vanilla) have larger degree of freedom (i.e., larger trainable parameters) than reconstructing caches?\n- In L.413-414, isn't this typo? Rev means K^i = K^1, V^i = V^8?\n- I believe overall presentation could be enhanced, like Figure captions or visualization of results).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U150fMhzRo", "reviewer": "Reviewer_7x2j", "summary": "This work proposes a training-based KV cache compression method that approximates top-layer caches by fusing caches from bottom layers. Two options are proposed: FusedKV, which creates a weighted sum of bottom and middle-layer caches to achieve better quality at the cost of some inference overhead, and FusedKV-Lite, which directly reuses a cache from a single, lower layer. The experiments are conducted on a range of models from 300M to 4B parameters, using training budgets on the order of 100B tokens.", "questions": "* How well would the proposed method perform in a post-training setting? For example, could one take a pre-trained model and simply estimate the fusion coefficients for FusedKV (or directly reuse bottom-layer caches for FusedKV-Lite) without full retraining?\n\n* Do I understand correctly that the trained models use Grouped-Query Attention (GQA) with 16 query heads and 2 KV heads? Or is the number of query heads and KV heads the same in experiments?\n \n> For evaluation, we adopt GQA with 128 query heads and 2 key–value heads as the baseline \n\n* What is the head dimension? This configuration is unusual, as large models typically have 8 or more KV heads. A setup with only 2 KV heads is more common for smaller models (e.g., those with 16 or 32 query heads).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The proposed method reduces KV cache memory usage by 50% without any performance drop relative to the baseline.\n\n* The observation of KV cache asymmetry appears to be novel within the compression literature.", "weaknesses": "* The overall approach lacks conceptual novelty. The idea of cache reuse has been explored in several prior works [1, 2] in a post-training setting, where cache similarity was used as the criterion for reuse. Approaches based on linear predictors [3] have achieved 4x compression with no performance drop and up to 8x compression with only minor degradation.\n\n* The idea of a weighted fusion of previous keys and values also appeared in [4] in the context of improving gradient flow. It is possible that the improvement in validation loss observed in this work stems from the same mechanism.\n\n* The proposed approach requires costly training from scratch, whereas numerous inexpensive post-hoc techniques already exist that achieve high KV cache compression rates [7, 8] with little to no performance decline (see 1st weakness).\n\n* The most practical application for KV compression techniques is in long-context tasks, where the KV cache incurs significant memory overhead and increases inference latency. To fully demonstrate the technique's efficacy, the authors should provide results on established long-context benchmarks, such as LongBench [5] or RULER [6].\n\n* *(minor)* For the smaller models, performance on benchmarks like MMLU (25% random guess) and ARC-C (25% random guess) is only slightly above chance. Given the high noise and potential for random results, along with the cost of estimating standard deviation, the authors might consider removing these tasks for the smaller models.\n\nReferences\n\n---\n\n[1] Yang Y. et al. Kvsharer: Efficient inference via layer-wise dissimilar kv cache sharing //arXiv preprint arXiv:2410.18517. – 2024.\n\n[2] Liu A. et al. Minicache: Kv cache compression in depth dimension for large language models //Advances in Neural Information Processing Systems. – 2024. – Т. 37. – С. 139997-140031.\n\n[3] Shutova, Alina, et al. \"Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models.\" arXiv preprint arXiv:2501.19392 (2025).\n\n[4] Pagliardini M. et al. Denseformer: Enhancing information flow in transformers via depth weighted averaging //Advances in neural information processing systems. – 2024. – Т. 37. – С. 136479-136508.\n\n[5] Bai Y. et al. Longbench: A bilingual, multitask benchmark for long context understanding //arXiv preprint arXiv:2308.14508. – 2023.\n\n[6] Hsieh, Cheng-Ping, et al. \"RULER: What's the Real Context Size of Your Long-Context Language Models?.\" arXiv preprint arXiv:2404.06654 (2024).\n\n[7] Liu, Zirui, et al. \"Kivi: A tuning-free asymmetric 2bit quantization for kv cache.\" arXiv preprint arXiv:2402.02750 (2024).\n\n[8] Hooper, Coleman, et al. \"Kvquant: Towards 10 million context length llm inference with kv cache quantization.\" Advances in Neural Information Processing Systems 37 (2024): 1270-1303.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ojyxqj7Bpz", "reviewer": "Reviewer_SbV3", "summary": "This paper introduces FusedKV and FusedKV-Lite, two methods for cross-layer KV cache reconstruction in Transformers to reduce memory footprint and prefilling cost. The approach leverages an observed asymmetry between keys and values—where higher-layer keys draw from mid-layer information, while values rely more on lower layers. Both methods halve KV cache memory and achieve 2× acceleration in prefilling while maintaining comparable perplexity to standard Transformers. Experiments across model sizes show consistent performance gains and favorable scaling behavior.", "questions": "1. Given that FusedKV-Lite already performs well, what is the intended use case for the heavier FusedKV model?\n2. How compatible is FusedKV with GQA methods? Can they be combined effectively, and what would be the impact on speedup and accuracy?\n3. The paper notes that KV sharing typically reduces parameters in MHA settings. Does the proposed design introduce additional parameters (e.g., in FFN layers or fusion weights), and if so, what is their quantitative effect?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The evaluation is extensive, covering multiple model scales, diverse tasks, and scaling laws. The inclusion of a scaling experiment convincingly demonstrates the effectiveness and generality of the approach.\n2. FusedKV introduces only a small architectural modification, where cross-layer fusion through linear weighting, and achieves measurable efficiency gains. The method integrates smoothly with existing architectures and remains conceptually clear.", "weaknesses": "1.\tThe paper focuses on achieving ~2× cache and prefilling acceleration, while prior methods like YOCO already explore two-stage attention designs (prefill and decode) achieving greater long-sequence efficiency. This limits the novelty in acceleration.\n2. Table 2 shows that FusedKV-Lite already provides strong accuracy and latency trade-offs. The heavier FusedKV variant adds learnable parameters and I/O overhead but yields only marginal improvement, making its necessity unclear.\n3. Since KV cache architecture primarily affects long-sequence efficiency, the evaluation mainly on perplexity and standard benchmarks doesn’t fully validate the benefit under ultra-long input settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)