Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads
Authors: Zhoutong Wu
Yuan Zhang
Yiming Dong
Chenheng Zhang
Cong Fang
Kun Yuan
Zhouchen Lin
Venue: NeurIPS 2025 poster ()
Keywords: Transformer architecture
representation
KV cache
skip connections
Primary area: deep_learning

Abstract: Transformer models have driven breakthroughs across various language tasks by their strong capability to learn rich contextual representations. Scaling them to improve representation, however, often demands substantial memory and compute costs, such as the Key-Value (KV) cache used during auto-regressive decoding. Skip connections offer a promising way to improve representation without bloating resource usage, yet most prior works either improve expressivity while leaving KV costs unchanged, or reduce memory at the cost of weaker representation. In this work, we propose SkipV1Former, a  Transformer variant that uses skip connections from the first layer's Value heads to strengthen model representation and reduce KV cache.  Specifically, from the second block onward, each layer reuses half of its Value heads from the very first layer, while computing the other half as usual-cutting Value projections and V cache by nearly 50 \%. Theoretically, we show that routing uncompressed first-layer Values into deeper layers restores information lost to compression and accelerates the model’s implicit mesa-optimization-a key pattern of Transformer in auto-regressive tasks. Empirically, across different model scales, SkipV1Former delivers consistent reductions of approximately 25 \% in KV cache while improving perplexity relative to standard Multi-Head Attention (MHA) Transformers and some advanced variants. Moreover, we propose a recipe for uptraining existing MHA Transformer checkpoints to SkipV1Former with only 10-15\% additional compute. Finally, SkipV1Former can seamlessly combine advanced methods like Group-Query Attention and Multi-Latent Attention to achieve further KV cache savings and performance improvement. When combined with YOCO, it cuts KV cache size by nearly 50 \% while still improving performance. The code is
 available at: https://github.com/Zhoutong-Wu/SkipV1Former.

Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads
Zhoutong Wu', Yuan Zhang', Yiming Dong", Chenheng Zhang?, Cong Fang?.3,
Kun Yuan Zhouchen Lin2,5,6'
{ztwu, zy1002, chenhengz) Bariggzz@ttt..e pku. euuu..ddu.c.. cn, (fangcong, kunyuan, zlin)@pku. edu. cn, yimingdong. www.:mmmlooooooooommmmmmooo
1 Academy for Advanced Interdisciplinary Studies, Peking University 2 State Key Lab of General AI, School Intelligence Science and Technology, Peking University AI for Science Institute, Beijing, China
National Engincering Laboratory for Big Data Analytics and Applications
5 Institute for Artificial Intelligence, Peking University
Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China
Abstract
Transformer models have driven breakthroughs across various language tasks by their strong capability to learn rich contextual representations. Scaling them to improve representation, however, often demands substantial memory and compute costs, such as the Key-Value (KV) cache used during auto-regressive decoding. Skip connections offer a promising way improve representation without bloating resource usage, yet most prior works either improve expressivity while leaving KV costs unchanged, or reduce memory at the cost of weaker representation. In this work, we propose SkipV [Former, Transformer variant that uses skip connections from the first layer's Value heads strengthen model representation and reduce KV cache. Specifically, from the second block onward, each layer reuses half of Value heads from the very first layer, while computing the other half as usual- cutting Value projections and V cache by nearly 50 %. Theoretically, we show that routing uncompressed first-layer Values into deeper layers restores information lost to compression and accelerates the model's implicit mesa-optimizationn2 key pattern of Transformer ruto-regressive tasks. Empirically, across different model scales, SkipV / Former delivers consistent reductions of approximately 25 % in KV cache while improving perplexity relative to standard Multi-Head Attention (MHA) Transformers and some advanced variants. Morcover, propose recipe for uptraining existing MHA Translormer checkpoints to SkipVIFormer [Former with only 10- 15% additional compute. Finally, Skip Former can seamlessly combine advanced methods like Group- Suupp-uurr Attention and Multi-Laten Attention to achieve further KV cache savings and performance improvement. When combined with YOCO, it cuts KV cache size by nearly 50 % while still improving performance. The code is Wu.
available at: https:///:ii tiggthhub.... http:/hhouooog-W//Skppp
Introduction
Large Language Models (LLMs) built on Transformer architectures | have achieved impressive performance in wide range of language tasks such dialogue generation and complex reasoning.
'Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

At their core, multi-head attention (MHA) [ endows these models with powerful ability to capture rich contextual representations. While modern models are scaling to billions trillions of parameters in pursuit of ever-stronger expressivity, their memory and computational demands grow excessively large, limiting practical deployment. In particular, the need to cache large Key and Value (KV) tensors in auto-regressive decoding presents a notable challenge for GPU memory [5].
2 3 4
Moreover, recent empirical studies suggest that beyond certain scale, simply increasing model size yields diminishing returns in downstream performance [6]. As result, a growing body of works is focusing on methods that boost model performance without corresponding surge in resource consumption. One promising technique is (cross-layer) skip connections. which route features between non-adjacent layers promote reuse. Existing skip-connection techniques in the Transformer family mainly fall into two categories:
1. Feature-augmennt augmentation approaches: these methods concatenate or linearly fuse representa- tions from earlier layers into later layers 7., analogous to DenseNet ( or U-Net [[O] designs in convolutional networks. By reusing feature maps, these skip connections bolster model representation and improve performance. However, they fail reduce KV cache footprint compared to standard Transformer of the same width and depth.
2. Layer-replacement approaches: these techniques selectively replace portions of a layer's Key Value (or other components) with those drawn from other layers [2] They can reduce the KV parameter counts and thus the KV caching consumption. However, naively substituting layer outputs often leads to a drop in model quality, it often weakens the model representation.
This raises a natural question: Can we design Akip-connection scheme that simultancously enriches representations and reduces KV-cache/storage requirements? To achieve both goals simultaneously, one must decide which parameters are relatively redundant to be replaced, while identifying which intermediate features are critical to reuse for expressivity. In fact, by the No Free Lunch theorem [3]. there is no universal recipe for shrinking parameter counts and boosting representation across all tasks. Such dual gains must exploit both the architecture and the task characteristics.
To address the problem, we propose SkipV (skip connection with Value-1), a simple yet effective MHA Transformer variant that strategically integrates both objectives. In an L-layer SkipV 1 it replaces half of each layer's Value heads (layer to L) with the corresponding heads from layer 1. This design cuts Value cache size and the trainable Value projection parameters by nearly 50%, while preserving the total head count and architectural depth.
Based on the characteristics of auto-regressive tasks, we provide theoretical analysis of the expres- sivity of SkipV Former by viewing Transformers as mesa-optimizer [14, i.e., self-attention block can be interpreted as performing an optimization step latent objective function. We show that feeding uncompressed first-layer Value signals to deeper blocks via skip connections restores lost information and accelerates the mesa-optimization process.
Experiments on GPT-2 models demonstrate that SkipV consistently lowers perplexity compared to standard MHA Transformers and even matches or outperforms advanced feature- augmentation methods-wind ~25 % less KV cache. Under the same cache-saving regimes, significantly outperforms layer-replacement approaches like YOCO [15 and CLA [16]. Scaling to the 3B I.LMAA model, SkipV1 Former maintains improved perplexity and KV-cache efficiency versus the vanilla MHA Transformer.
Moreover, we propose an uptraining method to train a SkipV Ioorerr from the checkpoint of an MHA Transformer, using only around 10- 15% of original pre-training compute. Finally, SkipVIFormi can be seamlessly combined with other advanced K K--saving approaches like Group-Query Attention and Multi-Latent Attention to attain even greater savings in KV cache and performance boost. By combining the Key reusing mechanisms in YOCO, we further propose a variant of Skip Former that saves ~ 50% KV cache while improving model performance.
2 Related Works
Skip Connections in Transformers. Skip connections, which route the output one layer directly into deeper layers or sublayers, are utilized in the original Transformer for stabilizing training [I

---

Subsequent Transformer variants have leveraged skip connections to enrich representations by reusing and propagating low-level features into higher layers
19 20 leading improved performance. Some recent works further generalize skip connections to "hyper" connections that also fuse connections from width [9]] Another main thread leverages skip connections to reduce inference costs by parameter sharing or substituting [16] 15 21. In particular, sharing Keys and Values across layers yields a straightforward and effective K KV-cache reduction
[16] [5]. Other approaches approximate attention scores for similar efficiency gains (22). In addition, skip connections have also been applied to combine Pre-LN and Post-LN advantages [23] or mitigate over-smoothing problem (24]. Among the above, the most related to our approach is ResFormer [25], which linearly adds the first layer's Value to every subsequent layer's Value, acquiring lower perplexity but without theoretical grounding KV cache savings. Our FormerFormer Former simultaneously enhances expressivity and compresses KV cache, backed by both theoretical analysis and empirical validation.
KV Cache Efficient Methods 26
Key-Value (KV) cache, which stores the intermediate attention
Keys and Values during auto-regressive generation to avoid redundant computation, often dominates GPU memory usage. To alleviate this, range of compression techniques have been proposed,
including KV quantization [27 [28) efficient attention (2 30 B1], sparsity methods [32 33 34 [35]
and KV sharing
36
38, (9]. Our approach falls into the KV sharing category, reducing
the number of cache entries via feature reuse and substitution. Representative KV sharing methods
include Multi- calti-Query Attention (MQA) 36 and Group Query Attention (GQA) [. which share the Keys and Values among all or subset of heads in the same layer. Other works explore cross-layer reuse-recycling KV states from adjacent layers-such as YOCO [15), Cross-Layer Attention (CLA) [16]. and Layer-Condensed KV (LCKV) [21]. Some works also attempt to share latent cache across different layers [40]. SkipV SkipVIFoud Former is the first to reuse half of every layer's 's Value heads directly from Value-1, which boosts model performance using less parameters.
Transformer as Mesa-Optimizer. In studies on in-context learning of Transformers, recent works reveal that learned Transformers exhibit a "copying" behavior, whereby they aggregate or replicate context tokens into one representation [4), then implicitly perform optimization steps-akin to gradient descent 41 or Newton's 's method 45 on a context-dependent loss. Subsequent work has validated this meta-optimization view across a range of sequence- sequence-modeling tasks [14] 44). Building on the mesa-optimizer viewpoint, we analyze Skip V Former and show that its cross-layer Value skip effectively accelerates implicit optimization with uncompressed first-layer signals.
3 SkiprIFormer
3.1 Background and Notations
We begin by reviewing the architecture of multi-head attention (MHA) Transformer. Let X E Rddx be the input to Transformer block, where d is the embedding dimension and n is the sequence length. Neglecting layer normalization for clarity, a standard -ead self-attention sublayer followed by feed-forward network (FFN) is
Q* WXX,
Khh = ===W,X 4h= -WWx, h 1=1,... H,
H
Attn(X)= = x + Σ " h=1
softmax ( TQ²),
FFN(X) = X + W2 Rel,U(WiX),
where W E RRdxd Wh k- W Rdu Wi Rrxd W2 Rdxr, du is the dimension of each head and r is the dimension of the hidden layer. Stacking L such blocks yields the full Transformer architecture. We focus on decoder-only Transformers in this work.
3.2 Our Methods
Architecture. We now introduce our Skip I Former. In each layer i 2.... L., Skip Former interleaves half that layer's Value heads with the corresponding heads from layer while computing the remaining parts in the standard MHA manner. Formally, letting H' = H/2, the attention in block
3

---

Layer L
FFN Attention
0%
K1 K
V2
Layer 3
Layer 2
Attn Score
FFN
Attention
R: 0
K2
V,
Layer 1
Figure 1: Overview of the SkipV Former architecture. Left: A full L-layer decoder, where each layer from to interleaves half its Value heads with the corresponding heads from layer 1. Center: A zoomed-in view of the first and L-th layers, showing both the attention and the FFN sublayers. Right:
Detailed illustration of the cross-layer Value skip connection, with orange shading indicating the first-layer Value heads that are injected into subsequent layers' head sets.
i becomes
Attn(X) = X + Σ W
Q²))
E k=H++1
wh Wottsotta T²
A=1
where V refers to the value of the f-th head of the first layer. Figure gives illustration of the architecture SkipVIFormer.
Formerr. Specifically, we use fixed deterministic selection strategy: for layers 2-7 we always keep heads 0-.. H' and concatenate them with heads H' +1 H from layer 1. This continuity ensures a stable and aligned head ordering across layers, which more hardware-friendly compared other discontinuous methods (45] such random selection. We also compare other head injection strategies in Appendix C.4
Since half of the heads in layers 2 L are drawn from layer 1, SkipV IFormer cuts the number of Wy parameters and Value cache by about 50% (i.e., 25% KV cache) without changing the total head count H layer depth L. We further propose variant of Skip Former that utilizes the Keys sharing mechanism of YOCO (You Only Cache Once) [15] in Appendix C.5] which saves another 25% KV cache on top of Former (50% KV cache in total). We focus this paper Skip Former and leave the YOCO-based variant to Appendix C.5
Relations to Previous Methods. Conceptually, SkipVIForma can be seen an extension of ResFormer 25 which blends every layer's Value via scalar interpolation, i.e.,
Attn(X) X Σ
wh (Av + +1 -))') softmax (
for some À E R. In contrast, SkipV IFormer inserts H first-layer heads directly into each deeper layers, thereby saving nearly 25% of KV cache whereas ResFormer cannot. Section 4 we provide a theoretical analysis which not only motivates the specific design of SkipV but also offers insights into why ResFormer's interpolation with first Value benefits model performance.
Why Skip Connection with First Value Head Helps
While seems natural view SkipV Former as more expressive than a variant that only reduces half the heads without injecting first-layer Values, it is shown that skipping from the first layer's Value yields notably better performance than skipping from later layers (see Section 6.5) This suggests that the first layer is uniquely informative for information propagation.

---

MHA Transformer
FkomVrFormer
Meso-ppimizee
Mesa Optimbs
Deep Layers
By WAX
Deep Layers
y W
Rou Jane
Copying
Copying
Shallow Layers
Shallow Layers
PM
Figure 2: Standard MHA Transformer (left) compresses token pairs actoss layers, incurring informa- tion loss vesa-optimization. whereas SkipV V Former (right) reintroduces raw token values into deep layers, preserving information fidelity.
potential clue lies in the "copying mechanism" studied in in-context learning (ICL) [46]: early Transformer layers often act to bind multiple adjacent tokens into compact representations [14 47). This effect is especially prominent in shallow layers and underlies many ICL models' assumptions. For example, it is commonly assumed that input tokens take the form C, (xj-y;) Rd where X, Rdd and y, Rd are the sample and label of training pair. Under this formulation, Oswald et al. [41] show that, for linear regression problem, i.c., y = W x +E, a single linear attention layer approximates one-step gradient descent:
ej (X5; yj VT(W)x)), L(W)=
Wx yr
2N
That is, the layer updates the token if performing step of gradient descent on W to minimize the regression loss. Subsequent works [8 extend this mesa-optimizatiin view to deeper and
nonlinear settings 48 49 42
However, a key assumption in this analysis is that each token fully encodes its (x, y) pair- condition difficult to satisfy in practice due to limited embedding dimensionality. Compressing multiple tokens into one inevitably incurs information loss, which can lead to degraded quality of optimization updates in deeper layers.
SkipV Former mitigates this by explicitly routing the first layer's Value into every deeper layer. Since the first layer processes the raw input tokens directly, its Value retain higher- fideliyy representations of the sample and label. The skip connection with first-layer Value allows the model to restore much of the information lost through copying whereas existing residual connections cannot. This preserves more of the causal structure between samples and labels, which in turn enhances the model's reasoning capabilities. The overall intuition is illustrated in Figure 2 Moreover, prior works [48] show that mesa- resa-oopiiizzaiion gradients in Transformers primarily flow through the Value pathway. This supports our design choice to interleave deeper layers' Value with the first-layer Value.
We further formalize this intuition simplified model to derive concrete performance improvement. Consider the linear regression problem in Eq. (T). where x,, V: E Rd and W ~ (000 I2). Assume that the embedding dimension is d, so there will be information loss in the compression for MHA Transformer. For the model, consider a two-layer Attention-onle Transformer with two heads per layer and linear attention. Denote by L1 and L2 the expected squared-error loss on predicting y of MHA Transformer and SkipV Former, respectively. We show that under this setup:
Theorem 1 (Informal). Assume that the first layer performs the copying mechanism, then there exists an independent constant > O, such that
min Wh W...W.W...
Le (e(( W&. Wb), H)
min
L((W W W.), H)
W.. W...W.
W..H
5

---

Multi-Hiad Attention
Value Projection N?
Mean
Value Projection WV
Value Projection w7
Mean
Pool
Value Projection BY
Figure 3: Left: Uptraining checkpoint conversion: we apply mean pooling over every two Wy heads to reduce dimensional mismatch. Middle: Integration of SkipV Former with GQA. Right: Integration of S [Former with MLA; shaded regions follow the style of (50) and indicate cached components.
formal statement and proof of the theorem are provided in Appendix A Theorem D shows that SkipV ScipVIFormen Former can drive the prediction error at least c lower than MHA Transformer on this task. In the proof, we show that SkipVIForme can realize an "uncompressed" GD GD-jur just as in the ideal one- step update in Eq. (I). In this sense, SkipV Former performs equivalently as MHA Transformer with extra embedding dimension that can fully encode the y) pair. Further discussions are provided in Appendix
Extensions & Practical Recipes
5.1 Uptraining from Pretrained MHA Checkpoints
Modern large language models are typically released with standard MHA Transformer checkpoints. To train SkipVIFormes from MHA Transformer checkpoint, we follow similar uptraining strategy to that Ainslie et al. [[]] for Multi-Query Attention [36). Concretely, 1) Convert an MHA pool across every two Transformer checkpoint into Skip V Former checkpoint by inserting Former layers with these head projections in layers beyond the first. 2) Initialize the new SkipVIFormer
pooled weights while preserving the original first-layer Value projections and continue pretraining. An illustration shown in Figure 3 shown in Section 6.4 that this uptraining can match the perplexity of training from scratch using additional 10% 15% of the original compute budget.
5.2 Combining with KV-Cache--fffcient Methods
Our frst-value-head skip connection is compatible with most existing KV cache compression methods, such as sparsity methods and same-layer KV sharing methods. In this Section, we consider Group- Query Attention (GQA) [1T] and Multi-Latent Attention (MLA) 50 We also consider YOCO in Appendix C.5
Group-Query Attention: GQA divides query heads into groups, each sharing a single key and value head. To integrate, we first replace half of the Value heads with those from the first layer before grouping. Each group thus shares either a current-layer or first-layer Value head, enabling shallow-layer reuse without disrupting GQA semantics.
Multi-Latent Attention: MLA uses low-rank joint compression e for attention keys and values. To integrate with our technique, we interleave half of each layer's c-dimensional latent vector with the corresponding first-layer latent features, redueing the c-cache footprint.
See Figure for illustration. It is shown in Section 6.4 that combining our methods with GQA or MILA leads to another 25% or 50% reduction on KV cache and attaining lower perplexity. We also experiment TinyLlama- 1. IB model in Appendix B.2 and observe similar improvements.

---

Dacelre tre
Decelre
SepViTemer
SepuiTemer
140
Donue mime
Aaaherma
1.2
320
35c
33C
30
25
466
13C
236
256
466
(a) GPT2-125M Model
(b) GPT2-200M Model
2.25
Villdea temation
Byssire
Sept/
3.0
map
3.20
3.2
3.15
3320
3.
246
200
230
(c) GGP22335M Model
(d) GPT2GPP5550 Model
Figure 4: Validation loss curves across four GPT-2 model sizes (125M, 200M, 355M, 550M) for six architectures plotted over training iterations. Solid lines indicate KV-cache efficient variants (YOCO-V, CLA-V, SkimVIFormer), while dashed lines correspond to non- we----Va---efffiiiiie models (MHA Transformer, DenseFormer, ResFormer).
Experiments
In this section, we evaluate SkimYrFormer Former on GPT-2 [3] and LLaMA models [4, examining primarily pre-training dynamics and losses, inference memory, uptraining, and integration with KV-efficient techniques.
6.1 Experiments GPT-2 Series
Setup. We pretrain the models on OpenWebText2 [51], an open-source replication of OpenWeb TexCCorpus comprising around 17B tokens. We also evaluate the pretrained checkpoints standard downstream tasks in a zero-shot manner. We experiment on four model scales: 1) GPT2-S (125 M) and GPT2-M (355 M), following the configurations in Radford et al. [ 2) Two additional intermediate variants for finer granularity: 200M and 550M.
set of
For the architectures, we choose MHA Transformer as the baseline model. We pick two recent regture-augmentatioo methods: FonseFormer 7 and ResFormer (25], and two KV-sharing methods: YOCO [1] and Cross-Layer Attention (CLA) [16 for comparisons. Too ensure fair comparison in the EV-cache-saving regime, we consider V-only YOCO and V-only CLA, aligning with FkipViFormer's design. All models are trained using Adam W. More details are provided in Appendix B.
The per-iteration validation losses for six architectures during pretraining presented in Former matches ResFormer in terms of final loss acssvvchiiinng the lowest value overall. Importantly, SkipVIFormer Former consistently outperforms both the baseline model and FenseFormer, highlighting the benefit of our skip connection with the first-layer Value. the same KV-saving regime, SkipVIForma significantly surpasses the lightweight cache-efficient methods YOCO and CLA. Additionally, the validation-Il curve for Skip Former is as smooth as those the regular models, demonstrating that our skip connection does introduce any additional training instability.
Results.
Figure FFomerFormer
We further evaluate these pretrained models zero-shot setting across standard downstream tasks in Appendix 1Former and ResFormer yield the strongest average performance.
7

---

(a) Validation Loss and Perplexity
(b) Val. Loss vs. Model Size
Validation Loss
Model Size
60M
130M
350M
1B
3B
Baseline
2.4
Baseline
3.549
3.229
2.931
2.723
2.664
SXipVIForne
3,504
3.192
2.885
2,711
2.652
Perplexity
Model Size
60M
130M
350M
1B
3B
Baseline
34.78
25.25
18.75
15.23
14.35
FkomeeFormer
33.25
24.34
17.90
15.04
14.18
Parameters
Figure 5: Validation Loss, Perplexity, and Loss Curve for LLaMA Models scaling from 60M to 3B. Skip Former consistently outperformed MHA Transformer with with reduced KV cache and fewer parameters.
6.2 Experiments on LLaMA Series
Setup. We further compare SkipVIFormer Former and the baseline model MHA Transformer on LLaMA series to validate the effectiveness of our architecture. We pre-train C4 dataset, colossal, cleaned version Common Crawl's web crawl corpus [52]. We evaluate sizes from 60 M up to B parameters to test scalability. More details provided in Appendix B.2
Results. Table aad Figure5bjrepon validation loss and perplexity across all sizes. SkipVIFormet Former outperforms the baseline at every scale. On the small and medium models (60M-350M), it achieves notable reduction in validation loss of approximately 0.03-0.05. For larger models (IB-3B), SkipV Former maintains noticeable improvement regarding the model scale. Importantly, these performance gains come alongside a reduction of approximately 25% in KV cache usage and around 4% in total parameter count. Furthermore, Table reports the downstream task accuracy of the models, with results for other scales provided in Appendix C.3 SkipV1Former Former also surpasses the baseline in average downstream performance.
Table 1: Test accuracies (%) of 3B-scale models on downstream tasks, with overall average.
Model
ARC-C
ARC-E
BoolQ
Hella
OBQA
PIQA
RTE
SciQ
Winogr
Avg
Baseline
21.0
49,5
35.5
18.6
68.4
53.1
77.5
49.3
47.5
SkipVIForma
21.0
49.4
60.3
35.1
17.6
69.3
53.8
79.1
49.5
48.3
6.3 Inference
GPU Memory. To validate real-world savings, we measure and compare GPU memory consump- tion of SkipViFormer and a baseline MHA Transformer across different dimensions. All experiments are conducted on LLaMA variants using a single NVIDIA RTX A6000 GPU with 48 GB of memory, and the results are summarized in Figure
Figure 66 plots KV cache size (in MB) as a function of sequence length. While both SkipV IkipVIFormer Former and MHA Transformer exhibit linear cache growth, SkipV Former's slope is ~25 % lower than that of MHA across all LLaMA variants, aligning with our theoretical predictions. Figure 6b plots per-token KV cache (KB) against model size (ranging from 350M to 7B parameters). Again, SkipVIFormee maintains a ~5 % reduction at all scales, demonstrating uniform savings regardless of model capacity. Finally, Figure be provides a stacked breakdown of peak allocated memory in inference without activation offloading. SkipVIForma not only shrinks KV cache size but also lowers the overall peak memory by noticeable margin of 5.6GB.
Prefill Time and Throughput. We also evaluate inference speed in terms of prefill latency (time to build the KV cache) and decoding throughput (tokens/s with cached KV states). Figure reports reportss results across 60M-7B models. Skip V Former reduces prefill time by 5-8 ms/100 tokens due to halved Wy projections, and achieves throughput within +2% of the baseline at practical batch sizes (B = 8, 16), with occasional gains. These modest but consistent improvements are achieved without

---

XX Cache Sequence Lenge: for LLANA
KY Cedhs por tooe Nede Set
Peat Planas Brokllis infrome 113 Hedes
22
Cockee EAAAAT (281
20003
Ohem
33881
Sequenoe Longth 1000e1s"
Mode Size
5056
(a) KV cache size across different
(b) Per-token KV cache as func- tion of model size.
(c) Memory breakdown at inference for 1B LLaMA.
sequence length.
Figure Inference memory evaluation of SkipV Former vs. standard MHA Transformer on LLaMA.
Boar
100
Thouptout 0 5--2 1100
10
(a) Prefill (B=1): lower is better.
(b) Throughput (B=8).
(c) Throughput (B=16). 7B OOM.
Figure 7: Inference speed of SkipV Former vs. baseline.
the additional computational overhead typically introduced by alternative KV-cache compression methods (e.g.. SVD-based), underscoring Fforme''''r'' efficiency. This overhead is expected to shrink with dedicated fused kernels. More details are provided in Appendix B.2
6.4 Extensions
Uptraining. We uptrain SkipV IFormers from the 125M 355M GPT2-MHA Transformer check- points using our uptraining strategy in Section 5.1 Results are shown in Figure g and further experimental details are included in Appendix B.I
As shown in Figure uptraining requires only 10%-15% of the original compute budget to reach the same validation loss training from scratch. Furthermore, to match the final performance of fully trained MHA Transformer, SkipV S1ipV1Forme Former requires only 5 %-10 % of the original training compute. The checkpoint conversion process incurs negligible resources compared full training. We also compare alternative checkpoint conversion approaches in Appendix B.2
Combining with Existing Methods. The skip connection mechanism our S Former can be combined with other layer-rellacement or low-rank methods such GQA and MLA. We evaluate the SkipVIFormer Formerr-GQO and Skip SkipVIFormi FFrmerr MLA against their vanilla counterparts on the LlaMA 350M architecture Open WebText.. As shown in Table the SkipV [oormer- of GQA and
Velicatte Lova Lec
Upleiiiin
Unbetinng
Upbeniing
skperoore
332
227
(a) GPT2-125M Model
GPT2-200M Model
(c) GPT2-355M Model
Figure 8: Validation loss versus uptraining proportion for GPT2-125M, GPT2-200M, and GPT2- 355M models. The horizontal dashed lines mark the final validation losses of the pretrained baseline (blue) and SkipV ToroVFoormer Former (red) when trained from scratch.

---

Table 2: Performance and KV cache reduction when composing SkipV IFormer with GQA and MLA
on GPT2-355M. "A KV %" indicates cache reduction relative to the Non-SkipVI methods.
Models Transformer-GQAA
Val. Loss
PPL
Params
KV Bytes token
ÁKV
2.912
18.39
334.7 M
304 B 74 752 B
SKipV/Former-GIA
2.893
18.04
328.9 M
-24.0%
Transformer-MLA.
2.896
18.10
337.4 M
13 824 B
2.888
17.95
334.4 M
680
-44.4%
MLA achieve lower validation loss and reduce the KV bytes per token by a large margin. The total parameter count also reduced, benefiting from the lightweight nature of our skip mechanism. To further align with recent practice, we also experiment with GQA using group-of-4 configuration on TinyLLaMA models. Results Appendix B.2 show that SkipVIFormer Iormer continues to outperform the baseline under this setting, confirming the robustness of the method across different GQA schemes. Additional results on combining SkipV Former with YOCO are provided Appendix C.5
6.5 Ablations
Skip-Head Ratio. We vary the fraction of attention heads whose Value are replaced by first-layer Values-from 25 % up 75 %-on GPT2-355M. As shown in Figure 9a| skipping more than 50 % of heads begins to hurt validation loss, while skipping fewer than 50 % also underperforms the 50% setting while consuming more memory. Hence, skipping exactly half of the heads seems to achieve an optimal tradeoff between model quality and memory saving. Additional experiments on LlaMA- 1B models and discussions are provided in Appendix B.2
Relestve Validation Proportion Sclp
0.000
Relatwve loss
-0.005
341010
25.0%
37.55 17.5%
Propertion Sktp Mouth
0%%
62.3 5755%
75.0%
38030
30dcs
(a) Relative validation loss for cross-layer head ratio compared with baseline. The asterisk denotes the default ratio of skip heads.
(b) Validation loss per iteration for cross-layer at different layers and comparison with no skip (SkipVO) and baseline.
Figure 9: Comparison of validation loss under different head and layer configurations.
Skip Connection with Different Layers. We further investigate the performance of reusing Values of other layers (layer 2, 3, 4) as well as reusing no Values VO). As shown in Figure 96] reusing second-layer Values (SkipV2) brings only marginal gains, and removing skip connections while halving the Value projection (SkipVO) severely degrades performance. Extending the skip to deeper layers (SkipV3 V4 similarly yields weaker results than SIipVI. Together, these findings highlight the unique importance of first-layer Values for cross-layer reuse.
7 Conclusions and Future Works
SkipVIFormet demonstrates that strategic reuse of first-layer Value can simultancously improve the model's representation capability and reduce KV cache size. This design enables deeper layers to access information typically lost to aggressive compression, thereby enhancing the model's inherent mesa-optimization. Across different model scales, SkipV Former consistently outperforms MHA Transformer and related variants while using less KV memory. Its "uptraining" methodology and seamless integration with existing techniques further demonstrate its practical value and flexibility.
Looking forward, promising directions include kernel-level optimizations for faster inference, training- free transformations to retrofit existing models, and applications in fine-tuning. We leave these to future investigation and hope our study motivates broader exploration of cross-layer skip connections in scale Transformer architectures.
10

---

Acknowledgements
K. Yuan is supported by the NSF China under Grants 92370121, 12301392, and W2441021. Lin is supported by the NSF China under Grants 62276004 and the State Key Laboratory of General Artificial Intelligence.
References
[1] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in Neural Information Processing Systems, 2017.
[2] A. Radford, Wu, Child, D. Luan, D. Amodei, I. Sutskever et al., "Language models are unsupervised multitask learners," OpenAl blog, 2019.
[3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell el al., "Language models are few-shot learners," " Advances in Neural Information Processing Systems, 2020.
[4] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., "Llama: Open and efficient foundation language models," ArXiv preprint arXiv:2302. 13971, 2023.
[5] R. Pope, Douglas, A. Chowdhery, J. Devlin, Bradbury, Heek, K. Xiao, S. Agrawal, and J. Dean, "Efficiently scaling transformer inference," Proceedings of Machine Learning and Systems, 2023.
[6] Petty. S. van Steenkiste, Sha, Dasgupta, D. Garrette, and T. Linzen, "The impact of depth and width on transformer language model generalization," ArXiv preprint arXiv:2310.19956, 2023.
[7] M. Pagliardini, A. Mohtashami, F. Fleuret, and M. Jaggi, "DenseFormer: Enhancing information flow in transformers via depth weighted averaging," Advances in Neural Information Processing Systems, 2024.
[8] R. He, A. Ravula, B. Kanagal, and J. Ainslie, "RealFormer:
Transformer likes residual attention," "
in Findings of the Association for Computational Linguistics, 2021.
[9] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[10] O. Ronneberger, P. Fischer, and T. Brox, "U-Net: Convolutional networks for biomedical image segmentation," in Medical Image Computing and Computer-Assisted Intervention, 2015.
[11] J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai, "GQA: training generalized multi-query transformer models from multi-head checkpoints," Conference on Empirical Methods in Natural Language Processing, 2023.
[12] W. Brandon, M. Mishra, A. Nrusimha, R. Panda, and J. Bagan-Kelley, "Reducing transformer key-value cache size with cross-layer attention," Advances in Neural Information Processing Systems, 2024.
[13] D. H. Wolpert and W. G. Macready, "No free lunch theorems for optimization," IEEE Transac- tions on Evolutionary Computation, 1997.
[14] J. Von Oswald, M. Schlegel, A. Meulemans, S. Kobayashi, E. Niklasson, N. Zucchet, N. Scherrer, N. Miller, M. Sandler, M. Vladymyrov et al., "Uncovering mesa-optimization algorithms in transformers," ArXiv preprint 2xXi:2339900858 2023.
[15] Y. Sun, L. Dong, Y. Zhu, S. Huang, W. Wang, S. Ma, Q. Zhang, J. Wang, and F. Wei, "You only cache once: decoder-desoder architectures for language models," Advances in Neural Information Processing Systems, 2024.

---

[16] W. Brandon, M. Mishra, A. Nrusimha, R. Panda, and J. Ragan-Kelley, "Reducing transformer key-value cache size with cross-layer attention," Advances in Neural Information Processing Systems, 2024.
[17] F. Liu, X. Ren, Z. Zhang, X. Sun, and Y. Zou, "Rethinking skip connection with layer normal- ization in transformers and resnets," ArXiv preprint arXiv:2105.07200. 07200, 2021.
[18] Q. Chen, W. Wang. Q. Zhang. S. Zheng, S. Zhang, C. Deng. H. Yu, J. Liu, Y. Ma, and C. Zhang. "Skip-layer attention: Bridging abstract and detailed dependencies in transformers," ArXiv preprint arXiv:2406. 11300.11220 11274, 2024.
[19] D. Zhu, H. Huang, Z. Huang, Y. Zeng, Y. Mao, B. Wu, Q. Min, and X. Zhou, "Hyper- connections," " ArXiv preprint arXiv:2409. 19606, 2024.
[20] D. Dai, Y. Sun, L. Dong, Y. Hao, S. Ma, Z. Sui, and F. Wei, "Why can GPT learn in-context? language models secretly perform gradient descent meta-optimizers." in Findings of the Association for Computational Linguisties, A. Rogers, J. L. Boyd-Graber, and N. Okazaki, Eds., 2023.
[21] H. Wu and K. Tu, "Layer-condensed KV cache for efficient inference of large language models," in Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics, L. Ku, A. Martins, and V. Srikumar, Eds., 2024.
[22] S. Venkataramanan, A. Ghodrati, Y. M. Asano, F. Porikli, and A. Habibian, "Skip-attention: Im- proving vision transformers by paying less attention," in International Conference on Learning Representations, 2024.
[23] S. Xie, H. Zhang, J. Guo, X. Tan, J. Bian, H. H. Awadalla, A. Menezes, T. Qin, and R. Yan, "Residual: Transformer with dual residual connections," " ArXiv preprint arXiv:2304.1480. 2023.
[24] T. Nguyen, T. Nguyen, and R. Baraniuk, "Mitigating over-smoothing in transformers via regularized nonlocal functionals," Advances in Neural Information Prucessing Systems, 2023.
[25] Z. Zhou, Wu, Z. Jiang, and Z. Lan, "Value residual learning for alleviating attention concen- tration in transformers," ArXiv preprint arXiv:2410. 2024.
[26] X. Zhu, Li, Y. Liu, C. Ma, and W. Wang. "A survey on model compression for large language models," " Transactions of the Association for Computational Linguistics, 2024.
[27] C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, S. Shao, K. Keutzer, and A. Gholami, "KVquant: Towards 10 million context length LLM inference with KV cache quantization," " Advances in Neural Information Processing Systems, 2024.
[28] Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, Stoica, and C. Zhang, "Flexgen: High-throughput generative inference of large language models with a single GPU," in International Conference on Machine Learning, 2023.
[29] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, "Transformers RNNs: Fast autore- gressive transformers with linear attention," in International Conference on Machine Learning, 2020.
[30] R. Child. S. Gray, A. Radford, and I. Sutskever, "Generating long sequences with sparse transformers," ArXiv preprint arXiv: 10509, 2019.
[31] H. Wang, Z. Zhang, and Han, "Spatten: Efficient sparse attention architecture with cascade to- Computer
ken and head pruning." in 2021 IEEE International Symposium on Architecture (HPCA), 2021.
[32] Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett et al., "H20: Heavy- oracle for efficient generative inference of large language models," Advances in Neural Information Processing Systems, 2023.
[33] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, "Efficient streaming language models with attention sinks," in International Conference on Learning Representations, 2024.
12

---

[34] Z. Liu, A. Desai, F. Liao, W. Wang, V. Xie, Z. Xu, A. Kyrillidis. and A. Shrivastava, "Scis- sorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time," Advances in Neural Information Processing Systems, 2023.
[35] H. Kang, Q. Zhang, S. Kundu, G. Jeong, Z. Liu, T. Krishna, and T. Zhao, "Gear: An efficient KV cache compression recipe for near-lossless generative inference of LLM," ArXiv preprini MxXiv:2403055527 2024.
[36] N. Shazeer, "Fast transformer decoding: One write-head all you need." ArXiv preprint arXiv:1911.02150, 2019.
[37] Y. Wu, H. Wu, and K. Tu, "A systematic study of cross-layer kv sharing for efficient Ilm inference," ArXiv preprint (rXiv:2410.1444) 2024.
[38] Y. Yang, Z. Cao, Q. Chen, L. Qin, D. Yang, H. Zhao, and Z. Chen, "KVsharer: Efficient inference via layer-wise dissimilar KV cache sharing," ArXiv preprint arXiv:2410./85/7,, 2024.
[39] Z. M. K. Zuhri, M. F. Adilazuarda, A. Purwarianti, and A. F. Aji, "MLKV: Multi-layer key-value heads for memory efficient transformer decoding," " ArXiv preprint XXXx:244660092977 2024.
[40] Z. Yang, J. Han, K. Wu, R. Xie, A. Wang, X. Sun, and Z. Kang, "Lossless kv cache compression 2%."" " ArXiv preprint arXiv:2410.15252, 2024.
[41] J. Von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov, "Transformers learn in-context by gradient descent," in International Conference on Machine Learning, 2023.
[42] K. Ahn, X. Cheng, H. Daneshmand, and S. Sra, "Transformers learn to implement precondi- tioned gradient descent for in-context learning," Advances Neural Information Processing Systems, 2023.
[43] D. Fu, T.-q. Chen, R. Jia, and V. Sharan, "Transformers learn to achieve second-order con- vergence rates for in-context linear regression," Advances in Neural Information Processing Systems, 2024.
[44] X. Wu and I. R. Varshney, "A meta-learning perspective on transformers for causal language modeling," in Findings of the Association for Computational Linguistics, L. Ku, A. Martins, and V. Srikumar, Eds., 2024.
[45] J. Yuan, H. Gao, D. Dai, J. Luo, L. Zhao, Z. Zhang, Z. Xie, Y. Wei, L. Wang, Z. Xiao, Y. Wang, C. Ruan, M. Zhang, W. Liang, and W. Zeng. "Native sparse attention: Hardware-aligned and natively trainable sparse attention," in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, W. Che, Nabende, Shutova, and M. Pilehvar, Eds., 2025.
[46] Q Dong, I. Li, D. Dai, C. Zheng, J. Ma, R. Li, H. Xia, J. Xu, 7. Wu, B. Chang, X. Sun, L., Li, and Z. Sui, "A survey on in-context learning," in Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen, Eds., 2024.
[47] C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen et al., "In-context learning and induction heads," ArXiv preprint arXiv:2209.11880 2022.
[48] A. V. Mahankali, T. Hashimoto, and T. Ma. "One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention." in International Conference on Learning Representations, 2024.
[49] R. Zhang, S. Frei, and P. L.. Bartlett, "Trrined transformers learn linear models in-context," Journal of Machine Learning Research, 2024.
[50] A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Dengr, C. Ruan, D. Dai, D. Guo el al., "Deepseek-VP" V2: A strong, economical, and efficient mixture-of-experts language model," ArXiv preprint arXiv:2405.044 044444 2024.
13

---

[51] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe. C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima al., "OpenWebText" dataset, as part of 'the pile: An 800GB dataset of diverse text for language modeling"." ArXiv preprint arXiv: 10X.00200.00022, 2020.
[52] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer," Journal of machine learning research, 2020.
[53] C. Eckart and G. Young, "The approximation of one matrix by another of lower rank," Psy- chometrika, 1936.
[54] A. Edelman, "Eigenvalues and condition numbers of random matrices," " SIAM Journal on Matrix Analysis and Applications, 1988.
[55] X. Chen, Zhao, and Zou, "How transformers utilize multi-head attention in in-context learning? A case study sparse linear regression," Advances in Neural Information Processing Systems, 2024.
[56] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization, in International Confer- ence Learning Representations, 2019.
[57] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, "Flashattention: Fast and memory-efficient exact attention with io-awareness," Advances in Neural Information Processing Systems, 2022.
[58] J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Y. Tian, "GaLore: Memory- efficient LLM training by gradient low-rank projection," " in International Conference on Machine Learning, 2024.
[59] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu, "RoFormer: Enhanced transformer with rotary position embedding," Neurocomputing. 2024.
[60] G. Alain and Y. Bengio, "Understanding intermediate layers using linear classifier probes," in International Conference Learning Representations. OpenReview.net, 2017.
[61] T. Guo, W. Hu, S. Mei, H. Wang, C. Xiong, S. Savarese, and Y. Bai, "How do transformers learn in-context beyond simple functions? case study on learning with representations" " in International Conference Learning Representations, 2024.
[62] Y. Dong, J.-B. Cordonnier, and A. Loukas, "Attention not all you need: Pure attention loses rank doubly exponentially with depth," in International conference on machine learning, 2021.
[63] P. Wang, W. Zheng, T. Chen, and Z. Wang. "Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice," in International Conference on Learning Representations, 2022.
[64] K. Wu, J. Zhang, H. Peng, M. Liu, B. Xiao, J. Fu, and L. Yuan, ViinyVit: Fast pretraining distil- lation for small vision transformers," in European Conference on Computer Vision. Springer, 2022.
[65] A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, Zhai, T. Unterthiner, M. Dehghani. M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An image is worth 16x16 words: Transformers for image recognition at scale," in International Conference on Learning Representations, 2021.
14

---

A Theoretical Analysis
A.1 Results and Proofs
In this section, give formal statement of Theorem and the detailed proof. We first restate the problem settings and assumptions.
We consider set of in-context examples {(xi, Yt)), where xx, Y; E R and W' + Ej. Here, W* is shared coefficient matrix and E; is the additive Gaussian noise for each sequence. The training input to the model is given by
= (X1,N1, .Yn, xn+1) E Rdx(2x+1),
where each x, is sampled independently from (o0, Ia), W ~ N(W.. I22) for some deterministic Wo E Rdxdd and E; ~ (0, La) are i.i.d. Gaussian noise. For the model, we follow the common considering two-layer linear attention Transformer with two heads per layer and no residual connections. This simplified model preserves the key component- multi head attention and is sufficient to highlight the differences in information flow between the stan- dard MHA Transformer and SkipV Former. We train the model using only the in-context tokens (x1, Y1," ... .vn.
simplifications 41 48 42
For the MHA Transformer, the output of each transformer block is
TF((E) = wht EP E T) 1= 1,2,
(2)
where
R(22+1)x(2x+))) whth Whh with E Raxd for 1, 1,2, and we omit Wo since it can be incorporated into Wh', making no difference in expressivity analysis. Denote by T the distribution over input sequences E. For SkipV with skip head, the output can be expressed as:
I2n 0
Dout = W. TF)
TF(())
(3)
+W2EPP TF
where TF1()) is defined as in Eq. (2. The training loss function is:
L (whh whe w//).. H) E, HH yn-1
(4)
where & = 1 refers to the loss of standard Transformer, and 2 refers to that of Skip Former. As stated in Section we admit the copying mechanism and introduce the following assumption: Assumption 1. The output of the first attention block is
Axi
AAxa By
( [Axx+1 ))
TF(EE) = X1:
By
Nn,
where Raxd B (dd-a)xd and a
Assumption iis close real scenarios, as the embedding dimension of attention is kept the same as that of token embedding []] Although we do not consider causal mask explicitly, we assume that the copying mechanism is conducted only y,s with previous tokens, maintaining the causal structure. In Assumption [ the information of the raw samples are compressed as Ax and By. Intuitively, MHA can only solve the compressed regression problem with sample-label pairs (Ax, By), whereas can access the uncompressed x and y and solve the original problem and attain lower
loss.
Under Assumption D we drop the superscript in Eq. and (3) The formal version of our main theorem is stated follows:
15

---

Theorem 2 (Formal). Assume that oa(Wo) > 2v 2vdo do and d is sufficiently large. Under Assumption there exists a constant c > O independent of A, B, 8. W WV, H, such that
min W..WW.WW.... L2( Wk. WV), HI)
min L W WVJ,
w W WR.
where Lı and L2 are defined as in Eq.
The proof follows the analysis framework in 48
We begin by rewriting the loss function in more
A and set
convenient form. Define W HWV,AD A
(W ( w
GL
Σ x;x,
AAxx AT By.x A
Axay B By.y, BT
1=1
for each input sequence as
(X1,y1:""
.Yn). ;yn) Then the loss of MHA Transformer can simplified
2
B(, E
Wh W Yn+1
(5)
F
Similarly, loss of Former be expressed
2
B2 = E,
Σ
wh G2 Xn+1
vntl
(6)
F
Ax;
where G3 GI,D and G Ei X + yı
By;
To prove the theorem, we present the following lemmas.
Lemma 1. Let WD be the solution ridge regression with regularization strength d² the exemplars (X1,"" ". =Yn Xa+1). For any matrices and B, there exists a constant C independent of A, B, W, M, such that
W, C ED Σ
w WhG WD
1,2.
F
Proof. We denote
W (W), W²), GTD-= (Gr. G. G2.D
G22D G2DD)
M M2
M
Using the law of total expectation, the loss can be written as
Ls(A, B, W, M) = ED, x.+: By.ss [|xx+11 WG 13 D. D,xn+1. k 1,2.
Define the function
g(U)
Eyn.s [[|xx++1 vn+ill? D. xx+]
Since 9()) is convex function, the minimizer of the function exists and we denote (any of) by Wp. The first-order optimality condition gives
Vug(U) Ry.., [2(Uxat1 y=+1)x D, x.+1].
For any matrix U, by taking the dot product of both sides with U WD. obtain
Ey.1 11 22Uxx++1 - Vx+1)xT+110 -WD) D, D,xn+1| O.
16

---

Therefore, we have the following decomposition
Eyn+1
||y+11 - W M.,, D, D,xn+1
Eynti
yat1 WDxx+1 + WDx+11 WG D, xn+11
yn+1 - WDxx+1 +1 11 + || Dxn+1 - W D, D,x Eyn11
+ yB (yn+1 WDX (WoXa+1) WG D,x++1
By setting U = -WG&, D MM in Eq.
77, the last term vanishes. Hence,
B, M) Dxxx
E Bynts [I DG l1 D,xnt1
]]
+ +C,
where
C = EDDx.. Fyn-1
1ly+1- Wox++1 D, D,
is independent of A, B, W.
Finally, since X,+1 (0, Ta), have
La B., C ED WD -WG
We denote T = (W wh) and M = -MP. M2)), where WP: M E Rdxa and W M E (d)()-a)) With these notations, the following lemma provides further simplification of the loss for both the standard MHA Transformer and the SkipV Former..
Lemma 2. For the standard Transformer, there exists a constant C1 B, W, M) > such that
H1 WD 2
B..AB.W.M). C + C1 ED
F
where
H1 Σ AXX B M + + BYX A M!) h=1 Similarly, for the SkimVrFormer there exists a constant C2(A, B, W. M) >
such that
B. C Ed H2 WD F
where
H2 Σ w A M 4=1
Proof. From Lemma and the definition of GL ( (L,D), have
(A, B, M) C ED
WD W
F
2
+ ED
WD
Wh
XXT +
AX BY
(XTAT AT YTBT) Mh
h=1
(8)
For any fixed A, B, W, M, define
(x))) =
(W"XXT M WTAXXT A MM),
h=1
(₂ Σ
wh BYBYYTO T B M
(9)
h=1
N₂(XYY) Σ
(W B MA + T M
17

---

Then Eq. 8 becomes
((A.B.W,M) C / N2())))))))))))))))()) - - Ns Nx
NaM)
From the definition of W. it is known that WD this identity, we observe that Wp -----M -N, Y) an odd-degree function in Y, while + an even-degree function in Y. Hence,
almost everywhere. Using
C' + ED WD - Na(x, + EDD (XN1(X) + N2(2)]
The first part of the lemma follows by setting H1 N3 and C₁ ED [ + NY
Similarly, for the SkipV / Former, we can decompose W W D. M into components of odd and even degree in Y. A direct calculation yields
T A, B, W, M) C + ED I.v - N5(X. + ED [I + NI
where
Ni(X)= = Σ
M + W. ATM)).
h=1
(10)
NA(Y) = W BYY T B M + W B M2
N WXY B M W: ByX x A T + w A
The second part of the lemma follows by taking H₂ = N2 and C2 = ED (X) + N3(Y)
MM
Next, for any matrices Wh (WT, Wh) e Rdx2d and M1hh
2dd for
2, we
define
XY
(WWM. Σ Wh
Nrhh.
YXT
h=1
We focus
i(W, M) ED | - h(W, M))
(11)
in the following analysis. We first present Lemmas 3 and 4 from 48 and treat them as one lemma as follows.
Lemma 3 (Lemma and 4 in 48 There exists scalars and 12, such that for any i, it holds that
Ep )., tile, ED [X(YT) (. tzlo,
where Ye, denotes the i-th row and Y., denotes the i-th column of Y. Moreover, for any i, by setting
[E((W))..X(")) ED.Y),,, XT X(TT)
holds that
ED (Xx (Y, (Y), X(Y") (W) 0. Lemma 4. There exists constant C3 > 0 which is independent of W, M, such that
LOW, M) = C3 + En AYXT n(W M
where A = diag A1, ... Ad) and
(D. A ED(Y),, iY, XT (TX(T))
18

---

Proof. We denote L'WW M) = Ep L - h(W, An) n(W,,,n)
=Ed
and G.D (yx² xvr) for
simplicity. Our goal is to show that
Viwr,i L(W, M) V (iV Mi) T'WW,
We first consider the gradient with respect to W
V Wn L(W, M) 2ED (N(W, WD)
V W B'W,, M) 2ED (M(W,, M) AVX))
Thus it suffices to prove
ED [ [o -ED AYX (
(12)
which breaks down into:
ED
W WD( T EDD
AYX (MTH) TXYT
(13)
E
W (D(M)) En
AYX AYXT(M) (M)
(14)
We first prove Eq. (13) suffices to show:
EEn (XX
(ED (,XXXX)))))))) Vi,j.
When # j. using the i.i.d. Gaussianity of W and 6s we obtain:
ED [(TT) k.y,, x] = O.
Thus have
ED (XY Yis (WD) ED ) (AY Vi
When = j. by Lemma 3 we also have
ED (XY D.. = ED [(XY]) )..X (XXTT.)XXXX 1A.
Thus Eq. 13p holds.
For Eq.
14 it is equivalent to
En (M)) TXX = EE, (M.)))
ii,
which can be further written as
ED
(WD).
ED (AYXT) ).
Σ (M
Vi.
J=1
By the commutativity of the dot product, the above expressions can be rearranged as
ED Σ (M)) (1))), (WD)..
ED
Σ (M (M))). (Y
From the above discussion, we obtain
ED (WVD).. )ss] EE [XXYT)) (YXT)) Vi.j.
which completes the proof of Eq. (14).
It remains to show that V iin L(W, M) V
in i'(W, M). Similarly, this is equivalent to showing
Ep (Gb)
( (W/1) wo] Ep (cb)] (W"))
This can be shown using essentially the same technique as above. particular, by setting W = 0, we have
C3 = En
W 2
En ||AXX
19

---

We now turn back to the proof of Theorem !
Proof of Theorem
By the definition of loss of standard MHA Transformer and SkipV1Former Former in Eq. and O and the form of L(W, .M in Eq.
(II) we immediately obtain
WD WD-WW (10o-N(xx,y)) x3
min BBB.MM
ED
min L(W,M) W.M
On the other hand, when
1 A, W A, I A,
A= A=1,B=0,
2
it follows from Eq. 10 that
(x) = 0, = 0, =AX
Therefore, from Lemma 4 we know that
min A,B.W.M
L2 M) C C3.
For the MHA Transformer, when a = d, implying B 0, we can conclude from Lemma that
L(,, 2 C En [
2
C + C3 + En ED I|
(0). the rank of Na Y) is at most a.
When a < d, noting from the definition that M W
Moreover, by Lemma 4 and the fact that N2(X, Y) is a special case of h(W, M), it holds
Ep WD Na(XX C3 + ED | - Na(X,
2 C ED|AAYX RAY
where R2 T) denotes the best rank-o approximation AY under the Frobenius norm.
Denote by Ep RDlAAXX Ra (AY X T)) which is constant independent of A, B, W, M.
Since C2 A, B, W, M) > 0, we have
L2, 2 CC
remains to show that c > 0. First, show that |\, > Recall that W, VXT so we have
ED (WD)), ):r) ED [ (XXT + -0 ):))))))
ED X (XXT + o² -Xx ):d
ED [ei, X (XXT + ):a)
By diagonalizing XXT, one can show that
xx (XXT +0211-1XX" 2 XXT -021.
Thus, we can derive
Ex [Xx T +o 12111-1xx"" Ex o²1] (d O
We also have X X 02 -x
Combining these results, we conclude that |>g] > 0.
Now, since A is a full-rank matrix, we obtain
Ep| - Ra, X )1 2 min A.R.EDIJY RY 1.
By the Eckart- Young theorem we have
2 ED ED [)](
Ep T XYX M1 ED
Σ (x(X)))
i=a+1
20

---

where () denotes the i-th singular value of YX". Since YXT = WXXT +EX by
Weyl's inequality, we further have
(x((Y aa(W" XX ) S ||X||2
For (WxX)x since (ABB 2 (a(A)ou(B)), we have
va(W"XX aa (W
By Edelman's Theorem [54]. for X (0, Lp), it holds that oa(X) + (()))))) when d is large.
Thus, we have
aa( XXT) 2 oa(W")) (d=t),
with high probability. Since W (U, from the assumption that Fa (a())) > 2 we have
Ep ( xx" -| O.
Combining all the above results, we have shown that c O, thus completing the proof.
A.2 Discussions
In this section, we discuss the implications and relations to the analysis in other works of our theoretical findings. A high-level basis of our analysis centers on the interplay between Transformer architecture and the tasks it solves, in particular, auto-regressive tasks. shows that SkipV FForerr fails to boost performance on non-autoregressive tasks, indirectly support its design's 's focus on auto-regressive. In addition, the importance of the first layer is also confirmed in (55). They similarly attribute in-context learning two-phase process: the first layer preprocesses context examples, while deeper layers perform iterative optimization steps.
In another related work, Zhou al. 225 explain ResFormer's (akin to our skip connection with first-layer Values) gains via its ability in alleviating attention concentration in multi-head attention. While it is not clear whether there theoretical connection between their explanations and our analysis, Zhou et al. also confirm first-layer information loss, which is consistent with our analysis of copying-induced compression.
Due to the essential difficulty in analyzing the behavior multi-layer Transformers in ICL [42], focus simplified two-layer model without residual connection in Theorem a tractable yet still instructive framework. While one may worry that omitting residual connections could affect deeper layers, several clues suggest that our analysis may still extend to standard Transformers. First, the "linear copying" mechanism in Assumption [ appears at the first-layer output of standard deep Transformers with residual connections (see Figure of [14). Second, for simplified one-layer Transformers with and without residuals, it has been shown, respectively in [48] and 422, that both models implement a single pradient-descens step on a least-squares objective optimality. other words, removing residual connections does not appear to impair the representational capacity of Transformers for in-context learning, suggesting that our simplification is reasonable to an extent. Regarding the two-layer simplified model, if we assume that a multi-layer Transformer performs one gradient-descett step per layer, our analysis naturally generalizes deeper models by applying the same proof technique as in Theorem 2 Furthermore, Appendix presents linear- linear-probe experiment on deeper models that provides empirical support for our analysis.
B Details and Additional Results for Section 6
B.1
GPT-2 Series
Pretraining. We provide implementation details of our GPT-2 pretraining setup, following the
training framework in 77. used in training. The hidden dimension of the FFN is set as 4x the embedding size. All the models are trained using BF16 format and AdamW optimizer (56) with 31 = 0.9, f2 = 0.95 and weight decay 0.1. We adopt warmup ratio of 10% and cosine annealing schedule with decay factor of
the main hyperparameters of the model and optimizer
21

---

Params
Embed
Heads
Layers
Steps
Data Amount
Peak LR
I25M
768
12
12
40K
5B
1e-3
200M
1024
16
12
40K
le-3
355M
1024
16
24
40K
5B
le-3
550M
1200
20
28
40K
6e-4
Table 3:
Hyperparameters of GPT-2 models in pretraining. "Data Amount" refers the total number
of tokens used during pretraining.
10% of the peak learning rate. The batchsize is 120 and the sequence length 1024. We also use dropout as 0.2 and flash attention [57] The experiments are conducted on 6x48 GB A6000 GPUs.
The architectures used in experiments (DenseFormer, YOCO, CLA) as follows:
DenseFormer enhances Transformer efficiency by introducing Depth Weighted Averaging (DWA) after each block. This mechanism computes weighted average of the current and past representations, allowing the model to achieve lower perplexity without increasing its size.
DenseFormer:
YOCO (You Only Cache Once): YOCO proposes decoder-decoder architecture that caches key-value pairs only once. It consists of a self-decoder and a cross-decoder, where the self-decoder computes the first layers using efficient attention and the cross-decoder computes the rest & layers by replacing the K, V in each of last &/2 with 4 K, V.
CLA (Cross-Layer Attention): CLA reduces the size of the key-value cache in Trans- formers by sharing Keys and Values for every adjacent layers. i typically set as 2 or 3.
To maintain the same K KV-cache saving regime, we consider V-only YOCO, which does not incorporate efficient attention and only reuses the value in the last & layers, and V-only CLA, which shares the value across every two adjacent layers, achieving 25% reduction KV cache size compared to the baseline.
Uptraining. For uptraining GPT2- to GPT2-355M models, we perform a grid search over learning rates (8e-5, le-4, 1.5e-4, 2e-4), where le-4 is the final learning rate of our pretraining. Among these, a learning rate of 1.5e-4 yields the best performance consistently across all three model sizes. Dropout is disabled during uptraining, while all other hyperparameters remain consistent with the pretraining configuration. To initialize the uptrained models, we also experiment with several strategies for converting the pretrained checkpoints:
Mean VO: Apply average pooling over every two head projections and their corresponding columns in Wo. and zero out the remaining columns.
Top V: Select the H' rows with the largest norm of head projections Wv to be the new head projections.
Top VO: Select the H' head projections in Wy and the H' output columns in Wo with the highest norms.
SVD: Construct best -rank approximation to the original WoWy using SVD.
All the above approaches are conducted every layer beyond the first. Empirically, all of the above strategies are outperformed by our method introduced in Section C.3 as evidenced by their higher initial losses. The comparisons of the converted checkpoints' initial loss are shown in Figure I0
B.2 LlaMA Series
Pretraining. We adopt the training framework from 588 and provide the following details regarding the models and hyperparameters. Table Summarizes the key hyperparameters of the models and optimizers used during training. All the models are trained using BF16 format and Adam W optimizer with 31 0.9, 32 0.95 and dropout. For the 60M 350M models, we apply no weight decay and gradient clipping, with a peak learning rate 2e-3. For the 1B and 3B models, we set weight decay to 0.1 and gradient clipping to 1.0, with peak learning rate of 1e-3 and 5e-4, respectively. A
22

---

Initial Loss of Different Methods
7,078
7:322
7-
6-
Initial Loss
4.836
4.316
Mean V
Moan VD
Top V Method
Top vo
SVD
Figure 10: Comparison of initial loss values across different methods. The "Mean V method corresponds to our baseline strategy described in the main text.
warmup ratio of 10% is used, along with a cosine annealing schedule decaying to 10% of the peak learning rate. The batchsize 512 and sequence length is 1024 for all models. We also use Rotary Positional Embedding (RoPE) [59] and flash attention. The experiments are conducted on 8880 GB
A800 GPUs
Params
Hidden
Intermediate
Heads
Layers
Steps
Data Amount
Peak LR
60M
512
1376
8
8
2.5K
1.3B
2e-3
130M
768
2048
12
12
5K
2.6B
2c-3
350M
1024
2736
16
24
15K
7.8B
2e-3
1.3B
2048
5461
32
24
25K
13.1B
1e-3
3B
2560
6848
32
32
30K
15.7B
5e-4
Table 4: Hyperparameters of LlaMA models pretraining. "Data Amount" refers to the total number of tokens used during pretraining.
We tune the learning rate from a set (2e-3, 1c-3, 5c-4) for all models and choose the best learning rate based on the validation loss of the baseline model. We observe that the influence of hyperparameters on Skip FFormer nearly identical to that on the standard MHA Transformer.
To further assess robustness, we also pretrain LLaMA-1B models with different random seeds and perform preliminary significance check. As shown in Table [5 SkipV SkipVIFormer FFormer consistently outperforms the baseline by similar margin under both seeds, indicating that the observed gains not seed-dependent. In addition, similar trends are observed on Tiny LaayLLAM (see Table 6). These results confirm the improvement of SkipV Former on accuracy.
Combining with GQA and MLA. We first provide brief introduction GQA and MIA. GQA divides the query heads into G groups. Each group shares single KV pair, reducing memory usage while maintaining performance. Mathematically, for each group g. the attention output is computed as:
head,.g = Attention (W2 Qi, W K, V),
Table 5: Validation loss of LLaMA-1B models under different random seeds. Reported mean + standard deviation is across three runs.
Model
Seed 42
Seed 100
Seed 2025
Mean + Std
Baseline
2.723
2.753
2.744
2.740 + 0.015
SkipV Former
2.711
2.743
2.732
2.729 + 0.016
23

---

Table Validation loss for TinyLLaMA models with GQA-4 configuration.
Model
1.1B
315M
125M
LiMALLaMA (baseline)
2.750
2.936
3,420
TinyLLaMA-Skin SSkipV
2.742
2.916
3,370
where Qi is the i-th query head, W.O, W W are the projection matrices for the g-th group. K g. y and V are the shared Key and Value for the group. The final output of GQA is computed in the same manner as that of MHA Transformer.
MLA introduces a low-rank compression of the Key and Value to reduce the memory footprint of the KV cache during inference. Instead of caching the full Keys and Values, MLA projects them into lower-dimensional latent spaces. Mathematically, let X E Ikdxn be the input of an MLA layer. The inference of the layer can be expressed
CKV -WDKVX X,
RCC -UK ORV
K" = -ROPE(WK'X),
K = =KKR, K
VC -WUV
where we omit the splitting of multi-heads. Here, CKV E RRde is the compressed latent tensor for Keys and Values; de < d indicates the KV compression dimension; WDDY E Rd denotes the down-projection matrix; WUK WUV Rdxde are the up- up-projection matrices for the Keys and Values, respectively; WKR E _gdaxd is the matrix used to produce the decoupled key that carries Rotary Positional Embedding (RoPE); and [ dentoes concatenation. In MLA, only the CKV and K need to be cached during generation.
We pretrain Base-GQA and Mase-MLA, as well their SkipV Former counterparts, on LLaMA- 350M with the OpenWebText dataset. The optimizer hyperparameters follow those used for GPT2-355M. In the experiments, GQA models share the same keys and values between every two heads for all layers, i.e., G = H/2. MIA models adopt d, = 256 and d = 32.
For clarity, we detail how the total parameter counts in Table 2 are obtained. Bach Transformer block contains an attention module and a feed-forward network (FFN). In the GPT2-355M setting, dmodel 1024, head dimension 64, and FFN hidden size du 4096.
GQA baseline. With 24 layers and 16 query heads, we have Wq E R1024X1024, WK. Wv E R1024x512 (due to head sharing), and Wo E I0102111024, giving ~3.7M parameters per layer for attention. The FFN adds ~8.4M per layer from W1 E 1006x10021 and W2 E R1021X Over 24 layers this yields ~00M (attention) ~010 (FFN), plus embeddingssnormss for a total of 33. FFormr halves Wy from layer 2 onward, i.e., 23 x (1024 x 256) z 6.0M fewer parameters, giving 328.9M.
MLA baseline. Here Values use low-rank latent d, = 256, with W-down E 10024x256 and W E 2256x1021 (~0.52M per layer), plus Wo, WK, Wo. so attention totals 3.9M per layer and FFN 8.4M. This gives about 93M 201 M 337.4M overall. SkipV Former-MLA halves d, to 128 from layer onward, reducing ~3.0M parameters, for a total of 334.4M.
As the above setting corresponds the group-of-2 GQA scheme in LLaMA (v1), while more recent models such as LLaMA-2 and LLaMA-3 employ a group-of-4 configuration, where keys and values are further shared across four heads. To align with this more realistic setting, we additionally conduct experiments on LiMY models with group-of-4 GQA. Specifically, we compare SkipV SkipV1Fomie Former against the baseline TinyLLaMA-1.. variants with the same modification strategy as used for LLaMA- 1MMMA-1300/350M with LLaMA-1.1B
and also evaluate smaller 315M and 125M TinyLLaMA
The results, summarized in Table 6 show that SI Former continues to outperform the baseline models by a margin comparable to that observed for GQA-2 (Table 2 This confirms that the effectiveness of cross-layer Value injection remains robust under more advanced GQA configurations.
24

---

second buffer
first buffer
cached V buffer
V[1]
pdw/2 [1
V[4]
y pta
plare
V[c 1]
p
[4]
p8:212af
Layer
Layer !
concat
v[c]
concat
(a) Standard Transformer: KV cache exten-
sion during decoding. Each new token's (b) SkipV Foomer: KV caching with first first-layer reuse. Deeper layers Value is appended along the sequence di- concatenate their current helf-heads with the cached half half-heads from the first layer.
mension.
Figure 11: Comparison of KV caching mechanisms during autoregressive decoding.
Table 7: Validation loss of LLaMA-1B models under different skip-head ratios.
Skip Ratio 25%
50% (default)
75%
Bascline
Val. Loss 2.716
2.711
2.717
2.723
KV Caching Mechanism in ViForer During autoregressive decoding, Skip Former differs from the standard Transformer in how Value tensors are cached and retrieved.
Standard Transformer. For each new token 2; Rdd, the cached Value tensor is V(1 t- [ which stores the Values of tokens Ii...., 2t-1- The Value of x, is simply appended along the sequence dimension, producing [[1 4).
Former: /ormerFormer In contrast, FormerFormer maintains two sets of cached buffers: (i) the first half of the heads for every layer, [1,1: t - 1] for 1 = 1...., L, and (ii) the second-half heads of the first layer, V:1.f/2+1:1f [1 t 1]. When processing 2,, these buffers are extended along the sequence dimension to include the new token, yielding ( t] and 11,4/2+1:H [] f). They are then concatenated along the head dimension form the full Values used in attention step An illustration is provided in Figure 11
SkipV/
This design effectively reuses first-layer Value information across all deeper layers via per-head concatenation, which is absent in the standard Transformer. Conceptually, the difference is not in appending along the sequence simeenioon-whih both models share-but butt in the additional cross- layer head stitching. While this introduces minor overhead compared to standard V-cache extension, it is implementation detail rather than kernel-level limitation, and can be optimized further with fused kernels.
Ablations. In the main text, we adopt a 50% skip-head ratio as the default setting. Since the total number of concatenated V heads is fixed, a larger skip ratio implies a stronger reduction in the embedding dimension of Wy from layer 2 onward. Skipping fewer than 50% of heads offers limited memory savings, while skipping all heads corresponds to SVFormer (25). which suffers from clear performance degradation. Intuitively, too few skipped heads restrict deeper layers by limiting the copying mechanism, whereas too many skipped heads hinder their ability to process the full data flow. Empirically, a 50% ratio achieves near-optimal tradeoff between memory efficiency and model quality.
To further validate this finding, we extend the ablation to larger IB 1B-parameter LLaMA model. Table reports validation loss under 25% 50%, and 75% skip ratios, together with the baseline. Consistent with the GPT2-355M results, the 50% ratio again provides a near-optimal tradeoff. Notably, the 75% ratio still improves upon the baseline while further reducing memory cost compared to the default 50% ratio, making it viable option in highly memory- constriie scenarios.
25

---

C Additional Experiments
C.1 Linear Probe
In this section, we validate our theoretical findings from byy examining the mesa-optimizaion behavior in Transformers using linear probes [60]. We follow the methodology outlined in [14 43). Specifically, we extract the output from each layer of a pretrained GPT2-125M model and train a linear probe on top. The probe consists of layer normalization followed by linear classification head. The learning rate is tuned over the set (1e-4, 2e-4, 3e-4), and training is conducted for 4000 steps, with all other hyperparameters remaining consistent with those used during pretraining.
Figure 12 illustrates the validation loss of the probe across each layer. For both the baseline model and S I Former, the probe loss decreases monotonically with layer depth, indicating progressively refined internal representations. The trend can be roughly divided into two phases:
Layers 0-6: In these early layers, the probe loss fluctuates between the base and FFormrr models. This behavior suggests that the shallow layers are transforming the input into representations conducive to mesa-optimization [6]].
Layers 7-11: In these deeper layers, the probe loss for Skip Former shows consistently steeper downward trend compared to the baseline. This indicates that SkipV1 Former has accelerated the mesa-optimization process, resulting in more rapidly improving representa- tions.
These observations empirically support our theoretical analysis, demonstrating that Skip Former enhances the efficiency of mesa-optimization in deeper layers.
Layerwise Loos (9-11)
Zoom Unior Rt (7-11)
locebe
4.2
SkyttFereee
40
6.3
3.0
3
3
3.64
4.2
3.4
32
30
Layor PMox
Layo' BObK
Figure 12: Layerwise linear probe on a 12-layer GPT2 model. We train a linear probe on the output of each layer (1-12) and plot validation loss for both the Baseline model and SkipV1Forman On the right, we zoom into layers 7-12 and overlay best-fit lines. The steeper negative slope for SkipV Former indicates a faster decrease in probe losseevidence that its cross-layer V reuse strengthens the model's 's mesa-optimization, yielding more rapidly improving representations at deeper layers.
C.2 Visualization
To further interpret how cross-layer Value skips in SkipV affect attention patterns and information propagation, we visualize both head-head and token-token similarity matrices at selected layers. These visualizations reveal changes in head diversity and the degree of "oversmoothing"
62
63
across layers.
Figure 13 shows that SkipV1Forma exhibits lower inter-head similarity, indicating higher head diversity. This can be attributed to the injection of uncompressed first-layer Values into deeper layers, which promotes the diversity of Value in deeper layers. Meanwhile, Figure [4] showw that, apart from a few tokens, SkipV Former reduces the overall token similarity compared to the baseline, alleviating the oversmoothing effect. By supplying deeper layers with original first-layer Values, tokens are less dependent on copying redundant information from neighboring positions, thus preserving more distinct features.
26

---

Beyond these similarity analyses, we also provide additional insights into the role of selected heads SkipV Former. Prior works have identified previous-token copying heads and induction heads in autoregressive tasks 47 Specifically, Olsson et al. found that there exists circuit implemented by two sorts of heads:
Previous-token copying head: finds the previous instance of the current token A and the token that came after (call it B3.
Induction head: it "completes the pattern" by predicting B after the current A.
This characterization is closely related our theoretical analysis of 'copying mechanism' in Section We hypothesize that skipping heads in SkipV play a similar role to previous-token copying heads--leveraging layer I's complete raw information to facilitate pattern recall-while the remaining heads in layers 2-L operate more like induction heads. In this way, SkipVIForme effectively replaces some of the redundant copying heads in deeper layers and preserves the more functionally important induction heads.
Overall, our visualizations and analysis together suggest that SkipV 1arV/FForerrss 's cross-layer Value reuse encourages greater head specialization and mitigates representation collapse across tokens two key factors underpinning its improved representational capacity.
Similanity (Layer
Similarity Laaer
100003
1.000
0.90995
0
0.99925
89996
0.90902
0.955
0.99075
0.99
0.9925
(a) Baseline, Layer 12.
(b) SkipV Layer 12.
Figure 13:
cosine similarity at layer 12, comparing Baseline and SkipV Former.
Seee.Thiee Codhee Sielarty
Thkken Riken Costee lartty 12.
Cairy locan rodax
15
04
Key tricen Indle:
Kay tobae indes
(a) Baseline, Layer 12.
(b) SkipV Layer 12.
Figure 14: Token-token cosine similarity at layer 12, comparing Baseline and Skip Former.
27

---

C.3 Downstream Evaluation
GPT2-Series. We assess the performance of our pretrained GPT-2 models across several standard downstream tasks, including HellaSward ARC-Challengy ARC-Easy, PIQA, Winogrande, OBQA, and SciQ. The results are presented in Tables Notably, SkipV Iormer and ResFormer models consistently achieve the highest average accuracy across various model scales, aligning with their performance during pretraining.
LlaMA-Series. Similarly, we evaluate the 60M to IB parameter LLaMA models on downstream tasks such ARC-Challenge, ARC-Easy, BoolQ, HellaSwag, OBQA, PIQA, RTE, SciQ, and Winogrande. As detailed in Table 12 SkomPIFormer Ioommr consistently outperforms the baseline models, demonstrating improved average accuracy across these tasks.
Table 8: Zero-shot accuracy six GPT2-125Ms variants across seven benchmark tasks.
"Avg."
reports the mean accuracy over all tasks. Best performance per column is in bold.
Model
HellaSwag
ARC-C
ARC-E
PIQA
Winogrande
Obqa
Sciq
Avg.
Bascline
27.2
17.5
40.9
60.1
51.0
14,6
67.7
39.9
SkipVIFormes
27.3
18.0
40.9
60.1
51.5
15.0
67.6
40.1
ResFormer
27.3
18.2
42.9
60.2
50.5
14.2
67.2
40.1
DenseFormer
27.3
18.6
40.7
59.7
51.1
15.8
65.3
39.8
YOCO
27.2
17.6
39.4
59.6
52.0
14.4
66.2
39.5
CLA
27.2
19.1
39.9
59.4
49.3
14.4
67.0
39.5
Table 9: Zero-shot accuracy of six GPT2-200Ms variants across seven benchmark tasks. "Avg." reports the mean accuracy over all tasks. Best performance per column in bold.
Model
HellaSwag
ARC-C
ARC-E
PIQA
Winogrande
Obqa
Seig
Avg.
Baseline
27.6
18.9
42.1
61.4
50.4
13.8
69.2
40.5
SkipVIFormer
28.0
17.7
43.7
60.1
49.0
14.8
72.2
40.8
ResFormer
27.9
18.6
42.0
61.8
51.2
16.4
69.6
41.1
DenseFormer
27.6
18.5
41.2
60.6
51.3
16.2
66.8
40.3
YOCO
27.6
18.3
41.4
60.3
51.5
16.0
69.3
40.6
CLA
27.5
17.7
40.7
60.2
50.7
15.2
66.9
39.8
Table 10: Zero-shot accuracy of six GPT2-355M variants across seven benchmark tasks. "Avg." reports the mean accuracy over all tasks. Best performance per column is in bold. Model
HellaSwag
ARC-C
ARC-E
PIQA
Winogrande
Obqa
Scig
Avg.
Baseline 28.6
21.0
42.2
61.5
51.3
16.6
71.6
41.8
SkipVIFormes
29.1
18.9
43.6
62.6
52.5
15.8
73.5
42.3
ResFormer
28.8
19.8
42.5
62.2
50.9
14.8
71.8
41.5
DenseFormer
28.7
19.7
44.4
61.3
51.5
16.4
70.7
41.8
YOCO
28.5
18.8
42.4
61.4
51.0
15.4
68.6
40.9
CLA
28.5
19.4
41.7
61.3
51.0
15.8
69.3
41.0
C.4 Alternative Head Injection Strategies
In this section, investigate different head injection strategies other than our default strategy, which directly concatenates the second-haf Value heads of layer to the first half Value heads of layer 2 L. Specifically, we evaluate
Pooling: Inject the head-wise average of the first half and second half heads layer 1.
Dynamic: For the i-th layer, inject head from layer 1 into position H/2 in rotating manner.
Odd/Even: Use layer I's first-half heads for odd-indexed layers and its second-half heads for even-indexed layers.
SkipVI ResFormer: top S Former, linearly add the first-half heads from layer to the first-half heads of all subsequent layers.
28

---

Table 11: Zero-shot accuracy six GPT2-550Ms variants across seven benchmark tasks. "Avg."
reports the mean accuracy over all tasks. Best performance per column is in bold.
Model
HellaSwag
ARC-C
ARC-E
PIQA
Winogrande
Obqa
Seiq
Avg.
Baseline
29.5
19.9
46.9
62.8
51.1
16.6
72.4
42.7
SkipViFormi
29.6
18.5
46.1
62.7
51.4
17.6
73.9
42.8
ResFormer
30.1
19.3
47.1
63.7
52.2
17.0
73.4
43.2
DenseFormer
29.4
20.6
44.2
62.2
49.8
15.8
72.7
42.1
YOCO
29.2
19.7
45.0
62.2
50.5
15.0
71.5
41.9
CLA
28.8
19.0
43.2
62.1
51.5
15.8
72.1
41.8
Table 12: Test accuracies (%)
60M
1B scale models on downstream tasks, with overall average.
Model
ARC-C
ARC-E
BoolQ
Hella
OBQA
PIQA
RIE
SciQ
Winogr
Avg
Baseline-1B
19.3
44.5
60.2
31.0
17.2
66.3
71.6
46.2
SkipV Former--B
19.3
44.1
59.6
31.2
17.4
66.9
52.3
73.8
50.9
46.2
Baseline-350M
19.7
45.1
59.4
30.1
16.6
64.3
50.9
71.8
51.4
45.5
SkipV 35MM
20.8
44.9
53.0
30.9
18.0
66.3
52.7
74.5
51.5
45.8
aaseline-130MM
19.4
37.1
50.3
27.2
16.0
60.9
49.8
65.6
52.0
42.0
SkipVIFormn 130m
17.8
39.3
61.3
27.7
14.8
61.9
53.8
68.5
49.8
43.9
Baseline-60M
17.8
33.0
45.3
13.6
58.8
SkipV F0Mmmrr-MMM
16.7
33.0
12.8
58.9
52.3
64.3
51.0
41.7
Table shows that there is no substantial improvement these strategies over the simple second half' scheme. Moreover, our default injection strategy selects preserves consistent head ordering across layers, which is more hardware friendly than discontinuous schemes like random head selection 45) In all, simply injecting only the second half heads has achieved certain model performance improvement while being more hardware-friendly.
C.5 Skip Connections with both and
In this section, we investigate methods to further reduce the Key-Value (KV) cache size by using skip connections on both Keys and Values. straightforward approach to reuse the first layer's and V in all subsequent layers via skip connections. We term this architecture as SkipK ( Additionally, we propose SkipVI-YOCO which combines the Keys sharing mechanism in YOCO [15] (see Appendix [BI] with our skip connection approach for the first Value in Skip Former. Specifically, the first 5 layers of SKipVI-YOCK are the same as those in Skip I while the rest 4 layers computes attention by:
H'
H Σ H'+1
Attn(X) X
whv' softmax (Kkt)))) Q²)
+
wav' softmax
QM)
hm1
Both FIooRee and SkipV1-YOCO reduce the KV cache size for approximately 50%. We pretrain SriKKVIoorme and SkipV1-YOCO on GPT2-125M to GPT2-355M using the same hyper- parameter settings as our GPT2 pretraining experiments.
Figure 15 illustrates the performance of these models across three different scales. Both AKipKVTFormer and SkipVI-YOCO are outperformed by SkipVIFormer indicating that skip con- nections on Keys may lead to performance degradation. On the other hand, SkipVI-YOCO shows only slight performance drop compared to SkipVI Forrmer and still outperforms the baseline model, whereas SkipKV IFomer exhibits inferior performance even compared to the baseline model. The relatively strong performance of SkipVI-YOCO suggests that selective reuse of Keys (as in YOCO) may mitigate performance degradation, pointing to a promising direction for further KV cache reduction.
Method
VI
Pooling Dynamic Odd/Even SkipVI ResFormer
Val. Loss 2.885
2.881
2.885
2.883
2.889
Table 13: Validation loss of different head injection strategies on LLaMA-350M model.
29

---

19 Coricc LOT SOPNOC
VNIORSET LOT
(Nforter LOST SertOc
Canora
Eanera
1.2
(a) GPT2-125M Model
(b) GPT2-200M Model
(c) GPT2-355M Model
Figure 15: Validation loss over training iterations for GPT2-125M GPT2-200M2 and GPT2-355M models, comparing SkipVI-YOCO and SkipKVIFormer with the MHA baseline (blue line) and SkipVIFormer (red line).
C.6 Vision Transformer
Although the design and theoretical analysis of SkipV Formeer target towards auto-regressive tasks, in this section we conduct experiments on vision tasks for completeness. We train a ViT- model [64] on CIFAR-I0 and CIFAR-IOU from scratch to preliminarily test the performance of SkipV Iooreer on vision tasks. Specifically, our Vit-Tiny model has 12 layers and 3 heads per layer, with hidden size 192 and MLP of dimension 768. The total number of parameters 5.5M. Though CIFAR datasets and the fair. train the models using AdamW with 3 = 0.9, 32 = 0.999 and weight decay 0.1. The peak learning rate is le-3, with 10% warmup ratio and cosine annealing that decays 10% of the peak. We use batch size of 1024 and train for 15000 steps, i.e., 300 epochs.
are generally considered too small for VIT [65] our comparisons between 4KipV1Forme
baseline model are controlled
Figure shows accuracy over training on CIFAR-I0 and CIFAR-100. SkipV Former does not outperform the standard MHA Transformer. This indirectly supports our theoretical analysis, which rooted in the mesa-optimization behavior observed in trained Transformers for auto-regressive tasks. In aon-autoreeressive tasks, accessing information from the first layer cannot preserve the causal structure between samples and labels. Instead, the raw features from the first layer are less processed and may disrupt the hierarchical feature extraction that vision Transformers rely leading to degraded performance when being injected deeper layers. Designing an effective skip connection scheme for vision Transformers may require deeper understanding of how information flows through the learned architecture.
Validation Accuracy vs. Iteration
Validation Accuracy Iteration
Wel0ee on locurary
0.6
Wiltet
6.
91a03romm
Stan3Fomm
8020
12302
24003
teradon
beradion (Clebial Stern)
(a) Accuracy per iteration CIFAR-I0.
Accuracu per iteration CIFAR-100.
Figure 16: Accuracy per iteration of the baseline versus SkipV1Forme on Vision Transformer: results on CIFAR-10 (left) and CIFAR- 100 (right).
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Abstract and Introduction, where enumerates the contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section f and Appendix A We mention that we study simplified 2-layer Transformer due technical issues.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
31

---

Justification:
See Appendix A for all the assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Justification: See Section 6 and Appendix
We provide details of our
experiment.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [Yes]
Justification: We promise to release the code after publication. We have provided sufficient details and instructions for the repreduction of the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssionpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section aad Appendix B Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We tune the learning rate for most our experiments. Due to limited computing resources, we are not able run the experiments many times derive the error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
33

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:p/pw..aarr.....o//////////a
Answer: [Yes]
Justification: This paper only include experiments on public open datasets, i.e., OpenWeb- Text2 and C4.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This paper mainly focus on fundamental architecture Transformer.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
34

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper focuses on fundamental Transformer architecture.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
35

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We introduce new assets, including model code and training implementations. However, we do not release them at submission time. We commit releasing the full codebase upon publication.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
36

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We only use LLM for writing and formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: for what should or should not be described.
37


Reviews:
[{"review_id": "pRz76CZNVF", "reviewer": "Reviewer_HPMS", "summary": "The paper proposes SkipV1Former, a Transformer variant that reuses half of the Value heads from the first layer in subsequent layers, cutting KV cache by while improving perplexity. It theoretically shows this design restores compressed information and accelerates mesa-optimization, validated by experiments on GPT-2 and LLaMA models, which demonstrate consistent performance gains. An uptraining method converts pre-trained MHAs with extra compute, and SkipV1Former integrates with techniques like GQA for further cache savings. However, it underperforms on non-autoregressive vision tasks, highlighting domain-specific effectiveness.", "questions": "Mention in weakness", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. **Innovative Architecture Design**: Introduces SkipV1Former, which strategically reuses half of the Value heads from the first layer in subsequent layers, reducing KV cache while improving model representation. This addresses the trade-off between representation and memory efficiency, outperforming prior skip-connection methods that either sacrificed performance for memory savings or vice versa.\n2. **Compatibility with Advanced Methods**: Seamlessly integrates with techniques like Group-Query Attention and Multi-Latent Attention, achieving further KV cache reductions while maintaining or improving performance.\n\n### Weaknesses\n1. **Lack comprehensive comparison with other methods**: Though the paper shows the validation loss comparison with related methods (ResFormer) comprehensively in Figure 4, it would be better to report a more comprehensive comparison with other methods (ResFormer) on downstream tasks in Table 1.\n2. **Assumption for the theoretical analysis is too strict**: Appendix A attempts to proof that SkipV1Former gains strength in the aspect of \"copying mechanism\" and in-context-learning. However, the assumption requires no residual connection, which might already have impact on deep layer. Moreover, though authors attempt to proof SkipV1Former benefits deep layers, the assumption stands scenarios with two layers, which might differ from the actual deep layers.", "was_revised": true, "final_justification": "My concerns have been partially addressed (W2). Regarding Weakness 1, I don't think a 24% reduction in KV cache is worth the performance degradation (from 47.2 to 46.2). Therefore, I will maintain my rating.", "rebuttal": "Thank you for your careful review and thoughtful questions and comments. Please see our response below.\n\n> Lack comprehensive comparison with other methods: Though the paper shows the validation loss comparison with related methods (ResFormer) comprehensively in Figure 4, it would be better to report a more comprehensive comparison with other methods (ResFormer) on downstream tasks in Table 1.\n\nIn addition to Figure 4 and Table 1, we include detailed downstream task comparisons for GPT models with ResFormer and other methods in Tables 5–8 in Appendix C.3. It is shown that SkipV1Former and ResFormer achieve the highest average accuracies across various model scales. \n\nDue to our limited computing resources, we further train a 1 B parameter ResFormer and evaluated it on our downstream benchmarks (see the following table) alongside the 1 B SkipV1Former. Similar to the trend shown on GPT models, ResFormer achieves a marginally higher mean accuracy compared to SkipV1Former. This is reasonable because ResFormer linearly adds all Value heads from layer 1 to subsequent layers without reducing the number of Value heads, thus offering no KV cache savings. On the other hand, SkipV1Former preserves model quality while cutting KV cache by roughly 25 %, demonstrating superior memory efficiency with minimal performance trade off. We will integrate these results in the final version of the paper to give a more comprehensive comparison.\n\n**Table: Test accuracies (%) of 1B scale models on downstream tasks, with overall average.**\n\n|   | $\\Delta$ KV  |  ARC-C | ARC-E | BoolQ | Hella | OBQA | PIQA | RTE | SciQ | Winogr | Avg.\n|:-----:|:-----:|:----------:|:----------:|:-----------:|:-----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|\n| ResFormer-1B | -0%|     20.9   |     47.0   |     56.6    |     32.2    |    18.8    |    68.1    |    53.0    |    77.0    |    52.1    |    47.2    |\n| SkipV1Former-1B | -24%|     19.3   |     44.1   |     59.6    |     31.2    |    17.4    |    66.9    |    52.3    |    73.8    |    50.9    |    46.2    |\n\n\n\n> Assumption for the theoretical analysis is too strict: Appendix A attempts to proof that SkipV1Former gains strength in the aspect of \"copying mechanism\" and in-context-learning. However, the assumption requires no residual connection, which might already have impact on deep layer. \n\nWhile our assumption has some limitations, there are some clues supporting that our analysis may apply to standard Transformer. \n\n- It is empirically identified in Ref. [14] that the `linear copying’ mechanism in Assumption 1 appears on the **first‑layer output in standard deep Transformer with residual connections** (See Fig. 2 in Ref. [14] for details). \n- For a simplified one-layer Transformer **with or without** residuals, it is shown respectively in Ref. [46] and Ref. [41] that **both models** implement a step of gradient descent on a least-squares objective when achieving optimality. In other words, it seems that removing residual connections does not impair the representation capability of Transformer for in-context learning. \n\nTogether, these findings suggest that omitting residual connections for analytical simplicity does not undermine the core insight we build on in SkipV1Former. We hope these clues can address your concerns and we will include these discussions in the final version of the paper.\n\n> Moreover, though authors attempt to proof SkipV1Former benefits deep layers, the assumption stands scenarios with two layers, which might differ from the actual deep layers.\n\nThank you for your feedback. Theoretical analyses usually need to start with simple cases to build intuition. Our analysis takes a step towards understanding the benefit of direct injection of Values of layer 1 subsequent layers, and we have provided some discussions on the limitations in Appendix A.2. Extending our analysis to deeper models is an interesting direction, and we will leave this to future works."}, {"review_id": "JgttqioOnz", "reviewer": "Reviewer_t3Ns", "summary": "SkipV1Former introduces a simple yet powerful twist to Transformers: starting with the second block, half of every layer’s value heads are borrowed directly from the first layer. This “Value-1 skip” trims both value-projection parameters and the value cache by roughly 50 %, which translates to 25% of total KV-Cache memory saving. Converting an existing checkpoint costs only 10–15 % of the original pre-training compute, and the technique is also able to stack cleanly with methods like Group-Query Attention or YOCO for enabling more memory-efficient inference.", "questions": "Q1: How are the heads selected in each layer? What about dynamical selection for different layers? \n\nQ2: Can the author provide some insights on the selected heads? Some other works have shown that different kv_heads play different roles. Are the selected value heads for replacement the relatively redundant ones? \n\nQ3: What is the batch size for measuring the throughputs in Fig. 14 (b)? \n\nQ4: If lowering the compression rate, is SkipV1Former able to operate in a training-free fashion? Some related works, such as MiniCache [1] and xKV [2], have uncovered the opportunity in enabling cross-layer KV-Cache sharing in a post-training fashion. Can authors provide some discussion on this?\n\n[1] MiniCache: KV Cache Compression in Depth Dimension for Large Language Models\n\n[2] xKV: Cross-Layer SVD for KV-Cache Compression", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength:\n1. Interesting insights and good mathematical analysis for why skipping the first layer works, I really like this part in Section 4\n2. Seamlessly combined with various KV cache efficient architectures, gqa, mla, etc. Proves this method is versatile.\n3. Experiments are sufficient, from pretraining to uptraining and memory analysis are good, in combination with other works, YOCO.\n\n## Weakness:\n1. Limited Efficiency Gain: The benefit of SkipV1Former mainly lies in memory saving for throughputs/latency; the proposed methods only bring marginal improvements. \n2. Insufficient explanation on how heads are selected. It seems that you just keep the indices 0,1,2..., H/2 for the following layers. \n3. Some missed related works.\n    + At CLLA [1], the proposal of sharing the latent cache of MLA across different layers has been explored. I also suggested including this paper in the discussion. \n\n[1] Lossless KV Cache Compression to 2%", "was_revised": true, "final_justification": "The SkipV1Former provides a compelling architectural insight by introducing skip connections from the Value projections of the first layer directly to later layers. This design is supported by solid theoretical derivations and justifications. Empirically, the model demonstrates improved perplexity compared to baseline methods when trained from scratch, highlighting its effectiveness. While the gains in computational efficiency are modest, the conceptual novelty has the potential to inspire future work in both system-level optimizations and architectural innovations.", "rebuttal": "Thank you for your careful review and thoughtful questions and comments. Please see our response below.\n\n> Limited Efficiency Gain: The benefit of SkipV1Former mainly lies in memory saving for throughputs/latency; the proposed methods only bring marginal improvements.\n\nWe view SkipV1Former’s core contribution as the **combined gains** in accuracy and ~25% KV‑cache reduction. We acknowledge that throughput and latency improvements are modest. On the other hand, many KV-cache saving methods—such as SVD based compression— often require extra computation, which degrades throughputs. It is an interesting direction to further improve the inference efficiency of SkipV1Former. Please see our response to Question 3 below for more discussions on throughputs and latency. \n\n> Insufficient explanation on how heads are selected. It seems that you just keep the indices 0,1,2..., H/2 for the following layers.\n\nWe use a **fixed deterministic selection** strategy: for layers 2-L we always keep heads 0…H/2 and concatenate them with heads H/2+1 … H from layer 1. This continuity ensures a stable and aligned head ordering across layers, which is more hardware-friendly compared to other discontinuous methods [1] such as random selection. We will clarify this choice and include the core code implementation in the Appendix. \n\n\n[1] Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention\n\n> Some missed related works. At CLLA [1], the proposal of sharing the latent cache of MLA across different layers has been explored. I also suggested including this paper in the discussion.\n\nThank you for highlighting CLLA. We will cite and discuss its cross‑layer MLA cache sharing in the final version of our paper.\n\n> Q1: How are the heads selected in each layer? What about dynamical selection for different layers?\n\nPlease refer to our “fixed deterministic selection” answer above (Weakness 2) for how heads are selected. For a discussion of dynamic or random selection schemes and their comparative evaluation, please refer to our response to Weakness 1.2 to Reviewer 8cpz.\n\n> Q2: Can the author provide some insights on the selected heads? Some other works have shown that different kv_heads play different roles. Are the selected value heads for replacement the relatively redundant ones?\n\nThank you for your valuable questions. In Appendix C.2, we visualize the **cosine similarities** of all the heads in Layer 1 and 12 of SkipV1Former and standard Transformer. It is shown that SkipV1Former exhibits **lower** inter-head similarity, suggesting some reductions on the redundancy of the heads in standard Transformer. \n\nRegarding the role of heads, prior works have identified previous-token copying heads and induction heads in autoregressive tasks [2]. Specifically, [2] found that there exists a circuit implemented by two sorts of heads: \n- **previous-token copying head**: it finds the previous instance of the current token $A$ and the token that came after $A$ (call it $B$).\n- **induction head**: it “completes the pattern” by predicting $B$ after the current $A$. \n\nThis is closely related to our theoretical analysis based on a similar `copying mechanism’ in Section 4. We hypothesize that the skipping heads in SkipV1Former function analogously to previous token copying heads—leveraging layer 1’s complete raw information to facilitate pattern recall—while the remaining heads in layers 2–L perform more similar to induction heads. In effect, SkipV1Former replace some of the less important copying heads in layers 2–L and preserve the more important induction heads. We will include this discussion in the final version of the paper and validate our hypothesis or study other possible interpretation of the selected heads in the future. \n\n\n[2] In-context Learning and Induction Heads\n\n> Q3: What is the batch size for measuring the throughputs in Fig. 14 (b)?\n\nThank you for your valuable question. The batch size for this experiment is 1, which may lead to underutilization of GPU parallelism and is not representative of steady‑state performance. We have therefore re‑benchmarked inference throughput at batch sizes 8 and 16. As shown in the following table, SkipV1Former performs on par with—and in some cases slightly better than—the baseline (all differences within $\\pm 5$%). It seems that the per‑head concatenation of SkipV1Former does not incur measurable overhead as we stated in Appendix C.5, because SkipV1Former has reduced V‑projection matmul FLOPs by approximately 50\\%. We expect that the throughput can be further improved for SkipV1Former and leave this to future work. We will include these expanded results and discussions in the final version.\n\n**Table: Inference throughput (tokens/sec) at batch sizes 8 and 16 on an 48G A6000 GPU.**\n\n| Model | Base (B=8) | Skip (B=8) | Base (B=16) | Skip (B=16) |\n|:-----:|:----------:|:----------:|:-----------:|:-----------:|\n| 130M  |    118.6   |    117.9   |    127.0    |    124.8    |\n| 350M  |     59.0   |     61.5   |     53.0    |     54.9    |\n| 1B    |     48.2   |     45.7   |     30.7    |     29.5    |\n| 3B    |     27.3   |     26.2   |     16.2    |     16.7    |\n| 7B    |     13.0   |     13.2   |      —      |      —      |\n\n> Q4: If lowering the compression rate, is SkipV1Former able to operate in a training-free fashion?\n\nThank you for your valuable question. From our understanding, Minicache and xKV are based on the high similarity between adjacent KV states on the **token/feature dimensions** (or spectral components) of the cache. **Neither method changes the model architecture**. By contrast, SkipV1Former **modifies** the architecture, reducing redundancy **in the head dimension** of multi-head attention (or its variants).  After uptraining, SkipV1Former not only saves memory but also improves overall performance.\n\nInspired by Minicache and xKV, we think that if a pretrained Transformer already exhibited strong similarity between certain heads in layers 2–L and those in layer 1, one could inject those layer 1 heads directly—effectively a training free SkipV1Former. Prior studies have observed moderate head similarity across layers, but naive head sharing typically causes a marked accuracy drop [3]. Exploring similarity driven, training free methods to turn a standard model into SkipV1Former is an interesting direction. We leave this to future work and we will add a discussion of this potential extension in the final version of the paper. \n\n\n[3] Head-wise Shareable Attention for Large Language Models"}, {"review_id": "m9lefB8zg9", "reviewer": "Reviewer_8cpz", "summary": "This paper proposed a KV reduction/sharing technique which injects half of the V of first layer into later layers so that the computation and cache of later V layers will be reduced.  This technique requires retraining from scratch or reduced fine-tuning (uptraining) from existing MHA checkpoints in order to achieve comparable or slightly better performance than baseline models.  Furthermore, the author tested the compatibility of the proposed method with other popular KV cache reduction works, such as GQA and MLA.  A theoretical analysis was also provided in Appendix to give insights on how and when this method may or may not work, such as the case of ViT.", "questions": "see Weakness above", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n1.\tA good amount of decent works, including theoretical analysis, experiments with relevant techniques, tested on a range of model sizes, and the comprehensive details in Appendix.\n2.\tMethod seems simple enough but yet works reasonably fine.\n\n\nWeakness:\n\n1.\ta tad counter-intuitive\n\n    - based on author's analysis, the way this method works is to inject \"raw\" information from V1 into later layers which may enhance model's representation capability. In other words, the existing skip connections (residual path) in transformer architecture, i.e., after attention block and MLP block, are not sufficient. Intuitively the \"raw\" information could be mostly preserved through an identity-like head and the relative mixing ratio could be modulated by the magnitude of last Linear layer in each block before adding the residual, e.g. \"o_proj\" and \"down_proj\" in Llama. These should have been optimized during model training, therefore, the need for additional direct injection, especially only partial \"raw\" information, doesn't seem necessary.\n    - Even if we need to do a direct injection, why is 50% a lot better than others, (Fig 8a)? and why only inject the second half of V1, wouldn't some information still get lost this way? maybe something like an average pool of every 2 heads, as used by author's uptraining initialization, would make more sense? Also will it make any differences to inject the first half, or even/odd, or randomly selected heads? Maybe the author could provide more experimental data on this part to reinforce the hypothesis?\n    - It would be nice to include this discussion in the manuscript/appendix so that the readers could have a clearer picture of how this method works. \n\n2.\tgeneralization with newer architectures\n\nThe GQA example given in the manuscript is based on Llama (seems to be version 1) with group-of-two scheme (Appendix B2.) However, in newer Llama (version 2 70b or version 3), GQA is applied with group-of-4 configuration, i.e. K and V has 8 heads instead of 32. It would be more beneficial to demonstrate with the more recent and realistic settings. As many readers may assume that when the redundancy is reduced in such GQA scheme, SkipV1Former method might not be able to maintain 50% injection from V1, hence, the benefit of this method could be reduced. \n\n3.\tother minor suggestions\n    - the parameter count reduction from SkipV1Former in Table 2 seems to be too small. Almost all the parameters in the V layers are halved, but parameter counts is only reduced by ~6M out of ~335M? maybe add a few sentences in that paragraph or in Appendix to demonstrate the calculation?\n    - Appendix C5 provides some useful inference speed test results. Even though the throughput at decoding stage is not improved as expected, it would be nice to include a little more information regarding the preliminary analysis of the source of the overhead, e.g. the author seems to have performed a profiling and suggests that the stitching of half of the V1 into later Vs is causing the slow-down. However, it's unclear how this stitching is different from normal V cache retrieval and concatination. Maybe a little more clarification would help the readers to understand if this is a kernel optimization problem or something else.", "was_revised": true, "final_justification": "Concerns about the selection of heads to be injected is addressed with new experimental data. Question regarding GQA with group-of-4 configuration is partly address with new data using smaller models. Hence the adjustment of the rating.", "rebuttal": "Thank you very much for your careful review and thoughtful questions and comments. Please see our response below.\n\n> a tad counter-intuitive… The need for additional direct injection, especially only partial \"raw\" information, doesn't seem necessary\n\nThank you for your feedback. We found that some of the terminology such as “identity like head” is a bit confusing, so let us first restate our understanding of your concern: you are asking whether the existing residual paths already suffice to preserve “raw” first layer information in deeper layers, making our cross layer head injection unnecessary. In fact, there is an essential difference between the residual connections in standard model and our first-layer head injection: \n- In a standard Transformer, each layer’s input $X^k$ is linearly transformed by the attention and MLP sublayers and then **added back** to the original $X^k$ via the residual connection. This process **fuses only adjacent layers** and does not establish any direct Value to Value linkage between layer 1 and layers 2–L. \n- By contrast, SkipV1Former explicitly **concatenates** half of the Value heads from layer 1 with the Value heads of every subsequent layer (note that SkipV1Former also utilizes the conventional residual connections). \n\nAs shown in Ref. [14] (Fig. 2) and Section 4, even with residual connections, standard Transformer exhibit a “copying mechanism” in early layers that incurs information loss. On the other hand, direct head injection preserves these raw features, enabling deeper layers to optimize with more complete information. Empirically, across our experiments on GPT and LlaMA models, SkipV1Former consistently improves perplexity and downstream metrics over the baseline. This confirms that first layer head injection is not redundant but rather a distinct and beneficial mechanism beyond conventional residual connections.\n\n> Even if we need to do a direct injection, why is 50% a lot better than others, (Fig 8a)? \n\nThank you for your question. The choice of the skip ratio is based on both model performance and memory savings. Since we maintain the same number of heads of the concatenated $V$, a larger ratio of skipping heads implies a larger reduction on the embedding dimension of $W_V$ from layer 2-L. Skipping fewer than 50% heads yields less memory savings, while skipping 100% heads causes a clear drop in accuracy (see SVFormer in Ref. [25]). Intuitively, skipping too few heads results in certain information loss caused by `copying mechanism’ for deeper layers, whereas skipping too many heads hinders the capability of deeper layer to process the full data flow. Empirically, a 50% skip ratio strikes a good balance between memory savings and model quality as shown in Fig. 8a, supporting this design choice. We are extending these ablation studies to larger LLaMA models and will include the full results in the final paper (please see also our detailed response to Weakness 1 to Reviewer oBgN).\n\n\n> And why only inject the second half of V1, wouldn't some information still get lost this way? … Maybe the author could provide more experimental data on this part to reinforce the hypothesis?\n\nThank you for your valuable question. We choose to inject heads $H/2+1$ to $H$ from layer 1 because **continuous, block-wise selection** preserves a consistent head ordering across layers, which is more hardware friendly than discontinuous schemes like random head selection [1]. \n\nIn early experiments, we also tried injecting the first half of layer 1’s heads via dynamic strategies and ResFormer style mixing, but these variants delivered only marginal gains.  \nTo explore further, we conducted an extended ablation on LLaMA 350M with several head injection strategies:\n\n- **Pooling**: Injecting the head-wise average of the first half and second half heads in layer 1.\n- **Dynamic**: Injecting head $i$ to $i + H/2$ in layer 1 for the $i$-th layer in a rotating manner.\n- **Odd/Even**: Injecting layer 1’s first half heads for odd-indexed layers and the second half heads for even-indexed layers.\n- **SkipV1 + ResFormer**: Linearly add the first half heads to all the first half heads in subsequent layers upon SkipV1Former.\n\nAs the table shows, we found no substantial improvement of these strategies over the simple second half scheme. In short, simply injecting only the second half heads has achieved certain model performance and may be more hardware-friendly.  We will incorporate this discussion and the full experimental results in the final manuscript.\n\n**Table: Validation loss of different head injection strategies on LLaMA-350M model.**\n\n| Method | SkipV1 | Pooling | Dynamic | Odd/Even | SkipV1 + ResFormer | \n|:-----:|:----------:|:----------:|:-----------:|:-----------:|:-----------:|\n| Val. Loss  |    2.885   |    2.881   |    2.885    |    2.883    |    2.889   |\n\n\n\n[1] Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention\n\n> generalization with newer architectures: It would be more beneficial to demonstrate with the more recent and realistic settings\n\nThank you for your valuable suggestion. We agree that it will be better to experiment on more advanced models such as GQA-4 Transformers. However, due to our limited computing resources, we now only able to conduct the uptraining experiments on relatively small models. \nSpecifically, we compare the performance of SkipV1Former and the baseline model utilizing GQA with group-of-4 configuration on TinyLlama 1.1B models. We also test the performance on 315M and 125M TinyLlama models by modifying the configurations in the same manner as Llama 130M/350M with Llama 1B.\nIt is shown in the following table that SkipV1Former still outperforms the baseline model by a similar margin as that in Table 2 for GQA-2 models. We hope these results can address your concerns and we will include them in the final version of the paper. \n\n**Table: Validation loss across 3 TinyLlaMA model sizes (125M, 315M, 1.1B).**\n\n| Model | T-LlaMA (1.1B) | T-LlaMA-SkipV1 (1.1B) | T-LlaMA (315M) | T-LlaMA-SkipV1 (315M) | T-LlaMA (125M) | T-LlaMA-SkipV1 (125M) |\n|:-----:|:----------:|:----------:|:-----------:|:-----------:|:-----------:|:-----------:|\n| Val. Loss  |    2.750   |    2.742   |    2.936    |    2.916    |     3.420    |     3.370    |\n\n\n> The parameter count reduction from SkipV1Former in Table 2 seems to be too small.\n\nThank you for your feedback. We provide the detailed calculations as follows. \n\n- GQA model: the number of layers is 24, the total number of heads (of Queries) is 16, and the dimension of each head is 64, implying that $W_Q \\in \\mathbb{R}^{1024 \\times 1024}$. Since it share the same keys and values between every two Query heads, the number of heads of Keys and Values is 8, implying that $W_V\\in \\mathbb{R}^{1024\\times 512}$. As SkipV1Former reduces $W_V$ parameters by half from layer 2 - L, the reduced number of parameters is $1024\\times 256 \\times 23 = 6.03$M. \n- MLA model: the number of layers, total heads and dimension of value head is the same as those of GQA model. The kv_lora_rank, aka, $d_c$ in line 661 is set as 256 following the shrinking ratio of deepseek-v3 model. SkipV1Former-MLA cuts $d_c$ to 128 from layer 2 - L, resulting in the reduced number of parameters as $1024 \\times 128 \\times 23 = 3.02$M. \n\nNote that the reductions on parameter count is a by-product of our skip connection schemes, and the main contribution of SkipV1Former lies in performance and KV-cache savings. We will include all these calculations as well as those of the total number of parameters in Appendix in the final version of the paper. \n\n> However, it's unclear how this stitching is different from normal V cache retrieval and concatenation. Maybe a little more clarification would help the readers to understand if this is a kernel optimization problem or something else.\n\nThank you for your valuable suggestion. We clarify this as follows: during decoding, for each new token $x_t$,\n- Standard Transformer: it extends the cached V tensor $V[1:t-1] \\in \\mathbb{R}^{bs\\times H\\times (t-1) \\times d}$ of token $x_1$-$x_{t-1}$ by appending $x_t$’s Value along the **seq_len** dimension, yielding $ V[1:t-1]$.\n- SkipV1Former: it caches Head 1 to H/2 from layer 1 to L (denoted by $V^{l, 1:H/2}[1:t-1]$ for $l=1,\\cdots, L$) and the second half heads of layer 1 (denoted by $V^{1, H/2+1:H}[1:t-1]$). When computing attention for $x_t$, it first concatenates $V^{l, 1:H/2}[1:t-1]$ and $V^{1, H/2+1:H}[1:t-1]$ to those heads of $x_t$ along **seq_len** dimension, yielding $V^{l, 1:H/2}[1:t]$ and $V^{1, H/2+1:H}[1:t]$. Then it concatenates these two buffers along the **head** dimension, implementing the cross-layer skip connections.\n\nTo better illustrate this, we will include a schematic diagram in the revised Appendix. \n\nAdditionally, we have updated our throughput benchmarks at batch sizes 8 and 16, which are more representative of real world inference. Under these settings, SkipV1Former’s decoding throughput matchess or slightly exceeds the baseline (±5%), rather than the average 5% slowdown reported in Appendix C.5. Please see our response to Q3 to Reviewer t3Ns for details.\n\nSince SkipV1Former reduces V projection matmuls by nearly 50%, we think that the throughput of can be further improved through techniques like kernel optimization (e.g., fused concat). We leave this exploration to future work and we will include the discussions in the final version of the paper."}, {"review_id": "ymLg0SwBW6", "reviewer": "Reviewer_oBgN", "summary": "They provide a method called SkipV1Former, which allows each layer to reuse half of its Value heads from the very first layer. Their method utilizes the property of self-attention can save KV cache and improve GPU memory usage, and at the same time not degrade the performance.", "questions": "1. Based on Figures 14a and 14b, it seems that the larger model gets improved less than the smaller model with SkipV1Former. Also, in Figure 5a, it seems the larger model also has a worse perplexity improvement. I wonder if 50% is still the best for the larger models (1B, 3B)\n2. What is the reason for the intuition that shallow layers would be more effective than deeper layers.", "limitations": "1. The improvement for the larger model is a bit small. How they choose how much of the Value to be replaced seems to only depend on the ablation study of GPT2-355M.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The motivation is clear. The authors clearly state the drawbacks of the two categories of skip-connection techniques. Their designed method solves those two problems simultaneously.\n2. Their proposed method is flexible, which can also be combined with other KV-saving approaches to attain greater savings.\n\n## Weaknesses\n1. The effectiveness of the method on larger models is limited. Based on Table 1 and Figure 5a, the absolute value of the improvement of the test accuracy and the perplexity seems to be a bit small. It would be better if the author could include a significance test. \n2. It is unclear why the method seems to be more effective for larger models. In the ablation study (Section 6.5), different skip-head ratios are only tested on GPT2-355M, which is relatively small compared to LLaMa models. Therefore, the reasons for choosing 50% seem not very convincing. It would be better if authors could include ablation studies for larger models.\n3. The author discusses why the first layer matters in Section 4. In line 136, the authors argue that \"it seems natural that injecting shallow-layer features into deeper layers enriches representation.\" However, I am confused why this is natural. It would be better if the author could provide some references. In line 137, it points to an ablation study, which only studies the first three layers. It would be better if the author could provide a study on deeper layers to prove their claim. \n4. For clarity, it is confusing how the proposed method improved efficiency. Discussion of how it improved prefill time, throughput, and GPU memory is not emphasized in the main text. Most of them are in the appendix. I would suggest the author move those discussion back to main text", "was_revised": true, "final_justification": "The author addressed most of my concerns. I have increased my score.", "rebuttal": "Thank you for your careful review and thoughtful questions and comments. Please see our response below.\n> The effectiveness of the method on larger models is limited. Based on Table 1 and Figure 5a, the absolute value of the improvement of the test accuracy and the perplexity seems to be a bit small. It would be better if the author could include a significance test.\n\nThank you for your feedback. We deem that the main contribution of SkipV1Former is the **simultaneous improvement in improving accuracy and reducing KV cache**. While SkipV1Former’s absolute gains in loss and perplexity may appear modest on larger models, it consistently reduces KV cache by ~25% and delivers reliable quality improvements across scales. \n\nTo address your concerns, we retrain 1B SkipV1Former and baseline model using another random seed (The original random seed is 42. Due to our limited computation resources, now we are only able to run one set of experiment). It is shown in the following table that SkipV1Former outperforms the baseline by a similar margin. In addition, we also conduct experiments on TinyLlaMA 1.1B and find similar results. Please refer to our response to Weakness 2 to Reviewer 8cpz. We will complete the significance test on 1B models in the final version of the paper.  \n\n**Table: Validation loss of 1B-LlaMA models under different random seed.**\n\n| Model | Baseline (seed 42) | SkipV1 (seed 42) | Baseline (seed 100) | SkipV1 (seed 100) | \n|:-----:|:----------:|:----------:|:-----------:|:-----------:|\n| Val. Loss  |    2.723   |    2.711   |    2.753    |    2.743    |\n\n\n> It is unclear why the method seems to be more effective for larger models. In the ablation study (Section 6.5), different skip-head ratios are only tested on GPT2-355M, which is relatively small compared to LLaMa models. Therefore, the reasons for choosing 50% seem not very convincing. It would be better if authors could include ablation studies for larger models.\n\nFor SkipV1Former, we choose a 50% skip head ratio to balance memory savings and model performance. Injecting fewer than 50% of the first layer heads (e.g., 25%) yields limited improvement in memory usage, whereas injecting all heads—as in SVFormer (see Ref. [25]. SVFormer is a variant of ResFormer, which happens to be an extreme variant of SkipV1Former that choose 100% skip-head ratio)—introduces significant performance degradation compared to the baseline. \n\nTo further validate this trade off, we conducted a preliminary ablation on the 1B parameter LLaMA model. \nIt can be seen that, although a 75% skip ratio does provide additional memory reduction, it incurs a noticeable drop in accuracy. Conversely, a 25% skip ratio fails to outperform the 50% setting. These results suggest that 50% may represents a near-optimal tradeoff between memory efficiency and model quality. We are currently extending these experiments to explore additional skip ratios, and will include the complete ablation study in the final version of the paper. We hope our additional results can address your concerns. \n\n**Table: Validation loss of 1B-LlaMA models with different skip-head ratios.**\n\n| Skip Head Ratio/Model | 25% | 50% (default) | 75% | baseline |\n|:-----:|:----------:|:----------:|:-----------:|:-----------:|\n| Val. Loss  |    2.716   |    2.711   |    2.717    |    2.723    |\n\n\n> The author discusses why the first layer matters in Section 4. In line 136, the authors argue that \"it seems natural that injecting shallow-layer features into deeper layers enriches representation.\" However, I am confused why this is natural. It would be better if the author could provide some references. In line 137, it points to an ablation study, which only studies the first three layers. It would be better if the author could provide a study on deeper layers to prove their claim.\n\n\nThank you for pointing out this ambiguity. The comparison here is between **SkipV1Former**, which injects half of layer 1’s heads into every deeper layer, and **SkipV0** in Fig. 8(b), Section 6.5, which simply drops those heads from layers 2–L without any first‐layer reuse. Our intent is to highlight the importance of the first layer. Specifically, the attention of SkipV1 can be expressed as the equation in line 122 and the attention of SkipV0 is $\\text{Attn}(X) = X + \\sum_{h=1}^{H’} W_O^h V^h \\text{softmax}((K^h)^{\\top}Q^h)$. Therefore, SkipV1Former’s attention can express SkipV0’s output (e.g., $W^h_O = \\textbf{0}$ for $h=H'+1,\\cdots,H$), implying an improved representation capability. This is why we say `naturally enriches representation’. Similar injection/concatenation techniques have been used in DenseNet [1] to improve representation. We will modify the statement of this paragraph in the final version of the paper.\n\nTo validate our claim beyond the first two layers (Fig. 8(b)), we extended the ablation studies at layer 3, 4, and 6 (Since inserting Value heads at deeper layer can only cut heads from that layer onward, leading to less V cache and parameter savings, we stop at a layer 6). As shown in the tale below, injecting heads of layer 1 consistently outperforms all other variants, confirming the unique importance of first layer head reuse. We will include these detailed results and discussions in the final version of the paper.\n\n**Table: Relative validation loss for cross-layer V at different layers.**\n\n| Model | SkipV1 | SkipV0 | SkipV2 | SkipV3 | SkipV4 | SkipV6 | \n|:-----:|:----------:|:----------:|:-----------:|:-----------:|:-----------:|:-----------:|\n| Relative Val. Loss  |    0   |      +0.042    |     +0.025    |     +0.025    |     +0.019   |  + 0.041  | \n\n\n[1] Densely Connected Convolutional Networks\n\n> For clarity, it is confusing how the proposed method improved efficiency. Discussion of how it improved prefill time, throughput, and GPU memory is not emphasized in the main text. Most of them are in the appendix. I would suggest the author move those discussions back to main text.\n\nThank you for your valuable suggestions. As shown in line 122-125 and Figure 1 (Right), SkipV1Former cuts half of the $W_V$ parameters from layer 2 - L compared to standard Transformer. This removal directly reduces the **number of heads of every V cache** (aka., the embedding dimension of V) from layer 2 - L and also the **corresponding matmul computations** of $W_V$, resulting in improvement on GPU memory, prefill time and throughput. Due to the limit of pages, we had to move some discussions on prefill time and throughput to the Appendix in the original manuscript, as the main contribution of SkipV1Former lies in performance boosting and KV cache savings. We will modify our discussions on how SkipV1Former improves inference and memory and move the discussions in the Appendix back to the main text. \n\n\n> Based on Figures 14a and 14b, it seems that the larger model gets improved less than the smaller model with SkipV1Former. Also, in Figure 5a, it seems the larger model also has a worse perplexity improvement. I wonder if 50% is still the best for the larger models (1B, 3B)\n\nPlease see our response for Weakness 2 above. We think that 50% is a near-optimal ratio for the tradeoff between performance and memory savings. We extend our ablation on skip-head ratio on 1B models and find that 50% is still the best (please see our response to Weakness 2 above). We will conduct more ablation studies on larger models in the final version of the paper.\n\n\n> What is the reason for the intuition that shallow layers would be more effective than deeper layers.\n\nThank you for your feedback. We first clarify that we do **not** claim that shallow layers are **strictly** more effective than deep layers as we only inject half of the heads of layer 1 instead of all heads nor heads from other shallow layers. In Section 4 and Appendix A, we present some discussions and theoretical analysis to show why injecting certain information from shallow layers directly can be beneficial. In short, trained Transformer performs `copying mechanism’ in shallow layers in auto-regressive tasks, which incurs certain information loss in the propagation and thus slower down the mesa-optimization. By contrast, by directly injecting half of layer 1’s heads, SkipV1Former restores access to the raw information of the input, enabling deeper layers to perform mesa-optimization with more complete information.\nMoreover, deeper layers themselves tend to develop certain redundant representations in standard Transformer (e.g., see Ref. [15]). Thus, removing half heads from layer 2-L and concatenate the rest with the second half of the layer 1 brings gains. We will refine this discussion in the final paper.\n\n> The improvement for the larger model is a bit small. How they choose how much of the Value to be replaced seems to only depend on the ablation study of GPT2-355M.\n\nPlease see our response to Weakness 1 and 2 above. Importantly, SkipV1Former consistently reduces KV cache by ~25% while still delivering accuracy gains compared to the standard Transformer—even at larger model scales."}]

Author rebuttal:

Meta-review: All reviewers agreed this paper should be accepted: it includes a nice mathematical analysis motivating their proposed solution, it modularly combines with other efficient methods, and has useful experiments. The main concerns were around (a) limited improvements: e.g., reviewer oBgN pointed out the marginal improvements in accuracy. The reviewers responded that the method provides both improvements in accuracy and KV-cache reduction, pointing out that other KV-cache-saving methods, e.g., SVD-based compression methods, require additional computation. This answers the concern. Another concern was around (b) comparison with other methods: e.g., reviewer HPMS wanted to see downstream task comparisons. These were provided by the authors, addressing this concern as well. Given these and other responses by authors to reviewer concerns, the paper is a clear accept. Authors please remember to include all reviewer requests into the camera-ready paper. Thank you!
Decision: Accept (poster)