Title: Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads
Venue: NeurIPS 2025
Accepted: yes
Keywords: Transformer architecture

representation

KV cache

skip connections
Abstract: Transformer models have driven breakthroughs across various language tasks by their strong capability to learn rich contextual representations. Scaling them to improve representation, however, often demands substantial memory and compute costs, such as the Key-Value (KV) cache used during auto-regressive decoding. Skip connections offer a promising way to improve representation without bloating resource usage, yet most prior works either improve expressivity while leaving KV costs unchanged, or reduce memory at the cost of weaker representation. In this work, we propose SkipV1Former, a  Transformer variant that uses skip connections from the first layer's Value heads to strengthen model representation and reduce KV cache.  Specifically, from the second block onward, each layer reuses half of its Value heads from the very first layer, while computing the other half as usual-cutting Value projections and V cache by nearly 50 \%. Theoretically, we show that routing uncompressed first-layer Values into deeper layers restores information lost to compression and accelerates the model’s implicit mesa-optimization-a key pattern of Transformer in auto-regressive tasks. Empirically, across different model scales, SkipV1Former delivers consistent reductions of approximately 25 \% in KV cache while improving perplexity relative to standard Multi-Head Attention (MHA) Transformers and some advanced variants. Moreover, we propose a recipe for uptraining existing MHA Transformer checkpoints to SkipV1Former with only 10-15\% additional compute. Finally, SkipV1Former can seamlessly combine advanced methods like Group-Query Attention and Multi-Latent Attention to achieve further KV cache savings and performance improvement. When combined with YOCO, it cuts KV cache size by nearly 50 \% while still improving performance. The code is
 available at: https://github.com/Zhoutong-Wu/SkipV1Former.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThey provide a method called SkipV1Former, which allows each layer to reuse half of its Value heads from the very first layer. Their method utilizes the property of self-attention can save KV cache and improve GPU memory usage, and at the same time not degrade the performance.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The motivation is clear. The authors clearly state the drawbacks of the two categories of skip-connection techniques. Their designed method solves those two problems simultaneously.\n2. Their proposed method is flexible, which can also be combined with other KV-saving approaches to attain greater savings.\n\n## Weaknesses\n1. The effectiveness of the method on larger models is limited. Based on Table 1 and Figure 5a, the absolute value of the improvement of the test accuracy and the perplexity seems to be a bit small. It would be better if the author could include a significance test. \n2. It is unclear why the method seems to be more effective for larger models. In the ablation study (Section 6.5), different skip-head ratios are only tested on GPT2-355M, which is relatively small compared to LLaMa models. Therefore, the reasons for choosing 50% seem not very convincing. It would be better if authors could include ablation studies for larger models.\n3. The author discusses why the first layer matters in Section 4. In line 136, the authors argue that \"it seems natural that injecting shallow-layer features into deeper layers enriches representation.\" However, I am confused why this is natural. It would be better if the author could provide some references. In line 137, it points to an ablation study, which only studies the first three layers. It would be better if the author could provide a study on deeper layers to prove their claim. \n4. For clarity, it is confusing how the proposed method improved efficiency. Discussion of how it improved prefill time, throughput, and GPU memory is not emphasized in the main text. Most of them are in the appendix. I would suggest the author move those discussion back to main text\n\nQUESTIONS:\n1. Based on Figures 14a and 14b, it seems that the larger model gets improved less than the smaller model with SkipV1Former. Also, in Figure 5a, it seems the larger model also has a worse perplexity improvement. I wonder if 50% is still the best for the larger models (1B, 3B)\n2. What is the reason for the intuition that shallow layers would be more effective than deeper layers.\n\nLIMITATIONS:\n1. The improvement for the larger model is a bit small. How they choose how much of the Value to be replaced seems to only depend on the ablation study of GPT2-355M.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a KV reduction/sharing technique which injects half of the V of first layer into later layers so that the computation and cache of later V layers will be reduced.  This technique requires retraining from scratch or reduced fine-tuning (uptraining) from existing MHA checkpoints in order to achieve comparable or slightly better performance than baseline models.  Furthermore, the author tested the compatibility of the proposed method with other popular KV cache reduction works, such as GQA and MLA.  A theoretical analysis was also provided in Appendix to give insights on how and when this method may or may not work, such as the case of ViT.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1.\tA good amount of decent works, including theoretical analysis, experiments with relevant techniques, tested on a range of model sizes, and the comprehensive details in Appendix.\n2.\tMethod seems simple enough but yet works reasonably fine.\n\n\nWeakness:\n\n1.\ta tad counter-intuitive\n\n    - based on author's analysis, the way this method works is to inject \"raw\" information from V1 into later layers which may enhance model's representation capability. In other words, the existing skip connections (residual path) in transformer architecture, i.e., after attention block and MLP block, are not sufficient. Intuitively the \"raw\" information could be mostly preserved through an identity-like head and the relative mixing ratio could be modulated by the magnitude of last Linear layer in each block before adding the residual, e.g. \"o_proj\" and \"down_proj\" in Llama. These should have been optimized during model training, therefore, the need for additional direct injection, especially only partial \"raw\" information, doesn't seem necessary.\n    - Even if we need to do a direct injection, why is 50% a lot better than others, (Fig 8a)? and why only inject the second half of V1, wouldn't some information still get lost this way? maybe something like an average pool of every 2 heads, as used by author's uptraining initialization, would make more sense? Also will it make any differences to inject the first half, or even/odd, or randomly selected heads? Maybe the author could provide more experimental data on this part to reinforce the hypothesis?\n    - It would be nice to include this discussion in the manuscript/appendix so that the readers could have a clearer picture of how this method works. \n\n2.\tgeneralization with newer architectures\n\nThe GQA example given in the manuscript is based on Llama (seems to be version 1) with group-of-two scheme (Appendix B2.) However, in newer Llama (version 2 70b or version 3), GQA is applied with group-of-4 configuration, i.e. K and V has 8 heads instead of 32. It would be more beneficial to demonstrate with the more recent and realistic settings. As many readers may assume that when the redundancy is reduced in such GQA scheme, SkipV1Former method might not be able to maintain 50% injection from V1, hence, the benefit of this method could be reduced. \n\n3.\tother minor suggestions\n    - the parameter count reduction from SkipV1Former in Table 2 seems to be too small. Almost all the parameters in the V layers are halved, but parameter counts is only reduced by ~6M out of ~335M? maybe add a few sentences in that paragraph or in Appendix to demonstrate the calculation?\n    - Appendix C5 provides some useful inference speed test results. Even though the throughput at decoding stage is not improved as expected, it would be nice to include a little more information regarding the preliminary analysis of the source of the overhead, e.g. the author seems to have performed a profiling and suggests that the stitching of half of the V1 into later Vs is causing the slow-down. However, it's unclear how this stitching is different from normal V cache retrieval and concatination. Maybe a little more clarification would help the readers to understand if this is a kernel optimization problem or something else.\n\nQUESTIONS:\nsee Weakness above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSkipV1Former introduces a simple yet powerful twist to Transformers: starting with the second block, half of every layer’s value heads are borrowed directly from the first layer. This “Value-1 skip” trims both value-projection parameters and the value cache by roughly 50 %, which translates to 25% of total KV-Cache memory saving. Converting an existing checkpoint costs only 10–15 % of the original pre-training compute, and the technique is also able to stack cleanly with methods like Group-Query Attention or YOCO for enabling more memory-efficient inference.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength:\n1. Interesting insights and good mathematical analysis for why skipping the first layer works, I really like this part in Section 4\n2. Seamlessly combined with various KV cache efficient architectures, gqa, mla, etc. Proves this method is versatile.\n3. Experiments are sufficient, from pretraining to uptraining and memory analysis are good, in combination with other works, YOCO.\n\n## Weakness:\n1. Limited Efficiency Gain: The benefit of SkipV1Former mainly lies in memory saving for throughputs/latency; the proposed methods only bring marginal improvements. \n2. Insufficient explanation on how heads are selected. It seems that you just keep the indices 0,1,2..., H/2 for the following layers. \n3. Some missed related works.\n    + At CLLA [1], the proposal of sharing the latent cache of MLA across different layers has been explored. I also suggested including this paper in the discussion. \n\n[1] Lossless KV Cache Compression to 2%\n\nQUESTIONS:\nQ1: How are the heads selected in each layer? What about dynamical selection for different layers? \n\nQ2: Can the author provide some insights on the selected heads? Some other works have shown that different kv_heads play different roles. Are the selected value heads for replacement the relatively redundant ones? \n\nQ3: What is the batch size for measuring the throughputs in Fig. 14 (b)? \n\nQ4: If lowering the compression rate, is SkipV1Former able to operate in a training-free fashion? Some related works, such as MiniCache [1] and xKV [2], have uncovered the opportunity in enabling cross-layer KV-Cache sharing in a post-training fashion. Can authors provide some discussion on this?\n\n[1] MiniCache: KV Cache Compression in Depth Dimension for Large Language Models\n\n[2] xKV: Cross-Layer SVD for KV-Cache Compression\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes SkipV1Former, a Transformer variant that reuses half of the Value heads from the first layer in subsequent layers, cutting KV cache by while improving perplexity. It theoretically shows this design restores compressed information and accelerates mesa-optimization, validated by experiments on GPT-2 and LLaMA models, which demonstrate consistent performance gains. An uptraining method converts pre-trained MHAs with extra compute, and SkipV1Former integrates with techniques like GQA for further cache savings. However, it underperforms on non-autoregressive vision tasks, highlighting domain-specific effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. **Innovative Architecture Design**: Introduces SkipV1Former, which strategically reuses half of the Value heads from the first layer in subsequent layers, reducing KV cache while improving model representation. This addresses the trade-off between representation and memory efficiency, outperforming prior skip-connection methods that either sacrificed performance for memory savings or vice versa.\n2. **Compatibility with Advanced Methods**: Seamlessly integrates with techniques like Group-Query Attention and Multi-Latent Attention, achieving further KV cache reductions while maintaining or improving performance.\n\n### Weaknesses\n1. **Lack comprehensive comparison with other methods**: Though the paper shows the validation loss comparison with related methods (ResFormer) comprehensively in Figure 4, it would be better to report a more comprehensive comparison with other methods (ResFormer) on downstream tasks in Table 1.\n2. **Assumption for the theoretical analysis is too strict**: Appendix A attempts to proof that SkipV1Former gains strength in the aspect of \"copying mechanism\" and in-context-learning. However, the assumption requires no residual connection, which might already have impact on deep layer. Moreover, though authors attempt to proof SkipV1Former benefits deep layers, the assumption stands scenarios with two layers, which might differ from the actual deep layers.\n\nQUESTIONS:\nMention in weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}