Uncovering the Redundancy in Transformers via a Unified Study of Layer Dropping
Authors: Shwai He
Guoheng Sun
Zheyu Shen
Ang Li
Venue: Accepted by TMLR ()

Abstract: While scaling Transformer-based large language models (LLMs) has demonstrated promising performance across various tasks, it also introduces redundant architectures, posing efficiency challenges for real-world deployment. Despite some recognition of redundancy in LLMs, the variability of redundancy across different architectures in transformers, such as MLP and Attention layers, is under-explored. In this work, we investigate redundancy across different Transformer modules, including blocks, MLP layers, and attention layers, through the lens of layer dropping. Surprisingly, despite the pivotal role of attention mechanisms in distinguishing Transformers from other architectures, we find that a large portion of attention layers exhibit excessively high redundancy and can be pruned without degrading performance. For example, LLaMA-3-70B achieves a 43.4\% speedup with only a 1.8\% drop in performance by pruning half of its attention layers. In contrast, dropping MLP layers severely impairs the model's ability to distinguish between tokens, leading to catastrophic performance degradation. Moreover, our analysis reveals that attention layer redundancy persists not only throughout training but is also evident in randomly initialized models. We attribute this redundancy to three key factors that constrain representational updates from attention layers: sparse attention patterns, over-smoothed token embeddings, and the low representational magnitude of attention outputs. Overall, our findings offer valuable insights into the internal redundancy of Transformer architectures and provide practical guidance for designing more efficient LLMs. The code is released at: https://github.com/CASE-Lab-UMD/LLM-Drop.

Published in Transactions on Machine Learning Research (02/2026)
Uncovering the Redundancy in Transformers via a Unified Study of Layer Dropping
Shwai He", Guoheng Sun", Zheyu Shen, Ang Li
University of Maryland, College Park
{shwaihe, ghsun, zyshen, anglicce) ) @umd. edu
Reviewed on OpenReview:
https: openreview. net/ forun? id= 117PCb0Pfe
Abstract
While scaling Transformer- based large language models (LLMs) has demonstrated promising performance across various tasks, it also introduces redundant architectures, posing efficiency challenges for real-world deployment. Despite some recognition of redundancy in LLMs, the variability of redundancy across different architectures in transformers, such MLP and Attention layers, is under-explored. In this work, we investigate redundancy across different Transformer modules, including blocks, MLP layers, and attention layers, through the lens of layer dropping. Surprisingly, despite the pivotal role of attention mechanisms distinguishing Transformers from other architectures, we find that large portion of attention layers exhibit excessively high redundancy and can be pruned without degrading performance. For example, LLaMA-3- 30B achieves 43.4% speedup with only 1.8% drop in performance by pruning half of its attention layers. In contrast, dropping MLP layers severely impairs the model's ability to distinguish between tokens, leading to catastrophic performance degradation. Moreover, our analysis reveals that attention layer redundancy persists not only throughout training but is also evident in randomly initialized models. We attribute this redundancy to three key factors that constrain representational updates from attention layers: sparse attention patterns, ovor-smoothed token embeddings, and the low representational magnitude of attention outputs. Overall, our findings offer valuable insights into the internal redundancy of Transformer architectures and provide practical guidance for designing more efficient LLMs. The code is released at: https:///ii //gttb/b/ com -AbS/--a---UDD//AAAA
1 Introduction
Transformer-based large language models (LLMs) have significantly advanced Al research, achieving re- markable performance across various domains (OpenAT 2024
Team 2024). However, scaling these models also introduces redundant architectures, namely overarching, leading to inefficiencies that complicate their e.g., inflating deployment costs and resource demands. Although several previous works have been proposed to promote LLM efficiency via removing redundant parameters or architectures Frantarr et al. 2023 Sun et al. 2023). these approaches often employ universal techniques that overlook the unique characteristies of transformer architectures. Specifically, trans- former (Vaswani 2017) architectures are composed of multiple stacked blocks, each containing an MLP layer and an Attention layer, which serve distinct functions and exhibit corresponding different levols of redundancy (Wang et al.) 2023 Shi et al.] 2023) This motivates a deeper investigation into the specific redundancies within Transformers, with the goal of identifying and addressing the most critical modules.
real-world deployment (Frantar al. 2023 Sun al.
2023
In this work, we systematically explore the redundancy in three key Transformer components: Block, MLP, and Attention. Using similarity-basse metric (Gromov et al. 2024 Men al. 2024 evaluate the importance of each component and progressively drop those identified as redundant. We first apply a " BBookk
"Equal contribution

---

Published in Transactions on Machine Learning Research (02/2026)
Drop" approach but observe that removing entire blocks leads to significant performance degradation. This suggests a need for more fine-grained strategy.
Upon further examination, we explore the separate pruning of MLP and Attention layers. Our findings reveal that while dropping MLP layers negatively affects performance, a substantial portion of Attention layers, i.e., the core of Transformer architectures which distinguish it from other mainstream architectures (e.g., RWKV (Peng et al., 2023b) and Mamba (Gu & Dao 2024)), can be pruned without degrading the model's performance. For instance, dropping 50% of the Attention layers in Llama-3-70B (Touvron et al., 2023) results in comparable performance to the full model, indicating high degree of redundancy in these layers.
Our work provides unified analysis of two key architectural components in Transformers: MLP and attention layers. On the one hand, MLP layers, which perform token transformations in high-dimensional space, contribute significantly to the hidden state representations. Without them, Transformer models suffer degraded sensitivity to token distinctions, resulting in substantial performance loss. In contrast, attention layers display notable redundancy. This redundancy persists across various training stages and is even observable in randomly initialized models, indicating that it is an inherent property of the architecture. We attribute this phenomenon to three key factors in attention architectures: (1) the sparsity of attention matrices, (2) the excessively high similarity between token embeddings, and (3) the low reprosentational magnitude of the attention branch. These factors collectively constrain the representational updates contributed by attention layers to the residual branch, thereby making certain attention layers amenable to pruning. These findings offer valuable insights into the internal redundancy of Transformer architectures and provide practical guidance for designing more efficient LLMs. In summary, our key contributions are follows:
Through an in-depth analysis of redundancy in three key Transformer components, we uncover a surprising level of redundancy within the Attention.
We propose Allention Drop, a simple yet effective algorithm for removing less important atlention layers in a training-fiee free manner, significantly improving efficiency without sacrificing performance.
Our extensive experiments demonstrate the effectiveness of dropping Attention, for instance, removing 50% of the attention layers in Llama-3-70B results in only a 1.8% performance reduction while achieving up to a 43.4% speodup.
We further demonstrate that attention layers possess inherent properties that lead to consistently high redundancy, both at initialization and throughout the training process. This observation offers valuable insights for future architectural design.
2 Related Works
Large Language Models
Although Transformer-bant Large Language Models (LLMs) have demonstrated promising performance across various tasks, their deployment costs still remain significant challenge for practical usage (Sun et al. 2023 Lin et al.) 2024 Gromov et al.) 2024). Transformer [Vaswani 2017 models consist of multiple blocks, which include Attention layers and MLP layers. Attention layers compute the contextual information between input tokens with quadratic complexity concerning the input sequence length
Li et al. 2020 KV-Cache (Pope et al.] 2022) mitigates the computational issuc but results in excessive memory costs iZhang el al.) 2023). MLP layers (Liu al.] 2021 Mai et al., 2022) transform each Loken independently, using an up-projection followed by (own-proojction and contribute most of the model parameters. Recent works have revealed that not all blocks or layers are equally important (Men al., 2024 Chen et al.] 2024), which urges us to reffect on the structured redundancy within LLMs and the potential design of more compact architectures.
Model Compression LLMs can be compressed to promote their efficiency in memory and computation. Quantization (Frantar et al., 2023 Lin et al. 2024 and Pruning (Sun et al. 2023 Frantar & Alistarh 2023 are the most widely used techniques to compress LLMss. Specifically, quantization transforms the data type into low-bit but remains potentially redundant architecture and parameters. Pruning can be categorized into
unstructured pruning (Kusupati et al. 2020 Sanh et al. 2020) and structured pruning (Zhuang al. 2020
2

---

Published in Transactions on Machine Learning Research (02/2026)
Kwon et al. 2020). While unstructured pruning maintains better performance than structured pruning, it cannot be effectively applied to hardware, limiting its practical usage. Our methods, Block Drop and Layer Drop, focus on removing structured modules rather than fine-grained parameters, creating hardware-frient efficient architectures while maintaining comparable performance. Additionally, Block Drop and Layer Drop are orthogonal to quantization, and their integration with quantization significantly enhances elficiency.
3 Methodology
In this section, we present the methodology for identifying and removing redundant modules in LLMs. We begin by introducing a similarity-based metric to assess redundancy for layer importance. Based on the insights gained from this analysis, we develop two targeted techniques, i.e., MLP Drop and Attention Drop, efficiently eliminate redundant components while preserving model performance.
3.1 Preliminaries
Similarity-based Drop To assess the redundancy of modules in LLMs, we employ a similarity-based metric that evaluates the importance of each module by measuring the similarity between its input and output [Gromov et al. 2024). The underlying hypothesis is that redundant modules produce outputs that are similar to their inputs, implying minimal transformation. In contrast, important modules are expected significantly alter their inputs and thus should be preserved. The similarity between the hidden states of the imput X and output Y of module is quantified using cosine similarity. The importance score S of the module is computed as:
S 1 ( X, Y).
(1)
Modules with higher cosine similarity exhibit lower importance scores, indicating redundancy. We identify and prune the mod- ules with the lowest importance scores according to a predefined prunting ratio. A complete evaluation of the effectiveness of cosine similarity as the dropping metric provided in Table
Head
112
Block Drop Transformer models are composed stacked blocks, where each block shares a common architecture and be viewed as subnetwork. To reduce complexity, we first consider dropping entire blocks that are deemed unimportant.
As shown in Figure 2 Transformer blocks operate sequentially, with each block's ' output feeding into the next. To evaluate redundancy, we compute the similarity between the input and output of each block. For the I-th block, the importance score is calculated as:
Embedding
Embedding
Figure 1: Visualization of Block Drop. where we use t to denote the blocks with high similarity scores. The dropped blocks are blurred.
1/B = 1 CosineSim(XX) ( Y2),,
(2)
where X/2 and Y denote the input and output of the I-th block, respectively. Since the similarity scores are computed locally, we can offload irrelevant modules to save memory. By iteratively computing the importance scores for each block from shallow to deep, we identify and drop blocks with the lowest. scores, thus saving memory and computational resources.
Dotasee
CA
Mathinstruct
27
Layer Index
0.0
0.1
0.2
03
Figure 2: Importance scores of Blocks.
3.2 Motivation
Block Drop is an aggressive technique that risks removing essential layers, as it overlooks the internal ine-grained architectures within each block. Given a transformer block consists of both an Attention layer and an ML.P layer. These layers perform distinct functions, with the Attention layer facilitating contextual
3

---

Published in Transactions on Machine Learning Research (02/2026)
UIA
Dalazes
LIA
FadeAlpaca
EodeAlaace
11 19
31
11 15
27
29
Layer Index
Layer Index
0.2
Figure 3: Importance scores for MLP (left) and Attention (right) layers, using various calibration datasets for comprehensive analysis.
information flow between tokens and the MLP layer transforming the token representations. Given their distinct roles, we assess the redundancy of cach layer separately by mcasuring the importance scores of Attention and MLP layers individually. Specifically, we leverage multiple calibration datasets to measure the importance scores, ranging from the pretraining dataset (e.g., C4 (Raffel et al. (020 to instruction fine-tuning datasets (e.g., CodeAlpaca-200: (Rozière et al. 2024), Mathinstruct (Yue al. 2024 and LIMA (Zhou et al,, 2024)). Figure 2 and 3 illustrate the varying trend of importance scores for Attention layers compared MLP layers across multiple datasets. This observation motivates us to consider the varying levels of redundancy between MLP and Attention layers and develop more fino-grained dropping techniques accordingly, namcly, MLP Drop and Attention Drop.
3.3 Layer Drop.
MLP Drop As illustrated in Figure 4 each MLP layer follows LayerNorm operation and involves a residual connection, ensuring that part of the input is preserved in the final output. Given the input X of the LayerNorm before MLP at the (-th Block, the output Y can be formulated as:
Y== = X MTP(LayerNorm
3
Since the output Y contains both the residual and the MLP transformation, evaluating similarity based solely on the MLP's s output can be misleading. To address this, consider the MT.P layer and its associated LayerNorm as single unit and compute their importance score as follows:
Sy 1 CosineSim(X$,r Ytr)
By treating these layers a single entily, we ensure a more accurate measure of importance. MLP Drop removes both the unimportant MLP and associated LayerNorm layers. Further validation of this approach can be found in Appendix A
Attention Drop Similarly, Attention layers also operate within a residual connection. The output of the [-th Attention layer computed as:
+ (aayetin (x.)))
5
where X is the inputs of the corresponding LayerNorm layers and Y overall outputs that involves residual connections. Like MLP Drop, we assess both the Attention layer and its associated LayerNorm as a single unit. The importance score for the Attention layer is:
S 1- - CosineSim Y
(6)
Layer Drop, for both MLP and Attention layers, is performed in a one-shot manner, calculating importance scores once and removing redundant layers in a single step. This approach avoids the sssoureeinntesivee and time-consuming iterative pruning process. The effectiveness this simple one-shot technique is evaluated in Appendix
4 Implementation Details
Models We utilize Llama-3 (Grattafiori et al.) 2024 and Mistral (Jiang et al., 2023) the default models, given their competitive performance and wide usage. We also evaluated the completely open-source model
4

---

Published in Transactions on Machine Learning Research (02/2026)
MLP Layer
MLP Layer
MLP Layer
Layer Norm
Layer Norm
Layer Norm
Attention Drop
MLP Drop
Attention Layer
Attention Layer
Attention Layer
Layer Norm
Layer Norm
Layer Norm
Figure Visualization of Layer Drop, where we visualize dropping either MLP or Attention Layers, Given the residual connection, take LayerNorm together with the corresponding layers. The dropped layers with high similarity scores are blurred.
MAP-Nco (Zhang et al. 2024 to explore the redundancy variations in the modules of the entire pre-training phase. Additionally, we experimented with the newly released Llama-3 to verify the effectiveness of model dropping on the latest models.
Datasets For the calibration dataset, we used the validation set of C4 dataset (Raffel al. 2019), with 256 samples and an input sequence length of 2,048, following the setup in (Sun et al. 2023). The setting well-supported by Appendix report normalized zero-shot or few-shot accuracy on the LM-harness benchmark, which includes multiple tasks: ARC-C (Clark et al. 2018), 2019).
Table Experimental scttings for evalu- ation tasks. "Norm" refers to the normal- ization performed with respect to the length of the input.
To evaluate model performance,
Task BoolQ RTE OBQA
Number few-shot
Metric
Accuracy
Accuracy
BoolQ CClark et al.) 2019). HellaSwag (Zellers et al.
Accuracy (Norm)
MMLU (Hendrycks et al. 2021), OBQA (Mihaylov et al.
PIQA
Accuracy (Norm)
2018), PIQA (Bisk al. 2019). RTE (Wang al.
2019). and WinoGrande (ai2] 2019). Please refer Table for detailed information. The evaluation code is based EleutherAl LM
MMLU
Accuracy
WinoGrande
Accuracy
GSM8K
Exact Match
HellaSwag
Accuracy (Norm)
Harness (Gao et al. 2023).
ARC-C
Accuracy (Norm)
5 Investigation of Dropping Different Target Modules
In this section, we conduct. comprehensive investigation into the effects of dropping different larget modules. To quantify the trade-off between performance degradation and speedup, we introduce new metric, iee. Specdup Degradation Ratio (SDR), defined as:
Avvg.
(7)
ASpeedup
where AAvg. represents the percentage change in average performance across the evaluated tasks, and ASpeedup denotes the corresponding percentage of speedup achieved by cach method. Therefore, y mcasures the amount of performance degradation incurred for each 1% increase in speedup. A lower y value indicates that the model achieves speedup with minimal performance loss, making it more efficient. In contrast,
5

---

Published in Transactions on Machine Learning Research (02/2026)
Masru-F8
Limma-2-188
Random Guasa
Hullehway
Dronddoos
Dhoppic RCP
(a) Block Drop
(b) MLP Drop
(c) Attention Drop
Figure 5: Performance with respect to Dropping Ratios. The solid lines represent. the impact of dropping the n modules with the lowest importance scores in Mistral-7B and Llama-2-13B, and the dotted lines represent the performances of the baseline and random guessing.
higher value suggests that the performance loss is substantial relative to the speedup gained, implying less favorable trade-off.
Table 2: Experimental results of dropping different modules. We drop fixed number of modules (e.g., 4 8) in the Mistral-7B model. The Block Drop baseline is adapted from ShortGPT (Men al. 2024
Rows with averaged performance lower than 95% of the original performance are grayed.
Method
ARC-C
BookQ
HellaSwag
MMLU
OBQA
PIQA
RTE
WinoGrande
Avg. (t)
SpeedUp (t) (4)
Baseline
61.5
83.7
83.2
62.5
43.8
82.0
66.8
78.5
70.3
1.00x
Block-4
53.1
80.4
77.5
61.6
40.0
77.6
70.0
76.6
67.1
1.14x
0.23
Block-8
40.0
71.6
63.9
60.0
30.6
69.3
63.9
69.7
58.6
1.32x
0.37
MLP-4
53.2
80.3
77.7
61.7
40.0
77.6
67.5
77.3
66.9
1.03x
1.13
MLP-S
36.7
71.8
33.6
53.3
30.6
68.0
66.8
66.6
53.4
1.06x
2.82
Attn-4
61.0
83.5
82.9
62.5
44.6
82.0
64.6
78.0
69.9
1.10x
0.04
Attn-8
60.2
82.7
82.3
62.2
44.2
81.3
66.8
78.8
69.8
1.23x
0.02
Attn-12
57.2
76.8
80.2
59.4
41.8
79.1
66.1
77.7
67.3
1.40x
0.08
Table and Figure summarize the results of dropping different target. modules, such Block, MLP, and Attention layers. Specifically, Table 2 shows the performance impact of dropping fixed number of modules (c.g., 4 and 8 layers), while Figure extends this analysis by evaluating a broader range of dropping ratios (0% to 100%).
Cosine Similarity as an Effective Metric for Layer Dropping Building on the implementation details provided in Appendix ! we first compare the effectiveness of various layer dropping metrics against cosine similarity. Specifically, Reverse Order (dropping from the deepest layers upward) and Relative Magnitude (the magnitude of the layer output relative to the residual branch) metrics proposed by ShortGPT (Men al. 2024) and apply them the Attention Drop setting. In addition, take the random dropping of attention layers as baseline. All experiments were conducted using the Mistral-7B model, with performance averaged over five different random seeds. As shown in Table our proposed metric, Cosine Similarity, consistently outperformed the others and is therefore selected as the default metric for dropping decisions.
Table 3: Comparison of Dropping Metrics. The re- ported results reflect the average performance across HellaSwag, MMLU, OBQA, and WinoGrande.
evaluate
Metric
Attn-4
Attn-8
Attn-12
Random
61.5
49.6
39.4
Reverse Order
66.9
66.9
61.5
Relative Magnitude
67.0
66.8
62.3
Cosine Similarity
67.0
66.9
64.8
Block and MLP Drop: Significant Performance Degradation with Moderate Speedup Block Drop and MLP Drop both lead notable performance declines across both models, despite achicving moderate speedups. For instance, dropping blocks causes 11.7% performance drop, which is unacceptable in practice. Similarly, MLP Drop exhibits comparable trend, with small decline at layers (3.4%,

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: Comparison with Wanda under the same parameter budget. Even with performance- favorable unstructured sparsity, Wanda is consis- tently outperforeed by Attention Drop.
Table 5: Experimental results on Llama-3-8h and 1ama---70B.
Method
HellaSwag
ODQA WmoGrande
Ang. (1)
SpeedUp (1)
LAum-3-888
Dasclinc
82.2
03.5
45.0
11.7
GLB
1.00x
Method
HellaSwag
MMLU
OBQA
Winogronde
Aug.
SpoddUp
Artn
SL.
08.1
45.0
78.4
67.
1.16x
0.31
Wands
82.3
51.
45.2
77.6
65.0
1.00x
Actn-12
79.4
63.9
42.2
77.8
65.8
1.25%
0.00
Adin-
82.0
54.7
46.2
77.2
65.0
1.06x
Altn-16
71.2
38.2
20.4
72.8
55.4
1.88%
0.32
82.4
Llama
Arto-8
82.2
64.5
47.0
77.4
65.3
1.13x
Doccline
48.4
85.4
26.1
1,00x
Wanda
82.4
51.8
46.2
77.4
65.2
LO0K
Acto 32
48.8
85.3
25.2
1.5xx
0.00
Asta-12
82.7
54.4
48.0
76.6
65.
1,20x
Artn-18
73.9
07.4
813
21.
0.07
1.13), but much larger drop at layers (13.5%, 7= 7 These results suggest that while Block and MLP Drop provide moderate speedup, they do so at the cost of significant performance degradation, especially at higher drop ratios.
y= 1.13),
Attention Drop: Minimal Performance Impact with High Efficiency Surprisingly, despite the critical role of attention layers in Transformer architectures, dropping attention layers is highly effective. Mistral-7B maintains over 95% of the original performance even after dropping 12 attention layers with a speedup of 1.40x and low 7 of 0.08, as shown in Table 2 These results demonstrate that attention layers highly redundant, and their removal has minimal impact on model accuracy, making Attention Drop a highly efficient pruning strategy.
To verify the consistency of our Attention Drop findings on larger models, we evaluate LLaMA-3-70B as shown in Table 5 which demonstrates a speedup of 1.55 and a of 0.07 when dropping 40 out of 80 attention layers. This robustness indicates that larger models can also tolerate the removal of a significant proportion of Attention layers without degrading performance.
Attention Drop achieves substantial speedup with minimal performance degradation and outperforms other mainstream compression methods. For instance, in Table | we compare Attention Drop with the mainstream pruning method Wanda (Sun et al. 2023 which removes less important paramcters based on activation-awaro criteria. Under the same parameter budget, our methods outperform Wanda with unstructured sparsity. Additionally, Wanda contributes to fine-grained sparsity, which is not hardware-friendly and has limited practical usage.
6 Joint Layer Drop Further Enhances the Performance
While a significant proportion of attention layers exhibit high redundancy, our findings also show that some MLP layers have low importance. To further optimize model efficiency, we introduce Joint Layer Drop, which combines both Attention Drop and MLP Drop strategies. This approach leverages the redundancy in both attention and MLP layers to enhance the overall performance of the model.
Combining Attention and MLP Drop The Joint Layer Drop method is implemented by first calculating the importance scores for both attention layers (S)) and MLP layers (S), individually. By dropping both MLP and allention layers based on the combined set of scores, = [S] {. this joint approach simultancously removes the most redundant components from both layer types, thereby enhancing model efficiency while preserving performance.
Methodology:
Superior Performance with Joint Layer Drop As demonstrated in Figure 6 Joint Layer Drop consistently achieves better performance than cither Attention Drop or MLP Drop alone. The process begins by exclusively dropping al allention layers. However, as the dropping ratio increases and the more redundant. attention layers are pruned, MLP layers start to become the next most redundant components. At this point, Joint Layer Drop begins to remove MLP layers, leading to further reductions redundant layers without. significant performance loss, e.g., after dropping 31 layers (Attention MLP), of the performance on the MMLU task.
still retains 90%
7

---

Published in Transactions on Machine Learning Research (02/2026)
HellaSwag
MMLU
OBGA
WinGGrando
60-
30"
12
The Dropped
Number Dropped Moriuies
The Numbbrr of Dropoed Morillos
Dropond Moduless
Figure 6: Accuracy Curves of Dropping Different Target Modules, where we consider dropping single types of modules and Joint dropping (Attn MLP). In the line of Joint Drop, * represents the step where the MLP is dropped, while the represents the step where the Attention is dropped.
Comparison with other Compression Techniques Table 6 first compares our Joint Layer Drop with published sparse models pruned by Shortened LLaMA (Kim et al. 2024a), which estimates the importance of transformer blocks using a perplexity-based metric and performs pruning followed by retraining. Specifically, we prune the original Vicuna-13B-vL.. model (Chiang et al. 2023 to maintain the same parameter budget. While Shortened LLaMA involves post-compression retraining, training-free Joint Drop performs better on average performance.
Table 6: Comparison with Shortened LLaMA Kim et al. 2024a), where Joint Layer Drop achieves significantly higher speedup.
Method
HellaSwag
MMLU
OBQA
Winogrande
Avg. (f)
SpeedUp (t)
Joint Layer Drop
76.0
49.6
42.4
74.9
60.7
1.45
Shortened LLaMA-PPL
75.3
47.7
42.2
74.0
59.8
1.23
Shortened LLaMA-Taylor
75.8
47.0
42.4
74.3
59.9
1.22
7 Efficiency of Attention Drop
In this section, we evaluate the efficiency of Attention Drop in terms of both memory usage and inference speed. Specifically, we examine the reduction in memory overhead due to the key-value (KV) cache and measure the speed-up during the entire generation phase.
KV-cache Memory Reduction Given the auto- regressive nature of attention, where outputs are generated token by token, the KV-cache is used to store intermediate representations of input sequences. This cache helps ac- celerate inference by preventing redundant computations but comes with a significant memory cost, especially with longer sequence lengths or larger batch sizes. Our proposed Attention Drop method efficiently removes unimportant attention layers, reducing the corresponding KCccche Table 7 provides comparison of 16-bit precision KV- cache memory usage before and after Attention Drop for various models, where we use 8 Nvidia RTX A6000 Ada GPUs for the 70B models and Nvidia RTX A6000
Table 7: Comparison of KV-ccche sizes be- fore and after Attention Drop across different models, with sequence length of 2048. Since Llama-2- 2-13B does not use grouped-query atten- tion, its KV-cache costs are higher.
Model
Datch Size
Wo/Alta Drop
*/Atta: Deap
Layers
RV cuche
Layers
RV rathe
Mistral
64
32
18GB
20
10GB
201
Liuma 2-71B
32
801
21GB
40
10GB
LJamm----83
64
32
160B
20
10GB
80
2009
40
10GB
Ada GPUs for other smaller models. As shown, Attention Drop results in substantial memory savings across all tested models. For instance, the KV-cache decreases from 20GB to 10GB in Llama-3-70B, Note the reported results are based on resource-constrained scenarios. In resourco-sufficiet cases, where larger batch sizes and longer sequence lengths can be applied, the memory usage savings from Attention Drop become even more significant..
8

---

Published in Transactions on Machine Learning Research (02/2026)
Speed Measurement We also evaluate the run-time speed improvements achieved through Attention Drop. The inference speed is measured throughout the entire generation process, starting from the input prompt. to the generation of the final token. To ensure that the results accurately reflect the speed improvements, we follow two key principles in our sctup: (1) all operations are performed on a single Nvidia RTX A6000 Ada GPU, avoiding any communication overhead caused by multi-GPU selups; and (2) we increase the batch sizes to maximize GPU utilization for each model. Specifically, for Llama-3-70B, we employ 4-bit quantization due to its large model size, while noting that Attention Drop orthogonal quantization shown in Appendix use 16-bit precision. We use an input sequence of 2048 tokens and autoregressively generate an additional 2048 tokens. This setup allows us to capture the full inference process, ensuring that both the prefill and the decoding stages are included in the speed measurements.
B For Llama-3-8B and Mistral-7B,
The speed-up ratios achieved through Attention Drop are presented in Tables 2 and 13 Our results show that Attention Drop provides up 40% speed-up while retaining more than 95% of the original model's performance. Additionally, as demonstrated in Table the y values for Attention Drop are significantly lower than those for MLP Drop and Block Drop, especially higher speed-up ratios. This indicates that Attention Drop achieves more efficient trade-off between speed and performance, making it superior method for model accelcration.
8 Visualization and Analysis of Layer Importance
In this section, we visualize the importance scores and the corresponding dropping order of pretrained models. We then trace back through historical checkpoints to explore the dynamics of importance scores throughout the training process.
Drop4 4
Drop
Drup-8
Dup-8
Drap-12
(a) MLP Drop
(b) Attention Drop
Figure 7: Laycr-wise cosinc similarity between the output embeddings of the original model and models with varying numbers of dropped (a) MLP or (b) Attention layers. Since all dropped layers are located in the second half of the network, only the corresponding layers are shown for clarity.
8.1 Impact of Layer Drop on Embedding Representations
We have demonsttrated the effectiveness of using layer-wise cosine similarity in the middle layers to guide layer dropping. We now examine how layer dropping alters the embeddings of the original model. Specifically, we extract the output embeddings of each layer from both the original and compressed models and compute the cosine similarity between them. As shown in Figure 7 dropping MLP layers leads to substantial fluctuations in the embeddings, whereas dropping attention layers maintains high degree of similarity with the original model. The low similarity observed in the model with MLP dropping results in significant divergence in the predicted tokens and failure to produce the correct answer, as illustrated in Figure Tn contrast, both the original model and the model with 12 dropped attention layers are able to generate the correct answer. These findings align with the observation that MLP layers are primarily responsible for storing knowledge
Beseline
MuP-Drap
A25
Aln-Drop
A/6
A0C
Figure 8: Output token probability dis- tributions of models before and after drop- ping 12 layers. The input query is "The capital of California is".
Geva
ct al. 2021 Gromov et al. 2024).

---

Published in Transactions on Machine Learning Research (02/2026)
On the other hand, since MLP layers perform token trans- formations in high-dimensional space, MLP dropping di- rectly impacts the model's sensitivity to different tokens. Since the embedding of the last token depends on both it- self and its preceding context, we switch the last token and monitor the resulting embodding from different forwarding passes. As shown in Figure g the original model exhibits low similarity across different imputs, which is consistent with the generative diversity of language models, where varying the current token leads to different subsequent tokens. In contrast, models with MLP dropping lack this capability and produce highly similar embeddings for dif- ferent input tokens. This suggests that such models losc the ability to distinguish between different tokens, result- ing in degraded performance. Meanwhile, models with attention dropping exhibit trends comparable to those of the original model, indicating the presence of less impactful attention layers, which we further investigate in the following sections.
08- 08- 02-
Codtre Smtaty
Base Ine
ZLA MLP-Drop
1523
Attn-Drop
Figure 9: Cosine similarity of the last-token embedding before and after replacing the last token with different candidates, given the input query: "In mammals, the diaphragm is the pri- mary muscle responsible for the act of
8.2 Deeper Modules with Higher Redundancy
Based on Figure 2 and 17 we observe that the deeper layers (excluding the last ones) often exhibit
3
excessively low importance across Block, MLP, and Attention modules.
To further analyze the dropped modules, we visualize the dropped layers or blocks with different dropping ratios. Figure 10 visualizes the remaining and dropped layers/blode /locks as the number of dropped modules increases. Llama-2- 2-1B and Mistral-7B exhibit similar patterns in Layer Drop and Block Drop: initially, both models tend to drop the deeper layers, followed by the shallower ones. These findings are consistent. with Men et al. (2024), which suggests that deeper layers tend to be more redundant. Larger models (e.g., Llama-2-70B) also showcase a similar trend in Appendix
LoVer Frdion
Lavar Fndex
Lase Indo
Lave hecc
Layer Ree
(a) Block Drop
(b) MLP Drop
(c) Attention Drop
Figure 10: Visualization of Dropping Order, where each row represents the corresponding retained and dropped layers/ soyers/blocks blocks for specific number of dropped layers/blocks.
Previous studies on layer analysis in neural net- works DDalvi et al. 2022 Sajjad et al. 2022 have demonstrated that shallow layers are more im- portant than deeper layers, since they primarily model local word interactions, which for capturing morphology and lexical semantics. These layers provide essential input to the higher layers, and their removal can therefore be catas- trophic. Building on this, we further investigate the differing behaviors of shallow and deep at- tention layers. Figure compares the attention matrices of shallow layer and deep layer.
400
A06
A06
4.20
critical
405
A02
100
(a) 4th Layer
(b) 25th Layer
Figure 11: Attention scores in the shallow and deep layer.
The shallow layer effectively captures relationships between
10

---

Published in Transactions on Machine Learning Research (02/2026)
tokens, whereas the deep layer exhibits sparse pattern, with attention concentrated along the diagonal and in specific columns.
8.3 Consistent Redundancy of Attention Layers Throughout Training
Now that the deep layers exhibit high redundancy, to investigate how such pattern is achieved, revisit the historical checkpoints to track the dynamic changing of layer-wise importance scores.
Specifically, we use checkpoints released by MAP-Neo-7B (Zhang et al. 2024), since it released continuous chockpoints during training stages. Figure 12 presents the importance scores of Blocks and Layers at different training stages, where Attention layers demonstrate consistently lower importance scores than MLP and Block at all training stages. While the importance scores for ML. layers and Blocks gradually increase as training progresses, the importance scores of Attention layers change much more slowly, consistently exhibiting low importance throughout the training process.
0.08
01E
8.5
R15
024
635
631
TainndTooorrs
Procpss
16
24
8
26
28
Modile Iniex
(a) Attention
(b) MLP
(c) Block
Figure 12: Visualization of Importance Scores in Checkpoints during the Pre-training Process of MAP-Neo-7B where lighter areas represent low importance scores (i.e., high similarity scores). We present the entire Training Process (checkpoints for every 500B trained tokens). We independently visualize the importance scores at the module index 0, since they are significantly higher.
addition, since randomly initialized models prior to training also exhibit low importance in certain attention layers (as shown in Figure 21). we attribute this phenomenon intrinsic propertics of the attention mechanism, which can be formulated weighted linear combination:
= Σ An
(XWa)(XWx)) ddz
Wo Σ As (X,WwWo), A softmax fm1
(8)
where Y, denotes the output of the i-th token, which depends the first tokens. However, shown by the attention sinks in Figure II attention dispro- portionately focuses on carly tokens (c.g., the t-th token), causing the term Aq to dominate the sum Σj X, This sparse localized focus effectively limits the contribution of the attention branch to the residual connections.
10
32
13
30
62
On the other hand, Attention also has the potential increase similarity between neighboring tokens, e.g., both the i-th token and i-th aggregate the first with the complete visualization in Appendix C
(a) Randomly Init.
(b) Pretrained
Figure 13: Token-to-token similarity deep layers,
tokens. Through progressively aggregation in mul- tiple layers, even distant tokens can maintain a high degree of similarity, as shown in Figure 13 Notably, the high token-to-tokee similarity observed in a randomly initialized model further highlights the inherent aggregation behavior the attention mechanism. More detailed visualizations and analyses are provided in Appendix
11

---

Published in Transactions on Machine Learning Research (02/2026)
Consequently, such high similarity between tokens (i.e.,
oversmoothed tokens (Wu et al. (023)) limits the de- gree of representational updates in attention layers, since aggregating nearly identical tokens has a dimin- ished effect on the hidden states. Moreover, due to the sparse patterns of attention weights and the low mag- nitude of the attention branch, shown in Figure [4] the updates are further constrained. This limitation diminishes the functional contribution of certain atten- tion layers, thereby reducing their relative importance within the model.
Beathe Moontuds
0.6
Attention
0.3
0.4
0.2
6
10
14
18
26
30
Layer
Figure 14: Relative magnitude of layer outputs com- pared to the residual branch.
9 Discussions and Conclusion
Insights for Future Network Architecture Design Despite the success of scaling up LLMs, our work provides valuable insights for scaling down models to achieve more efficient architectures. One key insight is the high redundancy in attention layers, particularly in deeper layers, which suggests that future works could reduce the number of attention layers without sacrificing performance, rather than maintaining parity with MLP layers. Morcover, unlike MLP layers, attention layers exhibit consistent redundancy as training progresses. This consistency may pose bottleneck in training large models. To address this, future research could explore replacing attention layers with alternative mechanisms develop new training techniques that capitalize on this redundancy to further enhance language model capacity.
Limitations While our proposed dropping tech- niques improve efficiency with minimal performance degradation, several limitations remain. An important direction for future work to evaluate the applica- bility of these techniques across broader range of architectures, such as vision transformers and vision
Zi
Baseline
APA Athn Drop
MLP Drop
Average Performace (%)
80
language models. Although our method is designed for free dropping without retraining, additional fine-tuning may further restore, or even improve, per- formance following pruning. As a preliminary study, we conduct both LoRA and full-model fine-tuning on Llama-3.2-3B using Tulu (Lambert et al.] 2024) as the training dataset. The results, shown in Figure 15 and detailed in Appendix B highlight the strong po- tential of fine-tuning for restoring the performance of compressed models. Finally, integrating layer dropping with large-scale pretraining represents a promising direction for future research. Identifying and removing redundancy at early training stages may enable models to be trained with inherently compact architectures, improving training efficiency while supporting more fundamental optimization of model capabilitics.
Bassir
LoRA Firelure
Full Finetune
Figure 15: Average performance of Llama-3.2-
across post-training strategies. Fine-tuning effectively mitigates the degradation introduced by layer dropping, with full-model fine-tuning achieving the most substantial recovery.
Conclusion In this work, we systematically revisited transformer architectures by investigating the effects of dropping three types of structures: Blocks, MLP layers, and Attention layers. Our findings reveal that attention layers display significant redundancy and can be removed in large proportions without compromising performance. To build on this, we introduced Joint Layer Drop, method that further increases both dropping ratios and performance by targeting redundant layers across both MLP and Attention layers. This si study empirically demonstrates the potential for creating more compact and efficient transformer models, providing valuable insights for future network design within the NLP community. By exploring structured redundancy, we open up new avenues for designing more efficient, scalable models that maintain high performance even under resource constraints.
12

---

Published in Transactions on Machine Learning Research (02/2026)
References
Winogrande: An adversarial winograd schema challenge at scale. 2019.
01. AT, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangeheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yanpeng Tı, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai.
Yi: Open foundation models by 01.ai, 2025. URL httpp///aaaxiy.gg///////////
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfong Gao, and Yejin Choi. Piga: Reasoning about physical commonsense natural language, 2019.
Iñigo Casanueva, Tadas Temčinas, Danicla Gerz, Matthew Henderson, and Ivan Vulié. Efficient intent detection with dual sentence encoders. arXie preprint arXiv:2033.040 2020.
Xiaodong Chen, Yuxuan Hu, and Jing Zhang. Compressing large language models by streamlining the unimportant layer, Xiv preprint (1v:2403.11115, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source org/
chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL 0023-03-30-vicuna//
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019.
Peter Clark, Isaac Cowhey Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018.
Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. Discovering latent concepts learned in BERT. In International Conference on Learning Representations, 2022. URL https: http://ppwueereee..eee//////aaa/ net//forum?tt==<PMthyYYxxxx
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thile, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dalaset of diverse text for language modeling, 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https:://enodo. htapss///aamdd..or//reoood..... wtt.///eorr////1266633999
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer foed-forward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Melhods in Natural Language Processing, pp. 5484 5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1
446. URL http::://aclatttolooot org/2021 mainn.pmmai...mm
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark,
13

---

Published in Transactions on Machine Learning Research (02/2026)
Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Damiel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakonkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovie, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Tvan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jemnifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajiwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosalc, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitmann, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgion, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li, Yuning Mao, Zacharic Delpicrre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadch Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Bealy, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizow, Guangyi, Zhang, Guna Lakshminarayanauı, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Tgor Molybog, Igor Tufanov, Ilias Leontiadis, Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean- -aptistt Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan
14

---

Published in Transactions on Machine Learning Research (02/2026)
McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Ti, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhalt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L.. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Batley, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shugiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kunar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The llama herd of models, 2024. URL atttp:://arxi> http//aaxvv..oaaaaaee www./bo////////// 21783
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. The unreasonable ineffectiveness of the deeper layers, 2024.
Albert Gu and Tri Dao. Mamba: Lincar-time sequence modeling with selective state spaces, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021.
Albert Q, Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaune Lample, Lucile Saulnier, Lélio Renard Lavaud, Maric-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Bo- Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung- Kyu Song. Shortened Ilaaa: A simple depth pruning for large language models. ICLR Workshop on Mathematicul and Empirical Understanding of Foundation Models (ME-FoMo), 2024a. URL https //openreview.co //opeareriee.nne//Eorrurrta wett:/orrumiidd1Braue
Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seoughoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, and Sunghun Kim. Solar 10.7b: Scaling large language models with simple yet effective depth up-scaling, 2024b. URL https://arxiv: org/ at///////2//3 15166
Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontaek Oh, Beomjun Kim, Taehwan Yoo, Seongjin Shin, Dongyoon Han, Jinwoo Shin, and Kang Min Yoo. Peri-LN: Revisiting normalization layer in the transformer architecture. In Forty-second International Conference on Machine Learning, 2025. URL http://pweeeeeee...ee.//rr/rr//r
15

---

Published in Transactions on Machine Learning Research (02/2026)
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparamederivation for learnable sparsity. In Proceedings of the International Conference on Machine Learning, July 2020.
Se Jung Kwon, Dongsoo Lee, Byeongwook Kim, Parichay Kapoor, Baeseong Park, and Yeon Wei. Structured compression by weight encryption for unstructured pruming and quantization. In 2020 IEEE/CVE Conference on Computer Vision and Pallern Recognition (CVPR), pp. 1906 1915, 2020. doi: 10.1109/ 0PPR42600.2020.0000
Nathan Lambert, Jacob Morrison, Valentina Pyalkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. 2024.
Rui Li, Jianlin Su, Chenxi Duan, and Shunyi Zheng. Linear attention mechanism: An efficient attention for semantic segmentation, 2020.
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context 1ms struggle with long in-context learning. arXiv preprint arXiv:2404 002220044000 2024.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and accelcration. In MLSys, 2024.
Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. Pay attention to mlps 2021.
Florian Mai, Arnaud Pannatier, Fabio Fehr, Haolin Chen, Francois Marelli, Francois Fleuret, and James Henderson. HyperMixer: An MLP-based Green Al Alternative to Transformers. arXiv preprint arXiv:2203.00620 033691 2022.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect, 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering, 2018.
OpenAL. Gpt-4 technical report, 2024.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. Xrviv preprint arXiv:2304.022 2023a.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. Rwkv: Reinventing rens for the transformer era, 2023b.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Auselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
Colin Raffel, Noam Shazoer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(14)):1-61 2020.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaove. Code llama: Open foundation models for code, 2024. URL https///aaxxivorg/abbbbeoe
12950
Hassan Sajjad, Nadir Durrani, Fahim Dalvi, Firoj Alam, Abdul Khan, and Jia. Xu. Analyzing encoded concepts in transformer language models. In Marine Carpuat, Marie-Catherr de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Compulational Linguislics: Human Language Technologies, pp- 3082 3101, Seattle, United States, July 2022. Association for Computational Linguisties. doi: 10.18653 v1 /202222222222 www.:aaaclllmm2200 225. URL naacl-main 225/
Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fino-tuning, 2020. URL https://arxxv.orrg//aaae// http///ab2200..000//
Dachuan Shi, Chaofan Tao, Ying Jin, Zhendong Yang, Chun Yuan, and Jiaqi Wang. UPop: Unified and
progressive pruming for compressing vision- lasioanlanguage transformers. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 31292-31311. PMLR, 2023.
Mingjic Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. orXiv preprint arXiv:2306. 1 11695, 2023.
Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rívière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kencaly, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Buse-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Tain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan- Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Char Danielle Eisenbud, Dee Cattle, Derck Cheng, Dimitris Paparas, Divyashree Shivakumar Srecpathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimizak--Pluciiska,, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, lan Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wicting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanezyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq lqbal, Shashir Reddy, Shruti Sheth, Siim Pöder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evei, Vedant Misra, Vincent Roscberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanscvicro, Lucas Gonzalez, Zach Gloicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet,
CJ Carey, Cormac Brick, Daniel Deutsch,
17

---

Published in Transactions on Machine Learning Research (02/2026)
Elena Buchatskaya, -ean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025. URL Itttp:///arxiv.rrt org. wttg/a//////...o..////// 19786
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Mario-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Ashish Vaswani. Attention is all you need. aixiv preprint (XX:::70660033 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. 2019. In the Proceedings of ICLR.
Tiannan Wang, Wangchunshu Zhou, Yan Zeng, and Xinsong Zhang. AffiiietVLLM Fast and accurate vision-language models via knowledge distillation and modal-adaptive pruming. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 13899-13913, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: (0.18653/v1/2022 hind//iin////..iiinnnsnisaa ac1.873. URL
www.:ong...iddiin-neeeeee ac1.873
Xinyi Wu, Amir Ajorlou, Zihui Wu, and Ali Jadbabaie. Demystifying oversmoothing attention-baser graph neural networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
URL http::///pppereeeee...eeee///ees mett/forummidakkosiieee
https://openreviev:
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference Learning Representations, 2024.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, Jun Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin, Nuolan Nie, Peidong Guo, Ruiyang Sun, Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuehai Pan, Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, and Zhiying Wu. Baichuan 2: Open large-scale language models, 2025a. URL lhttts:///arttt org/ abs/ 2309. 10305
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Peng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025b. URL httthtttt///rrkt org/ abs abss/2055.00.. 09388
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 0a..809..22.... 12284, 2023.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MAmmo Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference Learning Representations, 2024. URL https:://openrevvev. wttp://wwwwwwweeerrreeeeeeeeeeee htt.://ww...eetreooooe//ooooo//o
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019.
18

---

Published in Transactions on Machine Learning Research (02/2026)
Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming Liang, Yinghao Ma, Yizhi Li, Ziyang Ma, Bill Lin, Emmanouil Benetos, Huan Yang, Junting Zhou, Kaijing Ma, Minghao Liu, Morry Niu, Noah Wang, Quchry Que, Ruibo Liu, Sine Liu, Shawn Guo, Soren Gao, Wangchunshu Zhou, Xinyue Zhang Yizhi Zhou, Yubo Wang, Yuelin Bai, Yuhan Zhang, Yuxiang Zhang, Zenith Wang, Zhenzhu Yang, Zijian Zhao, Jiajun Zhang, Wanli Ouyang, Wenhao Huang, and Wenhu Chen. Map-neo: Highly capable and transparent bilingual large language model series, 2024.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy-hitter oracle for efficient generative inference of large language models, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuron-level struc- tured pruning using polarization regularizer. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 9865 9877. Cur- ran Associates, Inc., 2020. URL https://proceeddngg www.ine..ruiooeoooo http:///ecccpppp...ea//////////p /illes/paper//200/fiee 7099777666eea/7/0000000000066100 pdf
19

---

Published in Transactions on Machine Learning Research (02/2026)
A
Ablation Studies
One-Shot v.8. Iterative One-shot and iterative approaches are the two most common methods for model compression. In the onc-shot approach, importance scores are computed once, and the model is pruned in a single step. In contrast, the iteralive method compules importance scores and prunes the model incrementally over multiple iterations. In Figure 16 we empirically compare Iterative Dropping and One-Shot Dropping, where in Iterative Dropping, layers are removed one by one in cach itcration.
Drop Attention Layers
Drop Attention Layers
Iterative
82.3
Iterative
79.1
70.0
70
67.0
66.9
62.2
62.5
R2.1
60
$2.4
30
44.6
40.0
40
HellSwag
MMLU
0B0A
Winograrde
AVG.
HellaS mag
MKLU
CBQA
Winegrande
AVG.
Figure 16: Ablation Study on Dropping Strategies, i.e., Iterative and One-Shot, where One-Shot Dropping achieves comparable performance with Iterative Dropping.
As shown in Figure 16 Iterative Dropping achieves performance that is merely comparable to One-Shot Dropping, without offering any significant enhancement. This occurs because removing a layer lowers the similarity scores of deeper layers, making them more critical. As a result, subsequent dropping steps increasingly target shallower layers, accelerating the model's collapse. Given the simplicity and efficiency, Onc-Shot Dropping emerges as the superior choice.
Table 8: Ablation Study on the Residual Connection, where we report the average performance MMLU, WinoGrande, HellaSwag, and OpenbookQA. denoles the number of dropped modules. The notation "w/ res" indicates the involvement of the residual connection, while "w/o" indicates dropping without considering it.
Attn Drop
MLP Drop
n
w/o res
w/ res
w/o res
w/ res
4
39.4
65.0
31.2
64.5
8
37.7
65.3
31.1
61.9
12
36.8
65.4
31.1
55.6
16
32.2
63.4
30.8
49.9
20
32.0
62.9
30.9
42.1
Residual Connection The involvement of the residual connection ensures accurate estimation by accounting for the overall inputs and outpuls. To explore its impact performance, we also consider dropping modules without involving the residual connection. In this case, the importance scores are measured solely by the inputs and outputs of the Attention or MLP layers. As shown in Table g the involvement of the residual connection essential for Layer Drop.
Calibration Datasets Figure demonstrates the robustness of the importance scores across different datascts. In Figure 17 we further verify that the importance scores remain relatively stable across various modules of Mistral-7B the sample size increases. This stability indicates that both Block Drop and Layer Drop maintain consistency regardless of the number of samples. Consequently, we confirm that using 256 samples is sufficient for computing similarity, which serves as the standard adopted for all our experiments.
20

---

Published in Transactions on Machine Learning Research (02/2020)
carooie
13 Layer Rdte
-ayer Irdee
Loger I1d0
0.0
(a) Block
(b) MLP
(c)
Attention
Figure 17: The impact of sample quantity on the importance scores of Block, MLP and Attention.
Table 10: Performance of various models under layer dropping.
Yi-BB
Solar-10.7B
Method
HellaSwag
MMLU
OBQA
WinoGrande
Avg.
HellaSwag
MMLU
OBQA
WinoGrande
Avg.
Baseline
69.6
79.0
84.6
64.0
82.6
68.7
MLP-4
75.1
67.2
38.6
74.0
63.7
84.0
63.5
43.6
81.9
68.3
MLP-8
63.7
71.7
58.0
79.0
63.1
40.2
80.3
65.6
MLP-12
63.4
60.2
69.9
55.6
73.3
61.5
36.2
76.6
61.9
Attn-4
79.9
69.6
44.0
79.2
68.2
84.4
64.1
43.8
82.2
68.6
Atta-8
79.1
68.2
78.2
67.0
84.4
64.1
44.0
83.0
68.9
Attn 12
78.0
66.3
42.0
77.0
65.8
83.8
63.6
44.0
82.0
68.3
Baichuan-22 2-BB
Qwen3-8B
Method
HellaSwag
MMLU
OBQA
WinoGrande
Avg.
HellaSwag
MMLU
OBQA
W WinoGrande
Avg.
Baseline
73.8
54.0
39.6
70.7
59.5
76.3
75.0
38.8
75.1
65.4
MLP-4
67.7
44.7
36.4
69.1
54.5
75.6
75.7
39.0
74.5
65.6
MLP-8
54.2
36.8
66.6
66.5
36.8
59.2
MLP-12
44.2
47.9
28.2
62.0
45.6
50.4
49.4
30.4
60.1
45.5
Actu-1
54.1
39.8
71.3
79.7
76.5
41.4
77.0
68.5
Attu-8
73.4
53.8
71.8
60.0
79.1
76.1
39.8
76.0
67.5
Astn-12
69.7
57.0
37.2
55.9
Consistent Effectiveness Across Different Models Tables 12 and 13 report additional results on LLaMA-2- 13B and LJaMA-2-70B, which exhibit trends consistent with those observed on Mistral and Llama-3. Beyond base models, we further evaluate the instruction-tuned variant Llama-3-8B-Instrucc in Table g In addition, explore broader set of architectures, including Yi (Al et al. 2025
Table 9: Experimental Results Attention Drop on Llama-3-88 8--B--Instrtt
Method
HellaSnng
OBQA
WinCCrank
Arg. (?)
Spoodiipp (7)
GG
Baeelhee
78.7
05.6
47.5
75.6
$5.11
1.000
Atlu-
766
1.07x
1.16x
1.88x
Amn
68.2
1.62x
Solar (Kmm et al. 2024b), Baichuan-2 (Yang et al. 2025a),
and Qwen3 Yang et al. 2025b), as shown in Table 10
Moreover, given that Gemma-3 combines both post-norm and pre-norm designs, we extend Attention Drop to this model to examine its robustness under different normalization strategies. Overall, these results consistently validate the effectiveness of Attention Drop across diverse model families and configurations.
Table 11: Experimental results of Attention Drop on Gemma-3 models (Peri-LN),
Model
OBQA
PIQA
RTE
WinoGrande BoolQ
ARC-C
HellaSwag
MMLU
Avg.
Gemma-3-1B
Baseline
38.4
71.9
68.2
58.6
75.9
40.1
55.9
39.9
56.1
Attn-8
38.6
72.3
68.0
60.
76.7
37.4
55.5
39.7
56.0
A2t
38.0
71.8
68.6
60.0
73.0
38.7
55.0
38.9
55.5
Gemma-3-a4
Baseline
46.8
76.4
74.0
69.1
84.1
60.4
75.4
58.3
68.1
Attn-8
45.0
78.0
73.1
69.2
83.9
60.1
75.4
57.9
67.8
Atun-12
43.4
75.3
71.4
67.9
82.4
59.8
74.6
56.0
66.4
21

---

Published in Transactions on Machine Learning Research (02/2026)
Effectiveness Different Normalization Layouts Attention Drop is not restricted to Pre-LN architectures. Adapting to different normalization layouts, including Post-LN or Peri-LN (Pre + Post LayerNorm) (Kim et al. 2025), only requires shifting the measurement point of input/output features. The core logic and pruning criterion of Attention Drop remain unchanged.
While the main experiments in the paper focus on standard Pre-LN models such as LLaMA, we also evaluate Attention Drop on Gemma (Team et al. 2025), which adopts a Peri-LN structure. 'hee results in Table [] demonstrate consistent redundancy patterns and pruning tolerance, confirming the broader compatibility of approach across normalization designs.
Experimental Results of Dropping Different Modules, where we drop the fixed number (e.g., 4 and 8) of modules Llama-2-13B. Here, Block, MLP, and Attn are corresponding modules. Rows with averaged performance lower than 95% of the original performance are grayed.
Table 12:
Llama 138
Method
ARC-C
BoolQ
HellaSwag
MMLU
OBQA
PIQA
RTE
WinoGrande
Avg. (1)
SpredUp (f)
7 (L)
Hoseline
58.9
1.00x
Block-4
54.8
73.3
80.6
54.8
45.8
79.1
60.3
77.5
65.8
1.11x
0.22
Block-8
48.0
56.8
75.3
53.8
41.2
75.3
39.9
75.6
60.7
1.24x
0.31
MLP-4
54.9
76.1
80.4
54.8
45.4
79.3
66.4
77.3
1.0xx
0.32
MLP-8
49.2
63.4
75.6
54.5
42.2
76.0
59.2
75.1
61.9
1.08x
0.79
Allu-4
58.8
80.4
82.0
54.7
46.2
80.5
67.9
77.2
68.5
1.05x
-0.05
58.2
47.0
1.13x
Attn-16 16
56.4
79.2
81.9
48.2
47.4
79.5
59.9
76.2
66.1
1.29x
0.07
Attu-20
53.8
76.9
78.6
51.5
41.4
77.6
1.40x
0.08
Table 13: Block Drop and Layer Drop on Larger Models, where we drop a series of numbers (from 48) of modules on (lama-2-70B. Rows with averaged performance lower than 95% of the original performance are grayed.
Llama-2-70B
Method
ARC-C
BoolQ
HellaSwag
MMLU
OBQA
PIQA
RTE
WinoGrande
Avg. (t)
SpeedUp (f)
(4)
Baseline
67.4
83.8
87.1
68.5
48.6
82.5
69.3
83.7
73.9
1.00x
Block-4
63.8
80.4
84.6
60.2
48.0
81.6
71.1
78.0
71.0
1.07x
0.41
Block-8
59.1
77.5
81.3
55.1
46.2
81.0
68.2
73.2
67.7
1.14x
0.44
Block-16 16
44.6
64.6
69.9
29.3
40.0
75.2
51.6
59.7
54.4
1.30x
0.65
Block-32
35.1
58.8
56.7
25.7
36.8
71.7
54.5
55.3
49.3
1.67x
0.37
MLP-4
65.4
84.0
86.1
68.7
46.6
82.9
73.2
1.04x
0.18
MLP-8
64.4
83.9
84.9
68.7
47.6
81.7
82.2
72.5
1.05x
0.28
MLP-16
57.5
81.6
69.1
65. ,9
1.08x
MLP-32
40.6
64.2
29.8
52.7
72.7
Atta-4
67.2
84.0
87.0
68.6
48.8
82.5
69.3
83.3
73.8
1.06x
0.02
Atta-8
67.3
83.8
86.9
68.5
48.4
82.9
69.0
82.6
73.7
1.12x
0.02
Atta-16
67.8
83.9
87.2
68.5
49.0
$3.0
68.2
82.8
73.8
1.21x
0.00
Atta-32
67.2
84.8
87.2
68.4
49.6
81.8
67.5
83.5
73.8
1.35x
0.00
Attn-40
63.7
82.8
84.4
66.2
46.8
80.1
66.8
81.3
71.5
1.48x
0.05
Atta-da 48
58.5
73.7
80.6
56.8
45.0
79.8
59.6
81.0
66.9
1.62x
0.11
B Additional Experimental Results
Magnitude Consideration in Cosine Similarity Our method primarily relies on cosine similarity to identify redundant layers. Although magnitude information is not explicitly included in the criterion, it is implicitly captured through the residual structure. In residual connections f(x) + æ, the cosine similarity between x and depends on both the direction and the magnitude of f(x); a large f(x) naturally reduces similarity.
Empirically, as shown in Figures 3 and many attention layers exhibit cosine similarity greater than 0.95 or even 0.99. Such high similarity values can only occur when f(x) is both small magnitude and well-aligned with v, providing strong evidence that these layers are functionally redundant. Furthermore, Figure 14 shows
22

---

Published in Transactions on Machine Learning Research (02/2026)
that attention outputs consistently have lower magnitude compared to MLPs, especially in deeper layers, aligning with the trend observed in layer-wise cosine similarity.
While cosine-based ranking may not be the most sophisticated criterion, it is sufficient given the small search space (only L layers) and the competitive performance achieved by the pruned models. Unlike weight- or neuron-level compression, layer-level pruning is a relatively low-complexity problem, where even simple heuristics can be effective and efficient.
Table 14: Output Cases of Model with Dropping Some Attention Layers, where correct answers are marked in green, while incorrect answers are marked in red Note that "# # #" is used to extract the correct answer.
Question
Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
Golden Answer
Janet sells 16 $49*2=18m1 18 every day at the farmer's market. An # #### 18
x16-3-1-9*9 duck eggs day. AnSh makes * 2
Method
Output
Baseline
Janet's ducks lay 16 eggs per day. n She eats 3 eggs for breakfast and bakes 4 eggs for muffins, so she uses + 4 = <3+4=7w7 eggs per day. d Therefore, she sells 16 7 = a 116-7=9w9 eggs at the farmers' market. \nShe sells each egg for per day. \n # #### 18
$2, she makes 9 x $2 $*9*2=18
Attn-2
Janet's ducks lay 16 eggs per day. \nSh cats 3 eggs for breakfast and bakes 4 eggs for muffins, so she uses 3 + 4 = <3+4-7x7 eggs per day. |nTherefore, she sells 16 7 = a x16-7=9x9 eggs at the farmers' market. \nShe sells each egg for $2, so she makes 9 $2 $*9*2=18*15 per day. #### 18
Attn-4
Janet's s ducks lay 16 eggs per day. ,nShe eats 3 eggs for breakfast and bakes 4 eggs for muffins, so she uses a total of 3 + 4 = *3+4=7w7 eggs per day. \n Therefore, she sells 16 - = 116-7=9*9 eggs at the farmers' market. \nShe sells each egg for $2 so she makes $2 $*9*2= == 18 per day. \n #### 18
Attn-6
Janet's ducks lay 16 eggs per day: SShe cats eggs for breakfast and bakes 4 eggs for muffins, so she uses a total of 3 4 +3+4-7w7 eggs per day. \n Therefore,, she sells 16 7 11--7=9*9 eggs at the farmers' market. \nSe sells each egg for $2, so she makes x $2 = 2= 18 per day.)n #### 18
Attn-8
Janet's ducks lay 16 eggs per day: hnSh eats eggs for breakfast and bakes 4 eggs for muffins, so she uses a total of
*3+4=7w7 eggs per day. |hhhrrffre,, the farmers' market, \nShe sells each
Janet sells 16 7 x16-7=9%9 eggs
egg for $2, so Janet makes 9 $2 $ -x9*2=18>11 18 per day at the farmers' market. \n # # # 18
Attn-10
The total number of eggs the ducks lay is 16. \nJane eats 3 eggs for breakfast, so there are 16 3 = 13 eggs left. AnJaneJae bakes 4 eggs for muffins, so there are 13 4 = 9 eggs left. \ nnJane sells the remaining 9 eggs at $2 each, so she makes 9 * $2 = 2x9=2-18}
Attn-12
Dividing the total number of eggs to the number of eggs used for personal use, we get 3- = 11 eggs for selling, nIIf each egg is sold for $2, then the total amount earned from selling day at the farmers' market. \n # #### 22
* = $22. (22.).nTTheeee.. ATThrrefre,, she makes $22 every
23

---

Published in Transactions on Machine Learning Research (02/2026)
Performance Knowledge Intensive Tasks To assess the impact of Attention Drop complex technical tasks, we evaluated Llama-2-7B and Mistral-7B, and two corresponding instruction fine-tuned models, Betaaath-77B-LLO and AitaMMtth-iistra---- (Yu et al. 2023). The results in Figure 18 indicate that, except for Llama-2-BB-Mathe which MotaMath-7B-VLl all the models do not experience significant performance degradation when dropping fewer than Attention layers. We speculate that this because Llama-2-7B- Math is initialized with Llama-2-7B and undergoes instruction fine-tuning to improve mathematical ability. Llama-2-7B-Bas exhibits poor performance mathematics, and the ability obtained solely through fine-tuning appears to be superficial. Therefore, when dropping Attention layers, Llama-22 2-7B-Math's ability rapidly deteriorates.
Mistral-7B-Baa
We select the first sample from the test set of GSM8K as the imput and display the MiaaMatt-Misttrl-FB5 raw output in Table 14 Even when dropping attention layers, the model could still give the correct answer of this question, but failed to adhere the correct output format. However, when dropping 12 attention layers, the model is no longer able to produce the correct answer of this question. Despite this, post- training offers a potential recovery path for performance loss due to layer dropping. For instance, wightweight. LoRA fine- tuning on Meta.atthQA significantly boosts Mistral- 7B-Mhhsss accuracy from 11.6% 58.2%, demonstrating its effectiveness and making promising avenue for further exploration.
Llama-2-78-Base
Mistral-BB Math
Liama 2-7B 7B-Math
Accuracy
Dropped Attention Layers
Figure 18: Accuracy Curves on GSM8K.
Attention Drop is Orthogonal to Quantization Given that quantization simplifies data types and enhances efficiency in memory usage and inference speed, integrate module dropping with quantization to verify whether the quantized models can maintain the performance achieved by Attention Drop. Specifically, use the mainstrrea AWQ algorithm (Lin el al.) 2024) for 4-bit quantization, following its default settings, which involve using 128 samples from the Pile dataset (Gao et al. (2200) as the calibration dataset. In Table 15 the integration of quantization still maintains the performance of Attention Drop, i.c., only less than 1% difference average performance.
Table 16: Performance on Long In-Context Task.
Table 15: Integration of Module Dropping and Quan- tization, "w/Quant" denotes quantized models.
Sequence Length
Method
Avg.
Method
ARC-C HellaSwag
OBQA
WinoGrande
Avg.
2k
4k
7k
9k
14k
Llama 2 13B
Baseline
59.9
82.2
45.6
77.0
66.2
Baseline
26
70
75
81
65.6
w/ w/Quant
59.5
81.7
45.8
77.1
66.0
Attn-4
58.8
82.0
46.2
77.2
66.
Attn-2
33
68
77
65.4
w/ w/Quant
58.0
81.7
46.0
76.2
05.5
Attn-4
28
63
68
75
73
61.4
Attn-8
58.2
82.2
47.0
77.4
w/Quant
57.7
81.9
47.0
77.0
65.9
Attn-6
24
63
65
72
69
58.6
Mistral-P2 7B
Attn-8
15
50
53
58
64
48.0
Baseline
61.5
83.2
13.8
78.5
66.8
MIP-2
29
64
71
70
78
62.4
w/Q y/Quant
61.2
82.5
42.8
78.0
Attn-4
61.0
82.9
44.6
78.0
66.6
MLP-4
21
57
64
65
66
54.6
w /uQuant
61.0
82.8
43.6
77.6
66.3
MLP-6
9
41
48
47
48
38.6
Attn-8
60.2
82.3
44.2
78.8
66.4
w/ w/Quant
60.1
82.0
43.8
77.5
65.9
MLP-8
11
42
43
48
51
39.0
Attention Drop on Long In-Context Task We evaluate the performance of Attention Drop on the long in-context benchmark. Following LongICLBench (Li et al., 2024), we present the results on BANKING77 (Casanueva et al. 2020),, with the only distinction being that sample 100 examples from the test set. BANKING77 is banking-domain intent detection dataset comprising 77 classes. We evaluate
24

---

Published in Transactions on Machine Learning Research (02/2026)
from 1-shot/label to 5-shot /label resulting in contextual lengths of 2k, 4k, 7k, 9k, 14k We employ the for Layer Drop, which enlarges the context window of Llama-2 to 32k using position interpolation. From the results in Table 16 we observe that Attention Drop maintains performance and outperforms MLP Drop.
Table 17:
Post-training performance of L.MMA-322-3B under both LoRA fine-tuning and full-model fine-
tuning on a 500K-sample subset of Tulu 3 Lambert et al. 2024
Model
ARC-C
BoolQ
HellaSwag
MMLU
OBQA
PIQA
RTE
WinoGrande
Avg.
Baseline
51.1
74.1
76.6
56.5
40.8
78.1
53.8
72.1
58.7
w/ LoRA FT
50.2
77.2
75.6
55.4
42.4
77.4
60.7
71.4
59.4
w/ Full FT
53.7
79.5
77.3
55.2
41.8
77.3
67.2
70.8
60.6
Attn-12
44.4
74.8
68.2
31.7
36.2
72.8
63.2
69.6
52.8
w/ LoRA FT
46.3
75.5
71.1
41.4
41.2
75.0
69.0
71.2
57.0
w/ Full FT
50.3
77.5
73.0
51.9
39.4
76.8
73.3
69.3
59.2
MLP-12
29.2
61.3
37.6
53.6
30.0
57.3
52.4
64.0
44.3
w/
LoRA FT
33.0
62.3
52.2
44.5
30.8
64.3
56.0
64.0
47.0
w/ Full FT
39.9
73.2
60.4
53.0
32.6
67.6
63.2
65.5
51.8
Post-Traine Dest-Training on Comprossed Models Post-training has the potential to recover the performance degradation caused by Layer Drop. We consider two post-training scenarios: supervised fine-tuning on compressed base models and on compressed instruct models. For the former, we compress Llama-3.2-32 and perform both full-model fine-tuning and LoRA fine-tuning on Tulu 3 (Lambert et al., 2024), subsampling up to 500K training examples from the mixture. Table 17 shows that the compressed base model still provides strong initialization for subsequent fine-tuning. We further apply LoRA fine-tuning to the compressed instruct models for one epoch, as reported in Tables 18 and 19 After lightweight fine-tuning, the compressed models demonstrate substantial performance recovery. We anticipate that continual pretraining or full-model fine-tuning could further unlock their potential.
Method HellaSwag MMLU OBQA WinoGrande
Avg.
Beschnn
Aln--
Alti LRRA
A21-12
A2--2 LURA
Baseline
82.2
65.5
45.0
77.7
67.6
+ LoRA
81.6
65.9
46.6
77.9
68.0
Attn-12
79.4
63.9
42.2
77.8
65.8
Aocurecy (%)
+ LoRA
81.8
64.9
45.4
77.8
67.5
Attn-16
71.2
38.2
39.4
72.8
55.4
+ LoRA
78.3
53.3
43.6
75.7
62.7
20
Attn-20
42.2
23.0
30.6
58.7
38.6
+ LoRA
72.8
31.9
43.4
69.2
54.3
Lama 2878 Math
Mistal Math
Table 18:
Post-finetuning of Llama-3-8B Alpaca-
Figure 19: Post-Finetuning of Llama-2-7B-M Math and Mistral-7B-Matt on MetaMathQA Yu et al. (2023)
GPT4 Peng et al. (2023a).
C Additional Visualization
Dropping Order on Larger Models We present the dropping order Block Drop and Layer Drop for the 70B Llama models Figure 20 Similar to smaller models, larger models also tend to drop deeper layers first. While the dropping order Blocks differs between Llama-2-70B and Llama-3-70B, we believe this is attributed to different training techniques, e.g., different numbers of training tokens.
25

---

Published in Transactions on Machine Learning Research (02/2026)
Lbno F/
Livi 2 148
Lino 2M
Ulvr
12
29 19
(a) Block
(b) MT.P
(c)
Attention
Figure 20: Visualization of Dropping Order for Block Drop and Layer Drop on Larger Models,
i.., Llama-2-1 2-70B and Llama-3-70B.
To provide comprehensive view of the attention patterns and pairwise token similarity, we present the attention matrices of all attention heads in Figures 22 and
23
along with the token-wise similarity across all
layers in Figures 27
28 and 26
Randomly Initialized Model Before training, the model has not yet learned how different tokens contribute to lexical or semantic ures. In addition, since the model parameters are randomly initialized, attention be- haves approximately as an averaging mechanism, i.c.,
1.0
Cosine Similanty
y; = x, KWWyWo Noting that attention layers are deployed across stacked blocks, token embeddings are progressively aggregated along the depth of the model. Depending on the magnitude of Wy Wo, two scenarios arise: (1) with high magnitude, token embeddings ag- gregate rapidly, leading to high token-to-token similarity (Figure 27 (2) with low magnitude, limited information flow between tokens results in low similarity among token embeddings (Figure 28). Both scenarios demonstrate the redundancy of attention layers: in the first case, aggregating highly similar tokens provides minimal additional information; in the second, the low magnilude of the attention branch restricts feature updates. Together, these findings reinforce the inherent redundancy of the attention mechanism. As a result, deeper attention layers tend to exhibit excessively high input-output cosine similarity as shown in Figure 21 suggesting potential redundancy.
0.001
0.0001
20
Layer Index
Figure 21: Layer-wiss input- output cosine simi- larity on randomly initialized models with varying values of standard deviations a.
Pretrained Model With sufficient training, the model Iearns to identify both lexical and semantic features, with atlention sinks emerging at specific tokens (Xiao et al.) 2024), which contribute signifieantly to overall model performance. Moreover, attention sinks in early tokens create an impactful flow of information to later tokens, such that both early and later tokens incorporate the representations of these sinked tokens. This phenomenon enhances token similarity between distant tokens. Despite this, the sparse attention pattern introduces redundancy, and the resulting high token similarity at certain depths eliminates the need for some attention operations.
26

---

Published in Transactions on Machine Learning Research (02/2026)
Hoxd
0
0
108
138
108
135
162
162
216
216
216
54
108
162
216
54
108
162
218
54
105
162
216
54
108
162
216
0-
109
108
108
108
218
216
216
216
64
108
162
216
54
109
162
215
54
138
162
216
54
108
162
216
108
108
108
108
215
216
216
216
54
108
162
2.16
$4
108
162
216
54
108
162
216
54
108
162
216
Head
Heed
54-
100
108
100
106
162
216
216
216
216
64
108
162
2116
84
108
162
215
54
162
216
54
108
162
216
0.02
0.04
0.06
0.08
0.10
0.12
Figure 22: Attention map at the 4-th layer.
27

---

Published in Transactions on Machine Learning Research (02/2026)
Hoxd
0
0
108
138
108
135
162
216
216
216
54
108
162
216
54
108
162
218
54
105
162
216
54
108
162
216
0
0
0-
109
108
108
108
218
216
216
216
64
108
162
216
54
109
162
215
54
138
162
216
54
108
162
216
0-
108
108
108
108
218
216
216
216
54
108
162
2:16
$4
108
162
216
54
108
162
216
54
108
162
216
Head
Heed
54-
100
108
100
106
162
216
216
216
216
64
108
162
2116
64
108
162
215
54
162
216
54
108
162
216
0.05
0.10
0.15
0.20
0.25
0.30
0.35
Figure 23: Atlention map at the 24-th layer.
28

---

Published in Transactions on Machine Learning Research (02/2026)
HasE
108
108
108
108
182
162
216
216
216
216
64
108
162
216
54
108
162
215
54
162
216
54
108
162
216
Fleed
Headl
Flas?
54
54-
108
108
100
108
162
162
216
216
216
216
54
105
162
2/16
$4
108
162
216
54
162
100
162
Moud
Head
Moad 10
Moad
04-
54-
54-
108
162
162
162
162
216
216
210
216
216
54
Heat 12
Hear13
Head 14
Heee 15
162
182
162
152-
218
216
216
0.04
Figure 24: Attention map for randomly initialized map-neo at the 4-th layer.
29

---

Published in Transactions on Machine Learning Research (02/2026)
Hoxd
0
0-
108
138
108
138
162
216
216
216
54
108
162
216
54
108
162
218
54
105
162
216
54
108
162
216
0
109
108
108
108
218
216
216
216
64
108
162
216
54
109
162
215
54
138
162
216
54
108
162
216
108
108
108
108
218
216
216
216
54
108
162
2:16
$4
108
162
216
$4
108
162
216
S4
108
162
216
Head
llkad
Heed
54-
100
108
100
100
162
216
216
216
216
64
108
162
216
64
108
162
215
54
162
54
100
162
216
0.02
0.04
0.06
0.08
0.10
Figure 25: Attention map for randomly initialized map-neo at the 24-th layer.
30

---

Published in Transactions on Machine Learning Research (02/2026)
Layer
Layer 2
Layer 3
Layer 4
Layar 5
32
224
204
224
Layer
Layer?
Layer
Layer
Layer 10
120
120
123
120
100
160
180
152
100
150
102
Layer 11
Layer 12
Layer 13
Layer 14
Layer 15
0-
182
110
100
231
204
234
224
Layer
Layer
Layer
Layer 19
Layer
35-
30
50
134
128
135
16
Layer 21
Layer 22
Layer 23
Layer 24
Layer 25
150
154
100
132
190
182
190
102
204
234
204
204
204
Layer 27
Layer 26
Layer 29
32
0.2
1.0
Figure 26: Moken-to-tokee similarity in trained MAP-Neo-7B.
31

---

Published in Transactions on Machine Learning Research (02/2026)
Layer
Layer 3
Layer 4
5
Layer
Layer?
Layer
Layer 9
Layer
135
100
150
100
150
100
122
100
Layer 11
Layer 12
Layer 13
Layer 14
Layer 15
231
201
Layer
Layer
Layer
Layer
124
Layer 21
Layer 22
Layer 23
Layer 24
Layer 25
0-
150
15
100
182
19
204
124
204
224
204
Layer 27
25
Layer 29
0.4
0.6
0.8
1.0
Figure 27: Token-to-tte token similarity in randomly initialized Meo-Neo-7B, where the parameters of the attention layers are initialized with a high standard deviation of 0.02.
32

---

Published in Transactions on Machine Learning Research (02/2026)
Layer 1
Layer
Layer
Layer 4
Layer
Lay er 6
Layer
Layer
Layer 9
Layer 10
24
22
32
32-
04-
166
Layer
Layer
Layer
32-
04
36-
138
120
100
100
301
35
300
192
192
31
244
224
Layer
Layer
Layer
Layer
Layer
126
222
180
165
152
182
282
204
231
Layer 21
Layer 22
Layer
Layer 24
Layer 25
224
Layer 26
Layer 27
Layer 28
Layer 29
32-
0.0
0.7
0.4
0.6
0.8
1.0
Figure 28: Token-to-token similarity randomly initialized MAP-Neo- 7B, where the parameters of the attention layers are initialized with a low standard deviation of 0.0002.
33


Reviews:
[{"review_id": "IcamGTRQyN", "reviewer": "Reviewer_pEvQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Authors explore redundancy in transformer models via block, MLP, and attention modules and find attention has a surprising redundancy. With this property, the authors designed Attention Drop which improves efficiency without sacrificing accuracy. The paper shows that transformer models have certain redundancy and have a lot of spares for improving performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Most contributions in the paper were supported by analysis and experimental results. However, I have some further comments:\n\n1. A fundamental question the authors neglect is whether KVCache can keep the model sustainable. Authors may neglect one impact of scaling law and sparsity. As we can see from LoRA and other examples, sparse neural networks are easier for post-training and instruction tuning. Further training can embed more knowledge easier. To show attention drop is effective, one important study is that of investigating that after attention drop and than conducting post-training tasks and seeing how performance is.\n\n2. Authors show that attention has sparsity and redundancy but this does not directly correlate to inefficiency. For example, DeepSeek-V2 / V2.5 leverages sparse attention to improve the time complexity of computation of attention from O(n^2) to O(nk) where k<<n. In other words, having attention sparsity may improve efficiency.", "audience_yn": "Yes", "audience_explanation": "A wide range of audiences would be interested in the findings of the paper as redundancy can help instruct the design of new efficient LLM and transformer architectures. The findings over KV-cache efficiency improvement are also important.", "requested_changes": "Methods:\n1. I am a little bit confused by the block-drop. How do you calculate the cosine similarity. Is it the average for all samples where you inference all X with the model? If so, how many samples do you use? If not, how do you make sure that the cosine similarity results apply to other data samples?\n2. Any theoretical analysis of why cosine similarity is a good metric? For example, let's say for an MLP layer X'=MLP(X). If cos(X', X) is small, what is the relationship with the redundancy of this layer of MLP?\n\nExperiments:\n1. Experiments show that attention drop will not affect post-training.\n2. Authors should show either attention drop is orthogonal to sparse attention and can work with sparse attention (e.g. applied to an open sourced DeepSeek sparse attention model) or prove that for the same backbone model, attention drop is more efficient than sparse attention. I think the former one is easier as usually we don't have enough compute resources to do the later experiments.\n\nWriting:\n1. (Minor). The fonts in figure 1,7,8,9,10,11 are too small.\n\nOptional:\n1. The paper studied block, MLP, and attention. I am quite interested in whether MOE will contain any redundancy as experts are supposed to be designed for efficient usage of knowledge in neural networks.", "broader_impact_concerns": "No ethical concerns need to get addressed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ifrk8GnyLA", "reviewer": "Reviewer_tArF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an investigation into the redundancy across different transformer modules. The paper finds that there’s a substantially larger amount of redundancy in the attention modules compared to the MLP. Significant speedup can be achieved by pruning attention layers with only minor performance loss.", "claims_and_evidence_yn": "No", "claims_explanation": "The pruning baselines are relatively weak -- I provide more details in the Requested Changes.", "audience_yn": "Yes", "audience_explanation": "The paper tackles an important question in improving efficiency in transformer models and would be interesting to researchers in this area", "requested_changes": "Critical:\n1. The framing/analyses:\nThe framing of the paper is a bit unclear. Is this a pruning paper or an interpretability paper? In the current framing, neither aspect is strong enough. For a pruning paper, the baseline of dropping entire transformer blocks is somewhat weak — I’d encourage the authors to discuss SOTA pruning approaches and to compare their approach to the current SOTA. Otherwise, it's very hard to put the performance of the proposed approach into the perspective. If this is an interpretability paper, I’d like to see a deeper analysis of redundancy. Currently, most analyses are validating previous observations. \n\n2. Methods:\na) I’m confused about the Block Drop. P. 2 suggests that this method is proposed by the authors but Table 1 states that it’s adapted from Men et al. ’24. This needs to be clarified in the paper.\nb) Table 2: what are reverse order and relative magnitude? These baselines need to be explained so that the reader can understand them without having to look up the provided reference.\nc) Section 6.3 — what is the importance score here? Cosine similarity?\n\n3. Results:\na) Table 1: How are the numbers of dropped layers determined? Is there a reason for why the number isn't held constant across all manipulations?\n\nb) Table 4: How was the batch size and the number of layers to drop determined?\n\n\nNice-to-have’s:\n1. I generally find the results a bit hard to parse since different tables present only a subset of findings. For instance, Table 1 presents the results for Block, Att, and MLP drop for Mistral-7B and Table 2 presents results for the Llama models. However, Table 1 is missing Block and MLP but presents a larger number of Att drop results. The results would be easier to parse if they were consolidated into one table and showed a consistent set of experiments.", "broader_impact_concerns": "no concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VL5KNXK5td", "reviewer": "Reviewer_bdwR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The pruning of LLM is vital for the community because: 1). the acceleration of inference and training of LLM. 2). Further understanding of the intrinsic properties of LLM. The paper studies the pruning of LLM by carefully designed experiments and observations. Despite that the metric inherits cosine similarity, key factors (e.g., why attention layers are less important than MLP layers) revealed by the paper are impressive. Overall, it is a good and valuable paper.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims in the abstract are **partially** supported by the experiments, e.g., \"We attribute this redundancy to three key factors that constrain representational updates from attention layers: sparse attention patterns, over-smoothed token embeddings, and the low representational magnitude of attention outputs\". Please refer to the section \"Explain your answer above\".", "audience_yn": "Yes", "audience_explanation": "Three summarized reasons for \"pruning more attention layers\" are presented in the abstract, i.e., sparse attention layers, over-smoothed token embeddings, and low representational magnitude of attention outputs. \"low-representational magnitude of attention outputs\" needs to be proved by more evidences. Deep attention layers indeed show sparse attention scores (Figure 10) and over-smoother token embeddings (Figure 12).", "requested_changes": "I have the following concerns:\n\n1. About the metric of pruning, is the cosine similarity enough? The input of a module (e.g., MLP layer, Attention layer) is similar to the output of the module under the cosine metric, where the method regards the module as the module to be pruned. That may have two flaws: 1). The cosine similarity cannot measure the high-level semantic information. 2). Is removing the module that outputs results, which do not shift from the input by a large margin, feasible? Many previous methods utilize the gradients as the metric. Do the authors consider hybrid pruning metric?\n\n2. About the phenomenon that deep attention layers are less important and informative, that is a very valuable observation for the community.\n    \n    1). Maybe, more models can be tested for the observation, e.g., Qwen3-8B. If authors have enough GPUs, more experiments to verify the relation between the depth of the layer and the importance of the layer are expected.\n\n    2). More discussions about less important properties of deep layers are expected. Is that related to gradient vanish in the training?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)