High-Layer Attention Pruning with Rescaling
Authors: Songtao Liu
Peng Liu
Venue: Accepted by TMLR ()

Abstract: Pruning is a highly effective approach for compressing large language models (LLMs), significantly reducing inference latency. However, conventional training-free structured pruning methods often employ a heuristic metric that indiscriminately removes some attention heads across all pruning layers, without considering their positions within the network architecture. In this work, we propose a novel pruning algorithm that strategically prunes attention heads in the model's higher layers. Since the removal of attention heads can alter the magnitude of token representations, we introduce an adaptive rescaling parameter that calibrates the representation scale post-pruning to counteract this effect. We conduct comprehensive experiments on a wide range of LLMs, including LLaMA3.1-8B, Mistral-7B-v0.3, Qwen2-7B, and Gemma2-9B. Our evaluation includes both generation and discriminative tasks across 27 datasets. The results consistently demonstrate that our method outperforms existing structured pruning methods. This improvement is particularly notable in generation tasks, where our approach significantly outperforms existing baselines. Code is available at \url{https://github.com/SongtaoLiu0823/HARP}.

Published in Transactions on Machine Learning Research (01/2026)
High-Layer Attention Pruning with Rescaling
Songtao Liu' Peng Liu
The Pennsyluania State University
Reviewed on OpenReview: https: // openreview. net/ form?? id= jkPBI
Abstract
Pruming is a highly effective approach for compressing large language models (LLMs), significantly reducing inference latency. However, conventional training-free structured pruning methods often employ heuristic metric that indiscriminately removes some attention heads across all pruning layers, without considering their positions within the network architecture. In this work, we propose novel pruning algorithm that strategically prunes attention heads in the model's higher layers. Since the removal of attention heads can alter the magnitude of token representations, we introduce adaptive rescaling parameter that calibrates the representation scale post-pruning to countcract this effect. We conduct comprehensive experiments a wide range of LLMs, including LLaMA3.1-8B Mistral-7B- v0.3, Qwen2-7B, and Gemma2-99 1-an.. Our evaluation includes both generation and discriminative tasks across 27 datasets. The results consistently demonstrate that our method outperforms existing structured pruning methods. This improvement is particularly notable in generation lasks, where our approach significantly outperforms existing baselines. Code available at http:////ggttbbb.cco//nnttttooae
1 Introduction
Large language models (LLMs) Touvron et al. 2023 OpenAl et al. 2023 Jiang et al. 2023 Yang et al.
2024 Gemma2-Teeme Team et al.) 2024). pre-trained on extensive text data, have achieved surprising performance in downstream tasks such as information retrieval (Aaa et al. 2024 code generation (Guo et al. 2024h, and mathematical reasoning [Wang el al., 2023 Yang et al. 2023b Huang el al. 2024). These LLMs, however, contain a huge nunber of parameters, resulting in substantially slower inference speed compared to their smaller counterparts.
Many efforts have leveraged pruning algorithms Frantar & Alistarh 2023 Jaiswal et al. 2023
Xia et al.
2024 Ashkboos et al. Yin et al. 2024a b Zhao et al. 2024) to reduce redundant parameters and lower inference latency without signilicantly compromising the performance. Among these, training- structured pruning methods (Ma)
2024 Xu et al.) 2024 Jaiswal et al.) 2024aa] Zhang et al. 2024b Dong et al. 2024b
ct al. 2023 An et al.) 2024 Sun et al. 2024b) estimate the importance of parameters using predefined pruning metrics, allowing them remove less important components without retraining. While most pruned parameters typically come from feedforward networks, these methods also remove some low-importance attention heads. This is especially beneficial for long-context tasks, where the attention mechanism becomes the main computational bottleneck duc to its quadratic time complexity with respect to sequence length.
However, current pruning algorithms generally prune attention heads across all targeted layers, either by uniformly removing the same heads from each layer (Ma et al. 2023) or by adaptively pruming more heads in certain layers and fewer others (An et al., 2024). These approaches do not consider which attention head positions are actually more redundant and thus better suited for pruning. We argue that the pruning metrics used in these methods not accurate enough to reliably identify the most redundant attention heads. In this work, we aim to address the following question:
Can develop structured pruning algoritiom that effectively removes atlention heads?
'Correspondence Songtao Liu <sk15761 edpu..duuz.

---

Published in Transactions on Machine Learning Research (01/2026)
Generation Task Performance on LLaMA3,1-$B (%)
Rescaling Performance Comparison Across Benchmarks (%)
wio rescaling resccalig
61.45
27
18.35
36.16
15.05
12.61
9
12.43
1.29
0
Magesp
LLM-Prruner
Wanda-sp
FLAP
HARP
0
Triviaqu
Naturaluuettions
GSM8K
Figure 1: Left: Performance comparison of our method against state-of-the-art basclines on LLaMA3.1-BB, demonstrating superior generation capabilities. Right: Ablation study on the rescaling parameter across benchmarks, showing that applying rescaling consistently leads to significant performance improvements over the variant without rescaling.
Wc approach this problem by analyzing the relative importance of different layers in LLMs. Recent studies Men jet al. 2024 Zhong al. 2024 Gromov et al.] 2024 He al.) 2024; Siddiqui al. 2024) demonstrate that pruning blocks entire layers in the high layers of LLMs often yields better performance than pruning those in the low layers. Building these findings, we propose training- free structured pruning algorithm, called HARP (High-layer Attention Rescaled Pruning). that removes the attention heads, specifically the query and key parameters, in the higher layers. After pruming, the attention mechanism in these layers can be skipped entirely. However, bypassing the attention computation may alter the output magnitude, as the value represcntations are no longer combined through weightod averaging based on attention scores over the full sequence. To mitigate this effect, we introduce rescaling parameter in the attention residual block calibrate the token representation magnitude after pruning-
We evaluate the effectiveness of our approach by applying it to LLaMA3.1-SB. Mistral-7B-v0.3, Qwen2-7B. and Gemma2-9B. Extensive experiments on 27 representative downstream tasks show that our method outperforms popular training- free structured pruning algorithms, including Magnitude Pruning (Han et al., 2015), LLM-Pruner (Ma et al. )023), FLAP (An et al. 2024), and Wanda (Sun et al. 2024b). As illustrated on the left. side of Figure 1 our method, HARP, significantly outperforms these baselines on generat ion tasks under the same attention head pruning ratio. More notably, the results on the right side of Figure [ show that applying rescaling significantly improves performance compared to the unscaled variant, demonstrating the importance of rescaling for properly adjusting the represenlation magnitude.
2 High-Layer Attention Pruning
In this section, we first demonstrate the importance of pruning attention heads in LLMs as part of structured pruming algorithms to reduce latency in downstream long-context tasks (Section 2.1). Next, we provide a theoretical and empirical explanation for why attention heads in higher layers are less important (Section 2.2 Finally, we present the details of our pruning algorithm (Section 2.3
2.1 Pruning Attention Heads for Long- Context Modeling
Most existing structured pruning algorithms for LLMs primarily focus pruning parameters in the Feed- Forward Network (FFN), as they contain the majority of the model's parameters. By pruning the FFN, we can significantly reduce the latency. However, pruning allention heads is also crucial. In long-context tasks such as retrieval-augmentted generation (RAG), long-document summarization, and many-shot in-context lcarning (ICL), the attention mechanism often becomes the primary computational bottlenock due to its complexity O (N2) with respect to the sequence length N. We provide a detailed analysis of the time complexity associated with each component of L.LMs.
2

---

Published in Transactions on Machine Learning Research (01/2026)
Theoretical Computation Complexity Analysis. Let N be the sequence length, d the hidden size, and du the intermediate dimension in the FFN. In a standard transformer block, the linear projections in the attention module, including query, key, value, and output projections, have time complexity of O (Nd)). The self-attention mechanism, which includes computing attention scores and aggregating valuos, has a time complexity of O (N²d). The FFN, consisting of two linear transformations with an intermediate dimension du, has a time complexity of (Nddg).
When N dR, the FFN's O (Ndd#) complexity becomes the main time bottleneck. Conversely, when N > du, the O (Nºd) complexity of the self-attention mechanism dominates. In long cong-context tasks, prompt lengths can reach up to 2 million tokens (Jin et al., 22025). making the operations in the self-attention module extremely time-expensive. By pruning attention heads (Bansal et al.) 2022), we can achieve significant improvements in computational efficiency for long-conters tasks while only pruning a small fraction of the model's parameters.
2.2 High-Layer Attention Heads Are Less Important
While many structured pruming algorithms focus on pruning attention heads, they often overlook the attention head position within the network. In this section, we show that attention heads in higher layers are typically less important than those in lower layers.
Theoretical Analysis. In this subsection, we formalize what we mean by "tokens becoming similar" in layer and how this affects the usefulness of self-attention. Let H()) E Nxd denote the hidden representation matrix at layer i, whose i-th row h, corresponds to the representation of the i-th token. Throughout this paper, when we say that "the token representations are (almost) the same", specifically mean that their pairwise cosine similarities are close to 1, i.e., all token vectors are nearly parallel in the representation space. Definition 1. The averuge pairwise cosine similarity among tokens in a matric H E R Nxd is defined as
2 Sim (H) N(N 1)
Σ
h
(1)
Isi<jsN
A value Sim x 1 indicates that all token representations are almost colinear (up to scaling), while smaller value means that the representations point in more diverse directions. This quantity therefore measures the global level of similarity among token representations.
When (") + 1, token representations in layer C become nearly parallel. Since queries and keys are linear projections of H(), their dot products across positions become almost constant, so the row-wise softmaæ yields nearly uniform attention weights. The head thus degenerates into averaging value vectors and cannot distinguish tokens. Conversely, if (im(H(O) is bounded away from 1, attention remains non-uniform and can model meaningful token-to-toker dependencies.
Theorem 1.
A formal proof of Theorem D is provided in Appendix A Intuitively, as the hidden states within a layer become more and more aligned (large (im)H the attention operation behaves like a simple averaging operator that mixes already similar representations. This phenomenon is analogous to over-smoothing in deep GNNs, where repeated averaging causes node embeddings to collapse into a low-dimensional subspace. As a result, attention heads in very high layers, where representations are highly aligned, are less informative than those in lower layers where representations remain more diverse.
He et al. Empirical Analysis. Recent studies Men et al. 2024 Zhong et al. 2024 Gromov et al. 2024
2024 Siddiqui el al. 2024 Liu et al. 2024b Zhang et al. 2024a Jaiswal et al. 2024b Chen el al. 2024b: Kim et al. 2024a) have introduced various heuristic metrics to evaluate the importance of blocks within each layer of LLMs. A common finding across these works that the parameters in lower layers tend to be more important than those in higher layers, lending empirical support our theoretical analysis. One such metric is the Block Importance (BT) score (Men al. 2024):
(H H. ((()
(1) 1 -EH,
(2)
H
2
3

---

Published in Transactions on Machine Learning Research (01/2026)
Algorithm 1 Top-down search for HARP
Input: Base LLM, number of layers to prune P, search values a {0, 0.1, ... 1.0) Initialize: Wbest E <R [1.0, ..0,...,1. 1,0] L total mumber of layers in the model
for - down to L. - P do
PPLbes = oo
for each a E {0,0.1,..... 1.0) do
Set rescaling parameter Obbestl..11 [] = for layer PPL PerplexittyLLM. abbss)
if PPL PPLtes then
PPLbest = PPL
Obest [L - a
end if
end for end for Output: Optimal rescaling parameters Obbest Rp for cach pruned layer
The BI score typically decreases with increasing layer index E, indicating the reduced impact of higher-layer parameters. In this work, we use these empirical observations as additional evidence that pruning higher-layer attention heads tends to be less harmful than pruning lower-layer ones, and we explicitly incorporate this layer-wise asymmelry into our pruning design.
2.3 Methodology
In this section, we introduce HARP, a novel structured pruning algorithm specifically designed remove attention heads in the higher layers of LLMs. Our approach focuses on pruning only the query and key parameters in these layers. As result, we can bypass the self-attention mechanism for these pruned layers. Concretely, when the attention of layer is fully pruned, the layer no longer computes attention scores or performs aggregation; instead, it only applies the value and output projections. The computation within the self-attention module can then be formulated as H = HWyWo.
Rescaling. The aggregation step in self-attention computes, for each token, weighted average of value vectors over all positions in the sequence. This operation not only mixes information across tokens but can also change the overall scale of the hidden representations (Kipf & Welling 2017). We measure this scale using the Frobenius norm of the hidden state matrix H.
Proposition 1. Let H E Nxd denote the token representations, and let Á N be row-normalized attention matrix (i.e., each row is non-negative and sums to 1). Consider the aggregation step in self- self-attention defined by H' = ÁH. In general, we have
H H
(3)
and equality holds if and only if every of À is one-hot vector (so that each output token simply copies one inpul loken without averaging). This means that, except for this degenerale case, the aggregalion step changes the magnitude of token representations.
A detailed proof of Proposition is also given in Appendix A ("Proof of Proposition 17) The proposition implies that removing the aggregation step (by pruning queries and keys and bypassing attention) will in general alter the scale of the hidden states compared to the original model.
Based on this proposition, we argue that the magnilude of token representations can change after pruning, since the aggregation step bypassed. Inspired by normalization approaches (Loffe Szegedy 2015 Ba ct al.) 22166). which maintain the magnitude of token representations within stable range to prevent gradient explosion or vanishing during pretraining, we similarly adjust the magnitude after pruning. This adjustment keeps the token representation magnitude within a stable range relative to their pre-pruning state, helping
4

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Comparison of FLAP with existing pruning algorithms on LLMs.
Method
Weight Update
Calibration Data
Pruning Metric
Magnitude
X
Wayl WWill ||
Wanda
X
FLAP
X
MN X (
X x,,+)
W /
N-1
HARP
x
/
ppl
preserve model performance during inference. We introduce a rescaling parameter into the residual block of the self-attention module as H = + a.
Determining the optimal value of a is challenging, as we treat a as a non-differentiable, layer-wise hyperpa- rameter rather than a trainable parameter. Instead of retraining the model, we propose a simple and efficient greedy search strategy that selects by directly evaluating the perplexity of the pruned LLM on held-out corpus.
To simplify the search process, we adopt a top-down approach: starting from the topmost layer (layer L), we determine the optimal a for that layer while keeping all other layers unpruned or fixed; we then fix this value and proceed to search for a in the next lower layer. This layer-by-layer greedy strategy forms a sequential for-loop that significantly reduces the overall search space, as shown in Algorithm I
In practice, implement this procedure a grid search over small set of candidate values (0,0.1, 1.0), which requires only a modest number of forward passes (perplexity evaluations) and is therefore much cheaper than any training- based method. Designing more advanced search strategies (e.g., Bayesian optimization or differentiable relaxations) is an interesting open direction for future work.
3 Experiments
3.1 Experimental Setup
Baselines. We evaluate HARP against four baselines: Magnitude pruning (an et al. 2015 LLM- Pruner (aa et al.] 2023). FLAP (An et al., 2024), and Wanda (Sun et al. 2024b). as shown in Table We build upon the FLAP to extend Magnitude Pruning and Wanda to structured pruning, referring to these variants as Mag-sp and Wanda-sp. Mag-SP, LLM-Pruner, and Wanda-SP prune attention heads uniformly across all targeted layers by removing the samee number of heads from each layer, whereas FLAP adaplively prunes more heads in some layers and fewer in others.
Model Setting. We conduct experiments on four GQA-based LLMs: LLAMA3.1-BB Touvron el al. 2023) Mistral-7B-v0.33 (Jiang et al., 2023), Qwen2-7B (Yang al.) 2024), and Gemma2-9B Gemmma2 Team et al,, 2024h. For LLaMA3.1-BB and Mistral-7B-v0.3. Qwen2-7B and Gemma2-9B, it is set to 1/7. Therefore, we use HARP to prune 8 layers for LLa 4 layers for Qwen2-7B, 8 layers for Mistral-7B-vO.1 and 6 layers for Gemma2-9B. We find that our method achieves a smaller pruning ratio compared to the baselines. To address this and ensure a fair comparison, we integrate FLAP with our HARP, leveraging FLAP to prune FFN paramcters in our implementation. Aside from this, FFN parameters are not pruned in the baseline methods.
set the attention head pruning ratio to 1/4, whereas for
Baseline Implementation. is important to note that the FLAP official implementation D does not support Gas-based LLMs. Therefore, we use a modified implementation for our experiments. Furthermore, the official implementatior]) of LLM-Pruner is available only for 1--88..1-BB. Our HARP is compatible with any attention mechanism.
hhttps//ggtaaabooo///AAAAAAA..LL Phttee//gthhuu.com/yymuA///LLLL 3 http://ghhhubbcm////oeeee/LLLLLP
5

---

Published in Transactions on Machine Learning Research (01/2026)
Table Performance on generation tasks across LLaMA3.1-8B, Mistral-7B-vO.3, Qwen2-7B, and Gemma2-9B
(%). Best is bold.
LaMA331-88
Method
Parameter Pruning Ratlo
Attention Head Pruning Ratio TriviaQA
Naturalquestios
GSMsK MATH-bard
BDH
Average
Deman
70.58
27.70
49.66
4.61
62.56
43.08
Mag-sp
1/4
2.58
LLM Pruner
1.8%
1/4
55.25
17.67
2.58
0.38
27.03
20.56
Wanda-sp
4.8%
46.06
13.38
0.76
0.25
3.06
FLAP
4.8%
1/4
56.41
17.84
0.83
0.00
0.14
15.05
HARP
FLAP
4.8%
1/4
60.06
21.50
31.77
1.36
30.93
29.12
HARP
3.3%
1.36
Mistral-7H--ve Attention Head Pruning Ratio TriviaQA
Method
Parameter Pruning Ratio
NaturalQuestions
GSM6K MATH-bard
BBH
Average
Donae
0
100%
68.67
28.45
36.54
2.64
08.01
39.06
Mag-sp
4.8%
1/4
1.35
14.44
Wandh-sp
4.8%
1/4
63.15
23.16
0.76
0.00
0.02
17.42
FLAP
4.8%
63.18
22.47
1.67
HARP
FLAP
4.8%
1/4
65.64
26.40
19.71
2.64
39.98
30.88
HARP
3.0%
1/4
66.19
26.90
23.88
2.49
40.35
32.00
Qwen2-7B
Method
Parameter Pruning Ratio
Attention Head Pruning Ratio TriviaQA
NaturalQuestions
GSMsK
MATH-bard
BBH
Dease
61.16
26.02
78.32
20.62
39.87
09.32
Mag-sp
1.8%
1/7
54.32
20.33
62.84
6.87
36.74
34.22
Wanda
1.8%
1/7
FLAP
1.8%
1/7
49.73
19.09
18.35
2.64
37.69
25.50
HARP
FLAP
1.8%
57.76
23.46
12.61
36.20
38.79
HARP
0.9%
1/7
57.61
23.27
02.24
12:16
35.63
38.18
Gemma2-9B
Metbod
Paraneter Pruning Ratio|
Atlention Head Pruning Ratio TriviaQA NaturalQuestions
GSMSK MATH-bard BBH
Average
Deuse
31.02
68.31
14.27
70.80
51.18
Maggegg
3.2%%
1/7
64:41
22.801
8.00
0.15
3.12
18.10
3.2%
1/7
60.29
22.05
0.00
0.00
5.58
17.58
FLAP
HARP
FLAP
3.2%
1/7
67.37
27.81
30.63
4.61
32.97
36.72
HARP
1.0%
67.00
28.42
30.02
4.00
53.02
36.63
Please note that baseline approaches sometimes apply pruning to either all layers or only a subset of them. To ensure a fair comparison and optimize baseline performance, we also tune the pruming layer range for each baseline model by model based on perplexity, since perplexity a commonly-used metric for evaluating the performance pruning algorithms. Details of the experimental setup and the tuning results are provided in Appendix B All experiments are conducted using NVIDIA H100 80G GPUs.
Cost analysis and comparison with baselines. Our implementation performs layer-wise grid search, evaluating K candidate values per layer using perplexity-based forward passes. Thus, the total cost scales as 1.x K forward passes for an L-layer model. For the 8-layer setting used in our experiments with K=10, this amounts to 8 x 10 80 forward passes in total. In contrast, inference-only pruning baselines such as Mag-sp, Wanda-sp, and FLAP generally require only a single calibration aaliiionniifeeness /inference pass to collect imporlance statistics, ie., roughly 1x the cost of one forward pass. Overall, HARP introduces a one-time search overhead (about 80x more forward passes than singlo-pass basclines this 8-layer casc), but remains substantially cheaper than training- based methods; moreover, this cost is amortized since the search performed only once to select good
For evaluation, employ the widely-used 1m an-evaluutioon-hareee package (Gao et al, 2024] to conduct experiments both generation and discriminative tasks. Our generation tasks include 5-shot TriviaQA (Joshi et al. 2017), 5-shot NaturalQuestion
Benchmarks.
Kwiatkowski ct al. 2019), 5-shot GSMSK (Cobbe et al.
2,21b, 4-shol MATH- hard (Hendrycks et. al. 2021b Lewkowyez et al. 2022 3-shot COT (Wei el. al.) 2022a
and BBH (Suzgun et al. 2023). For discriminative tasks, we use 5-shot WinoGrande (Sakaguchi et al. 2021
25-shot ARC-Challenge ( (Clar et al. 2018)) (-shot BoolQ (Clark et al.
2019 0-shot OpenBookQA (Mihaylov
let al.] 2218)), 0-shot PIQA (Bisk al.] 2220),, and 5-shot MMLU Hendrycks al. 2021a). We report
the accuracy for these tasks recommended by the In-evallattion-haneee package. We also evaluate

---

Published in Transactions on Machine Learning Research (01/2026)
Table Performance on discriminative tasks across LLaMA3.1-8B, Mistral-7B-v0.3, Qwen2-7B, and Gemma2- 9B (%). Best is bold
LLaMA3.1-SD
Meshod
Attention Head Pruning Ratio
WinoGrande
ARC-Challeuge
BoolQ
OpenBookQQA
PIQA
MMLU
Average
Mag.sp
1/4
77.43
19.15
78.26
31.50
78.02
58.20
62.14
LLM-Prumer
1/4
Wanda
1/4
76,61
45.99
50.76
28.00
76.93
39.45
58.03
FLAP
1/4
HARP
FLAP
1/4
76,61
51.02
79.82
31,50
78.89
61.96
63.35
HARP
1/4
Mistral
Meshod
Attention Head Pruning Ratio
WinoGrande
ARC-Challegge
BoolQ
OpeuBook(QA
PIQA
MMLU
Average
Mag-sp
1/4
76,48
53.33
69
31.10
79.65
57.01
61.28
Wandhaspp
1/4
39.31
FLAP
1/4
62: 43
41.98
66
29.20
78.99
39.46
52,91
HARP
FLAP
1/4
33.60
60.68
HARP
1/4
77:90
55.55
80.49
33.20
79:22
61.32
64.64
Qwen2-7B
Mechod
Attention Head Pruming Ratio
WinoGrande
ARC-Challenge
OpenBuokQA
PIQA
MMLU
Average
Desee
0
77.03
38.11
84.80
33.20
79.92
70.34
67.07
Mag.sp
1/7
77.37
Wandhesp
1/7
63.30
53.92
81.19
31.00
79.05
02:23
61.78
FLAP
82.79
30.00
66:67
HARP
FLAP
1/7
75.61
67.08
85.23
35.00
79.33
67.97
06.70
HARP
1/7
Gemmo2-901
Meshod
Atlention Head Pruning Ratio
WinoGrande
ARC-Challenge
BoolQ
OpenBookQQA
PIQA
MMLU
Averaage
61.76
70.64
Mag-ap
1/7
66.22
58.19
79.91
34.00
80.30
58.56
62.03
Wandlassp
79.64
65.02
36.20
81.30
70.60
FLAP
1/7
76.09
53.41
78.44
29.00
78.36
67.82
63.89
HARP
FLAP
1/7
65.44
35.20
80.36
70:27
HARP
1/7
79.87
64.85
83.00
34.60
80.06
70.60
68.98
long-context generation performance the LongBench benchmark Bai et al. 2024 which includes 16 datasets. Details about benchmarks can be found in Appendix C
3.2 Main Results
Generation Tasks. Table summarizes the results of all pruming algorithms on generation tasks, showing the superior performance of our proposed method, HARP, across a range of generation tasks and language models. Our method consistently purperfssm all four baselines across different models and benchmarks.
Further analysis reveals notable performance gap between GSM8K and TriviaQA. This difference can be largely attributed to the varying response lengths required by each task. For TriviaQA, unpruned LLMs typically encounter the end-of-sequence (EOS) token within just 16 tokens. In contrast, GSM8K often requires up to 256 tokens to generate a complete solution. Since pruned LLMs generate output one token time, they are more vulnerable error propagation, especially in tasks that demand longer responses. Each additional token introduces more opportunily for cumulative errors. As result, the extended response length required for GSMSK leads to more significant performance drop in pruned models, compared to the relatively short responses needed for TriviaQA.
Discriminative Tasks. Table 3 presents the performance of four LLMs on discriminative lasks, along with the average results across six benchmarks. The results show that our pruning algorithm achieves the highest average performance on LLLMA3.1-8B, Mistral-8B-v0.3, and Qwen2-7B, and performs on par with the best bascline on Gemma2-9B, Interestingly, the performance drop on discriminative tasks is less pronounced compared to generation tasks. This can be attributed to the nature of discriminative tasks, which typically

---

Published in Transactions on Machine Learning Research (01/2026)
Table 4: Performance on long contect generation tasks across LLaMA3.1-8B and Mistral-7B-v0.3. Best is bold
Long-Context Generation Performance Evaluation
Single- Docunentt QA
Multi- Document QA
Surmmarzsation
Few-shot Learning
Synthetie
Code
Model
Method
NrtvQA
Quapes
MF--on
Hotpud
ZWiLMQQ
Musique
GoeReport
QMSim
MolliNews
TREC
TriviatAA
SA SAMSum
PCCouu
PRa
RB-P
Donee
20.32
11.53
12:10
14.51
20.
23.11
00:49
46.45
3.62
19.30
68.
62.
Mistral-BB-003 HaMA3-1-50
Mag-tgi
11.31
8:97
23.98
7:31
7.25
6.22
14.44
19.75
2.96
55.00
80.02
55.17
3.75
8.25
36.
39.16
LLM-Pruner
9.88
6.46
5.69
4.51
50.50
36.67
38 38.81
Wandaepp
14.05
5.98
17.48
6.90
6.27
3.83
8.28
19:05
0.48
00
68.
20. 20.84
4.70
6.12
28.
FLAP
7,85
5.14
13.12
6.06
5,50
4.37
5.39
8.31
2.87
15.50
73.15
10.06
1.55
1.01
38.05
31.57
HARP: FLAP
FLAP
12.88
10.02
24.49
HARP
13.21
10.20
23.98
10:21
10.31
9.88
19:19
19.50
0.00
61.00
$2.57
25.18
6.00
18.38
14:47
25.19
Dense
15.21
6.33
28.52
10:59
11.06
5.15
27.61
21.36
25.88
72,00
90.25
45.89
1.00
12.00
64.85
57.16
Magssp
12.34
9:34
8.75
31.70
4.67
49.68
39.31
Wandaesp
2.37
4.78
6.98
7:18
3.65
7:41
14:19
27.50
63.93
20.25
2.83
3.29
30.
FLAP
1.93
4.48
10.21
9.06
5.85
3.50
9.33
16.91
13.03
25.00
72.91
9.21
2.50
4.25
45.61
36.63
HARP+FLAP
HARP
12.79
8.87
22.42
10:40
10.49
5.01
23.32
19.78
23.25
57.00
88.35
32.
1.00
8.71
40.22
43.19
Table 5: Ablation study layer selection strategies for pruning attention heads, evaluated on six discriminative tasks using LLaMA3.1-8B (%).
LLaMA3.1-BB
Method
WinoGrande
ARC-Challenge
BoolQ
OpenBookQA
PIQA
MMLU
Average
Bottom-8
50.43
37.83
13.40
22.95
32.72
H
52.88
22.35
62.26
19.20
59.09
23.01
39.80
Similarity-8
72.30
43.94
79.91
25.80
72.25
57.53
58.62
Top-8
72.53
46.59
82.14
28.20
74.16
58.73
60.39
involve selecting a single correct answer from small set of options. Such tasks are generally less difficult than generation tasks, which require predicting the next token from the full vocabulary, making them more sensitive to pruning.
3.3 Long-Come Context Modeling
In this section, we evaluate the performance of structured pruning algorithms on long-context generation tasks. We use LLaMA3.1-8B and Mistral-8B-v0.33 for our experiments. For evaluation, we use LongBench (Bai et al., 2024), which encompasses diverse range of tasks categorized as follows: single-document QA (NrtvQA, Qasper, MF-en), multi-document QA (HotpotQA, 2WikiMQA, Musique), summarization (GovReport, QMSum, MultiNews), few-shot learning (TREC, TriviaQA, SAMSum), synthetic tasks (PCount, PRe), and code-related tasks (Lee, RB-P). This comprehensive benchmark allows us to assess performance across various long-context scenarios.
Table demonstraats the performance of various pruning algorithms across long-context benchmarks. Our proposed method outperforms baseline approaches in most benchmarks on both LLaMA3.1-8B and Mistral- 7B-v0.3 architectures. The improvements are particularly significant in multi-document QA tasks such HotpotQA and 2WikiMQA, indicating better preservation of reasoning abilities and langranne dependencies. While other methods like Mag-SP achieve strong results in specific areas (e.g., MF-en), our method attains the highest number of rank-1 performances across diverse benchmarks, highlighting its robustness for long-context. processing.
3.4 Exploring Layer Selection for Pruning Attention Heads
In this section, we conduct an ablation study to support our claim that attention heads higher layers are less important than those in lower layers. We evaluate four different strategies for selecting layers to prune, using 1.LBMA3.-8BB for experiments. The first strategy prunes attention heads in the lowest 8 layers
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table Ablation study of HARP on tasks using LLaMA3.1-8B, comparing performance with and without scarch (%). Best is bold
Discriminative
Method HARP
WinoGrande
ARC- ChC-Chaleeuge
BoolQ
OpenBookQA
PIQA
MMLU
Average
72.53
#2.14
28-20
74.16
58.73 64.83
01:30
searched
78.06
53.75
75.02
33.20
78.89
63.96
Short-Context Goneration Tasks
Method
TrivinQA
NaturalQuestions
GSM8K
MATHI-hard
Average
HARP
1.0 searched
12:43
3.24
1.29
0.00
0.03 34.53
3.40
61.45
21.99
36.16
1.36
31.10
(Bottom-8). The second selects 8 layers using Hessian-based metric proposed in (Yang al.) 2023a (Hessian-8). The third leverages similarity-basee metric for the self-attention module, proposed in (He ct al. 2024), to select 8 layers (Similarity-8). The fourth strategy, which is our approach, prunes attention heads in the highest 8 layers (Top-8). We set a = 1 for all layers. Additional details on the Hessian-based and similarity- based metrics are provided in Appendix D
For both the Hcssian-8 and Similarity-8 strategies, we compute the respective metrics using the WikiText dataset (Merity et al. 2017). The selected layer indices for pruning are [12, 13, 14, 15, 19, 22, 24, 26) based the Hessian-based metric, and [23, 24, 25, 26, 27, 28, 29, 30] based on the similarity- based metric. Based the selected layer indices, observe that the Hessian-8 strategy prunes attention heads in the middle 8 layers, the Similarity-8 strategy targets layers 23 to 30, and the Top-8 strategy focuses on layers 24 to 31. Table | reports the results these four strategies across six discriminative lasks. The results indicate that pruning attention heads in the higher layers yields better performance than pruning those in the lower layers. These findings support our theorctical claim that attention heads higher layers are less important overall model performance.
3.5 Search for Rescaling
To determine the optimal values of a, compute perplexity the WikiText (Merity et al. 2017) and Pile10K (Nanda 2022) datasets using LLaMA3.1-SB. The resulting optimal values are [0.8, 0.2, 0.1, 0.0, 0.0, for WikiText and [0.8, 0.2. 0.1, 0.0, 0.1, 0.1, 0.0] for Pile10K, with in- dices corresponding to layers counted from the top. The similarity between the two sets of values suggests a consistent trend in the layer-wise importance.
We further evaluate both configurations by measuring perplexity on both datasets. Using the values obtained from WikiText, the perplexity scores are 12.16 on WikiText and 28.94 Pilc10K. Conversely, using the values derived from Pile10K yields scores of 12.23 on WikiText and 28.50 on Pile10K. These results demonstrate that both configurations produce comparable perplexity each dataset, confirming the robustness of the searched a values. For simplicity, we adopt the a values derived from WikiText throughout this work. Figure 2 presents the searched a values for LLaMA3.1-8B, Mistral-7B-v0.3, Qwen2-7B, and Gemma2-9B. Our results show that the optimal a values vary across models and pruned layers. For LLaMA3. 1-8B, Mistral-7B-v0.3, and Qwen2-7B, we observe a general trend where the scarched a values increase with the layer index. Even for models with the same architecture but different sizes, such as LLaMA3.1 and ALaMA3.1-70B. the optimal values can be different. For LLaMA3,1-705B the searched values are [0.0,0.1, 0.0, 0.0, 0.2, 0.0, 0.0. 0.0), which is noticeably different from those obtained for LLaMA3.1-8B.
Please note that the WikiText dataset provided by dnr-evallationnhhaaraee LLaMA3.1-8B, our search involves 8 separate for-loops, each sweeping a values from to 1.0. As a result, the total number of evaluations amounts to 5,456 sub-tasks. We conduct the experiments using an NVIDIA H100 GPU, and the total runtime remains under two hours, which is comparable the time required for a typical experiment generation task.
includes only 62 sub-tasks. For
9

---

Published in Transactions on Machine Learning Research (01/2026)
Searched Alpha on LLaMA3.1-80
Searched Alpha Mistral-7B-vol
0.9
0.7
0.7
0.5
Alpha
0.5
Alpha
0.3
0.3
0.1
0.1
0.1
-0.1
24
26
28
30
32
24
26
28
30
32
Layer Index
Layer Index
Searched Alpha Qwen2-7B
Searched Alpha Gemma2-95
0.9
0.7
HARP
HARP
0.7
0.6
0.5
Alpha
0.5
Alpha
0.4
0.3
0.3
0.1
0.2
-0.1
0.1
24
25
27
36
38
40
42
Layer Index
Layer Index
Figure 2: Searched alpha on LLaMA3.1-8B, Mistral-7B-v0.3, Qwen2-7B and Gemma2-9B.
Ablation Study, We conduct an ablation study to demonstrate the effectiveness of a search for rescaling. As shown in the Table 0 incorporating search consistently improves performance, particularly on generation tasks, where we observe substantial performance gain.
3.6 Efficiency Analysis
In this section, we analyze the time complexity of our prumed model. We conduct experiments with varying sequence lengths to measure inference processing time. Using LLaMA3.1-8B as our test model, we evaluate sequences of lengths 1024, 2048, 4096, 8192, 16384, 32768, and 65536. All measurements are conducted on single NVIDIA H100 S0G GPU with each experiment repeated 10 times to ensure statistical reliability. For comparison purposes, test both pruned model (with 8 layer attention heads pruned) and the original model (with 0 attention heads pruned) to quantify the efficiency gains. Our measurements reveal consistent performance across all runs, with tight 95% confidence intervals: for the original model, inference times ranged from 0.0371 + 0.0001s for 1024 tokens to 5.4466 + 0.0022s for 65536 tokens; for the pruned model, times ranged from 0.0355 + 0.0001s for 1024 tokens to 4.5394 + 0.0063s for 65536 tokens. Figure 3 demonstrates the significant speed gains achieved by our pruned model. As sequence length increases, our speed improvement becomes more substantial. With just a 3.3% parameter reduction, we achieve 16.7% improvement in processing speed for t5,5en-token sequences. This highlights how targeted attention head pruning can substantially reduce inference latency in long-context tasks.
4 Related Work
Pruning. Pruning is a widely adopted and efficient technique in both Computer Vision and Large Language Modcls. It can be categorized into two main types: structurod pruning and unstructured pruning. structured pruning [Lagunas al.) 2021 Xia et al. 2022 Kurtic et al. 2023 He Xiao 2023 Xia al. 2024) involves removing entire filters from neural networks, making it particularly conducive to model deployment. On
10

---

Published in Transactions on Machine Learning Research (01/2026)
Inference Time Comparison
Original Model
16
5
Pruned Model layer attention heads)
14
Inference Time (s)
4
12
3
10
2
Spa
1
6
0
1024
2048 4096 8192 16384 32768 65536
Sequence Length (log2)
Figure Time complexity and runtime scaling with sequence length for the original and pruned models. Solid lines show the absolute runtime (left. y-axis), while the green dashed line denotes the relative speedup (%) of the pruned model over the original, plotted on the sccondary y-axis (right).
the other hand, unstructured pruning (Chen et al.
2020 Sanh al. 2020)
focuses on removing individual
neurons within the network. Some recent works (Men al. 2024
Zhong et al. 2024 Gromov et al. 2024
He et al.
2024 Siddiqui et al. 2024 Liu et al. 2024b Zhang al.
2024a Jaiswal all 2024b Chen et al.
2024b
Kim et al. 2024a have been proposed to prune blocks in the higher layers of LLMs.
Quantization. Quantization approaches Wei et al.
2022b Yao et al. 2022 Frantar et al. 2023 Xiao
et al.
2024 Lin et al. 2024g compress language models by converting weights and activations to lower precision formats such as 8-bit, 4-bit, or even 2-bit integers. This
2023 Dettmers & Zettlemoyer 2023 Park et al.
reduction in precision substantially decreases memory requirements and enhances inference speed, enabling the deployment of large language models in environments with limited computational resources.
KV Cache Compression.
Recent works
Liuu el al.
2023 Anagnostidis el a.. 2023 Zhang et al. 2023
Ge et al.
2024
Xiao et al. 2024
Kim et al..
2024b
Zhang et al. 2024c
Nawrot et al.
2024
Tang et al.
2024
Liu et al.
2024d
Dong et al.
2024a
Yue et al.
2024 Cai et al. 2024 Liu et al. 2024c a
Hooper et al, have focused
2024
Sun et al.
2024c
Chen et al.
2024a
Sun et al.
2024a Jiang et al. 2024 Li et al.
2024
on compressing
the KV cache reduce GPU memory lootprint. Some (Liu et. al.,
2024d
Hooper al.,
2024 Yue approaches like H2O (Zhang al.]
2024 of these methods utilize quantization techniques achieve compression. Additionally
2023) eviet less important tokens, while LESS (Dong al., 2024a
generates condensed representations through training optimize memory usage.
Dettmers et al. 2023 Low-rank Approximation. Low-rank approximation methods Hu et al., 2022
Lin et al. 2025: Wang et al., 2025: Chang et al. 2025) project token representations into a low-dimensional latent space and subsequently reconstruct the full representations via up up-projection. This class of methods significantly reduces the number of trainable paramcters during fino-tuning.
5 Conclusion
In this work, we propose HARP, a novel pruning algorithm that removes attention heads in the higher layers of LLMss. Extensive experiments across short-context and long-context, generation tasks, well
11

---

Published in Transactions on Machine Learning Research (01/2026)
discriminative benchmarks, demonstrate the superiority of our proposed method. Ablation studies further reveal the importance of incorporating a rescaling parameter to maintain performance.
Beyond these empirical gains, HARP has clear practical implications: by reducing inference cost without requiring architectural changes or retraining from scratch, it can facilitate the deployment of LLMs in resource-constrained environments, enable faster serving for interactive applications, and make long-context. reasoning more affordable in real-world systems.
This study has limitations: mainly focus attention-head pruning in higher layers, leaving other components underexplored. Future work includes extending HARP to larger and multimodal models, combining it with techniques such as quantization or KV-cache compression, and developing more task-aware pruning criteria to further improve robustness and efficiency.
References
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-based adaptive structured pruning for large language models. In AAAI Conference Artificial Intelligence, 2024.
Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, and Thomas Hofmann. Dynamic context pruning for efficient and interpretable autoregressive transformers. In Advances in Neural Information Processing Systems, 2023.
Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannanch Hajishirzi, and Wen-tar tau Yih. Reliable, adaptable, and attributable language models with retrieval. orXic preprint 2403.03187, 2024.
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns. In International Conference on Learning Representations, 2024.
Jimmy Lei Ba, Jamic Ryan Kiros, and Geoffrey Hinton. wrXXi:1607.064/ 2016.
Layer normalization. arXiv preprint
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng. Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Association for Computational Linguistics, 2024.
Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. Rethinking the role of scale for in-context learning: An interpretability-onn case study at 66 billion scale. arXiv preprint arXic:2212.00 09095,, 2022.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence, 2020.
Ruisi Cai, Yuandong Tian, Zhangyang Wang, and Beidi Chen. Lococo: Dropping in convolutions for long context compression. In International Conference on Machine Learning, 2024.
Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Yu Lån, Chong- Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, and Kai-Chiang Wu. Palu: KV-cache compression with low-rank projection. In International Conference on Learning Representations, 2025.
Renze Chen, Zhuofeng Wang, Beiquan Cao, Tong Wu, Size Zheng, Xiuhong Li, Xuechao Wei, Shengen Yan, Meng Li, and Yun Liang. Arkvale: Efficient generative LLM inference with recallable key-value eviction. In Advances in Neural Information Processing Systems, 2024a.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. In Advances in Neural Information Processing Systems, 2020.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Xiaodong Chen, Yuxuan Hu, and Jing Zhang. Compressing large language models by streamlining the unimportant layer. Xixiv preprint arxin:2403. 19135, 2024b.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toulanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In North American Association for Computational Linguistics, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 8808005557, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ai preprint arXiv:2110.14100 2021.
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning, 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems, 2023.
Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, and Beidi Chen. Get more with less: Syuthesizing recurrence with kv cache compression for efficient Ilm inference. In International Conference on Machine Learning, 2024a.
Poijic Dong, Lujun Li, Zhenhong Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruncr-zcro: Evolving symbolic pruning metric from scratch for large language models. In International Conference on Machine Learning, 2024b.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot.
In International Conference on Machine Learning, 2023.
Elias Frantar, Salch Ashkboos, Torsten Hoefter, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In International Conference on Learning Representations, 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas MuennighofT, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thitc, Ben Wang, Kevin Wang, and Andy Zou. A framework for fow-shot language model evaluation, 2024.
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawci Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In International Conference on Learning Representations, 2024.
Cemmaa-Teemm, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L'eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram'e, Johan Ferret, Peter Liu, Pouya Dehghani Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Staúczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Boxi Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Christoper A. Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozi' nska, D. Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimezak- Pluci nska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Bocker, Joe Fernandez, Joost R. van Amersfoort, Josh Gordon, Josh Lipschultz, Joshua Newlan, Junsong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican,
13

---

Published in Transactions on Machine Learning Research (01/2026)
Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, L. Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Gorner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthow Watson, Meg Risdal, Mchran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengehong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, S. Mc Carthy, Sarah Perrin, S'ebastien Arnold, Scbastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomás Koeiský, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phocbe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Brian Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeffrey Dean, Demis Hassabis, Koray Kavukcuoglu, Cl'ement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at a practical size. arXiv preprint 2Xx::2408000118 2024.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprinl arXic:2403. 17887, 2024.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wi,, YK Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming- the rise of code intelligence. arXiv preprint or Xiv:2244 14196, 2024.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advonces in Neural Information Processing Systems, 2015.
Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li. What malters in transformers? not all attention is needed. arXiv preprint arxiv:2406.10 15786, 2024.
Yang He and Lingao Xiao. Structured pruning for deep convolutional neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Advances in Newral Information Processing Systems, 2021b.
Coleman Richard Charles Hooper, Schoon Kim, Hiva Mohammadzadch, Michael W. Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In Advances in Newral Information Processing Systems, 2024.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In International Conference on Learning Representations, 2024.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
Ajay Jaiswal, Shiwei Liu, Tianlong Chen, and Zhangyang Wang. The emergence of essential sparsity in large pre-trained models: The weights that matter. In Advances in Neural Information Processing Systems, 2023.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Ajay Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, and Yinfci Yang. Compressing LLMs: The truth is rarely pure and never simple. In International Conference on Learning Represenlations, 2024a.
Ajay Kumar Jaiswal, Bodun Hu, La Yin, Yeonju Ro, Tianlong Chen, Shiwei Liu, and Aditya Akella. FFN- SkipLLM: A hidden gem for autoregressive decoding with adaptive feed forward skipping. In Empirical Methods in Natural Language Processing, 2024b.
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Maric-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. arXiv preprini ar Xrv:22300 066222 2023.
Huiqiang Jiang, YUCHENG LI, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yow Lin, Yuqing Yang, and Lili Qiu. Minference 1.0: Accelerating pre- filling for long-context LLMs via dynamic sparse allention. In Advonces in Newral Information Processing Systems, 2024.
Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. Long-context LLMs meet RAG: Overcoming challenges for long inputs in RAG. In International Conference on Learning Representations, 2025.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics, 2017.
Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung- Kyu Song. Shortened Ilama: A simple depth pruning for large language models. arXiv preprint arXiv:2402.00 02884. 2024a.
Jang-Hyun Kim, Junyoung Yeom, Sangdoo Yun, and Hyun Oh Song. Compressed context memory for online language model interaction. In International Conference on Learning Represendations, 2024b.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Eldar Kurtic, Elias Frantar, and Dan Alistarh. ZipLM: Tnference-aware structured pruning of language models. In Advances in Neural Information Processing Systems, 2023.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 2019.
François Lagunas, Ella Charlaix, Victor Sanh, and Alexander Rush. Block pruning for faster transformers. In Empirical Methods in Natural Language Processing, 2021.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo. Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems, 2022.
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. In Advances Neural Information Processing Systems, 2024.
Chi-Heng Lin, Shangqian Gao, James Scale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen- Yen-Chang Hsu. ModeGPT: Modular decomposition for large language model compression. In International Conference on Learning Representations, 2025.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device 1lm compression and acceleration. In Conference on Machine Learning and Systems, 2024.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J.L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Mong Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R.J. Chen, R.L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S.S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W.L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X.Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. Wu, Y.K. Li, Y.X. Wei, Y.X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yonggiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z.Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, and Ziwei Xie. Deepseek-v2: A strong, economical, and efficient mixture-of-exerts language model. arxiv preprinl 2024a.
Songwei Liu, Chao Zeng, Lianqiang Li, Chenqian Yan, Tean Fu, Xing Mei, and Fangmin Chen. Foldgpt: Simple and effective large language model compression scheme. arXiv preprint arXiv:2407.0080 2024b.
Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Anant hanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang, Cachegen: Ky cache compression and streaming for fast large language model serving. In Conference of the ACM Special Interest Group on Data Communication, 2024c.
Zichang Liu, Adilya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyriliddis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. In Advances in Neural Information Processing Systems, 2023.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-freo asymmetric 2bit quantization for KV cache. In International Conference on Machine Learning, 2024d.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-pruer: On the structural pruning of large language models. In Advances in Neural Information Processing Systems, 2023.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. Xiv preprint arXiv:2403.0385. 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Empirical Methods in Natural Language Processing. 2018.
Neel Nanda, NeelNanda/pile-- 10k datasets at hugging face. N 2022.
attpe://nnginggcccco//attattts
16

---

Published in Transactions on Machine Learning Research (01/2026)
Piotr Nawrot, Adrian Lańcucki, Marcin Chochowski, David Tarjan, and Edoardo Ponti. Dynamic memory compression: Retrofitting LLMs for accelerated inference. In International Conference on Machine Learning, 2024.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haim ing Bao, Mo Bavarian, Jeff Belgum, Trwan Bello, Jake Berdine, Gabriel Bernadett-Shape Shapeio,,,, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Benjamin Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Flcti, Tyna Eloundou, David Farhi, Liam Fodus, Niko Felix, Sim 'n Posada Fishman, Juston Forte, Is abella Fulford, Loo Gao, Elic Georges, Christian Gibson, Vik Gool, Tarun Gogineni, Gabriel Goh, Raphael Gontijo-Lopes, Jonathan Gordon, Morgan Graßstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Ryan Kiros, Matthew Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosie, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Ma teusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Adeola Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel P. Mossing, Tong Mu, Mira Murati, Oleg Murk, David M'ely, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Ouyang Long, Cullen O'Keefe, Jakub W. Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alexandre Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Pondé de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea. Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack W. Rac, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario D. Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin D. Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, llya Sutskever, Jie Tang, Nikolas A. Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tsong, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer'on on Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll L. Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qim ing Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report. arXiv preprint ar Xrxi:22903. nate:22333.8774,, 2023.
Gunho Park, Bacscong park, Minsub Kim, Sungjac Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. LUT-GEMM: Quantized matrix multiplication based on LUT's for efficient inference in large-scale generative language models. In International Conference on Learning Representations, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Commonications of the ACM, 2021.
17

---

Published in Transactions on Machine Learning Research (01/2026)
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. In Advances in Newral Information Processing Systems, 2020.
Han Shi, Jiahui Gao, Hang Xu, Xiaodan Liang, Zhenguo Li, Lingpeng Kong, Stephen M. S. Lee, and James Kwok. Revisiting over-smoothing in BERT from the perspective of graph. In International Conference Learning Representations, 2022.
Shoaib Ahmed Siddiqui, Xin Dong, Greg Heinrich, Thomas Breuel, Jan Kautz, David Krueger, and Pavlo Molchanov. A deeper look at depth pruning of llms. arXiv preprint arXiv:2407. 16286, 2024.
Hanshi Sun, Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. Shadowkv: Kv cache in shadows for high-throughput long-context Ilm inference. arXiv preprint arXte:2410.21466 2024a.
Mingjic Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In International Conference on Learning Representations, 2024b.
Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. Tn Advances in Neural Information Processing Systems, 2024c.
Mirac Suzgun, Nathan Scales, Nathanacl Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether thouggt can solve them. In Association for Computational Linguistics, 2023.
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. QUEST: Query-aware sparsity for efficient long-context LLM inference. In International Conference on Machine Learning, 2024.
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Authony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Tnan, Marein Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint ar urXiv:2307.09900 2023.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang, SVD-LLM: Fruncation-awarr singular value decompo- sition for large language model compression. In International Conference on Learning Representations, 2025.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In Internalional Conference on Learning Representations, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advonces in Neural Information Processing Systems, 2022a.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. In Advances in Newral Information Processing Systems, 2022b.
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Association for Compulational Linguistics, 2022.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerating language model pre-training via structured pruning. In International Conference on Learning Representations, 2024.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, 2023.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, 2024.
Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang, Kaipeng Zhang, Peng Gao, Fengwei An, Yu Qiao, and Ping Luo. BESA: Pruming large language models with blockwise parametor-cfficien sparsity allocation. In International Conference on Learning Representations, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Ke-Yang Chen, Kexin Yang, Mei Li, Min Xue, Na. Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yunyang Wan, Yunfei Chu, Zeyu Cui, Zhenru Zhang, and Zhi-Wei Fan. Qwen2 technical report. arXio preprint 2ix:224477 10671, 2024.
Huanrui Yang, Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision transformer pruning with hessian-aware saliency. In IEEE/CVF Conference on Computer Vision and Pottern Recognition, 2023a.
Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan J Prenger, and Animashree Anandkumar. Leandojo: Theorem proving with rotrieval-augmented language models. In Advances in Neural Information Processing Systems, 2023b.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient. and affordable post-training quantization for large-scale transformers. In Advances in Newral Information Processing Systems, 2022.
Lu Yin, Ajay Kumar Jaiswal, Shiwei Liu, Souvik Kundu, and Zhangyang Wang. Junk DNA hypothesis: Pruning small pre-trained weights Irreversibly and Monotonically impairs "difficult" downstream tasks in LLMs. In International Conference on Machine Learning, 2024a.
Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Kumar, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning LLMs to high sparsity. In International Conference Machine Learning, 2024b.
Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. Wkvquant: Quan- tizing weight and key lea/value cache for large language models gains more. arXiv preprint or Xto:2442.12265, 2024.
Yang Zhang, Yawei Li, Xinpeng Wang, Qianli Shen, Barbara Plank, Bernd Bischl, Mina Rezaei, and Kenji Kawaguchi. Finercut: Finer-grained interpretable layer pruning for large language models. arXiv preprinl arXiv:2405.1880 18218, 2024a.
Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug-and-play- An efficient post-training pruning method for large language models. In International Conference on Learning Representations, 2024b.
Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, and Rongrong Ji. Cam: Cache merging for memory-efficient LLMs inference. In International Conference on Machine Learning, 2024c.
19

---

Published in Transactions on Machine Learning Research (01/2026)
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Re, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems, 2023.
Bowen Zhao, Hannanch Hajishirzi, and Qingqing Cao. APT: Adaptive pruning and tuning pretrained language models for efficient training and inference. In International Conference on Machine Learning, 2024.
Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, and Liangzhi Li. Blockpruner: Fino-grained pruning for large language models. ar preprinl arXi::2406. 10594, 2024.
20

---

Published in Transactions on Machine Learning Research (01/2026)
A
Theoretical Analysis
Definition 2. Shi et al. 2022) Define the over-smoothing subspace
M: R Y== eC, C (R
where [1,1,... 1]T RNx1, N is the number of lokens, and d is the dimension of loken represenlations.
Intuitively, any matrix Y M has identical rows: every token in the sequence shares exactly the same representation vector. We therefore regard M as the set of "maximally over-smoothed" token representations, where no loken-specific information is preserved.
Shi et al. 2022) Define the distance between matriz H R Nxd and the over-smoothing subspace M as
Definition 3.
d (H) = min H YIFF.
YEM
where || ||r denoles the Frobenius norm.
Lemma 1
(hi et al. 2022 For a self-attention matriz A, any H.B E Nxd and any 01, 02 2 0, we have:
d (HW) S s ssdM(H)
dm(o(H)) & (H)
(4)
dm(a, a s and M (H) a2dd
dmHAH) S Ammax d M (H),
(5)
where Amax is the largest eigenvalue of Á (I- ee ) and s is the largest singular value of W.
By Lemma 1 we have the following result.
Lemma 2 ( SShi et al. 2022 ). For a Transformer block with h heads, we have
(H(())))) S vvd((H
(6)
where =1+ (81 s s> 0 is the largest element among all singular values of all W A is the largest eigenwalue of all A (I ) for each self-allention matrie A, and B1, B2 are the minimumm stondard deviations for the two layer normalization operations.
The proofs of these two lemmas can be found in (Sh al., 2022
shows that 1, then d (H((+)) < d (H(())) Lee. the hidden representations move closer M as the depth l increases. In other words, deeper layers become progressively more over-smoothed.
Lemma 2
Definition 1. (Average pairwise cosine similarity) Define the average pairwise cosine similarity among lokens in matrir H E R as
h
Sim H N(N 1) 1siejeN
Σ
A value Sim x indicates that all token vectors are nearly parallel (up to scaling), while smaller values correspond more diverse token directions. Thus, Sim(H) measures the global similarity of token representations.
Lemma 3. For any loken representation matria Nxd, if ils distance the over-smoothing subspace M tends zero, then its average pairwise similarity tends to one, i.e.,
( -
Sim(H) +
Proof. This the proof of Lemma 3 By definition, M (Y Nxd Y eC, C e Ixd),, where all token embeddings are equal (each row of is C). If (HH 0, then there exists some C such that H eC, meaning h, - C for all i. Hence,
h. |ICI2 and [|>||2||b,ll ||CI|3
21

---

Published in Transactions on Machine Learning Research (01/2026)
for any # we have
h hy h l2 hyll2
1.
Averaging over all pairs (i. then yields Sim H)
-1
Theorem 1 When ())) - 1, token representations in layer I become nearly parallel. Since queries and keys are lineur projections of H()), their dol products across posilions become almost constanl, so the row-wise softmax yields nearly uniform attention weights. The head thus degenerates into averaging value vectors and cannot distinguish tokens. Conversely, Sim(H(() is bounded away from 1, attention remains non-uniform and can model meaningful token-to-token dependenctes.
Proof. This is the proof of Theorem Suppose (i) 1. By Lemma 3 this implics (m(H(() 0, all token embeddings (c) converge a shared direction, ie., they become almost parallel.
In self-attention layer, the query and key matrices are obtained from H()) by linear projections:
Q Q HO
K(O = =
Since linear maps preserve the property of vectors being almost parallel, the rows of Q and K(O) also become nearly parallel when (i) - 1. Consequently, the inner products q ky for all i, converge to (almost) the same value:
k = for all i.j.
The attention weights for fixed query position are given by cxplq,ky/VVdx
aij exp(a ky Vdx)
If all logits q. jj are (spproximately) equal, then the softmax output [d...... ain] approaches the uniform distribution over NJ. Therefore, the attention weights no longer distinguish between different tokens, and the corresponding attention head effectively reduces to averaging the value vectors across positions.
As a result, when (i)(H()) is close to 1 in high layers, self-atlention behaves like a simple smoothing operator that mixes already similar representations, contributing little to further feature refinement or discrimination. In contrast, lower layers where (i)( bounded away from 1, attention weights remain non-uniform and capture meaningful loken-to-token dependencies, making those layers more informative.
Proof of Proposition 1
Proof. Recall that Á E NNN is row-normalized attention matrix and
H' AH, Hed
We first show that if every row Å a onc-hot vector (i.c., has exactly one entry equal to 1 and all others equal to 0), then the Frobenius norm is preserved for any H. this case, À is a row-permutation matrix: there exists a permutation æ of (I. such that
(AH). =H H 100 Vi.
Thus Å only permutes the rows of H, and we have
N d
A F (AHH),) Hx
H? |
Therefore, if each row of À is one-hot, then | ||]] for all H.
22

---

Published in Transactions on Machine Learning Research (01/2026)
We now prove the converse: if row-normalized attention matrix À preserves the Frobenius norm for all inputs, then each row of A must be one-hot. Assume that
|AH|| F ||]]| for all H E R
Consider the special case where H has only one nonzero column: let x RERN be arbitrary and define Then
H 0
0] e ERRx
AH = [Ax
0 0J.
and the Frobenius norm condition reduces to
||xx||2 |xx|2 for all E
Hence Å is isometry RN, which implies
Ix Vx. x x = || ||x||2
so ÄTÁ=I Thus Å an orthogonal matrix.
On the other hand, as an attention matrix, Å row-normalized with nonnegative entries:
Á.j 2
Σ A,==1 for cach row i.
Let a, denote the i-th row of Å. Orthogonality implics ||a:||2 1 for all i, while row-normalization gives Σ, ar and ay 0. Therefore,
N aij = 1,
N
Σ
a Σ
=1
aij 2 O.
J=1
For any a < 1, we have a² a. Hence, if any entry of a, satisfies ai < 1, then
N
N
Σ
aij
1
J=1
which contradicts 2 1. Thus every entry of a, must belong to (0, 1}. Since the entries are nonnegative and the row sum is each row has exactly one entry equal to and all others equal to 0, i.e., each row is one-hot.
Combining both directions, conclude that a row-normalized attention matrix preserves the Frobenius norm of all input representations if and only if each of its rows is a one-hot vector. In practical self-attention, the learned attention weights are not row-permutation matrices, so the aggregation step H' = ÁH typically changes the magnitude (Frobenius norm) of the token representations, as claimed in Proposition | 0
B Baseline Method Pruning Layer Range Tuning
To fairly compare our method against existing pruming approaches, we use perplexity tune the pruning layer range for all baseline methods. For Mag-sp, Wanda-sp, and LLM-Pruner, the pruming ratio is the same across all pruning layers, whereas FLAP employs adaptive pruning strategy, allowing different pruning ratios for each layer.
LLaMA3.1-88 1-8T and Mistral-7B-vO... have 32 layers, each with 8 key/value attention heads. Qwen2-7R8 consists of 28 layers with 4 key/value attention heads per layer, while Gemma2 includes 42 layers, each
hhttps://huuginggge http///gggggggc...ooooo wwt.c////mmctaaaaeeeeeeeee
woll///aan//oooee/o / /Mistral-7B--OO.. 1/B3/00//bllb/// /albb//aao///ee/ee json
/ hwwgs/////ggggggccc....ooo co/ mistmmlaiiaaa/a/ammammaaaiaiiaaa
"https://huggingghee co/ /wen///ee///eeeeeeeeeeeeeeee Beeea///wee22//bbbee /B/bb////////////// maann maain/config,ii json
7 hattps://huggingfiaee wogle//geeeee httpe///emmaaam///eaaaa /aan// main/connfig.jon
23

---

Published in Transactions on Machine Learning Research (01/2026)
Table The pruning layer range (closed on the left, open on the right) of pruning algorithms for LLaMA3.1- 1- 8B, Mistral-7B-v0.3, Qwen2-7B, and Gemma2-9B.
LLaMA3.1-8B
Method
Starting Layer Index
End Layer Index
Mag-sp
15
31
Wanda-sp
15
31
LLM-Pruner
14
30
FLAP
0
32
Mistral-7B-v0.33
Method
Starting Layer Index
End Layer Index
Mag-sp
15
31
Wanda-sp
32
FLAP
32
Qwen2-7B
Method
Starting Layer Index
End Layer Index
Mag-sp
12
Wanda-sp
11
FLAP
27
Gemma2-9B
Method
Starting Layer Index
End Layer Index
Mag-sp
2
18
Wanda-sp
30
42
FLAP
3
39
with 8 key/valut attention heads. For LLaMA3.1-8B and Mistral-7B-v0.3, we allocate a KV cache budget of 3/4, while Qwen2-77 7B and Gemma2-9B have a KV cache budget of 6/7. This requires removing 64 key/valut attention heads for ALaMA3.1-8B and Mistral--B-v0.3, 16 for Qwen2-7B, and 48 for Gemma2-9B.
To determine the number of key/ key/value attention heads to prune in each layer, we restrict the range of removable key/ /va/ualleeee attention heads to maintain integer pruning layer counts. For LLaMA3.1-8) and Mistral-7B--O.
2 or 4 heads are removed per layer. Qwen2-7B allows 1 or 2 heads per layer, while Gemma2-99 permits 2, 3,
4, or 6 heads per layer. As a result, the pruning layer counts are configured as follows: for LLaMA3.1 1-8B and Mistral-7B-v0.3, the pruning layer number is 16 or 32; for Qwen2-7B, it is 8 or 16; and for Gemma2-9B, it is 8, 12, 16, or 24. For FLAP, where the pruning ratio can vary across layers, we similarly tune the number of pruned layers to obtain strong perplexity under the same KV cache budget. Specifically, for L-aMA3.1-8B and Mistral-7B--0.33 the number of pruning layers is set to 12, 16, 20, 24, 28, or 32. For Qwen2-7B, the pruning layer counts are 8, 12, 16, 20, 24, or 28. Similarly, for Gemma2-91B, the options are 9, 12, 18, 24, 30, 36, or 42 layers.
For a fixed-length interval with undefined start and end points, we begin with the starting layer index set to 0. In each step, we increment both the starting layer index and the end layer index by 1 until the end layer index reaches the model's highest layer. The results are presented in Table 7 Note that we also integrate FLAP into HARP to prune FFN parameters. However, for our method we do not tune the pruming layer range and instcad always prune the highest layers, which makes the comparison with tuned basclines consorvative.
C Benchmark Details
For our evaluation, we employ the In-evaluation--arrees package (version 0.4.7) developed in (Gao et al. 2024). This framework provides unified interface to commonly used benchmarks and implements task-specific preproccssing and scoring. It is important to note that the aarevalluttion-harneee two accuracy metrics, "acc" and "ace norm", for the ARC-Challenge, OpenBookQA, and PIQA benchmarks. Unless herwiie stated, for these benchmarks, we report the "acc" accuracy results. Table g reports the number of tasks and the number of choices for each discriminative task.
provides
For the GSM8K and TriviaQA benchmarks, this package offers two accuracy metrics: "exact match,strict- In our reporting, use the "exact. match,strict-match accuracy
match" and "exact
24

---

Published in Transactions on Machine Learning Research (01/2026)
Table 8: Dataset Statistics
Metric
WinoGrande
ARC-Challenge
BoolQ
OpenBookQA
PIQA
MMLU
Tasks
1267
1172
3270
500
1838
11973
Choices
2
4
2
4
2
results for these benchmarks. The number of tasks for GSM8K, NaturalQuestions, TriviaQA, MATH-hard, and BBH are 1319, 3610, 17944, 1324, and 6511, respectively.
For long-context generation evaluation, we adopt the LongBench benchmark (Bai et al., 2024 which offers comprehensive suite of tasks designed to assess the ability of language models to handle long sequences across diverse domains. LongBench includes 23 tasks categorized into six groups: (T) Single-Document QA, (II) Multi-Document QA, (III) Long In-Context Learning, (IV) Long Dialogue History Understanding, (V) Code Repository Understanding, and (VI) Long Structured Data Understanding. Each task varies in domain, document length, and complexity, providing a broad challenge spectrum. According to Bai et al. (2024), expert performance ranges from 22% to 89% accuracy, with input lengths spanning from 13k to 167k tokens, and average human solving times belween and 13 minutes. This benchmark enables a rigorous evaluation of model's capacity to reason, retrieve, and generate over extended contexts.
D Layer Selection with Heuristic Metric for Pruning Attention Heads
Hessian-bascd Metric. The Hessian-basee metric uses gradient-basod importance score evaluate the contribution of each self-attention layer to overall model performance. To quantify the importance of a given layer, we define the following:
2
I(1)
Σ xE(g.k,)
a
W5
aw
Here, I()) denotes the importance score of layer I, and y refers to the query, key, and value projection matrices in the self-attention mechanism. W represents the weight matrix of projection s, while ØL is the gradient aw
of the loss function L with respect to the weight element W
This metric is derived from first-order Taylor expansion and approximates the effect of removing a layer by computing the squared sum of the element-wii wise product between weights and their corresponding gradients. Layers with lower importance scores are considered to contribute less to the model's performance.
Similarity-based Metric. The similarity-basee metric evaluates the importance each self-attention layer by measuring the extent to which the layer transforms its input representations. To quantify this transformation, define the following score:
(())) A
$0O 1 --EE
||X
1
Here, S denotes the importance score of layer (0( the input to the self-attention module, and (!) is its output, which can be expressed
Y = x + Attention (LayerNorm (xx)
This metric captures the degree of change introduced by the self-attention layer via cosine similarity. A higher score indicates a larger transformation between input and output, implying that the layer plays a more critical role in representation learning. Conversely, layers with lower scores perform minimal transformations.
In our experiments, we use these heuristic metrics only for layer selection (i.c., decide which layers are more suitable for pruning) and keep the head-level pruning and rescaling strategy fixed as described in the
25

---

Published in Transactions on Machine Learning Research (01/2026)
main text. This separation of concerns allows us to clearly attribute performance gains to the proposed
high-layer attention pruning scheme rather than to particular choice of heuristic.
26


Reviews:
[{"review_id": "IzY1QhgeuV", "reviewer": "Reviewer_KJ8C", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript proposes a training-free structured prunnig algorithm called HARP (High-layger Attention Rescaled Pruning). It removes the attention heads, specifically the query and key parameters in the higher layers. After pruning, rescaling in the residual block is used to rescale the magnitude. The effectiveness is evaluated with LLaMA3.1-8B, Mistral-7B-v0.3, Qwen2-7B, and Gemma2-9B on 27 downstream tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The algorithm is evaluated with four LLMs on many downstream tasks with popular training-free methods.", "audience_yn": "Yes", "audience_explanation": "The topic of LLM pruning is of great empirical interest in today's large model age.", "requested_changes": "1. The writing is not always clear. For example, the sentence \"It is important to note that the official implementation does not support GQA-based LLMs. Therefore, we use a modified implementation for our experiments\" even does not tell the readers it is about FLAP. Please go through a more careful proofreading. \n\n2. The reported performance is promising. However, the authors did not provide any code to reproduce the results. I will not suggest acceptance of a paper which claims SOTA but provides no code. \n\n3. The first proof environment in Page 20 is unprofessionally written. Proof of what?", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JPgmwSmiou", "reviewer": "Reviewer_egY7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces High-layer Attention Rescaled Pruning (HARP), a novel training-free structured pruning algorithm for Large Language Models (LLMs). HARP strategically removes attention heads in higher model layers, arguing they have reduced discriminative capacity due to increasing token similarity. A key feature is an adaptive rescaling parameter $\\alpha$ introduced into the attention residual block to maintain token representation magnitude post-pruning and mitigate performance loss. Experiments across 27 tasks on different LLMs consistently show HARP outperforms existing structured pruning methods, offering significant speed improvements, especially for long-context tasks.\n\nStrength:\n- The core idea of strategically pruning attention heads in the higher layers is a novel and effective design choice for structured pruning. It challenges conventional approaches that prune heads indiscriminately or based solely on per-layer importance metrics.\n- This work demonstrates superior experimental results. The proposed HARP significantly outperforms all compared baselines in generation tasks across all four tested LLMs (LLaMA3.1-8B, Mistral-7B-v0.3, Qwen2-7B, and Gemma2-9B).\n\nWeakness:\nThe determination of the optimal alpha is achieved via a greedy search strategy (grid search, layer-by-layer) evaluating perplexity, which is a heuristic and sub-optimal approach for finding a globally optimal configuration.\nThe notations in the method part are not clear enough. For example, in equation (3), I cannot directly understand what do W_v and W_o refer to. Similar situations exist in other formulas.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Generally, the claims and contributions in the paper is supported by accurate, convincing and clear evidence.", "audience_yn": "Yes", "audience_explanation": "The researchers that focus on efficient LLM reasoning will be interested on the finding of this paper.", "requested_changes": "A table to compare with HARP with other related pruning methods to make related work comparison clearer. \nCompare the proposed greedy search strategy with other possible hyperparameter search strategies. \nExperiment on different LLM size. Currently, the parameter scale of tested LLMs is between 7B and 9B. It is necessary to conduct experiments on both smaller and larger LLMs to verify the robustness of the proposed pruning strategy.", "broader_impact_concerns": "There are not concerns that that would require adding a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4NnneWm299", "reviewer": "Reviewer_tZup", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces High-layer Attention Rescaled Pruning (HARP), a structured pruning approach that enhances LLM inference efficiency by selectively removing query and key parameters in upper layers, a strategy grounded in theoretical insights regarding token over-smoothing and reduced discriminative capacity at depth. To mitigate the representational magnitude shifts caused by bypassing attention blocks, the authors propose an adaptive rescaling mechanism using a scalar parameter optimized via a greedy search. Extensive evaluations across multiple architectures demonstrate that HARP outperforms existing baselines like Wanda and FLAP, particularly in generation tasks sensitive to error propagation, while achieving notable inference speedups.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims presented in this paper are supported by robust and comprehensive evidence. The authors establish a convincing motivation for high-layer attention pruning by effectively synthesizing theoretical analysis, specifically regarding the reduced discriminative capacity of high-layer heads due to over-smoothing and the computational bottlenecks of long-context modeling, with existing literature. The proposed HARP framework is rigorously designed to align with this motivation and introduces a clear solution (adaptive rescaling) to address the specific issue of representational magnitude shifts caused by pruning. The empirical validation is extensive, demonstrating the method's consistent superiority over baselines across a diverse set of LLM architectures (e.g., LLaMA3.1, Mistral, Qwen) and 27 datasets. Furthermore, the inclusion of detailed ablation studies regarding layer selection strategies and efficiency analysis provides a clear confirmation of the algorithm's effectiveness and practical viability.", "audience_yn": "Yes", "audience_explanation": "The strong motivation behind HARP, combined with its algorithmic simplicity and the robust experimental evidence supporting its efficacy, makes this work highly relevant to the TMLR audience. It will be of particular interest to researchers focused on efficient LLM inference and model compression. Moreover, the theoretical insights and empirical findings presented in this study provide a solid foundation that will likely facilitate and inspire subsequent research in the field.", "requested_changes": "While I positively evaluate the strong motivation, the simplicity and effectiveness of the proposed algorithm, and the solid empirical evidence presented in this work, I believe the following adjustments are necessary to meet the acceptance criteria.\n\n**Critical to securing a recommendation for acceptance:**\n\n1.  Cost comparison with baselines: In Section 3.5 (Page 9), the authors mention that the $\\alpha$ search for HARP requires approximately two hours on an NVIDIA H100 GPU. To rigorously assess the efficiency of the proposed method, it is essential to compare this computational cost against the setup or calibration times required for the baseline pruning algorithms (Mag-sp, LLM-Pruner, Wanda-sp, and FLAP). Providing a cost comparison will allow for a clearer evaluation of the trade-off between HARP’s performance gains and its additional search overhead.\n\n**Would strengthen the work:**\n\n2.  Table captions: Please explicitly state the meaning of the bolded text (e.g., \"Bold indicates the best performance\") in the captions of all performance tables (Tables 1, 2, 3, etc.) to ensure the results are immediately interpretable.\n\n3.  Clarification of Figure 3: In Figure 3, the green dashed line appears to represent the relative speed improvement between the original and pruned models. Please expand the figure caption to explicitly describe what this line and the secondary y-axis represent to aid reader comprehension.\n\n4.  Expansion of conclusion: The current Conclusion section is relatively brief. I suggest expanding it to go beyond a simple summary of the algorithm and results. It would be beneficial to discuss the expected impact of the proposed method on real-world tasks, explicitly address any known limitations, and outline potential directions for future work.", "broader_impact_concerns": "I do not see any broader impact concerns that would require a specific ethical statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)