GradPruner: Gradient-guided Layer Pruning Enabling Efficient Fine-Tuning and Inference for LLMs
Authors: Wei Huang
Anda Cheng
Yinggui Wang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Fine-Tuning
Layer Pruning
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Fine-tuning Large Language Models (LLMs) with downstream data is often considered time-consuming and expensive. Structured pruning methods are primarily employed to improve the inference efficiency of pre-trained models. Meanwhile, they often require additional time and memory for training, knowledge distillation, structure search, and other strategies, making efficient model fine-tuning challenging to achieve. To simultaneously enhance the training and inference efficiency of downstream task fine-tuning, we introduce GradPruner, which can prune layers of LLMs guided by gradients in the early stages of fine-tuning. GradPruner uses the cumulative gradients of each parameter during the initial phase of fine-tuning to compute the Initial Gradient Information Accumulation Matrix (IGIA-Matrix) to assess the importance of layers and perform pruning. We sparsify the pruned layers based on the IGIA-Matrix and merge them with the remaining layers. Only elements with the same sign are merged to reduce interference from sign variations. We conducted extensive experiments on two LLMs across eight well-known datasets in downstream tasks. Including medical, financial, and general benchmark tasks. The results demonstrate that GradPruner has achieved a parameter reduction of 40% with only a 0.99% decrease in accuracy. Our code is available at https://anonymous.4open.science/r/LLM-GradPrune-436D.

Published as a conference paper at ICLR 2026
GRADPRUNER: GRADIENT-GUIDED LAYER PRUNING ENABLING EFFICIENT FINE-TUNING AND INFERENCE FOR LLMS
Wei Huang" Ant Group
Anda Cheng"
Yinggui Wang
Beijing, China
(hw1 9970202, andacheng. cad, sylinguuiJ@gmal com
ABSTRACT
Fine-tuning Large Language Models (LLMs) with downstream data is often con- sidered time-consuming and expensive. Structured pruning methods are primarily employed to improve the inference efficiency of pre-trained models. Meanwhile, they often require additional time and memory for training, knowledge distillation, structure search, and other strategies, making efficient model fine- thne-tnnngg challeng- ing to achieve. To simultaneously enhance the training and inference efficiency of downstream task fine-tuning, We introduce GradPruner, which can prune layers of LLMs guided by gradients in the early stages of fine-tuning, GradPruner uses the cumulative gradients of each parameter during the initial phase of fine-tuning to compute the Initial Gradient Information Accumulation Matrix (IGIA-Matri) to assess the importance of layers and perform pruning. We sparsify the pruned layers based on the IGIA-Matrix and merge them with the remaining layers. Only elements with the same sign are merged to reduce interference from sign vari- ations. We conducted extensive experiments on two LLMs across eight down- stream datasets. Including medical, financial, and general benchmark tasks. The results demonstrate that GradPruner has achieved parameter reduction of 40% with only a 0.99% decrease accuracy. Our codes are publicly available
INTRODUCTION
LLMs have currently gained remarkable performance across various tasks Grattatiori et al. (2024) However, when handling more specialized tasks, such medical financial domains, LLMs often exhibit a decline in performance (Zhang et al. 2023). We can fine-tune them on downstream data to enhance their capabilities. Fine-tuning I.LMs domain-specific data typically requires substantial time and is expensive. For instance. Yang et al (Yang et al.) (024a). trained a medical LLM that took approximately 221 hours. Furthermore, LoRA fine-tuning over pretrained LMs reduces training memory but does not improve inference efficiency (Han et al. 2024).
Structured pruning improves inference efficiency by removing parameter blocks. These structured pruning approaches typically involve two steps Muralidharan et al. (2024): Ma et al. (2023). The first step uses calibration data identify important parameters within the model. Since this process generally does not introduce additional training, it relies more the L.M's inherent ability to process the calibration data. However, due to the I.LM's suboptimal performance on domain -oeaiin-peciffic data, this may lead to significant biases. The second step involves training or distilling the pruned model. which requires more time and memory. Some works have also explored structured pruning for efficient LLM training and inference, such as APT Zhao al. (2024) and SAT MATMa et al. (2024). APT can only be applied to LoRA fine-tuning. In SAT, the model structure varies across different training steps, and the final training step restores the model its dense form, meaning it cannot accelerate inference.
"Equal contribution. i Corresponding author. thw.. com/ http:://hh.ttoooocceeeeeeeeeetee

---

Published as a conference paper at ICLR 2026
To ensure the accuracy of downstream tasks while efficiently training and inference with LLMs, we address three key challenges: 1) Developing a method to measure the importance of model parameters tailored to specific downstream data and models without increasing memory training time, 2) Preserving the original model structure as much as possible while maximizing parameter pruning, and 3) Supporting both full fine-tuning and LoRA fine-tuning. In this paper, introduce GradPruner, an efficient fine-tuning approach inspired by the observation that loss decreases sharply in the initial fine-tuning steps, indicating rapid learning of downstream tasks. Because different parameters hold varying levels of importance for downstream tasks, this leads to differences in learning capabilities Zhao al.] (2024). Leveraging this insight, simultaneously save time and memory, we employ LoRA fine-tuning Hu et al. (2022) to compute the accumulation of gradients during the initial phase of training (which is significantly fewer than the total number of training steps) to obtain the Initial Gradient Information Accumulation Matrix (IGIA-Matrix), which is used to assess the importance of each parameter.
We adopted layer-level pruning of the model to address the second and third issues. Through ab- lation studies, we found that pruning 30% of the layers has almost no impact on the accuracy of downstream tasks, but pruning increase the pruning ratios, we introduced a layer merging method. This method involves sparsify- ing the pruned layers using the IGIA-Matrix and then merging them with the remaining layers. To minimize interference from sign conflicts, we only merge elements with the same sign. By employ- ing layer merging, we are able to prune three additional layers while maintaining the accuracy of downstream tasks.
additional layer causes sharp decline in accuracy. To further
We conducted extensive experiments on two LLMs and eight downstream datasets. The results demonstrate that GradPruner can prune 40% of the parameters while ensuring that the accuracy on downstream tasks decreases by only 0.99%. Compared to structured pruning methods, our approach clearly outperforms them. Additionally, the pruned Llama3. 1-8B model achieves better accuracy on downstream tasks than the 1lmma32--3B.
This paper makes the following key contributions: 1) We propose GradPruner, which calculates the IGIA-Matrix using the initial gradients from LoRA fine-tuning to evaluate the importance of each model parameter for downstream tasks. 2) To prune many layers possible, we sparsify the pruned layers based on the IGIA-Matrix and then merge them with the remaining layers, only com- bining elements with the same sign. 3) Extensive experiments show that GradPruner has achieved parameter reduction of 40% with only a 0.99% decrease in downstream tasks accuracy.
2 BACKGROUND AND MOTIVATION
2.1 PROBLEM FORMULATION
Our goal is to enhance the efficiency of fine-tuning and inference for LLMs while ensuring the ac- curacy of downstream tasks. Intuitively, the more parameters that are pruned, the faster the training and inference will be, but the accuracy on downstream tasks will decrease. Conversely, the fewer parameters that are pruned, the better the accuracy on downstream tasks, but the training and infer- ence speed will slow down. However, numerous researchers have pointed out that LLMs contain a significant number of redundant parameters Yadav et al. 2023); y Muralidharan et al. (2024). There- fore, we aim to identify the minimal set of model parameters that can be retained to preserve the accuracy of downstream tasks.
We define the downstream task dataset as D, consisting of samples (I1, vi).... (IN, (IN.UN). yN). The objective of our problem achieve the maximum pruning ratio & while ensuring the minimization of the task loss L. We describe the optimization process as:
arg min
Σ ..NED
(1)
Oagh
D
where 0 represents all the model parameters.
2

---

Published as a conference paper at ICLR 2026
8
sharp decline
Loss valuu
sharp decline
6
4
0 100 200 300 400 500 600 700
0
200
400
600
800
Training steps
Training stops
(b)
Figure I: The loss training while LoRA fine-tuning PubMedQA (a) and PIQA (b) on Llama3. 1- 8B. The loss value showed rapid decrease during the initial training steps.
2.2 MOTIVATION
Many existing pruning methods rely on calibration data, using forward propagation (such interme- diate layer outputs) to assess the importance of various parameters. Such judgments depend heavily on the LLM's ability to process the calibration data. Nevertheless, when dealing with domain- specific tasks, the LLM's limited processing capabilities may introduce biases.
Gradient Sensitivity Analysis (Llama3.1-82)
We found that during experiments, when an LLM is fine-tuned on downstream tasks, the loss sharply drops within the initial 1% of the training steps (as shown in Figure D.. This indicates that the model quickly grasps the knowledge required for the downstream task, and different parameters contribute variably to the learning process. This phenomenon provides us with insights.
1.0- 1%
Accuracy
0.2
NubMudOA 0A
Based on the above observations and reason- ing, it is natural to consider using the ini- tial gradients from LoRA fine-tuning to mea- sure the importance of different parameters. This approach can not only accurately iden- tify parameters that crucial for down- stream tasks but also save time and reduce memory consumption.
0.0
500
1000
1500
2000
2500
Training steps
Figure 2: Gradient Sensitivity Analysis of the IGIA-Matrix on L-ama3.1-8B. We can see that the layer importance measured at such an early stage can accurately reflect the results after the entire training process.
2.3 GRADIENT SENSITIVITY ANALYSIS OF THE IGIA-MATRIX
Existing research has demonstrated that performing full training and subsequently using gradient information to assess the importance of different parameters is reasonable approach Matena & Raffel (2022): y Daheim al. (2023). However, our proposed IGIA-Matrix method requires only the initial 1% of the training steps measure the importance of each layer. To analyze the relationship between the layer importance derived from our method and that obtained after complete training, conducted Gradient Sensitivity Analysis of the IGIA-Matrix.
In our study, we recorded the layer importance at various stages of the training process and used the layer importance obtained after full training as the reference label list. Since need to prune layers, only the indices of the top 20 most important layers were retained the label list. We then compared the top 20 layers identified each stage with this label list. If any layer among the top 20 from given stage was not present in the label list. we considered that layer's 's result as mismatched with the labels, resulting in drop in accuracy. The experimental results of this analysis are presented
3

---

Published as a conference paper at ICLR 2026
Inital LeRA gradient acquisition (model training)
MatAriatri compucation
OutputLoss
Len and Loea A yod
Mergal
Layer
Lora Gand
compune the KAl Mainx of Litear
Layeri12
Gradient Caleu lation and Backward
Add Noma
Add & Noen
Forwar computation
Looug Lony
Tor R Aar
KilA--Maiis
Tayeriei
(
Layer praningg
Malti Head Attertinn
Food Forward
Layer
Layer
Layer HH
Layer H-2
Layer t+3
Layer
Loyari
Duiaset
The framework diagram of Parameter Importance Evaluation and Layer Pruning in Grad- Pruner. The first step involves obtaining gradients through a small amount of LoRA fine-tuning. The second step calculates the IGIA-Matrix based on gradients. the third step. assess the importance of each parameter and each layer based on the IGIA-Matrix and subsequently prune the layers accordingly.
Figure 3:
in Figure 2 From the figure, we can conclude that the layer importance measured such early stage can accurately reflect the results after the entire training process.
3 METHODOLOGY
this section, we provide detailed explanation GradPruner. GradPruner consists three steps: (1) Parameter Importance Measurement Phase. This step focuses identifying the model parameters that are important for downstream tasks. (2) Layer Pruning Phase. Once the importance of different parameters is determined, the second step assesses the importance of various layers and prunes them accordingly. (3) Layer Merging Phase. This step involves merging the pruned layers with the remaining layers.
3.1 PARAMETER IMPORTANCE EVALUATION AND LAYER PRUNING
Motivated by the observations from Figure I we observe that the loss decreases rapidly during the initial training phase as the model quickly adapts to downstream tasks. Recognizing that different parameters hold varying levels of importance, reflected in their gradient update magnitudes. Based on this insight, we measure parameter significance by capturing gradient values in each step of the early LoRA training phase. The overall process of the algorithm is illustrated in Figure
Parameter Importance Evaluation: Formally, we consider a linear layer with weight W.The cor- responding LoRA weights are WA and Wn. We freeze the parameter W and train the model using the downstream dataset D. We define the model to undergo a total T training steps; however, only need to obtain the gradients for the first steps (t << meanning that the model training terminates after steps. After i. rounds of training, we obtain the per-step gradient values for WA and WB. denoted V Ww, [(xx y) and V W, [(., respectivey. Specifically, V W I(x,, y) consists of gradients: {V w. L(z, 1)1, Vw, (1, Vw, L(I, w)+ After obtaining Vw, L(x, y) and Vw, L(x, 3). we need evaluate the importance of each parameter in W for the downstream task. First, we align the matrix dimensions of V W L(x, y) and V Vwa L(x, y) with those of W. We Inspire for LoRA be mergeable with the original parameters after tine-tuning, so we align them with W by performing matrix multiplication on Vw, L(x, y) and Vw L(x, y) to simulate the gradient of W. The formula as follows:
V w L(x, v). sim Vw, L(x, v): Vw, I(,,)); iE (1,...tt
(2)
4

---

Published as a conference paper at ICLR 2026
Sparsification
Unmorged Parmmetors
3
2
2
-2
-1
-2
-1
-2
0
W;
Spursification
Merged W/= Wy+ Wy=s" W/+2
3
3
2
5
2
-7
-1
0
9
8
5
8
W/+2
Far
Merged
Figure 4: The framework diagram of Layer Merging in GradPruner. The first step is sparsify the pruned modules using the IGIA-Matrix as the criterion. The second step is to merge the pruned layer with the preceding retained layer based on their signs.
Where, sim represents the simulated gradient.
Using Equation (2), have obtained the gradients of the W at each step during the first t training steps. To comprehensively evaluate the relationship between the gradients from the initial / training steps and parameter importance, calculate the IGIA-Matrix using the following formula.
Fw ((,
We obtain the IGIA-Matrix Fw for W through the steps above. Similarly, all linear layers in LLM can acquire their corresponding IGIA-Matrix using this approach.
Layer Pruning:
We choose to prune the model's layers based the following considera-
tions: 1) Preservation of the Model's Overall Structure: To ensure that the overall architecture of the model remains unchanged possible. 2) Impact Downstream Task Accuracy: Our exper- iments demonstrate that pruning the parameters of important layers adversely affects the accuracy of downstream tasks (see ablation study). We sum the IGIA-Matrix of all linear layers within each layer of the LLM to obtain the importance score of the current layer, as shown in the following:
Layer,
Fwa
k= 1=1
Where Layer, represents the importance score of the j-th layer, M denotes the number of linear layers within the j-th layer, and H signifies the number parameters in each linear layer. Based on the above formula. we can obtain the importance score for each layer in the model. The pseudocode for layer pruning can be found in Appendix
3.2 LAYER MERGING
When only performing layer pruning, our ablation study indicates that pruning more than 30% of the layers results in a significant decrease in accuracy. To prune as many layers of the model as possible, we do not directly discard the pruned layers but instead merge them with the remaining layers. The overall process of the algorithm is illustrated in Figure 4
To merge multiple linear layers [ from different layers. We designate W, as the linear layer be retained and (W,.... W,) perform the merging.
the linear layers to be pruned. Our method follows two steps to
Sparsification aims to reduce interference between pruned and retained layers while preserving downstream task accuracy. Research shows that keeping only subset of model parameters can maintain this accuracy. The key challenge is determining which parameters are most important. To tackle this, of each parameter for downstream tasks. The IGIA-Matrix corresponding to (...... W,, } are (Fwa...., Fw..). ). We retain the top-p% of parameters based the magnitude of the IGIA-Matrix
I) Sparsification:
the IGIA-Matri as metric to evaluate the importance
5

---

Published as a conference paper at ICLR 2026
Method
PubMcdQA
MedMCQA
BillSum
FinGPT
HellaSwag
WinoGrande
ARC
PIQA
Avg.
Llama3. T-8B
Full Fine- Fine-Iuning (FFT)
Dense Model
0.593
0.572
0.696
0.369
0.943
0.868
0.865
0.867
0.784
LLMPruner
0.560
0.521
0.641
0.818
0.898
0.810
0.817
0.805
0.734
Laco
0.356
0.514
0.649
0.814
0.901
0.818
0.244
0.809
0.736
MINITRON
0.555
0.527
0.640
0.808
0.898
0.814
0.826
0.803
0.734
SAT
0.567
0.552
0.656
0.832
0.908
0.835
0.822
0.833
0.750
FT FTLLama3.2)
0.591
0.568
0.088
0.367
0.932
0.836
0.874
0.861
0.777
)
0.591
0,586
0.687
0.867
0.939
0.861
0.849
0.876
0.782
Parameter- Eficcet Fine Tuming (LoRA)
Dense Model
0.607
0.633
0.677
0.831
0.959
0.821
0,931
0.893
0.794
LLMPruner
0.538
0.555
0.636
0.766
0.883
0.790
0.870
0.826
0.733
Laco
0.353
0.356
0.631
0.775
0.907
0.792
0.875
0.837
0.740
MINTTRON
0.546
0.557
0.631
0.765
0.899
0.781
0.873
0.822
0.734
SAT
0.550
0.596
0.621
0.779
0.909
0.780
0.882
0.844
0.745
APT
0.561
0.607
0.045
0.802
0.922
0.790
0.891
0.859
0.759
FT(Llama3.2)
0.392
0.619
0.662
0.825
0.926
0.808
0.902
0.839
0.774
Grad (our))
0.594
0.637
0.659
0.817
0.954
0.812
0.923
0.891
0.786
Mistrul 7B
Full Fine Tuning (PFT)
Model
0.591
0.862
0.841
0.903
LLMPruner
0.547
0.512
0.639
0.817
0.810
0.815
0.867
0.839
0.730
Laco
0.552
0.525
0.641
0.819
0.825
0.828
0.864
0.857
0.738
MINTTRON
0.544
0.511
0.650
0.233
0.822
0.80
0.851
0.846
0.734
SAT
0.561
0.543
0.823
0.834
0.836
0.870
GradPruner(ours)
0.586
0.568
0.670
0,846
0.840
0.860
0.895
0.897
0.770
Porameter Ceeeee---fffiiee Efficient Fine Tuming (LoRA)
Dense Model
0.607
0.565
0.681
0.853
0.963
0.846
0.909
0.896
0.790
0.827
Laco
0.541
0.499
0.643
0.804
0.904
0.812
0.851
0.846
0.737
MINTTRON
0.533
0.504
0.633
0.802
0.900
0.805
0,840
0.835
0.731
SAT
0.545
0.524
0.629
0.807
0.931
0.815
0.857
0.843
0.743
APT
0.555
0.533
0.631
0.815
0.866
0.854
0.750
GradPruner(ours)
0.588
0.565
0.659
0,840
0.963
0.832
0.893
0.896
0.780
The main results of our experiments under 40% sparsity pruning. "Avg." refers to the average score between eight datasets. "Dense Model" represents the results of the unpruned LLMs after fine-tuning "FT" represents fine-tuning Llama3.2-3 B.BB... Since APT can only be applied to LoRA fine-tuning, only report the results of APT in the context of LoRA fine-tuning.
Table I:
and set the remaining parameters to zero, thereby creating { sparsify Wi, is a more critical linear layer, and sparsifying it could adversely affect the accuracy of downstream tasks.
Wr,, Notably, we do not need
2) Symbol-based merging. given parameter may have positive values for some layers and nega- tive values for others. In both cases, simply merging these values can lead to interference, thereby shrinking value that parameter the merged layer. Therefore, we using the parameter signs with signs matching the total sign
of W1 as the total sign. Only the parameters in ( Wr), merged with W1. For more details, please refer to Equation 5.
(Wy)a (Wy)a+ + (W (Wy)a + (We+a)s
if (1))al (1941)ed (754n)er Kf (1y)a == (144)a = (75-n)at
M (Wy)
(5)
if (1)aa== (1s41) == (7o-n)
Where (W,). Wy denotes the I-th parameter of the k-th linear layer in the j-th layer, and represents the sign matrix corresponding the parameter matrix of the linear layer. Similarly, each linear layer in the pruned layer can be merged with the corresponding linear layer of the retained layer following the above process. The pruned layer is only merged with the preceding retained layer.
4 EXPERIMENTAL SETTINGS
4.1 DATASETS AND SETTING
To comprehensively evaluate the effectiveness GradPruner in the vertical domain, we conducted extensive experiments using two widely adopted LLMS and eight downstream datasets. For the

---

Published as a conference paper at ICLR 2026
Method
Train Time
Train Mem(U)
Inf Time(
Inf Meni 4)
Full "ine-Tuning (FFT)
Dense Model
100.0%
100.0%
100.0%
100.0%
SAT
75.5%
79.4%
98.9%
103.6%
Laco
73.8%
64.4%
59.7%
61.3%
MPruner
78.3%
284.4%
67.4%
65.3%
GradPruner
62.4%
65.8%
61.5%
60.9%
Poruneter Efficient Thuning (LoRA)
Dense Model
100.0%
100.0%
100.0%
100.0%
APT 158.0%
65.9%
87,5%
62.4%
SAT
74.8%
82.9%
102. 3%2
99.4%
Laco 71.5%
65.7%
61.4%
LLMPruner
81.8%
266.4%
69.7%
64.6%
GradPriner
66.9%
64.4%
61.8%
60.7%
Table Comparison of GradPruner with other methods terms of training and inference time, as well as GPU memory usage. Our measurement method is based on APT's paper. All efficiency met- rics are normalized Dense Model, that is, the relative time or memory overhead compared to the Dense Model. J denotes smaller is better. We do not compare to distillation method (MINITRON) because the training cost distillation is too large.
LLMs, we selected Llama3.1 1-8B Dubey et al. (2024) and Mistral-7B-v0.33 Jiang et al. (2023). Re- Jiang garding the downstream datasets, included four specialized domain datasets: PubMedQA Jin et al. 2019) and MedMCQA Pal et al. (2022), which focus on medical tasks, and BillSum Ko-
milova & Eidelman 2019) and finggt-sentiment-rrii
(FinGPT)Yang et al. (2023). which pertain to financial tasks. Additionally, we incorporated four general-domain reasoning benchmark datasets:
HellaSwag, WinoGrande, ARC, and PIQA Yao al. (2024).
Due to space constraints in the main paper. detailed description of the datasets and the setup of our experiment is presented in the appendix C
Our evaluation metrics are formulated based on the characteristies of the dataset. The QA data in the medical and financial fields, we adopt the method of evaluating the similarity between the output from LLMs and the standard label. We evaluate using 1/2 BerttSore (Zhang* et al., 2020) ROUGE-L (Lin 2004)). As for other reasoning benchmarks, we directly calculate the Accuracy score.
4.2 BASELINES
We conducted extensive comparisons between GradPruner and six baselines. Categorizing the base- lines into three groups. The first group consists of structured pruning methods focused on addressing how to efficiently train and infer when LLMs are adapted to downstream tasks. APT Zhao et al. 2024),, which dynamically adds salient tuning parameters for fast and accurate convergence while discarding unimportant parameters to improve efficiency; SAT Ma et al. (2024). which extends existing neuron importance evaluation metrics and introduces a ladder omission rate scheduler.
The second group of structured pruning methods focuses on accelerating inference. 1) LLM- Mr et al. (2023) adopts structural pruning that selectively removes non-critical coupled struc- tures based on gradient information. 2) LaCo Yang et al. (2024b), a concise layer-wise structured pruner called Layer Collapse, in which rear model layers collapse into prior layer; 3) MINI- TRON Muralidharan et al. (2024). which combines depth, width, attention, and MLP pruning with knowledge distillation- iatillatioo-based retraining. We fine-tuned the models pruned by LLMPruner, LaCo, and MINITRON on downstream data for comparison. Since MINITRON requires pre-training data for knowledge distillation, which could not access, used the Alpaca dataset as a substitute.
The third group of baselines consists of smaller parameter models. We used the fine-tuning of Llama3.2-3 BB Dubey et al. (2024) downstream tasks as comparison method.
is noted that, to ensure a fair comparison, the pruning baselines also utilize the downstream datasets calibration data.

---

Published as a conference paper at ICLR 2026
Number
Method
PuhModQA
McdMCQA
BillSum
FinGPT
HellaSwag
WinoGrande
ARC
PIQA
Avg.
Llama3. 1-8B
Full Fine- Tuning (FFT)
Dense Model
0.593
0.572
0.696
0.869
0.43
0.868
0.865
0.867
0.784
GindPrunner
0.591
0.586
0.687
0.867
0339
0.861
0.849
0.876
0.782
w/o Merging
0,560
0.535
0.663
0.816
0.893
0.803
0,830
0.826
0.741
2
GradPrrnner
0.593
0.861
0.786
w/o Merging
0.581
0.566
0.677
0.841
0.930
0.840
0.846
0.843
0.767
GraiPPruner 0.590
0.863
0.856
0.868
0.785
w/o Merging 0.585
0.580
0.688
0.855
0.936
0.852
0.853
0.851
0.775
Table 3: Experimental results on the impact of the number of merged layers on downstream task accuracy. w/o Merging indicates that no layer merging. The experimental results of LoRA fine- tuning and Mistral-7B are presented in the appendix D
5
MAIN RESULTS
Table M evaluates different pruning methods and fine-tuning approaches across multiple datasets. The results show that GradPruner consistently outperforms other methods, achieving the best per- formance on all datasets. To summarize the findings, we compute the average scores of each pruner, revealing that GradPruner incurs only 0.99% average performance drop comparing with the dense model. Notably, our pruned models outperform directly fine-tuned versions of Llama3.2-3BB demon- strating competitiveness with advanced pre-trained LLMs. GradPruner exhibits particularly strong gains, improving accuracy by approximately 5 percentage points on average for LLMPruner, Laco, and MINITRON, while also surpassing SAT and APT in accuracy.
In addition to accuracy, Table Zcompares GradPruner with other methods in terms of time and GPU memory usage during training and inference. From Table 2 GradPruner achieves approximately 36% reductions both training time and memory usage, along with 39% savings in inference time and memory, performing comparably to Laco. In contrast, the APT method shows significantly higher training time due to its reliance on knowledge distillation, while the SAT method exhibits similar inference time and memory usage the dense model, as it gradually restores pruned param- eters during training.
ABLATION STUDY AND ANALYSIS
We conducted five ablation experiments validate the design decisions and effectiveness of the proposed method.
Perform Only the Layer Pruning in GradPruner: Figure presents the results of using only the layer pruning strategy. For 1lamm33.-88. pruning up to 10 layers has minimal impact on downstream task accuracy. However, beyond 10 layers, accuracy begins decline significantly, critical layers are removed. These findings suggest that while layer pruning alone is effective to certain extent, it becomes insufficient for aggressive pruning. The experimental results of Mistral-7B can be found in the appendix D
increasingly
The Impact of the Number of Merged Layers on Accuracy: Table demonstrates the impact on downstream task accuracy after incorporating the layer merging algorithm on top of layer pruning. From the table, we can observe that for both models, whether fine-tuned using FFT or LoRA, layer merging significantly improves accuracy. Specifically, for Llama3. merging to 3 layers on top of pruning 10 layers results substantial accuracy improvement, nearly matching the dense model accuracy. Overall, layer merging ensures the accuracy of downstream tasks while further increasing the pruning rate.
The Impact of Different Pruning Ratios:
To investigate the impact of different sparsity rates on downstream task accuracy, we conducted relevant experiments. Specifically, we set the sparsity rates to 50%, 60%, 70%, 80%, and 90% to observe their effects task performance. The experimental results are shown in Figure 6 By analyzing the data in the figure, we found that both excessively high and low sparsity rates have varying degrees of impact task accuracy. When the sparsity rate
8

---

Published as a conference paper at ICLR 2026
Downstream Performance (Llama3.1-88
The Impact of Different Pruning Ratios
0.79
0.7
0.78
0.78
AAu.
AccurocytArg
0.77
0.77
0.76
Deseline/F7)
0.75
Herg ngifFTi
0.75
Luma2 1 3-555.2AA
Lama2 2 --157771
Mstrul78::+77
Mitrall 7B1
10 The number of pruned layers
6
0.74
The parsiyy ratio (%)
Figure 5: Experimental results of perform- ing Only the layer pruning in GradPruner. We report the average accuracy across eight datasets.
Figure 6: Experimental results of different pruning ratios GradPruner. We report the average accuracy across eight datasets.
is too high, too many important parame- ters pruned; conversely, when the spar- sity rate too low, too many redundant pa- rameters retained, leading increased interference among parameters during layer merging. Based the ablation study results, we selected an 80% sparsity rate as the opti- mal setting.
0,78
gited Aeeigging
Accuracy(Aug)
0.78
0:78
Replacing Symbol-Based Merging with Weighted Averaging: GradPruner, during layer merging, employs a sign-based merg- ing strategy on top of sparsification. To demonstrate its superiority, sign-based merging with a weight-averaging method for comparison. The results are shown in Figure 7
Uhma3 Lamms LoRs Mieril Meril LB Method
replaced the
Figure 7: Experimental results of replacing symbol-based merging with weighted aver- aging in GradPruner. We report the average accuracy across eight datasets.
is evident that sign-based merging significantly outperforms weight averaging. This indicates that merging only parameters with matching signs effectively reduce interference among different parameters, thereby enhancing the accuracy of downstream tasks.
Replacing Layer Pruning with Kernel Pruning: Readers may wonder why GradPruner focuses on pruning layers rather than adopting finer-grained approach, such as pruning kernels (rows or columns). Due to space constraints in the main paper, have included this portion of the experi- mental results in the appendix D
RELATED WORK
Structured Model Pruning refers to techniques for improving model efficiency by sparsification or parameter removal. While some considerations for pruning are from the perspective of hard- ware (Xia et al.] 2023). Generally, considerations are made from the depth width of the LLMs (Ling et al. 2024). During the pruning process, the parameters the target part are typi- cally directly deleted without considering the reliability of the parameter-judgmentt process (Men) et al. 2025). and the performance of the model often suffers significant losses. As a result, some works consider merging parameters into more important layers (Liu et al. 2024 Yang et al.) 2024b). However, these works always consider pruning in depth and width respectively. NASH constructs a narrow encoder and shallow decoder for T5 models (Ko al. 2023).
He et al.
2025 investigate

---

Published as a conference paper at ICLR 2026
redundancy across different modules within Transformers, including Layers, MLP, and Attention layers, none of them consider jointly pruning by a combination of multiple importance scores.
Sparse Training is training approach that makes model parameters sparsely distributed during training (Graesser et al. 2022). Frankle & Carbin (2019); : [vcii et al. (2019) discovered that training sparse networks from random initialization is difficult compared to dense neural networks. Despite this, dynamic sparse training (Liu et al.] [022], and one-shot pruning (Tanaka et al.] 2020) were proposed improve sparse training. Sparse Training needs change the structure of the model during initialization, so it is hard to adapt to downstream tasks for well-trained LLMs.
8 CONCLUSION
In this study, we propose GradPruner to address the challenges of time-consuming fine-tuning and high GPU memory usage in LLMs. We compute the initial training gradients to obtain the IGIA- Matrix, which is used to evaluate the importance of different layers. To further increase the number of pruned layers, we introduce a layer merging technique, which includes sparsification using the IGIA-Matrix and resolving sign interference issues. We conducted experiments various down- stream datasets and models. The results demonstrate that GradPruner can maintain nearly the same accuracy while reducing the number of parameters by 40%.
REFERENCES
Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference Artificial
Intelligence, 34(05):7432-7439, Apr. 2020. doi: (0.16099aaiivv34ii00 v34.0556639. URL. https::/
jjs. wwaiaaa............. , org /wargg/indexr www.aaix.g/inndxx.pppip php/AAAT art www.wwew/wweeeec.. 6239
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1803.0/8000.05 080305 2018.
Nico Daheim, Thomas Möllenhoff, Edoardo Maria Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. Model merging by uncertainty-based gradient matching. arXiv preprint arXiv:2310. /2808, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. In ICML 20/9 Workshop on Identifying and Understanding Deep Learning net/ www.rrmiddssyyypatee.e
Phenomena, 2019. URL
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2019. URL https://
openreview net/ wwweteeetootieeee
Laura Graesser, Utku Evei, Erich Elsen, and Pablo Samuel Castro. The state of sparse training in deep reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Confer- ence on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 7766-7792. PMLR, 17-23 Jul 2022. URL htttps://proceeeeeee ngs. ml prwss/vv6sl... graesser22a.1 html
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 2024.
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. /4608, 2024.
10

---

Published as a conference paper at ICLR 2026
Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li. What matters in transformers? not all attention htt.tppp//ooeneeiiw.ntttt/eeeeee
is needed, 2025.
[kdx
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):, 2022.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
06825
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2567-2577, 1077-257,,015 2019.
Hyo Seo Kim, Dongyoon Han, and Junsuk Choe. Negmerge: Sign-consensual weight merging for machine unlearning. In International Conference on Machine Learning. 2025.
Jongwoo Ko, Seungjoon Park, Yujin Kim, Sumyeong Ahn, Du-Seong Chang, Euijai Ahn, and Se- Young Yun. NASH: A simple unified framework of structured pruning for accelerating encoder- decoder language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 6076-6099 Singapore, December 2023. Association for Computational Linguisties. doi: 10. 10.18633/112023 wtt.://2....nnngseeeeee findings-emnlp.404 URL /tatts://accanett
ogy. 202..2223... find merdinns-emmt
404/
Anastassia Kornilova and Vladimir Eidelman. BillSum: A corpus for automatic summarization of US legislation. In Lu Wang. Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu (eds.), Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp. 48-56, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18655/11DD19-5000.
URL
54
Chin-Yew Lin. ROUGE: package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. /t..tttsss//ccctttoooooogg
WO4 -0111/13331113
Gui Ling, Ziyang Wang, Yuliang and Qingwen Liu. SlimGPT: Layer-wise structured prun- ing for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Deyuan Liu, Zhanyue Qin, Hairu Wang, Zhao Yang, Zecheng Wang, Fangying Rong, Qingbin Liu, Yanchao Hao, Bo Li, Xi Chen, Cunhang Fan, Zhao Lv. Dianhui Chu, Zhiying Tu, and Dianbo Sui. Pruming via merging: Compressing LLMs via manifold alignment based layer merging. In Yaser Al- Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1781 11217-7829, 7829, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 20118653/112222.emml main. 987. URL htttp:://aclantthoorry... www.:olloyyooo/g/////oooo
emn 1p main. 987/
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense peraatteeettni in-time pereraaraettiizotooonn in sparse training. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6989-7000 PMLR, 18-24 Jul 2021. URL htt.itttps///rrodedinggg ml.
http://nn.........es//r///3/1/// www.ppesssvv399 httm
Da Ma. Lu Chen, Pengyu Wang, Hongshen Xu, Hanqi Li, Liangtai Sun, Su Zhu, Shuai Fan, and Kai Yu. Sparsity- training for large language models. In Findings of the Association for Computational Linguistics ACL 2024, pp. 14696-14709 2024.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21720, 2023.

---

Published as a conference paper at ICLR 2026
Michael S Matena and Colin A Raffel. Merging models with fisher- weighte averaging. Advances in Neural Information Processing Systems, 35:17703- 2022.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and weipeng chen. ShortGPT: Layers in large language models are more redundant than you expect, 2025.
wtwp:erwwwwwwwweeeeeeeeeeeeeeeee d= 35mcG
Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Com- pact language models via pruning and knowledge distillation. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neu- ral Information Processing Systems, volume 37, pp. 41076-41100 102. Curran Associates, Inc., nwwrsssss.eeuuiss.pe.. cctps/cccppppppeeee. files/papee//022//// pdf
2024. URL E2222222225530000000000000000220
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann (eds.). Proceedings of the Con- ference on Health, Inference, and Learning, volume 174 of Proceeddnggs of Machine Learning Re- search, pp. 248-260. PMLR, 07-08 Apr 2022. v174 #174/pal22a.htm html
http::/ppooooeecce ngs. ml r.rssss/
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020.
Curran Associates Inc. ISBN 9781713829546.
Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafci Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. Flash-llm: Enabling cost-effective and highly-efficient large gen- erative model inference with unstructured sparsity. Proc. VLDB Endow. 17(2):211-22) October 2023. ISSN 2150-8097, doi: 10. 10.477883622222.326333 URL https://doi.crgg
362630
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Re- solving interference when merging models. Advances in Neural Information Processing Systems, 36:7093-7115, 2023.
Hongyang Yang, Xiao Yang Liu, and Christina Dan Wang. Fingpt: Open-source financial large language models. FinLLM Symposium IJCAI 2023, 2023.
Songhua Yang, Hanjie Zhao, Senbin Zhu, Guangyu Zhou, Hongfei Xu, Yuxiang Jía, and Hongying Zan. Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp- 19368-19376, 19376, 2024a.
Yifei Yang, Zouying Cao, and Hai Zhao. LaCo: Large language model pruning via layer col- lapse. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 6401-6417, Miami, Florida, USA, November 2024b. Association for Computational Linguisties. doi: 10.1 18653/11/2024 htt.5//////2222...nnnng-m-mmm findings-emnlp.372 URL UtL.rcpsss//aaaattooooo http://aaaaanttoooo.oeoe wtt.:/gggooro/oooooooooo 2024 findings eeaiinggs-ennnie 1p. 372/
Kai Yao, Zhaorui Tan, Tiandi Ye, Lichun Li, Yuan Zhao, Wenyan Liu, Wei Wang. and Jianke Zhu. Scaleot: sainyyuttitty-acllbbie offsite with dynamic layerreplac and selective rank com- pression, 2024. URL httpttttss//xxi......// org abs 2412.442 09812
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluis Märquez (eds.). Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 18653/v1/P19- 1472. URL https: / anthol og.. wttpchoooooy...ro/11111111
12

---

Published as a conference paper at ICLR 2026
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey, arXiv preprint arXiv:2308.10792, 2023.
Tianyi Zhang*, Varsha Kishore*, Felix Wu* Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. BtLtttt///p/ppeeeeee/ http://www.......oooooooooceeeaa
Bowen Zhao, Hannaneh Hajishirzi, and Qingqing Cao. Apt: adaptive pruning and tuning pretrained language models for efficient training and inference. In Proceedings of the 41 International Conference on Machine Learning, pp. 60812-60831, 2024.
13

---

Published as a conference paper at ICLR 2026
A PSEUDOCODE FOR LAYER PRUNING
Algorithm Layer Pruning
Dataset D, total training steps T, initial training steps t (with << 7), model M, linear layer W within M, the corresponding LoRA weights Wa and WB n for W, the number of pruned layers N, and the total number of layers K
Input:
Output: Pruned Model Mprone
1: Initialize the weights of WA and WB-
2: E> Step I: Obtain the gradient corresponding W at every training step during training.
3: for i in {I.....} do
4: Vw,L(a, y)s. W, L(x. y) + Training-and -rttainnngggaddinnttL((Z,,, # Obtaining
Gradients of Wa and WB
5: Vw Lw.La, v): Vw, [xx,, y); : Vw, Ix,g))
6: end for
7: En Step 2: Calculate the IGIA-Matrix corresponding to W.
8: Fw = Ei=1 (Vw L(x, ))
9: D. Step 3: Obtain each layer's importance score based the IGIA-Matrix.
10: for jrrjnn in 1....,K do
11: Layer, = ΣM k= 21=1 FW
end for
13: & Step 4: Prune model M based on the layer importance scores.
Mprune +- Prune(M, Layer,, N) # Pruned layer
15:
return M pprane
B THEORETICAL ANALYSIS
B.1 THRORY OF EARLY GRADIENT ACCUMULATION
During the early stages of fine-tuning, the model rapidly shift from its pre-trained state toward configuration better suited for the downstream task. The accumulated gradient information during this phase capture the most informative weight updates, reflecting the critical sub-networks that con- tribute significantly task-specific learning. This phenomenon resonates with the foundation of the Lottery Ticket Hypothesis (LTH; Frankle Carbin 2019). which posits that a randomly initialized, dense neural network contains a subnetwork referred to "winning ticket" that, when trained in isolation, can achieve performance comparable to the original network. LTH-style pruning typi- cally involves training the network for few steps and then identifying important weights based their magnitudes. Notably, our proposed method can be interpreted as gradient-based generaliza- tion of this idea. The magnitude of accumulated gradient information over initial fine-tuning steps serves as proxy for parameter sensitivity to task-specific signals. This information aligns closely with the most salient features in the loss landscape, and pruning along the less sensitive directions preserves the core learning capacity of the model. Therefore, our gradient-based early pruning method is consistent with LTH's principle of identifying winning subnetworks through short peri- ods of training. This connection supports the effectiveness of our approach in capturing long-term parameter importance through early gradient accumulation.
B.2 THEORY OF MERGING IN GRADPRUNER
In principle, GradPruner theoretically equal to the isotropic merging method with adaptive im- portance weights. Furthermore, this adaptive importance weighting scheme can also be similarly applied to Fisher merging. We will elaborate on these two parts in detail below.
Isotropic merging with adaptive importance weights. To merge M layers, isotropic merging with per-layer weights (Matena & Raffel 2022) approximates the posterior distribution each layer using an isotropic Gaussian whose mean is set to the layer's parameters. This approach introduces
14

---

Published as a conference paper at ICLR 2026
layer-specific scalar hyperparameters X; for i (1,...,.M which can be formally expressed as M
0- = arg max
X logp(0 0,,1).
where p(0 0,, denntes the probability density function of the isotropic Gaussian posterior, and the hyperparameters satisfy X 2 Oand Σi= X = 1. These hyperparameters govern the relative importance assigned to each layer during the merging process. For instance, if all layers assumed to contribute equally, one may set X; = for all i. However, this approach suffers from two primary limitations: (1) How should the importance weights be adaptively determined when the layers exhibit differing levels of importance? (2) When individual weights within the same layer possess varying degrees of importance. how can adaptive weight-specific coefficients be assigned?
GradPruner introduces an adaptive weighting scheme to solve the above challenges. Specifically, consider two layers be merged: let W, denote the base layer to be retained, and W,,, the layer to be merged. To assign appropriate weights Xand X to the j-th parameter entries W/and W GradPruner can be formulated as the following adaptive weight assignment function:
(0.5,0.3 0.5), if (Fw..) T and Sign(W..)) Sign (1.0,0.0), otherwise.
(6)
where Tdenotes pruning threshold. This formulation implies that if the squared gradient infor- mation of a weight in the merged layer exceeds the threshold T and its sign aligns with that of the corresponding weight in the retained layer, both weights deemed equally important and are signed equal coefficients. Otherwise, the weight from the merged layer is considered negligible, and only the retained weight is preserved. GradPruner's importance criterion simultaneously leverages (1) magnitude- - quantified by the squared gradient information - and (2) directional consistency captured by the agreement in sign between the two weights as complementary indicators of parameter significance. This design integrates insights from prior work: methods such as Matena & Raffel 2022); ; Daheim al. (2023) emphasize the utility of gradient magnitude in assessing pa- rameter importance, while approaches like Kim et al. (2025); Yadav et al. (2023) demonstrate the efficacy of sign consistency as relevance signal. GradPruner thus unifies both perspectives into a single adaptive framework. Consequently, the merged weight under isotropic merging with adaptive importance weights is given by 0J X + Nr
Integrating GradPruner's Adaptive Weights with Fisher Merging. As outlined above, Grad- Pruner fundamentally constitutes an adaptive strategy to determine importance weights. In this work, we apply this strategy to isotropic merging. Moreover, demonstrate that GradPruner's adaptive weighting mechanism can be seamlessly extended to Fisher Merging (FM; Matena & Raffel 2022p. Similar to isotropic merging, FM also formulates the merging objective 0
arg max; EM X, log 6;, but differs in that it employs the Laplace approximation to the posterior p. yielding a Gaussian approximation (0, H ") where H the Hessian matrix. To ren- der computation tractable, FM approximates the Hessian using the diagonal of the Fisher informa-
tion matrix F. The resulting closed-form expression for the merged parameter 0 >M A.F/ D., X.F! When integrating GradPruner with FM, the adaptive weighting scheme modifies only the assignment of Ar leaving the computation of the Fisher matrix unchanged. Therefore, in the case of merging two layers, the final merged weight becomes 6² determined via GradPruner described above.
X. a + Fh where (Ni,, Jare adaptively F2 Ph.
C SUPPLEMENTARY EXPERIMENTAL. SETUP
C.I DATASET DESCRIPTION
1) PubMedQA Jin et al. 2019) consists of 19717 scientific publications from the PubMed database of diabetes classified into one of three classes, which is in the medical domain.
2) MedMCQA Pal et al. (2022) is a large-scale, Question Answering (QA) dataset designed to address real-wor world medical questions. We randomly selected 40,000 samples the training set.
15

---

Published as a conference paper at ICLR 2026
Number
Method
PuhModQA
McdMCQA
BillSum
FinGPT
HellaSwag
WinoGrande
ARC
PIQA
Avg.
Llama3.11- 1-88
Porameter Efficient Fine Tuning (LoRA)
Dense Model
0.607
0.633
0.677
0831
0959
0.821
0.931
0.893
0.794
GindPruner
0.594
0.637
0.639
0817
0354
0.812
0.923
0.891
0.786
w/o Mereing
0.576
0.599
0.646
0.787
0.902
0.776
0.885
0.857
0.755
2
GradPPrnner
0.597
0.886
0.789
w/o Merging
0.587
0.601
0.656
0.808
0.931
0.802
0.899
0.858
0.768
GraiPPruner
0.885
0.792
w/o Merging
0.593
0.616
0.665
0.816
0.944
0.811
0.919
0.877
0.779
Mistral 7B
Full Fine- Tuning (PFT)
Model
0.591
0.583
0.684
0.862
0.841
0,878
0.905
0.903
0.781
GradPrrnner 0.586
0.568
0.670
0.846
0.840
0.860
0.895
0.897
0.770
2
Merping
0.863
0.740
GradPruner
0,585
0.578
0.685
0.800
0.840
0.872
0.902
0.897
0.777
Merging
0,579
0.826
0.853
0.877
0.878
0.760
Porumeter- Elficient Fine Tuning (oRRA)
Dense Model
0.607
0.565
0.68
0.853
0.963
0.846
0.909
0.896
0.790
GradPrrnner
0,388
0.840
0.963
0.832
0.893
0.896
0.780
w/o Merging
0.561
0.540
0.632
0.827
0.911
0.799
0.858
0.860
0.749
GradPruner
0.606
0.960
0.846
0.789
w/o Merging
0.590
0.544
0.657
0.826
0.933
0.819
0.875
0.877
0.765
Table 4: Experimental results on the impact of the number of merged layers on downstream task accuracy (LoRA fine-tuning).
3) BillSum (Kornlova & Eidelman 2019) is the first dataset for summarization of US Congressional and California state bills, which is the financial domain.
4) fingpt- seeet-sentiment-traie Yang et al. 2023 is a financial sentiment analysis question-answerin dataset containing 76,000 samples. We randomly selected 40,000 samples as the training set.
5) HellaSwag (Zellers et al. 2019) is a challenging dataset for evaluating commonsense NLI that especially hard for state-of state-of-the-art models, though its questions trivial for humans (,95% accuracy).
6) WinoGrande (Sakaguchi al. 2021 consists of 10.2k training samples for cloze tests, measuring performance with accuracy.
7) ARC (ARC-Easy) (Clark al. 2018 is multiple-choice question-answering benchmark de- signed to test the model's ability to reason about scientific knowledge.
8) PIQA (Bisk al. 2020 is physical commonsense reasoning dataset that is designed to the model's ability to build, craft, or manipulate objects using everyday physical knowledge.
C.2 TRAINING DETAILS
There two ways of fine-tuning One is Full Fine-Tuning (FFT). The other is Parameter-Efficien Tuning and we adopt the LoRA method. During training, we set the learning rate to le-5 and the batch size 64. Each dataset trained for epochs. The AdamW optimizer
used for fine- tuning. We employed SWIFT as the training platform and vLLM for inference. We set the sparsity ratio during merging 80% and pruned 13 layers (approximately 40% of the total parameters), three of which applied the layer merging technique.
D SUPPLEMENTARY ABLATION STUDY
Perform Only the Layer Pruning in GradPruner: Figure g presents the results of using only the layer pruning strategy. For Mistral-7B, pruning up 11 layers has minimal impact downstream task accuracy. However, beyond 11 layers, accuracy begins to decline significantly, as increasingly critical layers removed.
The Impact of the Number of Merged Layers on Accuracy: Table demonstrates the impact on downstream task accuracy after incorporating the layer merging algorithm top of layer pruning.
16

---

Published as a conference paper at ICLR 2026
From the table, can observe that for both models, whether fine-tuned using FFT or LoRA, layer merging significantly improves accuracy. For Mistral-7B, merging layer after pruning 11 layers aligns its accuracy with the dense model. Although merging layers introduces some accuracy differences, the model still maintains relatively high performance.
Readers may wonder why GradPruner focuses on pruning layers rather than adopting finer-grained approach, such as pruning kernels (rows or columns) within parameter matrices. To address this question, we applied pruning the Hidden Size (HZ) within the GradPruner framework. Specifically, we used IGIA-Matrix to compute the importance of each hidden size, pruned the less important parts, and merged the pruned hidden sizes with the unpruned ones. The pruning rate was set to 40%. The experimental results are shown in Figure 9 below. Compared to layer pruning, using the finer-grained kernel pruning leads to a noticeable decline in accuracy. We believe this phenomenon is mainly caused by two reasons: 1) The importance of layers may take precedence over that of kernels, and pruning kernels in critical layers could lead to a drop in accuracy. 2) Pruning hidden sizes requires merging larger number of hidden sizes, which is less conducive to maintaining accuracy.
Replacing Layer Pruning with Kernel Pruning:
Downstream mn Task Performance (M stral-78)
6.70
Prame
2.79
Accuracy(Aug.)
6.76
AccuracyAugg
0.344
0.76
6:72-
0.75
Saseinel..00
Seselral/T::
474
Nergingi.oRg
Lamaa. Uamaa Mitraa Method
The number pruned layers
Figure 9: Experimental results of replacing layer pruning with kernel pruning in Grad- Pruner. We report the average accuracy across eight datasets.
Figure 8: Experimental results of performing Only the layer pruning in GradPruner.
Layer-importance estimation for tasks with varying gradient noise: To further explore the dif- ferent behaviors of layer importance estimation for downstream tasks under various noise levels, we supplemented the following experiments: Based the WinoGrande dataset (original size 10.2k), we randomly sampled 1 16, 10%, 30%, and 100% of its data as training sets, systematically evaluating the different behaviors of layer importance estimation under different training data sizes (different gradient variances), paying particular attention to whether early gradient signals remain reliable with very small amounts of data. The relevant results are shown in Tab. 5
Table 5: Ablation experiments with different training data sizes conducted on the WinoGrande dataset, based on Llama 3-B (Full Fine-tuning). The Dense Model represents the accuracy of the unpruned model after fine-tuning downstream tasks.
Training data ratio and size
1% (102)
10% (1.02k)
30% (3.06k)
100% (10.2k)
Dense Model
0.853
0.863
0.866
0.868
GradPruner (ours)
0.842
0.857
0.862
0.861
Experiments show that even with a significant reduction in training data, our layer importance esti- mation maintains good stability. Only when the data volume extremely low (c.g., only about 100 samples) does the model performance fluctuate slightly, but the overall trend remains robust. This indicates that our method's 's reliance on early-step gradients is reasonable under typical data scales.
To comprehensively evaluate the relative advantages of Grad- Pruner across different sparsity levels, Tab. [xxtends the main experiment at 40% sparsity by addi- tional comparisons against baseline methods at 30% and 20% sparsity
Performance across Sparsity Ratio:
17

---

Published as a conference paper at ICLR 2026
We did not consider sparsity levels higher than 40% because, under which settings most meth- ods including GradPruner-eee exhibb exhibit significant performance degradation. Existing baseline meth- ods already struggle to maintain reasonable accuracy at 40% sparsity, and increasing sparsity further would exacerbate this performance drop, rendering comparisons at higher sparsity levels uninfor- mative.
Table 6: Comparative experiments between our method and other approaches under different spar- sity levels, using the Llama3.1-8B model with full fine-tuning on the HellaSwag dataset. "Dense Model denotes the accuracy of the unpruned model after fine-tuning on the downstream task.
Methods
Dense Model
LLMPruner
Laco
MINITRON
SAT
GradPruner (ours)
Sparsity Ratio (30%)
0.943
0.916
0.923
0.923
0.934
0,940
Sparsity rat))
0.943
0.927
0.936
0.929
0.940
0.942
The experimental results in Tab. sisow the following:
At 30% sparsity. GradPruner continues to maintain a clear advantage over all baseline methods. At 20% sparsity (i.e., under milder pruning), some methods such Laco and SAT also demonstrate strong performance; however, our method remains competitive and exhibits consistently robust over- all performance. These findings indicate that the advantages GradPruner are not limited mod- erate sparsity levels (e.g., 40%) and can be generalized to lighter pruning settings, demonstrating its effectiveness and robustness across a range of sparsity regimes.
Performance across Model Scales:To verify the applicability of GradPruner across different model scales, Tab. 7 shows experiments on both smaller (approximately IB) and larger (approximately 13B) models, using Llama-3.2-ID and Llama-2-13B as testbeds, respectively. The experiments demonstrate that our method performs effectively across both model scales: it not only maintains strong performance on the larger Llama-2-13B but also exhibits on the smaller Llama-3.2-1B. This indicates that GradPruner's pruning mechanism is highly scal- able, with its effectiveness consistently extending from IB-scale models those exceeding 10B parameters, showcasing strong generality.
more pronounced advantages
We evaluate GradPruner models of different sizes: for Llama-3.2-IB (originally 16 layers), we prune layers; for Llama-2-13B (originally 40 layers), we prune 16 layers. "Dense Model" denotes the accuracy of the unpruned model after fine-tuning the downstream task.
Table 7:
Datasets
MedMCQA
FinGPT
HellaSwag
Llama-3.2 3.2- 1 Model)
0.434
0.543
0.584
-.ama-3.2-1 9.2-1B(GradPPrnee
0.434
0.538
0.585
Model)
0.542
0,806
0.907
14amm=2213B(GrrdPPnnnn))
0.532
0.798
0.894
The Impact of the Number of Initial Fine-tuning Steps: We conducted ablation experiments on the HellaSwag dataset with different gradient accumulation steps. The steps we selected were 0.02%, 0.05%, 0.08%, and 1%. The experimental results are shown in Tab. 8] Experimental re- sults show that method has certain requirements for the number of gradient accumulation steps; performance will decrease when the number is less than 0.05%, but this requirement is not strin- gent experiments show that only about 1% of the training steps needed to achieve good align- ment results and stable performance.
Table 8: Ablation experiments with different gradient accumulation steps were conducted on the HellaSwag dataset, based on Llama -.B (Full Fine-tuning). The Dense Model represents the accuracy of the unpruned model after fine-tuning on downstream tasks.
0.02%
0.05%
0.08%
1%
Dense Model
0.943
0.943
0.943
0.943
GradPruner (ours)
0.911
0.929
0.939
0.939
18

---

Published as a conference paper at ICLR 2026
To check the effectiveness with even fewer gradient accumulation steps, conducted ablation experiments on the HellaSwag dataset with different gradient accumulation steps. The steps we selected were 0.02%, 0.05%, 0.08%, and 1%. The experimental results shown in Tab.
Experimental results show that our method has certain requirements for the number of gradient accu- mulation steps; performance will decrease when the number is less than 0.05%, but this requirement not eeggeentexxprimeent show that only about 1% of the training steps are needed to achieve good alignment results and stable performance.
Table 9: Ablation experiments with different gradient accumulation steps conducted the HellaSwag dataset using the Llama 3.1-8B model. The Dense Model represents the accuracy of the unpruned model after fine-tuning downstream tasks.
0.02%
0.05%
0.08%
1 1%
Dense Model 0.943
0.943
0,943
0.943
GradPruner (ours)
0.911
0.929
0.939
0.939
D.1 STABILITY OF EARLY-STEP GRADIENTS
To further explore the stability of early-step gradients, added the following experiments: Based on the WinoGrande dataset (original size 10.2k), we randomly sampled 1%, 10%, 30%, and 100% of its data training sets, systematically evaluating the stability of early-step gradients under different training data sizes, paying particular attention to whether the early gradient signals remain reliable with extremely small datasets. The relevant results are shown the Tab. IO
Experiments show that even with a significant reduction in training data, layer importance estimation based on early-step gradients maintains good stability. Only when the amount of data extremely low (e.g., only about 100 samples) does the model performance fluctuate slightly, but the overall trend remains robust. This indicates that our method's 's reliance on early-step gradients is reasonable under typical data scales.
Table 10: Ablation experiments with different training data sizes were conducted on the WinoGrande dataset, based on Llama 3.B (Full Fine-tuning). The Dense Model represents the accuracy of the unpruned model after fine-tuning on downstream tasks.
training data ration and size
1% (102)
10% (1.02k)
30% (3.06k)
100% (10.2k)
Dense Model GradPruner (ours)
0,853
0,863
0.866
0,868
0.842
0.857
0.862
0.861
D.2 EFFICIENCY FROM A FLOPS PERSPECTIVE
Tab. above shows comparison of the efficiency of the original Ilama3.1-88 model and the model pruned using GradPruner. During testing, we used an input length of 128 tokens. The results show that the pruned model reduced the number of parameters by approximately 38%, and saved approximately 40% FLOPs and MACs.
D.3 GENERALIZATION IN CROSS-DATASE" SCENARIOS
To verify the generalization ability of the method in cross-dataset scenarios, supplemented it with cross-dataset evaluation experiments within the domain: pruning and fine-tuning were per-
Table 11: Two metrics, FLOPs and MACs, represent the efficiency of GradPruner
Ilama3. 1-8B
FLOPs (TFLOPS)
MACs (GMACs)
Params (B)
original
1.79
893.35
7.5
pruned
1.06
530.43
4.67
19

---

Published as a conference paper at ICLR 2026
formed using the original medical domain training set MedMCQA from the paper, and testing was conducted on the medical subset clinical knowledge in MMLU. The results are shown in Tab. [2] Despite a slight performance drop compared to Dense models in cross-dataset settings (i.e., tran- ing and testing distributions are not perfectly consistent), GradPruner still maintains a significant accuracy advantage other existing pruning methods.
Table 12:
Cross-dataset evaluations in MedMCQA, based Llama 3.1-8B (Full Fine-tuning).
Training data is MedMCQA, and test data is clinical knowledge. The Dense Model represents the accuracy of the unpruned model after fine-tuning on downstream tasks.
Methods
Dense Model
LLMPruner
Laco
MINITRON
SAT GradPruner (ours)
0.735
0.688
0.711
0.703
0.709
0.717
D.4 GENERALIZATION WITH OTHER ADAPTER METHODS
To verify the stability of the IGIA matrix under different adapter methods, we conducted additional experiments: we replaced the strategy in GradPruner that originally used LoRA fine-tuning to eval- uate the importance of pre-trained parameters with QLoRA and DoRA, respectively. The relevant results are summarized Tab. [3 As the experimental results show, the performance of DoRA is comparable to LoRA, with almost no significant difference; however, a slight decrease in accuracy occurs when using QLoRA. This phenomenon can attributed to the technical characteristics of the two methods: DoRA decomposes the pre-trained weights, achieving optimization behavior closer to full parameter fine-tuning than LoRA, thus better evaluating the importance of pre-trained parameters; in contrast, QLoRA introduces quantization operations the pre-trained weights top of LoRA, which introduces quantization noise, leading to some bias in the estimated parameter importance.
Table 13: Stability tests were conducted on the HellaSwag dataset using different adapters based on Llama 3.I-8B (Full Fine-tuning). The Dense Model represents the accuracy of the unpruned model after fine-tuning on downstream tasks.
LoRA
DoRA
QLoRA
Dense Model
0.943
0.943
0.943
GradPruner (ours)
0.939
0.941
0.925
20


Reviews:
[{"review_id": "knuFeDltP6", "reviewer": "Reviewer_RvRE", "summary": "The paper proposes **GradPruner**, a lora-based gradient-guided **layer pruning + sign-based layer merging** method for efficient fine-tuning and inference. The method (i) computes an **Initial Gradient Information Accumulation (IGIA)** matrix from early-step LoRA gradients to score parameter/layer importance, (ii) prunes layers with low summed IGIA (over linear sublayers), and (iii) **sparsify-then-merge** pruned layers into the preceding retained layer using a **sign-consistency rule**. Experiments on two LLMs (Llama-3.1-8B, Mistral-7B) and eight datasets report ~**40% sparsity** with ~**0.99%** average degradation; the headline comparison (Table 1) is at **40% sparsity**.", "questions": "please see weakness above.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- **Simple, pragmatic pipeline.** Early-step gradient accumulation → IGIA-based layer scoring (sum over linear sublayers) → pruning → sign-based merging. The pruning score is clearly defined.\n- **Operationally clear merging.** “Top-p% by IGIA then sign-consistent addition into the preceding kept layer” is straightforward to implement and shown with a framework figure.\n- **Broad task coverage with efficiency reporting.** Ablations include number of merged layers and sparsity-rate sweeps for the proposed method.", "weaknesses": "### (A) Insufficient theoretical grounding for **merging** (most important)\n- **Self-inconsistency between pruning and merging.** The paper emphasizes that **layers differ in importance** and uses **IGIA** to make importance-aware pruning decisions. However, during **merging**, contributions from pruned layers are **added with equal weight** whenever signs match—**without any sensitivity weighting** (e.g., IGIA- or Fisher-based) for either donor or receiver layers. This disconnect undermines the rationale that sensitivity should matter.\n- **Cross-layer addition lacks justification.** There is no theoretical argument that **elementwise addition across different Transformer blocks** preserves function or yields bounded error, even after sparsification. A first-order approximation, Hessian/Fisher weighting, or Lipschitz-based stability discussion is missing.\n- **Actionable request.** Provide a clear **merging objective** (e.g., minimize a first-order loss surrogate), an **error bound**, or at least **IGIA-weighted** or **Fisher-weighted** merges. Otherwise, the method uses importance for pruning but not for merging.\n\n### (B) Comparisons are narrow in **pruning ratio** and **model scale**\n- **Single-ratio comparisons.** The main table compares methods **only at 40% sparsity**. While internal ablations vary sparsity for GradPruner, there is **no multi-ratio cross-method** comparison to show whether the advantage holds when pruning is milder or more aggressive.\n- **Limited model scaling.** Results center on **7–8B** models (plus a 3B FT reference). It is unclear whether gains **transfer down** to ~1–2B or **up** to ~14B+ models.\n- **Actionable request.** Report **accuracy–sparsity curves** for **multiple baselines**, and include **smaller (≈1.7B)** and **larger (≈14B)** models.\n\n### (C) Training recipe & domain generalization\n- The domain-specific fine-tuning often appears to train and test within the **same dataset**. This setup makes it hard to assess **out-of-distribution** robustness within the domain.\n- **Actionable request.** Include **cross-dataset** evaluations per domain (e.g., train on one medical QA dataset, test on another; for math-style settings: train on MetaMathQA-40k, evaluate on GSM8K / GSM-Plus) to demonstrate generalization beyond the training distribution.\n\n### (D) Minor Issues / Clarity\n- **Equation (2) ambiguity/typo.** The equation seems to multiply **the B-gradient twice**, which is likely a typo and inconsistent with LoRA structure. Please clarify the intended mapping and dimensional compatibility.\n- **Citation formatting.** Around **line ~310**, the first paragraph’s citation is not enclosed in parentheses, unlike others; please standardize the style.\n- **Small typos.** A few truncated terms (e.g., “IGIA-Matri”) and punctuation/spacing glitches around the merging equation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "su1mW5S49D", "reviewer": "Reviewer_87rX", "summary": "This paper proposes GradPruner, a gradient-guided structured pruning framework designed to enhance both fine-tuning and inference efficiency for large language models (LLMs). Unlike most existing structured pruning methods that rely on calibration data or additional distillation steps, GradPruner leverages initial gradient information obtained during the first few LoRA fine-tuning steps to estimate parameter and layer importance.\n\nThe key innovation lies in computing an Initial Gradient Information Accumulation Matrix (IGIA-Matrix) to quantify layer importance early in training. Based on this metric, GradPruner performs layer-level pruning followed by a layer merging step that sparsifies and merges pruned layers into adjacent retained layers while resolving sign conflicts to reduce destructive interference.\n\nEmpirically, GradPruner achieves a 40% reduction in parameters with only 0.99% loss in downstream accuracy, tested on two LLMs (LLaMA3.1-8B and Mistral-7B) across eight diverse benchmarks spanning medical, financial, and general reasoning tasks. It shows consistent performance improvements over strong structured pruning baselines (LLM-Pruner, LaCo, SAT, APT, and MINITRON), and reduces both training and inference time/memory by over 35%.", "questions": "1. How sensitive is GradPruner’s performance to the number of initial fine-tuning steps ttt?\n\n\n2. Have the authors tested GradPruner when using other adapter methods (e.g., QLoRA, DoRA) to verify whether IGIA-Matrix remains stable?\n\n\n3. In the merging phase, how is the sparsity rate ppp selected? Could adaptive sparsity (learned from IGIA statistics) yield further improvements?\n\n\n4. Does GradPruner preserve the same inference graph (number of layers) after merging, or does merging affect sequence length/runtime at deployment?\n\n\n5. Can GradPruner be integrated with post-training quantization or low-rank compression? If so, how does the gradient-based importance interact with quantization noise?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The use of IGIA-Matrix computed from <1% of training steps is original and empirically justified by gradient sensitivity analysis.\n\n\n2. The proposed sign-consistent merging technique effectively preserves accuracy even under 40% pruning.\n\n\n3. The authors test across multiple domains, model sizes, and fine-tuning regimes with strong baselines, demonstrating robustness.\n\n\n4. Substantial reductions in both training and inference costs (~35–40%) while maintaining accuracy are practically valuable.\n\n\n5. The paper analyzes pruning ratios, merging counts, and alternatives such as kernel pruning and weighted averaging.", "weaknesses": "1. While the empirical gradient correlation study is convincing, the paper lacks a deeper theoretical analysis of why early gradient accumulation correlates with long-term importance, beyond empirical observation.\n\n\n2. The method assumes access to LoRA gradients and may not generalize to non-LoRA or adapter-free fine-tuning setups.\n\n\n3. The layer-importance estimation could behave differently for tasks with varying gradient noise; this is not fully explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jreQZXCgk9", "reviewer": "Reviewer_QPt3", "summary": "GradPruner uses the IGIA-Matrix computed from the early stages of fine-tuning (within the first 1% of steps) to evaluate the importance of parameters and layers. Layers with low importance scores are pruned, and to further retain performance under high pruning rates, the parameters of pruned layers are merged into the retained layer using a sign-based merging rule. Experiments on two LLMs across eight datasets show that GradPruner achieves approximately 40% parameter reduction with less than 1% average accuracy drop, while also significantly reducing fine-tuning time, inference latency, and memory consumption compared to representative pruning baselines.", "questions": "* Can you clarify the exact computation and shape alignment for simulating the gradient of W via LoRA gradients?\n* Why is raw summation chosen for IGIA-matrix aggregation across all linear layers, rather than normalization (e.g., mean or norm per parameter or per layer size)? \n* Could the authors elaborate on the layer merging procedure, perhaps with an example?\n* Can the approach still be effective with even fewer gradient accumulation steps (e.g., less than 0.02% steps)？", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* GradPruner jointly improves both training and inference efficiency. Unlike many pruning approaches that focus only on inference speed-ups, GradPruner is explicitly designed to reduce fine-tuning time and memory consumption as well. I believe this aligns well with the practical need for LLMs to quickly adapt to new downstream tasks.\n* GradPruner is simple to understand yet conceptually novel. The paper introduces a new gradient-based importance metric computed from less than 1% of the early fine-tuning steps, which effectively reduces the computational cost of pruning. In addition, it seems like the proposed pruning strategy is generally applicable to all transformer-based architectures.\n* The experiments are comprehensive. Evaluations are conducted on two representative LLMs across eight datasets, which enhances the reliability of the results. The paper also provides detailed ablation studies that clearly demonstrate the contribution of each component.", "weaknesses": "* Limited theoretical grounding. Lines 201–208 simulate the gradient of W via a matrix multiplication, yet the rationale for this simulation is not theoretically justified. In addition, the sign-based merging rule in Equation (5) is presented as a heuristic with little theoretical explanation. Clearer derivations would strengthen the contribution.\n* Concern over the stability of early-step gradients. The method relies on gradient statistics collected within the first 1% of fine-tuning steps to estimate layer importance, yet such early gradients may not always provide a stable or reliable signal. For example, in small or noisy datasets, gradient variance can be high, potentially causing the IGIA-Matrix to mis-rank layer importance and resulting in suboptimal pruning decisions.\n* Incomplete efficiency metrics. The paper aims to make layer pruning more efficient and reports average time and parameter reduction. Including FLOPs would provide a more standardized and hardware-agnostic measure of computational efficiency.\n* Clarity of the method description could be improved. The paper does not explicitly detail how each transformer sub-module is handled during merging. It appears that each linear sub-module of a pruned layer is merged into the corresponding sub-module of the preceding retained layer, but this should be stated unambiguously. Moreover, Equation (5), which specifies merging based on sign agreement, is difficult to parse. A more thorough explanation and a small illustrative example would aid the reader's understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)