DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models
Authors: Wenlong Deng
Yize Zhao
Vala Vakilian
Minghui Chen
Xiaoxiao Li
Christos Thrampoulidis
Venue: ICLR 2025 Spotlight (2025)
Keywords: Delta parameter pruning
Efficiency
Large Language Models
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Storing open-source fine-tuned models separately introduces redundancy and increases response times in applications utilizing multiple models. Delta-parameter pruning (DPP), particularly the random drop and rescale (DARE) method proposed by Yu et al., addresses this by pruning the majority of delta parameters—the differences between fine-tuned and pre-trained model weights—while typically maintaining minimal performance loss. However, DARE fails when either the pruning rate or the magnitude of the delta parameters is large. We highlight two key reasons for this failure: (1) an excessively large rescaling factor as pruning rates increase, and (2) high mean and variance in the delta parameters. To push DARE’s limits, we introduce DAREx (DARE the eXtreme), which features two algorithmic improvements: (1) DAREx-q, a rescaling factor modification that significantly boosts performance at high pruning rates (e.g., > 30% on COLA and SST2 for encoder models, with even greater gains in decoder models), and (2) DAREx-L2, which combines DARE with AdamR, an in-training method that applies appropriate delta regularization before DPP. We also demonstrate that DAREx-q can be seamlessly combined with vanilla parameter-efficient fine-tuning techniques like LoRA and can facilitate structural DPP. Additionally, we revisit the application of importance-based pruning techniques within DPP, demonstrating that they outperform random-based methods when delta parameters are large. Through this comprehensive study, we develop a pipeline for selecting the most appropriate DPP method under various practical scenarios.

Published as a conference paper ICLR 2025
DARE THE EXTREME 3ं. REVISITING DELTA-PARAMETER PRUNING FOR FINE-TUNEE MODELS
Wenlong Dengl.2 1.2, Yize Zhao', Vala Vakilian', Minghui Chen Xiaoxiao Li
Christos Thrampoulidis!
'The University of British Columbia
2 Vector Institute
https://github. com/ htt////egddggg///eee DAREx. git
ABSTRACT
Storing open-source fine-tuned models separately introduces redundancy and increases response times in applications utilizing multiple models. Delta-parameter pruning (DPP), particularly the random drop and rescale (DARE) method proposed by Yu et al., addresses this by pruning the majority of delta parameters the differences between fine-tuned and pre-trained model veights-whild typically maintaining minimal performance loss. However, DARE fails when either the pruning rate or the magnitude of the delta parameters is large. We highlight two key reasons for this failure: (1) an excessively large rescaling factor as pruning rates increase, and (2) high mean and variance in the delta parameters. To push DARE's limits, we introduce DAREx (DARE the eXtreme), which features two algorithmic improvements: (1) DAREX-q. a rescaling factor modification that significantly boosts performance at high pruning rates (e.g., 30% on COLA and SST2 for encoder models, with even greater gains decoder models), and (2) DAREX-L2, which combines DARE with AdamR, in-training method that applies appropriate delta regularization before DPP. We also demonstrate that DAREX-Q can be seamlessly combined with vanilla parameter-efficient fine-tuning techniques like LoRA and can facilitate structural DPP. Additionally, we revisit the application of importance-bant DPP, demonstrating that they outperform random- based methods when delta parameters are large. Through this comprehensive study, we develop pipeline for selecting the most appropriate DPP method under various practical scenarios.
pruning techniques within
INTRODUCTION
Large Language Models (LLMs) like BERT (Devlin al. 2018 GPT (Floridi & Chiriatti 2020), and Llama (Touvron et al., 2023) excel in various language-modeling
tasks. Finetuning these for specific downstream tasks enhances personalized experiences (Sanh et al., 202 Labrak al. 2024) and has become a standard practice in natural language processing (Dodge et al. 2020 Zhao et al. 2023). ). Indeed, most entries on the OpenLLM Leaderboard involve full-parameter finetunes or their combinations (Liu et al. 2024), underscoring the widespread adoption and availability of fine-tuned models online. Decomposing fine-tuned model weights into the original parameters of the pre-trained model yields delta parameters (DP) (Yu et al., 2023a Liu et al. 2024: Yao & Klimovic 2023). Reducing the size of DPs, which are as large as the base model and can number in the hundreds of millions of parameters for LLMs, could significantly enhance communication efficiency in federated learning, minimize task conflicts in model merging, accelerate multi-task serving, and decrease storage needs for new fine-tuned models (see Related Work).
Delta- pattmeerrreter pruning (DPP) drops fraction p of the DPs towards realizing these benefits. Naturally, pruning, which compresses neural networks
DPP can be seen as an instance of generic by eliminating weights that contribute minimally, resulting in sparse architectures (LeCun al. 1989 Han

---

Published as a conference paper at ICLR 2025
et al. 2015b). Traditional pruning methods typically remove weights post-training based on importance criteria like weight magnitude or activation levels. While these techniques could naturally extend to DPP, their integration into this context remains largely unexplored.
Random-basse pruning strategies, which provide more flexibility and efficiency in implementation, also offer a competitive alternative. For instance, Random Drop and Rescale (DARE) (Yu et al. 2023a), recently introduced randomized DPP method, reduces DP size through random pruning followed by rescaling. DARE has been quickly adopted across various applications, including model merging libraries (Goddard et al. 2024), state-of-the of-the-arrrrrr art medical LLMs @Labrak et al. 2024). and Japanese LLMs (Akiba et al. (2024).
However, as we demonstrate in this paper, DARE struggles when the pruning rate is high or when DPs are large. This observation prompts several key questions:
What are the key factors contributing to DARE's failure modes? Can these issues be addressed to push the limits of effective rundom-based DPP? Additionally, can importance crrseaecee--edd model pruning techniques be applied to DPP in ways that compete with random- -based methods?
1.1 CONTRIBUTIONS
We address these questions through principled analysis and extensive experimentation on large-scale lan- guage models and datasets. Our contributions are summarized fol-
Pruning Rate
p
0.99
Delta Size
417.7MB
11,4MB
Method
COLA
SST2
(p -0= 0,99)
Test Performance (%)
Test Performance (%)
Pruming
56.24
90.25
DARE Yuu ul.
2023a
4.25
$1.00
lows:
L MP Han et al
201 (ours)
12.30 (+8.05)
83.14 (+32.14)
DAREX-La (ours)
57.24 (+52.99)
88.17 (+37.00)
Analysis of DARE: By examining the absolute change in intermediate output model representations result- ing from the application of DARE the DPs, we identify two primary factors that influence this change: (a) a large pruning rate (p), which re- sults in excessively high rescaling factor of 1/(1- p). and (b) a high mean and variance of the DPs rel- ative to input activations. Through experiments on both controlled se- tups and LLMs, we show that the absolute change in intermediate rep- resentations is a reliable proxy for test performance (see Fig. D.. Thus, these two factors emerge as the main contributors to DARE's failure modes. This analysis inspires two new algorithms that significantly improve DARE's performance, as follows.
DAREX-q (L/g.) (ours)
48.96 (+44.71)
85.64 (+34.64)
DAREX-q (1/g,) (ours)
45.20
85.81 (+34.81)
Table I: Impact of our key contributions on BERT and two datasets COLA and SST2: 'DAREX-L2" applies DARE after AdamR-L2 fine- tuning, 'LIMPP' applies magnitude pruning after AdamR-L,, the factors qu and qe adjust DARE's rescaling using a validation set and unlabeled data, respectively. All four proposed methods significantly outperform vanilla DARE (Yu et al., 2023a demonstrating that (randomized) DPP can still be highly effective, even at extreme pruning rates of % parameter reduction, when incorporating our modifications. Notably, our new post-hoc rescaling achieves a 40-fold model size reduction with minimal impact on performance compared to the unpruned model. See Tables 2 for more details.
Drop and rescale with 1/9 (DAREX-q): To ensure efficiency at high pruning-rates (e.g., p > 0.9), where DPP offers significant savings, our first algorithm DAREX-q q modifies DARE's ' rescaling factor from 1/(1- ( p) to 1/g. where 9 > 1- p. While (Yu et al. 2023a) recommended using the factor 1/(1 p) to maintain zero-expectation changes in intermediate representations (accounting for DARE's randomness), they neglected the importance of controlling the absolute changes and the impact of randomness, particularly since the pruning is applied only once.A large rescaling factor inflate the variance of these changes; therefore, by tuning the rescaling factor l/g. we can effectively balance the mean and variance, thereby minimizing performance degradation. We develop and evaluate four variants of DAREX-q (detailed in Sec. 4. to suit different scenarios along two orthogonal axes: (a) whether a labeled validation set is available and (b) whether tune global 1/q across all layers per-layer 1/g. All four methods provide rescaling factors that
2

---

Published as a conference paper ICLR 2025
Mean Output Change VS BERT)p-0.99)
Test Peformence VS BERT1p=0.99/
COLA MRPC
0.0120.016
0.034
SST2
(eeea Dusput thange
0.1
STS-8
1-2 0.01 (CARE)
Test Peformance
0:016
0.018
044
p. 01.5
0.3
NRPC
SS12
02
0:015
00-
1 0.01 (DARE)
0.000
0.010
0LO2>
0020
0.003
0.030
0.200
0,02
0.05
0.000
Invorse Rescaling Factor
Inverse Rescaling Factor
(a) Last-layer output changes on batch of training data.
(b) Test performance.
Figure 1: Our DAREX-q improves on DARE's performance by tuning the rescaling factor 1/g. Experiments with BERT at pruning rate of = 0.99. Right: The optimal rescaling factor (asterisk), which maximizes test performance, differs from the standard 1 (1 - p) across all four datasets and yields up to 10-fold gains. Left: The rescaling factor that minimizes the last-layer output change, averaged over last-layer neurons, serves as an excellent proxy for the optimal factor maximizing performance and can be determined by inference single training batch.
significantly improve upon vanilla DARE while preserving its purely post-training nature, demonstrating that effective DPP remains possible even at high pruning rates.
Our second algorithm controls the mean and variance of DPs (averaged over input features at each intermediate layer) prior to applying DARE. Specifically, our AdamR, a modified Adam optimizerr, applies L2 regularization directly to the DPs during fine-tuming, Combining AdamR with DARE (DAREx-L2) yields highly competitive performance (see Table I]. While this requires in in-training modification, it significantly broadens the scenarios where randomized DPP can be effectively applied.
AdamR fine-tuning:
Extensive experiments: We conduct extensive experiments on both encoder-decoder and decoder-only models across range of downstream tasks. The results demonstrate the effectiveness of both DAREX-g and D2REx-L2 algorithms. As summarized in Table applying these techniques to a fine-tuned BERT model on the CoLA and SST-2 datasets leads to substantial performance improvements, consistently exceeding 35%. Additional results are presented in Tables 2357 and Figures 23
importance-based DPP: We incorporate importance-based methods such as magnitude- nagnitude-pruning and WANDA into DPP. Our analysis reveals that while random-based DPP generally outperforms important-basee DPP, especially under higher pruning rates, combining the latter with AdamR-L, regularization (as opposed to L2) forms competitive alternative (see Tables I and 77. Moreover, importance-based DPP can outperform random-based DPP when DPs are large and eustomized fine-tuning is not an option.
2 RELATED WORK
patta--aramtter pruning. DPs represent the parameter changes induced by fine-tuning and are critical to various operations. In Federated and Distributed Learning, DPs are continuously exchanged between central server and distributed clients during training (Li et al. 2021 2020 Khirirat et al.) 2018 Chen et al. 2021). model merging, the goal to merge multiple task-specific models, fine-tuned from the same pretrained backbone, by combining their DPs into single model with diverse capabilities (Wortsman et al. 2022 Ilharco et al., (002).. Similarly, in multi-task serving. DPs are processed separately while the base model weights computed once during the forward pass (ao & Klimovic 2023 Liu et al. 2024). (Yu et al.
2023a Liu et al. 2024; Yao & Klimovic 2023) focus on reducing the size of DPs, and recently Yu et al. 2023a) has popularized DARE a powerful candom-based DPP method. However, DARE fails with large
3

---

Published as a conference paper at ICLR 2025
DPs or high pruning rates. We propose techniques that, applied during both fine-tuning and post-hoc stages, restore performance and significantly improve upon vanilla DARE. Our results demonstrate, for the first time, that DPP remains effective even at extreme pruning rates, reducing up to 99% of the model parameters. Thanks to our modification, we also show that random-based DPP can be combined with LoRA and can be used for structured pruning.
Model parameter pruning. There are two primary approaches to model parameter pruning: regularization- based and importance based methods; however, an exhaustive review is beyond the scope of this paper. Among iuportance-baseee methods, early works (LeCun et al., 1989 Hassibi & Stork 1992) introduced importance scores based on second-orde information from the loss function. In deep neural networks (DNNs), methods such (Han al., 2013a Li et al. 2018 Han et al. 2015b) utilize magnitude pruning to eliminate weights based on their magnitude. More recently, (Sun et al. (023) developed an importance score called WANDA, which is proportional to the product of weight magnitude and the norm of the input feature, specifically targeting LLMs. However, the application of these techniques to DPP remains poorly understood. We address this gap by studying both the classical magnitude pruning method and the recently proposed
approach, WANDA, within the context of DPP.
3 DELTA PARAMETER PRUNING
DPP aims to efficiently store fine-tuned model O. initialized from pre-trained base model 0p by pruning the difference known as the "delta parameter" (DP), defined as Ag = 0r - -0p between the fine-tuned and the base model. Successful DP pruning (DPP) removes parameters from An resulting in reduced storage requirements while minimally compromising performance. Sec. 3.1 presents the theoretical model used analyze various DPP methods. Sec. 3.2 and Appendix provide theoretical analysis of randomized and aepoortance-bsee DPP methods, respectively. These theoretical insights are tested through detailed controlled experimental analysis using a two-layer neural network in App. B and, more importantly, applied in Sec. 4 to guide the design of new algorithms that significantly improve LLM performance in Sec.
3.1 FORMULATION
To gain analytical insights, we consider the application of DPP to perceptron operations in neural network. This focus is partly motivated by the observation that multi-layer perceptron (MLP) operations, both in the projection layers of attention mechanisms and ferwaro networks, constitute the majority of operations LLMs. Consider a perceptron layer, where E R" is the input activation vector connecting to m output neurons, W E Rm*X" is the hidden-layer weight matrix, and a is a Lipschitz activation non non-lineane -lnecarrity (c.g.. ReLU, sigmoid) Let Aw E R denote the delta parameters of the weight matrix. A pruning mapping P mR - R aims to reduce the parameters in A w. leading to a sparsified P (W)). The desired sparsity level, k E [m], is critical design specification for DPP, often dictated by factors such as storage constraints relative to the number of fine-tuned models. For randomized DPP, elements of Aw are dropped with a probability p& E [o, 1]. resulting an average sparsity level of A: (1-p)mn. The objective the delta parameters to a sparsity level & (or 1 - for randomized DPP) while maintaining the performance of the pruned model, represented by Wp + P(Aw) Aw) ). close to that of the original model Wp. To quantify the impact of pruning, we define the difference vector diiff which measures the change in intermediate output model representations between the pruned and original models:
prune
h diff = Aww - P(Aw)æ.
(1)
We require that output after pruning (WW + ((Aw))( ))²) does not differ much from the output before pruning a ((Wp+ + Aw )z). By Lipschitzness of the activation, this translates to each entry hdlif of the difference vector h diff being small in absolute value. Thus, we study how P affects |Adifff i E [m]]
Bias terms can be included by augmenting the input and DPs to x' ='= -xx and A' Aw Ao). respectively, but recent LLM architectures often omit bias terms for training stability (Touvron 2023 Chowdhery et al. 2023
4

---

Published as a conference paper ICLR 2025
3.2 RANDOMIZED DPP: RANDOM DROP AND RESCALE (DARE)
DARE randomly sets each element of (P(Aw) w las to zero (dropped) with probability p, and rescales non-zero elements [P( P w/ k, = A w / p) Yu et al. 2023a). While DARE has been tested on a wide range of fine-tuned models across various tasks, it exhibits performance degradation under high pruning rates, We seek the underlying causes of these limitations and explore potential improvements. Our approach involves quantifying the difference vector h hdiff in relation to key variables: the dropout rate, the DPs, and the input data. As per Eq.[] [
shown in Table 7 and Table S
the effect of DARE on the output becomes
diff = Σ jc(m)
AWijoy - Σ JE[n] 1
8.g AgA Jo
(2)
where p the drop rate, (615)xe(m) [e(m),)e) are iid Bernoulli(1 pp random variables, and 1/(1 - p) is the rescaling factor. We denote , the (i entry of Aw E Rmxx It is easy see that E[ 0 (expectation over &ty), and Yu al. (0023a) use this as justification for DARE's effective performance. However, this alone overlooks the sources of DARE's s failure modes, which we reveal by instead establishing bounds on the absolute output changes [h The following theorem addresses this by providing high-probability bound on these changes, with respect to DARE's randomness, which arises from the Bernoulli variables. See App. F. the proof.
Theorem 3.1. Denote helft as the i-th component of hdiff in Eq. 2 For E [m],j E [n], let cjj = AW Jj represent the change in influence of the j-th feature on the i-th output neuron after fine-tuning. Define the mean eN and variance of of these as: ē (1/n) ΣjE(n] Cij and o3 = (1/n) Ej -,) Then, for any E [m] and y E (0,1), 1), holds with probability at least 1 y that
|ng diffff S (4(0)/(1-))) p)) nnc + +ot) log (2/7)
where V(()) 2p)/ log((1 p)/p) ifp S 1/2, otherwise (p) /2p(1-p))
Thus, the key factors influencing the magnitude of [hghiff are: (i) the rescaling factor 1/(1- p) and (ii) the mean and variance (averages are over input dimension j E [n]) of the influence parameters {ct;} ). Specifically, increasing the rescaling factor (eqv. increase drop-rate p) increases Infiff at most rate of O p) à), which can be large in demanding pruning scenarios when is large. Also DARE yields smaller | values when Gjj AWajæ, exhibit low first (mean) and second order (variance) averages. We validate these conclusions through elaborate experiments on a controlled setup in Appendix B More importantly, our algorithms in Sec. []] further tested on LLMs, are also built upon the insights from this theorem and its observations.
3.3 IMPORTANCE-BASED BASED DPP
Using the same analytical framework, we extend the application of importance-bassee pruning methods, specifically magnitude pruning and WANDA, to the DPP setting. Due to space constraints, we provide a detailed discussion in App. A In brief, importance-based DPP can achieve strong performance when the distribution of the coefficients Gjj exhibits light tails and sharp-peakedness. We provide empirical validation of these findings in Sec. C.5 and Appendix B
4 ALGORITHMS
To extend the applicability of DPP at high pruning rates and in cases where DPs of fine-tuned models exhibit undesirable statistics, we propose here two strategies based on the theoretical insights discussed in Sec. B
We call 'mean' l'arrancee here the first/sccond -order summary statisties of (ce, Hje(m) over the input dimension. On the other hand, the theorem's 'tigh-probability' statement is w.r.t. the randomness of random variables (6:))sc(w
5

---

Published as a conference paper at ICLR 2025
4.1 ADJUSTING THE RESCALING FACTOR
Motivation. Recall from Thm. 3.1 as p increases, the absolute difference in outputs | grows at at most rate of (((1 p)- j. Setting the rescaling factor to 1 - p eliminates the mean of Kglif, but does not minimize its magnitude. Through empirical analysis, we demonstrate in Fig. I how various values of q influence model outputs in terms of both mean output change (i.e., the average of |nghtf| across all last-layer neurons) in Fig. [a and, test performance Fig. Tb across different datasets. For 99% pruning rate (1-p- 0.01) we observe these quantities over a range of q values from 0.005 to 0.04. The results depicted in Fig. La for mean output change, indicate a convex-like trend, with minimum points (marked with asterisks) achieved at values higher than the vanilla 1 p. F.g., the optimal q for COLA and SST2 is x 0.018 and 0.033, respectively. Recall now that in our analysis, the amount of change in output activations is used as a proxy for test performance: smaller mean output change corresponds to better performance. Fig. [bb validates this: the test-performance curves show a concave-like shape, with the peak performance also occurring at q > 1- p and numerically consistent with the values of that minimize the mean absolute output change.
Drop and rescale with 1/q (DAREX-q). Motivated by these observations, we propose two variants of DAREx-q: tuning based on (1) maximizing performance on a labeled validation set, and (2) minimizing mean output change over unlabeled data.
DAREX-q with labeled validation data (1/q,): We use validation dataset (#e: yo E V to determine the best rescaling factor 1 1/q. that maximizes test performance (eqv. minimizes test error) on the validation set. Specifically, we select g, = arg min, (f (z.)) A yo), where fa represents the pruned model rescaled by 1/g. We then randomly drop a faction p of the model DPs and rescale those that are not dropped with 1 1/qu- Further details can found in Algorithm 2 (1) in the Appendix.
DAREX-q with unlabeled data (1/qe): This method selects q by optimizing an unsupervised objective measuring the mean output difference | across all last-layer neurons of the model. This approach is based on the observation in Fig. that mean output difference is an effective unsupervised proxy for test performance. The key advantage of this method over qe is that it does not require labeled data. Specifically, our implementations, selecting ge involves running inference on single fixed batch of data (with texts from either the training data or any other natural dataset) using both the pruned and original models. Refer to Algorithm 2 in the Appendix for further details.
DAREX-q with per-layer tuning. The implementations of DAREX-q described above select a single, global rescaling factor (1/qg or 1/q²) that is then applied to all surviving DPs across all layers of the network.
However, since the DPs in intermediate layers each have their own mean output change over their respective output layers, a natural question arises: Can we select a different rescaling factor for each layer E [T.]? Moreover, can this be done efficiently without requiring I. hyperparameter searches (one for each layer), which would be computationally prohibitive? In Appendix C.9 we demonstrate that the answer to both of these questions is affirmative. To circumvent the computational challenge of L searches, we leverage theoretical insights by extending Thm. Thm.3.1 to provide Like Thm. 3.1. the new bound still depends on the first- and second-order statistics of the coefficients cly. which means the optimal ge can vary per layer based on these statistics. We denote this vector of per-layer rescaling factors as 1/q [1/g].... 1/g"1 and 1/qe, respectively, for two implementations depending on whether the performance on validation set or mean output change is optimized. Please refer to App. C.9 for details. In Sec. 5.1] we show that this approach performs very well encoder models and is comparable to the global 1/q approach for decoder models.
gigh-brbbabilityy bound for arbitrary q and then optimizing over it.
4.2 ADAMR FINETUNING
Motivation. While DAREX-q addresses the issue of large rescaling factors at high pruning rates, we' 'v seen that for absolute output changes |لcli to remain small, the first- and second-order averages of the influence
'The rate of growth with p in the bigh-probabillty upper bound of Thm. 3.1 is tight and thus predictive of the output change taking larger values as p increases; see App. E.
6

---

Published as a conference paper ICLR 2025
Table 2: D consistently outperforms DARE across four datasets and two encoder models. The pruning rate is set to p = 0.99. Bold indicates the best performance, while underline marks the second-best.
Dataset
SST2
COLA
MRPC
STS-B
Models
BERT
RoBERTa
BERT
RoBERTa
BERT
RoBERTa
BERT
RoBERTa
Pruning
$0.25 [-]
92.09 (-)
(6.24(-)
62.10 (-)
85.01 [-]
90.05 (-)
88.51
90.57 [-)
DARE
51.00 (0.64)
51.81 (1.00)
4.25
7.65 (3.63)
79.96 (6.07)
87.32 (0.69)
82.56
81.59 (1.44)
DAREX-q (1/g.)
85.64 (3.07)
86.56 (4.39)
(4.93)
(1.58
83,92 (0.62)
87.30
(1.60
88.07 (0.70)
DAREX-q (1/q.)
89.60 (0.67)
90.40 (1.36)
53.12 (2.01)
58.38 (2.05)
83.86 (0.66)
89.80 (0.60)
87,55 (0.12)
89.10 (0.32)
DAREX-q (1/g.)
85.81 (2.72)
81.05 (4.96)
45.20
(5.00
83.12 (1.11)
88.85 (0.26)
87.03 (0.12)
87.29 (0.57)
DAREx-q (1/qg)
87.39 (0.86)
89.62 (1.35)
31.70
54.43 (2.49)
83.25 (0.94)
89.59 (0.42)
87.59 (0.19)
87.60 (0.49)
Table 3: Similar to Table
DAREX-q yields significant gains for decoder models.
Dataset
GSM8K
Abel-7B
MetaMath-7B
WizardMath-75
MetaMath Qwen2-Q.nnnn
Methods
p= 0.95
0.99
0.95
= 0.99
p p= 0.95
p 0.99
= 0.95
p 0.99
No Pruning
58.32
65.50
54.90
42.00
DARE
37.30
0.00
58.22
0.00
47.10
0.00
0.00
0.00
DAREx-q (1/g.)
47.20
20.20
59.05
34.87
49.81
35.63
30.70
19.17
DAREX-Q q (1/qg)
44.50
20.00
59.28
32.06
50.64
34.34
29.56
18.65
DAREX-q (1/g.)
42.99
21.30
60.34
32.60
49.05
33.97
30.32
18.95
DAREX-Q (1/9c)
42.84
19.63
59.28
28.05
50.64
33.58
28.80
17.66
factors [E()) must also be small. These factors directly depend on the magnitude of the DPs AW which are determined during fine-tuning. We propose an in-training method that fine-tunes the model keep these statistics small, thereby guaranteeing that the post-hoc application of DARE maintains performance.
A AdamR-L2. Our approach is fine-ture mun-tune the model with L2 regularization on the DPs A = 0r 0p.
As mentioned, the motivation behind this is penalizing |اممك directly translates to smaller total second- order averages Σjenn < which, as previously noted, capture the extent of change in the output layer post-finetuning and factor into the bound in Thm. 3.1 To implement this, we replace the weight decay of AdamW-th standard choice for training LLMs with our custom regularization. Specifically, we adjust the regularization strength based on the gradient norm, as Xie et al. (2024) recently found that weight decay can cause large gradient norms during the final phase of training. Our modified decay step of AdamW is:
θ: 0
m
Vo + A(0,-1 0p),
(3)
V +E
where 0, represents model parameters at iteration f, m, and ú, are the first and second moments of gradients, and 0, is the mean of the second moment used to adjust regularization. See also Algorithm [ Appendix.
R We extend our approach to fine-tune the model in preparation for importance-based pruning, rather than random-based pruning, post-training. In this case, we regularize using the L1 norm of the DPs. This is again motivated by the analysis in Sec. 3.3 The detailed algorithm, referred to as AdamR-L,, is provided Algorithm in the Appendix.
5 EXPERIMENTS
We validate the effectiveness of our analysis and proposed algorithms through comprehensive experiments. Datasets and models for Encoder-based and Decoder-based LMs. For Encoder-based LMs, we utilize four datasets sentence acceptability dataset CoLA (Warstadt al., 2019). sentiment detection dataset SST- 2 (Socher et al. 2013)., paraphrase dataset MRPC (Dolan & Brockett) 2005), and sentence similarity dataset STS-B (Cer al. (017). For the selection of pretrained backbones, jet al. 2018) and KoBERTa-bbss (Liu et al., 2019), fine-tuning them on these task-specific datasets to obtain supervised finetuned models. For Decoder-bart based LMs, focus on mathematical reasoning tasks. Due
choose BERR-baas--unaasee (Devlin
7

---

Published as conference paper ICLR 2025
Pruning Rate Performance SST-2 and STS-8 Datasets
Pruning vs Performance on GSMBK
90
90
Test Accuracy
Test Accuracy
86
Test Accuracy
80
84
75
DARE BERT 5ST-2
80
DARE BERT STS-8
DARE Abel
SSF-2
RoBERTa
BERT SST-2
BERT STS-8
Abel
65
9. RoBERTa SST-2
76
RoBERTa STS-B
Meta-Math
0.96
0.90
0.02
0.94
0.96
0.98
Pruning Rate (p)
Pruning Rate
Pruning Rate
(a) Encoder models
(b) Decoder model
Figure Across pruning rates p, DAREX-Q performs at least as well as vanilla DARE and significantly outperforms it at higher pruning rates.
constraints on resources for tne-ttunig, we opted to fine-tum tne-tune smaller Qwen2-0.5B (Yang 2024 model on the MetaMath dataset YYu et al. (203b)) Additionally, we utilize publicly available mathematical reasoning models, including the MetaMatthllemm-11 (Yu et al. 2023b). MetaMath-7BB (Yu et al. 2023b), WizardMath- 7B (Luo et al., 2023) and Abel-7B (Chern et al. 2023), 3), all based on the Llama2-7B architecture (Touvron et al. 2023). We then use the GSMSK (Cobbe et al. 2021) to test these models.
Evaluation metries. Following Yuu et al.) 2023a),
evaluate performance using various metrics; the Matthews correlation coefficient for CoLA, accuracy for SST-2, a combined score of accuracy and Fl for MRPC, the mean of Pearson and Spearman correlations for STS-B, BB, and zero-shot accuracy for GSM8K.
5.1
RESCALING-PARAMEEN
MODIFICATION
We evaluate our DARq algorithm, discussed in Section 4.1 Recall the four proposed variations for choosing
the rescaling factor, categorized by two distinct factors: (a) the use of labeled validation data or not (1/9, or 1/g.. respectively); and (b) global tuning versus per-layer tuning (1/q versus 1/q).
Encoder models. Table 2 compares DAREX-q (all four variants) with vanilla DARE (with rescaling factor 1/(1 p)). We report average performance and standard deviation runs. For reference, we also report performance of the finetuned model before pruning. We find that: (1) All DAREX-q variants outperform vanilla DARE in all cases and by large margin (e.g., 40% COLA). (2) 1/g, and 1/q, deliver competitive performance compared to 1 1/9, and 1/q,- making it an alternative when a labeled validation set is unavailable. (3) Per-layer rescaling (1/q, and 1/q.) outperforms global rescaling. Decoder models. Table evaluaaes DAREX-q on decoder models. We find that: (1) DAREX-q consistently boosts performance over vanilla DARE across all models and both pruning rates p 0.95, 0.99. (2) Higher pruning rates yield striking improvements; vanilla rescaling fails completely, while DAREX-g yields non-trivial performance; (3) Global rescaling (1/q) provides performance comparable to that of per-layer rescaling (1/q), making it more efficient recommendation for decoder models.
the test set over four independent
Performance across different pruning
Table 4: DAREX-q is orthogonal to and can be combined with LoRA, leading to significant improvements over vanilla DARE.
rates. Fig. 2 compares performance of DAREX-q to vanilla DARE for varying values of pruning rates for both encoder and decoder models. For all examined values of p. DAREx-q is at least good as vanilla DARE. More importantly, we confirm that DAREX-q significantly out- performs vanilla DARE at higher pruning rates.
Datasets (p 0.9)
STS-B
COLA
SST2
MRPC
No Pruning
90.07
59.56
94.27
87.09
DARE Yu et al.
2023a
81.49
50.96
48.05
80.03
DAREX- (L/qg) (ours)
85.06
56.73
90.25
82.53
DAREX-g (1 (1/g.) (ours)
85.00
56.73
90.02
82.53
8

---

Published as a conference paper ICLR 2025
Table 5: DARE with Ad finetuning (DAREx-Lg) significantly improves performance on encoder models compared to DARE without it.
BERT
RoBERTa
Dataset
P
original
L2
difference
original
L2
difference
No Pruning
85.01 (-)
84.31 (-)
-0.70
90.05 (-)
90.69 (-)
+0.64
MRPC
0.99
79,96 (6.07)
84.30 (0.48)
+4.34
87.32 (0.69)
90.02 (0.41)
+2.70
No Pruning 0.99
8-)51
87.99 (-)
-0.52
90.57 (-)
90.25 (-)
-0.32
STS-B
82.56 (1.13)
87.62 (0.27)
+5.06
81.59 (1.44)
89.88 (0.29)
+8,29
No Pruning
90.25 (-)
89)91
-0.34
92)
(-)
+0.80
SST2
0.99
51,00 (0.64)
88.17 (0,44)
+37.17
51.81 (1.00)
91.95 (0.24)
+40.14
No Pruning
5-)24
59.27 (-)
+3.03
(-)
59.27 (-)
-2.83
COLA
0.99
4.25 (2.83)
57.24 (0.62)
+52.99
7.65 (3.63)
59.01 (1.26)
+51.36
LoRA+DAREX-q: As a post-training DPP, DAREX-q is orthogonal to and can be combined with parameter- efficient tuning approaches like LoRA (Hu et al. 2021). To demonstrate this, we fine-tune the BERT-Base model by training only the LoRA module, then prune the parameters of LoRA rate p 0.9. The results, shown in Table reveal that applying vanilla DARE does not yield good performance. In contrast, our
DAREX-q significantly improves the results. In addition these, demonstrate in Appendix Sec. C.11 that
DAREX-Q is even comparable to sparse fine-tuning, despite being a purely post-training method. Structural DPP with DAREX-q. Our random based DAREx-q can also be utilized for structural DPP for
enhanced hardware efficiency (Shen et al. 2022; Yao & Klimovic, 2023). In this setup, we randomly select a% of the input dimensions of MLP layer and randomly retain 0% of the DPs within the selected dimensions, achieving a total structured DP pruning rate of
b%. Table in the Appendix, usesa = 5, b = 20 for an equivalent p = 0.99, to show that DAREX-q outperforms DARE with a 40% improvement.
5.2
AD FINETUNING
We now show that our proposed in-training method of Sec. 4.2 Encoder models. Table 55 the performance of DARE ap- plied to the original finetuned model against its performance when applied to models finetuned with AdamR-L2 across two encoder models and four datasets. We set the pruming rate at 0.99 and, for reference, also report the performance the finetuned models without pruning, both with and without AdamR- L2. Consistent with the insights of Sec. [42]] applying DARE after AdamR- (DAREX-L2) outperforms vanilla DARE without regularization. Additionally, the performance is only marginally lower than that of the original finetuned model. Figure 7 in the Appendix further shows that increasing the regularization strength allows pruning up 99.9% of parameters with only minor performance drop. See Appendix C.4 for further ablations on regularization weight. Finally, Table 9 the Appendix demonstrates that DAREX-L2 unlocks the potential of random-based DPP for structural pruning, improving DARE without L2 finetuning by over 40%. We also conduct training analysis of DAREX-L, L2 in Appendix F Decoder models. To demonstrate the effectiveness of DAREX- L2 on decoder-based LMs,
successfully achieves highlv prunable DPs.
Adam8: on Owen-028 GSMBK)
Teak Rocuraayy
Mcg120 wiigiccee00
Log Scated 1-0g1001
Applying DARE on de- Figure 3:
coder models finetuned with AdamR-Ta (DAREX-I.2) at varying regularization strengths demonstrates significant per- formance improvements for p > 0.9.
finetune Qwen-O.5B using the MetaMath dataset (Yu et al., 2023b). Figure shows the L2 fine-tuned Qwen-0.5B model with different regularization weights on the MetaMath datasets. We used pruning rates p E [0, 0.5, 0.5. 0.7. 0.9, 0.99) to prune the DPs. For visualization, the x-axis uses a log-scaled pruning value of log(1 11111pp0 p) /2- = {0, 0.15, 0.26, 0.5. 1.0). When AdamR-I. is not applied (regularization weight 0), performance GSM8K drops significantly
9

---

Published as a conference paper at ICLR 2025
p 0.9. By increasing the regularization weight, higher pruning rates can be achieved. With regularization weight of le- (green line), we can reach 99% pruning rate while maintaining performance.
5.3 How TO APPLY IMPORTANCE BASED DPP AND WHEN IS IT COMPETITIVE?
We have seen, both theoretically and empirically, that DARE is effective at pruning DPs when their first and second order statistics (e.g. as summarized by 2; and of in Thm. 3.1) are small. While most fine-tuned LLMs typically introduce small DPs Yao & Klimovic (2023); Lee et al. (2019), exceptions exist. For instance, Table 8 in the App. shows that the MeetaMathLLLaMA.
model, fine-tuned on large datasets (MetaMathQA Yu et al. (2023b) and Proof-Pile-2 Azerbayev et al. (2023)), ). has a mean delta parameter magnitude of 0.017, notably larger than that of Abel-7B and MletaMath-7B In such cases, demon- strated in Table 6 DARE experiences a substantial performance drop when the pruning rate exceeds 0.1.
Can importance-based DPP methods serve as alternatives? Yes, as shown meth-
Table 6: Peformance of pruning methods on MetaMath-llema.
in Table [ Importance-basse ods like MP and WANDA can main- tain pruning rates of up to 0.5 with only minimal performance degrada- tion. Thus, these methods are effective alternatives to DARE when DPs are large and fine-tuning with customized methods like AdamR- is not feasible.
Methods
GSM8K
(MetaMath-Tenma) Ilema-7B)
p 0.0
p= 0.1
p= 0.3
p 0.5
0.6
MP
63.00
61.00
47.50
32,00
WANDA
63.00
62.50
54.50
32.50
64.50
DARE
51.00
0.00
0.00
0,00
Random Drop
52.50
6.50
0.00
0.00
Improvements with AdamR-Lj- For completeness of our study, we also demonstrate that finetuning with AdamR- _, can boost performance of importance-bart baasaancee-eeed DPP as suggested in Sec. 4.2] Specifically, in Table 7 in App. C.5 show elaborate experiments comparing the performance of MP and WANDA (applied to DPs) when used on a model finetuned with/without AdamR-L, for varying values of pruning rates and two encoder models across four datasets. In summary, we find: (1) AdamR-La consistently increases performance of MP. (2) While AdamR- L1 improves WANDA's performance for most datasets, this is not always the case. We attribute this to WANDA's importance score being impacted not only by DPs AWsy but also by input magnitude |ızل and important DPs not always being aligned with outlier activations (see App. D
6 CONCLUSIONS AND LIMITATIONS
Our study of DPP methods, provides theoretical insights, ex- tensive experiments in controlled environments, and demon- strates practical applications on large-scale models and datasets. Starting with an in-depth analysis of random-based DPP methods, we identify key factors that degrade their per- formance. Based on these insights, we propose: (i) DAREX-q, a post-training rescaling modification strategy that signifi- cantly outperforms the state-of-the-ar and performs at least as well across other rates; and (ii) AdamR finetuning to enhance existing models that struggle with DPP, ensuring the production of highly prunable DPs. Our results, including dramatic gains over baseline methods shown in Table [ suggest that these strategies warrant further investigation across different tasks and modalities. Addi- tionally, preliminary results suggest that DAREX-q can be effectively combined with parameter-efficien methods like LoRA and utilized for structural DPP, both of which highlight the potential for further study. Finally, while finetuning LLMs typically results in small DPs favoring random-basee DPP, we demonstrate that importance- based DPP methods can serve as robust alternatives for larger DPs. This highlights the need for further exploration of alternatives. In conclusion, comprehensive study of DPP methods provides a framework for selecting appropriate DPP methods under different scenarios, which we summarize in Fig. 4 (details in App. C.3). Broader impact see App. C.13
Start
Allow Fine- Turing?
No
Are Lar go?
Yeu
WANDA DP
Yos
No
DAREX-
AdamR-I,
AdamR-1, R-
high pruning rates
MP-DP
DAREX-
Validation Date?
No
DAREX (1/e.)
Youi
Flowchart for DPP
DAREX-Q
Methods Selection
(1/%))
Figure 4: Flowchart for selecting appropriate DPP methods based on different scenarios.
10

---

Published as conference paper ICLR 2025
Acknowledgments: This work was partially funded by an Alliance Mission Grant, the NSERC Discovery Grant No. 2021-03677, the Alliance Grant ALLRP 581098-22, NFRFE-2023-00923 Natural Sciences and Engineering Research Council of Canada (NSERC), Canada CIFAR AI Chairs program, Canada Research Chair program, MITACS-CIFAR Catalyst Grant Program, the Digital Research Alliance of Canada, Institute of Information & communications Technology Planning & Evaluation (ITP)) grant funded by the Korea government (MSIT).
REFERENCES
Takuya Akiba, Makoto Shing, Yujin Tang. Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. arXiv preprint arXiv:2403.13187 2024.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint Xrv: 10631, 2023.
Daniel Berend and Aryeh Kontorovich. On the concentration of the missing mass. Electronic Communications in Probability, 18(3):1-7, 2013.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio. and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity similarity-mulilingee multilingual and cross-lingual focused evaluation. arXiv preprint arXiv: 1708.0005, 2017.
Guanzheng Chen, Fangyu Liu, Zaiqiao Meng. and Shangsong Liang. Revisiting parameter-efficient tuning: Are we really there yet? arXiv preprint arXiv:2202.07962. 2022.
Mingzhe Chen, Deniz Gündüz, Kaibin Huang, Walid Saad, Mehdi Bennis, Aneta Vulgarakis Feljan, and Vincent Poor. Distributed learning in wireless networks: Recent progress and future challenges. IEEE Journal on Selected Areas in Communications, 39(12):3579-360) 2021.
Ethan Chern, Haoyang Zou, Xuefeng Li, Jiewen Hu, Kehua Feng, Junlong Li, and Pengfei Liu. Generative ai for math: Abel. /ttps::/qittee
com www.m//III.. G
abel
2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.144: 14168, 2021.
Wenlong Deng, Christos Thrampoulidis, and Xiaoxiao Li. Unlocking the potential of prompt-tuning in bridging generalized and personalized federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6087-6097, 2024.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zeulemoyer. Gpt3, int8 O: 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318-30332. 2022.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:
2018.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):200-2355 2023.

---

Published as a conference paper ICLR 2025
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannanch Hajishirzi, and Noah Smith. Fine- tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002. 06305, 2020.
Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP200S), 2005.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 19999, 2020.
Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681-694, 2020.
Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Wai Lam, Lidong Bing, and Nigel Collier. On the effectiveness of parfieter-efficient fine-tuning. In Proceedings of the AAAI conference artificial intelligence, volume 37, pp. 12799-12807, 2023.
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's mergekit: toolkit for merging large language models. arXiv preprini arXiv:2403. 13257, 2024.
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXhv:2012.07463 2020.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffmman coding. arXiv preprint arXiv:1570.00149, 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015b.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1xXiv:2/06.0960) 2021.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint urXiv:22/2.04089 2022.
Michael Kearns and Lawrence Saul. Large deviation methods for approximate probabilistic inference. arXiv preprint orXiv:1301.72: 73900,.773.,000 2013.
Sarit Khirirat, Hamid Reza Feyzmahdavian, and Mikael Johansson. Distributed learning with compressed gradients. arXiv preprint arXiv: 06573, 2018.
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01533 2024.
Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoing Gourraud, Mickael Rouvier, and Richard Dufour. Biomistral: collection of open-source pretrained large language models for medical domains. arXiv preprini arXiv:2402.1 10373, 2024.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural proooesnionprrccesiingg processing systems, 2, 1989.
12

---

Published as a conference paper ICLR 2025
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent.
Advances neural information processing systems, 32, 2019.
Guiying Li, Chao Qian, Chunhui Jiang, Xiaofen Lu, and Tang. Optimization based layer-wise magnitude- based pruning for dan compression. In IJCAI, volume 330, pp. 2383-2389, 2018.
Li Li, Yuxi Fan, Mike Tse, and Kuo- Yi Lin. A review of applications in federated learning. Computers & Industrial Engineering, 149:106854, 2020.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.000 2021.
Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07622 2021.
Baohao Liao, Yan Meng. and Christof Monz. Parameter-efficien fine-tuning without introducing new latency.
arXiv preprint arXiv:2305. 16742, 2023.
James Liu, Guangxuan Xiao, Kai Li, Jason Lee, Song Han, Tri Dao, and Tianle Cai. Bitdelta: Your fine-tune may only be worth one bit. arXiv preprint arXiv:2402. 10193, 2024.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint
2019.
Yadong Lu, Chunyuan Li, Haotian Liu, Jianwei Yang, Jianfeng Gao, and Yelong Shen. An empirical study of scaling instruct-tuned large multimodal models. arXiv preprint arXiv:2309.0990 2023.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced wool-instruct. arXiv preprint arXiv:2308.09583, 2023.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Manni.-Manor. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104. 10972, 2021.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika. Zaid Alyafeai, Antoine Chaffin, Arnau Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:21 10.08207 2021.
Pierre Sermanet, Soumith Chintala, and Yann LeCun. Convolutional neural networks applied to house numbers digit classification. In Proceedings of the 21st international conference on pattern recognition (ICPR2012). pp. 3288 3291. IEEE. 2012.
Maying Shen, Hongxu Yin, Pavlo Molchanov, Lei Mao, Jianna Liu, and Jose M Alvarez. Structural pruning via latency-saliency knapsack. Advances in Neural Information Processing Systems, 35:12894-111 12908, 2022.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pp. 1631-1642, 2013.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306./ 1695, 2023.
13

---

Published as conference paper ICLR 2025
Mingjie Sun, Xinlei Chen, Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXi::2402.17762 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09228 2023.
Roman Vershynin. High-dimensional probability. University of California, Irvine, 10:11, 2020.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transac- tions of the Association for Computational Linguistics, 7:625 14625-61, 2019.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35: 17412-1741444 2022.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lppes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korbllith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965-23998, PMLR, 2022.
Zeke Xie, Zhiqiang Xu, Jingzhao Zhang, Issei Sato, and Masashi Sugiyama. On the overlooked pitfalls of weight decay and how mitigate them: A gradient-norm perspective. Advances in Neural Information Processing Systems, 36, 2024.
Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. Advances in neural information processing systems, 32, 2019.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024.
Xiaozhe Yao and Ana Klimovic. Deltazip: Multi-tenant language model serving via delta compression. arXiv preprint arXiv:2312.05215. 2023.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as free lunch. arXiv preprint arXiv:2311.03099 2023a.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 1224424, 2023b.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yinggian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint a303v /8223, 2023.
14

---

Contents
Introduction
1
Contributions
2
2
Related work
3
3
Delta parameter pruning
3.1
Formulation
4
3.2
Randomized DPP: Random Drop and Rescale (DARE)
5
3.3
Importance-based DPP
5
4
Algorithms
4.1
Adjusting the rescaling factor
6
4.2
AdamR finetuning
5
Experiments
7
5.1
Rescaling-paramell modification
8
5.2
AdamR-I-2 finetuning
5.3
How to apply coportance-bbsse DPP and when is it competitive?
10
6
Conclusions and limitations
10
Appendix
15
A
Importance-basee DPP
17
Analysis on two-layer neural-netword
17
Additional details
18
C.1
Additional related Work
18
C.2
Implementation details
19
C33
Proposed framework for DPP method selection.
19
C.4
Controlling pruning with regularizaiton
19
C.5
AdamR-L, L, finetuning
20
C.6
Scale of delta parameters
21
C.7
AdamR algorithm
21
C.8
DAREX-q algorithm)
23
C.9
Analytical Per-layer 1/q
24
C.10
Structural pruning
25
C.1
Comparison to sparse finetuning
25
12 Random-ban based methods generally outperforms importance-bul based DPP
26
C.13 Broader impact
26
D Massive activation and outlier features
27
D.I Outlier feature (activation)
27
D.2 Massive activation!
27
E
Proofs
28
Proof of l'heorem 3.1
28
E.1.1 Lower bound
31
15

---

Published as conference paper ICLR 2025
E.2
Finding a rescaling factor balancing mean and variance
31
F
DAREX-L2 Training Analysis
33
Additional Experiments on Scalability and Applicability
33
G.1
Performance on computer vision task
34
G.2
Performance on larger models
34
G.3
LoRA with AdamR-L2
35
16

---

Published as a conference paper ICLR 2025
A IMPORTANCE-BASED DPP
In this section, we adapt importance-basel pruming, which is traditionally popular for model parameter pruning. c.g. Han et al. 2015b); Li et al. (2018); Sun et al. (2023), ), into DPP.
Magnitude pruning (MP). Magnitude pruning Li et al. (2018) drops model weights according to their magnitudes: less "important" weights with smaller magnitudes are dropped. To extend its application to pruning delta parameters evaluate importance in terms of the magnitudes of delta parameters, represented as ASi = AWyl-
Pruning based on both weights and activations (WANDA). Recently, WANDA was introduced by |un et al. (2023) and was empirically found to achieve state-of-the-an pruning performance for L.Mss. It evaluates the importance each model weight using a score metric that scales the magnitude of the weight by the Euclidean norm of the input feature, thus accounting for the input activations. We extend WANDA to encompass delta parameters by adjusting the importance score A.S.) = 1, ||,|||2-
For their respective scores, MP and WANDA (and any other aeporraace--ampliingg method adapted to DPP) maintain the top-k parameters in Aw with the highest scores. Let Sş C [m]] x [n]] be the set of indices (i. j) that correspond to weights with the top-k scores AS,j. Moreover, let l := (1 p)mn so that the method retains the same number delta parameters as DARÉ does in expectation. Accordingly, denote the set of selected weights with respect to p as Sp == Se. Then, putting in Eq. ( the change for each output activation i E [m]] can be expressed as:
A ddifff
E E(ponnes, AWajay
(4)
where the summation extends over all input dimensions j for which the entry of Aw is dropped due a low score. An importance sampling method can perform effectively even for large z if the cumulative contributions from the summation are approximately zero out. This is guaranteed when the distribution of the entries [AWayailje has a light tail and high peakedness, We validate this in Secs. C.5 accordingly propose A fine-tuning to enhance their pruning performance (algorithm detailed in in Sec.
Secs. and B and
B ANALYSIS ON TWO-LAYER NEURAL-NETWORK
Having gained analytical insights into the key factors that influence the performance of DPP methods in Sec. 3.22 and Sec. A we now explore in detail how these factors influence DPP in a two-layer neural network. Concretely, for input x: TR.,, the model output is f(r W. N( Wi N(wr) b by- Here, N denotes layer normalization (RMSnorm Zhang & Sennrich (2019) in our case), ø is the ReLU activation function, W. / b, and W1 / by are the weights biases the output and the hidden layer respectively, and are all trainable (during both pre- pre-training and fine-tuning). In our experiments, we pre-train the model on the CIFAR- dataset and use the SVNH dataset [Sermanet et al. (2012) for the supervised fine-tuning task.
Influence of variance and mean statistics on DARE (Fig. 5a): Thm. 3.1 establishes how DARE's performance, approximated by the magnitude of helfft is influenced by the mean and variance statistics of {cg)) Specifically, smaller values are favorable for performance. To verify this, we compare performance of DARE when the model is fine-tuned using L,/L2 regularization with respect to the pre-trained parameters. Formally, recalling that A = O, - 0p denotes the delta parameter of all trainable weights, we use regularization that penalizes the following: || r E {1,2). L2 regularization directly corresponds a smaller total energy (Eip <.)) of the (cig) parameters, which recall capture the degree of change in the output layer after fine-tuning. This enters the bound Thm. 3.1 and suggests that Lz regularization improves DARE's performance. On the other hand, L, regularization favors sparse Ag. thus also {ciz] }. Intuitively,
17

---

Published as a conference paper ICLR 2025
Moan and Varlance infeence on DARE Pruning
hormatination Infuence en DARE Pruning
Roqular zallo benefi importance Funiing
Perfommnce
nare WMIC
De82
12 EUOR un mas cearn 1-)
WARDL
06
Rusiig haa
60
(5)
Figure 5: Controlled experiments of DPP performance on two-layer neural net. (a) Influence of variance and mean statistics on DARE. (b) Influence of normalization layer. (c) L1 regularization for importance based pruning. (d) Methods with best-fitting regularization.
this increases the variance, thus we expect from Thm. that L1 regularization might hurt performance. Fig. (a) shows the test accuracy of DARE for varying values of pruning rate p for three finetuning settings: no regularization (blue). L2 regularization (orange), and L1 regularization. The results confirm the insights above and thus support Thm. 3.1
Influence of normalization layer Fig.5p)) Normalization layers have become standard practice in deep learning due to their role in stabilizing training by normalizing the variance of activations, as established in Xu al. (2019). In this study, we explore the impact of these layers on the performance of DARE. According to Thm. 3.1 reduced variance in activations (denoted as z,s in our theorem setup) leads to decreased variance the (cgg coefficients, which turn is predicted to enhance DARE's effectiveness. This hypothesis confirmed in Fig. (b), where the absence of normalization layer is shown to expedite the performance decline of DARE as pruning rates increase. Conversely, importance- based methods discussed in Sec. [A] demonstrate a higher tolerance to increased pruning rates without normalization. These observations are consistent with the discussions in Sec. A
Li regularization for importance based pruning (Figure In Sec. we discussed how importance- based DPP methods are favored. Here, we demonstrate this is indeed the case by using L1 regularization during finetuning to induce sparse delta parameters, which increases the variance. Concretely, as illustrated in Fig. introducing I,, regularization improves the performance of importance- methods. Note also that, in this scenario, magnitude pruning outperforms WANDA due to the absence of outlier features Wei et al. (2022) Sun al. (2023).
Methods with best-fitting regularization(Fig 5d): In Fig 5d we compare three methods with their corresponding best-fit regularization. As discussed above, DARE benefits from L2 regularization, while importance-based methods benefit from Ln regularization. We find that that DARE with L2 regularization more robust with respect to the increase in pruning rate. On the other hand, importance based methods with Li regularization perform the best for medium-range pruning rates.
C ADDITIONAL DETAILS
C.I ADDITIONAL. RELATED WORK
Supervised fine-tuning of language models. Supervised fine-tuning (SFT) of pre-trained LLMs is designed to enhance their capabilities by training on task-specific data, establishing a standard in natural language processing Dodge al. (2020); Zhao al. (2023). SFT can be categorized into full fine-tuning Devlin lt al.] (2018); Liu et al. 2019) and parfmeter-effiicent fine-tuning (PEFT) Ding et al. (2023): Li & Liang
(2021 Hu et al. 2222 Deng et al. (2024). However, recent studiesChen al. (2022): b Lu et al. (2023)
18

---

Published as a conference paper ICLR 2025
Yao & Klimovic (2023) suggest that PEFT methods may not yet achieve the model quality of full parameter fine-tuning, particularly in high-resource tasks Chen et al. (2022). Additionally, Liu et al. (2024) indicates that most models on the Open LLM Leaderboard are derived from full parameter fine-tunes or their merges. Consequently, this paper focuses on full model fine-tuning.
Sparse finetuning. An orthogonal approach to DPP is sparse fine-tuning, which reduces the size of delta parameters by modifying the fine-tuning process itself. This is achieved through techniques such as iterative
masking and fine-tuning to create sparse DP (Guo et al. 2020: Liao al. 2023 Fu et al. (023). In contrast,
DPP methods like DARE Yu et al. (2023a) are primarily post-hoc procedures that focus pruning the delta weights of models that have already been fine-tuned, making DPP particularly valuable for the many fine-tuned models available on platforms like Hugging Face.
Foplow-up work on DARE. The recent development of the DARE (Drops delta parameters And REscales) technique has significantly advanced the efficiency of finetuned-mode pruning. This method sets most delta parameters to zero, maintaining the efficacy of Supervised Fine-Tuning without loss of performance. Highlighted in Goddard et al. (2024) within Arcee's MergeKit, toolkit for merging large LMs, DARE has shown great potential enhancing model merging processes and has various practical applications. It has been successfully implemented projects such as Prometheus Kim al. (2024), an open-source LM for evaluating other LMs, and in medical LMs like Biomistral Labrak et al. (2024), which develops LMs for medical applications. The technique also supports specialized domains, as seen in Akiba al. (2024).
which focuses on a Japanese LLM with advanced reasoning capabilities. These implementations highlight the broad applicability and effectiveness DARE in enhancing model merging strategies, thus making our improvements particularly relevant and practical.
C.2 IMPLEMENTATION DETAILS
For decoder LLMs, following Yu et al. (202a)), we set the temperature to 0.0 for greedy decoding and limit the maximum number of generated tokens to 1,024 GSM8K. For encoder-based LMs, we fine-tune unn-sseeee-u BERT-base- uncased and RoBERTa- Experiments are conducted on NVIDIA A100 GPUs.
for 10 epochs using warmup strategy and learning rate of le-4.
C.3 PROPOSED FRAMEWORK FOR DPP METHOD SELECTION.
We outline the procedure for selecting appropriate DPP methods in practice. As illustrated in Fig if fine-tuning is not permitted and the data points (DPs) have large statistics, recommend using WANDA for DPP (see Sections 5.3 If the DPs are not large, DAREX-q with 1/g, should be applied when a validation set is available, otherwise, DAREX-q with 1/ge is recommended (see Sec. 4.1. If the existing DPs are insufficient and fine-tuning is allowed, we suggest using AdamR-L9 or L1 with appropriate regularization weights to generate highly prunable DPs for DAREX-q and MP, respectively (see Sec. C.7). Among the two, AdamR- LDgR-DDDDDREE... (DAREX-L2-4) should be prefered due to flexibility and better performance as shown in Table 9 and Table
C.4 CONTROLLING PRUNING WITH REGULARIZAITON
In this section, we demonstrate the magnitude of regularization weight can control the degree of pruning that is required.
Fig (a-d) illustrates the AdamR-L2 fintuned RoBERTa model with different regularization weights on SST2, MRPC, COLA and STS-B datasets respectively and we use pruning rate p 0.9, 0.99, 0.999] to prune the delta parameters. To separate different pruning rates in the figure, we set log scaled pruning -log(1 p)/3 [0, 0.33, 0.67, 1.00 in axis. With moderate regularization the model has close
19

---

Published as a conference paper ICLR 2025
Canted the repuar (RodEtt
62
u 4415
0.02
CD
24
(0)
(b)
(c)
(d)
Figure Control DARE pruning performance with L2 regularizaiton on (oBERTa.(a-d) demonstrate results on SST2, MRPC, COLA, and STS-B, respectively. Aligning the degree of regularization proportionally to the pruning rate can lead to optimal performance.
performance to models without regularizaiton (weight 0) (even better i.e. weight 0.01 in Fig (a)). indicate adding moderate regularization won't hurt the model performance. To achieve moderate pruning performance p 0.99, we can choose moderate regularization. For example, in fig (c). choose weight 0.05 achieves the best performance when pruning rate p = 0.99 and outperforms the no regularization by more than 40%. When there need for extreme delta parameters pruning, is recommended to increase the regularization weight. As shown in Figure 6 (d), weight 0.1 achieves the best performance when pruning rate p = 0.999 and outperforms no regualrizaiton by 80 %. It is notable with L2 regualrization, all dataset can achieve 99.9% delta parameter pruning with minor performance drop. Finally, shown in Fig (d), a strong regularization weight 0.5 although face original performance drop, but demonstrate robustness (minor performance drop) to different level of pruning rate.
(a)) This
Fig 7 (a-d) depicts Le fine-tuned BERT models with varied regularization weights on SST2, MRPC, COLA, and STS-B datasets, respectively. Pruning rates p E [0, 0.9, 0.99, 0.999] are used. Log- scaled pruning -log(1-p) /3 = [0, 0.33, 0.67, 1.00] are plot to separate different rates on the x-axis. Moderate regularization yields performance close to models without regularization (weight 0), sometimes even better (e.g., weight 0.05 in Fig (a)). suggesting it won't it hurt model performance. For moderate pruning (p = 0.99), moderate regularization suffices. For instance, in Fig (a), weight 0.05 achieves optimal performance, outperforming no regularization by about 40%. For extreme pruning needs, increase the regularization weight. As in Fig (b), weight 0.1 achieves peak performance at p = 0.999, surpassing no regularization by 60%. Notably, with AdamR- L2, all datasets can achieve 99.9% DPP with minimal performance drop. Lastly, Fig Fig](() shows that a strong weight of 0.1, despite an initial performance drop, exhibits robustness (minor drop) across different pruning rates.
C.5 ADAMR-LI FINETUNING
In this section, we focus on MP (Han et al. 2015a and WANDA (Sun et al. 2023) utilizing AdamR- L1 to enhance pruning performance for importance- based methods.
MP: We demonstrate AdamR-L, results in Table the pruning rate consistently increased on datasets when applying magnitude pruning on L1 fintuned models. This indicates AdamR- L, an effective strategy in producing highly prunbale DPs for MP.
WANDA: As shown in Table 7 AdamR-L; improved WANDA's performance in most datasets. However, in the SST2 and STS-B datasets (Table 7 adding AdamR-I, negatively impacted pruning performance, as indicated by the numbers in red. This is because WANDA considers outlier activations Dettmers et al.) (2022) in LLMs and uses AS, =AW, |æy] as its metrie. Thus, some important DPs may be pruned
20

---

Published as a conference paper ICLR 2025
Control the pruning with ingutatzation (HHH s512)
Control ingulatration (MERT MSPCI
Control
pouning regudariatio
0.53
37
0.75
04
042
025
viggEAL vigeRAce
3.2
033
souiigg Pate (1m0001--000
02
02
Rate beppsttaun
02
6500
(4)
(b)
(6)
(d)
Figure 7: Control DARE pruning performance with L2 regularization on BERT. (a-d) demonstrate results on SST2, MRPC, COLA, and STS-B, respectively. Aligning the degree of regularization proportionally to the pruning rate can lead optimal performance. Table 7: MP and WANDA with AdamR-L, on BERT and RoBERTa across various datasets
SSP2
COLA
Models
p=0.0
p-0.00
p-1.05
p=1.00
pC00
|=0.00
p-0.55
PS0.09
original
41
original
4g
original
4g
original
A,
original
original
1,
oraginal
BERT-WANDA
89.91
90.51
80.68
63.53
86.93
79.93
57.64
63.19
37.34
47.49
45.29
19.24
44.59
46.09
8.37
12.30
10.85
RoBERTa WANDA
92.09
92.55
92.12
90227
91.24
92.78
92.35
54.66
92.78
$9.45
65.00
76.15
91.36
72,82
62,10
60.16
34,64
14.85
30.78
42.54
6.56
0.07
43.31
36.55
0.00
0.00
123T
29,90
MRFC.
STS-B
Models
pabl
p-0.00
p-1.95
p-1.99
p-110
p-1.00
p-1.55
p-0.99
Original
original
1.1
original
1,1
original
1,1
original
origital
1.1
original
1.1
original
7.1
84.35
64.77
67.22
26.52
23.49
15.81
88.51
80.71
$4.98
710.00
40.93
RoBERTa WANDA
MP
90.05
90.07
62.21
76.39
$4.68
K5.17
76.24
75.17
78.86
78.93
74.80
74.80
74.80
74.80
90.57
89.87
84.56
32,70
87.43
78.13
76.17
74.58
86.25
65.42
21.26
20.91
38.33
25.87
when AdamR-Li is applied due their small [xy]] This suggests that important DPs are not always aligned with outlier activations, and that AdamR-Ly may not be the optimal strategy for improving WANDA. For further analysis of outlier activations, see Appendix [)
C.6 SCALE OF DELTA PARAMETERS
As demonstrated in Theorem 3.1 DARE is effective at pruning DPs when the mean and variance of Ctj are
small, and it generally outperforms importance- -assed methods such conditions, as shown in Table 12 While most fine-tuned LLMs typically introduce small DPs Yao & Klimovic (2023); [ee et al.) 2011 and x is small, as indicated in Table 8 where 1 < AW1jl. there are still instances where delta parameters are large. Specifically, Table 8 reports the magnitudes of AW and ej across various LLMs. For instance, the MetaMatt-LLLaMA-
Yu et al. (2023b), which fine-tunes LLaMA2-7B using two large datasets-MettMMtthe MetaMathQA Yu et al. (2023b) and Proof-Pilee Pile-2 Azerbayev et al. - mean delta parameter magnitude of 0.017, sigmificantly larger than that of Abel-7B and MetaMath-7BB As illustrated in Table 6 DARE experiences substantial performance drop when the pruning rate exceeds 0.1.
C.7 ADAMR ALGORIITM
With a simple model. Sec. B demonstrated how L1 and Le regularization enhance DPP. We now show that when modifications to model finetuning are allowed, our approach can scale up to LLMs. In order to implement regularization strategy, we propose replacing the weight decay of AdamW- -the standard choice for training LLMs LLMs--ith our custom regularization, as described in Sec. [.2 and Sec. C.3] However, we do this carefully since Xie et al. (2024) found recently that weight decay can lead to large gradient norms
21

---

Published as conference paper ICLR 2025
Table 8: Magnitude Statistics of ( and AWyy of Llama2-7B fintuned LLMs on all dimensions and layers.
Models
Magnitude
mcan (AW (l)
mean
AW
Abel-7B
7.3c-4 (4.3e-7)
2.89e-5 (5.41c-9)
MetaMath-7B
7.2c-4 (3.3e-7)
2.85e-5 (6.02e-9)
Wizardmath- 7B
4.0e-4 (1.0e-7)
1.56e-5 (1.67e-9)
MetaMath-Ilemm 7B
0.017 (1.9e-4)
7.0e-4 (3.02 e-6)
in the final training phase, deteriorating convergence. To address this issue, following Xie et al. (2024) adjust our regularization strength based on the gradient norm. We call our new optimizer as AdamR. Note
regularization here is based the DP rather than simply the weights of the finetuned model. The detailed algorithm, which we call AdamR, is presented in Algorithm]]
Algorithm 1 AdamR
Input: Pre-trained model 8p.
1: 9: VL(
2: m. = 81m4-1 + (1 81 8119
3: 0t 32V1-1
(1 82)gi
4: m, I-ØT
5: 6,
Vr
I-3H
6: G mean (t))
7: 0, = 0,-1
m.
7 1(2,1 - 0p)
L2 regularization
Vbgto Vo,+t
8: 0, 0
Vo mi
Jo,te Asign(Or-1 0p)
11 regularization
Recall from Sec. B that Le regularization promotes small mean and variance of delta parameters, while L, regularization encourages sparse updates important parameters, leading to larger variance. We also emphasize that our regularization here is based on the DP rather than simply on the weights of the finetuned model.
22

---

Published as conference paper ICLR 2025
C.8 DAREX-Q ALGORITHM
Algorithm 2 Empirically find q
Input: Delta Weight As, Input Dataset D,, Pre-trained Model 8p, Pruning Rate Rounds N.
Step Size Ag (Ar))
1: initialization: error inf
2: for 1. 1--1 + N do
3: Find global then = 1 -p+tx
4:
1- p > Prune and rescale by gr
5:
Ag = P(As. 9r)
6:
else
7:
= t x An
8:
q, Algorithm mp)
Resolve per-layer rescaling factor. &. Prune and rescale by q
9:
A = P(A.,
10:
end if
11:
0' = 0p + Ao'
> Add pruned delta parameters to Op
12:
Dr is Validation Dataset then
13:
error = E() (f((f) + y)
(1) Validation d.
else
15:
error EzED, (Os( x)* s(0p + 20,x)4)
D (2) Last Layer's change de
16:
end
17: if error < mir then
18:
errormin = error and Qbest = 9t (qbest = qi)
19:
end if
20:
end for
21:
Return Qbest (9best)
Algorithm 3 Analytically calculate
Input: Delta Weight 4g, Input æ, Pre-trained Model 0p, Pruning Rate p, Step Size Ag, Rounds N s 1/Aq. Empty List q = []. Probability 7, Constant 7y:
1: initialization: x' x
Initialize layer input b Loop each MI.P layer
2: for Aw, W E Ag. 0p do
3: initialization: errormán inf
4: for f < + N do
at = - 6: eтror = Eq (5)
b 2 p b Evaluate the objective in Eq 55) for the given 4 =
Y, 7,21,11 error errormin and f N then errormin error and qbest Qt
7:
8:
end if 10: end for
9:
11: q.append(qhes. (best)
Add best q for current layer Calculate the output of the current layer of the fine-tuned model.
12: x' f'= + Wr,z)
13:
end for
14:
Return q
this section, we propose and describe three strategies to determine the global rescaling factor. The per-layer rescaling factor follows similar approach to the global one, with slight adjustments in determining the and
23

---

Published as a conference paper ICLR 2025
resolving 1/q (see details in Algorithm 2 step 6.)
Empirically finding 1 1/q. using validation data: In this approach, we use validation dataset {Ze, Be to empirically determine the best rescaling factor q. The goal to solve the optimization problem arg min, (xx) yo), where La represents the pruned model rescaled by 1/g. The process is as follows:
Optimization Process: We iterate over range of q values, adjusting the pruned model by rescaling with 1/q. For each 9, we measure the model's performance on the validation set by comparing the model's predictions fg (zx) to the true labels yo-
Outcome: The q value that results in the best performance the validation set is selected as 1 1/ge- This 1/ is expected to provide the best overall performance across all layers when evaluated against the validation set.
Algorithm Details: This process is detailed in Algorithm 2 with objective (1), which outlines the specific steps taken to find 1/qe-
Empirically finding 1/q. using output change: This method builds on the core idea from Thm 3.1 aiming empirically determine global rescaling factor de that balances the mean and variance in output embedding differences. Unlike the validation-basee approach, this method does not rely on labeled data. Instead, uses unsupervised objective function based on the difference in the last layer's 's output embedding before and after pruning and rescaling. Procedure:
Data Usage: We select single fixed batch of data, which could drawn from the training set or any natural dataset.
Inference: The selected batch is passed through both the pruned and original models, and the output embeddings are compared.
Optimization: The goal is find the rescaling factor qe that minimizes the difference between these embeddings, ensuring that the rescaled pruned model closely approximates the original model's output.
Efficiency: This method is highly efficient as it only requires inference on single batch of data, avoiding the need for a full validation set. Algorithm Details: The specific steps for this method are outlined in Algorithm 2 with objective (2), which provides detailed procedure for determining 1/qe-
C.9 ANALYTICAL PER-LAYER 1/q
The challenge in naively implementing per-layer tuning of the rescaling factor is that if of were selected to optimize, say, the mean output change [Aglff]] averaged over all neurons in the f-th layer, this would require searches, which is computationally intractable. Instead, we turn to theory for more efficient solution.
Our approach extends Thm which bounds [|dlil|| for q = 1 - p. to obtain high-probability bound for arbitrary q. This involves bounding both the mean (which is now non-zero) and the variance, then optimizing this bound over q for each layer. In view of Thm. [.I] is unsurprising that the bound depends on the first- and second-order statistics of the coefficients Cij- Consequently, the optimal q can indeed vary per layer, depending on these statistics.
We need a high probabbibybi bound that holds for all values of 9, even & becomes small (on the order of, but still larger than, 1 - p). In more detail, following the machinery of Thm. 3.1 to bound [A J. we apply Markov's inequality to the Laplace transform Efe")s""") "I], O. After some careful calculations (see App. E.2h. for each fixed 7) we can define g by solving a dimennonall minimization problem involving 7,
P. and the statistics of cty of the respective f-th layer (which are measurable). Specifically, this is given
24

---

Published as a conference paper ICLR 2025
follows, for some probability of failure 7 E (0, 1):
((7) == arg min log + (1 (1-) Σ Ciy + , Σ,
2
(5)
?
4q²
In typical concentration bounds, one selects the optimal y by minimizing the upper bound obtained via Markov's inequality. However, this optimal is inversely proportional to q². For the small values we are interested in here (on the order of 1 p), this makes Markov's inequality loose. Instead, we propose selecting 7 by a grid search over the mean output change of the last layer after rescaling with (n) This yields a value of 17 Tec
To give flexibility to the rescaling factors of other layers, allowing them to adjust based on the individual statis- ties of Cij. while avoiding grid searches for all layers, use the same value of 17 1/e and rescale each layer with (1e). We denote the vector of selected per-layer rescaling factors 1/q. [1/q' (ne). ........................ ..... 1//" (ne)l Additionally, by selecting y by optimizing the performance on validation set rather than the mean output change (denote this n)), can arrive an alternative implementation, 1/q.- = [1/9 (ne)...... ... 1// when labeled validation data are available.
C.10 STRUCTURAL PRUNING
Our random-based DAREx-q can also be utilized for structural DPP for enhanced hardware effi-
Table 9: Structural Pruning on BERT.
ciency (Shen et al., 2022 Yao & Klimovic 2023) this setup, we randomly select a% of the input dimensions of a MLP layer and randomly retain 5% of the DPs within the selected dimensions, achieving total structured DP pruning rate of p= a 5%. In Table 9 we use = 5, b = 20, resulting in an equivalent pruning ratio 0.99. Our results
Dataset 0.99)
SST2
COLA
DARE
51.00 (0.23)
6.25 (1,41)
Struct-DAREx-1 (1/g,)
83.40 (1.92)
39.09 (1.50)
Struct-DAREx-1-4 (1/g.)
87.64 (0.47)
53.71 (1.74)
show that DAREX-q significantly outperforms DARE, achieving over a 30% improvement. AdamR- L2 unlocks the potential of random- based DPP for structural pruning, improving DARE without L2 finetuning
by over 40%.
C.11 COMPARISON TO SPARSE FINETUNING
Sparse Finetuning (SFT) selects key weights by directly altering the fine-tuning process, using methods like iterative masking and fine-tuning achieve sparse parameterization. In contrast, our DAREX-q are primarily post-hoc techniques that prune the weights of models after they have been fine-tuned.
We compare DPP with sparse fine-tuning [uoo al.) (2020), evaluating both on the BERT-Large model. Consistent with Guo et al. (202)). we assess the performance of vanilla DARE and our DAREX-q with a rescaling factor of 1/gu. using pruning rate of p = 0.995 (retaining only 0.5% of weights). As shown in Table TO, for MRPC and SST2, vanilla DARE performs slightly below the Diff-struct method, while DAREX-Q remains competitive with Diff-struct and surpasses Diff. For STS-B and COLA, vanilla DARE underperforms, but our DAREX-q restores performance, making it competitive with Diff-struct and superior to Diff. Moreover, a key advantage of our DAREx-q is that it a post-hoc method, allowing for easy application pre-fine-tuned models, and is simpler to implement.
Next, we conduct an additional experiment to examine how our proposed srrthod-oombinng the post-hoc DARE with AdamR-La Lo comparss to sparse fine-tuning. As shown in Table 11 for pruning rate of = 0.999, DAREx-LA L2 outperforms the baseline across both datasets. It's important to note that, similar to sparse fine-tuning, AdamR- L2 modifies the fine-tuning process. However, AdamR- only replaces
25

---

Published as conference paper ICLR 2025
Table 10: Comparison DAREX-q with SFT
Table 11: Comparison DAREX-L2 with SFT
p 0.995
MRPC
STS-B
COLA
SST2
p == 0.999
MRPC
STS-B
Full-tune
91.0
86.9
61.2
93.2
Full-tune
91.0
86.9
Diff
87.0
83.5
60.5
92.5
Diff
86.2
82.9
Diff-struct
89.7
86.0
61.1
93.1
Diff-struct
88.2
85.2
DARE
89.6
0.00
53.2
90.1
DARE
70.2
0.00
DAREX-q (1/q.)
89.9
84.2
60.1
92.2
DAREX-T12 (ours)
88.5
86.3
Table 12: Similar Table
2 DAREx-q consistently provides significant gains for decoder models.
Dataset
GSMBK
Abel-7B
MetaMath-7B
WizardMath-7B
MetaMath- Qwen2-0.3B
Methods
p 0.95
p 0.99
0.95
p 0.99
p 01.5
p 0.99
0.95
P 0.99
No Pruning
58.32
65.50
54.90
42.00
WANDA
18.20
0.00
23.58
0.00
21.50
0.00
0.00
0.00
MP
15.16
0.00
22.82
0.00
12.00
14.50
0.00
0.00
DARE
37.30
0.00
58.22
0,00
47.10
0.00
0.00
0,00
DAREX-q (1/g.)
47.20
20.20
59.05
34.87
49.81
35.63
30.70
19.17
DAREX-Q qq))
44.50
20.00
59.28
32.06
50.64
34.34
29.56
18.65
DAREX-Q (1/q.) q.)
42.99
21.30
60.34
32,60
49.05
33,97
30.32
18.95
DAREX-q (1/9a)
42.84
19.63
59.28
28.05
50.64
33.58
28.80
17.66
the original AdamW optimizer while keeping the same task loss objective, making it simpler implementation compared to methods like Diff, which require additional modifications.
C.12 RANDOM-BASED METHODS GENERALLY OUTPERFORMS IMPORTANCE- BASED DPP
When DPs are not large, we show that random- aandod methods consistently outperform importance-based methods across all decoder models, as presented Table 12 with DAREX-q achieving more than a 30% improvement.
C.13 BROADER IMPACT
As discussed in Section I and demonstrated in prior work, DPP offers potential advantages for online serving, gradient communication in Federated Learning, and storage efficiency. Our systematic study of DPP methods and the proposed strategies that improve performance of existing methods can thus positively impact the usage L.MMs. We analyzed the impact of our approach on the efficiency of the machine learning system.
As stated in the introduction, high pruming rate can benefit online serving, gradient communication in Federated Learning, and storage saving. To showcase this effectiveness, we compare the delta parameter loading time, the number of communication parameters, and the storage size at various pruning rates, as detailed Table 13 Firstly, we found that time to load the sparse compressed model decreases significantly the pruning rate increases. In Federated Learning, only p% of the parameters need to be communicated, greatly enhancing communication efficiency. Additionally, when applied to storage saving, further reduce memory usage by converting sparse tensors to CSR format, resulting in only 1.7 MB and 7.5 MB of storage with a 99.9% pruning rate for BERT and RoBERTa respectively. Consequently, pruning delta parameters effectively enhance the efficiency of machine learning systems.
26

---

Published as a conference paper ICLR 2025
Table 13: Loading, Communication, Storage Efficiency with different pruning rate
Loading Time (8)
Communcctton Size (# parameters)
CSR Storage Size (MBB
Models
pruming
pruning
pruming
Na Pruming
0.3
0.39
0.309
No Pruning
0.9
0.9
0.009
No Pruning
0.9
0.30
0.099
BERT
0.143
0.120
0.035
IIM
M.ITM
417.7
108.7
1.7
RoBERIa
0.165
0.125
0.038
0.026
125M
12.3M
& 125M
0.125M
475.6
102.3
16.2
7.3
outler oupput features from different transformer layer
outier features delba from difierett NIP across
308
500
0014
6012-
100
0.00
0006
25
0.1
0034
00
30
kppds
(c)
Figure 8: Analysis Outlier Features and the AWz with SST2 fintuned RoBERTa. Mean (green line) means average of all feature dimensions. The AW* is extremely small on average.
D MASSIVE ACTIVATION AND OUTLIER FEATURES
Since mennsformer-basee models contain significantly large outliers that tend to concentrate in few embed- ding dimensions Wei et al. (2022), and LLMs exhibit massive activations with high magnitude values on unique dimensions for specific tokens Sun et al. (2024), we analyze the influence of these large magnitude values in our study.
D.1 OUTLIER FEATURE (ACTIVATION)
In this section, we empirically show the influence of outlier features. As shown in Table 7 WANDA outperforms MP on SST2, MRPC in original fintuned model, which indicates the validity of considering outlier features in DPP. Although outlier features will introduce larger activation inputs than normal features (Fig (a)), with small delta parameter change, the mean and variance of Á AWy is still small (Fig S (b-c) thus making DARE works well in delta parameter pruning. As result, DARE results (Table and outperforms that of WANDA and MP.
D.2 MASSIVE ACTIVATION
The decoder based large language models (larger than 7-B) have massive activation Sun et (024) that embrace large value on unique dimension on unique tokens (i.e. firstt token of a input). We leverage MetaMath- Ilema-7B Yu al. (2033)) which finetune Llama2-7B with math datasets. We analyze each attention layer's input activations and identified the 2533-th, 1-l-thhh1115th and 1512-th dimension as the massive activations, and shown in Fig 9 (a).
notable 2533-th and 1415-th dimension aligned with the pretrain model Llama2-7B's 's massive activation dimension Sun et al. (2024). but we have one new 1512-th dimension in MetaMath- Meatthh-lema-P- 7B. When then studied the impact of layer normalization and found the massive activations is heavily downscaled by the layer normalizaiton's 's scale multiplier, which may fail to being considered as important by Wanda. We furtherly study the influence on removing the delta parameters that correspond the those large manitude features by evaluating the results GSM8K dataset. As shown in Table [4]
27

---

Published as conference paper ICLR 2025
Norm of Massive Activation and Outlier Features
Multiplication Factor in Layer Normarization
1.2
2235
1.0
3431
2533
2415
1512
mean
1000
1076
scaling
3431
0.2
0.0
10
15
20
25
30
20
layers
layers
(a)
(b)
Figure 9: Analysis on Massive Activation and Outlier Features on MetaaMath-Ilema.
removing the 1512-th massive activation features will bring significant performance drop. As result, suggest to maintain the delta parameters related to new massive activation features.
Table 14: Influnece of the delta parameters that correspond to large manitude features. Evaluation accuracy GSM8K dataset.
ID
Prune Status
Accuracy
None
No Pruning
64.50
2533
Massive Activations
62.50
1512
Massive Activations
48,00
1415
Massive Activations
65.50
3431
Outlier Feature
61.00
2158
Outlier Feature
64.50
2350
Outlier Feature
62.50
E PROOFS
E. PROOF OF THEOREM 3.1
The proof employs Hoeffding's inequality for subgaussian random variables. However, straightforward application results in suboptimal dependence of the bound on the drop-rate p. Instead, we utilize refinement of Hoeffding's 's bound tailored for (generalized) Bernoulli random variables, developed by Kearns and Saul Kearns & Saul (2013). For large values of p. we adopt the improved bound proposed by Berend and Kontorovich Berend & Kontorovich 2013). These adaptations detailed below.
28

---

Published as conference paper ICLR 2025
Recall from Eq. ( that the entries of ddiff are given
diff
Σ
AW
GTg Σ
1
8yg
1y Σ (1-
1
5zg) (,))
1-
==1
where 8. are iid Bernoulli(1 p) random variables. Fix any i E [m]. Note iight is weighted sum of n iid random variables
A, Ay= -1 d.g)
We make the following observations about these random variables. First, simple calculation gives that they are zero-mean, i.e. [[A 0 for all j E [n). Second, the random variables are bounded satisfying Third, simple calculations yields their variance V(Ag) p/(1 p).
- Ai S
Based on these, perhaps a first attempt in bounding !hş is applying Chebychev's 's inequality: For any t > 0,
V(h
Pr(
(h diff >
21
12
Using the mean and variance calculations above, this yields with probability at least yy:
lightff
P
aW
The drawback is of course that this scales poorly with 7.
The natural way to improve the dependence of the bound on is to use stronger concentration as materialized by a Chernoff bound. Since A.s are bounded, we can immediately apply Hoeffdings inequality for bounded random variables /Vershynin 2020, Thm. 2.2.6) to find that for any > 0:
2t
Pr (Ihg
E( diff )
S exp
(I-p)² Σ Gy
Therefore, for any 7 E (0, 1) the following holds with probability at least 1 7
|n diff
1/V2 (1 - p)
Σ JE(M]
log
2
Note the significant improvement over Chebychev's bound with respect to the scaling factor 7. However, the bound remains relatively loose in terms of p for values of near the extremes of its range. To this note that as p Rs 0, for which Aij Rs 0 and V(A;,) p/(1 p) Re O. Conversely, Hoeffding's bound, which scales 1/(1 - p), does not decrease as p gets smaller. To get an improved bound, we can apply the Kearns-Saul ineqaulity (Kearns& & Saul 20133, Theorem 2), which is better suited for managing the characteristics of the distribution as p approaches extremal values.
Theorem E.1 (Kearns & Saul (2013)). Ler R3 {0,1} be iid Bernoulli(1 - Pÿ) for j E [n]. Let also constants J [m). Then, for all t
12
Pr
ag -1 p;)) >t s 2exp
ag
(6)
x2
JE(n]
where x2 ;= Eje(n) (p,) and
- 2x
(xx)
(7)
log
1=xt
29

---

Published as conference paper ICLR 2025
Hoeffding Kears
Berend Kontorevich
Figure 10: Visualization of the Hoeffding, Kearns-Saul and Berend- Kontorovich bounds. Note that the last is significantly improved over the vanilla Hoeffding bound for small/large values of p. Large values of p corresponding to large drop-rates three bounds diverge as p - This is because of the rescaling factor 1/(1 - p). However, note that the bound diverges significantly slower. This further signifies the importance of using this versus the vanilla Hoeffding bound to provide an explanation on why DARE performs well even for large values p. See proof of Thm. 3.1 for details.
particularly interesting in the DPP setting. Note also that all
Applied to setting, let Rj +- dig and aj T- Then, the sum the LHS of (6) is equal to (pp)
-jee( Ai and x2 (1-p)² Eje(n) ciy. Put together, this gives for any >> O:
(- (-
1
Pr diff
(nh
t) < 2xxp
(1-p)² p(p) E jj 22
where (pp)
1-2p
Thus, for any E (0, 1) the following holds with probability at least 1 - 7:
(1-((1-))p))
(p))
2
[nghttf
log
(8)
1-p
jEIn]
To see why this improves upon Hoeffding's bound, note that the following hold for ((p):
1. (p) 1/2 for all j0, 1]
2. (0) - 0 as p 0 or 1.
The first property shows that the bound is strictly better than Hoeffding's for all values of p. The second property shows that the bound approaches as p As O, which is intuitive since in that case Ai 0 and V(Ar)) x On the other hand, for p + 1, it easy to show that / /d(p)/(1-) p) + too. Hence, both Hoeffding and this improved bound predict that [hdiff can be large as - It turns out that the Kearns-Sa Saul bound can be further improved for p 2 1/2. Such improvement is
30

---

Published as conference paper ICLR 2025
important in our setting, since large drop-rates p >> 1/2 are of particular interest in DPP. The improved inequality is due to (Berend & Kontorovich, 2013 Lemma 5). When applied to our setting, it improves the first term in to p-I
To arrive now at the advertised statement of the theorem simply rewrite Eje(n) oj in terms of the mean and
variance of the influence statistics
AW2
1
Σ C. - +
17
7
n
==1
1=1
This completes the proof of the theorem.
For completeness, we visualize the three bounds that we discussed above in Figure 10 The lower the better, thus note how the Kearns-Saul bound improves over naive Hoeffding and the Bereend-onttrovvch bound improves over both for > 1/2. fact, this bound is tight as it can seen from the central limit theorem follows: From Eq. (2) the output change can be expressed (dropping the subscript i for convenience):
h = D jejn cy(1 1 [1 p)o,) where dy are iid Bernoulli( (1 - p) RVs, we have dropped the subscript i from Eq. (2) for simplicity and we have denoted cj = A AW Wyyy ,. Our goal to quantify the growth of the absolute output change 1h/. Note that the random variables Y, = 1 1/(1 p)6, have E[Y,] 0 and Var[Y,] cpp/(1 p). Thus, by the central limit theorem, in the limit of large n. the (normalized) output change (l/vn)(h)
(pp(1-p)a V) O G where G is standard normal. Note the scaling vpp(L - p) Eg o precisely matches the bound Thm 3.1 for p > 1/2, demonstrating that it captures the true behavior of the output change with respect p. The exponential nigh-probabilijg bound of Thm 3.1 follows by a Hoeffding-tyre bound, but as demonstrated in Fig 10, to achieve this optimal p-scaling, we use a bound by Berend and Kontorovich that improves upon Hoeffding 's bound the moment generating function for generalized Bernoulli RVs.
| is distributed
E.1.1 LOWER BOUND
E.2 FINDING RESCALING FACTOR BALANCING MEAN AND VARIANCE
This section complements Sec. C.9 that describes the per-layer version of DAREX-q. Specifically, show how to derive Eq. (5).
We study more general version of DARE that rescales with parameter q that is not necessarily set 1/(1 - p). In this case, we have Aij = 1 _ Cij. which we can rewrite as follows:
1 (1 p)
1
Aij
1
cq
Cij (ōej
p))
q
R:R
Following Kearns & Saul 2013), for any random variable X, it holds that Pr(X > 0) = 1 [
Observe that (1 + [ S v for any positive number 1) >> 0, wheren is small positive number to ensure apply Markov's inequality the
tightness of Eq. 99. Then, obtain (alternatively, but equivalently,
31

---

Published as a conference paper ICLR 2025
Laplace transform):
Pri PriE Aij > = 2 poy 1asss-ayaae 2 - -(-- ne
(9)
= Eg H
III (o, - Hu
E
Jou (8., (8, -1-p))
(11-p)))
=e - M MD II [pee (pe + (1-ppee -80
3 ( +)+ + Es o (pp) Kearns & Saul 2013 Lemma 1).
(10)
Since # (p) 2 0, choosing the optimal 1)
2 3 p13-5 Σ, cg
minimizes Eq. I0 However, this may result in
-2(E, Ni a large especially for small q that is of interest to us. When 1y 41p)
for all <> 0, we have:
Σ, +
Pri Σ A c 2e
(11)
Since Maj = (1 [1 p)) Caj. then with probability 1 1-7, we have:
(1 - p) E, at //2 (oog(( city (p)) if (1- p) E3 Cij - 0 ( 1 (1 - p)) Σ, Eg 1 (og(())) E,
(q))=
Cy. if (1 (1-1 [1 p) E3 Cij -55
Since 17 2 0, the second condition is always satisfied. Therefore,
If
log
ΣS city (p) 0, then - p is simply the optimal rescaling.
If
log
2
Σ, (p)) > 0, we can increase q to compute the optimal scaling factor:
| 1 - p) Σ ci +
= arg min
log (p() Σ c32
3
arg min
Σ cas +
2 logl (p(pp)
Edd c4 (1 - p) Σ csS
?
Then the min value can be obtained by taking
10g( (2)) E, 2t
q = 1--p
E,0t Cij
32

---

Published as conference paper ICLR 2025
Curves
(LB-3)
Similar Training Loss Curve
duta weihht Encry
decay veigtt dticey
viggt 2000
30
6.5
022
01.
20
120
1.0
350
Figure 11: The training loss curves delta decay and weight decay on the MRPC, COLA, and STSB datasets using BERT are compared. Both delta decay and weight decay exhibit similar training loss dynamics.
If EE 0 and V log (2) (()) E, oj 11- E, Cij. since the optimal q > 0, this suggests that
increasing q towards infinity is favorable. However, given 7) -2(E, mo-=) 7) also tends to infinity, causing
Σ, T
the loosening of Eq. 99 Thus, we propose selecting a fixed 17, ensure the overall consistency of the inequalities in Eq. (9) and Eq. (TO). Specifically, in our implementation of DAREX-q with per-layer tuning, W) is computed grid search (see Sec.
C.9). Substituting 17 = 1/+ into Eq. (TO). we obtain:
6 log + +1+
(1 1
1
>)}
p)
Σ Cij + 2 (p) ct
(12)
4q²
Given 7h we can find the optimal to minimize |e| in Eq.
12 which arrives at Eq. (5)
F DAREX-L2 TRAINING ANALYSIS
this section, we analyze the training dynamics AdamR, with focus on DAREX-L2- To align with the weight decay terminology and for simplicity, refer our DAREX-L, L2 as "delta decay" in this section.
Concretely, we empirically inspect the training loss curve of delta decay and compare with the (vanilla) weight decay. To be specific, we finetune pretrained BERT-base model MRPC, COLA and STSB datasets by using weight decay and delta decay respectively, then we plot the training loss curves for three fine-tuning tasks under different regularization strengths.
Similar training dynamic to weight decay. When using a reasonable regularization strength, as shown in Fig. [T delta decay maintains stability and convergence on par with standard weight decay. This supports that delta decay having similar convergence and training dynamics.
Large regularization. We further increasing the regularization weight 1e-2 2 and demonstrate the results in Fig 12 The results show that delta decay maintains stable training under a large regularization strength, whereas standard weight decay struggles and fails converge.
G ADDITIONAL EXPERIMENTS ON SCALABILITY AND APPLICABILITY
In this section, conducted extra experiments show method's strong scalability and applicability.
33

---

Published as conference paper ICLR 2025
Accommodate Large Regslurizatio
with lled) welt decay
Smoothed Loss Curves COLA with 11e22
Smroothed Loss Curves on STSB BEAT (Le-2)
27
wogredoo
3
3.4
8.-
02
120
340
Emidoon
Saraton
Figure 12: The training loss curves for delta decay and weight decay on the MRPC, COLA, and STSB datasets using BERT are analyzed under higher regularization strength. Delta decay is able to adapt to a larger regularization weight, whereas weight decay fails.
Table 15: Performance CIFAR-100-ViT and Meta-Math-1331 13B models.
Dataset
CIFAR- 100
GSM8K
CIFAR 100- ViT
MetaMath- 13B
Methods
p p= 0.9
p 0.99
p = 0.9
p 0.99
No Pruning
91.98
72.32
DARE
90.53
32.74
67.09
0.00
DAREX-Q (1/qx)
91.68
84.11
68.84
44.73
DAREX-L2
91.01
83.84
G.1 PERFORMANCE COMPUTER VISION TASK
We evaluated our method on computer vision tasks using a Vision Transformer (ViT) Dosovitskiy 2020
model pretrained ImageNet-211 Ridnik al. 2222]) and fine-tuned the CIFAR-100 dataset. As shown in Table T5| our approach consistently outperforms DARE across all scenarios, demonstrating substantial performance gains. Most notably, achieves an improvement of over 50% a pruning rate of 0.99 compared to baseline methods. These findings highlight the versatility of our method, showcasing its effectiveness not only in NI.P tasks but also in the domain of computer vision.
G.2 PERFORMANCE ON LARGER MODELS
We extended our method to the larger Llama2-13B model, fine-tuned the MetaMath dataset Yu et al. (2023b), and assessed its performance on GSM8K. Despite computational constraints that limited us from tine-tuning the 13B model, DARq consistently outperformed other methods at both pruming rates (0.9 and 0.99), as detailed in Table 15 Notably, at pruning rate of 0,99, DARq achieved an impressive performance improvement of over 44 %. These results, alongside the consistent gains observed across different tasks (e.g., computer vision and NLP) and model scales (e.g., 0.5B, 7B and 13B), highlight the scalability and versatility of our method and demonstrate its potential for even larger models.
34

---

Published as conference paper ICLR 2025
Table 16: Performance of AdamR-L2 on LoRA.
p 0.99
SST2
COLA
MRPC 49.77
STSB
DARE
50.57
3.62
5.23
AdamR-L2+DARE
87.73
42.21
77.82
85.93
G.3 LORA WITH ADAMR-L2
This section presents the results of DARE on LoRA tniing and AdamR-L2 regularized LoRA tuning for SST2, COLA, MRPC, and STSB a pruning rate of p 0.99. As shown in Table 16 AdamR-L2 significantly improves DARE's performance across all tasks, achieving over 30% improvement on each task and an impressive 80% gain on STSB.
35


Reviews:
[{"review_id": "pESRXp1Eom", "reviewer": "Reviewer_S9Bb", "summary": "The paper proposes improved algorithms for Delta-Parameter Pruning (DPP), notably DARq and AdamR, which achieve substantial performance gains at high pruning rates.   It begins with a comprehensive analysis of DARE, an existing DPP approach, and leverages insights from this analysis to develop two enhanced algorithms. Furthermore, the proposed DARq method can be integrated with a parameter-efficient fine-tuning strategy to further minimize delta parameters. Additionally, the paper provides guidance on applying DPP across various scenarios.", "questions": "1 could you explain why you present a different range of regulation strength instead of a unity strength range to better understand the trend of the scale of dp in the Figure 6 ?\n\n2 how do you conduct the experiments on the different regulation strengths in Figure 4? do they keep all other hyperparameters the same?\n\n3 have you try lora + adamr-l2/adamr-l1?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 The  paper provides a detailed analysis of DARE about the failure cases \n\n2 base on the observation of DARE, they proposed two improved methods\n\n3 their present the effectiveness of their method through theoretical and experimental points.\n\n4 good presentation for reading \n\n5 consider different scenarios", "weaknesses": "see questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dC3S1Tsl1w", "reviewer": "Reviewer_CUxB", "summary": "This work makes progress over DARE to obtain Prunable Delta Paramters at higher pruning rates. Towards this, the authors propose DARq, that modifies, the rescaling factor based on the insight that it is more prudent to look at output layer perturbations when pruning DPs as opposed to worrying about the expected change in delta parameters. The authors also propose to use AdamR, different from AdamW since it uses regularisation based on gradient norm, to obtain DPs that are more prunable than those obtained by AdamW. The authors also experiment with importance based pruning methods to show that they can obtain better result than random pruning. They also show that DARq works with structural pruning, LoRA.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "S1. The problem considered is quite interesting and not well-addressed in the literature\n\nS2. The paper is well written\n\nS3. Empirical results evidently show that DARq is performs quite well. The other insights provided are valuable too.", "weaknesses": "W1. This paper seems quite exploratory. The authors do not conclude by providing a single method that they recommend to use for obtaining DPPs.\n\nW2. Conclusions derived from Theorem 3.1 seem misleading. The authors provide an upper bound as opposed to a lower bound. On line 229, authors claim \"from Thm. 3.1 as p increases, the absolute difference in outputs $|h^{diff}_{i} |$ grows at a rate of $\\mathcal{O}((1 − p)^{-\\frac{1}{2}})$, which is incorrect. But rather, $|h^{diff}_{i} |$ grows at most by $\\mathcal{O}((1 − p)^{-\\frac{1}{2}})$. It's not clear how tight this bound is to precisely make subsequent analysis hold true.\n\n\n\nTypos:\nLine 137: powerful", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QvCqRBsYGy", "reviewer": "Reviewer_xpsx", "summary": "The paper addresses limitations in the random-rescaling Delta-parameter pruning method (DARE), which struggles with high pruning rates or large mean/variance in delta parameters. To tackle this, the authors propose two new approaches: (1) DARq, a post-training rescaling modification strategy, and (2) AdamR, a DDP-aware fine-tuning method. They validate the effectiveness of these methods through experiments on various model architectures and datasets.", "questions": "* In Table 3, could you elaborate on why global rescaling outperforms local rescaling?\n* In Table 5, after a 0.99 pruning rate, the pruned model seems to be able to achieve better performance than the original and AdamR-L2 fine-tuned models. For instance, in the RoBERTa-COLA cell, it reaches 59.01 at a 0.99 pruning rate with a standard deviation of 1.26, indicating that it can outperform the unpruned model (59.27) in optimal test runs. Does this suggest the potential for further pruning? Could you experiment with even higher pruning rates to assess the method's limits?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This work provides a clear analysis of why DARE struggles with high pruning rates and large delta parameter mean/variance cases. Additionally, it includes a theoretical proof of DARq’s effectiveness as a modified rescaling strategy.\n* The paper presents comprehensive experiments on both encoder- and decoder-based language models across multiple datasets. DARq, with various rescaling variations, consistently outperforms other state-of-the-art methods across models and datasets, demonstrating its robustness. Moreover, DARq can be combined with parameter-efficient tuning approaches, such as LoRA, broadening its applicability.\n* The AdamR fine-tuning method achieves promising results, effectively maintaining performance even at high pruning rates.", "weaknesses": "* This paper only compares their method with DARE, but there are lots of other methods proposed in the neural network pruning community. I wonder how this method compares to some existing pruning methods. For example, measuring the importance of weights in the original model and pruning the delta parameters based on that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VtJNM26OJy", "reviewer": "Reviewer_3h3b", "summary": "The authors revisit delta-parameter pruning (DPP) for fine-tuned models, focusing on the limitations of the existing Random Drop and Rescale (DARE) method when dealing with high pruning rates or large delta parameters. They identify two key issues causing DARE's performance degradation: the excessively large rescaling factor at high pruning rates and the high mean and variance in the delta parameters. To address these challenges, they propose two algorithmic improvements:\n\n1. **DARq**: A modification of DARE that adjusts the rescaling factor, leading to significant performance gains at high pruning rates.\n2. **AdamR**: An in-training regularization technique that incorporates delta (L2) regularization during fine-tuning to control the mean and variance of delta parameters before applying DPP.\n\nThey demonstrate that their methods outperform the original DARE, especially at extreme pruning rates, and can be combined with parameter-efficient fine-tuning techniques like LoRA. Additionally, they show that importance-based pruning methods can outperform random-based methods when delta parameters are large.\n\nThe findings of this work seem very high utility and are of substantial increase in performance in edge cases where the original methods failed very strongly. A very good step forward for DPP methods.", "questions": "1. **Scalability and Applicability to Other Domains**: While your methods have shown impressive results on language models, have you considered how DARq and AdamR might scale to even larger models or different architectures beyond NLP? Specifically, do you anticipate any challenges or necessary modifications when applying these techniques to other domains such as computer vision or speech recognition?\n\n2. **Automated Selection of Rescaling Factor in DARq**: The selection of the rescaling factor \\( q \\) in DARq appears to be crucial for maintaining performance after pruning. Have you explored automated methods for selecting \\( q \\) without relying on a validation set or manual tuning? For instance, could analyzing the statistics of the delta parameters during pruning provide a way to determine an optimal \\( q \\) more systematically?\n\n3. **Impact of AdamR on Training Dynamics**: Introducing delta regularization through AdamR alters the fine-tuning process. How does this affect the convergence, stability, and overall training dynamics compared to standard fine-tuning methods? Are there any trade-offs, such as increased training time or complexity, that practitioners should be aware of when integrating AdamR into their training pipelines?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- **Comprehensive Analysis**: The paper provides a thorough analysis of the limitations of the existing DARE method, identifying the root causes of its failure at high pruning rates.\n- **Innovative Solutions**: The proposed methods, DARq and AdamR, are well-motivated and offer practical improvements over existing techniques.\n- **Empirical Validation**: Extensive experiments on both encoder and decoder models across various datasets showcase the effectiveness of the proposed methods, with significant performance gains.\n- **Flexibility and Compatibility**: The authors demonstrate that DARq can be combined with existing fine-tuning techniques like LoRA and can be used for structural pruning, increasing its practical utility.\n- **Revisiting Importance-Based Pruning**: By exploring the application of importance-based pruning methods within DPP, they offer valuable insights into scenarios where these methods may be more effective.\n- **Clear Recommendations**: The paper provides a practical pipeline for selecting the appropriate DPP method under different scenarios, which is valuable for practitioners.", "weaknesses": "- **Complexity of Methods**: While the proposed methods are effective, they introduce additional complexity, such as tuning the rescaling factor \\( q \\) in DARq, which may require validation data or additional computations.\n- **Limited Theoretical Depth**: The theoretical analysis, while helpful, could be further deepened to provide more rigorous guarantees or insights into why the proposed methods work.\n- **Applicability Scope**: The methods are primarily validated on language models and NLP tasks; it would strengthen the paper to demonstrate their applicability to other domains or types of neural networks.\n- **Clarity in Presentation**: Some sections, particularly the methodology and theoretical analysis, could benefit from clearer explanations and more intuitive descriptions to make them accessible to a broader audience.\n- **Dependence on Regularization**: The AdamR method relies on in-training modifications and specific regularization strategies, which may not always be feasible or may interact unpredictably with other training dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)