Revisiting Delta-Parameter Pruning For Fine-Tuned Models

## Abstract

Abstract Storing open-source fine-tuned models separately introduces redundancy and increases response times in applications utilizing multiple models. Delta-parameter pruning (DPP), particularly the random drop and rescale (DARE) method proposed by Yu et al., addresses this by pruning the majority of delta parameters—the differences between fine-tuned and pre-trained model weights—while typically maintaining minimal performance loss. However, DARE fails when either the pruning rate or the magnitude of the delta parameters is large. We highlight two key reasons for this failure: (1) an excessively large rescaling factor as pruning rates increase, and (2) high mean and variance in the delta parameters.
To push DARE’s limits, we introduce DAREx (DARE the eXtreme), which features two algorithmic improvements: (1) DAREx-q, a rescaling factor modification that significantly boosts performance at high pruning rates (e.g., > 30 % absent percent 30 >30\% > 30 % on COLA and SST2 for encoder models, with even greater gains in decoder models), and (2) DAREx- L 2 subscript 𝐿 2 L_{2} italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , which combines DARE with AdamR, an in-training method that applies appropriate delta regularization before DPP. We also demonstrate that DAREx-q can be seamlessly combined with vanilla parameter-efficient fine-tuning techniques like LoRA and can facilitate structural DPP. Additionally, we revisit the application of importance-based pruning techniques within DPP, demonstrating that they outperform random-based methods when delta parameters are large. Through this comprehensive study, we develop a pipeline for selecting the most appropriate DPP method under various practical scenarios.

## 1 Introduction

Large Language Models (LLMs) like BERT (Devlin et al., ), GPT (Floridi & Chiriatti, ), and Llama (Touvron et al., ) excel in various language-modeling tasks. Finetuning these for specific downstream tasks enhances personalized experiences (Sanh et al., ; Labrak et al., ) and has become a standard practice in natural language processing (Dodge et al., ; Zhao et al., ). Indeed, most entries on the OpenLLM Leaderboard involve full-parameter finetunes or their combinations (Liu et al., ), underscoring the widespread adoption and availability of fine-tuned models online.
Decomposing fine-tuned model weights into the original parameters of the pre-trained model yields delta parameters (DP) (Yu et al., ; Liu et al., ; Yao & Klimovic, ).
Reducing the size of DPs, which are as large as the base model and can number in the hundreds of millions of parameters for LLMs, could significantly enhance communication efficiency in federated learning, minimize task conflicts in model merging, accelerate multi-task serving, and decrease storage needs for new fine-tuned models (see *Related Work*).
Delta-parameter pruning (DPP) drops a fraction $p$
of the DPs towards realizing these benefits. Naturally, DPP can be seen as an instance of generic model-parameter pruning, which compresses neural networks by eliminating weights that contribute minimally, resulting in sparse architectures (LeCun et al., ; Han et al., ). Traditional pruning methods typically remove weights post-training based on *importance criteria* like weight magnitude or activation levels. While these techniques could naturally extend to DPP, their integration into this context remains largely unexplored.
Random-based pruning strategies, which provide more flexibility and efficiency in implementation, also offer a competitive alternative. For instance, Random Drop and Rescale (DARE) (Yu et al., ), a recently introduced randomized DPP method, reduces DP size through random pruning followed by rescaling. DARE has been quickly adopted across various applications, including model merging libraries (Goddard et al., ), state-of-the-art medical LLMs (Labrak et al., ), and Japanese LLMs (Akiba et al., ).
However, as we demonstrate in this paper, DARE struggles when the pruning rate is high or when DPs are large. This observation prompts several key questions:

### 1.1 Contributions

**Table 1: *Impact of our key contributions on BERT and two datasets COLA and SST2*: ‘DAREx-$L_{2}$ ’ applies DARE after AdamR-$L_{2}$ fine-tuning, ‘$L_{1}$+MP’ applies magnitude pruning after AdamR-$L_{1}$, the factors $q_{v}$ and $q_{e}$ adjust DARE’s rescaling using a validation set and unlabeled data, respectively. All four proposed methods significantly outperform vanilla DARE (Yu et al., ), demonstrating that (randomized) DPP can still be highly effective, even at extreme pruning rates of 99% parameter reduction, when incorporating our modifications. Notably, our new post-hoc rescaling achieves a 40-fold model size reduction with minimal impact on performance compared to the unpruned model. See Tables 2,3 for more details.**
| Pruning Rate | $p=0$ | $p=0.99$ |
| --- | --- | --- |
| Delta Size | 417.7MB | 11.4MB |
| Method | COLA | SST2 |
| ($p=0.99$) | Test Performance (%) | Test Performance (%) |
| No Pruning | 56.24 | 90.25 |
| DARE (Yu et al., ) | 4.25 | 51.00 |
| $L_{1}$+MP (Han et al., ) (ours) | 12.30 (+8.05) | 83.14 (+32.14) |
| DAREx-$L_{2}$ (ours) | 57.24 (+52.99) | 88.17 (+37.00) |
| DAREx-q ($1/q_{v}$) (ours) | 48.96 (+44.71) | 85.64 (+34.64) |
| DAREx-q ($1/q_{e}$) (ours) | 45.20 (+41.05) | 85.81 (+34.81) |

We address these questions through principled analysis and extensive experimentation on large-scale language models and datasets. Our contributions are summarized as follows:

$\bullet$ Analysis of DARE: By examining the *absolute change in intermediate output model representations* resulting from the application of DARE to the DPs, we identify two primary factors that influence this change: (a) a large pruning rate ($p$), which results in an excessively high rescaling factor of $1/(1-p)$, and (b) a high mean and variance of the DPs relative to input activations. Through experiments on both controlled setups and LLMs, we show that the absolute change in intermediate representations is a reliable proxy for test performance (see Fig. 1). Thus, these two factors emerge as the main contributors to DARE’s failure modes. This analysis inspires two new algorithms that significantly improve DARE’s performance, as follows.

$\bullet$ Drop and rescale with $1/q$ (DAREx-q): To ensure efficiency at high pruning-rates (e.g., $p>0.9$), where DPP offers significant savings, our first algorithm DAREx-q modifies DARE’s rescaling factor from $1/(1-p)$ to $1/q$, where $q>1-p$. While (Yu et al., ) recommended using the factor $1/(1-p)$ to maintain zero-expectation changes in intermediate representations (accounting for DARE’s randomness), they neglected the importance of controlling the *absolute* changes and the impact of randomness, particularly since the pruning is applied only once.A large rescaling factor can inflate the variance of these changes; therefore, by tuning the rescaling factor $1/q$, we can effectively balance the mean and variance, thereby minimizing performance degradation. We develop and evaluate four variants of DAREx-q (detailed in Sec. 4.1) to suit different scenarios along two orthogonal axes: (a) whether a *labeled* validation set is available and (b) whether to tune a global $1/q$ across all layers or a per-layer $1/q$. All four methods provide rescaling factors that significantly improve upon vanilla DARE while preserving its purely post-training nature, demonstrating that effective DPP remains possible even at high pruning rates.

$\bullet$ AdamR fine-tuning: Our second algorithm controls the mean and variance of DPs (averaged over input features at each intermediate layer) prior to applying DARE. Specifically, our AdamR, a modified Adam optimizer, applies $L_{2}$ regularization directly to the DPs during fine-tuning. Combining AdamR with DARE (DAREx-$L_{2}$) yields highly competitive performance (see Table 1). While this requires in-training modification, it significantly broadens the scenarios where randomized DPP can be effectively applied.

$\bullet$ Extensive experiments: We conduct extensive experiments on both encoder-decoder and decoder-only models across a range of downstream tasks. The results demonstrate the effectiveness of both DAREx-q and DAREx-$L_{2}$ algorithms. As summarized in Table 1, applying these techniques to a fine-tuned BERT model on the CoLA and SST-2 datasets leads to substantial performance improvements, consistently exceeding 35%. Additional results are presented in Tables 2,3,5,7 and Figures 2,3.

$\bullet$ Importance-based DPP: We incorporate importance-based methods such as magnitude-pruning and WANDA into DPP.
Our analysis reveals that while random-based DPP generally outperforms important-based DPP, especially under higher pruning rates, combining the latter with AdamR-$L_{1}$ regularization (as opposed to $L_{2}$) forms a competitive alternative (see Tables 1 and  7). Moreover, importance-based DPP can outperform random-based DPP when DPs are large and customized fine-tuning is not an option.

Figure: (a) Last-layer output changes on one batch of train data.
Refer to caption: extracted/5917509/images/output_Change_099_v3.png

## 2 Related work

Delta-parameter pruning. DPs represent the parameter changes induced by fine-tuning and are critical to various operations. In Federated and Distributed Learning, DPs are continuously exchanged between a central server and distributed clients during training (Li et al., , ; Khirirat et al., ; Chen et al., ). In model merging, the goal is to merge multiple task-specific models, fine-tuned from the same pretrained backbone, by combining their DPs into a single model with diverse capabilities (Wortsman et al., ; Ilharco et al., ). Similarly, in multi-task serving, DPs are processed separately while the base model weights are computed once during the forward pass (Yao & Klimovic, ; Liu et al., ). (Yu et al., ; Liu et al., ; Yao & Klimovic, ) focus on reducing the size of DPs, and recently(Yu et al., ) has popularized DARE as a poerful random-based DPP method. However, DARE fails with large DPs or high pruning rates. We propose techniques that, applied during both fine-tuning and post-hoc stages, restore performance and significantly improve upon vanilla DARE. Our results demonstrate, for the first time, that DPP remains effective even at extreme pruning rates, reducing up to 99% of the model parameters. Thanks to our modification, we also show that random-based DPP can be combined with LoRA and can be used for structured pruning.
Model parameter pruning.
There are two primary approaches to model parameter pruning: regularization-based and importance-based methods; however, an exhaustive review is beyond the scope of this paper. Among importance-based methods, early works (LeCun et al., ; Hassibi & Stork, ) introduced importance scores based on second-order information from the loss function. In deep neural networks (DNNs), methods such as (Han et al., ; Li et al., ; Han et al., ) utilize magnitude pruning to eliminate weights based on their magnitude. More recently, (Sun et al., ) developed an importance score called WANDA, which is proportional to the product of weight magnitude and the norm of the input feature, specifically targeting LLMs. However, the application of these techniques to DPP remains poorly understood. We address this gap by studying both the classical magnitude pruning method and the recently proposed LLM-targeted approach, WANDA, within the context of DPP.

## 3 Delta parameter pruning

DPP aims to efficiently store a fine-tuned model $\bm{\theta}_{F}$ initialized from a pre-trained base model $\bm{\theta}_{P}$ by pruning the difference known as the ”delta parameter” (DP), defined as $\bm{\Delta}_{\bm{\theta}}=\bm{\theta}_{F}-\bm{\theta}_{P}$ between the fine-tuned and the base model.
Successful DP pruning (DPP) removes parameters from $\bm{\Delta}_{\bm{\theta}}$ resulting in reduced storage requirements while minimally compromising performance.
Sec. 3.1 presents the theoretical model used to analyze various DPP methods. Sec. 3.2 and Appendix A provide a theoretical analysis of randomized and importance-based DPP methods, respectively. These theoretical insights are tested through a detailed controlled experimental analysis using a two-layer neural network in App. B, and, more importantly, applied in Sec. 4 to guide the design of new algorithms that significantly improve LLM performance in Sec. 5.

### 3.1 Formulation

To gain analytical insights, we consider the application of DPP to perceptron operations in a neural network. This focus is partly motivated by the observation that multi-layer perceptron (MLP) operations, both in the projection layers of attention mechanisms and in feed-forward networks, constitute the majority of operations in LLMs. Consider a perceptron layer, where $\bm{x}\in\mathbb{R}^{n}$ is the input activation vector connecting to $m$ output neurons, $\bm{W}\in\mathbb{R}^{m\times n}$ is the hidden-layer weight matrix, and $\sigma$ is a Lipschitz activation non-linearity (e.g., ReLU, sigmoid).(^1^11Bias terms can be included by augmenting the input and DPs to $\bm{x}^{\prime}=[\bm{x},1]$ and $\bm{\Delta}_{\bm{W}}^{\prime}=[\bm{\Delta}_{\bm{W}},\bm{\Delta_{b}}]$, respectively, but recent LLM architectures often omit bias terms for training stability (Touvron et al., ; Chowdhery et al., ))
Let $\bm{\Delta}_{\bm{W}}\in\mathbb{R}^{m\times n}$ denote the delta parameters of the weight matrix. A pruning mapping $\mathcal{P}:\mathbb{R}^{m\times n}\rightarrow\mathbb{R}^{m\times n}$ aims to reduce the parameters in $\bm{\Delta}_{\bm{W}}$, leading to a sparsified $\mathcal{P}(\bm{\Delta}_{\bm{W}})$. The desired sparsity level, $k\in[m]$, is a critical design specification for DPP, often dictated by factors such as storage constraints relative to the number of fine-tuned models. For randomized DPP, elements of $\bm{\Delta}_{\bm{W}}$ are dropped with a probability $p\in[0,1]$, resulting in an average sparsity level of $k=(1-p)mn$. The objective is to prune the delta parameters to a sparsity level $k$ (or $1-p$ for randomized DPP) while maintaining the performance of the pruned model, represented by $\bm{W}_{P}+\mathcal{P}(\bm{\Delta}_{\bm{W}})$, close to that of the original model $\bm{W}_{P}$. To quantify the impact of pruning, we define the difference vector $\bm{h}^{\rm{diff}}$, which measures the change in intermediate output model representations between the pruned and original models:

$$ $\displaystyle\bm{h}^{\rm{diff}}:=\bm{\Delta}_{\bm{W}}\bm{x}-\mathcal{P}(\bm{ \Delta}_{\bm{W}})\bm{x}.\vspace{-0.05in}$ (1) $$

We require that output after pruning $\sigma\left((\bm{W}_{P}+\mathcal{P}(\bm{\Delta}_{\bm{W}}))\bm{x}\right)$ does *not* differ much from the output before pruning $\sigma\left((\bm{W}_{P}+\bm{\Delta}_{\bm{W}})\bm{x}\right)$. By Lipschitzness of the activation, this translates to each entry $h_{i}^{\rm{diff}}$ of the difference vector $\bm{h}^{\rm{diff}}$ being small in absolute value. Thus, we study how $\mathcal{P}$ affects $|h_{i}^{\rm{diff}}|,i\in[m]$.

### 3.2 Randomized DPP: Random Drop and Rescale (DARE)

DARE randomly sets each element of $[\mathcal{P}(\bm{\Delta}_{\bm{W}})]_{ij}$ to zero (dropped) with probability $p$, and *rescales* non-zero elements to $[\mathcal{P}(\bm{\Delta}_{\bm{W}})]_{ij}=[\bm{\Delta}_{\bm{W}}]_{ij}/(1-p)$(Yu et al., ).
While DARE has been tested on a wide range of fine-tuned models across various tasks, it exhibits performance degradation under high pruning rates, as shown in Table 7 and Table 8.
We seek the underlying causes of these limitations and explore potential improvements.
Our approach involves quantifying the difference vector $\bm{h}^{\rm{diff}}$ in relation to key variables: the dropout rate, the DPs, and the input data. As per Eq. 1, the effect of DARE on the output becomes

$$ $\displaystyle h_{i}^{\rm{diff}}=\sum\nolimits_{j\in[n]}\Delta W_{ij}x_{j}-\sum \nolimits_{j\in[n]}\frac{1}{1-p}\delta_{ij}\Delta W_{ij}x_{j},\vspace{-0.1in}$ (2) $$

where $p$ is the *drop rate*, $\{\delta_{ij}\}_{i\in[m],j\in[n]}$ are iid $\rm{Bernoulli}(1-p)$ random variables, and $1/(1-p)$ is the *rescaling factor*. We denote $\Delta W_{ij}$ the $(i,j)$ entry of $\bm{\Delta}_{\bm{W}}\in\mathbb{R}^{m\times n}$.
It is easy to see that $\mathbb{E}[h_{i}^{\rm{diff}}]=0$ (expectation over $\delta_{ij}$), and Yu et al. () use this as justification for DARE’s effective performance. However, this alone overlooks the sources of DARE’s failure modes, which we reveal by instead establishing bounds on the *absolute* output changes $|h_{i}^{\rm{diff}}|$. The following theorem addresses this by providing a high-probability bound on these changes, with respect to DARE’s randomness, which arises from the Bernoulli variables. See App. E.1 for the proof.

###### Theorem 3.1 .

Denote $h_{i}^{\rm{diff}}$ as the $i$-th component of $\bm{h}^{\rm{diff}}$ in Eq. (2). For $i\in[m],j\in[n]$, let $c_{ij}=\Delta W_{ij}x_{j}$ represent the change in influence of the $j$-th feature on the $i$-th output neuron after fine-tuning. Define(^2^22We call ‘mean’/‘variance’ here the first/second -order summary statistics of $\{c_{ij}\}_{j\in[n]}$ over the input dimension. On the other hand, the theorem’s ‘high-probability’ statement is w.r.t. the randomness of random variables $\{\delta_{ij}\}_{j\in[n]}$) the mean $\bar{c}_{i}$ and variance $\sigma_{i}^{2}$ of these as:
$\bar{c}_{i}=(1/n)\sum\nolimits_{j\in[n]}c_{ij}$ and $\sigma_{i}^{2}=(1/n)\sum\nolimits_{j\in[n]}(c_{ij}-\bar{c}_{i})^{2}.$
Then, for any $i\in[m]$ and $\gamma\in(0,1),$ it holds with probability at least $1-\gamma$ that

$$ $|h_{i}^{\rm{diff}}|\leq{\left({\Psi(p)}/({1-p})\right)}\sqrt{n(\bar{c}_{i}^{2} +\sigma_{i}^{2})}\sqrt{\log\left({2}/{\gamma}\right)}\,\vspace{-0.05in}\,,$ $$

where $\Psi(p)=(1-2p)/\log((1-p)/p)$ if $p\leq 1/2$, otherwise $\Psi(p)=\sqrt{2p(1-p)}$.

Thus, the key factors influencing the magnitude of $|h_{i}^{\rm{diff}}|$ are: (i) the rescaling factor $1/(1-p)$ and (ii) the mean and variance (averages are over input dimension $j\in[n]$) of the influence parameters $\{c_{ij}\}$. Specifically, increasing the rescaling factor (eqv. increase drop-rate $p$) increases $|h_{i}^{\rm{diff}}|$ at a rate of $\mathcal{O}((1-p)^{-\frac{1}{2}})$, which can be large in demanding pruning scenarios when $p$ is large. Also DARE yields smaller $|h_{i}^{\rm{diff}}|$ values when $c_{ij}=\Delta W_{ij}x_{j}$ exhibit low first (mean) and second order (variance) averages.
We validate these conclusions through elaborate experiments on a controlled setup in Appendix  B. More importantly, our algorithms in Sec. 5, further tested on LLMs, are also built upon the insights from this theorem and its observations.

###### Theorem 3.1 .

### 3.3 Importance-based DPP

Using the same analytical framework, we extend the application of importance-based pruning methods, specifically magnitude pruning and WANDA, to the DPP setting. Due to space constraints, we provide a detailed discussion in App. A. In brief, importance-based DPP can achieve strong performance when the distribution of the coefficients $c_{ij}$ exhibits light tails and sharp-peakedness. We provide empirical validation of these findings in Sec. C.4 and Appendix B.

## 4 Algorithms

To extend the applicability of DPP at high pruning rates and in cases where DPs of fine-tuned models exhibit undesirable statistics, we propose here two strategies based on the theoretical insights discussed in Sec. 3.

### 4.1 Adjusting the rescaling factor

Motivation. Recall from Thm. 3.1 as $p$ increases, the absolute difference in outputs $|h_{i}^{\rm{diff}}|$ grows at a rate of $\mathcal{O}((1-p)^{-\frac{1}{2}})$. Setting the rescaling factor to $q=1-p$ eliminates the mean of $h_{i}^{\rm{diff}}$, but does not minimize its magnitude.
Through empirical analysis, we demonstrate in Fig. 1 how various values of $q$ influence model outputs in terms of both mean output change (i.e., the average of $|h_{i}^{\rm{diff}}|$ across all last-layer neurons) in Fig. 1(a), and, test performance in Fig. 1(b) across different datasets. For
99% pruning rate ($1-p=0.01$) we observe these quantities over a range of $q$ values from $0.005$ to $0.04$. The results depicted in Fig. 1(a) for mean output change, indicate a convex-like trend, with minimum points (marked with asterisks) achieved at $q$ values higher than the vanilla $1-p$. E.g., the optimal $q$ for COLA and SST2 is $\approx 0.018$ and $0.033$, respectively. Recall now that in our analysis, the amount of change in output activations is used as a proxy for test performance: smaller mean output change corresponds to better performance. Fig. 1(b) validates this: the test-performance curves show a concave-like shape, with the peak performance also occurring at $q>1-p$ and numerically consistent with the values of that minimize the mean absolute output change.

Drop and rescale with $1/q$ (DAREx-q). Motivated by these observations, we propose two variants of DAREx-q: tuning based on (1) maximizing performance on a labeled validation set, and (2) minimizing mean output change over unlabeled data.
$\bullet$ *DAREx-q with labeled validation data $(1/q_{v})$*: We use a validation dataset $\{x_{v},y_{v}\}\in\mathcal{V}$ to determine the best rescaling factor $1/q_{v}$ that maximizes test performance (eqv. minimizes test error) on the validation set. Specifically, we select $q_{v}=\arg\min_{q}\mathbb{P}_{\mathcal{V}}(f_{q}(x_{v})\neq y_{v})$,
where $f_{q}$ represents the pruned model rescaled by $1/q$. We then randomly drop a faction $p$ of the model DPs and rescale those that are not dropped with $1/q_{v}$. Further details can be found in Algorithm 2 (1) in the Appendix.
$\bullet$ *DAREx-q with unlabeled data $(1/q_{e})$*: This method selects $q$ by optimizing an unsupervised objective measuring the mean output difference $|h_{i}^{\rm{diff}}|$ across all last-layer neurons of the model. This approach is based on the observation in Fig. 1 that mean output difference is as an effective unsupervised proxy for test performance. The key advantage of this method over $q_{v}$ is that it does *not* require labeled data. Specifically, in our implementations, selecting $q_{e}$ involves running inference on a single fixed batch of data (with texts from either the training data or any other natural dataset) using both the pruned and original models. Refer to Algorithm 2 in the Appendix for further details.

DAREx-q with per-layer tuning. The implementations of DAREx-q described above select a single, global rescaling factor ($1/q_{v}$ or $1/q_{e}$) that is then applied to all surviving DPs across all layers of the network. However, since the DPs in intermediate layers each have their own mean output change over their respective output layers, a natural question arises: Can we improve performance by selecting a different rescaling factor for each layer $\ell\in[L]$? Moreover, can this be done efficiently without requiring $L$ hyperparameter searches (one for each layer), which would be computationally prohibitive?
In Appendix C.8, we demonstrate that the answer to both of these questions is affirmative. To circumvent the computational challenge of $L$ searches, we leverage theoretical insights by extending Thm. 3.1 to provide a high-probability bound for arbitrary $q$ and then optimizing over it. Like Thm. 3.1, the new bound still depends on the first- and second-order statistics of the coefficients $c^{\ell}_{ij}$, which means the optimal $q^{\ell}$ can vary per layer $\ell$ based on these statistics.
We denote this vector of per-layer rescaling factors as $1/\mathbf{q}_{v}=[1/q_{v}^{1},\ldots,1/q_{v}^{L}]$ and $1/\mathbf{q}_{e}$, respectively, for two implementations depending on whether the performance on validation set or mean output change is optimized. Please refer to App. C.8 for details. In Sec. 5.1, we show that this approach performs very well on encoder models and is comparable to the global $1/q$ approach for decoder models.

### 4.2 AdamR finetuning

Motivation. While DAREx-q addresses the issue of large rescaling factors at high pruning rates, we’ve seen that for absolute output changes $|h_{i}^{\rm{diff}}|$ to remain small, the first- and second-order averages of the influence factors $\{c_{ij}\}_{j\in[n]}$ must also be small. These factors directly depend on the magnitude of the DPs $\Delta W_{ij}$, which are determined during fine-tuning.
We propose an in-training method that fine-tunes the model to keep these statistics small, thereby guaranteeing that the post-hoc application of DARE maintains performance.

AdamR-$L_{2}$. Our approach is to fine-tune the model with $L_{2}$ regularization on the DPs $\bm{\Delta}_{\bm{\theta}}=\bm{\theta}_{F}-\bm{\theta}_{P}$. As mentioned, the motivation behind this is penalizing $\|\bm{\Delta}_{\bm{\theta}}\|_{2}$ directly translates to smaller total second-order averages $\sum_{j\in[n]}c_{ij}^{2}$, which, as previously noted, capture the extent of change in the output layer post-finetuning and factor into the bound in Thm. 3.1. To implement this, we replace the weight decay of AdamW—the standard choice for training LLMs—with our custom regularization. Specifically, we adjust the regularization strength based on the gradient norm, as Xie et al. () recently found that weight decay can cause large gradient norms during the final phase of training. Our modified decay step of AdamW is:

$$ $\displaystyle\bm{\theta}_{t}=\bm{\theta}_{t-1}-\frac{\eta}{\sqrt{\hat{\bm{v}}_ {t}}+\epsilon}\hat{\bm{m}}_{t}-\frac{\eta}{\sqrt{\bar{v}_{t}}+\epsilon}\lambda {\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}(\bm {\theta}_{t-1}-\bm{\theta}_{P})},$ (3) $$

where $\bm{\theta}_{t}$ represents model parameters at iteration $t$, $\hat{\bm{m}}_{t}$ and $\hat{\bm{v}}_{t}$ are the first and second moments of gradients, and $\bar{v}_{t}$ is the mean of the second moment used to adjust regularization. See also Algorithm 1 in Appendix.

AdamR-$L_{1}$. We extend our approach to fine-tune the model in preparation for importance-based pruning, rather than random-based pruning, post-training. In this case, we regularize using the $L_{1}$ norm of the DPs. This is again motivated by the analysis in Sec. 3.3.
The detailed algorithm, referred to as AdamR-$L_{1}$, is provided in Algorithm 1 in the Appendix.

**Table 2: DAREx-q consistently outperforms DARE across four datasets and two encoder models. The pruning rate is set to $p=0.99$. Bold indicates the best performance, while underline marks the second-best.**
| Dataset | SST2 | COLA | MRPC | STS-B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Models | BERT | RoBERTa | BERT | RoBERTa | BERT | RoBERTa | BERT | RoBERTa |
| No Pruning | 90.25 $(-)$ | 92.09 $(-)$ | 56.24 $(-)$ | 62.10 $(-)$ | 85.01 $(-)$ | 90.05 $(-)$ | 88.51 $(-)$ | 90.57 $(-)$ |
| DARE | 51.00 (0.64) | 51.81 (1.00) | 4.25 (2.83) | 7.65 (3.63) | 79.96 (6.07) | 87.32 (0.69) | 82.56 (1.13) | 81.59 (1.44) |
| DAREx-q ($1/q_{v}$) | 85.64 (3.07) | 86.56 (4.59) | 48.96 (4.13) | 53.58 (1.97) | 83.92 (0.62) | 87.30 (0.58) | 86.60 (1.61) | 88.07 (0.70) |
| DAREx-q ($1/\mathbf{q}_{v}$) | 89.60 (0.67) | 90.40 (1.36) | 53.12 (2.01) | 58.38 (2.05) | 83.86 (0.66) | 89.80 (0.60) | 87.55 (0.12) | 89.10 (0.32) |
| DAREx-q ($1/q_{e}$) | 85.81 (2.72) | 81.05 (4.96) | 45.20 (2.04) | 52.10 (5.01) | 83.12 (1.11) | 88.85 (0.26) | 87.03 (0.12) | 87.29 (0.57) |
| DAREx-q ($1/\mathbf{q}_{e}$) | 87.39 (0.86) | 89.62 (1.35) | 51.70 (2.57) | 54.43 (2.49) | 83.25 (0.94) | 89.59 (0.42) | 87.59 (0.19) | 87.60 (0.49) |

**Table 3: Similar to Table 2, DAREx-q yields significant gains for decoder models.**
| Dataset | GSM8K |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | Abel-7B | MetaMath-7B | WizardMath-7B | MetaMath-Qwen2-0.5B |  |  |  |  |
| $p=0.95$ | $p=0.99$ | $p=0.95$ | $p=0.99$ | $p=0.95$ | $p=0.99$ | $p=0.95$ | $p=0.99$ |  |
| No Pruning | 58.32 | 65.50 | 54.90 | 42.00 |  |  |  |  |
| DARE | 37.30 | 0.00 | 58.22 | 0.00 | 47.10 | 0.00 | 0.00 | 0.00 |
| DAREx-q ($1/q_{v}$) | 47.20 | 20.20 | 59.05 | 34.87 | 49.81 | 35.63 | 30.70 | 19.17 |
| DAREx-q ($1/\mathbf{q}_{v}$) | 44.50 | 20.00 | 59.28 | 32.06 | 50.64 | 34.34 | 29.56 | 18.65 |
| DAREx-q ($1/q_{e}$) | 42.99 | 21.30 | 60.34 | 32.60 | 49.05 | 33.97 | 30.32 | 18.95 |
| DAREx-q ($1/\mathbf{q}_{e}$) | 42.84 | 19.63 | 59.28 | 28.05 | 50.64 | 33.58 | 28.80 | 17.66 |

## 5 Experiments

We validate the effectiveness of our analysis and proposed algorithms through comprehensive experiments.
Datasets and models for Encoder-based and Decoder-based LMs. For Encoder-based LMs, we utilize four datasets—sentence acceptability dataset CoLA (Warstadt et al., ), sentiment detection dataset SST-2 (Socher et al., ), paraphrase dataset MRPC (Dolan & Brockett, ), and sentence similarity dataset STS-B (Cer et al., ). For the selection of pretrained backbones, we choose BERT-base-uncased (Devlin et al., ) and RoBERTa-base (Liu et al., ), fine-tuning them on these task-specific datasets to obtain supervised finetuned models. For Decoder-based LMs, we focus on mathematical reasoning tasks. Due to constraints on resources for fine-tuning, we opted to fine-tune a smaller Qwen2-0.5B (Yang et al., ) model on the MetaMath dataset (Yu et al., ). Additionally, we utilized publicly available mathematical reasoning models, including the MetaMath-llema-7B (Yu et al., ), MetaMath-7B (Yu et al., ), WizardMath-7B (Luo et al., ) and Abel-7B (Chern et al., ), all based on the Llama2-7B architecture (Touvron et al., ). We then use the GSM8K (Cobbe et al., ) to test these models.
Evaluation metrics. Following (Yu et al., ), we evaluate performance using the Matthews correlation coefficient for CoLA, accuracy for SST-2, the average of accuracy and F1 score for MRPC, the average of Pearson and Spearman correlation for STS-B, and zero-shot accuracy for GSM8K.

### 5.1 Rescaling-parameter modification

We evaluate our DAREx-q algorithm. From Sec. 4.1, recall the four proposed variations (i.e., $1/q_{v},1/q_{e},1/\mathbf{q}_{v}$ and $1/\mathbf{q}_{e}$) for choosing the rescaling factor along two orthogonal axes: (a) whether to use labeled validation data or not ($1/q_{v}$ or $1/q_{e}$, respectively); and (b) global tuning versus per-layer tuning ($1/q$ vs $1/\mathbf{q}$).

Figure: (a) Encoder models
Refer to caption: extracted/5917509/images/prune_qv_encoder_v2.png

Encoder models. Table 2 compares DAREx-q (all four variants) with vanilla DARE (with rescaling factor $1/(1-p))$. We report average performance and standard deviation on the test set over four independent runs. For reference, we also report performance of the finetuned model before pruning. We find that: (1) *All* DAREx-q variants outperform vanilla DARE in *all* cases and by a large margin (e.g., $>40\%$ on COLA). (2) $1/q_{e}$ and $1/\mathbf{q}_{e}$ deliver competitive performance compared to $1/q_{v}$ and $1/\mathbf{q}_{v}$, making it an effective alternative when a labeled validation set is unavailable. (3) Per-layer rescaling ($1/\mathbf{q}_{v}$ and $1/\mathbf{q}_{e}$) outperforms global rescaling.
Decoder models. Table 3 evaluates DAREx-q on decoder models. We find that: (1) DAREx-q consistently boosts performance over vanilla DARE across *all* models and both pruning rates $p=0.95,0.99$. (2) Higher pruning rates yield striking improvements: vanilla rescaling fails completely, while DAREx-q yields non-trivial performance; (3) Global rescaling ($1/q$) provides performance comparable to that of per-layer rescaling ($1/\mathbf{q}$), making it a more efficient recommendation for decoder models.

**Table 4: DAREx-q is orthogonal to and can be combined with LoRA, leading to significant improvements over vanilla DARE.**
| Datasets<br>($p=0.9$) | STS-B | COLA | SST2 | MRPC |
| --- | --- | --- | --- | --- |
| No Pruning | 90.07 | 59.56 | 94.27 | 87.09 |
| DARE (Yu et al., ) | 81.49 | 50.96 | 48.05 | 80.03 |
| DAREx-q ($1/q_{v}$) (ours) | 85.06 | 56.73 | 90.25 | 82.53 |
| DAREx-q ($1/q_{e}$) (ours) | 85.00 | 56.73 | 90.02 | 82.53 |

Performance across different pruning rates. Fig. 2 compares performance of DAREx-q to vanilla DARE for varying values of pruning rates $p$ for both encoder and decoder models. For *all* examined values of $p$, DAREx-q is at least as good as vanilla DARE. More importantly, we confirm that DAREx-q significantly outperforms vanilla DARE at higher pruning rates.

LoRA+DAREx-q: As a post-training DPP, DAREx-q is orthogonal to and can be combined with parameter-efficient tuning approaches like LoRA (Hu et al., ). To demonstrate this, we fine-tune the BERT-Base model by training only the LoRA module, then prune the parameters of LoRA at rate $p=0.9$. The results, shown in Table 4, reveal that applying vanilla DARE does *not* yield good performance.
In contrast, our DAREx-q significantly improves the results.
In addition to these, we demonstrate in Appendix Sec. C.10 that DAREx-q is even comparable to sparse fine-tuning, despite being a purely post-training method.

**Table 5: DARE with AdamR-$L_{2}$ finetuning (DAREx-$L_{2}$) significantly improves performance on encoder models compared to DARE without it.**
| Dataset | p | BERT | RoBERTa |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| original | $L_{2}$ | difference | original | $L_{2}$ | difference |  |  |
| MRPC | No Pruning | 85.01 (-) | 84.31 (-) | -0.70 | 90.05 (-) | 90.69 (-) | +0.64 |
| 0.99 | 79.96 (6.07) | 84.30 (0.48) | +4.34 | 87.32 (0.69) | 90.02 (0.41) | +2.70 |  |
| STS-B | No Pruning | 88.51 (-) | 87.99 (-) | -0.52 | 90.57 (-) | 90.25 (-) | -0.32 |
| 0.99 | 82.56 (1.13) | 87.62 (0.27) | +5.06 | 81.59 (1.44) | 89.88 (0.29) | +8.29 |  |
| SST2 | No Pruning | 90.25 (-) | 89.91 (-) | -0.34 | 92.09 (-) | 92.89 (-) | +0.80 |
| 0.99 | 51.00 (0.64) | 88.17 (0.44) | +37.17 | 51.81 (1.00) | 91.95 (0.24) | +40.14 |  |
| COLA | No Pruning | 56.24 (-) | 59.27 (-) | +3.03 | 62.10 (-) | 59.27 (-) | -2.83 |
| 0.99 | 4.25 (2.83) | 57.24 (0.62) | +52.99 | 7.65 (3.63) | 59.01 (1.26) | +51.36 |  |

Structural DPP with DAREx-q. Our random-based DAREx-q can also be utilized for structural DPP for enhanced hardware efficiency (Shen et al., ; Yao & Klimovic, ). In this setup, we randomly select $a\%$ of the input dimensions of a MLP layer and randomly retain $b\%$ of the DPs within the selected dimensions, achieving a total structured DP pruning rate of $p=a\cdot b\%$. Table 9 in the Appendix, uses$a=5,b=20$ for an equivalent $p=0.99$, to show that DAREx-q outperforms DARE with a 40% improvement.

### 5.2 AdamR- L 2 subscript 𝐿 2 L_{2} italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT finetuning

We now show that our proposed in-training method of Sec. 4.2 successfully achieves highly prunable DPs.

Figure: Figure 3: Applying DARE on decoder models finetuned with AdamR-$L_{2}$ (DAREx-$L_{2}$) at varying regularization strengths demonstrates significant performance improvements for $p\geq 0.9$.
Refer to caption: extracted/5917509/images/Qwen05-gsm8k_new.png

Encoder models. Table 5 compares the performance of DARE applied to the original finetuned model against its performance when applied to models finetuned with AdamR-$L_{2}$ across two encoder models and four datasets. We set the pruning rate at $p=0.99$ and, for reference, also report the performance of the finetuned models without pruning, both with and without AdamR-$L_{2}$. Consistent with the insights of Sec. 4.2, applying DARE after AdamR-$L_{2}$ (DAREx-$L_{2}$) outperforms vanilla DARE without regularization. Additionally, the performance is only marginally lower than that of the original finetuned model. Figure 7 in the Appendix further shows that increasing the regularization strength allows pruning up to 99.9% of parameters with only a minor performance drop. See Appendix C.3 for further ablations on regularization weight. Finally, Table 9 in the Appendix demonstrates that DAREx-$L_{2}$ unlocks the potential of random-based DPP for structural pruning, improving DARE without $L_{2}$ finetuning by over 40%.

Decoder models. To demonstrate the effectiveness of DAREx-$L_{2}$ on decoder-based LMs, we finetune Qwen-0.5B using the MetaMath dataset (Yu et al., ). Figure 3 shows the $L_{2}$ fine-tuned Qwen-0.5B model with different regularization weights on the MetaMath datasets. We used pruning rates $p\in[0,0.5,0.7,0.9,0.99]$ to prune the DPs. For visualization, the x-axis uses a log-scaled pruning value of $-\log(1-p)/2=[0,0.15,0.26,0.5,1.0]$. When AdamR-$L_{2}$ is not applied (regularization weight $=0$), performance on GSM8K drops significantly at $p=0.9$. By increasing the regularization weight, higher pruning rates can be achieved. With a regularization weight of $1e^{-4}$ (green line), we can reach a $99\%$ pruning rate while maintaining performance.

### 5.3 How to apply importance-based
DPP and when is it competitive?

We have seen, both theoretically and empirically, that DARE is effective at pruning DPs when their first and second order statistics (e.g. as summarized by $\bar{c}_{i}$ and $\sigma_{i}^{2}$ in Thm. 3.1) are small. While most fine-tuned LLMs typically introduce small DPs Yao & Klimovic (); Lee et al. (), exceptions exist. For instance, Table 8 in the App. shows that the MetaMath-LLaMA-7B model, fine-tuned on large datasets (MetaMathQA Yu et al. () and Proof-Pile-2 Azerbayev et al. ()), has a mean delta parameter magnitude of 0.017, notably larger than that of Abel-7B and MetaMath-7B. In such cases, as demonstrated in Table 6, DARE experiences a substantial performance drop when the pruning rate exceeds $0.1$.

**Table 6: Peformance of pruning methods on MetaMath-llema.**
| Methods | GSM8K |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| (MetaMath-llema-7B) | $p=0.0$ | $p=0.1$ | $p=0.3$ | $p=0.5$ | $p=0.6$ |
| MP | 64.50 | 63.00 | 61.00 | 47.50 | 32.00 |
| WANDA | 63.00 | 62.50 | 54.50 | 32.50 |  |
| DARE | 51.00 | 0.00 | 0.00 | 0.00 |  |
| Random Drop | 52.50 | 6.50 | 0.00 | 0.00 |  |

Can importance-based DPP methods serve as alternatives? Yes, as shown in Table 6. Importance-based methods like MP and WANDA can maintain pruning rates of up to 0.5 with only minimal performance degradation. Thus, these methods are effective alternatives to DARE when DPs are large and fine-tuning with customized methods like AdamR-$L_{2}$ is not feasible.

Improvements with AdamR-$L_{1}$. For completeness of our study, we also demonstrate that finetuning with AdamR-$L_{1}$ can boost performance of importance-based DPP as suggested in Sec. 4.2. Specifically, in Table 7 in App. C.4, we show elaborate experiments comparing the performance of MP and WANDA (applied to DPs) when used on a model finetuned with/without AdamR-$L_{1}$ for varying values of pruning rates and two encoder models across four datasets. In summary, we find: (1) AdamR-$L_{1}$ consistently increases performance of MP. (2) While AdamR-$L_{1}$ improves WANDA’s performance for most datasets, this is not always the case. We attribute this to WANDA’s importance score being impacted not only by DPs $\Delta W_{ij}$ but also by input magnitude $|x_{j}|$ and important DPs not always being aligned with outlier activations (see App. D).

## 6 Conclusions and limitations

Figure: Figure 4: Flowchart for selecting appropriate DPP methods based on different scenarios.
Refer to caption: extracted/5917509/images/frame_work_v3.png

Our systematic study of DPP methods, provides theoretical insights, extensive experiments in controlled environments, and demonstrates practical applications on large-scale models and datasets.
Starting with an in-depth analysis of random-based DPP methods, we identify key factors that degrade their performance. Based on these insights, we propose: (i) DAREx-q, a post-training rescaling modification strategy that significantly outperforms the state-of-the-art at high pruning rates and performs at least as well across other rates; and (ii) AdamR finetuning to enhance existing models that struggle with DPP, ensuring the production of highly prunable DPs. Our results, including dramatic gains over baseline methods shown in Table 1, suggest that these strategies warrant further investigation across different tasks and modalities. Additionally, preliminary results suggest that DAREx-q can be effectively combined with parameter-efficient finetuning methods like LoRA and utilized for structural DPP, both of which highlight the potential for further research.
Finally, while finetuning LLMs typically results in small DPs favoring random-based DPP, we demonstrate that importance-based DPP methods can serve as robust alternatives for larger DPs. This highlights the need for further exploration of alternatives.
In conclusion, our comprehensive study of DPP methods provides a framework for selecting appropriate DPP methods under different practical scenarios, which we summarize in Fig. 4.
Framework for DPP method selection: We outline the procedure for selecting appropriate DPP methods in practice. As illustrated in Fig 4, if fine-tuning is not permitted and the data points (DPs) have large statistics, we recommend using WANDA for DPP (see Sections 5.3). If the DPs are not large, DAREx-q with $1/q_{v}$ should be applied when a validation set is available, otherwise, DAREx-q with $1/q_{e}$ is recommended (see Sec. 4.1). If the existing DPs are insufficient and fine-tuning is allowed, we suggest using AdamR-$L_{2}$ or $L_{1}$ with appropriate regularization weights to generate highly prunable DPs for DAREx-q and MP, respectively (see Sec. C.6). Among the two, AdamR-$L_{2}$+DAREx-q (DAREx-$L_{2}$-q) should be prefered due to flexibility and better performance as shown in Table 9 and Table 5.
Broader impact: As discussed in Section 1, DPP offers potential advantages for online serving, gradient communication in Federated Learning, and storage saving. Our systematic study of DPP methods and the proposed strategies that improve performance of existing methods can positively impact the usage of LLMs. We also analyzed the impact of our approach on the efficiency of the machine learning system in App.  C.12.
Acknowledgments: This work was partially funded by an Alliance Mission Grant, the NSERC Discovery Grant No. 2021-03677, the Alliance Grant ALLRP 581098-22, NFRFE-2023-00936, Natural Sciences and Engineering Research Council of Canada (NSERC), Compute Canada, and Vector Institute.

## Appendix A Importance-based DPP

In this section, we adapt importance-based pruning, which is traditionally popular for model parameter pruning, e.g. Han et al. (); Li et al. (); Sun et al. (), into DPP.

Magnitude pruning (MP). Magnitude pruning Li et al. () drops model weights according to their magnitudes: less “important” weights with smaller magnitudes are dropped.
To extend its application to pruning delta parameters we evaluate importance in terms of the magnitudes of delta parameters, represented as $\Delta S_{ij}:=|\Delta W_{ij}|$.

Pruning based on both weights and activations (WANDA). Recently, WANDA was introduced by Sun et al. () and was empirically found to achieve state-of-the-art pruning performance for LLMs. It evaluates the importance of each model weight using a score metric that scales the magnitude of the weight by the Euclidean norm of the input feature, thus accounting for the input activations.
We extend WANDA to encompass delta parameters by adjusting the importance score to $\Delta S_{ij}=|\Delta W_{ij}|\cdot\|\bm{x}_{j}\|_{2}$.

For their respective scores, MP and WANDA (and any other importance-sampling method adapted to DPP) maintain the top-$k$ parameters in $\bm{\Delta}_{\bm{W}}$ with the highest scores. Let ${\mathcal{S}}_{k}\subset[m]\times[n]$ be the set of indices $(i,j)$ that correspond to weights with the top-$k$ scores $\Delta S_{ij}$. Moreover,
let $k:=(1-p)mn$ so that the method retains the same number of delta parameters as DARE does in expectation. Accordingly, denote the set of selected weights with respect to $p$ as ${\mathcal{S}}_{p}:={\mathcal{S}}_{k}$. Then, putting in Eq. (1), the change for each output activation $i\in[m]$ can be expressed as:

$$ $\displaystyle h_{i}^{\rm{diff}}=\sum\nolimits_{\{j:(i,j)\notin{\mathcal{S}}_{p }\}}\Delta W_{ij}x_{j}\,,$ (4) $$

where the summation extends over all input dimensions $j$ for which the $(i,j)$ entry of $\bm{\Delta}_{\bm{W}}$ is dropped due to a low score. An importance sampling method can perform effectively even for large
$p\approx 1$ if the cumulative contributions from the summation are approximately zero out. This is guaranteed when the distribution of the entries $[\Delta W_{ij}x_{i}]_{j\in[n]}$ has a light tail and high peakedness. We validate this in Secs. C.4 and B and accordingly propose AdamR-$L_{1}$ fine-tuning to enhance their pruning performance (algorithm detailed in in Sec. C.6)

## Appendix B Analysis on two-layer neural-network

Having gained analytical insights into the key factors that influence the performance of
DPP methods in Sec. 3.2 and Sec. A, we now explore in detail how these factors influence DPP in a two-layer neural network. Concretely, for an input $\bm{x}\in\mathbb{R}^{n}$, the model output is
$f(\bm{x})=\bm{W}_{o}\operatorname{N}(\phi(\bm{W}_{1}\operatorname{N}(\bm{x})+
\bm{b}_{1}))+\bm{b}_{o}$. Here, $\operatorname{N}$ denotes layer normalization (RMSnorm Zhang & Sennrich () in our case), $\phi$ is the ReLU activation function, $\bm{W}_{o}$ / $\bm{b}_{o}$ and $\bm{W}_{1}$ / $\bm{b}_{1}$ are the weights / biases of the output and the hidden layer respectively, and are all trainable (during both pre-training and fine-tuning). In our experiments, we pre-train the model on the CIFAR-10 dataset and use the SVNH dataset Sermanet et al. () for the supervised fine-tuning task.

Influence of variance and mean statistics on DARE (Fig. 5a):
Thm. 3.1 establishes how DARE’s performance, approximated by the magnitude of $h_{i}^{\rm{diff}}$, is influenced by the mean and variance statistics of $\{c_{ij}\}$. Specifically, smaller values are favorable for performance. To verify this, we compare performance of DARE when the model is fine-tuned using $L_{1}$/$L_{2}$ regularization with respect to the pre-trained parameters. Formally, recalling that $\bm{\Delta}_{\bm{\theta}}=\bm{\theta}_{F}-\bm{\theta}_{P}$ denotes the delta parameter of all trainable weights, we use regularization that penalizes
the following:
$\|\bm{\Delta}_{\bm{\theta}}\|_{r},~{}r\in\{1,2\}$.
$L_{2}$ regularization directly corresponds to a smaller total energy ($\sum_{ij}c_{ij}^{2}$) of the $\{c_{ij}\}$ parameters, which recall capture the degree of change in the output layer after fine-tuning. This enters the bound in Thm. 3.1 and suggests that $L_{2}$ regularization improves DARE’s performance. On the other hand, $L_{1}$ regularization favors sparse $\bm{\Delta}_{\bm{\theta}}$, thus also $\{c_{ij}\}$. Intuitively, this increases the variance, thus we expect from Thm. 3.1 that $L_{1}$ regularization might hurt performance. Fig. 5 (a) shows the test accuracy of DARE for varying values of pruning rate $p$ for three finetuning settings: no regularization (blue), $L_{2}$ regularization (orange), and $L_{1}$ regularization. The results confirm the insights above and thus support Thm. 3.1.

Figure: Figure 5: Controlled experiments of DPP performance on two-layer neural net. (a) Influence of variance and mean statistics on DARE. (b) Influence of normalization layer. (c) $L_{1}$ regularization for importance based pruning. (d) Methods with best-fitting regularization.
Refer to caption: x1.png

Influence of normalization layer (Fig. 5b):
Normalization layers have become standard practice in deep learning due to their role in stabilizing training by normalizing the variance of activations, as established in Xu et al. (). In this study, we explore the impact of these layers on the performance of DARE. According to Thm. 3.1, reduced variance in activations (denoted as $x_{j}$s in our theorem setup) leads to decreased variance in the $\{c_{ij}\}$ coefficients, which in turn is predicted to enhance DARE’s effectiveness. This hypothesis is confirmed in Fig. 5 (b), where the absence of a normalization layer is shown to expedite the performance decline of DARE as pruning rates increase. Conversely, importance-based methods discussed in Sec. A demonstrate a higher tolerance to increased pruning rates without normalization. These observations are consistent with the discussions in Sec. A.

$L_{1}$ regularization for importance based pruning (Figure 5c):
In Sec. A, we discussed how importance-based DPP methods are favored. Here, we demonstrate this is indeed the case by using $L_{1}$ regularization during finetuning to induce sparse delta parameters, which increases the variance.
Concretely, as illustrated in Fig. 5c, introducing $L_{1}$ regularization improves the performance of importance-based methods. Note also that, in this scenario, magnitude pruning outperforms WANDA due to the absence of outlier features Wei et al. (); Sun et al. ().

Methods with best-fitting regularization(Fig. 5d): In Fig 5d, we compare three methods with their corresponding best-fit regularization. As discussed above, DARE benefits from $L_{2}$ regularization, while importance-based methods benefit from $L_{1}$ regularization. We find that that DARE with $L_{2}$ regularization is more robust with respect to the increase in pruning rate. On the other hand, importance based methods with $L_{1}$ regularization perform the best for medium-range pruning rates.

## Appendix C Additional details

### C.1 Additional related Work

Supervised fine-tuning of language models.
Supervised fine-tuning (SFT) of pre-trained LLMs is designed to enhance their capabilities by training on task-specific data, establishing a standard in natural language processing Dodge et al. (); Zhao et al. (). SFT can be categorized into full fine-tuning Devlin et al. (); Liu et al. () and parameter-efficient fine-tuning (PEFT) Ding et al. (); Li & Liang (); Hu et al. (); Deng et al. (). However, recent studiesChen et al. (); Lu et al. (); Yao & Klimovic () suggest that PEFT methods may not yet achieve the model quality of full parameter fine-tuning, particularly in high-resource tasks Chen et al. (). Additionally,  Liu et al. () indicates that most models on the Open LLM Leaderboard are derived from full parameter fine-tunes or their merges. Consequently, this paper focuses on full model fine-tuning.

Sparse finetuning. An orthogonal approach to DPP is sparse fine-tuning, which reduces the size of delta parameters by modifying the fine-tuning process itself. This is achieved through techniques such as iterative masking and fine-tuning to create sparse DP (Guo et al., ; Liao et al., ; Fu et al., ). In contrast, DPP methods like DARE Yu et al. () are primarily post-hoc procedures that focus on pruning the delta weights of models that have already been fine-tuned, making DPP particularly valuable for the many fine-tuned models available on platforms like Hugging Face.

Follow-up work on DARE.
The recent development of the DARE (Drops delta parameters And REscales) technique has significantly advanced the efficiency of finetuned-model pruning. This method sets most delta parameters to zero, maintaining the efficacy of Supervised Fine-Tuning without loss of performance. Highlighted in Goddard et al. () within Arcee’s MergeKit, a toolkit for merging large LMs, DARE has shown great potential enhancing model merging processes and has various practical applications. It has been successfully implemented in projects such as Prometheus Kim et al. (), an open-source LM for evaluating other LMs, and in medical LMs like Biomistral Labrak et al. (), which develops LMs for medical applications. The technique also supports specialized domains, as seen in Akiba et al. (), which focuses on a Japanese LLM with advanced reasoning capabilities. These implementations highlight the broad applicability and effectiveness of DARE in enhancing model merging strategies, thus making our improvements particularly relevant and practical.

### C.2 Implementation details

For decoder LLMs, following Yu et al. (), we set the temperature to 0.0 for greedy decoding and limit the maximum number of generated tokens to 1,024 on GSM8K. For encoder-based LMs, we fine-tune BERT-base-uncased and RoBERTa-base for 10 epochs using a warmup strategy and a learning rate of 1e-4. Experiments are conducted on NVIDIA A100 GPUs.

### C.3 Controlling pruning with regularizaiton

In this section, we demonstrate the magnitude of regularization weight can control the degree of pruning that is required.

Fig 6 (a-d) illustrates the AdamR-$L_{2}$ fintuned RoBERTa model with different regularization weights on SST2, MRPC, COLA and STS-B datasets respectively and we use pruning rate $p\in[0,0.9,0.99,0.999]$ to prune the delta parameters. To separate different pruning rates in the figure, we set log scaled pruning $-log(1-p)/3=[0,0.33,0.67,1.00]$ in x axis. With moderate regularization the model has a close performance to models without regularizaiton (weight 0 0) (even better i.e.weight $0.01$ in Fig 6 (a)). This indicate adding moderate regularization won’t hurt the model performance. To achieve moderate pruning performance $p=0.99$, we can choose moderate regularization. For example, in fig 6 (c), choose weight $0.05$ achieves the best performance when pruning rate $p=0.99$ and outperforms the no regularization by more than 40% . When there is a need for extreme delta parameters pruning, it is recommended to increase the regularization weight. As shown in Figure 6 (d), weight $0.1$ achieves the best performance when pruning rate $p=0.999$ and outperforms no regualrizaiton by 80 %. It is notable with $L_{2}$ regualrization, all dataset can achieve $99.9\%$ delta parameter pruning with minor performance drop. Finally, as shown in Fig 6 (d), a strong regularization weight $0.5$ although face original performance drop, but demonstrate robustness (minor performance drop) to different level of pruning rate.

Fig 7 (a-d) depicts $L_{2}$ fine-tuned BERT models with varied regularization weights on SST2, MRPC, COLA, and STS-B datasets, respectively. Pruning rates $p\in[0,0.9,0.99,0.999]$ are used. Log-scaled pruning $-log(1-p)/3=[0,0.33,0.67,1.00]$ are plot to separate different rates on the x-axis. Moderate regularization yields performance close to models without regularization (weight 0 0), sometimes even better (e.g., weight $0.05$ in Fig 7 (a)), suggesting it won’t hurt model performance. For moderate pruning ($p=0.99$), moderate regularization suffices. For instance, in Fig 7 (a), weight $0.05$ achieves optimal performance, outperforming no regularization by about 40%. For extreme pruning needs, increase the regularization weight. As in Fig 7 (b), weight $0.1$ achieves peak performance at $p=0.999$, surpassing no regularization by 60%. Notably, with AdamR-$L_{2}$, all datasets can achieve 99.9% DPP with minimal performance drop. Lastly, Fig 7 (c) shows that a strong weight of $0.1$, despite an initial performance drop, exhibits robustness (minor drop) across different pruning rates.

Figure: Figure 6: Control DARE pruning performance with $L_{2}$ regularizaiton on RoBERTa.(a-d) demonstrate results on SST2, MRPC, COLA, and STS-B, respectively. Aligning the degree of regularization proportionally to the pruning rate can lead to optimal performance.
Refer to caption: x2.png

Figure: Figure 7: Control DARE pruning performance with $L_{2}$ regularization on BERT. (a-d) demonstrate results on SST2, MRPC, COLA, and STS-B, respectively. Aligning the degree of regularization proportionally to the pruning rate can lead to optimal performance.
Refer to caption: x3.png

### C.4 AdamR- L 1 subscript 𝐿 1 L_{1} italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT finetuning

In this section, we focus on MP (Han et al., ) and WANDA (Sun et al., ), utilizing AdamR-$L_{1}$ to enhance pruning performance for importance-based methods.
MP: We demonstrate AdamR-$L_{1}$ results in Table 7: the pruning rate consistently increased on datasets when applying magnitude pruning on $L_{1}$ fintuned models. This indicates AdamR-$L_{1}$ is an effective strategy in producing highly prunbale DPs for MP.
WANDA:
As shown in Table 7, AdamR-$L_{1}$ improved WANDA’s performance in most datasets. However, in the SST2 and STS-B datasets (Table 7), adding AdamR-$L_{1}$ negatively impacted pruning performance, as indicated by the numbers in red. This is because WANDA considers outlier activations Dettmers et al. () in LLMs and uses $\Delta S_{ij}=|\Delta W_{ij}|\cdot|x_{j}|$ as its metric. Thus, some important DPs may be pruned when AdamR-$L_{1}$ is applied due to their small $|x_{j}|$. This suggests that important DPs are not always aligned with outlier activations, and that AdamR-$L_{1}$ may not be the optimal strategy for improving WANDA. For further analysis of outlier activations, see Appendix D.

**Table 7: MP and WANDA with AdamR-$L_{1}$ on BERT and RoBERTa across various datasets**
|  | SST2 | COLA |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Models | p=0.0 | p=0.90 | p=0.95 | p=0.99 | p=0.0 | p=0.90 | p=0.95 | p=0.99 |  |  |  |  |  |  |  |  |
|  | original | $L_{1}$ | original | $L_{1}$ | original | $L_{1}$ | original | $L_{1}$ | original | $L_{1}$ | original | $L_{1}$ | original | $L_{1}$ | original | $L_{1}$ |
| BERT-MP | 90.25 | 89.91 | 91.40 | 89.68 | 84.29 | 89.22 | 51.15 | 83.14 | 56.24 | 56.52 | 37.47 | 47.49 | 21.58 | 44.59 | 8.49 | 12.30 |
| BERT-WANDA | 90.51 | 88.53 | 86.93 | 79.93 | 52.64 | 63.19 | 37.34 | 45.29 | 19.24 | 46.09 | 8.37 | 10.85 |  |  |  |  |
| RoBERTa-MP | 92.09 | 92.55 | 90.27 | 92.78 | 84.86 | 92.78 | 68.00 | 91.86 | 62.10 | 60.16 | 24.88 | 50.78 | 6.56 | 43.81 | 0.00 | 32.81 |
| RoBERTa-WANDA | 92.12 | 91.74 | 92.35 | 89.45 | 76.15 | 72.82 | 34.64 | 42.54 | 0.07 | 36.55 | 0.00 | 29.80 |  |  |  |  |
|  | MRPC | STS-B |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Models | p=0.0 | p=0.90 | p=0.95 | p=0.99 | p=0.0 | p=0.90 | p=0.95 | p=0.99 |  |  |  |  |  |  |  |  |
|  | original | $L_{1}$ | original | $L_{1}$ | original | $L_{1}$ | original | $L_{1}$ | original | $L_{1}$ | original | $L_{1}$ | original | $L_{1}$ | original | $L_{1}$ |
| BERT-MP | 85.01 | 84.35 | 66.14 | 67.22 | 15.81 | 26.52 | 15.81 | 15.81 | 88.51 | 85.12 | 82.79 | 84.98 | 42.31 | 84.42 | 55.80 | 75.71 |
| BERT-WANDA | 72.85 | 64.22 | 22.75 | 23.49 | 15.81 | 15.81 | 80.71 | 76.11 | 50.70 | 70.80 | 40.92 | 54.21 |  |  |  |  |
| RoBERTa-MP | 90.05 | 90.07 | 76.39 | 85.17 | 75.17 | 78.93 | 74.80 | 74.80 | 90.57 | 89.87 | 82.70 | 87.43 | 74.58 | 86.25 | 21.26 | 58.53 |
| RoBERTa-WANDA | 82.21 | 84.68 | 76.24 | 78.86 | 74.80 | 74.80 | 84.56 | 78.13 | 76.17 | 65.42 | 20.91 | 25.87 |  |  |  |  |

### C.5 Scale of delta parameters

As demonstrated in Theorem 3.1, DARE is effective at pruning DPs when the mean and variance of $c_{ij}$ are small, and it generally outperforms importance-based methods in such conditions, as shown in Table 12. While most fine-tuned LLMs typically introduce small DPs Yao & Klimovic (); Lee et al. (), and $x$ is small, as indicated in Table 8, where $|\Delta W_{ij}x_{j}|\ll|\Delta W_{ij}|$, there are still instances where delta parameters are large. Specifically, Table 8 reports the magnitudes of $\Delta W$ and $c_{ij}$ across various LLMs. For instance, the MetaMath-LLaMA-7B Yu et al. (), which fine-tunes LLaMA2-7B using two large datasets—MetaMathQA Yu et al. () and Proof-Pile-2 Azerbayev et al. ()—yields a mean delta parameter magnitude of 0.017, significantly larger than that of Abel-7B and MetaMath-7B. As illustrated in Table 6, DARE experiences a substantial performance drop when the pruning rate exceeds 0.1.

**Table 8: Magnitude Statistics of $\Delta W_{ij}$ and $\Delta W_{ij}x_{j}$ of Llama2-7B fintuned LLMs on all dimensions and layers.**
| Models | Magnitude |  |
| --- | --- | --- |
|  | $\rm{mean}(|\Delta W_{ij}|)$ | $\rm{mean}(|\Delta W_{ij}x_{j}|)$ |
| Abel-7B | 7.3e-4 (4.3e-7) | 2.89e-5 (5.41e-9) |
| MetaMath-7B | 7.2e-4 (3.3e-7) | 2.85e-5 (6.02e-9) |
| Wizardmath-7B | 4.0e-4 (1.0e-7) | 1.56e-5 (1.67e-9) |
| MetaMath-llema-7B | 0.017 (1.9e-4) | 7.0e-4 (3.02 e-6) |

### C.6 AdamR algorithm

With a simple model, Sec. B demonstrated how $L_{1}$ and $L_{2}$ regularization can enhance DPP. We now show that when modifications to model finetuning are allowed, our approach can scale up to LLMs. In order to implement our regularization strategy, we propose replacing the weight decay of AdamW—the standard choice for training LLMs—with our custom regularization, as described in Sec. 5.2 and Sec. C.4. However, we do this carefully since Xie et al. () found recently that weight decay can lead to large gradient norms in the final training phase, deteriorating convergence. To address this issue, following Xie et al. () we adjust our regularization strength based on the gradient norm. We call our new optimizer as AdamR. Note regularization here is based on the DP rather than simply on the weights of the finetuned model. The detailed algorithm, which we call AdamR, is presented in Algorithm 1.

Figure: Algorithm 1 AdamR

Recall from Sec. B that $L_{2}$ regularization promotes a small mean and variance of delta parameters, while $L_{1}$ regularization encourages sparse updates on important parameters, leading to larger variance. We also emphasize that our regularization here is based on the DP rather than simply on the weights of the finetuned model.

### C.7 DAREx-q algorithm

Figure: Algorithm 2 Empirically find $q$

Figure: Algorithm 3 Analytically calculate $q$

In this section, we propose and describe three strategies to determine the global rescaling factor. The per-layer rescaling factor follows a similar approach to the global one, with slight adjustments in determining the $\eta$ and resolving $1/\mathbf{q}$ (see details in Algorithm 2 step 6.)
Empirically finding $1/q_{v}$ using validation data:
In this approach, we use a validation dataset $\{x_{v},y_{v}\}\in\mathcal{V}$ to empirically determine the best rescaling factor $q$. The goal is to solve the optimization problem $\arg\min_{q}\mathbb{E}_{\mathcal{V}}(f_{q}(x_{v})=y_{v})$, where $f_{q}$ represents the pruned model rescaled by $1/q$. The process is as follows:

- •
Optimization Process: We iterate over a range of $q$ values, adjusting the pruned model by rescaling with $1/q$. For each $q$, we measure the model’s performance on the validation set by comparing the model’s predictions $f_{q}(x_{v})$ to the true labels $y_{v}$.
- •
Outcome: The $q$ value that results in the best performance on the validation set is selected as $1/q_{v}$. This $1/q_{v}$ is expected to provide the best overall performance across all layers when evaluated against the validation set.
- •
Algorithm Details: This process is detailed in Algorithm 2 with objective (1), which outlines the specific steps taken to find $1/q_{v}$.

Empirically finding $1/q_{e}$ using output change:
This method builds on the core idea from Thm 3.1, aiming to empirically determine a global rescaling factor $q_{e}$ that balances the mean and variance in output embedding differences. Unlike the validation-based approach, this method does not rely on labeled data. Instead, it uses an unsupervised objective function based on the difference in the last layer’s output embedding before and after pruning and rescaling.
Procedure:

- •
Data Usage: We select a single fixed batch of data, which could be drawn from the training set or any natural dataset.
- •
Inference: The selected batch is passed through both the pruned and original models, and the output embeddings are compared.
- •
Optimization: The goal is to find the rescaling factor $q_{e}$ that minimizes the difference between these embeddings, ensuring that the rescaled pruned model closely approximates the original model’s output.

Efficiency: This method is highly efficient as it only requires inference on a single batch of data, avoiding the need for a full validation set.
Algorithm Details: The specific steps for this method are outlined in Algorithm 2 with objective (2), which provides a detailed procedure for determining $1/q_{e}$.

### C.8 Per-layer 1 / 𝐪 1 𝐪 1/\mathbf{q} 1 / bold_q

The challenge in naively implementing a per-layer tuning of the rescaling factor is that if $q^{\ell}$ were selected to optimize, say, the mean output change $|h_{i}^{\rm{diff}}|$ averaged over all neurons in the $\ell$-th layer, this would require $L$ searches, which is computationally intractable. Instead, we turn to theory for a more efficient solution.

Our approach extends Thm.3.1
, which bounds $|h_{i}^{\rm{diff}}|$ for $q=1-p$, to obtain a high-probability bound for arbitrary $q$. This involves bounding both the mean (which is now non-zero) and the variance, then optimizing this bound over $q$ for each layer. In view of Thm. 3.1, it is unsurprising that the bound depends on the first- and second-order statistics of the coefficients $c_{ij}$. Consequently, the optimal $q$ can indeed vary per layer, depending on these statistics.

We need a high-probability bound that holds for all values of $q$, even as $q$ becomes small (on the order of, but still larger than, $1-p$). In more detail, following the machinery of Thm. 3.1 to bound $|h_{i}^{\rm{diff}}|$, we apply Markov’s inequality to the Laplace transform $\mathbb{E}[e^{\eta|h_{i}^{\rm{diff}}|}]$, $\eta>0$. After some careful calculations (see App. E.2), for each fixed $\eta$, we can define $q^{\ell}(\eta)$ by solving a one-dimensional minimization problem involving $\eta$, $p$, and the statistics of $c^{\ell}_{ij}$ of the respective $\ell$-th layer (which are measurable). Specifically, this is given as follows, for some probability of failure $\gamma\in(0,1)$:

$$ $\displaystyle q(\eta):=\arg\min_{q}|\log\frac{2}{\gamma}+\eta\left(1-\frac{1}{ q}(1-p)\right)\sum_{j}c_{ij}+\frac{\eta^{2}\Phi(p)\sum_{j}c^{2}_{ij}}{4q^{2}}|$ (5) $$

In typical concentration bounds, one selects the optimal $\eta$ by minimizing the upper bound obtained via Markov’s inequality. However, this optimal $\eta$ is inversely proportional to $q^{2}$. For the small $q$ values we are interested in here (on the order of $1-p$), this makes Markov’s inequality loose. Instead, we propose selecting $\eta$ by a grid search over the mean output change of the last layer after rescaling with $q^{L}(\eta)$. This yields a value of $\eta=\eta_{e}$.

To give flexibility to the rescaling factors of other layers, allowing them to adjust based on the individual statistics of $c_{ij}$, while avoiding grid searches for all layers, we use the same value of $\eta=\eta_{e}$ and rescale each layer with $q^{\ell}(\eta_{e})$. We denote the vector of selected per-layer rescaling factors as $1/\mathbf{q}_{e}=[1/q^{1}(\eta_{e}),\ldots,1/q^{L}(\eta_{e})]$. Additionally, by selecting $\eta$ by optimizing the performance on validation set rather than the mean output change (denote this $\eta_{v}$), we can arrive at an alternative implementation, $1/\mathbf{q}_{v}=[1/q^{1}(\eta_{v}),\ldots,1/q^{L}(\eta_{v})]$, when labeled validation data are available.

### C.9 Structural pruning

**Table 9: Structural Pruning on BERT.**
| Dataset ($p=0.99$) | SST2 | COLA |
| --- | --- | --- |
| DARE | 51.00 (0.23) | 6.25 (1.41) |
| Struct-DAREx-q ($1/q_{v}$) | 83.40 (1.92) | 39.09 (1.50) |
| Struct-DAREx-$L_{2}$-q ($1/q_{v}$) | 87.64 (0.47) | 53.71 (1.74) |

Our random-based DAREx-q can also be utilized for structural DPP for enhanced hardware efficiency (Shen et al., ; Yao & Klimovic, ). In this setup, we randomly select $a\%$ of the input dimensions of a MLP layer and randomly retain $b\%$ of the DPs within the selected dimensions, achieving a total structured DP pruning rate of $p=a\cdot b\%$. In Table 9 we use $a=5,b=20$, resulting in an equivalent pruning ratio $p=0.99$. Our results show that DAREx-q significantly outperforms DARE, achieving over a 30% improvement. AdamR-$L_{2}$ unlocks the potential of random-based DPP for structural pruning, improving DARE without $L_{2}$ finetuning by over 40%.

### C.10 Comparison to sparse finetuning

Sparse Finetuning (SFT) selects key weights by directly altering the fine-tuning process, using methods like iterative masking and fine-tuning to achieve sparse parameterization. In contrast, our DAREx-q are primarily post-hoc techniques that prune the weights of models after they have been fine-tuned.
We compare DPP with sparse fine-tuning Guo et al. (), evaluating both on the BERT-Large model. Consistent with Guo et al. (), we assess the performance of vanilla DARE and our DAREx-q with a rescaling factor of $1/q_{v}$, using a pruning rate of $p=0.995$ (retaining only 0.5% of weights). As shown in Table 11, for MRPC and SST2, vanilla DARE performs slightly below the Diff-struct method, while DAREx-q remains competitive with Diff-struct and surpasses Diff. For STS-B and COLA, vanilla DARE underperforms, but our DAREx-q restores performance, making it competitive with Diff-struct and superior to Diff. Moreover, a key advantage of our DAREx-q is that it is a post-hoc method, allowing for easy application to pre-fine-tuned models, and is simpler to implement.

**Table 10: Comparison DAREx-q with SFT**
| $p=0.995$ | MRPC | STS-B | COLA | SST2 |
| --- | --- | --- | --- | --- |
| Full-tune | 91.0 | 86.9 | 61.2 | 93.2 |
| Diff | 87.0 | 83.5 | 60.5 | 92.5 |
| Diff-struct | 89.7 | 86.0 | 61.1 | 93.1 |
| DARE | 89.6 | 0.00 | 53.2 | 90.1 |
| DAREx-q ($1/q_{v}$) | 89.9 | 84.2 | 60.1 | 92.2 |

Next, we conduct an additional experiment to examine how our proposed method—combining the post-hoc DARE with AdamR-$L_{2}$ (DAREx-$L_{2}$)—compares to sparse fine-tuning. As shown in Table 11, for a pruning rate of $p=0.999$, DAREx-$L_{2}$ outperforms the baseline across both datasets. It’s important to note that, similar to sparse fine-tuning, AdamR-$L_{2}$
modifies the fine-tuning process. However, AdamR-$L_{2}$ only replaces the original AdamW optimizer while keeping the same task loss objective, making it a simpler implementation compared to methods like Diff, which require additional modifications.

### C.11 Random-based methods generally outperforms importance-based DPP

When DPs are not large, we show that random-based methods consistently outperform importance-based methods across all cases on decoder models, as presented in Table 12, with our DAREx-q achieving more than a 30% improvement.

**Table 12: Similar to Table 2, DAREx-q consistently provides significant gains for decoder models.**
| Dataset | GSM8K |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | Abel-7B | MetaMath-7B | WizardMath-7B | MetaMath-Qwen2-0.5B |  |  |  |  |
| $p=0.95$ | $p=0.99$ | $p=0.95$ | $p=0.99$ | $p=0.95$ | $p=0.99$ | $p=0.95$ | $p=0.99$ |  |
| No Pruning | 58.32 | 65.50 | 54.90 | 42.00 |  |  |  |  |
| WANDA | 18.20 | 0.00 | 23.58 | 0.00 | 21.50 | 0.00 | 0.00 | 0.00 |
| MP | 15.16 | 0.00 | 22.82 | 0.00 | 12.00 | 14.50 | 0.00 | 0.00 |
| DARE | 37.30 | 0.00 | 58.22 | 0.00 | 47.10 | 0.00 | 0.00 | 0.00 |
| DAREx-q ($1/q_{v}$) | 47.20 | 20.20 | 59.05 | 34.87 | 49.81 | 35.63 | 30.70 | 19.17 |
| DAREx-q ($1/\mathbf{q}_{v}$) | 44.50 | 20.00 | 59.28 | 32.06 | 50.64 | 34.34 | 29.56 | 18.65 |
| DAREx-q ($1/q_{e}$) | 42.99 | 21.30 | 60.34 | 32.60 | 49.05 | 33.97 | 30.32 | 18.95 |
| DAREx-q ($1/\mathbf{q}_{e}$) | 42.84 | 19.63 | 59.28 | 28.05 | 50.64 | 33.58 | 28.80 | 17.66 |

### C.12 Broader impact

As discussed in Section 1 and demonstrated in prior work, DPP offers potential advantages for online serving, gradient communication in Federated Learning, and storage efficiency. Our systematic study of DPP methods and the proposed strategies that improve performance of existing methods can thus positively impact the usage of LLMs. We analyzed the impact of our approach on the efficiency of the machine learning system.

As stated in the introduction, a high pruning rate can benefit online serving, gradient communication in Federated Learning, and storage saving. To showcase this effectiveness, we compare the delta parameter loading time, the number of communication parameters, and the storage size at various pruning rates, as detailed in Table 13. Firstly, we found that the time to load the sparse compressed model decreases significantly as the pruning rate increases. In Federated Learning, only $p\%$ of the parameters need to be communicated, greatly enhancing communication efficiency. Additionally, when applied to storage saving, we further reduce memory usage by converting sparse tensors to CSR format, resulting in only 1.7 MB and 7.5 MB of storage with a 99.9% pruning rate for BERT and RoBERTa respectively. Consequently, pruning delta parameters can effectively enhance the efficiency of machine learning systems.

**Table 13: Loading, Communication, Storage Efficiency with different pruning rate**
| Models | Loading Time (s) | Communication Size (# parameters) | CSR Storage Size (MB) |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| pruning rate $p$ | pruning rate $p$ | pruning rate $p$ |  |  |  |  |  |  |  |  |  |  |
| No Pruning | $0.9$ | $0.99$ | $0.999$ | No Pruning | $0.9$ | $0.99$ | $0.999$ | No Pruning | $0.9$ | $0.99$ | $0.999$ |  |
| BERT | 0.143 | 0.120 | 0.035 | 0.023 | $\approx$ 110 M | $\approx$ 11 M | $\approx$ 1.1 M | $\approx$ 0.11 M | 417.7 | 108.7 | 11.4 | 1.7 |
| RoBERTa | 0.165 | 0.125 | 0.038 | 0.026 | $\approx$ 125M | $\approx$ 12.5M | $\approx$ 1.25M | $\approx$ 0.125M | 475.6 | 102.3 | 16.2 | 7.5 |

## Appendix D Massive activation and outlier features

Since Transformer-based models contain significantly large outliers that tend to concentrate in a few embedding dimensions Wei et al. (), and LLMs exhibit massive activations with high magnitude values on unique dimensions for specific tokens Sun et al. (), we analyze the influence of these large magnitude values in our study.

### D.1 Outlier feature (activation)

In this section, we empirically show the influence of outlier features. As shown in Table 7, WANDA outperforms MP on SST2, MRPC in original fintuned model, which indicates the validity of considering outlier features in DPP. Although outlier features will introduce larger activation inputs than normal features (Fig 8 (a)), with small delta parameter change, the mean and variance of $\Delta W_{ij}x_{ij}$ is still small (Fig 8 (b-c)) thus making DARE works well in delta parameter pruning. As a result, DARE results (Table 5 and Fig 7) outperforms that of WANDA and MP.

Figure: Figure 8: Analysis Outlier Features and the $\Delta Wx$ with SST2 fintuned RoBERTa. Mean (green line) means average of all feature dimensions. The $\Delta Wx$ is extremely small on average.
Refer to caption: x4.png

### D.2 Massive activation

Figure: Figure 9: Analysis on Massive Activation and Outlier Features on MetaMath-llema-7B
Refer to caption: extracted/5917509/images/massive_outlier.png

The decoder based large language models (larger than 7-B) have massive activation Sun et al. () that embrace large value on unique dimension on unique tokens (i.e.first token of a input). We leverage MetaMath-llema-7B Yu et al. () which finetune Llama2-7B with math datasets. We analyze each attention layer’s input activations and identified the $2533$-th,$1415$-th and $1512$-th dimension as the massive activations, and shown in Fig 9 (a). It is notable $2533$-th and $1415$-th dimension is aligned with the pretrain model Llama2-7B’s massive activation dimension Sun et al. (), but we have one new $1512$-th dimension in MetaMath-llema-7B. When then studied the impact of layer normalization and found the massive activations is heavily downscaled by the layer normalizaiton’s scale multiplier, which may fail to being considered as important by Wanda. We furtherly study the influence on removing the delta parameters that correspond to the those large manitude features by evaluating the results on GSM8K dataset. As shown in Table 14, removing the $1512$-th massive activation features will bring significant performance drop. As a result, we suggest to maintain the delta parameters related to new massive activation features.

**Table 14: Influnece of the delta parameters that correspond to large manitude features. Evaluation accuracy on GSM8K dataset.**
| ID | Prune Status | Accuracy |
| --- | --- | --- |
| None | No Pruning | 64.50 |
| 2533 | Massive Activations | 62.50 |
| 1512 | Massive Activations | 48.00 |
| 1415 | Massive Activations | 65.50 |
| 3431 | Outlier Feature | 61.00 |
| 2158 | Outlier Feature | 64.50 |
| 2350 | Outlier Feature | 62.50 |

## Appendix E Proofs

### E.1 Proof of Theorem 3.1

The proof employs Hoeffding’s inequality for subgaussian random variables. However, a straightforward application results in suboptimal dependence of the bound on the drop-rate $p$. Instead, we utilize a refinement of Hoeffding’s bound tailored for (generalized) Bernoulli random variables, as developed by Kearns and Saul Kearns & Saul (). For large values of $p$, we adopt the improved bound proposed by Berend and Kontorovich Berend & Kontorovich (). These adaptations are detailed below.

Recall from Eq. (2) that the entries of $\bm{h}^{\rm{diff}}$ are given as

$$ $h_{i}^{\rm{diff}}=\sum^{n}_{j=1}\Delta W_{ij}x_{j}-\sum^{n}_{j=1}\frac{1}{1-p} \delta_{ij}\Delta W_{ij}x_{j}=\sum^{n}_{j=1}\big{(}1-\frac{1}{1-p}\delta_{ij} \big{)}\Delta W_{ij}x_{j},$ $$

where $\delta_{ij}$ are iid $\rm{Bernoulli}(1-p)$ random variables. Fix any $i\in[m]$. Note $h_{i}^{\rm{diff}}$ is weighted sum of $n$ iid random variables

$$ $A_{ij}:=\big{(}1-\frac{1}{1-p}\delta_{ij}\big{)}.$ $$

We make the following observations about these random variables.
First, a simple calculation gives that they are zero-mean, i.e. $\mathbb{E}[A_{ij}]=0$ for all $j\in[n]$.
Second, the random variables are bounded satisfying
$-\frac{p}{1-p}\leq A_{ij}\leq 1$. Third, a simple calculations yields their variance $\mathbb{V}(A_{ij})=p/(1-p)$.

Based on these, perhaps a first attempt in bounding $|h_{i}^{\rm{diff}}|$ is applying Chebychev’s inequality: For any $t>0$,

$$ $\displaystyle\Pr(|h_{i}^{\rm{diff}}-\mathbb{E}(h_{i}^{\rm{diff}})|\geq t)\leq \frac{\mathbb{V}(h_{i}^{\rm{diff}})}{t^{2}}.$ $$

Using the mean and variance calculations above, this yields with probability at least $1-\gamma$:

$$ $|h_{i}^{\rm{diff}}|\leq\frac{1}{\sqrt{\gamma}}\sqrt{\frac{p}{1-p}}\sqrt{\sum_{ j=1}^{n}\Delta W_{ij}^{2}x_{j}^{2}}\,.$ $$

The drawback is of course that this scales poorly with $\gamma$.

The natural way to improve the dependence of the bound on $\gamma$ is to use stronger concentration as materialized by a Chernoff bound. Since $A_{ij}$s are bounded,
we can immediately apply Hoeffdings inequality for bounded random variables (Vershynin, , Thm. 2.2.6) to find that for any $t>0$:

$$ $\Pr\left(|h_{i}^{\rm{diff}}-\mathbb{E}(h_{i}^{\rm{diff}})|>t\right)\leq 2\exp \left(-\frac{2t^{2}}{\frac{1}{(1-p)^{2}}\sum_{j\in[n]}c_{ij}^{2}}\right)\,.$ $$

Therefore, for any $\gamma\in(0,1)$ the following holds with probability at least $1-\gamma$:

$$ $|h_{i}^{\rm{diff}}|\leq{\left(\frac{1/\sqrt{2}}{(1-p)}\right)}\sqrt{\sum_{j\in [n]}c_{ij}^{2}}\sqrt{\log\left(\frac{2}{\gamma}\right)}\,.$ $$

Note the significant improvement over Chebychev’s bound with respect to the scaling factor $\gamma$. However, the bound remains relatively loose in terms of $p$ for values of $p$ near the extremes of its range. To see this note that as $p\approx 0$, for which $A_{ij}\approx 0$ and $\mathbb{V}(A_{ij})=p/(1-p)\approx 0$. Conversely, Hoeffding’s bound, which scales as $1/(1-p)$, does not decrease as $p$ gets smaller. To get an improved bound, we can apply the Kearns-Saul ineqaulity (Kearns & Saul, , Theorem 2), which is better suited for managing the characteristics of the distribution as $p$ approaches extremal values.

###### Theorem E.1 ( Kearns & Saul ( 2013 ) ) .

Let $R_{j}\in\{0,1\}$ be iid $\rm{Bernoulli}(1-p_{j})$ for $j\in[n]$. Let also constants $\alpha_{j},j\in[n]$. Then, for all $t>0$

$$ $\displaystyle\Pr\left(\Big{|}\sum_{j\in[n]}\alpha_{j}(R_{j}-(1-p_{j}))\Big{|}> t\right)\leq 2\exp\left(-\frac{t^{2}}{\chi^{2}}\right)\,,$ (6) $$

where $\chi^{2}:=\sum_{j\in[n]}\alpha_{j}^{2}\Phi(p_{j})$, and

$$ $\displaystyle\Phi(x)=\frac{1-2x}{\log\left(\frac{1-x}{x}\right)}\,.$ (7) $$

Applied to our setting, let $R_{j}\leftarrow\delta_{ij}$ and $\alpha_{j}\leftarrow\frac{1}{1-p}c_{ij}$. Then, the sum on the LHS of (6) is equal to $-\sum_{j\in[n]}A_{ij}$ and $\chi^{2}=\frac{\Phi(p)}{(1-p)^{2}}\sum_{j\in[n]}c_{ij}^{2}$. Put together, this gives
for any $t>0$:

$$ $\Pr\left(|h_{i}^{\rm{diff}}|>t\right)\leq 2\exp\left(-\frac{t^{2}}{\frac{1}{(1 -p)^{2}}\Phi(p)\sum_{j\in[n]}c_{ij}^{2}}\right)\,,$ $$

where $\Phi(p):=\frac{1-2p}{\log\left((1-p)/p\right)}$. Thus,
for any $\gamma\in(0,1)$ the following holds with probability at least $1-\gamma$:

$$ $\displaystyle|h_{i}^{\rm{diff}}|\leq{\left(\frac{\sqrt{\Phi(p)}}{1-p}\right)} \sqrt{\sum_{j\in[n]}c_{ij}^{2}}\sqrt{\log\left(\frac{2}{\gamma}\right)}\,.$ (8) $$

Figure: Figure 10: Visualization of the Hoeffding, Kearns-Saul and Berend-Kontorovich bounds. Note that the last is significantly improved over the vanilla Hoeffding bound for small/large values of $p$. Large values of $p$ corresponding to large drop-rates are particularly interesting in the DPP setting. Note also that all three bounds diverge as $p\rightarrow 1$. This is because of the rescaling factor $1/(1-p)$. However, note that the Berend-Kontorovich bound diverges significantly slower. This further signifies the importance of using this versus the vanilla Hoeffding bound to provide an explanation on why DARE performs well even for large values of $p$. See proof of Thm. 3.1 for details.
Refer to caption: x5.png

To see why this improves upon Hoeffding’s bound, note that the following hold for $\Phi(p)$:

- 1.
$\Phi(p)\leq 1/2$ for all $p\in[0,1]$
- 2.
$\Phi(0)\rightarrow 0$ as $p\rightarrow 0$ or $p\rightarrow 1$.

The first property shows that the bound is strictly better than Hoeffding’s for all values of $p$. The second property shows that the bound approaches 0 0 as $p\approx 0$, which is intuitive since in that case $A_{ij}\approx 0$ and $\mathbb{V}(A_{ij})\approx 0$. On the other hand, for $p\rightarrow 1$, it is easy to show that $\sqrt{\Phi(p)}/(1-p)\rightarrow+\infty$. Hence, both Hoeffding and this improved bound predict that $|h_{i}^{\rm{diff}}|$ can be large as $p\rightarrow 1$.
It turns out that the Kearns-Saul bound can be further improved for $p\geq 1/2$. Such an improvement is important in our setting, since large drop-rates $p\gg 1/2$ are of particular interest in DPP. The improved inequality is due to (Berend & Kontorovich, , Lemma 5). When applied to our setting, it improves the first term in to $\sqrt{\frac{{2p}}{p-1}}$.
To arrive now at the advertised statement of the theorem simply rewrite $\sum_{j\in[n]}c_{ij}^{2}$ in terms of the mean and variance of the influence statistics

$$ $\frac{1}{n}\sum_{j=1}^{n}\Delta W_{ij}^{2}x_{j}^{2}=\frac{1}{n}\sum_{j=1}^{n}c _{ij}^{2}=\bar{c}_{i}^{2}+\sigma_{i}^{2}.$ $$

This completes the proof of the theorem.

For completeness, we visualize the three bounds that we discussed above in Figure 10. The lower the better, thus note how the Kearns-Saul bound improves over naive Hoeffding and the Berend-Kontorovich bound improves over both for $p>1/2$.

###### Theorem E.1 ( Kearns & Saul ( 2013 ) ) .

### E.2 Finding a rescaling factor balancing mean and variance

This section complements Sec. C.8 that describes the per-layer version of DAREx-q. Specifically, we show how to derive Eq. (5).

We study a more general version of DARE that rescales with a parameter $q$ that is not necessarily set to $1/(1-p)$. In this case, we have
$A_{ij}=\left(1-\frac{1}{q}\delta_{ij}\right)c_{ij},$
which we can rewrite as follows:

$$ $A_{ij}=\underbrace{\left(1-\frac{1}{q}(1-p)\right)c_{ij}}_{=:\mu_{ij}}- \underbrace{\frac{1}{q}c_{ij}\left(\delta_{ij}-(1-p)\right)}_{=:R^{\prime}_{j} }\,.$ $$

Following Kearns & Saul (), for any random variable $X$, it holds that $\Pr(X>0)=\frac{1}{2}\mathbb{E}\left[1+\frac{X}{|X|}\right]$. Observe that $\frac{1}{2}(1+\frac{X}{|X|})\leq e^{\eta X}$ for any positive number $\eta>0,where\eta$ is a small positive number to ensure tightness of Eq. (9). Then, we obtain (alternatively, but equivalently, we can apply Markov’s inequality to the Laplace transform):

$$ $\displaystyle Pr[\sum_{j}A_{ij}>\epsilon]$ $\displaystyle\leq\mathbb{E}\left[e^{\eta\sum_{j}\mu_{ij}-\frac{1}{q}c_{ij} \left(\delta_{ij}-(1-p)\right)-\eta\epsilon}\right]$ (9) $\displaystyle=e^{-\eta\epsilon+\eta\sum_{j}\mu_{ij}}\mathbb{E}\left[\prod_{j}e ^{-\frac{\eta}{q}c_{ij}\left(\delta_{ij}-(1-p)\right)}\right]=e^{-\eta\epsilon +\eta\sum_{j}\mu_{ij}}\prod_{j}\mathbb{E}\left[e^{-\frac{\eta}{q}c_{ij}\left( \delta_{ij}-(1-p)\right)}\right]$ $\displaystyle=e^{-\eta\epsilon+\eta\sum_{j}\mu_{ij}}\prod_{j}(pe^{\frac{\eta}{ q}c_{ij}(1-p)}+(1-p)e^{-\frac{\eta}{q}c_{ij}p})$ $\displaystyle\leq e^{\eta\left(\sum_{j}\mu_{ij}-\epsilon\right)+\frac{\eta^{2} }{4}\sum_{j}\frac{c^{2}_{ij}}{q^{2}}\Phi(p)},\text{\cite[cite]{(\@@bibref{Auth orsPhrase1Year}{kearns}{\@@citephrase{, }}{}, Lemma 1)}}\,.$ (10) $$

Since $\frac{c^{2}_{ij}}{q^{2}}\Phi(p)\geq 0$, choosing the optimal $\eta=\frac{-2\left(\sum_{j}\mu_{ij}-\epsilon\right)}{\frac{\Phi(p)}{q^{2}}\sum
_{j}{c^{2}_{ij}}}$ minimizes Eq. (10). However, this may result in a large $\eta$ especially for small $q$ that is of interest to us. When $\eta=\frac{-2\left(\sum_{j}\mu_{ij}-\epsilon\right)}{\frac{\Phi(p)}{q^{2}}\sum
_{j}{c^{2}_{ij}}}$, for all $\epsilon>0$, we have:

$$ $\displaystyle Pr[|\sum_{j}A_{ij}|>\epsilon]$ $\displaystyle\leq 2e^{-\frac{q^{2}\left(\sum_{j}\mu_{ij}-\epsilon\right)^{2}}{ \Phi(p)\sum_{j}c^{2}_{ij}}}\,.$ (11) $$

Since $\mu_{ij}:=\left(1-\frac{1}{q}(1-p)\right)c_{ij}$, then with probability $1-\gamma$, we have:

$$ $\displaystyle\epsilon(q)=\begin{cases}\left(1-\frac{1}{q}(1-p)\right)\sum_{j}c _{ij}-\frac{1}{q}\sqrt{\log(\frac{2}{\gamma})\sum_{j}c^{2}_{ij}\Phi(p)},~{}~{} \text{if $\left(1-\frac{1}{q}(1-p)\right)\sum_{j}c_{ij}-\epsilon>0$}\\ \left(1-\frac{1}{q}(1-p)\right)\sum_{j}c_{ij}+\frac{1}{q}\sqrt{\log(\frac{2}{ \gamma})\Phi(p)\sum_{j}c^{2}_{ij}},~{}~{}\text{if $\left(1-\frac{1}{q}(1-p) \right)\sum_{j}c_{ij}-\epsilon\leq 0$}\end{cases}$ $$

Since $\eta\geq 0$, the second condition is always satisfied. Therefore,

- •
If $\frac{1}{q}\sqrt{\log\left(\frac{2}{\gamma}\right)\sum_{j}c_{ij}^{2}\Phi(p)}=0$, then $1-p$ is simply the optimal rescaling.
- •
If $\frac{1}{q}\sqrt{\log\left(\frac{2}{\gamma}\right)\sum_{j}c_{ij}^{2}\Phi(p)}>0$, we can increase $q$ to compute the optimal scaling factor:

$$ $\displaystyle q$ $\displaystyle=\arg\min_{q}\left(1-\frac{1}{q}(1-p)\right)\sum_{j}c_{ij}+\frac{ 1}{q}\sqrt{\log(\frac{2}{\gamma})\Phi(p)\sum_{j}c^{2}_{ij}}$ $\displaystyle=\arg\min_{q}\sum_{j}c_{ij}+\frac{1}{q}\left(\sqrt{\log(\frac{2}{ \gamma})\Phi(p)\sum_{j}c^{2}_{ij}}-(1-p)\sum_{j}c_{ij}\right)$ $$

Then the min value can be obtained by taking

$$ $\displaystyle q=1-p-\frac{\sqrt{\log(\frac{2}{\gamma})\Phi(p)\sum_{j}c^{2}_{ij }}}{\sum_{j}c_{ij}}$ $$

If $\sum_{j}c_{ij}>0$ and $\sqrt{\log\left(\frac{2}{\gamma}\right)\Phi(p)\sum_{j}c^{2}_{ij}}>(1-p)\sum_{j
}c_{ij}$, since the optimal $q>0$, this suggests that increasing $q$ towards infinity is favorable. However, given $\eta=\frac{-2\left(\sum_{j}\mu_{ij}-\epsilon\right)}{\sum_{j}\frac{c^{2}_{ij}}
{q^{2}}\Phi(p)}$, $\eta$ also tends to infinity, causing the loosening of Eq. (9).
Thus, we propose selecting a fixed $\eta_{\star}$ to ensure the overall consistency of the inequalities in Eq. (9) and Eq. (10). Specifically, in our implementation of DAREx-q with per-layer tuning, $\eta_{*}$ is computed via grid search (see Sec. C.8). Substituting $\eta=\eta_{\star}$ into Eq. (10), we obtain:

$$ $\displaystyle\epsilon=\log\frac{2}{\gamma}+\eta_{\star}\left(1-\frac{1}{q}(1-p )\right)\sum_{j}c_{ij}+\frac{\eta^{2}_{\star}\Phi(p)\sum_{j}c^{2}_{ij}}{4q^{2}}.$ (12) $$

Given $\eta_{\star}$, we can find the optimal $q$ to minimize $|\epsilon|$ in Eq. (12), which arrives at Eq. (5).

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1 could you explain why you present a different range of regulation strength instead of a unity strength range to better understand the trend of the scale of dp in the Figure 6 ?

2 how do you conduct the experiments on the different regulation strengths in Figure 4? do they keep all other hyperparameters the same?

3 have you try lora + adamr-l2/adamr-l1?
soundness: 3
strengths: 1 The  paper provides a detailed analysis of DARE about the failure cases 

2 base on the observation of DARE, they proposed two improved methods

3 their present the effectiveness of their method through theoretical and experimental points.

4 good presentation for reading 

5 consider different scenarios
summary: The paper proposes improved algorithms for Delta-Parameter Pruning (DPP), notably DARq and AdamR, which achieve substantial performance gains at high pruning rates.   It begins with a comprehensive analysis of DARE, an existing DPP approach, and leverages insights from this analysis to develop two enhanced algorithms. Furthermore, the proposed DARq method can be integrated with a parameter-efficient fine-tuning strategy to further minimize delta parameters. Additionally, the paper provides guidance on applying DPP across various scenarios.
weaknesses: see questions

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Please refer to the weaknesses.
soundness: 2
strengths: S1. The problem considered is quite interesting and not well-addressed in the literature

S2. The paper is well written

S3. Empirical results evidently show that DARq is performs quite well. The other insights provided are valuable too.
summary: This work makes progress over DARE to obtain Prunable Delta Paramters at higher pruning rates. Towards this, the authors propose DARq, that modifies, the rescaling factor based on the insight that it is more prudent to look at output layer perturbations when pruning DPs as opposed to worrying about the expected change in delta parameters. The authors also propose to use AdamR, different from AdamW since it uses regularisation based on gradient norm, to obtain DPs that are more prunable than those obtained by AdamW. The authors also experiment with importance based pruning methods to show that they can obtain better result than random pruning. They also show that DARq works with structural pruning, LoRA.
weaknesses: W1. This paper seems quite exploratory. The authors do not conclude by providing a single method that they recommend to use for obtaining DPPs.

W2. Conclusions derived from Theorem 3.1 seem misleading. The authors provide an upper bound as opposed to a lower bound. On line 229, authors claim "from Thm. 3.1 as p increases, the absolute difference in outputs $|h^{diff}_{i} |$ grows at a rate of $\mathcal{O}((1 − p)^{-\frac{1}{2}})$, which is incorrect. But rather, $|h^{diff}_{i} |$ grows at most by $\mathcal{O}((1 − p)^{-\frac{1}{2}})$. It's not clear how tight this bound is to precisely make subsequent analysis hold true.



Typos:
Line 137: powerful

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: * In Table 3, could you elaborate on why global rescaling outperforms local rescaling?
* In Table 5, after a 0.99 pruning rate, the pruned model seems to be able to achieve better performance than the original and AdamR-L2 fine-tuned models. For instance, in the RoBERTa-COLA cell, it reaches 59.01 at a 0.99 pruning rate with a standard deviation of 1.26, indicating that it can outperform the unpruned model (59.27) in optimal test runs. Does this suggest the potential for further pruning? Could you experiment with even higher pruning rates to assess the method's limits?
soundness: 3
strengths: * This work provides a clear analysis of why DARE struggles with high pruning rates and large delta parameter mean/variance cases. Additionally, it includes a theoretical proof of DARq’s effectiveness as a modified rescaling strategy.
* The paper presents comprehensive experiments on both encoder- and decoder-based language models across multiple datasets. DARq, with various rescaling variations, consistently outperforms other state-of-the-art methods across models and datasets, demonstrating its robustness. Moreover, DARq can be combined with parameter-efficient tuning approaches, such as LoRA, broadening its applicability.
* The AdamR fine-tuning method achieves promising results, effectively maintaining performance even at high pruning rates.
summary: The paper addresses limitations in the random-rescaling Delta-parameter pruning method (DARE), which struggles with high pruning rates or large mean/variance in delta parameters. To tackle this, the authors propose two new approaches: (1) DARq, a post-training rescaling modification strategy, and (2) AdamR, a DDP-aware fine-tuning method. They validate the effectiveness of these methods through experiments on various model architectures and datasets.
weaknesses: * This paper only compares their method with DARE, but there are lots of other methods proposed in the neural network pruning community. I wonder how this method compares to some existing pruning methods. For example, measuring the importance of weights in the original model and pruning the delta parameters based on that.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: 1. **Scalability and Applicability to Other Domains**: While your methods have shown impressive results on language models, have you considered how DARq and AdamR might scale to even larger models or different architectures beyond NLP? Specifically, do you anticipate any challenges or necessary modifications when applying these techniques to other domains such as computer vision or speech recognition?

2. **Automated Selection of Rescaling Factor in DARq**: The selection of the rescaling factor \( q \) in DARq appears to be crucial for maintaining performance after pruning. Have you explored automated methods for selecting \( q \) without relying on a validation set or manual tuning? For instance, could analyzing the statistics of the delta parameters during pruning provide a way to determine an optimal \( q \) more systematically?

3. **Impact of AdamR on Training Dynamics**: Introducing delta regularization through AdamR alters the fine-tuning process. How does this affect the convergence, stability, and overall training dynamics compared to standard fine-tuning methods? Are there any trade-offs, such as increased training time or complexity, that practitioners should be aware of when integrating AdamR into their training pipelines?
soundness: 4
strengths: - **Comprehensive Analysis**: The paper provides a thorough analysis of the limitations of the existing DARE method, identifying the root causes of its failure at high pruning rates.
- **Innovative Solutions**: The proposed methods, DARq and AdamR, are well-motivated and offer practical improvements over existing techniques.
- **Empirical Validation**: Extensive experiments on both encoder and decoder models across various datasets showcase the effectiveness of the proposed methods, with significant performance gains.
- **Flexibility and Compatibility**: The authors demonstrate that DARq can be combined with existing fine-tuning techniques like LoRA and can be used for structural pruning, increasing its practical utility.
- **Revisiting Importance-Based Pruning**: By exploring the application of importance-based pruning methods within DPP, they offer valuable insights into scenarios where these methods may be more effective.
- **Clear Recommendations**: The paper provides a practical pipeline for selecting the appropriate DPP method under different scenarios, which is valuable for practitioners.
summary: The authors revisit delta-parameter pruning (DPP) for fine-tuned models, focusing on the limitations of the existing Random Drop and Rescale (DARE) method when dealing with high pruning rates or large delta parameters. They identify two key issues causing DARE's performance degradation: the excessively large rescaling factor at high pruning rates and the high mean and variance in the delta parameters. To address these challenges, they propose two algorithmic improvements:

1. **DARq**: A modification of DARE that adjusts the rescaling factor, leading to significant performance gains at high pruning rates.
2. **AdamR**: An in-training regularization technique that incorporates delta (L2) regularization during fine-tuning to control the mean and variance of delta parameters before applying DPP.

They demonstrate that their methods outperform the original DARE, especially at extreme pruning rates, and can be combined with parameter-efficient fine-tuning techniques like LoRA. Additionally, they show that importance-based pruning methods can outperform random-based methods when delta parameters are large.

The findings of this work seem very high utility and are of substantial increase in performance in edge cases where the original methods failed very strongly. A very good step forward for DPP methods.
weaknesses: - **Complexity of Methods**: While the proposed methods are effective, they introduce additional complexity, such as tuning the rescaling factor \( q \) in DARq, which may require validation data or additional computations.
- **Limited Theoretical Depth**: The theoretical analysis, while helpful, could be further deepened to provide more rigorous guarantees or insights into why the proposed methods work.
- **Applicability Scope**: The methods are primarily validated on language models and NLP tasks; it would strengthen the paper to demonstrate their applicability to other domains or types of neural networks.
- **Clarity in Presentation**: Some sections, particularly the methodology and theoretical analysis, could benefit from clearer explanations and more intuitive descriptions to make them accessible to a broader audience.
- **Dependence on Regularization**: The AdamR method relies on in-training modifications and specific regularization strategies, which may not always be feasible or may interact unpredictably with other training dynamics.

Meta-review: This paper analyses the "DARE" delta-parameter pruning method, and its failure mechanisms. It then proposes two algorithmic improvements in the form of "DARq" and "AdamR". Reviewers appreciated the motivation and the analysis of DARE, as well as strong, comprehensive experimental results. Criticisms were fairly minor, but included a lack of comparison to other pruning methods, issues with the theory, and a lack of validation outside of language domains (points that were later addressed by the authors). The authors did well with their responses, leading to two reviewers increasing their scores. With entirely positive reviews, solid experiments, and a lack of major weaknesses, I believe this paper should be accepted.

Decision: Accept (Spotlight)