Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models
Authors: Xinxin Liu, Aaron Thomas, Cheng Zhang, Jianyi Cheng, Yiren Zhao, Xitong Gao
Accepted at: acl25

Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models

Xinxin Liu1,2    Aaron Thomas3    Cheng Zhang4
Jianyi Cheng5    Yiren Zhao4    Xitong Gao2,6*
1 Southern University of Science and Technology
2 Shenzhen Institutes of Advanced Technology, CAS
3 University of Birmingham   4 Imperial College London   5 University of Edinburgh
6 Shenzhen University of Advanced Technology

Abstract

Parameter-Efficient Fine-Tuning (PEFT) has gained prominence through low-rank adaptation methods like LoRA. In this paper, we focus on sparsity-based PEFT (SPEFT), which introduces trainable sparse adaptations to the weight matrices in the model, offering greater flexibility in selecting fine-tuned parameters compared to low-rank methods. We conduct the first systematic evaluation of salience metrics for SPEFT, inspired by zero-cost NAS proxies, and identify simple gradient-based metrics is reliable, and results are on par with the best alternatives, offering both computational efficiency and robust performance. Additionally, we compare static and dynamic masking strategies, finding that static masking, which predetermines non-zero entries before training, delivers efficiency without sacrificing performance, while dynamic masking offers no substantial benefits. Across NLP tasks, a simple gradient-based, static SPEFT consistently outperforms other fine-tuning methods for LLMs, providing a simple yet effective baseline for SPEFT. Our work challenges the notion that complexity is necessary for effective PEFT, while our open-source framework establishes a reproducible benchmark for future research1.

1 Introduction

Pretrained large language models (LLMs) have demonstrated strong performance across various natural language processing (NLP) tasks (Brown et al., 2020). A typical approach for adapting these LLMs to specific downstream tasks involves fine-tuning their trainable parameters. However, this process can be prohibitively expensive on consumer-grade hardwares, if we consider training all free parameters, especially on LLMs exceeding a billion parameters. For example, models with over 100 billion parameters, such as BLOOM, required training with 384 GPUs across 48 distributed computing nodes (Luccioni et al., 2023). Instead of training all parameters, an alternative fine-tuning paradigm that enables model training on new tasks with minimal computational resources is Parameter-Efficient Fine-Tuning (PEFT). This method aims to learn only a small set of parameters in order to adapt the model to the new task, substantially lowers the computational resource requirements (Ansell et al., 2021; Hu et al., 2021).

Existing effort on PEFT methods mainly focuses on two categorizes, low-rank-based and sparsity-based adaptation approaches. LoRA (Hu et al., 2021), a popular low-rank adaptation method reparameterizes the model weight of each layer (\( \theta \in \mathbb{R}^{d_1 \times d_2} \)) as \( \theta \triangleq \theta_0 + BA \), where \( \theta_0 \) denotes the pretrained weight matrix which remains fixed during fine-tuning, \( B \in \mathbb{R}^{d_1 \times r} \) and \( A \in \mathbb{R}^{r \times d_2} \) are trainable weights of a lower rank with \( r \ll \min\{d_1, d_2\} \). Recently, sparsity-based PEFT (SPEFT) has emerged as an alternative approach which constructs an alternate reparameterization, \( \theta \triangleq \theta_0 + \theta_{sp} \), where \( \theta_{sp} \) is an extremely sparse matrix, and updates solely its non-zero entries. Figure 1 illustrates the distinction between the two categories of PEFT methods. Previous sparse PEFT methods (Guo et al., 2020; Sung et al., 2021; Ansell et al., 2021) have employed various first- and second-order metrics for determining these non-zero entries and adopted distinct approaches for handling the sparsity mask during training. The varying constructions and training-time treatments of the sparsity mask lead us to the following research questions on the basic design principles for SPEFT:

• Which salience metric or proxy is optimal for determining a sparsity mask?

• Is a static mask determined prior to the start
Figure 1: Comparison between LoRA (Hu et al., 2021) and SPEFT. LoRA freezes pretrained weights \( \theta_0 \) and updates the low-rank terms \( A \) and \( B \), while SPEFT adopts zero-cost proxies to build a sparse adapter \( \theta_{sp} \), to update the weight elements that contribute most to the downstream task.

of training sufficient, or is a dynamically updated pruning mask preferable?

In this paper, we systematically re-examine the design principles for SPEFT and conduct an evaluation across distinct salience metrics. Drawing inspiration from recent advancements in zero-cost Network Architecture Search (NAS) proxies, which explore diverse low-cost proxies for determining parameter importance that has incorporated both first-order (\( e.g. \), weight magnitude, gradients, SNIP (Lee et al., 2019b), etc.) and second-order estimators (\( e.g. \), GRaSP (Wang et al., 2020), Fisher information (Sung et al., 2021), etc.), we discovered that these NAS proxies encompasses many salience metrics used in SPEFT for sparsity mask construction (DiffPruning (Guo et al., 2020), FishMASK (Sung et al., 2021), etc.). Consequently, inspired by recent zero-cost NAS metrics that have shown strong performance to construct sparsity masks, we are the first to comprehensively evaluate 8 different salience metrics in the context of SPEFT for LLMs. Furthermore, we investigate both dynamic and static masking approaches, where a dynamic mask matrix \( \tau \) changes during training, while a static mask maintains a static \( \tau \) binary matrix throughout the PEFT process. We make the following contributions:

• We systematically evaluate 8 different salience metrics for constructing sparsity masks in SPEFT and empirically show that gradient-based SPEFT offers strong performance, while second-order metrics, such as Fisher information, do not significantly enhance SPEFT performance.

• We found that dynamic masking strategies do not surpass the effectiveness of a simple static mask predefined before training in SPEFT. This approach affords greater acceleration opportunities, as fixed indices are predetermined and this avoids the mask re-computation cost.

• Our results indicate that a simple gradient-based, static SPEFT method delivers the best trade-off between effectiveness and efficiency. For instance, for RoBERTa-base (Liu et al., 2019) on MRPC (Dolan and Brockett, 2005) task, our method achieves 0.98% higher than the baseline given the same amount of trainable parameters. Gradient-based SPEFT outperforms LoRA by 22.6% on GSM8k (Cobbe et al., 2021) when trained on MetaMathQA (Yu et al., 2024). Consequently, we advocate for this SPEFT variant to be considered a strong baseline for subsequent developments in this field.

2 Related Work

2.1 PEFT Methods

With the advent of large language models, fine-tuning these models on downstream tasks can be prohibitively expensive due to the sheer number of trainable parameters. A suite of parameter-efficient fine-tuning (PEFT) methods have been proposed to address this issue.

Low-rank adaptation (Hu et al., 2021) is a popular method in PEFT which reparameterizes
the weight matrix of each layer (\( \theta \in \mathbb{R}^{d_1 \times d_2} \)) as \( \theta = \theta_0 + B A \). Here, \( \theta_0 \in \mathbb{R}^{d_1 \times d_2} \) is the pretrained weight matrix, and \( B \in \mathbb{R}^{d_1 \times r} \) and \( A \in \mathbb{R}^{r \times d_2} \) are lower-rank matrices with \( r \ll \min(d_1, d_2) \). By making only \( A \) and \( B \) trainable, this method significantly reduces the number of trainable parameters, thereby lowering computational resource requirements. LoRA has demonstrated effectiveness in reducing trainable parameters for fine-tuning large language models, while maintaining strong fine-tuned performance across various downstream tasks compared to full fine-tuning.

Sparsity-based adaptation Since the advent of low-rank adaptation, sparsity-based adaptation has emerged as an alternative approach to PEFT. It constructs sparse trainable matrix \( \theta_{sp} \) reparameterization for each layer weight \( \theta = \theta_0 + \theta_{sp} \), where \( |\theta_{sp}|_0 \leq s \ll d_1 \times d_2 \), and \( s \) represents the number of non-zero entries. The gradient updates only happen to the non-zero entries of the sparse matrices during fine-tuning. Since the sparse matrix \( \theta_{sp} \) is typically constructed to be extremely sparse, this approach can also achieve notable parameter efficiency, and the sparsity masking strategy plays a crucial role in determining impactful trainable parameters for fine-tuning.

This approach has been explored in various forms in the literature. Earlier works such as Diff-Pruning (Guo et al., 2020) learns a sparsity mask with straight-through gradient estimator (Bengio et al., 2013; Hubara et al., 2016) to select important parameters for downstream tasks. FishMASK (Sung et al., 2021) applies a static sparsity mask from training outset, guided by Fisher information to measure sparsity. Beyond static masks, Fish-DIP (Das et al., 2023) further allows the Fisher information-based mask to be updated dynamically during training. Inspired by the lottery ticket hypothesis (Frankle and Carbin, 2019), LF-SFT (Ansell et al., 2021) finds that sparse masks obtained by selecting the parameters with the largest changes after fine-tuning on a task can be transferred to other tasks. However, this approach requires full fine-tuning on an initial task, which may not be feasible for resource-constrained settings. This paper explores the design principles for constructing the sparsity mask with low-cost salience metrics and the impact of dynamic versus static masks on the fine-tuning process.

Finally, sparsity-based adapters also allow highly granular control over trainable parameters, and can enable the use of existing knowledge transfer techniques, such as mixtures of sparse experts (Xu et al., 2024) and multi-task learning with sparse masks (Sun et al., 2020) in LLMs.

2.2 Salience Proxies for Sparsity Masking

The extensive research on low-cost salience metrics for fine-grained network pruning has provided a rich set of pruning-at-initialization metrics to determine the importance of neural network parameters. These metrics can be broadly classified into first- and second-order categories. First-order metrics include weight magnitude (Han et al., 2015), connection sensitivity (SNIP) (Lee et al., 2019a), foresight connection sensitivity (FORCE) (de Jorge et al., 2021), Taylor-FO (Molchanov et al., 2019), SynFlow (Tanaka et al., 2020), and finally, the gradient of the loss with respect to the weight. Second-order metrics comprise GRApSP (Wang et al., 2020) and Fisher information-based metrics (Liu et al., 2021). Coincidentally, both FishMASK (Sung et al., 2021) and Fish-DIP (Das et al., 2023) propose to use Fisher information to construct the sparsity mask: while FishMASK uses a static mask, Fish-DIP further allows the mask to be updated periodically during fine-tuning. These metrics are designed to identify important parameters or connections in a neural network. In this paper, we explore the impact of these salience metrics on fine-tuning by using them to construct sparse masks for PEFT.

3 Method

3.1 Problem Formulation

Given a pretrained model \( f_{\theta_0} \) with initial parameters \( \theta_0 \), a dataset \( \mathcal{D}_{train} \), and a downstream task loss function \( \mathcal{L} \), the goal of sparse parameter-efficient fine-tuning (SPEFT) is to find a set of sparse trainable parameters \( \theta_{sp} \), that minimizes the loss function on the training dataset \( \mathcal{D}_{train} \):

\[
\theta_{sp}^* = \arg\min_{\theta_{sp}} \mathbb{E}_{(\mathbf{x}, y) \sim \mathcal{D}_{train}} \left[ \mathcal{L}(f_{\theta_0 + \theta_{sp}}(\mathbf{x}); y) \right].
\]

To ensure the sparsity of \( \theta_{sp} \), we constrain it with \( 1[\theta_{sp} \neq 0] = \tau \), where \( 1[\cdot] \) is the indicator function, \( \tau \in \{0, 1\}^{d_1 \times d_2} \) is the sparsity mask with \( |\tau|_0 \leq \rho \ll d_1 \times d_2 \), where \( \rho \) is the number of non-zero entries. This opens up the flexibility of \( \tau \) design, i.e., selecting the non-zero locations in \( \theta_{sp} \) to update during fine-tuning, which can be determined by various salience metrics as discussed below in Section 3.2.
3.2 Salience Metrics

In this section, we describe the 8 salience metrics which can be used to determine the importance of weights \( \theta \). Assume that \( x \) is the sampled input, \( \ell \triangleq \mathcal{L}(f_{\theta}(x); y) \) is the loss function, \( \odot \) denotes element-wise multiplication, and \( |\cdot| \) denotes the element-wise absolute value. For simplicity, we also assume all data-aware metrics to be expectations over the training dataset \( (x, y) \sim \mathcal{D}_{\text{train}} \), which can be approximated by sampling from it. We have the following 6 1st-order salience metrics:

• Magnitude: \( |\theta| \), where simply the magnitude (\emph{i.e.}, absolute value) of the weight is used.

• Gradient: \( \frac{\partial \ell}{\partial \theta} \), which is the gradient of the loss with respect to the weight \( \theta \).

• SNIP (single-shot network pruning): \( |\frac{\partial \ell}{\partial \theta} \odot \theta| \), the connection sensitivity metric proposed in (Lee et al., 2019a) to determine the importance of weights.

• FORCE (foresight connection sensitivity): \( -\frac{\partial \ell}{\partial \theta} \odot \theta \), introduced in (de Jorge et al., 2021).

• Taylor-FO (Taylor first-order expansion): \( (\frac{\partial \ell}{\partial \theta} \odot \theta)^2 \), derived from the 1st-order Taylor expansion of the loss (Molchanov et al., 2019).

• SynFlow (iterative synaptic flow pruning): \( \frac{\partial}{\partial \theta} \left[ \mathbf{1}^\top \left( \Pi_{l=1}^L |\theta^{(l)}| \right) \mathbf{1} \right] \odot \theta \), where \( \theta^{(l)} \) denotes the weights of the \( l \)th layer, and \( L \) denotes the number of layers. A data-free metric proposed in (Tanaka et al., 2020) to model synaptic flow.

In addition, the 2nd-order salience metrics are computed as follows, where \( H \triangleq \frac{\partial^2 \mathcal{L}(f_{\theta}(x); y)}{\partial \theta \partial \theta^\top} \) denotes the Hessian matrix:

• GRaSP (gradient signal preservation): \( -(H \frac{\partial \ell}{\partial \theta}) \odot \theta \), which is a 2nd-order metric proposed in (Wang et al., 2020) that aims to preserve gradient signals rather than the loss value.

• Fisher information: \( \left( \frac{\partial \ell}{\partial \theta} \right)^2 \), which uses the Fisher information to determine the importance of weights (Sung et al., 2021; Das et al., 2023).

3.3 Sparsity Masking

Global Sparsity Masking Given a salience metric \( S(\theta) \) of the weight \( \theta \) defined in Section 3.2, we can construct the sparse binary mask \( \tau \) by selecting the top \( \rho \in (0, 1] \) fraction of the salience metric values, \emph{i.e.}, \( \rho \) denotes the density level, namely:

\[
\tau = \mathbf{1}[s \geq \text{top}_\rho(s)], \text{ where } s = S(\theta).
\]

Here \( \mathbf{1} \) is the indicator function, and \( \text{top}_\rho \) selects the top \( \rho \) values.

Local Sparsity Masking Instead of ranking the salience metric values across all weight values, alternatively, we can construct layer-wise masks \( \tau^{(l)} \) for the individual weights \( \theta^{(l)} \) in each layer \( l \), where each layer has a shared sparsity \( \rho \), and the top \( \rho \) values are selected from the salience metric values of the weights in that layer:

\[
\tau^{(l)} = \mathbf{1}\left[ s^{(l)} \geq \text{top}_\rho(s^{(l)}) \right], \text{ where } s^{(l)} = S(\theta^{(l)}).
\]

Here, \( \theta \) is decomposed into layer-wise weights \( [\theta^{(1)}, \ldots, \theta^{(L)}] \) and \( \tau^{(l)} \) and \( \theta^{(l)} \) respectively denotes the mask and weights of the \( l \)th layer.

3.4 Static vs. Dynamic Masks

Beyond generating a static mask using the above approach prior to fine-tuning, which remains fixed throughout the training process, we can also explore the use of dynamic masks, which are updated periodically during training. The dynamic mask can be refreshed at specific intervals by the following procedure: first, we apply the current trained weights to the model; we then re-rank the salience metric values with these weights, the top \( \rho \) values are then selected to form a new mask using the updated salience metric values; subsequently, the fine-tuning process continues with the new mask. Notably, after updating the dynamic masks, we also need to reinitialize memory-based optimizers in order to avoid applying incorrect momentum values to the newly adapted sparse weights.

3.5 The SPEFT Algorithm

Algorithm 1 provides an overview of the proposed SPEFT algorithm to fine-tune models with sparse weight adaptations. The algorithm takes as input a pretrained model \( f_{\theta_0} \), an optimizer Opt, a training dataset \( \mathcal{D}_{\text{train}} \), a batch size \( B \), a loss function \( \mathcal{L} \), a salience metric \( S \), a sparsity level \( \rho \), the number of fine-tuning steps \( T \), the learning rate \( \alpha \), and the mask update interval \( I \). The algorithm begins by
Algorithm 1 Sparse Parameter-Efficient Fine-Tuning (SPEFT)

Require: Pretrained model \( f_{\theta_0} \), training dataset \( \mathcal{D}_{\text{train}} \), batch size \( B \), loss function \( \mathcal{L} \), salience metric \( \mathcal{S} \), sparsity level \( \rho \), fine-tuning steps \( T \), fine-tuning learning rate \( \alpha \), mask update interval \( I \)
1: \( \theta_{sp} \leftarrow 0; \theta \leftarrow \theta_0 \) \( \triangleright \) Initialize weights
2: for \( t = 1 \) to \( T \) do \( \triangleright \) For each fine-tuning step…
3:   if \( t = 1 \lor (I \geq 0 \land (t \bmod I = 0)) \) then \( \triangleright \) If salience masks should update… (Section 3.4)
4:     \( (\theta, \theta_{sp}) \leftarrow (\theta + \theta_{sp}, 0) \) \( \triangleright \) Apply sparse weights to model
5:     \( s \leftarrow \mathcal{S}(\theta) \) \( \triangleright \) Compute salience values for all weights (Section 3.2)
6:     \( \tau \leftarrow 1[s \geq \top_\rho(s)] \) \( \triangleright \) Update mask by top-\( \rho \) values (Section 3.3)
7:   end if
8:   \( (\mathbf{x}_{[1:B]}, y_{[1:B]}) \leftarrow \) minibatch(\( \mathcal{D}_{\text{train}} \)) \( \triangleright \) Sample mini-batch
9:   \( \ell \leftarrow \frac{1}{B} \sum_{b=1}^B \mathcal{L}(f_{\theta+\theta_{sp}}(\mathbf{x}_b); y_b) \) \( \triangleright \) Forward pass
10:  \( \theta_{sp} \leftarrow \text{Opt}\left(\alpha, \theta_{sp}, \tau \odot \frac{\partial \ell}{\partial \theta_{sp}}\right) \) \( \triangleright \) Parameter-efficient optimization of sparse weights
11: end for \( \triangleright \) NOTE: only need to compute non-zero entries of \( \tau \) for the gradient
12: return \( \theta + \theta_{sp} \) \( \triangleright \) Return fine-tuned model

initializing the sparse weights \( \theta_{sp} \) to zero (line 1), and then iterates for \( T \) steps (line 2). In each iteration, the algorithm first checks if it is the initial iteration, which requires updating the mask, or if it is at the correct interval for iterative dynamic mask updates (line 3). If either of these conditions is true, the algorithm applies the current sparse weights to the model (line 4), evaluates the new salience values \( s \) (line 5), and updates the salience mask \( \tau \) for the updated weights, on the sparsity level \( \rho \) (line 6). After updating the mask, the training step follows by sampling a mini-batch \( \{x, y\} \) from the training dataset (line 8), and learns the sparse weights \( \theta_{sp} \) (line 9) using the optimizer Opt (e.g., stochastic gradient descent, Adam, etc.). Here, \( \tau \odot \alpha \frac{\partial \ell}{\partial \theta_{sp}} \) where \( \odot \) denotes element-wise multiplication. In terms of actual implementation, only the non-zero entries in \( \frac{\partial \ell}{\partial \theta_{sp}} \) dictated by the mask \( \tau \) are computed and updated. Finally, the algorithm returns the fine-tuned model \( \theta_0 + \theta_{sp} \).

4 Experimental Results

Models We evaluated our approaches and baselines over a set of models, including fine-tuned OPT variants (-125m, -350m, and -1.3b) (Zhang et al., 2022), BERT-base-uncased (Devlin et al., 2019) and RoBERTa-base (Liu et al., 2019), for the GLUE (Wang et al., 2019) benchmark, and fine-tuned Gemma2-2b (Team et al., 2024) and Qwen2-7b (Yang et al., 2024), to evaluate on the Massive Multitask Language Understanding (MMLU) benchmark (Hendrycks et al., 2021) and GSM8K (Cobbe et al., 2021), a dataset of grade school math problems. Moreover, we fine-tuned Llama3-8b (Grattafiori et al., 2024) to evaluate on the HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) benchmarks. In addition to sparse PEFT methods presented in this paper, we further include LoRA (Hu et al., 2021) and PiSSA (Meng et al., 2024) as low-rank adapter baselines for comparison.

Benchmarks To show the generality of our approach, we chose GLUE, MMLU, GSM8K, HumanEval and MBPP as benchmarks for evaluation. For the GLUE (Wang et al., 2019) benchmark, six representative tasks with large sizes are selected: single-sentence task SST-2, inference tasks QNLI, MNLI, similarity and paraphrase tasks MRPC, STS-B and QQP2. For the MMLU (Hendrycks et al., 2021) benchmark, it contains questions covering 57 subjects across STEM, the humanities, the social sciences, and others. It is designed to test the model’s ability to handle various types of language data and complex problems. We fine-tuned Gemma2-2b, Qwen2-7b on either the Alpaca (Taori et al., 2023) or OASST2 (Köpf et al., 2023) conversational datasets, and then evaluated them on all tasks in MMLU. We fine-tuned Gemma2-2b on the MetaMathQA (Yu et al., 2024) dataset and evaluated on GSM8K (5-shots) to assess the models’ multi-step mathematical reasoning ability. Furthermore, we fine-tuned Llama3-8b on the CodeFeedback (Zheng et al., 2024), to evaluate on the HumanEval and MBPP which are aimed to test the code generation ability of models. In the

2We did not evaluate CoLA and RTE because these datasets are too small and require special treatments such as fine-tuning RTE using an MNLI checkpoint (Lan et al., 2019).
results, we reported the match accuracy for MNLI, Pearson correlation for STS-B, flexible extract and strict match scores for GSM8K, Pass@1 for HumanEval and MBPP, and accuracy values for other tasks.

Baselines We chose LoRA (Hu et al., 2021) and PiSSA (Meng et al., 2024) as the competing low-rank baselines across models and benchmarks. By default in all comparisons, SPEFT methods use global sparsity ranking with static masks. For statistical significance, we repeated each experiment 3 times for OPT-{125m,350m}, BERT-base-uncased, and RoBERTa-base, and reported average metrics and their standard deviations.

Ablation Analyses We used the most reliable salience metric, i.e., gradient-based, in further experiments to explore questions related to dynamic vs. static masks, and global vs. local sparsity in Section 4.3. Additionally, we also explored the efficiency-performance trade-off between LoRA, PiSSA and sparse baselines in Section C.

Hyperparameters Our SPEFT methods introduce a hyperparameter \( \rho \), the percentage of trainable parameters. To ensure a fair comparison, we fixed \( \rho \) of our SPEFT methods to use the same amounts of trainable parameters as LoRA and PiSSA on every model, and kept the remaining hyperparameters always the same. For example, for the RoBERTa-base model, we performed a grid sweep over learning rates from \( 5 \times 10^{-4} \) to \( 5 \times 10^{-5} \) to search for the best. Details about the hyperparameter settings can be found in Section A.

4.1 Main Results

Our experiments results on OPT-350m and BERT-base-uncased can be seen in Table 1. For additional results on RoBERTa-base, OPT-125m and OPT-1.3b, please refer to Tables 9 to 11 in Section B. Across all models, we observed that among all the approaches, gradient-based SPEFT has the best average accuracy, higher than LoRA and PiSSA. For instance, in OPT-125m and OPT-350m, gradient-based SPEFT achieves 86.92% and 88.45%, that are higher than the best competing SPEFT methods by 0.73% and 0.85% respectively. Particularly on OPT-350m, gradient-based SPEFT has the best performance on MNLI, MRPC, SST-2, and STS-B, On QNLI and QQP, LoRA has the best performance while gradient-based SPEFT has a good performance close to it. This shows that although LoRA shows excellent performance on certain tasks, SPEFT methods, particularly with the gradient salience metric, could further push the limit, achieving better results in accuracy. On BERT-base-uncased, we found that while SPEFT with Fisher-Info salience metric outperforms gradient-based SPEFT on QNLI, QQP and SST-2, it has a large gap in performance in the remaining tasks, making gradient-based SPEFT a more reliable and desirable choice. Similar results are also observed for other OPT variants in Tables 10 and 11 and RoBERTa-base in Table 9 of Section B.

Notably, for both causal and masked language models, sparsity-based PEFT can outperform low-rank adapters, and the gradient-based SPEFT shows the strongest performance compared to other methods, closely followed by LoRA and PiSSA, which is consistent across all models. In addition, the gradient-based SPEFT outperformed LoRA and PiSSA in several tasks, highlighting its effectiveness across different model sizes. The comprehensive results table for these models and tasks underlines the consistent performance edge of gradient-based SPEFT, making it a reliable choice for a wide range of NLP tasks.

4.2 Larger Scale Models

For larger models, we evaluated all methods on Gemma2-2b and Qwen2-7b, and show the results in Table 2. The results indicate that larger models can also benefit from SPEFT with the gradient-based saliency method, which outperforms other sparse training methods and LoRA.

To evaluate on the text generation task, We fine-tuned Gemma2-2b with our methods on MetaMathQA and evaluated on 5-shot GSM8K. We also provide the results of the pretrained model (without fine-tuning) and LoRA as baselines. The results are shown in Table 3. It can be seen that the sparse adapters outperformed the LoRA baseline, with the gradient-based SPEFT method leading the pack with the best performance. Furthermore, for code generation tasks, we fine-tuned Llama3-8b with our methods and evaluated on HumanEval and MBPP benchmarks. The results are shown in Table 12 of Section B. Notably, the lead by sparse adapters widens as the task complexity increases, which demands token sequence generation with multi-step reasoning.
<table>
  <tr>
    <th>Method</th>
    <th>MNLI</th>
    <th>MRPC</th>
    <th>QNLI</th>
    <th>QQP</th>
    <th>SST-2</th>
    <th>STS-B</th>
    <th>Avg.</th>
    <th>#</th>
  </tr>
  <tr>
    <th colspan="9">OPT-350m (Trainable = 0.35%)</th>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>83.56±.07</td>
    <td>84.56±.49</td>
    <td><b>89.69±.11</b></td>
    <td><b>89.66±.04</b></td>
    <td>93.87±.06</td>
    <td>88.57±.99</td>
    <td>88.32±.29</td>
    <td>2</td>
  </tr>
  <tr>
    <td>PiSSA</td>
    <td>83.45±.06</td>
    <td>83.09±.52</td>
    <td>89.38±.06</td>
    <td><b>89.66±.02</b></td>
    <td>93.58±.09</td>
    <td>88.39±.52</td>
    <td>87.93±.21</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Magnitude</td>
    <td>79.34±.41</td>
    <td>71.57±.13</td>
    <td>86.45±.06</td>
    <td>87.68±.01</td>
    <td>91.98±.12</td>
    <td>45.04±.39</td>
    <td>77.01±.51</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Gradient</td>
    <td><b>83.86±.06</b></td>
    <td><b>84.80±.55</b></td>
    <td>89.68±.01</td>
    <td>89.51±.01</td>
    <td>93.93±.12</td>
    <td><b>88.95±.25</b></td>
    <td><b>88.45±.02</b></td>
    <td>3</td>
  </tr>
  <tr>
    <td>SynFlow</td>
    <td>77.45±.05</td>
    <td>77.94±.49</td>
    <td>83.19±.03</td>
    <td>88.03±.02</td>
    <td>92.32±.18</td>
    <td>79.18±.63</td>
    <td>83.02±.22</td>
    <td>0</td>
  </tr>
  <tr>
    <td>SNIP</td>
    <td>83.40±.05</td>
    <td>83.09±.37</td>
    <td>89.68±.22</td>
    <td>89.37±.02</td>
    <td>93.75±.06</td>
    <td>86.32±.04</td>
    <td>87.60±.10</td>
    <td>0</td>
  </tr>
  <tr>
    <td>FORCE</td>
    <td>83.25±.08</td>
    <td>82.60±.62</td>
    <td>89.75±.30</td>
    <td>89.50±.03</td>
    <td><b>94.04±.69</b></td>
    <td>85.53±.18</td>
    <td>87.44±.26</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Taylor-FO</td>
    <td>83.31±.08</td>
    <td>83.09±.37</td>
    <td>89.68±.22</td>
    <td>89.37±.02</td>
    <td>93.75±.06</td>
    <td>86.32±.04</td>
    <td>87.59±.12</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GRaSP</td>
    <td>74.78±.27</td>
    <td>83.58±.49</td>
    <td>84.46±.39</td>
    <td>89.38±.03</td>
    <td><b>94.04±.01</b></td>
    <td>86.97±.01</td>
    <td>85.54±.20</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Fisher-Info</td>
    <td>35.45±1.35</td>
    <td>84.31±.61</td>
    <td>88.12±.34</td>
    <td>86.34±.41</td>
    <td>87.16±.35</td>
    <td>88.61±.02</td>
    <td>78.33±.51</td>
    <td>0</td>
  </tr>
  <tr>
    <th colspan="9">BERT-base-uncased (Trainable = 0.27%)</th>
  </tr>
  <tr>
    <td>LoRA</td>
    <td><b>81.45±.41</b></td>
    <td>88.48±1.03</td>
    <td>89.57±.35</td>
    <td>87.77±.54</td>
    <td>91.82±.14</td>
    <td>84.07±1.11</td>
    <td>87.19±.30</td>
    <td>1</td>
  </tr>
  <tr>
    <td>PiSSA</td>
    <td>81.08±.27</td>
    <td>87.75±.43</td>
    <td>90.19±.30</td>
    <td>88.14±.33</td>
    <td>91.51±.08</td>
    <td><b>85.12±.26</b></td>
    <td>87.30±.18</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Magnitude</td>
    <td>77.09±.24</td>
    <td>68.88±.25</td>
    <td>86.60±.07</td>
    <td>85.56±.50</td>
    <td>90.14±.02</td>
    <td>37.59±.193</td>
    <td>74.31±.33</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Gradient</td>
    <td>80.99±.12</td>
    <td><b>89.46±.48</b></td>
    <td>89.90±.26</td>
    <td>87.48±.13</td>
    <td>91.63±.01</td>
    <td>85.08±.06</td>
    <td><b>87.42±.15</b></td>
    <td>2</td>
  </tr>
  <tr>
    <td>SynFlow</td>
    <td>70.85±.21</td>
    <td>71.33±.25</td>
    <td>83.49±.04</td>
    <td>83.69±.16</td>
    <td>90.08±.29</td>
    <td>74.55±.36</td>
    <td>79.00±.12</td>
    <td>0</td>
  </tr>
  <tr>
    <td>SNIP</td>
    <td>80.74±.20</td>
    <td>79.90±.17</td>
    <td>89.39±.08</td>
    <td>87.27±.25</td>
    <td>91.57±.06</td>
    <td>80.92±.41</td>
    <td>84.96±.18</td>
    <td>0</td>
  </tr>
  <tr>
    <td>FORCE</td>
    <td>80.25±.09</td>
    <td>78.31±.86</td>
    <td>88.98±.15</td>
    <td>87.04±.38</td>
    <td>91.57±.17</td>
    <td>79.21±.24</td>
    <td>84.23±.15</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Taylor-FO</td>
    <td>80.74±.20</td>
    <td>79.90±.17</td>
    <td>89.39±.08</td>
    <td>87.27±.25</td>
    <td>91.57±.06</td>
    <td>80.87±.46</td>
    <td>84.96±.18</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GRaSP</td>
    <td>79.37±.27</td>
    <td>77.95±1.72</td>
    <td>87.50±1.12</td>
    <td>87.03±.41</td>
    <td>91.35±.52</td>
    <td>79.67±1.43</td>
    <td>83.81±.59</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Fisher-Info</td>
    <td>79.83±.16</td>
    <td>87.75±.74</td>
    <td><b>90.46±.22</b></td>
    <td><b>88.78±.25</b></td>
    <td><b>91.86±.34</b></td>
    <td>82.79±.63</td>
    <td><b>86.91±.18</b></td>
    <td><b>3</b></td>
  </tr>
</table>

Table 1: Comparing the salience metrics on OPT-350m (with 0.35% trainable parameters) and BERT-base-uncased (with 0.27% trainable parameters) for various GLUE tasks. For reference, we provide the LoRA and PiSSA baselines with the same number of trainable parameters for each model. The “#” column denotes the number of best performing tasks for each method. The best result of each column is highlighted in bold. “Avg.” reports the average score across all tasks, and their average standard deviations.

4.3 Exploration of masking strategies

Based on the comparisons with SPEFT in Section 4.1, which showed that gradient-based SPEFT is the best-performing method, we would use it for ablation studies of dynamic vs. static masks, and global vs. local sparsity. In this section, we delve into the comparisons between global and local sparsity (Section 3.3) and also static and dynamic masking strategies (Section 3.4) using gradient-based SPEFT, the best-performing salience metric, across OPT-125m, OPT-350m, and BERT-base-uncased. Here, we periodically update the masks every \( I = 1000 \) steps with 1024 training examples to estimate the salience metrics. The results are shown in Table 4.

Dynamic vs. static masking The findings reveal that dynamic masking offers only a slight performance advantage in smaller models like BERT-base-uncased but does not significantly outperform static masking in larger models. For instance, on OPT-350m, we actually see static masking provides us a better averaged accuracy (88.46 and 88.71) compared to dynamic masking (86.14 and 81.76). Given that dynamic masking requires more computational resources, because of the periodic update on sparsity masks, the marginal performance gain does not justify the extra cost, especially for larger models. Therefore, static masking emerges as a more practical and resource-efficient strategy, providing substantial performance benefits without the additional computational overhead.

Global vs. local sparsity With global sparsity, SPEFT calculates the metrics across all transformer layers, ranks them collectively, and makes only
<table>
  <tr>
    <th rowspan="2">Model Dataset</th>
    <th colspan="2">Gemma2-2b</th>
    <th colspan="2">Qwen2-7b</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Alpaca</th>
    <th>OASST2</th>
    <th>Alpaca</th>
    <th>OASST2</th>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>53.07</td>
    <td>52.59</td>
    <td>69.77</td>
    <td>70.42</td>
    <td>61.46</td>
  </tr>
  <tr>
    <td>Gradient</td>
    <td><b>53.11</b></td>
    <td><b>53.11</b></td>
    <td><b>70.96</b></td>
    <td>70.55</td>
    <td><b>61.93</b></td>
  </tr>
  <tr>
    <td>SynFlow</td>
    <td>52.84</td>
    <td>53.07</td>
    <td>69.80</td>
    <td>70.66</td>
    <td>61.59</td>
  </tr>
  <tr>
    <td>Magnitude</td>
    <td>52.97</td>
    <td>53.03</td>
    <td>70.12</td>
    <td><b>70.76</b></td>
    <td>61.72</td>
  </tr>
  <tr>
    <td>SNIP</td>
    <td>52.81</td>
    <td>52.89</td>
    <td>68.75</td>
    <td>70.52</td>
    <td>61.24</td>
  </tr>
  <tr>
    <td>FORCE</td>
    <td>52.79</td>
    <td>52.88</td>
    <td>69.01</td>
    <td>70.53</td>
    <td>61.30</td>
  </tr>
  <tr>
    <td>Taylor-FO</td>
    <td>52.81</td>
    <td>52.96</td>
    <td>68.75</td>
    <td>69.10</td>
    <td>60.91</td>
  </tr>
  <tr>
    <td>GRaSP</td>
    <td>52.38</td>
    <td>52.60</td>
    <td>66.69</td>
    <td>69.91</td>
    <td>60.40</td>
  </tr>
  <tr>
    <td>Fisher-Info</td>
    <td>52.70</td>
    <td>52.65</td>
    <td>66.45</td>
    <td>69.10</td>
    <td>60.23</td>
  </tr>
</table>

Table 2: Comparing the salience metrics on Gemma2-2b and Qwen2-7b respectively with 0.97% and 0.53% trainable parameters. We fine-tuned models on either Alpaca or OASST2 and evaluated on 5-shot MMLU. For reference, we provide the LoRA baselines with the same number of trainable parameters for each combination.

<table>
  <tr>
    <th>Method</th>
    <th>Flexible Extract</th>
    <th>Strict Match</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Pretrained</td>
    <td>24.56</td>
    <td>17.66</td>
    <td>21.11</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>39.20</td>
    <td>28.81</td>
    <td>34.00</td>
  </tr>
  <tr>
    <td>Gradient</td>
    <td><b>50.27</b></td>
    <td><b>37.15</b></td>
    <td><b>43.71</b></td>
  </tr>
  <tr>
    <td>SynFlow</td>
    <td>37.76</td>
    <td>27.75</td>
    <td>32.75</td>
  </tr>
  <tr>
    <td>Magnitude</td>
    <td>37.45</td>
    <td>27.07</td>
    <td>32.26</td>
  </tr>
  <tr>
    <td>SNIP</td>
    <td>39.80</td>
    <td>29.64</td>
    <td>34.72</td>
  </tr>
  <tr>
    <td>FORCE</td>
    <td>39.88</td>
    <td>29.95</td>
    <td>34.91</td>
  </tr>
  <tr>
    <td>Taylor-FO</td>
    <td>40.33</td>
    <td>30.25</td>
    <td>35.29</td>
  </tr>
  <tr>
    <td>GRaSP</td>
    <td>50.15</td>
    <td>37.03</td>
    <td>43.59</td>
  </tr>
  <tr>
    <td>Fisher-Info</td>
    <td>41.47</td>
    <td>30.25</td>
    <td>35.86</td>
  </tr>
</table>

Table 3: Comparing the salience metrics on Gemma2-2b with 0.97% trainable parameters. We fine-tuned the model on MetaMathQA and evaluated on 5-shot GSM8K. For reference, we provide pretrained model (without fine-tuning) and the LoRA baseline with the same number of trainable parameters.

<table>
  <tr>
    <th></th>
    <th>MNLI</th>
    <th>MRPC</th>
    <th>QNLI</th>
    <th>QQP</th>
    <th>SST-2</th>
    <th>STS-B</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <th colspan="8">OPT-125m (Trainable = 0.35%)</th>
  </tr>
  <tr>
    <td>SG</td>
    <td>81.41</td>
    <td>83.82</td>
    <td>88.58</td>
    <td>88.71</td>
    <td>91.44</td>
    <td>87.55</td>
    <td><b>86.92</b></td>
  </tr>
  <tr>
    <td>SL</td>
    <td>81.41</td>
    <td>81.86</td>
    <td><b>88.94</b></td>
    <td><b>88.76</b></td>
    <td>91.40</td>
    <td>87.38</td>
    <td>86.63</td>
  </tr>
  <tr>
    <td>DG</td>
    <td>77.71</td>
    <td>82.84</td>
    <td>83.80</td>
    <td>87.36</td>
    <td>89.33</td>
    <td><b>88.28</b></td>
    <td>84.89</td>
  </tr>
  <tr>
    <td>DL</td>
    <td>69.26</td>
    <td>73.53</td>
    <td>80.56</td>
    <td>84.82</td>
    <td>86.35</td>
    <td>87.15</td>
    <td>80.28</td>
  </tr>
  <tr>
    <th colspan="8">OPT-350m (Trainable = 0.35%)</th>
  </tr>
  <tr>
    <td>SG</td>
    <td>83.86</td>
    <td>84.80</td>
    <td><b>89.68</b></td>
    <td>89.51</td>
    <td>93.93</td>
    <td><b>88.95</b></td>
    <td>88.46</td>
  </tr>
  <tr>
    <td>SL</td>
    <td><b>84.31</b></td>
    <td>83.33</td>
    <td>90.63</td>
    <td><b>90.97</b></td>
    <td><b>94.50</b></td>
    <td>88.52</td>
    <td><b>88.71</b></td>
  </tr>
  <tr>
    <td>DG</td>
    <td>78.03</td>
    <td>85.29</td>
    <td>89.22</td>
    <td>84.24</td>
    <td>91.51</td>
    <td>88.54</td>
    <td>86.14</td>
  </tr>
  <tr>
    <td>DL</td>
    <td>78.86</td>
    <td>71.57</td>
    <td>80.84</td>
    <td>84.52</td>
    <td>87.27</td>
    <td>87.52</td>
    <td>81.76</td>
  </tr>
  <tr>
    <th colspan="8">BERT-base-uncased (Trainable = 0.27%)</th>
  </tr>
  <tr>
    <td>SG</td>
    <td>80.99</td>
    <td><b>89.46</b></td>
    <td>89.90</td>
    <td>87.48</td>
    <td>91.63</td>
    <td>85.08</td>
    <td>87.42</td>
  </tr>
  <tr>
    <td>SL</td>
    <td>74.58</td>
    <td>85.54</td>
    <td>89.62</td>
    <td>83.41</td>
    <td>91.06</td>
    <td>85.79</td>
    <td>85.00</td>
  </tr>
  <tr>
    <td>DG</td>
    <td><b>83.17</b></td>
    <td><b>89.46</b></td>
    <td><b>90.32</b></td>
    <td><b>90.27</b></td>
    <td><b>92.20</b></td>
    <td>84.20</td>
    <td><b>88.27</b></td>
  </tr>
  <tr>
    <td>DL</td>
    <td>72.80</td>
    <td>86.52</td>
    <td>83.49</td>
    <td>82.51</td>
    <td>90.25</td>
    <td><b>85.95</b></td>
    <td>83.59</td>
  </tr>
</table>

Table 4: Results of OPT-125m, OPT-350m and BERT-base-uncased with fixed or dynamic gradient masks and global or local sparsity on various GLUE tasks. The dynamic strategy will update the gradient mask every 1000 train steps. “S / D”: static / dynamic masks, “G / L”: global / local sparsity. Runs were repeated 3 times and all results have a standard deviation of < 0.5%.

the highest-ranked ones trainable. In the local approach, metrics are sorted and ranked within each individual layer. Our results showed no significant difference in performance between the two strategies. For instance, the results in BERT-base-uncased suggests that global is superior, by showing a better averaged accuracy across the six GLUE tasks, but the numbers in OPT-350m suggest the reverse under the static masking strategy.

4.4 Minimal Overhead for SPEFT

Computational overhead For all first-order salience metrics, we use a few gradient evaluations to compute the salience scores. Specifically, only 64 steps with a batch size of 16 per estimation are needed (1024 examples), which is negligible compared to the overall training cost. For example, this represents only 0.26% and 0.97% of the training time for one epoch on MNLI and QNLI, respectively. For static masks, this computation is performed once before training; for dynamic masking, it is repeated once per \( I = 1000 \) steps. Second-order metrics such as GRaSP and Fisher-Info require \( 2 \times \) the number of gradient evaluations of first-order metrics to compute the second-order gradients. The magnitude metric requires no additional computation. Finally, we observed no statistically significant difference in training time between the sparse methods and the LoRA baseline.

Memory overhead As we aligned the number of trainable parameters across LoRA and the SPEFT methods, the peak memory usage for both methods are mostly identical, except that the SPEFT methods require a small amount of additional memory overhead to store the indices in CSR format. In all experiments, the overhead is less than 0.5% of the peak memory usage.

5 Discussion

The Trend of Supporting Sparse Computation as Hardware Intrinsics Numerous hardware vendors have introduced specialized hardware features with instruction set extensions tailored for sparse matrix multiplication. Especially in recently announced hardware devices. Mainstream devices
like NVIDIA’s A100 (Choquette et al., 2021), H100 (Choquette, 2023), and H200, as well as offerings from other major vendors or emerging competitors such as AMD’s MI300 (AMD) and Cerebras’ WSE2 (Selig, 2022), are embracing this trend. As hardware support for sparse computation advances, the utility of sparsity-based PEFT, or generally sparse training, is poised to improve substantially. This development will enable both current and future strategies to attain performance levels closer to their full potential, as these calculations won’t require emulation via dense computations, allowing for closer realization of theoretical speedups and savings on FLOPs.

The Role of Salience Measurements  A fundamental element of this study involves reevaluating certain design choices in SPEFT, leading to the discovery that straightforward designs, such as first-order salience proxies, emerge as the most effective methods. Intriguingly, selecting the most salient weights in a neural network has being a long-standing challenge, one that dates back to early weight pruning research by LeCun et al. in 1989 (LeCun et al., 1989). It’s notable that the optimal saliency metric seems to differ – or arguably should differ – among different task setups, such as post-training weight pruning (LeCun et al., 1989), pruning at initialization (Lee et al., 2019b; de Jorge et al., 2021), and zero-cost NAS proxies (Siems et al., 2020). The suggested practice then should be to systematically review a range of known and established proxies to set a solid baseline before designing a complex salience metric.

6 Conclusion

We explored the efficacy of various sparse parameter-efficient fine-tuning (SPEFT) methods in enhancing the performance of LLMs. Our experiments compared LoRA and PiSSA against SPEFT methods with a range salience metrics, and demonstrated that gradient-based SPEFT consistently achieved superior accuracy across different tasks and model architectures. This demonstrates that, although LoRA and PiSSA is effective in certain contexts, SPEFT methods that leverage gradient information can further optimize performance. We also investigated the impact of static versus dynamic sparsity masks, concluding that while dynamic masks do not significantly outperform static masks, and they introduce additional training overhead. Our findings suggest that static masks, combined with the gradient-based salience metric, provide a practical balance between computational efficiency and model accuracy. Overall, our research contributes to the ongoing efforts in making model fine-tuning more efficient and accessible, particularly in resource-constrained settings.

7 Acknowledgments

This work is supported in part by the National Key R&D Program of China (2023YFC3321600), National Natural Science Foundation of China (62376263, 62372443 and 62271496), Guangdong Basic and Applied Basic Research Foundation (2023B1515130002), Natural Science Foundation of Guangdong (2024A1515030209 and 2024A1515011970), Shenzhen Science and Technology Innovation Commission (JCYJ20230807140507015 and JCYJ20220531100804009), and Yu-Liang Lu’s Project Team Development Funding (KY23A102).

8 Limitations

During the experiments, we found that in a few training runs, SPEFT seems less sensitive to hyperparameter changes than LoRA, i.e., on a range of hyperparameter sets, SPEFT always improves model performance, but LoRA fails. Due to limited resources and time, we did not run additional experiments to explore this interesting observation. We leave this exploration for future work. Moreover, similar investigations on parameter efficient fine-tuning could be conducted with non-language-based models or other multimodal models, such as vision large language models (VLLMs), however, these explorations are beyond the current scope of this paper and thus is left as future work.

References

AMD Instinct MI300 Series Accelerators. https://www.amd.com/en/products/accelerators/instinct/mi300.html. Accessed: 2024-03-03.

Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulić. 2021. Composable sparse fine-tuning for cross-lingual transfer. arXiv preprint arXiv:2110.07560.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

Jack Choquette. 2023. NVIDIA Hopper H100 GPU: Scaling Performance. IEEE Micro, (3):9–17.

Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. 2021. NVIDIA A100 tensor core GPU: Performance and innovation. IEEE Micro, (2):29–35.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reijichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.

Sarkar Snigdha Sarathi Das, Ranran Haoran Zhang, Peng Shi, Wenpeng Yin, and Rui Zhang. 2023. Unified low-resource sequence labeling by sample-aware dynamic sparse finetuning. arXiv preprint arXiv:2311.03748.

Pau de Jorge, Amartya Sanyal, Harkirat Behl, Philip Torr, Grégory Rogez, and Puneet K. Dokania. 2021. Progressive skeletonization: Trimming more fat from a network at initialization. In International Conference on Learning Representations.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint, arXiv:1810.04805.

William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).

Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Demi Guo, Alexander M Rush, and Yoon Kim. 2020. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463.

Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2016. Binarized neural networks. Advances in neural information processing systems, 29.

Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. 2023. Openassistant conversations – democratizing large language model alignment. Preprint, arXiv:2304.07327.

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.

Yann LeCun, John Denker, and Sara Solla. 1989. Optimal brain damage. Advances in neural information processing systems, 2.

Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. 2019a. SNIP: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations.

Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. 2019b. Snip: Single-shot network pruning based on connection sensitivity. Preprint, arXiv:1810.02340.

Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. 2021. Group fisher pruning for practical network compression. In International Conference on Machine Learning, pages 7021–7032. PMLR.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692.
Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. 2023. Estimating the carbon footprint of bloom, a 176b parameter language model. Journal of Machine Learning Research, 24(253):1–15.

Fanxu Meng, Zhaohui Wang, and Muhan Zhang. 2024. Pissa: Principal singular values and singular vectors adaptation of large language models. Preprint, arXiv:2404.02948.

Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. 2019. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11264–11272.

Justin Selig. 2022. The cerebras software development kit: A technical overview. Technical Report, Cerebras.

Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. 2020. Nas-bench-301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint arXiv:2008.09777, 4:14.

Tianxiang Sun, Yunfan Shao, Xiaonan Li, Pengfei Liu, Hang Yan, Xipeng Qiu, and Xuanning Huang. 2020. Learning sparse sharing architectures for multiple tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8936–8943.

Yi-Lin Sung, Varun Nair, and Colin A Raffel. 2021. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193–24205.

Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. 2020. Pruning neural networks without any data by iteratively conserving synaptic flow. In International Conference on Learning Representations.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. Preprint, arXiv:1804.07461.

Chaoqi Wang, Guodong Zhang, and Roger Grosse. 2020. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations.

Jiahui Xu, Lu Sun, and Dengji Zhao. 2024. MoME: Mixture-of-masked-experts for efficient multi-task recommendation. In SIGIR, pages 2527–2531.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhengguo Li, Adrian Weller, and Weiyang Liu. 2024. Metamath: Bootstrap your own mathematical questions for large language models. Preprint, arXiv:2309.12284.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuhui Chen, Christopher DeWan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.

Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 2024. Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658.
A Hyperparameters

The hyperparameters we used for all models are shown in Tables 5 to 8. Notably, for all models, the density \( \rho \) was set to make sure the number of trainable parameters across all methods was the same as the LoRA baseline.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Dataset</th>
    <th colspan="3">QNLI</th>
  </tr>
  <tr>
    <th>MRPC</th>
    <th>SST-2</th>
    <th>STS-B</th>
    <th>MNLI</th>
    <th>QQP</th>
  </tr>
  <tr>
    <td rowspan="6">Shared</td>
    <td>Optimizer</td>
    <td>AdamW</td>
    <td>AdamW</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Warmup Ratio</td>
    <td>0</td>
    <td>0</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LR Schedule</td>
    <td>Linear</td>
    <td>Linear</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>16</td>
    <td>64</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td># Epochs</td>
    <td>30</td>
    <td>30</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>4E-4</td>
    <td>5E-5</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Max Seq. Len.</td>
    <td>512</td>
    <td>196</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">LoRA</td>
    <td>LoRA \( r \)</td>
    <td>8</td>
    <td>8</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LoRA \( \alpha \)</td>
    <td>16</td>
    <td>8</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>OPT-125m</td>
    <td>Sparse \( \rho \)</td>
    <td>0.35%</td>
    <td>0.35%</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>OPT-350m</td>
    <td>Sparse \( \rho \)</td>
    <td>0.35%</td>
    <td>0.35%</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>BERT-base</td>
    <td>Sparse \( \rho \)</td>
    <td>0.27%</td>
    <td>0.27%</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>RoBERTa-base</td>
    <td>Sparse \( \rho \)</td>
    <td>0.24%</td>
    <td>0.24%</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

<table>
  <tr>
    <th>Model (Method)</th>
    <th>Hyperparameters</th>
    <th>All datasets</th>
  </tr>
  <tr>
    <td rowspan="6">Shared</td>
    <td>Optimizer</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>Warmup Ratio</td>
    <td>0</td>
  </tr>
  <tr>
    <td>LR Schedule</td>
    <td>Linear</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>5E-5</td>
  </tr>
  <tr>
    <td># Epochs</td>
    <td>30</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>16</td>
  </tr>
  <tr>
    <td>OPT-1.3b (LoRA)</td>
    <td>LoRA \( r \)</td>
    <td>8</td>
  </tr>
  <tr>
    <td>OPT-1.3b (Sparse)</td>
    <td>LoRA \( \alpha \)</td>
    <td>8</td>
  </tr>
  <tr>
    <td>OPT-1.3b (Sparse)</td>
    <td>Sparse \( \rho \)</td>
    <td>0.18%</td>
  </tr>
</table>

Table 5: The hyperparameters we used for all models evaluated on the GLUE benchmark. The percentage of trainable parameters (\( \rho \)) for the sparse models are chosen to be the same as the LoRA models.

<table>
  <tr>
    <th>Model (Method)</th>
    <th>Hyperparameters</th>
    <th>Alpaca</th>
    <th>OASST2</th>
  </tr>
  <tr>
    <td rowspan="5">Shared</td>
    <td>Optimizer</td>
    <td>AdamW</td>
    <td></td>
  </tr>
  <tr>
    <td>Warmup Ratio</td>
    <td>0.03</td>
    <td></td>
  </tr>
  <tr>
    <td>LR Schedule</td>
    <td>Constant</td>
    <td></td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>16</td>
    <td></td>
  </tr>
  <tr>
    <td>Max Seq. Len.</td>
    <td>1024</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3">Gemma2-2b (LoRA)</td>
    <td># Steps</td>
    <td>2000</td>
    <td></td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>5E-5</td>
    <td></td>
  </tr>
  <tr>
    <td>LoRA \( r \)</td>
    <td>64</td>
    <td></td>
  </tr>
  <tr>
    <td>LoRA \( \alpha \)</td>
    <td>16</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Gemma2-2b (Sparse)</td>
    <td># Steps</td>
    <td>2000</td>
    <td></td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>1E-5</td>
    <td>5E-6</td>
  </tr>
  <tr>
    <td>Sparse \( \rho \)</td>
    <td>0.97%</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3">Qwen2-7b (LoRA)</td>
    <td># Epochs/Steps</td>
    <td>3 Epochs</td>
    <td>2000 Steps</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>5E-5</td>
    <td></td>
  </tr>
  <tr>
    <td>LoRA \( r \)</td>
    <td>64</td>
    <td></td>
  </tr>
  <tr>
    <td>LoRA \( \alpha \)</td>
    <td>16</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Qwen2-7b (Sparse)</td>
    <td># Epochs/Steps</td>
    <td>3 Epochs</td>
    <td>2000 Steps</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>5E-5</td>
    <td>5E-6</td>
  </tr>
  <tr>
    <td>Sparse \( \rho \)</td>
    <td>0.53%</td>
    <td></td>
  </tr>
</table>

Table 6: The hyperparameters we used for Gemma2-2b and Qwen2-7b on Alpaca and OASST2. The percentage of trainable parameters (\( \rho \)) for the sparse models are chosen to be the same as the LoRA models.

and strict match metrics. In order to explore the efficiency-performance trade-off, we varied for LoRA \( r \) from 4 to 128 and compare it against SPEFT methods with the same trainable parameters for each config. The LoRA \( \alpha \) was always kept the same as \( r \).

With the same numbers of training parameters, LoRA and SPEFT would use almost identical FLOPs per step, as the added overheads of both are of the same magnitude and much smaller (<0.5% in all of our main experiments) than the base model. There was no noticeable difference between LoRA and SPEFT in terms of computational and memory footprint for all runs.

As is shown in Figure 2, the performance of SPEFT methods improve with increasing trainable parameters while LoRA results are mostly constant with increased parameter budget. Overall, the gradient-based SPEFT outperformed LoRA using fewer trainable parameters, but also widens the gap further as the budget increases.

B Additional Experimental Results

Tables 9 to 11 provide additional respective results on GLUE tasks for the OPT-125m and OPT-1.3b variants, and BERT-base-uncased. Table 12 shows the results on HumanEval and MBPP benchmarks for Llama3-8b model.

C Additional Ablation Studies

We fine-tuned the Gemma2-2b model on the MetaMathQA dataset and evaluated it on the GSM8K_cot task (5-shot) using flexible extract

D Computational Resources

We performed all experiments on a cluster of NVIDIA A100 40GB GPUs. The experiments took around 486 GPU-hours for a single model on all
GLUE subsets and all salient metrics. Besides, it took around 40 GPU-hours for a single model on Alpaca or OASST2 training on all low-rank and sparse PEFT methods. It also took around 80 GPU-hours to train with all methods on MetaMath for GSM8k evaluation. We also spent around 500 GPU-hours aligning the baseline results with the literature and determining fine-tuning hyperparameters.

![Bar charts comparing accuracy vs. trainable parameters for different methods](page_176_393_563_312.png)
(a) Flexible Extract.

![Bar charts comparing accuracy vs. trainable parameters for different methods](page_176_726_563_312.png)
(b) Strict Match.

Figure 2: Varing the number of trainable parameters on Gemma2-2b and GSM8K_cot (5-shot) with LoRA, PiSSA and SPEFT methods. The x-axis represents the percentage of trainable parameters, while the y-axis denotes accuracy.

<table>
  <tr>
    <th>Model (Method)</th>
    <th>Hyperparameters</th>
    <th>MetaMathQA</th>
  </tr>
  <tr>
    <td rowspan="3">Shared</td>
    <td>Optimizer</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>Warmup Ratio</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td>LR Schedule</td>
    <td>Linear</td>
  </tr>
  <tr>
    <td rowspan="5">Gemma2-2b (LoRA)</td>
    <td>Batch Size</td>
    <td>16</td>
  </tr>
  <tr>
    <td># Epochs</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>2E-5</td>
  </tr>
  <tr>
    <td>LoRA \( r \)</td>
    <td>64</td>
  </tr>
  <tr>
    <td>LoRA \( \alpha \)</td>
    <td>16</td>
  </tr>
  <tr>
    <td rowspan="5">Gemma2-2b (Sparse)</td>
    <td>Max Seq. Len.</td>
    <td>1024</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>16</td>
  </tr>
  <tr>
    <td># Epochs</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>2E-5</td>
  </tr>
  <tr>
    <td>Sparse Top \( k \)</td>
    <td>0.18%</td>
  </tr>
  <tr>
    <td>Max Seq. Len.</td>
    <td>1024</td>
  </tr>
</table>

Table 7: The hyperparameters we used for Gemma2-2b on MetaMathQA. The percentage of trainable parameters (\( \rho \)) for the sparse models are chosen to be the same as the LoRA models.

<table>
  <tr>
    <th>Model (Method)</th>
    <th>Hyperparameters</th>
    <th>CodeFeedback</th>
  </tr>
  <tr>
    <td rowspan="3">Shared</td>
    <td>Optimizer</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>Warmup Ratio</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td>LR Schedule</td>
    <td>Linear</td>
  </tr>
  <tr>
    <td rowspan="5">Llama3-8b (LoRA)</td>
    <td>Batch Size</td>
    <td>16</td>
  </tr>
  <tr>
    <td># Epochs</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>2E-5</td>
  </tr>
  <tr>
    <td>LoRA \( r \)</td>
    <td>64</td>
  </tr>
  <tr>
    <td>LoRA \( \alpha \)</td>
    <td>16</td>
  </tr>
  <tr>
    <td rowspan="5">Llama3-8b (Sparse)</td>
    <td>Max Seq. Len.</td>
    <td>512</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>16</td>
  </tr>
  <tr>
    <td># Epochs</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>2E-5</td>
  </tr>
  <tr>
    <td>Sparse Top \( k \)</td>
    <td>0.67%</td>
  </tr>
  <tr>
    <td>Max Seq. Len.</td>
    <td>512</td>
  </tr>
</table>

Table 8: The hyperparameters we used for Llama3-8b on CodeFeedback. The percentage of trainable parameters (\( \rho \)) for the sparse models are chosen to be the same as the LoRA models.
<table>
  <tr>
    <th>Method</th>
    <th>MNLI</th>
    <th>MRPC</th>
    <th>QNLI</th>
    <th>QQP</th>
    <th>SST-2</th>
    <th>STS-B</th>
    <th>Avg.</th>
    <th>#</th>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>86.52±.06</td>
    <td>89.46±.73</td>
    <td>92.11±.29</td>
    <td>88.70±.15</td>
    <td>93.81±.23</td>
    <td>90.30±.01</td>
    <td>90.15±.25</td>
    <td>0</td>
  </tr>
  <tr>
    <td>PiSSA</td>
    <td><b>86.71±.02</b></td>
    <td>89.47±.42</td>
    <td><b>92.20±.09</b></td>
    <td>88.46±.10</td>
    <td>93.75±.14</td>
    <td><b>90.78±.02</b></td>
    <td>90.23±.13</td>
    <td><b>3</b></td>
  </tr>
  <tr>
    <td>Magnitude</td>
    <td>82.58±.46</td>
    <td>31.62±2.05</td>
    <td>88.03±.35</td>
    <td>86.37±.36</td>
    <td>90.60±.23</td>
    <td>15.16±2.64</td>
    <td>65.73±1.01</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Gradient</td>
    <td>86.00±.05</td>
    <td><b>90.44±.11</b></td>
    <td>91.89±.13</td>
    <td>88.78±.05</td>
    <td><b>94.16±.06</b></td>
    <td>90.29±.02</td>
    <td><b>90.26±.04</b></td>
    <td>2</td>
  </tr>
  <tr>
    <td>SynFlow</td>
    <td>75.53±.02</td>
    <td>70.34±.12</td>
    <td>84.37±.01</td>
    <td>85.19±.02</td>
    <td>91.80±.29</td>
    <td>76.92±.44</td>
    <td>80.69±.17</td>
    <td>0</td>
  </tr>
  <tr>
    <td>SNIP</td>
    <td>85.97±.01</td>
    <td>87.01±.25</td>
    <td>91.34±.01</td>
    <td>88.31±.06</td>
    <td>93.92±.29</td>
    <td>87.52±.16</td>
    <td>89.01±.08</td>
    <td>0</td>
  </tr>
  <tr>
    <td>FORCE</td>
    <td>85.64±.05</td>
    <td>85.29±.37</td>
    <td>91.31±.04</td>
    <td>88.39±.04</td>
    <td>93.75±.06</td>
    <td>86.52±.15</td>
    <td>88.48±.07</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Taylor-FO</td>
    <td>85.97±.01</td>
    <td>87.01±.25</td>
    <td>91.34±.01</td>
    <td>88.31±.06</td>
    <td>93.92±.29</td>
    <td>87.52±.16</td>
    <td>89.01±.08</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GRaSP</td>
    <td>79.07±.02</td>
    <td>84.80±.25</td>
    <td>87.88±.02</td>
    <td>88.45±.12</td>
    <td>93.52±.06</td>
    <td>86.81±.24</td>
    <td>86.76±.04</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Fisher-Info</td>
    <td>85.52±.15</td>
    <td>86.76±.35</td>
    <td>91.82±.06</td>
    <td><b>89.16±.03</b></td>
    <td>93.92±.28</td>
    <td>87.51±.05</td>
    <td>89.12±.15</td>
    <td>1</td>
  </tr>
</table>

Table 9: Comparing the salience metrics on RoBERTa-base for various GLUE tasks with 0.24% trainable parameters, following the same format as Table 1.

<table>
  <tr>
    <th>Method</th>
    <th>MNLI</th>
    <th>MRPC</th>
    <th>QNLI</th>
    <th>QQP</th>
    <th>SST-2</th>
    <th>STS-B</th>
    <th>Avg.</th>
    <th>#</th>
  </tr>
  <tr>
    <td>LoRA</td>
    <td><b>81.94±.22</b></td>
    <td>82.84±.23</td>
    <td>88.23±.30</td>
    <td>88.45±.20</td>
    <td><b>91.97±.18</b></td>
    <td>87.25±.47</td>
    <td>86.78±.21</td>
    <td>2</td>
  </tr>
  <tr>
    <td>PiSSA</td>
    <td>81.56±.11</td>
    <td>83.33±.30</td>
    <td>87.99±.32</td>
    <td>88.17±.15</td>
    <td><b>91.97±.11</b></td>
    <td>86.87±.39</td>
    <td>86.65±.15</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Magnitude</td>
    <td>78.03±.14</td>
    <td>76.35±.45</td>
    <td>85.46±1.62</td>
    <td>86.56±1.15</td>
    <td>90.40±.85</td>
    <td>50.32±2.42</td>
    <td>77.85±3.27</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Gradient</td>
    <td>81.41±.01</td>
    <td><b>83.82±.37</b></td>
    <td><b>88.58±.37</b></td>
    <td><b>88.71±.09</b></td>
    <td>91.46±.05</td>
    <td>87.55±.34</td>
    <td><b>86.92±.05</b></td>
    <td><b>3</b></td>
  </tr>
  <tr>
    <td>SynFlow</td>
    <td>81.05±.05</td>
    <td>81.01±.37</td>
    <td>87.92±.07</td>
    <td>88.35±.04</td>
    <td>91.21±.14</td>
    <td>85.47±.75</td>
    <td>85.83±.16</td>
    <td>0</td>
  </tr>
  <tr>
    <td>SNIP</td>
    <td>81.21±.01</td>
    <td>81.62±.74</td>
    <td>88.31±.12</td>
    <td>88.58±.04</td>
    <td>91.32±.53</td>
    <td>86.11±.40</td>
    <td>86.19±.06</td>
    <td>0</td>
  </tr>
  <tr>
    <td>FORCE</td>
    <td>81.31±.09</td>
    <td>79.91±.74</td>
    <td>88.31±.08</td>
    <td>88.46±.04</td>
    <td>91.44±.23</td>
    <td>85.62±.48</td>
    <td>85.84±.02</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Taylor-FO</td>
    <td>81.21±.01</td>
    <td>81.62±.74</td>
    <td>88.31±.12</td>
    <td>88.58±.04</td>
    <td>91.32±.53</td>
    <td>86.11±.40</td>
    <td>86.19±.06</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GRaSP</td>
    <td>81.36±.14</td>
    <td>81.25±.61</td>
    <td>88.11±.03</td>
    <td>88.52±.12</td>
    <td>91.40±.28</td>
    <td>85.69±.35</td>
    <td>86.05±.20</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Fisher-Info</td>
    <td>74.43±.15</td>
    <td>80.39±.61</td>
    <td>80.63±.64</td>
    <td>86.81±.03</td>
    <td>87.50±.91</td>
    <td><b>87.59±.38</b></td>
    <td>72.31±.45</td>
    <td>1</td>
  </tr>
</table>

Table 10: Comparing the salience metrics on OPT-125m with 0.35% trainable parameters on various GLUE tasks, following the same format as Table 1.

<table>
  <tr>
    <th>Method</th>
    <th>MRPC</th>
    <th>QNLI</th>
    <th>SST-2</th>
    <th>STS-B</th>
    <th>QQP</th>
    <th>Avg.</th>
    <th>#</th>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>83.33</td>
    <td><b>92.48</b></td>
    <td>95.99</td>
    <td>89.03</td>
    <td>89.97</td>
    <td>90.16</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Magnitude</td>
    <td>77.45</td>
    <td>90.43</td>
    <td>95.18</td>
    <td>80.33</td>
    <td><b>90.41</b></td>
    <td>86.76</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Gradient</td>
    <td><b>87.25</b></td>
    <td>92.11</td>
    <td>95.53</td>
    <td><b>90.30</b></td>
    <td>89.02</td>
    <td><b>90.84</b></td>
    <td>2</td>
  </tr>
  <tr>
    <td>SynFlow</td>
    <td>78.68</td>
    <td>90.85</td>
    <td><b>96.10</b></td>
    <td>81.66</td>
    <td>88.56</td>
    <td>87.17</td>
    <td>1</td>
  </tr>
  <tr>
    <td>SNIP</td>
    <td>83.82</td>
    <td><b>92.48</b></td>
    <td>75.23</td>
    <td>89.44</td>
    <td>85.93</td>
    <td>85.38</td>
    <td>1</td>
  </tr>
  <tr>
    <td>FORCE</td>
    <td>83.58</td>
    <td>92.39</td>
    <td>89.56</td>
    <td>88.83</td>
    <td>88.31</td>
    <td>88.53</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Taylor-FO</td>
    <td>83.82</td>
    <td><b>92.48</b></td>
    <td>75.23</td>
    <td>89.44</td>
    <td>85.93</td>
    <td>85.38</td>
    <td>1</td>
  </tr>
  <tr>
    <td>GRaSP</td>
    <td>84.80</td>
    <td>92.46</td>
    <td>87.96</td>
    <td>89.54</td>
    <td>88.09</td>
    <td>88.57</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Fisher-Info</td>
    <td>81.37</td>
    <td>90.74</td>
    <td>83.26</td>
    <td>84.86</td>
    <td>85.27</td>
    <td>85.10</td>
    <td>0</td>
  </tr>
</table>

Table 11: Comparing the salience metrics on OPT-1.3b with 0.18% trainable parameters on a subset of the GLUE benchmark, following the same format as Table 1.

<table>
  <tr>
    <th>Method</th>
    <th>HumanEval</th>
    <th>MBPP</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>40.85</td>
    <td>48.8</td>
    <td>44.83</td>
  </tr>
  <tr>
    <td>PiSSA</td>
    <td>38.41</td>
    <td>48.0</td>
    <td>43.21</td>
  </tr>
  <tr>
    <td>Gradient</td>
    <td>48.78</td>
    <td>50.0</td>
    <td>49.39</td>
  </tr>
  <tr>
    <td>SynFlow</td>
    <td>40.85</td>
    <td>49.0</td>
    <td>44.93</td>
  </tr>
  <tr>
    <td>Magnitude</td>
    <td>39.02</td>
    <td>49.2</td>
    <td>44.11</td>
  </tr>
  <tr>
    <td>SNIP</td>
    <td>46.95</td>
    <td>51.0</td>
    <td>48.98</td>
  </tr>
  <tr>
    <td>FORCE</td>
    <td>46.95</td>
    <td>50.4</td>
    <td>49.19</td>
  </tr>
  <tr>
    <td>Taylor-FO</td>
    <td>49.39</td>
    <td>49.4</td>
    <td>49.40</td>
  </tr>
  <tr>
    <td>GRaSP</td>
    <td>46.34</td>
    <td>49.8</td>
    <td>48.07</td>
  </tr>
  <tr>
    <td>Fisher-Info</td>
    <td>46.34</td>
    <td>47.2</td>
    <td>46.77</td>
  </tr>
</table>

Table 12: Comparing the salience metrics on Llama3-8b with 0.67% trainable parameters. We fine-tuned the model on CodeFeedback and evaluated on HumanEval and MBPP. For reference, we provide LoRA and PiSSA as baselines with the same number of trainable parameters.

Reviewer paper summaries: This study summarizes popular sparsity-based PEFT (SPEFT) methods in a unified framework and investigates 8 mainstream salience metrics employed in those methods to select sparsity masks.

Analysis conducted in

* BERT models on a subset of tasks in GLUE
* Gemma2-2B, Qwen2-7B on MMLU
* Gemma2-2B on MataMathQA, GSM8K

shows several interesting conclusions, including

* Gradient-based SPEFT has a favorable performance in most tasks
* Fisher information does not improve too much as a comparison
* Static masking strategies are sufficiently strong, whereas dynamic masking strategies do not improve significantly compared to them
* No significant difference between local masking and global masking strategies

This paper explores different aspects of existing sparse PEFT (PEFT) methods' effectiveness derived from zero-shot Neural Architecture Search (NAS) proxies, mainly on the selection of salience metrics in language model (LM) finetuning. The findings suggest that among all salience metrics derived from NAS, gradient-based metrics are the most effective for constructing the sparse tuning mask. Experiments also demonstrate that SPEFT methods can outperform the LoRA baseline on various LMs and end-task selections. Furthermore, the result indicates the masking strategy (dynamic v.s. static, global v.s. local) poses little effectiveness to the downstream task performance of SPEFT.
Summary of strengths: The paper is organized in a way that is very easy for readers to follow. The experimental design covers different tasks and model choices, and draws clear conclusions from the results. Those conclusions shed light on the future direction for SPEFT research and provide insightful principles for the underlying hardware design. The paper itself also serves as a good survey of SPEFT.

Specifically, the effectiveness of static masking strategies looks surprising and novel to me, and it could drive the direction of SPEFT research from developing stronger and more complicated methods to finding more efficient and simple algorithms.

- This paper presents a set of solid and overwhelming experiments to suggest the effectiveness of different salience metrics rooted in NAS, thoroughly comparing different metrics on varied (small and large) model backbones and tasks.
- The findings in the paper suggest that using gradient-based salience metric is strongest in SPEFT, yet the selection of masking strategies poses little effectiveness in LM finetuning, which sheds light on the future research of PEFT in LMs.
Summary of weaknesses: The experimental design of large-scale settings can be improved. Specifically, in Table 3, the accuracy of GSM8k looks too low (<0.5) when compared to the reported performance of [Qwen2-7B](https://huggingface.co/Qwen/Qwen2-7B) (0.799). Given most popular models have achieved over 0.9 accuracy in this saturated benchmark, I am wondering what's the underlying reason for this phenomenon.

Also, I would suggest that since the conclusions drawn from the paper could significantly influence the future direction of SPEFT, it would be better to have even more extensive sets of experiments in the following aspects:

* More 8B-sized models, and even larger ones, such as 13B or 70B. Currently, only Qwen2-7B is investigated, whereas llama-3-8B and Gemma-2-9B are also encouraged to be included.

* Experiments in more aspects, such as out-of-distribution (OOD) performance of different SPEFT algorithms.

* Ablation studies in different levels of sparsity.

I would consider raising my score if the aforementioned concerns can be well addressed.

- In general, the paper lacks the major motivation of using SPEFT strategies to improve the PEFT methods in using LMs. The results shown in Table 1 and Table 2 indicate minimal improvement in using SPEFT over LoRA. Therefore, whether SPEFT is indeed better than LoRA in the overall PEFT paradigm remains a question.
- The hyperparameters that influence the number of training parameters in the experiments are predefined (e.g., LoRA r is fixed at 64 for Gemma and Qwen models). However, smaller r in LoRA might achieve similar performance and consume less memory footprint. Therefore, I would suggest authors experiment with LoRA and SPEFT with various trainable parameters with different models and tasks, and show that SPEFT could achieve better memory-performance tradeoff in general.
Comments, suggestions and typos: I am wondering what is the sensitivity to randomness for different SPEFT metrics, i.e. performance variance under different random seeds.

See weaknesses.
Overall assessment scores: 3.5

3.5
Metareview score: 4
Metareview: The paper has a comprehensive study of sparse parameter-efficient fine-tuning, It included several matrices such as weight salience metrics, and sparsity masking strategies to explore the optimum strategies. Experimental results have been shown on several datasets. The authors also provided some analysis of the results.
Summary of reasons to publish: 1. The paper provides a comprehensive comparison of SPEFT technical choices.

2. This paper presents a set of solid and overwhelming experiments to support their claims.