Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning (2024)

Abstract: Parameter-efficient fine-tuning (PEFT) has emerged as the predominant technique for finetuning in the era of large language models. However, existing PEFT methods still have inadequate training efficiency. Firstly, the utilization of large-scale foundation models during the training process is excessively redundant for certain fine-tuning tasks. Secondly, as the model size increases, the growth in trainable parameters of empirically added PEFT modules becomes non-negligible and redundant, leading to inefficiency. To achieve taskspecific efficient fine-tuning, we propose the Light-PEFT framework, which includes two methods: Masked Early Pruning of the Foundation Model and Multi-Granularity Early Pruning of PEFT. The Light-PEFT framework allows for the simultaneous estimation of redundant parameters in both the foundation model and PEFT modules during the early stage of training. These parameters can then be pruned for more efficient fine-tuning. We validate our approach on GLUE, SuperGLUE, QA tasks, and various models. With Light-PEFT, parameters of the foundation model can be pruned by up to over 40%, while still controlling trainable parameters to be only 25% of the original PEFT method. Compared to utilizing the PEFT method directly, Light-PEFT achieves training and inference speedup, reduces memory usage, and maintains comparable performance and the plug-and-play feature of PEFT1.

heading: 1 Introduction
text: Large-scale pre-trained language models have demonstrated outstanding performance in various natural language processing domains (Liu et al., 2019; Brown et al., 2020; Touvron et al., 2023; OpenAI, 2023). Along with the performance improvements, the scale of model parameters contin-
* Corresponding author: Peng Fu. 1Our code is available at https://github.com/gccnlp/
Light-PEFT.
ues to grow, making the cost of fine-tuning models increasingly expensive. Moreover, the practice of maintaining a separate copy of the large model for each task in conventional fine-tuning incurs substantial storage costs.
To address these challenges, parameter-efficient fine-tuning (PEFT) has been proposed: freezing most parameters of the foundation model and finetuning only a small number of parameters (Houlsby et al., 2019; Li and Liang, 2021; Liu et al., 2022a; Hu et al., 2022), thereby reducing the computational resource requirements during training and performing nearly full-parameter fine-tuning. In addition, this technique eliminates the need to save an entire model copy for each task. During inference, task-specific models can be obtained by switching directly to the appropriate parameter-efficient module for the given task.
However, the training efficiency of existing PEFT methods still needs improvement. The first problem lies in the excessive redundancy of using a large-scale foundation model during fine-tuning for specific tasks, which results in substantial computational costs. A typical strategy is to integrate PEFT with quantization (Dettmers et al., 2023; Kim et al., 2023). Nonetheless, these methods only quantize parameters to low-bit in memory, without reducing the number of parameters and they still need to be dequantized to high-bit during training, leading to wasted training time. Another more direct approach for reducing parameters is model structured pruning (Hedegaard et al., 2022; Zhao et al., 2023). However, most methods mainly focus on the inference efficiency of the model, which means they may result in higher training costs.
The second problem is that as the size of the foundation model increases, the number of parameters in added trainable modules also increases significantly. This introduces a lot of redundancy in trainable parameters, leading to inefficiency in finetuning. For instance, the commonly used methods
7528
LoRA (Hu et al., 2022) and QLoRA (Dettmers et al., 2023) empirically insert the low-rank modules onto fixed weight. However, there is no need to uniformly add trainable modules of the same rank to all weights for fine-tuning each task. An improved approach is the dynamic rank method (Zhang et al., 2023; Valipour et al., 2023; Ding et al., 2023), which adaptively allocates module parameters by progressively calculating the importance of the rank during training. However, these methods require continuous estimation during training and show limited improvement in actual training efficiency.
In this paper, we introduce a novel framework named Light-PEFT, which aims to enhance the efficiency of the PEFT technique during fine-tuning. The framework consists of two methods: Masked Early Pruning of Foundation Model and MultiGranularity Early Pruning of PEFT. In the early training stage, Light-PEFT estimates redundant parameters in both the foundation model (heads and intermediate dimensions) and the PEFT modules (module importance and rank importance) simultaneously. Structured pruning is used to eliminate this redundancy, resulting in a lighter foundation model and PEFT module for more efficient finetuning.
To validate the effectiveness of our Light-PEFT framework, we conduct extensive evaluations on various foundation models (RoBERTa, OPT1.3B, OPT-6.7B), different PEFT structures (LoRA, Adapter), and on diverse benchmarks (GLUE, SuperGLUE, and question-answering tasks). The empirical results indicate that the proposed LightPEFT framework outperforms other baseline methods in performance. It significantly improves training efficiency that reduces training memory usage by 39% and accelerates training to 1.6×. Additionally, the Light-PEFT framework improves inference efficiency that reduces inference memory by 48% and increases inference speed to 1.6×.

heading: 2 Related Works

heading: 2.1 Parameter-Efficient Fine-Tuning
text: Parameter-Efficient Fine-Tuning has been proposed to reduce the computational cost of fine-tuning entire model parameters (Houlsby et al., 2019; Li and Liang, 2021; Hu et al., 2022). Following works aim to further improve the efficiency of PEFT. Improvements to the PEFT module. The motivation behind of this category of methods is that
previous works often insert trainable modules empirically, resulting in uniform ranks for all inserted modules that are not parameter-efficient. AdaLoRA (Zhang et al., 2023) proposes obtaining the optimal rank for each module by iteratively pruning ranks during training. DyLoRA (Valipour et al., 2023) achieves this through dynamic training on a range of ranks. AutoPEFT (Zhou et al., 2023) automatically selects PEFT configurations through Bayesian optimization. Recently, SoRA (Ding et al., 2023) introduces masks on the ranks and gradually makes each module sparse. However, all of these methods gradually calculate the rank allocation during training, which does not improve the actual training efficiency in fine-tuning. Our method estimates the rank allocation for each module in the early stage of training and utilizes the pruned parameterefficient modules to improve training efficiency during fine-tuning. Improvements to the training paradigm of PEFT. To enhance training efficiency, one idea is to further reduce the memory footprint during training. QLoRA (Dettmers et al., 2023) and PEQA (Kim et al., 2023) reduce memory usage by quantizing the foundation model, while LST (Sung et al., 2022) and MEFT (Liao et al., 2023), respectively alleviate the memory footprint of intermediate activations in the foundation model through methods ladder side-tuning and reversible structures. Our approach is orthogonal to these methods from a memory perspective and can be combined with them. We explore early-stage pruning of the foundation model to reduce memory usage. Moreover, our approach can lower computational costs, speed up training, and improve inference efficiency.
Combining PEFT with pruning, most of works focus on improving inference efficiency. PST (Li et al., 2022) and DSEE (Chen et al., 2023) propose combining unstructured pruning and PEFT, which hardly achieves acceleration on practical hardware. SPAs (Hedegaard et al., 2022) integrates structured pruning of the foundation model with PEFT, while CPET (Zhao et al., 2023) proposes distilling knowledge into PEFT modules simultaneously with pruning to reduce performance degradation. Concurrently to our works, APT (Zhao et al., 2024) reduces the training cost of the CPET method, presenting more efficient distillation and pruning. However, these methods, including APT, still require higher training time and memory costs compared to the original PEFT methods. Our approach aims to reduce the original PEFT training
costs, including speed and memory, by employing early-stage structured pruning to train a nonredundant PEFT model efficiently, while improving inference efficiency simultaneously.

heading: 2.2 Structured Pruning of Models
text: Model pruning has been proposed to compress redundant parameters in models (LeCun et al., 1989; Kurtic et al., 2022; Liu et al., 2022b; Ma et al., 2023), with structured pruning being the most straightforward method to achieve acceleration on actual hardware. For the structured pruning of Transformer models, the focus lies in pruning components of the model, such as attention heads and feed-forward dimensions (Liu et al., 2021; Xia et al., 2022; Tao et al., 2023; Xia et al., 2024). However, most structured pruning works require additional costs during training to obtain smaller and more accurate models for inference efficiency. In terms of training efficiency, You et al. (2020) base on the lottery ticket hypothesis (Frankle and Carbin, 2019) and discover the existence of early winning tickets in DNN models, allowing early pruning to enhance subsequent training efficiency. Subsequently, Chen et al. (2021) identify early tickets in BERT models (Devlin et al., 2019) to enhance the efficiency of BERT’s pre-training and fine-tuning. We follow these works and explore early pruning in parameter-efficient fine-tuning and generative foundation models.

heading: 3 Preliminaries

heading: 3.1 Parameter-Efficient Fine-Tuning
text: In our framework, we choose two of the most widely used methods: Adapter (Houlsby et al., 2019) and LoRA (Hu et al., 2022) to validate our approach. Adapter. For each layer in the foundation model, including the attention sub-layer and the feedforward sub-layer, Adapter inserts a trainable MLP module after each sub-layer. It consists of a downprojection layer Wdown ∈ Rd×r, followed by a non-linear activation function f , and finally an upprojection layer Wup ∈ Rr×d, where d is the hidden size of the foundation model, and r is the bottleneck dimension in the trainable module, with r ≪ d. The Adapter method can be formulated as follows:
h← h+ f(hWdown)Wup (1)
where h is the output of the inserted sub-layer.
LoRA. For each linear weight matrix W ∈ Rd×d in the foundation model, the LoRA method adds trainable MLP modules in parallel to W . The trainable module includes a down-projection layer Wdown and an up-projection layer Wup. The LoRA method can be be formulated as follows:
h← h+ s ·XWdownWup (2) where X represents the input to the linear weight matrix W and s is a hyper-parameter scaling factor.

heading: 3.2 PEFT Training Efficiency
text: In this section, we present observations on the training efficiency of PEFT. We utilize LoRA to observe the results on two foundation models, RoBERTa (Liu et al., 2019) and OPT (Zhang et al., 2022). For training samples, we set the length to 128 with a batch size of 32 and the time is the sum of 10 batches. All tests are conducted on a single NVIDIA RTX 3090 GPU. The impact of foundation models size. From the perspective of training speed (Figures 1a), PEFT methods reduce the gradient computation time, so the forward pass time gradually surpasses the backward pass time. Nonetheless, the forward calculation is still unchanged and needs to use all model parameters to propagate the state forward and backpropagate the loss through the entire model, becoming slower as the model size increases. From the memory perspective (Figure 1b), although PEFT techniques reduce the memory consumption of optimizer states and gradients, the model weights and intermediate activations still occupy a significant amount of memory during training. Compressing the foundation model to a smaller size can better alleviate it. This highlights the importance of reducing the parameter redundancy of the foundation model for training efficiency.
The impact of PEFT modules. We explore the impact of intra-module rank and the number of PEFT modules on training efficiency. From the perspective of training speed, Figure 2a presents experiments where we keep same modules and only increase the rank. Figure 2b shows experiments where we keep the same trainable parameter, adding structured PEFT modules to different weights. It can be observed that when increasing the number of PEFT modules compared to varying the rank, both forward and backward times significantly increased. This indicates that, during training, the impact on speed of adding more structured PEFT modules is significantly larger than that of increasing in rank of a single structured module. From a memory perspective, the trainable parameters affect the memory consumption of optimizer states and gradients during training. As the size of the foundation model increases, the redundancy introduced by empirically adding trainable parameter modules impacts training efficiency.

heading: 4 Methodology

heading: 4.1 Overview of Light-PEFT
text: Our goal is to eliminate parameters redundancies in the early stage, thereby reducing the computational costs of fine-tuning. Thus, we propose the LightPEFT framework as shown in Figure 3, which consists of two methods: Masked Early Pruning of Foundation Model to reduce the redundancy of the foundation model and Multi-Granularity Early
Pruning of PEFT to reduce the redundancy of the trainable parameters. First, both methods in our framework simultaneously estimate redundancies during the early stage of training, where the total training steps are denoted as t, and the estimation for early pruning steps denoted as t′, t′ ≪ t. After estimation, we prune redundancies in both, thus obtain a non-redundant foundation model and PEFT modules for more efficient fine-tuning. Besides the PEFT parameters, we only need to additionally save mask vectors, which are much smaller than PEFT modules, to record the pruning index of the foundation model. During inference, our method allows the masks and PEFT modules to be easily changed, maintaining the plug-and-play feature.

heading: 4.2 Masked Early Pruning of Foundation Model
text: A typical Transformer model (Vaswani et al., 2017) consists of L layers, each with a multi-head attention (MHA) sub-layer and a feed-forward network (FFN) sub-layer. A MHA sub-layer contains NH attention heads and weight matrices W
(i) Q ,W (i) K ,W (i) V ∈ Rd×dH , WO ∈ Rd×d are used for query, key, value and output, where d is the hidden size and dH = d/NH is the hidden size of a head. In parameter-efficient fine-tuning, the weights of the foundation model are frozen, and we add the PEFT module’s ∆W to these matrices. Taking the LoRA module as an example, for an input X the output of the MHA is calculated as follows:
head(i) = (W (i) Q +∆W (i) Q ,
W (i) K +∆W (i) K ,W (i) V +∆W (i) V , X)
(3)
MHA(X) = Concat(head(1), ...,
head(NH))(WO +∆WO) (4)
To identify redundancy in attention heads, we introduce a trainable scalar mask mA in each layer’s MHA sub-layer. Now the MHA becomes:
head(i) = m (i) A · (W (i) Q +∆W (i) Q ,
W (i) K +∆W (i) K ,W (i) V +∆W (i) V , X)
(5)
For a FFN sub-layer, which contains activation function Act(·) and weight matrices Wfc1 and Wfc2 which denote up-projection and downprojection. With PEFT modules, for an input X the output of the FFN is calculated as follows:
FFN(X) =Act(X(Wfc1 +∆Wfc1))
· (Wfc2 +∆Wfc2) (6)
We also introduce a trainable scalar mask mF in each layer’s FFN sub-layer to eliminate redundancy in intermediate dimension. Now the FFN become:
FFN(X) =Act(X(Wfc1 +∆Wfc1)) ·mF · (Wfc2 +∆Wfc2) (7)
Inspired by Liu et al. (2017), we then use L1 regularization to learn masks mA and mF . During the mask learning, the PEFT module and the mask are trained jointly using gradient descent, which allows the mask to better present the impact of PEFT to the foundation model training on the target task. The loss function is as follows:
Lmask = L+ λA∥mA∥1 + λF ∥mF ∥1 (8)
where L is the original loss in fine-tuning, λA and λF are hyper-parameters to control the penalty of regularization (see Appendix A.4 for details). The masks are initialized to 1 at the beginning of training.
After estimating, we perform structured pruning on attention heads with pruning ratio ρA layerwise and intermediate dimensions with ρF globally based on the magnitudes of mA and mF .

heading: 4.3 Multi-Granularity Early Pruning of PEFT
text: In comparison to the fine-grained sparsity (i.e. rank allocation) that is the focus of most previous works
(Zhang et al., 2023; Valipour et al., 2023), our preliminary observation also confirms the significance of coarse-grained module pruning for training speed. Therefore, we propose multi-granularity PEFT pruning to consider both aspects simultaneously. Furthermore, we perform pruning PEFT in the early stage to maximize efficiency during training.

heading: 4.3.1 Modules Pruning
text: To achieve coarse-grained module pruning, we begin with the original design of PEFT, where we believe that the importance of a module is primarily determined by the change it brings to the original information. Specifically, for the LoRA method, we add a trainable module WdownWup on the weight W . Thus, given an input X , the importance ratio IM is defined as:
IM = ∥X ·WdownWup∥2 ∥X ·W∥2
(9)
where ∥·∥2 represents the L2 norm, measuring the magnitude of the vector output from the PEFT module. Because one of the weight matrices in the PEFT module, such as Wup in the LoRA method, is typically initialized to zero. Therefore, during training, the ratio of the output magnitude of the LoRA module to the weight W ’s output magnitude
indicates the importance of the changes required by the module added at that position.
For the Adapter method, a trainable module is added after a sub-layer. Given the output h of the previous sub-layer, the importance ratio IM is defined as:
IM = ∥f(hWdown)Wup∥2
∥h∥2 (10)
where IM represents the change in information of the Adapter module on the output information h of the previous sub-layer.
In the implementation, to better estimate the importance of all added positions for the LoRA method, we add LoRA modules on all weights of the foundation model. This may result in higher costs compared to the original LoRA in the short term, but our early estimation steps are significantly smaller than the total training steps, allowing for a substantial reduction in total costs. For the Adapter method, we follow the original approach by adding them after both the MHA and FFN sub-layers. After estimation, we use IM to globally prune the entire PEFT modules with the pruning rate ρM .

heading: 4.3.2 Ranks Pruning
text: In addition to coarse-grained pruning, we further perform fine-grained pruning on the rank of the modules. This allows us to reduce more trainable parameters and enhance training efficiency. Our motivation is based on the fact that not all modules require the same rank allocation. To eliminate redundant ranks, we use the first-order Taylor expansion (Molchanov et al., 2017) to estimate the importance IWi,j of each parameter connected to the rank in the PEFT module:
IWi,j = ∣∣∣∣ ∂L
∂Wi,j Wi,j
∣∣∣∣ (11)
where Wi,j represents the i-th row and j-th column of parameters in Wdown or Wup of the PEFT module. The importance of the rank IR is the sum of the importance IWi,j of all parameters corresponding to the rank in the column of Wdown and the row of Wup. After estimation, we globally prune the unimportant ranks with pruning rate ρR.

heading: 5 Experiments

heading: 5.1 Experimental Setup
text: Datasets and evaluation. We conduct experiments on eight natural language understanding (NLU)
tasks from GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a) and six questionanswering (QA) tasks. Because our goal is to enhance training efficiency, training on small datasets does not hold much significance. As a result, we choose four larger datasets from GLUE including MNLI (Williams et al., 2018), QNLI (Rajpurkar et al., 2016), QQP2, and SST-2 (Socher et al., 2013), and four larger datasets from SuperGLUE comprising ReCord (Zhang et al., 2018), WiC (Pilehvar and Camacho-Collados, 2019), BoolQ (Clark et al., 2019), and MultiRC (Khashabi et al., 2018). For MNLI, we report accuracy on the matched validation set. For QNLI, QQP, SST-2, WiC and BoolQ we report accuracy. For ReCord we report F1 and for MultiRC we report F1 over all answeroptions (F1a). The QA tasks include OpenBookQA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), ARC-Easy and ARC-Challenge (Clark et al., 2018), SciQ (Welbl et al., 2017) and WebQuestions (Berant et al., 2013). We report accuracy on all QA tasks by lm-evaluation-harness (Gao et al., 2023). Baselines. We use RoBERTa-Large for NLU tasks, OPT-1.3B and OPT-6.7B for QA tasks as foundation models. We choose several baselines to verify the effectiveness of our method. Full-FT is the conventional approach for fine-tuning. Adapter (Houlsby et al., 2019) and LoRA (Hu et al., 2022) are original structures we used in our framework. LayerDrop (Fan et al., 2020) is a strong baseline method that enhances training efficiency by dynamically dropout layers during training. We reimplement it combining with LoRA method. LST (Sung et al., 2022) improves model training efficiency by avoiding backpropagation in the foundation model. Offsite-Tuning (Xiao et al., 2023) uses a emulator derived from the foundation model for efficient training, and replaces the emulator’s layers back into the foundation model for inference. LLM-Pruner (Ma et al., 2023) prunes model on small amount of task-agnostic corpora and restores performance using LoRA, thereby improving training efficiency. We re-implement their original taskagnostic pruning and add a task-specific pruning implementation using 1k random samples from task data. Implementation. For the GLUE benchmark, we control the estimation steps for early pruning to be around 5% of the total training steps. For the more
2https://data.quora.com/First-Quora-Dataset-ReleaseQuestion-Pairs
challenging SuperGLUE benchmark, we set the estimation steps to be within 10%. For QA tasks, we uniformly use 10% of the training steps. Please refer to the Appendix A.1 for detailed pruning settings, as well as other training details.

heading: 5.2 Experimental Results

heading: 5.2.1 Experiments on NLU Tasks
text: We first evaluate our method on the GLUE benchmark. As shown in Table 1, we achieve comparable performance with the original method while using 72% of the foundation model parameters (pruning 5/16 of the heads and 1/3 of the FFN intermediate dimensions) and 0.3M trainable parameters by pruning PEFT modules and ranks. This results in a 1.4× training speedup and improvements in memory usage due to pruning. Furthermore, our method outperforms the baseline methods with the same speed, having fewer trainable parameters. When increasing the pruning rate and retaining 67% of the parameters in the foundation model, Light-PEFT achieves a 1.6× training speedup while still ensuring slightly better performance than the baselines. On the more challenging SuperGLUE benchmark, as shown in Table 2, we prune 4/16 of the heads and 30% of the FFN intermediate dimensions, retaining 76% of the parameters in the foundation model and 0.3M trainable parameters. This achieves performance comparable to the original PEFT method , demonstrating the effectiveness of our method Masked Early Pruning of Foundation Model.

heading: 5.2.2 Experiments on QA Tasks
text: For the QA tasks (Table 3), we first conduct experiments on OPT-1.3B. We prune parameters (12/32
heads and 2/5 intermediate dimensions), retaining 64% of the foundation model parameters and 1.5M trainable parameters, achieving comparable performance to the original method. When the trainable parameter in the original LoRA method is set to 1.57M (r=8), our method outperforms the original LoRA under fewer foundation model parameters, which demonstrates the effectiveness of our method Multi-Granularity Early Pruning of PEFT.
Compared to Offsite-Tuning, our method achieves better performance without the high training costs of the distillation. Compared to LLMPruner, our method outperforms both task-agnostic and specific implementations, and our pruning process does not require the large model’s gradients, leading to significantly reduced computational costs. Even when pruning it to 54%, we maintain better performance than the baselines.
On the larger OPT-6.7B model, pruning more foundation model parameters than OPT-1.3B and using 5.2M trainable parameters, we achieve performance comparable to the original method. When reducing trainable parameters to 2M, our method still demonstrates good performance. These ex-
perimental results demonstrate that in QA tasks, we can use the Light-PEFT framework to remove more redundant parameters from the foundation model and trainable modules, improving training efficiency while ensuring performance.

heading: 5.3 Analysis

heading: 5.3.1 Ablation Study
text: In the Section 5.2, we have demonstrated the performance of foundation model pruning (more experiments in Appendix A.2). Here, we conduct ablation study to examine two PEFT pruning strategies, module pruning and rank pruning (Table 4). Compared to not using any PEFT pruning, using module pruning or rank pruning generally improves generalization and thus enhances performance in most cases, indicating the effectiveness of the two proposed pruning strategies. Moreover, by combin-
ing the two pruning strategies, the model maintains a comparable level of performance despite having more pruned trainable parameters.

heading: 5.3.2 Training and Inference Efficiency
text: We validate the training and inference efficiency of our method on NVIDIA RTX 3090. In terms of training efficiency (Figure 4), we conduct experiments on RoBERTa-Large, retaining 67% of foundation model parameters and 0.3M trainable parameters that resulted in 32% reduction in model weight memory, 40% reduction in activations memory, and 39% reduction in peak memory. Calculating the total time for 10 batches, we achieve 2.2× speedup in forward and backward pass time compared to the original LoRA.
In terms of inference efficiency (Table 5), we conduct experiments on OPT-6.7B, representing widely used generative LLMs. Compared to the common practice of adding LoRA modules onto all matrices in the fine-tuning of LLMs (Vanilla), our proposed foundation model pruning and PEFT module pruning can effectively increase inference speed by up to 1.6×. Additionally, foundation model pruning can effectively reduce the model loading memory usage by up to 48%.

heading: 6 Conclusion
text: This paper introduces Light-PEFT, a novel framework designed to improve the efficiency of the PEFT technique during fine-tuning. The framework comprises two methods: Masked Early Pruning of Foundation Model and Multi-Granularity Early Pruning of PEFT. The Light-PEFT framework estimates redundant parameters in both the foundation model and PEFT modules during the early stage of training and prunes them to achieve more efficient fine-tuning. We validate our approach on GLUE, SuperGLUE, and QA tasks using various models. The experiments demonstrate that Light-PEFT achieves training and inference speedup, reduces memory usage, and maintains comparable performance.
Limitations
Although Light-PEFT has achieved improved training and inference efficiency along with good performance, our work primarily focuses on the singletask fine-tuning scenario. A future direction worth exploring is the estimation and early pruning of redundant parameters on the multi-task learning scenario, enabling efficient fine-tuning across multiple tasks.
Ethics Statement
The goal of our Light-PEFT framework is to enhance training efficiency and reduce computational resource costs, which has positive impacts.

heading: Acknowledgements
text: The authors thank Yuanxin Liu from Peking University and Jiaxuan Zhao from Institute of Information Engineering for the help and the anonymous reviewers for their valuable feedback on our paper.

heading: A Appendix
text: A.1 Details of Experimental Setup Hardware. We use NVIDIA TITAN RTX and NVIDIA RTX 3090 for NLU experiments and experiments using OPT-1.3B in QA Tasks. Additionally, we use NVIDIA A800 for experiments using OPT-6.7B in QA Tasks. Implementation. The implementation of LightPEFT is based on Transformers (Wolf et al., 2020), LLM-Adapters (Hu et al., 2023), and EarlyBERT (Chen et al., 2021). The data processing for SuperGLUE and QA tasks follows Liu et al. (2022a) and Xiao et al. (2023), respectively. Hyper-parameters. We use AdamW as the optimizer for training. Other detailed settings for NLU tasks are provided in Table 7, while the settings for QA tasks can be found in Table 8 and Table 9.
A.2 The impact of the pruning rate on the foundation model.
We analyze the impact of different foundation model pruning rates on performance on the WiC dataset (Figure 5). It is observed that within a certain range (above 62.5%), pruning results in a relatively minor decrease in performance. However, once this threshold is exceeded, a significant performance decline occurs, demonstrating that pruning within this range removes redundant parameters.
A.3 The impact of the estimation steps of early pruning
We analyze the impact of the early pruning estimation steps on performance using the BoolQ dataset (Figure 6). It is observed that once the estimation steps exceed 6.8% of the total training steps, further estimation does not lead to performance improvement. This demonstrates that our method can
effectively identify redundant parameters in both the foundation model and PEFT modules during the early stage of training.
A.4 The settings of mask learning penalty In practice, we keep λA and λF consistent and assess the impact of these hyper-parameters in pilot experiments (Table 6). Based on this result, we uniformly set λA and λF to 1× 10−4 and achieve good task performance in our main experiments.

heading: 1 ∗ 10−2 95.8 91.9 93.85

heading: Future Work
text: A future direction worth exploring is the estimation and early pruning of redundant parameters on the multi-task learning scenario, enabling efficient fine-tuning across multiple tasks