LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning (2024)

Abstract: Large Language Models (LLMs), such as LLaMA and T5, have shown exceptional performance across various tasks through finetuning. Although low-rank adaption (LoRA) has emerged to cheaply fine-tune these LLMs on downstream tasks, their deployment is still hindered by the vast model scale and computational costs. Post-training model pruning offers a way to compress LLMs. However, the current pruning methods designed for LLMs are not compatible with LoRA. This is due to their utilization of unstructured pruning on LLMs, impeding the merging of LoRA weights, or their dependence on the gradients of pre-trained weights to guide pruning, which can impose significant memory overhead. To this end, we propose LoRAPrune, a new framework that delivers an accurate structured pruned model in a highly memory-efficient manner. Specifically, we first design a LoRA-guided pruning criterion, which uses the weights and gradients of LoRA, rather than the gradients of pre-trained weights for importance estimation. We subsequently integrate this criterion into an iterative pruning process, effectively removing redundant channels and heads. Extensive experimental results demonstrate the superior performance of our LoRAPrune over existing approaches on the LLaMA series models. At a 50% compression rate, LoRAPrune demonstrates superior performance over LLM-Pruner, achieving a reduction in perplexity by 4.81 on WikiText2 and 3.46 on PTB, while also decreasing memory usage by 52.6%. Besides, LoRAPrune also matches semi-structural pruning across multiple LLMs, proving its wide applicability. The code is available at https: //github.com/aim-uofa/LoRAPrune.

heading: 1 Introduction
text: Large Language Models (LLMs) (Touvron et al., 2023; Du et al., 2022; Frantar et al., 2022) have
∗X. Yu is the corresponding author.
showcased remarkable prowess, exhibiting outstanding performance across numerous tasks. To enable LLMs to perform specific tasks, such as chat-bots (Du et al., 2022; Zeng et al., 2022), they are often efficiently fine-tuned on downstream datasets (Taori et al., 2023; Chenghao Fan and Tian, 2023) by parameter-efficient fine-tuning (PEFT) methods (Jia et al., 2022; Hu et al., 2022; Chen et al., 2022), among which LoRA-based fine-tuning methods (Hu et al., 2022; Luo et al., 2023; He et al., 2023) have gained widespread use. However, the remarkable success of LLMs is accompanied by obstacles from their vast scale and substantial computational costs, making deployment exceedingly arduous (Frantar and Alistarh, 2023). Neural network pruning (Li et al., 2017; Molchanov et al., 2017), a prevailing technique for model compression, can significantly reduce the model size and complexity. Recently, the post-training pruning literature, such as SparseGPT (Frantar and Alistarh, 2023) and WANDA (Sun et al., 2023), have achieved high-performance unstructured sparse LLMs. However, unstructured sparse models face two critical issues: 1) Unstructured sparse models are hard to obtain direct inference speedup. They often require specialized hardware support to achieve satisfying acceleration benefits, which leads to unstructured pruning not benefiting legacy off-the-shelf platforms, e.g., CPUs, DSPs, and GPUs (Fang et al., 2023; You et al., 2023; Zhou et al., 2022). 2) Unstructured sparse models are not compatible with LoRA. As shown in Figure 1
3013
(a), since the weights BA produced by LoRA are dense, it poses challenges when trying to merge BA into the unstructured sparse weights. For instance, LoRA without merging increases inference time by nearly 54% (see Table 3), diminishing the benefits of pruning. One potential solution is to perform fine-tuning using LoRA on downstream tasks first and then carry out post-training pruning. However, separating tuning and pruning can lead to sub-optimal results (Molchanov et al., 2019; Sanh et al., 2020). To tackle this challenge, PST (Li et al., 2022) combines unstructured pruning with efficient fine-tuning, which simultaneously prunes LoRA and pre-trained weights. This method ensures a seamless merge of LoRA weights and avoids additional computational overhead that comes from LoRA. However, unstructured pruning of LoRA necessitates computing BA first and then doing Hadamard product with a binary mask M, which results in significant memory overhead (see Table 1) since BA and M share the same shape with pre-trained weights. For instance, when pruning LLaMA-65b, the intermediate variables necessitate the storage capacity of three NVIDIA A100 (80G) GPUs. This poses a significant memory challenge when adapting PST to LLMs. Instead, structured pruning can mitigate this issue since we can directly prune the structured weights of A in LoRA without storing BA. Therefore, it is significant to combine LoRA with structured pruning to achieve simultaneous PEFT and direct acceleration on general hardware platforms with high performance.
To this end, we propose a unified framework for LoRA and structured pruning, named LoRAPrune.
As shown in Figure 1 (c), LoRAPrune not only prunes the structured weights (e.g., heads, channels) from the pre-trained model weights W0 but also trims the corresponding weights in LoRA weight A without computing BA first. Consequently, after pruning and fine-tuning, the weights of LoRA can be seamlessly merged with the pretrained weights, ensuring that no additional computations are needed during inference. To identify weight connections of structural importance, the criterion used in the structured pruning methods (Ma et al., 2023; Molchanov et al., 2019, 2017) is often estimated by gradients or its variants, as shown in Figure 1 (b). However, LoRA typically requires frozen pre-trained weights without computing their gradients, thus pruning approaches that rely on gradients of the pre-trained weights cannot be directly applied. To efficiently estimate the importance of pre-trained weights, LoRAPrune introduces a novel criterion that exclusively utilizes the gradients of LoRA. In contrast to the vanilla gradient-guided pruning method, LoRAPrune leverages LoRA’s gradients as the approximation for the gradients of the pre-trained weights. Based on the presented criterion, we can iteratively perform pruning while simultaneously conducting efficient fine-tuning to restore the performance of the pruned LLMs, requiring only a small calibration set. Specifically, we compute the importance of every batch of data and update the importance using a moving average. Every few iterations, we remove a portion of unimportant structured weights until the desired sparsity is achieved. Through extensive experiments on diverse bench-
mark datasets and various scales of LLMs, we demonstrate that LoRAPrune consistently outperforms other structured pruning techniques tailored for LLMs. Furthermore, compared to the vanilla gradient-guided pruning, LoRAPrune significantly diminishes memory and computational overhead, facilitating efficient pruning and fine-tuning of LLaMA-65b on a single GPU concurrently. This paper has the following key contributions:
• We introduce a novel memory-efficient pruning criterion tailored for LLMs, termed the LoRAguided criterion, which seamlessly integrates with LoRA. Leveraging the gradients of LoRA, we can efficiently approximate the importance of pre-trained weights without needing to compute their gradients.
• As we can efficiently approximate gradients and update weights using LoRA, LoRAPrune facilitates iterative structured pruning, resulting in precise small models. Our framework ensures both high memory efficiency during pruning and incurs efficient inference.
• Pruning experiments conducted on the LLaMA models demonstrate that LoRAPrune can efficiently perform structured pruning with up to 65 billion weights on a single GPU. Furthermore, the pruning results achieved by LoRAPrune significantly surpass other pruning methods. For example, against LLM-Pruner, LoRAPrune uses only 52.6% of the memory yet scores lower perplexities by 4.81 on WikiText2 and 3.46 on PTB. LoRAPrune also matches semi-structural prun-
ing performance across various LLMs, proving its broad applicability.

heading: 2 Related Work
text: Parameter-efficient fine-tuning. PEFT methods (Jia et al., 2022; Wu et al., 2022; Chen et al., 2022; Hu et al., 2022; Luo et al., 2023; He et al., 2023) have received increasing attention from both academia and industry. Among them, LoRA (Hu et al., 2022) proposes injecting trainable low-rank decomposition matrices into each layer which can be merged into the pre-trained weights, avoiding extra computation in inference. Since inference efficiency, many methods based on LoRA have emerged. For instance, LongLoRA (Chen et al., 2023) improves upon LoRA, enabling efficient finetuning of LLMs on long contexts. AnimateDiff (Guo et al., 2023b) obtains a personalized generator by inserting LoRA into the frozen text-to-image model. Quantizing the pre-trained weights into 4- bit, QLoRA (Dettmers et al., 2023) employs LoRA for fine-tuning LLMs in downstream tasks while maintaining efficient memory usage. Therefore, LoRA is indispensable for fine-tuning LLMs. Our method seamlessly integrates LoRA and pruning, making it easily extensible to other PEFT methods based on LoRA. Neural network pruning. Removing unimportant weights from LLMs to reduce memory and the computational cost of deployment has become a common approach for model compression. Unstructured pruning (Dong et al., 2017; Lee et al., 2019; Wang et al., 2020; Sun et al., 2023; Frantar and Alistarh, 2023; Li et al., 2022) can ob-
tain highly compressed models by directly pruning neurons, which also causes unstructured sparsity and hard deployment. In contrast, structured pruning (Ma et al., 2023; Xia et al., 2023; Guo et al., 2023a) directly discards the whole grouped parameters (e.g.heads, channels) and leaves a model with deploy-friendly structures. However, structured pruning models require extensive finetuning to regain their performance levels. For example, Xia et al. (2023) utilized 50B tokens sampled for continued pretraining of their pruned model, a process that proves to be prohibitively expensive in terms of hardware resources. In contrast, our approach leverages structured pruning, enabling direct inference acceleration while maintaining training expenses at an acceptable level.
Pruning criterion. Determining the importance of weights in a network is still an open question (Blalock et al., 2020). A common approach to model pruning is to use parameter magnitude (Li et al., 2018; Lee et al., 2020; Elesedy et al., 2020; Han et al., 2015; Li et al., 2017) as a criterion. However, the small weights can still have a significant impact on the model output due to the complex structure of neural networks, while large weights may not be as important. Many methods (Sanh et al., 2020; Yu et al., 2022a; Zhang et al., 2022; Lee et al., 2019; Yu et al., 2022b; Wang et al., 2020; LeCun et al., 1989; Hassibi et al., 1993) employ Taylor expansion to approximate the errors introduced by pruning and use this as the criterion for importance estimation. To avoid computing the Hessian matrix (Hassibi et al., 1993) or Hessian inverse (LeCun et al., 1989) in Taylor expansion, (Molchanov et al., 2017, 2019) only use the first-order term in Taylor expansion. Furthermore, LLM-Pruner (Ma et al., 2023) similarly utilizes the first-order expansion for pruning and extends the pruning technique to LLMs. However, the firstorder term in Taylor expansion still requires gradients of the pre-trained weights. As shown in Table 1, computing and storing the gradients of pretrained weights significantly increases the pruning cost. To avoid computing gradients of pre-trained weights, PST (Li et al., 2022) learns the gradients of pre-trained weights by an extra low-rank matrix, which is motivated by LoRA. Nevertheless, PST conducts unstructured pruning and needs to compute a substantial mask with the equivalent shape of pre-trained weights in each forward pass, which is memory-intensive and hard to be adapted to LLMs.
Different from LLM-Pruner (Ma et al., 2023) and PST (Li et al., 2022), our criterion only relies on LoRA’s gradients and does not require expensive mask computation, making it memory-efficient.

heading: 3 Method

heading: 3.1 Preliminary
text: We define the notation used in this paper. Bold letters represent matrices and vectors. Lower-case letters indicate scalars. “Subscripts” identify the index of elements within a matrix, and “superscripts” indicate the layer index in a network. Low-rank adaptation. To efficiently fine-tune LLMs, low-rank adapter LoRA (Hu et al., 2022) constrains the update of model parameters to maintain a low intrinsic rank. During fine-tuning, the pre-trained weights remain frozen, abstaining from gradient computation, while the inserted LoRA is kept trainable. Given two low-rank matrices A ∈ Rr×k and B ∈ Rd×r (r ≪ min(d, k)), the update of a linear module can be written as
z = xW0 + xBA, (1)
where W0 ∈ Rd×k, z ∈ Rn×k and x ∈ Rn×d denote the pre-trained weights, outputs and inputs, respectively. After adaption, the new weights W can be re-parameterized as W = W0 +BA. Pruning with Taylor expansion. In vanilla pruning approaches (Molchanov et al., 2017, 2019), the importance of a weight Wi,j ∈W0 can be quantified by measuring the impact of its removal on the loss. For an input x and the ground-truth prediction y, the induced error of Wi,j can be given as:
Ii,j = [L(x,y,W0)−L(x,y,W0|Wi,j = 0)]2. (2) Computing Ii,j for each weight is computationally expensive. Following (Molchanov et al., 2019), we can use first-order Taylor expansion to approximate the importance Îi,j by:
Îi,j = ( ∂L
∂Wi,j Wi,j)
2. (3)
Dependency-aware structured pruning. In structured pruning, it is crucial to consider that pruned neurons can exhibit dependencies with other neurons due to their interconnected nature. The dependencies of weights are illustrated in Figure 5. We organize the connected weights as a group and estimate the group importance by accumulating the
Algorithm 1: Progressive pruning with LoRA-guided criterion Require :Calibration data D; Pre-trained
weights W0; Randomly initialized low-rank matrices A and B; Loss function L; Target sparsity level S; Fine-tuning iterations T .
Output :Trained low-rank adaption A and B; Binary mask M. Ḡ l g ← 0, Mlg ← 1 for ∀l,∀g; // Initialization for masks and group importance s← 0; // Initialize sparsity level for t ∈ [1, . . . , T ] do
Clear gradient; Forward and backward via Eq. (13); Update A and B via AdamW; Calculate Î|t via Eq. (10); Calculate Ĝ|t via Eq. (4); Calculate Ḡ|t via Eq. (11); for l ∈ [1, . . . , L] do
p← SortDescending(Ḡ)s; // Set threshold Mlg ← 0 where Ḡ l g ≤ p, and
g ∈ {1, . . . , G} end // Remove unimportant groups Progressively increase s until ||M||0 > S;
end
weight importance within the same group. Formally, the importance for the g-th group can be expressed as
Ĝg = ∑
Wi,j∈G Îi,j , (4)
where Ĝ ∈ R1×G represents the importance of groups, G denotes a set of weights within a group and G is the candidate group number in a layer.

heading: 3.2 Pruning with Low-rank Adaption
text: Motivation. To achieve highly-compressed LLMs, it is essential to accurately evaluate the importance of pre-trained weights. A key approach is to utilize the criteria in Eq. (3) for this evaluation. However, obtaining the gradient of W0 in a LLM is difficult since it requires a lot of computing power and storage space. Fine-tuning LLMs with LoRA
is becoming prevalent (Taori et al., 2023; Chenghao Fan and Tian, 2023). During LoRA fine-tuning, only the gradients of LoRA’s weights are computed, yielding remarkable computation and memory efficiency. Therefore, can we rely solely on the weights and gradients of LoRA to accurately estimate the importance of pre-trained weights? LoRA-guided criterion. In this work, we discuss how to estimate the importance of W0 by inserting the learnable matrices A and B in the downstream task adaption. The core idea lies in setting the element (BA)ij = −Wij if the element Wij ∈W0 is removed. The importance of each parameter in Eq. (2) can be reformulated as follows
Ii,j = [L(x,y,W)−L(x,y,W|(BA)i,j = −Wi,j ]2. (5) Exploiting the first-order Taylor expansion with (BA)i,j = −Wi,j to approximate Eq. (5), the estimated importance Îi,j of parameter Wi,j can be represented by
Îi,j = [ ∂L
∂(BA)i,j ((BA)i,j +Wi,j)]
2. (6)
However, as shown in Eq. (1), the LoRA computation sequence involves first multiplying by B and then by A, which means that BA cannot be obtained during the forward and backward pass. Besides, preserving ∂L∂(BA)i,j still entails the same level of complexity as ∂L∂Wi,j since BA shares the same shape of W0. Here, we only save and use the gradients of two low-rank matrices A and B to approximate
∂L ∂(BA) . We can rely on the gradient update that
(BA)i,j |t = (BA)i,j |t−1 − η ∂L∂(BA)i,j to estimate the gradient, where (BA)i,j |t and (BA)i,j |t−1 represents the (BA)i,j in t-th and (t− 1)-th step, respectively. Apparently, η ∂L∂(BA)i,j is equal to the change of BA, which can be written as
η ∂L
∂(BA)i,j = [(BA)i,j |t−1 − (BA)i,j |t]. (7)
Here, (BA)i,j |t = Bi,:|tA:,j |t is generated by the multiplication of the i-th row of B|t with the j-th column of A|t. Using the above assumption, we can also estimate η ∂L∂A:,j = A:,j |t−1 −A:,j |t and η ∂L∂Bi,: = Bi,:|t−1 − Bi,:|t, respectively. Further, we can obtain that A:,j |t = A:,j |t−1 − η ∂L∂A:,j and
Bi,:|t = Bi,:|t−1 − η ∂L∂Bi,: . Subsequently, we can calculate
(BA)i,j |t = Bi,:|tA:,j |t
= (Bi,:|t−1 − η ∂L
∂Bi,: )(A:,j |t−1 − η
∂L
∂A:,j ) (8)
Substitute (BA)i,j |t in Eq. (8) to Eq. (7) and obtain
∂L
∂(BA)i,j =[
∂L
∂Bi,: A:,j |t−1 +Bi,:|t−1
∂L
∂A:,j
− η ∂L ∂Bi,: ∂L ∂A:,j ].
(9)
For simplicity, we set the learning rate η = 1. Substitute Eq. (9) to Eq. (6), we can estimate the importance in a gradient-based manner Îi,j = [(
∂L ∂Bi,: A:,j+Bi,: ∂L ∂A:,j − ∂L∂Bi,: ∂L ∂A:,j )(Wi,j+
(BA)i,j)] 2.
Îi,j =[( ∂L
∂Bi,: A:,j +Bi,:
∂L ∂A:,j − ∂L ∂Bi,: ∂L ∂A:,j )
(Wi,j + (BA)i,j)] 2.
(10)
As shown in Figure 2, the LoRA-guided criterion only needs to compute the gradients of A and B
with the approximation in Eq. (10), which saves memory and computation compared with the gradients of pre-trained weights W0. Progressive pruning. To efficiently obtain group importance for structured pruning, we can substitute Eq. (10) into Eq. (4). However, estimating importance and pruning weights with a single batch of data can lead to significant bias and performance loss. To mitigate this, we apply moving average to evaluate group importance G and incrementally prune less critical groups. Specifically, the group importance at t-th iteration is computed as follows:
Ḡ|t = λḠ|t−1 + (1− λ)Ĝ|t. (11)
Here, Ḡ|t denotes the group importance scores calculated by Eq. (10) and Eq. (4) at the t-th iteration, and λ ∈ [0, 1] balances the importance between historical and current statistics. To this end, we can efficiently and accurately estimate the importance of each group. We then prune the unimportant groups by setting a binary mask M ∈ {0, 1}1×G for each pruned layer. The binary mask M is obtained by
Mg = { 1 Ḡg > p 0 Ḡg ≤ p , (12)
where the index g ∈ {1, . . . , G} denotes the g-th group in the layer, and p represents the threshold
of importance. Groups falling below this threshold will be pruned. After setting the mask, the forward process of each pruned layer can be written as
z = (xW0 + xBA)⊙M, (13)
where ⊙ denotes Hardamard product and can be calculated by broadcast. The complete algorithm of LoRAPrune is given in Algorithm 1.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Models and metrics. Our method is applied to the LLaMA-1 model family (Touvron et al., 2023), which comprises LLaMA-7B, LLaMA-13B, LLaMA-30B and LLaMA-65B. Following (Frantar and Alistarh, 2023), we evaluate models on the perplexity metric with WikiText (Merity et al., 2016) and PTB (Marcus et al., 1993) dataset. To assess the zero-shot ability of LLMs, we follow LLaMA to perform zero-shot task classification on common sense reasoning datasets: PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), ARC-easy (Clark et al., 2018), ARC-challenge (Clark et al., 2018), OpenbookQA (Mihaylov et al., 2018). We evaluate the in-context learning ability under a 5-shot setting on MMLU (Hendrycks et al., 2020).
Implementation details. We provide results for LoRAPrune as a single-shot method and with posttraining recovery fine-tuning. We iteratively prune models on LaMini instruction dataset (Wu et al., 2023) for LLaMA-7b and 20k sampled C4 dataset (Raffel et al., 2020) for LLaMA-13b, LLaMA-30B and LLaMA-65B. Our training configuration includes a batch size of 128, a learning rate set to 1e-4, and a total of 2 training epochs. As the pretrained weights remain frozen, there is the option to quantize them into 8-bit values to save memory. All models are optimized by AdamW optimizer (He et al., 2020) with a cosine learning rate decay. Contenders. We compare LoRAPrune with the following pruning methods in both fine-tuning and without fine-tuning settings: 1) Magnitude Pruning: pruning based on the absolute values of model weights. 2) LLM-Pruner (Ma et al., 2023): pruning using criterion in Eq. (3). 3) WANDA (Sun et al., 2023): pruning based on the magnitude of input features and pre-trained weights. 4) Compresso (Guo et al., 2023a): pruning based on a set of learnable masks.

heading: 4.2 Main Results
text: Zero-shot performance. Table 2 demonstrates the effectiveness of our proposed method. Our LoRAPrune far surpasses other large model prun-
ing methods under structured sparsity. For instance, at a 50% compression rate, LoRAPrune achieves a perplexity of 11.60 on WikiText2, significantly outperforming LLM-Pruner’s perplexity of 16.41. We also replicate the experimental results of WANDA under structured pruning scenarios. Our findings reveal that the pruning outcomes achieved by WANDA fell short in comparison to gradientbased pruning methods such as LLM-Pruner and LoRAPrune. This observation underscores the superior performance and effectiveness of gradientbased pruning approaches in our experiments.
It’s worth noting that LoRAPrune’s efficient approximation for the gradients of the pre-trained weights allows for 8-bit quantization of those weights, greatly reducing the memory requirements for pruning. Moreover, LoRAPrune demonstrates superior pruning results even when models are quantized to 8 bits. These findings underscore the effectiveness and versatility of LoRAPrune in achieving impressive pruning results across various scenarios and compression rates.
Few-shot performance. To verify whether the pruned LLMs retain the in context learning capability, we evaluate on the MMLU with 5-shot. As shown in Table 2, LoRAPrune consistently achieves a higher score than other pruning methods across all sparsity ratios. Notably, LoRAPrune achieves performance on par with the unpruned LLaMA-7B model at a 20% sparsity ratio.
Acceleration for pruned LLMs. Models with structured pruning can be directly sped up in general GPU devices. We conducted tests with 2048 tokens, averaging the results over 100 trials. We specifically examined the inference time with and without merging LoRA weights into the pre-trained weights. As shown in Table 3, we observed that when pruning 20% weights, LLM without merging LoRA has an even slower inference speed than LLM with LoRA merged without pruning. In addition, through structured pruning, the model achieves reductions in inference time of 24.7% and 49.5% at compression rates of 20% and 50%.
Pruning on large-scale LLMs. Due to the ef-
ficient approximation of the pre-trained weights’ gradients, LoRAPrune enables iterative pruning on larger-scale LLMs. To ensure that all experiments can be conducted on one GPU, we quantize the pre-trained weights of LLaMA-30b and LLaMA65b to 8 bits. The experimental results are shown in Figure 3. We observe that, in comparison to the magnitude-based method, LoRAPrune exhibits significant superiority across various scales. Furthermore, in comparison to the 2:4 sparsity model, LoRAPrune achieves comparable pruning results at a 50% sparsity rate. However, it is worth noting that the 2:4 sparsity model also faces challenges in direct weight merging with LoRA, resulting in additional computational overhead during inference. Besides, accelerating 2:4 sparsity models requires specialized hardware support, such as NVIDIA GPUs based on the Ampere architecture, which significantly increases the deployment constraints for 2:4 sparsity models.

heading: 4.3 Ablation Study
text: Efficiency of LoRA-guided criterion vs. vanilla criterion. We conduct a comparative analysis of different pruning criteria with respect to their resource requirements and computational efficiency, including GPU memory and throughput. We adopt the vanilla criterion, as outlined in Eq. (3), as our baseline. For each forward pass, we set the batch size to 1, and we accumulate this process iteratively until we reach a total of 128 accumulations. To ensure robustness and reliability, we compute and subsequently average the results obtained over a span of 100 steps. The comparison results can be found in Table 4. Compared to the vanilla criterion, LoRA-guided and LoRA-guided (8bit) criteria demonstrate a significant reduction in GPU memory usage, saving 52.6% and 64.2% of the memory, respectively. Moreover, as the LoRAguided criterion does not require the computation of original gradients, it achieves a 64.6% increase in throughput compared to the vanilla criterion with comparable performance, greatly enhancing the speed of the pruning process. Efficacy of LoRA-guided criterion vs. vanilla
criterion. Since the LoRA-guided criterion in Eq. (10) is an efficient approximation of the vanilla criterion in Eq. (3), we evaluate the effectiveness of the proposed LoRA-guided criterion by comparing mask similarity with the vanilla criterion. We randomly sample 128 data and then perform oneshot pruning with both LoRA gradient and vanilla criterion. Figure 4 illustrates that in the case of low compression rates (Ratio=10%), the masks generated by these two criteria exhibit a high degree of consistency. As the compression rates increase, the mask similarity may decrease. However, it is crucial to emphasize that LoRAPrune follows an iterative pruning approach. In each pruning iteration, it only needs to precisely identify the least important weights (about top-5%), thus ensuring the accurate approximation. Hence, the LoRA-guided criterion can attain results that are on par with those of the vanilla criterion while incurring reduced costs.

heading: 5 Conclusion
text: In this paper, we have proposed a method to effectively prune and fine-tune LLMs simultaneously, achieving state-of-the-art efficiency-accuracy tradeoffs. Specifically, we have proposed a novel LoRAguided criterion, for evaluating the parameter importance by only computing the LoRA gradients, which greatly reduces the computational resources required for pruning LLMs. Building upon the proposed criterion, we have presented LoRAPrune, a technique that performs efficient joint pruning and fine-tuning without the need for computing gradients of the pre-trained weights. Finally, comprehensive experiments on various LLMs and benchmarks have demonstrated the superiority of LoRAPrune over other pruning methods. In terms of comparison with the vanilla criterion, the LoRA-guided criterion shows its efficiency and effectiveness. In the future, we aim to further enhance the pruning results of LoRAPrune at higher compression rates. Limitation. LoRAPrune requires fine-tuning to restore model performance. This limitation can restrict the application of LoRAPrune in scenarios where fine-tuning is unavailable. Acknowledgements: This work was supported by National Key R&D Program of China (No. 2022- ZD0118700), National Natural Science Foundation of China (No. 62373329) and Baima Lake Laboratory Joint Funds of the Zhejiang Provincial Natural Science Foundation of China (No. LBMHD24F030002).

heading: A Weight Dependency for LLaMA
text: Here, we use LLaMA architecture as an example to explain the weight dependency. The dependency details are shown in Figure 5. In terms of the Attention module, when we decide to prune a specific head of weights in the Query layer, it is imperative that the corresponding weights with the same index in the Key, Value and Out layers are also pruned. Similarly, for the Feed-Forward Network (FFN) module, when pruning a particular channel of weights in the Up layer, it is essential to prune the weights with matching indices in the Gate and Down layers. This meticulous coordination ensures that pruning maintains the structural integrity and functionality of the model. Following (Ma et al., 2023) and (Fang et al., 2023), we prune heads for Attention and channels for FFN, respectively. The dependency details are shown in Figure 5.

heading: B More Ablation Studies
text: Pruning on 20k sampled C4 dataset. We also evaluate LoRAPrune on a tiny dataset that randomly samples 20k data from C4 dataset. As presented in Table 5, LoRAPrune outperforms both LLM-Pruner and WANDA across the majority of zero-shot reasoning datasets, thereby securing the highest average score overall. Specifically, LoRAPrune exceeds the performance of LLM-Pruner by margins of 0.82% and 1.02%, respectively.
Effectiveness of the moving average. We verify the rationale behind the moving average through the setting of different values for λ. These experiments were conducted on LLaMA-7b with 20k sampled C4 dataset. The experimental results, as shown in Figure 6 (a), reveal that as λ increases, the pruning results exhibit a significant reduction in perplexity. This effect is especially pronounced when λ = 0 where pruning is solely determined by the importance of the current batch, confirming the effectiveness of the moving average.
Impact of iterations. To assess the impact of the pruning iterations on pruning results, we conducted experiments on the LLaMA-7b model with different iterations on 20k sampled C4 dataset. The results are shown in Figure 6 (b), which indicates that excessive iterations can lead to a decrease in the model’s zero-shot performance, potentially due to overfitting on the calibration dataset. Furthermore, we observe that the model requires more
iterations to regain its performance when pruning with high compression (e.g., ratio=50%).
LoRAPrune vs. LLM-Pruner with gradients off-loading. The gradient off-loading strategy can partially mitigate LLM-Pruner’s memory demands, such as transferring certain gradients to CPU memory. However, the memory access cost and computational overhead are substantial. Table 6 shows LoRAPrune outperforms LLM-Pruner in efficiency, being 8.19× faster with CPU offloading and 2.75× faster without it. This speed allows iterative pruning to counteract the performance drop due to structured sparsity.
Joint vs. separate. To demonstrate the necessity of integrating pruning and fine-tuning, we conducted experiments that sequentially performed pruning followed by fine-tuning, specifically applying one-shot pruning to the LLaMA-7b model and then employing LoRA fine-tuning to recover the model’s performance. The experimental results presented in Table 7 indicate that joint pruning and fine-tuning yields much better performance than the separate counterpart, especially under the high compression ratio.
Pruning frequency. We explore the impact of different pruning frequencies, i.e., how many iterations of fine-tuning before pruning, on the final performance. The experimental results, as shown in Table 8, indicate that our default frequency (frequency=10) obtains the best pruning result. Additionally, we observe that if pruning is too frequent (frequency=1), the model may not have enough iterations to recover through fine-tuning, leading to inaccurate importance estimation. Furthermore, excessive fine-tuning between pruning iterations (frequency=20) leads to overfitting on the calibration data.

heading: C Generative Results
text: We show the generalization capability of the LoRAPrune by some instructions encompass common sense, translation, and coding tasks in Table 9.

heading: Future Work
text: In the future, we aim to further enhance the pruning results of LoRAPrune at higher compression rates. Limitation. LoRAPrune requires fine-tuning to restore model performance. This limitation can restrict the application of LoRAPrune in scenarios where fine-tuning is unavailable