Memory-Efficient Fine-Tuning via Structured Neural Network Pruning

Authors: Cen-Jhih Li, Aditya Bhaskara
Primary area: foundation or frontier models, including LLMs
Keywords: Transformer, Fine-tuning, Memory efficient learning

Abstract:
Fine-tuning is an important step in adapting foundation models such as large language models to downstream tasks. To make this step more accessible to users with limited computational budgets, it is crucial to develop fine-tuning methods that are memory and computationally efficient. Sparse Fine-tuning (SFT) and Low-rank adaptation (LoRA) are two frameworks that have emerged for addressing this problem, and have been adopted widely in practice. In this work, we develop a new SFT framework, based on ideas from neural network pruning. At a high level, we first identify "important" neurons/nodes using feature importance metrics from network pruning (specifically, we use the structural pruning method), and then perform fine-tuning by restricting to weights involving these neurons. Using experiments on both vision and language tasks, we demonstrate that our method significantly improves the memory efficiency of SFT without increasing training time complexity and implementation complexity, while achieving accuracy comparable to state-of-the-art methods such as LoRA and its variants.

Decision: reject
Reviewer scores: 3, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730236706834

thread_type:
official_review

<thread_object>
creator: reviewer_3hdw

<content>
summary:
The paper proposes an improvement to the the efficiency of parameter-efficient finetuning (PEFT) by using a finetuning neuron-level mask, rather than a low-rank decomposition, to perform finetuning. This allows the authors to finetune only a small number of parameters, freezing the rest. The fact that the parameters are organized in matrix rows (corresponding to entire neurons) makes the process efficient, and more so than LoRA for the same number of parameters.

strengths:
The paper clearly lays out its method and the arguments for its method and the method itself is overall well-explained. The authors discuss their experimental choices and add an additional, helpful section on the practical considerations of memory requirements when using the Adam optimizer.

The idea of a class-aware Taylor importance is nice and is a useful side contribution of the paper.

weaknesses:
1. This paper unfortunately does not position itself appropriately with regard to the state of the field. In particular, RoSA [1] considers using a sparse finetuning mask combined with a low-rank adaptation, finding that sharing the parameter budget between the two approaches works better than either in isolation.

    Other relevant work is missing as well. For example, model compression methods are also used for model adaptation even without finetuning [2].  The related work section also fully omits any mention of sparse training methods, of which there are many; most relevant to this work might be the Movement Pruning method [3] and the FISH-Mask method [4].

    Overall, I would suggest removing the claim “ours is the first work that can integrate the literature on model compression for morel fine-tuning”. Further, the proposed method should be benchmarked against RoSA, or, better yet, a combination of this paper’s pruning approach with a low-rank component.

1. The claim that “the computational overhead required for computing Taylor importances is already prohibitively large!” (line 187) is not correct. The SparseGPT method, which the authors cite, relies on an approximate Taylor importance. So does the ZipLM method, which applies to structured pruning.

1. For the image classification task, assuming that the models were pretrained on ImageNet, the transfer tasks are very similar. It would have been more informative to also see cases where the downstream tasks are from a quite different distribution - for instance, some of the ones used in [5], such as Aircraft, flowers, etc.

1. The section on considering parameter dependencies in importance evaluation (line 262-271) is not clearly written, and the computation of the parameter budget was confusing. The whole idea of parameter dependencies doesn’t seem to add much, since the approach is actually unhelpful (per the experimental results). I am not sure why the authors make the claim “structured pruning considers parameter dependencies in its importance evaluation” (line 262), and there is no citation given.

1. Some experimental details are lacking. For the image classification tasks, the authors omit any mention of what dataset the models are pretrained on, if any. Additionally, the results do not have any sort of error bars, making it difficult to judge whether the improvements have any significance.

References:

[1] Nikdan, Mahdi et al. “RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation.” ArXiv abs/2401.04679 (2024)

[2] Mallya, Arun et al. “Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights.” European Conference on Computer Vision (2018).

[3] Sanh, Victor et al. “Movement Pruning: Adaptive Sparsity by Fine-Tuning.” ArXiv abs/2005.07683 (2020)

[4] Sung, Yi-Lin et al. “Training Neural Networks with Fixed Sparse Masks.” ArXiv abs/2111.09839 (2021)

[5] Kornblith, Simon et al. “Do Better ImageNet Models Transfer Better?” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

questions:
How does the proposed method compare with RoSA? Can even better results be achieved from combining this method with a low-rank component?

Please expand the related work section as outlined in the 'weaknesses', above

Please expand the performance evaluation in computer vision to more OOD tasks as in [5], above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730301640633

thread_type:
official_review

<thread_object>
creator: reviewer_ntjt

<content>
summary:
The paper proposes a Structured-Pruning-based Sparse Fine-Tuning (SPSFT) method that combines neural network compression and matrix decomposition techniques to significantly reduce memory and parameter requirements while achieving performance comparable to existing methods such as LoRA in both vision and language tasks. It is the first to apply network pruning to fine-tune large models, particularly by using structured pruning to identify and select important neurons for fine-tuning, achieving parameter and memory efficiency. The paper also provides guidance on hyperparameter and training configuration choices to help users achieve efficient and high-accuracy fine-tuning results across different tasks.

strengths:
(1): The SPSFT method uses structured pruning and matrix decomposition to greatly reduce memory and parameter requirements while maintaining performance, suitable for scenarios with limited computational resources.
(2): The approach is modular, allowing different pruning strategies to be used, making it adaptable to various task requirements.
(3): The method performs well across both vision and language tasks, demonstrating good generalizability.

weaknesses:
(1): Please clarify the term "QTaylor" mentioned in line 365. Ensure it is defined within the context of your study or provide relevant references. Consistent use and clear definition across the manuscript are crucial for reader comprehension.
(2): Could you elaborate on your choice of datasets for the classification tasks? Additionally, please discuss the potential applicability or comparability of your findings to larger, established datasets like ImageNet to enhance the generalizability of your results.
(3) Consider incorporating the standard 8 zero-shot datasets commonly used in fine-tuning tasks on Llama, if feasible. Additionally, including an AVG (average) metric could provide a holistic view of the model’s performance across tasks. If these datasets were not used, please provide a rationale for your selection.
(4): Please provide a detailed comparison of your model's parameter count and performance with LoRA, particularly at comparable scales. Discuss any trade-offs involved, which would help elucidate the practical implications of your approach.
(5):There are several LoRA variants that have reported improvements in parameter efficiency. Could you compare your approach against key variants such as [DoRA,  VeRA, SVFT, LoRA-XS]? This comparison would be valuable in highlighting the distinct advantages or potential areas for enhancement in your method.
(6): The figure presented in the main text could be enhanced for better clarity and information delivery. Consider adding detailed legends, clearer labels, or supplementary figures that demonstrate performance comparisons or efficiency gains. Visual aids are instrumental in conveying complex information effectively.

questions:
(1): Carefully check for spelling errors in the paper.
(2): Provide a fair comparison with LoRA at the same parameter scale, and report performance on 8 datasets along with the AVG metric.
(3): Consider comparing with some of the LoRA variants.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730346933814

thread_type:
official_review

<thread_object>
creator: reviewer_5q3j

<content>
summary:
The paper presented a parameter-efficient fine-tuning method inspired by techniques used in network pruning. The proposed method identifies and fine-tunes only the most important neurons selected using a pre-defined importance metric. The method is similar to low-rank adaptation (LoRA) in that it also formulates the weight updates as the product of two low-rank matrices. However, one of the low-rank matrices is fixed to binary values determined from the importance metric, while the other matrix is initialised as zeros and trained. Therefore, the method achieves high parameter efficiency, and the experiments also demonstrate competitive performance over vision and language models.

strengths:
1. The method is well-grounded in network pruning research, which has long sought to reduce model complexity while retaining performance by focusing on the more important weights of a model. By leveraging structured pruning and importance metrics, i.e., the Taylor score and L2 norm, the proposed method identifies the output channels that contribute most to the performance. The method is thus very well motivated.

2. PEFT methods such as LoRAs already are already parameter efficient. The proposed method further reduces the number of learnable parameters by fixing one of the low-rank matrices. This leads to even lower learnable parameter count and additional memory save.

3. The experiments are fairly comprehensive, covering both vision models, e.g., ViT, ResNet, and language models, e.g., DeBERTa, Llama. These experiments provide ample evidence on the model's efficiency and performance across a wide range of tasks.

weaknesses:
1. There is a lack of strong evidence on the advantage of the proposed method. The title of the paper focuses on the memory efficiency of the method. However, in most of the tables, the memory consumption of the proposed method is not included. There is only a small section (Sec. 5.4) that briefly mentions the memory footprint. This only provides one data point, and is not comprehensive enough to justify the title. More specifically, Table 6 shows a comparison against LoRAs using Llama models, without any conclusive evidence. The proposed method leads to lower parameter count, but also significantly lower performance in many cases. It is perhaps a more fair comparison to either scale both methods to the same parameter count and compare the performance, or the other way around. In addition to this, showing a curve of performance vs. parameter count/memory consumption of the two methods would largely benefit the argument.

2. A few more minor comments.
(a) Tables and figures are placed at the top of the page as a convention.
(b) Some notations are sloppy. In L161, there is no reason to enumerate the values of rows for matrix $M$. Simply use i as the index. In Eq.6, $F - c_i$ is a very crude notation. It's better to just use a new symbol.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730556783706

thread_type:
official_review

<thread_object>
creator: reviewer_h135

<content>
summary:
This paper discusses a method where they start by identifying "important" neurons or nodes using structural pruning metrics and then fine-tune by focusing only on the weights associated with these neurons. The proposed method performs well on the datasets selected by the authors. However, the paper appears to have been completed rashly, with significant issues in the coherence between sections, a lack of basic comparisons with other baseline methods, and missing experiments on more datasets. I believe the paper requires substantial revision to be considered for acceptance. Although I am inclined against accepting it, I am open to changing my view if persuaded.

strengths:
1. The article validates the proposed method with extensive experiments on both vision and language tasks, which helps in establishing the robustness and applicability of the technique across different domains.

2. It provides a comprehensive comparison with existing state-of-the-art methods like LoRA, demonstrating improvements in memory efficiency methods and maintaining comparable accuracy, which is crucial for practical applications.

weaknesses:
1. The layout of tables in the paper has issues. For instance, there is a large gap between Table 2 and Table 3. Additionally, the full names of "dft" and "hft" are not provided in Table 2, and placing your method in the middle of the table makes feels confusing.
2. Lack of experiments on ImageNet-10/CUB200/Caltech101. At least a dataset with high resolution. Since your experiments are mainly conducted on the CIFAR-100 or TinyImageNet, I have a concern that your method only work on the these datasets.
3. Lack of comparisons with existing baseline methods. e.g. LoRAPrune[1].
4. Limited novelty. In light of the extensive discussions around LoRA and its variants, the contribution of these hybrid methods seems insufficient. 

[1] Zhang, Mingyang, et al. "Loraprune: Pruning meets low-rank parameter-efficient fine-tuning." arXiv preprint arXiv:2305.18403 (2023).

questions:
Please convince me and make me believe that the weaknesses I identified in the paper do not actually exist.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732594370863

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
**Rebuttal change summary:**

1. We added paragraphs in section1 (line 60-68, 87-89) and section 2 (line 123-137) to more clarify the challenge of sparse fine-tuning. We also added some words at the research questions (line 69-72).
2. We described more on the memory benefit of our approach in detail (line 209-211, re-wrote section 5.4, and added appendix D). We also added memory footprints to all experimental results of Llama.
3. We included the discussions and corresponding descriptions for LoRA’s variants (line 148-152, 214-215, 281-284).  
4. We changed the setting of fine-tuning Llama using our approach to be similar settings compared with LoRA (line 315-316, Table 6, 9, 10).
5. We incorporated additional experiments (line 261-262, 309-310, 871-878, Table 2 and 4, and appendix D).
6. We rewrote, added texts, and added citations for some minor improvements (line 21-24, 76-77, 90-98, 269, 296-297, 322, 333-336, 459-470, 526-527, 800, and Equation 6).

Overall, in the revision, we included the memory footprint for Llama and additional experimental results. These results underscore the remarkable memory efficiency of our approach: even when the trainable parameters of our approach are 3 times that of LoRA, the memory usage of our method remains smaller than LoRA's.

We have also uploaded a **revision track** highlighting the revised text in red, as included in the **supplementary material**.


**Shared feedbacks for all reviewers:**

We thank all the reviewers for the effort and the valuable and constructive feedback. We “presented a parameter-efficient fine-tuning method inspired by techniques used in network pruning” (see reviewer 5q3j) and we “validate the proposed method with extensive experiments on both vision and language tasks” (see reviewer H135). Our approach “uses structured pruning and matrix decomposition to greatly reduce memory and parameter requirements while maintaining performance, suitable for scenarios with limited computational resources” (see reviewer nTJT). In the experimental results for Llama, we “provide a comprehensive comparison with existing state-of-the-art methods like LoRA, demonstrating improvements in memory efficiency methods and maintaining comparable accuracy, which is crucial for practical applications” (see reviewer H135). We also “add an additional, helpful section on the practical considerations of memory requirements when using the Adam optimizer” (see reviewer 3hdW). 

The main concerns of the reviewers were related to argument and experimental comparisons. In our revision, we re-structured and re-wrote the text and added some experimental results to improve the argument. The modification is summarized in the rebuttal change summary. Then, we address each point individually below.

We will be happy to engage in further discussion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732771310571

thread_type:
official_comment

<thread_object>
creator: area_chair_xtgc

<content>
title:
Reviewers please respond

comment:
Dear Reviewers,

If you have not responded to author's rebuttal, please kindly do so as soon as possible. The deadline is Dec 2, but the authors can potentially further clarify questions if you respond earlier. Thanks!

Best,\
AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Summary: a method for PEFT that only tunes weights on "important" weights, based on insights from neural pruning; experiments show improvement over existing sparse-fine-tuning methods.

(b) Strengths: experiments over various settings; higher efficiency in memory; practical method.

(c) Weaknesses: lack of comparisons with key LoRA variants; trailing performance with LoRA; more fine-grained downstream datasets result missing; missing details on memory efficiency

(d) reasons for decision
missing important experiments; efficiency not better than LoRA. Not enough support from reviewers.