Dynamic Low-Rank Sparse Adaptation for Large Language Models

Authors: Weizhong Huang, Yuxin Zhang, Xiawu Zheng, Liuyang, Jing Lin, Yiwu Yao, Rongrong Ji
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models; Network Sparsity; Low-Rank Adaptation

Abstract:
Despite the efficacy of network sparsity in alleviating the deployment strain of Large Language Models (LLMs), it endures significant performance degradation. Applying Low-Rank Adaptation (LoRA) to fine-tune the sparse LLMs offers an intuitive approach to counter this predicament, while it holds shortcomings include: 1) The inability to integrate LoRA weights into sparse LLMs post-training, and 2) Insufficient performance recovery at high sparsity ratios. In this paper, we introduces dynamic $\textbf{Lo}$w-rank $\textbf{S}$parse $\textbf{A}$daptation $\textbf{(LoSA)}$, a novel method that seamlessly integrates low-rank adaptation into LLM sparsity within a unified framework, thereby enhancing the performance of sparse LLMs without increasing the inference latency. In particular, LoSA dynamically sparsifies the LoRA outcomes based on the corresponding sparse weights during fine-tuning, thus guaranteeing that the LoRA module can be integrated into the sparse LLMs post-training. Besides, to achieve the optimal sparse model architecture, LoSA leverages Representation Mutual Information (RMI) as an indicator to determine the importance of layers, thereby dynamically determining the optimal layer-wise sparsity rates during fine-tuning. Predicated on this, LoSA adjusts the rank of the LoRA module based on the variability in layer-wise reconstruction errors, allocating an appropriate fine-tuning for each layer to reduce the output discrepancies between dense and sparse LLMs. Extensive experiments tell that LoSA can efficiently boost the efficacy of sparse LLMs within a few hours, without introducing any additional inferential burden. For example, LoSA reduced the perplexity of sparse LLaMA-2-7B by $\textbf{68.73}$$\downarrow$ and increased zero-shot accuracy by $\textbf{16.32}$%$\uparrow$, achieving a $\textbf{2.60$\times$}$ speedup on CPU and $\textbf{2.23$\times$}$ speedup on GPU, requiring only $\textbf{45 minutes}$ of fine-tuning on $\textbf{a single}$ NVIDIA A100 80GB GPU. Code is available at https://github.com/wzhuang-xmu/LoSA.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1730341018851

thread_type:
official_review

<thread_object>
creator: reviewer_vonn

<content>
summary:
This paper introduces dynamic Low-rank Sparse Adaptation (LoSA), a method that integrates low-rank adaptation into LLM sparsity within a unified framework, thereby enhancing the performance of sparse LLMs without increasing the inference latency. In particular, LoSA dynamically sparsifies the LoRA outcomes based on the corresponding sparse weights during fine-tuning, thus guaranteeing that the LoRA module can be integrated into the sparse LLMs post-training. Besides, LoSA leverages Representation Mutual Information (RMI) as
an indicator to determine the importance of layers, thereby efficiently determining the layer-wise sparsity rates during fine-tuning. Predicated on this, LoSA adjusts the rank of the LoRA module based on the variability in layer-wise reconstruction errors, allocating an appropriate fine-tuning for each layer to reduce the output discrepancies between dense and sparse LLMs.

strengths:
Extensive experiments tell that LoSA can efficiently boost the efficacy of sparse LLMs within a few hours, without introducing any additional inferential burden. For example, LoSA reduced the perplexity of sparse LLaMA-2-7B by 68.73 and increased zero-shot accuracy by
16.32%, achieving a 2.60× speedup on CPU and 2.23× speedup on GPU.

weaknesses:
The novelty may be limited. This paper proposes three improvements including dynamic sparsity rates across layers with RMI, dynamic rank allocation, and progressive pruning. Theses ideas have been known to be effective for pruning. For example, [A1] atomically determines the   layer-wise sparsity ratio and [A2,A3] uses MI (and HSIC) in model pruning.  [A4,A5] investigates rank allocation. Progressive pruning has been commonly used in pruning and proved to be effective such as [A6,A7]. It mentions to merge sparse lora with sparse weights for actual acceleration. This idea is straightforward and [A7] already investigates sparse lora with sparse weights so that they can be merged. The adoption of RMI generally follows [Zheng et al., An Information Theory-inspired Strategy for
Automatic Network Pruning, 2021]. It seems to change the CNN model to LLMs with almost the same RMI method in [Zheng et al, 2021].  The technical contribution may be limited. It is better to highlight the unique contributions. 


The baseline is not enough. The proposed method is a finetuning method. SparseGPT and wanda are PTQ methods without finetuning. It may not be fair to compare with  SparseGPT or wanda. The actual baseline only has lora. It may be better to compare with other finetuning based pruning methods for LLMs such as [A7], LLM-pruner or SliceGPT. Although most methods are structured pruning, it is easy to adapt the proposed method for structured pruning with such as wanda-sp of structured mask. It adopts multiple improvements and it is not surprising to be better than the naïve lora. It may be better to compare with related works focusing on pruning and finetuning LLMs. 

The setting with  2:8 or mixed 2:8 sparsity may not be solid. The GPU compiler can accelerate a specific N:M sparsity such as 2:4. It may not be able to accelerate 2:8 or mixed 2:8 sparsity. The experiments for this part only demonstrate the accuracy performance. But the actual acceleration may be limited without the support from compiler.  It may be better to discuss  this setting. 


The lora in the baseline finetunes the whole model so that the model is not sparse if merged. It may be a bit strange to compare between a dense model with lora and a sparse model with losa. It may be better to provide a new baseline such as SparseGPT + sparse lora or wanda + sparse lora. The sparse lora can be merged with the sparse model. It should be a more direct baseline with uniform sparsity rates and the same ranks in the merged model, in contrast to the dynamic sparsity and dynamic ranks. It may be more reasonable to compare the speed of this baseline and the proposed method as they are both sparse.  
 
Reference

[A1] Layer-adaptive sparsity for the Magnitude-based Pruning

[A2] Layer-wise Model Pruning based on Mutual Information

[A3] MPruner: Optimizing Neural Network Size with CKA-Based Mutual Information Pruning

[A4] AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

[A5] ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models

[A6] Fast and Effective Weight Update for Pruned Large Language Models

[A7] LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning

questions:
see the weakness.

It is better to highlight the unique contributions. 

 It may be better to compare with related works focusing on pruning and finetuning LLMs. 

 It may be better to discuss  this setting. 

The baseline can be improved.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730522359449

thread_type:
official_review

<thread_object>
creator: reviewer_7hsm

<content>
summary:
This paper presents an approach called "Dynamic Low-Rank Sparse Adaptation" (LoSA) that seeks to address the challenges of performance degradation associated with sparsifying LLMs. The paper introduces a unified framework that combines low-rank adaptation with sparsity, aiming to improve efficiency while maintaining model performance.

strengths:
1. The paper offers extensive experimental results across various models (e.g., LLaMA-2, Vicuna, OPT) and datasets.

2. The authors provide an explanation of their methodology, from dynamic sparsification to adaptive rank allocation.

weaknesses:
1. This paper ignores previous works [1-4] on sparse PEFT using Low-Rank Adaptation. I did not see any comparisons or discussions with previous similar works. It's extremely hard to tell if this work has enough contributions to the area since the author choose such narrow baselines (SparseGPT, Wanda).

2. Without through discussions with previous similar works, it's hard to judge the novelty of this work.

3. The author could also compare with some latest works of LLM post-training pruning works.

4. The improvements over baselines are incremental, some of them are very marginal.

Given the current state of this paper, it's a clear reject for me.

[1] Sparse Low-rank Adaptation of Pre-trained Language Models, EMNLP'23

[2] LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning, ACL'24

[3] LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery

[4] LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation

questions:
Please check the weakness. Please provide comprehensive experimental results and discussions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730621997630

thread_type:
official_review

<thread_object>
creator: reviewer_mfoh

<content>
summary:
The paper proposes Dynamic Low-Rank Sparse Adaptation (LoSA). This framework enhances sparse Large Language Models (LLMs) by integrating low-rank adaptation (LoRA) into the sparsity framework with dynamically adjusted layer-wise sparsity rates and rank allocations. LoSA utilizes representation mutual information (RMI) to determine layer importance for sparsity and reconstruction errors to allocate ranks, which purportedly improves performance without increasing inference latency.  Experimental results demonstrate that LoSA achieves considerable gains in accuracy, perplexity, and inference efficiency across various architectures and sizes.

strengths:
1.  LoSA introduces a combined dynamic sparsity and rank adjustment mechanism for fine-tuning sparse LLMs. Using RMI for layer-wise sparsity rate determination and reconstruction errors for rank allocation seems a reasonable approach for preserving model performance under sparse conditions. Moreover, trying to match the sparsity pattern of the adaptation path BA and the pre-trained weight W is novel. 
   
2. Comprehensive Empirical Evaluation: The paper’s experiments cover multiple architectures (LLaMA, Vicuna, OPT) and sizes, presenting results across different sparsity ratios. Performance improvements in both perplexity (Table 1) and zero-shot accuracy (Table 2) underscore LoSA’s adaptability across models and contexts, although some improvements at lower sparsity rates are modest.

3. Inference Efficiency: LoSA reportedly achieves significant inference speedups, which is crucial for deploying LLMs in resource-constrained environments. The throughput gains shown in Table 9 demonstrate LoSA’s potential in reducing inference latency compared to other sparsification methods, although these gains are achieved using specific sparse inference engines.

weaknesses:
1. The paper lacks comparisons with adaptive LoRA methods like AdaLoRA[1] and SoRA[2], which are critical for evaluating LoSA’s performance among recent dynamic rank approaches. Without these comparisons, LoSA’s relative advantage remains unclear.

2.  The optimization setup in Eq. 5 (Section 2.2) assigns higher sparsity rates to layers with higher importance, which contradicts standard practices that seek to preserve the most important layers. This questionable logic may weaken the model’s representational power and suggest a potential flaw in LoSA’s sparsity allocation strategy.

3. The paper does not clearly explain how LoSA ensures consistency in sparsity across LoRA weights (BA matrices) and model weights (W matrices). Given the critical need for alignment in the sparse structure, the mechanism by which LoSA achieves this alignment is unclear, especially in the context of using SparseGPT or Wanda. This missing detail may complicate LoSA’s practical applicability.

4. While some ablation studies are presented, further analysis is needed on the soundness and effectiveness of the RMI-based sparsity and reconstruction-based rank allocation across different architectures.

[1] Zhang, Qingru, et al. "AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning." arXiv preprint arXiv:2303.10512 (2023).

[2] Ding, Ning, et al. "Sparse low-rank adaptation of pre-trained language models." arXiv preprint arXiv:2311.11696 (2023).

questions:
1   - Could authors provide results comparing LoSA with adaptive LoRA baselines like AdaLoRA and SoRA? These comparisons would help clarify how LoSA performs relative to other dynamic sparsity approaches.

2- In Eq. 5, why is the sparsity rate set to increase with the layer importance score? This setup seems counterintuitive since it would sparsify important layers more, potentially impacting the model’s performance. Could you explain this choice and its implications?

3- How does LoSA ensure consistent sparsity patterns between the LoRA (BA) and LLM (W) weights? Does the method sparsify entire rows or columns to maintain this alignment, or is there another approach? Further detail would clarify how LoSA integrates with existing sparse methods like SparseGPT and Wanda.

4- It would be helpful if authors could also elaborate on the computational overhead introduced by the RMI and reconstruction error-based adaptations. Specifically, how does this additional computation scale with model size?

 5- Could authors also clarify the interpretability of RMI as a sparsity allocation metric? While the Information Bottleneck principle supports its use, it would be helpful to see additional justification or experiments demonstrating that RMI consistently aligns with real layer importance (e.g., gradient-based layer importance) across diverse architectures.

6- Given that LoSA’s speedups are measured using specific sparse inference engines (e.g., DeepSparse, nm-vllm), how generalizable are these results to other deployment environments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718388072

thread_type:
official_review

<thread_object>
creator: reviewer_fsq8

<content>
summary:
This paper propose an adaption method for fine-tuning a dense LLM resulting into a sparse LLM which is more powerful and faster. The proposed algorithm works on sparsity design and low rank allocation. Experiements are evaluated on LLaMA and OPT models to demonstrate the better model performance and faster inference speed while comparing with other method at the same sparsity rate.

strengths:
1. reasonable motivation, studying the sparsification of LLM while applying LoRA. 
2. clear problem formulation and related work introduction at each subproblem. 
3. detailed and summarized pseudocode for connecting each step and explaining the overall algorithm. 
4. strong and promising experimental results on LLMs regarding both model performance and speedup.

weaknesses:
1. the algorithm consists of many heuristics and is lack of step by step derivation, e.g., Eq. 7 and Eq. 9
2. some experiment details are missing and unclear.

questions:
1. section "RMI for Sparsity" with Eq. 2 and Eq. 3 are not used in the proposed algorithm which can confuse readers for their purpose. this paper could shorten this presentation. 
2. Eq. 7 has a hidden assumption that higher rank setting can help reduce reconstruction loss. According to Eq. 1, under current problem formulation, higher rank does not change anything given the sparsity mask M. Thus, the design of Eq. 7 is not useful. Similarly, this question also applies to Eq. 9. Why Eq. 9 is performed at each step and what happens if Eq. 9 became too large?
3. It is unclear how rank increase and decrease is implemented in practice. For example, if current rank is 3 and you want to increase to 4, do you initialize the additional vector randomly? if current rank is 4 and you want to decrease to 3, do you perform rank reduction by singular values magnitude (if you use svd)? 
4. It should be discussed the computation complexity. In particular, the sparsity mask computation can be slow. Given the algorithm is executed at each fine-tuning step, the overall computation time should be reported. 
5. Experiment result table report "SparseGPT with LoSA", while the Algorithm 1 input is dense weight of LLM. Which part of dense weight of SparseGPT you work on?
6. Which layer of LLM and weight matrix (Q,K,V, etc) you run experiment with?
7. What is the outcome of SparseGPT with LoRA? Does it become a dense LLM? If so, what does sparsity 50% mean for SparseGPT with LoRA?
8. Is there any guideline for setting fine-tuning steps? In Table 1, T=5, why 5 steps can be sufficient for completing LoRA training?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730822173817

thread_type:
official_review

<thread_object>
creator: reviewer_vhbk

<content>
summary:
The paper introduces LoSA (dynamic Low-rank Sparse Adaptation), a novel method for fine-tuning sparse Large Language Models (LLMs). LoSA addresses the performance degradation often observed in sparse LLMs by integrating low-rank adaptation (specifically inspired by LoRA) directly into the sparsity process.  It does so by dynamically sparsifying the low-rank adaptation outcomes to ensure compatibility with the sparse LLM weights and enable merging post-training.  Furthermore, LoSA uses Representation Mutual Information (RMI) to dynamically adjust layer-wise sparsity rates and allocates the rank of low-rank adaptation based on layer-wise reconstruction errors.  Experiments on various LLMs (LLaMA variants, OPT, Vicuna) demonstrate that LoSA significantly improves perplexity and zero-shot accuracy compared to existing sparsity methods and LoRA, while maintaining inference speedup.

strengths:
* Addresses a relevant problem: Performance degradation in sparse LLMs is a known issue, and LoSA offers a practical solution.

* Novelty: Integrating sparsification into the low-rank adaptation process and dynamically adjusting sparsity/rank based on RMI and reconstruction errors are novel ideas.

* Strong empirical results: The experimental results show consistent improvements across various LLMs and sparsity levels.

* Inference efficiency: LoSA preserves the inference speed advantages of sparsity by merging the adapted weights.

weaknesses:
na

questions:
1. Can you comment (qualitiatively) on the applicability of the mutual information based layer-sensitivity method to other compression techniques -- e.g., would it work for quantization, or if we were jointly quantizing and sparsifying?

2. You have a brief section on N:M sparsity, where you fix M=8. How easy is it to extend this to also determine the right value of M for different layers? Relatedly, any notes on smaller values of M (like M=4, found in Nvidia GPUs as 2:4 sparsity)?

3. Can you add more color to the cost of your proposed method during training (i.e., impact on training time). There is an annotation that it takes 48 seconds for LLama-2-7B. Can you clarify: (i) what is the relative contribution to the overall step time, (ii) if the computations are performed in every step (or less frequently like every k steps -- or if it is possible to do it less frequently).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732221470807

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers and ACs

comment:
We extend our heartfelt thanks to all reviewers and area chairs for their time and efforts in reviewing our paper and providing insightful comments. We are pleased that the reviewers have recognized our contributions, which we summarize below:
1. **Novelty.** We propose a novel and efficient fine-tuning method named LoSA for sparse LLMs, which seamlessly integrates low-rank adaptation into LLM sparsity within a unified framework and dynamically adjusting sparsity/rank based on mutual information and reconstruction errors. (Reviewer VhBK and MFoH)
2. **Strong and comprehensive empirical results.** The experimental results show consistent improvements across various LLMs and sparsity levels. Our paper’s experiments cover multiple architectures (LLaMA, Vicuna, OPT) and sizes, presenting results across different sparsity ratios. Performance improvements in both perplexity and zero-shot accuracy underscore LoSA’s adaptability across models and size. (All reviewers)
3. **Inference efficiency.** Our LoSA method preserves the inference speed advantages of sparsity by merging the low-rank adaptation weights. LoSA achieves significant inference speedups on CPUs and GPUs, which is crucial for deploying LLMs in resource-constrained environments. (Reviewer VhBK, FSq8, MFoH and VoNN)

We also appreciate the constructive suggestions and concerns raised by the reviewers, which have led to productive discussions and additional experiments. We have made appropriate revisions, with all changes marked in blue for easy reference. We believe these revisions further strengthen our paper. We summarize our major changes as follows:

1. Make minor corrections to the errors in Eq.1 and Eq.5. (Section 2.1 and 2.2)
2. Add implementation details of parameter initialization when increasing or decreasing the rank of low-rank adaptations. (Section 2.3)
3. Add comparisons and discussions with previous similar works. (Section B)
4. Add comparison results of LoSA and Sparse LoRA. (Section C.5)
5. Add ablation experiments on the OPT model to verify the effectiveness of RMI-based sparsity and reconstruction-based rank allocation. (Section C.6)
6. Extend our LoSA method to structured pruning and compare with LLM-Pruner, SliceGPT, LoraPrune, and LoRAShear baselines. (Section D)
7. Add experimental results of LoSA fine-tuning some latest LLM post-training sparsity works. (Section E)
8. Add experimental comparison with SoRA and AdaLoRA baselines. (Section F)
9. Add discussion on mixed N:M sparsity setting experimental results. (Section G)
10. Add computational cost statistics of RMI-based sparsity and reconstruction-based rank allocation for models with different parameter sizes. (Section H)
11. Add analysis of layer-wise sparsity rate obtained by RMI. (Section I)

Finally, **we have addressed all the raised questions and concerns**, and we present detailed responses to individual reviewers below. We look forward to any additional comments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732503343122

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A kind reminder regarding our response

comment:
Dear Reviewers,

We have put in many efforts to address the raised questions and concerns. As the ICLR rebuttal period is approaching its end, we kindly remind you to review our submitted response. Your feedback is essential for finalizing our work.

Thank you for your attention.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces "Dynamic Low-Rank Sparse Adaptation" (LoSA), a novel framework designed to enhance the performance of sparse LMMs by integrating low-rank adaptation within their sparsity structures. LoSA dynamically adjusts layer-wise sparsity rates and low-rank allocations based on Representation Mutual Information (RMI) and layer-specific reconstruction errors. This approach ensures efficient compatibility between sparse and low-rank structures, maintaining inference speed while significantly improving model accuracy and perplexity across various LLM architectures like LLaMA, OPT, and Vicuna, without increasing inference latency.

**Strengths.** 

All reviewers found the paper well-motivated, clearly written, and straightforward to follow. Also, they admired the extensive and promising experimental results across LLM architectures like LLaMA, Vicuna, and OPT, which demonstrate significant performance improvements and inference speedups, showing potential for deployment in resource-constrained environments.

**Weaknesses** 

The reviewers highlighted key weaknesses in the submission, such as the absence of comparisons with recent pruning studies (raised by most reviewers), limited novelty, inadequate computational analysis of the RMI method, and insufficient implementation details. Additionally, they noted marginal improvements in several experimental results. Some minor issues, including the unconventional definition of sparsity rate, also caused confusion. 

**Conclusion** 

In my view, the authors have effectively addressed most of the previously identified weaknesses and have made significant enhancements to the experimental section of their paper. The proposed method, while relatively simple, has merit when compared to recent baselines. Regrettably, despite repeated reminders from the Area Chair, several reviewers have not acknowledged the authors' rebuttal efforts. Notably, the review by Reviewer 7hsM, which received a rating of 3 and a confidence level of 4, was less substantial and focused solely on comparisons with recent baselines, a concern that the authors have adequately addressed in their rebuttal. Furthermore, Reviewer VoNN's primary critique is that different components of the paper have appeared in previous work; hence, the paper lacks novelty. However, the authors have appropriately credited prior work in the revised paper, particularly regarding components like the RMI. I agree with the authors that integrating these methods into their proposed framework does introduce a novel element, even if it's not groundbreaking. Considering these factors, I believe the paper is a step in the right direction. I join the majority of the reviewers in positively evaluating this work and believe that it will contribute value to our esteemed conference. Consequently, I vote for accepting the paper.