SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs

Authors: Mohammad Mozaffari, Amir Yazdanbakhsh, Zhao Zhang, Maryam Mehri Dehnavi
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: sparse training, low rank adapter, LLM, optimization

Abstract:
We propose SLoPe, a Double-Pruned **S**parse Plus **L**azy L**o**w-rank Adapter **P**r**e**training method for LLMs that improves the accuracy of sparse LLMs while accelerating their pretraining and inference and reducing their memory footprint. Sparse pretraining of LLMs reduces the accuracy of the model, to overcome this, prior work uses dense models during fine-tuning. SLoPe improves the accuracy of sparsely pretrained models by adding low-rank adapters in the final 1% iterations of pretraining without adding significant overheads to the model pretraining and inference. In addition, SLoPe uses a double-pruned backward pass formulation that prunes the transposed weight matrix using N:M sparsity structures to enable an accelerated sparse backward pass. SLoPe accelerates the training and inference of models with billions of parameters up to 1.25× and 1.54× respectively (OPT-33B and OPT-66B) while reducing their memory usage by up to 0.63× and 0.61× for training and inference respectively.

Decision: accept (poster)
Reviewer scores: 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730050712282

thread_type:
official_review

<thread_object>
creator: reviewer_b3ze

<content>
summary:
This paper proposes SLoPE, a Double-Pruned Sparse Plus Lazy Low-rank Adapter pretraining method that improves the accuracy and accelerates training and inference of sparse LLMs. By incorporating a low-rank adapter during the final 1% of pretraining iterations, the model accuracy is well maintained. Experimental results show that the proposed method significantly speedups in both pretraining and inference phases and efficiently reduces memory footprint.

strengths:
1. The authors propose a double-pruned backward pass to improve model quality and reduce mask search overhead. They introduce additional low-rank adapters in the final 1% iterations of pretraining and improving model capacity.

2. The paper is well-written and easy to follow up. The finding of adding low-rank adapter in the last 1% iterations achieving better performance is interesting!

weaknesses:
1. Why does introducing additional zeros in an N:M pattern during the computation of the input gradient in the backward pass of the first 99% of iterations not severely affect accuracy?

2. The memory footprint for the pretraining should be higher than for inference, so why are the memory footprint reduction multiples similar? In the backward pass, calculating weight gradient involves a dense computation. While double-pruning reduces the computation of input gradients, the overall memory reduction should be more substantial.

3. As discussed in section 2.2, the authors introduce a hyperparameter to balance the memory footprint, computational efficiency, and model quality. In the experiments on the effects of low-rank adapters, ranks are set to 4,16, and 64. Why weren’t larger ranks used? Since the low-rank adapter is only applied in the final 1% of iterations, the computational overhead should be minimal.

questions:
Please see above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730131758783

thread_type:
official_review

<thread_object>
creator: reviewer_nkfr

<content>
summary:
This paper introduces double-pruned sparsity, a technique that applies two rounds of N:M sparsity to the weight matrices of a neural network: one for accelerated forward pass and another on the transpose of the already N:M sparse matrix for an accelerated backward pass. The proposed method is combined with LoRA-tuning in the final 0.01 iterations of pre-training resulting in the method termed Double-Pruned Sparse Plus Lazy Low-rank Adapter Pre-training (SLoPE). The authors use efficient tiling of upsample tensors and kernel fusion to reduce overhead, resulting in further boost in training and inference speed. They show that their approach results in speedup and memory reduction of pre-training LLMs of various sizes. They also show their method improves zero-shot accuracy of GPT2-small compared to a sparse pre-training baseline.

strengths:
1-The proposed double-pruned N:M sparsity for both forward and backward acceleration offers a novel perspective on N:M
sparsity, improving flexibility compared to existing transposable N:M masks.

2-The tiling and kernel fusion solution that integrates LoRAs with sparse weights for enhanced efficiency is interesting.
The authors effectively demonstrate the acceleration and memory reduction benefits of their approach across various models, outperforming FST [1].

3- The method achieves performance surpassing SR-RTE[2] and approaches that of a densely pre-trained model in tasks like MMLU, Arch Challenge, and OpenQA.

[1] Accelerating Transformer Pre-Training with 2:4 Sparsity, Hu et all, ICML 2024

[2] Learning N:M Fine-grained Structured Sparse Neural Networks from Scratch, Zhou et. all, ICLR 2021

weaknesses:
1- The most significant weakness is the limited evaluation of how the proposed sparsity impacts performance. The sparse pre-training method negatively affects performance, and the use of a static sparse matrix further limits flexibility. While the authors claim this does not hinder flexibility, more evidence across a broader range of downstream tasks is needed to support this claim. Given the inherent trade-off between speed and performance, the current experiments do not sufficiently demonstrate the method's effectiveness. Table 3 should include more tasks (tasks in FST[1]) to give the reader a clearer understanding of the impact.

2- Even in the limited downstream task evaluation, some important baselines are missing. The authors put FST in parentheses after SR-STE in the text, although these are distinct methods: FST refers to [1] while ST-RTE is [2]. Zero-shot FST results in their own setting on these tasks are not reported. Additionally, while comparing their method to a larger dense model may be unfair, it would be informative to see how a smaller dense model performs without sparse pre-training.

3- The impact of LoRA fine-tuning in the final iterations appears minimal. The benchmarks (MMLU, Arch Challenge, and OpenQA) are not reported for the LoRA ablations. Since a major contribution of this work is the tiling and kernel fusion solution designed to mitigate issues caused by adding LoRAs, it is concerning that adding LoRAs seems to have little effect, especially based on the current results.

[1] Accelerating Transformer Pre-Training with 2:4 Sparsity, Hu et. all, ICML 2024

[2] Learning N:M Fine-grained Structured Sparse Neural Networks from Scratch, Zhou et. all, ICLR 2021

questions:
1-The authors show that their method is faster than FST during pre-training, which is expected since FST dynamically searches for a mask, whereas their method uses a static one, making the training column in Table 1 clear. However, the inference column of Table 1 suggests that their method is also faster than FST for inference. All sparse pre-training methods use the full model for inference. Does this imply that their method uses the sparse matrix for inference (e.g., during zero-shot evaluations)? Then what is the point of sparse pre-training? In the table caption, it’s mentioned that "the lack of inference speedup in FST is due to the final dense pre-training during the final iterations, resulting in a dense model for inference." Could the authors clarify if you have included fine-tuning on a downstream task as inference? and how are the SLOPE values in the inference columns calculated?

2-Is flash attention used for the baselines as well (line 312)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712118074

thread_type:
official_review

<thread_object>
creator: reviewer_ywjx

<content>
summary:
This paper introduces SLOPE, a novel pretraining method aimed at improving the efficiency of large language models (LLMs) by combining sparsity and low-rank approximations. SLOPE employs double-pruning techniques—applying N:M sparsity masks during both forward and backward passes—and introduces lazy low-rank adapters in the final training stages. The double-pruning approach reduces memory and computational overheads, while the lazy low-rank adapters help maintain model accuracy by mitigating the quality loss from sparsity.

strengths:
1. This is a technical solid article that designs optimized CUDA kernels that jointly optimize Nvidia 2:4 sparse kernels and low-rank calls through efficient tiling and scheduling.

2. SLOPE yields up to 1.25x faster training and 1.54x faster inference on models with billions of parameters, with a memory reduction of up to 63% during training.

weaknesses:
1. Structured and semi-structured sparsity with low-rank adapters, are not new concepts in the literature (e.g., see Losparse [1]).

2. This article lacks an introduction to related work and baseline methods, such as FST. Readers unfamiliar with the field may find it challenging to follow the content.

3. Could you provide results on additional zero-shot datasets in GLUE beyond MMLU, ARC-c, and OpenBookQA for GPT-2 when comparing SR-STE?


[1] Li, Yixiao, et al. "Losparse: Structured compression of large language models based on low-rank and sparse approximation." In International Conference on Machine Learning, pp. 20336-20350. PMLR, 2023.

Minor: 
1. L 430 typo of "accuracy"

2. The bold values in the Tables are misleading:

Table 1: Why are some values of FST also bolded?

Table 3: Why are the values of SR-STE not bolded on the MMLU dataset?

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731817433635

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you!

comment:
Dear Area Chairs and Reviewers,

We are grateful for the time you have dedicated to reviewing our manuscript and for your insightful feedback. We appreciate the positive remarks from the reviewers, praising SLoPe for its (1) novelty and a technical solidity with optimized CUDA kernels with efficient tiling and scheduling [Reviewer ywjx](https://openreview.net/forum?id=lqHv6dxBkj&noteId=f2zY0XLoGq), (2) surpassing other sparse and dense methods in accuracy and effective demonstration of acceleration and memory reduction [Reviewer NKfR](https://openreview.net/forum?id=lqHv6dxBkj&noteId=1lfuUA3d4g), (3) significant speedup in both pretraining and inference with a well-written and easy to follow manuscript [Reviewer B3ze](https://openreview.net/forum?id=lqHv6dxBkj&noteId=XBJWBmV8Ay). Your comments have been instrumental in refining our work. Consequently, we have added further clarifications and additional results, which are summarized below:

- [Rebuttal-A1](https://openreview.net/forum?id=lqHv6dxBkj&noteId=MxWgE5lxsL), [Rebuttal-A2](https://openreview.net/forum?id=lqHv6dxBkj&noteId=Qykay4JQDw): Testing SLoPe and other benchmarks on the GLUE zero-shot downstream task.

- [Rebuttal-A2](https://openreview.net/forum?id=lqHv6dxBkj&noteId=ND6rig1pY3): Pretraining a fully dense model with the same size as the sparse models and reporting GLUE, MMLU, Arch Challenge, and OpenBookQA to further clarify the effectiveness of SLoPe.

We are confident that we have responded to each of the reviewers' comments individually. We look forward to engaging in a constructive discussion during the author response period.

Best regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732325903172

thread_type:
official_comment

<thread_object>
creator: area_chair_kaqf

<content>
title:
Follow-Up Discussion on Author Feedback

comment:
Dear PC Members,

Thank you for your valuable comments during the review period, which raised many interesting and insightful questions. The authors have now posted their feedback, and I encourage you to review their responses and engage in further discussion if necessary.

I understand that you may have a busy schedule, but your additional input is highly appreciated, particularly for papers that are initially on the borderline. Your contributions are crucial in ensuring a fair and well-rounded decision-making process.

Thank you once again for your continued support and dedication to ICLR.

Best regards,
AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel pruning technique based on double-direction masks, which is further combined with LoRA in the final stages, resulting in the Double-Pruned Sparse Plus Lazy Low-rank Adapter Pre-training (SLoPE). The reviewers recognized the solid technical contributions of this work. However, there are lingering concerns about the experiments, particularly regarding the use of a dense training baseline and static masks. While some of the reviewers' questions were addressed during the rebuttal, the responses only partially resolved their concerns. As a result, the overall scores are generally positive, though one reviewer with a score of 5 noted that they might increase their score if other reviewers had no remaining doubts about the experiments.

Overall, I believe this is a borderline paper: it offers technical contributions, but they are not highly significant. I suggest acceptance, but I am open to any decision made by the PC/SAC.