Title: 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Sparse plus Low-Rank

Model Compression

Large Language Models

LoRA

PEFT

ADMM

Optimization
Abstract: Sparse plus Low-Rank $(\mathbf{S} + \mathbf{L}\mathbf{R})$ decomposition of Large Language Models (LLMs) has emerged as a promising direction in $\textit{model compression}$, aiming to decompose pre-trained model weights into a sum of sparse and low-rank matrices $\mathbf{W} \approx \mathbf{S} + \mathbf{LR}$. Despite recent progress, existing methods often suffer from substantial performance degradation compared to dense models. In this work, we introduce $\texttt{3BASiL-TM}$, an efficient one-shot post-training method for $(\mathbf{S} + \mathbf{L}\mathbf{R})$ decomposition of LLMs that addresses this gap. Our approach first introduces a novel 3-Block Alternating Direction Method of Multipliers (ADMM) method, termed $\texttt{3BASiL}$, to minimize the layer-wise reconstruction error with convergence guarantees. 
We then design a transformer-matching ($\texttt{TM}$) refinement step that jointly optimizes the sparse and low-rank components across transformer layers. This step minimizes a novel memory-efficient  loss that aligns outputs at the transformer level.
Notably, the $\texttt{TM}$ procedure is universal as it can enhance any $(\mathbf{S} + \mathbf{L}\mathbf{R})$ decomposition, including pure sparsity. Our numerical experiments show that $\texttt{3BASiL-TM}$ reduces the WikiText2 perplexity gap to dense LLaMA-8B model by over 30% under a (2:4 Sparse + 64 LR) configuration, compared to prior methods. Moreover, our method achieves over 2.5x faster compression runtime on an A100 GPU compared to SOTA $(\mathbf{S} + \mathbf{L}\mathbf{R})$ method.
Our code is available at https://github.com/mazumder-lab/3BASiL.


Peer reviews:
comments: SUMMARY:
In this paper, the authors propose 3BASiL, which introduces a unified algorithmic framework for compressing LLM by decomposing weight matrices into sparse + low rank form. Its key innovations include: 

1) Propose a 3-block ADMM algorithm that jointly updates sparse, low-rank, and the dual variables with a closed-form update for each.

2) Propose a block-level weight fine-tune step that is able to further improve performance.

STRENGTHS AND WEAKNESSES:
**Strengths:**

1) The method offers closed-form updates for each component (sparse, low-rank, dual), and is supported by theoretical convergence guarantees.

2) Compared to prior work, it achieves up to 40% lower perplexity and 2–3× faster compression, demonstrating both efficiency and effectiveness.

**Weaknesses:**

1. The necessity of adding the term $\|\hat{W} - (S + L)\|$ in Equation (1) is not clearly explained. When computing updates, doesn't it behave similarly to $\|(X + \lambda)\hat{W} - (X + \lambda)(S + L)\|$?

2. While the method focuses on N:M sparsity, it seems generalizable. Have you evaluated its performance under unstructured or other structured sparsity formats? Additionally, could it adapt to non-uniform sparsity distributions, such as OWL[1]?

3. The paper mentions 80 alternating minimization steps during compression. Have you conducted sensitivity analysis on the number of steps?

[1] Lu et al. Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity.

QUESTIONS:
Please see the weakness.

LIMITATIONS:
The authors addressed the limitations in the Conclusion and limitation section.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-25

comments: SUMMARY:
This paper proposes a sparse + low-rank approach for model compression and introduces cross-layer joint optimization to improve the performance of the compressed model.

STRENGTHS AND WEAKNESSES:
**Strengths**：

1.This work proposes a novel three-block variable decomposition framework that jointly optimizes sparsity and low-rank constraints, which address the theoretical limitations of existing alternating minimization methods.

2.Introduced a TM fine-tuning step after model compression to enhance the performance of the compressed model.

**Weaknesses**：

1.This method is only applicable to semi-structured conditions, which limits its scope of application

2.Experiments were only conducted on models of 1-8B size. The performance on larger-scale models remains to be further verified

QUESTIONS:
1.The paper primarily presents experimental results for the semi-structured sparse component. To demonstrate the method's general applicability, could additional comparisons with unstructured sparsity be attempted?

2.Could the actual acceleration effects of the compressed model in practical usage be provided to show that the low-rank + sparse method is indeed well-adapted to hardware?

3.If conditions permit, a more extensive assessment can be conducted appropriately.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, an ADMM algorithm is studied for compressing dense weights into sparse-plus-low-rank (S+LR) representations. With particular sparse patterns, such as 2:4 sparsity, S+LR matrices can be computed efficiently on GPUs. The paper proposes a multi-block ADMM procedure to initialize the factors using calibration data, which provides a good initialization and thus improves the model quality after fine-tuning. Each factor is obtained by minimizing the augmented Lagrangian while fixing the others, with convergence guarantees as the penalty term rho increases. To address misalignment in layer-wise output matching, the paper also introduces module-wise output matching. Experimental results demonstrate that 3BASiL improves both one-shot and fine-tuned compression outcomes across various structured sparsity patterns on large language models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper presents both theoretical and practical contributions. The proposed algorithm appears to outperform other S+LR decomposition models, and the experimental results show that the compressed models maintain decent performance. Theorem 1 provides convergence guarantees, which is a strong point.\n2. The writing quality is good, making the paper easy to understand.\n\n**Weaknesses**\n\nMost of my concerns relate to the experimental results.\n\n1. Although Theorem 1 proves convergence of the proposed ADMM steps, it is unclear whether convergence occurs in practice. Please consider including plots showing the diminishing difference between L and S at each step or the convergence of the augmented Lagrangian.\n2. The actual inference speedup is not reported. While it seems likely that the compressed network will run efficiently on GPUs supporting 2:4 or 4:8 sparsity, reporting real inference speed measurements would help emphasize the practical impact of the proposed method.\n\nQUESTIONS:\n1. Why is 3BASiL designed with a 3-block ADMM? In the Robust PCA literature, an ADMM algorithm to solve robust PCA $\\min_{L,S} ||L||_* + ||S||_1$  s.t.  $L+S=W$ can be formulated as a 2-block method. Is the use of 3 blocks in 3BASiL due to the hard sparsity constraint, unlike the l1 loss in RPCA?\n2. The last inequality of Equation 18 uses $||S^{(t+1)} - S^{(t)}||_F \\le \\frac{C_A}{\\rho_t}$, whereas the bound in Equation 17 is $\\frac{3C_A}{\\rho{t-1}}$. How is the former inequality obtained? Can’t the bound in Equation 17 be used directly?\n3. Transformer-level Matching is proposed to update the factors with a better proxy than layer-level matching. If the quality of this proxy is a concern, couldn’t the S and L factors be updated directly based on the original training loss (e.g., cross-entropy) or a knowledge distillation loss on a subset of C4, instead of splitting the process into Transformer-level Matching and LoRA fine-tuning?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe large scale of Large Language Models (LLMs) makes them a big target for compression, and one common technique is sparsity which, when coupled with low-rank decomposition, shows promise at matching the dense model's quality.  Despite many developments in this direction, there still exists a large gap in model quality.  The authors present 3BASiL-TM to help close this gap.  First, 3BASiL is a 3-block ADMM algorithm which alternates between updating the sparse version of the weights, the low-rank component, and the sparse mask.  Treating L and D as a single variable block reformulates the approach as standard 2-block ADMM and helps guarantee convergence.  Next, the authors propose Transformer-level Matching (denoted by a -TM suffix), which serves to further refine the sparse and low-rank components, using the nonlinear behavior of each transformer block, so that the low-rank components are well-initialized for LoRA adaptation.  Empirical studies show that TM generalizes to many one-shot S+LR methods, that its application to 3BASiL delivers top-quality results, and that the benefit of 3BASiL-TM is maintained, even after LoRA fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\nThis submission is outstanding.  It presents a novel collection of related ideas, clearly motivating and proving the benefit of each.  Treating each metric in turn:\n### Quality\nI couldn't find any missing links in the chain of reasoning from claim to claim.  Each claim was supported by theory or empirical results, as appropriate, with one exception (noted below).  This is clearly a complete package and not a work in progress.  (Future work is indicated, but well beyond the scope of a single endeavor.)\n### Clarity\nI believe that sufficient detail is presented such that a dedicated practitioner could implement the technique and reproduce results.  Fully understanding each step does take careful reading, but the organization and exposition is not to blame; the mechanisms used are simply nontrivial.\n### Significance\n3BASiL-TM represents a major step forward in recovering model quality.  There are still gaps to the dense baselines, but they are greatly reduced without undue computational load. Care has been taken to not only find compelling new techniques, but to make them practical for use - runtimes in single-digit hours is entirely reasonable that users shouldn't need to shy away from one-time applications to their models due to limited resources.  (Larger models will require more time, but will also require more resources to train and deploy.)  As I discuss immediately below, there are findings within the results that I consider very useful for advancing the field.\n### Originality\nI only noted one missing piece of related work (below); otherwise, the comparison to prior work is complete and shows that the collection of techniques in 3BASiL-TM is both new and important.  \n\nA couple observations show the importance of the submission goes beyond its ability to improve zero-shot model quality:\n- Table 3 is particularly interesting: for a given compression rate, it's important to find the right balance between sparsity and the rank of the LR.\n- Figure 4 is also important: I've observed \"head-starts\" by clever one-shot schemes become washed out after fine-tuning.  It's good to see this doesn't happen to 3BASiL-TM (at least for the limited fine-tuning performed).\n\n## Weaknesses\nI noted one missing comparison - EoRA (1), which seems like it might compete with the \"smart initialization\" of LoRA parameters offered by transformer matching.\n\nI also could not find evidence to support the claim on line 181 that TM can apply to sparse, but not LR, networks.  I can imagine how it will apply, but this is not supported in any of the presented results.\n\nMinor issues:\n- Inconsistent citation styles are confusing - particularly (but not limited to) lines 213-221.  Some citations are missing years (e.g. lines 290, 292).\n- The wrong entry in Table 2's 2:4+64LR RTE results is bolded (or there's a typo in one of the 3BASiL results).\n- Citing the N:M sparse format (and the hardware advancements that make it useful) (e.g., 2) would make the background information complete.\n\n1. Liu et al., EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation, https://arxiv.org/abs/2410.21271\n2. Mishra et al., Accelerating Sparse Deep Neural Networks, https://arxiv.org/abs/2104.08378\n\nQUESTIONS:\nThese two questions directly relate to the quality of the submission:\n1. How does TM improve, say, the 2:4 and 4:8 results in Table 3?\n2. How does EoRA interact with 3BASiL as a whole, and specifically with using TM as a smart initialization for LoRA fine-tuning?\n\nLIMITATIONS:\nThe limitations are satisfactorily discussed in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}