3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs
Authors: Mehdi Makni
Xiang Meng
Rahul Mazumder
Venue: NeurIPS 2025 poster ()
Keywords: Sparse plus Low-Rank
Model Compression
Large Language Models
LoRA
PEFT
ADMM
Optimization
Primary area: optimization
TL;DR: We introduce 3BASiL-TM, a highly efficient one-shot post-training method for Sparse plus Low-Rank decomposition of LLMs that reduces the WikiText2 perplexity gap to dense model by over $30\%$ compared to prior methods.

Abstract: Sparse plus Low-Rank $(\mathbf{S} + \mathbf{L}\mathbf{R})$ decomposition of Large Language Models (LLMs) has emerged as a promising direction in $\textit{model compression}$, aiming to decompose pre-trained model weights into a sum of sparse and low-rank matrices $\mathbf{W} \approx \mathbf{S} + \mathbf{LR}$. Despite recent progress, existing methods often suffer from substantial performance degradation compared to dense models. In this work, we introduce $\texttt{3BASiL-TM}$, an efficient one-shot post-training method for $(\mathbf{S} + \mathbf{L}\mathbf{R})$ decomposition of LLMs that addresses this gap. Our approach first introduces a novel 3-Block Alternating Direction Method of Multipliers (ADMM) method, termed $\texttt{3BASiL}$, to minimize the layer-wise reconstruction error with convergence guarantees. 
We then design a transformer-matching ($\texttt{TM}$) refinement step that jointly optimizes the sparse and low-rank components across transformer layers. This step minimizes a novel memory-efficient  loss that aligns outputs at the transformer level.
Notably, the $\texttt{TM}$ procedure is universal as it can enhance any $(\mathbf{S} + \mathbf{L}\mathbf{R})$ decomposition, including pure sparsity. Our numerical experiments show that $\texttt{3BASiL-TM}$ reduces the WikiText2 perplexity gap to dense LLaMA-8B model by over 30% under a (2:4 Sparse + 64 LR) configuration, compared to prior methods. Moreover, our method achieves over 2.5x faster compression runtime on an A100 GPU compared to SOTA $(\mathbf{S} + \mathbf{L}\mathbf{R})$ method.
Our code is available at https://github.com/mazumder-lab/3BASiL.

3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs
Mehdi Makni
Xiang Meng
Operations Research Center
Operations Research Center
Massachusetts Institute of Technology mmakn @mit maittedu edu
Massachusetts Institute of Technology
mengx@mit.. brngxmmit.ec edu
Rahul Mazumder
Operations Research Center
Massachusetts Institute of Technology rahulmaz@mit... edu
Abstract
Sparse plus Low-Rank (S LR) decomposition of Large Language Models (LLMs) has emerged as a promising direction in model compression, aiming to decompose pre-trained model weights into sum of sparse and low-rank matri- ces W ~ S + LR.. Despite recent progress, existing methods often suffer from substantial performance degradation compared dense models. In this work, we introduce 3BASiL-TM, an efficient one-shot post-training method for (S + LR) decomposition of LLMs that addresses this gap. Our approach first introduces a novel 3-Block Alternating Direction Method of Multipliers (ADMM) method, termed 3BASiL, minimize the layer-wise reconstruction error with convergence guarantees. We then design an efficient transformer matching (TM) refinement step that jointly optimizes the sparse and low-rank components across transformer layers. This step minimizes novel memory-efficient loss that aligns outputs at the transformer level. Notably, the TM procedure is universal as can enhance any (S + LR) decomposition, including pure sparsity. Our numerical exper- iments show that 3BASiL-TM reduces the WikiText2 perplexity gap relative dense LLaMA-8B model by over 30% under (2:4 Sparse + 64 LR) configura- tion, compared to prior methods. Moreover, our method achieves over 2.5x faster compression runtime on A100 GPU compared to SOTA (S + LR) method. Our code is available at httpt::///// //t::////ittaee wom/mazumder......ooooooooomom.m lab/3BAS1L
1 Introduction
Large Language Models (LLMs) have demonstrated exceptional performance across diverse tasks including complex reasoming [Xu al. 2025). text generation [Achiam et al. [023), mathematical
problem-solving (Romera-Paredes et al. 2024 and code synthesis |Rozirre et al. 2023 How-
ever, state-of-the-art L.LMs [Achiam et al. 2023 Dubey al. 2024 Google 2023] with billions of
parameters face substantial deployment challenges due to their computational and memory require- ments. These constraints substantially limit real-time applications and deployment on resource- constrained devices. Consequently, model compression techniques have emerged research direction increase LLM accessibility while preserving their accuracy and functionality.
essential
Established methods for model compression primarily include neural network pruning LeCun et al.)
1989 Hassibi and Stork 1992. Han al. 20155) and quantization [Han al. 2015a [2016). For
LLMs. recent research has focused on one-shot post-training compression methods FFantar and
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Alistarh, 2023 Dettmers et al. 2023 Lin et al., 2024, Frantar et al. 2022 Behdin et al. 2023] Meng et al. 2024a b] that compress model weights using minimal calibration dataset without expensive retraining. These approaches have become particularly attractive as they enable efficient compression of modern L.MS even on a single commodity GPU.
An exciting recent line of research in one-shot compression studies the task of decomposing pre- trained weight matrices W into a compressed backbone component (e.g., sparse or quantized) and a low-rank component: W 2 C(W) + LR. This LoRA-aware formulation effectively integrates with Low-Rank Adaptation (LoRA) methods (Hu et al. 2022], allowing efficient downstream adap- tation by freezing C( C(W) and tine-tuning only the low-rank components, which serve as a smart- inirialization to LoRA. Guo et al.] [2024] [ al. [2024) demonstrate that this approach outper- forms the sequential approach of first compressing the model W = C(W) (under similar backbone components constraints) followed by standard LoRA fine-tuning (Noise & Zero adapter).
One-shot Quantized plus Low-Rank decomposition methods for LLMs [Guo et al.) 2024 Li et al.) 2024 Saha et al., 2024] have demonstrated exceptional efficiency. These works decompose pre- trained LLM matrices into low-rank components and memory-efficient quantized backbones, en- abling aggressive quantization while preserving model performance.
Parallel advances in Sparse plus Low-Rank (S + LR) decomposition Zhang and Papyan 2025
Makni et al.] [025 combine the strengths of pruming and matrix factorization. OATS (Zhang and Papyan 2025 pioneered post-training (S + +L)) compression for LLMs, demonstrating its viability as alternative to unstructured pruning at the same compression rates and achieving CPU accelera- tion via SpapSp Magira 2021]. HASSLF-Free [Makni et al.) 2025) established a unified (S + LR) framework showing an underlying connection between OATS and various LLM prun- ing methods-they focus on N:M sparse plus low-rank decompositions for GPU acceleration using specialized CUDA kernels from |Mozafaari al. 2024). Despite their promise, existing (S + LR) algorithms for LLMs rely exclusively on alternating minimization approaches. Due to the complex- ity of the underlying optimization problem, these procedures have limited convergence guarantees and may perform poorly in joint optimization of the sparse and low-rank components. Indeed, our empirical evidence in Figure[aalnde Saland [ngure/5bssugeeee that our proposed algorithm more effectively optimizes the sparse and low-rank parts compared to an alternating minimization approach.
In this paper, we propose 3BASiI an elegant 3-block ADMM approach tailored for (S + LR) decomposition of LLMMs. Unlike prior approaches that separate pruning and low-rank fitting steps, 3BASiL explicitly models their interaction through simultaneous optimization under unified objec- tive with provable convergence guarantees. We formulate the weight decomposition problem with explicit sparsity and rank constraints, decomposing into three variable sets sets-sparse component, low-rank component, and original weights- weights-optimized within an iterative ADMM framework. This approach precisely enforces sparsity pattern and rank constraints at each iteration via closed-form proximal updates while minimizing reconstruction error with respect to the original model weights.
(TM) procedure that refines sparse and low-rank components by aligning transformer block outputs with the dense model. In contrast to prior (S + LR) methods, which allow only low-rank components refinement via LoRA after layerwise compression, TM enables joint refinement of both sparse and low-rank components at the transformer level. This procedure is compatible with any existing (S + LR) method and can be applied prior to LoRA fine-tuning with minimal computational overhead, providing more effective initialization for downstream adaptation.
Additionally, propose (memory--effiiient)) transformer-mateen
Our contributions can be summarized as follows:
1. 3-Block ADMM We introduce 3BASiL, a novel 3-Block Alternating Direction Method of Mul- tipliers (ADMM) algorithm specifically designed for Sparse plus Low-Rank (S LR) decom- position of Language Models. Our method explicitly captures interactions between sparse and low-rank components within a unified optimization framework, while providing theoretical con- vergence guarantees as well. Moreover, 3BASiL offers remarkable computational advantages, achieving over 7x speedup compared to the strong MASSLE-frre-APPP bascline, when com- pressing a Llama3.2-33 3-BBBB model on an L.40 48GB GPU.
'3BASiL: 3Block ADMM for Sparsity and Low-Rank Constraints.

---

Trandormer Output
Kx
Add & Norm
W
XW
Update via GD
Linear
ADMM
Linear
Smart Initialization
Norm
2-4
Multi-hean attention
Sparse
plus
Low-RRak
Transformer
Inpat
Figure I: Overview of the proposed 3BASiL framework. (Left) For each layer in Transformer, employ multi-Blosd ADMM to efficiently decompose weights into high-quality Sparse plus Low- Rank components by minimizing the layer reconstruction objective. (Right) At the Transformer level, we apply gradient-based optimization to jointly refine all sparse and low-rank components across layers match the original transformer's output, with the resulting low-rank components serving as smart initialization for subsequent LoRA fine-tuning.
Transformer matching and Universality We introduce TM, novel (memory-efficient) refine- ment procedure that jointly optimizes sparse and low-rank components across transformer lay- ers. This approach significantly improves sparse component quality with minimal computational cost by directly leveraging transformer-level outputs, addressing major limitation in current (S + LR) methods. Crucially, our TM procedure is universally applicable and can enhance any existing (S LR) decomposition method, including purely sparse compression, providing superior initialization for subsequent LoRA fine-tuning.
state- of-the-art method for (S + LR) one-shot decomposition Large Language Models. signifi- cantly improves LLM evaluation benchmarks including perplexity of different datasets and var- ious zero-shot tasks. Specifically, our numerical experiments show that 3BASiL-TH reduces the WikiText2 perplexity gap dense model by over 30% compared to prior methods for Llama- 8B model under a (2:4 Sparse 64 LR) configuration. It also provides significant compression runtime speedups compared other (S LR) decomposition techniques for LLMs.
Empirical Validation and State-of-the-Art Results We introduce 3BASiL-TM
Highly effective Sparse plus Low- decomposition via ADMM
2.1 Problem formulation
We compress the layers an LLM sequentially, one at time by minimizing the reconstruction error between the outputs of pre- pre-trained weights and compressed ones on a set of given input activations. Formally, let W represent the pre-trained weight matrix of given layer, and X denote the input activations (i.e., output of previous layers) on set of N calibration samples. In our setting, the goal of layer-wine wise reconstruction is find (S LR) decomposition that minimizes the lg error between the outputs of the original and decomposed weights-this can be formulated as follows:
min S.L
XW x (S 1
X 2
W (S+ S.L. SES, rank (L) Sr. (1)
2
Above || denotes the Frobenius norm, S denotes the set of matrices satisfying a specified sparsity constraint (e.g., unstructured sparsity with given sparsity level or N:M sparsity); and denote the sparse and low-rank components, respectively. Parameter À 0 encourages the decomposed weights to remain close to the pre-train pre-trained ones.
2.2
A multi-Block ADMM approach for layer-wise reconstruction
The primary challenge in optimizing problem (I) lies in the joint minimization of S and L under two complex constraints cesstaintt--sparsittyy and low-rank. To address this, we employ the Alternating Direction Method of Multipliers (ADMM), which enables separate updates of S and L at each iteration while

---

maintaining their interdependence through a Lagrangian multiplier. This approach preserves the power of joint optimization while making the problem tractable. Our 3-block ADMM introduces an auxiliary variable D as copy of the sparse component S, reformulating problem ①
as:
| +
A w LS) (s()
2
min S.D.L.
XW XS (S+
2
S.t.
S rank Sr.
Sr.
where lls (D) is an indicator function that equals to infinity when D & and zero otherwise. The augmented Lagrangian function with dual variable V and a quadratic penalty parameter p > reads:
L,, = xw L À
w (S (S+ L) + Ts(D) P s D+
The method proceeds by minimizing the augmented Lagrangian with respect three variables quentially: the sparse component S. the low-rank component L, and sparse component's ' constrained copy D, followed by dual update (in variable V). This sequential optimization over three variable blocks gives the method its name: 3-Block ADMM. At iteration t, the updates are:
((++) = arg ming (S, (O), D(O), V())
L ((+1) = arg ming C, (S(+1),, L, D v(1)
D(++1) arg minp (S((tI) ((+1), D, V(") V v p(S D((+))).
Below, we derive the updates. For notational simplicity, we denote = X X
S-block update Since (S, L((), D v(t) is a quadratic function of S, we obtain the closed- form solution by setting the gradient zero:
((++1) pI) ((( - v + +D(
(3)
L-block update Note that the L-optimization subproblem can be reformulated as minimizing /22 (W- ----S S((+1) - [) subject to the rank constraint. When H is full-rank (satisfied for any
>> 0), this problem has the closed-form solution (see Section5]for a discussion about rank-reduced regression results):
L (1+1) = H- H ( (S
(4)
where P. denotes the best rank-r approximation, which can be computed via SVD
D-block update The optimization over D involves projecting S(++1) + V(0 /p onto the sparsity constraint set S, which corresponds to magnitude-based pruning of (S(++1 + V(0 sort [(S(+1) /(0)p) and retain only the largest values. For unstructured pruning, keep predetermined fraction of the largest values; for N:M structured sparsity, we retain N largest values of every M consecutive weights.
In practice, we employ an iteration penalty parameter Pt. giving the following updates:
= (H p. p.t) )(++1)
(H(W- - ()) (( + (() L (x+1) -H - - 1/2 /2 g)++
+ v(0)p.))
((+1)_ p. (1-1).
(5)
Computational complexity We implement several tricks to reduce the computational cost in the and L-update steps, which constitute the major computational cost of 3-Block ADMM algorithm. For the S-update step, we adopt the approach of Meng et al. [2024a] by pre-computing (once) and storing the eigenvalue decomposition H UΣU This allows us to efficiently calculate the ma- trix inverse (H + pl) 1 U pI) -UT for varying values of p across iterations. For an efficient L-update step, we store the matrices H H-1/2 1E2 and H E and em- ploy a randomized-Si
procedure [Halko et al. 2011] for numerical efficiency. In the context of LLMs, the weight matrices scale with the transformer's hidden dimension N. Our algorithm's 's per- iteration time complexity comprises: five -mtrix multiplications with complexity O(N*), Randomized-SVD operation with complexity O(N2r) to enforce rank constraint (using constant oversampling and power iterations as in (Halko et al. 201T)), and a projection onto S requiring at most ( (N)) for sorting and thresholding operations across the entire matrix for unstruc- tured sparsity or within blocks for semi remi-structured sparsity. The overall time complexity is O(N3).
"The closed-fore solution in Equation can be used to improve other (S LR) methods like HASSLE- free (which employs gradient-descent reparameterized L TVT)

---

2.3 Convergence of ADMM
Despite its appeal and usage, the convergence properties of 3-Block ADMM remain theoretically challenging. Chen et al. [20T6] demonstrated that without additional conditions, the algorithm may fail to converge, while later works [Lin et al. 2015 Wang et al. 2018) established various sufficient conditions for convergence.
We observe that our proposed 3-block ADMM approach can be reformulated as a standard 2-block ADMM by treating (L.D) as single variable block. This reformulation is valid because the La- grangian is separable with respect to L and D. meaning their joint minimization yields equivalent updates to sequential optimization (although 3-blocks remain the "natural" way to conceptualize the updates). [Mng et al. [2024a] established convergence guarantees for ADMM applied to lay- erwise pruning, their analysis addresses different problem formulation than ours. Specifically, they apply ADMM solely to unstructured pruning, whereas our approach extends to (S LR) decom- position. Our framework includes low-rank component with relatively complex updates in each iteration, which introduces additional mathematical challenges in convergence analysis that prevent direct application of the results in Meng et al.) [024a].
To address this gap, we establish the following novel convergence guarantee that ensures the de- composition converges as long as we choose penalty parameter Pt that increases sufficiently rapidly (refer to Appendix A for a complete proof).
Theorem 1. Ler { J==o and [ be the sequence generated according to update rule (5)
Suppose the penalty parameter Pt chosen at iteration is non-decreasing and satisfies Et=o 1/ < D0. Then for any 2
max [||S - -S(0) " (++1+) - - [10}
C/P--1,
(6)
where C is a constant depending on X, W. X, Pa. and Ei 1/pv- In particular, there exists matrix W such that (tt) + L() W t oc.
3 Transformer-level matching
After layer-wise pruning, LoRA can directly refine the low-rank components in the (S+ LR) de- composition for task adaptation. However, the sparse components not well-optimized by this process, as they are determined solely via layer-wise objectives. These layer-wise objectives are im- perfect proxies for the true end-to-end loss function. On the other hand, fully optimizing the sparse components using the true end-to-end loss is computationally expensive and requires a full back- propagation on the entire network. To address this limitation, introduce efficient transformer- matching refinement step that leverages transformer-level information to enhance the sparse compo- nents. This procedure is efficient because it requires comparable CUDA memory and runtime the compression algorithms themselves.
Our ceaeffommermmttchin procedure jointly optimizes all sparse and low-rank components across layers within a transformer block to better match the original transformer's output. It acts as an intermediate loss function between layer-wise proxies and the true end-to-end loss. This approach can enhance any (S LR) decomposition, including pruning (where LR = 0). Figure 2 illustrates the performance gains obtained after applying TK to state-of-the-art one-shot (S + LR decompo-
sition algorithms. In Table 3] we show results of applying transformer-e. saanfformrr-mttchinn to pruning algorithms with pure sparsity constraints like WandA [Sun et al. 2024). SparseGPT FFrnntar and
Alistarh 2023). and ALPS [Meng et al. 2024a highlighted in dark red.
Formally, for each transformer block T. with L layers, after obtaining sparse and low-rank compo- nents {S (t.(.)) L (1.00 (1 M6=1 through layer-wise pruning, we denote the support of sparse components as S(+,)) Supp(S (S (t,)). ). Let X; represent the outputs from the previously compressed transformer
block T:-1. We then refine these components using a transformer-leve reconstruction loss:
min L.
I: X (W(1)) )E=) T: X (S
(0.0 (=00)
s.t. Supp(S((.)) c S(1,)) ra (..0) (1.0)
where this constraint optimizes the weights of the decomposed components. Due to the non-linear activations between layers, we use gradient-based optimization methods such as Adam. Nonethe- less, this optimization remains computationally efficient as it is performed using iteratively chunks
5

---

of the small calibration dataset used for compression. Additionally, the forward/backward passes are limited to only one transformer block. The transformer transformer-matching approach offers two key ad- vantages. First, it creates a more accurate proxy of the original loss function by directly minimiz- ing the discrepancy between the original and compressed transformer outputs, resulting in higher- performance pruned models. Second, it reduces accumulated inrors-introduccdd in layer-wise prun- ing where input activations X are computed from outputs of previously pruned layers- by ensuring that activations fed into subsequent layers more faithfully match those of the dense model:
T: x (S + )( X1+1
x = T; X;; (W).)))))
(8)
therefore providing better activation statistics for compression on subsequent transformers compared layer-wise reconstruction which only matches weight matrices layer by layer.
After transformer-ne mathhiinnn,,, the refined sparse components S(1.0) remain fixed during downstream fine-tuning, while the low-rank components ((,)) serve as smart initializations for efficient LoRA adaptation to specific tasks.
Transformer-Matching Universality
Meta-L ama-3-8B 2:4 Sparse 64 Low-Rank
Uama3.2-38 4:8 Sparse 64 Low-Rank
WT2 Perplexity
12.5
13
12
12
A109 80GB
>2x
11.5
1.40 48GB
-1
11.0
10
10.5
OATS
Hf SparseGPT 6.65 h 10.44
Hf ALPS 15.71 h
38ASIL 4.08
OATS
Hff SparseGPT 2.04 h 2.87 h
Hf ALPS 7.73 h
3BASiL
Runtime: 8 3.60 h
2,88 h
1.06 h
RunfimerTM: 12.38
19.67
7.77 h
3,80
8.62 h
1.99
Figure 2: Our transformer matching (TM) procedure improves any one-shot (S LR) decomposition method (sce baselines in Section 4 with small computational overhead. Circled markers represent standard (S + (R) methods, while filled markers indicate their TX-enhanced versions. Black arrows illustrate performance gains due to TM. The compression runtimes are reported in hours. Llama3-8B models were run on a A100 GPU, while Llama3.2-33 were run on a L40 GPU. Our proposal 3BASiL-TM, remains significantly faster: (left) over 2x speedup on an A100 80GB for the Alama3-8B model decomposed to (2:4+64LR) configuration, and (right) over 3x speedup on an L40 48GB for the Llama3.2-3B model decomposed to (4:8+64LR) configuration (both compared to Hf-ALPS).
4
Experimental results
4.1 Experimental setup
Models and LLM Evaluation Protocol To rigorously assess the effectiveness of our proposed ap- proach 3 and fransformer-mutchinng (TM) procedure, we conducted extensive experiments the Llama-3 and Llama-3,2 model families [Dubey et al. [2024] and scaled results in one experiment OPT-30B (Zhang al. 20022 model, hence covering architectures with number of parameters ranging from IB to 30B. Following the widely adopted setup introduced by Frantar and Alistarh 2023) for one-shot compression, we select the calibration set consisting of 128 randomly sampled text segments (2048 tokens each) from the C4 [Raffel et al. 2020 train dataset's s first shard. This calibration set shared across all evaluated compression methods ensure consistency.
We adopt two evaluation criteria: (1) perplexity as a foundational measure of language model- ing quality, and (2) zero-shot lask performance assess practical downstream capabilities post- compression. Perplexity is measured using three standard benchmarks: WikiText2 [Merity et al.) 2017). Penn Treebank [Marcus al. 1994]. and C4 validation samples, computed using Hugging- Face's s full-stride perplexity protocol Per 2022]. For zero-shot evaluation, we utilize the LM Har- ness framework [Gao al.] on diverse suite of eight zero-shot tasks: PIQA (Bisk et al. 2020], ARC- ERC-EasyyChallengg [Clark et al. 2018), HellaSwag [Zellers et al. 2019)] Winogrande [Sakaguchi
al.
2021], RTE [Poliak] 2020]. OpenbookQA [Banerjee et al., 2019]. and BoolQ [Clark et al.
2019
We report individual scores for each benchmark and the average across all tasks.
For perplexity, (4) lower values are preferred. For zero-shot tasks, (f) higher values are preferred.
Baselines
Our main baselines are OATS (Zhang and Papyan| 2025), IASSL.--rree-ppareGPP (Hff-SparsGGPT) and HASSL.E-free-AP.. (Hf-ALPS) the latter two use pruning approaches

---

SparseGPT Frantar and Alistarh 2023] and ALPS [Meng et al. 2024a respectively, in the sparsi- fication step of the alternating minimization algorithm proposed by Makni et al. [202
For all these baselines, we follow the original configuration and perform 80 steps of alternating minimization. For HASSLE-free methods, propose an improved implementation that replaces their original parameterization of L UV and gradient-based optimization with the closed-form solution provided in Equation 4 This modification leads to improved compression runtime and better downstream LLM evaluation metrics-see Table 6 Under this improved implementation, the method EoRA [Luu et al. [024], which applies the update in Equation (4) once after one round of compression, reduces to HASSLE-free (alternating minimization approach) with a number of itera- tions equal to one. EoRA is the fastest (S + LR) method but underperforms HASSLE-free which uses more alternating minimization steps (default=80), and hence there is a large gap compared to our approach on most model/confiiuration settings. We show some results of EoRA in Table
More details the implementation of 3BASiL, TM and the baselines (with improved implementa- tion) are provided in Appendix B
4.2 Numerical results
Our evaluation focuses primarily on (N:M + LR) decompositions. which enable efficient GPU ac- celeration via specialized CUDA kemnlss (Mozafíari et al.. 2024 Makni et al. 2025 We evaluate both one-shot compression performance and downstream LoRA fine-tuning capabilities. Addition- ally, demonstrate the generality of our approach through experiments with unstructured sparsity and integration with sparsity allocation methods. The downstream LoRA A experiments have been motivated by recent studies [Li et al.. 2024 Guo al.
2024 Saha al., 2024 suggesting that de- compositions of the form C C(W) (W++ LR are LoRA-aware: i.e. Tow-rank components obtained from compression can act as smart initialization to improve downstream LoRA fine-tuning. Further nu- merical experiments where we ablate on TM and LoRA fine-tuning for (S + +R) methods can be
found in Appendix C
One-shot (Sparse + LR) results We compare 3BASiL to prior (S + LR) decomposition methods in the one-shot compression setting-i.e. without fine-tuning. Table I results for LLaMA3.2 family under various (N:M + 64LR) configurations. Table and Figure show results for similar configurations for the LLaMA3-SB model. 3BASiL reduces perplexity by up 8% compared to previous SOTA (due to better layer-wise reconstruction-see Figure 5 and Figure 5b), with the TM step yielding further dramatic improvements of up 40% perplexity reduction.
We also compare (S + LR) decompositions with semi-structured pure pruming methods under a fixed compression ratio p 50%. Results in Table 3 show that 3BASiL-TM achieves the best wewprrsssioo-eerfomraeee
trade-off under (3:8 + LR) configurations among different (S + LR) methods. Additionally, we expand our (S LR) experiments to include (2:4 + 112) configuration for OPT-30B model (Zhang et al.] 2022). This configuration uses a 1.56% Low-Rank Adapter (hidden size 7168). Under this configuration, Mozaffari et al. 2024 report 1.53x speedup well as a 0.63x memory reduction compared to dense model. Results are reported in Table
For unstructured sparsity configurations, we benchmark 3BASiL against prior (S LR) methods "less aggressive" (50%+ + 128) compression for both Llama3.2- 1B and Llama3-8B models. Table 5 shows that our proposed method maintains its advantage even in this near-lossless configuration regime. We further evaluate 3BASiL under high sparsity ratios with (Unstructured 64) configu- rations and demonstrate how our method integrates with the sparsity allocation method OWL Yin
et al. 2024 for the Llama3-88 -8B model see
see Table 13 in Appendix B
These results highlight the effectiveness and flexibility of our method 3BASiL.
LoRA fine-tuning after one-shot compression After applying (S + LR) decomposition, the resulting low-rank components can serve as initialization for LoRA fine-tuning on downstream tasks recover lost performance. We conducted limited LoRA fine-tuning on 10% of the first C4 training dataset shard (approximately 15 million tokens), with detailed hyperparameters in Ap- pendix B Figure 4a demonstrates that LFT-3BASiL-TT TM significantly reduces the C4 perplexity of (S+LR) dccompositions, particularly under aggressive compression regimes like 2:8+64LR. More- over, while LoRA fine-tuning can recover large portion of the performance lost due compression, an advanced one-shot decomposition approach retains its advantage post fine-tuning. For instance, AFT-3BASiL-TV still outperforms competing decomposition methods after LoRA fine-tuning of 2:8+64LR configurations, achieving approximately 8% lower perplexity.
7

---

Table 3: One-shot (N:M Sparse LR) decomposition performance Figure 3: One-shot C4 perplexity analy- sis of Llama3-8B under different (N:M + 64LR) configurations.
Table 1: Perplexity of Llama-3.2 family
Method
Contig
Llama-33-11
L.ama-32--0
W12
PTB
C4
WT2
PTB 1
DATS
640.35
605.3 20
$31.47
494.31
674.71
162.45
134.21
106.07
106.17
151.92
Meta-Llama-3-88 C4 Before LoRA FT
HZ ALPE
2:8+5LRR
107.
94.71
124.17
69.96
65.34
103.68
3BASSL
97.50
85.59
100:35
73.00
72.26
110.10
3BAS1L-TH
$5.24
69.74
09.88
45.35
42.38
68:39
80
H1-SparreePP
HHALPS
H2 SparredP?
DATS
125.91
43.50
92.13
34.18
115.55 155.00
51.16
65.08
34.65
26.50
47.27
39.76
C4 Perplexity
70
3BASLL
SBASL-TM
II A1PS
31A511
3:8+6ILR
37.80
26.96
29.00
43.50
26.
27.94
22.77
30:56
31:77
34.55 5.
60
02
38A511 TN
26.26
20.75
52,09
20.89
17.18
25.31
50
Hf-5pa TT
38.06
19.69
15.90
27.35
17.09
19.25
12.30
21.67
19.19
40
H2 ALPS
3BASSL
4:8-5LL
20.71
20.04
14.90
14:26
24.25
24.75
16.04
15.65
11.51
10.97
18:17
17:39
30
MBAS1L-TH
18.66
13.19
22.46
14.89
10.29
16.52
DATS
41.80
28:45
45.35
25.18
17:41
28.60
20
Ht Spa-paerPPP Itf
27.25
19:45
52.63
20. 38
15,00
23.23
2-4+64ER
23.00
17.66
28.00 96
18:45
13.79
20.50
2.8+64LA
3:8464LR 2:4R
Configuration
4:8+64LR
38AS1L 38AS11 TM
23.16
20.46
17:27
15.23
24.60
27.77
17.89
16.37
13:12
11.79
20.10
18.34
Dense
14.01
9.75
17.99
11:33
7.81
13.53
Table 2: One-shot (N:M Sparse + LR) decomposition performance for Meta-Llama-3--
Method
Config
C4.
WI2
PIB 1 PIQA
115
ARC-E
ARC-C11
WG1
RIE
OQA1 1
BoolQ 1
Avg 1
DATS
58.88
40.76
67.35
03.71
29,45
42.68
24.32
53.91
52.71
25.40
63.95
46.15
20.32
21.46
12.06
68.66
$1.99
50.97
50.38
63.85
53.07
$2.00
71.31
52.78
Hassle ALPS
3:8+6LR
23.93
18:20
26.31
70.82
56.54
54.42
30.12
64.72
55.23
32.80
71.96
$4.55
SBAS1L
23.077
18.00
24.84
71.06
32.59
54.51
33.00
66.70
54.90
38A91L-7H
18.11
74,05
34.73
54.51
34,80
76.91
57.94
17.23
60.29
38.20
61.84
Mapele- free- SparsoOPT
14.65
15.21
72.99
71.27
56.32
79.39
1ree
4:864LR
14.94
76.82
72.77
56.68
78.13
384811,
13:74
9.21
14.24
76.88
72.14
01
30.89
30AS1L-WM I
13.02
8.64
13:70
78.24
72.59
73.11
47.35
71,98
63.18
42.40
$0.49
66.17
DA78
21.59
14.76
23.41
72.74
60.70
60.86
$4.81
65.51
57.76
35.20
68.32
56.99
Hossle-froo-Bpe tw----pappp
17.77
12.38
18.71
74.81
63.04
66.16
33.57
70.19
54.87
35.40
77.71
60.71
Haxx frr---------a 11.8
2:44641R R
16.15
11.38
16.71
75.
67.10
64.44
38.91
69:53
59.93
30.40
78. 38
61.61
3BAS1L
15.76
11:23
16.25
76.50
67.51
67.21
40.10
70.24
64.26
38.20
78, 78,29
62:80
38AS1L-7M
14.34
9.78
14,88
77.48
69.55
67.21
40.53
71.27
61.37
39.80
79.51
63.34
Meta-1 1att--Lmm----888 Dense
9:44
6.14
11.18
30.79
79.17
77.69
53.33
72.85
69 68
45.00
$1:44
69.99
Method
Config
C4
WT2
PTB.
PIQA
ARC-E.
ARC-C1
Wanda
38.21
26.89
47.13
67.63
49.37
29.01
Wanda-TK TM
15.91
11.03
17.50
75.14
63.97
40.19
SparseGPT
22.65
16.22
25.15
71.16
56.48
32.59
of Llama3-8B model. The com- pression ratio (percentage of non-
SP..SGGP.TM
15.30
10.83
16:77
ALPS
19.62
14.50
21.73
73.78
00.06
35.84
ALPS-TM
14.96
10.64
16.35
26.38
65.03
39.85
zero parameters retained) is fixed to be P 0.5. For Perplex- ity, (1) lower values are preferred.
Wandu
22.70
15.558
26.62
72.03
58.63
36.09
Wanda-TM
13.99
15.09
77.48
42.75
SpaneGPTT
SpanseGPT-TM
4:8
17.59
13.68
12.29
9.28
14.51
18.4%
78.02
75.68
63.38
70.29
43.94
39.59
For zero-shot tasks, (t) higher val-
ALPS
16.06
11.17
16.50
76.12
66.25
40.57
ALPS-TM
13.59
14.18
69.57
43.94
ues are preferred. Bolded val- ues correspond to the overall best compression scheme that satisfies P 0.5. Underlined values cor- respond to the best pure pruning algorithm for the same compres- sion. This shows the universal- to
OATS
21.08
14.54
24.15
73.67
59
37.12 12.12
HF- SpurseeeP
20,05
15.03
22.01
74.05
60.52
36.18 18
HE-ALPS
2.
17.89
13:00
19.11
74.54
65.53
39.08
TRASIE
15.20
10.64
15.80
76.71
70:08
43.52
3BASIL-TM
13.81
9.50
14.74
77.15
73.36
44.54
OATS
16,87
11.43
18.53
75.24
65.91
39.85
HI-SSpuneGPPT
16.16
HE-ALPP
3/44ER
14.85
10.20
15.42
77.15
69:40
3BASIL.
13:75
9.29
14.62
78.45
71.42
43.4%
3BASiL-TM
13.01
8.69
13.74
77,80
75.00
47.44
ity of transformer-matchee
Llama3-88 Douse
11.18
77.69
pure sparsity constraints.
Table 4: One-shot (2:4 + 112) decomposition of OPT-30B model. This configuration results in ef- ficient inference. We limit the compression run- time to 6 A GPU hours. 3BASiL-TM largely exceeds this period. We limit the alternating min- imization steps of Hf ALPS and DATS to 10 to fit within the time constraint.
Method
C4
WT2
PTB
Time (hrs)
DATS-10 10
11.75
10.48
14.65
5.81
HE SparreGPTT
11.58
10.17
14.39
5.97
10
11.56
10.05
14.33
4.33
3BASiL
11.53
10.04
14,26
4.20
Dense
11.44
9.56
14.04
8

---

Table 5: One-shot (50% + 128) decomposition for Llama3.22 B.BBBB and
models.
Method
Config
WT2.
PPQA
HS
ARC-E.
ARC-C1
WG
0QA1
Avg
17:99
21.40
71.71
57.25
63:91
53.80
RE-SpaneePPT
17:25
20.87
72.91
59.04
56.82
$3.11
57.76
57.03
HF-ALPS
528++128
16.81
11:66
20:
72. 80
59.92
57.62
33.11
58.64
53.96
35.30
59.06
54.11
SBASIL
16.17
11:16
20.00
73.83
09:42
58,04
34.47
60.28
53.79
36.50
58.20
54.39
3BASII-TM
15.78
10.87
19.33
73.25
60.66
59.26
34.56
61.00
59.21
36.00
64.13
56.08
LIama-3 3-2-1 Dense
14.01
9.75
17.59
74.59
63.06
00.48
36.26
60.69
36,08
57.20
63:98
56.69
OATS
12.25
7.78
12.92
78.40
75.32
13.99
69.15
73,50
58,84
41,30
79.42
66.34
HE-SpareGPPT
11.98
12:85
79.11
75.00
75.32
78.32
67.25
HF-ALPS
5216+128
12.09
56
78.75
76.52
81.62
67.29
SBASIL SBASI
11:51
7:47
12.
79.54
74.75
03.72
72.69
67.87
11.27
7.30
12.36
79.65
76.07
75.84
47.78
71.98
70.40
80.70
68.33
Meta-T 3ama-338 Dense
9.44
80.79
79.17
77.69
53.33
72.85
81:44
60.99
Meta-Llamaa
C4 ppl After LoRA FT
C4 Perplexity: Methods Without/ith LoRA Finetuning vs Dense Model
Kittaal Leki Moidd
35
LFT SparseGPP
-e- LFT HP ALPS
LFT-3BASIL
3BASILTM
117
14:61
Perplextry
> 35% reduction
25
25.49
20
15-
Denss
14
14.11
2:0+64LR
3:8+64LR
2:4+64LR
4:8+64LR
GATS
HEALPS
30ASIL
3BASL-IM
Derie
Configuration
Rardirer [M000 8908)
9.44
0.90
6:77.
928
1:30
[Uama 18)
Runtima LoRA FT 113M tolenn on 11100 GB GPU): 0.70n
(a) C4 ppl Llama3-8B model under different (S + LR) configurations after LoRA.
(b) C4 perplexity gap to dense model (Llama3.2-1BB under (50%+128LB contiguration.
Figure 4: C4 perplexity performance of Llama3-8B & Llama3.2 IB before/after LoRA fine-tuning.
5 Related Work
One-shot Sparse/Quantized plus Low-Rank compression The seminal works Yu et al. [017 proposed compression technique for neural network using sparsity plus low-rank constraints. However, the authors study small-scale vision models. In addition, they consider a compression that needs to be repeated over multiple rounds (decomposing selected layers and followed by training process). Our focus is different; we are interested in compressing at LLM-scale in one-shot (no expensive retraining). Recent methods in LLM compression have focused on effectively com- bining low-rank decomposition with quantization or sparsity. EoRA [Liu et al. 2024) has been proposed as a method to compensate for the loss produced by a geneera--uupoose compressed weight C(W) using a low-rank component, it does the low-rank fitting step once post the initial weight compression, which could include combinations of sparsity and quantization. LoftQ [Li et al. 2024) jointly optimizes quantization and LoRA initialization by solving minq.1 1. IIW - (Q+L)) where W represents the original weights, Q the quantized component, and L the low-rank com- ponent. LQ-LoRA [Guo et al., 2024 extends this by incorporating Fisher information weighting, approximately solving minq. F (W- - (Q + L))... CALDERA (Saha et al. (024) further considers the layer-wise reconstruction error, optimizing minq. ||WW - -X(Q + L)[lF
F to main- tain the outputs of individual layers rather than mere weight approximation. From the (S + LR) perspective, OATS [Zhang and Papyan 2025) proposes an outlier-aware alternating minimization, effectively reducing to solving mins DW D + D)llF F with D = diag (X" as noted by
Makni et al. 2025 HASSLE-free [Makni et al. 2025 directly tackles layer-wise reconstruction error ming.L. L JXW- ((S + L) using alternating minimization. While methods such as OATS and HASSLE-free separately optimize sparse and low-rank components, our proposed approach, 3BASiL, distinctly utilizes a unified optimization framework via a 3-block ADMM formulation, jointly optimizing sparse and low-rank components simultaneously.
Sparse plus Low-Rank structures in transformers Beyond model compression, sparse plus low- rank structures have a strong presence in the context of L.MSs. LoRA (Zhang et al. 2024 is a purse sparsification method, which prunes model (iteratively) by designing memory A-gide (low-rank structure) pruning criterion. In contrast, LoSA (low-rank Sparse Adapta-

---

tion) [Huang et al. 2025] jointly applies LoRA fine-tuning and pruning in a unified framework to obtain fine-tuned sparse-only (as opposed to (S + -LR) model. by dynamically sparsifying the LoRA weights and adjusting their rank. SLTrain (Han et al. 2024] addresses (S + LR) from training perspective. pre-trains an L.LM using a fixed random sparse mask plus trainable low-rank factors (similar to LoRA), achieving comparable accuracy to dense training with far fewer parame- ters. SLTrain demonstrates the benefits of (S + LR) structure for pre-training but it doesn't solve the post-hoc decomposition problem of dense model. There are connections between our transformer- matching step and SLTrain as they both train sparse (fixed support) and low-rank components, but they minimize different loss functions and serve different purposes.
ADMM approaches compress networks The Alteraatnn Direction Method of Multipliers (ADMM) [Boyd et al., [011 Davis and Yim 2016) is an effective optimization technique for prob- lems with coupled variables that has been successfully applied to neural network compression. Ye et al. [2018] introduced ADMM-based progressive weight pruning that optimizes the original loss function under sparsity constraints, which Ye et al. (2019) extended to preserve adversarial robust- ness during compression. In contrast, recent methods have scaled ADMM to LLMs through layer- wise reconstruction: [Boža 2024 employed ADMM to solve a convex problem recovering optimal weights on fixed support of the weight matrix, while Meng et al. [024a]] utilized ADMM for non-convex problem that jointly optimizes both support and weights. Our proposed method differs from these prior works as explore 3-block ADMM in model compression that simultaneously optimizes (S + LR) components with theoretical convergence guarantees.
Exact Low-Rank updates for layer-wise compression The problem of exact low-rank updates found in Equation 44) has original roots from classical reduced-rank regression methods [Tzenman] 1975 Reinsel and Velu, 1998]. which provide closed-forn solutions for optimally approximating
linear regression models under rank constraints. Recent work, including CALDERA [Saha et al.) 2024 and the low-rank correction method by |Scetbon and Hensman [2024), applies these closed-
form updates to compress large language models into W R3 Q++ We also use these exact low-rank updates by integrating them directly in Equation (4) within our ADMM framework for
(S + (R) decomposition.
6 Conclusion and limitations
We present 3BASiL as highly-efficient (S + LR) decomposition algorithm with theoretical con- vergence guarantees. provides high-quality solutions to the layer-wise decomposition problem presented in Equation (T) in terms of objective minimization (Figure Sal and Figure (5b) compared to competing (S + LR) decomposition methods. We further refine these decomposed weights with our novel (memory-efficiienl transformer matching step TM that can enhance any (S + LR) decom- position. This shows that one route for optimal compression results (in the context of C(W) + LR) is to unfold the LLM compression into 3 minimization steps: (i) [layer-wise reconstruction] this is the loss that has been considered in many SOTA pruning/quantizatinn
algorithms (Frantar and Alistarh 2023 Meng al.) 2024a Saha al., 2024 Frantar et al.) 2022 Meng al.) 2024b), (ii) [rransormer--natching this is an intermediate loss function (to be optimized in a memory-efficient manner) which is more reliable approximation to the true loss function than simple layer-wise reconstruction, and (iii) [LoRA fine-tuning] plugs the obtained low-rank components as smart ini- tialization for LoRA to minimize the true LLM loss function. We believe that our 3-block ADMM approach and TM can generalize to quantization or quantized-sparse constraints. We leave these ex- plorations for future works. While we have shown how integrate sparsity allocation mechanisms like OWL to our framework, it remains to explore dedicated methods that can algorithmically al- locate different sparsity/rank configurations to different layers to further improve efficiency-uiie -utility- computations tradeoffs.
Acknowledgements
This research is supported in part by grants from the Office of Naval Research (N000142512504, N000142212665) We acknowledge the MIT Engaging cluster for providing HPC resources that have contributed to the research results reported within this paper. Additionally, we thank Google for providing us with Google Cloud Credits We thank Shibal Ibrahim, Ryan Lucas, and Gabriel Afriat for their helpful discussions.
10

---

References
Perplexity of fixed-length models, 2022.
URL
transformers/perplexiii
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.088: 2023.
Pratyay Banerjee, Kuntal Kumar Pal, Arindam Mitra, and Chitta Baral. Careful selection of knowl- edge to solve open book question answering. arXiv preprint arXiv:1907.1075:: 2019.
Kayhan Behdin, Ayan Acharya, Sathiya Keerthi Aman Gupta, and Rahul Mazumder. Quantease: Optimization-based quantization for language models-an efficient and intuitive algorithm. stat, 1050:5, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
Stephen Boyd. Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimiza- tion and statistical learning via the alternating direction method of multipliers. Foundations and Trends@ in Machine learning. 3-1):1-122, 2011.
Vladimír Boža. Fast and optimal weight update for pruned large language models. arXiv preprint arXiv:2401.02938, 2024.
Caihua Chen, Bingsheng He, Yinyu Ye, and Xiaoming Yuan. The direct extension of admm for multi-block convex minimization problems is not necessarily convergent. Mathematical Pro- gramming, 155(1):57-79, 2016.
Christopher Clark, Kenton Lee, Ming-Was Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring surprising difficulty of natural yes/no questions. Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume I (Long and Short Papers), pages 2924-2936, Minneapolis, Min- nesota, June 2019, Association for Computational Linguistics. doi: 10.18653/ 1300. URL ittps://aclaathooloyy org N11//N991100 NNNNNN11100 1300/
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1e03:04455055 2018.
Damek Davis and Wotao Yin. Convergence rate analysis of several splitting schemes. Splitting methods communication, imaging, science, and engineering, pages 115-163, 2016.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Salch Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized repre- sentation for near-lossless Ilm weight compression. arXiv preprint ar Xrxiv22306.33078 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21738 2024.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323-10337. PMLR, 2023,
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint 0rXi::2210..732.2 2022.
I. Gao, Tow, B Abbasi, S Biderman, S Black, A DiPofi, C Foster, I. Golding, Hsu, A Le Noac" et al. framework for few-shot language model evaluation, 12 2023. URL https://zenoo. org//ecoodds/10556866 7.

---

Gemini Team Google. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
Han Guo, Phillip Greengard, Eric Xing, and Yoon Kim. LQ-IoRA: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In The Twelfth International Conference on Leurning Representations, 2024. URL
Xw29VvOMmU
Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53 (2):217-288, 2011.
Andi Han, Jiaxiang Li, Wei Huang, Mingyi Hong, Akiko Takeda, Pratik Jawanpuria, and Bamdev Mishra. SLTrain: sparse plus low rank approach for parameter and memory efficient pretraining. In The Thirty-eight Annual Conference on Neural Information Processing Systems, 2024. URL
https://openreviees wwt.terruurmeee... iid=MXzeHH" opg
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffiman coding, arXiv preprint arXiv:1510.00149 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015b.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark Horowitz, and William Dally. Eie: Efficient inference engine compressed deep neural network. ACM SIGARCH Computer Architecture News, 44(3):243-25 2016.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https::/opeerevieeeee net/ ceet/forum?dd nZeVKeeFYrg
Weizhong Huang, Yuxin Zhang, Xiawu Zheng, Liuyang, Jing Lin, Yiwu Yao, and Rongrong Ji. Dynamic low-rank sparse adaptation for large language models. In The Thirleenth International Conference on Learning Representations, 2025. URL
oXh0939Zzq
Alan Julian Izenman. Reduced-rank regression for the multivariate linear model. Journal of multi- variate analysis, 5(2):248-264, 1975.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Yixiao Li. Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He. Weizhu Chen, and Tuo Zhao. Loftq: LoRA-fine- fine-tuninng-awree quantization for large language models, In The Tweifth In- ternational Conference on Learning Representations, 2024. URL (tRL:htts:://// httttss://pperevieeee .eteee/wwwweeee.e..........ee
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device IIm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024.
Tian Yi Lin, Shi- Shi-Qian Ma. and Shu-Zhong Zhang. On the sublinear convergence rate of multi-block. admm. Journal of the Operations Research Society of China, 3:251-274, 2015.
Shih- Yang Liu, Huck Yang, Chein-Yi Wang, Nai Chit Fung, Hongxu Yin, Charbel Sakr, Saurav Muralidharan, Kwang-Ting Cheng, Jan Kautz, Yu-Chiang Frank Wang, Pavlo Molchanov, and Min-Hung Chen. Eora: Training-free compensation for compressed I1m with eigenspace low-rank approximation. CoRR, abs/2410.21271, 2024. URL https::///// //doi./ddiooeeo org/ 48550/arXiv.. 2410. 21271
12

---

Mehdi Makni, Kayhan Behdin. Zheng Xu, Natalia Ponomareva, and Rahul Mazumder. A unified framework for sparse plus low-rank matrix decomposition for LLMs. In The Second Confer- ence on Parsimony and Learning (Proceedings Track), 2025. BRRl.ptts:///peereeee Forum??ie-hyW7/SAJJ1
net/
Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Fer- guson, Karen Katz, and Britta Schasberger. The penn treebank: Annotating predicate argu- ment structure. Proceedings of the Workshop on Human Language Technology. HITT '94, page 114-119, USA, 1994. Association for Computational Linguistics. ISBN 1558603573, doi: 1ttp://..111111111211111111 1075835
10.3115/1058112.0753333 URL httph:ttp///dt
Xiang Meng, Kayhan Behdin, Haoyue Wang, and Rahul Mazumder. Alps: Improved optimization for highly sparse one-shot pruning for large language models. In Annual Conference on Neural Information Processing Systems, NeurlPS, 2024a.
Xiang Meng, Shibal Ibrahim, Kayhan Behdin, Hussein Hazimeh, Natalia Ponomareva, and Rahul Mazumder. Osscar: One-shot structured pruning in vision and language models with combinato- rial optimization. In Forty-first International Conference on Machine Learning, 2024b.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mix- ture models. In International Conference Learning Representations, 2017. URL https:
//openreview. www.eew..eeffoooe meet/foruuniddByyjaun
Mohammad Mozaffari, Amir Yazdanbakhsh, Zhao Zhang, and Maryam Mehri Dehnavi. Slope: Double-pruned sparse plus lazy low-rank adapter pretraining of llms. arXiv preprint arXiv:2405.16325. 2024.
NeuralMagic. Deepsparse: A cpu runtime for sparse inference of neural networks., 2021. URL https://github.comm com/nuraa http:/tthwb..ooo//eeaaaaaaaaaeea httpe//aaaaaaaeeeeeeeeaeeeaaaeae
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Adam Poliak. A survey on recognizing textual entailment as an nlp evaluation. arXiv preprint arXiv:2010.3306, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020, URL http:
//jmlr /ttp://rr////ppppee////2//000000
Gregory C Reinsel and Raja Velu. Multivariate reduced-rank regression. Springer. 1998.
Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models. Nat., 625(7995):4688444
January 2024. URL https:
//doi.org/ 10. 1038/ /418//41166--0 023 0692446)
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea Goldsmith, and Mert Pilanci. Compress- ing large language models using low rank and low precision decomposition. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: //openreview. net nett/forumiid-lkxxopppsss
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver-
sarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
Meyer Scetbon and James Hensman. Low-rank correction for quantized lmms. arXiv preprint arXiv:2412.07902, 2024.
13

---

Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representattons, 2024. URL wtt.tttts://///// htttps::/openteeieew ntt.:forumm/dddxoouu53333
Fenghui Wang, Wenfei Cao, and Zongben Xu. Convergence of multi-block bregman admm for nonconvex composite problems. Science China Information Sciences, 61:1-12, 2018.
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint wrXiv:2501.09688 2025.
Shaokai Ye, Tianyun Zhang, Kaiqi Zhang, Jiayu Li, Kaidi Xu, Yunfei Yang, Fuxun Yu, Jian Tang, Makan Fardad, Sijia Liu, et al. Progressive weight pruning of deep neural networks using admm. arXiv preprint arXiv:1810.0733
2018.
Shaokai Ye, Kaidi Xu, Sijia Liu, Hao Cheng, Jan-Henrik Lambrechts, Huan Zhang, Aojun Zhou, Kaisheng Ma, Yanzhi Wang, and Xue Lin. Adversarial robustness vs. model compression, or both? In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 111-120, 2019.
Lu Yin, You Wu. Zhenyu Zhang, Cheng- Yu Hsich, Yaqing Wang, Yiling Jia, Mykola Pochenizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning LLMs to high sparsity, 2024. URL httts:
net/forumiid-p0B3 1PxFd
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7370-7379, 2017.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez, editors, Proceedings of the 57th Annual Meeting the Association for Computational Linguistics, pages 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-111 URL httph::t////tt http:////laaatatooy.oooooo org/ 1472/
Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou. Xinyi Yu, and Bohan Zhuang. LoRAPrune: Pruning meets low-rank parameter-efficient fine-tuning, 2024. URL
Stephen Zhang and Vardan Papyan. OATS: Outlier-aware pruning through sparse and low rank decomposition. In The Thirteenth International Conference Learning Representations, 2025.
UURLhtttp:://peerrrrii .et. www.:eeee/foooooe het.://oooromm///---DDVu......
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et Opt: Open pre-trained transformer language models. arXiv preprint 2205v:2005.0106 2022..
2022.
14

---

Appendix
A Proofs of Theorem I
Proof. For conciseness, throughout the proof, we denote X W. We denote Cp as large constant such that
X + AI and G
max (1, H 1/2//112, |2 [|]][>] Cr.
To establish the theorem, first present the following three lemmas.
Lemma A.1. Let { t- { J--n and { {V()) It=o be the sequence generated according to update rule (5 Then for any 2 1, holds
|V(0|E
[[V(--))
I C 1 +
(10)
Pt-1
Pt-1
Lemma A.2. Let {D(1) 1=0 and [V(') ] be the sequence generated according to update rule
Then for any t 1, it
holds
[[V(-D1EE
|V $ (CF C+)
[1+ ||
+
(11)
Pr-1
Pe-1
and
2Cp + 20
|V
IIVit-)
-D -D F
1+ |+
(12)
Pt.
Pt-1
Pt-1
Lemma A.3. Let (D(t) 00 and [v(t) j be the sequence generated according to update rule
1==
Then for any? 1, it holds
|V(--1)|E
D (2)
Pd-
Pt-1
I exp
C(C
Σ
t-
)
| +
|
||
3(CF + C1.)
Pa-1)
Po
Pa
Ps-1
(13)
Returning to the proof of the main theorem, define
=A=2(CC c+)
1 + exp
3(CF + +
Pa-1
(14)
| |
| ||
3(Cr C+.)
[ ||r
Po
Pu
Ps-1
follows from the update rules that CA is constant depending on X. W. Xo po. and 1/pc.
Lemma A.2 together with Lemma A.3
yields
D (+11 - -D(1) < 2CF + 2C
| |E
CA
[1 |
(15)
Pr
Pe-1
Pt-1
Pt
and
|V
|
CA
[V(+1) 1| < (Cp + 204)
1+ |
+
<
(16)
Pt-1
Pe-1
2
then follows from V-update rule and triangle inequality that
[s(++1) SS F < |[(++)] ((++)) + (t- - D + ||(0 -((
F
|V(+1) + ( F + D ((+1) -
|V(0 + | (t-1)) F
Pt
Pt-1
3CA <
Pt-1
(17)
15

---

According to Luuddate rule, we have
L ((+1) L 10
H
1/2
H
( S((+1))) - H - - -W
||H~1/21|
| PHH1 - - -W-(
11 - 1
Cr
(18)
-S
30 CA
Pt-1
Therefore, with constant C = =33C we obtain
C P1-1
max(1|g((+1)) -s()) |F, (++1) - [(() s
(19)
Since Ex 1/ps < oo, both ] and [[ are Cauchy sequences. Therefore, there exist matrices S and L such that ((O) + and L (t) + L as t + co. Setting W + L, we conclude
that (()) + (() + W co.
A.1 Proof of Lemma
A.1
Proof. The L-update rule in (). together with ( yields
L |F
H 1/2 ( W s(
SH 1/1111
P. 12 ( -S F
Cr
H 1/2 11/
- 1//(W-SSt
5(0) F
(20)
CF
1/2
| + CF H//2
(0
S +
where the second inequality follows from the non-expansiveness of rank-r projection operator P, in Frobenius norm. It then follows from the V-update rule in (5) that
5 W F + C
D V(O) v(t-1)
c F + C7
(21)
P
|VO0|e + |V((-I)
t02 1 1
Pe-1
P
16

---

A.2 Proof of Lemma A.2
Proof. According the S-update rule in 5 holds
- + v(()
H + p+l (G HL V + p.D( - ( + V(
Pt
Pt
( PcI) - p: 1) ( HH + pcl) -1 IG HL ( v(0) + V(
Pr.
H
(1) v + V
+
HD
(4)
I+
GG
HL
Pt
Pt)
Pt
-1
H
I
(G
HL
(
HD
(1)
+
I
V
Pt
Pt
Pt
Pt.
H
-1
HV(0)
I+
G
HL
HD
Ps
Pt
Pt
(22)
Therefore, we obtain
s((+1) D
V(O)
1
I I+
H
-1
G
HL
(
HD
())
HV
F
Pt
S
1
G
HL
HD
(0)
+
HV
(23)
Pt
Pt
1
|G
HL
(0) HD
F
Pt
Pt
Denote ī == {(t,j) E [N [Nou]]
D (0)
0}. It follows from the D-update rule and the
definition of the projection operator that
V(9
V( 2
D D(++)) s((+1)_
Σ
(((+1)
min Nin In[Nin|x(NNuu [Z] N.n Nout (i-))Z
Pt
Pt
i
V(t)
2
V(O) 2
(((++))
+
=
Σ
(((+1) -
+
(24)
Pe
Pt 4j
(k,))t
(Wee
V(t)
((++))
D
Pt
Together with 23 we get
D(+1) s((+) V(t)
<
1
|G HL
HDD
|
(25)
F
Pr
Pt.
It then follows from the V-update rule that |V(++)]|
D(++) - g(t+1)
(4)
G (IG-HL HL -HD
< -
Pt
Pt
Pt
(26)
According to Lemma A.1 and the monotonicity of {pr 12 it holds
H
2||2l|V(0) |r
|G HL (0
HD
< |G]|e + H H|| (0)
Ie + HD
Pt
Pe
|V
< CC 1 + | ||
CFIIL(
Pr-1
<Cr c+) (1+ [++
|V
|F
||V(--))|
+
Pt-1
Pi-1
(27)
Together with inequality 26
this establishes the first inequality of the lemma. Furthermore, by summing up 23 and 25 and applying the triangle incquality, verify the second inequality. L
17

---

A.3
Proof of Lemma A.3
Proof. It follows from Lemma A.2 that
|
(1+ [++ID('0 + Cr + CF
||
(28)
Pt
PC-1
PC-1
and
[D(+
[D
+ [[(++1] - |D
D (0)
+ 2Cp 20
(1+ [+||D(11
|
(29)
Pt
P
PC-
Summing up these inequalities yields
|
|V
D + |
|
1I +
Pt
Pt
Pr-1
3Cp + 304
|V
+ |
|
||
1 |++|D(00
+ [|D0
(30)
Pr-1
P-1
Pr-
3CF +30
|DD(0
||
|IV(-
3CF + +30
P111
P1-
Pt-1
Pc-1
Denote at = |D || + || + -|V(1-1) F Pt-11 then the above inequality can be
rewritten
1+ 3C 30 +30%
3Cr + 3C
at
(31)
Pt-1
P
Therefore,
a4+1 II. (1 3(CF + Ct)/p--1)
3(Cp+ +
(1 + 3(Cp + Ct)/
Pr- 0r-1II=-(1++((
3(CF +
at
3(Cr CC)
(1 3(CF + C#)/pa-1)
PL-1
(32)
then follows from telescoping that
3(Cr + Cp.)
a2
S a1 +
(33)
(1 3(CF C4)/
Po-1
Note that
t-1 (1 3(CF + CT)/ exp
1-1
II
3(CF + +
Σ
(34)
Pa-1
==1
recalling definition of a, completes the proof.
18

---

B Additional Experimental Details
Computing environments All experiments were conducted on computing cluster. Unless oth- erwise specified, we utilized an Intel Xeon Gold 6248 machine with 16 CPU cores and a single NVIDIA L40 48GB A10 80GB / H100 80GB GPU. When runtime compression results are re- ported, all experiments have been run on the same node (including GPU) configuration. All language models and pruning methods were implemented using the PyTorch library Paszke al. 2017].
Implementation Details of 3BASiL We H' H 0.005diag X X) 0.005 Tr(X X)I.
In practice, we employ an iteration-dependent penalty parameter Pt: giving the following updates at iteration t:
( v(O + pi) ((+) = (((+))) + v(t)
D
v(+))= + (. -D (D(1)))
(35)
The initial = 0.1. The p-update for ADMM depends on the support change similar to what was proposed by Meng et al. [2024a]. The (S + LR) decomposition is more "sensitive" to increasing aggressively compared to pure pruning in the works of Meng et al. [222aa]. We use the following o update rules. We update P every 10 iteration based on step function that depends on the current value of P: and s, := Supp (D()) A Supp (+))=0) ) which represents the number of elements in the symmetric difference between Supp (D()) and Supp (D(+-1)). Specifically, set
1.1p:
if 81 2 0.1k;
P++1
1.05pt
if st 0.005k,
(36)
1.02pt
if st 2 0.5.
is worth noting that the algorithm can converge significantly faster if we set these parameters to the ones proposed by Meng et al. [0224a] (ADMM for pruning) but the solution quality can be slightly compromised.
Implementation Details of transformer matching (TM) Given a transformer T₃ with input tivations X; (obtained from the outputs of the previously compressed transformer block Ti-1, we start by creating copy of Tü termed 7or We then compress T. layers using (S + LR) method. We now replace dense layers with LoRA layers that contain new linear sparse layers and low-rank components A, B. We set all parameters in transformer block T; be trainable and minimize ing Adam the loss [X;, - -T(ori) If. The input activations fed into subsequent transformer blocks are T.(TM) (X4), where T((TM) is the transformer block after (S LR) decomposition and TM refinement steps.
For TM step, we employ the Adam optimizer with PyTorch's default hyperparameters. We use 20 epochs (on the 128 calibration data points selected for compression). The batch size used is 8. The learning rate is -5 using Cosine Annealing Scheduler with Mmmn 4e
Baseline Implementation Details
Below are the implementation specifications for:
OATS: We adopt the official implementation from Zhang and Papyan [2025] (accessible via GitHub) and apply the default hyperparameters and 80 alternating minimization steps.
MASSSLE-free-Spaseee
We adopt the official implementation from Makni et al. [2025] (ac-
cessible via GitHub) and provide an improved implementation that uses the closed-form solution Equation (4) for the low-rank fitting step. We apply the default hyperparameters and 80 alternating minimization steps.
HASSL.E-free-ALPS. We adopt the official implementation from Makni et al. 2025 (accessible via GitHub) and provide an improved implementation that uses the closed-form solution Equa- tion (4) for the low-rank fitting step. We apply the default hyperparameters and 80 alternating minimization steps.
In Table 3 we use the values reported in Makni et al. 2025 For all other reported values, instead of minimizing |X(W- - M)]le s.t. rank(M) S by reparameterizing M = UVT and optimizing
19

---

Perplesity ())
Zero-shot (f)
Model
Algorithm
PIQA
0QA
BoolQ
Avg
Hi SpaneGPP-oriiinnl
18.06
12.66
18.66
71.86
61.77
63.85
37.37
69.22
56.68
36.40
76.12
59.91
Llama3-S0
SpuneGiT-oour
17.77
12.38
18.71
74,81
65.04
66.16
38.57
70.09
54.87
38.40
77.71
60.71
72.25
34.56
PS-oripinal
75.08
77.89
Hf- F.--ALPS-our
16.15
11.38
16.71
15.19
67.10
64.44
38.91
09.53
59
39.40
78.38
61.61
EoRA-ALPS
18.09
13.61
20.55
73.99
62.21
61.07
37.20
68.59
57.40
36,00
74.16
58.83
Table 6: Comparison of (original paper), reproduced results (with improved implementation) and EoRA (reduces to HASSLE-free with alternating minimization steps set to 1) for Llama3-8B under the configuration (2:4 64). Perplexity (lower better), Zero-shot accuracy (higher better).
with gradient-descent on U and V proposed by the authors, tion of HASSLE-free with closed-form solution Equation (4). This results in sigmificant speedup improvements. A slight improvement in LLM evaluation benchmarks is also sometimes observed using the improved implementation. This is expected because gradient-descent on U and V ap- proximately solves the reduced-rank regression problem, whereas the closed-form solution is an optimal solution.
improved implementa-
Table 6 shows an extract of the differences between the implementation of HASSLE-free proposed in Makni et al. [2025] and ours (using closed-form solution for low-rank update). Moreover, the original paper reports compression runtime (of I Llama3-8B under 2:4+64LR configuration) of 20.13 3 hours using a single A100 80GB GPU, whereas we report a compression runtime (for the same setup) of 15.71 hours in Figure 2 (using single A100 80GB GPU) thanks the efficiency of the closed-form solution. It is worth noting that 3BASiL and 3BASiL-TM are still over 7 times and times, respectively, faster than HASSLE-free-ALPS even when using the improved implementation for HASSLE-free.
LoRA Finetuning Details We follow similar LoRA fine-tuning pipeline to the one introduced in Guo et al. [2024]. For LoRA fine-tuning use learning rate of 2e-5 and a batch size of size 64 per step. The block size used 1024 tokens per batch. The effective batch size obtained by using physical batch size of on GPU with 32 gradient accumulation steps before each weight update. Training conducted on 10% of the first shard of the C4 training dataset, which contains over 15 million tokens. We employ the Adam optimizer with PyTorch's 's default hyperparameters. A cosine learning rate scheduler used, with a warm-up ratio of 0.03 and no weight decay applied.
Layer-wise reconstruction error of 3BASiL Tigure Figure Sb show the objective of Equa- tion (1) attained by 3BASiL and other (S + LR) methods for the first transformer block of 2:4+ decomposition of lamma-8BB model for Attention and MLP layers, respectively.
Comparioo Layers
Compartson for MLP Layers
DATS
DATS
648
#"sparnder
500
True Look Dowee boster))
2463
200
15.2
Adtedoon Fri
AlterBen Oulpui
HLP trg)
VL2
Ouayy
CUl Frg
Attencion Layer Typas
MLP Layer Types
(a) True loss for attention layers (lincar scale).
(b) True loss MLP layers (log scale).
Figure 5: Comparison of true loss values introduced in Equation T across different (S + LR) methods. Lower values indicate better optimization quality. 3BASiL consistently outperforms other methods, particularly for attention layers.
20

---

C Additional Experimental Results
We provide additional performance results considered in Section 4 algorithms and their TM-enhanced versions (apply TM an add-on to the decomposition algorithm). In that case, we add the suffix -TM to the algorithm. We mark algorithms with TM in gray. We also study the results of the (S + LR) decomposition after LoRA fine-tuning as described in Appendix B In that case, we add the prefix LFT- to the algorithm.
We compare different (S+ LR)
Example: LFT-OATS-TTE TM denotes the results of (S + LR) decomposition after (i) using OATS to obtain sparse and low-rank components, (ii) refine these decomposed components with TM and (iii) LoRA fine-tunes the model by using the low-rank components from the (S + LR)) decomposition as a smart initialization.
Method
Cenlia
WT2
165
ARC-E1
ARC-CT
WGG
DNTS
640.56
615.3 30
77V30
52.01
27.67
25.00
23.13
$9.16
52.71
25.00
37:33
37.12
DATE-11 DATE-TH
116.29
59.93
120-10
55.55
11:44
20.55
91:38
50.21
26-40
66.3 7M
19:25
Haznle Sparsedrg
162445
170.12
54.10
31.40
22.01
40.11
53.71
26.30
57.25
25:64LK
167.14
67.59
04.71
124
55. 1g
53.03
20.83
52.77
53,07
27.30
49 27
40.16
Bazs AL TX!
59.30
$2.30
71.97
59.51
12.21
34.93
21.39
$2.50
$2.71
25,20
62.20
42.68
SEABAL
97.50
$6.59
100:35
$5.91
30.00
52.37
21.05
53.75
$3.87
24.40
61.74
41.73
3RAC11-TM
58.24
49.74
69.49
38.81
43.05
92.13
$7.15
22.61
51.99
62.51
42.58
DAT2-1N
36.32
27.69
41.96
63.17
35.47
-3.09
23.21
$2.88
$2.71
29.30
62.08
45.05
Hane www.lee-----tee Frne-Sgarredd
03:40
14.18
$1.16
61.86
16.24
41.%%
74
55.75
63:21
75.70
62.18
45.47
Haze
Baze Aee----ee--ee Seap--paaeeePP
M6664LR
30.45
24:18
37.28
65.18
64.04
41.40
+5:41
25.65
25:85
55.96
50.71
$4.51
30400
62.12
62.26
46.56
47.38
$4.81
38.
41.55
64.91
42.54
-5.24
27.47
$3.01
$1.30
62.99
47.67
85A812 TH
26.36
31.75
33.09
66:43
45.47
477477
27.05
$7.77
52.71
33,80
62.20
48.89
DATS
75.05
19:00
11.50
96.70
90.10
24:50
31:40
51.54
SparsetP2
22:34
15,50
70.51
34,20
62.63
Baaee Le--eee------te
4.8:64RR
30:71
19.1
13.85
14.00
34.17
24.55
78:34
69.59
55.12
$5.20
$3.07
55.05
20.76
57.77
16.85
$4.81
$3.70
51.50
$1.60
65 15.15
63.53
$2.28
$1.78
Basolo ALP
19007
13:70
23.00
71.11
54.50
55.30
31
58.64
50.73
31,00
02.42
$2.50
SEACAL
20:04
24:37
70.52
30.73
35:23
34,00
63.06
GATS
41.30
25.45
45.36
63.28
41,59
47.03
26,37
53.51
51.25
25,40
63,19
46.85
DAT5-TN
23:00
17:04
23:09
66:25
41:06
51.01
27.33
55.64
56.31
33.60
62:32
50.20
Basolo free- Sya--8ys
27.25
19:45
33.65
67.63
47.70
-5.96
26:96
55.99
52.73
3040
(2.14
48.67
Manole- Teee-e------pppe Spar--pareedPPP
3:4461
16:41
25.5m
69:10
50.58
Hazzie 1oo-----------111 BRAS12
20.91
15.35
25.15
78.46
51.14
$1.09
28.24
58.33
$7.84
34,60
63.55
51.81
23.16
17:37
77:37
69.00
11.78
77 13
56.72
51.81
31.40
(2
11.33
SEA21L-TH TH
20.46
15.33
24.60
70.18
$2.91
$2,06
30.13
58.96
52.71
33.40
62.50
$1.59
LAama-33 12-11 Dense
1400
0.75
17.59
74.59
63.00
00:48
30.76
50.09
56.08
37.20
03.18
56.00
Table 7: One-shot (N:M Sparse + LR) decomposition performance for Llama-3.2-18 For Perplexity, (1) lower values are better. For zero-shot tasks, (t) higher values are better.
Method
Cenfu
C4_
WT2
PIQA
HSS
ARC-E.1
ARC-C
WGG 1
O0A1
BuolQ
Avy
DATS
331,47
014.31
674.71
$2.50
25.16
23.29
40.57
52.71
25,60
39.00
$7.47
OKTE-IW
100.7
$7.20
120.95
56.64
29.01
36.13
20.65
50.99
52.71
25,00
(2.11
41.00
Hane ie thee- Spare-rrr So
106.07
105.17
151 02
$4.82
20.67
20.02
21.63
$4.28
33.21
26.60
61.00 1..03
40.05
18:64LR
81.50
90:37
38.96
22:10
51.78
26:40
20
44.00
61.26
23.12
SEAE4L
73,00
72:36
110.10
$7.29
32.62
34.01
21.42
$1.14
$2.71
26.80
62.35 12.30
42.27
2RAS1T-TH
65.35
42.58
64:79
61.10
36.90
38.17
22.78
55.12
53,87
36.00
62.66
44.23
UNTS
47:27
61.75
23.30
50.21
Hancle feee- SpareedP7
34.60
26.00
39:76
65.94
46.19
47.77
26.45
53.07
25.00
65102
49.10
Hazzle see----ee--e Seee---pareerr---
23
19.54
27:45
69:70
51.54
$2.78
29.35
$0.22
$5.96
30.60
62.72
$1.52
Mhanle-frwa
22.54
22.77
34.59
69.15
50.18
$3.32
20.03
$1.48
$2.71
33,00
63.55
$1.45
Hone Te- Fre-------t---e A.
36:47
71.00
$7.37
30.30
56.33
33.40
16
53.64
65.41
ONTS
19:25
13:40
21.67
72.47
61.10
35.15
56.34
57,80
34,00
73.24
$7.56
DAT2-1N
15.02
11.00
17.82
24.21
63.46
37.80
56.
64,63
35380
72.26
60.11
Hazzlee Sparaedrr
19.19
13.83
63.01
36.52
38.12
72.00
4:646-1.
16.04
11:51
14.17
34. 14
64.61
61:76
96.95
96.51
$6.51
12.15
Bazule ALP
15.07
10.54
15.84
75.19
65.95
66.34
48.02
67.40
60.65
40:00
74.01
61.25
SRAS41
15.55
10.90
13:19
15.68
65.97
66-26
19-43
57:25
50.91
15.50
52
3RAC1L-TH
14.89
10.19
16.52
35.79
$6.46
67.15
38.33
56.66
30:30
50.70
17.41
25.60
70.59
32.76
53,07
70.40
$4.20
DWTS-TH
18.0
12:85
N6.35
32.42
19:46
11.34
60.22
56.34
94.00
T0.21
Bazzle Lcen- SparsedP2
20.36
15.00
23.13
71.55
59.62
50.53
32.94
63.85
$7.40
33,60
00.04
55.00
Hasole www.- Spa---aaaee---- llaze
2:4+6LL
17.24
12:66
30-10
19/41
13.15
73.76
61.50
60.82
61.65
63.36
34.84
$1.40
$3.00
38.45
37.20
31.30
08.02
72.76
$7.45
it
13.12
20.10
13.34
35.07
16.37
11.79
18.34
13.78
63.38
63.05
38.05
64.25
30.57
36.30
71.1 13
18.05
Llamto Dease
7.81
13.55
77.48
54.51
43.00
73.99
63.08
Table 8: One-shot (N:M Sparse + I.) decomposition performance for Llama-3.2-3B. For Perplexity,
(J) lower values are better. For zero-shot tasks, (f) higher values are better.
21

---

Method
Config
C4
WT2.
PTR.
PIQA
185
ARC-E.1
ARC-C
WG
RTR.
0QA
BeoiQ
DA78
434.55
431.81
590.28
51.63
28.11
27.61
23.72
49.72
52.71
27.03
35.26
37.34
0478 IM
73:42
64.21
100.59
55.43
31.23
32.30
20.22
51.38
52.71
26.03
62.29
41.31
Bassle Iree SparscdFT
88.39
96.61
109.71
54.95
30.77
31.48
20.85
50.75
52.71
26.20
61.59
41.14
Bassle free SparseOPT TK
Rascle free 1LF3
28+64LR
46.25
60.16
56.00
42:75
71 16.66
77.11
61:45
57.94
36.76
34.92
36.74
34.64
23.12
21.67
53.51
54.82
$2.71
$2.71
27.
27.60
56.12
63.15
44.38
42.55
Massle-free free- PMP3-IM
16.30
34:31
50.14
64.91
41.18
40.28
24.40
56.99
52.71
28.20
59.60
46.00
56.99
53.83
72 44
59.25
15.
15:50
21. 598
54.05
52.71
22.
64.62
43.67
3RAS11-TV
36.16
33.51
52.87
63.60
41.47
19.51
24.32
58.41
52.71
26.80
63.67
46.35
DATE
58.88
401.76
67.35
63.71
39.48
42.68
24.32
53.91
52.71
25.40
63.98
46.15
DAT8- IN
22.67
17:37
24.46
71.22
$4.29
54,88
31,48
03:22
54.35
32,80
71.38
54.18
Bossle free SparscOPT
29.32
21.46
32.06
65.66
$1.99
50.97
30.38
63.83
53.00
32,00
71.31
52.78
Bassle free SparscOPT TV
Rancle o free- 1LPS
388664R
23.93
19:97
18.20
15:51
26. 31
22.22
20.62
72.85
59.72
56.54
54.42
58.12
33.96
30.12
64.72
00:87
55.22
61.37
32.80
32.00
71 1.96
74.92
57.20
54.55
fassle-tree- PM
18:38
14:52
20.15
74.48
61.55 56
59.01
53:28
67.48
57:40
38.20
11
57.94
SRAS1.
23.07
18:03
71.06
56.96
52.70
35.00
54.00
SRASHI-TV
14.26
00.47
74.05
61,83
60.73
34.73
65.98
76.91
17.23
70.88
73.61
14:37
76.82
70:15
43.52
70.88
77.89
free SparscOFT
41.81
71.27
56.32
Hassle tree SparsoOFT
71.42
Hansle free-
Mascle-tree-
14.76
23:41
72:74
60.70
34.81
65.51
57.76
10.61
16.31
68.599
36.60
75.69
Hascle free SparscOFT
17:37
12:38
18.71
74.81
38,57
70.09
54.87
38.40
77.71
Hassle free-
Hassle free SparscOFT
24464LR
16.95
16.15
10:28
11:38
15:97
16.71
78.88
75.19
68.18
67.10
41.81
38.91
09.66
69.53
64.96
59.93
38.20
39
78.11
78.36
63.19
61.11
Mhssle-free-- RLPS-TM SIAAH1
14:45
10.00
15:25
77.09
69.3 50
67.36
40.10
00.17
60.85
38.80
75.75
62.39
15.76
11:33
16.75
76. 50
62.31
40. 10.10
70.24
64.76
38.20
78:
60.50
14.34
9.78
14.58
77.48
69.58
67.31
40.53
71.27
61.3 37
39.80
79.51
63.34
Metsol Iuma--38lt Deroe
6.14
11:18
$0.79
09.17
77.60
53.33
72.85
69.58
45.00
69.09
Table 9: One-shot (N:M Sparse + LR) decomposition performance for -ama--Jama---8B- For Per-
plexity, (4) lower values better. For zero-shot tasks, (f) higher values are better.
Method
Confiz
1
PTB
ARC-E
ARC-CT
RTE1
BoolQ
LPT-DATE
71.41
63.58
95.05
56.58
29.22
30.72
22.10
53.83
52.71
23.00
61.10
41.16
1PU-DATB-11
53.01
47.58
08.98
59.09
31.07
36,15
21.84
$2.49
52.71
24.00
62.14
42.51
$1.19
57.00
63.87
90:09
$1.16
35.14
32.18
$0.83
$1.71
26.00
60
47.20
Lit- -tw.eeeeee-rre--.lig
28864LR
44.15
42:85
41.00
50:46
$5.53
57.45
68.55
90.36
33.22
53:37
35.23
34.68
22.18
22:18
$1.03
50:36
53.71
52.71
26.60
28.00
62.14
60.95
43.07
42.70
LFT Hasste free- ALPB
37.54
35.50
50
00:28
35.80
37.50
23.63
50.00
50.71
27:30
62.14
64.29
LFT-1BAB1C
38.09
33.42
45.13
69.58
35.52
37.71
25.21
53.12
53,07
27.00
62.02
64.00
36.03
$4.13
600077
36.16
$1.20
24.66
$8.41
29.40
44.90
39:71
LITT-ONTS-TM
Hawsle-aaaaae -parusoPT
27.17
65.72
44.09
47.71
Hassle SparosoPT
Saoolo ALPE
3:8664L
24.74
25.30
20.00
65.01
67.03
45.71
46,19
LIT- www.wll-fwwww.......
23:42
15.51
30.33
68.23
47.55
46.50
27.22
$5.25
$4.51
$1.30
62.25
40.14
LPA-SBABLL
22.97
18.33
29.74
67.36
48.
48.36
29.52
$5.72
$4.67
51.50
62.72
49.83
LFT-3BABI
22.73
18:29
29 94
68.01
49.32
45.86
29.35
$6.83
53.71
33.30
63.21
56.17
LPP--NTS
28.06
10.50
32.00
67. 67.30
40.07
45.43
27.65
56.20
$3.71
30.20
62.45
40.26
LPP--NTB-TN
18.95
13:77
23.63
70.73
54.25
56.36
32:51
58.50
54.87
33.60
62.72
$2.98
LFT- Ha----nne---eere weltee-fre--pppee Sparr--roo
19.58
14.15
24.17
71.11
51.18
51.54
30,80
56.15
59.06
$3.30
6061
51.23
LFT. Hassle- Exee-
4.8461RR
15.77
18:05
15:35
15:55
56.00
32.94
56:43
51.00
56.00
51.08
LFT.
ALP8-10
15:11
22.77
71.93
56.97
3EAB1C
17:98
73.52
LFT DAT9
23:55
17.52
67.08
48,04
39.25
27.90
31,00
09.59
LIMT-NTS-TM
21.00
15:40
35.77
69:50
50.02
$2.15
29.01
$5.06
55.23
53.00
62.63
$1.00
L-T- Ho--HHenllleteee Spar-ppa
21.56
15.59
26.81
69.53
$1.03
45.65
26.67
$6.04
$1.35
31.40
62.14
40.38
LST- Ra---Bnne---- Tonste--ren A
LPT- CTT-Haalle-dae free Spars--p
2:4-64LR
20.35
20.16
15.16
15.57
25.23
25.0 4)
71.11
71.16
$3.40
52.59
51.41
52.86
29.69
29.69
$7.23
56.99
56,68
54.97
35.00
$3.00
62.17
62.84
$1.89
$2.27
LIT- www.-ee----eeeeoo www---r------------ MLP--TM
19:47
14.51
24 91
71.
$4001
$1.16
35.9
S8.01
$4.15
$4.30
6041
$1.25
19:25
14.00
24.5 56
71.76
54.73
$3.16
2901
57.77
51.49
$4.00
63.15
$2.12
LFT SEABSL TV
19007
14.37
23.88
71.44
55.12
51.04
29.95
58.17
53.43
33.50
62.69
$2.90
2.-18 Donse
14:01
9.75
17.90
74.59
63.66
60.63
36.26
50.89
56.08
$7.30
63.95
$6.00
Table 10: (N:M Sparse- + LR) decomposition performance for Llama-3.2-1B after LoRa Fine-Tuning
(LFT). For Perplexity, (4) lower values are better. For zero-shot tasks, (t) higher values are better.
22

---

Mathed
Cumliz
C41
W12.
PEB
PIQA
188
ARC-F11
ARC-C1
WG1
KTF.1
(QA
BuslQ
Ang
LFT-OATS
48.35
64.51
62.21
58.27
12.84
16.95
22:16
53.90
52.21
25.60
51:46
42.54
LFT-OAT3-TX
100.7
87.20
120.98
56,64
29:01
50.13
20.65
50:99
52.71
26.00
52.11
43.03
LFT San-le--eee -eoee-p
35.91
33.44
46.64
61.48
37:33
36.70
23.46
51.34
$2.25
26.20
52.17
43,90
13T--Naweee- wrr.--paraaaa-t-- 28664LR
90:77
40:19
18:76
54.78
58.12
31.45
63.38
735
13. 46
55.72
46.06
Saosle
28:22
26.81
55.72
991-9845LL-SN
27.48
26.18
35.35
LFT OAT5
21.98
16.44
27.21
69
52.26
52.75
29. 66
50.32
31.00
LIT--OTSSITH
27.09
20.94
30.21
67.68
47.36
51.26
28:41
$7.46
52.71
29.20
64.65
49.63
LFT TT1-Banle--ree weeelleddeeeeee -pa-aaaa---
20.01
15.72
24.87
70.46
56.25
$4.59
30.72
60:22
58.12
33.20
63.64
53.40
EST- BoT--aaste free www.---aaaaaaa---
3:5+64LR
18.85
18775
15:18
15.54
25.25
23.94
71:00
71.00
58.27
58.49
56.21
52.15
12.51
$2.51
61:72
63.22
59:31
$4.87
15.80
15-40
55.00
56.64
53.19
$4.76
LET. Sassle free ALP8
18.04
14,08
23.05
72.91
00:26
60.25
34.23
61.09
62.88
36.00
58,50
55.89
LET 3BASLL
18.40
14.51
22.
72.30
19.70
57.85
32.17
02.67
55:23
35.20
68.47
55.43
L9T-3BASSL-SN
17.69
14.38
72.96
57.32
10:77
65.24
1532
$7.50
14.97
16.62
65.87
30.25
38,60
LFT -aassaeeeeete free PS
Tassle messte-fee teeeeee SparselPT-TH
R
14.47
14,00
10:25
10.53
16,M1
16.71
75.19
75.08
64:21
06.31
17.63
19.76
66.34
42.80
18:20
55.76
67.36
55.47
LFT TTT-Basle--er Iree ALPS-TN
14.30
10.11
16.30
75.63
68.15
66.75
41.21
07.58
59.31
40.00
69.33
61.03
LFT 3EABLL
14.38
10.10
15.55
76.77
68.00
67.26
40.70
07.48
59.57
39.60
68.25
60.97
L3T-MMASHL-TT
14.35
9.89
16.29
77.36
68.44
66.20
39.59
66.49
62.52
19:80
72.29
61.64
LFT-OATS
25:18
12.08
20.26
73.99
61.58
62.88
16.19
61:75
16.60
1IPT-ONTS-TH
1978
71:43
59:46
60:17
70.21
LFT. Sansle SporsefFt
19.43
63.51
35.07
Bassle SparsesFT
75.14
62.38
15.51
11.57
36.52
37.80
$5.20
L9T-Bassle-1zee ALP5--1
15:37
11.22
74.70
64.56
LFT- 3BAS1.
15:52
11:25
19.80
75.08
63.05
16,00
66.00
64.96
39.50
08.25
LST-MMASIL-TM
18.19
18.96
17.59
74.46
66.46
63.01
39,88
61.06
61.17
15.
69.37
50.99
L.ama-32-30
11.33
7.81
13.53
77.48
73.61
71.63
45.99
09.85
43.00
73.39
63.68
Table 11: (N:M Sparse + LR) decomposition performance for Llama-3.2-3B after LoRa Fine-Tuning
(LFT). For Perplexity, (J) lower values are better. For zero-shot tasks, (f) higher values are better.
Metlod
Coulit
C4
WT2..
PTB
PIOA
HS'
ARC-ET
ARC-C1 1
WG1
RTE.
00A 1
Bool)) 1
Ave
LFT. DATS
37.06
$1.49
71
62. 14.24
36.38
38.72
24.25
51.90
53.71
27.20
62.02
64.59
LPPI-ONT-TM
28:23
LPt-Baanle-
28.80
24.91
25664LK
24.89
22:14
66.00
48.49
29.50
22:85
$1.20
SSASLL
24.51
30.05
LFT 3BAB12-77 TV
22.45
20,00
29.00
68,12
$2.97
LIY--ATS LFT: DATS TH
17.87
12.65
20.90
72.36
61.30
$7.45
35.58
63.98
$3.00
35.00
70.58
56.15
15.18
11:43
18.34
73.61
65.43
60.65
37.05
56.54
$7.40
36,80
75.72
59.30
LFT Ransle tree- SparrooFT
16.05
12.00
15.54
73.54
64.77
58.00
37.20
16.05
61.01
35.000
69:79
56.90
LPT- H-L--aanle--ae wew.nl----------- 12P8-ALP
LST- Wanetee peee-ree--ppe www.-rree-prrro-------
3-86641.
15.63
15.92
13.77
15.51
17.89
18.05
74.59
75.63
66.56
66.50
67.04
60.33
36.43
$7.54
80.225
68.67
78:40
63.34
37.20
35.20
74.71
76.79
61.42
60.52
LPT- Bwsste- free- ALPS-11 -7
15:29
11:46
17.74
75,90
68.34
61.06
37.03
68.96
62.45
36.20
69:33
59.99
LIMI-WAR1..
15.64
38.65
57.44
LI9--BABLL-TY
15.11
11.38
75.41
58.03
DATS
13:09
14.64
72.31
75.60
76.18
12:38
76.78
74.29
Raoste ALPS
48864LK
12:55
14.45
76.82
71.19
71.04
72.77
Basste-fren free- ALP8-7H
12:31
8:29
13.41
78.54
74.92
73.86
48:29
79.51
07.00
LPT- 3RAB1..
12:99
K:51
13.41
78.51
74.55
21.00
45.56
72.45
43.40
77.91
LPT- SEAB1L- TV
12:17
8.25
13.41
79.33
74,40
73.23
48.73
71.50
61.73
41.60
78.90
66.23
LFT-DNTS
14:34
9:67
16:24
77.15
09:54
60.66
40.70
08.21
66-43
58. 90
73.73
61.54
LPT-ONTB-TM
13:45
9.09
14.81
77.69
71.25
09.99
44.28
59.77
58.84
40:50
78007
63.81
LFT- Bwssle--eaae-oe free- Exee-ppazooPPP
13.83
9.50
15.35
77.04
71.42
68.86
41.72
70.01
61.37
39.40
76.02
63.25
LFT. Rassbe: free @parssOFT TH
2:4464L
13:35
13.49
9:15
14.84
14.61
71:45
79.00
72.57
67.80
67.09
45.60
42.92
70.01
70.56
63.90
50.00
40.40
75.69
64.12
13.20
76:34
72.51
67.38
41.89
70.58
63.67
LP9-SBABLL
13.35
14.54
77.37
43.52
1307
Coetu-LLamm--- 3-88 Dense
9.44
6.14
13.18
80.79
79.17
77.69
53.33
72.85
69.08
45.00
$1.44
69.99
Table 12: (N:M Sparse LR) decomposition performance for Mlta--Lam after LoRa Fine-
Tuning (LFT). For Perplexity, (4) lower values are better. For zero-shot tasks, (1) higher values are better.
23

---

Config
C4
WT2
PIQA
HS
ARC-E1
ARC-CT
WG1
RTK
OQA 1
BoolQ
Avg
3BASIL+OWL
23.21
27.09
71.71
56.69
33.53
54.15
77.86
56.60
3BASIL-TM
70%
19.55
16.44
21.64
74.16
59.76
58.04
33.28
66.38
58.12
35.20
73.15
$7.26
3BASIL 3BASIL-TM-OWT
19.52
16.22
21.37
73.56
59.92
58.88
31,40
64.33
56.32
35.40
70.30
56.33
SBASIL 3BASIL
62.85
61.088
79.49
59.52
35.07
35.40
22.27
$4.22
52.71
27.00
60.95
43.39
30.51
58.16
79.09
61.70
39.42
37.63
23.98
59.35
55.60
28,00
68.23
46.74
3BASIL TM 3BASIL-TM TM-UWL
80%
36.51
19.32
57.94
65.07
42.05
39.94
25.00
58.80
52.71
26.00
64.39
66.77
36.32
47.08
Meta Liama 3- 8B Dense
9.88
6.14
11.18
30.79
79.17
77.69
53.33
72.85
69.88
45.00
$1.44
69.99
Table 13: Impact of OWL on 3BASiL for (Unstructured + 64) decompositions of Meta-I.ama-33
24

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We conclude the introduction with a paragraph that explicitly outlines the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of our work at the end of Section ??.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificatton, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [es]]
25

---

Justification:
We clearly state all assumptions in Theorem and provide a rigorous proof in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We present a detailed description of the proposed 3-Block ADMM algorithm, including update rules and computational procedures, in Section and describe the Trans- former matching procedure in Section Additional implementation details necessary for reproducing our results are provided in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can accomplished various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
also be provided via
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results way to produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
26

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will release the codes if the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips www.cc/ppplcgppeeeeccccooo wwwpc//ggues//eegg httggguddeeeeeuuueeeeeeeee
wwwpuumisssiooooiiiii..o for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide detailed training and evaluation settings for both proposed pipeline and the baseline methods in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question:
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: While we aimed to provide rigorous evaluation, we were constrained by com- putational resources and thus could not include statistical significance measures. We have, however, ensured consistent settings and fair comparisons across all baselines.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative еπτог rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details of the computational resources used for our experiments in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp: ctcpippp.ccpppuieae / http://ppwpppp.ppooocccccccccaaa
Answer: [Ye]]
Justification: We have carefully reviewed the NeurlPS Code of Ethics and confirm that all research presented in this paper adheres to its principles.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
To the best of our knowledge, our work has no societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
28

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
To the best of our knowledge, our work poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with
necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines restrictions access the model or implementing safery filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: At the start of Section 4 reference all datasets and models involved in our experiments. The sources of the code used are listed in Appendix B
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, pattastts.paperswiiiae paperswithcooeee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
30

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Although our work focuses on pruning LLMs, the core methods proposed do not involve LLMs as important, original, or non-standard components of the algorithm itself.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https://neur)) lnttps//nerrigssccc//ffeeeeeee wwtpee/cc/oooonncaeeeeeee 2025/LLM for what should or should be described.
31


Reviews:
[{"review_id": "NmEwOWpiHV", "reviewer": "Reviewer_JUTM", "summary": "The large scale of Large Language Models (LLMs) makes them a big target for compression, and one common technique is sparsity which, when coupled with low-rank decomposition, shows promise at matching the dense model's quality.  Despite many developments in this direction, there still exists a large gap in model quality.  The authors present 3BASiL-TM to help close this gap.  First, 3BASiL is a 3-block ADMM algorithm which alternates between updating the sparse version of the weights, the low-rank component, and the sparse mask.  Treating L and D as a single variable block reformulates the approach as standard 2-block ADMM and helps guarantee convergence.  Next, the authors propose Transformer-level Matching (denoted by a -TM suffix), which serves to further refine the sparse and low-rank components, using the nonlinear behavior of each transformer block, so that the low-rank components are well-initialized for LoRA adaptation.  Empirical studies show that TM generalizes to many one-shot S+LR methods, that its application to 3BASiL delivers top-quality results, and that the benefit of 3BASiL-TM is maintained, even after LoRA fine-tuning.", "questions": "These two questions directly relate to the quality of the submission:\n1. How does TM improve, say, the 2:4 and 4:8 results in Table 3?\n2. How does EoRA interact with 3BASiL as a whole, and specifically with using TM as a smart initialization for LoRA fine-tuning?", "limitations": "The limitations are satisfactorily discussed in Section 6.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths\nThis submission is outstanding.  It presents a novel collection of related ideas, clearly motivating and proving the benefit of each.  Treating each metric in turn:\n### Quality\nI couldn't find any missing links in the chain of reasoning from claim to claim.  Each claim was supported by theory or empirical results, as appropriate, with one exception (noted below).  This is clearly a complete package and not a work in progress.  (Future work is indicated, but well beyond the scope of a single endeavor.)\n### Clarity\nI believe that sufficient detail is presented such that a dedicated practitioner could implement the technique and reproduce results.  Fully understanding each step does take careful reading, but the organization and exposition is not to blame; the mechanisms used are simply nontrivial.\n### Significance\n3BASiL-TM represents a major step forward in recovering model quality.  There are still gaps to the dense baselines, but they are greatly reduced without undue computational load. Care has been taken to not only find compelling new techniques, but to make them practical for use - runtimes in single-digit hours is entirely reasonable that users shouldn't need to shy away from one-time applications to their models due to limited resources.  (Larger models will require more time, but will also require more resources to train and deploy.)  As I discuss immediately below, there are findings within the results that I consider very useful for advancing the field.\n### Originality\nI only noted one missing piece of related work (below); otherwise, the comparison to prior work is complete and shows that the collection of techniques in 3BASiL-TM is both new and important.  \n\nA couple observations show the importance of the submission goes beyond its ability to improve zero-shot model quality:\n- Table 3 is particularly interesting: for a given compression rate, it's important to find the right balance between sparsity and the rank of the LR.\n- Figure 4 is also important: I've observed \"head-starts\" by clever one-shot schemes become washed out after fine-tuning.  It's good to see this doesn't happen to 3BASiL-TM (at least for the limited fine-tuning performed).\n\n## Weaknesses\nI noted one missing comparison - EoRA (1), which seems like it might compete with the \"smart initialization\" of LoRA parameters offered by transformer matching.\n\nI also could not find evidence to support the claim on line 181 that TM can apply to sparse, but not LR, networks.  I can imagine how it will apply, but this is not supported in any of the presented results.\n\nMinor issues:\n- Inconsistent citation styles are confusing - particularly (but not limited to) lines 213-221.  Some citations are missing years (e.g. lines 290, 292).\n- The wrong entry in Table 2's 2:4+64LR RTE results is bolded (or there's a typo in one of the 3BASiL results).\n- Citing the N:M sparse format (and the hardware advancements that make it useful) (e.g., 2) would make the background information complete.\n\n1. Liu et al., EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation, https://arxiv.org/abs/2410.21271\n2. Mishra et al., Accelerating Sparse Deep Neural Networks, https://arxiv.org/abs/2104.08378", "was_revised": true, "final_justification": "The authors' rebuttal entirely satisfactorily answered my questions, and reading other reviews and responses has not changed my initial review's conclusions.", "rebuttal": "We would first like to thank the reviewer for the thorough examination of the paper, highlighting the strengths of our submission and proposing missing pieces that would improve the manuscript.\n\n**Weaknesses -- Questions**\n\n***W1 & Q2:*** Thank you very much for pointing EoRA which should indeed be included in the related works of our paper. Upon inspection of this important method, we noted that HASSLE-free [a considered competing method] with a special configuration (alternating minimization steps=1 and using our improved implementation--see Table 4 discussion) reduces to EoRA. In that case, Hf-SparseGPT-ours would (i) compress the model weights to 2:4 sparsity using the SparseGPT algorithm and (ii) use the closed-form rank-update (equation 4) for \"compensation\" as discussed in the EoRA paper [1]. In EoRA, $Q^\\prime$ corresponds to $H^{1/2}$ in our paper (up to a minor notation difference, as they consider $XX^T$ to be the hessian whereas we use $X^TX$).\n\nWe launched experiments with EoRA, Hf-SparseGPT-ours (default AM steps=80) and 3BASiL for Llama3-8B under a (2:4 + 128) configuration [similar to Table 2 in EoRA table] (we also compare the effect of using TM on top of EoRA to show universality).\n\nWe obtain the following results:\n\nTable 1: Comparison of (S+LR) methods with (2:4 + 128) configuration on Llama3-8B. For perplexity (lower is better, ↓). For accuracy (higher is better, ↑).\n\n| Method              | C4 PPL ↓ | WT2 PPL ↓ | PTB PPL ↓ | ARC-C ↑ | HellaSwag ↑ | WinoGrande ↑ | Avg. ↑ |\n|:--------------------|:--------:|:---------:|:---------:|:-------:|:-----------:|:------------:|:------:|\n| EoRA                | 20.34    | 14.49     | 21.86     | 35.75   | 61.02       | 67.17        | 58.23  |\n| EoRA-TM             | 16.00        | 11.41         | 17.32         | 38.82   | 65.11       | 66.54        | 59.91  |\n| Hf-SparseGPT-ours   | 15.46        |10.50         | 16.05         | 40.61   | 69.27       | 70.72        | 63.30  |\n| 3BASiL              | 14.45    | 9.56     | 14.46     |43.94   | 71.29       | 70.65        | 65.06  |\n| **3BASiL-TM** | **13.30**| **8.82** | **13.92** | **44.88**| **72.28** | **71.59** | **65.40** |\n\nNote that the results of our implementation of EoRA are slightly different than reported in the EoRA paper.\n\n(i) Our EoRA implementation has a slightly worse WikiText ppl than reported in the EoRA paper (11.07) because our calibration data (X matrix) is C4 whereas they consider WikiText2 (training data). Despite this fact, 3BASiL-TM [with calibration C4] largely outperforms EoRA [with calibration WikiText2] on the WikiText2 test perplexity task (from 11.07 to 8.82) under the same constraints.\n\n(ii) Our EoRA implementation has a slightly better ARC-C accuracy than reported in the original EoRA paper. For this metric, we both use the same calibration data C4. That being said, we use more calibration data (128 segments vs. 64 in EoRA paper). Our method 3BASiL-TM improves the accuracy post-compression from 35.75 to 44.88.\n\n\nWe would like to thank the reviewer again for pointing us to this important method. We intend to add it to the related works discussion (Exact Low-Rank updates for layer-wise compression). We also think it is helpful for the compression community if we add a paragraph discussing the underlying connections between our improved implementation of Hf-SparseGPT-ours and EoRA as well as more experiments in the Appendix.\n\n***W2 & Q1:***\nThis is a very good remark. We have launched preliminary results for the universality of Transformer Matching to pure pruning methods. The results in Table 3 of the paper for pure pruning methods have been extracted from HASSLE-free paper [2]. During the rebuttal, we have added support for SparseGPT and ALPS with TM. The results are below.\n\nTable 2: Comparison of pure pruning methods with 2:4 and 4:8 configurations on Llama3-8B. For perplexity (lower is better, ↓). For accuracy (higher is better, ↑).\n\n| **Method** | **Config** | **C4 ↓** | **WT2 ↓** | **PTB ↓** | **PIQA ↑** | **ARC-E ↑** | **ARC-C ↑** |\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| SparseGPT | 2:4 | 22.62 | 16.14 | 25.31 | 71.49 | 56.19 | 34.04 |\n| SparseGPT-TM | 2:4 | 15.31 | 10.82 | 17.07 | 76.50 | 66.04 | 40.70 |\n| ALPS | 2:4 | 19.80 | 14.59 | 22.42 | 73.56 | 59.60 | 35.15 |\n| ALPS-TM | 2:4 | **14.99** | **10.69** | **16.53** | **77.31** | **66.88** | **40.87** |\n|---|---|---|---|---|---|---|---|\n| SparseGPT | 4:8 | 17.56 | 12.27 | 18.63 | 74.86 | 62.92 | 38.82 |\n| SparseGPT-TM | 4:8 | 13.68 | 9.25 | 14.51 | 77.97 | **69.99** | **44.11** |\n| ALPS | 4:8 | 16.08 | 11.23 | 16.58 | 75.90 | 65.07 | 40.27 |\n| ALPS-TM | 4:8 | **13.61** | **9.18** | **14.28** | **78.35** | 69.74 | 42.75 |\n\n\nWe agree with the reviewer that further experiments should be included to pure sparsity algorithms to showcase the universality of our proposed Transformer Matching procedure. In the revised paper, we will add support to all reported pruning methods and their enhanced TM versions.\n\n\n***W3:*** \nThank you for pointing out these issues. They have been fixed in the revised manuscript. \n\nWe want to reiterate our appreciation for the thorough feedback of the reviewer. We believe that incorporating the suggestions and addressing the questions and weaknesses discussed above will greatly improve the quality of the revised paper.\n\n\n\n[1] Liu et al., EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation.\n\n[2] Makni et al., A unified framework for Sparse plus Low-Rank Matrix Decomposition for LLMs (CPAL'25)."}, {"review_id": "UfYK5jxj5Y", "reviewer": "Reviewer_pjgM", "summary": "In this paper, an ADMM algorithm is studied for compressing dense weights into sparse-plus-low-rank (S+LR) representations. With particular sparse patterns, such as 2:4 sparsity, S+LR matrices can be computed efficiently on GPUs. The paper proposes a multi-block ADMM procedure to initialize the factors using calibration data, which provides a good initialization and thus improves the model quality after fine-tuning. Each factor is obtained by minimizing the augmented Lagrangian while fixing the others, with convergence guarantees as the penalty term rho increases. To address misalignment in layer-wise output matching, the paper also introduces module-wise output matching. Experimental results demonstrate that 3BASiL improves both one-shot and fine-tuned compression outcomes across various structured sparsity patterns on large language models.", "questions": "1. Why is 3BASiL designed with a 3-block ADMM? In the Robust PCA literature, an ADMM algorithm to solve robust PCA $\\min_{L,S} ||L||_* + ||S||_1$  s.t.  $L+S=W$ can be formulated as a 2-block method. Is the use of 3 blocks in 3BASiL due to the hard sparsity constraint, unlike the l1 loss in RPCA?\n2. The last inequality of Equation 18 uses $||S^{(t+1)} - S^{(t)}||_F \\le \\frac{C_A}{\\rho_t}$, whereas the bound in Equation 17 is $\\frac{3C_A}{\\rho{t-1}}$. How is the former inequality obtained? Can’t the bound in Equation 17 be used directly?\n3. Transformer-level Matching is proposed to update the factors with a better proxy than layer-level matching. If the quality of this proxy is a concern, couldn’t the S and L factors be updated directly based on the original training loss (e.g., cross-entropy) or a knowledge distillation loss on a subset of C4, instead of splitting the process into Transformer-level Matching and LoRA fine-tuning?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper presents both theoretical and practical contributions. The proposed algorithm appears to outperform other S+LR decomposition models, and the experimental results show that the compressed models maintain decent performance. Theorem 1 provides convergence guarantees, which is a strong point.\n2. The writing quality is good, making the paper easy to understand.\n\n**Weaknesses**\n\nMost of my concerns relate to the experimental results.\n\n1. Although Theorem 1 proves convergence of the proposed ADMM steps, it is unclear whether convergence occurs in practice. Please consider including plots showing the diminishing difference between L and S at each step or the convergence of the augmented Lagrangian.\n2. The actual inference speedup is not reported. While it seems likely that the compressed network will run efficiently on GPUs supporting 2:4 or 4:8 sparsity, reporting real inference speed measurements would help emphasize the practical impact of the proposed method.", "was_revised": true, "final_justification": "All of my concerns are resolved.", "rebuttal": "We would like to thank the reviewer for the thorough review, for highlighting the strengths of our submission, and for raising important fundamental questions about our compression algorithm.\n\n**Weaknesses & Questions**\n\n***W1:*** Thanks for pointing out this remark. We also think that including plots for the convergence of the objective and iterates will further improve the quality of the paper, we will add such plots in the revised version.\n\nSince we are not allowed to include plots for this rebuttal per NeurIPS policy, we wanted to share simple logs we used to follow the convergence of 3BASiL [for the compression of self_attn.q_proj of the first transformer of Llama3-8B under a (2:4 + 128) configuration].\n\nFor iteration 009, the true loss ||X(W_old - (W_S + B @ A))||_F^2 = 1952.1258544921875\n\nFor iteration 049, the true loss ||X(W_old - (W_S + B @ A))||_F^2 = 664.7432861328125\n\nFor iteration 149, the true loss ||X(W_old - (W_S + B @ A))||_F^2 = 176.23971557617188\n\nFor iteration 199, the true loss ||X(W_old - (W_S + B @ A))||_F^2 = 119.56678771972656\n\nFor iteration 299, the true loss ||X(W_old - (W_S + B @ A))||_F^2 = 87.93989562988281\n\nFor iteration 399, the true loss ||X(W_old - (W_S + B @ A))||_F^2 = 87.01194763183594\n\nFor iteration 479, the true loss ||X(W_old - (W_S + B @ A))||_F^2 = 86.82907104492188 \n\n[479 is the last iteration, note that iterations in our algorithm are extremely efficient and are not comparable to the default value 80 of alternating minimization steps used in HASSLE-free and OATS].\n\n***W2:*** Thank you for pointing out this very important point! We are currently running experiments on the model OPT-30B which we intend to include in the revised paper [see Table 2 below for a comparison between 3BASiL and Hf-SparseGPT--it seems that this type of compression has less impact on performance for larger models]. \n\nAccording to SLOPE [1], we should expect an inference acceleration of up to 1.25× and up to 0.63 memory reduction for this model if we use their custom designed CUDA kernels during inference.\n\nTable 2: Comparison of the the perplexity of Hf-SparseGPT and 3BASiL on (2:4 + 64) configuration for a OPT-30B model.\n\n| **Method** | **Config** | **C4 ↓** | **WT2 ↓** | **PTB ↓** |\n|:---|:---:|:---:|:---:|:---:|\n| Hf-SparseGPT | 2:4 + 64 | 11.64 | 10.34 | 14.54 |\n| 3BASiL | 2:4 + 64 | **11.55** | **10.04** | **14.32** |\n| Dense | - | 11.44 | 9.56 | 14.04 |\n\n***Q1:*** Thank you for making the connection between our optimization formulation and the RPCA literature. Our problem is quite different from RPCA as we aim to match the $\\textbf{outputs}$ of each weight matrix $||X(W - S - L)||_F$ as opposed to the actual matrix weights matching $||W - S - L||_F$. A block in ADMM is introduced to this end. Please also note [line 154 of our paper] that our 3-block ADMM approach can be reformulated as a standard 2-block because the Lagrangian is separable with respect to 2 blocks introduced for the optimization. \n\n***Q2:*** This is a typo, thanks for catching it. The correct inequality should be \n\n$| L^{(t+1)} - L^{(t)} |_F \\leq \\frac{3C_F^2 C_A}{\\rho_{t-1}}$, \n\nwhich directly uses the bound in Eq (17). The constant $C$ should be correspondingly changed to \n$C=3C_F^2C_A$ \n(since $C_F\\ge 1$). We deeply appreciate the reviewer's careful examination and have corrected this typo in our revised manuscript.\n\n***Q3:*** This is a fundamental question related to one-shot compression methods! It is true that $S$ and $L$ can be updated with the original loss function ($L$ is updated during LoRA fine-tuning for example) or a knowledge distillation loss from the original model. However, if one aims to update $S$, one needs to do a full back-propagation on the entire LLM which is very memory intensive. Our entire pipeline: compression of LLama3-8B with 3BASiL, refinement with TM and LoRA fine-tuning can be done on a single A100 GPU. However, full back-propagation even with batch=1 results in a cuda memory error. That's why Transformer Matching introduces an intermediate [memory-efficient] loss function which provides is a good trade-off between the local-layerwise reconstruction loss and the original training loss.\n\n\n[1] Mozaffari et al., SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs (ICLR'25)."}, {"review_id": "1BkWoaepUU", "reviewer": "Reviewer_UVpP", "summary": "This paper proposes a sparse + low-rank approach for model compression and introduces cross-layer joint optimization to improve the performance of the compressed model.", "questions": "1.The paper primarily presents experimental results for the semi-structured sparse component. To demonstrate the method's general applicability, could additional comparisons with unstructured sparsity be attempted?\n\n2.Could the actual acceleration effects of the compressed model in practical usage be provided to show that the low-rank + sparse method is indeed well-adapted to hardware?\n\n3.If conditions permit, a more extensive assessment can be conducted appropriately.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**：\n\n1.This work proposes a novel three-block variable decomposition framework that jointly optimizes sparsity and low-rank constraints, which address the theoretical limitations of existing alternating minimization methods.\n\n2.Introduced a TM fine-tuning step after model compression to enhance the performance of the compressed model.\n\n**Weaknesses**：\n\n1.This method is only applicable to semi-structured conditions, which limits its scope of application\n\n2.Experiments were only conducted on models of 1-8B size. The performance on larger-scale models remains to be further verified", "was_revised": true, "final_justification": "These responses have addressed most of my concerns. Given that the current evaluation is already positive, we will not modify the score.", "rebuttal": "We would like to thank the reviewer for the extensive feedback, highlighting the strengths of our paper, and the valuable suggestions to improve the manuscript.\n\n\n**Weaknesses and Questions**\n\n***W1 & Q1:*** Our method is indeed generalizable to unstructured sparsity patterns. We experiment with a \"less aggressive compression\" configuration (0.5 + 128) and study the differences after further LoRA fine-tuning. We believe that it is interesting to further explore these settings as they are \"near-lossless\" configurations.\n\nTable 1: Comparison of the the perplexity of (S+LR) algorithms before and after LoRA fine-tuning on the configuration (0.5 + 128) on the model Llama3.2-1B. For perplexity (lower is better, ↓). If a dataset has the suffix \"-LFT\", we report the perplexity on that dataset after further LoRA fine-tuning on the C4 dataset [this can decrease the performance on WT2 and PTB datasets if the model has a performance comparable to dense model--3BASiL seems to be the only one to have this property].\n\n| **Method** | **Config** | **C4 ↓** | **C4-LFT ↓** | **WT2 ↓** | **WT2-LFT ↓** | **PTB ↓** | **PTB-LFT ↓** |\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| OATS | 0.5 + 128 | 17.99 | 16.71 | 12.29 | 11.76 | 21.68 | 20.97 |\n| Hassle-free-SparseGPT | 0.5 + 128 | 17.25 | 16.38 | 11.91 | 11.53 | 21.01 | 20.53 |\n| Hassle-free-ALPS | 0.5 + 128 | 16.81 | 16.12 | 11.62 | 11.41 | 20.59 | 20.54 |\n| 3BASiL | 0.5 + 128 | 16.17 | 15.69 | 11.17 | 11.13 | 19.84 | 20.00 |\n| 3BASiL-TM | 0.5 + 128 | **15.78** | **15.44** | **10.89** | **10.91** | **19.19** | **19.43** |\n| dense | -- | 14.01 | 14.01 | 9.75 | 9.75 | 17.59 | 17.59 |\n\n\n***W2 & Q2:*** Thank you for pointing out this very important point! We are currently running experiments on the model OPT-30B which we intend to include in the revised paper [see Table 2 below for a comparison between 3BASiL and Hf-SparseGPT--it seems that this type of compression has less impact on performance for larger models]. \n\nAccording to SLOPE [1], we should expect an inference acceleration of up to 1.25× and up to 0.63 memory reduction for this model if we use their custom designed CUDA kernels during inference.\n\nTable 2: Comparison of the the perplexity of Hf-SparseGPT and 3BASiL on (2:4 + 64) configuration for a OPT-30B model.\n\n| **Method** | **Config** | **C4 ↓** | **WT2 ↓** | **PTB ↓** |\n|:---|:---:|:---:|:---:|:---:|\n| Hf-SparseGPT | 2:4 + 64 | 11.64 | 10.34 | 14.54 |\n| 3BASiL | 2:4 + 64 | **11.55** | **10.04** | **14.32** |\n| Dense | - | 11.44 | 9.56 | 14.04 |\n\n\n***Q3:*** In the revised paper, we are going to expand our experiments to report more unstructured sparsity (S+LR) configurations similar to Table 1 above. We intend to include more pure pruning results to showcase the universality of Transformer Matching (see Table 2 of our response to reviewer JUTM). In addition, we are going to include a subsection showcasing the applicability of our method to non-uniform transformer compression (see Table 2 of our response to reviewer J8EX). Finally, we will report some results on the model OPT-30B to see the performance of 3BASiL on larger models [similar to Table 2 above].\n\n[1] Mozaffari et al., SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs (ICLR'25)."}, {"review_id": "RWR35hBPAW", "reviewer": "Reviewer_J8EX", "summary": "In this paper, the authors propose 3BASiL, which introduces a unified algorithmic framework for compressing LLM by decomposing weight matrices into sparse + low rank form. Its key innovations include: \n\n1) Propose a 3-block ADMM algorithm that jointly updates sparse, low-rank, and the dual variables with a closed-form update for each.\n\n2) Propose a block-level weight fine-tune step that is able to further improve performance.", "questions": "Please see the weakness.", "limitations": "The authors addressed the limitations in the Conclusion and limitation section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1) The method offers closed-form updates for each component (sparse, low-rank, dual), and is supported by theoretical convergence guarantees.\n\n2) Compared to prior work, it achieves up to 40% lower perplexity and 2–3× faster compression, demonstrating both efficiency and effectiveness.\n\n**Weaknesses:**\n\n1. The necessity of adding the term $\\|\\hat{W} - (S + L)\\|$ in Equation (1) is not clearly explained. When computing updates, doesn't it behave similarly to $\\|(X + \\lambda)\\hat{W} - (X + \\lambda)(S + L)\\|$?\n\n2. While the method focuses on N:M sparsity, it seems generalizable. Have you evaluated its performance under unstructured or other structured sparsity formats? Additionally, could it adapt to non-uniform sparsity distributions, such as OWL[1]?\n\n3. The paper mentions 80 alternating minimization steps during compression. Have you conducted sensitivity analysis on the number of steps?\n\n[1] Lu et al. Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity.", "was_revised": true, "final_justification": "My concerns are addressed. I lean to accept.", "rebuttal": "We would like to thank the reviewer for the high-quality review, highlighting the strengths of our paper, and the useful suggestions to improve the manuscript that led to an insightful finding regarding using OWL for (S+LR) methods.\n\n**Weaknesses:**\n\n***W1:*** You are making a very good point. If one expands the expression of equation (1), it is equivalent to minimizing $1/2Tr((\\hat{W} - S - L)^T (X^TX+\\lambda I) (\\hat{W} - S - L))$. So it behaves like adding a regularization term to the local layer-wise reconstruction error Hessian $X^TX$. The aim here is to make the Hessian matrix $X^TX + \\lambda I$ invertible (which is satisfied for any $\\lambda > 0$) and reduce the numerical errors when applying equation (4). \n\nFrom a global optimization perspective, we aim to minimize the layer-wise reconstruction error $\\textit{without deviating too far}$ from the original weights. It has also been used in previous (S + LR) compression methods like Hassle-free.\n\n\n***W2.1:*** Our method is indeed generalizable to unstructured sparsity patterns. We experiment with a \"less aggressive compression\" configuration (0.5 + 128) and study the differences after further LoRA fine-tuning. We believe that it is interesting to further explore these settings as they are \"near-lossless\" configurations.\n\nTable 1: Comparison of the the perplexity of (S+LR) algorithms before and after LoRA fine-tuning on the configuration (0.5 + 128) on the model Llama3.2-1B. For perplexity (lower is better, ↓). If a dataset has the suffix \"-LFT\", we report the perplexity on that dataset after further LoRA fine-tuning on the C4 dataset [this can decrease the performance on WT2 and PTB datasets if the model has a performance comparable to dense model--3BASiL seems to be the only one to have this property].\n\n\n| **Method** | **Config** | **C4 ↓** | **C4-LFT ↓** | **WT2 ↓** | **WT2-LFT ↓** | **PTB ↓** | **PTB-LFT ↓** |\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| OATS | 0.5 + 128 | 17.99 | 16.71 | 12.29 | 11.76 | 21.68 | 20.97 |\n| Hassle-free-SparseGPT | 0.5 + 128 | 17.25 | 16.38 | 11.91 | 11.53 | 21.01 | 20.53 |\n| Hassle-free-ALPS | 0.5 + 128 | 16.81 | 16.12 | 11.62 | 11.41 | 20.59 | 20.54 |\n| 3BASiL | 0.5 + 128 | 16.17 | 15.69 | 11.17 | 11.13 | 19.84 | 20.00 |\n| 3BASiL-TM | 0.5 + 128 | **15.78** | **15.44** | **10.89** | **10.91** | **19.19** | **19.43** |\n| dense | -- | 14.01 | 14.01 | 9.75 | 9.75 | 17.59 | 17.59 |\n\n\n***W2.2:*** Our method can also support non-uniform sparsity distributions and we have added support for OWL for unstructured sparsity. This is a great insight and we would like to thank the reviewer for raising this important remark. We have tried to activate OWL for the configuration (0.7 + 64) for Llama3-8B. It turns out that this does improve the results of 3BASiL. This is not trivially true because OWL deltas were optimized for pure pruning methods. We believe that this is an insightful finding for the community. It opens further research directions as to how to create similar deltas for the low-rank components, so thanks to the reviewer for this suggestion!\n\nTable 2: Comparison of the performance of our proposed method 3BASiL with and without activating OWL on the configuration (0.7 + 64) on Llama3-8B. For perplexity (lower is better, ↓). For accuracy (higher is better, ↑).\n\n| **Method** | **Config** | **C4 ↓** | **WT2 ↓** | **PTB ↓** | **PIQA ↑** | **ARC-E ↑** | **ARC-C ↑** | **BoolQ ↑** | **HellaSwag ↑** | **OpenBookQA ↑** | **RTE ↑** | **WinoGrande ↑** | **Avg ZS ↑** |\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 3BASiL | 0.7 + 64 | 25.33 | 20.51 | 28.01 | 71.16 | 53.49 | 31.83 | 74.22 | 55.32 | 32.60 | **55.96** | 65.90 | 55.06 |\n| 3BASiL-OWL | 0.7 + 64 | **23.32** | **19.84** | **27.09** | **71.93** | **59.81** | **32.68** | **74.71** | **58.50** | **33.80** | 53.79 | **66.22** | **56.43** |\n\nWe intend to add a section showcasing the value of activating OWL for highly sparse unstructured configuration [e.g. (0.7 + 64)] and adding a discussion motivating the question of non-uniform sparsity and non-constant rank allocation as a future research direction.\n\n***W3:*** 80 alternating minimization steps is the default value in the papers of OATS and Hassle-free. A sensitivity analysis has been conducted in OATS for instance [see figure 1 of OATS paper [1]], and shows that the performance plateaus after 80 minimization steps. For these alternating-minimization based methods, increasing the number of steps generally lowers the objective (1) and improves the performance. If we were to reduce the number of steps, the performance would degrade.If we were to increase it, the performance gain is negligible (plateau reached) in our experiments but the runtime still increases.\n\nOur algorithm is different from alternating minimization [used by OATS/HASSLE-free] and the number of iterations is not directly comparable to OATS and HASSLE-free. That being said, 3BASiL is both faster and achieves a better performance than OATS and HASSLE-free [see figure 2 in our submission].\n\n\n[1] Zhang et al., OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition (ICLR'25)"}]

Author rebuttal:

Meta-review: The paper proposes 3BASiL, an efficient one-shot post-training method for decomposing large language models (LLMs). Section 2 formulates the optimization problem, which is solved using the ADMM framework. A key component of the method is the Transformer Matching (TM) step, which jointly optimizes all sparse and low-rank components across layers within a transformer block to better approximate the original model’s output. The paper presents an extensive set of experiments demonstrating the effectiveness of the proposed approach.

All reviewers appreciated the novelty of the method and the rigor of the experimental evaluation. Initial concerns raised during the review process were addressed during the discussion phase, including clarifications and additional results. These responses were well-received, and there was broad consensus among reviewers that the paper meets the standards for acceptance.

I recommend acceptance. The paper offers a well-motivated and technically sound contribution, supported by thorough experimentation. For the camera-ready version, the authors should incorporate all requested changes, including the additional experiments presented during the rebuttal.
Decision: Accept (poster)