ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization
Authors: Lawrence Liu
Alexander Liu
Mengdi Wang
Tuo Zhao
Lin F. Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Pruning
Model Compression
Block Coordinate Descent
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: New LLM pruning method that utilizes block diagonal matricies to preserve more model performance post-pruning.

Abstract: Large language models (LLMs) present significant deployment challenges due to their immense computational and memory requirements. While semi-structured pruning, particularly 2:4 sparsity, offers a path to practical hardware acceleration, existing methods often incur substantial performance degradation. To bridge this gap, we introduce ARMOR: (Adaptive Representation with Matrix-
factORization), a novel one-shot post-training pruning algorithm. Instead of directly pruning weights, ARMOR factorizes each weight matrix into a 2:4 sparse core wrapped by two low-overhead, block diagonal matrices. These wrappers act as efficient pre- and post-transformation error correctors, offering greater flexibility to preserve model quality compared to conventional 2:4 pruning techniques. The sparse core and block diagonal wrappers are chosen through a block coordinate descent algorithm that minimizes a layer-wise proxy loss. We prove this optimization is guaranteed to converge to a solution with a proxy loss less than or equal to state-of-the-art pruning algorithms. Experiments on Llama (Touvron et al., 2023; Dubey et al., 2024) and Qwen (Yang et al., 2025) model families demonstrate that ARMOR consistently and significantly outperforms state-of-the-art 2:4 pruning methods across a wide range of downstream tasks and perplexity evaluations, and generalizes to provide improvements for general N:M patterns and unstructured sparsity. ARMOR achieves this superior performance while retaining the inference speedups and substantial memory usage reductions of 2:4 pruning, establishing a more effective trade-off between model compression and task accuracy.

Published as a conference paper at ICLR 2026
ARMOR: HIGH-PERFORMANCE SEMI-STRUCTURED PRUNING VIA ADAPTIVE MATRIX FACTORIZATION
Lawrence Liu' Alexander Liu² University of California, Los Angeles
Mengdi Wang*
Tuo Zhao
Lin F. Yang'
IIdependent
3 Princeton University
Georgia Institute of Technology
ABSTRACT
Large language models (LLMs) present significant deployment challenges due to their immense computational and memory requirements. While semi-structured pruning, particularly 2:4 sparsity, offers a path to practical hardware acceler- ation, existing methods often incur substantial performance degradation. To bridge this gap, we introduce ARMOR: (Adaptive Representation with Matrix- factORization), a novel one-shot post-training pruning algorithm. Instead of di- rectly pruning weights, ARMOR factorizes each weight matrix into a 2:4 sparse core wrapped by two low-overhead, block diagonal matrices. These wrappers act as efficient pre- and post-tran post-transformation error correctors, offering greater flexibil- ity to preserve model quality compared to conventional 2:4 pruning techniques. The sparse core and block diagonal wrappers are chosen through a block coor- dinate descent algorithm that minimizes layer-wise proxy loss. We prove this optimization is guaranteed to converge to solution with a proxy loss less than or equal to state-of-the-art pruning algorithms. Experiments on Llama (Touvron et al., 2023; Dubey et al., 2024) and Qwen (Yang et al., 2025) model families demonstrate that ARMOR consistently and significantly outperforms state-of- of-the- art 2:4 pruning methods across wide range of downstream tasks and perplexity evaluations, and generalizes to provide improvements for general N:M patterns and unstructured sparsity. ARMOR achieves this superior performance while re- taining the inference speedups and substantial memory usage reductions of 2:4 pruning, establishing a more effective trade-off between model compression and task accuracy.
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable capabilities (Park et al., 2023; Huang& & Yang, 2025), yet their immense computational and memory requirements pose significant barriers to practical deployment. As result, techniques reducing model sizes and computa- tional costs while retaining performance are of significant interest for the research community. A particular area of focus are one-shot post training compression techniques, a highly efficient model compression regime where already trained models compressed in single pass without itera- tive fine-tuning. Pruning, the removal of model parameters is a particularly compelling avenue of compression as it offers direct path to inference acceleration via dedicated hardware support for specific sparsity patterns (Kwon et al., 2022), and its benefits can be compounded with orthogonal methods like quantization (Frantar al., 2022; Lin al., 2024; Li et al., 2025a). However, a critical trade-off plagues existing pruning techniques. Methods capable of delivering tangible inference ac- celeration do by sacrificing significant model accuracy, while the most accurate techniques offer largely theoretical speedups. Bridging this gap is the central focus of our work.
Pruning algorithms can be broadly divided into three categories: structured, unstructured, and semi- structured pruning. Structured pruning removes entire weight structures, such as rows or columns of weight matrices (Ashkboos et al., 2024), attention heads (Ma et al., 2023), or even full layers (Men et al., 2024). This coarse-grained approach is highly compatible with existing hardware and soft- ware, as it results in smaller, dense matrices that can be processed efficiently by standard libraries,
"Correspondence 1awrencer1 fu uuucla e.edu

---

Published as a conference paper at ICLR 2026
Compressed Weight W
Nahe Approach
Pruned LLM
Individual Original Original Weights
W
AEMOR Elctotication
Mock Ulgend
2-4 Sperse Care
Midi Pligeaa
Pruned LLM
A
(W M)
B
Figure I: Illustration of proposed ARMOR factorization. For given LLM, each weight matrix W pruned individually. Instead naively pruning the weight matrix, ARMOR wraps the sparse core with pair of block diagonal matrices and uses a unique optimization algorithm to find the optimal structured pruning mask. M E {0, 1) dous x represents the 2:4 binary mask.
leading to direct improvements in inference speed (Ma et al., 2023). However, this rigidity comes at cost; by removing large, contiguous blocks, structured pruning can lead to significant degra- dation model accuracy and often has lower limit on the achievable sparsity before performance collapses (Ashkboos et al., 2024).
At the other end the spectrum, unstructured pruning offers the highest flexibility by removing in- dividual weights from any part of the model. Leading unstructured pruning algorithms have shown that is possible to remove up to 50% of the weights from an LLM with minimal loss in per- formance (Frantar & Alistarh, 2023). However, the resulting irregular, sparse matrices disrupt the parallel processing capabilities of modern hardware like GPUs, which are optimized for dense ma- trix operations. Thus difficult translate the theoretical model size reductions into practical inference speedup (Xia et al., 2023).
To bridge the gap between hardware efficiency and model performance, semi-structured pruning has emerged as compelling compromise. A popular variant is N:M sparsity, which enforces regular pattern by ensuring that within any contiguous block of weights, only N weights are non- zero. This regularity is key, as it can be directly accelerated by specialized hardware. For instance, NVIDIA's Ampere and subsequent GPU architectures provide native support for 2:4 semi-structured sparsity, which can theoretically double the throughput of matrix operations (Hu et al., 2024; Mishra et al., 2021). However, the constraint of a fixed pruning pattern within each small block restricts the algorithm's ability to retain the most critical weights, leading to significantly increased drop in performance. For example, applying a state-of-the-art 2:4 pruning method to Llama-7B increases Wikitext2 perplexity by nearly 59% its 50% unstructured counterpart, creating an undesirable choice between theoretical efficiency and practical accuracy (Sun et al., 2024).
In this work, we seek close this performance gap by introducing ARMOR: (Adaptive Representation with Matrix-factORizztionn) an theoretically grounded one-shot model pruning al- gorithm. We reframe semi-structured pruning pruning weights, our key insight is to factor each weight matrix into constrained sparse core that adheres the 2:4 hardware pattern, pre- and post-multiplied by lightweight block diagonal matri- ces, this factorization is illustrated in Figure 1. These block diagonal matrices, which are highly parameter-efficient due to their sparse structure (containing only O(N) parameters compared to O(N²) for dense matrix), can be multiplied with activations efficiently modern hardware. They act as learned, low-overhead linear transformations that rotate the activation and weight spaces into basis where the 2:4 pruning constraint is less lossy, providing enhanced flexibility and preserving model quality more effectively than naive 2:4 pruning algorithms.
matrix factorization problem. Instead of directly
Through extensive experiments Llama (Dubey et al., 2024; Touvron al.. 2023) and Qwen (Yang et al., 2025) family models, we show that ARMOR consistently and significantly outperforms existing 2:4 pruning methods on both perplexity and downstream tasks. For example, Llama-2-
2

---

Published as a conference paper at ICLR 2026
13B, ARMOR reduces the perplexity gap between the 2:4 pruned model and the dense original by almost 50%. We show that these accuracy gains are achieved while preserving the practical inference speedups and memory reduction inherent to native 2:4 sparsity, and that ARMOR generalizes to general N:M and unstructured sparsity with consisted improvements over SOTA. Our work suggests that rethinking the fundamental representation of weights, rather than simply removing them, is promising direction for future hardware-software co-design in efficient deep learning.
1.1 RELATED WORK
Existing one-shot unstructturedseemi-srrucccaa weight matrix W as the element-wise product of a dense matrix W and binary mask M, where W = W' C M. Research within this paradigm focuses on two primary challenges: identifying an optimal mask M (importance scoring) and updating the unpruned weights W" to compensate for the removed connections. The simplest approaches are weight-update-free methods like Wanda, which fix the unpruned weights to their original values (W" W) and focus on finding a good mask M through clement-wisc metrics (Sun et al., 2024; Liu et al., 2025; Liu et al.; Dong et al., 2024; Das et al., 2023; Zhang et al., 2024). In contrast, more complex weight-update methods such as SparseGPT (Frantar & Alistarh, 2023) aim for higher accuracy by iteratively pruning weights while simultaneously adjusting the remaining ones to minimize a proxy reconstruction loss, often based on Hessian sketch. While effective, this introduces significant computational overhead, such as the costly inversion of the Hessian sketch matrix. Both methods suffer significantly increased performance loss when applied to 2:4 semi-structured pruning compared with their unstructured counterparts.
pruning methods largely formulate the compressed
This gap has led alternative formulations beyond elementwise masking. These included sparse matrix factorization approaches such as DSF (Boža & Macko, 2024), decomposition based works like WRP, and hybrid low-rank structures such as LoSparse (Li et al., 2023) and Targeted Low Rank Refinement (Shen et al., 2025). However such approaches suffer from their own challenges, DSF and outieer-adapive schemes like OWL (Yin et al., 2023) are incompatible with accelerated 2:4 ker- nels. Similarly, WRP's performance is bottlenecked by its reliance
highly unstructured sparse matrix component, for which no efficient matrix-multiplee multiplication kernels exist. Directly learnable ap- proaches such as LoSparse and MaskLLM (Fang al., 2024) typically rely expensive iterative retraining, for instance MaskLLM uses x0000 more data and ~80 more compute than ARMOR. Finally, rotation-based methods such RotPruner (Chen & Wang, 2025) and DenoiseRotator (Gu et al., 2025) introduce fixed overheads. Unlike ARMOR 's tunable block-diagonal wrappers, these rotations cannot be easily adjusted to balance accuracy and latency, leaving a gap between high compression performance and tangible speedups.
2 PROBLEM STATEMENT
To remain computationally tractable, one-shot compression pruning methods regularly adopt layer- by-layer framework. For a given linear layer with weight matrix W E RRdon Xdm, the objective is to find a compressed representation W that minimizes a data-aware proxy loss, (W.) This loss quantifies the approximation error using small calibration dataset X"" E RRaxd,, The exact formulation of this proxy loss a key design choice and varies across different pruming algorithms, we define our proxy loss in Section 3.2.
We constrain W to leverage a 2:4 semi etri-structured sparse component to guarantee hardware acceler- ation. This requires that the underlying binary mask M E {0, Aou xdi defining this component's pattern has exactly two non-zero entries in every group of four consecutive columns per row. For- mally we express this constraint as M.M.illloo = 2, Vi [done], k E [din/4] where we adopt MiJp Mr 1:4k shorthand. The general layer-wise optimization problem is therefore:
min Lw.x s.t. |M = 2, Vi E [dous] l: [dsn/4]
params
Regardless of optimization algorithm, this layer-wise approach is inherently one-shot because the compression is completed in single pass over the network's layers without any global retraining.
3

---

Published as a conference paper at ICLR 2026
3 METHODS
In this section we introduce the ARMOR pruning process. Section 3.1 introduces the ARMOR factorization and notation. In Section 3.2 discuss the proxy loss optimization objective and initialization. In Section 3.3 we introduce the ARMOR optimization algorithm to optimize the ARMOR factorization. This algorithm consists of two alternating steps, the continuous parameter update step, Section 3.3.1, and the sparse core update step, Section 3.3.2. Finally we introduce the main theoretical result in 3.4.
3.1 ARMOR MATRIX FACTORIZATION
For each layer in an LLM, let the original weight matrix be W E Rd Xdin, Our approach seeks to find a compressed matrix, W E Rdon xdi,, with the following factorization:
W(A,B,W, M) A (W" M). B,
where the goal is to optimize for the parameters A, B, W and M. Specifically, W Rd x din a dense matrix representing transformed weights, and M E {0,1) 1ood xdin is a binary mask that imposes sparsity through element-wis multiplication (O). The matrices A E Redou X and B Red xde are block-diagonal. The block size, dblock. is chosen hyperparameter, selected such that it divides both dout and din- We refer the set of all learnable parameters as 0 (A, B, W M).
The key to ARMOR is the diagonal matrix wrappers, and B, that surround the sparse "core" W" @ M. Compared to the naive approach of directly pruning diagonal only wrappers, these block diagonal matrix wrappers offer additional flexibility, while having low overhead and existing implementations on hardware for storage and inference. We can store and B as tensors of size (dout/dblock] dblocc)) dblock X dblock and (in/dklock) x dblock X dislokk respectively. Matrix multiplication inference time can be performed as batched matrix multiplication. storing A and B and performing inference grows with (((out + di dolock))) which is sublinear to number of original parameters in the layer, dounthn-
a result the overhead of
Notation We denote the individual blocks of A and as A(1) Rddttox Xdotock and B(i) pidstock Xdblock ie: A = diag (A(2), ...., (ddoss/dtooc))) and likewise for B. More gener- ally for any matrix C E Rdou xd. we denote the ditock X dolock matrix blocks Clis), ie Vi E [dout/dblock] j E [din/ detook]) A de- tailed illustration this notation can be found in appendix A.
((l,) ((--)debsoca + L:iddsaa,
3.2 OPTIMIZATION OBJECTIVE
We optimize W by minimizing the NoWag layerwise proxy loss (Liu al., 2025).
Lw.x (0) = ||W - W Ii didgg = ΣΣ (W., W.,) )1,)) f
(2)
where W E Bout xdu is the row and column normalized version of W:
dea
du Σ
W 1
2
W4
W fJ
Vjj E [dmns
r!
Vi E [dou]]
(2)
We adopt the Wo objective function due to its data-aware nature and decomposable structure. The diag (Xx*) term weights the squared Frobenius norm by the magnitude of the corresponding input activations, which focuses the optimization preserving weights that are most influential for the given calibration data. Furthermore, compared to Hessian sketch based methods, this objective function requires less calibration data.
A key advantage of this objective is that the loss can be decomposed into independent element- wise subproblems. While ARMOR factorization's wrapper matrices prevent direct clement-wise optimization, this structure is still critical as allows for the problem to be broken down into in- dependent block-level subproblems, which is cornerstone of our greedy sparse core optimization strategy. After optimizing, we scale W back by denormalizing, which is performed by pre-scaling the rows and columns of A and B by (1) and (2) respectively before inference.
4

---

Published as a conference paper at ICLR 2026
3.3
OPTIMIZATION ALGORITHM
Algorithm ARMOR Optimization Algorithm
Require: Original weight W. calibration data X, tolerance E.
Ensure: Optimized factorization parameters 0 = (A, B, W MM.
1:
> Initialization Using and column normalization from NoWag
2: W Normalize (W)
3: (0)o < Initialize(W,JX
Initialize according Eq
4: for [ 1,2,3, 3, Mittes do
5: (A)r, (B),, [W))tfoon < 1 (B) (W")-1, (M)-11 W, X)
6: (W"),, (M); f SpansCCoreUpdate(AA), (B)), (W")ejcont: (M)r-1, X
7: end for
8: return A,, Bt, W7, Me
To optimize the ARMOR factorization, utilize block coordinate descent (Wright, 2015) al- gorithm that alternates between updating the continuous parameters A, W and the sparse core W' @ M. These alternating updates are performed for Ritees iterations. The overall algorithm is outlined in Algorithm 1. detailed technical description of the optimization algorithm can be found in appendix B.
Initialization
We initialize our factorization (0)o = ((A)o, (B)o, (W"),, (A))o) as:
(A)o= I, (B)o I, (W')o W. (Mij)o (Ir, top2 (Iz.je)) Vi [dout]. [diu]s (3)
where [5/4]. W2 and I denotes the identity matrix of appropriate size. This initialization is chosen as it is the optimal solution to equation 2 if W only consisted of the naive sparse core, and is equivalent to the pruning result of NoWag- pruning algorithm (Liu al., 2025).
3.3. UPDATING OF CONTINUOUS PARAMETERS
This step uses sequential gradient descent update A, B, and W sequentially, with the learning rate determined via local B-smoothness. Algorithm depicts single step in detail. In practice, we replace these sequential steps with joint Adam (Kingma & Ba, 2014) optimization that updates A, B, and W" simultaneously, a choice driven primarily by efficiency. This approach requires only one forward/backward pass per iteration and eliminates the need to recalculate local {-smoothness at each step. While Adam introduces minor additional memory overhead, this is negligible since we optimize each layer independently. We present the sequential gradient descent version here to provide strict theoretical guarantees of convergence. In practice, however, joint Adam optimization yields no significant differences compared to sequential gradient descent.
3.3.2 UPDATING THE SPARSE CORE
This step updates the sparse core W" @ M to reduce the proxy loss. To avoid the exponential large search space, adopt greedy approach, where select and update fraction of the elements of the sparse core to reduce the proxy loss. An illustration of the algorithm is depicted in Figure 2 and mathematical description can be found in Appendix B.1. Below elaborate the technical novelty and efficiency of the procedure.
Leveraging the 2:4 Pattern If we freeze all of the sparse core beyond single consecutive sparse group. finding the optimal values for this sparse group can be performed as follows. Since there are only (+) = 6 possible mask choices for group, it is computationally tractable to consider each possible mask. For possible mask choice. finding the proxy loss minimizing values for the 2 non- zero elements the group is least squares problem. Thus we can solve the least squares for all possible mask choices and select the with the smallest minimal proxy loss.
5

---

Published as a conference paper at ICLR 2026
ARMUR Factonsatton
Poesible Musics
Values
B:00
0.00
0.9
3:00
0.00
0:27
0:90
8:00
0.13
Bock Wite Subgroelem
6:00
A-LL
2.01
0.00
0.71
8:00
0.13
6.90
-222
L.1
Figure 2: An illustration of the sparse update step of the ARMOR optimization algorithm
Leveraging the Elementwise Property of the Proxy Loss Updating single 4 element sparse group usually has minimal impact on the layerwise proxy loss, since 4 elements account for less than 0.2 x 10-6 of the parameters of typical LLM layer. To enable updates of more than 4 elements at time, we leverage the element wise property of the proxy loss. As discussed previously, this allows us break the optimization problem into broken independent block-level subproblems:
Cw.x(0) = E> (1.)) (ali))) j=1
ΣΣ
| A (Wilas) o Mn bb)l2 =1
(4)
where bbut/dblock)) j E /din/ddiood /dbtcck). Thus we can optimize each block, (W"(i.)) M(-))),
independently through the greedy least squares process outlined above. This allows for parallel
updates of (out) / d block groups a time, which for standard LLM translates on the order of 10ss more elements updated at once.
Selecting The Sparse Group We select the sparse groups randomly, with their selection probabil- (6,)) of selecting group ities weighted by their proxy loss gradient. For block (i,j). the probability P(ark))
i', k is:
(i.) P( x (Verroaaaaae
V
[dstock], E [ditock/4]
Such selection heuristic results in more focus the important sparse groups through proxy loss gradient weighting. Additionally, the randomness helps prevent selecting the same group over and over again. Empirically
observer that this leads faster and better convergence of the proxy loss, and thus better overall LLM performance retention. An ablation of this selection heuristic can be found in Appendix E.I.
3.4 THFORETICAL RESULTS
In this section, we establish the following theorem to show that our algorithm guarantees conver- gence
Theorem 3.1. (Convergence of the ARMOR optimization algorithm). v t>
The sequence
(Cw.x((0))) converges and Lw.x ((0)) <
The proof can be found in Appendix C. Since ARMOR factorization initialization is equivalent to NoWag-P, Lw.x (w)))(())))))))))))) is the proxy loss of NoWag-P. Thus ARMOR will perform at least equiva- lently to NoWag-P SOTA pruning algorithm, in terms of proxy loss.

---

Published as a conference paper at ICLR 2026
4 RESULTS
Models and Experimental Setup We demonstrate the efficacy of our pruning method on two con- temporary model families: Qwen 2.5 (7B, 14B, 32B, and 72B) and Qwen (8B and 14B) (Yang et al., 2025). Our investigation targets foundational base models of 7B parameters and larger. This focus on base models allows for a direct assessment of our algorithm's 's effect on the core knowledge acquired during pre pre-training. removing confounding variables introduced by instruction tuning or other post-trainnt modifications. Furthermore, we concentrate on dense architectures, excluding Mixture-of-Experte (MoE) variants, recent work suggests that MoE models benefit from special- ized pruning strategies (Xie et al., 2024; Li et al., 2025b). For all pruning experiments, we config- ured the block size, dotock, to 128 and executed the proxy loss optimization for 20,000 iterations. Additional implementation details are provided in Appendix H.
Evaluation: To comprehensively assess performance degradation, we employed a two-pronged evaluation strategy. First, to measure practical performance on downstream tasks, we evaluated the pruned Qwen models on suite of seven mndusty--standary benchmarks using the LM Eval Harness (Gao et al., 2024). These benchmarks cover range of capabilities, including commonsense and complex reasoning. mathematical problem-solving, and world knowledge. A detailed description of each benchmark is available in Appendix G. Second, to ensure comparability with the broader model compression literature, which often relies on perplexity metrics, we conducted an additional set of experiments. For this, we pruned models from the Llama-2 (7B, 13B, and 70B) (Touvron et al., 2023) and Llama-3 (8B and 70B) (Dubey et al., 2024) families. We then evaluated their perplexity on the test split of Wikitext2 (Merity al., 2016) and subset of the C4 validation split (Dodge et al., 2021). 1), following standard evaluation protocols in the field.
Baselines We compare ARMOR against 3 leading pruning methods, SparseGPT (Frantar & Alistarh, 2023), Wanda (Sun et al., 2024), WaWag-P (Liu et al., 2025). Using NoWag-P is of particular interest, since as discussed previously, ARMOR uses the same proxy loss and is initalized at NoWag- P. Therefore comparing ARMOR against NoWag-P servers as an ablation to evaluate the empirical effectiveness of the ARMOR factorization and optimization algorithm.
4.1 TASK BASED EVALUATIONS
Results of the task based evaluations on Qwen 2.5 (7B/14B/32B/77BB and Qwen (8B/14B) models shown in Tables 1 and 2 respectively. Across all 7 tasks and all models evaluated, ARMOR consistently and significantly outperforms the state-of-the-art pruning methods. For example, on GPQA with the Qwen 2.5-32B model, ARMOR achieves score of 39.51, outperforming even the dense model's performance (38.84) and vastly exceeding the next best pruning method, SparseGPT, which scored 30.36. The performance gains are especially pronounced in reasoning or domain expertise heavy tasks like GSM8K, BBH, GQPA, demonstrating that our factorization approach is more effective preserving the complex capabilities of the model compared to simply removing weights.
4.2 PERPLEXITY BASED EVALUATIONS
Results of perplexity based evaluations on Llama-2 (7B/13B/70B) and Llama-3 (8B/70B) (8B/700) models reported in Table 3. Closely reflecting the task based results, ARMOR consistently and significantly outperforms the state-of-the-art pruning methods in retaining lower perplexity, key indicator of lan- guage modeling quality. For example, on L.lam-22 13B evaluated on Wikitext2, ARMOR achieves a perplexity of 6.37. This is dramatic improvement over the next-best baseline (NoWag-P 8.28) and represents a reduction of nearly 50% in the perplexity gap relative to the original dense model. We observe similar substantial gains across all evaluated Llama models and datasets, reinforcing that the ARMOR factorization preserves model quality more effectively than existing 2:4 pruning techniques.
4.3 COMPARISONS WITH LEARNABLE BASELINES
We further compare ARMOR against recent learnable semi-structured pruning methods: RotPruner (Chen & Wang, 2025) and DenoiseRotator (Gu et al., 2025). For these specific comparisons,
7

---

Published as a conference paper at ICLR 2026
Task Accuracy (%) (1)
Method
Sparsity
MMLU
GSM8K
BBH
GPQA
ARC-C
Wino
Hclla
Dense (2.5-7B)
74.19
82.33
69.16
33.03
59.55
76.09
60.03
Dense (2.5-148)
79.8
88.02
75.18
38.17
64.51
80.9
63.46
Dense (2.5-328)
0
83.24
88.78
81.72
38.84
66.30
81.22
65.12
Dense (2.5-72B)
0
86.06
89.54
85.01
42.63
68.52
82.16
67.63
SparseGPT (2.5-7B)
2:4
56.91
36.69
46.31
29.69
43.43
68.35
47.04
Wanda (2.5-7B)
2:4
52.21
31,00
41.39
25.45
37.80
63.37
43.81
NoWag-P (2.5-7B)
2:4
53.51
28.28
39.98
27.23
39.16
64.01
44.33
ARMOR (2.5-7B)
2:4+4.95%
65.56
53.28
55.11
31.47
48.63
70,96
$1.67
SparseGPT (2:5-14B)
2:4
64.21
46.55
56.44
31.92
48.21
71.51
49.73
Wanda (2.5-148)
2:4
59.98
49.36
54.51
29.91
46.16
69.38
48.24
NoWag-P (2.5-14B)
2:4
58.45
45.87
52.1
30.36
43.77
68.67
48.42
ARMOR (2.5-14B)
2:444.17%
70.55
67.17
67.24
33.48
53.75
74.82
55.18
SparseGPT (2.5-32B)
2:4
75.26
66.03
71.31
30.36
57.08
79.01
55.73
Wanda (2.5-328)
2:4
75.45
72.93
70.17
35.27
55.72
77.82
55.71
NoWag-P (2.5-328)
2:4
74.89
68.69
69.53
27.01
55.29
76.64
55.69
ARMOR (2.5-32B
2:4+3.44%
78.18
78.77
76,56
39.51
60.15
79.32
59.78
SparseGPT (2.5-72B)
2:4
78.71
72.27
75.31
27.46
62.29
80.03
58.71
Wanda (2.5-72B)
2:4
79.61
75.66
23.88
62.37
80.19
59.43
NoWag-P P (2.5-72B)
2:4
78.93
75.13
76.04
28.35
60.84
79.08
59.31
ARMOR (2.5-72B)
2:4+2.4%
82.40
82.11
79.42
40.40
63.40
80.90
62.64
Table I: Results of Qwen-2.5 7B/14B/32B/72B The additional 0% for ARMOR pruned models represent the relative overhead of the block diagonal matricies. ARC-C is short for ARC-Challenge, Wino is short for WinoGrande, Hella is short for HellaSwag. (2.5-7B) denotes Qwen 2.5-7B, (2.5- 14B) denotes Qwen 2.5-14B, etc.
Task Accuracy (9) (t)
Method
Sparsity
MMLU
GSM8K
BBH
GPQA
ARC-C
Wino
Hella
Dense (3-8B)
76.82
84.91
77.42
42.86
63.65
76.87
58.92
Dense (3-14B)
0
80.47
84.0
78.50
39.29
66.64
79.08
61.81
SparseGPT (3-8B)
2:4
55.77
33.36
$2.96
32.14
44.54
66.46
44.74
Wanda (3-8B)
2:4
55.75
27.45
46.32
29.46
41.55
62.75
42.92
NoWag-P (3-8B)
2:4
54.1
28.28
43.37
28.57
40.02
61.48
42.78
Ours (3-8B)
2:4+5.03%
66.22
60.13
33,93
50.34
68.59
49.55
SparseGPT (3-14B)
2:4
64.73
48.22
61.5
27.9
$3.58
71.27
50.22
Wanda (3-14B)
2:4
62.93
52.16
$7.53
29.02
50.51
69.14
48.61
NoWag-P (3-14B)
2:4
61.69
46.63
56.11
27.46
48.89
68.27
48.42
Ours (3-14B)
2:443.89%
71.43
63.38
68.28
29.91
56.31
74.35
53.77
Table 2: Results of Qwen-3 3 8B/14B Base. Same setup the Qwen 2.5 results, Once again (3-8B)
denotes Qwen-3 8B etc
adopt the baselines' evaluation protocol: reporting Wikitext-2 perplexity at sequence length of 2048 (in contrast to the native 4096/8192 used in our main results in Table 3). We exclude MaskLLM (Fang et al., 2024) as it operates in retraining regime requiring orders of mangitude more data and compute than ARMOR. We also omit Targeted Low-Rank Refinement (Shen et al., 2025) due to lack of publicly available code. As shown in Table 5, ARMOR significantly outperforms RotPruner across all models. Against the concurrent work DenoiseRotator. ARMOR consistently outperforms the Wanda-based variant and remains highly competitive with the SparseGPT variant, surpassing on Llama-2 7B and 13B. Crucially, ARMOR achieves this parity without the rigid inference con- straints of rotation-basee based methods. While DenoiseRotator enforces fixed computational overhead
8

---

Published as a conference paper at ICLR 2026
Wikitext 2 (4)
C4 (U)
Method
Spursity
2-7B
2-13B
2-70B
3-8B
3-70B
2-7B
2-13B
2-70B
3-8B
3-70B
Dense
0%
5.12
4.57
3.12
5.54
2.58
6.63
6.05
4.97
7.10
5.78
SparseGPT
2:4
10.16
8.39
5.39
14.18
8.65
11.98
10.22
7.20
13.88
9.27
Wanda
2:4
11.35
8,36
5.20
22.42
8.29
13.80
10.96
7.19
21.63
9.63
NoWag-P
2:4
11.14
8.28
5.17
24.0
7.52
13.91
11.05
7.23
23.5
9.18
ARMOR
2:4+0
7.21
6.37
4.55
10.10
5.95
9.36
8.59
6.44
11.22
7.50
Table 3: Wikitext 2 and C4 perplexities on Llama-2 7B/13B/70B and Llama-3 8B/70B. Perplexity evaluations performed at 4096 and 8192 context length for Llama-2 and Llama-3 models respec- tively. o denotes the relative overhead of the block diagonal matricies, which 4.94%, 3.95%, 2.42% for the 7B, 13B, and 70B parameter models respectively. Following established notation 2- 7B denotes Llama-2 7B etc
Qwen 2.5 7B
Qwen 2.5 14B
Batched MatVec
Tokens/s
Max VRAM
Model Size
Tokens/s
Max VRAM
Model Size
(ms)
Dense
4461
32.84GB
14,23GB
2013
41.13GB
27.65GB
9.04
2:4
5430 (1.217x)
27.52GB
8.89GB
2157 (1.07Ix)
30.29GBB
16.81GB
4.85 (1.86x)
ARMOR
(1.141x)
28.11GB
9.25GB
2096 (1.04xx)
31.32GB
17.85GB
5.77 (1.57x)
Table Inference speed, Max VRAM, and Model Size for Dense, naive 2:4 pruning, and ARMOR pruned Qwen 2.5 7B/14B. Qwen2.5 7B and Qwen2.5 14B generation was performed at batch size 2048 and 512 respectively, Rightmost column lists the timings and speedup of batched Matrix Vector multiplication between batch of 8192 input activations and a dense, 2:4 pruned, and ARMOR factorized matrix for standard gate.pro layer of Qwen 2.5 14B.
via orthogonal matrices, ARMOR offers a tunable trade-off between accuracy and latency through the block size hyperparameter dblock.
4.4 INFERENCE EFFICIENCY
To quantify the practical efficiency of ARMOR, we implemented a PyTorch only proof of concept
and benchmarked its generation speed, max VRAM, and model size for Qwen 2.5 7B and 14B (Table 4), alongside batched Matrix-Vector timing (batch size 8192) for a Qwen 2.5- 14B layer. ARMOR explicitly trades a small fraction of theoretical throughput for improved model quality: for this layer, ARMOR's block diagonal wrappers introduce a 3.4% additional flop overhead, reducing the theoretical maximum speedup from 2.0x for naive 2:4 1.87x. Our implementation achieves 844 of this theoretical limit, compared to 93% for highly optimized native 2:4 kernels. other words, even using an unoptimized proof of concept, ARMOR retains the majority of the speed and memory benefits of 2:4 sparsity while delivering fate-of-the-arr recovery.
4.5 EXTENSIONS To N:M AND UNSTRUCTURED
ARMOR extends naturally to general N:M patterns. Furthermore, while the sparse core update not computationally tractable for general unstructured sparsity, ARMOR can be applied un- structured sparsity as well by only performing the continuous update step. In table 6 we present experiments comparing ARMOR with NoWag-P for 50% unstructured, 4:8, 5:8, and 6:8 patterns Llama-2 7B/13B. ARMOR consistently and significantly outperforms No across all sparsity patterns and types. These experiments were ran with significantly less iterations, 5000 for unstruc- tured, 2000 for semi-structured, than the main 2:4 results, as a result they only serve as lower bound on ARMOR's 's performance on sparsity structures beyond 2:4 sparsity.
4.6 ABLATIONS
To validate our proxy loss choice, we track its relative value against the model's relative C4 per- plexity during 20,000 iterations of the ARMOR proxy loss optimization algorithm. The left plot

---

Published as a conference paper at ICLR 2026
Wikitext 2 (4)
C4 (L)
Method
Sparsity
2-7B
2-13B
2-7B
2-13B
Wikitext 2 (4)
Dense
0%
5.12
4.57
6.63
6.05
Method
2-7B
2-13B
2-70B
3-8B
NoWag-P P
50%
6.37
5.49
8.27
7.35
Dense
5.47
4.88
3.32
6.13
ARMOR
50%+
6.13
5.42
8.01
7.23
NoWag-P P
4:8
8.04
6.47
10.17
8.67
RotPruner
9.20
11.65
Wanda +
ARMOR
4:8+0
6.74
5.96
8,78
7.98
RetoiseRotator
9.88
8.77
4.85
11.41
NoWag-P
5:8
5.89
5.14
7.61
6.80
SparseGPT
ARMOR
5:8+0
5.62
5.02
7.29
6.67
8.70
6.81
4.64
10.01
RetoiseRotator
NoWag-P P
6:8
5.35
4.76
6.92
6.27
ARMOR
7.65
6.80
4.80
11.27
ARMOR
6:8+o
5.26
4.72
6.80
6.23
Table 5: Comparison of Wikitext 2 perplex- ity (context length = 2048) of 2:4 pruned Llama-2 7B/13B/70B and Llama-3 8B be- tween ARMOR, RotPruner and DenoiseRo- tator methods. Same ARMOR models as Ta- ble 3 evaluated at shorter context length.
Table 6:
Comparison of ARMOR vs.
No across 50% and various N:M spar- sity patterns on Llama-2 7B/13B. Relative overhead o is the same as Table 3.
Relative Aug Prosy LossPeratesiyy Plaay
Relative Perplesity Block
Rolatto deg LnneTuptret
Telaboo C4 Poyly
1080
1758
3008
Figure 3: Left: Relative average Proxy Loss and C4 Perplexity of Llama-2 7B across 20,000 it- erations of the ARMOR Proxy Loss optimization algorithm with block size 128. Right: Relative C4 Perplexity for Lama-2 7B/13B, and Llama-3 8B across block sizes of (NoWag-P), 8, 16, 32, 64, and 128. ARMOR was ran for 5000 itcrations for each block size. Relative perplexity is with respect to initial and optimal (dense) perplexities.
in Figure shows strong correlation between the two metrics; as the proxy loss decreases, so does perplexity, confirming its utility as surrogate for overall model performance. Furthermore, we observe that the majority of the performance loss reduction was achieved within the first 2,500 iterations. We also performed an ablation study on block size to understand impact on perplexity (Figure 3, right). This ablation reveals clear trend across all models: increasing the block size im- proves performance by lowering perplexity in an exponential decaying manner. Additional ablations are detailed in Appendix E.
CONCLUSION
In this work, we introduce ARMOR, a novel one-shot algorithm that addresses the significant per- formance degradation of hardwarr-acceleraated 2:4 pruning. Instead of simply removing weights, ARMOR reframes the problem by factorizing each weight matrix into 2:4 sparse core and adap- tive, low-overhead block diagonal wrappers that act as error correctors. This approach is theoret- ically guaranteed to converge to solution with a proxy loss less than or equal to state-of-the-ar methods and empirically validated on Llama and Qwen family models, where it consistently and significantly outperforms existing 2:4 pruning techniques on both perplexity and downstream tasks. Crucially, ARMOR achieves these accuracy gains while retaining the majority of the infer- ence speedups and memory reduction of native 2:4 sparsity. Our work demonstrates that rethinking weight representation is a powerful path toward establishing a more effective tra between the performance and efficiency of large language models.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
Our code for ARMOR is available on Github at ARMOR/ tree /main. To facilitate replication, have also provided detailed descriptions of our architecture, algorithms, hyperparameter settings, and experimental setup in Appendix H.
com/ Coom/LarenceRLi
7
ACKNOWLEDGMENTS
LY and LL are supported in part by NSF Grant 2221871. LY is also supported by an Amazon Faculty Award.
REFERENCES
Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprinl arXiv:2401.1002, 2024.
Vladimír Boža and Vladimír Macko. Two sparse matrices are better than one: Sparsifying neural networks with double sparse factorization. arXiv preprint arXiv:2409.18850 2024.
Haoxian Chen and Limin Wang. Rotpruner: Large language model pruning rotated space, 2025. net met/forumts forum? i d=wV9 www.Viiiyyy.. M
URL hww.hhtt:/ssetttc /opennevv///evevvieeeee
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, ai2 reasoning challenge. arXiv preprini arXiv: arXiv: 08057 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 1tt::2100.11166 2021.
Rocktim Jyoti Das, Mingjie Sun, Liqun Ma, and Zhiqiang Shen. Beyond size: How gradients shape pruning decisions in large language models. arXiv preprint arXiv:2311.04902. 2023.
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, and Matt Gardner. Documenting the english colossal clean crawled corpus. ArXiv, abs/2104.08758, 2021. URL https: / /tppi.ap.aa/////// http/ppp.aaaaaaaaaaa semant icscholar
Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. arXiv preprint arXiv:2406.0299 2024.
Abhimanyu Dubey, Abhinav Jaubri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. Maskllm: Learnable semi-structured sparsity for large language models. Advances in Neural Information Processing Systems, 37:7736-7758. 2024.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models be accurately pruned in one-shot. In International conference on machine learning, pp. 10323-10337. PMLR, 2023.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.1332. 2022.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https:/ /wweoooo..ooo wwwpo///w/r...oo/oooooo/oooooort 02608866//226203

---

Published as a conference paper at ICLR 2026
Tianteng Gu, Bei Liu, Bo Xiao, Ke Zeng, Jiacheng Liu, and Yanmin Qian. Denoiserotator: Enhance pruning robustness for Ilms via importance concentration. arXiv preprint arXiv:2505.23049, 2025.
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Muelllr, Sourab Man- grulkar, Mare Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made sim- ple, efficient and adaptable. https: : / thub. com/huge i nn face/ accce erate, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021b.
Yuezhou Hu, Kang Zhao, Weiyu Huang, Jianfei Chen, and Jun Zhu. Accelerating transformer pre- training with 2: 4 sparsity, arXiv preprint arXiv:240..01847, 2024.
Yichen Huang and Lin F Yang. Gemini 2.5 pro capable of winning gold at imo 2025, arXiv preprint arXiv:2507.15855, 2025.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980, 2014.
Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gho- lami. A fast post-training pruning framework for transformers. Advances in Neural Information Processing Systems, 35:24101-24116, 2022.
Xinlin Li, Osama Hanna, Christina Fragouli, and Suhas Diggavi. Icquant: Index coding enables low-bit IIm quantization. arXiv preprint arXiv:2505. 00850, 2025a.
Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengeheng He, Weizhu Chen, and Tuo Zhao Losparse: Structured compression of large language models based on low-rank and sparse ap- proximation. In International Conference on Machine Learning, pp. 20336-202000 20350. PMLR, 2023.
Zichong Li, Chen Liang, Zixuan Zhang, Ilgee Hong, Young Jin Kim, Weizhu Chen, and Tuo Zhao. Slimmoe: Structured compression of large moe models via expert slimming and distillation. arXiv preprint arXiv:2506. 18349, 2025b.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awan weight quantization for on-device Ilm compression and acceleration. Proceedings of machine learning and systems, 6:87-100, 2024.
Lawrence Liu, Inesh Chakrabarti, Yixiao Li, Mengdi Wang, Tuo Zhao, and Lin F Yang. Nowag: A unified framework for shape preserving compression of large language models. arXiv preprint arXiv:2504.14569 14569, 2025.
Lian Liu, Xiandong Zhao, Guanchen Li, Dong Li, Mengdi Wang, Yinhe Han, Xiaowei Li, et al. Bawa: Automatic optimizing pruning metric for large language models with balanced weight and activation. In Forty-second International Conference on Machine Learning.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702- 2023.
Xin Men, Mingyu Xu. Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.088. 003.03333 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
12

---

Published as a conference paper at ICLR 2026
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arxiv preprint 10XY1:2/04.08378, 2021.
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. Proceedings of the 36th annual acm symposium user interface software and technology, pp. 1-22, 2023,
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. In First Conference on Language Modeling, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99- 106, 2021.
Li Shen, Anke Tang, Yong Luo, Tao Sun, Han Hu, and Xiaochun Cao. Targeted low-rank refinement: Enhancing sparse language models with precision. In Forty-second International Conference on Machine Learning, 2025. URL https: wwwpereeeeeeweeeeooo/p/p. net/ forum? d=SOncZdwcI.t.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers. Jacob R Steeves, Joel Hes- tness, and Nolan Dey. plicated version of RedPajama. sl alimpajJi jama-a-aa------------aa- a- 62 2023. URL https: httgggggggaaaeeeeeeeeo /tt///ttteet//eerrrrr////eeeeee/
SlimPajama:
A 627B token cleaned and dedu-
https: http://wwwwebbbass.//e..e /
627B.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: / /wop/ppeeviieee net
3dWl.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint 1rX1v:2210.09221, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint wiXiv2230.00928% 2023.
Stephen J Wright. Coordinate descent algorithms. Mathematical programming, 151(():3-34, 2015.
Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. Flash-llm: Enabling cost cost-effective and highly-efficient large gen- erative model inference with unstructured sparsity. arXiv preprint arXiv:2309. 2023.
Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu. Moe-pruner: Pruning vixture-of-experr large language model using the hints from its router. arXiv preprint 24X 2024.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 0rXiv:2505.09388, 2025.
Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yo Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, et al. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning Ilms to high sparsity. arXiv preprint arXiv:2310.05175. 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? arXiv preprint
1905.07830 2019.
Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug- and-play An efficient post- training pruning method for large language models. 2024.
13

---

Published as a conference paper at ICLR 2026
A
NOTATION IN DEPTH
As established in section 3.1, for a block diagonal matrix D E Rdxd with block size dillckk, we denote the individual blocks as:
D 0
0
D
0
0
+bllokk
where D(6) E RR Edhinck A i E [1, d/dntock). We choose block diagonal wrappers over diagonal only because diagonal only offers no additional expressivity because commutes with the element-wisc pruning mask, mathematically:
diag (a) (W' diii (b) W" 3 M, A a R b Rdo.
(5)
where W diag (a) W' diag (b). In other words, Block-Diagonal matrices are minimal, hard- ware acceleratable, structure that enable flexibility, which diagonal only matrices do not.
More generally for matrix C Rddxd denote the dltock dolock matrix blocks as:
c( (lasttoe Aplock
C(.1)
((122)
==
Cl dalack .1)
C( dplack 2)
C Aptock dplack
B ARMOR OPTIMIZATION ALGORITHM IN DEPTH
In section B. elaborate the sparse update step more formally. Additionally provided pseudo code for both the sparse core update and the continuous parameter update steps in B.2.
B.1 THE SPARSE CORE UPDATE STEP IN DEPTH
Decomposing equation into independent subproblems at the block matrix level:
(0) = >> 1.5) ( =Σ
(r(kk) A(t) (With)) M(hh)) B() 2
F,diag( x x
(6)
where i E [dout/dbia [btock] and E [din/dblock] We optimize each block subproblem in parallel, for the remainder this section we will consider block (i.j). Optimizing these subproblems still requires sweep over exponential search space, we adopt greedy approach that optimizes single sparse group for each subproblem.
Optimizing the Sparse Group After selecting group V, k. we freeze all values beyond (W" @ 1). 2.[k] which we optimize with the 2:4 constraint. This is computationally feasible since we only have to sweep over (t) = possible mask choices. Let us consider one such mask m E {0, 1]4 s.t. ||m//o- = 2, which has unmasked indices i, 22 [4] ie m, = mi, = 1. Let us denote the unmasked elements for this mask WR? and &'' = 4(k 1). For this mask choice, the corresponding best-case proxy loss is given by:
(45 (m) min (1 - (0 BD) A w B
Wi
where ())).)) represents the frozen remainder of the sparse core:
0
in = +15+51ps V n. E [chtock] otherwise
( n.
( Wa.
(GJ)
14

---

Published as a conference paper at ICLR 2026
Equation 7 can be arranged into linear least squares problem with closed form solution. For ease of notation, let AW W((d) ( B B' B (6 a A and D(1 diag (XX+) 6), Solving the least squares problem results in optimal e
(B'DOAATTO) (BDD)) (BDDAAW)) |al12
f = |[Wl]
(8)
The first term is independent of the mask choice for the group, thus when sweeping the possible masks, we only compute the second term for efficiency. The optimum is achieved with unmasked weights w
BT) (BIDDAWTT)) |al
After sweeping over all 6 possible masks, we select the mask m². with nonzero values at i i2* that achieves the minimum best-case proxy loss as given by equation 8. We substitute the corresponding optimal wi. given by equation 9 into W to fully update sparse core, i.c. set (W-"(E.))
w1 and W ((..)) new
W2-
The 2:4 semi-structured sparsity allows for efficient calculation of equations 8 and 9. Since there
are only unmasked values for each group (B' DU' B"T) and (B' D() AWT a) are of shape 2 and 2 1. Thus the computational cost for solving for (B'D5)B') (B'D()) AWT a) are neg- ligible. Rather, calculating equations 8 and 9 are dominated by calculating (B'D() B"T) and B AWT a). which scale with d We update all dout / /atckk blocks at once, thus the overall computational complexity is Oodad)out) scaling linearly with parameter size.
B.2 ALGORITHM PSEUDOCODE
Algorithm Continuous Optimization via Sequential Gradient Descent
Require: Current parameters A, B, W', mask M, normalized weight W. calibration data X.
Ensure: Updated continuous parameters A, B, W'.
1: function CONINNOUSUUPDATEA, B, W M, XX)
2:
W < A(W" O M)B
3:
L - W F,
b Compute proxy loss from Eq.
4:
Update block diagonal matrix A
5:
< ComputeLcaminnRRat(AA.
>> Based on local smoothness, Eq. 10
6:
A
nA AL
7:
Update block diagonal matrix B
8:
dв + CompuceLeamingRate((
> Eq. 11
9:
B - B 1)8 V BL
10:
b Update underlying weight matrix W"
11:
Tp ComputeLearmingRate((/ L))
> Eq. 12
12:
W' + W TT V
return A, B, W
14:
end function
15

---

Published as a conference paper at ICLR 2026
Algorithm 3 Greedy Sparse Core Update
Require: Current parameters A, B, W mask M, normalized weight W, calibration data X. Ensure: Updated sparse core parameters W M.
1: function SPARSECOREUPDATHH, A, B, W" M,W,W
2:
for each block (i. in parallel do
3:
Select single 2:4 sparse group to update within the block
4:
for each sparse group (i', k) in block (i,j) do (i,f) 5: P(tt,) x
V (Mown) 1.
g(i5)
Calculate selection probability
6:
end for
7:
Select group kk" based on probabilities p(t.))
8:
Find the best mask and weights for the selected group Let be the of all ( possible masks for group of 4.
9:
10:
Thest f- OC
for each candidate mask m .M do Calculate the resulting loss it, using Eq. 8
12:
1. <. lhest then
14:
lest f- 1.
15:
m < 772 end if end for
16:
17:
18:
Calculate optimal weights w for the unmasked entries using Eq. 9 > Update the mask M and weights W" for the chosen group Update mask for group (**, in ( with m.
19:
20:
21:
Update corresponding weights in W/(E,) with w.
22:
end for
23:
return W W .M
24:
end function
C PROOFS
PROOF FOR PROPOSITION
Proposition The proxy loss is bounded below by and is convex with respect to A, B, and W individually. A formal proof is provided in Appendix C
Proof. The proxy loss is defined as:
Lw.x (0) !|W - W W XX7)
(( 1
where W- A(W' @ M)B.
1. Bounded Below by O: For each term in the summation, (W (W,-wW,)² > O it squared real number. The weighting term ||X is the squared Euclidean norm of vector, which is also negategae sum of negatege terms non -egatvvaevve. Thus, Lw.x (W) 2 0.
2. Convexity with respect to A: When B, W and M are held fixed, let S (W' @ M)B. The proxy loss function can be written as function of A:
Lw. (4) WW AS 2
This a weighted least squares objective. The function f(A) = W - AS an affine (linear) transformation of A. The function g(Z) = || l?. is a squared weighted Frobenius a convex function with an affine transformation is convex. Therefore, the proxy loss is convex with respect A.
where D
which is convex function. The composition of
16

---

Published as a conference paper at ICLR 2026
3. Convexity with respect to B: Similarly, when A, W and M held fixed, let S' A(W" @ M). The proxy loss function can be written as a function of B:
LB) - I|W BI
This is also a weighted least squares objective. Using the same reasoning as for A, the function is a composition of a convex function (squared norm) with an affine transformation B, and thus is convex with respect to B.
4. Convexity with respect to W' When A, B, and M are held fixed, the reconstructed weight matrix W = A(W" @ M) is linear function of the elements of W Let w' = vec (W),, the vector of elements in W Then vec(W) is a linear transformation of w'. The objective function (w.x(W) is quadratic function of the elements of W, and therefore a quadratic function of the elements of W'. Specifically, it is a positive semidefinite quadratic form, which is convex. Therefore, the proxy loss is convex with respect to W
This completes the proof.
PROOF FOR LEMMA C.I
Lemma C.1. The continuous parameter update step results equal or lower proxy loss:
Cw.x W ((A))41,(B) (B)+11, (W")++1poonl (M)+)) Cw.x (w ((A)s, (B)e, (W"),, (M)))
Proof. The paper states that the continuous optimization step updates the parameters A, B, and W using sequential gradient descent (Algorithm 2). The process is iterative:
1. Update A: (A)c+1 = (A); - nA V V.L., ((A),, (B),, (W"),, (M)+)
2. Update B: (B)e+1 (B), - 1/n L ((A)x+1, (B)r, (W") (M))
3. Update W [W")==1/oom (W"), ?w Ww ((A)++1, (B)++1, (W"),, (M))
From Proposition 1, the loss function Lw.x is convex with respect to each of A, B, and W indi- vidually. Algorithm 2 states, the learning rate 7) determined via local B-smoothness, which are calculated in D.
For convex and 3-smooth function (xx), the gradient descent update #1+1 = 2' - V/ (ax)) with step size 0 < S 1/B guarantees that ((=++1) < [(xx).
Applying this property to each step of the sequential update:
1. The update of A ensures that
Lw. ((A)441 (B),, (W").. (M)) s (w.x((A),, (W), (W"),, (M)))
2. The update of B, starting from the new A, ensures that
Lw,x ((A) (B)e+1, (W').. (M)+) & Lw.x((A))+). (B),,W") (W"),, (M)+)
3. The update of W starting from the new A and B, ensures that Lw,x ((A) (1) (W") (|/oont: (M)r (A)x (M)+)
Chaining these inequalities together, we get:
Lw.x Lw.x((A)(+1) (B)) (W") )e41 (M)x) S Lw.x((A) (B))))))(()))))))))))))))))))))( (W'),, < Lw. $w..x(AA)(x1 (B),, (W"), (M)) & ((Ax((A),. (B),, (W"), (M))
Thus, each continuous optimization step is guaranteed not increase the proxy loss.
17

---

Published as a conference paper at ICLR 2026
PROOF FOR LEMMA C.2
Lemma C.2. The sparse core update step results in a equal or lower proxy loss:
Lw;x (wr ((A))+1, (B)++1, (W")++1; (M))+1))))))))))) S Lw,x (W ((A)++I, (B)+11, (W") WW11oonn (M)+)
Proof. The discrete optimization step (Algorithm 3) seeks to improve the sparse core W @ M. The total proxy loss is decomposable into independent subproblems for each matrix block (i,j), as shown in Equation
daut dalockk dua dalock
(w) Σ
Σ
(Wb)) ((W' C ( dx
The algorithm updates a single 2:4 sparse group (i', k) within each block (i, Let's focus on one such update. Let the loss before the update for this specific group be laefore- The current mask for this group is mold, which is one of the (-) 6 possible masks.
The algorithm proceeds as follows:
1. For each of the 6 possible masks in the set of valid masks M, it calculates the opti- mal weights w that minimize the loss for that group, assuming that mask m is chosen (Equation 9). This results in the best possible loss it for that mask choice (Equation 8).
2. It then selects the mask m² that yields the minimum loss among all possibilities: lhest minmeM im-
3. The algorithm updates the mask for group (i', k) to m and its corresponding weights in W' -
The loss with the original mask mold and its weights before the update is fore- The optimized loss for this original mask, bimm must be less than or equal fores since Equation finds the optimal weights for any given mask.
E S lbefore
By definition, the selected mask m" is the one that minimizes this optimal loss across all 6 choices. Therefore, its loss best must be less than or equal to the optimal loss for the old mask 1 mass"
lestt L
Combining these, have leeat I lbe fore. The update for the selected group can only decrease or maintain the proxy loss. Since this holds for the update in each block, and the block losses are additive, the total proxy loss after the discrete optimization step is guaranteed to be less than equal to the loss before the step.
PROOF FOR THEOREM 3.1 (CONVERGENCE OF THE ARMOR OPTIMIZATION ALGORITHM)
Theorem 3.1. (Convergence of the ARMOR optimization algorithm). [Lw.x ((0)))20 converges and Lw.x ((0))) < Lw.x A t>0.
The sequence
Proof. Let L, = Cw.x((0), be the value of the proxy loss at the end iteration t. Each iteration of the ARMOR optimization algorithm consists of continuous optimization step followed by discrete optimization step.
From Lemma C.I, the continuous step does not increase the loss. Let the loss after the continuous step at iteration be Lz+1/oont We have:
L++1/cont S Lt
From Lemma C.2, the sparse core step, which follows the continuous step, also does not increase loss. Let the loss after the discrete step be L.+1- We have:
T441 T441lcont
18

---

Published as a conference paper at ICLR 2026
Combining these two results gives:
T++1 S T
This shows that the sequence of proxy losses (T4)220 is monotonically non-increasing. By induc- tion, this implies that for any t > 0, L S Lo-
Furthermore, from Proposition the proxy loss is bounded below by 0. Therefore, {L 1:20 is a monotonically non-increasing sequence that is bounded below. By the Monotone Convergence Theorem, any such sequence must converge to limit.
Therefore, the sequence (Cw:x((0):)) converges, and its value is always less than or equal to its initial value.
D BETA SMOOTHNESS OF PROXY LOSS
In this section we present the proofs that the proxy loss is ß-smooth with respect to each of the parameters A, B. W'. And using them, derive the learning rates for the sequential gradient descent based continous optimization step.
D.1 PROXY Loss 3-SMOOTH WITH RESPECT TO A
First let us consider the B-smoothness of the proxy loss for block (i.j) with respect to AQ, We have that:
((55) (D(W0)== = W//hh)) - A (id) o wubD)oll (xdia((XX7)0))
Where ((v,3) ((0,, W ((t..),,M...)) We have that:
vaco (B(:))) 2A (5t D(S) g(a.s)) 2w/(. D(I DU) glr.s
Where ((i.) = (M @ W(ld)) and DG) = diag (XXT) (1) Consdier two A(1), A(2) E Rduuc xduuck Let us denote 0(h.1) (A (i): BU), Wi(ls), M(d)) ((( and 0( (2)
(1)
(4()) (2), B0), Wils),, M...).. Then we have that: Then we have that:
V Vawelii (1:)) (0(1g)) v Vime (0 =2
=2| 1100) (G)
$2 163 F | DU1
Where the inequality follows from the submultiplicatiitty of the Frobenius norm. Thus we have that e the proxy loss (((-)) (g(++)) is f-smooth with respect to A with ((1-3) = =(il)( ( 5(1.) p- We now consider the overall proxy loss, for some A(1)- A(2) E Rduu xdin
Let 0())
(Ag), B, W", M)) and (2) (Az)) B, W M). Then have that:
dant
Jolock
V V)Cwx((())) &a
Ac ((35) (0(()))))))))
F
dost dblook
dislook
Atte (K-)) (0(())) - v efo
dunn Jhlock
Jolack
< $2 A2 Awall
(((,)) g(
19

---

Published as a conference paper at ICLR 2026
Where the first inequality follows from the triangle inequality and the second follows from the previous result. Therefore the overall loss Lw.x (f) is B-smooth with respect to A with BA =
Most
Ain Bllock
2 2 blookk
|| (()))( This leads us to a learning rate of:
1=
1==
71
BA
boutt bloock
Ain Jalack
(10)
2 2E dil
Σ _==1
D.2 PROXY LOss B-SMOOTH WITH RESPECT TO B
Once again, we start by considering the B-smoothness of the proxy loss for block (i,j) with respect to Bú), Reusing the notation we introduced in section, we have that:
V B (15) X (o(+))) 28 g(+-3 B D (5( wolld)
Where SM(E)) (M C w(ld)) and D = diag (XX7) (1) Consdier two B(1, B(2) E Reulect xdulack Let us denote (())) = (A(), BB]) Whlts), Mbx) and ((5))) =
(4(9, (2) Wilds), M(..)). Then we have that:
Bon (1.)) (0(1)))) B ((3) (101))) 2 ( (B -B(3) D || | - ( 1, ((O)
x (2)
F
2 S(LNTT
Where the inequality follows from the submultiplicatiiity of the Frobenius norm. Thus we have that the proxy loss (6.9) is 3-smooth with respect to B6) with (6.2) = stall
2 (( | F. Following the same procedure as the A case, consider some B() B(2) E Rdu Let i() = (A, B(), W .)) and 0(2) = (A, B(2), W W', Then we have that for the overall proxy loss:
Auat
Blook
Aolock
VDCW.x(0()) VBLw,x(92))11 42|BB() -Ba))l
Σ
SWi g(t.)) |
Therefore the overall loss (w) is 3-smooth with respect to with 3B
2 ②
dblock
dblock
g(o)))) | |()0] p- This leads to learning rate of:
==1
1
18
(11)
3B
dour Aotock
2 >
Mulock
|| |D6)
D.3 PROXY LOss 3-SMOOTH WITH RESPECT TO W
Finally, we consider the 3-smoothness of the proxy loss with respect to W". We have that:
Lw. (0) = (2 A(W" @ M)B (Xx*) B" 2A4 W Wiaa (XX*) B²) o M
Consdier two W B. (2), M). Then we have that:
(1): W
(2) RR xdin Let us denote 0(1) = B. M) and ( =
Lw,(2x Vww.w.x(0 2 (A" ((W)) - W(2) M) Bdiag (xx?) BT) о M
2 ATA (WW) - -W2)) o M) Bdiag (XXT) BT
ATT All Bediag (XXT) BTT (W - W/2) @ N
<
2 A |ATllll |Bdia (xx²) B² W/) WEn
20

---

Published as a conference paper at ICLR 2026
Where the first and third inequalities follow from the property that |V o M I [M max IIV F for any matrices V, M of the same shape, and the second follows from the submuliipliatiiitty of the Frobenius norm. Thus have that the proxy loss Lw,x x (0) is 3-smooth with respect to W with Bw 2 A All [diag (XXT) B") This leads to learning rate of:
1 Bw
TW"
2 1| All Bdiag (XXT) B² 1F
(12)
E ABLATIONS
E.I SELECTION HEURISTIC
We investigated four choices for the sparse group selection heuristic used in the Discrete Optimiza- tion steps:
Uniform Random Selection (Random). We select a sparse group from the available &boock/ / sparse groups within a sparse core block.
L.1 Greedy Selection, We directly select the sparse group based on which group has the maximum 1.1 gradient norm.
L2 Random Selection, We draw a sparse group with probability that is based on the L2 norm of the gradient, instead of the LI norm:
(i.j) k(e,k)
V (MMWW) 2.M
A 1' [dbtock],
k E [dbtook/ [lbboock/]]. /4].
L11 Random Selection, This the selection heuristic discussed the main text and that
we use.
The results of this ablation is listed in table 7. L1 random and 1.2 random perform roughly equiva- lently.
Wikitext 2 (1)
C4 (4)
Method
2-7B
2-13B
2-7B
2-13B
Random
8.17
7.05
10.49
9.48
1.1 Greedy
8.46
7.28
10.87
9.72
L2 Random
7.95
7.00
10.27
9.38
Random
7.99
6.99
10.30
9.39
Table C4 and Wikitext 2 perplexities of ARMOR pruned Llama-2-7B/133 for different sparse group selection heuristics. Block size of 128 was used, and ARMOR optimization was ran for 2000 iterations expedite runtime.
E.2 CALIBRATION DATASET
Dataset
Wikitext 2 (1)
C4 (4)
RedPajama-Data-
7.59
SlimPajama-622
7.68
9.88
Table 8: Perplexity of ARMOR pruned Llama-2-7B Wikitext2 and C4 Datasets when using RedPajama-Dater vs SlimPajama--62) 627B for our calibration dataset. ARMOR is minimally sensi- tive the choice of dataset, so long as is representative of the pre-training dataset. Block size of 128 was used, and ARMOR optimization was ran for 5000 iterations.
E.3 CALIBRATION DATASET SAMPLE SIZE
By default, we use 128 samples from the SlimPajama-633 627B dataset, with sample length 4096 for Llama-2 and 8192 for Llama-3 and Qwen models. This amounts 524.29K and 1.049M tokens
21

---

Published as a conference paper at ICLR 2026
respectively. In table 9, additionally evaluated the performance of ARMOR pruned Llama-2 7B using 16, 32, and 64 samples from SlimPajama-627B There is < 1% change both Wikitext and C4 perplexity across the entire range of sample sizes, indicating that ARMOR is highly data efficient.
Num Samples
Tokens
Wikitext (4)
C4 (4)
16
65.54K
7.61
9.86
32
131.07K
7.66
9.86
64
262.14K
7.63
9.86
128 (Default)
524.29K
7.66
9.93
Table 9: Wikitext 2 and C4 perplexities of ARMOR pruned Llama-2 7B using various numbers of
calibration dataset samples (sample length 4096). Block size of 128 was used, and 5000 iterations were used for each number samples
F EXTENSION TO MIXTURE OF EXPERT MODELS
ARMOR supports MoE models out-of-the-boo In Table 10, we applied ARMOR to the Qwen3- 30B-A3B MoE model and evaluated it MMLU, comparing against NoWag-P. Due time straints, we used a highly restricted setup: 1,000 iterations (vs. 20,000 for the main results presented in Tables 1,2, and 3) and block size 32 (vs. 128). We compared this against the NoWag-P baseline on the same model. To account for the sparse routing of MoE architectures, we increased the calibration dataset size 512 samples (vs. 128) for both ARMOR and the baseline ensure sufficient data coverage for all experts. ARMOR outperformed the NoWag-P baseline by +5.83% Furthermore, the degradation of ARMOR on this MoE model 8.7%) remarkably consistent with the degrada- tion we observe dense models (e.g.. 8.6% for Qwen-2.5-7B in Table 1). This indicates that the ARMOR factorization and optimization strategy is robust across both Dense and Mixture-of-Fxpene architectures.
Method
MMLU (%) (t)
Gap (L)
Dense
81.16
NoWag-P -P
66.67
14.5%
ARMOR
72.50
8.67 %
Table 10: ARMOR and NoWag 2:4 pruned Dwen3-30B-an MoE model performance on MMLU. Block size 32 and 1000 iterations were used for ARMOR
G FEW SHOT TASK DESCRIPTIONS
ARMOR prund Qwen2.5 and Qwen models were evaluated shot task benchmarks. The results were reported in Tables 2 and 1. detailed description of each task is included below. HuggingFace accelerate (Gugger et al., 2022) used for parallel process- ing, and the EleutherAl LM evaluation harness was used (Gao et al., 2024). max 4096 for all benchmarks for speedup.
industry standard suite of 7 few
max_mode sot__moell. len
1. MMLU (Hendrycks et al., 2021b;a) - The Massive Multitask Language Understanding (MMLU) benchmark is 57-subject multiple-choice benchmark spanning STEM, human- ities, social sciences, and professional domains that evaluates broad knowledge and basic reasoning. We report the 5-shot accuracy. (acc, none).
2. GSM8K (Cobbe et al., 2021) - Grade School Math 8K (GSM8K) dataset of 8.5K high quality linguistically diverse grade school math word problems. Problems require concepts beyond the level of early Algebra, and solutions are provided in natural language. We report the 8-shot strict match: (exa coract_match,,. strict strict-matcc).
22

---

Published as a conference paper at ICLR 2026
3. ARC-e (Clark et al., 2018) The challenge subset of the A12 Reasoning Challenge (ARC), which is composed of grade-school science questions authored for human tests. The challenge subset is composed of questions that baseline algorithms have failed on. We report the 25-shot accuracy. (acc, none).
4. HellaSwag (Zellers et al., 2019) - The HellaSwag challenge dataset has models choose the most commonsense continuation of everyday scenarios among four options. The bench- mark intentionally challenging for models. We report the O-shot acurracy acc, none.
5. BBH (Suzgun et al., 2022) BIG-Bench Hard (BBH) is suite of 23 especially chal- lenging tasks from the Beyond the Imitation Game benchmark (BIG-Bench). These tasks require multi-step reasoning and compositional generalization. We report the shot match (exact_match, get answer).
6. WinoGrande (Sakaguchi et al., 2021) WinoGrande is large dataset of 44k problems inspired by those of the Winograd Schema Challenge, systematically designed to mini- mize bias. The questions are of 2-choice fill in the blank format that tests commonsense reasoning. report the 5-shot accuracy (acc, none).
7. GPQA (Main Set) (Rein et al., 2024) The Graduate-Level Google-Proof Q&A bench- mark (GPQA) main set is a multiple-choice benchmark of 448 graduate-level biology, physics, and chemistry questions authored by experts. The main excludes questions that are likely to be not objective and difficult. We report the 5-shot accuracy (acc, none).
H IMPLEMENTATION DETAILS
We used block size of 128 for all models. For optimization, we used learning rate of 10-1 for ADAM, and ran for 20,000 iterations. We used 128 samples of the SlimPajama-6277 dataset (Soboleva et al., 2023) as calibration dataset. Each sample had context length of 4096 for Llama-2 family models, and 8192 for Llama-3, Qwen 2.5, and Qwen family models. Inference benchmarks were performed 48GB 4090.
I
LLM USAGE
We used LLMs to polish and refine language. Some icons, such as the robot in Figure 1 and the fire and snowflake icon in Figure 2 were generated with generative models. Furthermore, we used LLMs to conduct some literature review to supplement and double check manual literature review.
23


Reviews:
[{"review_id": "r15ZQqXN5U", "reviewer": "Reviewer_HqUF", "summary": "The paper proposes **ARMOR**, a one-shot, *semi-structured* pruning method for N:M (notably **2:4**) that re-parameterizes each weight matrix as\n\\[\n\\hat W \\;=\\; A \\; (W' \\odot M) \\; B,\n\\]\nwhere \\(M\\) is a hardware-friendly **2:4** mask, and \\(A,B\\) are block-diagonal wrappers that act as lightweight, learnable pre-/post-transformations to **correct pruning error. The optimization alternates between: (i) continuous updates of \\(A,B,W'\\) under a NoWag-style proxy loss, and (ii) a greedy sparse-core update that exhaustively selects the best 2-of-4 configuration per group via small least-squares solves. The authors prove monotone convergence of the proxy loss and initialize at **NoWag-P**, guaranteeing a proxy loss no worse than that baseline. Empirically, on Qwen/Llama families, ARMOR reports **consistent accuracy gains over state-of-the-art **2:4** methods (Wanda / SparseGPT / NoWag-P) and provides inference measurements showing it largely retains 2:4 speed/memory benefits despite wrapper overhead.", "questions": "1. **Where exactly is the runtime gap vs. 2:4?** Can wrapper fusion (e.g., pre-/post-accumulation, tiling into the 2:4 kernel) close the 1.57× vs 1.86× gap? \n2. **Post-training recovery.** Have you tried small **LoRA/SFT** passes after ARMOR to close the accuracy gap without losing efficiency?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Theoretically supported and rigorous.** Clear proxy-loss formulation, greedy 2:4 subproblem with closed-form LS updates, and a convergence theorem (monotone decrease of proxy loss; initialization at NoWag-P).\n- **Strong accuracy vs. 2:4 baselines.** On Qwen/Llama, ARMOR consistently outperforms Wanda / SparseGPT / NoWag-P under 2:4 across PPL and downstream tasks; in some settings it narrows the dense–pruned gap by ~50%. \n- **Insightful idea for future work.** Reframing pruning as adaptive factorization (sparse core + learnable wrappers) suggests a promising path for co-design—optimizing both representational form and hardware mapping, not only masks.", "weaknesses": "1. **Speedups are modest vs. expectations.** Reported generation speedups are **~1.041×–1.141×** on Qwen-2.5-14B/7B, and **batched MatVec** shows **1.57×** vs **1.86×** for plain 2:4—indicating nontrivial overheads or kernel gaps for wrappers. This is **far below** what a dense model with 50% fewer parameters could achieve, and below the **best-case 2:4** micro-kernel gains. A detailed **profile** of where time is spent (wrapper GEMMs, cache/bandwidth, kernel launch) is needed. :contentReference[oaicite:7]{index=7}  \n2. **Performance degration is significant compared to dense model and far below equal-parameter dense counterparts.** For example, ARMOR-pruned Qwen-2.5-14B can underperform Qwen-2.5-7B on some tasks; the absolute accuracy gap to dense remains notable in places, limiting immediate deployability. Consider exploring targeted post-training (e.g., LoRA or small SFT) to recover quality while preserving efficiency.\n3. **Efficiency gap vs. plain 2:4 in microbenches.** The 1.57× batched MatVec vs 1.86× for 2:4 suggests wrapper-cost or fusion inefficiency. Without kernel-level fusion/tiling tailored to (block-diag × 2:4 × block-diag), ARMOR may remain below the Pareto frontier on pure efficiency. More benchmarks are needed to claim a better performance–efficiency trade-off overall. \n4. **Tied strongly to NoWag-P and 2:4.** While the method is framed generally, in practice it heavily relies on the NoWag proxy + NoWag-P init and target **2:4**. Please clarify **algorithmic novelty** beyond NoWag-P, and quantify how much of the gain derives from more effective parameterization rather than more effective parameter count.\n5. **Limited exploration.** It remains unclear whether combining ARMOR with structured pruning (e.g., head/column removal) and quantization (e.g., AWQ/GPTQ) yields **practical** end-to-end speedups while keeping quality. A hybrid study would strengthen deployability claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6G20UtNKBW", "reviewer": "Reviewer_4fMQ", "summary": "The paper proposes ARMOR, a one-shot 2:4 semi-structured pruning method for large language models. Instead of directly pruning weights, each matrix is factorized into a 2:4 sparse core surrounded by two block-diagonal “wrapper” matrices that act as lightweight error correctors. The method is optimized via a block coordinate descent alternating between gradient updates for the wrappers and greedy least-squares updates for the sparse core. ARMOR achieves large accuracy gains over prior 2:4 pruning methods (Wanda, SparseGPT, NoWag-P) while keeping almost identical inference speedups and memory savings. This is an excellent paper that combines a simple yet powerful idea with strong empirical validation and theoretical rigor. It clearly advances the state of the art in semi-structured pruning, a practically important domain for LLM deployment.", "questions": "1. How does the performance of ARMOR change with smaller calibration datasets or fewer proxy-loss iterations (e.g., 5k vs 20k)? Is the method robust to low-data calibration settings?\n2. Could the block-diagonal wrappers be fine-tuned jointly with the model in a lightweight manner (e.g., LoRA-style), further improving performance?\n3. Have you explored extending the framework to N:M patterns beyond 2:4? Since the design is general, it may adapt naturally.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Recasts semi-structured pruning as a matrix-factorization problem instead of a masking problem, providing a new theoretical and practical perspective. The idea of block-diagonal wrappers acting as error-correcting transformations is elegant and hardware-friendly.\n2. Consistent, substantial improvements over multiple strong baselines (Wanda, SparseGPT, NoWag-P). Performance nearly matches dense models on some reasoning benchmarks.\n3. Provides a convergence guarantee showing that ARMOR’s proxy loss never exceeds that of the initialization (NoWag-P). Proxy loss design is interpretable and data-aware (weighted Frobenius norm).\n4. Includes both perplexity and task-based evaluation, block-size ablation, and correlation between proxy loss and perplexity. Shows convincing analysis of trade-offs between accuracy, overhead, and speed.", "weaknesses": "1. Although the proxy loss correlates with perplexity, a stronger justification (e.g., empirical correlation coefficients across models) would improve interpretability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ex9fdFy6Sw", "reviewer": "Reviewer_Ze6L", "summary": "The paper proposes ARMOR, a post-training semi-structured pruning method for LLMs that factorizes each weight matrix into a 2:4 sparse core surrounded by block-diagonal transformation matrices (A and B) to mitigate accuracy loss.", "questions": "1. Does the proposed method support other models, such as MoE models? Does ARMOR degrade on them?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The factorization approach is a novel way to add flexibility to rigid 2:4 pruning, potentially bridging the gap between unstructured and semi-structured methods.\n2. The theoretical analysis in Appendix C is sound.", "weaknesses": "1. The proposed ARMOR method does not support unstructured pruning. It is specifically designed for semi-structured pruning (with a focus on 2:4 sparsity patterns), as the core optimization algorithm relies on the constrained group structure of semi-structured masks to remain computationally tractable. However, baseline methods such as Wanda support unstructured pruning.\n2. The related work is sufficient. There are other refinement methods that also mitigate the gap between a dense model and a pruned model. For example, low-rank refinement [1, 2] and adaptive layer-wise sparsity control [3].\n3. Minor concern: evaluations are restricted to dense models.\n\n[1] Y. Li et al., “LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation,” 2023.  \n[2] L. Shen, A. Tang, Y. Luo, T. Sun, H. Hu, and X. Cao, “Targeted Low-rank Refinement: Enhancing Sparse Language Model with Precision,”  2025.   \n[3] L. Yin et al., “Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity,” 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VIlm53wuRu", "reviewer": "Reviewer_G27K", "summary": "In this paper, the authors address the trade-off between hardware acceleration and performance degradation in semi-structured pruning. They propose an adaptive matrix factorization framework that decomposes the original weight matrix W into three components (A, W’xM, B). Then they employ a block coordinate descent algorithm to iteratively optimize these parameters. Experimental results on LLaMA and Qwen demonstrate the effectiveness of the proposed approach compared to existing methods. The inference analysis further indicates that the method introduces a slight computational overhead relative to standard 2:4 pruning. Moreover, it provides a theoretical proof to guarantee the algorithm’s convergence.\n\nOverall, the paper presents a decomposable and learnable 2:4 pruning scheme for LLMs, offering a meaningful alternative for LLM compression research. Therefore, I lean to marginally accept this paper, and I would be willing to raise my score if the authors address my main concerns during the rebuttal phase.", "questions": "1. Comparison with SOTA methods:\nCan the authors include results comparing their approach with recent learnable semi-structured pruning methods such as MaskLLM [1], RotPruner [2], and DenoiseRotator [3]? This would help clarify whether the proposed learnable transformation provides measurable benefits beyond these baselines.\n\n2. Generality across N:M patterns:\nDoes the proposed method extend naturally to other N:M pruning settings (e.g., 4:8, 5:8, or 6:8)? If so, how should hyperparameters like block size and iteration count be adjusted?\n\n3. Structure of transformation matrices:\nWhat is the rationale behind adopting a block-diagonal structure for matrices A and B? Would using a pure diagonal structure simplify computation while maintaining comparable performance?\n\n4. Inference efficiency:\nThe observed inference speedup for 2:4 sparsity is much smaller than the theoretical 2× expectation. Are there software or hardware factors limiting this gain? Could kernel fusion or custom sparse-GEMM implementations improve it?\n\nMinor issue:\nPlease correct the syntactic error at lines 78–79 (“...speedups is difficult”).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\nThis paper proposes a decomposable and learnable 2:4 pruning scheme for large language models (LLMs), which is both novel and valuable for LLM compression research. It provides a promising approach to enhance the performance of 2:4 pruning while preserving practical inference efficiency.\n\nQuality:\nThe paper is well-written and well-organized, presenting its ideas in a coherent and logical manner.\n\nClarity:\nThe content is clearly presented, and the mathematical formulations are easy to read and follow.\n\nSignificance:\nThe study introduces an alternative approach to improving the performance of 2:4 pruning for LLMs, making it a meaningful contribution toward efficient and practical model deployment.", "weaknesses": "Main concerns:\n1. The paper’s main weakness lies in limited novelty relative to existing learnable semi-structured pruning frameworks. The proposed method introduces learnable transformations (A, W' B) to improve 2:4 LLM pruning, but this concept is similar to prior learnable pruning works such as MaskLLM [1], RotPruner [2], and DenoiseRotator [3], which also learn rotation or masking matrices to enhance pruning robustness and recovery. To convincingly demonstrate novelty, the authors should explicitly position their method against these approaches—highlighting what differentiates their transformation learning (e.g., new optimization formulation, stability, or efficiency aspects).\n\n2. The main tables compare against SparseGPT, Wanda, and NoWag, which are not the current state-of-the-art methods for semi-structured pruning. Without comparisons to MaskLLM, RotPruner, and DenoiseRotator, it is difficult to assess the real performance improvement or contribution of the proposed method.\n\n3. It lacks insufficient explanation of why a block-diagonal structure is preferred over a simple diagonal form for A and B matrix.\n\n4. the inference acceleration results appear underwhelming. For 2:4 sparsity, the theoretical speedup ratio should ideally approach 2×, but the reported gain is marginal. This suggests the need for more detailed discussion or optimization strategies to bridge the gap between theoretical sparsity and actual hardware acceleration.\n\nMinor comments:\n1. Line 78-79 syntax error, “Thus it is difficult to translate the theoretical model size reductions into practical inference speedups is difficult”\n\n\n[1] MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models. NeurIPS 2024.  \n[2] RotPruner: Large Language Model Pruning in Rotated Space.  \n[3] DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration. NeurIPS 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)