Learning Semi-Structured Sparsity for LLMs via Shared and Context-Aware Hypernetwork
Authors: Lu Sun
Jun Sakuma
Venue: ICLR 2026 Poster (2026)
Keywords: LLM pruning
semi-structured sparsity
hypernetwork
continual learning
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose a resource-efficient framework, HyperPrune, that uses a lightweight hypernetwork to create structured sparse masks for large language models, achieving a good balance between accuracy and efficiency.

Abstract: Large Language Models (LLMs) achieve state-of-the-art performance but are costly to deploy in resource-constrained environments. Pruning with $n:m$ semi-structured sparsity reduces computation and enables hardware acceleration, yet existing methods face a trade-off: one-shot approaches are efficient but heuristic, while optimization-based methods are accurate but expensive.  
We introduce \textbf{HyperPrune}, a resource-efficient framework that directly optimizes $n:m$ sparsity. A lightweight hypernetwork, shared across layers and conditioned on learnable embeddings, generates structured masks in a one-shot, layer-wise manner. \textit{Continual pruning} preserves cross-layer knowledge, and \textit{feature outlier regularization} retains critical activations, unifying the strengths of heuristic and optimization-based methods.  
Experiments on LLaMA-7B to 70B show state-of-the-art accuracy–sparsity trade-offs on a single A100 GPU, achieving higher efficiency, accuracy, and scalability than prior approaches. HyperPrune offers a practical, scalable, and hardware-friendly solution for structured LLM pruning.

Published as a conference paper at ICLR 2026
LEARNING SEMI-STRUCTURED SPARSITY FOR LLMs VIA SHARED AND CONTEXT-AWARE HYPERNETWORK
Lu Sun Jun Sakuma
1 Center for Advanced Intelligence Project (AIP), RIKEN, Tokyo 103-0027, Japan School of Computing. Institute of Science Tokyo, Tokyo 152-8550, Japan
& UUnprecedented-scall Data Analytics Center, Tohoku University, Sendai 980-0845, Japan 1u. sun. e@tohhok. ac. jp, sakumaěc. wwwett.ee.........
ABSTRACT
Large Language Models (LLMs) achieve state-of-the-art performance but are costly to deploy in resource-constrained environments. Pruning with n m semi- structured sparsity reduces computation and enables hardware acceleration, yet existing methods face trade-off: one-shot approaches are efficient but heuristic. while optimization basedd methods are accurate but expensive. We introduce Hy- perPrune, eesturce-efficieent framework that directly optimizes n : m sparsity. lightweight hypernetwork, shared across layers and conditioned on learnable embeddings, generates structured masks in one-shot, layer-wise wise manner. Con- tinual pruning preserves cross-layer knowledge, and feature outlier regularization retains critical activations, unifying the strengths of heuristic and optimization- based methods. Experiments on LLaMA-7B to 70B show state-of-the-art accu- racy-sparsity trade-offs single A100 GPU, achieving higher efficiency, accu- racy, and scalability than prior approaches. HyperPrune offers a practical, scal- able, and frrewale-friendly solution for structured LLM pruning.
INTRODUCTION
Large Language Models (LLMs) have transformed artificial intelligence, advancing state-of-the-arr performance in language understanding, reasoning, generation, and diverse applications (Brown al. 2020 OpenAl 2023 Grattafioni al. 2024). Models such as GPT-4 (OpenAI 2023), LLaMA (Touvron et al., 2023a). and OPT (Zhang et al. 2022). with billions to trillions of pa- rameters, excel in translation, multi-step reasoning, and creative generation. However, their massive scale hinders deployment in cooource-constrainee. environments: for example, LLaMA-70BB requires over 130 GB of GPU memory, exceeding the limits of edge devices, mobile platforms, or embedded
systems CCa et al. 2020 Frantar & Alistarh 2023 Dettmers et al. 2023 Cheng al. 2024). This
bottleneck motivates efficient compression and acceleration techniques.
To address LLM scaling challenges, the community has developed diverse compression techniques, including quantization (Dettmers et al.) 2023] Frantar et al.) (2022a), knowledge distillation (Hin-
ton et al. 2015; Jiao et al. (222)). low-rank approximation (Hu et al. 2021 Xu et al. 2024a) and
pruning FFrantar et al. 2022b Dong et al. 2024). Among these, pruning is particularly attractive: preserves the original architecture, offers fin-grriined interpretability, complements quantization, and is increasingly supported by specialized hardware. Recent NVIDIA GPUs (e.g., A100, H100) natively support n m structured sparsity, enabling up 2x throughput gains matrix multipli- cations, lower memory usage, and accuracy comparable to dense models (Hubara et al. 2021 Zhou al., [021] Fang et al., 2024) Sun al. 2025). Empirical evidence further reinforces pruning: over peeeeaaramtee networks often generalize as well or better when pruned (Reed) 1993 Arora
al.] 2018); compressing a pretrained dense model outperforms training sparse from scratch (Zhu & Gupta 2017); and many layers tolerate aggressive pruning without performance loss (Wang al. 2018 These factors make pruning a practical and powerful tool for LLM compression.
WWok done while RIKEN ATP. Current affiliation: Tohoku University.

---

Published as a conference paper at ICLR 2026
LosS
W
X
FFN
1
Layer
11
Xt-1
htg
MHSA
Continual Pruning
Layer l
Regularized
Hypernetwort
Embeddings
X/-1
Loss 1
Context
Layer 1 -1
X1-2
FFN
x
M O
Layer
e1-1
ha
Xg
MHSA
W
t
LLM
Figure 1: The HyperPrune Framework. HyperPrune learns semi-structured sparse patterns for LLM weight matrices, illustrated here with 2:4 sparsity. prunes matrices (W) across components (e.g., Q. K, V, projections) in layer-wise manner. For each layer f, shared hypernetwork ho generates a binary mask Mr, producing sparse weights W, We @ M,. These are optimized via the layer loss Loss,, which is then minimized by optimizing 0. The hypernetwork is context-aware through layer (eş) and component (t) embeddings, and further regularized by feature-outlier and continual prruning objectives to retain important features and prevent catastrophic forgetting.
Yet pruning for LLMs remains nascent. Early iterative prune-retrain methods (Han et al. 2015 Louizos et al., 2018) infeasible for today's billion-parameter models, prompting
efficient strategies. One-shot heuristics such as SparseGPT (Frantar & Alistarh (002)) and Wanda (Sun et al.) 2024) derive sparsity masks from small calibration sets but suffer from reconstruction loss, poor scalability, and lack of native n m support. Optimization based approaches like MaskLLM (Fang al. 2024) achieve higher accuracy by learning n m masks on larger calibration sets, but require massive resources (e.g.. thousands of GPU hours for I.LaMA-2 (Touvron et al. (023b)). Recent methods such as MaskPro (Sun et al.) 2025) reduce costs but still struggle with policy gradient variance and linearly increased memory size. Overall, current pruning approaches face two key challenges: (i) an accuracy-efficiency trade-off, where heuristics degrade high sparsity and op- timization methods are costly; and (ii) lack of efficient, direct
support. Since hardware accelerators benefit most from n m sparsity, an effective framework must enforce such structure efficiently, within realistic budgets, without sacrificing accuracy.
In this work, we present HyperPrune, a resource-efficiemt optimization based framework for n : m semi-structured pruning of L.MMs. While heuristic methods efficient, they cannot guarantee n m sparsity, and existing optimization- approaches enforce at high computational cost for billion-scale models. HyperPrune overcomes these challenges with a shared, lightweight hyper- network he that generates layer- and component-specific masks M, conditioned on context-aware embeddings e and t, producing sparse weights We M,, as illustrated in Fig. This design enables adaptive pruning across layers and components, promotes cross-layer knowledge sharing, mitigates catastrophic forgetting, and scales efficiently large LLMs. Crucially, we show theo- retically that minimizing the reconstruction loss of pruned layers is equivalent to maximizing the mutual information between dense and pruned models under the n:m constraint (Theorem [ pro- viding principled relaxation of discrete mask optimization. Combined with fcature-outlier and continual- parnnln-runng regularizations, HyperPrune preserves critical activations and ensures stable se- quential pruning. Together, the context-aware hypernetwork, information-thes novel regularizations offer an efficient, scalable, and principled approach to structured LLM pruning. Experiments demonstrate strong accuracy securacy-sparsity trade-offs models from 7B to 70B parameters, all on a single 1000 GPU (80GB).
foundation, and
Our main contributions are:
2

---

Published as a conference paper at ICLR 2026
Hypernetwork-basedd structured pruning. We propose HyperPrune, the first optimization-bant framework that generates n m sparse masks using a shared, context- aware hypemeetwrr, enabling adaptive and layer- structured pruning across LLM components.
cntrmmttion-hhoreetii foundation for n m sparsity. We show that maximizing mutual information under m constraints is approximately equivalent minimizing recon- struction loss via a differentiable relaxation (Theorem [T, providing the first information- theoretic justification for structured mask learning.
Regularization for feature and knowledge preservation. We introduce feature outlier regularization to preserve critical activations and continual pruning regularization to miti- gate catastrophic forgetting during sequential layer-wise pruning.
Scalable and practical pruning. HyperPrune scales to LLMs up to 70B parameters a single A 100 GPU and consistently outperforms both heuristic and optimization-based baselines on zero-shot and generation benchmarks.
RELATED WORK
LLM Pruning. Pruning reduces redundant parameters while aiming to preserve performance, of- ten motivated by the Lottery Ticket Hypothesis (Frankle & Carbin] 2018 2019). Recent L.L. prun- ing methods fall into two main categories: One-shot, heuristic methods. Magnitude pruning (Han) cet al. 2015)) removes the smallest weights and remains strong baseline. SparseGPT (Frantar & and Wanda (Sun et al. 2024) prune in single pass using layer-wise reconstruction
magnitude heuristics, reaching 50% sparsity without retraining but degrading higher levels.
Extensions include OWL (Yin et al., 2024) for outlier weights, ShortGT (Men et al. 2024) and
AdaInfer (Fan et al.) 2024) for layer skipping, and LaCo (Yang et al. 2024b) and SliceGPT (Ashk-
boos et al. 2024) for embedding compression. Pruner-Zero (Dong et al. 2024) evolves symbolic pruning metries via genetic programming, while RIA (Zhang al. (2024) leverages relative impor- tance, activations, and channel permutation to outperform heuristics. Optimization-bant
meth-
ods. ALPS (Meng et al. 2024) enforces unstructured sparsity with ADMM (Boyd et al. (011). and BESA (Xu et al.] 2024b) differentiably allocates layer- wise sparsity. ShearedLLMM (Xia et al.) 2024) prunes efficiently at scale, while LoSA (Huang et al. 2025) integrates pruning with LoRA fine-tuning. These methods achieve higher accuracy under extreme unstructured sparsity but re- quire thousands GPU hours on 7B-70B models and do not natively support n m structured sparsity. MaskLLM (Fang et al. 2024 enforces strict n patterns via probabilistic masking but incurs prohibitive memory and computational overhead due to its large mask parameters. MaskPro (Sun et al. 2025) mitigates some of this cost with linear-space probabilistic formulation, yet it still suffers from high-variance gradients and memory usage that grows linearly with model size. In contrast, heuristic methods are efficient but brittle, while existing optimization-base approaches are accurate yet resource-intensive and often cannot scale to large LLMs with : m sparsity. Our proposed HyperPrune overcomes these limitations through layer-wise optimization with a shared, context-awarr hypernetwork, whose memory footprint depends only on individual layer size rather than the full model. This design reduces both memory and compute demands while retaining high pruning quality and scalability.
Seternetworr-Baseer Methods. Hypernetworks (Ha et al. 2017 7 Chauhan et al. 2024 generate parameters for target network, enabling continual learning (Wang et al. 2024). few-shot adap- tation (Wang et al., 2020), and multi-task modeling (Zhang & Yang. 2022). By conditioning on embeddings. they can produce task-specific weights, masks, or adapters. Examples include Hy- perShot (Sendera et al. 2023) and HyperTransformer (Karimi Mahabadi et al.) 202T) for few-shot learning, and Polyhistor (Liu et al. 2022) for multi-task vision-language tasks. Continual learning approaches (von Oswald et al. (0200 use low-dinensional task embeddings to compress knowledge and transfer across tasks. Hypernetworks have also been used for compression: LoRA (Hu et al.) 2021) and Text-to-LoRA (Charakorn et al. (2025) generate low-rank adapters, HyperMask (Ksiazek) & Spurek 2025) learns binary masks, HyperLogic (Yang et al. 2024a) generates logical rules, and Bayesian Krueger et al., 2017) or noise-modulating (Eyring et al. 2025) hypemnetorrks improve robustness or efficiency. Despite their versatility, hypernetworks have not been applied to LLM pruning due to the enormous output space, structured sparsity requirements, and heterogeneous lay-
3

---

Published as a conference paper at ICLR 2026
HyperPrune overcomes these challenges by generating n masks for each group of m consecutive weights using a shared hypemnetorr,, reducing the effective output space from the full weight matrix to size m, while conditioning layer- and component-specifee embeddings to handle heterogeneous layers efficiently.
PRELIMINARIES
Consider LLM parameterized by W. The goal of pruning is to learn binary mask M of the same dimension, where Mij = 0 removes the corresponding parameter and M, = 1 keeps it. Given an input x from calibration set D, the standard pruning objective is:
min Ex~D (x W @ M)] s.t. |M < C. M
where LLLM is the LLM loss and the to constraint enforces sparsity and @ denotes the element-wiss product. Despite its simplicity, Eq. i intractable at LLM scale due to the combinatorial &o con- straint, and it yields unstructured sparsity that offers little acceleration on modern GPUs (Sun al, 2024 Fang et al. 2024). Two main directions address these issues: heuristic one-shot pruning and optimization-basee pruning.
One-Shot, Heuristic Pruning. Instead of learning a global mask, one-shot methods prune by directly optimizing layer-specific mask M, minimize the reconstruction enror:
min Ex M2
W WWxe-11 - (We @ M,)xx-11??
s.t. ||Mlll <C,
where 1 denotes the input to the ε-th layer Various score-based heuristics (Frantar & Alistarh, 2023 Sun et al., 2024) approximate this NP-hard problem using a small calibration set. However, the resulting masks are usually unstructured, incompatible with hardware-friene nam sparsity. Enforcing group-wise sparsity heuristically approximates n:m but often incurs accuracy loss.
Optimization-basee based Pruning. Optimization based methods enforce n.m semi-structures sparsity by directly constraining each group of m consecutive weights to retain exactly n nonzero entries. Reformulating Eq. [1 the objective becomes
min (M, [M EXND [LLM(X; [ @ M.))..
where a weight matrix W E dgd. xd2 (superscript e and subscript é omitted for clarity) partitioned row-wise into did2 disjoint groups of consecutive elements. Each group W; is paired with
binary mask M, and the collection MM, E Smmm 1did//m specifies masks for all groups with constraints M, E Sremm Vi. Here, Srem denotes the set of valid nom sparsity patterns. For example, in the widely adopted 2:4 case, the feasible mask set is
[i=1
S2:4 {2,,82,--- ... 56} ([1, 1, 0), [1, 0, 1, [1. 0,0,0 1], 1, 1, 0j. 00 1, 0, 1), 0, 1]] (4) In general, | (*)) (e.g.. 2 6. |
70). To make the discrete optimization in Eq. 4 tractable, Fang et 2024): Sun et al. (2025) sample masks from a relaxed categorical distribution over STEm and apply the Gumbel-Softma trick (Jang et al., 2017] Maddison et al.) 2017) for differentiability, enabling more effective scaling large calibration sets D than heuristics.
METHODOLOGGY
Heuristic pruning is efficient but cannot guarantee : m structured masks, while optimization-basse methods enforce n m sparsity but are costly for billion-scale models. We propose HyperPrune, framework for efficient structured pruning of LLMs. HyperPrune uses lightweight shared hyper- network with contextual embeddings generate m masks sequentially, incorporates continual pruning to preserve cross-layer knowledge, and applies feature-outlier regularization to retain crit- ical weights. We further enable end-to-end optimization via continuous relaxation and improve efficiency and generalization through
transfer.
SStrictly speaking, superscript e should be used for both X and W to indicate the component E C. Here, C {g. k, v, o, u, d. g} refers to the set of Transformer subcomponents, where k o correspond to multi-head self-attention (MHSA) and d, g correspond to the feed-forward network (FFN). For readability, omit these superscripts throughout the paper.
4

---

Published as a conference paper at ICLR 2026
4.1 EFFICIENT LAYER-WISS MASK OPTIMIZATION VIA HYPERNETWORK
Layer-wise Mask Optimization. Unlike one-shot pruning in Eq. that relies on heuristic scores or global methods in Eq. 3 optimize masks for the entire model, we adopt a layer-wise strategy. For each Transformer layer l E [74] with function f. the objective is:
min (Me.) E8 ES Ex-D [I xx-1) f((W Medh x
(5)
where {M J is the set of groups (defined in Eq. of the &-th layer mask Mr. This approach scales under resource constraints, only the current layer's weights and activations need GPU memory, and it provides a stronger objective than one-shot pruning by optimizing at the layer rather than component level. Two challenges remain: (i) pruning layers independently may lose cross-layer knowledge, and (ii) directly optimizing large binary masks M2 is infeasible for modern L.MMs for instance, a single FFN projection in LLaMA--7BB contains over 1.7B parameters, making naive layer-wise optimization impractical without additional parameterization or compression.
Shared Hypernetwork for m Sparsity. To overcome the limitations of Eq. rather than directly learning independent laser-wise masks Mr, HyperPrune employs a lightweight shared hy- structured masks across all layers, enhancing both memory efficiency
pernetwork to generate and cross-layer knowledge transfer. For a Transformer layer & with weight matrix We, and under the grouping scheme of Eq. 3] the i-th group mask Me,; sampled from its corresponding weight group We, via the shared hy hypernetwork hg R + R":
M ho W e,a), Vi E
d: m
ve [7.]-
For each group We,, the hypernetwork outputs logits Pe, over the valid nam mask set Sum {s; E {0, 1)mh ||s n based on P(M 8j) Pt,jo 8j E S. The mask for group i is generated contextually from its weights W e.::
M7 ~ (pt. Pr, = Softmax(ha) (We.))
where he 1. + RT is the shared hypernetwork and n the valid mask set defined in Eq. This shared, group-wise design drastically reduces hypemeetwrk size while capturing local dependencies within each m-sized block. For example, producing 6 logits from m=44 weights requires only a few thousand parameters, whereas a naïve full-matrix hypernetwork for the FFN up-projection (W,, 40066 11008 in LLaMA-7B) would demand tens of millions of input/output dimensions-billion of parameters, several orders larger. Handling extremely high-dimensional LLM layers under strict n.m sparsity makes this shared, local-mask generation a necessary and non-trivial innovation for memory- and compute-effin efficcien structured pruning.
Context-Aware Embeddings. To adapt the shared hypernetwork different pruning contexts across layers and components, we introduce trainable conditional embeddings. A layer embedding og E Rd identifies the f-th layer for depth-specific adaptation, while a component embedding e Rd specifies the pruned and Up-projection (t,,), Down-projection (t,z), Gating (t,g) for FFN. These embeddings are learned jointly with the hypernetwork, adding d x (L + parameters:
Beapoonnntnuueee (tg). Key (t&), Value (t,), Output (t,,) for MHSA,
per layer and seven shared
globally. Initialization and optimization details are in Appendix D.4
Using the shared hypernetwork and embeddings, HyperPrune generates context-aware masks:
min (MM (Wee..sett)) ([[(We))..... f([We,, @ M,,).xx- (8) 8,e.t
This formulation captures structural regularities across layers while remaining flexible to component-level variations. HyperPrune thus balances generalization (via shared parameters) and specialization (via embeddings), yielding more effective and transferable pruning.
Information-Theoreds Interpretations. From an information -heeoretiotthhoretiic view, optimizing Eq. corresponds to maximizing the mutual information between dense and pruned models, formalized in Theorem
5

---

Published as a conference paper at ICLR 2026
Layer- wise Loss
Hypernetwork ha
D
Frozen
Pruned weights W
Mask set
Logits p
Probability z
Mask M
weights W
Figure 2: Illustration of generating n m semi-structured sparsity masks via continuous relaxation with a hypernetwork (example: 2 4). The hypemneworr takes frozen weights W input and outputs logits p over mask patterns. Probabilities z and the relaxed mask M are computed via Eq. 37 yielding pruned weights W = w @ M used in Eq. 12 This relaxation allows gradients to flow through mask selection, enabling end-to-end updates of
Theorem (Stochastic Relaxation of n m Masked Linear Approximation). Lel W E Bdxdd and D. Partition binary mask M into d² m disjoint blocks {M.} sfze m, each constrained to satisfy n.m sparsity. Let E M denote the block-wise expectation of these random masks. Then,
max II; x); f(W M,x)) + MESm.m
minEx~pllf(W... {p.}
f(W O [IM]x))l??
where {p:} parameterize a differentiable relaxation of the discrete n: mask selection.
Proof Sketch. Let Z f(W, x) and = f(W M,)) x). Under small perturbation assumption, the sparse output can be viewed noisy version of the dense output, and for linear layers with Gaussian inputs, maximizing their mutual information is approximately equivalent to minimizing the expected squared difference | -- 2112. To enable differentiable optimization under n:m con- straints, introduce block-wise categorical distribution feasible masks and replace M with its expectation, which by linearity yields the surrogate objective Ex f( f @E[M] x)| The full proof is provided Appendix [A]
This theorem shows that learning stochastic n:m masks via soft probabilities is equivalent to maxi- mizing the information preserved between dense and pruned layers. As result, discrete structured sparsity can be optimized through smooth reconstruction objective, enabling efficient end-to-end training of the hypernetwork.
4.2 REGULARIZED HYPERNETWORK FOR ENIIANCED MASK LEARNING
Continual Pruning Regularization. Layer-wise pruning risks catastrophic forgetting: when op- timizing the hypemettwrk h for the current layer É, knowledge from earlier layers .I....,f
may be overwritten. The formulation in Eq. S targets only the current layer and lacks consistency with prior ones. Inspired by continual learning (Wang et al.] 2024). HyperPrune preserves past knowledge by regularizing the hypernetwork remain consistent previously pruned layers. In this continual pruning setting (Mallya et al.] [0188] Golkar et al. 2019 Ksiazek & Spurek 2025), the hypernetwork parameters 6' after pruning layer forgetting, we penalize discrepancies between the outputs of hg and hy over earlier layers:
initialize those for layer L. To mitigate
1
(Wr (Wr,er, () ha (We, er, t)]? 8=1
Reontimual Ew-DW
(9)
(E
The factor (E-1) normalizes across layers, preventing scale-up of the regularization. In practice, since only W loaded during pruning, the expectation can be approximated by caching small subset Dw of W from previous layers (discussed in Appendix D.S). This regularization serves as functional knowledge distillation, aligning past and current hypernetworks. More broadly, it can be seen as an information-theorchi constraint (e.g., KL divergence or mutual information preservation) that stabilizes behavior across layers and prevents abrupt shifts that degrade mask quality.

---

Published as a conference paper at ICLR 2026
Feature Outlier Regularization. Large-scale LLMs (typically >6B parameters) often produce hidden features with unusually high magnitudes. These feature outliers are not noise; they encode key semantic and structural information and strongly influence predictions (Dettmers et al. 2022) Naively pruning weights linked to such features can severely degrade performance, especially under structured sparsity. To mitigate this, we introduce feature-aware regularization that favors preserv- ing weights interacting with high-magnitude activations (large weight and input norms). Formally:
Routier EE-DhBMMh [I(We @ Me) Diag(
(10)
where Diag(-) forms diagonal matrix. This biases pruning toward directions aligned with high- magnitude activations, protecting critical semantic channels and preserving accuracy and robustness. Interestingly, for single component (ignoring EM for simplicity), this regularization can be refor- mulated as
ExND
W Diag(xe-)
[1]
-
Σ (Wess Bx-D 2
Σ Weis Varl (11)
with W. We Mr. The first term recovers Wanda's importance score (Snn et al. 2024 based on average weight-activation alignment. The second term captures feature variance, ensuring even zero-mean features affecting outputs are preserved. Together, they provide a variance-awe aware importance measure for more robust pruning under stochastic or heterogeneous inputs.
Formulation of HyperPrune. By combining continual pruning regularization 99. feature outlier regularization (T)), and hypernetwork-basse pruning (8), the full objective HyperPrune is:
min Ex DEM || [( xe-1) f( 0,e,t
A Routier + A2 Ronniinnall
s.t. We, We, 2 Me,,, M Categorical(pt.)) Peạ = Softmax(ho ( t)), PM,,. 55) = pe,ty Sy E S,
d1 le [L].
(12)
m
where \1, A2 0 are regularization hyperparameters.
HyperPrune is a hypernetwork-bant based, poatinullleaming-insppi framework for n:m semi- structured pruming of LLMs.. Rather than jointly optimizing masks across all layers, it prunes each layer sequentially via a shared hypernetwork, retaining knowledge from previous layers to miti- gate catastrophic forgetting and reduce memory usage. Categorical mask sampling is relaxed with the Gumbel-Softmax trick for end-to-end gradient- based optimization (Algorithm [}: details are in Appendix B Fig. illlustrates this context-aware, hypernetwork- based mask generation.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Models and Baselines. We evaluate HyperPrung on diverse set of pre-trained large language models, including LLaMA-2 (7B/13B/70B) (Touvron et al., 2023b), to assess both performance and scalability. We compare against representative heuristic and optimization-based pruning meth- ods: Magnitude (Han et al.] 2015), Wanda (Sun et al. 2024). SparseGPT (Frantar & Alistarh 2023). Pruner-Zero (Dong al. 2024), and MaskPro (Sun et al. 2025). We exclude MaskLLM FFng et al. 2024) due to its prohibitive computational cost, which requires thousands of GPU hours and 520K calibration samples even for LLaMA-2-133 13B. Instead, we adopt MaskPro as strong
optimization-baser baseline that achieves competitive results with substantially lower cost (Sun)
al. 2025). Full experimental configurations and additional results are provided in Appendix D
Evaluation and Datasets.
For methods requiring calibration data, either for pruning- scoing com- putation or optimization, we follow SparseGPT and Wanda and use 128 sequences sampled from the C4 training set (Raffel et al. 2020), with context length matching the model. For HyperPrune,

---

Published as a conference paper at ICLR 2026
Algorithm HyperPrune: Semi-Structured Pruning via Shared and Context-Aware Hypernetwork
1: Input: Pretrained LLM with L layers, calibration data D
2: Initialize: Shared hypernetwork hgg layer embeddings {ez}. task embeddings {te}
3: for layer & t to L do
4: Load W. into GPU
5: while not converged do
6: Get input activation x from for block i in We do
7:
8:
Compute logits: log Pr, = ha ( We,,,
9:
Sample structured mask M a via Gumbel-Softmax
10:
Prune weights: W We @ M end for
11:
12:
Forward pass: X = [Wexx,
Compute reconstruction loss Lreeon and outlier regularization Routleer
14:
Compute continual pruning regularization from past layers Reontinual
Update 6, Cz2 and (t) using total loss f freeom d Routlier + X2 Recuntiual
16:
end while
17:
Freeze he as har for future regularization
18:
Unload W, to free memory
19:
end for
20:
return Final pruned model with structured sparse weights
optimize the Gumbel logits for 200 steps while keeping LLM weights frozen. Consistent with prior work, C4 is used for pruning calibration and Wikitext (Merity et al. (017)) for perplexity evaluation (PPL). We further adopt the Al LM Harness (Gao et al. 2024) for zero-shot evaluation on downstream tasks (measured by accuracy), providing standardized and reproducible benchmark.
All experiments are conducted on a single NVIDIA A100 GPU (80GB).
Sparse Pattern. We exclude the embedding layer and the final classification head from pruning, as pruning them significantly harms performance while only accounting for about 1% of model parameters. All remaining linear layers pruned under the 2:4 semi-structured sparsity pattern. For heuristic methods that do not natively support semi-structured sparsity, we follow the procedure in (Frantar & Alistarh 2023 Sun et al. 2024), selecting two elements retain within each group of four consecutive weights based on their pruning scores.
5.2 EXPERIMENTAL RESULTS
Performance Evaluation. This section evaluates the efficacy of our proposed pruning method with respect to two key questions: (i) whether generative performance is preserved after struc- tured pruning, measured by perplexity on standard language modeling benchmarks such as Wiki- Text (Merity al.) 2017), and (ii) whether zero-shot generalization maintained across diverse downstream tasks using the EleutherAl LM Harness (Gao et al., 2024). As shown in Table D our method, HyperPrune, consistently achieves the best trade-off between perplexity and accuracy across all model scales. For LLaMA-2 7B, HyperPrune reduces Wikitext perplexity to 10.11 and attains the highest mean accuracy of 53.76, outperforming heuristic methods (c.g., SparseGPT at 51.00) and optimization-be saasddd methods (c.g., MaskPro at 52.81). On LLaMA-2 13B, HyperPrune leads with mean accuracy 59.25, slightly surpassing MaskPro (58.97) while maintaining com- petitive perplexity (7.60). For LLaMA-2 70B, HyperPrune achieves mean accuracy of 68.57 with low perplexity (5.13), exceeding all baselines. These results demonstrate that end-to-end training of compact shared hypernetwork for mask generation enables HyperPrune to robustly retain gener- ative quality and cross-task task generalization. In contrast, traditional heuristics suffer from estimation errors in weight scoring, and optimization-basee methods such MaskPro face memory limitations on large models (c.g., LLaMA-2 70B) or MaskLLM require substantially more calibration samples and GPU resources than HyperPrune, highlighting its effectiveness and efficiency (See Table 7)
8

---

Published as conference paper at ICLR 2026
Table 1: Evaluation of language modeling and zero-shot task performance under 2:4 sparsity on models with frozen weights. Pruning methods were calibrated on the C4 dataset and evaluated on Wikitext-2 (perplexity) and seven zero-shot tasks (accuracy), following the methodology of (Frantar & Alistarh 2023 Sun et al. 2024). Best results are in bold and second-best are underlined.
Method
Wikitext
BoolQ
RTE
HellaS.
WinoG.1 f
ARC-E1 t
ARC-C1
OBQA
Mean
LLaMA-2
5.12
77.74
62.82
57.17
76.39
43.52
31.40
59.71
Magnitude
52.85
56.33
51.38
42.47
61.03
59.68
27.61
21.98
45.78
SparseGPT
10.39
70.56
58,90
43.46
66.71
64.17
29.95
23.22
51.00
Wanda
11.09
67.62
53.17
40.95
62.38
61.76
31.23
24.32
48.78
Pruner-Zero
10.35
69.14
53.43
$4.68
60.54
61.57
32.17
32.60
52,02
MaskPro
12.29
71.12
61.37
46.18
65.82
66.12
32.85
26.20
52.81
HyperPrune
10.11
70.69
60.22
$5.28
64.50
66.32
32.55
26.76
53.76
1.aMM-2 13B
4.57
80.52
65.34
60.06
72.22
79.42
48.46
35.20
63.03
Magnitude
8.31
65.70
54.25
50.17
62.06
62.51
31.72
23.23
49,95
SparseGPT
8.27
76.78
59.48
46.61
68.69
70.64
36.66
25.50
54.91
Wanda
8.29
76.83
61.26
47.83
66.92
69.34
36.87
26.43
55.07
Pruner-Zero
7,53
77.89
56.68
63.37
67.72
69.70
37.29
36.00
58.38
MaskPPo
8.16
78.05
62.22
63.18
67.67
37,56
33.18
58.97
HyperPrune
7.60
77.63
61.84
64,20
67.77
71.00
37,50
34.80
$9.25
LLaMA-2 70B
3.12
83,40
67.87
66.10
78.06
82.55
54.44
37.20
67.08
Magnitude
6.33
73.22
57.03
58.41
74.30
76.17
45.23
35.46
59.97
SpanseGPT
5.44
79.53
70.75
59.03
76.62
78.96 88.96
48.57
33.82
64.61
82,18
69.83
59.37
76.25
47.29
64.16
Pruner-ZZer
4.87
81,40
67.87
77.60
74.11
79.45
50.00
43.40
67.69
HyperPrune
5.13
81.75
70.20
77.83
77.60
79.50
49.62
43.51
68.57
LLaMA-7B
LLaMA-7B
54
12.5
SparseGPT
SparseGPT
Wande
Mean Accuracy 1
Wanda
12.0
Prunar-Zero
Pruner-Zero
Perplexity
MaskPro
MaskPru
11.5
Hyper/rune
51
HyperTrune
11.0
50
49
10.5
10.0
256 512
Callbration Samples
CCallbrttion Samples
(a) Scalability on language modeling
(b) Scalability on zero-shot generalization
Figure 3: Perplexity on Wikitext (a) and mean accuracy across seven zero-shot tasks (b) for different pruning methods, evaluated with increasing numbers of calibration samples (1, 2, 4, 8, 16, 32, 64, 128. 256, and 512). The results illustrate how performance scales with additional calibration data.
Scalability Evaluation. To evaluate whether HyperPrune generate scalable pruning masks that exploit structural redundancy across layers and modules while minimizing performance degra- dation, adopt held-out calibration protocol following MaskLLM. As shown in Fig. 3 Hyper- Prune consistently outperforms competing methods in both Wikitext perplexity and zero-shot mean accuracy, with performance steadily improving as more calibration samples are used. With 512 samples, it achieves the lowest perplexity ( 10.2) and highest mean accuracy (>> 53%). indicating strong data efficiency and scalability. Although MaskPro performs competitively, it is consistently surpassed by HyperPrune, while heuristic approaches such as Wanda and SparseGPT show weaker scalability and limited gains from additional calibration data. These results demonstrate that Hy- perPrune not only delivers superior overall performance but also effectively leverages increasing calibration data, making a robust and scalable solution for pruning large language models.
Ablation Studies. We evaluate the contribution of each HyperPrune component LLaMA-2-7B by removing layer embeddings (le), component embeddings (ce), continual pruning (cp), or feature- outlier (fo) regularization. Removing either embedding consistently degrades performance (e.g..

---

Published as a conference paper at ICLR 2026
12
HyperPrune
HyperPrune
11
HyperPrune w/o le
HyperPrune w/o cp
Perplexity
10
HyperPrune w/o ce
Perplexity
10
HyperPrune w/o fo
Llaa-2-78
Lama-2-138
Uama-2-7CB
Uama-2-78
Uama-2-139
Llama-2-708
Models
Models
(a) Effect context-aware embedding
(b) Effect of regularization
Figure 4: Ablation study of HyperPrune the Wikitext dataset. (a) Impact of context-aware em- beddings, where le and ce denote layer and component embeddings. (b) Impact of regularizations, where ep and fo denote continual pruning and feature outlier retention.
perplexity increases from 10.11 to 11.36), confirming that contextual conditioning is important for generating high-quality structured masks, with layer embeddings having slightly larger impact than component embeddings. Similarly, disabling cp fo increases perplexity (e.g.. 10.11 to 11006 without fo), highlighting their role stabilizing sequential pruning and preserving salient features during sparsification. Overall, these ablations show that both context-aware embeddings and the proposed regularization strategies are necessary to maintain low perplexity and achieve effective layer-wise structured pruning.
Computational and Memory Efficiency. We evaluate the practical efficiency of HyperPrune un- der 2:4 structured sparsity. On A100 GPU, kernel-level speedups of 1.55-1.65x translate into an end-to-end latency reduction for LLaMA-2-7B from 248 ms to 174 ms x1 see Appendix D.9). demonstrating tangible inference gains under realistic deployment settings. Importantly, these im- provements arise from hardware-aligned structured sparsity, ensuring that theoretical sparsity trans- lates into actual runtime acceleration. HyperPrune is also highly efficient during mask optimization (Appendix D.II. Tables 7 Tad 8 show that it reduces pruning mask training cost by nearly order of magnitude, requiring only 7-15 GPU hours and 15 5222 GB memory for LLaMA-2 7B/13B. This is substantially lower than MaskPro and dramatically more efficient than MaskLLM, which quires 1200-2300 GPU hours and hundreds GB memory, while HyperPrune remains feasible 70B scale. Overall, these results demonstrate that HyperPrune achieves strong computational and memory efficiency without sacrificing scalability to modern large-scale LLMs.
CONCLUSION AND FUTURE WORK
We introduced HyperPrune, a practical framework for learning n semi-structured sparsity in LLMs. HyperPrune employs compact shared hypernetwork generate block-wise masks in sliding, layer-wise fashion, conditioned on learned embeddings. It incorporates and feature-outlier regularizers to preserve cross-layer knowledge and protect critical activations. This enables direct optimization for hardware-friendd n sparsity with low memory and compute overhead, supporting models from 7B to 70B parameters on a single A100 GPU. Experiments on LLaMA show that HyperPrune maintains language modeling performance and provides tangible speedups under 2:4 sparsity, demonstrating its efficiency for LLM deployment.
Nonetheless, HyperPrune has limitations. It relies sufficient calibration data, potentially reducing robustness under domain shift, and the lightweight hypernetork faces a capacity- seittyyggneaalzzttion trade-off, motivating adaptive hierarchical designs. Future work includes evaluation diverse inference stacks and hardware, and integration with complementary techniques such as quantization or low-rank adaptation to maximize practical impact.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We sincerely thank the anonymous reviewers for their constructive feedback and valuable sugges- tions, which have helped improve the clarity and quality of this work. This work was supported in part by JST (JPMJNX25C2, JPMJKP24C3, JPMJCR23M4, JPMICR2ID3), JSPS (23H00483), and 120251002.
REFERENCES
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via compression approach. In International conference machine learning, pp- 254-263. PMLR, 2018.
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns. In The Twelfth International Conference Learning Representations, 2024. URL https:
//openreview.o Sepeerreieewnne/forroeeeeeeeeeee taat/fforume http://oooommddd-aaaaaaaaaa
Stephen Boyd, Neal Parikh, Eric Chu, Borja Pelcato, and Jonathan Eckstein. Distributed optimiza- tion and statistical learning via the alternating direction method of multipliers. Foundations and Trendso in Machine Learning, 3 2011.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voor Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., wwtps//apeeeeeeeee.e... pdf
2020. URL https://prroeeedinsseneeiiioooos File sile/ 121/114545 21t.::00ccbbba111111110000011110
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang. and Song Han. Once for all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2020. URL
019000001110 pdf
Rujikorn Charakorn, Edoardo Cetin, Yujin Tang, and Robert Tjarko Lange. Text-to-loRA: Instant transformer adaption. In Forty-second International Conference on Machine Learning, 2025. URL
net / forum? F id= dd=2sskCd300
Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A Clifton. A brief review of hypernetworks in deep learning. Artificial Intelligence Review, 57(9)2250, 2024.
Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE Trans. Pattern Anal. Mach. Intell., 46(12):10558-10578 December 2024. ISSN 0162-8828. doi: 10.1009/PAMM.22244447000 URL httts::// /ttp:::///oiioooo/// org 10. 1109/TPAM, 3444434708444
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int80: 8-bit matrix multi- plication for transformers scale. Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: efficient finetuning of quantized 1lms. Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023, Curran Associates Inc
Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner-zerr zero: evolving symbolic pruning metric from scratch for large language models. In Pro- ceedings of the 41st International Conference Machine Learning, ICML'24, JMLR. 2024.

---

Published as a conference paper at ICLR 2026
Luca Eyring, Shyamgopal Karthik, Alexey Dosovitskiy, Nataniel Ruiz, and Zeynep Akata. Noise hypemetworks: Amortizing test-time compute diffusion models. arXiv preprint, 2025.
Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng. Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. Not all layers of Ilms are necessary during inference. arXiv preprint arXiv:2403.0218/ 2024.
Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. MaskLLM: Learnable semi-structured sparsity for large lan- guage models. In The Thirty-eighth Annual Conference on Neural Information Processing Sys- tems, 2024. URL https:
7b
Jonathan Frankle and Michael Carbin, The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: /03.1v5880,.0363,, 2018.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR). 2019.
Elias Frantar and Dan Alistarh. Sparsegpt: massive language models can be accurately pruned in one-shot. In Proceedings of the 40th International Conference on Machine Learning, ICMI'23.
JMrR.orgg 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210.17333, 2022a.
Elias Frantar, Sidak Pal Singh, and Dan Alistarh. Optimal brain compression: a framework for accurate post-training quantization and pruning. In Proceedings of the 36th International Confer- ence on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022b. Curran Associates Inc. ISBN 97817138711 1088..
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL httph:t///// pattts://zenodee: http:///nnoooooooeeeeerre//////e
Siavash Golkar, Micheal Kagan, and Kyunghyun Cho. Continual learning via neural pruning. In Real Neurons & Hidden Units: Future directions the intersection of neuroscience and artificial intelligence @ NeurlPS 2019, 2019. URLhtttps://ppeeeeeeeee
/t/p//pppeviie..eooo hwt.:eiwwwwwweeeeeeeee....ee wwt.:tettoooooooe i d-
XXYLIB
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Ko- renev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak. Chloc Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind That- tai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jong- soo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasami. Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu. Kunal Bhalla, Kushal Lakhotia. Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi,
12

---

Published as a conference paper at ICLR 2026
Mahesh Pasupuleti, Mannat Singh. Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Ku- mar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoy- chev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ra- mon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Ro- hit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang. Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mi- haylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Gold- schlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert. Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexii Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, An- drew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, An- nie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leon- hardi, Bernie Huang. Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Mon- talvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang. Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harri- son Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog. Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean -aptisee Gaya, Jeff Mareus, Jeff Tang, Jen- nifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Jun- jie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena. Keqian Li, Kiran Jagadeesh. Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov. Mikayel Samvelyan. Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo,, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang. Rachad Alao, Rachel Ro- driguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani. Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai layesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ra-
13

---

Published as a conference paper at ICLR 2026
maswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe. Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Ku- mar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang. Wes Bouaziz, Will Constable, Xiaocheng Tang. Xiao- jian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi. Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary VeVito Zef Rosnbrick, Zhao- duo Wen. Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https: /ttas://aaxiii org/
wtwp//////////oooo//./////aaaae/ abs 2407 21783
David Ha, Andrew M. Dai, and Quoe V. Le. Hypernetworks. In International Conference on Learn- Seeee/oorumiidd--kA000
ing Representations, 2017. uttppptss//oppeeeeiii
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 29th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, pp. 1135-1143, Cambridge, MA. USA, 2015. MIT Press.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015. URL httth:://aaxiii .rg. orgg/bbs/503..00 02531
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 11Xi:22006.09655.. 2021.
Weizhong Huang, Yuxin Zhang, Xiawu Zheng, Liuyang, Jing Lin, Yiwu Yao, and Rongrong Ji. Dynamic low-rank sparse adaptation for large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL http:///ppp//eeeee/////s// net forum?
Id==oh09397.7q
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph (Seffi) Naor, and Daniel Soudry. Ac- celerated sparse neural training: provable and efficient method to find n.m transposable masks. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS 21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845393.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztion with gumbel-softmax. softmax. In International Conference on Learning Representations, 2017. http:s/tt:t////t ptatttss://ppeeeevvieee
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language understanding. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4163-4174, Online, November 2020. Association for Computational Linguistics. 20g/2020. doi: 10018663//1/2020.iii findings-emmll.372. URL https: httt:s://aclanttcllee findings-emn wp.
Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetoorks. In Annual Meeting of the Association for Computational Linguistics, 2021.
David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint 1rXiv:1710..4773 2017.
Kamil Ksiazek and Przemystaw Spurek. Hypermask: Adaptive hypernetwork-basee masks for con- tinual learning. Neural Networks, 191: 2025. ISSN 0893-6080. doi: https://doi.onn org/ n0.1016/jnemmet2225..07777. URL https: www.wciinneeiieet.ttt com/ wwt.sciencccccc////cccccco
ntticcle//i1/30093330222220
14

---

Published as a conference paper at ICLR 2026
Yen- Yen-Cheng Liu, Thih-Yao Ma, Junjiao Tian, Zijian He, and Zsolt Kira. Polyhistor: parameter- efficient multi-task adaptation for dense vision tasks. NIPS '22, Red Hook, NY, USA, 2022.
Curran Associates Inc. ISBN 9781713871088 1088.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through 4 regularization. In International Conference on Learning Representations, 2018. URL https:
h/opennevvew.nee.// hwtp://tt/ooouumoomomeo d= Y8hhhghg
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In ICLR, 2017.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to mul- tiple tasks by learning to mask weights. In Computer Vision ECCV 2018: 15th European Con- ference, Munich, Germany, September 14, 2018, Proceedings, Purt IV, pp. 72-88, Berlin, Hei- delberg, 2018. Springer- Verlag. ISBN 978 978-3-030-01224-3 doi: 10.1007/978-3-000
10w./000/9900 URL https:: /tttps://do..og:: 1t..:///..1110/9171
01225-0
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect, 2024.
Xiang Meng, Kayhan Behdin, Haoyue Wang, and Rahul Mazumder. ALPS: Improved optimiza- tion for highly sparse one-shot pruning for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL 4upd
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mix- ture models. International Conference on Learning Representations, 2017. URL https:
//openreview.net/Booee Byy
OpenAl. GPT4 Technical Report. arXiv preprint arXiv:2303.08774, 2023. URL https://
arxiv. Dexivorrgabbs/2300 08774
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 21(1), January 2020. ISSN 1532-4435.
R. Reed. Pruning algorithms--a survey. IEEE Transactions on Neural Networks, 4(5):740-747, 1993. doi: 10. 10.1109/72.248452.
Marcin Sendera, Marcin Przewiezlikowski, Konrad Karanowski, Maciej Zieba, Jacek Tabor, and Przemyslaw Spurek. Hypershot: Few-shot learning by kernel hypernetworks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2469-2478, January 2023.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.052000 2020. URL
htttvvorggabb////00/ http://as/202...52222 http:s///arrvv.orrrr//////22222.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models. In The Twelfth International Conference Learning Representations, 2024.
Yan Sun, Qixin Zhang, Zhiyuan Yu, Xikun Zhang, Li Shen, and Dacheng Tao. Maskpro: Linear- space probabilistic learning for strict (n:m)-sparsity on large language models, 2025. URL
https::/aaxx:: org ala/ 12876
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a. URL /tt.tttess//aaavv..oo//bba///
15

---

Published as a conference paper at ICLR 2026
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL
/tt///bb//233333/ 09288
Johannes von Oswald, Christian Henning, Benjamin F. Grewe, and João Sacramento. Continual learning with hypernetworks. In International Conference on Learning Representations, 2020.
URL https::////aziii org. http//bs/99990 00695
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A Comprehensive Survey of Continual Learning: Theory, Method and Application. IEEE Transactions on Pattern Analysis & Ma- chine Intelligence, 46(08):5362 5383, August 2024. ISSN 1939-3539, doi: 10.1
2024.3367329. URL https://doi wececoopttersooieee..rr/./..0.
2024. 3361329
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E. Gonzalez. Skipnet: Learning dy- namic routing in convolutional networks. In Computer Vision ECCV 2018: 15th European Con- ference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII, pp. 420-436446 Berlin, Heidelberg, 2018. Springer-Verlage ISBN 978- 978-3-033--0126--1... doi: 25. URL httlt:tts
10. 00-.000077770--0 00010//0000000711--00000-0-0--00
25
Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. Generalizing from a few exam- ples: A survey on shot learning. ACM Comput. Surv., 53(3), June 2020. ISSN 0360-0300. doi: 10.1145/3386252. URL WRRLhttp:: http:///oo.ooogg..// 1145/338622
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerat- ing language model pre-training via structured pruning. In The Twelfth International Confer- ence on Learning Representations, 2024. URL 0910dae0zp
Mingxue Xu, Yao Lei Xu, and Danilo P. Mandic. Tensorgpt: Efficient compression of large lan- guage models based on tensor-train decomposition, 2024a. URL www.:/rumteeeteteeeoooeo forum?id
abs/2307 00526
Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang, Kaipeng Zhang, Peng Gao, Fengwei An, Yu Qiao, and Ping Luo. Besa: Pruning large language models with blockwise parameter-efficient sparsity allocation. In The Twelfth International Conference Learning Representations (ICLR),
2024b. URL httttps://ppenreviiie httppp/wwwnnaan..eeeeooooattaaaa
net =odrm:2
Yang Yang. Wendi Ren, and Shuang Li. Hyperlogic: Enhancing diversity and accuracy in rule leam- ing with hypernets. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL http///ppeerevveeee net. wtwt://frummiieeem d=g.2
Yifei Yang, Zouying Cao, and Hai Zhao. LaCo: Large language model pruning via layer col- lapse. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 6401-6417, Miami, Florida, USA, November 2024b. Association for Computational Linguisties. doi: 10. 1tt.///////222..nnnnnnn. URL http::tttt/ccctt htttttsss/aaaaaahhooyy.oooo hwwpt/hhoooo..ooo///eoo 200pg/2224..22
372/
Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsieh, Yaqing Wang. Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (owl): missing secret sauce for pruning Ilms to high sparsity. In Proceedings of the 41st International Conference on Machine Learning, ICML'24 JMLR.org, 2024.
16

---

Published as a conference paper at ICLR 2026
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moxian Chen, Shuohui Chen, Christo- pher Choi, Daniel Lakhotia, Anthony Li, Yanzhe Malykhin, Bryan O' Horo, Anjali Pandey, Sesh Santhanam, Stephen Santhanam, Vivek Sharma. Kevin Shleifer, Eric Wenzek, and Luke Zettle- moyer. OPT: Open Pre-trained Transformer I Language Models. arXiv preprint arXiv:2205.0 2022. pttttps:/arxiv..oot. /2000 01068
Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug- and-play: An efficient post-training pruning method for large language models. In The Twelfth International Conference Learning Representations, 2024. htt.hhttss///ppnrreeeee
fest//orum?iid-e P 9xoo
Yu Zhang and Qiang Yang. A survey on multi-task learning. JEEE Transactions on Knowledge and Data Engineering, 34( 34(1 2):5586)66600 2022. doi: 10.1 10.1009/KKDE::222:30000000
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hong- sheng Li. Learning n:m fine-grained structured sparse neural networks from scratch. In Interna- tional Conference on Learning Representations, 2021.
Michael Zhu and Suyog Gupta. To prune, not to prune: exploring the efficacy of pruning for model compression, 2017. URL https: htt.:httt://arxtee http:///rrav.oo///////1.1.11
A PROOF OF THEOREM It STOCHASTIC RELAXATION OF n SPARSE MASKING VIA EXPECTED MSE Loss
Here we present the full version of Theorem along with its detailed proof, which were omitted from the main text for clarity.
Theorem (Stochastic Relaxation of n m Masked Linear Approximation). Let W E Rdxd and D. Partition a binary mask M E {0, ] into B = dL /m blocks {Mi) of size m, with
M, ~ Categoricall Pt E 111-1
Snm {s E {0, 1)"h ||s|lo = n).
Define the relaxed mask E[M] [E, P158J...... -,.,,.,,,,,,,, PB}Ss] [0, 1]axd Then
MAX MES
TU(W,X)) O M,x)) e min (p:)
E / (W, x) - S(w o EMM), x)I?:
where {p1} provide differentiable relaxation of discrete n:m sparsity and f denotes MHSA/FFN layers (linear activation).
Proof. We split the proof into two main parts: (i) the reduction from mutual information to a mean squared error (MSE) objective under a small-noise approximation, and (ii) the stochastic relaxation of the discrete mask into continuous surrogate via expectation.
Part 1: From Mutual Information to Expected MSE. We consider the mutual information be-
tween the dense and sparse linear layer outputs: ((((W, f( o M,,), where f denotes the function of MHSA or FFN with linear activation, and we omit the layer subscript L for brevity. We aim to show that minimizing the MSE between the outputs of a dense model and a pruned model is equivalent to maximizing the mutual information between them under mild assumptions. Let us denote:
. Z = f(W, x) - the output of the original dense layer,
2 = f(W O M, x) - the output of the pruned sparse layer,
. E Snmm the binary sparsity mask constrained by the n:m sparsity pattern.
Step 1: Additive Noise Model
We model the sparse output as a noisy version of the dense output:
Ż Z E,
(13)
17

---

Published as a conference paper at ICLR 2026
where the noise term
E E f(W O M,x) - f( f(We @ (M 11 ).x)
(14)
represents the information loss due masking. We assume:
E zero-mean, isotropic, and independent of x,
the noise magnitude is small (i.e., pruning induces small perturbations).
Step 2: Gaussian Assumption and Joint Distribution
Assuming the input
X ~ 00
(15)
both outputs Z and Z are linear Gaussian transformations of jointly Gaussian random vectors.
Step Mutual Information for Jointly Gaussian Variables
Recall that for jointly Gaussian variables Z and Z, the mutual information is given by
Σg
I(Z; Z)
log
(16)
where
Σg Cov(Z),
Σz = Cov(2/Z).
(17)
Since = Z + e aadd e is independent of Z.
Σz E.:
Σg = Ez + Ec.
(18)
Therefore,
1
Σz + Ee
I(Z; 2) = -ogg
(19)
Ee
Step 4: Approximation for Small Noise Covariance
When the noise covariance Σ. small compared to Σz, we apply the matrix determinant lemma or first-order Taylor expansion:
Ez + E./ Ez (1 + Tr Ez Ex))
(20)
Thus,
Σz (1 + Tr(Ei (ET Σx,
[(Z; 2) 23
log
Eel
Σz| EE|
log
+ log (1 + Tr E.)
(21)
Ignoring the weak dependence on the trace term in the small noise regime, the dominant term
[Zz 2) ss const 2 log [E].
(22)
Step 5: Trace Approximation and Connection to MSE
Σg is diagonal or near-diagonal, the log-determinant can be approximated by the trace:
log Σgl Tr(E.).
(23)
Recalling that
Tr(E.) Eex [1]
(24)
18

---

Published as a conference paper at ICLR 2026
arrive at the key equivalence:
((z x const 2lbix Z 2
(25)
Final Conclusion:
Maximizing the mutual information between dense and sparse outputs approximately equivalent to minimizing their expected squared difference:
max [ff(W,)) x): @ M.)) MES.im
min
Ex Ex f(W + M,x) (26)
MES.IM
This provides an information theoretic justification for using MSE-based objectives in n:m sparsity mask learning and pruning.
Part 2: Stochastic Relaxation via Expected Mask. To enable differentiable optimization, introduce a stochastic relaxation. Let the binary mask M E 10, d be partitioned into B = d d/m non-overlapping blocks {M,) of size m. Each block is constrained to belong to the mask set:
m
Spm == {sy E {0,
||sy|lo = =n}, with K = S =
(27)
Instead of selecting a fixed M,, we sample it from a categorical distribution over Spm:
M, Categorical(p.), where p. (pa1.... ... ..K)) A k-1
Σ
Ply
1.
==1
MMM 8,) Pij. with s, E nmm
(28)
The expected value of each block becomes:
K
E
Σ
PtjSy.
(29)
j=1
Then, the full mask is formed as M MM,.... M.....,.)]]] E Spams and its expectation given by:
K
EM I E[Ma]] =
PljS
Σ pass; E 1j
(30)
Let := /(W @ M, x) again denote the sparse output. Taking the expectation the mask:
EM 2 = (W @ x) = S(W O EM~p[M] [M],
(31)
using the linearity of expectation and the independence of x and M. Note that here f denotes a function of MHSA or FFN with linear activation.
The expected squared enror between the dense output and the expected sparse output becomes:
C(p) Ex [[/(W, x) |M O M, x)]]
(32)
= Ex [I x) - -(W @ E.M] x)
(33)
Ex || (11 -EMM) x x)
(34)
The relaxation transforms discrete selection problem into continuous optimization categor- ical parameters p:- When ps becomes one-hot (i.e., Pij = for some j), the relaxation recovers a valid deterministic n:m sparse mask. Thus, optimizing the surrogate loss:
min Ex 1/(W.x) f(W ④ B x) (pc)
(35)
19

---

Published as a conference paper at ICLR 2026
serves smooth surrogate to the original discrete problem:
min MESa.m.
Ex FxllfW.x) f(W M.x)
(36)
This surrogate can be optimized using standard gradient-based methods by parameterizing (c.g., via logits and softmax). This result shows that our relaxation bridges discrete sparsity and differen- tiable training, justifying mask learning as an ieformatiionpreservviii approximation while retaining n m structure. This concludes the proof.
Remark 3 (On the Linear and Gaussian Assumptions). Theorem is derived under linear-layer and Gaussian imput assumption to obtain a clean fonitull-infommatioo-t------- relaxation. Although Transformer blocks include nonlinearities (c.g., SwiGLU (Shazeer 2020)), the justification remains valid in practice for two reasons. First, pruning is performed in small- pentul-petuurbation regime, where a first-order Taylor expansion provides an accurate local linearization of the nonlinear layer, allowing the additive-noise and reconstruction-ent error analysis to apply to the Second, the Gaussianity assumption concerns high-dimensional pre-activations (e.g.. FFN inputs), which are empirically close to Gaussian due to aggregation through residual connections and the stabilizing effect LayerNorm (a Central Limit Theorem (CLT)-like phenomenon). Empirically, observe that the pruning-induced induced reconstruction enror closely follows Gaussian distribution across layers, further supporting the validity of the approximation in practical LLM settings.
-inbbiiiee dynamics.
B OPTIMIZATION
Continuous Relaxation via the Gumbel-Sootmax Trick. Optimizing Eq. 12 is challenging be- cause mask variables are sampled from categorical distribution, making the problem discrete and non-differentiable. To enable end-to-end training, we reparameterize mask selection using the Gumbel-Softmi relaxation (Jang al., 2017 Maddison al. 2017). The categorical distribution over S with logits pr, is relaxed as:
[S]
exp x(loggPP.... + Te.,5/T) Σ, exp (log + (T,..)))
M =
2E.5 Sj.
20, =
(37)
where Mejij Gumbel(O, and 0 is temperature controlling the approximation to discrete- ness. The mask set is S [s....... As - O, the relaxation converges to discrete selection; during training, T is annealed to transition from soft interpolation to near-discrete pruning. With this relaxation, masked weights are updated as W., = We, @ M Fig- illustrates the relaxation process, which permits gradient flow through mask selection, enabling gradient-based optimization. Algorithm summarizes the HyperPrune framework.
Temperature Annealing Schedule in Gumbel-Softmax Relaxation. In experiments, the standard exponential temperature annealing schedule for the Gumbel-Softmax relaxation. The tem- perature 7 is initialized at a relatively high value to enable smooth gradients and is gradually an- nealed toward a small, non-zero final value. Concretely, we adopt
rtt) = max(T/, To (eeeppp)n))
where we set To = 1.0, Ty = 0.5, and use a small decay rate (typically the range 10-5 5 to 10-4), following common practice and the schedule recommended in (Jang al. 2017). Here is the current training step (or epoch). This annealing strategy stabilizes early training while progressively encouraging the hypernetwrr to produce near-discrete mask selections.
Optimization Algorithm. Algorithm | details the training procedure of HyperPrune. The algo- rithm designed to efficiently generate semi-structured pruning masks through a shared, context- aware hypernetwork while maintaining memory scalability across large LLMs. At each layer, the pretrained weights are loaded onto the GPU in blocks, and the hypernetwork conntk-iondiidedd on both layer embeddings and task embeddings produces logits that parameterize Gumbel-Softne Softmaxx dis- tribution for sampling structured 2:4 masks. This stochastic relaxation enables differentiable mask learning while respecting hardware-friendly sparsity. During optimization, the model minimizes
20

---

Published as a conference paper at ICLR 2026
reconstruction loss to preserve activation fidelity, combined with two key regularizations: (i) outlier retention to safeguard critical features, and (ii) continual pruning to enforce cross-layer consistency and knowledge transfer. After training each layer, the hypemetwork parameters to stabilize future pruning decisions, and the corresponding weights are offloaded to reduce memory usage, allowing the method to scale models large LLaMA-2 70B single A100 GPU. By jointly leveraging context-aware embeddings, structured mask sampling, and memory- training, HyperPrune directly aligns pruning with both accuracy retention and practical deployabil- ity.
partially frozen
C TRANSFER LEARNING FOR SPARSE PRIORS
accelerate HyperPrune training and leverage prior knowledge, we incorporate sparse priors into the relaxed distribution candidate masks s, S. Such priors can be obtained from fast heuristic pruning methods like Wanda (Sun et al. 2024). We bias the logits of candidate masks toward reference prior M using cosine similarity:
To
exp(B cos cis-sssmsmma)) co,
log + Pt,j + y log
(38)
E, exp(3 cos-sim M,, Sy:))
where " controls the prior strength and 33 sharpens the similarity- softmax. This preserves the probabilistic semantics while softly biasing selection toward masks structurally aligned with the prior. Setting = 0 ignores the prior, whereas larger "y interpolates smoothly toward prior-guided pruning.
To allow the model to gradually transition from relying on the prior to relying on its own learned signal, we anneal the prior influence parameter y over time. Specifically, decay 7 according to exponential schedule:
Automatically Adjusting 7.
Y = To exp(-a t),
(39)
> decay rate hyperparameter, and is the current training step (or epoch). This annealing scheme ensures that the model initially benefits from the guidance of the prior mask, but progressively shifts toward data-driven learning as training advances.
where "o is the initial prior strength,
The decay rate a can be tuned to control how quickly the prior influence vanishes. A smaller a leads slower decay, allowing the prior remain active longer, which is useful in low-data or noisy regimes. This dynamic adjustment supports smooth interpolation between prior-driven exploration and model-driver convergence.
D EXPERIMENTS
D.1 TRAINING ENVIRONMENT
Table 2: Technical Specifications of the Training Environment (Hardware and Software) used for all
Main Paper Experiments.
GPU
CPU
CUDA
OS
PyTorch
A100 (80GB Memory)
AMD EPYC 7742 2.25GHz cores
11.4
Ubuntu 20.04
2.6.0
All experiments for HyperPrune are conducted on single NVIDIA A100 GPU with 80GB memory, supported by dual AMD EPYC 7742 CPUs (64 cores, 2.25GHz). The software stack includes CUDA 11.4, PyTorch 2.6.0, and Ubuntu 20.04, as summarized in Table 2 This environment is representative of widely available high-performance computing clusters used for large-scale model training. The ample GPU memory of the A100 allows efficient handling of models up to 14B parameters without gradient checkpointing offloading, ensuring stable and reproducible training. Meanwhile, the combination of optimized CUDA kernels and the latest PyTorch backend provides reliable support for semi-structured sparsity and mixed-precision training. both of which are crucial
21

---

Published as a conference paper at ICLR 2026
for accelerating pruning and fine-tuning. We note that HyperPrune does not rely on customized kernels beyond the standard PyTorch and CUTLASS implementations, making our method portable to other modern GPU architectures that support 2:4 sparsity.
D.2 COMPARISON METHODS
We evaluate the proposed HyperPrune against four heuristic pruning methods: Magnitude (Han et al. 2015). Wanda (Sun et al. 2024). SparseGPT (Frantar & Alistarh (023). and Pruner- Zero (Dong et al. 2024p. as well as one optimization- based pruning baseline: MaskPrd (Sun
al. (2025). For all methods, use the publicly available implementations provided by the original authors and follow their recommended configurations. The only exception Magnitude, for which use the implementation provided in the Wanda library.
D.3 HYPERPARAMETER CONFIGURATION
Table 3: Training details and hyper-parameters for mask training in HyperPrune.
Model
Optimizer
#ltcrations
Hiddon dim
A
A2
Prior
LLaMA-2 7B
AdamW(Se-4, wd=0.1)
2,000
(256,512)
[1e-3, 1e3]
[le-3, 1e3]
SparseGPT
2
le-5
LLaMA-2 13B
AdamW(Se-4, wd=0.1)
2,000
(256, 512)
[le-2, 1e2]
[le-3, 1e3]
SparseGPT
le-5
LLaMA-2 70B
AdamW(5e-4, wd=0.1)
2,000
2256, 512)
[le-2, 1e2]
[le-3, 1c3]
SparscGPT
lc-5
Table summarizes the syper-parameters used for training pruning masks in HyperPrune across different model scales. We employ the AdamW optimizer with a learning rate of 5e-4 and weight decay of 0.1, and fix the training schedule to 2,000 steps, which we found sufficient for convergence of mask learning without overfitting to the calibration data. Embeddings in the hypernetwork are initialized from a zero-mean Gaussian distribution with variance 0.01 to ensure stable early training. Specifically, the lightweight hypernetwork employs four-layered feedforward network. We explore two configurations for the hidden layer size: 256 and 512 neurons. The regularization coefficients d1 and Ag. which balance continual pruming and feature outlier retention, selected from broad logarithmic ranges to capture model-specific sensitivities. For LLaMA-2 7B, stronger regularization (A1 E [10-3 3 109) was required to stabilize pruning, whereas for larger models (13B and 70B), narrower range ()1 E [10-2, 10a) proved more effective, reflecting improved robustness at scale. The prior is initialized with SparseGPT masks to provide a strong starting point, and the scaling factors a = 2 and 7 = 10 in Eq. 39 are kept fixed across models for consistent sparsity control. confirms that our method is not overly sensitive to hyperparameter tuning and scales reliably across model sizes.
For architectural conflagration of the hypernetwork HyperPrune, we performed small-scale hy- perparameter search over the hypernetwork architecture during development. Specifically, we eval- uated variants with 1-3 hidden layers and widths {128, 256, 512, 1024). This grid search showed that beyond certain capacity, e.g., more than two hidden layers widths larger than 512, the per- formance of HyperPrune remained essentially unchanged, while training cost increased. The 4-layer FFN with 256/512 hidden dimensions therefore represents stable operating point: it is expressive enough to model structured masks across layers. yet compact enough to maintain efficiency.
D.4 CONTEXT-AWARE EMBEDDING INITIALIZATION AND OPTIMIZATION
Initialization. Jointly optimizing the task embedding e and hypernetwork parameters 6 can be unstable. Careful initialization of e, improves convergence and generalization. For initializing e a one-hot vector with added small Gaussian noise is recommended (e 1, + e ~ (0 I²),, < 1). This approach combines the benefits of clear task separation from one-hot encoding with the advantages of smooth optimization provided by the noise, which also helps break
22

---

Published as a conference paper at ICLR 2026
symmetry and facilitates early-stage gradient flow. When training, consider applying regu- larization (A][ez]l3) using smaller learning rates specifically for e,, and optionally freezing e, during the initial warmup phase.
Unlike discrete and untrainable one-hot embeddings, which create a hard separation between tasks and ignore their relationships, trainable embeddings offer continuous, dense representation that can be updated with gradients. This allows them to capture the similarity between tasks, leading improved generalization and scalability. The trainable approach enables soft parameter sharing across tasks, making it a more efficient and effective method for hypernetoorss.
Configuration of Embedding Dimensionality. In our hyperparameter search, we experimented with several embedding dimensions for both the layer-specific embeddings and the component embeddings t, specifically {48, 64, 96, 1283. We found that once the embedding size exceeds a relatively small threshold (e.g.. d > 96), further increasing the dimensionality brings noticeable performance improvement while incurring additional computational cost. Based on this study, we recommend using d = 64 for the layer embeddings and the component embeddings.
Optimization of Context-Award Embeddings. Both embeddings are learned jointly with the hy- pernetwork parameters via backpropagation, forming part of the input hg Eq. [T] Specifically, for each weight group We., the hypernetwork generates a probabilistic nwm mask conditioned on corresponding embeddings:
Pt, Softmax (WW,,, Cg2 t))
where t is the component embedding for the current subcomponent. Gradients of the reconstruc- tion loss and regularization terms are backpropagated through hgg to update both the hypernetwork parameters and the embeddings, allowing the embeddings to capture layer- and component-speciin pruning patterns. This results in small set of context-aware parameters (d x (L + 7) in total) that modulate mask generation adaptively across the network.
D.5 GENERALIZATION TO LLAMA-3 MODELS
To assess whether HyperPrune extends beyond LLaMA-2, we further evaluate its performance the more recent LLaMA-3 8B (Grattafiori et al. 2024), which incorporates revised activation functions and architectural updates. As shown in Table HyperPrune maintains strong pruning performance on this modern architecture, achieving a Wikitext-2 perplexity of 15.63. This surpasses Wanda, SparseGPT, and MaskPro, and closely approaches the Pruner-Zero baseline (15.21). These results indicate that HyperPrune's sparsity patterns and pruning dynamics transfer effectively to contempo- rary LLM families.
Table Wikitext-2 perplexity of pruning methods applied to LLaMA-3 8B under identical sparsity settings. HyperPrune generalizes well to the updated LLaMA-3 architecture, outperforming prior baselines and closely matching the strong Pruner-Zero method.
Method
Wikitext-2 PPL
LLaMA-3 8B (Dense)
5.76
Magnitude Pruning
2.43
SParseGPT
18.38
Wanda
21.34
Pruner-Zero
15.21
MaskPro
16.70
HyperPrune
15.63
D.6 SCALABILITY OF THE CALIBRATION SET
We evaluate the effect of calibration dataset size zero-shot performance for LLaMA-2-7B. as shown in Table 5 The results indicate a consistent improvement across all tasks as the dataset size increases from 32 256 samples. For example, accuracy BoolQ rises from 64.83% 70.85%.
23

---

Published as a conference paper at ICLR 2026
Table Zero-shot evaluations of masks trained with different dataset sizes on LLaMA-2-7B.
Dataset Size
BoolQ
RTE f
HellaS, f
WinoG.
ARC-E
ARC-C1
OBQA 1
Mean
32 samples
64.83
53.95
50.65
59.11
61.84
24.34
21.51
48.03
64 samples
67.94
57.72
53.64
62.21
63,94
29.99
24.54
51.43
128 samples
70.69
60.22
55.28
64.50
66.32
32.55
26.76
53.76
256 samples
70.85
60.46
56.34
64.70
67.20
32.78
27.20
54.22
RTE from 53.95% to 60.46%, HellaSwag from 50.65% 56.34%. WinoGrande from 59.11% to 64.70% ARC-Easy from 61.84% to 67.20%, ARC-Challenge from 24.34% to 32.78%, and Open- BookQA from 21.51% 27.20% Correspondingly. the overall mean accuracy improves steadily from 48.03% with 32 samples to 54.22% with 256 samples. These findings highlight two key ob- servations. First, while increasing the calibration dataset size consistently boosts performance, the gains are relatively modest, suggesting diminishing returns beyond few hundred samples. Second, even with only 32 samples, HyperPrune achieves reasonably strong zero-shot performance across diverse tasks, demonstrating its robustness and ability to learn effective masks from limited cali- bration data. Overall, these results underscore HyperPrune's efficiency in leveraging small datasets while maintaining strong generalization.
D.7
SUPPLEMENTARY RESULTS FOR ABLATION STUDY
Hyperfrune
llyperPrune
70
HyperPrune
70
HyperPrune
Mean Accuracy
HyperPrune
Mean Accuracy
HyperPrune
65
60
60
55
35.
50
Llama22778
Llama-2-138 Models
Lama-22778
Llama-2-78
Llama-2-138 Models
Lama-2-778
(a) Effect of regularization
(b) Effect of context-aware embedding
Figure 5: Perplexity on Wikitext (a) and Mean Accuracy on seven zero-shot tasks (b) for different pruning methods, evaluated using 2, 4, 8, 16, 32, 64, 128, 256, and 512 calibration samples.
Fig. present ablation studies on HyperPrune on zero-shot tasks, which analyzes the contribution of its core components across LLaMA-2-7B models of different sizes. Panel (a) evaluates the impact of regularization, showing that removing either continual pruning (w/o cp) or feature orthogonality (w/o fo) generally reduces Mean Accuracy, with the full HyperPrune consistently achieving the best results, especially on the 7B and 70B models. Panel (b) examines the effect of context-aware embeddings, where excluding the learnable embedding (w/o 1e) or cortexts (w/o ce) substantially degrades performance on the 7B and 13B models, confirming their importance for adapting pruning downstream tasks.
D.8 EVALUATION OF THE SAMPLING NUMBER OF PARAMETERS
In Eq. 12 we maintain set of layer-wise model parameters in the LLM to compute the expecta- tion over W for continual pruning. Figure exxllores the effect of pruning density by varying the number of sampled parameters per layer. As expected, perplexity on Wikitext decreases spar- sity is reduced, but plateaus after roughly 50 sampled parameters, while Mean Accuracy on seven zero-shot tasks follows a similar trend. Importantly, LLaMA-2-70B retains strong performance even under high sparsity, highlighting HyperPrune's robustness and efficiency in preserving critical weights across model scales.
24

---

Published as a conference paper at ICLR 2026
12
67.5
Uama 2-78
2-13-22-130
65.0
Llama 2-78
10
Perplexity
62.5
Uama-2-78
9
Llama 138
60.0
8
57.5
7
55.0
6
52.5
5
50.0
10
50
110
10
50
70
110
Sampled Parameters per Layer
Sampled Parameters per Layer
(a) Effect of regularization
(b) Effect of context-aware embedding
Figure 6: Perplexity on Wikitext (a) and Mean Accuracy on seven zero-shot tasks (b) for Hyper- Prune, evaluated using 10, 30, 50, 70, 90, and 110 sampled parameters per layer of LLaMAA2-7B.
D.9 INFERENCE SPEEDUP SIMULATION.
Table 6: Acceleration of matrix multiplication (in milliseconds) for LLaMA-2-7B with semi-
structured 2:4 sparsity obtained by HyperPrune on an NVIDIA A100 GPU.
LLaMA Layer
Dense (ms)
2:4 Sparsity (ms)
Speedup
q/k/v/o projection
2.15
1.30
1.65x
up/gate projection
5.95
3.70
1.61 x
down projection 6.05
3.90
1.55x
We evaluate the practical inference benefits of HyperPrune on an NVIDIA A100 GPU by leverag- ing frilear--rriedly 2:4 structured sparsity. Following (Frantar & Alistarh 2023 measure latency for matrix multiplications linear layers using the high-performance CUTLASS GEMM kernel. As reported in Table 6 the g/k/vo, up/gate, and down projection layers achieve speedups of 1.55-1.65% x, resulting in an average per-layer acceleration of approximately 1.60x. Applied end- to-end on LLaMA-2-7B, this corresponds to 1.43 reduction overall inference latency (248 + 174 ms). These results demonstrate that HyperPrune's learned pruning masks not only induce structured sparsity but also provide substantial computational efficiency on modern accelerators, highlighting their practical value for real-world deployment.
D.10 MEMORY SCALABILITY
Table Memory (GB) required for training with 2:4 sparsity.
Model
MaskLLM
MaskPro
HyperPrune (est.)
LLaMA-2 7B
339.56 630.80
39.36 68.41
15.46
LLaMA-2 13B
21.63
LLaMA-2 70B
33.74
Table reports the memory required for training LLaMA-2 models (7B, 13B, 70B) under 2:4 spar- sity across 8x A100 GPUs. The results highlight sharp contrast in scalability among methods. MaskLIM suffers from extremely high memory overhead due to its dense mask parameteriza- tion, exceeding 600 GB for LLaMA-2 13B and failing with out-of-memory (OOM) errors at 70B scale, making impractical for very large models. MaskPro, in contrast, maintains nearly con- stant memory consumption because of its lightweight linear probability modeling, requiring only 39-68 GB for 7B-13B. However, it cannot scale to the 70B model within a single-node setting. HyperPrune achieves better balance: by using compact shared hypemneworr with context- aware embeddings and layer-wise memory offloading, incurs only modest overhead compared to MaskPro, requiring 15.46 GB for 7B and 21.63 GB for 13B, while remaining feasible even at 70B
25

---

Published as a conference paper at ICLR 2026
with 33.74 GB. These results demonstrate that HyperPrune is both memory-efficiem and scalable, enabling structured pruning of trillion-scale models on realistic hardware budgets, unlike methods such MaskLLM that become prohibitive at large scales.
D.11 GPU-HOUR AND WALL-CLOCK COSTS FOR MASK TRAINING.
Table 8 provides a detailed comparison of computational and memory overhead for 2 : 4 sparsity on LLaMA-2 models, reporting GPU hours and memory usage for mask training under identical calibration sizes. As shown, HyperPrune has significantly lower computational and memory costs compared to MaskPro and M.skI.LM. For LLaMA-2 7B, MaskLLM requires R 1200 GPU hours and 339.56 GB of memory across 8x A100 GPUs, while MaskPro uses 10 GPU hours and 39.36 GB. HyperPrune achieves the same task using only GPU hours and 15.46 GB. Similarly, for LLaMA-2 13B, MaskLLM takes 2300 GPU hours, MaskPro 22 GPU hours, whereas HyperPrune only 15 GPU hours with 21.63 GB. We do not report LLaMA-2 70B here because MaskLLM and MaskPro cannot complete pruning single A100 GPU. These results demonstrate HyperPrune's practical advantage in both computation and memory.
Table 8: GPU hours and memory usage required for training pruning masks under 2 4 structured sparsity on LLaMA-2 models. HyperPrune achieves the lowest computational and memory cost across all settings, while MaskLLM requires orders of magnitude more resources and fails to scale larger models.
LLaMA-2 7B
LLaMA-2 13B
Method
Memory (GB)
GPU Hours
Memory (GB)
GPU Hours
MaskLLM
339.56
1200
630.80
2300
MaskPro
39.36
10
68.41
22
HyperPrune 15.46
21.63
D.12 ABLATION STUDY: EFFECT BRSOR-BASED INITIALIZATION
We investigate the impact of prior-based initialization on HyperPrune by testing three variants: ini- tialization from SparseGPT prior (Frantar & Alistarh (023), from Wanda prior (Sun al.] 2024), and with no prior. Experiments were conducted on A-a----- and I.LaMA-2-13B using Wikitext-2 perplexity. As shown Table D prior-based initialization consistently improves prun- ing quality, with SparseGPT providing the strongest gains. These results indicate that HyperPrune effectively leverages upstream sparsity patterns while remaining robust to the choice of prior.
Table 9: Ablation on prior-based initialization for HyperPrune. Using a SparseGPT or Wanda prior improves Wikitext-2 perplexity both LLaMA-2-7B and 13B, while HyperPrune without prior performs worse, indicating the benefit but not strict necessity of prior information.
Initialization Method
LLaMA-2 7B
LLaMA-2 13B
HyperPrune SparseGPT prior
10.11
7.60
HyperPrune Wanda prior
10.28
7.79
HyperPrune (no prior)
10.75
8.04
26


Reviews:
[{"review_id": "iPuL6RTNOc", "reviewer": "Reviewer_c5Ha", "summary": "HyperPrune introduces a hypernetwork-based framework for n:m semi-structured pruning of LLMs. A shared, context-aware hypernetwork generates block-wise masks conditioned on layer embeddings, combined with continual-pruning and feature-outlier regularizers. The method provides an information-theoretic view linking reconstruction loss to mutual-information preservation, and achieves competitive performance on LLaMA-2 (7B/13B/70B) with up to 1.43× end-to-end speedup (2:4 sparsity) on A100 GPUs.", "questions": "1. Provide GPU-hour and wall-clock costs per model (7B/13B/70B) for mask training; how does this compare to MaskPro under identical calibration sizes? MaskLLM is excluded for cost; while MaskPro is compared, a tighter apples-to-apples system study (same calibration sizes, wall-clock, memory footprint, failure cases) would clarify where HyperPrune truly wins.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea of targeting n:m semi-structured sparsity with a shared, context-aware hypernetwork is interesting and novel to me.\n\n2. Results across 7B/13B/70B with comparisons to magnitude, SparseGPT, Wanda, Pruner-Zero, and MaskPro; includes data-scaling ablation\n\n3. An MI-based justification shows the reconstruction objective as a surrogate for information preservation under n:m constraints (Theorem 1)", "weaknesses": "1. All experiments are conducted exclusively on LLaMA-2, an outdated architecture. Given the architectural and activation differences in modern models (LLaMA-3, Qwen3, Gemma-3), it is unclear whether HyperPrune’s sparsity patterns or latency gains generalize to contemporary LLMs.\n\n2. The appendix reports per-layer GEMM latency improvements (≈1.6×) against the dense baseline on A100 using CUTLASS kernels, but provides no end-to-end inference benchmarks on realistic engines (vLLM, TensorRT-LLM) or comparisons with other 2:4 pruning methods. Because Ampere tensor cores inherently yield ~1.5× speedups for any valid 2:4 pattern, these numbers mostly reflect hardware properties rather than advantages of HyperPrune’s mask design. Without full throughput measurements (tokens /s, batching, attention caching), the claimed 1.43× overall speedup remains a proxy rather than a demonstrated system-level gain.\n\n3. Although the shared hypernetwork reduces parameter count, it brings typical drawbacks of hypernetwork-based methods—training instability, sensitivity to learning-rate and regularization hyperparameters, and potential overfitting to limited calibration data. Because the hypernetwork must generate masks for all layers jointly, cross-layer interference and catastrophic forgetting remain possible despite the continual-pruning regularizer. These issues make the approach potentially fragile and difficult to reproduce without careful hyperparameter tuning and large-scale validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gUSZvINWN2", "reviewer": "Reviewer_2WfL", "summary": "This paper proposes HyperPrune, a post-training framework for learning n:m semi-structured sparsity in LLMs. The method employs a lightweight shared hypernetwork conditioned on learnable layer and component embeddings to generate structured masks in a layer-wise manner. Two key regularizations are introduced: (1) continual pruning regularization to preserve cross-layer knowledge during sequential pruning, and (2) feature outlier regularization to retain critical activations. The authors provide a theoretical justification by showing that minimizing reconstruction loss is equivalent to maximizing mutual information between dense and pruned models under n:m constraints (Theorem 1). Experiments on LLaMA-2 models (7B/13B/70B) demonstrate competitive performance compared to both heuristic methods (SparseGPT, Wanda) and optimization-based methods (MaskPro) on a single A100 GPU.", "questions": "1.\tThe proof of Theorem1 assumes linear activation, but transformers use SwiGLU (LLaMA). How does this affect the theoretical justification in practice? Can you provide empirical validation that the Gaussian assumptions hold for actual LLM activations?\n2.\tFor hypernetwork, why is a 4-layer FFN chosen?  The hidden dimension is either 256 or 512. Is there a systematic way to choose this, or was it selected via trial and error?\n3.\tAll experiments use LLaMA-2. How well does HyperPrune transfer to other model families?\n4.\tSection C introduces prior-based initialization but provides no empirical evaluation. How much does this improve convergence speed or final performance? Is the method sensitive to prior quality?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tWell-motivated approach: The use of a shared hypernetwork with context-aware embeddings provides an efficient solution to the memory scalability challenge of optimization-based pruning methods, enabling layer-wise mask generation without requiring massive GPU resources.\n2.\tComprehensive experimental validation: The paper provides extensive evaluation across multiple model sizes (7B/13B/70B), diverse benchmarks (perplexity and zero-shot tasks), and thorough ablation studies analyzing the contribution of each component.\n3.\tTheoretical foundation: Theorem 1 provides an information-theoretic justification connecting mutual information maximization with reconstruction loss minimization, offering a principled view of the relaxation.", "weaknesses": "1.\tInferior perplexity on large models: On LLaMA-2-70B, HyperPrune achieves worse Wikitext perplexity (5.13) compared to Wanda (5.21), and significantly worse than Pruner-Zero (4.87). This challenges the claimed scalability advantage and suggests that the method's benefits diminish or even reverse at larger scales.\n2.\tInsufficient analysis of design choices: The hypernetwork architecture (4-layer FFN with hidden size 256/512) appears arbitrary with no justification or ablation on architecture depth/width.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aigLosoLgj", "reviewer": "Reviewer_8US3", "summary": "This paper relates to pruning LLMs under the constraint of n:m sparsity. This aims at leveraging, e.g. 2:4 sparsity that some computers support in hardware, potentially increasing math throughput by 2x.\n\nThis paper sets itself in the context of:\n* heuristic methods: fast, but approximate, and unable to support structured sparsity\n* optimization methods: accurate but computationally expensive, as they usually operate on the full model for a joint optimization.\n\nThe paper proposes an intermediate solution which applies a block-wise optimization, thus only one block (of e.g. 4 parameters) at a time needs to be loaded in memory for the optimization to take place. This is done via a shared hypernetwork (an MLP). Each weight is partitioned into sets of m parameters, and each set is provided as input into the hypernetwork, which outputs logits for mask selection (in the case of 2:4 sparsity, there are 6 possible masks, thus we need 6 logits). \nIn order to further improve the efficiency of the hypernetwork, the type of weight (q,k,v, o for attention, u,v,d for FFN) is provided as an input by ways of a contextual embedding.\nUnder the formulation, the hypernetwork would ignore cross-layer dependencies, thus a regularization loss is applied to encourage the hypernetwork to remain consistent with its pruning of previous layers.\nAdditionally, the authors observe that LLMs often exhibit intermediate features with very large magnitude. Given that these features are thought to be critically important, a regularization term is used to encourage the hypernetwork to preserve these features. \n\nThe method is validated with experiments that show competitive results against a set of heuristic and one optimization baselines (though MaskLLM is not included in the comparison due to the computational complexity). Results are shown for both perplexity (next token prediction) and accuracy (common benchmarks).", "questions": "On line 237, did you mean \"producing *6* logits from m=4\"?\n\nWhat is the embedding dimension $d$ for the layer-specific embeddings $e_\\ell$ and component embeddings $t$, and how sensitive is the method's performance to different values of $d$?\n\nWhat temperature annealing schedule is used for the Gumbel-Softmax relaxation during hypernetwork training, including initial/final values and decay strategy?\n\nFrom algorithm 1, I understand you train the hypernetwork for each layer until convergence (the \"for layer ℓ = 1 to L do\" is the outer scope). Did you try a round-robin approach (i.e. with the loop over calibration data being the outer scope?)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written and proposes an interesting intermediate solution between heuristic and optimization methods for n:m sparsity pruning.\n\nThe use of a shared hypernetwork to predict n:m masks is, from what I can tell, novel.\n\nThe theoretical (Theorem 1) provides a principled information-theoretic justification for using the reconstruction loss as a surrogate objective under n:m sparsity constraints.\n\nAblation studies provide justification for each component of the method.", "weaknesses": "No major weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Pndi6TNyK", "reviewer": "Reviewer_1Unw", "summary": "The paper introduces HyperPrune, a new framework for n:m semi-structured pruning of large language models (LLMs). Unlike traditional one-shot heuristics (e.g., SparseGPT, Wanda) or heavy optimization-based methods (e.g., MaskLLM, MaskPro), HyperPrune uses a lightweight shared hypernetwork conditioned on layer and component embeddings to generate pruning masks in a context-aware, layer-wise fashion.", "questions": "1. While the paper highlights HyperPrune’s scalability to 70B models on a single A100 GPU, it lacks a detailed breakdown of training or optimization costs (e.g., wall-clock time, computational budget) compared to other optimization-based pruning methods such as MaskPro or MaskLLM. A more explicit cost–performance analysis would strengthen the paper’s practicality claim.\n\n2. The experimental evaluation primarily focuses on the LLaMA-2 family. It remains unclear how well HyperPrune generalizes to other architectures.\n\n3. The current ablation section briefly evaluates the effects of embeddings and regularizations but does not isolate the impact of the proposed mutual information relaxation. Including such an analysis would better justify its theoretical contribution and clarify how much it contributes to the overall improvement.\n\n4. Although the paper compares HyperPrune against strong baselines like SparseGPT, Wanda, and MaskPro, adding more recent state-of-the-art pruning methods[1][2][3] would further contextualize its advantage.\n\n[1]Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Models (ICML 2024) \n\n[2]AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024)  \n\n[3]Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity (ICML2024)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The shared hypernetwork with conditional embeddings is a novel solution to the scalability problem in structured pruning, enabling n:m sparsity without massive parameter overhead.\n2. The mutual information theorem provides an interpretable bridge between discrete mask learning and differentiable optimization.\n3. Extensive experiment results on the LlaMA-2 family demonstrate the effectiveness of the proposed method.", "weaknesses": "1. The experiments focus primarily on LLaMA-2. It’s unclear how HyperPrune performs on other architectures (e.g., LLaMA-3, Qwen, Mistral, OPT)\n2. The hypernetwork setting and training detail is not sufficiently illustrated in the paper, including the network structure, parameters, and training settings.\n3. Some equations and notation (e.g., in Section 4.1) are dense and difficult to follow. Simplifying or visually decomposing them would improve readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)