WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference
Authors: Sihan Chen
Dan Zhao
Jongwoo Ko
Colby Banbury
Huiping Zhuang
Luming Liang
Pashmina Cameron
Tianyi Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Sparse Activation
Efficient Inference
Primary area: foundation or frontier models, including LLMs

Abstract: The ever-increasing computational demands of large language models (LLMs) make efficient inference a central challenge. While recent advances leverage specialized architectures or selective activation, they typically require (re)training or architectural modifications, limiting their broad applicability. Training-free sparse activation, in contrast, offers a plug-and-play pathway to efficiency; however, existing methods often rely solely on hidden state magnitudes, leading to significant approximation error and performance degradation. To address this, we introduce WINA (Weight-Informed Neuron Activation): a simple framework for training-free sparse activation that incorporates both hidden state magnitudes and weight matrix structure. By also leveraging the ℓ2-norm of the model’s weight matrices, WINA yields a principled sparsification strategy with provably optimal approximation error bounds, offering better and tighter theoretical guarantees than prior state-of-the-art approaches. Overall, WINA also empirically outperforms many previous training-free methods across diverse LLM architectures and datasets: not only matching or exceeding their accuracy at comparable sparsity levels, but also sustaining performance better at more extreme sparsity levels. Together, these results position WINA as a practical, theoretically grounded, and broadly deployable solution for efficient inference. Our source code is available at https://github.com/microsoft/wina.

Published as a conference paper at ICLR 2026
WINA: WEIGHT INFORMED NEURON ACTIVATION FOR ACCELERATING LARGE LANGUAGE MODEL IN- FERENCE
Sihan Chen²*, Dan Jongwoo Ko¹, Colby Banbury', Huiping Zhuang,, Luming Liang', Pashmina Cameron', Tianyi Chen Microsoff, RRemmin University of China, South China University of Technology thensihan@@run edu. en, Tiany Cwweeheeennneeeeeccee wwweeneiiiroooootccomm
ABSTRACT
The ever-increasing computational demands of large language models (LLMs) make efficient inference central challenge. While recent advances leverage spe- cialized architectures or selective activation, they typically require (re)training or architectural modifications, limiting their broad applicability. Training-fre free sparse activation, in contrast, offers plug-and-play pathway to efficiency; however, ex- isting methods often rely solely on hidden state magnitudes, leading significant approximation eтror and performance degradation. To address this, we introduce WINA WWight--Iffrrmet Neuron Activation): a simple framework for training- free sparse activation that incorporates both hidden state magnitudes and weight matrix structure. By also leveraging the f--n of the model's weight matrices, WINA yields a principled sparsification strategy with provably optimal approxi- mation error bounds, offering better and tighter theoretical guarantees than prior state-of-the-an approaches. Overall, WINA also empirically outperforms many previous training-free -fre methods across diverse LLM architectures and datasets: not only matching or exceeding their accuracy at comparable sparsity levels, but also sustaining performance better at more extreme sparsity levels. Together, these position WINA as a practical, theoretically grounded, and broadly deployable lution for efficient inference. Our code is available at gjihub.com/microsonn..
INTRODUCTION
While large language models (LLMs) have demonstrated impressive capabilities in variety of ap- plications such as text generation (Li al., 2024b; Cheng et al., 2025), translation (Hendy et al., 2023; sea, 2025), understanding (Chang et al., 2024; Tschannen et al., 2025), and computer us- ing agent (Xie et al., 2024; Hui et al., 2025). Their growing size and complexity often translate into a need for substantial computational resources, particularly during inference, making reducing inference costs without degrading output quality a key challenge.
One strategy is to only activate sub-network (Jacobs et al., 1991) during inference through archi- tectural changes such as Mixture of Experts (MoE), which has already seen adoption LLMs like GPT4 (Achiam et al., 2023), Mistral (Jiang et al., 2023), etc., or through model distillation (Moslemi et al., 2024) where a smaller model is trained using knowledge distilled from a larger teacher model to route inference requests more efficiently. However, these approaches typically require consid- erable amount of training which, in itself, can be computationally intense.
One alternative is training-free sparse activation (Liu et al., 2024a; Lee et al., 2024) which retains the original dense model but selectively omits weights/neurons at inference time. These methods avoid (re)training and be applied to off-the-shelf models, leveraging criteria such as hidden- state magnitudes, weight importance, weight statistics, or additional validation data to determine which parts of the model select to accelerate inference.
Equal contributions. +Corresponding author.

---

Published as a conference paper at ICLR 2026
Nonetheless, current training-free sparse-methode face several limitations. Most notably, they ignore the influence of weight matrices on error propagation: these approaches fail to account for how interactions between imput elements and the weight matrix during forward propagation can affect model outputs, which lead to accumulating approximation errors in sparse activation.
Contributions. We propose WINA: a simple, easy-to-use, training-free framework that performs sparse activation based both the magnitude of hidden states and the column- wise fo-norm of the weight matrix. By combining activation strength with weight importance, our thresholds directly account for how much each activation influence the next layer. This design provides theoretical guarantees on bounding the total approximation error in a way better than that of other approaches.
In contrast, methods like TEAL (Liu et al., 2024a) rely exclusively the distribution of hidden- state magnitudes to decide which activations to keep. However, ignoring weight magnitudes in this way can discard highly influential activations while retaining many low-impact ones, leading to sub- optimal trade-offs between efficiency and output quality. These types of approaches overlook how the weight matrix directly influences imput features during the forward pass; by focusing solely on activations, this omission can result in compounding approximation errors that ultimately skew the calculation of sparse activations and degrade the accuracy of the sparse model more than necessary. Our framework overcomes these limitations by integrating weight statistics into the selection pro- cess, achieving finer control over sparsity and tighter bounds on the resulting approximation error.
We evaluate WINA on several widely-used LLMs (ranging from 7B to 14B) across sev- eral popular benchmarks. Compared with ex- isting training-free sparse activation methods like TEAL (Liu et al., 2024a), CATS (Lee et al., 2024), and R-Sparse (Zhang et al., 2025), WINA achieves superior model performance at comparable sparsity levels with significantly less performance degradation. We also establish tight theoretical error bounds for WINA, providing formal support for our experimental results validate its effectiveness. In short, our contributions can be summarized as follows:
Table I: Comparison between WINA and others.
WINA
TEAL
CATS
R-Spurse
Tight Approx Error Layer-A Aayee--gnostle Application
Layer Speper--Secihceee Spursity
Some methods (i... CATS) are only adogted specific types layers.
Weighted- Informed W Activation. We introduce novel sparse activation method that jointly siders hidden state magnitudes and the column-wise f2-norms of weight matrices. This allows for selecting neurons that are not only strongly activated but also those that have a larger influence downstream layers, leading to a more informed construction of a sub-network during inference.
We conduct formal analysis to demonstrate that our veight-informed activation mechanism yields a lower expected output error compared to prior methods (e.g., TEAL) under mild assumptions.
Theoretically Tighter Approximation Error.
Numerical Experiments. We conduct extensive evaluations across multiple LLM families, in- cluding the Illama series (Touvron et al., 2023), Phi-4 (Abdin et al., 2024), and Mistral (Jiang al., 2023), and show that our method consistently achieves superior accuracy across range of sparsity levels. In particular, WINA sustains higher performance as sparsity increases, high- lighting its robustness and practical applicability across diverse tasks and model scale. Moreover, we also demonstrate WINA's 's compatibility with quantization, achieving promising results under both 4-bit and 8-bit settings, WINA's impact on models' long-context reasoning abilities, and its impact on social biases in models-aspects largely overlooked in prior works. Lastly, we provide a Triton kernel to benchmark WINA's competitive speed-ups against methods like TEAL.
The rest of our paper is organized as follows. We begin with related works in Section 2 and detail our methodology Section 3. We present our results in Section and conclude in Section 5.
RELATED WORK
Sparse Activation. Modern sparse activation approaches fall into two principal paradigms: training- methods and training-free free methods. Training-based methods typically employ trainable router to learn to dynamically select activated experts for each token, with the Mixture- of- of-Fxperts (MoR) architecture (Jacobs et al., 1991) serving the foundational framework.
2

---

Published as a conference paper at ICLR 2026
Dense Linear Layer
INT.
WINNA
WINA Linear Layer
Norm
Activation
LLM
Weight
Reduced FLOPs
Colamn-wise fyrtorm of W
Figure 1: Overview of WINA. WINA performs training- free sparse activation by selecting the most influential input dimensions based on both activation magnitude and the column- wise forms of weight matrices. This joint criterion effectively extends to layers of varying precision, including quantized layers, ensuring accurate sub-network activation at each layer during inference.
This paradigm has been expanded through many iterations and variants. The sparsely-gated mixture of experts layer (Shazeer et al., 2017) 7) integrates MoE into recurring neural networks (RNNs). Works like GShard (Lepikhin et al., 2020) and the Switch Transformer (Fedus al., 2022) extend MoEs to the Transformer architecture (Raffel et al., 2020) while others combine several approaches, such as WideNet (Xue et al., 2022), reduces the size of the MoE model by initially compressing the model before transitioning into MoE. Works like MoEBert (Zuo et al., 2022) decomposes the FFN layer of a pre-trainee dense model into multiple experts based on importance-guider adaptation and then refines the model through distillation. LLM in Flash (Alizadeh et al., 2023) employs a low-rank predictor to determine which intermediate neurons are activated.
Training-free methods, in contrast, do not rely teria to perform sparse activation such as TEAL (Liu et al., 2024a), TDA (Ma et al., 2024), and SEAP (Liang et al., 2025). Methods (Han al., 2015b) utilize magnitude-base ing or global activation pruning (Wen et al., 2016b) to apply a fixed sparsity pattern regardless of input. For instance, Q-Sparse (Wang et al., 2024) produces sparsity as function of input magni- tudes, achieving high sparsity with reasonable performance degradation. CATS (Lee et al., 2024) applies sparse activation on SwiGLU within gated MLP layers, achieving reasonable sparsity while maintaining performance. In contrast, TEAL (Liu et al., 2024a) extends magnitude-baze sparsity all layers, achieving high model-wied sparsity. However, current sparse activation meth- ods tend to focus solely on selecting activation elements based on the magnitude of hidden states, which can result suboptimal performance.
learnable router, instead using predefined cri-
weight prun-
activation
Model Pruning. Another line of related work is model pruning, which removes redundant neurons from deep neural networks (Han et al., 2015a; Frankle and Carbin, 2018; Frantar and Alistarh, 2023; Li et al., 2024a) to obtain compact yet high-performing sub-networks. While effective, pruning typically requires fine-tuning 2016a; Li et al., 2020; Zhuang et al., 2020; Chen et al., 2021b; 2024), introducing additional training overhead. Such retraining stages make pruning less practical for large foundation models.
recover accuracy (Lin et al., 2019; He al., 2018; Wen et al.,
METHODOLOGY
We now present WINA, a framework for sparse activation that preserves critical elements while zeroing out non-essential components in each layer's input. As illustrated in Figure 1, WINA WINA jointly considers both the imput activation and the associated weight matrix, rather than relying solely on activation magnitudes. During inference, it activates only the most influential neurons, effectively constructing a sparse sub-network that maintains the expressive power of the original model.
3.1 PROBLEM STATEMENT
Problem. Consider a deep neural network (DNN) M consisting of T. layers. We denote the weight matrix of the [-th layer as w(() e Rmixn and the corresponding input as an arbitrary tensor X e Reixs for 1 {1,..., L}, representing the full information content. Our goal is identify set of binary activation gates (1) (").) (f)) where each ( (0, 1) th is binary vector, such that
3

---

Published as a conference paper at ICLR 2026
the deviation between the model's original output and the gated output is minimized:
minimize g()))=g(()))
[IM(X)) - MM 1))l2-
(1)
Since obtaining the complete of possible inputs X is generally infeasible, we instead sampled subset X for approximation. The activation gating operates in the input vector space to reduce output deviation. With this observation, we can reformulate the original problem into more tractable per-layer version.
Given a weight matrix W e mmxx and input vector x' E R", the standard linear transformation is given by y Wæ. Our objective then becomes identifying an activation gate or mask g {0, 1 such that the masked output Bg + (g o x) approximates the original:
Refined Problem.
minimize ge(0,1)
| x (gg 2 22
3.2 WEIGHT INFORMED GATE FUNCTION
Motivation. Many current sparse activation methods (e.g., Q-sparse (Wang al., 2024), CATS (Lee et al., 2024), TEAL (Liu et al., 2024a)) operate via a top-K gating mechanism governed by the absolute values of the hidden states:
if | is among the top- K values in |z],
otherwise.
However, this approach ignores the critical role that weight matrices themselves play: more specifi- cally, it ignores how each element of the preceding imput interacts with the weight matrix W during the forward pass. This mismatch is what motivates us to propose WINA, a method that jointly con- siders both inputs and weight matrices to minimize the approximation error for better performance.
In WINA, we construct binary activation gates by selecting the top-K components via:
[SwINA]s = {1 |z,c,| among the top- K values in |x O cl.
(4)
otherwise,
where C R.t represents the column-wise f2-norm of W and @ denotes the Hadamard or element- wise product. A smaller K results in deactivated neurons, thereby saving more FLOPs while potentially sacrificing performance. The choice of K is flexible and adaptable, ranging from coarse-grained universal criterion where a shared K applied across all layers to a fine-grained layer-specific strategy that assigns K individually to better minimize performance degradation. In the meantime, WINA is architecture agnostie, applicable across different layers, such as attention layers, multi-layer perceptions (MLPs), residual connections, etc.
3.3 THEORETICALLY OPTIMAL APPROXIMATION ERROR
WINA also comes with theoretical guarantees, establishing a tighter bound on the approximation error than prior approaches under relatively mild assumptions. We first show that for single linear layer network, WINA yields an optimal solution to the target problem (Eqn. 2).
Lemma 3.1 (Optimal approximation error over single linear layer). Let e IR" be an input vector and W e moxn be a matrix satisfying column-wise orthogonality: W is diagonal matrix. For any target density level N satisfying K n, the deviation between the original network output and the gated output via WINA is optimal. Formally,
gWINA (x: argmin g=(0,1}"
I (le-W(goz)lo
where is the input. and gWINA (x) is the gating function of WINA retaining the K elements activated with the largest (2.. 22 for (1."..n).
Proof. See Appendix A.7.
Building upon our single linear-layer Lemma 3.1, we now extend it to deep L linear layer networks M and present that WINA could tighten the gated error upper bound.
4

---

Published as a conference paper at ICLR 2026
Theorem 3.2 (WINA minimizes a provable upper bound output deviation). Consider an L-layer linear network M(x) W where for each i 2 2 the weight matrix satisfies W(OT = D(9) with D(O) diagonal (column-orthogondiig Let G = =(g(1) "(g(t).....
be gating variables with g()) 10,1)dr and M( ()) Let M(x 9) denote the gated network output. Define the output deviation E) = (
G) Then there exists a separable upper bound ((x;9) s (xxgg)) Moreover, minimizing U 9) reduces selecting, each layer, the k largest coordinates weighted by squared column norms. Therefore WINA satisfies
GWINA = arg mx:
g
Proof. See Appendix A.8 for details.
Remarks on Column-Wise Orthogonality. Our analysis relies on the column-wise orthogonal- ity of the relevant weight matrices. We applied an efficient one-off offline tensor transforma- tion from (Ashkboos al., 2024a) to enforce column orthogonality. This pre-pocessiin step lightweight, does not change the functional capacity of the model, and enables our theoretical guar- antees to translate effectively to practical settings. We leave details Appendix A.2.
3.4 THEORETICAL VALIDATION WITH SYNTHETIC EXPERIMENTS
To validate our theoretical analysis, we first conduct controlled synthetic experiments on randomly initialized networks under the assumptions of Lemma 3.1 and Theorem 3.2.
We initialize input vectors and weight matrices using Kaiming initialization with the SiLU activation function, and enforce column-wise orthogonality of weight matrices via the tensor transformation described in Appendix A.2. We then compare the dense network output against sparsified outputs generated by CATS, TEAL, and R- R-Sparse. Since CATS and TEAL share the same underlying mech- anism and differ only in their sparsity distribution, we group them together as baseline.
Performance is quantified by the l2 deviation between the dense output and the sparsified out- puts across varying sparsity ra- tios. Each experiment is re- peated with 20 random seeds. report averaged results as- sociated with detailed error bar analysis in Table 2. WINA con- sistently achieves lower approx- imation error than competing methods across all sparsity levels and theoretical settings. Notably, WINA reduces eиror about 50% compared to others, aligning with our theoretical guarantees. Build- ing on this, we turn next to evaluating WINA on LLMs more realistic settings.
Table 2: Approximation errors of different methods over ran-
domly initialized networks and sparsity levels. Lower is better.
Theory
Method
25%
40%
50%
68%
CATS/TEAL
0.11
+ 0.20
4,86 = 0.23
7.55 +
Lemma 3.1
R- R-Sparse
0.12
3.48 0.20
5.01 = 0.30
7.75 + 0.35
WINA
0.05
1.73 17330..00
2.70 0.13
4.75
CATS/TEAL.
0.73
1.44 0.06
2.04 0.11
3.02 + 0.18
Theorem 3.2
R SpSrae
0.77 0.04
1.51 0.07
2.11 0.11
3.13 0.16
WINA
0.38 0.02
0.76 a 0.04
1,09 = 0.06
1,76 = 0.08
EXPERIMENTS
4.1 SETUP
Models. To demonstrate WINA's effectiveness across different model families and sizes, pro- vide our results on four commonly used LLMs: Llama-2-7B (Touvron et al., 2023), Llama-3- 8B (Dubey al., 2024), Mistral-7B (Jiang et al., 2023), and Phi-4-14B (Abdin et al., 2024).
Evaluation. We use the Im-evaluation-haree
pipeline (Gao et al., 2023) to assess WINA across a diverse suite of tasks. In addition to the commonsense reasoning examined works like R- Sparse (Zhang et al., 2025), we also consider general reasoning, mathematics, and code generation. For commonsense reasoning, we evaluate on PIQA (Bisk et al., 2020). WinoGrande (Sakaguchi et al., 2019), HellaSwag (Zellers al., 2019), BoolQ (Clark et al., 2019), Arc Challenge and Are Easy (Clark et al., 2018). SciQ (Johannes Welbl, 2017), and OpenBookQA (Mihaylov et al., 2018), which measure capabilities such as applying everyday knowledge and resolving coreferences to and
5

---

Published as a conference paper at ICLR 2026
answer questions. For general reasoning and knowledge, consider MMLU (Hendrycks et al., 2020), which spans domains across STEM, humamities, and social sciences, providing a compre- hensive test of broad reasoning and knowledge transfer. For math, adopt GSM8K (Cobbe et al., 2021), which examines multi-step arithmetic problem solving. For code generation, we use Hu- manEval (Chen et al., 2021a) which requires synthesizing executable Python code. We note that WINA has a closed-form mechanism characterized in Eqn. (4), making it deterministic given an input. Therefore, there no error bars report for WINA's performance.
Baselines. We compare WINA with three recent, representative sparse activation methods: CATS (Lee et al., 2024), R-Sparse (Zhang al., 2025), and TEAL (Liu et al., 2024a). We as- sign layer-specific sparsity ratios instead of a uniform sparsity across the model so that, given a global sparsity target, we leverage the greedy algorithm proposed in TEAL (Liu et al., 2024a) to iteratively configure per-layer sparsity levels so that the aggregate sparsity meets the global budget.
4.2 RESULTS ON COMMONSENSE REASONING
Following the experimental setup of works like R-Sparse (Zhang al., 2025), we provide em- pirical comparison of WINA against various baselines (e.g.. CATS, R-Sparse, and TEAL) across different sparsity levels, ranging from 25% 65% on commonsense reasoning tasks demonstrate effectiveness across various experimental settings. Figure 2 provides an comparative overview of the performance seroorrtnne-sprssttttyy trade-offs across the different methods.
TEAL
WYNA (Durs)
R-Sparse
CATS
Liaaa 2-7B
Liama-3-80
Mistral-78
70
Coro Speraity 12.32
2rm Spaccty
68
70.0
Deftomeree
66
87.5
00.0
57.5
25
40
50
65
25
40
50
05
25
40
50
Sparsty (%)
Spansty 1961
Sparshy (96)
Figure 2: Performance-Sparrsity Frontier. Mapping average performance at each sparsity level for various techniques. WINA outperforms other methods (TEAL, R-Sparse, CATS) across sparsity levels with the performance gap increasing as sparsity increases. We note that CATS is unable to achieve higher model sparsity due to its applicability to only certain layers.
Table 3:
Results over Llama-2-7B on commensense reasoning.
Spursity
Method
PIQA
Are-C
WinoGrande
HellaSwag
SciQ
OBQA
BoolQ
Arc-E
Avg
0%
Baseline (full model)
79.05
46.25
68.90
76.00
91.00
44,20
77.77
74.58
69.72
CATS'
77.20
44.20
67.48
75.75
90.10
43.40
74.31
71.89
68.04
25%
R-Sparse
78.89
45.99
68.35
75.60
91.70
43.00
77.71
74.58
69.48
TEAL
78.73
68.90
75.98
77.09
74.24
WINA
78.40
46.16
69.38
75.93
90.90
44,00
77.03
74.92
69.59
CATS'
74.70
36.86
61.88
69.25
79.90
39.00
66.06
58.96
60.83
40%
R- R-Sparse
78.18
43.60
67.09
74.57
91,40
42,80
75.72
73.32
68. 34
TEAI.
78.07
45.22
66.85
75.31
91.40
43,20
76.48
72.69
68.65
WINA
78.40
45.31
68.59
75.48
91.50
42.40
76.79
74.28
69.09
R-Sparse
76.93
42.32
65.82
72.25
91.60
41.00
74.46
71.84
67.03
50%
TEAL
77.09
42.92
68.03
73.54
90.20
43.20
74.07
71.55
67.58
WINA
77.80
44.71
68.51
74.43
91.00
44.60
75.84
73.44
68.79
R-Sparse
71.93
34,22
59.04
59.63
87,50
34,20
66.91
61.53
59.37
65%
TEAL
73.61
36.09
62.12
63.58
84.00
38.00
69.24
61.95
61.07
WINA
75.35
39.68
65.82
69.03
89.50
40.40
72.78
68.60
65.14
t CATS is unable to reach 50%
65% sparsity since it only achieves sparse activations over MLP layers.

---

Published as a conference paper at ICLR 2026
Table 4:
Results over Llama-3-8B on commonsense reasoning.
Spursity
Method
PIQA
Arc-C
WinoGrande
HellaSwag
SciQ
OBQA
BoolQ
Arc-E
Avg
0%
Baseline (full model)
80.79
53.33
72.61
79.17
93.90
45.00
81,38
77.74
72.99
CATS'
78.62
48.04
70.64
76.32
91.90
41.80
78.13
71.09
69.57
25%
R-Sparse
79.82
$2.05
72.38
78.69
93.50
44,40
80.92
78.75
72.56
80.20
53.16
73.32
94.10
45.20
80.83
76.89
72.82
WINA
80.41
52.82
73.80
78.99
94.00
44,60
82.05
78.03
73.09
CATS'
59.96
27.82
51.30
40.18
46.10
29.80
42.26
38,09
41.94
R-Sparse
79.05
50.26
72.14
76,91
94,10
43.00
79.14
77.86
71.56
40%
TEAL
79.00
48.98
71.82
77.45
93.30
45.00
80.03
77.19
71.60
WINA
79.87
50.68
72.30
77.91
93.90
45.00
82.23
77.57
72.43
R-Sparse
76.22
45.73
66.61
73.22
93.80
42.20
76.70
74.83
68.66
50%
TEAL
78.29
48.12
70.09
74.83
93.70
42.60
78.23
74.41
70.03
WINA
79.16
48.81
70.64
76.44
93.50
43.60
81.25
75.00
71.05
R-Sparse
68.50
33,36
57.38
51.48
86.00
31.80
65.23
58.80
56.57
65%
TEAL
73.34
37.37
63.46
61.76
88.90
37.00
69.85
64.48
62.02
WINA
74.65
41.98
64,48
67.89
90.70
41,60
76,73
67.00
65.63
t CATS is unable to reach 50% 65% sparsity since it only achieves sparse activations over MLP layers.
Llama-2-7B. On Llama-2-7B, WINA shows strong performance under various sparsity con- straints. As shown in Table 3, WINA achieves the highest average accuracy at 25% sparsity, out- performing CATS, R-Sparse and TEAL. While performance naturally degrades at the more extreme 65% sparsity level, WINA still offers the best accuracy, scoring +5.77 % higher than R-Sparse and +4.07 % higher than TEAL, suggesting its robustness under aggressive pruning-
Llama-3-8B. Across all sparsity levels, WINA (Table 4) consistently outperforms or matches best baseline methods. At 25% sparsity, WINA achieves the highest average score of 73.09%, surpassing the full baseline model (72.99%) and other sparse methods. This trend continues at higher sparsity levels: WINA maintains strong performance at 40% and 50% sparsity with average scores 72.43% and 71.05% respectively. Even at 65% sparsity, WINA remains competitive while other methods experience significant degradation, on average scoring +3.61 % higher than TEAL and +9.06 % higher than R-Sparse.
Table 5: Results over Mistral-7B on commensense reasoning.
Sparsity
Method
PIQA Are-C
WinoGGrande
HellaSwag
SciQ
OBQA
BoolQ
Arc-E E
Avg
0%
Basline (full model)
82.05
54.01
73.88
81.06
93.90
43.80
83.61
79.59
73.99
CATS
81.01
52.22
72.38
80.86
92.00
42.40
82.60
77.06
72.57
R-S R-Sparse
81.61
53.41
74.19
81.06
94,10
45.60
83.73
78.83
74.07
25%
TEAL
82,21
53.75
73.48
80.93
94.10
44.00
83.52
78.87
73.86
WINA
82.10
53.24
80.88
83.30
73.90
CATS'
75.95
41.13
63.61
69.27
78.90
36,20
68.07
60.86
61.75
R-Sparse
81.77
51.62
70.09
77.56
91.50
39,20
83.09
78 54
71.67
40%
TEAL
81.56
52.30
73,40
80.03
93.80
43.00
83.21
78.07
73.17
WINA
82.05
53,33
72.38
80.41
94.30
43.60
82.87
78.83
73.47
R- Sparre
81.07
49.74
66.22
69.39
89.50
34,20
82.14
77.36
68.70
50%
TEAL
79.76
49.57
70.80
78.84
93.90
43.00
82.75
71.93
WINA
81.34
52.30
70.88
79.60
94.50
41.80
81.59
77.57
72.45
Sparse
63.60
36.52
63.77
59.30
88.00
30.20
74.16
43.64
57.40
65%
TEAL
77.97
42.49
66.22
72.26
92.20
40.40
79.27
72.47
67.91
WINA
78.24
48.12
66.38
75.11
92.90
42.00
77.65
74.75
69.39
+ CATS unable to reach 50% 65% sparsity since it only achieves sparse activations over MLP layers.
Mistral-7B. Overall, WINA matches or outperforms or matches the best performing methods across different sparsity levels (Table 5). At moderate sparsity (25-40%), WINA maintains per- formance nearly identical to the full model baseline, showing strong robustness to pruning. WINA's performance becomes most evident under extreme sparsity: at 65% sparsity. WINA achieves erage score 69.39%, which is +9.48 % higher than R-Sparse (59.91 (9.91% %) and +1.48% higher than TEAL (67.91 %), demonstrating that WINA more effective under aggressive compression.

---

Published as a conference paper at ICLR 2026
4.3 ADDITIONAL ANALYSIS AND ABLATIONS
Additional Benchmarks and Architecture. We conduct additional experiments several ad- vanced tasks, including general reasoning (MMLU), mathematics (GSM8K), and coding (Hu- manEval). Additionally, we use Phi-4-14B architectures. To focus our analysis and reduce clutter, we compare against TEAL, the second best performing method Section 4.2 as an upper- upper-bound. bound. We detail our results in Table 6.
the adaptability of WINA different model
We observe that WINA consistently delivers better performance than TEAL across all sparsity levels and tasks, highlighting robustness on diverse applications, ranging from commonsense reason- ing and subject knowledge to mathematical reasoning and code generation. Similar to our findings from Section 4.2, WINA's performance advantage becomes even more pronounced at higher spar- sity levels, demonstrating its superior scalability. At 65% sparsity, the improvements over TEAL considerable: +1.65% for Commonsense Reasoning, +4.88 for MMLU, +2.73% for GSM8K, and +9.14% % for HumanEval. Notably, HumanEval, WINA even surpasses the baseline, a phe- nomenon often observed pruning literature (e.g.. see Section 2) as its sparse activation may also be implicitly regularizing or suppressing harmful neurons, thereby enhancing performance.
Table 6: Results over Phi-4-14B over additional benchmarks.
Sparsity
Method
Commonsense (avg)
MMLU
GSMBK
HumanEval
0%
Baseline (full model)
74.38
77.06
90.22
$0.61
TEAL
74.12
76.63
89
46,95
25%
WINA
74.25
76.60
90.22
50.00
TEAL
73.49
75.10
88.02
45.73
40%
WINA
73.76
76.44
90.67
53,00
TEAL
72.73
73.52
86. 13
41.46
50%
WINA
73.51
87.57
51.83
TEAL
69.93
65.17
74.37
32.32
65%
WINA
77.10
41.46
Computational Savings. In addition performance gains, WINA also yields computational sav- ings; as shown in Figure 3, WINA reduces the overall (G)sLOPs by up to 63.7% on Llama-2-7B and Mistral-7B, 60.4% on Llama-3-8B, and 62.7% Phi-4-144 14B 65% sparsity, potentially translating to faster inference and lower computational costs under tight resource constraints.
7%6
62. 7%
63.7%
63
40.4%
60
49.0%
46.5%
48.2%
48 0%
Reduction (% GFLOPS) Full
50
37.215
40
24.5%
23.2%
24.1%
2450
30
20
10
25%
40%
50%
65%
Sparsity
Llama-2-78
Phi-4-14B
Llama-3-88
Mistral-7B
Figure 3: Computational Savings from WINA. Percentages indicate the reduction GFLOPs at the specified sparsity level vs. the full dense model (higher is better).
8

---

Published as a conference paper at ICLR 2026
Compatibility with Quantization. In re- source constrained environments, LLMs are typically deployed in quantized form lower precision (Ding et al., 2023). Although prior works rarely evaluate the robustness of their methods under quantization, we show that WINA is naturally compatible and apply it onto quantized versions of Llama-2-78 2-7B to assess performance on diverse commonsense reason- ing tasks. As shown in Table 7, WINA incurs only minimal degradation, consistently outper- forming TEAL;; similar to before, the perfor- mance gap between WINA and TEAL increases as sparsity increases. Most notably, at 65% sparsity, WINA outperforms TEAL by +4.07% on 16-bit, +3.86% % on 8-bit, and +3.68 on 4-bit quantized models. We leave the treatment of sparsification-awal quantization (e.g., joint sparse- aciivatin-quaatizatioo (Lin et al., 2024)), to future work.
Table 7: Llama-2-7B results across sparsity and quantization levels on commonsense reasoning.
Sparsity
Method
FP
INTs
INT,
0%
Baseline (full model)
69.72
69.39
68.06
TEAL
69.52
68.09
67.93
25%
WINA
69.59
68.39
68.06
TEAL
68.65
67,66
67.39
40%
WINA
69.09
68.16
TEAL WINA
67.58
66.41
66.00
68.79
67.28
66.50
TEAL
61.07
38
59,99
65%
WINA
65.14
64.24
63.67
Ablative Study on Orthogonalization. To control for potential confounding effects from the transformation process, introduce additional baseline, TEAL-Transform, where TEAL is also applied the transformed model, retaining the K elements with the largest ab- solute values |zc]. Table 8 shows the average performance over commonsense datasets from our main experiments, Consistent with earlier results, WINA still outperforms in most cases with the gap widening as sparsity increases, demonstrating that gains arise primarily from WINA's gating function.
Table 8: Ablating the effects of orthogonalization.
Meihod
Spursity
Llama-2:17
Llanna 3-8B
25%
TEAL (transform)
69.70
72.91
WINA
69.59
73.09
TEAL (transform)
69.11
72.05
40%
WINA
69.09
72.43
TEAL (transform)
67.97
70.36
50%
WINA
68.79
71.05
TEAL (transform)
63.05
63.52
65%
WINA
65.14
65.63
Social Bias. To account for the effect of model compression on potential social/model biases, following (Gonçalves and Strubell, 2023), we evaluate our sparsified models on CrowS-Pairs (Ta- ble 13). Across all models and sparsity levels, we observe no systematic increase in bias, suggesting WINA does not exacerbate bias and can modestly mitigate it. We leave full details to Appendix A.6.
4.4 HARDWARE ACCELERATION
To achieve realistic speed-ups, we develop a dedicated sparse GEMV kernel for WINA using Tri- ton (Tillet al., 2019). This kernel takes the input tensor æ, matrix W. pre-computed column- wise norm e of W, and a desired sparsity level to return an output x @ g. WINA gating is computed via Eq. (4), selecting subset of columns of W before the input is fed into the matrix multiply. The weight norms e are pre-computed offline once during model loading. Compared to TEAL's sparse kernel, WINA only introduces an additional element-wise product
c, whose cost negligible.
In particular, let d be the hidden dimension, B be the batch size, and T' be the sequence length so that standard linear layer incurs O(Bdd² FLOPs during the forward pass while WINA's 's gating mechanism requires (BBdd) FLOPs. Comparing the two, their ratio is on order of (BBT)))(BBT&) O(1d). Since typically large modern LLMs. (e.g, 2048, 4096 or larger), WINA consumes only << 0.1% additional overhead, which is negligible compared to the acceleration and accuracy improvements that its sparsification provides.
To empirically verify the negligible runtime overhead, we benchmark our Triton WINA kernel across commonly used matrix shapes in modern LLMs and batch sizes 1, 64, 256. To avoid hardware- specific bias, we profile on three GPU architectures: A800, A100, and the RTX PRO 6000 Black- well. Results in Figure 4 show that WINA matches TEAL's speed/latency almost identically. This phenomenon is consistent across all tested hardware platforms and batch sizes. Due space siderations, we leave results on A800 and additional details to Appendix A. Additionally, we in- vestigate how the speed-up up ratio varies with the batch size (Appendix A.4). As batch size increases

---

Published as a conference paper at ICLR 2026
Bm
B=256
0.34
Time (ms)
0.12
1100
0.30
0.30
RTX PRO 6000 Blackwelll
0.84
Time (ins)
0.82
0.26
0.80
0.24
0.78
0.2
0.3
044
0.2
0.3
0.4
0.5
0.3
0.6
Sparsity (96)
Sparsity (96)
(%)
Figure 4: Sparsity vs. lateney across different batch sizes B E {1, 64, 256} for GEMV (generalized matrix-vector multiplication) of sizes x and 5120x 17920. WINA's Triton kernel performance consistently matches that TEAL across different GPU architectures (A100 top, RTX PRO 6000 Blackwell bottom), achieving similar speedups across our sparsity levels and sparsity increases.
and GPU memory bandwidth becomes saturated, the achievable speed-up gradually decreases. Nev- ertheless, degradation is mild and WINA still delivers substantial acceleration, achieving approxi- mately up to 42% speed-up A100 and up to 57% RTX PRO 6000 Blackwell across different batch sizes, similar to TEAL.
4.5 PERFORMANCE ON LONG CONTEXT TASKS
We further evaluate the long-context reasoning ability of models under WINA on LongBench (Bai et al., 2024), multi-task benchmark with realistic long-context scenarios involving code comple- tion, summarization, as well single and multi-document QA. LongBench contains 4,750 examples with average task length ranging from 5-15K. We evaluated Llama-2-7B. Llama-3-8B, and Phi-4- 14B, which support context windows of 4K, 8K, and 16K tokens, respectively. Table shows results for Llama-2-7B at 4K where, across all settings, WINA consistently outperforms TEAL even more demanding scenarios. We leave additional results to Appendix A.5.
Table Performance on LongBench 2-7--k2-7B-4K)
Spariity
Method
Code Completion
Fow-shot Learning
Summari- zation
Multi- Document
Single- Document
Synthetic Tasks
Overall
QA
QA
Baseline
52.09
11.63
13.00
22.62
TEAL
51.80
11.83
7.24
12.87
22.59
0.25
WINA
62.54
52.46
12.48
7.07
12.95
5.26
22,89
TEAL
52.23
22.83
D.4
WINA
61.52
51.76
13.81
7.63
13.64
4.97
23.11
TEAL
58.41
50.10
12.97
8.39
12,80
4.28
22.23
WINA
61.15
$2.16
14.62
7.41
12.09
3.13
22.71
TEAL
44.20
44.15
11.16
7.09
8.17
1.62
17.88
0.65
WINA
48.93
5 CONCLUSION
WINA offers not only a simple yet highly effective training-free sparse-activation for accelerating LLMs but also solid thcoretical guarantees on its approximation error. Our results show that consistently outperforms strong baselines across many benchmarks, while maintaining robustness even under aggressive 4-bit and 8-bit quantization. These results highlight both the practicality and the plug-and play usability of WINA, making it ideal for sparse- aactivationacceleeese inference.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our work introduces a training-free sparse activation framework designed to improve the efficiency of large language model inference. The primary ethical consideration is ensuring that such efficiency gains are not misused to scale potentially harmful or biased models more widely without proper safeguards. We emphasize that WINA is a general-purposs acceleration method and does not alter the underlying datasets or model outputs. As such, any societal risks or biases present in the base models remain unchanged, and practitioners must remain vigilant about responsible deployment. Additionally, by lowering computational costs, our approach may promote accessibility of research to institutions with limited resources, contributing to broader inclusivity in Al research.
REPRODUCIBILITY STATEMENT
We have taken steps to ensure the reproducibility our results. All experiments were conducted on publicly available benchmark datasets. We used standard open-source evaluation pipelines such as im-evaluation-hamae for benchmarking. Detailed experimental settings are reported in Section 4, including models (e.g.. Llama-2, Llama-3, Mistral-7B, and Phi-4), sparsity configurations, and com- putational resources. Comparisons were made against established baselines under controlled spar- sity levels, and we provide theoretical analyses and ablation studies support our claims. The total run time of our experiments were run using one A100 80GB GPU for several days along with a few hours on a RTX PRO 6000 Blackwell and A800 for minor benchmarking, Our source code made anonymously available at URL.
REFERENCES
Joint speech and text machine translation for up to 100 languages. Nature, 637(80466:587-593, 2025.
M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Java- heripi, P. Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024.
Achiam, S. Adler, Agarwal, I., Ahmad, I. Akkaya, L.. Aleman, D. Almeida, Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
K. Alizadeh, I. Mirzadeh, D. Belenko, K. Khatamifard, M. Cho, C. C. Del Mundo. M. Rastegari, and M. Farajtabar. Llm in a flash: Efficient large language model inference with limited memory. arXiv preprint 23 /114, 2023.
Ashkboos, M. Croci, M. G. do Nascimento, Hoefler, and Hensman, Slicegpt: Compress large language models by deleting rows and columns, 2024a. URL btttps://arxiv..r /20/22001. 15024.
S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, P. Cameron, M. Jaggi, D. Alistarh, T. Hoefler, and Hensman. Quarot: Outlier-free 4-bit inference in rotated 1mss. Advances in Neural Information Processing Systems, 37:100213-100222 2024b.
Y. Bai. X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, Tang, and J. Li. LongBench: bilingual, multitask benchmark for long context under- standing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1119-3137, Bangkok, Thailand, Aug. 2024. As- sociation for Computational Linguistics. doi: 10. 11116663//1/0222.aclloooooo //acl anthol ogy. wrroor////0 2024. ac1-1ong.
URL https:
Y. Bisk, R. Zellers, J. Gao, Y. Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45, 2024.
11

---

Published as a conference paper at ICLR 2026
J. Chee, Y. Cai, V. Kuleshov, and C. M. De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36:4396 4429, 2023.
M. Chen, J. Tworek, H. Jun, Q Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, Such, Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang. I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, Leike, Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and
W. Zaremba. Evaluating large language models trained on code, 2021a.
T. Chen, B. Ji, T. Ding, B. Fang, G. Wang, Z. Zhu, L. Liang, Y. Shi. S. Yi, and X. Tu. Only train once: A one-shot neural network training and pruning framework. In Advances in Neural Information Processing Systems, 2021b.
T. Chen, X. Qu, D. Aponte, C. Banbury, J. Ko, T. Ding, Y. Ma, V. Lyapunov, I. Zharkov, and L. Liang. Hesso: Towards automatic efficient and user friendly any neural network training and pruning. arXiv preprint wxXv:2449.00083, 2024.
M. Cheng. S. L. Blodgett, A. Vrio, Egede, and A. Olteanu. Dehumanizing machines: Miti- gating anthropomorphic behaviors in text generation systems. arXiv preprint arXiv:2502.14019, 2025.
C. Clark, K. Lee. M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
Clark, Cowhey, O. Etzioni, T. Khot, A. Sabharwal, Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. S.Kiv:/803.0545777 2018.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021. 14168.
URL https: http://axivv...o/// org/
T. Ding, Т. Chen, H. Zhu, J. Jiang, Y. Zhong, J. Zhou, G. Wang. Z. Zhu, Zharkov, and L. Liang. The efficiency spectrum of large language models: An algorithmic survey. CoRR, 2023.
A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120))1-39, 2022
Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint
2018.
E. Frantar and D. Alistarh. Sparsegpt: Massive language models can be accurately pruned in one- shot. In International Conference on Machine Learning, pages 10323-10337. PMLR, 2023.
L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac'h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou. A framework for few-shot language model evaluation, 12 2023. URL https: //zeodo. orq/records/ wtteecordd12255e0 110256866.66
G. Gonçalves and E. Strubell. Understanding the effect of model compression social bias in large language models. In H. Bouamor, J. Pino, and K. Bali, editors. Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Processing, pages 2663-2675. Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10. https: / antho ogy wttg//222002 emnlp-ma 161/.
161. URL.
S. Han, H. Mao, and W. J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 15Xiv:1510.001 2015a.
12

---

Published as a conference paper at ICLR 2026
S. Han, J. Pool, J. Tran, and W. J. Dally, Learning both weights and connections for efficient neural networks, 2015b.
Y. He, G. Kang, X. Dong, Y. Fu, and Y. Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint 1rXiv:8808. 06866, 2018.
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.0338 03300, 2020.
A. Hendy, M. Abdelrehim, A. Sharaf, V. Raunak, M. Gabr, H. Matsushita, J. Kim, M. Afff,, and H. H. Awadalla. How good are gpt models machine translation? comprehensive evaluation, 2023. URL httph:://// / arxiv. org/ abs / 22302222222 223022202 09210.
X. Hu, Y. Cheng, D. Yang, Z. Xu, Z. Yuan, J. Yu, C. Xu, Z. Jiang, and S. Zhou. Ostquant: Re- fining large language model quantization with orthogonal and scaling transformations for better distribution fitting. arXiv preprini arXiv:2501. 13987, 2025.
L. Huang, L. Liu, F. Zhu, D. Wan, Z. Yuan, B. Li, and L. Shao. Controllable orthogonalization in training dnns. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2020.
Z. Hui, Y. Li, T. Chen, C. Banbury, K. Koishida, et al. Winelick: Gui grounding with multimodal large language models. arXiv preprint MiXi::25033047300 2025.
R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-88) 87, 1991.
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. I. Casas, Bressand,
G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682: 10.006200, 2023,
M. G. Johannes Welbl, Nelson F. Liu. Crowdsourcing multiple choice science questions. 2017.
D. Lee, J.-Y. Lee, G. Zhang. M. Tiwari, and A. Mirhoseini. Cats: Contextually-aware threshold- ing for sparsity in large language models, 2024. URL htttpss//aaxiv..rgg///o 08763.
D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
J. Li, Y. Dong, and Q. Lei. Greedy output approximation: Towards efficient structured pruning for Imms without retraining, 2024a. URL httpt: http://aaavvve.ee//aeeeeeeaaaela orr./rrgg/a.o..... /togg//ab//00.... /tt.b////47..11222. 19126.
J. Li, T. Tang, W. X. Zhao, J.-Y. Nic, and J.-R. Wen. Pre-trained language models for text generation:
A survey. ACM Computing Surveys, 56(9): 1-39, 2024b.
Y. Li, S. Gu, C. Mayer, L. V. Gool, and R. Timofte, Group sparsity: The hinge between filter pruning and decomposition for network compression. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8018-8027, 2020.
X. Liang, H. Wang, H. Lai, S. Niu, S. Song, J. Yang, J. Zhao, F. Xiong, B. Tang, and 7. Li. Seap: Training-free sparse expert activation pruning unlock the brainpower of large language models, 2025. URL https: / arxiv. org.o////a...... abss///// 2.503076660
J. Lin, J. Tang, H. Tang. S. Yang. W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Pro- ceedings of machine learning and systems, 6:87-100, 2024.
S. Lin, R. Ji, Y. Li. C. Deng, and X. Li. Toward compact convnets via structure-sparsity regularized filter pruning. IEEE transactions on neural networks and learning systems, 31(2):574-588, 2019.
J. Liu, P. Ponnusamy, T. Cai, H. Guo. Y. Kim, and B. Athiwaratkun. Training- free activation sparsity in large language models, 2024a. URL https: http///xxvvv/roee
1469..
13

---

Published as a conference paper at ICLR 2026
Z. Liu, C. Zhao, I. Fedorov, B. Soran, D. Choudhary, R. Krishnamoorthi, V. Chandra, Y. Tian, and T. Blankevoort. Spinquant: Lim quantization with learned rotations. arXiv preprint 2XXv:22005.66006 2024bb.
C. Ma, M. Huang. Y. Zhang, C. Wang, Y. Wang, L. Yu, C. Liu, and W. Lin. First activations matter: Training-free methods for dynamic activation in large language models, 2024. URL https:/ arxiv.iii.v...... org/ abs 24082.0.... 11393.
T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
A. Moslemi, A. Briskina, Z. Dang, and J. Li. A survey on knowledge distillation: Recent advance- ments, Machine Learning with Applications, 18:100605. 2024.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (1(10)):16) 2020.
K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge scale. arXiv preprint arXiv:1907. 10641, 2019.
N. Shazeer, Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: (0X1::/000.06338 2017.
Tillet, H.-T. Kung, and D. Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop Machine Learning and Programming Languages, pages 10-19, 2019.
H. Touvron, T. Lavril, G. Izacard, T. Lacroix. B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Ro- driguez, A. Joulin, E. Grave, and G. Lample. Llama 2: Open foundation and fine-tuned chat models, 2023.
M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa, et al. Siglip 2: Multilingual lisiun-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.
2025.
H. Wang, S. Ma, R. Wang, and F. Wei. Q-sparse: All large language models can be fully sparsely- activated. arXiv preprint arXiv:2407. 10969, 2024.
W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. Learning structured sparsity in deep neural networks. arXiv preprint arXiv: 00665,,006.... 2016a.
W. Wen, Wu, Y. Wang, Y. Chen, and H. Li. Learning structured sparsity in deep neural networks, 2016b.
D. Xie, J. Xiong, and S. Pu. All you need is beyond a good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation. In Proceedings of the IEEE Conference Computer Vision and Partern Recognition, pages 6176- 6185, 2017.
T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 2024.
F. Xue, Z. Shi, F. Wei, Y. Lou, Y. Liu, and Y. You. Go wider instead of deeper. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8779-87888 87899877777008 2022.
R. Zellers, A. Holtzman, Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Z. Zhang. Z. Liu, Y. Tian, H. Khaitan, Z. Wang, and S. Li. R-sparse: Rank-aware activation sparsity for efficient Um inference. arXiv preprint arXiv:2504. 19449, 2025.
14

---

Published as a conference paper at ICLR 2026
T. Zhuang, Z. Zhang, Y. Huang, X. Zeng. K. Shuang, and X. Li. Neuron-level structured pruning using polarization regularizer. Advances in Neural Information Processing Systems, 33, 2020.
S. Zuo, Q. Zhang, C. Liang, P. He, T. Zhao, and W. Chen. Moebert: from bert to mixture-of-experts via importance-guided adaptation. arXiv preprint arXiv:220.007675, 2022.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.I THE USE OF LARGE LANGUAGE MODELS (LLMS)
The use LLMs this paper was restricted to limited editing supports, such as detecting and conrecting grammar errors, typos, rephrasing sentences.
A.2 ORTHOGGNNAL TENSOR TRANSFORMATION
To enforee orthogonality in DNNs, there are two families of approaches. The first one is to augment orthogonality regularizer [WT I|| into the objective function (Xic et al., 2017; Huang al., 2020). This line of approach requires some model training. Another family of approaches uses matrix transformation rotation to enforce orthogonality. In particular, SliceGPT (Ashkboos al., 2024a) propose a transformation to bring orthogonality while preserve the numerical equivalence or computation invariance. Similar transformation is also widely used in efficient-Al literatures (Ashk- boos et al., 2024b; Chee et al., 2023; Liu et al., 2024b; Hu et al., 2025). In this work, we adopt the same transformation from (Ashkboos et al., 2024a) to produce computational invariant LLMs with enforced orthogonality to maximize the performance gain of WINA.
Overview. Given a weight matrix W, we can enforce column-wiss orthogonality by multiplying W from the right by an orthogonal matrix Q such that the product W has orthogonal columns. Specifically, we perform Singular Value Decomposition (SVD) on W, W 1 UVV,, where U and V are orthogonal matrices, and Σ is diagonal matrix containing the singular values of W. To achieve column-orthen -othooonnthooooalityy we set Q V and transform W as W WV. This transformation guarantees that the resulting matrix W satisfies the column- -othumonttooooaality
(5)
To ensure that the model's final output remains unchanged after this transformation, we compensate
for its effects using computational invariance (Ashkboos et al., 2024a). In particular, enforce column-wise orthogonality constraints over the matrices via SVD-basedd transformation. Detailed pseudoscope Algorithm present 1. Without loss of generality, we present pseudocode munsformer-based model M equipping with L layers. Each layer includes the following weight { (w((), w w w:("), w((t) w-(0) w:(e) down for f 1...., along with the output projection matrix W Wead of the final head layer.
matrices:
Runtime. This transformation is lightweight and efficient, taking less than four minutes on a single A100 GPU for models like Llama-2-33 Llama-3-8B, and Mistral, and less than twelve minutes for Phi-4-14B.
A.3
RESULTS OF DIFFERENT MODELS OVER GSM8K
Table 10: Results of different models over GSM8K
Sparsity
Method
Llama-2-7B
Llama-3-8B
Mistral-7B
Phi-4(144)
D
Raseline
13.95
49.96
38.74
90.22
TEAL
14.33
48.98
37.98
89.84
0.25
WINA
12.51
49.36
37.04
90.22
TEAL
13.12
39.88
33.97
88.02
0.4
WINA
13.34
40.26
35.41
TEAL
8.72
27.07
29.49
0.5
WINA
11.22
30.40
29.72
87.57
TEAL
2.50
2.73
10.99
74.37
WINA
4.62
5.91
12.36
77.10
16

---

Published as a conference paper at ICLR 2026
Algorithm Orthogonal Tensor Transformation
1: Input: Model M with matrix Weml of embedding layer, L layers with matrices (W(), W WhO W! W gate: uap wlt) V and matrix Whead of head layer.
down 2: Oufput: Orthogonally transformed model M' which computational invariant to M.
3: Perform SVD over ( w( -UVT.
4: O V.
266) 5: Wemb WWenl 1emb
6: for 8= 1,,,2....... do
W
Q()
W. + of
7:
8: Perform SVD over w1( gate
UEV..
Q V
9:
W!O -Q00) gate)) (W(())
W(( w:(O) gate gater
2
W W (())
gate
gate
12:
if then
Perform SVD over W ((+1) = UEVT
13:
((-1))
14:
(
15:
down
down
end
17: end for
WINA Speed-up Sparsity for Different Batch Sizes
1.8
Batch 1
Batch
Batch
Batch 54
1.6
256
Speed-up (WANA)
1.5
1.4
1.2
1.1
1.0
3.25
0.30
0.35
0.40
0.45
0.50
0.35
0.60
0.65
Sparsity
Figure 5: Speed-up ratio versus sparsity level WINA on 5120 17920.
A.4 SPEED-UPS ACROSS BATCH SIZES
We compute GEMV (General matrix-vector multiplication) latency and speedup via sparse activa- tion across different batch size. The experiment is conducted on A800. We employ matrix of shape 1120x17920, corresponding to the largest weight matrix in phi-4, the biggest LLM used in experiments. The sequence length is set to 128. We calculate the speedup based on latency, defined as the ratio the dense model's latency to the latency under sparse activation. The results in Figure show that TEAL and WINA achieve comparable speedups the same sparsity level, and the speedup for both decreases as the batch size increases.
17

---

Published as a conference paper at ICLR 2026
A.5
ADDITIONAL EXPERIMENTS ON LONGBENCH
Table 1I: Comparison of WINA and TEAL on LongBench tasks (Llama-3-8B-
Sparsity
Method
Code Completion
Few-shot Leaming
Summari- zation
Multi- Document
Single- Document
Synthetic Tisks
Overall
QA
Renchmark
TEAL
22.00
60.17
15.77
9.96
13.92
11.03
22.69
0.25
WINA
23.30
60.20
16.24
9.89
13.65
10.92
22.82
0.4
WINA
24.13
60.47
17.71
9.74
13.02
7.30
22.57
TEAL
17.17
60.50
15.92
9.18
14.22
4.41
21.28
WINA
23.88
60.24
19.29
9.08
11.82
3.95
21.97
TEAL WINA
8.29
51.25
13.86
7.85
9,78
3.48
17.05
0.65
53.61
14.66
7.07
7.73
18.13
Table 12: Comparison of WINA and TEAL on LongBench tasks (Phi-4-16k).
Sparsity
Method
Code Completion
Feo Leamnigg
Summari- zation
Multi- Document
Single- Document
Synthetic Tasks
Overall
QA
QA
D
Benchmark
29.58
56.03
8.41
4.80
18.59
59.59
28.06
TEAL
31.16
54.83
10.11
5.73
18.32
55.60
27.86
WINA
30.15
55.53
9.07
6.48
19.34
37.46
28.30
TEAL
30.65
55.20
12.53
9.95
21.38
48.70
28.74
0.4
TEAL
29.51
58.44
15.04
11.69
22.15
51.03
30.54
0.5
WINA
36.53
59.39
13.39
15.98
21.19
48.76
31.39
TEAL
19.25
30.07
0.65
WINA
37.60
59.22
17.36
12.24
16.30
46.58
30.26
18

---

Published as a conference paper at ICLR 2026
A.6 EXTENSIVE EXPERIMENTS ON SOCIAL BIAS
Prior work (Gonçalves and Strubell, 2023) has shown that model compression can unintentionally influence social bias in LLMs. To assess whether our sparsification method exhibits similar behavior, we follow the evaluation protocol and stereotype-score metric of (Gonçalves and Strubell, 2023)
and measure performance on the CrowS-Paii benchmark across GENDER, RACE, and RELIGION categories. Table 13 reports results for our models at varying sparsity levels.
Across all models, observe no systematic amplification of social bias sparsity increases. In- stead, bias scores typically change modestly (within 3-6 points) and often move closer to the 50% "unbiased" target defined in (Gonçalves and Strubell, 2023).
For instance, Llama-2-77 exhibits reductions in GENDER (59.92 53.82) and RACE (69.77 66.28) at 65% sparsity; Llama-3-8B shows mild decreases across all categories at higher sparsity levels; and Mistral-7B maintains largely stable behavior with slight improvements in RACE and RELIGION. Notably, Phi-4-14B demonstrates the strongest effect, with substantial reductions at 65% sparsity across all categories, particularly in the RELIGION category (74.29 60.00).
CrowS-Pairs stereotype scores for Gender, Race, and Religion for WINA different LLMs. For each metric, we report the stereotype scores and their absolute distance from the optimal unbiased value of 50% in parenthesis (smaller values indicate less bias from neutrality.). Scores further from 50% indicate more bias. Green indicates improved bias scores from baseline, red indicates worse, and grey indicates change from baseline.
Table 13:
Model
Sparsity
GENDER
RACE
RELIGION
Baseline
59.92 (9.92)
69.77 (19.77)
74.29 (24.29)
25%
59.92 (9.92)
67.83 (17.83)
77.14 (27.14)
Llama-2-7B
40%
58.40 (8.40)
71.51 (21.51)
78.10 (28. 10)
50%
59.54 (9.54)
68.02 (18.02)
74.29 (24.29)
65%
53.82 (3.82)
66.28 (16.28)
76.19 (26.19)
Baseline
60.31 (10.31)
66.28 (16.28)
74.29 (24.29)
25%
59.54 (9.54)
65.70 (15.70)
76.19 (26.19)
Llama-3-8B
40%
61.45 (11.45)
66.09 (16.09)
76.19 (26.19)
50%
57.63 (7.63)
65.50 (15.50)
71,43 (21.43)
65%
60.69 (10.69)
64.34 (14.34)
68.57 (18.57)
Baseline
62.98 (12.98)
67.25 (17.25)
69.52 (19.52)
25%
62.60 (12.60)
68.22 (18.22)
66.67 (16.67)
Mistral-7B
40%
62.21 (12.21)
64.53 (14.53)
68.57 (18.57)
50%
62.60 (12.60)
67.05 (17.05)
69.52 (19.52)
65%
61.83 (11.83)
64.15 (14.15)
64.76 (14.76)
Baseline
65.65 (15.65)
63.95 (13.95)
74.29 (24.29)
25%
63.36 (13.36)
63.57 (13.57)
71.43 (21.43)
Phi-4-14B
40%
61.83 (11.83)
60.66 (10.66)
67.62 (17.62)
50%
59.54 (9.54)
61.43 (11.43)
65.71 (15.71)
65%
59.54 (9.54)
59.88 (9.88)
60.00 (10.00)
Taken together, our results suggest that sparsification method does not exacerbate social bias and in some cases modestly reduces it, consistent with the behavior of certain compression settings reported in (Gonçalves and Strubell, 2023): all in all, we reduced bias in 12/16 experiments for GENDER, 14/16 for RACE, and 9/16 for RELIGION. We emphasize, however, that these benchmarks capture only limited set of bias dimensions. We view these findings indicative rather than comprehensive, and further research on fairness-aware compression remains an important direction.
19

---

Published as a conference paper at ICLR 2026
A.7 PROOF OF LEMMA 3.1
Proof. Let (x) := (i 0f be the set of indices of zero elements at æ. The output deviation between the original network output and the gated output via general-format sparsification is:
- (az-o-xx))
Σ
2;
W.W.
2
Σ 2,W
Σ
W
16
Σ T W. W.y.W. W.,
je
Σ =
o W.
Σ oソæ; W W,y. W.,
injeZ=0
The expected output deviation for WINA is:
W TT Wx
eWINA
WINA
Σ
I +
Σ
a. W.+-
se WINA
je-j TwiNA
Since W is assumed to be column orthogonal, the cross-term expectations vanish, and the expected output error is determined solely by the main term:
Σ SEZWINA
eWINA =
Because WINA sparsification sets the smallest [ terms zero, we have the mask of WINA reaches out the lower bound of approximation error for a single layer network, i.c.,
gWINA (a:) arg min [W(#og=))))
ge(U,13"
Thus, the above indicates that WINA sparsification achieves the tight lower bound of the approxi- mation error, including those of TEAL and CATS.
A.8
PROOF OF THEOREM 3.2
Proof. We consider an L-layer linear network (no activation):
(1) w(), x,
(++)) =w(e+)) l= 1-...,I-111
L-1,
so
M(x) W(L)...
Let (9) E 10 de be a gate and define M = (1). The gated network is defined recursively by
u (M0 y ())
y
Denote G (2)
( (
Key assumption. For each layer 2 2, assume columm-orthooonaaiiyy
(0T = ()() is diagonal.
(Equivalently, columns of (()) are mutually orthogonal; normalization is required.)
20

---

Published as a conference paper at ICLR 2026
Step 1: Base case N 2 and exact decomposition.
For two layers,
w(2) x,
Ug
Let the deviation be eg Vg (2) (2) Add and subtract ((O
(2)
9 = (M - T) M x+
(MM()))) (0 1)
Therefore,
12= |212 | + 2u'e v.
This identity is exact.
Separable expressions for lul2 and || Let D = ((2)T which is diagonal by as- sumption. Write W(2) a and v= = b where
a: CMWW N)c,
b: (IW() M4)
Then
||12 = a D a,
101 = 6o D
Because (2) is diagonal and (M(2) 1) keeps exactly the coordinates with 9j (2) 0,
|0122 = Σ
[ 12 ((W)))
59 =0
Moreover, letting A == (()) and noting u A(M() - -D),, we have
2
Σ |A,113 x??
eg.
The cross-term equals
u v = a 2
b ΣD azbj.
which in general depends on the gates and cannot be assumed vanish without additional structure.
A provable upper bound.
For any 0, by Young's 's inequality,
2u v 2ll 11012.
hence
1e (1+a) [1 4 lul2.
Define the upper bound objective
u. (x( (1+ (1+ 1 a |/1/2.
Since |u|2 and || have separable (top-k) forms above under column-orthogonnalty. WINA's se- lection rule minimizes UA (x:
Step Inductive extension to N 2. For N + layers, let the deviation eg (N+1) := yg (N).
(N-1)
(N+1), Add and subtract W(N+1 yg
(N) (N+1) (N++1) (N+1) 1)y (N), + wwN+1) (3) - (N)). Eg
:2
=
Thus,
le (N+11) | || + 1101 2 TT. v.
21

---

Published as a conference paper at ICLR 2026
and applying Young's inequality yields
(N+1)
2 (1+a)lu (1+22) |0|2 =: =: (x: g).
e
Under the column- ootooonatttyy assumption for W(N+1), || separable:
|
E
|w:(++1) 1 (
so minimizing it reduces keeping the k largest weighted coordinates. Moreover, 1 [W(N+1) (yg (N) - (1)) is minimized by recursively applying the same rule to the first N layers. Therefore WINA minimizes the upper bound U. (z; 9) for all
The above establishes that, under column-orthogonality WINA optimizes a provable upper bound Ua (x; 9) on the true deviation [[M())) MMx 9) In general, without additional assumptions
forcing u v 0 (or gate- satte-iddependent) one cannot claim exact optimality for the true deviation. O
A.9 LIMITATIONS
To maximize performance gains and ensure realistic acceleration, WINA requires a few additional operations. First, WINA needs the column-wis wise norms of the weight matrix W, which can be pre- computed once during model loading and reused throughout inference. Second, WINA performs an element-wise product between the input and the pre-computed norms; this operation extremely lightweight and adds negligible overhead.
The performance benefits of WINA maximized when the weight matrices in LLMs satisfy the column- orthogonality condition. To enforce this property, we adopt the efficient one-off or- thogonalization transformation proposed in (Ashkboos al., 2024a). This transformation is applied a single time prior to inference and produces a numerically equivalent LLMs for further usages.
22

---

Published as a conference paper at ICLR 2026
A.10
ADDITIONAL RESULTS ON HARDWARE ACCELERATION
b=4
b=16
0.14
0.18
Time (ms)
A100
0.12
0.16
0.10
0.14
WINA
TEAL
0.08
0.12
RTX PRO 6000 Blackwell
0.18
0.14
Time (ms)
0.16
0.12
0.14
0.10
0.12
0.08
0.10
0.0
0.2
0.4 Sparsity (%)
0.6
0.0
0.2
0.4 Sparsity (%)
0.6
Figure 6: 5120x 1720 for batch sizes and 4 on A100 and RTX PRO 6000 Blackwell.
b=1
b=4
0.08
Time (ms)
0.07
A100
0.07
0.06
0.06
WINA
0.05
TEAL
Dense
RTX PRO 6000 Blackwell
0.08
0.07
0.07
Time (ms)
0.06
0.06
0.05
0.05
0.04
0.0
0.2
0.4
0.6
0.0
0.2
0.4 Sparsity (%)
0.6
Sparsity (%)
Figure 7: 4096x 1008 GEMV for batch sizes and on A100 and RTX PRO 6000 Blackwell.
A.11 NOTE ON LAYER- WISE SPARSITY ALLOCATION
Additionally, in this work, do not focus specifically per-layer sparsity allocation since WINA generalized well and applicable over varying sparsity allocations. As future work, we believe dedicated sparsity-allocation strategies and focus on layer-wise sparsity using different sparsity assignment protocols are promising directions to further improve WINA's performance.
23


Reviews:
[{"review_id": "Ea4Wsc0JJT", "reviewer": "Reviewer_i21q", "summary": "- Proposes WINA, a training-free sparse activation method that gates neurons using both activation magnitude and column-wise weight ℓ₂ norms.", "questions": "- In your current implementation, is the WINA gate used before the matmul to index only a subset of columns of 𝑊 , or do you compute a dense 𝑊𝑥 and then apply the mask? If it’s the latter, how do you obtain any wall-clock speedup, especially in batch-1, memory-bound settings?\n\n- Comparison to TEAL/CATS kernels: TEAL/CATS explicitly discuss sparse kernels that reduce weight loading per token. Do you implement a comparable kernel for WINA, and can you report latency/throughput numbers vs TEAL/CATS on real hardware (A100, L40S, etc.), not just GFLOP estimates?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- Very simple, plug-and-play rule that is easy to implement on top of existing sparse-activation baselines.", "weaknesses": "- The paper reports GFLOP reductions but does not clearly explain whether WINA’s gating is used to avoid weight loads or only to mask post-matmul activations; without a truly sparse kernel and latency measurements, it is unclear how much real speedup WINA provides over TEAL/CATS in memory-bound, batch-1 inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "liI8CjRvSr", "reviewer": "Reviewer_zwra", "summary": "The paper addresses the important challenge of reducing inference cost in LLMs without degrading output quality. Existing training-free sparse activation methods often rely solely on hidden state magnitudes, which can lead to significant approximation error, particularly at high sparsity levels.\n\nThe authors propose WINA (Weight-Informed Neuron Activation), a simple yet effective framework that incorporates both hidden state magnitudes and the ℓ2-norm of weight matrices into neuron selection. This provides a principled sparsification strategy with provably optimal approximation error bounds, yielding tighter theoretical guarantees than prior methods.\n\nThe method is empirically validated across multiple widely used LLMs, including Llama-2-7B, Llama-3-8B, Mistral-7B, and Phi-4-14B, and evaluated on diverse tasks such as general reasoning (MMLU), mathematics (GSM8K), and coding (HumanEval). WINA is compared against several strong baselines, including TEAL, R-Sparse, and CATS. The results show that WINA performs comparably to prior methods at low sparsity and significantly better at high sparsity, achieving several percent improvement in commonsense reasoning accuracy and sustaining performance under extreme sparsity levels.\n\nOverall, WINA is presented as a practical, theoretically grounded, and broadly deployable approach for efficient inference in LLMs.", "questions": "The authors propose incorporating weight norms into the neuron selection process. Could the authors clarify whether this additional step increases computation amount during inference, and if so, provide benchmark comparisons to quantify the overhead relative to other training-free sparse activation methods?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem addressed is highly relevant, as reducing LLM inference cost without sacrificing output quality is an important challenge. The method is theoretically grounded, as incorporating weight norms provides a principled sparsification strategy with provable error bounds. The empirical evaluation is extensive, covering multiple LLMs, a range of tasks, and both low and high sparsity levels, and the method is compared to strong baselines including TEAL, R-Sparse, and CATS. The approach is practical and easy to deploy, as it is training-free and plug-and-play, making it broadly applicable. The results demonstrate robustness, as the method maintains competitive performance across different sparsity regimes and models.", "weaknesses": "The main contribution is a relatively straightforward extension of existing sparse activation methods, which could be considered incremental, though it is strengthened by solid theoretical and empirical support. The paper could benefit from a discussion of potential limitations, such as scenarios where weight-informed selection might be less effective or challenges when scaling to very large models beyond those tested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sMP6jmmOTK", "reviewer": "Reviewer_SJQR", "summary": "The paper proposes WINA (Weight Informed Neuron Activation), a training-free sparse activation method that combines hidden state magnitudes with the weight matrix structure to guide neuron selection. WINA is proven to minimize approximation error under column-wise orthogonality and monotonic activation assumptions. In the experiments, it outperforms other training-free methods like CATS, R-Sparse and TEAL across multiple LLM architectures (Llama2/3, Mistral, Phi-4) and benchmarks (MMLU, GSM8K, HumanEval), achieving over 60% FLOPs reduction at 65% sparsity while preserving accuracy.", "questions": "- How does WINA perform on large MoE models (e.g., DeepSeek, Llama4, GPT-OSS)? It would help to understand how the method scales to production scale LLMs.\n- Could you evaluate WINA’s actual latency or throughput in real inference scenarios? What’s the challenges to integrate this method to inference frameworks?\n- How does WINA perform under long-context settings (e.g., 16K–128K tokens)? Are the top-K activation patterns stable as sequence length increases?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method introduces a simple yet effective training-free sparse activation mechanism that combines both hidden-state magnitudes and the column-wise L2-norm of weight matrices to guide neuron selection.\n- The theoretical analysis is rigorous and well structured, providing provably optimal approximation error bounds under clear and interpretable assumptions (column-wise orthogonality and monotonic activation).\n- The experiments are comprehensive, covering multiple model architectures, quantization methods, and ablations, demonstrating consistent improvements that align with theoretical predictions.", "weaknesses": "- The models in the experiments are small dense LLMs. Large-scale or MoE architectures (e.g., DeepSeek-V3, Llama4, GPT-OSS) which are more common in product deployment workloads are not tested. It’s unclear whether WINA’s activation gating would maintain efficiency with expert routing sparsity in these larger models.\n- The evaluation focuses on theoretical FLOPs reduction but lacks real-world inference measurements such as latency or throughput on inference frameworks. Without kernel-level or runtime validation, the practical performance benefits of WINA remain unclear, especially given the hardware inefficiency of non-structured sparsity.\n- The theoretical assumptions rely on column-wise orthogonality and monotonic activation functions, which may not strictly satisfied in real transformer models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ADyOI3ABxA", "reviewer": "Reviewer_DJxS", "summary": "The paper proposes a new sparse activation method, WINA, to improve LLM inference efficiency. WINA or Weight Informed Neuron Activation uses both the hidden state magnitude and weight matrix structure while sparsifying the activation; previous work only relies on the hidden magnitude. \nWINA uses the product of l2 norm of the column vector of the weight matrix and the hidden state magnitude to select the top-K neuron with theoretical justification. Extensive experiments are provided with popular benchmark datasets.", "questions": "1. How is LayerNorm monotonically increasing? (Line 238)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow, and the idea is intuitive. \n\n2. Theoretical justification showing that using both the L2 norm of the column vector and the hidden state magnitude yields an optimal solution and reduces error (section 3). \n\n3. Results provided are quite extensive; the method is evaluated on multiple datasets and different downstream tasks. \n\n4. Results in tables 3 and 4 show consistent improvements, especially at higher sparsities, compared to other baselines, showing the efficacy of the method.\n\n5. Additional results showing on quantization are provided, showing WINA is compatible with quantization.", "weaknesses": "1. The technical novelty of the method is limited (however, results show it improves over baselines). \n\n2. It is not clear why the orthogonality of the weight matrix is enforced (in sec 3.4)? Does this orthogonality hold in a general setting as well?\n\n3. Recent works have shown that LLM compression has an unintended impact on the model bias. It would be helpful to also evaluate the impact of the proposed method on model bias. \n\n\n[1]. Strubell et al., Understanding the Effect of Model Compression on Social Bias in Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UcMZUIJTYb", "reviewer": "Reviewer_1iPj", "summary": "The paper aims to improve inference efficiency of LLMs and proposes skipping unnecessary neuron computations in feed-forward network (FFN) layers. The proposed method does this with a gating function that uses the magnitude of a neuron's output weight to assess its importance. Less important neurons (smaller output weight magnitue and smaller intermediate activation) are skipped. The paper has empirical results on Llama 3-8B and Qwen 2-7B, demonstrating significant speedups with minimal impact on model accuracy.", "questions": "Do the authors have any suggestions on how to tune the threshold efficiently?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method, WINA, is easy to apply to existing pre-trained LLMs since it is a post-training method that does not require any fine-tuning or retraining. \n\n- To the best of my knowledge, assigning importance scores based on output weight's magnitude is a novel idea. \n\n- Empirical results are strong across various benchmarks and model sizes.", "weaknesses": "- The linked source code is not available. \n\n- The success of WINA relies on the threshold used to determine which neurons to skip. Tuning this threshold would be costly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JzLMG2UPB6", "reviewer": "Reviewer_6SUJ", "summary": "This paper presents WINA (Weight-Informed Neuron Activation), a training-free sparse activation method that accelerates LLM inference by selecting neurons based on both hidden state magnitudes and weight matrix norms. This weight-informed approach achieves tighter theoretical error bounds and better accuracy than prior methods like TEAL and CATS, maintaining strong performance even at high sparsity levels across various LLMs and tasks, and showing compatibility with quantized models.", "questions": "See weaknesses above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and easy to follow.\n\n2. The figures are clearly and beautifully presented.\n\n3. The experiments conducted on extensive datasets provide strong validation and demonstrate the integrity of the proposed method.", "weaknesses": "1. My main concern is related to the performance measurement. The authors claim that WINA is more efficient than previous methods, \"potentially translating to faster inference speeds and lower computational costs.\" Could the authors provide empirical evidence, such as wall-clock time or GPU memory usage, to support this claim?\n\n2. The improvement in synthetic results shown in Table 2 is substantial, but the gains in real-world LLM experiments are relatively modest. Could the authors clarify the reason for this large discrepancy between synthetic and real-world results?\n\n3. How can the assumption of \"column-wise orthogonality\" in the theorems be verified? Is there any experimental evidence to support this assumption?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DOu8h7E4B9", "reviewer": "Reviewer_iVmy", "summary": "This paper proposed a framework WINA for training-free sparse activation that incorporates both hidden state magnitudes and weight matrix structure, which combines the magnitude of activations with the column-wise norm of the weight matrices to preserve the top-k activations. The authors claimed that WINA can achieve a lower approximation error bound under several assumptions and is model-agnostic. Methods are tested on Llama-2-7B, Llama-3-8B, Mistral-7B, and Phi-4-14B models across several benchmark datasets, which demonstrate that WINA can achieve superior performance under various sparsity ratios.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The combinatorial gating strategy is reasonable, which produces a tighter approximation error bound.\n2. WINA as a training-free method is friendly for deployment.\n3. The paper is well written and easy to follow.", "weaknesses": "1. There are no end-to-end latency performance comparisons between WINA and previous methods like TEAL, CATS, and R-Sparse.\n2. For the math reasoning task like GSM8K, the aggressive sparsity can induce significant damage to the model performance, dropping accuracy from 50 to 7, although WINA reported superior performance to the baseline methods.\n3. When the batch size is larger, will the sparsity be affected and further the speedup gain be degraded?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)