Universal Properties of Activation Sparsity in Modern Large Language Models
Authors: Filip Szatkowski
Patryk Będkowski
Alessio Devoto
Jan Dubiński
Pasquale Minervini
Mikołaj Piórczyński
Simone Scardapane
Bartosz Wójcik
Venue: ICLR 2026 Poster (2026)
Keywords: LLMs
activation sparsity
efficiency
representations
Primary area: foundation or frontier models, including LLMs
TL;DR: We provide an overview of activation sparsity in modern LLM architectures and highlight a set of interesting properties of the activations across different model families.

Abstract: Activation sparsity is an intriguing property of deep neural networks that has been extensively studied in ReLU-based models, due to its advantages for efficiency, robustness, and interpretability. 
However, methods relying on exact zero activations do not directly apply to modern Large Language Models (LLMs), leading to fragmented, model-specific strategies for LLM activation sparsity and a gap in its general understanding. 
In this work, we introduce a general framework for evaluating sparsity robustness in contemporary LLMs and conduct a systematic investigation of this phenomenon in their feedforward~(FFN) layers.
Our results uncover universal properties of activation sparsity across diverse model families and scales.
Importantly, we observe that the potential for effective activation sparsity grows with model size, highlighting its increasing relevance as models scale. 
Furthermore, we present the first study of activation sparsity in diffusion-based LLMs. 
Overall, our work provides a comprehensive perspective and practical guidance for harnessing activation sparsity in LLM design and acceleration.

Published as a conference paper at ICLR 2026
UNIVERSAL PROPERTIES OF ACTIVATION SPARSITY IN MODERN LARGE LANGUAGE MODELS
Filip Szatkowski!.
Patryk Bedkowski'
Alessio Devoto
Jan Dubiński
Pasquale MinerviniSo Mikolaj Piórczyúski!? Simone Scardapane
Bartosz Wójeik?
ABSTRACT
Activation sparsity is an intriguing property of deep neural networks that has been extensively studied in ReLU-based models, due to its advantages for efficiency, robustness, and interpretabillity However, methods relying on exact zero activations do not directly apply to modern Large Language Models (LLMs), leading to fragmented, model-specific strategies for LLM activation sparsity and a gap in its general understanding. In this work, we introduce general framework for evaluating sparsity robustness contemporary LLMs and conduct a systematic investigation of this phenomenon in their feedforward (FFN) layers. Our results uncover universal properties of activation sparsity across diverse model families and scales. Importantly, we observe that the potential for effective activation sparsity grows with model size, highlighting its increasing relevance as models scale. Furthermore, we present the first study of activation sparsity in diffusion- based LLMs. Overall, our work provides a comprehensive perspective and practical guidance for harnessing activation sparsity in LLM design and acceleration.
INTRODUCTION
An intriguing property of deep learning models is activation sparsity, the tendency of their hidden states to contain a significant fraction of zero near-zero values with input-dependent patterns.
This phenomenon has been connected to robustness (Dhillon et al.) 2018 et al. 2023), inter-
pretability (Geva et al.) 202 1 Zhang et al.] [023 Cunningham et al. 2023 Bricken et al. 2023
Gao et al. (2024), and potential efficiency gains from skipping redundant computations (Liu et al.
2023; Mirzadeh et al. 2024). Motivated by these benefits, prior work has studied activation sparsity
extensively in Resee- networks (Gilorot et al. 2011 2011] Awasthi et al.) 2024 Rhu et al. 2018 Kurtz
al. 2020 Zhang al. 2021] Li et al. 2023).
However, the landscape of modern deep learning has shifted with the rise of Large Language Models (LLMs), which predominantly use GLU-based architectures (Shazeer 2020) with SiLU or GELU activation functions (Gemma Team 2024 Touvron et al. 2023 Qwen Team 2023 Seep
AI 2024). Although the hidden states of these models often contain many near-zero values, methods developed for ReLU-based networks transfer poorly when applied to them directly. Recent works have attempted to adapt activation sparsity techniques to LLMs by retrofitting them to use ReLU or similar
activation functions that induce exact zero activations (Zhang et al. 2021 Mirzadeh et al. 2024
Song et al. 2024a b), or exploit approximate sparsity in existing architectures (Federici et al. 2024
Lee et al. 2024 Chua et al. 2024 Liu et al. 2025a b). However, retrofitting approaches often require additional training and risk degrading model quality during such training, while approximate sparsity approaches lack the principled guarantees of strict sparsity induced by ReLU, require calibration of thresholds, and may overfit to the held-out calibration dataset. Moreover, the literature on sparsity- based acceleration remains fragmented, with different methods targeting various components of the LLM modules, such as beed-forward network (FFN) input (Federici et al. 2024)] Liu et al. (2025 b). gate (Song et al. 2024a] Lee al. 2024), or intermediate activations (Liu et al. [2023] 2024). Consequently, the design choices behind LLM activation sparsity methods can seem arbitrary,
(0223)Akhauri Akhauri et al.
Warsaw University of Technology, VIDEAS Research Institute, 'Sapienza University of Rome, 'NASK National Research Institute, Poland, "University of Edinhurgh, MinimmLAT, 'Jagicllonian University, Kraków

---

Published as a conference paper at ICLR 2026
Laput based activation sparsity
Gate based activation sparsity
Predictor based actiration sparsity
W. g
Wa g
W, R
Wa R
(0,1)
W, g
Wa
2
We g
W
u- R
R MEE
Figure 1: Common strategies for exploiting activation sparsity to skip redundant computations in GLU-based FFN modules, with the origins of the sparse activation masks denoted with red borders. Input-based methods skip parts of the matrix multiplications corresponding to the low-magnitude components in the input vectors across all three linear layers. Gate-based and predictor-based methods instead omit computations associated with values that are either negligible in the gate activation vector or predicted to be negligible by an auxiliary predictor module.
and there is currently no comprehensive overview of the phenomenon in modern models, nor unified guidelines for leveraging it to enhance model efficiency.
Given the rapid advances in LLMs, argue that systematic study of activation sparsity in such models is necessary to understand their internal mechanisms, improve their efficiency, and guide their architectural design. In this work, we address this gap by analyzing the robustness of contemporary LLMs to activation sparsity across FFN components, architectures, and model sizes. We provide unified view of sparsity patterns and introduce a simple, general framework for assessing model robustness. To induce sparsity, we propose straightforward top-p method, applicable to any model without architectural assumptions or additional training. This allows us to assess models' activation sparsity from functional perspective and identify the highest sparsity levels that do not trigger meaningful quality loss. We formalize this concept of functional activation sparsity via critical spansity, metric defined as the maximum sparsity level that causes at most 1 1% performance drop. As prior work has already demonstrated that with appropriate implementation sparsity in FFNs activations can translate approximately linearly computational acceleration (Szatkowski al. 2024 Liu et al.) 2025a), in our work we focus on characterizing models' tolerance to sparsity and uncovering universal patterns across popular LLMs.
Using our proposed framework we find that the critical sparsity tends to increase with model size, and present empirical comparisons of sparsity tolerance across FFN activations. We further examine the impact of training strategies on sparsity robustness, highlighting the universal capacity for activation sparsity pretrained, instruction-tuned. and reasoning models. Additionally, we analyze activation sparsity and temporal consistencies in sparsity patterns in masked diffusion LLMs, revealing potential acceleration opportunities; to our knowledge, this is the first study of such phenomena in this class of models. Finally, we discuss our findings in the context of prior work and mention its implications for designing future methods that aim to leverage activation sparsity. Overall, our study uncovers universal patterns of activation sparsity in modern LLMs and offers practical insights and guidelines for researchers and practitioners seeking a deeper understanding of this intriguing phenomenon.
ACTIVATION SPARSITY IN MODERN TRANSFORMERS
2.1 SPARSE ACTIVATIONS IN DEEP NEURAL NETWORKS
Activation sparsity refers to the tendency of neural network hidden states to contain a substantial fraction of zero values, following input-dependent patterns. This phenomenon has been widely observed in ReLU-basee architectures, including MLPs (Glorot et al. 2011]] Davis & 2013) and CNNs (Rhu et al. 2018 Kurtz et al.) 2020). More recently, Zhang et al. (2021) and Li et al. (2023) reported that standard training also induces significant activation sparsity in Transformer forward networks (FFNs). Activation sparsity has been associated with several desirable properties. In MLPs, it enhance learnability and generalization (Awasthi et al.) 2024), while in CNNs and ViTs it has been shown to improve robustness against input corruptions (Ahmad & Scheinkman 2019 Li et al. 2023). Moreover, sparsity has been leveraged for interpretability by disentangling neurons
2

---

Published as a conference paper at ICLR 2026
corresponding to distinct concepts (Geva et al. 2021 Elhage et al. 2022 Cunningham al. 2023
Bricken et al. 2023 Gao et al. 2024a).
Modern LLMs typically use SiLU or GELU activations in combination with GLU-based FFNs (Shazeer 2020). and do not exhibit strictly zero activations. Nevertheless, effective activation sparsity still arises in these models even without explicit architectural or regularization constraints, and several studies have sought to theoretically explain this phenomenon (Andriushchenko et al.)
Peng et al. 2023). Zhang et al. (2024b) investigated how different activation functions during LLM pretraining influence emergent sparsity patterns, while Luo et al. (2024) derived scaling laws connecting activation sparsity to the size of pretraining datasets. Additionally, some works have ex- plored leveraging the approximate sparsity in LLM activations to improve model efficiency (Liu et al.)
2023
2023 Mirzadeh et al.) 2024), which we discuss in more detail in Section 2.3 Despite these efforts,
most studies focus on particular forms of activation sparsity, and unified, systematic investigation of the phenomenon in modern LLMs is still lacking.
2.2 ACTIVATION VECTORS IN GATED FEEDFORWARD TRANSFORMER LAYERS
Transformer blocks consist of attention and feedforward (FFN) sub-blocks (Vaswani et al. 2017). While the original Transformer FFN was composed of two projection layers separated by an ac- tivation function, modern LLMs typically employ FFNs based the Gated Linear Unit (GLU)
architecture (Shazeer 2020), which can be expressed as:
FFN(x) Wa((Wnz) O (WgF)).
where E R is the input vector, Wn E Raxd is the up-projection matrix, Wg E ggxdd is the gating projection matrix, Wa E Rddc is the down-projection matrix, and a is an activation function, usually SiLU or GELU, We use h and d to denote the model's hidden and interme- diate dimensions, respectively. In the subsequent sections, we refer to the above-mentioned ac- ( gT) gate and
tivation vectors in the FFN as x- --ipull u W. up-projection
i i= (W a @ o @ gz) intermediate vectors. Additionally, we consider the simultancous sparsi- fication of input and intermediate activations, and refer to the case where induce sparsity in both of these vectors the same time all FFN inputs sparsification.
2.3 LLM INFERENCE ACCELERATION WITH ACTIVATION SPARSITY
Modern models exhibit significant activation sparsity, meaning that a large portion of activations are effectively unused, resulting in wasted computation. This effect is especially pronounced in FFN layers, which dominate the computational cost of LLMs until extremely long contexts (Casson) 2023). Skipping these redundant computations can not only reduce computation, but also lower memory overhead, decrease the cost of loading weights, and even enable hardware-specine optimizations
such weight caching (Alizadeh et al. 2024 Federici et al. 2024 LLM acceleration methods that leverage activation sparsity can be grouped into three main categories outlined in Figure [
Input-based methods (Federici et al. 2024 Liu al. 2025a b) select subset of input activations for each linear layer and skip the operations corresponding to masked out multiplications. Since input vectors might lack natural sparsity, e.g., from activation functions, these methods calibrate skip thresholds or use heuristics to identify redundant activations. Gate-based methods (Lee et al., 2024, Song al., 2024a) exploit sparsity based on the gate activation vector. Since this vector is result of projection through an activation function, such methods assume that it should contain many near-zero values, which render portions of the up- and down-projection computations redundant. However, computing the gate vector accounts for roughly one-third of the FFN cost, which limits its overall efficiency. Predictor-
methods (Zhang et al., 2021 Liu et al. 2023 Szatkowski et al. (024) usually target the intermediate activations. Computing intermediate activation vectors directly requires over two-thirds of FFN computation, so these methods typically use a lightweight predictor (e.g., small linear network) instead to estimate which neurons (or groups) can be skipped. This approach can reduce computations in all FFN components, but requires training the predictor, induces slight inference overhead on the prediction, and introduces potential approximation errors.
Research on activation sparsity typically targets specific types of activations and focus on improving model efficiency. Prior studies have already demonstrated computational benefits of activation
3

---

Published as a conference paper at ICLR 2026
Normalized Accuncyy
Intermediste
Projectetoonn
All-inputs
200
150
120
302
200
28-
20:
93
50
18
92
25
02 Sparsity
00
102
20
40
50
200
25
40
50
102
20
42
60
300
29
40
50
60
100
Sparsity
Sparsity
Sparsity
Spersity
(a) Normalized accuracy and activation sparsity of FFN modules in Gemma3 models across the model sizes.
Semmelk1n
Gemmmw3-4
Gemmmi1-12
Fermei3:227
202
Inter Irgut
230
Sparsiy
Sparsity
Sparsity
Sparshy
(b) Normalized accuracy and activation sparsity for different sizes Gemma3 across FFN modules.
Average accuracy across downstream tasks with different induced activation sparsity for base Gemma3 models. We normalize the accuracy by the original performance of the dense models, and denote the highest (critical) sparsity where at least 99% performance is retained with marker.
Figure 2:
sparsity for dense FFN computation: with sufficiently optimized kemnel that can exploit sparsity in the linear computations, both FLOPs and execution latency of the linear modules scale approximately linearly with activation sparsity (e.g., see Figure 3 in Liu et al. (2025a) and Figure in Szatkowski ct al. (2024). Since these computational gains are already well-established, our work instead focuses on developing broader understanding of activation sparsity across modules and model architectures.
DETERMINING ACTIVATIONS TO SPARSIFY IN NON-RELU LLMS
Modern LLM architectures lack components that explicitly produce zero activations, which makes it difficult to study activation sparsity directly. However, prior work (Chua et al. 2024) Liu et al.) certain
2025a(b) by has shown that some activation vectors e E RR in such models can be spursified degree without incurring significant performance loss. To study the general impact of sparsification on FFN layers, we propose use simple top-p sparsification rule, where we obtain a sparsity mask mp from the maggnttde entries in e whose absolute values sum at least fraction p of the vector's 's total LI norm:
top-p(t) = my @ U3 mp = arg min |mlo s.t. I|m @ 0//1 2p p. ||l|/2 and m E {0. 13
The induced sparsity is then the fraction of zeros in mp: Sp(v) = " E; (mmp = 0). By evaluating model performance over range of p values, off curve and assess the functional activation sparsity of the model the level of sparsity at which the model still performs comparably well to the densely activated original. To this end, introduce the concept of critical sparsity the highest sparsity level at which the model retains at least 99% of the performance. This notion provides with a practical way characterize model's ability to tolerate activation sparsity while anchoring the analysis in realistic performance constraints.
obtain sparsity- perrstt-performance trade-
Our approach simple, general, and easy to interpret. Crucially, it can be applied any FFN module without requiring auxiliary training or calibration, which enables a fair comparison of models and modules. See Appendix for for further discussion and comparison between top-p and the alternatives.
EXPERIMENTS
To study the effects of activation sparsity in LLMs, we evaluate Gemma3 GGema Team 2025). Llama3.1/3.2 (Meta All 2024). and Qwen2.5 (Qwen Team, 2024) models using In-evallharnee Gao et al. 2024b) in a zero-shot setting. Unless otherwise specified, we use pretrained model variants and report the average performance across all the tasks from the task suite
4

---

Published as a conference paper at ICLR 2026
70
Sate
0:30
rput
Critical sparsity
Semmiu LLa/M
Chiticll Spersity
50-
-0-
Effective
Gerrmal Lu3993
10
30
3/
6.73
2:80
0.20
Model Size
Model Size
(a) Model size and critical sparsity.
(b) Activation effective ranks.
Sparsity and corresponding p.
Figure 3: Activation sparsity becomes more pronounced as model size increases. a) Average critical sparsity of FFN components across models, with least-squares trend lines. Larger models generally tolerate higher sparsity, suggesting greater potential benefits. b) Effective ranks (Roy & Vetterli (2007)) of activations on Winogrande, normalized by activation dimension. Larger models show lower effective dimensions, indicating greater redundancy available for sparsification. c) Critical sparsity under All-Inputs sparsification and the corresponding top-p thresholds at which performance degrades. Results are averaged across evaluation tasks, with marker size indicating model size.
from Mirzadeh et al. (2024), which includes: ARC-Easy CClark et al. 2018), ARC Challenge (C lark
ct al. 2018), PIQA (Bisk et al. 2020), BoolQ (Clark et al. 2019), HellaSwag (Zellers et al.. 201 9), WinoCrandd (Sakaguchi et al., 2021). Lambada (Paperno et al. 2016), SciQ (Welbl et al. 2017
and TriviaQA (Joshi et 2017) datasets. For clarity, throughout our study, we primarily focus on Gemma models, they exhibit the most uniform dimension and depth scaling with model sizes. We provide the corresponding results for LLaMa and Qwen in the appendices.
To measure the relation between sparsity and model performance for given activation type from Section 2.2 (input, gate, up-projection, intermediate, or all inputs), we apply the top-p rule uniformly across all layers over a range of thresholds p. For each threshold, we then measure the average induced sparsity and the corresponding performance drop, thereby obtaining the empirical relationship. To explicitly link sparsity with accuracy, we focus primarily on the critical sparsity introduced in Section 3 the highest empirical sparsity level at which models retain at least 99% accuracy. To compare the performance drop induced by activation sparsity applied to different models and tasks, we plot their accuracies normalized by the original performance.
4.1 ACTIVATION SPARSITY ROBUSTNESS IN FEN COMPONENTS
We begin our analysis by investigating the sparsification robustness of different types of activations in Gemma3 models. Specifically, we assess performance degradation under increasing sparsity induced by the top-p rule and show the results Figure 2 Among the FFN components, intermediate activations demonstrate the highest tolerance to sparsity. However, as discussed in Section 2.. leveraging this sparsity for computational acceleration in practice requires a sparsity predictor, which limits its practical benefits. Nevertheless, our results confirm that with sufficiently accurate predictors, exploiting intermediate sparsity remains the most promising strategy for reducing computation.
Interestingly, we observe that simple input-based methods, which use either the FFN input alone or combined with the down- pwojeeroeottion input, achieve sparsity levels comparable to or exceeding those of the gate-based approach often favored in prior work (Song et al. 2024a| Lee et al. 2024). Although the gate activation naturally shrinks activations, its sparsity does not surpass input sparsity; in practice, raw input and up-projection sparsity behave similarly to gates. This makes input- based sparsification the most practical predictor-free approach, as it can accelerate all FFN modules without introducing approximation error. By contrast, gate-based sparsification offers no clear advantage at the model scales studied here, though it may surpass basut- methods in models larger than 30B parameters. Overall, we find that the sparsity robustness of all FFN components generally improves with model size, trend we explore in more detail in the following section.
5

---

Published as a conference paper at ICLR 2026
Interadiate
Insut
Gate
Up- Proprreecioon
AK
Or Spariity
Cr tica Sparsity
Critical Sparsity
Ciitical Spursity
Critical Sparsit
Figure 4: Critical sparsity for Gemma3 models (IB, 4B, 12B, and 27B) across all evaluated modules and tasks. Marker size represents model scale, and tasks with higher accuracy are positioned toward the top. While tasks with higher baseline accuracy generally tolerate sparser activations, critical sparsity varies substantially across tasks, highlighting that activation sparsity is task-dependent.
4.2 MODEL SIZE AND ACTIVATION SPARSITY
Critical sparsity against model size. To assess the generality of our previous findings across families and scales, we consider pretrained LLaMA3. and Qwen2.5 models and plot their critical sparsity in Figure 3a] fitting trends with least squares (see Appendix [ for the table with the exact values). The trends outlined in the previous section remain roughly consistent across model sizes: intermediate activations are generally the most sparse, and the gate sparsity is comparable with input or up-projection sparsity until the larger model sizes. We attribute the slight deviations in the trends to the non-unin non-uniform depth-wide wiihh scaling, particularly prominent in the Qwen model family, where dimensions grow disproportionately with parameter count. Overall, activation sparsity tends to increase with model size, though cannot be directly determined based on the model size alone.
Effective ranks of the model activations. We further examine activation sparsity through effective ranks (Roy& & Vetterli 2007) of activations in Gemma and LLaMA models on the Winogrande task. In this experiment, we do not sparsify activations, and instead compute their effective rank across layers and report the mean value. In Figure 33 we show the effective ranks of intermediate, input, and gate activations. To enable comparisons between different modules and models, we normalize the ranks by the activation dimensionality (see Appendix D for computational details). Effective ranks consistently decrease with model size, reinforcing the observation that larger models exhibit greater capacity for activation sparsity. However, interestingly, gate activations exhibit effective ranks comparable to intermediates, despite their lower empirical sparsification capacity. This suggests that effective rank alone may be insufficient fully capture robustness to sparsification.
Relationship between critical sparsity threshold and model size. Finally, we investigate how the critical sparsity threshold p varies with model size. [c] we plot the relationship between the critical sparsity and p for all-inputs sparsification. Both the threshold and critical sparsity are averaged across all evaluation tasks, with marker size indicating model size. While the critical sparsity for a given model generally varies across tasks, larger models consistently tolerate lower top-p thresholds and exhibit higher critical sparsity. Therefore, as model size increases, maintaining performance requires smaller fraction of the total activation norm and smaller fraction of the overall activations. Together with the previously observed scaling of effective rank, these results have significant implications for efficiency and sparsification, indicating that larger models have an increasingly pronounced long tail of neurons that do not significantly contribute to their performance.
4.3
DOWNSTREAM TASK IMPACT ON ACTIVATION SPARSITY PATTERNS
Recent works on activation sparsity for LLM acceleration (Liu et al. 2023 Lee et al. 2024 Liu et al. b) typically assume that sparsity predictors activation thresholds, tuned small dataset, generalize reliably to downstream tasks. However, activation sparsity patterns are by definition dynamic and input-dependent, which calls this assumption into question. To study how sparsity robustness varies across tasks, we analyze the critical sparsity of the FFN modules in four Gemma variants (IB, 4B, 12B, and 27B parameters, indicated by marker size) in Figure 4 We report performance on separate evaluation tasks, as well as the average across tasks, and order the tasks by the dense model's 's accuracy, with aigher-accuracy ("easier") tasks at the toff
Comparable results for LLaMAA and Qwen models are shown in Appendix D

---

Published as a conference paper at ICLR 2026
onval query
Garmaz-12 10
Cemmu3-4t 40
Normall Accuracy
Normalizzed Accuracy
102
92-
0 5 8 y
02-
70-
62
Downd -DB Inttrut
Doan2 -B ai-.rasiing
LL1DA-03
Sparsity
Sparsity
(a) PT and IT models comparison. (b) IT and Thinking Qwen3-4B. (c) Sparsity in LLaDA LLaMA.
Figure 5: Activation sparsity is a prevalent property across different model types. a) Critical sparsity levels for pretrained and instruction-tuma Gemma3 models. b) Performance instruction-tung and reasoning variants of Qwen3-4B on generative tasks assessing general knowledge, mathematics, and factual accuracy, with critical sparsity indicated by the markers. c) Activation sparsity in LLaMA-8B and the diffusion- based LLaDA-8B, with critical sparsity similarly marked. We report normalized accuracy as the accuracy sparsified model divided by the accuracy of the dense version.
While the tasks that exhibit higher accuracy tend be more robust to sparsification, there no clear trend, and model's 's critical sparsity generally varies per task. This suggests that downstream appli- cations may be differently sensitive to sparsification, challenging the assumption that sparsification rules calibrated held-out dataset will generalize universally.
4.4 MODEL TRAINING AND ROBUSTNESS TO ACTIVATION SPARSITY
As established in the previous section, the model's activation sparsity depends on its architecture, size, sparsification method, and downstream task. The relation between the critical sparsity and the training recipe, however, remains underexplored. To address this, we study activation sparsity in instruction-tuned and reasoning variants of the models.
Critical sparsity in pretrained and instruction-tuned models. We analyze average critical sparsity achieved across our task suite for pretrained Gemma3 models with that instruction tuned variants in Figure Sa At larger scales, instruction-turne models exhibit greater tolerance to activation sparsity, indicating that the training recipe can substantially influence robustness even when the underlying architecture is unchanged. The differences between pretrained and instruction-tuned models are consistent across all tested architectures, with detailed results provided in Appendix B
Reasoning with Qwen-4B. We explore activation sparsity in reasoning models, focusing on Qwen3- 4B, a lightweight and cost-efficient model available in both Instruct and Thinking variants that can be fairly compared. We evaluate the performance of these models on three representative generative tasks: MMLU-Redux (knowledge) (Gema et al. 2024), GSM8K (math) (Cobbe al. 2021 }. and TruthfulQA (factuality) (Lin et al., 2021). We report normalized accuracy on the tasks as a function of activation sparsity in Figure [b] The reasoning model shows slightly greater robustness GSM8K and even modest improvements with moderate sparsity on TruthfulQA, suggesting potential benefits of activation sparsity for the robustness. However, the performance of the reasoning model on MMLU-Rede --eudux degrades more sharply, as higher sparsity tends to lengthen outputs, causing them to exceed the maximum generation limit imposed by our compute constraints.
Our results demonstrate that activation sparsity consistently emerges across multiple post-trained models, independent of their training recipe. In light of these findings, activation sparsity appears to be particularly promising yet relatively underexplored research avenue for enhancing the robustness and efficiency of reasoning models, which are becoming increasingly popular in modern applications.
4.5 EXPERT-WISE SPARSITY IN MIXTURE-OR-EXPERT MODELS
Sparse Mixture-off Experrs (MoE) models (Shazeer et al. 2016) have become a common choice for large-scale LLMs due their efficient capacity scaling, and many recent frontier systems adopt this design (DeepSeck-Al (024). In these architectures, a sparse MoE layer replaces the standard FFN block with multiple experts, only a small subset of which is selected for any given input. Although
7

---

Published as a conference paper at ICLR 2026
All Inees
inmmediste
140
AlHrcctt
Infermadi
112
112
(a) Critical sparsity across Qwen3-30B-A3B layers.
(b) Activation sparsity across the diffusion steps.
Mixture-of-Expere model layers. We show the critical sparsity for each layer averaged over experts, alongside the minimum and maximum values across 128 experts in each layer. b) Critical sparsity of selected LLaDA-8B layers under All-Inputs and Intermediate sparsification on HvmanEval. Overall activation sparsity remains relatively stable across diffusion steps. Interestingly, both early and late layers demonstrate the highest potential for sparsification, whereas middle layers exhibit slightly lower critical sparsity.
Figure 6: a) Critical sparsity in Awee3-33B-A3B
sparsity is usually discussed in terms of expert routing, each expert is still a standalone FFN and exhibits its own activation sparsity. This naturally leads to the question of how activation sparsity varies across experts within the same layer, and how pronounced these differences are in practice.
We study these questions using the Owen3-30B-AS model, 30.30B-apa aaBB-paameter MoE with 3B parame- ters active per forward pass (8 out of 128 experts per token). Using our standard evaluation suite, we measure the average critical sparsity across experts in each layer, along with the per-layer minimum and maximum sparsity of any single expert and the corresponding variance. We show the results for All-Inputs and Intermediate sparsification in Figure 6a] While layer-wise average sparsity across the experts stays relatively stable, individual experts within each layer exhibit notably higher, outlier sparsity levels. Interestingly, these critical sparsity levels exceed dense models and show surprising stability in the intertask variance, as shown in Appendix ( Overall, the results show that MoE experts reach activation sparsity levels comparable to dense LLMs, further reinforcing the universal existence of activation sparsity in modern LLMs and proving its viability for improving MoE's 's efficiency.
Table 1: Critical sparsity values for diffusion-based LLaDA-SB.
4.6 ACTIVATION SPARSITY IN DIFFUSION LLMS
Intermediate All-
Finally, we investigate activation sparsity in the increasingly popular class of diffusion based LLMs. While prior work has explored sparsity and caching in image diffusion models (Ma et al. 2024; Silveria et al.) 2025 Zhang al. 2024a Li et al. 2024a). to our knowledge this is the first analysis of activation sparsity diffusion- based language models. Masked diffusion LLMs generate full sequences in parallel by progressively de- noising across multiple diffusion steps, with each step producing distinct intermediate activations. To apply our sparsification scheme to such models, we compute separate sparsity masks every model forward pass, and report sparsity averaged across all forward passes. We use the official implementation of recently introduced LLaDA-8B (Nie al.) 2025). which adopts the same architecture as autoregressive LLaMA3-8B. This choice enables fair comparisons without additional interference that would stem from the differences in model design.
General Tasks
MMLU
69.46
62.72
ARC-C
56.48
31.74
HellaSwag
71.21
67.92
WinoGrande
50.75
42.35
PIQA
83.25
75.68
Maths and Science
GPQA
62.50
45.94
Coding
HumanEval MBPP
81.25 66.67
77.89
59.18
Chinese
CMMLU
69.01
$7.25
C-Eval
69.25
50.62
Average
68.13
56.79
Critical sparsity in diffusion LLMs. To compare activation sparsity in the autoregressive and diffusion LLMs, we investigate intermediate activations in LLLMM3.1-8B and LLaDA-8B. We examine sparsity characteristics across three tasks from our previous experiments: ARC-Challenge, HellaSwag, and PIQA, which are also analyzed in the LLaDA paper, and show the results in Fig Similar to the autoregressive model, LLaDA exhibits substantial activation sparsity and shows even slightly more favorable sparsity performance trade-offs. To investigate the phenomenon in more detail, measure critical sparsity (averaged over diffusion steps for generation or Monte- Carlo likelihood trials for scoring tasks) on tasks from the LLaDA evaluation suite, with both intermediate- and all-inputs activation sparsification. We present the results in Table T Average critical sparsity
pparsity-performance
8

---

Published as a conference paper at ICLR 2026
Iopit lnyers
lyers
Inpur byens
Interediate lyyers
8-0
Diflusion Step
Diflaton Stop
Lager Indie:
Layar Index
(a) Jaccard similaritics in cach diffusion step.
(b) Consecutive Jaccard similarity per model layer.
Figure 7: Jaccard similarity of the sparsity masks induced with critical sparsity in LLaDA-8B on HumanEval. We sparsify intermediate and both intermediate and FFN input activations, and plot the metrics for input and intermediate layers separately. a) Mask similarity between consecutive diffusion steps and between the current and the first step. We average the similarity across all modules of a given type. b) Consecutive similarity of each model layer, averaged over the whole diffusion process.
across all tasks is substantially higher than that observed in autoregressive LLaMA models in Figure 3al likely due to the denoising nature of diffusion-based generation, which can better tolerate noise introduced by sparsification. Although activation sparsity remains largely unexplored in diffusion LLMs, results indicate its considerable potential to accelerate computation in such models, to an even greater extent than in the autoregressive models.
Sparse activation patterns across diffusion steps. Finally, we analyze the temporal dynamics of activation sparsity in LLaDA. Previous work on autoregressive models has highlighted that sparsity patterns remain relatively stable after the prompt (Dong et al. 22024] Federici et al. 2024). which can be leveraged to make the model inference more efficient, for instance by reducing GPU weight-loading operations. We investigate whether similar efficiency techniques could be applied to diffusion-basee LLMs by analyzing sparsity masks across diffusion steps to assess the presence of comparable temporal stability.
First, we look at the overall activation sparsity levels LLaDA- 8B across the diffusion steps on HumanEval dataset. For each diffusion step f, we apply the top-p rule introduced in Section 3 with fixed p corresponding to the critical sparsity values obtained earlier, and obtain sparse masks mp.t for each diffusion step. We perform this experiment for both intermediate and all-inputs sparsification, and report the average sparsity level for each diffusion step across the whole dataset in Figure 6b While the average activation sparsity slightly decreases throughout the denoising process, every step remains highly sparse at around 80% critical sparsity on average across all layers. Interestingly, we observe that both the earliest and the latest layers show the highest critical sparsity, while the middle layers exhibit slightly less sparsity.
The average activation sparsity remains high throughout the steps of the diffusion process, but that alone does not capture how stable the underlying activation patterns are. To measure this, we compute the Jaccard similarity between the of neurons active for two time steps i and l2
mpp. O Mapa2
J1
Mpptt U Umpatl
We measure both consecutive Jaccard similarity, where io = t - 1, and drift Jaccard similarity, where we fix to O. Consecutive Jaccard measures local stability between adjacent diffusion steps, while drift Jaccard measures how sparsity patterns deviate from the initial sparsity mask.
In Figure 7a] we report the consecutive and drift Jaccard similarities computed over input and inter- mediate activation vectors, averaged across all model layers. While consecutive Jaccard similarity remains stable, drift similarity rapidly declines across diffusion steps, indicating that sparsity patterns undergo gradual yet substantial changes as the denoising process progresses. Notably, when spar- sification is applied only to intermediate activations, the similarity metrics are higher, highlighting that sparsifying both input and intermediate activations makes the sparsity patterns less stable. In Figure 7b] we further present the average consecutive similarity of sparse activation sets across all diffusion steps, analyzed per LLaDA layer. Interestingly, consecutive similarity decreases steadily until approximately fifthh of the model depth, after which it slightly increases near the final layer. Overall, despite the diffusion model maintaining consistently high overall sparsity across the

---

Published as a conference paper at ICLR 2026
denoising process, the similarity between consecutive masks remains too low to allow for their reuse across steps, with potential exception of very early layers which exhibit very high Jaccard similarity.
To the best of our knowledge, our work is the first to demonstrate that activation sparsity is highly prevalent phenomenon in diffusion- LLMs. Our results suggest that sparsity could be a promising avenue for accelerating such models, as the critical sparsity levels in LLaDA significantly exceed the sparsity achieved with autoregressive LLaMA3.1 of the same size. However, our analysis of temporal sparsity patterss indicates that the behaviors observed in diffusion LLMs do not always mirror those seen in autoregressive models. Therefore, effective sparsification techniques for diffusion LLMs are likely to require model-specific solutions that leverage the unique properties of this new class of models.
Sparsification in diffusion LLMs.
5 CONCLUSIONS AND DISCUSSION
Despite lacking any architectural bias toward explicitly sparse activations, modern LLMs consistently exhibit functional sparsity. We argue that functional sparsity is universal property of LLMs and advocate for its wider adaptation when designing efficient models.
We find that larger models tend exhibit higher sparsity, suggesting that frontier models will become sparser as scaling continues. Therefore, activation sparsity stands out as a promising tool for accelerating ever-growing LLMs, and it already starts appear in common model families, such as the recent Gemma3n release that includes activation sparsity-aware layers (Yu et al. 2025).
Our results show that input activations match or exceed the sparsity of gates and up-projections. Computing gates to choose sparsity patterns (Lee et al. inputs, and newer work (Liu et al. 2 et al.
2024) is wasteful if they are no sparser than
2024 Liu et al.) (0255) demonstrates stronger acceleration with purely input sparsity. Overall, our results suggest that input sparsification is the most practical training-free approach to leveraging activation sparsity for model acceleration.
Federici
The high variance of critical sparsity across evaluation tasks and training recipes calls into question methods that rely on extra training (Zhang et al. 202T; Liu et al. 2023) Szatkowski et al.. (024) or threshold calibration (Lee et al. [024 Liu et al. (2025a) on auxiliary datasets. Our results suggest that sparsification methods should be truly data-free, as both functional sparsity levels and resulting patterns can be prone to overfitting.
Our results should be seen as a lower bound on activation sparsity, as we adopt a simple, broadly applicable framework. While layer- or module-specific methods may achieve higher sparsity, top-p approach already reaches practical levels comparable to existing work. Given this and our earlier arguments on overfitting, argue that sparsification method design should favor simplicity.
Our work is also the first to examine functional sparsity in diffusion LLMs. We highlight sparsity as a promising avenue for improving their efficiency, and expect that activation sparsity could see increasing adoption in diffusion LLMs as their development advances.
We focus activation sparsity in feed-forward network (FFN) layers, excluding multi-head attention (MHA) to isolate the effect. In large models (28B parameters), FFNs dominate inference costs for typical context lengths, and the cost of MHA only becomes significant for extremely long se- quences (Casson 2023). Even in such cases, techniques like K KV-cache compression more effectively accelerate MHA due to its quadratic cost, making activation sparsity for attention less practical.
Finally, we argue that evaluations of activation sparsity methods should prioritize performance preservation, as captured by our notion of critical sparsity. Effective speedups from activation sparsity are limited to roughly 1.3-1.5x (Lee et al., 2024. Liu et al. 2025a b). while techniques such as speculative decoding enable lossless speedups of up to 4x (Li et al.) 20246] (025). Consequently, activation sparsity should be viewed as complementary other acceleration methods rather than as the sole focus of efficiency studies on model acceleration techniques.
We have systematically analyzed activation sparsity in LLMs, demonstrating that functional sparsity is pervasive and tends to increase with model scale. We hope our work highlights the potential of activation sparsity for efficiency and provides insights for future model design. Our findings emphasize the growing significance of activation sparsity, and we expect that, as models continue to scale, it will become an increasingly important tool for building efficient and high-performing L.Mss.
10

---

Published as a conference paper at ICLR 2026
Reproducibility statement. All experiments in this paper conducted NVIDIA A100 (40GB) or GH200 (80GB) GPUs. We used an open-source LLM evaluation framework along with publicly available datasets and models, introducing only minor code modifications to induce activation sparsity. To ensure reproducibility, we have published our code at https
on- -on--parstty--bbccmaaaa
Ethics statement. Our work aims to advance the understanding of activation sparsity in large language models (LLMs). This phenomenon can leveraged to obtain more efficient, robust, and interpretable models, thereby democratizing access to LLMs, reducing their cost, and making them safer. We do not identify any immediate ethical concerns specific to our method; however, as with any technique that improves the capabilities of LLMs, there is potential for misuse. We therefore urge that any deployment of resulting models be approached with caution and with consideration of broader societal impacts.
In line with the ICLR LLM policy, we disclose that LLMs were used only to refine the writing of this manuscript. All ideas and content originate from the authors.
Contributions statement. Filip was the primary contributor to this work, leading the development of the core ideas, the majority of the implementation, and the writing of the manuscript. Patryk contributed the code and experimental results and helped write the section on the diffusion LLMs. The remaining authors provided input on the research direction and contributed the writing and revision of the manuscript throughout the project. Author order beyond the first author is alphabetical.
ACKNOWLEDGEMENTS
This research was supported by National Science Centre (NCN, Poland) Grants No. 2022/45/B/STT602881 and 2024/53/N/ST6/0007788 This research was partially funded by National Science Centre, Poland, grant no: L/D///////////0244 This paper has been supported by the Horizon Europe Programme (HORIZON-CLEC) CHORZZONCCL--002-HMMMMND under the project "ELIAS: Eu- ropean Lighthouse of AI for Sustainability". GA no. 101120237. We gratefully acknowledge the Polish high- tigh-performance computing infrastructure PLGrrdd (HPC Center: ACK Cyfronet AGH) for providing computer facilities and support within the computational grants PLG/2024/012202. PLGG/025501 8230, Pl.G/2025/088391.

---

Published as a conference paper at ICLR 2026
REFERENCES
Subutai Ahmad and Luiz Scheinkman. How can we be so dense? the benefits of using highly sparse representations. arXiv preprint /903v 1257, 2019.
Yash Akhauri, Ahmed F AbouElhamayed, Jordan Dotzel, Zhiru Zhang, Alexander M Rush, Safeen Huda, and Mohamed S Abdelfattah. Shadowllm: Predictor-base baaee contextual sparsity for large language models. arXiv preprint arXiv:2406. 16635, 2024.
Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mchrdad Farajtabar. Lim in a flash: Efficient large language model inference with limited memory. In Proceedings the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 22622-15844 2024.
Maksym Andriushchenko, Aditya Vardhan Varre, Loucas Pillaud- Vivien, and Nicolas Flammarion. Sgd with large step sizes learns sparse features. In International Conference on Machine Learning, pp. 903-925, PMLR, 2023.
Pranjal Awasthi, Nishanth Dikkala, Pritish Kamath, and Raghu Meka. Learning Neural Networks with Sparse Activations. In Conference on Learming Theory (COLT), 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. In Proceedings of the AAAI conference on artificial intelligence, 2020.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Shiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah-E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards Monosemanticity: Decomposing Language Models with Dictionary Learning. Transformer Circuits Thread, 2023.
Adam Casson. Transformer flops, May 2023. URL http://wwwwdaaaaacooocccccoo//// (transformer- fl Accessed: 2025-09-08.
Vui Seng Chua, Yujie Pan, and Nilesh Jain. Post-Training Statistical Calibration for Higher Activation Sparsity. arXiv, 2024.
Christopher Clark, Kenton Lee, Ming-War Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint a9Xiv: 10044, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:
2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reïichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse Autoen- coders Find Highly Interpretable Features in Language Models, arXiv, 2023.
Andrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint 1rXi::1312.4461, 2013.
DeepSeek-AL. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. arXiv, 2024.
Guneet Dhillon, Kamyar Azizzadenesheli, Zachary C Lipton, Jeremy D Bernstein, Jean Kossaifi, Aran Khanna, and Animashree Anandkumar. Stochastic Activation Pruning for Robust Adversarial Defense. In International Conference Learning Representations (ICLR), 2018.
12

---

Published as a conference paper at ICLR 2026
Harry Dong, Beidi Chen, and Yuejie Chi. prmppt-promped Adaptive Structured Pruning for Efficient
LLM Generation. arXiv, 2024.
Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodds, Jackson Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Kadavath, Josh Jacobson, Eli Tran-Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCandlish, Dario Amodei, and Christopher Olah. Softmax linear units. Transformer Circuits Thread, 2022.
hutp::/trraastomee-cccruuus http://ttatommmmmmrrrmmiiiissb/d pub:2202225sou///dxxxx..u
Marco Federici, Davide Belli, Mart van Baalen, Amir Jalalirad, Andrii Skliar, Bence Major, Markus Nagel, and Paul Whatmough. Efficient LLM Inference using Dynamic Input Pruning and Cache- Aware Masking, arXiv, 2024.
Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and Evaluating Sparse Autoencoders. In International Conference on Learning Representations (ICLR), 2024a.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang. Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. The Language Model Evaluation Harness, 2024b.
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino. Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? arXiv preprint 24Xx::2406004127, 2024.
Gemma Team. Gemma: Open Models Based on Gemini Research and Technology. arXiv, 2024.
Gemma Team. Gemma 3 technical report. arXiv preprint arXiv:2503. 2025.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer Feed-Forward Layers Are Value Memories. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep Sparse Rectifier Neural Networks. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv: 1705.0351,, 2017.
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and Exploiting Activation Sparsity for Fast Inference on Deep Neural Networks. In International Conference on Machine Learning (ICML), 2020.
Je- Yong Lee, Donghyun Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini. -ontextually-Awaare Thresholding for Sparsity in Large Language Models. In Conference on Language Modeling (COLM), 2024.
CATS:
Bocheng Li. Zhujin Gao, Yongxin Zhu, Kun Yin, Haoyu Cao, Deqiang Jiang, and Linli Xu. Few-shot Temporal Pruning Accelerates Diffusion Models for Text Generation. In Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), 2024a.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. In International Conference on Machine Learning (ICML), 2024b.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-3: Scaling up Inference Acceleration of I Large Language Models via Training- Test. arXiv, 2025.
13

---

Published as a conference paper at ICLR 2026
Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix Chern Felix X. Yu, Ruiqi Guo, and Sanjiv Kumar. The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers. In International Conference on Learning Representations (ICI.R), 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07995: 2021.
James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, and Ben Athiwaratkun. Training- Free Activation Sparsity in Large Language Models. In International Conference on Learning Representations (ICLR), 2025a.
Kai Liu, Bowen Xu, Shaoyu Wu, Xin Chen, Hao Zhou, Yongliang Tao, et al. La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation. In International Conference on Machine Learning (ICML), 2025b.
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, and Beidi Chen. Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. In International Conference on Learning Representations (ICLR), 2023.
Yuqi Luo, Chenyang Song. Xu Han, Yingfa Chen, Chaojun Xiao, Xiaojun Meng, Liqun Deng. Jiansheng Wei, Zhiyuan Liu, and Maosong Sun. Sparsing law: Towards large language models with greater activation sparsity. arXiv preprint arXiv:2411.02333 2024.
Xinyin Ma. Gongfan Fang, and Xinchao Wang. DeepCache: Accelerating Diffusion Models for Free. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Meta AL. The Llama 3 Herd of Models. arXiv, 2024.
Seyed-Iman Mirzadeh, Keivan Alizadeh-Vaiie Vahid, Sachin Mehta, Carlo del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. ReLU Strikes Back: Exploit- ing Activation Sparsity Large Language Models. In International Conference on Learning Representations (ICLR), 2024.
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Evan Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-experts language models. In
The Thirteenth International Conference on Learning Representations, 2025.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, JUN ZHOU, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large Language Diffusion Models. In International Conference on Learning Representations (ICLR), 2025.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint arXiv: 00000, 2016
Ze Peng, Lei Qi. Yinghuan Shi, and Yang Gao. A theoretical explanation of activation sparsity through flat minima and adversarial robustness. arXiv preprint arXiv:2309.03004, 2023.
Qwen Team. Qwen Technical Report. arXiv, 2023.
Qwen Team. Qwen2.5 Technical Report. arXiv, 2024.
Minsoo Rhu, Mike O'Connor Niladrish Chatterjee, Jeff Pool, Youngeun Kwon, and Stephen W Keckler. Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks. In IEEE International Symposium on High Performance Computer Architecture (HPCA), 2018.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effeectvee dimensionality. In 2007 15th European signal processing conference, pp. 606-610. IEEE, 2007.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 2021.
14

---

Published as a conference paper at ICLR 2026
Noam Shazeer. GLU Variants Improve Transformer. arXiv, 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference Learning Representations, 2016.
Austin Silveria, Soham V. Govande, and Daniel Fu. Chipmunk: Training- Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas. arXiv, 2025.
Chenyang Song, Xu Han, Zhengyan Zhang, Shengding Hu, Xiyu Shi, Kuai Li, Chen Chen, Zhiyuan Liu, Guangli Li, Tao Yang, et al. Prosparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models. arXiv, 2024a.
Yixin Song, Haotong Xie, Zhengyan Zhang, Bo Wen, Li Ma, Zeyu Mi, and Haibo Chen. Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters, arXiv, 2024b.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive Activations in Large Language Models. First Conference on Language Modeling, 2024.
Filip Szatkowski, Bartosz Wójeik, Mikot aj Piórczyúski, and Simone Scardapane. Exploiting Activation Sparsity with Dense to Dynamic-k ofxtuure-ff-xpertt Conversion. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models. arXiv, 2023.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv: 7707706209 2017.
Chong You. Kan Wu, Zhipeng Jia. Lin Chen, Srinadh Bhojanapalli, Jiaxian Guo. Utku Evci, Jan Wassenberg, Praneeth Netrapalli, Jeremiah J. Willcock, Suvinay Subramanian, Felix Chern, Alek Andreev, Shreya Pathak, Felix Yu, Prateek Jain, David E. Culler, Henry M. Levy, and Sanjiv Kumar. Spark Transformer: Reactivating Sparsity in FFN and Attention. arXiv, 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905.07830 2019.
Evelyn Zhang, Bang Xiao, Jiayi Tang, Qianli Ma, Chang Zou, Xuefei Ning, Xuming Hu, and Linfeng Zhang. Token Pruning for Caching Better: 9 Times Acceleration Stable Diffusion for Free. arXiv, 2024a.
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Moefication: Transformer Feed-forward Layers are Mixtures of Experts. arXiv, 2021.
Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Chaojun Xiao, Xiaozhi Wang, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, and Jie Zhou. Emergent Modularity in Pre-trained Transformers. In Findings of the Association for Computational Linguistics (Findings of ACL), 2023.
Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, and Maosong Sun. Relu2 wins: Discovering efficient activation functions for sparse llms. arXiv preprint arXiv:2402.03804, 2024b.
15

---

Published as a conference paper at ICLR 2026
Appendix
A HOW To INDUCE ACTIVATION SPARSITY IN FFNS?
A.1 ALTERNATIVE SPARSIFICATION RULES
In Section 3 we propose to use the top-p sparsification rule to induce the sparsity in the activation vectors of the models. We opt for simple sparsification rule to avoid any data dependency or bias towards specific model or FFN module. However, top-p is not the only possible way to perform sparsification, and many other works opted for alternative methods to extract the sparse subsets of neurons, such top-k Zhang et al. 2021) max-p (Szatkowski et al. 2024).
For vector v E R", top-k finds & largest neurons in the vector and can be formally defined as transformation that multiplies with subset of k neurons which maximizes the norm of the sparsified vector:
top-k(v) ma O 0; my
arg max I| @ e/Iı s.t. ||m]lo = and m E {0, 13".
m
Similarly, max-p finds the subset of the neurons that satisfy the condition that their absolute values are at least p max(v):
mux-p(u)= mp C ot my arg min |mm/lo s.t. |ev| m, 2 p- 1/olloo Vi and m {0,1)"
where ||||, max; Jual denotes the maximum absolute entry of v, and the mask mp retains exactly those coordinates for which |el 2 p- |/ollc. Notably, the mask always selects the largest entry in the activation vector.
We empirically compare the three sparsification strategies in Figure g focusing on the spar- sity-aceuracy tradeoff averaged over our evaluation tasks for the smallest model in each family, Overall, top-p and top-k produce very similar curves, whereas max-p underperforms in certain settings. Therefore, we adopt top-p for our experiments, as is more interpretable than top-k and can more universally transfer across model sizes. In particular, larger models typically yield higher sparsity, requiring to carefully chosen as most values have no effect until critical sparsity reached. By contrast, with top-p performance degrades more smoothly and predictably, allowing us to evaluate fixed set of thresholds that transfer well across models.
irput
intemdatte
Gate
Up Projection
55.
Gummaz
Sonsiioatoo Pule
Part
Figure Comparison of sparsification rules for different models and different blocks.
16

---

Published as a conference paper at ICLR 2026
A.2 GENERALIZED top-p OPERATOR
We further investigate generalized top-p rule, denoted as top-p", which is given by:
top-p"(u) mp + i; mp arg min |mm]lo s.t. I/m @ +|1 2 p ||ö|| and m E {0, (1)
m
where ê = |1| / for L > O. This formulation potentially allows finer control over the sparsification algorithm, with > making the sparsification rule with given p select smaller subsets of large activations, while L < leading to rule usually selecting more neurons for given threshold. Notably for L = the rule equivalent to the top-p used in the main body of our paper.
We compare top-k, and top-p" with L {0.5, 1.0, 2.0} Qwen2.5 models, applying sparsification to all-inputs, intermediate and gate activations, and present the findings in Figure 9 There are slight fluctuations in the sparsity characteristics obtained with the tested rules, but all approaches produce very similar results. While particular values of or might transfer to different sparsity levels, with sufficiently dense sampling of the theresholds the characteristics for all the rules cover similar sparsity levels, and there are no meaningful perfomance gains from using one given approach. This further supports our choice standard top-p as the default sparsification operator, as its simply easier to practically apply, while performing as well as the other alternatives.
All-Inputs
intermeiatee
Gate
Qwen2 5.5.000 0.58
30-
Owen2.5 5.5 1.58
50
Qiven2.5 3-
Sparsflicaton Rulc
Tap-K
Tip-PIL-1.0
PIL-2.0)
40
80
100
100
20
100
Sparsity
Sparsity
Sparsity
Figure 9: Accuracy characteristics under sparsification with variants of generalized top-p and top-k for 5wen2.5 0.5B, 1.5B, and 3B.
17

---

Published as a conference paper at ICLR 2026
A.3 SPARSIFICATION RULE TRANSFERABILITY BETWEEN THE MODELS
To further study the transferability and impact of the threshold selection in different models, we investigate the activation sparsity induced in separate layers of Gemma3 and Qwen2.5 models. We select a subset of thresholds, and register the activation sparsity obtained at a given layer alongside the average of the accuracy under the threshold. We plot the results as heatmaps in Figures 10 and
Top-p threshold values and resulting sparsity induced in the gate activation vectors alongside the accuracy with the given threshold across different layers of Gemma3 and Qwen2.5.
Figure 10:
First, we investigate the sparsity of gate activations in the Gemma and Qwen models of corresponding sizes in Figure 10 Except for some early layers, the sparisty values obtained across the models appear similar for a given threshold across the middle layers.
Figure 11: Top-p threshold values and resulting sparsity for Gemma3-277 and Qwen2.5-32B models.
Then, in Figure 10 we plot the sparsity of all four investigated activation vector types in the largest models within each model family. Again, except for few layers, the sparsities obtained for given p appear similar between the models.
Both of these results support the universality of our approach and our decision choose top-p over top-k, as using top-k would require more manual threshold selection to find the critical sparsity, as outlined by the variance in the critical sparsity across different model sizes and types in Section While the resulting sparsity heatmaps show few high outliers, particularly for Gemma3-278 gates, we attribute the presence of these to the presence of massive activations (Sun al.) 2024). as for massive outlier values, the magnitude of the vector norms that we use will concentrate around very few large values and may even cause exact sparsity to appear at p = 1.0 as the nature of the numerical precision will make the smallest entries in the activation vector appear like zeros since they basically contribute nothing compared to the massive outlier. We not investigate this phenomenon further and leave it for future work. However, note that it can have important implications for the design of activation sparsity approaches, particularly those that rely on thresholding, as rules and thresholds devised for massive activations might be highly unstable upon encountering ait-of-distribution data.
18

---

Published as a conference paper at ICLR 2026
B CRITICAL ACTIVATION SPARSITY OF PRETRAINED AND INSTRUCTION- DINRUCTION--TNED MODELS
Table 2: Critical activation sparsity for pretrained and instruction- models. Sinters Sail_in in. Syates Simput and S Sup refer to intermediate, all inputs, gate, input and up-projection sparsification, respectively, and L, dm and d, stand for number of layers, model and intermediate dimensionality.
Pretrained
Instruction- Tuned
Model
din
Sinter
Sail_u .m
Spote
Siput
Sup,
Sister
Sail .m
Syote
Suget
Sup,
Gemmaš-18
1152
6912
28.53
32.96
49.98
23.23
32.14
30.27
Remma3-4B
34
2560
10240
58.56
40.46
28.50
34.63
39.72
62.82
47.78
35.99
42.29
40.82
Gemma3- 12B
48
3840
15360
69.46
52.54
42.05
43.03
42.03
78.77
61.45
55.45
50.74
54.26
Gemma3-27B
5376
21504
74.12
53.03
50.83
42.01
84.05
74.25
68.15
59.88
56.95
LLaMA3 I-BIB
2048
8192
44.44
30.93
26.51
28.09
28.82
45.02
32.52
24.30
29.65
29.70
LLaMA3.2-30
3072
8192
49.58
34.51
25.52
35.91
37.14
58.07
36.62
28.90
33.28
44.72
LLaMA3.1 1...3.11880 188
32
4096
14336
51.89
38.97
28.04
37.31
30.52
61.96
44.35
34,38
39.34
41,76
Qwen2.5-02
24
896
4864
46.54
36.66
17.16
42.01
29.20
43.92
33.20
24.60
32,80
32.12
Qwen2.5-1.5B
1536
8960
50.49
40.98
25.93
40.12
35.50
52.93
39.14
27.63
32.50
32,99
Qwen2.5-3B
2048
11008
71.16
39.58
39.40
43.46
59.80
47.47
36.90
44.16
36.32
Qwen2.5-7B
28
3584
18944
60.98
51.77
37.25
47.89
43.05
59.95
51.73
32.58
47.01
40,62
Qwen2.5-144 14H
5120
13824
71.66
54.66
48.04
47.39
52.25
69.35
56.89
41.87
50.10
49.04
Qwen2.5-32B
5120
27648
65.66
40.20
54.08
$2.46
68.77
40.54
55.17
57.35
In Table 2 we report the exact numerical values of the critical activation sparsity for all models con- sidered our experiments, including both pretrained and instruction-tuned variants. For context, we also include key model architectural parameters such the number of layers, model dimensionality, and intermediate dimensionality. While we do not observe clear, direct relationships between these parameters and the achieved critical sparsity, the general trend of sparsity increasing with model size remains evident. Notably, the Qwen family exhibits some fluctuations, which may partially be explained by the non-uniform scaling of its architectural parameters across model sizes.
LLaMA3-1B
Qwen2.5-0.58
Qwen2.5-1.58
Critical sparsity
Critical sparsity
80
60
40
40
20
20:
LLaMA3-B8 3B
Qwen2.5-33
Qwen2.5-78
Critical sparsity
Critical sparsity
60
60
60
40
40
LLaMA3-8B
Qwen2.5-14B
Qwen2.5-32B
Critical sparsity
Critical sparsity
80
80
80
40
Pretrainec
Pretrained
20
Instruction-Tuned
20
20
Instruction-Tuned
Inter.
Input
Gate
Up-Proj.
All-In-
Inter.
Input
Gate
Up-Pmoj.
All-In.
Inter.
Input
Gate
Up-Proj.
All-In.
Figure 12: Critical sparsity plots for LLaMA and Qwen models, corresponding to the Figure 5a
19

---

Published as a conference paper at ICLR 2026
C
AVERAGE ACCURACY OF PRETRAINED QWEN2.5 AND LLAMA3 MODELS.
Sparsity for different FFN modules at various model sizes.
Normalized Accurary
102
Intermediate
Input
192
102
Up-Projection
162
All-Inputs
02
SC-
92
SG-
38
8B
22
102
30
20
20
15
Spansity
Sparsity
Sparsity
Sparsity
Sparsity
(b) Sparsity for different models various modules.
LLMMA3-1B 18
LLaMA3-3B
LLaMA3-8B
1.2
152
Normalized nccurass
26
56
Inter
Irput
42-
Allin.
100
Sparsity
Sparsity
Sparsity
Figure 13: Average accuracy across downstream tasks normalized by the original performance with different induced activation sparsity for base LLaMA3 models, corresponding to the plots in Figure
(a) Sparsity for different FFN modules at various model sizes.
Nomalbed Accuracy
202
Intermediste
Input
152
Gate
102
Up Up-Projection
152
All-Inputs
100
102
24
94
P4
Sparsity
Sparsity
Sparsity
(b) Sparsity for different models at various modules.
Spank
Saundy
Figure 14: Average accuracy across downstream tasks normalized by the original performance with different induced activation sparsity for base Qwen2.5 models, corresponding the plots in Figure
20

---

Published as a conference paper at ICLR 2026
D PER-TASK SPARSITY FOR LLAMA AND QWEN MODELS
Intereditte
Insut
Gate
Up- Us-Prejction
AtHinues
Awea
Oriiicl Sparsity
Critical Soarsity
Citicall Sparstty
Crttical Soersiyy
Critical Sparsity
(a) Critical sparsity for LLaMA3 models (IB, 3B, and 8B).
Intermediate
Insul
Up-Prqeccion
Alinputs
Qatical
Cribial Spuraity
Ertiical
Criiicll Spersity
Critical
(b) Critical sparsity for Qwen2.5 models (0.5B, 1.5B, 3B, 7B, 14B, and 32B).
Figure 15: Critical sparsity of different tasks across all the evaluated modules. The marker size denotes the model size, and the tasks are ordered by their accuracy (the tasks at the top achieve the highest accuracy). The sparsity at which the model performance degrades varies considerably between the tasks. Although the "easier" tasks with higher accuracy tend to be more tolerant to sparsity, there is no clear correlation, which suggests that activation sparsity patterns tend to be highly data-dependent.
EFFECTIVE RANK COMPUTATION
To compute effective ranks of activations in Section 4.2 we follow the formula from (Roy & Vetterli 2007), which we restate here for the sake of the paper's 's self-containment. In our experiments, we apply the following formula to each batch and then report the effective rank as the average across all the batches.
To compute the effective rank of activation matrix E Raxd, where n refers to the number of tokens within batch, and d is the feature dimensionality of the model, we first compute singular value decomposition (SVD) of A:
A A=
where D E Raxd xd is diagonal matrix containing the singular values 02 2 ... 2 aQ 2 0, and Q is defined as Q = min(ni, d).
We can further define (a), 22 o2)"" and the singular value distribution is then given by:
ok for A = 1........ Q,
pk
||l||1
where denotes transposition and ||- || the LI norm.
The effective rank of A is then defined:
crank(A) = ( Pp.....pp))
where H(P1, (1,1,P2.... .P.. PQ) is the Shannon entropy of the singular value distribution.
.......
21

---

Published as a conference paper at ICLR 2026
F ADDITIONAL RESULTS FOR MOE MODELS
F.1 INTER-TASK VARIANCE IN CRITICAL SPARSITY LEVELS FOR QWEN3-30B-A3B
In this section, we present the results complementary to the ones presented Section 4.5 Similar Figure baj we investigate minimum, maximum, and average sparsity in each layer in Qwen3-30B- A3B across the isolated tasks from our task suite. Interestingly, we find that average sparsity is very similar across each task, and the only notable variance in the model behaviour can be observed in the maximum sparsity, which sometimes fluctuates per expert. While our previous results for the dense models indicated a large variance in critical sparsity per task, these results rather indicate that such variances average out across large numbers of experts MoE architectures.
All Inputs
Intermediate
M
Critical Sparsity
80
70
Challenge
HellSwag Lambadi
Easy
Winogrande
BoolQ
5dg
15
30
35
Layer
Layer
Figure 16: Critical sparsity across Oween-33B-A- layers measured per-task.
22

---

Published as a conference paper at ICLR 2026
F.2 RESULTS FOR OLMoE-7B-AN
To complement our analysis. we repeat Qwen3-30B-A33 experiments with smaller MoE, OLMoE- 7B-AIB IB (Muennighoff al. 2.25). OLMoE has 16 MoE layers and uses 8 out of 64 experts for a given token. Figures] 17 and 18 we present the plots corresponding to the Figure 6a] from the main paper and the per-task activation sparsity analysis presented Figure [6] Overall, the sparsity in the 7B-AIB model is smaller than in the 30B-A3B Qwen, and the per-expert variance is smaller, as also expected. Interestingly, in this model, intermediate sparsity close all-inputs, which can be explained by the lower expansion rate in OLMoE FFN, where the intermediate dimension is just two times larger than the hidden dimension. Nonetheless, the interesting property of experts showing small variance in sparsity across the tasks observed for the Qwen model persists in the smaller MoE.
All Inputs
Intermeiiate
90
80
Critical C Sparsity
70
60
50
Layer
Layer
Figure 17: Critical sparsity across OLMoE-7B-AII across the layers, with standard deviation shaded and the minimum and maximum sparsity for each expert marked with dotted lines.
All Inputs
Intermediate
90
Critical Sparsity
50
40
ARC Challenge
HellaSwag Lambada Scio)
TivaQA
30
Winegrande
BeelQ
20
10
12
14
10
12
Layer
Layer
Figure 18: Critical sparsity across OLMoE-7B-AII layers measured per-task.
23


Reviews:
[{"review_id": "MDi8pNIiDU", "reviewer": "Reviewer_7CAX", "summary": "This paper investigates activation sparsity in modern large models. Beyond ReLU-based networks that produce exact zeros, using SiLU/GELU activations in FFNs yields functional or approximate sparsity, with many activations near zero. The authors address the current fragmented understanding of this phenomenon by introducing a general, training-free framework to evaluate sparsity robustness.", "questions": "Please refer to \"Weaknesses\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "They propose a simple top-p sparsification to induce sparsity in various activations (input, gate, up-projection, intermediate). This allows for the introduction of critical sparsity, i.e., the maximum sparsity level that causes less than 1% performance drop. Through extensive experiments on models like Gemma, Llama, and Qwen across different scales (e.g., 1B to 32B parameters), they find that the potential for effective activation sparsity increases with model size. They also find that input-based sparsification is as effective as, or even better than, the more commonly studied gate-based methods, making it a more practical, predictor-free approach. The study also shows that critical sparsity is task-dependent, varying significantly across different downstream evaluations. It also persists across different model types, including instruction-tuned and reasoning-specialized variants. All these provide a comprehensive perspective and practical guidance for harnessing activation sparsity in model design and acceleration.", "weaknesses": "This paper's methodology introduces sparsity via the top-p rule and defines critical sparsity as the level at which performance degradation is less than 1%. This is similar to the paper \"Sparsing Law: Towards Large Language Models with Greater Activation Sparsity\". Further comparison of different sparsity definitions is crucial, as a better definition of sparsity will result in less reduction in model performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ktjQmA39VO", "reviewer": "Reviewer_YvVz", "summary": "The paper addresses the lack of a general understanding of activation sparsity in modern Large Language Models (LLMs) that utilize non-ReLU activations, which prohibits the use of traditional zero-based sparsity methods. To solve this, the authors introduce a novel, general framework to evaluate sparsity robustness in contemporary LLMs, focusing on the Feedforward Network (FFN) layers. Through systematic investigation across diverse models and scales, the work reveals a key universal property: the potential for effective activation sparsity increases with model size. Furthermore, the study presents the first analysis of activation sparsity in diffusion-based LLMs, ultimately offering a comprehensive perspective and practical guidance for leveraging this phenomenon in LLM design and acceleration.", "questions": "none", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a simple, training-free top-p sparsification method and the metric of critical sparsity (maximum sparsity retaining 99\\% performance), providing a unified and fair way to compare sparsity tolerance across different LLM architectures and FFN components.\n\n2. The work systematically confirms that the critical sparsity increases with model size. This is consistently reinforced by the finding that the effective rank of activations decreases with model size, offering strong evidence that larger models inherently possess greater exploitable redundancy.\n\n3. The investigation into diffusion-based LLMs (LLaDA) is novel, revealing that they exhibit substantial activation sparsity and even slightly more favorable sparsity-performance trade-offs than autoregressive models, highlighting a new acceleration opportunity.\n\n4. Based on empirical data, the authors provide the practical insight that input activation sparsification is the most effective training-free approach, as its sparsity tolerance is comparable to or greater than gate or up-projection activations.", "weaknesses": "1. A major finding is that the critical sparsity varies substantially across different downstream tasks and training recipes (e.g., instruction-tuning). This challenges the core assumption of many prior acceleration methods that sparsification rules calibrated on an auxiliary dataset will generalize universally without overfitting\n\n2. The paper acknowledges that the effective speedups from activation sparsity methods are practically limited to a factor of 1.3x to 1.5x, which is less compelling when compared to alternative lossless techniques like speculative decoding that can achieve up to 4x speedups.\n\n3. The analysis is explicitly constrained to only the FFN layers, intentionally excluding the Multi-Head Attention (MHA) module. While a cost justification is provided, this limits the comprehensiveness of the \"universal properties\" claim within the entire Transformer architecture.\n\n4. The use of effective rank as a theoretical proxy for redundancy is weakened by the observation that gate activations show a high effective rank yet exhibit a low empirical capacity for sparsification, suggesting that this metric is insufficient to fully capture robustness to sparsification", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "te0WqYasAP", "reviewer": "Reviewer_PkEJ", "summary": "This work proposes a unified framework for evaluating sparsity robustness in large language models and systematically examines activation sparsity in their feedforward layers. The study identifies consistent sparsity patterns across different model families and sizes, revealing that larger models exhibit greater potential for effective activation sparsity. It also provides the first analysis of sparsity in diffusion-based LLMs. Overall, the work offers insights and guidance for leveraging sparsity to improve LLM efficiency and design.", "questions": "- What is  \"functional sparsity\"? The authors not seem to provide good explanation across the paper, though this term is very important.\n- What is \"effective rank\"? Need short explanation in the paper to be self-contained. For instance, what \"effective rank of 0.1\" means? How it corresponds to your conclusion.\n\nPersonally, I like this type of paper giving inspiration. Please work on the quesitons and weakness points if appropriate.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper uncovers some universal properties of activation sparsity across diverse model families and scales. This could be quite helpful to the community.\n- The paper is well written.\n- Experiments are quite quantitive and informative.", "weaknesses": "- Pure experimentation-driven work without any theoretical analysis or others.\n- Larger model leads to higher critical sparsity. This point is actually very straightforward. For instance, usually larger model and smaller model are trained on the same pile of data. Of course the larger model admit more redandancy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3f4MMHsCUG", "reviewer": "Reviewer_BibT", "summary": "This paper presents a comprehensive study on the properties of sparsification in the FFN layers of transformer-based language models and diffusion language models. Specifically, they use a simple top-$p$ sparsification strategy and analyze the robustness against sparsity of various models from the Gemma3, Qwen2.5, and LLaMa3.1/3.2 family, by measuring at what sparsity levels does downstream task performance drop below 99% of its original performance.\n\nSome interesting findings from this paper include: input-based sparsification is often more attractive than gate-based sparsification, larger models are more robust to sparsification, and diffusion models are more robust to sparsification. The findings of this paper are valuable for designing language models with better robustness against sparsification or methods that better leverage sparsification to achieve inference speedup.", "questions": "- What does \"Normalized accuracy\" in Figure 5 mean?\n- Can you provide more insights into why MMLU-Redux is less robust to sparsification compared to GSM8K and TruthfulQA?\n- It seems that you apply the same $p$ value across all layers. However, I think using a different $p$ for different layer is more reasonable. Is it possible to repeat the experiments, but with a layer-specific $p$ value such that each layer can have its own critical sparsity?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-presented and generally easy to follow.\n- The findings of this paper is very interesting and highly valuable to researchers interesting in language models with sparse activations. \n- The methods and experimental setup is well designed, making the results reliable and convincing.", "weaknesses": "- How sparse activation patterns evolve across diffusion steps is an interesting phenomenon and the paper does a good job in bringing it up (starting with Line 406, and Figure 6). However, the conclusions of the experiments in this regard is inconclusive and it is unclear how they are useful for the community.\n- Considering how sparse mixture-of-experts (MoE) models have become very popular, I would love to see investigations in how the findings of this paper transfers to MoE models. I suggest repeating some of the experiments with models such as MoE variants of the Qwen3 series, GPT-OSS, GLM4.5, etc.\n- While the top-$p$ sparsification strategy is reasonable, I would like to see a comparison of it against an even simpler, top-$k$ sparsification strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)