Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
Authors: Yuqi Luo
Chenyang Song
Xu Han
Yingfa Chen
Chaojun Xiao
Xiaojun Meng
Liqun Deng
Jiansheng Wei
Zhiyuan Liu
Maosong Sun
Venue: ICML 2025 poster (2025)
Keywords: activation sparsity
large language model
Primary area: deep_learning->large_language_models
TL;DR: Comprensive studies on the measurement, influential factors, better architectural design and training practice for activation sparsity.

Abstract: Activation sparsity denotes the existence of substantial weakly-contributed neurons within feed-forward networks of large language models (LLMs), providing wide potential benefits such as computation acceleration. However, existing works lack thorough quantitative studies on this useful property, in terms of both its measurement and influential factors. In this paper, we address three underexplored research questions: (1) How can activation sparsity be measured more accurately? (2) How is activation sparsity affected by the model architecture and training process? (3) How can we build a more sparsely activated and efficient LLM? Specifically, we develop a generalizable and performance-friendly metric, named CETT-PPL-1\%, to measure activation sparsity. Based on CETT-PPL-1\%, we quantitatively study the influence of various factors and observe several important phenomena, such as the convergent power-law relationship between sparsity and training data amount, the higher competence of ReLU activation than mainstream SiLU activation, the potential sparsity merit of a small width-depth ratio, and the scale insensitivity of activation sparsity. Finally, we provide implications for building sparse and effective LLMs, and demonstrate the reliability of our findings by training a 2.4B model with a sparsity ratio of 93.52\%, showing 4.1$\times$ speedup compared with its dense version. The codes and checkpoints are available at https://github.com/thunlp/SparsingLaw/.

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
Yugi Luo"! "I
Chenyang Song" Xu Han'
Yingfa Chen Chaojun Xiao
Xiaojun Meng? Liqun Deng
Jiansheng Wei Zhiyuan Liu Maosong Sun'
[Luo-yg23, scy22 eewallemalllsl.....e..aaaaaaalle wtw.eetssiinuuuuteee edu, en,
{han-xi xu, liuzy) e singhua www.gghuaueee.e..eoo edu.cn
Abstract
activation sparsity
activation
strangly contributed
W dole
weakly contribute neurons
Activation sparsity denotes the existence of sub- stantial weakly-contributed neurons within feed- forward networks of large language models (LLMs), providing wide potential benefits such as computation acceleration. However, existing works lack thorough quantitative studies on this useful property, in terms of both its measurement and influential factors. In this paper, address three underexplored research questions: (1) How can activation sparsity be measured more accu- rately? (2) How is activation sparsity affected by the model architecture and training process? (3) How can we build a more sparsely activated and efficient L.M? Specifically, we develop gener- alizable and performance--frendll metric, named CETT-PPL-1%% measure activation sparsity. Based on CETT-PPL- %, we quantitatively study the influence of various factors and observe sev- eral important phenomena, such as the convergent power-law relationship between sparsity and train- ing data amount, the higher competence of ReL.U activation than mainstream SiLU activation, the potential sparsity merit of small width-depth ratio, and the scale insensitivity of activation spar- sity. Finally, we provide implications for building sparse and effective LLMs, and demonstrate the reliability of our findings by training a 2.4B model with sparsity ratio of 93.52%, showing 4.1 speedup compared with its dense version. The codes and checkpoints are available at https: //github.e com/ http//hhnlp///// hauunpp/Spariiggiaa.
matrix muttipication
alement multplication
activation function
FFNI))
input hidden state (one token)
W
W
Figure 1: A typical case of activation sparsity (with spar- sity ratio of 60%) in gated feed-forward network of LLMs.
1. Introduction
Activation sparsity refers to the phenomenon that consider- able elements within the output of activation layers (shown in Figure I) are zero or low values, and thus the correspond- ing neurons' contribute weakly to the final model output given a specific input. a prevalent property of many language and vision modeling architectures (Li et al., 2022), activation sparsity has wide practical values, such in- ference acceleration (Liu et al., 2023; Song et al., 2023), training acceleration (Zhang et al., 2024b), and LLM inter- pretation (Sajjad et al., 2022). For instance, Xue et al. (2024) achieves up to 27.8x inference acceleration on smartphones by utilizing activation sparsity, while Zhang et al. (2023)
reveals and interprets the emergent modularity LLMs by analyzing the specialization of sparsely-activated neurons.
Despite the extensive benefits of activation sparsity, there are few relevant comprehensive and quantitative studies. To fill this gap, we focus on three research questions:
'Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China Huawei Noah's Ark Lab, China. Correspondence to: Xu Han <han-xu @wsiinnuu.ccu..nn Zhiyuan Liu <liuzy @ tsinghua.edu.cn.. edd.
'Equal contribution
(Q1) How can activation sparsity be measured more accurately?
(Q2) How is activation sparsity affected by the model architecture and training process?
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
A neuron denotes certain column within the parameter matrices feed-forward networks (FFN).

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
(Q3) How can we build more sparsely activated and efficient LLM?
4. Given similar width-depth ratios, the limit of activation sparsity is weakly correlated the scale of L.Mss, while the convergence speed to the limit is much faster in smaller models. (Figure 7) We try to explain these phenomena in Section 5.3, indicating a similar neuron specialization pattern across models of distinct scales.
Metric for Activation Sparsity A "metric" for activa- tion is for measuring the sparsity ratio (i.c., the ratio of weakly-contributed neurons) of an LLM given specific in- puts. A good metric should be generalizable (i.e., applica- ble different activation functions), and its impact the model performance (after the neurons recognized weakly- contributed ones are skipped during calculation) should be negligible. To achieve this, we need an accurate algorithm to identify weakly-contribuue
Approach to More Sparsely-activated LLMs Based on the above findings, we answer the third question: To train LLM with greater activation sparsity from scratch, it is better to adopt the ReLU activation, use small width- depth ratio on the premise of stable training, and feed more training data so that the decreasing logspace power-law can help reduce the activation ratio.
neurons in each layer. The more accurate this algorithm is, the less performance degradation a model will suffer under specific sparsity ratio.
We experimentally demonstrate that CETT (Zhang al., 2024a), generalizable method adaptively setting layer- wise thresholds for activation values to identify weakly- contributed neurons (Section 2.2), achieves the best trade- off between performance and sparsity ratio among existing methods. To further ensure negligible performance degrada- tion, we develop CETT-PPL-1% as better sparsity metric. Specifically, this metric binary searches the CETT hyper- parameter (controlling the output error of FFNs in each layer after a PPL increase tolerance of 1% that causes nearly no harm to performance on downstream tasks. In the 0.8B ReLU- activated model (Figure 2). reduces the activation ratio by 61% relatively compared with the common ReLU-oorented metric using a zero threshold.
To validate the generalizability of our findings, we train a larger 2.4B model. With ReLU activation, near 800B train- ing tokens, and width-depth ratio close to the O.1B- experimental models (which is small enough and ensures training stability). the model undergoes a similar logspace power-law trend between activation and data, achieving a high limit sparsity ratio of 93.52% and 4.1 x speedup ratio to the dense model. This value is also close to the limit sparsity ratios of 0.1B-1.2B models, consistent with the previously found scale insensitivity of the limit sparsity.
neurons are skipped) under
To sum up, our work provides a better practice of measuring activation sparsity and then comprehensively studies its influential factors and scaling properties. The empirical laws found above can provide instructional values for designing and pre- pre-training an LLM with greater activation sparsity, which helps produce more efficient LLMs.
Influential Factors of Activation Sparsity Using CETT- PPL-1% we systematically study the correlation between activation sparsity and four influential factors, including the amount of pre-training data, the activation function, the width-depth ratio (i.e., the ratio of the hidden dimension to the layer number), and the parameter scale. Through comprehensive experiments models scaled from O.1B to 1 1.2B, we obtain the following observations:
2. Preliminaries and Related Works
2.1. Preliminaries of Activation Sparsity
Activation sparsity is a prevalent property existing in neural networks with activation layers, indicating the existence of considerable neurons, that correspond to zero or low activation values, having a limited impact on final network outputs given specific inputs. Mainstream LLMS present remarkable activation sparsity (Li et al., 2022; Zhang et al., 2022; Song et al., 2025; 2024).
1. There is an increasing power-law (mainstream SiLU- activated LLMs) or decreasing logspace power-law (Ret.v--ct LLM relationship between the ac- tivation ratio (1 sparsity ratio) and the amount of pre-training data. Both laws are convergent with a certain limit sparsity ratio as the amount of data ap- proaches infinity. Rel.U-activated LLMSs can be more sparsely activated with more data. (Figure 4)
Activation sparsity can help improve LLMs in many as- pects, such as efficiency and interpretability. Recent works manage to exploit activation sparsity for inference accelera- tion, mainly by saving the computation related to weakly- contributed parameters (Liu et al., 2023; Song et al., 2023; Xue et al., 2024). Zhang et al. (2024b) utilize the existence of activation sparsity throughout the majority of the L.M pre- training process to accelerate pre- pre-training. Besides accelera- tion, activation sparsity also helps improve interpretabilityy which is important for reliable and well-perfonming LLMs. Explanation of the sparse neuron activation patterns has
2. Given the same parameter scale, the sparsity ratio of ReLU-activated LLMs always surpasses that of SiL.U-activated LLMs, with comparable performance on downstream tasks. (Figure 4 and Table 1)
3. Given the same parameter scale, the activation ratio lin- carly increases with the width-depth ratio under a bot- tleneck (i.e., deeper models are sparser), above which activation fluctuates around a fixed level. (Figure 5)
2

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
long been a mainstream paradigm of interpreting LLM be- haviors (Sajjad al., 2022; Bills et al., 2023; Gao al., 2024; Lieberum et al., 2024).
The major drawback of this straightforward definition is the lack of generalizability. Concretely, it is unsuitable for activation functions with considerable non-negligible negative outputs, such as SiLU (Elfwing et al., 2018). In these cases, the straightforward metric can lose considerable negative neuron outputs and harm performance. A quick fix is to use the absolute value, D = {i||ss] < <). but a global threshold across layers is hard to determine. Zhang et al. (2024a) adaptively searches the layer-wise thresholds by introducing the cumulative errors of tail truncation (CETT), Defined as the L2 norm relative error caused by skipping weakly- conti--cotti neurons, CETT is computed as:
Notably, mnixture-of-expert (MoE) is popular paradigm of activation sparsity, enforeing each token to activate a limited number of experts. As such constraints can sacrifice flexibility and performance (see Appendix B), we focus on activation sparsity in vanilla Transformers in this work.
Besides, activation sparsity also has fundamental differences from weight pruning, which realizes acceleration by remov- ing certain parts of LLM parameters regardless of inputs. By contrast, activation sparsity adaptively determines important neurons for different inputs with no parameter permanently removed, causing considerably less performance degrada- tion (see Appendix C).
|1Eien milla CETT |[FN(X)]22
D= filllell < e).
(3)
where |12 is the L2 norm operator. As CETT increases monotonically with e, for each layer, we can use binary search to find the threshold e to meet the CETT value.
2.2. Metrics of Activation Sparsity
Building a satisfactory metric for measuring sparsity a nontrivial work. For the convenience of demonstrations, we formally introduce the following notations for the compu- tation process of FFNs (also see Figure 1). With hidden dimension dA and an intermediate dimension dy. a gated FFN (the FFN paradigm in mainstream LLMs (Dauphin et al., 2017; Shazeer, 2020) works as follows:
In addition to the above sparsity metries, we also note some special practices. For instance, CATS (Lee al., 2024) finds layer-wise adaptive thresholds but targets the same expected sparsity ratio at each layer. This is equivalent to the Top-k setting in Section 4. TEAL (Liu et al., 2024a) addresses a more general paradigm named input sparsity. This is largely different from output sparsity, which the paradigm handled in this work and mainly induced by acti- vation functions. We leave the quantitative studies input sparsity for future work.
o (Woatex) x), FFN Woul [s (W),,), (1)
where x E Rds, s E RR,, a, and @ denote the input hidden states, the activation values, the activation func- tion, and the element-wise multiplication, respectively. Waate, Win E xd and Wort E IR xd, are learn- able parameters. Next, we decompose the parameters of FFN along the dimension of y into y neurons. The output of the i-th neuron n, is calculated by
In this work, we experimentally demonstrate that CETT, when tuned to just make the validation PPI. increase by 1%, is a metric with great generalizability and the least performance degradation, even at a high sparsity ratio.
8 = o Lo (W x),
T2 Wo @ (Wi
2.3. Influential Factors of Activation Sparsity
dy
(2)
Despite the importance of activation sparsity, few works conduct comprehensive and quantitative studies on how it is affected by the model architecture and training process. Speculating that activation sparsity comes from the train- ing dynamic in the optimization process, Li et al. (2022) finds an increasing trend of sparsity with larger scale, depth, and width in T5 series (Raffel et al., 2020). Zhang et al. (2024a) dives into this problem from the aspect of activa- tion functions. Song et al. (2025) discovers that LLMs tend to be sparser on more formatted datasets such as codes and multiple choices. Other works have discussed the scal- ing properties of parameter-sparss models (Frantar et al., 2023), fine-grained MoE models (Krajewski et al., 2024), and sparse autoencoders (Gao et al., 2024). To the best of our knowledge, we present the first comprehensive quantita- tive study on the accurate measurement, influential factors, and training practice of activation sparsity for LLMss.
FFN(X)= = Σ n4.
1=1
where Wigate, Win. f.s+ Wond are the i-th row of W the i-th row of Woaae, and the i-th column of Wout, respectively. The FFN outputs equals the sum of all neuron outputs.
Activation sparsity is measured by the ratio of weakly- contributed neurons, namely [D]/dy, where D the in- dex set of weakly-contributed neurons. Different metrics mainly differ in determining whether a specific neuron con- tributes weakly. A straightforward metric, naturally adopted in Rel.U-activated models, regards neurons with zero activa- tion values as weakly-contributee namely D = {i|s 0}. To find non-zero weakly contributed activations, Kurtz et al. (2020) and Mirzadeh et al. (2023) introduce a positive threshold or bias, i..., D = (ils; < c), 6 <> <>0. 0.
3

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
0.18 ReLU
0.1B SiLU
0.8B ReLU
0.8B SiLU
20
20
35
GETT
Top-k
Perplexity
300
FAT-e
16
16
Stragghforward
RaLL
25
14
14
Strughtlowwrrd ReLU
12
Straightoowarrd ReLU
10
10
10
16
100
Activation Ratio (%)
Activation Ratio (%)
Activation Ratio (%)
Activation Ratio (%)
Figure 2: The PPL-activation ratio (i.e., 1 sparsity ratio) Pareto curve using different methods recognize weakly- contributed neurons. "Straightforward ReLU" is only applicable Rel.U-activated models using the zero threshold
0.18 ReLU on C.R.
0.1B SiLU C.R.
0. 0.1B ReLU on R.C.
0.1B SiLU R.C.
40
40
Evaluation Score
55
Staighttoward
55
Stra ghalonward ReLU
50
45
CETT
Top-k
40
FAT-
20
20
Strw ghiforwwrd Rel
35
35
90
96
100
75
100
80
95
100
100
Sparsity Ratio (%)
Sparsity Ratio (%)
Sparsity Ratio (%)
Sparsity Ratio (%)
Figure 3: The Pareto curves between downstream task performance and activation sparsity using different methods to recognize ceakly-contributed neurons. C.R. refers to commonsense reasoning and R.C. refers to reading comprehension.
3. Quantitative Study Settings
lated on models that only complete pre-training, and the task-specific performance is evaluated on checkpoints after the decay stage. See Appendix H, I, and K for more details about datasets and training settings.
To address the three issues in the Introduction section, we conduct extensive experiments, training, evaluating, and an- alyzing the models ranging from 0. to 1. I.2B. We introduce our experimental settings in detail as follows.
4. How can activation sparsity be measured more accurately?
Model settings We adopt the architecture of μP Trans- former (Hu al., 2024), which combines the mainstream LLaMA (Touvron et al., 2023) with gated FFN and pP parametrization (Yang et al., 2022) for training stability.
4.1. Recognition of CeattyyContributed Neurons
As stated above, the crucial part of metric for activation sparsity is the accurate recognition of weakly-contributed neurons. In this section, we compare the following four methods to determine weakly-contribute neurons in FFNs (Refer to Eq, (2) for the definition of neurons):
Training settings We mainly explore the activation spar- sity of foundation models, which only undergo the pre- training stage. However, before we evaluate models on task-specific benchmarks, we follow recent LLMs (Dubey al., 2024; Hu et al., 2024) conduct decay stage, where instruction-tuning data is added. Thereby, we can obtain more reasonable results on benchmarks. We also follow the optimal batch sizes and learning rates in recent LLMs.
(1) Straightforward ReLU is the most simple but com- monly used setting, which uses a zero threshold and is only applicable to ReLU: D
{i]s; 0}.
(2) Top-k, widely adopted in the MoE architectures (Fedus et al., 2022), enforces each layer to consistently maintain h activated neurons, whose absolute activation values rank in the top-k ones among all the neurons of that layer. Obvi- ously, we have [D] dy - k. and the Top-k sparsity method holds constant sparsity ratio across all layers.
Evaluation settings We introduce a tiny validation dataset and two groups of benchmarks for evaluation, including commonsense reasoning (C.R.) and reading comprehension (R.C.). For the measurement of sparsity, to eliminate the im- pact of stochastic factors (especially the sparsity fluctuations during the early stage), we employ a sparsity stabilizing strategy (see Appendix E).
(3) FAT-c (Kurtz al., 2020) (FAT denotes forced activation threshold) similarly introduces global hyper-parameter as the threshold shared by all layers, namely D = {ills;] < e). Note that this is slightly different from the original
If there are no special statements, the training loss, valida- tion loss, and perplexity (on validation data) are all calcu-

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
Table I: The average evaluation scores (%) on two task groups. The second column represents settings with different PPL increase tolerances p%,, with "Dense" indicating the most accurate case where p = The marker means the score difference the corresponding dense setting.
0.IB
0.2B
0,4B
0.8B
1.2B
Avg.
ReLU
SiLU
ReLU
SiLU
ReLU
SiLU
ReLU
SiLU
ReLU
SiLU
Dense
49.6
49.5
52.0
52.2
54.7
55.8
56.8
57.6
60.0
59.6
54.78
CETT-PPL--1% %
0.5
+0.4
-0.
40.2
-0.1
0.9
0
0.4
0
0.16
C.R.
CETT-PPL-5%
-0.5
0.3
-02
-0.4
0.5
0.7
0.50
CETT-PPL-10%
0.2
-0.8
0.4
0.3
+0.2
-0.6
0.2
1.2
-0.7
-0.3
0.45
Dense
27.7
40.7
40.2
44.0
44.8
43.3
54.8
41.87
CETT-PPL-11 %
+0.2
+0.3
-1.0
-0.6
-1.1
-0,9
- 1.6
+1.0
+0.1
+0.6
0.30
R.C.
A CETT-PPL-5%
1.3
1.2
-21
33,4
3322
3,66
2.6
2.6
+0.1
-22
2221
A CETT-PPL-I0%
2.0
2.9
-2.
-5.8
4.1
6.5
4.5
4.5
0.3
3.7
-3.64
RelU--Aciieeed Models
SiLU-Activated Models
40
Activation Ralio (%)
9
36
0.18
0.28
0.4B
0.88
1.28
32
Number Tokens Passed (B)
Number of Tokens Passed (B)
Figure 4: The trend of activation ratios of models with different parameter numbers and activation functions during the pre-training stage. The fitted curves are plotted in brown.
FATReLU by Kurtz et al. (2020) as we compute the absolute values of activation values to accommodate SiLU.
find the search algorithm is described Appendix F.
that just meets the tolerance. The
(4) CETT (Zhang et al., 2024a), as introduced in Section 2.2, requires each layer share the same L2 norm relative error after cretiyy-ontrrbbueed neurons are skipped in calculation, while the layer-wise activation thresholds adaptively searched accordingly.
find a proper p% tolerance, we inspect impact on downstream task performance. As shown in Table with PPL increasing (intrinsically promoting greater sparsity), the reading comprehension performance is considerably impaired, corresponding to the trade-off between sparsity and performance. Notably, in both task groups, the aver- age performance "CETT-PPL-1% is comparable to that of the theoretically most accurate "Dense" setting. There- fore, we assume CETT-PPL--1 sparsity as a reliable performance-frieut metric and employ it compute sparsity in the following discussions.
As demonstrated by Figure 2 and Figure 3, CETT obtains the best trade-off between sparsity and performance. Un- der the same target sparsity ratio, CETT can always achieve the lowest PPL, and comparable or higher evaluation score downstream tasks. These indicate the highest accuracy of CETT in recognizing weakly-come ceatly-contributed neurons.
5. How is activation sparsity affected by the model architecture and training process?
4.2. Introduction of PPL Increase Tolerance
Despite the advantages of CETT, remains problem how to choose the best parameer, namely the shared L2 norm relative error. Therefore, we introduce CETT-PPL-p%% which denotes the sparsity ratio measured by CETT when the PPL on validation data rises by p% compared with the dense setting (i.e., with all neurons activated). Given a PPI. increase tolerance of p%, we can conduct binary search to
5.1. Amount of Pre- Fre-Trainig Data and Choice of Activation Functions
To obtain the scaling relationship between the activation sparsity and the amount of pre-training data, we pre-train models with different parameter numbers and two activation functions (i.e., ReLU and SiLU), respectively, and then
5

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
3.50
Limit of Aclivation Ralio (%6
8.0
Limit Training Loss
3,46
7.0
8.5
3.42
8.0
3
200
200
300
830
Width-Depth Ratio
Width-Depth Ratio
Figure 5: Limits activation ratios on O.1B ReLU models.
Figure 6: Limit training loss on 0. ReLU models.
Limit of Activation Ratio (%)
SiLL--Adivated Models
Derivative of Sparsity-Dalas Curve
0.015
0.020
0.28
0.1B
50
ReLU-Activted Models
40
0.005
ReLU
30
0.000
SIUU
-0.000
10
-0.01
0.2
0.8
Number of Parameters (B)
Training Tokens to Parameters
Figure The limits activation ratios on models with different scales and activation functions.
Figure 8: The derivative trends of the sparsity-data curve with the increase of data-scale ratio.
evaluate the sparsity level of their checkpoints using CETT- PPL-1%. After careful attempts, we find that the curve of activation ratios to the amount of pre-training data is easier to fit than that sparsity ratios. Therefore, we will frequently use activation ratios instead of sparsity ratios in the following sections, whose trend is plotted in Figure 4.
sparsity achieved by ReLU is significantly greater than that of SiLU. Besides, the task-specific performance in Table I and the trend of training loss in Appendix reveal the com- parable performance between ReLU and SiLU activations. Based on the above observations, ReLU more compe- tent the activation function than SiLU due to three ad advantages: increasing trend of sparsity, significantly higher sparsity ratio, and comparable performance.
For ReLU models, we observe logspace power-law re- lationship between the activation ratio A netu ()) and the amount of pre-training data D, expressed as follows:
5.2. Width-Depth Ratio
A (D) exp(-cDa b) Ag
(4)
The width-depth ratio, defined as the ratio of the hidden dimension to the layer number, reflects the shape of Trans- former and is a key architectural property that potentially influences activation sparsity. To inspect its influence on the activation sparsity, we conduct experiments on the IB ReL -eatiiciietted model and select different width-depth ratios. The limits of the activation ratio and the training loss are plotted in Figure and Figure 6, respectively.
where Ag > O the limit of activation ratio with infinite training data and c, a > O. This a convergent decreasing function, indicating that more training data can poten- tially make ReLU models more aparsely-
By contrast, the activation ratio Asanu (D) of SiLU models exhibit a vanilla plw relationship:
ASiLU (D) = Do + Ao:
(5)
As demonstrated by Figure 5, under bottleneck point (about 114 for 0.1B), the activation ratio linearly in- creases with the width-depth ratio. After this bottleneck, the activation ratio fluctuates around 8%. From the spar- sity aspect, a smaller width-depth ratio more helpful. However, Figure 6 demonstrates that an extremely small width-depth ratio causes significant performance degrada- tion. This is attributed to the training instability of deep
where similarly, Ao > is the limit of activation ratio and O. Note that this is convergent increasing function, and thus more training data will impair the activation sparsity of SiLU models. See Appendix G for the algorithm of curve fitting and the results (i.c., coefficients).
As for the selection of activation functions, by comparing the sparsity dynamics, we can conclude that the activation
6

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
Code
Wikipedia
Math
Chiniee
Actoatoon Frequanty (S3)
100
100
100
100
0.18
602
80
0.28
0.80
1.28
Naurons (%)
Naurons (%)
Raito Neurons (%)
Ritto Nourons (N)
Figure The distribution of the neuron activation frequencies on four datasets within models of distinct scales.
20
20
20
20
0.25 ReLU
15
15
0.88 ReLU
15
1.28 RaLU
15
10
10
10
10
0.48
5
20
20
20
0.1B ReLU
0.1B ReLU
0.1B ReLU
0.18 ReLU
0.28 SILU
75
0.4B SILU
75
0.88 SIUU
75
1.28 SILU
75
25
25
25
25
0
25
50
100
0
100
75
100
50
75
100
0.18
SILU
0.18 SiLU
0.18 SILU
0.18 SILU
Figure 10:
The activation ratio (%) distributions of 71,549 tokens sampled from the vocabulary. We conduct a pair- pair-wise wise comparison of the average activation ratio of each token within models of different scales. The red line the y = x curve.
Transformers (Petty et al., 2023; Wu & Tang, 2024). There- fore, the best width-depth ratio should fall on the smallest point of the interval that ensures stable training and the best performance (e.g., from 74 to 282 for O.1B).
smaller models tend to converge faster than larger mod- els to the limit, as the absolute values of their derivatives are much larger. We try to explain the above observations.
Observation: Neurons within models of different scales present similar activation patterns. To support this point, conduct two experiments from the aspect of dataset-wise and token-wise activation patterns respectively.
5.3. Parameter Scale
To obtain comprehensive scaling properties of activation sparsity with the increase of seales (i.e., the number of non- embedding parameters), we obtain the limit of activation ratio on the above pre-trained models with distinet scales but similar width-depth ratios. From the results plotted reach the first observation that under similar width-depth ratios, the limit of activation ratio, as the amount of pre-training data approaches infinity, is weakly related to the parameter scale. For SiLU settings, the activation ratio decreases slightly by 2.7 points from 0.11 1B 1.2B. By contrast, for ReLU settings, the activation ratio marginally increases by 1.7 points from O.IB 1.2B.
To inspect the dataset-wise activation distribution, we con- sider four subsets of the pre-training data and compute the distribution of activation frequencies (i.e., the times that a neuron is activated divided by the total number of tokens) among the neurons within models of different scales. As demonstrated by Figure 9, for all datasets, the distribution patterns of neuron activation frequencies are similar across different scales. While this observation holds average, special cases exist in certain layers (see Appendix J).
in Figure
For the token-wise activation, we sample 71,549 tokens from the vocabulary and count their activation ratios on a sufficiently large amount of data. Next, we compare the activation ratios of each token among models of different scales in pair-wise manner. Figure 10 clearly shows that
To reflect the evolving dynamics of sparsity, compute the derivatives of the aparsity-data curve as fitted in Section 5.1 and plot the trend of derivatives with the increase of data- scale r The results in Figure 8 clearly demonstrate that
works have demonstrated the roughly proportional relationship between the optimal amount of pre-training data and the parameter scale (Hoffmann et al., 2022: Besiroglu al., 2024).
"The data-scale ratio means the ratio of the number of training tokens to the parameter scale. We choose this variable previous

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
most tokens maintain close activation ratio across models of various scales.
2.48 Rell U-Adiivated Model
exp(-11.56 10 +) -DL9 -3.46) 1 6.48%
8.5
The above two experiments both support the insensitivity of the activation pattern to the parameter scale. This can potentially provide one explanation for why the activation sparsity is quite weakly correlated with the model sizes.
8.0
Activalion Ratio
0.5
0.0
Neurons within models of different scales present similar specialization. The neurons in FFNs tend to specialize into certain functions during the train- ing progress (Li et al., 2022; Zhang et al., 2023). However, few works have studied how such specialization differs in models of distinct scales. As stated above, both the dataset- wise and token-wise activation patterns are insensitive to the parameter scale. In other words, the numerical distribution of neurons activated for a certain function (e.g., specific category of datasets syntactic elements) is similar. There- fore, it is reasonable to assume that the specialization of neurons is also scale-insensitive.
Assumption:
7.0
6.5
600
1000
Number of Tokons Possad (B)
Figure 11: The activation-data trend of the 2.4B ReLU model, with the fitted logspace power-law presented.
To validate the reliability and generalizability of our find- ings, we train a 2.4B model based on the above experience. Equipped with ReLU activation, near 800B training data, and width-depth ratio close to the above models (also small enough and close to the smallest point of the training stability interval), this model achieves low limit activation ratio of 6.48%, quite close to those values of models from O.1B to 1.2B. Moreover, its activation-datt trend can also be well fitted with a decreasing logspace power-law, as shown in Figure 11. These are all consistent with previous findings.
Deduction: Smaller models converge faster to the limit of activation ratio mainly due to their small amount of neu- rons. To simplify this problem, we model the specialization of neurons as grouping process, where each neuron can be placed into zero more groups (considering the potential existence of dead neurons and versatile neurons). Suppose the dy neurons should specialize into G groups, each of them having t1, 122 ..., tG neurons respectively. Based on the assumption of similar activation patterns and neuron spe- cialization, the ratio of neurons placed in each group (i.e., 0 l//dg & 1, i 1,2, G) should be shared across different parameter scales. We can obtain the number of all the possible grouping results T(dy) easily,
Moreover, demonstrate the practical acceleration value, we compare the decoding speed of the 2.4B model with PowerInfer (Song et al., 2023) and "Ilama.cpp" (Gerganov, 2023), respectively. While PowerInfer utilizes sparsity to save computation, "Ilama.cpp" only conducts dense FFN computation. Consequently, the former achieves 4. speedup compared with the latter, revealing the potential of activation sparsity for efficient LLMs. Refer to Appendix L for experiment details.
dy! 4
T(d,)
II C i=1
( 4.)!
7. Discussion
where Cj2, is the combinatorial number, the number of possi- bilities to select f; neurons from d, ones. Obviously, (ddy) grows in a factorial speed with dy, much faster than the linear function. For larger models, the number of neuron specialization possibilities is significantly greater, and thus more training expenses required to form stable neuron specialization and approach the limit of activation ratio.
In this section, we mainly discuss other potential values of this work, especially in terms of monitoring the model dynamics from new aspects.
Training-tine time predictable sparsity Our work enables the prediction of sparsity during the training stage. By fitting the power-law or logspace power-law between activation sparsity and the amount of pre-training data, model devel- opers can either predict the theoretical upper/lower bound sparsity of model evaluate its potential (c.g.. in inference acceleration), estimate the number of tokens required to achieve a desired sparsity ratio.
6. How can we build more sparsely activated and efficient LLM?
Based on the above findings, we finally come to the ap- proach to building an LLM with greater activation spar- sity: Use ReLU as the activation function with a larger amount of pre-training data, and a small width-depth ratio within the interval ensuring the training stability.
Lens for the convergence of neuron specialization Gen- erally, the loss curve is an important signal of the training state (e.g., at which point the model converges). However, if we compare the loss curve in Figure 13 and the sparsity
8

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
(activation ratio) curve in Figure 4, we will find that the convergence speed of activation sparsity is much slower than loss, indicating the ongoing of neuron specialization even when the loss changes little. Despite the wide recog- nition of the neuron specialization phenomenon (Li et al., 2022; Zhang et al., 2023), it is unclear when such special- ization converges and how to inspect this progress. Besides, the loss curve often not good inspector for convergence, especially for LLMs with trillion-level pre-training data. We believe that the trend of activation sparsity can pro- vide a new lens for inspecting the progress of neuron specialization and training convergence.
org/paff/204.10100
Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J., Wu, J., and Saunders, W. Language models can explain neurons in language models. URL https://openaiiubbicc blob. core. windows. nel/henroneexpiiner/ppprrrriiiie himl. accessed: 14.05. 2023), 2. 2023.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020. URL https://ojs. wtt.://os.aaaaaaaaaoooo org/ index. php/ htt.a//aticceeeee///////////////
8. Conclusion
Brown, T., Mann, B., Ryder, N., Subbiah, M., Ka- plan, J. D., Dhariwal, P. Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language mod- els are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. URL cc /cc/s/ppet_iillee/ppppr///////// 05577006bfaa44077111110042222233 pdf.
In this work, we address three important research questions about activation sparsity. First, prove CETT-PPL-190 % as good sparsity metric with the most accurate recognition of weakly-contributed neurons and negligible performance degradation. Next, quantitative studies are conducted on the influential factors of activation sparsity. Finally, towards a sparser LLM,, our findings substantiate the advantage of ReLU activation, more training data, and a small width- depth ratio. We also observe and explain the scale insensi- tivity of sparsity. These can better instruct LLM developers to build sparser LLMs and leverage their merits.
Chen, M., Tworek, J., Jun, H., Yuan, Q. Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03337, 2021. URL attps://arxiv.org/
Impact Statement
pdf.
This paper presents work whose goal advance the field of large language models and address underexplored issues of activation sparsity. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. For more rigorous studies, we also discuss the limitations of our work in Appendix A.
Clark, C.,, Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2924-2936, 2019. URL https: / //aclanthology . wwt//19-g1/0000 pdf..
Acknowledgments
This work is supported by the high-quality development project of MIIT, the National Natural Science Foundation of China (No. 62236004), and a grant from the Guoqiang Institute, Tsinghua University.
Clark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., and Palomaki, J. TyDi QA: A benchmark for information seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:4544470, 2020. URL wwtpt:ttp//tt...tt pesttss:/aclaatttnn http///clllathhh/c aoglllhhhhooyyy......m.ooo ogy. org/ 2020. tacl- 1..30.pp.
siformationnseeking
References
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E.. Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07733 2021. URL https: //arxiv. wowlvvorg//doeo wwtpoo/gppp/.....//// 07732. pdf.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H.,
Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110014116: 2021. URL https: //arxiv. org/ pdt 22100. 68. 6f..
Besiroglu, T., Erdil, E., Barnett, M., and You, J. Chin- chilla scaling: A replication attempt. arXiv preprint arXiv:2404. 10102, 2024. URL https://arxii:
Colombo, P., Pires, T. P., Boudiaf, M., Culver, D., Melo, R., Corro, C., Martins, A. F., Esposito, F., Raposo, V. L., Mor- gado, S., et al. SaulL pioneering large language

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
model for law. arXiv preprimi arXiv:2403.03388 2024. URL https::/p//......... http:::/axxvv.oo.. 0338000... 03883.
arXiv:2406.04093, 2024.
URL https://arxiy:
org/pdff2406.0er 04093.
Dauphin, Y. N.N. Fan, A., Auli, M., and Grang- ier, D. Language modeling with gated convo- lutional networks. In International Conference on Machine Learning, pp. 933-941. PMLR, 2017. mmlr.
Gerganov, G. llama.cpp. httpp://github.ce
Everganov/llame cpp, 2023.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M. Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint wrXiv2200.033000 2020. URL https: aarxiv.. ozq/pdf/2003/ 0ww.0.3330..... .a5500.pp.... pdf.
URL URLhttts::/prroeeee wtt.ggs........////...... press/v70/ pd...
Ding, N., Chen, Y. Xu, B., Qin, Y., Zheng, Z., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language models by scaling high-quality instructional conversa- tions. arXiv preprint arXiv:2305. 14233, 2023. URL https:
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford. E.. de Las Casas, D., Hendricks, I.. A., Welbl, J., Clark, A., et al. Training compute- optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pp. 30016-30030, 2022. URL 22tp:f/222...2
/ttp/aaxvvv////////////e////aaaa org/ pdf////p///ee 2305 14233.223. pdf..
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., al. The Llama herd of models. arXiv preprint arXiv:2407.21783 2024. URL https:// arxiv. mrxiv.org/pei pptp//pff/20/./// 21783.
petps://arxiv.ore
Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y.. Huang, Y., Zhao, W., et al. MiniCPM: Unveiling the potential of small language models with scalable train- ing strategies. arXiv preprint vrXiv:2404.06395 2024.
Elfwing, S., Uchibe, E., and Doya, K. Seggoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018. URL URLhetps: /wwww/wwwww//oo///w/w http///wwwwcccceeerettttttteteet com/ sci wwwpee/eeeeeeeeeeeieeiieeeeeeeee
Huang, Q., An, Z.,, Zhuang, N., Tao, M., Zhang, C.,, Jin, Y., Xu, K., Chen, L., Huang, S., and Feng, Y. Harder tasks need more experts: Dynamic routing in MoE models. arXiv preprint arXiv:2403.076622 2024. URL https: //arxiv. org http:////4.0066/66
Fedus, W., Zoph, B., and Shazeer, N. Switch Transformers: Scaling to trillion parameter models with simple and effi- cient sparsity. Journal of Machine Learning Research, 23 (120);1-39, 2022. URL https: /ww.. jmlr. /all..rg/////.. papers/volume22 - 0990099989122 011///221999.9.0 .df.
Kocetkov, D., Li, R., Jia, L., Mou, C., Jernite, Y., Mitchell, M., Ferrandis, C. M., Hughes, S., Wolf, T., Bahdanau, D., et al. The Stack: 3 TB of permissively licensed source code. Transactions on Machine Learning Re- search, 2022. URL https: www.review..eee.eeeeeeee pdF?id@pxpbTTUER..
Frantar, E. and Alistarh, D. SParseGPT: Massive language models can be accurately pruned in one- shot. In International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023. URL petps::/proceedinng mlr press/v202/ Frantar23a/ httpa///raaatataaaaaaaaa pdf.
Krajewski, J., Ludziejewski, J., Adamezewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebiera, K., Król, K., Odrzygóżdź, T., Sankowski, P., et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.0788 077.7711, 2024. URL bttps://arxit org/ peg/pdff/400/
Frantar, E., Ruiz, C. R., Houlsby, N., Alistarh, D., and Evci, U. Scaling laws for sparsely-connected foun- dation models. In The Twelfth International Confer- ence on Learning Representations, 2023. URL https: //openreview wwt.:/eett/ddeee
Kurtz, M., Kopinsky, J., Gelashvili, R., Matveev, A., Carr, J., Goin, M., Leiserson, W., Moore, S., Shavit, N., and Alistarh, D. Inducing and exploiting activation sparsity for fast inference on deep neural networks. In Interna- tional Conference on Machine Learning, pp. 5533-5543. PMLR, 2020. URL https: press/v119 htsss//ll///crttt////rtttttt//
Gao, L., Biderman, S., Black, S., Golding, L.. Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The Pile: An 800GB dataset of di- verse text for language modeling. arXiv preprint 10Xiv22101.00077 2020. URL https: //rsiv./ org/pdf/21C1 00027. pdf.
.mlr. //tpc/eeiccccceeesso/eoee AtRp:tttss//p
Lee, J.-Y., D., Zhang, G., Tiwari, M., and Mirhoseini, A. CATS: Contextually-awann thresholding for sparsity in large language models. CoRR, 2024. URL http: / //arxivvorrg wwr.xx.org//d//2000000 08763.
Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scal- ing and evaluating sparse autoencoders. arXiv preprint
10

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
Li, R., Allal, L.. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. StarCoder: may the source be with you! arXiv preprint arXivv2305.06611, 2023, URL https: www.xv.orggppoaoo /2305. 061661611 .dd..
Petty, J., van Steenkiste, S., Dasgupta, I., Sha, F,, Gar- rette, D., and Linzen, T. The impact of depth and width transformer language model generalization. arXiv preprint arXiv:2310.19955 2023. URL https: / //arxiv. org/ wttpr////pf//ee/eoo/ pttpp//f......//////// 19956.
Li, Z., You, C., Bhojanapalli, S., Li, D., Rawat, A. S., Reddi, S. J., Ye, K., Chern, F., Yu, F., Guo, R., et al. The lazy neuron phenomenon: On emergence activation sparsity in Transformers. In The Eleventh International Confer- ence on Learning Representations, 2022. URL https: //openreview net.riewww.eeeeeeeeee httpi/ddffiii TU2nxciickT
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W. and Liu, J. Exploring the limits of transfer learning with unified text-to-text Transformer. Journal of machine learning research, 21 (140):1-67, 2020. URL https:ttppc........t....ttp / /www. jm1 malr.org. papers wwwpssavvolmmee111
074/20 074 74. pdf.
Lieberum, T., Rajamanoharan, S., Conmy, A., Smith, L.,, Sonnerat, N., Varma, V., Kramár, J., Dragan, A., Shah, R., and Nanda, N. Gemma Scope: Open sparse au- toencoders everywhere all at once on Gemma 2. arXiv preprint arXiv:2408.0m 05 1477, 2024. URL https://
Roemmele, M., Bejan, C. A., and Gordon, A. S. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 20/ AAAI Spring Symposium Series, 2011. URL https:/ // /cdn/nnnn//////// waaai.orre. wwwg//s////ooooooooooo/o////// /t///cc//22 2oss/4188 2418/ 2418-1087 8- 1-0--1-PPP 1-P...-P.......... pdf.
arxiv. wwwiv.org/por wwt.:o/g//ppppt..... 2408 05147.
Sajjad, H., Durrani, N., and Dalvi, F. Neuron-level interpre- tation of deep NLP models: A survey. Transactions of the Association for Computational Linguistics, 10:1285- 1303, 2022. URL https: //tpt::://drrrtt...ptet wtw.eeetttt.ii.te..... tacl becl/artiileepddf/odolt http://.1....///aaeeeee 00519/2060745
Liu, J., Ponnusamy, P., Cai, T., Guo, H., Kim, Y., and Athi- waratkun, B. Training-free freee activation sparsity in large lan- guage models. arXiv preprint arXiv:2408 11088.444900 14690, 2024a. URL https::///// /awxivvv.v.///////////aa////aava org/ p2f/2408. 14690,
Liu, L., Kim, Y. J., Wang, S., Liang, C., Shen, Y., Cheng, H., Liu, X., Tanaka, M., Wu, X., Hu, W., Chaudhary, V., Lin, Z., Zhang, C., Xue, J., Awadalla, H., Gao, J., and Chen, W. GRIN: GRadient- MoE, 2024b. URL https:/ /arxivv. org/ abs/ 220224099 12136.
aaea__00119_11 pdf.
Sakaguchi, K., Le Bras, R., Bhagavatula, C., and Choi, Y. WinoGrande; An adversarial winograd schema chal- lenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8732-8740, 2020. URL https: aaai /ttgg/oss// 6399/ 1399-13-9624-111--2200000
Liu, Z.. Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Re, C., et al. Deja Vu: Contextual sparsity for efficient LLMs at inference time. In International Confer- ence on Machine Learning, pp. 22137-22176. PMLR, 2023. htRhttts:://// v202 httlllll2aammmmmm wtt.//mmmmmmmmmmmmmmamae 11m.m......m..mm.mmmmmmmmm. pdf..
pdf.
Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. SocialIQA: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process- ing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP) pp. 4463- 4473, 2019. URL https: /tt.tts::// http:////ccaatthhlogg.. org/ D19-1454, ..44.po. pdf..
Marquardt, D. W. An algorithm for least-squares estimation of nonlinear parameters. Journal of the society for Indus- trial and Applied Mathematics, 11(2))431 441, 1963.
Mirzadeh, I. Alizadeh, K., Mehta, S., Del Mundo, C. C., Tuzel, O., Samei, G., Rastegari, M., and Fara- jtabar, M. ReLU strikes back: Exploiting activation sparsity in large language models. arXiv preprint arXiv:2310.04564, 2023. URL https://arxiv org/pdf/2310 04564. .df.
Shazeer, N. GLU variants improve Transformer. arXiv preprint arXiv:2002.052003 2020. URL https:// arxiv.org/pdde 2ot.a///002ooe. 05202. pdf.
Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D.. Atkin son, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arX0w:2402.00155, 2024. URL https: //arxiv. http://p//222201155533
Paperno, D., Kruszewski, G., Lazaridou, A.. Pham, N.-Q.. Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word prediction requiring broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pp. 1525- 1534, 2016. URL https:://// /c//aalaaa aalaaccnnntoo www.annhologg..ooooo 1144 .pdf.
Song, C., Han, X., Zhang, Z., Hu, S., Shi, X., Li, K., Chen, C., Liu, Z,, Li, G., Yang, T., and Sun, M. ProSparse: Introducing and enhancing intrinsic activation sparsity within large language models. In

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
Proceedings of the 31st International Conference on Computational Linguistics, pp. 2626-2644, January 2025. URL https: www.aathhoogy.orggoooooo coling-main. 180. 180.pdf.
Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs V: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203 2022. URL https:/ /arxiv. org/pdf/2203.00
Song, Y., Mi, Z., Xie, H., and Chen, H. PowerInfer: Fast large language model serving with a consumer-grads GPU. arXiv preprint arXiv:2312.12456 2023. URL petps://arxiiee
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sen- tence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791- 4800, 2019. URL https: h/tps//ataaaatlooy...oo.o....//. org/ P19-1472 pdf.
22312. 12456.2455..
Song. Y. Xie, H., Zhang, Z., Wen, B., Ma, L., Mi, Z., and Chen, H. Turbo Sparse: Achieving LLM SOTA performance with minimal activated parameters. arXiv preprint arXiv:2406.05955 2024. URL https:// 05955
Zhang. Z. Lin, Y., Liu, Z., Li, P., Sun, M., and Zhou, J. MoEfication: Transformer feed-forward layers are mixtures of experts. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 877-890, 2022. URL URL https: /tass://alaaaaaa /aaaaaaaanthlllo celanthology...... wrny.org//022. f Findings-acl.. 71.ppf.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. Challenging big-bench tasks and whether chain-of-thougle can solve them. arXiv preprint urXiv:2210.09926: 2022. URL https:// arxiv. www.vorg//pce. http:/df//22 2ttp:22...........e.
Zhang. Z., Zeng, Z., Lin, Y., Xiao, C., Wang, X., Han, X., Liu, Z., Xie, R., Sun, M., and Zhou, Emergent modularity in pre-trained Transformers. In Findings of the Association for Computational Linguisties: ACL 2023, pp. 4066 4083, 2023. URL https: / / ac sclantholoog org/2023. for...inniiin fwnniinnnnnnanneee acl. 2500 pdf.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Ham- bro, E., Azhar, F., et al. LLaMA: Open and ef- ficient foundation language models. arXiv preprint (rXi::2302.13971, 2023. URL https://arxix org/pdf/2302.0m 2ww.222....1.....2 pdf.
Zhang, Z., Song, Y., Yu, G., Han, X., Lin, Y., Xiao, C., Song, C., Liu, Z., Mi, Z., and Sun, M. ReLU wins: Discovering efficient activation functions for sparse LLMs. arXiv preprint arXiv:2402.0300 arxiv. axxiv.org/pee httgg//df///00// 038.24422..0....22.
Wei, Y., Wang. Z., Liu, J., Ding, Y., and Zhang, L. Magi- coder: Empowering code generation with ISS-Instruct. In Forty-first International Conference on Machine Learn- ing. 2024. URL https: httpss///aavvv.e//e ortp / 2312. 02120.
2024a. URL https:// 03804.000.... .df.
Zhang, Z., Xiao, C., Qin, Q., Lin, Y., Zeng, Z., Han, X., Liu, Z., Xie, R., Sun, M., and Zhou, J. Exploring the benefit of activation sparsity in pre-training. In Forty- first International Conference on Machine Learning, bttps::/openrevieee wwtp://eeppeeeeeeeeeepeee. net net/pdf?
Wu, C. and Tang, R. Performance law of large language models. arXiv preprint wrwiv:2408....... 09895, 2024, URL attps::/aat /tts:////axvvv.///a http:/atwww.....dd///////a 0 09995.
2024b. URL https: 1d=KFxXPCoobh.
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. AGIEval: A human-centric benchmark for evaluating foundation mod- els. arXiv preprint arXiv:2304.0m 06364, 2023. URL 64 pdf....f.........p..............
Xia, M., Gao, T., Zeng. 7.,, and Chen, D. Sheared LLaMA: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694 2023. URL https:/ //apx//aavvv.///////ae/ea wttpiivoorr////.f//e//oooo 2211 06694...666...... .ddf.
https: www.xii.ogg/paae.
Xu, C., Sun, Q. Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. WizardLM: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023. URL https: //arxiv. org/ wttppdt//////ooooo httpppff/223000 4.0034440000 12244.
Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., and Fedus, W. ST-MoE: Designing stable and transferable sparse expert models. arXiv preprint 0xXiv:2202.08906 2022. URL https:// arxiv. arxiv.org/pdf /2w...///22202882
Xue, Z., Song, Y., Mi, Z., Chen, L., Xia, Y., and Chen, H. Powerlnfer-2: Fast large language model inference on a smartphone. arXiv preprint arXiv:2406.06220 2024. URL https:/ httpss://axxvv..mo Maraii.orrr http:p:////adiooo/////////0000 06682.
12

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
A. Limitations
PPL-Activation Pareto Curve
MoE (16 experts)
One drawback of our study is the absence of computation FLOPS) in some analyses, especially the experiments for width-depth ratios. In Section 5.2, we find a smaller width-depth ratio potentially produces sparser model. However, with the substantial increase in the number of layers, the training efficiency is significantly decreased, as we have observed in the training process. Therefore, in addi- tion to performance, the computation costs of a model also deserve consideration. Similarly, considering the values of activation sparsity in acceleration, it may be interesting to involve the inference computation variable in our study.
40.0
MoE (30 experts)
37.5
MoE (60 experts)
Vanilla (RaLU)
Perplexily
26.0
30.0
27.5
25.0
15
20
25
30
Activation Ratio (%, 1 Sparsity Ratio)
Figure 12: The PPL-activation Pareto curve of the O.IB MoB with different expert numbers versus the 0.IB vanilla decoder-only Transformer.
Another limitation of our CETT-PPL--% metric (as well as all the sparsity metrics relying on a validation dataset) is the sensitivity to different data distributions. Intuitively, the same model can have different sparsity ratios on dis- tinct datasets or tasks. The correlation between sparsity and influential factors (e.g., the form of power laws) can also have dataset-unique characteristics. A piece of evi- dence already presented in our paper is in Table I, where the performance on commonsense reasoning tasks is insen- sitive to y%6, largely different from the results on reading comprehension tasks. Moreover, the data mixing policies for pre-training can also have a considerable impact on the activation sparsity, which we leave for future work.
tion sparsity within vanilla decoder-only Transformer-based LLMs in this paper, such as GPT (Brown et al., 2020) and LLaMA (Touvron et al., 2023).
C. Activation Sparsity and Weight Pruning
Weight pruning, also a popular paradigm for inference ac- celeration, accelerates LLMs by removing certain elements from the model parameters (e.g., neurons, weights, or struc- tured blocks). Nevertheless, the sparsity introduced by weight pruning is fundamentally different from activation sparsity, as the pruning sparsity is completely static, namely independent of inputs. Specifically, weight pruning always drops the same part of parameters regardless of inputs. Con- sequently, enforcing high such static sparsity can easily cause considerable performance degradation (Frantar & Al- istarh, 2023; Xia et al., 2023).
B. Activation Sparsity and Mixture-of-Experts
Mixture-of-expere (MoE) is a mainstream method to achieve high activation sparsity by introducing constraints at the model architecture level. Typically, MoE uses a token- level top-k parameter selection router to assign a fixed spar- sity ratio for each token at each layer (Fedus et al., 2022: Zoph et al., 2022). However, these constraints often sacri- fice model flexibility and performance. Recent works reveal the potential performance degradation caused by such in- flexible sparsity assignment (Huang et al., 2024; Liu et al., 2024b). Moreover, to inspect the impact of such constraints, we plot the PPI.-activation (PPL denotes perplexity) Pareto curve of MoE in Figure 12 and compare it with vanilla decoder-only Transformer (Touvron et al., 2023) of the same parameter scale and amount of pre-training data?. MoE has a significantly worse performance-sparee sparsstty trade-off. More- over, the best sparsity ratio is also hard to predefine, since a too-high or too-low sparsity ratio may lead to more se- vere performance degradation or substantial unnecessary computation, respectively.
By contrast, activation sparsity dynamically determines neurons given specific input tokens, which potentially sacrifices less LLM capacity and perfor- mance. For example, Rett-activaeed models, with con- siderably higher activation sparsity, have comparable per- formance mainstream SiLU-activated ones. Moreover, weight pruning and activation sparsity can also be combined to further promote LLM efficiency.
D. Training Loss Dynamics
To present the comprehensive training dynamics, we plot the trend of loss with the increase of training data in Fig- ure 13. As can be clearly observed, larger models have smaller training loss. Besides, we also plot the limit values of the training loss with infinite training tokens, shown in Figure 14. As demonstrated in the above two figures, SiLU and ReLU models are well comparable from the loss aspect.
Therefore, to avoid negative impacts on flexibility and per- formance, in this work, we focus on the intrinsic activa-
"MoE models of different sparsity are obtained by tuning the number of activated experts, while for the vanilla setting, we adjust the CETT hyper- parametrr proposed by Zhang et al. (2024a).
13

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
0.1B
3.4
SIL SiLU-Adivated Models
3.75
0.2B
Training Loss
Accivaldd Models
Smoothed Training
0.4B
3.2
3.50
0.8B
1.28
3.25
ReLU
3.0
SILU
2.8
2.75
100
200
300
0.2
0.4
0.8
0.8
1.0
Number of Tokens Passed (B)
Number of Parameters (B)
Figure 13: The trend of pre-training loss for models with different scales and activations.
Figure 14: The limits of the training loss with the amount of pre-training data approaches infinity.
E. Sparsity Stabilizing Strategy
parameter p%,, we employ Algorithm 1 to find the CETT hyper-parameter that just makes the average PPL. these checkpoints on the validation dataset rise by exactly p%,, compared to the dense setting with all the neurons activated. The rationality of using binary search is substantiated by the monotonous relationship between PPL and CETT, as shown in Figure 15. Note that this algorithm can be applied to either a single checkpoint multiple checkpoints, as adopted in the strategy described in Appendix E.
We find that the stochastic factors in gradient descent during pre-training have significant impact on the metric of activa- tion sparsity. Especially, during the early training stage, the model is far from convergence with considerable sparsity fluctuations, and the magnitude- based sparsity metric can become unstable and cause problems to curve fitting.
To eliminate the influence of these unstable factors facil- itate smoother sparsity metric, first drop the sparsity points during the warmup stage for curve fitting. More- over, we mainly apply the CETT-PPL-p% on the last several checkpoints (specifically, the last five pre-trained check- points) as a whole, binary-searching the hyper-panne parameter value that controls the average PPL of these checkpoints to just increase by p%. Then this hyper-parameter is applied to all the checkpoints of this pre-training process to measure the sparsity ratio.
Algorithm 1 Find the CETT hyper- parameter for CETT- PPL-p% sparsity
Input: The input list of checkpoints C'pp List.
Input: The validation dataset VS.
Input: The PPL. increase tolerance p%.
Input: The error tolerance eps for binary search.
Output: The
CETT that just makes
the average PPL of LkstList on V rise by p%.
F. Binary Search Algorithm for CETT
+ 0, r f-
while r -1 eps do mid +- r2
100
0.18 ReLU
PP PPLRatioList +
for Ckpl. Ckkpl do
40
Activation Ratio
0.88 ReLU
lossåenae Lieesse (Ckpt, VS)
60
0.88 SLU
losssparse + L Lsparse (Ckpt, VS, cett = mid)
PPLRalio + exp(los8 sposs sporse lossdense)
20
PPL wtt.Rattoosss.oppendeee PPI.Rato))
end for
0.00
0.25
0.50
0.75
1.00
0.30
0.25
0.50
0.75
1.00
Mean PPLRatio +- Mean (PPLRatioList) if Mean < 1 p% then
CETT
CETT
l ( mid
Figure 15: Experiments show that both PPI. and the ac- tivation ratio change monotonously with CETT, the key hyper-parameter of CETT-PPL-y%. This provides ratio- nality for using a binary search to determine the CETT hyper-parameter.
else
r + mid
end if
end while
CETT +- (1+r)/2
return CETT
Given list of checkpoints, a validation dataset, and a hyper-
14

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
Table 2: Coefficients of activation-data (logspace) power-laws obtained from curve fitting. The curves of ReLU-iccivated and aittivctie models follow Eq. (4) and Eq. (5) respectively.
b
Au
O.IB
1.01
10
-1551
10
3.20 10
6.14 10 10-U2
0.2B
4.49 x 10 0z
-3.05 x 10
2.86 x 10-91
6.74 x 10
ReLU
0.4B
6.83
10-01
-3.46 x 10+00
7.90
10-02
6.90 10-02
0.8B
1.01 x 10+00
-3.49
10+000
7.97 10 10-03
7.21 10-02 02
1.2B
1.33 x 10
-3.89 x 10
9.03
10
7.82 10
2.4B
1.53 10
-3.46 10+00
1.56 10-04
6.48 10-02
0.1B
4.79 x 10-01 01
1.02 10
4.09 10
0.2B
8.44 x 10-01
2.08 x 10-01
3.90 x 10-01
SiLU
0.4B
1.03 10+00
4.20 10
3.85 10-01
0.8B
9.95 x 10 10-01
5.62 x 10
3.83 x 10
L.2B
9.67 10 10-01
5.38
10
3.82 10
Table 3: Hyper-parameters across various parameter scales.
Parameter Scale
0.1B
0.2B
0.4B
0.8B
1.2B
2.4B
# non-embedding parameters
1.08 x 10
2.41 x 10a
4.52 x 10a
7.60 x 10
1.18
10
2.44 x 109
batch size
3.27 x 10
5.90 x 10
7.86 x 10 1.18 10
1.57
10
2.10
10
G. Fitting Algorithm and Results
Evaluation benchmarks To evaluate the task-specific per- formance of models, we introduce the following two groups of benchmarks: (1) Commonsense reasoning: We compute the average O-shot accuracies on PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2020), and COPA (Roem- mele al., 201 1). (2) Reading comprehension: We report the average 0- accuracies BoolQ (Clark et al., 2019), LAMBADA (Paperno et al., 2016), and TyDi QA (Clark et al., 2020).
We employ the Levenberg- Gevenberg-Marquardtt method (Marquardt, 1963) fit the activation-data curves. To improve the stabil- ity of curve fitting, (i.e., the amount of pre-training data) by 10 to normalize its magnitude. All the results we obtained from fitting Eq. (4) (for Rea_tvvcte models) and Eq. (5) (for SiLU-acttvvtedd models) are shown in Table 2.
divide the number of tokens passed
H. Datasets and Benchmarks
We also evaluate our model on more complex tasks but fail to obtain performance above the random level. These include: the average pass @ scores on HumanEval (O-shot) (Chen et al., 2021) and MBPP (3-shot) (Austin et al., 2021), the average accuracies GSM8K (8-shot) (Cobbe et al., 2021), MMLU (5-shot) (Hendrycks et al., 2020), Big Bench Hard (BBH) (3-shot) (Suzgun et al.. 2022), and AGI-Eval (0- shot) (Zhong et al., 2023).
Training data The pre-training data is mixture of var- ious corpus, including a cleaned version of Common- Crawl, Dolma (Soldaini et al., 2024), C4 (Raffel et al., 2020), Pile (Gao et al., 2020), the Stack (Kocetkov et al., 2022), StarCoder (Li et al., 2023), and other collected raw corpus. In contrast, the decay data contains addi- tional instruction-tumn data, such UltraChat (Ding al., 2023), SlimOrca (Colombo et al., 2024), OssInstruct (Wei et al., 2024), Evollnstruct (Xu et al., 2023), and other col- lected datasets.
I. Detailed Training Settings
We utilize the μP Transformer (Hu et al., 2024) architec- ture and adopt its hyper-paramete policies, along with the WSD learning rate scheduling method. Across all parameter scales, the ratio of dy to d2 is equal to 2.5 consistently, the number of query heads always matches that of key and value heads, and the width-depth ratios range from 48 56, gen- erally similar across different scales. The specific number of parameters of various settings are shown in Table 3. We play the following pre- mre-training
Validation data To measure the CETT-PPL-p%% sparsity more precisely, we introduce a tiny validation dataset, which shares the same distribution as the pre- pre-training data. We con- duct deduplication to eliminate any intersections between validation and pre-training data.
across all
15

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
settings: peak learning rate Ir 0.01, 31 0.9, 82 0.95, weight decay = 0.1. The batch size depends the param- eter scale, presented in Table 3.
according the activation frequencies of different neu- rons and save redundant computation and time wasted on weakly-contributed neurons. By contrast, "Ilama.epp" does not utilize activation sparsity for acceleration, computing the FFNs in a dense manner.
J. Dataset-wise Activation Pattern
Both frameworks are compiled with CUDA enabled and run on the same machine with 104 CPUs and NVIDIA A800 GPU. Although "Ilama.cpp" does not support ReLU and thus cannot correctly conduct inference with our 2.4B model, this does not impact the acceleration experiment as the FLOPS remain the same as SiLU-activated model. We use 100 test prompts sampled from C4*, and each prompt is composed of 5 prefix tokens.
Although the overall distribution patterns of activation fre- quencies are similar in terms of the average scenario, they exhibit differences when focusing on neurons in specific lay- ers, such as the first, the last, or the exact middle layer. As shown in Figure 16, models with varying parameter scales have diverse neuron activation frequency distributions in the first layer and the last layer, while the patterns on the middle layer are still largely scale-insensitive.
Consequently, we found that PowerInfer can perform decod- ing at an average speed of 41.79 tokens per second, while "Ilama.cpp" can only reach 10.23 tokens per second. The 4. I x speedup of PowerInfer provides strong evidence of the acceleration potential offered by activation sparsity.
K. Performance on Independent Benchmarks
In Table 1, already provide the average performance on the two groups of commonsense reasoning and reading comprehension. In this section, we present the evaluation scores on independent benchmarks these two task groups, as shown Table and Table 5, respectively. From these tables, it can be observed that in commonsense reasoning benchmarks, as the number of model parameters increases from OB to 1.2B, the average evaluation score of the ReLU settings rises from 49.6 to 60.0, while the average score of the SiLU settings increases from 49.5 to 59.6. Similarly, in reading comprehension benchmarks, the score of ReLU settings goes from 28. 2 to 53.2, and the score of SiLU set- tings rises from 27.7 54.8. Additionally, models with these distinet activation functions demonstrate compa- rable performance the same parameter scale. Moreover, under the CETT-PPL-119 1% setting, the models are generally comparable to the dense setting with all neurons activated, whereas under the CETT-PPL-5% setting, they tend to suf- fer from significant performance on reading comprehension tasks, but the commonsense reasoning scores almost remain unaffected, which is a phenomenon worth studies.
We also evaluate our models several more complex tasks. However, due to the limited number of parameters, we are unable to obtain reliable results above the random level. The evaluation results for this part are shown in Table 6.
L. Details of Acceleration Experiments
To demonstrate the practical inference acceleration values of activation sparsity, run experiments with the 2.4B ReLU- activated model on two different acceleration frameworks: PowerInfer (Song et al., 2023) and "Ilama.cpp" (Gerganov, 2023). Specifically, PowerInfer, tailored for activation spar- sity, involves an offline profiler and online activation predic- tors forecast the activation pattern of each neuron. There- fore, PowerInfer can wisely allocate hardware resources
hncres//hoggingaae.c///aaass /lnaai/
16

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
The First Layer on Code
The First Layer
The Layer
The Layer
on Wikipedia
on Math
on Chinese
100
100
100
100
Aclivation Frequency
80
80
80
0.88
0.4B
0:88
1.88
40
40
40
SILU
20
20
20
25
75
100
50
100
0
50
75
100
25
75
100
The Exact Middle Layer
The Exact Middle Layer Wikipedia
The Exact Middle Layer Math
The Exact Middle Layer
100
Activation Frequeney (%)
80
80
60
80
80
60
40
0
0
25
100
100
The Last Layer on Code
The Last Layer Wikipedia
The Last Layer Math
The Last Layer
on Chinese
100
100
100
100
Activation Frequency
80
80
80
40
40
40
40
20
20
20
25
50
75
100
25
50
75
100
0
25
50
75
100
0
25
50
75
100
Neurons (%)
Neurans (%)
Ratio Neurons (%)
Ratio Neurons (%)
Figure 16: The distributions of average activation frequencies across three individual layers at different positions within models of distinct scales, including four datasets from the pre-training data.
17

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
Table 4: Evaluation scores (%) commonsense reasoning benchmarks.
PIQA
SIQA
HellaSwag
WinoGrande
COPA
Avg.
Dense
62.8
37.8
30.5
53.0
64.0
49.6
CE.T-PPL.-1%
62.7
37.4
30.5
52.6
62.0
49.1
ReLU
CETT-PPIL-5%
63.1
37.6
30.3
64.0
49.2
CETT-PPL--10%
63.0
38.0
30.5
51.5
64.0
49.4
0.IB
Dense
64.3
37.6
30.9
52.8
62.0
49.5
CETT-PPL-1%
64.3
37.5
30.7
$3.0
64.0
49.9
SiLU
CETT-PPL-5%
63.5
38.4
30.5
51.5
61.0
49.0
CETT-PPL-10%
63.8
38.1
30.4
$1.3
60.0
48.7
Dense
66.3
38.3
37.1
53.1
65.0
52.0
CETT-PPL-11%
66.3
38.1
37.2
52.7
64.0
$1.7
ReLU
CETT-PPL-5%
66.2
38.1
37.1
51.7
CETT-PPL-10%
66.0
37.9
37.0
51.9
65.0
51.6
0.2B
Dense
67.6
39.0
37.8
52.2
CETT-PPL-19%
68.2
39.2
37.7
52.0
65.0
52.4
SiLU
CETT-PPL-5%
67.4
38.2
37.7
51.8
65.0
52.0
CETT-PPL-10%%
66.8
38.8
37.9
52.1
64.0
51.9
Dense
68.8
39.9
42.7
51.9
70.0
54.7
CETT-PPL-1%
68.8
39.7
42.9
51.8
70.0
54.6
ReLU
CETT-PPL-5%%
68.3
39.9
42.7
52.5
68.0
54.3
CETT-PPL-10%
68.1
40.4
42.6
53.2
70.0
54.9
0.4B
Dense
69.0
39.6
44.5
51.9
55.8
CETT-PPL-193
68.7
39.4
44.6
52.2
74.0
55.8
SiLU
CETT-PPL-55%
68.9
39.4
44.6
$1.5
71.0
55.1
CETT-PPL--10%
68.7
39.3
44.9
51.0
72.0
55.2
Dense
70.1
41.8
50.4
53,6
68.0
56.8
CETT-PPI--15.
69.8
41.8
50.2
52.8
65.0
55.9
ReLU
CETT-PPL-55%
69.9
41.8
49.7
52,3
68.0
56.3
CETT-PPL--10%
69.6
41.8
50.0
51.8
65.0
55.6
0.8B
Dense
70.4
40.9
50.6
54.0
72.0
57.6
CETT-PPL-1%
70.3
41.4
50.6
$3.9
72.0
$7.6
SiLU
CETT-PPL-5%
69.9
41.3
51.0
54.1
69.0
57.1
CETT-PPL-10%
69.5
40.7
50.6
53.2
68.0
56.4
Dense
71.6
44.1
57.7
$6.4
70.0
60.0
CETT-PPL-11%
71.1
44.7
58.0
55.3
69.0
59.6
ReLU
CETT-PPL-5%
70.8
43.9
57.8
54.9
69.0
59.3
CETT-PPL-10%
70.2
43.6
57.1
53.7
72.0
59.3
1.2B
Dense
71.8
41.2
57.8
56.1
71.0
59.6
CETT-PPL-I99
71.8
40,9
57.8
59.6
SiLU
CETT-PPL-5%
71.8
41.3
57.9
55,9
67.0
58.8
CETT-PPL-10%
71.6
41.3
58.1
55.5
70.0
59.3
18

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
Table 5: Evaluation scores (%) on reuding comprehension benchmarks.
BoolQ
LAMBADA
TyDiQA
TyDiQA
Avg.
F1
Dense
60.8
30.1
17.9
4.1
28.2
CETT-PPL--1%
60.6
28.5
19.9
4.5
28.4
ReLU
CETT-PPIL-5%
17.9
26.9
CETT-PPL-100
60.1
24.6
16.4
3.9
26.2
0.IB
Dense
31.4
18.5
4.5
27.7
CETT-PPL-1%
56.2
31.1
19.1
5.5
28.0
SiLU
CETT-PPL-5%
53.6
28.9
18.0
5.5
26.5
CETT-PPL-10%
51.9
25.7
16.6
5.0
24.8
Dense
56.3
38.4
38.0
30.0
40.7
CETT-PPL-1%
56.2
35.8
36.8
30,0
39.7
ReLU
CETT-PPL-5%
36.3
38.6
CETT-PPL-10%
55.9
30.8
37.4
30.2
38.6
0.2B
Dense
57.5
36.3
28.2
40.2
CETT-PPL-198
57.5
38.3
35.3
27.5
39.6
SiLU
CETT-PPL-5%
55.2
36.0
31.6
24.3
36.8
CETT-PPL-110%
54.5
34.0
28.1
20.9
34.4
Dense
61.7
42.9
43.6
28.0
44.0
CETT-PPL-I90
61.6
41.3
42.1
26.6
42.9
ReLU
CETT-PPL-5%
60.8
39.1
39.9
23.4
40.8
CETT-PPL-10%
60.2
37.8
39.2
22.5
39.9
0.4B
Dense
57.6
41.1
25.4
41.8
CETT-PPL-1%
56.6
43.1
40,5
23.4
40,9
SiLU
CETT-PPL-5%%
55.2
39.2
38.1
20.4
38.2
CETT-PPIL-10%
52.7
35.9
35.0
17.7
35.3
Dense
62.1
47.3
42.6
27.3
44.8
CETT-PPL--19
61.7
45.7
41.0
24.6
43.2
ReLU
CETT-PPL-55%
60.9
43.8
40.0
24.1
42.2
CETT-PPL-110%
59.8
42.5
37.8
21.1
40.3
0.8B
Dense
63.1
46.9
41.0
22.1
43.3
CETT-PPL-I%
43.3
24.8
44.3
SiLU
CETT-PPL-5%
62.5
44.7
37.5
18.2
40.7
CETT-PPL--10%
62.7
43.0
34.6
15.0
38.8
Dense
63.3
$2.5
54.3
42.5
53.2
CETT-PPL-119 1%
63.4
52.2
55.0
42.7
53.3
ReLU
CETT-PPL-55%
62.1
49.5
56.3
45.2
53.3
CETT-PPL-10%
62.6
56.8
44.5
52.9
1.2B
Dense
63.2
53.4
55.2
47.3
54.8
CETT-PPL-199
56.1
47.5
55.4
SiLU
CETT-PPL-5%
62.2
51.2
53.1
43,9
52.6
CETT-PPL-10%
60.2
47.5
53.1
43.4
51.1
19

---

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
Table 6: Evaluation scores (%) on other more complex benchmarks.
AGIEval
HumanEval
MBPP
GSMSK
MMLU
BBH
Avg.
pass@1
pass
Dense
23.4
0.6
0.3
1.8
26.3
29.3
13.6
CETT-PPL.-1%
23.3
0.6
0.3
1.7
26.5
29.5
13.7
ReLU
CETT-PPI.-5%
0.1
1.9
28.7
13.5
CETT-PPIL-105
23.4
0.0
0.2
1.4
26.4
29.7
13.5
0.1B
Dense
0.8
1.6
29.2
13.7
CETT-PPL-110
23.5
0.6
0.4
2.1
25.6
28.5
13.4
SiLU
CETT-PPL-5%
23.6
0.6
0.3
1.4
25.8
30.6
13.7
CETT-PPL-10%
23.0
1.2
0.4
1.4
25.8
29.0
13.5
Dense
23.2
2.4
1.5
1.6
27.2
28.8
14.1
CETT-PPL--1%
22.8
2.4
1.2
2.1
26.9
30.3
14.3
ReLU
CETT-PPL--5%
1.0
29.7
CETT-PPL-10%
23.0
2.4
1.2
2.1
26.4
30.1
14.2
0.2B
Dense
24.2
4.3
1.0
25.7
29.6
14.5
CETT-PPL-11%
24.2
4.3
1.8
2.0
25.2
29.1
14.4
SiLU
CETT-PPL-5%
23.9
5.5
1.6
1.4
25.0
29.0
14.4
CETT-PPL--10%
23.2
3.0
0.5
2.4
24.2
28.4
13.6
Dense
24.6
6.7
2.3
2.1
26.1
30.3
15.3
CETT-PPL-190
24.3
7.9
3.1
1.9
26.2
30.1
15.6
ReLU
CETT-PPL-5%
24.6
7.9
2.9
2.2
26.6
30.2
15.7
CETT-PPL-10%
25.0
7.3
2.7
2.4
26.5
29.8
15.6
0.4B
Dense
24.4
5.5
3,2
24.9
30.6
15.2
CETT-PPL-1%
24.6
5.5
3.7
3.3
25.8
29.4
15.4
SiLU
CETT-PPL-55%
24.5
6.1
2.9
3.8
25.3
29.6
15.4
CETT-PPI--10%
24.2
4.9
2.3
2.7
24.6
30.1
14.8
Dense
25.4
9.2
5.3
4.2
26.3
30.1
16.7
CETT-PPL--15
25.7
9.2
5.8
4.5
26.3
30.0
16.9
ReLU
CETT-PPI-55%
25.3
8.5
5.4
4.5
26.5
29.8
16.7
CETT-PPL.-10%
25.8
8.5
5.0
29.2
16.5
0.8B
Dense
25.4
9.2
4,7
4.1
24.7
28.9
16.1
CETT-PPL-199
25.1
7.9
4.6
29.7
16.0
SiLU
CETT-PPL-5%
25.1
7.3
3.8
3.6
24.5
29.4
15.6
CETT-PPL--10%
24.8
7.3
3.9
3.0
24.2
28.8
15.3
Dense
26.6
7.3
6.2
6.4
33.4
29.9
18.3
CETT-PPL--15 %
26.5
9.8
7.8
7.7
33.9
30.3
19.3
ReLU
CETT-PPI-55%
25.8
7.9
7.4
6.3
34.3
30.2
18.6
CETT-PPL-10%
25.9
7.3
6.6
30.6
18.4
1.2B
Dense
26.2
9.8
9.0
5.2
32.6
30.9
18.9
CETT-PPL-I90
27.0
8.9
30.4
19.2
SiLU
CETT-PPL-5%
25.7
7.9
8.5
5.1
31.0
30.0
18.0
CETT-PPL--05%
25.6
9.2
6.9
4.0
30.7
30.1
17.8
20


Reviews:
[{"review_id": "sdl6CE8RNs", "reviewer": "Reviewer_BaNd", "summary": "This paper addresses three main directions related to activation sparsity. First, they introduce a new metric which they show to be better than existing activation sparsity metrics. Then, they explore the relationship between various details of the training process with the ability for a model to achieve high activation sparsity. Finally, they demonstrate that through a combination of various features discovered in the second direction, they are able to train a highly sparse LLM with an activation ratio of 6.48% with minimal performance degradation (ie at most 1% increase in perplexity).\n\n## Update after Rebuttal\nNo additional comments, the authors answered my questions and I maintain my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your excellent review. These will encourage us to further improve the quality of our work and continuously forge ahead on the research path.\n\n## Ablation Studies in \"Claims And Evidence\"\n\nThank you for reminding us of the ablation studies on the 2.4B model. As it is really expensive to pre-train a new 2.4B model from scratch, we'd like to re-state the results on smaller models for each ablation factor. Most of these are already presented in the current manuscript.\n\nAs mentioned in Section 6, we consider 3 factors: (1) ReLU as the activation function; (2) a larger amount of training data; (3) a small width-depth ratio within the interval ensuring the training stability.\n\n### Activation Function\n\nFor each size among \"0.1B, 0.2B, 0.4B, 0.8B, 1.2B\", as shown in Figure 7, **replacing ReLU with SiLU can cause a considerable increase in the activation ratio by more than 30%**, indicating worse sparsity. Note that these two activation functions do not have significant performance differences by task performance (Table 1) and training loss (Figure 14).\n\nSpecifically, we present the limit activation ratio (%) of the 0.1B/0.2B models with different activation functions in the following table. Clearly, **ReLU has significantly higher sparsity than commonly used SiLU and GELU**.\n\n|      | ReLU | SiLU | GELU |\n| :--: | :--: | :--: | :--: |\n| 0.1B | 6.14 | 40.9 | 33.3 |\n| 0.2B | 6.74 | 39.0 | 34.2 |\n\n### Amount of Training Data\n\nFor each size among \"0.1B, 0.2B, 0.4B, 0.8B, 1.2B\", as shown in Figure 4, the ReLU-activated models always display lower activation ratios (i.e., higher sparsity ratios) with the increase in the amount of training data. This fact is re-validated in the 2.4B model. Figure 11 indicates that 2.4B has a similar negative relationship between the activation ratio and the amount of training data. Therefore, **a decrease in the amount of training data can cause a sparsity drop for ReLU-activated models**.\n\n### Width-Depth Ratio\n\nWe conduct a systematic study on the relationship between sparsity and the width-depth ratio in Section 5.2. As shown in Figure 5 and Figure 6, for the 0.1B ReLU-activated model, the limit of activation ratios linearly increases with the width-depth ratio before a bottleneck point 114, and the lowest training loss with training stability can be achieved within the interval [74, 282]. The 2.4B model as well as other settings from 0.2B to 1.2B all adopt a width-depth ratio close to the smallest point of the training stability interval. In 0.1B experiments, this setting is demonstrated to have the lowest activation ratio under the premise of training stability. **A smaller width-depth ratio than this can cause training instability and worse performance, and a larger one brings about a sparsity drop.**", "claims_and_evidence": "Yes, for the most part I think the claims are supported by evidence. One additional experiment I would like to see is a side-by-side comparison of LLM training as proposed in section 6 with a baseline and/or ablations of each of the components. I understand this is likely infeasible in the rebuttal period but it would be nice to see a direct comparison of even a much smaller LLM trained from scratch if possible. To be clear, if this is not computationally feasible within the time frame, it will not affect my final score.", "methods_and_evaluation_criteria": "Yes, methods and evaluation criteria both make sense.", "theoretical_claims": "There are no theoretical claims that need to be checked.", "experimental_designs_or_analyses": "Yes, the experimental design and analysis seem sound. The paper clearly goes through each component, providing evidence for each claimed relationship to activation sparsity.", "relation_to_broader_scientific_literature": "The most relevant tie to previous literature is building upon the CETT metric to develop their proposed metric, CETT-PPL-1%. Otherwise, it seems to contribute a nice set of findings to both the activation sparsity and scaling law literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "I think this paper is well structured and written. It has novelty in the proposal of a new metric, it backs up this metric with experimental evidence of its superiority, and provides general principles/guidelines with experimental evidence.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NjjUdkb80Z", "reviewer": "Reviewer_RPBu", "summary": "This paper investigates activation sparsity in LLMs through extensive experiments. The main findings include:\n1) A quantitative analysis of sparsity patterns across model scales and width-depth ratios;\n2) The relationship between activation sparsity ratio and data scale;\n3) Achievement of a 93.52% sparsity ratio and 4.1× speedup compared to the dense model on a 2.4B parameter model using CETT-PPL-1%.\n\n## update after rebuttal\nThanks for the authors' responses, which address most of my concerns. I have decided to increase the score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your excellent review. These will encourage us to further improve the quality of our work and continuously forge ahead on the research path.\n\n## Practical Acceleration using Activation Sparsity\n\nIn Section 6, we present an acceleration experiment based on our 2.4B sparsely-activated model, achieving a 4.1$\\times$ speedup ratio. This is conducted on the machine with 1 NVIDIA A800 GPU and PowerInfer, a SOTA sparsity-based framework. As pointed out by the reviewer, there may exist gap between this setting and real-world application scenarios. Therefore, we conduct new acceleration experiments under the following setting: (1) We use the device **\"NVIDIA Jetson Orin NX\", a real-world representative end-side device tailored for AI**. (2) We combine activation sparsity with **Q4 quantization**, a mainstream acceleration technique.\n\nThe decoding speeds (token/sec) of the 2.4B model on NVIDIA Jetson Orin NX are shown in the following table:\n\n| Dense (llama.cpp) | Sparse (PowerInfer) | Sparse (PowerInfer+Q4) |\n| :---------------: | :-----------------: | :--------------------: |\n|5.42|8.76|10.97|\n\nUsing activation sparsity, we achieve a considerable speedup compared to the dense setting. Moreover, activation sparsity can also be combined with quantization and achieve an even higher decoding speedup.\n\nFinally, we admit that the combination of activation sparsity and some other acceleration methods is non-trivial. For example, we are already working on its combination with speculative decoding. In this case, we use a new auxiliary loss to increase the similarity between activation patterns of neighbor tokens. With such modification, we are able to effectively utilize activation sparsity to further promote the efficacy of speculative decoding.\n\n## Contributions of This Work\n\n**Our work is not a mere extension of previous work CETT.** Instead, we expect this work to be the foundation of future works involving the **measurement, analysis, and training of sparsely-activated LLMs**. Specifically, we give answers to three important questions:\n\n- How can activation sparsity be measured more accurately?\n\nWe present a new perspective on activation sparsity: **sparsity is the function of performance and must be measured under a specific tolerance of performance drop (i.e., PPL increase)**. Most existing works focus on ReLU-based activation sparsity, considering sparsity a fixed value with no connection to performance. After that, \"ReLU$^2$ Wins\" proposes CETT [1], enabling us to evaluate the sparsity of non-ReLU LLMs, as sparsity is considered a function of CETT.\n\nAdmittedly, in terms of methodology, our CETT-PPL-1% mainly introduces a hyper-parameter search process, finding an appropriate CETT value under a PPL increase ratio. This conversion is simple but important, as sparsity is finally linked to performance, and we are able to inspect the sparsity of LLMs under a specific expectation of performance. After all, it is more intuitive and reasonable to measure sparsity from the lens of performance rather than the ambiguous CETT. Otherwise, it will be complicated to compare the sparsity of models with different architectures (e.g., measuring the sparsity at multiple CETT points and comparing the Pareto curves).\n\n- How is activation sparsity affected by the model architecture and training process?\n\nWe present a systematic quantitative analysis of the influential factors of activation sparsity, including the activation function, amount of training data, parameter scale, and width-depth ratio. Analytical scaling laws are also found between sparsity and the amount of training data under ReLU and SiLU. These findings are the basis of the third question.\n\n- How can we build a more sparsely-activated and efficient LLM?\n\nAs the most important long-term contribution of this work, based on the above findings, **we propose the better approach to obtaining more sparsely-activated LLM**: Use ReLU as the activation function with a larger amount of pre-training data, and a small width-depth ratio within the interval ensuring the training stability. We also pre-train a 2.4B model from scratch, with an extremely low activation ratio of 6.48%, to re-validate our findings. Our work can provide instructional values for designing and pre-training an LLM with greater activation sparsity, which helps produce more efficient LLMs.\n\n## References\n\n[1] Zhang, Zhengyan, et al. \"ReLU$^ 2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs.\" *arXiv preprint arXiv:2402.03804* (2024).", "claims_and_evidence": "Yes, the sparsity analysis is backed by comprehensive measurements across different model scales.", "methods_and_evaluation_criteria": "The evaluation methods are appropriate. However, the evaluation framework for activation sparsity is directly adopted from previous work (CETT), with this study merely finding optimal hyperparameters under a specific experimental setting (PPL increase tolerance of p%). Besides, the reported 4.1× speedup on a 2.4B parameter model lacks convincing evidence for real-world application scenarios.", "theoretical_claims": "The paper does not present any theoretical proofs for its claims.", "experimental_designs_or_analyses": "The experimental designs are valid.", "relation_to_broader_scientific_literature": "The work appears to be an extension of CETT's findings on ReLU sparsity. While it provides more comprehensive analysis, it doesn't present fundamentally new insights or directions.", "essential_references_not_discussed": "The paper has appropriately cited and discussed the key related works. No significant omissions were found in the literature review.", "other_strengths_and_weaknesses": "The analysis is comprehensive, with experimental investigations across different model scales and detailed examinations of sparsity patterns. The paper is well-structured.\nHowever, the work primarily extends existing work (CETT) without introducing fundamentally new concepts and focuses more on analysis rather than proposing new solutions.", "other_comments_or_suggestions": "While the authors clearly differentiate their work from MoE and parameter pruning, two potential improvements were overlooked: (1) exploring combinations with other optimization techniques could yield more substantial improvements beyond the 4.1× speedup; (2) demonstrating practical applications would better justify the research value.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YNRIfhAy5P", "reviewer": "Reviewer_Po5c", "summary": "This paper tackles activation sparsity in LLMs to boost efficiency. They introduce CETT-PPL-1%, a sparsity metric that keeps perplexity within 1% of dense models, cutting activations. They explore four factors (pre-training data, activation function, width-depth ratio, model scale) across 0.1B to 1.2B models, finding ReLU outperforms SiLU, more data sparsifies ReLU models, deeper models help sparsity up to a limit, and scale barely shifts sparsity caps.\n\n## update after rebuttal\nAfter careful consideration, I have decided to increase the score to \"Accept.\" The lack of formal proof is not an issue at all. Also, I believe the paper has the potential to share valuable insights about increasing the activation sparsity. If the paper includes the references to the missing citations, the paper will give provide even more complete picture.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your excellent review. These will encourage us to further improve the quality of our work and continuously forge ahead on the research path.\n\n## Works in \"Essential References Not Discussed\"\n\nThank you for pointing out these two works! They are both related to our paper. We discuss these works here, and will cite them in our future versions.\n\nCATS [1] seems to be a post-training sparsification method based on thresholds. However, after carefully reading the paper and codes, we find that CATS is exactly equivalent to the Top-$k$ setting in our experiments in Section 4.1 of our paper. Specifically, for each FFN layer, CATS will find a corresponding cutoff threshold that just controls the sparsity ratio at a desired level. The activations with smaller absolute values will be dropped. Therefore, this is the same as Top-$k$, which reserves the same ratio of activation values for each layer according to the absolute values.\n\nTEAL [2] is a more interesting work. It proposes a completely different paradigm of activation sparsity. Concretely, TEAL focuses on the input sparsity, indicating that for each vector-matrix computation $\\mathbf{x}\\mathbf{W}^T$, the values at some positions of $\\mathbf{x}$ are small, and computation corresponding to these positions can be skipped for acceleration. By contrast, our work focuses on output sparsity, indicating the highly sparse patterns in the output $\\sigma(\\mathbf{x}\\mathbf{W}_{gate}^T)$ of FFN activation functions. Since these two paradigms have completely different definitions of activation sparsity, the sparsity ratios cannot be trivially compared, and the properties (e.g., scaling laws) of them are also different. We are willing to open a new work to study such input sparsity in the future.\n\n## Lack of Formal Proofs in \"Theoretical Claims\"\n\nThank you for pointing out the lack of formal math proofs in our work. We believe that more strict theoretical works are valuable and indispensable in the future. However, at present, LLMs are black-box systems, which are extremely difficult to interpret theoretically. Instead, mainstream works find empirical laws using statistical methods to make the LLM behavior more predictable. Strict math models are hard to build, as the huge parameter scales make LLMs a highly complex system. The study of human brains also encounters similar challenges, and the statistical analysis of signals like brain waves becomes the mainstream paradigm. Still, we admit the value of more rigorous and formal math modeling and will make our analyses more convincing.\n\n## Question 1\n\nConsidering the general ability, the PPL used to measure sparsity is evaluated on a general validation set with the same distribution as the training set, covering a wide variety of corpus. However, a lower average PPL on the general data cannot necessarily ensure better performance on a specific category of downstream tasks. In Table 1, the performance of reading comprehension consistently drops with an increasing PPL, but the scores of commonsense reasoning fluctuate. Such a phenomenon will be more significant for those tasks accounting for a smaller part of the training data.\n\n## Question 2\n\nSome coefficients in our activation-data power-laws can potentially be extrapolated to larger models.\n\nFirst, the limit activation ratio $A_0$ is clearly weakly correlated to the parameter scale. This is also an important finding already stated in this article.\n\nBesides, by Table 2, for ReLU-activated models, the coefficient $\\alpha$ monotonically increases, while $c$ monotonically decreases with the model scale. Coefficient $b$ seems to have little value fluctuation when the model scale is larger than 0.4B. These indicate that there probably exist quantitative relationships between these coefficients and the model scale, or in other words, we may incorporate the model scale into our scaling law. However, finding a well-fit law including the model scale is too expensive, as dozens of scales of models should be trained for data preparation. Even the most famous work on scaling laws by OpenAI [3] did not cover model scales larger than 2B.\n\n\n\n## References\n\n[1] Lee, Donghyun, et al. \"CATS: Contextually-aware thresholding for sparsity in large language models.\" *arXiv preprint arXiv:2404.08763* (2024).\n\n[2] Liu, James, et al. \"Training-free activation sparsity in large language models.\" *arXiv preprint arXiv:2408.14690* (2024).\n\n[3] Kaplan, Jared, et al. \"Scaling laws for neural language models.\" *arXiv preprint arXiv:2001.08361* (2020).", "claims_and_evidence": "The claims hold up. CETT-PPL-1% balances sparsity and performance—Figure 2 shows low PPL, and Table 1 keeps downstream scores near dense (avg. drop <0.5%). Factor findings are solid: ReLU beats SiLU (Figure 4), and the 2.4B model’s 93.52% sparsity (Section 6) follows their recipe. Thorough and extensive experiments were done to back the claims.", "methods_and_evaluation_criteria": "CETT-PPL-1% is clever—binary search tunes sparsity per layer works across ReLU and SiLU. They also conduct extensive experiments to find the activation sparsity scaling law. The extensive use of well-established benchmarks is also great.", "theoretical_claims": "No formal proofs, just fits. ReLU’s logspace power-law (Eq. 4) and SiLU’s power-law (Eq. 5) match Figure 4, with coefficients in Appendix G. Scale-insensitivity leans on similar neuron patterns (Section 5.3), supported by Figures 9-10. Smaller models converging faster (Figure 8) tracks, but the grouping model (Eq. 6) is speculative—no math locks in why sparsity caps across scales.", "experimental_designs_or_analyses": "Experiments are robust and extensive. A practical limitation would be that the experiment was limited to 2.4B model. Although not feasible, investigating the larger models could have provided more significant insights.", "relation_to_broader_scientific_literature": "They not only find the best activation sparsity technique, but also go beyond that by investigating the sparsity law. This sets them apart from the prior works that simply present a new sparsification technique. This paper has a potential to serve as a \"sparsity scaling law\" which can provide more insights into how to train a more sparse model. The paper's in-depth exploration of the four factors that affect activation sparisty is also helpful.", "essential_references_not_discussed": "\"CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models\" and \"Training-free activation sparsity in large language models\" are simple post-training techniques for SiLU that find a layer-wise absolute threshold to accelerate LLM inference, that were concurrent works to \"ReLU^2 wins.\" Citing this could provide a more complete picture of sparsifying SiLU.", "other_strengths_and_weaknesses": "While limited to smaller models (<=2.4B models), the paper provides insights into what affects the activation sparsity of language models. It is understandable that it is hard to study a model bigger than that. The experiments presented in the paper already suggests some meaningful insights on what affects activation sparsity.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Q1. In Table 1, for a smaller ReLU model, why does CETT1% perform worse than CETT10%?\n\nQ2. Can we extrapolate the scaling law for larger models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers recommend accepting the paper. The proposed CETT-PPL-1% is quite nice and seems effective. Some concerns were raised about not finding anything fundamentally new on top of CETT, because CETT-PPL-1% is just a hyperparameter search. But I think that moving towards practical and deployable activation sparsity is quite important, and this work does just that. 

I would recommend to the authors to revise and update the related work section to include a discussion of where this work fits in the broader catalogue of sparsity. Someone who reads this paper will want to know whether they can combine TEAL and CETT-PPL-1%, and you are the best people to answer this question proactively.
Decision: Accept (poster)