Training-Free Activation Sparsity in Large Language Models
Authors: James Liu
Pragaash Ponnusamy
Tianle Cai
Han Guo
Yoon Kim
Ben Athiwaratkun
Venue: ICLR 2025 Spotlight (2025)
Keywords: Large Language Models
Activation Sparsity
Efficiency
Primary area: foundation or frontier models, including LLMs
TL;DR: Training-free activation sparsity in LLMs, with a hardware-aware kernel to achieve speedup.

Abstract: Activation sparsity can enable practical inference speedups in large language models (LLMs) by reducing the compute and memory-movement required for  matrix multiplications during the forward pass. 
However, existing methods face limitations that inhibit widespread adoption. Some approaches are tailored towards older models with ReLU-based sparsity, while others require extensive continued pre-training on up to hundreds of billions of tokens. 
This paper describes TEAL (**T**raining-Fre**e** **A**ctivation Sparsity in **L**LMs), a simple training-free method that applies magnitude-based activation sparsity to hidden states throughout the entire model. TEAL achieves 40-50\% model-wide sparsity with minimal performance degradation across Llama-2, Llama-3, and Mistral families, with sizes varying from 7B to 70B. We improve existing sparse kernels and demonstrate wall-clock decoding speed-ups of up to 1.53× and 1.8× at 40\% and 50\% model-wide sparsity. TEAL is compatible with weight quantization, enabling further efficiency gains.

Published as a conference paper at ICLR 2025
TRAINING-FREE ACTIVATION SPARSITY IN LARGE LANGUAGE MODELS
James Liu Pragaash Ponnusamy?
Tianle Cai
Han Guo'
Yoon Kim
Ben Athiwaratkun
Massachusetts Institute of Technology Together Al
Princeton University
httts///gthhbb..mm/aatteeeeoognn
ABSTRACT
Activation sparsity can enable practical inference speedups in large language mod- els (LLMs) by reducing the compute and nemory-movenee required for matrix multiplications during the forward pass. However, existing methods face limi- tations that inhibit widespread adoption. Some approaches are tailored towards older models with ReL.Ubbased sparsity, while others require extensive continued pre-training on up to hundreds of billions of tokens. This paper describes TEAL (Training-Free Activation Sparsity in LLMs), a simple training-free method that applies magnitude-based activation sparsity to hidden states throughout the entire model. TEAL achieves 40-50% model-wide sparsity with minimal performance degradation across Llama-2, Llama-3, and Mistral families, with sizes varying from 7B to 70B. We improve existing sparse kernels and demonstrate wall-clock decoding speed-ups of up 1.53x and 1.8x at 40% and 50% model-wide sparsity. TEAL is compatible with weight quantization, enabling further efficiency gains.
INTRODUCTION
Large language models (LLMs) demonstrate that scaling in both parameter count and training data leads to capabilities that are useful for addressing variety of downstream tasks (Brown et | 2020). However, the large number of parameters in modern LLMs can lead to substantial challenges during inference. In typical small-batch deployment settings, autoregressive inference is memory- i... bottlenecked by the speed at which the parameters can be moved from off-chip to on-chip memory. This is in contrast to LLM training and prefill inference, which is generally compute- bound, i.c., bottlenecked by the speed at which computation can performed. A core strategy for overcoming this memory wall (Gholami et al. 2024) is through weight quantization (Frantar et al. 2022 (Shao et al.
2023
Yuan et al. [2023] L et al. 2024 Dettmers et al.
2023c Tseng 20.4 Liu et al. 2024) and sparsification (Wang et al., 2019] Frantar & Alistarh
Egiazarian et al.
2024
2023 Xia al. 2023 Ma al.] 2023), which can lead practical speed-ups when coupled with
specialized kernels that move the weights from off-chip to on-chip memory in quantized/sparse
formats (Kim et al. [2023] Dettmers et al. 2023b Frantar et al. 2024 Wang et al. 2024b Xia al.)
2024
Guo et al.) 2024).
The above methods directly compress a model's weights and apply the same (quantized/sparse) matrix to all inputs. Activation sparsity (Chen et al., 2023 Raihan & Aamodi 2020 Kurtz et al.) 2020 is an alternative method which enforces
structure on the weight matrices by
leveraging (or inducing) sparsity in the hidden states. Since the weight channels corresponding to zero-valued activations are not used in computation, speed-up can be realized by selectively omitting these weights during memory transfer, which is possible due to the hardware-frienae sparsity pattern. In older LLMs, activation sparsity is largely made possible by the high natural sparsity (around 95%) in the intermediate states of the MLP blocks in ReLU-based Transformer models (Li et al., 2023). Based on this, Liu et al. (2023) propose Deja Vu., which learns a small auxiliary model that predicts the contextual activation sparsity patterns of future layers, and realize 2x wall-clock speed-up on OPT-175B (Zhang et al. 2022a). Because the hidden state is extremely sparse, the less expressive auxiliary model can afford to overestimate non-zero activations while maintaining accuracy and efficiency (e.g., 20% predicted vs. 5% actual non zero entries).
channel-wise
"Correspondence jamesl 1@mi edu Work done during internship at Together AL.

---

Published as a conference paper at ICLR 2025
However, modern LLMs have largely moved away from Reaed feedforward layers due to their worse performance compared to vari- ants like SwiGLU (Shazeer (020)- In these models the activations are longer naturally sparse, making it difficult to apply methods like DejaVu. And while recent works have found that replacing SiLU with ReLU in the MLP blocks and performing continued pre-training can "recover" models that exhibit high activa- tion sparsity (thus making older methods ap- plicable) (Mirzadeh et al. 2023) [oong et al. 2024alb), b), such methods require training on up to hundreds of billions of tokens.
W
s(x)
Non sallent
Overview of TEAL. During decod- ing, TEAL thresholds low-magnitude activation entries to zero, which obviates the need to move the associated weight channels onto the registers, thus enabling wall-clock speed-ups.
Figure 1:
This work describes TEAL (Training-Free Activation Sparsity in LLMs), simple, training-free approach that applies activation sparsity based on magnitude pruning. TEAL is based on the observation that distributional shapes in L.L.AA-rcchittetture L.MMs are zero-mean uni- modal. By pruning low-magnitude, non-salient activations, we achieve 40-50% model-wide (input- dependent) sparsity, in contrast to prior work which only achieves sparsity in portions of the model (Lee al., [2024b) We realize wall-clock speed-ups of up to 1.53x and 1.8x 40% and 50% sparsity respectively through specialized kernels, and further demonstrate compatibility with weight quantization.
RELATED WORK
Conditional computation (Bengio 2013: Bengio et al. (016)) alleviates the burden of training and serving by selectively activating parts of model. (Shazeer et al. (2017) propose Mixture-of-Experts (MoE) in language models, applying conditional computation to feed forward networks, Mixture- of of-Experts models decouple parameter count with computational footprint (Fedus et al. 2022), and demonstrate superior scaling laws compared to dense baselines (Clark et al. 2022). Dense models can also be converted into MoE models after pre-training (Zhang et al. 2022b Szatkowski et al.,
2024 Zheng et al. 2024).
Activation sparsity occurs when a significant portion of a model's hidden states contain zero-valued entries, and can be seen as an instance of conditional computaton. Activation sparsity is known to naturally emerge in the intermediate states of ReLU-based MLPs (Li et al. 2023). Liu et al.
2023) leverage activation sparsity to accelcrate LLM inference by avoiding the transfer of weight channels associated with zero-valued entries to GPU registers. [Song et al. (2023) and Alizadeh et al. (2024) extend activation sparsity CPU offloading, reducing weight transfer from CPU to GPU memory. However, newer architectures typically make use of non-ReL MLPs (e.g.. SiiGLL,,SSazzes 2020), making these off-the-shelf methods difficult to in practice.
Recent work has thus focused on reintroducing activation sparsity in newer architectures. Mirzadeh al. (2023) replace SiLU GeLU activation functions with ReLU, followed by continued pretrain- ing on hundreds of billions of tokens. Zhang et al. (2024b) experiment with different activations and find Squared ReLU (So al.
2022 to be the most effective replacement. [Song al. (2024b) and Song et al. (2024a) introduce techniques such as activation regularization to push sparsity even higher adapted models. [Wang et al. (2024a) combine magnitude pruning with Squared ReLU and quantized activations, and establish scaling laws for sparsely activated L.MSs during pretraining.
Lee et al. (2024a) propose CATS, and realize training-fres activation sparsity on SwiGLU based ILMMs by applying magnitude pruning on the output of W ggae. with the intuition that in the training- free setting, ReLU-based methods suboptimally zero out nontrivial negative values but keep positive values with lower magnitude intact. They achieve up to 50% sparsity in W up and W Wdown for Mistral and -.lam---7BB. However, other matrices including W Waae and Wykvo are computed densely, result- ing in lower model-wide sparsity (roughly 25%). whereas we target every matrix in the model. We refer the reader to Appendix A.3 for formal definitions of the weight matrices and their interactions within each Transformer block.
2

---

Published as a conference paper at ICLR 2025
3 BACKGROUND: ACTIVATION SPARSITY IN NEURAL NETWORKS
The activation sparsity of hidden state is defined as the proportion of zero-valued entries, which interact with the model in two ways. The first is input sparsity: when computing y xWT for x IR",, W E _=**", the columns W., corresponding to zero-valued entries X, are unused. The second is output sparsity: when computing y = s @ (xWT) for the aforementioned parameters velu-valued entries s, unused. CATS makes (p gaie)) the mask and applying up- with the intuition that (-)) serves as a gating mechanism. Interestingly, for SwiGL.U.
and mask E R", the rows i,; corresponding
use of output sparsity on GLU variants, treating
output sparsity on xW
find in Section 5.4.1 that input sparsity still preferable in the training-free
In LLMs, the computation xW is memory-boune in the decoding phase due to the high memory footprint of weights, and thus reducing the transfer of unnecessary entries (i.e., rows/columns cor- responding zero-valued activations) can enable speed-ups. However, GPUs designed to fetch multiple consecutive memory entries in a single access to maximize memory bandwidth. When memory accesses are non-contiguous. as they are when unnecessary entries are scattered, this leads to inefficient use of memory bandwidth. To ensure memory coalescing and contiguous memory access, crucial to store weights associated with input sparsity in a column-major format, and weights associated with output sparsity in row-major format.
TEAL: TRAINING-FREE ACTIVATION SPARSITY IN LLMS
4.1
MOTIVATING STUDY: DISTRIBUTIONAL PROPERTIES OF ACTIVATIONS IN LLMS
MLP Wap
Waren
Attn
Attr
Block
000 001 -44
36
04
SA 06
O OTS 5010- ROSIOSS 500 M.30 D075
34
00
06
35
02
-.4
0.5
01
0.
Bock
Block 24
-30
-2.3
-0.0 -035 D02 31b
013 -05
1.0
01
Activation Value
Figure 2: Activation distributions of Llama-3-8B's four hidden states at Blocks 16, and 24. The activations preceding the Attention and MLP blocks typically exhibit Gaussian- shapes, while intermediate states within these blocks exhibit Laplacian-like shapes. The best-fit Gaussian/Laplace distributions are overlayed in blue.
We perform preliminary study of the distributional properties of activations of LLMs. We collect activations of Llama-3-8B (Dubey al.) 2024) sampled from C4 (Raffel et al. 2023) the four hidden states in Transformer block and visualize them in Figure As indicated by prior work. some of the activations are heavy-tailed and contain outliers (Dettmers et al. 2022 Xiao al. 2022] Wei et al. 2022 Nrusimha et al.] 2024). The hidden states are moreover zero-mean unimodal, and qualitatively fall into two distinctly shaped distributions. The hidden states before the Attention and the MLP layers tend to be boussian-llkk, while the hidden states in the intermediate of such layers tend to be Laplacian-like. The concentration the activations around zero motivates our magnitude-based activation pruning approach.
Remark. We do not attempt to explain why these distributions are shaped the way they are, nor do give the theoretical underpinnings of why activation sparsity works. However, make few general observations. LLM weights are typically Gaussian (Dettmers et al. 2023a), and multiplying
'Throughout, use "block" to refer entire Transformer block consisting of the seven matrices and "layer" refer an individual layer (corresponding a single matrix) within the Transformer block.
3

---

Published as a conference paper at ICLR 2025
an independent isotropic Gaussian vector with independent Gaussian matrix follows multi- variate generalized Laplace distribution Mattei (0117) (the weights and activations are clearly not independent in practice). Attention is data- data-dependent linear operator (Poli et al. 2023) which may have similar properties. Distributions may be zero-mean due to layer normalization (Ba et al. 2016). We further derive the expected error induced by pruning low-magnitude activations in Appendix A. under a more restrictive assumption that weights and activations are independent Gaussians.
4.2 TEAL
The above analysis motivates our simple approach for activation sparsity based on magnitude prun- ing. While magnitue activations could still have large effect on the output if the norms of corresponding channels of the weight matrix are large, we find that magnitude-base pruning is empirically effective. We first define a sparsification function for an activation vector as follows:
Definition 1. For random vector & threshold &p as
(ă1 (Z1..... (n)
and sparsity level p E. [0, 1]. define the
1 P( 54p
n
The sparsification function St, RRn - R" is defined as:
8t, (x) = (st, (z1),...,
.......,,, 81, (In))
where x is realization of x, and for each component:
0
i S lp
st, (xx)
otherwise
In practice we estimate lp using an empirical distribution constructed offline using activations from generic text. The sparsity level is characterized entirely by threshold (,,, which is useful in both implementation and kernel design (Section 4.4).
Let V be the set of matrices in the MLP and Attention blocks of a model, and further let N [ We define model-level sparsification configuration as (PI, (P,),,,,,,, where each E [0. I] represents the sparsity level for the corresponding matrix W, For each matrix W, E W, we define its layer-level sparsified forward pass as:
st, Wx
for input x and magnitude-basee sparsification function Stp, (-) as defined in Definition I We apply this sparsified forward pass all N matrices obtain the model-level sparsified forward pass.
4.3 BLOCK-WISS GREEDY OPTIMIZATION
Algorithm 1 Block-wise Greedy Optimization
Input: Block B, buse step size
How should find the optimal p? We initially tried a gradient-based approach to learning the thresholds based the straight through estima- tor (Bengio et al., 2013), but encountered opti- mization issues. We instead used a simple greedy approach illustrated in Algorithm [ which was found to be effective.
inpat E R xoexd, n matrices
Record size (memory footprint) of matrices
f. +- size(W,, )) for 1, 1.....
Dim fo Find size block Initialize block and layer sparsities
Om, P--0
Y, + B(X) # Forward pass through block B find ground truth output
while P 1 do
For each Transformer block, we seek to min- imize the block-wise & activation eтror sub- ject to block-level sparsity constraint. Each Transformer block consists of seven matrices: W,, W.. W. W.. W gate. W... Wd Algo- rithm Tinitializes the sparsity levels of all layers to zero, and attempts to increment the sparsity level of each layer by an amount inversely proportional to its memory footprint. The layer with the lowest l activation error is incremented, and the block- level sparsity plus associated layer-level sparsities
for i = to no
8; + a- (F/f+)
Error if we further sparsify this layer
P: +=
L(X. pl)
E. Y,
Y,Ilz
PL
end for
min, E;
+= 8, Increment layer with lowest eroor
Σ (p: f
Record p.
while
4

---

Published as a conference paper at ICLR 2025
recorded. We assign the same block-level sparsity level to every Transformer block; therefore, all blocks have the same target sparsity level, but the individual layer-level sparsities could be different across different blocks.
Cost. We describe the cost of our method. The time complexity is O (MM²² forward passes, where M is the number of samples, n is the number of matrices, and is the average step size. In practice, we use length 2048 samples and M, n, a = 10, 7, 0.05. The resulting cost over all blocks is therefore 10 7 0.05 9800 forward passes, which is less than one GPU-hour on an A A100 for Llama-3-8B.
It consumes minimal device memory due to its being block-wise and requiring no backpropagation.
4.4 HARDWARE AWARE ACCELERATION
We develop specialized sparse GEMV kernel achieve practical speed-ups, building the Triton- based (Tillet et al., 2019) kernel introduced by DejaVu (Liu et al. (023). This kernel takes in an input x, boolean sparsity mask and matrix W, and returns (x W Wall-clock speed-up is
realized in three ways: (1) W is stored in column major format for optimal memory coalescing; (2) Columns W., i are selectively loaded based the truth value of s; (3) SplitK work decomposition
is used, enabling finer-grained parallelism across thread blocks, combining partial results through atomic adds.
Our kernel makes the following improvements on top of the original kernel: (1) We fuse the mask creation process, as s = x[[x] > 1z] entirely characterized by x and lp in TEAL; (2) We accumu- late along the outer SplitK dimension in FP16 (keeping the inner in-register accumulation in FP32), as writing to global memory in FP32 results in significant traffic; (3) We specify an eviction policy in PTX. prioritizing cache retention for activations which reused across multiple thread blocks, and deprioritizing weights which are block-specific. This guarantees that activations are persistent in L2 cache.
NVIDIA A6000
NVIDIA A100
Latency (ms)
0.20
0.10
0.15
0.08
0.06
0.10
TEAL
0.04
0.05
torch.matmul
0.02
Theoretical Optimal
0.00
0.00
0.2
0.4
0.8
1.0
0.0
0.2
0.6
0.8
1.0
Sparsity
Figure 3: Latency vs. sparsity for matrix-vector multiplication (1x4096 x 4096x 14336). comparing TEAL to Deja Vu. 'Theoretical Optimal' shows the latency reduction for torch. forch.matmul. assuming perfect linear scaling with sparsity.
Figure shows small speed-up on A6000, and a larger speed-up on A100 over the Deja kernel. Note that torch. matmul is not the strongest baseline in small batch settings (Hong et al.) 20024) which why our kernel is faster at 0% sparsity for A100. We use stronger baseline for end-to- end evaluations (Section 5.2). The larger speed-up on A100 can be attributed to its higher memory bandwidth, which amplifies the impact of reducing other overhead factors. These overhead improve- ments become increasingly important memory bandwidth across device tiers improves over time, particularly for quantized models and in latency-sensitive or resource-constrainet applications.
5 RESULTS
Models and Datasets. We evaluate TEAL on the Mistral (Jiang et al., 2023). L.ama-2 (Touvron et al. 2023), and Llama-3 (Dubey et al., 2024) families. We measure the performance of sparsified models on language modeling using the WikiText MMeity et al. 2016) validation set, and on an aggregate of six downstream tasks using the EleutherAl LM Harness (Gao et al., 2023), including
5-shot MMLU, 25-shot ARC challenge. 10-shot HellaSwag, 5-shot GSMSK, zero-shot PiQA, and zero-shot Winogrande (Hendrycks et al., 2021 Clark et al.) 2018 Zellers et al. 2019: Cobbe al.,
2021 Bisk et al. 2019) SSkaguuhi et al. 2019). 1.. For language modeling, we evaluate all models on
the same 128 random samples, using 2048-token context and 512-token evaluation window.
5

---

Published as a conference paper at ICLR 2025
Baselines. We the block-wise greedily optimized sparsities from Section 4.3 for TEAL, and primarily compare to CATS (Lee et al. 2024a) in its training- frren configuration with no finetuning.
We report model-level sparsities for all methods.
CATS applies sparsity to MLP parameters, and does not apply sparsity to attention parameters. In particular, CATS sparsifies the output of W gate. replacing SiLU(( gate with st, (SiLU gate Waate
for sparsification function Stp associated with the distribution of SiLU(xW ). Overall, CATS sparsifies the intermediate state of the MLP
first performing dense computation on W gates en- forcing output sparsity on Wup- and then enforcing input sparsity on Woown This is in contrast with TEAL, which enforces input sparsity on all matrices.
Table I: Perplexity results. Results between Llama-3 and Llama-2/Mistral are not directly compa- rable due to differing vocabulary sizes.
3LaMA-3
LLaMA-2
Mistral
Method Model
8B
70B
7B
13B
70B
7B
Baseline (0%)
5.87
2.93
5.07
4.50
3.12
4.92
CATS 25%
6.78
3.64
5.52
4.99
3.42
5.87
TEAL 25%
5.94
3.02
5.09
4.51
3.13
5.01
CATS 40% TEAL 40%
10"
96.97
43.8
53.9
171
10
3.52
5.22
4.60
3,25
5.13
TEAL 50%
4.30
5,43
4.76
3,50
5.31
TEAL 65%
9.06
6.29
6.62
5.50
4.28
6.23
These methods are decoding solutions primarily, but some of the prefill needs to be sparsified for meaningful evaluation on log- ligellioooodd based tasks (such as language modeling and MMLU). For such tasks we sparsify the second half of prefill along the sequence length dimension. See Section prefill is associated with the initial tokens,
5.4.3 for a more detailed analysis most degradation which is likely related to the attention sink phenomenon (Xiao et al.) 2024), and we thus need to take care not to sparsify them. We do not sparsify prefill on generation tasks (such as GSMSK).
5.1 ACCURACY
Main Results. TEAL performant, as shown Tables and showcasing near zero degradation at 25%, and minimal degradation at 40% sparsity. At 50% sparsity, I.aam-3 variants show slightly more degradation compared to older Llama-2 2 and Mistral variants which are still fairly performant. This falls in line with prior work showing that quantization techniques are less effective on newer models trained on more tokens (Huang et al., 2024). Most models degrade significantly at 65% sparsity, with the exception of Llama-2-70k 70B which is still reasonably performant. In terms of down- stream task results, both of the 70B models are more sparsifiable than their smaller counterparts.
Table 2: Downstream task evaluation results. Reported results are averaged over six tasks. See Appendix A.2 for fine-grained results. We omit CATS 40% as it is degenerate.
LLaMA-3
LLaMA-2
Mistral
Method Model
8B
70B
7B
13B
70B
7B
Baseline (0%)
68.07
80.41
56.50
62.01
72.65
66.96
CATS 25%
64.15
79.25
54.60
60.48
71.93
64.25
TEAL 25%
67.73
80.22
56.42
62.21
72.67
66.63
TEAL 40%
66,21
79.29
55.45
61.27
72.57
65,46
TEAL 50%
63.42
78.26
54.26
60.41
72.02
64.16
TEAL 65%
52.59
73.07
48.16
55.71
69.30
58.93
ReLUfication is degenerate in the training-free setting. TEAL outperforms CATS at both 25% and 40% sparsity, which is mainly due to two factors. First and most importantly, TEAL sparsifies every matrix in the model, not just W up and W Vowaa allowing model. When applied to -lama-2-7B, CATS sparsifies the intermediate state MLPs to 56.2% at 25% overall sparsity, and 89.7% at 40% overall sparsity. TEAL avoids such extreme sparsity in
moderate sparsity levels across the

---

Published as a conference paper at ICLR 2025
any single component. Second, design choice to input sparsity instead of output sparsity for Wup yields lower error, which we analyze in Section 5.4.1
Single-batch end-to-end inference speed results, measured in tokens per second. We ex- clude Mistral-7B and Llama-2-70B as they are architecturally similar to Llama-3-8B and 70B. We utilize tensor parallelism for -lam--3770B TP2 for A100, and TP4 for A6000.
Table 3:
ILLMM-33
LLaMA-2
GPU
Sparsity
8B
70B
7B
13B
A6000
Baseline
45.32 x1.00x)
15.93 (1.00x)
50.54 (1.00x)
26.43 (1.00x)
0%
44.49 x0.98x)
15.57 (0.98x)
50.06 (0.99x)
26.25 (0.99
25%
55.38 (1.22x)
18.93 (1.19x)
64.54 (1.28x)
33.67 (1.27
40%
64.15 (1,42x)
20.86 (1.31x)
77.30 (1.53x)
40.20 (1.52x)
50%
73.94 (1.63x)
23.77 (1.49x)
89.91 (1.78x)
47.60 (1.80x)
A100
Baseline
100.79 (1.00x)
21.85 (1.00x)
110.15 (1.00x)
61.01 (1.00x)
0%
92.13 (0.91x)
20.32 (0.93x)
100.97 x0.92x)
56.33 (0.92x)
25%
112.11 (1.11x)
25.18 (1.15x)
126.14 (1.15x)
70.66 (1.16x)
40%
126.24 (1.25x)
28.78 (1.32x)
143.85 (1.31x)
81.90 (1.34x)
50%
134.29 (1.33x)
29.99 (1.37x)
154.02 (1.40x)
88.38 (1.45x)
5.2 END-TO-END DECODING SPEED-UP
We benchmark TEAL's end-to-end single-batch decoding latency by integrating with GPT-Fast (PyTorch 2024). We enable CUDA graphs and torch. compile. Tests use Llama-2 (7B, 13B) and Llama-3 (8B, 70B) models 0%, 25%, 40%, and 50% uniform sparsities. We use the standard inference benchmarking setup in GPT-Fast, which passes in roughly 5 input tokens and generates at most 200 output tokens. Our GPU power limit settings are 500W and 300W for A A100 and A6000 respectively. As shown in Table TEAL achieves significant speed-ups of up 1.53x and 1.8x 40% and 50% sparsity respectively. TEAL is slower than the baseline at 0% sparsity on A100 due to torch. compile strengthening the torch. torch.matmul matmul baseline. This suggests further room for opti- mization of our kernel. We find lower speedups for Llama-3-8B compared to Llama-2-7B partially due its larger LM Head, which we do not currently sparsify. We leave the sparsification of LM Head to future work.
5.3 COMPATIBILITY WITH QUANTIZATION
We demonstrate compatibility with quantization, which another promising direction for efficient LLM inference. We consider 8-bit channel-wise RTN, 4-bit AWQ (Lin et al. 2024). and 2/3-bit QulP# (Tseng et al., (2024), and plot the perplexity of Llama-2-7B on WikiText in Figure The point of sharp perplexity degradation is similar across bitwidths, suggesting that errors from activation sparsity and weight quantization compound somewhat independently. Combining activation sparsity with weight quantization unlocks new regimes with respect memory transferred GPU registers, allowing for higher inference speed-up. This requires developing specialized sparse + quantized kernels, which we leave future work.
16 bit (baseline) 8-bic RTN (INTB)
4-bit AWQ
Perplexity
7
QuiP#
b-bit QuiP#
5-
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.05
0.10
0.20
0.50
1.00
Sparsity
Effective Active Parameters (log scale)
Figure 4: Perplexity vs. sparsity for 1-mm-22-BB quantized to various bitwidths on WikiText. Left: Performance sparsity levels. Right: Performance normalized by bitwidth.

---

Published as a conference paper at ICLR 2025
5.4 ANALYSIS
5.4.1 SHOULD OU HAVE INPUT OR OUTPUT SPARSITY?
TEAL naturally differs from CATS in its treatment of W up- TEAL uses input sparsity, whereas CATS uses output sparsity with output mask s (SiLU(xW gue)), with the intuition that SiLU serves as a gating mechanism. We must choose one treatment over the other due to differing memory format constraints (see Section
10
1.0
TEAL CATS
Uniform Greedy
Relative Error
0.8
9-
Perplexity
0.6
8-
0.4
7
0.2
6-
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.2
Sparsity
Sparsity
Figure 5: Layer-level activation error for W up at Block 16 of Llama-3-8B: TEAL utilizing input sparsity, and CATS utilizing output sparsity.
Figure 6: Perplexity of Llama-3-8B on Wiki- Text under uniform and block- wise greedy spar- sity configurations.
We analyze the activation error in the intermediate state of MLPs, assuming W gate is computed densely, as it is in CATS. The error associated with TEAL is |[(x St, (x)) W o SiLU(xX> gate) and the error associated with CATS is ||x up @ [SiLU(x W gmee 84, (SiL.U gate ))]ll, where st, (.) and (-) are sparsification functions associated with and SiLU(( ( gate respectively. We additionally normalize errors by the norm of the unsparsified product. Figure shows that input sparsity outperforms across levels. This because output mask s has no information regarding the saliency of outputs with respect to W up, which is relevant since SiLU does not threshold exactly to zero. As a result, larger values of x W up may be unnecessarily pruned.
5.4.2 BLOCK- WISE GREEDY SPARSITIES
We observe in Figure that the block-level greedy method in Section 4.3 outperforms the uniform configuration across all sparsity levels. The resultant sparsities can be used to analyze the workings of modern LLMs. We make two interesting observations about Llama-3-70B, which tend to hold for the other models we analyze.
Attention
MLP
0.6
Sparsiyy
0.5
0.6
0.4
0.3
up
gate
down
10
60
80
70
Block Inclex
Figure 7: Greedy sparsities for Llama-3-70B at 50% model-level sparsity. Left: Attention parame-
ters. Right: MLP parameters.
Attention: We plot sparsities of Wq. WK. W., W. at 50% model-level sparsity. W,, Wx exhibit high sparsifiability in Block 0, followed by sharp decline. W.'s sparsifiability varies dynamically: starts at 50-60%, peaks at 80-90% mid-model, then returns to 50-60% in the final blocks. The blocks where W. exhibits high sparsifiability pruned FinerCut (Zhang et al. (224a),, suggesting that the sparsifiability of W, may have some correlation saliency in Attention modules.
to align with those of the Attention modules
8

---

Published as a conference paper at ICLR 2025
We plot sparsities of W up. W Waate: W down 50% model-level sparsity. Across all blocks, W down is more sparsifiable than W gate- which is more sparsifiable than Wup- Intuitively, Wdown is sparsifiable as it corresponds to Laplacian shaped distribution, which is more densely concentrated around zero than a Gaussian shaped distribution. W gaee may be more sparsifiable than W up due to SiLU decreasing the saliency of negative outputs.
MLP:
5.4.3 PREFILL SPARSIFICATION
We vary the proportion of prefill sparsified (along the sequence length dimension) in Figure Spar- sifying the second half of prefill is nearly identical to sparsifying 99% of prefill (all tokens besides the initial tokens). However, more severe degradation occurs when sparsifying the initial tokens. This is due to attention sinks (Xiao et al., (024), a phenomenon in LLMs where initial tokens are allocated an outsized amount of attention due the softmax operation. Degradation keys and values of initial "attention sink" tokens results in more substantial model degradation due to their greater importance (Hooper et al. 2024).
TEAL is a decoding solution so this is typically not an issue, but care must be taken when sparsifying prefill for evaluation on oggllikelhood based tasks.
7.5
1.0
Prefill 50%
Relative Error
0.8
2
Perplexity
7.0
Prefill 99%
Prefill 100%
0.6
6.5
8
0.4
256
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
Sparsity
Sparsity
Figure 8: Layer-level activation error for W down at Block 16 of Llama-2-7B, at varying batch sizes.
Figure 9: Perplexity of Llama-3-8B Wiki- Text, varying the proportion of prefill sparsified, using greedy sparsity configurations.
5.4.4 BATCHED SPARSIHICATION
We focus the single-batch case, but it may be valuable study activation sparsity in batched settings. The key challenge is that different inputs may prefer different sparsity patterns. We need find subset of weight columns associated with activations that relatively low-magnitude for the entire batch.
We propose to sparsify based the average magnitude of activations across the batch dimension, a natural extension from the single batch case. The resultant sparsification criterion is batch depen- dent, but still entirely characterized by threshold.
As preliminary analysis, we find the layer-level activation error for Wdown at Block 16 of Llama- 2-7B, ablated across batch sizes, in Figure
At low batch sizes above W doww still exhibits substantial sparsity. For example, in the single batch setting, W Wáonn is assigned roughly 60% spar- sity at 50% model-wide sparsity. To have the same eтrоr at batch size 4, W Waown assigned roughly 38% sparsity. As batch size tends to infinity, TEAL can be interpreted as a structured channel-wist pruning algorithm (Zhao (023), with simple pruning metric based on activation magnitde.
APPLICATIONS AND LIMITATIONS
Applications. The most immediate application of TEAL is accelerating inference in resource con- strained edge settings. These settings typically single-batch, which is where TEAL realizes the most salient speed-up. Furthermore, TEAL compatible with quantization (Section 5.3), which is another essential axis of efficiency in this setting.

---

Published as a conference paper at ICLR 2025
Limitations. TEAL exhibits substantial sparsity in the low-batch setting (Section 5.4.4) 4 but does not scale as well to higher batch sizes, which is limitation of most activation sparsity work A way to alleviate this is to push sparsities higher through continued pretraining. While TEAL focuses the training-free case, we provide many learnings that can aid future work in sparse aware adaptation.
A setting where batched inference is less difficult is in the low-batch setting of Mixture of Experts (Shazeer et al. (017)) based models, as the baseline itself does not scale well due to having to activate more experts and lowering the arithmetic intensity.
7 CONCLUSION
We propose TEAL, a simple method that applies magnitude-base basedd activation sparsity to modern LLMs without training, achieving 40-50% model-wide sparsity with minimal degradation. We ad- ditionally optimize per-layer sparsity levels, improve existing sparse kernels, and demonstrate com- patibility with quantization. We achieve wall-clock speed-ups in single-batch decoding, which is crucial in resource-constrained edge settings, We hope TEAL has impact in real-world applications and enhances our understanding of activation sparsity in LLMs.
ACKNOWLEDGMENTS
We thank Neil Movva, Jue Wang, and Yucheng Lu for helpful comments and discussion.
REFERENCES
Joshua Ainslie, James Lee- Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi multi-query transformer models from multi-head check- points, 2023. /ttL/htss///axxii.v
2305/ 13245
Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. Lim in a flash: Efficient large language model inference with limited memory, 2024. URL http:: httsss//axxvv.orrr org/abs//312. 11514
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL 06450
Emmanuel Bengio, Pierre-Lue Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models, 2016. URL ltttps://axxiii httpss//aarii.oor/abb///1111 wttp://///bbb///1..2222222
Yoshua Bengio. Deep learning of representations: Looking forward, 2013. URL https://aaxv.. org/abs/1305, 0445
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation, 2013. URL http:tt/////axiiorrggg// 1308. 3432
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense natural language, 2019. URL
11641
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https://arxte wwwp:/ivvorggg/aao www.orgg///////.oooo//ooo/...eb/ 2005. 14165
"We note that Wang et al. 2024a) propose to enforce structured n:m sparsity on activations to address batched inference, but this is applicable only if inference is compute bound instead of memory bound, and is outside the scope of our work. A regime where inference compute bound is with 1.58-hit models (Ma et al. 2024 in high-hatch settings.
10

---

Published as a conference paper at ICLR 2025
Xuanyao Chen, Zhijian Liu, Haotian Tang, Li Yi, Hang Zhao, and Song Han. Sparsevit: Revisiting activation sparsity for efficient high-resolution vision transformer, 2023. URL httpts//aax:..i. org/abs/23055 17685
Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hennigan, Matthew Johnson, Katie Millican, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Jack Rac, Erich Elsen, Koray Kavukeuuglu, and Karen Simonyan. Unified scaling laws for routed language models, 2022. URL http::httt:://tttte org/ abs/ 2tt.//2222221111
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL htttp::/arxiv.orrs
1803. 05457
Karl Cobbe. Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano. Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https::/arxiv..rrr org/
abs/2110. 14168
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.in800: 8-bit matrix multi- plication for transformers scale, 2022. URL URL http:p/s//aaxii..or///////////// 07339
Tim Dettmers, Artidoro Pagmoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finctuning of quantized Ilms, 2023a. URL
14314
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMss Advances in Neural Information Processing Systems, 36, 2023b.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. SpQR: A sparse-quantized repre- sentation for near-lossless LLM weight compression. arXiv preprint arXiv:2306.03077. 2023c.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits. David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arricta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lec, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini. Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajiwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa.
11

---

Published as a conference paper at ICLR 2025
Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petro- vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang. Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching- Hssing Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer. Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Di- ana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Es- teban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Flo- rez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Her- man, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanie Hamid Shojamazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Igor Molybog, Igor Tufanov, Irina-Flena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U. Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- ata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh. Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Rañ Ayub, Raghotham Murthy, Raghu Nayani. Rahul Mitra. Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li. Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xi- aocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin
12

---

Published as a conference paper at ICLR 2025
Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024. URL
https:///axizi org/ wwggabs///ee http///44.222222
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Al- istarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401. /00 18, 2024.
Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. Maskllm: Learnale semi-structured sparsity for large language models, 2024. URL https://aaxx httpps///aaaxi..oo/aa////////// wtt.ooo////aaa//e/oo/oe 17481
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022. URL httth:://aattitii org/ass//////eaaoooooooooeee abs/a111..... 03961
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp.
PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler. and Dan Alistarh. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210. 17323, 2022.
Elias Frantar, Roberto L. Castro, Jiale Chen, Torsten Hoefler, and Dan Alistarh. MARLIN: Mixed-precision auto-regressive parallel inference on large language models. arXiv preprint arXiv:2408. 17433 2024.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang. and Andy Zou. A framework for few-shot language model evaluation, 2023. WtLphttp:://zene wtwpodo.oorr//eeeeeeeeeeeeeeeeoe argg/rcoood///02633333
Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, and Kurt Keutzer. Ai and memory wall, 2024. [RLL/tttpss//axxivorr///as
14123
Han Guo, William Brandon, Radostin Cholakov, Jonathan Ragan-Kelley, Eric P Xing, and Yoon Kim. Fast matrix multiplications for lookup table-quantized llms. arXiv preprint arXiv:2407. 10960, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL httptt://///xii. org/abs/2009 03300
Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. Flashdecoding++:: Faster large language model inference on gpus, 2024. URL
https://aaxxi.orr/abbb/ 012222
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length 1lm inference with kv cache quantization, 2024. URL http:/ts///rvww.or///aaaa
2401. 18079
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhi ZZu, Yuanzhi Li, Shean Wang, Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
//arxiv. httrsdw.orrg/ab//106.... 09685
Wei Huang, Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. An empirical study of llama quantization: From llms to mllms, 2024. URL http::ts:///ari http::///axxvvooraaaaaaa org/abs/ 2404. 14847
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud. Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
06825
13

---

Published as a conference paper at ICLR 2025
Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. SqueezeLLM: Dense- bense-and-syp and-sparse quantization. arXiv preprint arXiv:2506.07629, 2023.
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Car,, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and exploiting acti- vation sparsity for fast inference on deep neural networks. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5533-5543. PMLR, 13-18 Jul 2020. URL
https://procceedings mlr www...pree/vvvvim wttpss//119kkurt222.00 html
Donghyun Lee, Jaeyong Lee, Genghan Zhang. Mo Tiwari, and Azalia Mirhoseini. CATS: Context- aware thresholding for sparsity large language models. In First Conference on Language Mod- eling, 2024a. URL /tthttss /tottps://opereviiiie
Je- Yong Lee, Donghyun Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini. Cats:
Contextually-awarr thresholding for sparsity in large language models, 2024b. URL https:
//arxiw.org/ab//240...o 08763
Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, and Sanjiv Kumar. The lazy neuron phenomenon: On emergence of activation sparsity in transformers, 2023. URL 06313
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration, 2024. URL
00978
James Liu, Guangxuan Xiao, Kai Li, Jason D. Lee, Song Han, Tri Dao, and Tianle Cai. Bitdelta: Your fine-tune may only be worth one bit, 2024.
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, and Beidi Chen. Deja vu: Contextual sparsity for efficient lims inference time, 2023. URL https:://arxw..orr/bbb////
17157
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang. Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit Ilms: All large language models are in 1.58 bits, 2024. URL httthttt://aakte wrg.abs/ 2402. 17764
Xinyin Ma. Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21722 2023.
Plerre-Alexandre Mattii. Multiplying gaussian matrix by a gaussian vector, 2017. URL https:
//arxivvorg//ab//100.000 02815
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL/https://arxiv www.:org//bbss... hrggabb/1609.... 07843
Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models, 2023. URL https://arxiv. www.v//.g///////////e/oo 04564
Aniruddha Nrusimha, Mayank Mishra, Naigang Wang, Dan Alistarh, Rameswar Panda, and Yoon Kim. Mitigating the impact of outlier channels for language model quantization with activation regularization. arXiv preprint arXii:2404..03600) 2024.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models, 2023. htt..httts://aaiiii wwwperwwwvv.orgaaaaea www.:orrg/abb//32/2/ 10866
Team Py Acceleratcn generative ai with pytorch i:: Gpt, fast, 2024. HttLhhtpss//pptoorr
14

---

Published as a conference paper at ICLR 2025
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL /tttps://axxiv.ort
abs/ 20os//991 1910. 10683
Md Aamir Raihan and Tor M. Aamodt. Sparse weight activation training, 2020. URL https:
//arxiv. barxiv..rgg/bss/2001 01969
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale, 2019. / horgg/bbb/999177 //tp://///rri...gg////////.. 10641
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. OmniQuant: Ommidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308. 13137, 2023.
Noam Shazeer. Glu variants improve transformer, 2020. URL httpss//aarrivv.rr//bb/2222. 05202
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageeosly large neural networks: The sparsely- eparsely-ggaed mixture-of-experts layer, 2017. URL h wttp:/ogggab/////// 1701. 06538
David R. So, Wojeiech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V. Le. Primer: Searching for efficient transformers for language modeling, 2022. URL https://arxiv.c org/
abs/2109. 086688
Chenyang Song, Xu Han, Zhengyan Zhang, Shengding Hu, Xiyu Shi, Kuai Li, Chen Chen, Zhiyuan Liu, Guangli Li, Tao Yang, and Maosong Sun. Prosparse: Introducing and enhancing intrinsic activation sparsity within large language models, 2024a. URL https://arxxv.org/aa////// 13516
Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. Powerinfer: Fast large language model serving with a consumer- grade gpu, 2023. URL htt.:ttps//axxv..or////////////
12456
Yixin Song, Haotong Xie, Zhengyan Zhang, Bo Wen, Li Ma, Zeyu Mi, and Haibo Chen. Turbo sparse: Achieving 11m sota performance with minimal activated parameters, 2024b.
//arxiv.org/abs//206 05955
Filip Szatkowski, Bartosz Wójeik, Mikołaj Piórczyński, and Simone Scardapane. Exploiting ac- tivation sparsity with dense to dynamic-k mixture-of-experts conversion, 2024. URL https:
04361
Philippe Tillet, H. T. Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPI. 2019, pp. 10-19, New York, NY, USA. 2019, Association for Computing Machinery. ISBN 9781450367 196. doi: (1145//311500.33299999
StRL/https:///// //dtps:://oi... org/10. 145/3315508 3329973
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel. Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross loy, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kamhadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. JRLLhttss://arxiv wwwperiww..../oe//eemmooooo abs/2300 09288
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipft: Even better I1m quantization with hadamard incoherence and lattice codebooks, 2024. URL https:
7/arxiv. org//bs//// 24020 04396
15

---

Published as a conference paper at ICLR 2025
Hongyu Wang, Shuming Ma, Ruiping Wang. and Furu Wei. Q Sparsee: All large language models can be fully sparsely-activated, 2024a. URL http:: //axxiv org/abs/2407. 10969
Lei Wang, Lingxiao Ma, Shijie Cao, Quanlu Zhang, Jilong Xue, Yining Shi, Ningxin Zheng, Ziming Miao, Fan Yang, Ting Cao, Yuqing Yang, and Mao Yang. Ladder: Enabling efficient low- new-precision deep learning computing through hardware-awart tensor transformation. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024b. URL https://www usenixx www.uueiiww..ooooooooe.reeeeee/e
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. arXiv preprint 19X.v::990.0440 2019.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Feng- wei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer lan- guage models. Advances in Neural Information Processing Systems, 35:17402-17414, 2022.
Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang. Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. Flash-LLM: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity. In Proceedings of VLDB, 2024.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng. and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.066/2 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant Accurate and efficient post-training quantization for large language models. arXiv:2211. 10448, 2022.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024. 999l:/ttpss///kkv..or//////22232 17453
Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.01089, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence?, 2019. URL https://arxiu wtwpgbr/g/g//////// 07830
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar. Tianlu Wang. and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022a. URL https://arxxv.orr/aaa 01068
Yang Zhang, Yawei Li, Xinpeng Wang, Qianli Shen, Barbara Plank, Bernd Bischl, Mina Rezaei, and Kenji Kawaguchi. Finercut: Finer grained interpretable layer pruning for large language models, 2024a. httpttpss//aaiivoorrra/aaasee//e
18218
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Moefication: Transformer feed-forward layers are mixtures of experts, 2022b. URL htttps//arrxvv.og.
abs/2110.01 01786
Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, and Maosong Sun. Relu² wins: Discovering efficient activation functions for sparse 1lms, 2024b. URL htt/httss://arxttt http:///aaawvv..ooaaaa////se orgg 03804
Kaiqi Zhao, Animesh Jain, and Ming Zhao. Adaptive activation-based structured pruning, 2023. URL http:///axxxvv..rg/bbb/s// 10520
Haizhong Zheng, Xiaoyan Bai, Xueshen Liu, Morley Mao, Beidi Chen, Fan Lai, and Atul Prakash. Learn to be efficient: Build structured sparsity in large language models, 2024. URL wttp:/wwwwww.///////2//////
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1
DERIVATION OF SPARSIFICATION ERROR
1.0
Theoretical Errors (random)
Theoretical Errors
Relative Error
0.8
Empirical Errors (Wup)
Empirical Errors (Waown)
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Sparsity
Figure 10: Errors Block 16 of Llama-3-8B. Gaussian based theoretical errors from random and magnitude based sparsification, empirical errors from W up and W
down-
We derive the error of magnitude- based activation sparsity for the case where W and X are inde- pendent Gaussian in Theorem A.1 Our error metric is Ex||YY-Y|zzl where X the input, Ý is Rx (||]]
the predicted output and is the ground truth output. We plot this error in Figure T0 along with empirical errors on W Wppddwwn Block 16 of Llama-3-8B, and the theoretical error obtained from random sparsification.
Definition A.1. For random vector X threshold Lp as
and sparsity level p [0, 1]. define the
n
P(Xx < =
The sparsification function St," R.n > T2 defined as:
St, (X) (st, (X1), 8tp (Xn))
where for each component:
(X,) =
X; otherwise
Lemma A.1 (Variance of Scalar Sparsified Error). For independent random normal variables X ~ (O, )² (0, ow) and sparsification function st, (.), the variance of (X - St, ( is given by:
26p pl
[r]
Var((X - (X)()) o
ox σx
where ppt)
1 _e- is the probability density function of the standard normal distribution. /2n
3/2²
Proof. For |xl I tp. X- st, (X) follows a truncated normal distribution with lower bound and upper bound given by -lp and lp respectively. We thus have:
Var((X stp (X))W) pVar(X St, (X) |X| Var(W)
xx 1 t(((t) -(- (-t)g(-t)
((t) (((-)) 2 oiv @(t) +
)
Φ(1) (
(1:
2 (1) -
-xoo
21p 4p
1p
p
σx
ox
17

---

Published as a conference paper at ICLR 2025
where t 1z and (t) (1 + erf(x//2)) is the cumulative density function of the standard ox
normal distribution.
Lemma A.2 (Expected l Norm of Sparsified Matrix- Vector Error). Let X Ik be a vector where
each X; (0, x) and W E IR be matrix where each Wj; N(0, ow with all entries
independent. For a sparsification function St, (-), let Y (X (t, (x)) WT. Then:
1) The variance of the j-th entry Y is:
2tp
tp
Var(Y,) xoxo
OX
ox
2) The expectation of the l2 norm of Y is:
21p 40
lp
B B 2 oxow / mn p
ox
ox
where tp is the threshold value satisfying F (t) = and cp(t)
- is the probability
V2a"
density function of the standard normal distribution.
Proof. For the variance of ł,: The j-th entry of Y is the sum of n independent products (X, St, (J.) Each product has variance o ot (p
oxp s0 is
26p
ox }}. Since variances of independent
terms add, we multiply this by get the result.
For the expectation of Y 22 We first show Cov(Y,, Y2)
for + k:
B[ = E
(x, St, ( W = 0
":WW;Wed for j * & due to independence and zero mean. Y, and Yn are uncorrelated for j * & and are thus independent. Therefore:
1 = E
Σ
E
Σ Var(Y;)
Substituting the variance from part summing over m components, and taking the square-root completes the proof.
Theorem A.1 (Distributional Relative Error). Let X T and W Raxm with elements inde- pendently drawn from N (O, xx) and N(0,ow) respectively. For a sparsification function 8t, (.), define Y St (X)WT and Y XWT. The distributional relative error is given by:
Ex Y Y [
2tp
tp
P
Ex Y|
where :p(t) 1 2 V2n
3.2t is the standard normal probabilty density function.
Proof.
From the previous theorem, we have Ex Y Y
axow
mn
21p
fp l
ox
For the unsparsified case, we have Ex the result.
|| ox mnn Dividing these expectations yields
18

---

Published as a conference paper at ICLR 2025
A.2 FULL DOWNSTREAM TASK RESULTS
We provide the full downstream task results for all evaluated models. For Llama-3-8B in Table also provide results obtained from the uniform sparsity configuration, showing that the greedy sparsity configuration outperforms across the board. For CATS, we additionally provide the sparsity of the hidden state in the intermediate of the MLP blocks.
Table 4: Full downstream task results for Llama-3-8B.
Method
Sparsity
MMLU
ARC
HellaSwag
GSM8K
PIQA
WinoGrande
Average
Baseline
0
65.08
57.68
82.20
49.81
80.79
72.85
68.07
25
64.48
57.34
81.63
48.52
79.92
73.88
40
62.35
54.86
79.98
43.59
79.11
72.22
65.35
Uniform
50
59.07
53.50
77.60
36.47
79.22
70.17
62.67
65
43.48
42.58
65.81
16.22
75.84
65.43
51.56
25
64.61
57.17
81.77
48.90
80.47
73.48
67.73
40
62.69
57.08
80.42
43.82
80.47
72.77
66.21
Greedy
50
59.68
53.84
78.44
38.06
78.94
71.59
63.42
65
44.54
43.86
68.85
17.51
77.15
66.06
52.99
CATS
(46.45)
61.18
54.61
80.20
39.88
79.76
69.30
64.15
Table 5: Full downstream task results for Llama-3-708
Method
Sparsity
MML.U
ARC
HellaSwag
GSMSK
PIQA
WinoGirande
Average
Baseline
0
78.70
69.71
87.94
81.12
84.55
80.43
80.41
25
78.44
69.37
87.83
80.36
84.55
80.74
80.22
77.92
68.52
87.35
78.77
83.79
79.40
79.29
Greedy
50
76.48
67.75
86.73
78.17
83.08
77.35
78.26
65
71.39
63.31
83.71
64.44
80.74
74.82
73.07
CATS
25 (45.54)
77.67
67.92
87.75
78.01
84.27
79.87
79.25
Table 6: Full downstream task results for Llama-2-7B.
Method Baseline
Sparsity
MMLU
ARC
HellaSwag
GSM8K
PiQA
WinoGrande
Average
45.78
52.47
78.96
13.95
78.94
68.90
56.50
25
45.34
52.56
78.66
14.25
78.78
68.90
56.42
40
42.81
53.16
78.28
12.66
78.40
67.40
55.45
Greedy
50
40.52
52.47
76.54
10.84
77.86
67.32
54.26
65
31.63
42.83
69.64
4.62
76.55
63.69
48.16
CATS
25 (56.2)
42.05
52.39
78.20
10.24
77.64
67.09
54.60
Table 7: Full downstream task results for Llama-2-13B.
Method
Sparsity
MMLU
ARC
HellaSwag
GSMSK
PiQA
WinoGrande
Average
Baseline
0
54.76
59.39
82.18
23.05
71.98
80.69
62.01
25
54.96
59.47
82.31
23.58
72.14
80.79
40
54.15
58.02
82.11
22.44
70.72
80.20
61.27
Greedy
50
52.13
57.85
81.38
19.71
71.11
80.25
60.41
65
44.37
53.24
76.95
12.81
68.59
78.29
55.71
CATS
(56.02)
52.31
57.76
82.73
20.32
70.24
79.49
60.48
Table 8: Full downstream task results for Llama-2-70B.
Method Baseline
Sparsity
MMLU
ARC
HellaSwag
GSMSK
PiQA
WinoGrande
Average
0
68.71
67 4949
87.02
52.38
82.70
77.58
72.65
25
68.80
67.24
86.93
52.38
82.70
77.98
72.67
40
67.75
66.81
86.88
53.90
82.48
77.58
72.57
Greedy
50
66.79
66.72
86.38
52.69
82.59
76.95
72.02
65
62.75
63.74
84.96
45.41
81.72
77.19
69.30
CATS
25 (45.54)
67.45
66.81
86.96
50.95
82.92
76.48
71.93
19

---

Published as a conference paper at ICLR 2025
Table 9: Full downstream task results for Mistral 7B.
Method
Sparsity
MMLU
ARC.
HellaSwag
GSM8K
PIQA
WinoCirande
Average
Baseline
62.46
61.43
83.47
38.21
82.10
74.11
25
62.02
61.52
83.35
37.53
81.77
73.56
66.63
40
61.00
60.84
82.65
33.89
81.28
73.09
65.46
Greedy
50
59.02
59.90
81.38
31.62
81.28
71.74
64.16
51.92
54.01
76.79
21.01
80.41
69.46
58.93
CATS
25 (46.45)
60.10
59.81
82.08
29.72
80.41
73.40
64.25
A.3 TRANSFORMER ARCHITECTURE OVERVIEW
Transformer block consists of an attention layer followed by a multilayer perceptron (MLP). Each block contains seven weight matrices that process the input R in sequence:
Attention: In Grouped Query Attention (GQA) Ainslle et al. 2023), the matrices Wg E Rdxd and Wk. W. E Rd, xd project the input into query, key, and value representations, which are fed into the attention operation. After the attention operation, W. Rdxdd projects the output back to the model dimension.
MLP: The SwiGLU (Shazeer, 2020 MLP variant uses three matrices: W gate; W up Rm which project to higher dimension, and W down ddd which projects back to the model dimen- sion. The computation flow is:
MLP(x) (Si
xW up
gale
where SiLUU(x) x ④ (x), is the sigmoid function, and @ denotes alement-wiss multiplication.
A.4 COMPARISON TO 2:4 WEIGHT SPARSITY
We compare TEAL to MaskLLM (Fang et al. 2024], a state-of-the-art approach to semi-structured 2:4 weight sparsity that learns weight masks through differentiable relaxation. The authors have not
released checkpoints as of the time of writing, so we trained our own masks on Llama-3-8B using
2B tokens from C4. We the greedily optimized sparsities described in Section 4.3 for TEAL, and evaluate both methods on WikiText:
Model
Perplexity
Llama-3 M-mma-3-8B (bascline)
5.870
Llama-3-8B + TEAL (50%)
6.673
Llama-3-8B + MaskLLM (2:4)
8.532
Llama-3-8B + TEAL MaskLLM
9.590
We observe that TEAL outperforms MaskLLM, while being training-free. In contrast, MaskLLM computationally cecensive-traininn Llama-3-8B requires 2B tokens and approximately 8B frozen parameters plus 12B learnable parameters, which took us roughly 800 H100 hours. The combination of both methods works well, suggesting they complementary rather than mutually exclusive.
We additionally compare single-batch decoding speed-up Llama-2-7B using single A6000 GPU (using MaskLLM's reported numbers from their Table 6):
Model
Speed-up
Llama-2- 2-7B TEAL (50%)
1.78x
Llama-2-1 2-7B MaskLLM (2:4)
1.4x
This comparison may not be fully representative as 2:4 weight sparsity is more performant high- batch settings and can additionally accelerate the prefill phase.
20

---

Published as a conference paper at ICLR 2025
A.5
COMPATIBILITY WITH FINE-TUNING
Table 10: Perplexity results with and without fine-tuning on Llama-3-8B.
Sparsity Level
PPL (No Fine-tuning)
PPL (With Fine-tuning)
0% (baseline)
5.870
50%
6.673
6.622
60%
7.827
7,515
70%
13.39
9,927
90%
4.141 10
4.589 10
While TEAL primarily designed as a training-free method, it can further enhanced with fine- tuning. We fine-tune Llama-3-8B using LoRA (Hu et al. (202T) with rank of 32 (approximately 1 % of parameters are trainable) and learning rate of 0.0002. The model is fine-tuned on 30M tokens from C4. We evaluate on WikiText and use the greedily optimized sparsities described in Section
We observe in Table 10 that fine-tuning provides marginal improvements lower sparsity levels (50-60%). The benefits are more pronounced at higher sparsity levels (70-90%), where fine-tuning helps recover some of the performance lost due aggressive sparsification.
21


Reviews:
[{"review_id": "ppsUi3o2Kk", "reviewer": "Reviewer_wbxr", "summary": "The authors introduce TEAL, a training-free method for inducing activation sparsity in LLMs, achieving 40-50% sparsity with minimal performance loss and up to 1.8× speed-up in decoding. Unlike previous methods, TEAL works across all model layers, supports modern activation functions, and is compatible with quantization for additional efficiency.", "questions": "**1. Calibration Set vs. Evaluation Set for Pruning Decisions:**\n - If I understand correctly, the decision to prune specific activations in TEAL is based on a calibration set, with the assumption that this set is representative enough for generalization. However, is it possible that an activation deemed unimportant (and thus pruned) based on the calibration set might behave differently on a different dataset?\n - It appears that C4 is used as the calibration set, while WikiText is used for evaluation. How similar are these datasets in structure and distribution? If you were to repeat the experiment on a different evaluation dataset, would you still observe the same levels of performance degradation?\n\n**2. Pruning Effectiveness in Highly Optimized Newer Models:**\n\n- In line 298, could it be that newer models are using their capacity more fully, meaning that fewer activations are redundant and available for pruning? If so, does this suggest that newer architectures could achieve comparable complexity with fewer parameters, making further compression both unnecessary and less effective?\n- In this scenario, how would TEAL still provide an advantage? Could you explain the benefits of TEAL for cases where newer models are already highly optimized?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- TEAL achieves high sparsity (40-50%) without retraining, resulting in faster execution (up to 1.8× speed-up) with minimal performance degradation.\n\n- A range of experiments across various models and model sizes shows that TEAL consistently outperforms other methods, demonstrating its robustness and general applicability\n\n- The proposed method is designed to work seamlessly with newer models that use modern activation functions like SwiGLU, making it applicable to the latest LLM architectures.", "weaknesses": "- Although TEAL is adapted for compatibility with newer activation functions like SwiGLU, the core concept of magnitude-based pruning is similar to prior work, which may limit its perceived innovation and contribution.\n- The paper does not mention whether the code is available for reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ABZut4cXQd", "reviewer": "Reviewer_rpTk", "summary": "The paper proposes a novel sparsification method called TEAL (Training-Free Activation Sparsity in LLMs) which aims to boost up inference speeds of a contemporary LLMs. It zeros out p-th layer-wise percentile of activations closest to 0 in all of the linear projection layers of the Transformer block. Specifically, it is catered to architectures with GeLU, SwiGLU and GeGLU activations (e.g. Llama and Mistral) which do not exhibit large level of natural sparseness, as opposed to older model families (e.g. GPT and OPT) with high sparseness induced by ReLU. The approach doesn't demand extensive fine-tuning or continual pre-training, as the sparsification is applied in online fashion to activations of the model during each forward pass. Only inexpensive greedy search over limited number of hyper-parameters is required to adapt the technique to a model.\n\nThe authors conduct experiments with several model families of different sizes, benchmarking speed, perplexity and reasoning abilities of models augmented with different degrees of TEAL activation sparsity. The results indicate that the method substantially increases inference  speeds while retaining good modeling and reasoning abilities of the models.", "questions": "1. Line 107: Which recent work?\n\n2. Could the authors please explicitly explain why activation sparsity error in figure 8 increases with the batch size? Does the algorithm nullify the activations independently across different samples of a batch or activations are zeroed according to some aggregate statistics?\n\n3. It is mentioned in the paper that the efficient low-level Triton kernel is implemented based on Deja vu kernel [1]. Is it possible to apply that kernel directly to the task of sparsifying SwiGLU and GeGLU transformers, and if it is, how it fares against the proposed method in terms of quality?\n\n4. The paper seems to imply but not to explicitly state that the result of (SiLU(xWgate ) ⊙ xWup is also sparsified. Are there any other places where activation sparsity is applied but not mentioned? \n\n\n[1] Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, and Beidi Chen. Deja vu: Contextual sparsity for efficient llms at inference time, 2023. URL https://arxiv.org/abs/2310.17157.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper successively addresses an important and relevant applied  problem: how to speed up inference of **contemporary** and widely used architectures of LLMs such as Llama model family with no or minimal quality loss. \n\n* Suggested solution is conceptually simple, and authors supply custom Triton kernels which makes their method suitable for easy adoption.\n\n* The paper is clear, legible and easy to follow. The results are solid, and the contribution is sound.\n\n* The authors analyze and visualize activation distributions for recent and largely popular Llama-3-8B architecture which gives an intuitive ground for justifying not only their own online training-free activation sparsity, but also various other existing and future algorithms.\n\n* The authors apply sparsity to all weight matrices in a Transformer block as compared to competing approach (CATS, [1]). TEAL induces greater overall sparsity with lesser degradation of performance. \n\n* The method is shown to work well in practice in conjunction with another extremely popular approach to accelerating the inference of LLMs: weight quantization.\n\n[1] Donghyun Lee, Jaeyong Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini. CATS: Context- aware thresholding for sparsity in large language models. In First Conference on Language Modeling, 2024.", "weaknesses": "1. It would be nice to include formal definitions to various matrices (such as $W_{up}$ and $W_{down}$) and layers (MLP, SwiGLU) which are referred to throughout the work.\n\n2. There are no comparisons with structured and semi-structured pruning (e.g. Mask-LLM [1] which was originally evaluated on the same benchmarks with Llama-2 models). Overall, it would be intriguing and instructive to compare differences in quality/ speed, and trade-offs between the TEAL and unstructured/ semi-structured/ structured pruning, and active sparsity inducing algorithms.\n\n3. In particular, comparison with fine-tuning free ReLUfication [2] seems to be a little bit unfair to the spirit of the method as it’s intended by the paper’s authors to be used with fine-tuning or continual pretraining. It would be nice to either completely discard this comparison in Table 1 or to replace it with a stronger baseline, such as full ReLUfication with fine-tuning.\n\n[1] Fang, G., Yin, H., Muralidharan, S., Heinrich, G., Pool, J., Kautz, J., Molchanov, P., & Wang, X. (2024). MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models. ArXiv, abs/2409.17481.\n[2] Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models, 2023. URL https://arxiv.org/abs/2310.04564.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ji9Niaja0f", "reviewer": "Reviewer_yw38", "summary": "This paper introduces TEAL, a novel magnitude-based pruning approach to achieve activation sparsity in Large Language Models (LLMs). It focuses on LLaMA-architecture LLMs with SiLU-like activation functions, identifying activation sparsity patterns through a greedy search method that does not require continued pre-training. The experiments demonstrate that TEAL achieves comparable performance on both accuracy and inference speed.", "questions": "- In Figure 3 (right), why is the theoretical Latency higher than TEAL at small sparsity levels?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper provides a high-quality exploration of sparsity patterns in SiLU-like activations, which are widely used in modern LLMs.\n- The greedy search-based solution presented is straightforward yet effective.\n- The authors demonstrate the accuracy results on several common downstream tasks at various scales LLMs, and the end-to-end throughput experiments show the practical value of TEAL in certain scenarios.", "weaknesses": "- While the main focus is on Llama-like LLMs, discussing the applicability of TEAL to other architectures, such as [Mixture-of-Experts](https://arxiv.org/abs/1701.06538) and [Mamba](https://arxiv.org/abs/2312.00752), would enhance the paper's scope.\n- The paper lacks a detailed analysis of the memory footprint of the TEAL method, especially in long context scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5MopMJNyYH", "reviewer": "Reviewer_5UWU", "summary": "The paper proposes TEAL, a training-free activation sparsification technique for Large Language Model (LLM) acceleration. Modern LLMs adopt more sophisticated activation functions in lieu of ReLU, thus their activations are no longer naturally sparse. TEAL is built upon the observation that the distribution of the inputs to the weight matrices closely follows either a Gaussian or Laplace distribution. The magnitude based sparsification follows, where the threshold is chosen by a block-level greedy optimization process based on the $\\ell_2$ error. The whole process does not involve back propagation. The customized GPU kernel attributes the acceleration of LLMs on the NVIDIA A6000 and A100 GPUs.", "questions": "1. Are there any limitations or difficulties caused by TEAL when it comes to training or fine-tuning the sparsified model?\n2. Does the Triton kernel need to be rewritten for different GPU architectures?\n3. In Table 3, how was the inference speed tested? Was it tested on autoregressive text generation? If so, what are the lengths of the given and the generated text?\n4. Does the sparsity level change based on the domain of the text (e.g., code, novel, scientific paper, etc.)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow. The challenge is stated clearly and addressed properly.\n2. The proposed method achieves meaningful progress on the activation sparsification of the non-ReLU functions in the training-free setting.\n3. The sparsified models show actual throughput improvement on the widely used GPUs. Overall, the proposed scheme seems quite practical and can be easily adopted by others.", "weaknesses": "A couple of minor concerns are listed below. \n1. Although I understand that the paper focuses on the training-free scheme, providing a discussion and/or experimental results on fine-tuned sparse activation models could make the paper stronger and more interesting. Since a couple of baseline papers (CATS and Relufication) adopt training or fine-tuning steps in their pipelines, I am curious about how much improvement a simple fine-tuning method, such as parameter-efficient fine-tuning (e.g., LoRA), using a moderate number of tokens (possibly <100K), can bring to TEAL. If post-sparsification fine-tuning is prohibited due to the attributes of TEAL, this should be discussed in the paper.\n2. The standard deviation of the sparsity by TEAL is not reported. Since the sparsity level changes from input to input, providing more statistical information can help in understanding the proposed method better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)