Training-Free Activation Sparsity in Large Language Models

## Abstract

Abstract Activation sparsity can enable practical inference speedups in large language models (LLMs) by reducing the compute and memory-movement required for matrix multiplications during the forward pass.
However, existing methods face limitations that inhibit widespread adoption. Some approaches are tailored towards older models with ReLU-based sparsity, while others require extensive continued pre-training on up to hundreds of billions of tokens.
This paper describes TEAL ( T raining-Fre e A ctivation Sparsity in L LMs), a simple training-free method that applies magnitude-based activation sparsity to hidden states throughout the entire model. TEAL achieves 40-50% model-wide sparsity with minimal performance degradation across Llama-2, Llama-3, and Mistral families, with sizes varying from 7B to 70B. We improve existing sparse kernels and demonstrate wall-clock decoding speed-ups of up to 1.53× and 1.8× at 40% and 50% model-wide sparsity. TEAL is compatible with weight quantization, enabling further efficiency gains.

## 1 Introduction

Large language models (LLMs) demonstrate that scaling in both parameter count and training data leads to capabilities that are useful for addressing a variety of downstream tasks (Brown et al., ). However, the large number of parameters in modern LLMs can lead to substantial challenges during inference. In typical small-batch deployment settings, autoregressive inference is memory-bound, i.e., bottlenecked by the speed at which the parameters can be moved from off-chip to on-chip memory. This is in contrast to LLM training and prefill inference, which is generally *compute-bound*, i.e., bottlenecked by the speed at which computation can performed. A core strategy for overcoming this *memory wall* (Gholami et al., ) is through weight quantization (Frantar et al., ; Shao et al., ; Yuan et al., ; Lin et al., ; Dettmers et al., ; Tseng et al., ; Egiazarian et al., ) and sparsification (Wang et al., ; Frantar & Alistarh, ; Xia et al., ; Ma et al., ), which can lead to practical speed-ups when coupled with specialized kernels that move the weights from off-chip to on-chip memory in quantized/sparse formats (Kim et al., ; Dettmers et al., ; Frantar et al., ; Wang et al., ; Xia et al., ; Guo et al., ).

The above methods directly compress a model’s weights and apply the same (quantized/sparse) matrix to all inputs.
Activation sparsity (Chen et al., ; Raihan & Aamodt, ; Kurtz et al., ) is an alternative method which enforces *input-dependent* structure on the weight matrices by leveraging (or inducing) sparsity in the hidden states. Since the weight channels corresponding to zero-valued activations are not used in computation, speed-up can be realized by selectively omitting these weights during memory transfer, which is possible due to the hardware-friendly channel-wise sparsity pattern.
In older LLMs, activation sparsity is largely made possible by the high natural sparsity (around 95%) in the intermediate states of the MLP blocks in ReLU-based Transformer models (Li et al., ).
Based on this, Liu et al. () propose DejaVu, which learns a small auxiliary model that predicts the contextual activation sparsity patterns of future layers, and realize a $2\times$ wall-clock speed-up on OPT-175B (Zhang et al., ). Because the hidden state is extremely sparse, the less expressive auxiliary model can afford to overestimate non-zero activations while maintaining accuracy and efficiency (e.g., 20% predicted vs. 5% actual non-zero entries).

Figure: Figure 1: Overview of TEAL. During decoding, TEAL thresholds low-magnitude activation entries to zero, which obviates the need to move the associated weight channels onto the registers, thus enabling wall-clock speed-ups.
Refer to caption: extracted/5920604/clickbait.png

However, modern LLMs have largely moved away from ReLU-based feedforward layers due to their worse performance compared to variants like SwiGLU (Shazeer, ). In these models the activations are no longer naturally sparse, making it difficult to apply methods like DejaVu. And while recent works have found that replacing SiLU with ReLU in the MLP blocks and performing continued pre-training can “recover” models that exhibit high activation sparsity (thus making older methods applicable) (Mirzadeh et al., ; Song et al., ; ), such methods require training on up to hundreds of billions of tokens.

This work describes TEAL (Training-Free Activation Sparsity in LLMs), a simple, training-free approach that applies activation sparsity based on magnitude pruning. TEAL is based on the observation that distributional shapes in LLaMA-architecture LLMs are zero-mean unimodal. By pruning low-magnitude, non-salient activations, we achieve 40-50% model-wide (input-dependent) sparsity, in contrast to prior work which only achieves sparsity in portions of the model (Lee et al., ). We realize wall-clock speed-ups of up to $1.53\times$ and $1.8\times$ at 40% and 50% sparsity respectively through specialized kernels, and further demonstrate compatibility with weight quantization.

## 2 Related Work

Conditional computation (Bengio, ; Bengio et al., ) alleviates the burden of training and serving by selectively activating parts of a model. Shazeer et al. () propose Mixture-of-Experts (MoE) in language models, applying conditional computation to the feed forward networks. Mixture-of-Experts models decouple parameter count with computational footprint (Fedus et al., ), and demonstrate superior scaling laws compared to dense baselines (Clark et al., ).

Activation sparsity occurs when a significant portion of a model’s hidden states contain zero-valued entries, and can be seen as an instance of conditional computaton. Activation sparsity is known to naturally emerge in the intermediate states of ReLU-based MLPs (Li et al., ). Liu et al. () leverage activation sparsity to accelerate LLM inference by avoiding the transfer of weight channels associated with zero-valued entries to GPU registers. Song et al. () and Alizadeh et al. () extend activation sparsity to CPU offloading, reducing weight transfer from CPU to GPU memory. However, newer architectures typically make use of non-ReLU-based MLPs (e.g., SwiGLU, Shazeer, ), making these off-the-shelf methods difficult to use in practice.

Recent work has thus focused on reintroducing activation sparsity in newer architectures. Mirzadeh et al. () replace SiLU or GeLU activation functions with ReLU, followed by continued pretraining on hundreds of billions of tokens. Zhang et al. () experiment with different activations and find Squared ReLU (So et al., ) to be the most effective replacement. Song et al. () and Song et al. () introduce techniques such as activation regularization to push sparsity even higher in adapted models. Wang et al. () combine magnitude pruning with Squared ReLU and quantized activations, and establish scaling laws for sparsely activated LLMs during pretraining.

Lee et al. () propose *CATS*, and realize training-free activation sparsity on SwiGLU based LLMs by applying magnitude pruning on the output of $\textbf{W}_{\text{gate}}$, with the intuition that in the training-free setting, ReLU-based methods suboptimally zero out nontrivial negative values but keep positive values with lower magnitude intact. They achieve up to 50% sparsity in $\textbf{W}_{\text{up}}$ and $\textbf{W}_{\text{down}}$ for Mistral and Llama-2-7B. However, other matrices including $\textbf{W}_{\text{gate}}$ and $\textbf{W}_{\text{q,k,v,o}}$ are computed densely, resulting in lower model-wide sparsity (roughly 25%), whereas we target every matrix in the model.

## 3 Background: Activation Sparsity in Neural Networks

The activation sparsity of a hidden state $\mathbf{x}$ is defined as the proportion of zero-valued entries, which can interact with the model in two ways. The first is input sparsity: when computing $\mathbf{y}=\mathbf{x}\mathbf{W}^{\top}$ for $\mathbf{x}\in\mathbb{R}^{m},\mathbf{W}\in\mathbb{R}^{n\times m}$, the columns $\mathbf{W}_{:,i}$ corresponding to zero-valued entries $\mathbf{x}_{i}$ are unused. The second is output sparsity: when computing $\mathbf{y}=\mathbf{s}\odot(\mathbf{x}\mathbf{W}^{\top})$ for the aforementioned parameters and mask $\mathbf{s}\in\mathbb{R}^{n}$, the rows $\mathbf{W}_{i,:}$ corresponding to zero-valued entries $\mathbf{s}_{i}$ are unused. Recent work makes use of output sparsity on GLU variants, treating $\mathbf{s}=\mathrm{sparsify}(\sigma(\mathbf{x}\mathbf{W}_{\text{gate}}^{\top}))$ as the mask and applying output sparsity on $\mathbf{xW}_{\text{up}}^{\top}$, with the intuition that $\sigma(\cdot)$ serves as a gating mechanism. Interestingly, we find in Section  5.4.1 that input sparsity is still preferable in the training-free case for SwiGLU.

In LLMs, the computation $\mathbf{xW}^{\top}$ is memory-bound in the decoding phase due to the high memory footprint of weights, and thus reducing the transfer of unnecessary entries (i.e., rows/columns corresponding to zero-valued activations) can enable speed-ups. However, GPUs are designed to fetch multiple consecutive memory entries in a single access to maximize memory bandwidth. When memory accesses are non-contiguous, as they are when unnecessary entries are scattered, this leads to inefficient use of memory bandwidth. To ensure memory coalescing and contiguous memory access, it is crucial to store weights associated with input sparsity in a column-major format, and weights associated with output sparsity in a row-major format.

## 4 TEAL: Training-Free Activation Sparsity in LLMs

### 4.1 Motivating Study: Distributional Properties of Activations in LLMs

Figure: Figure 2: Activation distributions of Llama-3-8B’s four hidden states at Blocks 8, 16, and 24. The activations preceding the Attention and MLP blocks typically exhibit Gaussian-like shapes, while intermediate states within these blocks exhibit Laplacian-like shapes. The best-fit Gaussian/Laplace distributions are overlayed in blue.
Refer to caption: extracted/5920604/distrs.png

We perform a preliminary study of the distributional properties of activations of LLMs.
We collect activations of Llama-3-8B (Dubey et al., ) sampled from C4 (Dodge et al., ) at the four hidden states in a Transformer block,(^1^11Throughout, we use “block” to refer to an entire Transformer block consisting of the seven matrices and “layer” to refer to an individual layer (corresponding to a single matrix) within the Transformer block.) and visualize them in Figure 2. As indicated by prior work, some of the activations are heavy-tailed and contain outliers (Dettmers et al., ; Xiao et al., ; Wei et al., ; Nrusimha et al., ). The hidden states are moreover zero-mean unimodal, and qualitatively fall into two distinctly shaped distributions. The hidden states before the Attention and the MLP layers tend to be Gaussian-like, while the hidden states in the intermediate of such layers tend to be Laplacian-like.
The concentration of the activations around zero motivates our magnitude-based activation pruning approach.

Remark. We do not attempt to explain why these distributions are shaped the way they are, nor do we give the theoretical underpinnings of why activation sparsity works. However, we make a few general observations. LLM weights are typically Gaussian
(Dettmers et al., ), and multiplying an independent isotropic Gaussian vector with an independent Gaussian matrix follows a multivariate generalized Laplace distribution Mattei () (the weights and activations are clearly not independent in practice). Attention is a data-dependent linear operator (Poli et al., ) which may have similar properties. Distributions may be zero-mean due to layer normalization (Ba et al., ).
We further derive the expected error induced by pruning low-magnitude activations in Appendix A.1, under a more restrictive assumption that weights and activations are independent Gaussians.

### 4.2 TEAL

The above analysis motivates our simple approach for activation sparsity based on magnitude pruning.
While small-magnitude activations could still have a large effect on the output if the norms of corresponding channels of the weight matrix are large, we find that magnitude-based pruning is empirically effective.
We first define a sparsification function for an activation vector as follows:

###### Definition 1 .

For a random vector $\tilde{\mathbf{x}}=(\tilde{x}_{1},\dots,\tilde{x}_{n})$ and sparsity level $p\in[0,1]$, define the threshold $t_{p}$ as

$$ $\frac{1}{n}\sum_{i=1}^{n}\mathbb{P}(|\tilde{x}_{i}|\leq t_{p})=p.$ $$

The sparsification function $s_{t_{p}}:\mathbb{R}^{n}\to\mathbb{R}^{n}$ is defined as:

$$ $s_{t_{p}}(\mathbf{x})=(s_{t_{p}}(x_{1}),\ldots,s_{t_{p}}(x_{n}))$ $$

where $\mathbf{x}$ is a realization of $\tilde{\mathbf{x}}$, and for each component:

$$ $s_{t_{p}}(x_{i})=\begin{cases}0&\text{if }|x_{i}|\leq t_{p}\\ x_{i}&\text{otherwise}\end{cases}$ $$

In practice we estimate $t_{p}$ using an empirical distribution constructed offline using activations from generic text. The sparsity level ${p}$ is characterized entirely by threshold $t_{p_{i}}$, which is useful in both implementation and kernel design (Section 4.4).

Let $\mathcal{W}$ be the set of matrices in the MLP and Attention blocks of a model, and further let $N=|\mathcal{W}|$. We define a model-level sparsification configuration as $\mathbf{p}=(p_{1},...,p_{N})$, where each $p_{i}\in[0,1]$ represents the sparsity level for the corresponding matrix $\mathbf{W}_{i}$.
For each matrix $\mathbf{W}_{i}\in\mathcal{W}$, we define its layer-level sparsified forward pass as:

$$ $\hat{\mathbf{Y}}=s_{t_{p_{i}}}(\mathbf{x})\mathbf{W}_{i}^{\top}$ $$

for input $\mathbf{x}$ and magnitude-based sparsification function $s_{t_{p_{i}}}(\cdot)$ as defined in Definition 1. We apply this sparsified forward pass to all $N$ matrices to obtain the model-level sparsified forward pass.

Figure: Algorithm 1 Block-wise Greedy Optimization

###### Definition 1 .

### 4.3 Block-wise Greedy Optimization

How should we find the optimal $\mathbf{p}$? We initially tried a gradient-based approach to learning the thresholds based on the straight through estimator (Bengio et al., ), but encountered optimization issues. We instead used a simple greedy approach illustrated in Algorithm 1, which was found to be effective.

For each Transformer block, we seek to minimize the block-wise $\ell_{2}$ activation error subject to a block-level sparsity constraint. Each Transformer block consists of seven matrices: $\mathbf{W}_{\text{q}},\mathbf{W}_{\text{k}},\mathbf{W}_{\text{v}},\mathbf{W}_{
\text{o}},\mathbf{W}_{\text{gate}},\mathbf{W}_{\text{up}},\mathbf{W}_{\text{
down}}$.
Algorithm 1 initializes the sparsity levels of all layers to zero, and attempts to increment the sparsity level of each layer by an amount inversely proportional to its memory footprint. The layer with the lowest $\ell_{2}$ activation error is incremented, and the block-level sparsity plus associated layer-level sparsities are recorded. We assign the same block-level sparsity level across all Transformer blocks; therefore, all blocks have the same target sparsity level, but the individual layer-level sparsities could be different across different blocks.

Cost. We describe the cost of our method. The time complexity is $\mathcal{O}(\frac{Mn^{2}}{\alpha})$ forward passes, where $M$ is the number of samples, $n$ is the number of matrices, and $\alpha$ is the average step size. In practice, we use length 2048 samples and $M,n,\alpha=10,7,0.05$. The resulting cost over all blocks is therefore $10\cdot 7^{2}\cdot\frac{1}{0.05}=9800$ forward passes, which is less than one GPU-hour on an A100 for Llama-3-8B. It consumes minimal device memory due to its being block-wise and requiring no backpropagation.

### 4.4 Hardware Aware Acceleration

We develop a specialized sparse GEMV kernel to achieve practical speed-ups, building on the Triton-based (Tillet et al., ) kernel introduced by DejaVu (Liu et al., ). This kernel takes in an input $\mathbf{x}$, boolean sparsity mask $\mathbf{s}$ and matrix $\mathbf{W}$, and returns $(\mathbf{x}\odot\mathbf{s})\mathbf{W}^{\top}$. Wall-clock speed-up is realized in three ways: (1) $\mathbf{W}$ is stored in column major format for optimal memory coalescing; (2) Columns $\mathbf{W}_{:,i}$ are selectively loaded based on the truth value of $\mathbf{s}_{i}$; (3) SplitK work decomposition is used, enabling finer-grained parallelism across thread blocks, combining partial results through atomic adds.

Our kernel makes the following improvements on top of the original kernel: (1) We fuse the mask creation process, as $\mathbf{s}=\mathbf{x}[|\mathbf{x}|>t_{p}]$ is entirely characterized by $\mathbf{x}$ and $t_{p}$ in TEAL; (2) We accumulate along the outer SplitK dimension in FP16 (keeping the inner in-register accumulation in FP32), as writing to global memory in FP32 results in significant traffic;
(3) We specify an eviction policy in NVIDIA PTX, prioritizing cache retention for activations which are reused across multiple thread blocks, and deprioritizing weights which are block-specific. This guarantees that activations are persistent in L2 cache.

Figure: Figure 3: Latency vs. sparsity for matrix-vector multiplication (1x4096 × 4096x14336), comparing TEAL to Deja Vu.
Refer to caption: extracted/5920604/kernel.png

Figure 3 shows a small speed-up on A6000, and a larger speed-up on A100 over the DejaVu kernel. Note that torch.matmul is not the strongest baseline in small batch settings (Hong et al., ), which is why our kernel is faster at 0% sparsity for A100. We use a stronger baseline for end-to-end evaluations (Section 5.2). The larger speed-up on A100 can be attributed to its higher memory bandwidth, which amplifies the impact of reducing other overhead factors.
These overhead improvements become increasingly important as memory bandwidth across device tiers improves over time, particularly for quantized models and in latency-sensitive or resource-constrained applications.

## 5 Results

##### Models and Datasets.

We evaluate TEAL on the Mistral (Jiang et al., ), Llama-2 (Touvron et al., ), and Llama-3 (Dubey et al., ) families. We measure the performance of sparsified models on language modeling using the WikiText (Merity et al., ) validation set, and on an aggregate of six downstream tasks using the EleutherAI LM Harness (Gao et al., ), including 5-shot MMLU, 25-shot ARC challenge, 10-shot HellaSwag, 5-shot GSM8K, zero-shot PiQA, and zero-shot Winogrande (Hendrycks et al., ; Clark et al., ; Zellers et al., ; Cobbe et al., ; Bisk et al., ; Sakaguchi et al., ). For language modeling, we evaluate all models on the same 128 random samples, using a 2048-token context and 512-token evaluation window.

##### Baselines.

We use the block-wise greedily optimized sparsities from Section 4.3 for TEAL, and primarily compare to CATS (Lee et al., ) and ReLUfication (Mirzadeh et al., ) in their training-free configurations with no finetuning. We report model-level sparsities for all methods.

Both CATS and ReLUfication apply sparsity to MLP parameters, and do not apply sparsity to attention parameters. For SwiGLU variants, the original dense forward pass is defined as: $\text{MLP}(\mathbf{x})=(\text{SiLU}(\mathbf{x}\mathbf{W}_{\text{gate}})\odot
\mathbf{x}\mathbf{W}_{\text{up}})\mathbf{W}_{\text{down}}$. CATS sparsifies the output of $\mathbf{W}_{\text{gate}}$, replacing $\text{SiLU}(\mathbf{x}\mathbf{W}_{\text{gate}})$ with $s_{t_{p}}(\text{SiLU}(\mathbf{x}\mathbf{W}_{\text{gate}}))$ for sparsification function $s_{t_{p}}$ associated with the distribution of $\text{SiLU}(\mathbf{x}\mathbf{W}_{\text{gate}})$. ReLUfication simply replaces the SiLU activation function with ReLU. Both methods end up sparsifying the intermediate state of the MLP by first performing dense computation on $\mathbf{W}_{\text{gate}}$, enforcing output sparsity on $\mathbf{W}_{\text{up}}$, and then enforcing input sparsity on $\mathbf{W}_{\text{down}}$. This is in contrast with TEAL, which enforces input sparsity on all matrices.

**Table 1: Perplexity results. Results between Llama-3 and Llama-2/Mistral are not directly comparable due to differing vocabulary sizes.**
|  | LLaMA-3 | LLaMA-2 | Mistral |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method / Model | 8B | 70B | 7B | 13B | 70B | 7B |
| Baseline (0%) | 5.87 | 2.93 | 5.07 | 4.50 | 3.12 | 4.92 |
| ReLUfication ($\sim$35%) | 1.5 · 10^5 | 1.5 · 10^4 | 1.8 · 10^4 | 6.3 · 10^3 | 9.2 · 10^3 | 3.4 · 10^4 |
| CATS 25% | 6.78 | 3.64 | 5.52 | 4.99 | 3.42 | 5.87 |
| TEAL 25% | 5.94 | 3.02 | 5.09 | 4.51 | 3.13 | 5.01 |
| CATS 40% | 7.6 · 10^4 | 96.97 | 43.8 | 53.9 | 171 | 2.8 · 10^4 |
| TEAL 40% | 6.21 | 3.52 | 5.22 | 4.60 | 3.25 | 5.13 |
| TEAL 50% | 6.67 | 4.30 | 5.43 | 4.76 | 3.50 | 5.31 |
| TEAL 65% | 9.06 | 6.29 | 6.62 | 5.50 | 4.28 | 6.23 |

These methods are decoding solutions primarily, but some of the prefill needs to be sparsified for meaningful evaluation on log-likelihood based tasks (such as language modeling and MMLU). For such tasks we sparsify the second half of prefill along the sequence length dimension.
See Section 5.4.3 for a more detailed analysis – most degradation in prefill is associated with the initial tokens, which is likely related to the attention sink phenomenon (Xiao et al., ), and we thus need to take care not to sparsify them. We do not sparsify prefill on generation tasks (such as GSM8K).

### 5.1 Accuracy

##### Main Results.

TEAL is performant, as shown in Tables 1 and 2, showcasing near zero degradation at 25%, and minimal degradation at 40% sparsity. At 50% sparsity, Llama-3 variants show slightly more degradation compared to older Llama-2 and Mistral variants which are still fairly performant. This falls in line with prior work showing that quantization techniques are less effective on newer models trained on more tokens (Huang et al., ). Most models degrade significantly at 65% sparsity, with the exception of Llama-2-70B which is still reasonably performant. In terms of downstream task results, both of the 70B models are more sparsifiable than their smaller counterparts.

**Table 2: Downstream task evaluation results. Reported results are averaged over six tasks. See Appendix A.2 for fine-grained results. We omit CATS 40% and ReLUfication as they are degenerate.**
|  | LLaMA-3 | LLaMA-2 | Mistral |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method / Model | 8B | 70B | 7B | 13B | 70B | 7B |
| Baseline (0%) | 68.07 | 80.41 | 56.50 | 62.01 | 72.65 | 66.96 |
| CATS 25% | 64.15 | 79.25 | 54.60 | 60.48 | 71.93 | 64.25 |
| TEAL 25% | 67.73 | 80.22 | 56.42 | 62.21 | 72.67 | 66.63 |
| TEAL 40% | 66.21 | 79.29 | 55.45 | 61.27 | 72.57 | 65.46 |
| TEAL 50% | 63.42 | 78.26 | 54.26 | 60.41 | 72.02 | 64.16 |
| TEAL 65% | 52.59 | 73.07 | 48.16 | 55.71 | 69.30 | 58.93 |

ReLUfication is degenerate in the training-free setting. TEAL outperforms CATS at both 25% and 40% sparsity, which is mainly due to two factors. First and most importantly, TEAL sparsifies every matrix in the model, not just $\mathbf{W}_{\text{up}}$ and $\mathbf{W}_{\text{down}}$, allowing us to moderate sparsity levels across the model. When applied to Llama-2-7B, CATS sparsifies the intermediate state of MLPs to 56.2% at 25% overall sparsity, and to 89.7% at 40% overall sparsity. TEAL avoids such extreme sparsity in any single component. Second, our design choice to use input sparsity instead of output sparsity for $\mathbf{W}_{\text{up}}$ yields lower error, which we analyze in Section 5.4.1.

**Table 3: Single-batch end-to-end inference speed results, measured in tokens per second. We exclude Mistral-7B and Llama-2-70B as they are architecturally similar to Llama-3-8B and 70B. We utilize tensor parallelism for Llama-3-70B: TP2 for A100, and TP4 for A6000.**
|  |  | LLaMA-3 | LLaMA-2 |  |  |
| --- | --- | --- | --- | --- | --- |
| GPU | Sparsity | 8B | 70B | 7B | 13B |
| A6000 | Baseline | 45.32 (1.00$\times$) | 15.93 (1.00$\times$) | 50.54 (1.00$\times$) | 26.43 (1.00$\times$) |
|  | 0% | 44.49 (0.98$\times$) | 15.57 (0.98$\times$) | 50.06 (0.99$\times$) | 26.25 (0.99$\times$) |
|  | 25% | 55.38 (1.22$\times$) | 18.93 (1.19$\times$) | 64.54 (1.28$\times$) | 33.67 (1.27$\times$) |
|  | 40% | 64.15 (1.42$\times$) | 20.86 (1.31$\times$) | 77.30 (1.53$\times$) | 40.20 (1.52$\times$) |
|  | 50% | 73.94 (1.63$\times$) | 23.77 (1.49$\times$) | 89.91 (1.78$\times$) | 47.60 (1.80$\times$) |
| A100 | Baseline | 100.79 (1.00$\times$) | 21.85 (1.00$\times$) | 110.15 (1.00$\times$) | 61.01 (1.00$\times$) |
|  | 0% | 92.13 (0.91$\times$) | 20.32 (0.93$\times$) | 100.97 (0.92$\times$) | 56.33 (0.92$\times$) |
|  | 25% | 112.11 (1.11$\times$) | 25.18 (1.15$\times$) | 126.14 (1.15$\times$) | 70.66 (1.16$\times$) |
|  | 40% | 126.24 (1.25$\times$) | 28.78 (1.32$\times$) | 143.85 (1.31$\times$) | 81.90 (1.34$\times$) |
|  | 50% | 134.29 (1.33$\times$) | 29.99 (1.37$\times$) | 154.02 (1.40$\times$) | 88.38 (1.45$\times$) |

### 5.2 End-to-end Decoding Speed-up

We benchmark TEAL’s end-to-end single-batch decoding latency by integrating it with GPT-Fast (PyTorch, ). We enable CUDA graphs and torch.compile. Tests use Llama-2 (7B, 13B) and Llama-3 (8B, 70B) models at 0%, 25%, 40%, and 50% uniform sparsities. Our GPU power limit settings are 500W and 300W for A100 and A6000 respectively. As shown in Table 3, TEAL achieves significant speed-ups of up to $1.53\times$ and $1.8\times$ at 40% and 50% sparsity respectively. TEAL is slower than the baseline at 0% sparsity on A100 due to torch.compile strengthening the torch.matmul baseline. This suggests further room for optimization of our kernel.
We find lower speedups for Llama-3-8B compared to Llama-2-7B partially due to its larger LM Head, which we do not currently sparsify.

### 5.3 Compatibility with Quantization

We demonstrate compatibility with quantization, which is another promising direction for efficient LLM inference. We consider 8-bit channel-wise RTN, 4-bit AWQ (Lin et al., ), and 2/3-bit QuIP# (Tseng et al., ), and plot the perplexity of Llama-2-7B on WikiText in Figure 4. The point of sharp perplexity degradation is similar across bitwidths, suggesting that errors from activation sparsity and weight quantization compound somewhat independently. Combining activation sparsity with weight quantization unlocks new regimes with respect to memory transferred to GPU registers, allowing for higher inference speed-up. This requires developing specialized sparse + quantized kernels, which we leave to future work.

Figure: Figure 4: Perplexity vs. sparsity for Llama-2-7B quantized to various bitwidths on WikiText. Left: Performance over sparsity levels. Right: Performance normalized by bitwidth.
Refer to caption: extracted/5920604/quant.png

### 5.4 Analysis

#### 5.4.1 Should W up subscript W up \textbf{W}_{\text{up}} W start_POSTSUBSCRIPT up end_POSTSUBSCRIPT have Input or Output Sparsity?

TEAL naturally differs from CATS in its treatment of $\mathbf{W}_{\text{up}}$. TEAL uses input sparsity, whereas CATS uses output sparsity with output mask $\mathbf{s}=s_{t_{p}}(\text{SiLU}(\mathbf{x}\mathbf{W}_{\text{gate}}^{\top}))$, with the intuition that SiLU serves as a gating mechanism. We must choose one treatment over the other due to differing memory format constraints (see Section 3).

Figure: Figure 5: Layer-level activation error for $\mathbf{W}_{\text{up}}$ at Block 16 of Llama-3-8B: TEAL utilizing input sparsity, and CATS utilizing output sparsity.
Refer to caption: extracted/5920604/zealvcats.png

We analyze the activation error in the intermediate state of MLPs, assuming $\mathbf{W}_{\text{gate}}$ is computed densely, as it is in CATS. The error associated with TEAL is $||(\mathbf{x}-s_{t_{p}}(\mathbf{x}))\mathbf{W}_{\text{up}}^{\top}\odot\text{
SiLU}(\mathbf{x}\mathbf{W}_{\text{gate}}^{\top})||_{2}$, and the error associated with CATS is $||\mathbf{x}\mathbf{W}_{\text{up}}^{\top}\odot[\text{SiLU}(\mathbf{x}\mathbf{W
}_{\text{gate}}^{\top})-s^{\prime}_{t_{p}}(\text{SiLU}(\mathbf{x}\mathbf{W}_{
\text{gate}}^{\top}))]||_{2}$, where $s_{t_{p}}(\cdot)$ and $s^{\prime}_{t_{p}}(\cdot)$ are sparsification functions associated with $\mathbf{x}$ and $\text{SiLU}(\mathbf{x}\mathbf{W}_{\text{gate}}^{\top})$ respectively. We additionally normalize errors by the norm of the unsparsified product.
Figure 6 shows that input sparsity outperforms across all levels. This is because output mask $\mathbf{s}$ has no information regarding the saliency of outputs with respect to $\mathbf{W}_{\text{up}}$, which is relevant since SiLU does not threshold exactly to zero. As a result, larger values of $\mathbf{x}\mathbf{W}_{\text{up}}$ may be unnecessarily pruned.

#### 5.4.2 Block-wise Greedy Sparsities

We observe in Figure 6 that the block-level greedy method in Section 4.3 outperforms the uniform configuration across all sparsity levels. The resultant sparsities can be used to analyze the workings of modern LLMs. We make two interesting observations about Llama-3-70B, which tend to hold for the other models we analyze.

Figure: Figure 7: Greedy sparsities for Llama-3-70B at 50% model-level sparsity. Left: Attention parameters. Right: MLP parameters.
Refer to caption: extracted/5920604/projs.png

Attention: We plot sparsities of $\mathbf{W}_{\text{q}},\mathbf{W}_{\text{k}},\mathbf{W}_{\text{v}},\mathbf{W}_{
\text{o}}$ at 50% model-level sparsity. $\mathbf{W}_{\text{q}},\mathbf{W}_{\text{k}}$ exhibit high sparsifiability in Block 0, followed by a sharp decline. $\mathbf{W}_{\text{o}}$’s sparsifiability varies dynamically: it starts at 50-60%, peaks at 80-90% mid-model, then returns to 50-60% in the final blocks. The blocks where $\mathbf{W}_{\text{o}}$ exhibits high sparsifiability seem to align with those of the Attention modules pruned in *FinerCut* (Zhang et al., ), suggesting that the sparsifiability of $\mathbf{W}_{\text{o}}$ may have some correlation to saliency in Attention modules.

MLP: We plot sparsities of $\mathbf{W}_{\text{up}},\mathbf{W}_{\text{gate}},\mathbf{W}_{\text{down}}$ at 50% model-level sparsity. Across all blocks, $\mathbf{W}_{\text{down}}$ is more sparsifiable than $\mathbf{W}_{\text{gate}}$, which is more sparsifiable than $\mathbf{W}_{\text{up}}$. Intuitively, $\mathbf{W}_{\text{down}}$ is sparsifiable as it corresponds to a Laplacian shaped distribution, which is more densely concentrated around zero than a Gaussian shaped distribution. $\mathbf{W}_{\text{gate}}$ may be more sparsifiable than $\mathbf{W}_{\text{up}}$ due to SiLU decreasing the saliency of negative outputs.

#### 5.4.3 Prefill Sparsification

We vary the proportion of prefill sparsified (along the sequence length dimension) in Figure 9. Sparsifying the second half of prefill is nearly identical to sparsifying 99% of prefill (all tokens besides the initial tokens). However, more severe degradation occurs when sparsifying the initial tokens. This is due to attention sinks (Xiao et al., ), a phenomenon in LLMs where initial tokens are allocated an outsized amount of attention due to the softmax operation. Degradation to keys and values of initial “attention sink” tokens results in more substantial model degradation due to their greater importance (Hooper et al., ).

TEAL is a decoding solution so this is typically not an issue, but care must be taken when sparsifying prefill for evaluation on log-likelihood based tasks.

Figure: Figure 8: Layer-level activation error for $\mathbf{W}_{\text{down}}$ at Block 16 of Llama-2-7B, at varying batch sizes.
Refer to caption: extracted/5920604/batch.png

#### 5.4.4 Batched Sparsification

We focus on the single-batch case, but it may be valuable to study activation sparsity in batched settings.
The key challenge is that different inputs may prefer different sparsity patterns. We need to find a subset of weight columns associated with activations that are relatively low-magnitude for the entire batch.

We propose to sparsify based on the average magnitude of activations across the batch dimension, a natural extension from the single batch case. The resultant sparsification criterion is batch dependent, but is still entirely characterized by a threshold.

As a preliminary analysis, we find the layer-level activation error for $\mathbf{W}_{\text{down}}$ at Block 16 of Llama-2-7B, ablated across batch sizes, in Figure 9. At low batch sizes above 1, $\mathbf{W}_{\text{down}}$ still exhibits substantial sparsity. For example, in the single batch setting, $\mathbf{W}_{\text{down}}$ is assigned roughly 60% sparsity at 50% model-wide sparsity. To have the same error at batch size 4, $\mathbf{W}_{\text{down}}$ is assigned roughly 38% sparsity.
As batch size tends to infinity, TEAL can be interpreted as a structured channel-wise pruning algorithm (Zhao et al., ), with a simple pruning metric based on activation magnitude.

## 6 Applications and Limitations

##### Applications.

The most immediate application of TEAL is accelerating inference in resource constrained edge settings. These settings are typically single-batch, which is where TEAL realizes the most salient speed-up. Furthermore, TEAL is compatible with quantization (Section 5.3), which is another essential axis of efficiency in this setting.

##### Limitations.

TEAL exhibits substantial sparsity in the low-batch setting (Section 5.4.4) but does not scale as well to higher batch sizes, which is a limitation of most activation sparsity work(^2^22We note that Wang et al. () propose to enforce structured n:m sparsity on activations to address batched inference, but this is applicable only if inference is compute bound instead of memory bound, and is outside the scope of our work. A regime where inference is compute bound is with 1.58-bit models (Ma et al., ) in high-batch settings.). A way to alleviate this is to push sparsities higher through continued pretraining.
While TEAL focuses on the training-free case, we provide many learnings that can aid future work in sparse aware adaptation.

A setting where batched inference is less difficult is in the low-batch setting of Mixture of Experts (Shazeer et al., ) based models,
as the baseline itself does not scale well due to having to activate more experts and lowering the arithmetic intensity.

## 7 Conclusion

We propose TEAL, a simple method that applies magnitude-based activation sparsity to modern LLMs without training, achieving 40-50% model-wide sparsity with minimal degradation. We additionally optimize per-layer sparsity levels, improve existing sparse kernels, and demonstrate compatibility with quantization. We achieve wall-clock speed-ups in single-batch decoding, which is crucial in resource-constrained edge settings. We hope TEAL has impact in real-world applications and enhances our understanding of activation sparsity in LLMs.

## Appendix A Appendix

### A.1 Derivation of Sparsification Error

Figure: Figure 10: Errors at Block 16 of Llama-3-8B: Gaussian-based theoretical errors from random and magnitude based sparsification, empirical errors from $\mathbf{W}_{\text{up}}$ and $\mathbf{W}_{\text{down}}$.
Refer to caption: extracted/5920604/error.png

We derive the error of magnitude-based activation sparsity for the case where $\mathbf{W}$ and $\mathbf{X}$ are independent Gaussian in Theorem A.1. Our error metric is $\frac{\mathbb{E}_{\mathbf{X}}[\|\mathbf{Y}-\hat{\mathbf{Y}}\|_{2}]}{\mathbb{E}
_{\mathbf{X}}[\|\mathbf{Y}\|_{2}]}$, where $\mathbf{X}$ is the input, $\hat{\mathbf{Y}}$ is the predicted output and $\mathbf{Y}$ is the ground truth output. We plot this error in Figure 10, along with empirical errors on $\mathbf{W}_{\text{up,down}}$ in Block 16 of Llama-3-8B, and the theoretical error obtained from random sparsification.

###### Definition A.1 .

For a random vector $\mathbf{X}=(X_{1},\dots,X_{n})$ and sparsity level $p\in[0,1]$, define the threshold $t_{p}$ as

$$ $\frac{1}{n}\sum_{i=1}^{n}\mathbb{P}(|X_{i}|\leq t_{p})=p.$ $$

The sparsification function $s_{t_{p}}:\mathbb{R}^{n}\to\mathbb{R}^{n}$ is defined as:

$$ $s_{t_{p}}(\mathbf{X})=(s_{t_{p}}(X_{1}),\ldots,s_{t_{p}}(X_{n}))$ $$

where for each component:

$$ $s_{t_{p}}(X_{i})=\begin{cases}0&\text{if }|X_{i}|\leq t_{p}\\ X_{i}&\text{otherwise}\end{cases}$ $$

###### Lemma A.1 (Variance of Scalar Sparsified Error) .

For independent random normal variables $X\sim N(0,\sigma_{X}^{2}),W\sim N(0,\sigma^{2}_{W})$ and sparsification function $s_{t_{p}}(\cdot)$, the variance of $(X-s_{t_{p}}(X))W$ is given by:

$$ $\mathrm{Var}((X-s_{t_{p}}(X))W)=\sigma_{X}^{2}\sigma_{W}^{2}\Big{[}p-\frac{2t_ {p}}{\sigma_{X}}\varphi(\frac{t_{p}}{\sigma_{X}})\Big{]}$ $$

where $\varphi(t)=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}t^{2}}$ is the probability density function of the standard normal distribution.

###### Proof.

For $|x|\leq t_{p}$, $X-s_{t_{p}}(X)$ follows a truncated normal distribution with lower bound and upper bound given by $-t_{p}$ and $t_{p}$ respectively. We thus have:

$$ $\mathrm{Var}((X-s_{t_{p}}(X))W)=p\mathrm{Var}(X-s_{t_{p}}(X)\,\big{|}\,\lvert X \rvert\leq t_{p})\mathrm{Var}(W)$ $$

$$ $\displaystyle=\sigma_{X}^{2}p\Bigg{[}1-\frac{t\varphi(t)-(-t)\varphi(-t)}{\Phi (t)-\Phi(-t)}-\bigg{(}\frac{\varphi(t)-\varphi(-t)}{\Phi(t)-\Phi(-t)}\bigg{)}^ {2}\Bigg{]}\sigma_{W}^{2}$ $\displaystyle=\sigma_{X}^{2}\sigma_{W}^{2}p\bigg{[}1-\frac{2t\varphi(t)}{2\Phi (t)-1}\bigg{]}$ $\displaystyle=\sigma_{X}^{2}\sigma_{W}^{2}\bigg{[}p-\frac{2t_{p}}{\sigma_{X}} \varphi\Big{(}\frac{t_{p}}{\sigma_{X}}\Big{)}\bigg{]}$ $$

where $t=\frac{t_{p}}{\sigma_{X}}$, and $\Phi(t)=\frac{1}{2}(1+\mathrm{erf}(x/\sqrt{2}))$ is the cumulative density function of the standard normal distribution.
∎

###### Lemma A.2 (Expected $\ell_{2}$ Norm of Sparsified Matrix-Vector Error) .

Let $\mathbf{X}\in\mathbb{R}^{m}$ be a vector where each $X_{i}\sim N(0,\sigma_{X}^{2})$, and $\mathbf{W}\in\mathbb{R}^{n\times m}$ be a matrix where each $W_{ji}\sim N(0,\sigma_{W}^{2})$, with all entries independent. For a sparsification function $s_{t_{p}}(\cdot)$, let $\hat{\mathbf{Y}}=(\mathbf{X}-s_{t_{p}}(\mathbf{X}))\mathbf{W}^{\top}$. Then:

1) The variance of the $j$-th entry of $\hat{\mathbf{Y}}$ is:

$$ $\text{Var}(\hat{Y}_{j})=n\sigma_{X}^{2}\sigma_{W}^{2}\left[p-\frac{2t_{p}}{ \sigma_{X}}\varphi\left(\frac{t_{p}}{\sigma_{X}}\right)\right]$ $$

2) The expectation of the $\ell_{2}$ norm of $\hat{\mathbf{Y}}$ is:

$$ $\mathbb{E}[\|\mathbf{\hat{Y}}\|_{2}]=\sigma_{X}\sigma_{W}\sqrt{mn\left[p-\frac {2t_{p}}{\sigma_{X}}\varphi\left(\frac{t_{p}}{\sigma_{X}}\right)\right]}$ $$

where $t_{p}$ is the threshold value satisfying $F_{|X|}(t_{p})=p$, and $\varphi(t)=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}t^{2}}$ is the probability density function of the standard normal distribution.

###### Proof.

For the variance of $\hat{Y}_{j}$:
The $j$-th entry of $\hat{\mathbf{Y}}$ is the sum of $n$ independent products $(X_{i}-s_{t_{p}}(X_{i}))W_{ji}$. Each product has variance $\sigma_{X}^{2}\sigma_{W}^{2}[p-\frac{2t_{p}}{\sigma_{X}p}\varphi(\frac{t_{p}}{
\sigma_{X}})]$. Since variances of independent terms add, we multiply this by $n$ to get the result.

For the expectation of $\|\hat{\mathbf{Y}}\|_{2}$:
We first show $\text{Cov}(Y_{j},Y_{k})=0$ for $j\neq k$:

$$ $\mathbb{E}[\hat{Y}_{j}\hat{Y}_{k}]=\mathbb{E}\left[\sum_{i=1}^{n}(X_{i}-s_{t_{ p}}(X_{i}))^{2}W_{ji}W_{ki}\right]=0$ $$

$\mathbb{E}[W_{ji}W_{ki}]=0$ for $j\neq k$ due to independence and zero mean. $\hat{Y}_{j}$ and $\hat{Y}_{k}$ are uncorrelated for $j\neq k$ and are thus independent. Therefore:

$$ $\mathbb{E}[\|\mathbf{\hat{Y}}\|^{2}]=\mathbb{E}\left[\sum_{j=1}^{m}\hat{Y}_{j} ^{2}\right]=\sum_{j=1}^{m}\mathbb{E}[\hat{Y}_{j}^{2}]=\sum_{j=1}^{m}\text{Var} (\hat{Y}_{j})$ $$

Substituting the variance from part 1, summing over $m$ components, and taking the square-root completes the proof.
∎

###### Theorem A.1 (Distributional Relative Error) .

Let $\mathbf{X}\in\mathbb{R}^{m}$ and $\mathbf{W}\in\mathbb{R}^{n\times m}$ with elements independently drawn from $N(0,\sigma_{X}^{2})$ and $N(0,\sigma_{W}^{2})$ respectively. For a sparsification function $s_{t_{p}}(\cdot)$, define $\hat{\mathbf{Y}}=s_{t_{p}}(\mathbf{X})\mathbf{W}^{\top}$ and $\mathbf{Y}=\mathbf{X}\mathbf{W}^{T}$. The distributional relative error is given by:

$$ $\frac{\mathbb{E}_{\mathbf{X}}[\|\mathbf{Y}-\hat{\mathbf{Y}}\|_{2}]}{\mathbb{E} _{\mathbf{X}}[\|\mathbf{Y}\|_{2}]}=\sqrt{p-\frac{2t_{p}}{\sigma_{X}}\varphi \left(\frac{t_{p}}{\sigma_{X}}\right)}$ $$

where $\varphi(t)=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}t^{2}}$ is the standard normal probabilty density function.

###### Proof.

From the previous theorem, we have $\mathbb{E}_{\mathbf{X}}[\|\mathbf{Y}-\hat{\mathbf{Y}}\|_{2}]=\sigma_{X}\sigma_
{W}\sqrt{mn\left[p-\frac{2t_{p}}{\sigma_{X}}\varphi(\frac{t_{p}}{\sigma_{X}})
\right]}$.

For the unsparsified case, we have $\mathbb{E}_{\mathbf{X}}[\|\mathbf{Y}\|_{2}]=\sigma_{X}\sigma_{W}\sqrt{mn}$. Dividing these expectations yields the result.
∎

###### Definition A.1 .

###### Lemma A.1 (Variance of Scalar Sparsified Error) .

###### Proof.

###### Lemma A.2 (Expected ℓ 2 subscript ℓ 2 \ell_{2} roman_ℓ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT Norm of Sparsified Matrix-Vector Error) .

###### Proof.

###### Theorem A.1 (Distributional Relative Error) .

###### Proof.

### A.2 Full Downstream Task Results

We provide the full downstream task results for all evaluated models. For Llama-3-8B in Table 4, we also provide results obtained from the uniform sparsity configuration, showing that the greedy sparsity configuration outperforms across the board. For CATS, we additionally provide the sparsity of the hidden state in the intermediate of the MLP blocks.

**Table 4: Full downstream task results for Llama-3-8B.**
| Method | Sparsity | MMLU | ARC | HellaSwag | GSM8K | PiQA | WinoGrande | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline | 0 | 65.08 | 57.68 | 82.20 | 49.81 | 80.79 | 72.85 | 68.07 |
| Uniform | 25 | 64.48 | 57.34 | 81.63 | 48.52 | 79.92 | 73.88 | 67.63 |
| 40 | 62.35 | 54.86 | 79.98 | 43.59 | 79.11 | 72.22 | 65.35 |  |
| 50 | 59.07 | 53.50 | 77.60 | 36.47 | 79.22 | 70.17 | 62.67 |  |
| 65 | 43.48 | 42.58 | 65.81 | 16.22 | 75.84 | 65.43 | 51.56 |  |
| Greedy | 25 | 64.61 | 57.17 | 81.77 | 48.90 | 80.47 | 73.48 | 67.73 |
| 40 | 62.69 | 57.08 | 80.42 | 43.82 | 80.47 | 72.77 | 66.21 |  |
| 50 | 59.68 | 53.84 | 78.44 | 38.06 | 78.94 | 71.59 | 63.42 |  |
| 65 | 44.54 | 43.86 | 68.85 | 17.51 | 77.15 | 66.06 | 52.99 |  |
| CATS | 25 (46.45) | 61.18 | 54.61 | 80.20 | 39.88 | 79.76 | 69.30 | 64.15 |

**Table 5: Full downstream task results for Llama-3-70B.**
| Method | Sparsity | MMLU | ARC | HellaSwag | GSM8K | PiQA | WinoGrande | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline | 0 | 78.70 | 69.71 | 87.94 | 81.12 | 84.55 | 80.43 | 80.41 |
| Greedy | 25 | 78.44 | 69.37 | 87.83 | 80.36 | 84.55 | 80.74 | 80.22 |
| 40 | 77.92 | 68.52 | 87.35 | 78.77 | 83.79 | 79.40 | 79.29 |  |
| 50 | 76.48 | 67.75 | 86.73 | 78.17 | 83.08 | 77.35 | 78.26 |  |
| 65 | 71.39 | 63.31 | 83.71 | 64.44 | 80.74 | 74.82 | 73.07 |  |
| CATS | 25 (45.54) | 77.67 | 67.92 | 87.75 | 78.01 | 84.27 | 79.87 | 79.25 |

**Table 6: Full downstream task results for Llama-2-7B.**
| Method | Sparsity | MMLU | ARC | HellaSwag | GSM8K | PiQA | WinoGrande | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline | 0 | 45.78 | 52.47 | 78.96 | 13.95 | 78.94 | 68.90 | 56.50 |
| Greedy | 25 | 45.34 | 52.56 | 78.66 | 14.25 | 78.78 | 68.90 | 56.42 |
| 40 | 42.81 | 53.16 | 78.28 | 12.66 | 78.40 | 67.40 | 55.45 |  |
| 50 | 40.52 | 52.47 | 76.54 | 10.84 | 77.86 | 67.32 | 54.26 |  |
| 65 | 31.63 | 42.83 | 69.64 | 4.62 | 76.55 | 63.69 | 48.16 |  |
| CATS | 25 (56.2) | 42.05 | 52.39 | 78.20 | 10.24 | 77.64 | 67.09 | 54.60 |

**Table 7: Full downstream task results for Llama-2-13B.**
| Method | Sparsity | MMLU | ARC | HellaSwag | GSM8K | PiQA | WinoGrande | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline | 0 | 54.76 | 59.39 | 82.18 | 23.05 | 71.98 | 80.69 | 62.01 |
| Greedy | 25 | 54.96 | 59.47 | 82.31 | 23.58 | 72.14 | 80.79 | 62.21 |
| 40 | 54.15 | 58.02 | 82.11 | 22.44 | 70.72 | 80.20 | 61.27 |  |
| 50 | 52.13 | 57.85 | 81.38 | 19.71 | 71.11 | 80.25 | 60.41 |  |
| 65 | 44.37 | 53.24 | 76.95 | 12.81 | 68.59 | 78.29 | 55.71 |  |
| CATS | 25 (56.02) | 52.31 | 57.76 | 82.73 | 20.32 | 70.24 | 79.49 | 60.48 |

**Table 8: Full downstream task results for Llama-2-70B.**
| Method | Sparsity | MMLU | ARC | HellaSwag | GSM8K | PiQA | WinoGrande | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline | 0 | 68.71 | 67.49 | 87.02 | 52.38 | 82.70 | 77.58 | 72.65 |
| Greedy | 25 | 68.80 | 67.24 | 86.93 | 52.38 | 82.70 | 77.98 | 72.67 |
| 40 | 67.75 | 66.81 | 86.88 | 53.90 | 82.48 | 77.58 | 72.57 |  |
| 50 | 66.79 | 66.72 | 86.38 | 52.69 | 82.59 | 76.95 | 72.02 |  |
| 65 | 62.75 | 63.74 | 84.96 | 45.41 | 81.72 | 77.19 | 69.30 |  |
| CATS | 25 (45.54) | 67.45 | 66.81 | 86.96 | 50.95 | 82.92 | 76.48 | 71.93 |

**Table 9: Full downstream task results for Mistral 7B.**
| Method | Sparsity | MMLU | ARC | HellaSwag | GSM8K | PiQA | WinoGrande | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline | 0 | 62.46 | 61.43 | 83.47 | 38.21 | 82.10 | 74.11 | 66.96 |
| Greedy | 25 | 62.02 | 61.52 | 83.35 | 37.53 | 81.77 | 73.56 | 66.63 |
| 40 | 61.00 | 60.84 | 82.65 | 33.89 | 81.28 | 73.09 | 65.46 |  |
| 50 | 59.02 | 59.90 | 81.38 | 31.62 | 81.28 | 71.74 | 64.16 |  |
| 65 | 51.92 | 54.01 | 76.79 | 21.01 | 80.41 | 69.46 | 58.93 |  |
| CATS | 25 (46.45) | 60.10 | 59.81 | 82.08 | 29.72 | 80.41 | 73.40 | 64.25 |

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: **1. Calibration Set vs. Evaluation Set for Pruning Decisions:**
 - If I understand correctly, the decision to prune specific activations in TEAL is based on a calibration set, with the assumption that this set is representative enough for generalization. However, is it possible that an activation deemed unimportant (and thus pruned) based on the calibration set might behave differently on a different dataset?
 - It appears that C4 is used as the calibration set, while WikiText is used for evaluation. How similar are these datasets in structure and distribution? If you were to repeat the experiment on a different evaluation dataset, would you still observe the same levels of performance degradation?

**2. Pruning Effectiveness in Highly Optimized Newer Models:**

- In line 298, could it be that newer models are using their capacity more fully, meaning that fewer activations are redundant and available for pruning? If so, does this suggest that newer architectures could achieve comparable complexity with fewer parameters, making further compression both unnecessary and less effective?
- In this scenario, how would TEAL still provide an advantage? Could you explain the benefits of TEAL for cases where newer models are already highly optimized?
soundness: 3
strengths: - TEAL achieves high sparsity (40-50%) without retraining, resulting in faster execution (up to 1.8× speed-up) with minimal performance degradation.

- A range of experiments across various models and model sizes shows that TEAL consistently outperforms other methods, demonstrating its robustness and general applicability

- The proposed method is designed to work seamlessly with newer models that use modern activation functions like SwiGLU, making it applicable to the latest LLM architectures.
summary: The authors introduce TEAL, a training-free method for inducing activation sparsity in LLMs, achieving 40-50% sparsity with minimal performance loss and up to 1.8× speed-up in decoding. Unlike previous methods, TEAL works across all model layers, supports modern activation functions, and is compatible with quantization for additional efficiency.
weaknesses: - Although TEAL is adapted for compatibility with newer activation functions like SwiGLU, the core concept of magnitude-based pruning is similar to prior work, which may limit its perceived innovation and contribution.
- The paper does not mention whether the code is available for reproducibility.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Line 107: Which recent work?

2. Could the authors please explicitly explain why activation sparsity error in figure 8 increases with the batch size? Does the algorithm nullify the activations independently across different samples of a batch or activations are zeroed according to some aggregate statistics?

3. It is mentioned in the paper that the efficient low-level Triton kernel is implemented based on Deja vu kernel [1]. Is it possible to apply that kernel directly to the task of sparsifying SwiGLU and GeGLU transformers, and if it is, how it fares against the proposed method in terms of quality?

4. The paper seems to imply but not to explicitly state that the result of (SiLU(xWgate ) ⊙ xWup is also sparsified. Are there any other places where activation sparsity is applied but not mentioned? 


[1] Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, and Beidi Chen. Deja vu: Contextual sparsity for efficient llms at inference time, 2023. URL https://arxiv.org/abs/2310.17157.
soundness: 4
strengths: * The paper successively addresses an important and relevant applied  problem: how to speed up inference of **contemporary** and widely used architectures of LLMs such as Llama model family with no or minimal quality loss. 

* Suggested solution is conceptually simple, and authors supply custom Triton kernels which makes their method suitable for easy adoption.

* The paper is clear, legible and easy to follow. The results are solid, and the contribution is sound.

* The authors analyze and visualize activation distributions for recent and largely popular Llama-3-8B architecture which gives an intuitive ground for justifying not only their own online training-free activation sparsity, but also various other existing and future algorithms.

* The authors apply sparsity to all weight matrices in a Transformer block as compared to competing approach (CATS, [1]). TEAL induces greater overall sparsity with lesser degradation of performance. 

* The method is shown to work well in practice in conjunction with another extremely popular approach to accelerating the inference of LLMs: weight quantization.

[1] Donghyun Lee, Jaeyong Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini. CATS: Context- aware thresholding for sparsity in large language models. In First Conference on Language Modeling, 2024.
summary: The paper proposes a novel sparsification method called TEAL (Training-Free Activation Sparsity in LLMs) which aims to boost up inference speeds of a contemporary LLMs. It zeros out p-th layer-wise percentile of activations closest to 0 in all of the linear projection layers of the Transformer block. Specifically, it is catered to architectures with GeLU, SwiGLU and GeGLU activations (e.g. Llama and Mistral) which do not exhibit large level of natural sparseness, as opposed to older model families (e.g. GPT and OPT) with high sparseness induced by ReLU. The approach doesn't demand extensive fine-tuning or continual pre-training, as the sparsification is applied in online fashion to activations of the model during each forward pass. Only inexpensive greedy search over limited number of hyper-parameters is required to adapt the technique to a model.

The authors conduct experiments with several model families of different sizes, benchmarking speed, perplexity and reasoning abilities of models augmented with different degrees of TEAL activation sparsity. The results indicate that the method substantially increases inference  speeds while retaining good modeling and reasoning abilities of the models.
weaknesses: 1. It would be nice to include formal definitions to various matrices (such as $W_{up}$ and $W_{down}$) and layers (MLP, SwiGLU) which are referred to throughout the work.

2. There are no comparisons with structured and semi-structured pruning (e.g. Mask-LLM [1] which was originally evaluated on the same benchmarks with Llama-2 models). Overall, it would be intriguing and instructive to compare differences in quality/ speed, and trade-offs between the TEAL and unstructured/ semi-structured/ structured pruning, and active sparsity inducing algorithms.

3. In particular, comparison with fine-tuning free ReLUfication [2] seems to be a little bit unfair to the spirit of the method as it’s intended by the paper’s authors to be used with fine-tuning or continual pretraining. It would be nice to either completely discard this comparison in Table 1 or to replace it with a stronger baseline, such as full ReLUfication with fine-tuning.

[1] Fang, G., Yin, H., Muralidharan, S., Heinrich, G., Pool, J., Kautz, J., Molchanov, P., & Wang, X. (2024). MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models. ArXiv, abs/2409.17481.
[2] Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models, 2023. URL https://arxiv.org/abs/2310.04564.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: - In Figure 3 (right), why is the theoretical Latency higher than TEAL at small sparsity levels?
soundness: 3
strengths: - The paper provides a high-quality exploration of sparsity patterns in SiLU-like activations, which are widely used in modern LLMs.
- The greedy search-based solution presented is straightforward yet effective.
- The authors demonstrate the accuracy results on several common downstream tasks at various scales LLMs, and the end-to-end throughput experiments show the practical value of TEAL in certain scenarios.
summary: This paper introduces TEAL, a novel magnitude-based pruning approach to achieve activation sparsity in Large Language Models (LLMs). It focuses on LLaMA-architecture LLMs with SiLU-like activation functions, identifying activation sparsity patterns through a greedy search method that does not require continued pre-training. The experiments demonstrate that TEAL achieves comparable performance on both accuracy and inference speed.
weaknesses: - While the main focus is on Llama-like LLMs, discussing the applicability of TEAL to other architectures, such as [Mixture-of-Experts](https://arxiv.org/abs/1701.06538) and [Mamba](https://arxiv.org/abs/2312.00752), would enhance the paper's scope.
- The paper lacks a detailed analysis of the memory footprint of the TEAL method, especially in long context scenarios.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Are there any limitations or difficulties caused by TEAL when it comes to training or fine-tuning the sparsified model?
2. Does the Triton kernel need to be rewritten for different GPU architectures?
3. In Table 3, how was the inference speed tested? Was it tested on autoregressive text generation? If so, what are the lengths of the given and the generated text?
4. Does the sparsity level change based on the domain of the text (e.g., code, novel, scientific paper, etc.)?
soundness: 3
strengths: 1. The paper is well-written and easy to follow. The challenge is stated clearly and addressed properly.
2. The proposed method achieves meaningful progress on the activation sparsification of the non-ReLU functions in the training-free setting.
3. The sparsified models show actual throughput improvement on the widely used GPUs. Overall, the proposed scheme seems quite practical and can be easily adopted by others.
summary: The paper proposes TEAL, a training-free activation sparsification technique for Large Language Model (LLM) acceleration. Modern LLMs adopt more sophisticated activation functions in lieu of ReLU, thus their activations are no longer naturally sparse. TEAL is built upon the observation that the distribution of the inputs to the weight matrices closely follows either a Gaussian or Laplace distribution. The magnitude based sparsification follows, where the threshold is chosen by a block-level greedy optimization process based on the $\ell_2$ error. The whole process does not involve back propagation. The customized GPU kernel attributes the acceleration of LLMs on the NVIDIA A6000 and A100 GPUs.
weaknesses: A couple of minor concerns are listed below. 
1. Although I understand that the paper focuses on the training-free scheme, providing a discussion and/or experimental results on fine-tuned sparse activation models could make the paper stronger and more interesting. Since a couple of baseline papers (CATS and Relufication) adopt training or fine-tuning steps in their pipelines, I am curious about how much improvement a simple fine-tuning method, such as parameter-efficient fine-tuning (e.g., LoRA), using a moderate number of tokens (possibly <100K), can bring to TEAL. If post-sparsification fine-tuning is prohibited due to the attributes of TEAL, this should be discussed in the paper.
2. The standard deviation of the sparsity by TEAL is not reported. Since the sparsity level changes from input to input, providing more statistical information can help in understanding the proposed method better.

Meta-review: Dear Authors,

Thank you for your valuable contribution to the ICLR and the ML community. Your submitted paper has undergone a rigorous review process, and I have carefully read and considered the feedback provided by the reviewers.

This paper proposes a training-free method for achieving activation sparsity in large language models, achieving around 50% sparsity with minimal performance loss. This leads to up to 1.8× speed-up in decoding.  Overall, the paper received mostly positive response from the reviewers (8,8,8,6) scores.

Given this positive assessment, I am willing to recommend the acceptance of your paper for publication.

I would like to remind you to carefully review the reviewer feedback and the resulting discussion. While most reviews were positive, the reviewers have offered valuable suggestions that can further strengthen the quality of the paper. Please take another careful look a the 'weaknesses' section of each reviewer comment. I encourage you to use this feedback to make any necessary improvements and refinements before submitting the final version of your paper.

Once again, thank you for submitting your work to ICLR.

Best,
Area Chair

Decision: Accept (Spotlight)