Training-Free Activation Sparsity in Large Language Models

Authors: James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, Ben Athiwaratkun
Primary area: foundation or frontier models, including LLMs
Keywords: large language models, activation sparsity, efficiency

Abstract:
Activation sparsity can enable practical inference speedups in large language models (LLMs) by reducing the compute and memory-movement required for  matrix multiplications during the forward pass. 
However, existing methods face limitations that inhibit widespread adoption. Some approaches are tailored towards older models with ReLU-based sparsity, while others require extensive continued pre-training on up to hundreds of billions of tokens. 
This paper describes TEAL (**T**raining-Fre**e** **A**ctivation Sparsity in **L**LMs), a simple training-free method that applies magnitude-based activation sparsity to hidden states throughout the entire model. TEAL achieves 40-50\% model-wide sparsity with minimal performance degradation across Llama-2, Llama-3, and Mistral families, with sizes varying from 7B to 70B. We improve existing sparse kernels and demonstrate wall-clock decoding speed-ups of up to 1.53× and 1.8× at 40\% and 50\% model-wide sparsity. TEAL is compatible with weight quantization, enabling further efficiency gains.

Decision: accept (spotlight)
Reviewer scores: 6, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730249603959

thread_type:
official_review

<thread_object>
creator: reviewer_5uwu

<content>
summary:
The paper proposes TEAL, a training-free activation sparsification technique for Large Language Model (LLM) acceleration. Modern LLMs adopt more sophisticated activation functions in lieu of ReLU, thus their activations are no longer naturally sparse. TEAL is built upon the observation that the distribution of the inputs to the weight matrices closely follows either a Gaussian or Laplace distribution. The magnitude based sparsification follows, where the threshold is chosen by a block-level greedy optimization process based on the $\ell_2$ error. The whole process does not involve back propagation. The customized GPU kernel attributes the acceleration of LLMs on the NVIDIA A6000 and A100 GPUs.

strengths:
1. The paper is well-written and easy to follow. The challenge is stated clearly and addressed properly.
2. The proposed method achieves meaningful progress on the activation sparsification of the non-ReLU functions in the training-free setting.
3. The sparsified models show actual throughput improvement on the widely used GPUs. Overall, the proposed scheme seems quite practical and can be easily adopted by others.

weaknesses:
A couple of minor concerns are listed below. 
1. Although I understand that the paper focuses on the training-free scheme, providing a discussion and/or experimental results on fine-tuned sparse activation models could make the paper stronger and more interesting. Since a couple of baseline papers (CATS and Relufication) adopt training or fine-tuning steps in their pipelines, I am curious about how much improvement a simple fine-tuning method, such as parameter-efficient fine-tuning (e.g., LoRA), using a moderate number of tokens (possibly <100K), can bring to TEAL. If post-sparsification fine-tuning is prohibited due to the attributes of TEAL, this should be discussed in the paper.
2. The standard deviation of the sparsity by TEAL is not reported. Since the sparsity level changes from input to input, providing more statistical information can help in understanding the proposed method better.

questions:
1. Are there any limitations or difficulties caused by TEAL when it comes to training or fine-tuning the sparsified model?
2. Does the Triton kernel need to be rewritten for different GPU architectures?
3. In Table 3, how was the inference speed tested? Was it tested on autoregressive text generation? If so, what are the lengths of the given and the generated text?
4. Does the sparsity level change based on the domain of the text (e.g., code, novel, scientific paper, etc.)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730297618588

thread_type:
official_review

<thread_object>
creator: reviewer_yw38

<content>
summary:
This paper introduces TEAL, a novel magnitude-based pruning approach to achieve activation sparsity in Large Language Models (LLMs). It focuses on LLaMA-architecture LLMs with SiLU-like activation functions, identifying activation sparsity patterns through a greedy search method that does not require continued pre-training. The experiments demonstrate that TEAL achieves comparable performance on both accuracy and inference speed.

strengths:
- The paper provides a high-quality exploration of sparsity patterns in SiLU-like activations, which are widely used in modern LLMs.
- The greedy search-based solution presented is straightforward yet effective.
- The authors demonstrate the accuracy results on several common downstream tasks at various scales LLMs, and the end-to-end throughput experiments show the practical value of TEAL in certain scenarios.

weaknesses:
- While the main focus is on Llama-like LLMs, discussing the applicability of TEAL to other architectures, such as [Mixture-of-Experts](https://arxiv.org/abs/1701.06538) and [Mamba](https://arxiv.org/abs/2312.00752), would enhance the paper's scope.
- The paper lacks a detailed analysis of the memory footprint of the TEAL method, especially in long context scenarios.

questions:
- In Figure 3 (right), why is the theoretical Latency higher than TEAL at small sparsity levels?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719734281

thread_type:
official_review

<thread_object>
creator: reviewer_rptk

<content>
summary:
The paper proposes a novel sparsification method called TEAL (Training-Free Activation Sparsity in LLMs) which aims to boost up inference speeds of a contemporary LLMs. It zeros out p-th layer-wise percentile of activations closest to 0 in all of the linear projection layers of the Transformer block. Specifically, it is catered to architectures with GeLU, SwiGLU and GeGLU activations (e.g. Llama and Mistral) which do not exhibit large level of natural sparseness, as opposed to older model families (e.g. GPT and OPT) with high sparseness induced by ReLU. The approach doesn't demand extensive fine-tuning or continual pre-training, as the sparsification is applied in online fashion to activations of the model during each forward pass. Only inexpensive greedy search over limited number of hyper-parameters is required to adapt the technique to a model.

The authors conduct experiments with several model families of different sizes, benchmarking speed, perplexity and reasoning abilities of models augmented with different degrees of TEAL activation sparsity. The results indicate that the method substantially increases inference  speeds while retaining good modeling and reasoning abilities of the models.

strengths:
* The paper successively addresses an important and relevant applied  problem: how to speed up inference of **contemporary** and widely used architectures of LLMs such as Llama model family with no or minimal quality loss. 

* Suggested solution is conceptually simple, and authors supply custom Triton kernels which makes their method suitable for easy adoption.

* The paper is clear, legible and easy to follow. The results are solid, and the contribution is sound.

* The authors analyze and visualize activation distributions for recent and largely popular Llama-3-8B architecture which gives an intuitive ground for justifying not only their own online training-free activation sparsity, but also various other existing and future algorithms.

* The authors apply sparsity to all weight matrices in a Transformer block as compared to competing approach (CATS, [1]). TEAL induces greater overall sparsity with lesser degradation of performance. 

* The method is shown to work well in practice in conjunction with another extremely popular approach to accelerating the inference of LLMs: weight quantization.

[1] Donghyun Lee, Jaeyong Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini. CATS: Context- aware thresholding for sparsity in large language models. In First Conference on Language Modeling, 2024.

weaknesses:
1. It would be nice to include formal definitions to various matrices (such as $W_{up}$ and $W_{down}$) and layers (MLP, SwiGLU) which are referred to throughout the work.

2. There are no comparisons with structured and semi-structured pruning (e.g. Mask-LLM [1] which was originally evaluated on the same benchmarks with Llama-2 models). Overall, it would be intriguing and instructive to compare differences in quality/ speed, and trade-offs between the TEAL and unstructured/ semi-structured/ structured pruning, and active sparsity inducing algorithms.

3. In particular, comparison with fine-tuning free ReLUfication [2] seems to be a little bit unfair to the spirit of the method as it’s intended by the paper’s authors to be used with fine-tuning or continual pretraining. It would be nice to either completely discard this comparison in Table 1 or to replace it with a stronger baseline, such as full ReLUfication with fine-tuning.

[1] Fang, G., Yin, H., Muralidharan, S., Heinrich, G., Pool, J., Kautz, J., Molchanov, P., & Wang, X. (2024). MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models. ArXiv, abs/2409.17481.
[2] Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models, 2023. URL https://arxiv.org/abs/2310.04564.

questions:
1. Line 107: Which recent work?

2. Could the authors please explicitly explain why activation sparsity error in figure 8 increases with the batch size? Does the algorithm nullify the activations independently across different samples of a batch or activations are zeroed according to some aggregate statistics?

3. It is mentioned in the paper that the efficient low-level Triton kernel is implemented based on Deja vu kernel [1]. Is it possible to apply that kernel directly to the task of sparsifying SwiGLU and GeGLU transformers, and if it is, how it fares against the proposed method in terms of quality?

4. The paper seems to imply but not to explicitly state that the result of (SiLU(xWgate ) ⊙ xWup is also sparsified. Are there any other places where activation sparsity is applied but not mentioned? 


[1] Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, and Beidi Chen. Deja vu: Contextual sparsity for efficient llms at inference time, 2023. URL https://arxiv.org/abs/2310.17157.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730730924351

thread_type:
official_review

<thread_object>
creator: reviewer_wbxr

<content>
summary:
The authors introduce TEAL, a training-free method for inducing activation sparsity in LLMs, achieving 40-50% sparsity with minimal performance loss and up to 1.8× speed-up in decoding. Unlike previous methods, TEAL works across all model layers, supports modern activation functions, and is compatible with quantization for additional efficiency.

strengths:
- TEAL achieves high sparsity (40-50%) without retraining, resulting in faster execution (up to 1.8× speed-up) with minimal performance degradation.

- A range of experiments across various models and model sizes shows that TEAL consistently outperforms other methods, demonstrating its robustness and general applicability

- The proposed method is designed to work seamlessly with newer models that use modern activation functions like SwiGLU, making it applicable to the latest LLM architectures.

weaknesses:
- Although TEAL is adapted for compatibility with newer activation functions like SwiGLU, the core concept of magnitude-based pruning is similar to prior work, which may limit its perceived innovation and contribution.
- The paper does not mention whether the code is available for reproducibility.

questions:
**1. Calibration Set vs. Evaluation Set for Pruning Decisions:**
 - If I understand correctly, the decision to prune specific activations in TEAL is based on a calibration set, with the assumption that this set is representative enough for generalization. However, is it possible that an activation deemed unimportant (and thus pruned) based on the calibration set might behave differently on a different dataset?
 - It appears that C4 is used as the calibration set, while WikiText is used for evaluation. How similar are these datasets in structure and distribution? If you were to repeat the experiment on a different evaluation dataset, would you still observe the same levels of performance degradation?

**2. Pruning Effectiveness in Highly Optimized Newer Models:**

- In line 298, could it be that newer models are using their capacity more fully, meaning that fewer activations are redundant and available for pruning? If so, does this suggest that newer architectures could achieve comparable complexity with fewer parameters, making further compression both unnecessary and less effective?
- In this scenario, how would TEAL still provide an advantage? Could you explain the benefits of TEAL for cases where newer models are already highly optimized?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732425178908

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We appreciate that the reviewers share our excitement about TEAL. Their thoughtful feedback has helped strengthen our paper while highlighting the potential opportunities for activation sparsity to accelerate modern LLM inference. Below we briefly recap our paper and detail our revision addressing the reviewer comments.

### **Recap**
We study activation sparsity in modern LLMs, which forgo the use of the sparsity-inducing ReLU function due to empirical reasons. In particular, we discover that hidden states in modern models are **almost sparse**, and realize significant wall-clock speedups by clipping small, low-magnitude activations to zero. We make further optimizations on both modeling (Section 4.3) and kernel (Section Section 4.4) fronts, and demonstrate compatibility with weight quantization. Our method is simple, cheap, and **training-free**, allowing for widespread adoption.

We are grateful that reviewers consistently acknowledged our method's **effectiveness** and **practical impact**, noting that our work:

* Provides a "**clear, legible and easy to follow**" solution to an "**important and relevant applied problem**", with a "straightforward yet effective" approach and "custom Triton kernels which makes their method **suitable for easy adoption**". (rpTk, yw38, 5UWU)
* "**Consistently outperforms other methods**" while demonstrating effectiveness "on several common downstream tasks at various scales". (wbxr, yw38)

* Provides valuable "analysis and visualization of activation distributions" that gives "**intuitive ground for justifying** not only their own online training-free activation sparsity, but also various other existing and future algorithms". (rpTk)
* Shows **significant speedups** through "**actual throughput improvement** on the widely used GPUs" with "up to 1.8× speed-up". (5UWU, wbxr)

### **Revision**

Thanks to reviewer feedback, we have uploaded a revision with updates highlighted in blue. We group these updates into two main themes:

**Writing clarity + presentation**
* We provide formal definitions for each mentioned weight matrix. (L99-101, App. A.3) (rpTk)
* We replace "Recent work" with "CATS". (L108) (rpTk)
* We clarify that sparsity is applied to all four hidden states of each Transformer block. (L189-191) (rpTk)
* We clarify the meaning of “Theoretical Optimal” and how it relates to `torch.matmul`. (L246-247) (yw38)
* We remove the comparison with ReLUfication. (Table 1) (rpTk)
* We specify the number of input and output tokens used in our latency benchmarking setup. (L343-345) (5UWU)

**Expanded experimental analysis**
* We include a comparison to MaskLLM, a state-of-the-art approach to semi-structured 2:4 weight sparsity. TEAL outperforms in both accuracy and single-batch decoding latency. (App. A.4) (rpTk)
* We demonstrate compatibility with fine-tuning. Fine-tuning provides marginal improvements at lower sparsities (50-60%), and more substantial improvements at higher sparsities (70-90%). (App. A.5) (5UWU)
* We add ablations on the calibration set. We find TEAL is fairly insensitive to choice calibration set. (App. A.6) (wbxr)

We thank all reviewers again for their constructive feedback. We are happy to follow up with any additional questions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Dear Authors,

Thank you for your valuable contribution to the ICLR and the ML community. Your submitted paper has undergone a rigorous review process, and I have carefully read and considered the feedback provided by the reviewers.

This paper proposes a training-free method for achieving activation sparsity in large language models, achieving around 50% sparsity with minimal performance loss. This leads to up to 1.8× speed-up in decoding.  Overall, the paper received mostly positive response from the reviewers (8,8,8,6) scores.

Given this positive assessment, I am willing to recommend the acceptance of your paper for publication.

I would like to remind you to carefully review the reviewer feedback and the resulting discussion. While most reviews were positive, the reviewers have offered valuable suggestions that can further strengthen the quality of the paper. Please take another careful look a the 'weaknesses' section of each reviewer comment. I encourage you to use this feedback to make any necessary improvements and refinements before submitting the final version of your paper.

Once again, thank you for submitting your work to ICLR.

Best,
Area Chair