Sparsing Law: Towards Large Language Models with Greater Activation Sparsity

Authors: Yuqi Luo, Chenyang Song, Xu Han, Yingfa Chen, Chaojun Xiao, Zhiyuan Liu, Maosong Sun
Primary area: foundation or frontier models, including LLMs
Keywords: activation sparsity, large language model

Abstract:
Activation sparsity denotes the existence of substantial weakly-contributed elements within activation outputs that can be eliminated, benefiting many important applications concerned with large language models (LLMs), such as computation acceleration and model interpretability. Although promoting greater activation sparsity within LLMs deserves deep studies, existing works lack comprehensive and quantitative research on the correlation between activation sparsity and potentially influential factors. In this paper, we present a comprehensive study on the quantitative scaling properties and influential factors of the activation sparsity within decoder-only Transformer-based LLMs. Specifically, we propose PPL-$p\%$ sparsity, a precise and performance-aware activation sparsity metric that is applicable to any activation function. Through extensive experiments, we find several important phenomena. Firstly, different activation functions (i.e., ReLU and SiLU) exhibit comparable performance but opposite training-time sparsity trends. The activation ratio (i.e., $1-\mathrm{sparsity\ ratio}$) evolves as a convergent increasing power-law and decreasing logspace power-law with the amount of training data for SiLU-activated and ReLU-activated LLMs, respectively. These demonstrate that ReLU is more efficient as the activation function than SiLU and can leverage more training data to improve activation sparsity. Secondly, the activation ratio linearly increases with the width-depth ratio below a certain bottleneck point, indicating the potential advantage of a deeper architecture at a fixed parameter scale. Finally, at similar width-depth ratios, we surprisingly find that the limit value of activation sparsity varies weakly with the parameter scale, i.e., the activation patterns within LLMs are insensitive to the parameter scale. These empirical laws towards LLMs with greater activation sparsity have important implications for making LLMs more efficient and interpretable.

Decision: reject
Reviewer scores: 5, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729245446001

thread_type:
official_review

<thread_object>
creator: reviewer_96rd

<content>
summary:
The paper identifies architectural choices that influence activation sparsity, which is measured by there own metric. They find that the ReLU activation function promotes sparsity more than SiLU, and propose relations that model activation sparsity as a function of training data. The influence of model depth and width is also analysed.

strengths:
1) The paper raises an interesting question, that is: when is a model sparse?

2) The paper points out potential influences that can lead to sparsity.

weaknesses:
1) The paper claims that creating sparse LLMs is of broad interest to the community, but the cited papers are mainly interested in removing sparse parts of the network to speed up e.g. inference of large models. The question of where sparse networks perform worse/better, or how the modifications in general affect performance, is not adequately addressed. 

2) The paper claims "empirical laws", but they are not sufficiently motivated and validated. Even if the parameters of a model can be determined empirically, the generality of the results must be questioned when a function with four parameters (Eq. 4) is fitted to a curve. 

3) One of the key results of the paper is that ReLU produces sparser networks than SiLU, which is not at all surprising given the "dying ReLU" problem.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660215744

thread_type:
official_review

<thread_object>
creator: reviewer_aag2

<content>
summary:
This paper addresses activation sparsity in large language models (LLMs), where a significant portion of elements in activation outputs have minimal contributions to the final output. The authors aim to enhance activation sparsity to improve computational efficiency, interpretability, and training dynamics. They propose a new metric, PPL-p% sparsity, which is performance-aware and adaptable across various architectures. Through extensive experiments, the paper studies factors influencing activation sparsity, including activation functions, width-depth ratios, and parameter scales, revealing patterns and scaling properties that can inform the design and training of efficient, sparse LLMs.

strengths:
1. Novel Metric for Performance-Aware Sparsity: The introduction of PPL-p% sparsity is a key contribution, offering a more performance-sensitive measurement of activation sparsity that is adaptable across model architectures. This metric brings a practical perspective to sparsity evaluation in LLMs.
2. Insightful Scaling Laws for Activation Sparsity: The proposed scaling laws establish patterns in activation sparsity across varying model parameters, helping to guide model design for optimized efficiency and sparsity. These scaling laws are particularly valuable for practitioners focused on resource-efficient model scaling.
3. Practical Design Guidelines for Sparse LLMs: The paper’s findings offer actionable design guidelines, such as optimal width-depth ratios, for promoting activation sparsity without compromising model performance. This provides a practical framework for building more efficient LLMs.

weaknesses:
1. Limited Exploration of Sparsity Effects on Downstream Tasks: While the paper extensively analyzes activation sparsity during training, it lacks exploration of how increased sparsity impacts downstream task performance. This leaves uncertainty about whether the sparsity benefits hold in real-world applications.
2. Inconsistency in Performance Across Different Width-Depth Ratios: Although the paper highlights optimal width-depth ratios, it lacks a detailed examination of potential performance trade-offs when deviating from these ratios. This makes it difficult to understand the flexibility of the proposed guidelines for diverse architectures.
3. Scalability Concerns for Very Large Models: The paper’s experiments are conducted on specific model scales, but it is unclear how well the findings scale to extremely large models (e.g., hundreds of billions of parameters). Additional validation on larger LLMs would enhance the paper’s applicability to state-of-the-art models.
4. Unverified Claims on Sparsity Ratios and Performance Degradation: The paper mentions that overly high or low sparsity ratios may lead to severe performance degradation or unnecessary computation. However, this claim lacks experimental validation. Empirically, in many kinds of MoE models, higher sparsity levels generally correlate with improved performance, which contradicts the paper’s statement.
5. Insufficient Training Tokens and Overreliance on Predicted Curves: The experiments use an insufficient number of training tokens, relying heavily on predicted scaling curves (e.g., Figure 4). For larger models (e.g., 0.8B and above), training up to 200B tokens is typically required to observe convergence. The lack of experiments on larger scales raises doubts about the reliability of the proposed scaling laws and their applicability to state-of-the-art models.
6. Limited Practicality in Reducing Computational Load: The proposed method may face challenges in reducing computational load in real applications. Empirically, different tokens activate different channels, making it difficult to apply a uniform activation pattern across all tokens. Since the method relies on precomputed PPL and activation patterns, these patterns may not generalize well to other tokens. In extreme cases, this would require all tokens to activate all channels to achieve their unique activation pattern, negating the intended efficiency gains.

questions:
1. Applicability of Scaling Laws to Extremely Large Models: The current experiments are conducted on a specific range of model scales. Do the authors plan to validate the proposed scaling laws on much larger models (e.g., hundreds of billions of parameters)? Such validation would enhance the reliability of the scaling laws for the latest state-of-the-art LLMs.
2.  Memory and Computational Efficiency Gains: Could the authors provide quantitative results on memory and computational efficiency improvements achieved through increased activation sparsity? Detailed comparisons would strengthen the practical impact of promoting sparsity in LLMs.
3. Broader Exploration of Activation Functions: The paper mainly discusses ReLU and SiLU activations. Have the authors considered examining other commonly used activation functions, such as GELU and Swish? Exploring a broader range of activations could help generalize the findings to a wider variety of LLM architectures
4.  Scaling Law Reliability with Limited Training Tokens: Given the limited number of training tokens used in the experiments, could the authors discuss the potential impact of this on the accuracy of the scaling laws? Would they consider conducting larger-scale experiments, ideally with 200B tokens for models above 0.8B parameters, to validate these scaling patterns more robustly?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693432164

thread_type:
official_review

<thread_object>
creator: reviewer_mqgb

<content>
summary:
This paper aims to analyze the relationship between activation sparsity and various other features of transformer-based LLMs. The authors introduce a novel metric, PPL-p% sparsity, based on a previous metric called CETT, that identifies a sparsity level at which perplexity is only increased by p% relative to a dense baseline. They study how several features relate to sparsity: amount of training data, choice of activation function, width-depth ratio of the network, and parameter scale. There are several findings, including that ReLU networks can achieve lower sparsity ratios than SiLU networks at the same performance level and convergence rates of activation (1-sparsity) ratios as the aforementioned features are varied.

strengths:
- This paper introduces a new metric, PPL-p%, that builds upon a previous metric, CETT, by finding a sparsity level for a desired perplexity score. 
- The paper demonstrates that using PPL-p% as a metric for measuring activation ratio resuilts in a lower perplexity score relative to other metrics for measuring sparsity. 
- The set of analyses are interesting and provide some valuable, albeit limited, insights into the behavior of some LLMs.

weaknesses:
- I believe that there is not enough evidence showing that PPL-p% is a better metric. The main comparison point between PPL-p% and other metrics evalutates the methods on perplexity of the resultant model. This seems a bit like the metric is simply overfitting to the downstream evaluation criterion. How do each of these methods do on other tasks such as the commonsense reasoning and reading comprehension tasks?
- A significant part of this paper involved identifying various relationships between aspects of the model and the activation sparsity. It seems like a reasonable next step would be to create a model that embodies all of these takeaways, ie has an optimal activation function, amount of training data, etc, and show the results of the activation sparsity and downstream performance relative to some baseline. This would demonstrate the practical value of the observations discussed in the paper. 
- It is not clear if the results generalize to other LLMs. It would be nice to see results on other models. 
- The authors state that the goal of this paper is to produce an LLM with greater activation sparsity, but I feel like this question is not quite answered. It seems as if the authors have conducted several (interesting and thorough) ablation studies, but do not tie all of their insights together to produce one most sparse model. 

Overall, this is a decent exploration of some phenomenology around activation ratios in neural networks, but the findings are not comprehensive or cohesive enough to warrant acceptance.

questions:
- I would recommend mentioning some more dataset details in the main paper rather than just “commonsense reasoning” and “reading comprehension”.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730732382712

thread_type:
official_review

<thread_object>
creator: reviewer_6uup

<content>
summary:
This paper presents a study on the activation sparsity in large language models (LLMs), particularly focusing on decoder-only Transformer-based models. The authors propose a new metric called PPL-p% sparsity, which is a performance-aware measure of activation sparsity. The paper contributes to the understanding of how to design and pretrain LLMs for greater activation sparsity, which has implications for efficiency and interpretability.

strengths:
1. The paper introduces a novel metric for measuring activation sparsity (PPL-p% sparsity) and uncovers empirical laws regarding the scaling properties of activation sparsity in LLMs.
2. The paper is well-written, and the findings are presented clearly with the aid of visualizations.
3. The insights gained from this study can inform the design and training of future LLMs, potentially leading to more efficient and interpretable models.

weaknesses:
1. The paper could benefit from additional experiments on larger-scale models (which is more commonly seen in applications) to confirm the generalizability of the findings.
2. The paper compares ReLU and SiLU activation functions, but in practical applications, a variety of activation functions such as SwiGLU may be used. The performance of these activation functions in terms of activation sparsity may differ, affecting the efficiency and performance of the model.

questions:
1. The paper mentions the calculation of PPL-p% using a binary search method, but does the PPL change monotonically with CETT? Is this an assumption, an intuition, or is there a rigorous proof to support this relationship?
2. The paper mentions that activation sparsity can improve the interpretability of models; are there specific examples or methods to demonstrate how this sparsity helps explain the model's decision-making process?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733270339042

thread_type:
official_comment

<thread_object>
creator: area_chair_7tid

<content>
title:
End of reviewer-author discussion phase

comment:
Dear reviewers,

As we near the conclusion of the reviewer-author discussion phase, I wanted to kindly follow up to see if you’ve had a chance to review the author responses on your comments. Could you confirm that you’ve read it and, if needed, update your review and scores accordingly?

Thank you for your time and effort!

Your AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
a) Summary:
The paper presents a comprehensive study of activation sparsity in large language models (LLMs), focusing on decoder-only Transformer architectures. The key claims include:
1. Introduction of PPL-p% sparsity as a new performance-aware metric for measuring activation sparsity
2. Different activation functions (ReLU vs SiLU) show opposite trends in training-time sparsity, with ReLU demonstrating better sparsity properties
3. The activation ratio follows power-law relationships with training data: decreasing for ReLU and increasing for SiLU
4. The activation ratio increases linearly with width-depth ratio below a bottleneck point
5. The limit value of activation sparsity shows weak correlation with model parameter scale

(b) Strengths:
1. Novel metric (PPL-p%) that considers both sparsity and performance impact
2. Extensive empirical analysis across different model scales (0.1B to 2.4B parameters)
3. Clear practical implications for LLM design, particularly regarding activation function choice
4. Well-documented experimental methodology with comprehensive ablation studies
5. Strong theoretical grounding in relating findings to existing work on model scaling

(c) Weaknesses:
1. Limited exploration of extremely large-scale models (>10B parameters)
2. Initial experiments had insufficient training tokens for larger models, though partially addressed during rebuttal
3. Unclear generalization beyond the specific architecture studied (MiniCPM/LLaMA-like models)
4. Some claims about performance impact of sparsity ratios needed stronger empirical validation
5. Limited exploration of practical acceleration benefits in real-world scenarios

(d) Reasons for Decision:
I recommend rejection primarily because:
1. The work, while thorough in its empirical analysis, presents findings that are somewhat incremental rather than transformative
2. The practical impact is limited by the focus on a specific architecture and moderate model scales
3. Some key claims about sparsity's impact on performance and scaling properties need stronger validation
4. The findings, while interesting, don't provide sufficient breakthrough insights to warrant acceptance at ICLR