R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference
Authors: Zhenyu Zhang
Zechun Liu
Yuandong Tian
Harshit Khaitan
Zhangyang Wang
Steven Li
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model; Efficient Inference; Activation Sparsity
Primary area: foundation or frontier models, including LLMs

Abstract: Large Language Models (LLMs), while demonstrating remarkable capabilities across various applications, present significant challenges during inference due to their substantial model size, especially when deployed on edge devices. Activation sparsity offers a promising solution to reduce computation and memory movement, enabling more efficient inference, particularly for small-batch on-device applications. However, current approaches face limitations with non-ReLU activation function, which are foundational to most advanced LLMs, or require heavy continual training. Additionally, the difficulty in predicting active channels and limited achievable sparsity ratios constrain the effectiveness of activation sparsity-based methods. In this paper, we introduce R-Sparse, a training-free activation sparsity approach capable of achieving high sparsity levels in advanced LLMs. We conducted two preliminary investigations into how different components contribute to the output within a single linear layer and found two key observations: (i) the non-sparse components of the input function can be regarded as a few bias terms, and (ii) The full computation can be effectively approximated by an appropriate combination of input channels and weight singular values. Building on this, we replace the linear layers in LLMs with a rank-aware sparse inference method that leverages the sparsity of input channels and singular value components, eliminating the need for active channel prediction like the output sparsity based approaches. Experiments on Llama-2/3 and Mistral models across ten diverse tasks demonstrate that R-Sparse achieves comparable performance at 50\% model-level sparsity, resulting in a significant 43\% end-to-end efficient improvements with customized kernels.

Published as a conference paper at ICLR 2025
R-SPARSE: RANK-AWARE ACTIVATION SPARSITY FOR EFFICIENT LLM INFERENCE
Zhenyu Zhang', Zechun Liu', Yuandong Tian', Harshit Khaitan², Zhangyang Wang', , Steven Li 'Tee University of Texas at Austin, MMeta Al zhenyu. chang@utexxe edu, stevenl www.lxx.mmtttmmmmmmmmmmmm..
ABSTRACT
Large Language Models (LLMs), while demonstrating remarkable capabilities across various applications, present significant challenges during inference due their substantial model size, especially when deployed on edge devices. Activation sparsity offers promising solution reduce computation and memory movement, enabling more efficient inference, particularly for small-batch on-device applica- tions. However, current approaches face limitations with non-ReLU activation function, which are foundational to most advanced LLMs, or require heavy contin- ual training. Additionally, the difficulty in predicting active channels and limited achievable sparsity ratios constrain the effectiveness of activation sparsity-base -based methods. In this paper, we introduce S-ppare rse, a training-fres frre activation sparsity approach capable of achieving high sparsity levels in advanced LLMss We con- ducted two preliminary investigations into how different components contribute to the output within single linear layer and found two key observations: (i) the non- sparse components of the input function can be regarded as a few bias terms, and (ii) The full computation can be effectively approximated by appropriate combi- nation of input channels and weight singular values. Building on this, we replace the linear layers in LLMs with rank-aware sparse inference method that leverages the sparsity of input channels and singular value components, eliminating the need for active channel prediction like the output sparsity based approaches. Experi- ments on 2/33 and Mistral models across ten diverse tasks demonstrate that R-Spars r se achieves comparable performance 50% model-level sparsity, result- ing in significant 43% end-to-ent to-d efficient improvements with customized kernels. The code available at wtlhtts:/// /ttp::////ithbb...// wwt.com////cc// /VIA Group/R- Sparse
INTRODUCTION
Large Language Models (LLMs) have become ubiquitous due to their remarkable capabilities, pow- ering applications from virtual assistants to automated content creation. However, their impressive performance comes with significant computational and memory costs due their enormous pa- rameter counts. This poses significant challenges for latency-sensitive applications, particularly for deployments on edge devices. To address this, network pruning or sparsity (Frantar & Alistarh| [023] Sun et al. 2023: Yin et al.) 2023: Ma et al. 2023) is an effective solution. These strategies operate data- indepenedd manner with different levels of pruning granularity, e-g., unstructured, semi- structured, or structured. While more structured pruning approaches leads to more limited sparsity levels, unstructured sparsity introduces greater challenges for efficient hardware implementation.
Recently, activation sparsity (Liu et al. 2023 Mirzadeh et al. 2023 Dong al. 2024 Lee et al. 2024) has emerged as a promising solution that dynamically loads only the active channels and their corresponding weight
or columns from off-chip HBM (NVIDIA] 2020) on-chip SRAM, significantly alleviate the latency and memory cost when equipped with optimized system implemen- tations (Song et al. 2023). Designing activation sparsity functions in structured, data-dependent way, can make the specified network more hardware-fird friendly while also achieving higher sparsity levels compared to traditional pruning techniques.
Despite the promising progress, several challenges remain: (i) Feasibility for non non-ReLU based LLMs: ReLU eliminates the negative part of activations, enabling lossless approximation when skipping

---

Published as a conference paper at ICLR 2025
the computation of corresponding channels (Liu et al. use non- non-ReLU activations like SiLU (Elfwing et al. 2018) and GELU (Hendrycks & Gimpel 2016), which retain small negative values, requirig extensive continual pre-traiming to obtain meaningful activation sparsity (Song et al.] 2024a| Zhang et al. 2024a Mirzadeh et al. 2023 Song et al. 2024b). Such training process can involve up 150B tokens, taking approximately one month on
2023 However, most advanced LLMs now
64 A100 GPUs. (ii) Difficulty in Predicting Active Chamnels: Previous approaches identify critical channels within the hidden activations of MLP blocks, facing significant challenges in predicting the active channels before performing the computation. Common strategies include exploiting the similarity of activated channels across semantically similar tokens (Dong al. 2024). leveraging
the activations after the gate projection (Lee et al. 2024), using a learnable predictor (Liu et al.
2023). while the accuracy of active channel prediction will highly affect their effectiveness. (iii) Limited Sparsity Levels: For approaches that do not rely on extensive retraining (Teee et al. 22244]
Dong et al. 2024), only 50% sparsity within MLP blocks can be achieved, leading to a model-level
sparsity of one third. Achieving higher levels of overall sparsity remains a significant challenge.
This paper targets a training-free activation sparsity approach that is: (i) feasible for non-ReLU based LLMs; (ii) unaffected by the difficulty of predicting active channels; and (iii) capable of achieving higher sparsity levels. While previous methods focus on output activation sparsity (Liu al. 2023 Lee al. 2024), requiring prior prediction of important channels, our approach leverages input activation sparsity, identifying active channels directly from the input without the need of prediction. Furthermore, recent studies (Mirzadeh et al. 2023] Song et al.) 2024a by have shown that directly removing the non-sparse components only achieves limited sparsity while with extensive training, sparsity ratios can be pushed as high as 90%. This sparsity gap raises an natural question: Is the non-sparse portion of the activation truly necessary for maintaining model performance, or can we employ lightweight strategy to mitigate the non-sparse part without resorting to heavy pre-training? Motivated by this, we apply a multi-phase ReLU function to the non-sparse channels, the corresponding activations will then rounded to a few discrete values. As the number of discrete values increases from 1 to 2, performance can be significantly improved, even sparsity level of 90%. The output components associated with the non-sparse portion can then be approximated by a few bias terms, indicating low-rank structure for these components.
To better understand the low-rank structure, we analyze the importance of each input channel in the activations and each singular value component of the weights, to the output activations. As shown in Figure [ we observe a highly sparse structure where an appropriate com- bination of input channels (green rectangle) and singular value components (yellow rect- angle) can effectively approximate the full computation. Building on these, we propose R-Spar Sisparse, a simple yet effective framework that decompose the computation of each lin- ear layer with sparse and low-rank compo- nents. For the sparse portion, our approach identifies sparse channels by selecting those with large magnitude values and loads only the corresponding rows of weights into SRAM for computation. For the low-rank compo- nents, we route the non-sparse channels to a low-rank modules that obtained from an of- fline low-rank decomposition of the original weights. R-Spaz se can be applied to both at- tention and MLP modules that achieves higher sparsity levels. Additionally, we find the pat- terns of sparse and low-rank combinations vary across different layers. With that, we employ an evolutionary search algorithm to identify the optimal ratios for the sparse com- ponents in each layer within LLMs, resulting in enhanced performance.
Layer 0, self attn.v proj
500
0.00020
1000
SVD Index
1500
-0,00015
2000
-.000000
2500
3000
0 00005
3500
00000
1000
2000
3000
4000
Input Channel Index
Figure 1: Contributions of each input channel and singular value components. The measurement metric is detailed in Section 3.3 Results are obtained from Llama-2-7B with 16 training samples from C4. Both the input channel and SVD components are sorted from small to large for better visualization.
2

---

Published as a conference paper at ICLR 2025
We conduct extensive experiments three representative LLM families: Llama-2 Touvron et al. 2023), Llama-3 (Dubey et al.) 2024). and Mistral (Jiagg et al. 2023), across ten tasks, including common-sense reasoning, language modeling, and text summanzattion. Our approach achieves 50% model-level sparsity while maintaining performance comparable to the full model. Additionally, by utilizing a customized kernel, we demonstrate up 43% end-to-end improvements in generation speed. Furthermore, R- Sparse is compatible with weight quantization for further efficiency gains.
2 RELATED WORKS
2.1 EFFICIENT LLM INFERENCE
The inference process of LLMs is typically memory-intensive due to the large number of parameters and the huge KV cache required to store intermediate key and value embeddings. To reduce memory overhead, various strategies have been investigated. including removing redundant components
through pruning or sparsification (Frantar & Alistarh 2023; 2 Sun et al., 2023 Yin et al., 2023 Ma
al. 2023 Zhang al. 2024b) Xiao al.. [023bb] lang
[002g; quantizing data into lower bit
formats (Frantar et al. 2022; Lin et al. 20244 Xiao et al. 2023a Chee et al., 2024 Kim et al. 2023
Egiazarian et 2024 Liu al. 2024b);; and distilling large models into smaller or more efficient
architectures (Bick et al. 2024 Hinton 2015 Sreenivas et al. 2024). Additionally, some approaches
focus on developing efficient architectures (Gu & Dao 2023 Peng et al., 2023 Yang et al., 2023)
or optimizing hardware (Dao et al.) [022] K woo et al. 2023 Alizadeh et al. (023), enhancing the
efficiency of LLM inference and making them more accessible on edge devices. This work focuses
on mitigating the overhead from the large model sizes while compression techniques for KV cache
are orthogonal to weight reduction and can be naturally combined that we will explore in the future.
2.2 ACTIVATION SPARSITY
Several studies have demonstrated that activations within the MLP blocks of transformers are highly sparse (Geva et al. 2020 Li et al. 2022 Dettmers al., 2022). This sparsity primarily arises from ReLU activations, where negative values are zeroed out, providing a natural, lossless opportunity for accelerating inference in LLMs like OPT (Zhang et al. 2022). However, most modern LLMs use activation functions like SiLU or GeLU, which retain small negative values. Directly replacing with the ReLU activation would impair model functionality, To address this challenge, a common strategy is "ReLUfication" where the original activations are replaced with ReLU, followed by extensive continual training to recover performance (Zhang et al., 2202aa] Mirzadch et al., [2033] Song et al. 2024b a). However, this approach introduces significant computational overhead, limiting its accessibility. Recent training-frre frre methods (Lee et al. 20044 Dong et al.) (202)) have made progress in applying sparsity non-ReLU models, achieving modest sparsity ratios (e.g., 50% in MLP blocks and up to 33% model-wide). Additionally, most previous works focus on the sparse structure of output activations, requiring extra effort to identify active channels before computation (Dong et al. 2024 Liu et al., [003] Lee et al. 2024), with the accuracy of channel prediction significantly affecting performance. In our work, we shift the focus to the sparse structure of input channels and singular value components, eliminating the need for active channel prediction while feasible for both attention and MLP blocks, leading to higher sparse ratios without additional training. One concurrent work (Liu al. 2024a) shares similar intuition but focuses solely input channels that can be viewed as a special case our framework.
METHODOLOGY
This section starts from a brief overview of LLM inference and the notations used throughout the paper. Following this, we present two interesting observations: (I) the contribution of non-sparse (i.e., magiitum input channels can be converted into biases, and (II) the full computation can be effectively approximated with an appropriate combination of input channels and singular value components. Motivated by these, we detail our proposed inference framework R- R-Sparse, along with the evolutionary search algorithm for determining the optimal sparsity recipe.
3

---

Published as a conference paper at ICLR 2025
3.1 PRELIMINARY
LLM inference typically consists of two stages: the pre-filling stage, where batch of prompts containing multiple tokens is processed by the model, and @ the decoding stage, where tokens generated incrementally. The decoding phase is often semorrybbundded and its iterative mechanism amplifies the overhead associated with loading parameters into on-chip memory, becoming the main bottleneck during inference. However, activation sparsity mitigates this by enabling the selective loading of only active rows or columns of the weights into SRAM at each decoding stage. In the following, we focus primarily on the decoding phase.
Consider typical LLM architecture, where each block contains seven linear layers. The attention part comprises four matrices: W Ws, W., u, W, Raxx, while the widely used MLP block Touvron al. p: Dubey et al. 2024) includes three matrices: W ap, W gatt E Rmxn and W down R (n and stands for the dimension of model embedding and hidden activations within MLP blocks, respectively). The computational process of the MLP block can be formulated as
Y HWT down. where H XWT up (XX gate).
3.2 MOTIVATION CASE I: NON-SPARSS COMPONENTS ARE BIASES
We first carry out preliminary investigation into how sparsification of input activations influences the final performance. We use soft multi-phase ReLU function (-()) approximate the non-ReLU activation functions (-)), which is defined as:
if 2 To
(T(x = T T++1+1
if S <
where T = (To, "i...,,TTii.. and / determines the softness of the sparsification operation. When To = 0 and = 1, this is equivalent to standard activation sparsity achieved by ReLU where all non-sparse part (æ: < 0) are masking out zero. Note that we define the sparse components as the values that remain unchanged after the activation function, while the sparsity ratio measured as the proportion of values that being changed. Ad- ditionally, we set T as the minimum value input and the sparsity is defined as the ratios of < To. As shown in Figure 2 By simply in- creasing from 2, the degraded performance can be easily recovered, even at sparsity ratio of 90%. Additionally, we use 24 to represent the subset of channels in that satisfy T++1 S Hr < <; (k E UA). The corresponding output Y can then be decomposed into the sparse part Y,, where Hy 2 To. and the residual part Y,, as:
OBIAA
AXC4
203
97.5
25.
Accuracy of Llama-2-7B on Open- BookQA (Mihaylov el al. 2018a) (OBQA) and ARC Challenge (Clark et al. 2018a) (ARC-C) tasks.
Figure 2:
1-2
T
Y, Y-
Σ j=0
Σ WT down [t. kEU
The subset of channels My is input-dependent and each term Ekeu, WTownl I. k) can be viewed as a data-dependent bias By. This allows the non-sparse components to be effectively approximated with a few biases. We will show later how these data-dependent biases can be converted into static biases and being pre-computed. With only two biases, the sparsity ratio is significantly increased to 90%.
3.3 MOTIVATION CASE II: RANK-AWARE ACTIVATION SPARSITY
Although it's costly to obtain the input-dependent biases on the fly. we observe that the space
spanned by the biases across thousands of tokens exhibits low-rank structure, e.g., for each token i, we use two biases to approximate the residual part Y Bg + B By concatenating 4000 biases from 2000 tokens, we obtain bias matrix M, where M 21] B3 and M|; [ 2i
Bj
4

---

Published as a conference paper at ICLR 2025
Figure 3: Importance of each input channel and singular value. Zoom in for better visualization. Results are ohtained with pretrained Llama-2-7B model and samples from the C4 training dataset, each with a sequence length of 4096. Each subfigure corresponds to the results of different layers, with the horizontal axis representing the input channel index and the vertical axis representing the singular value index. The top, middle, and bottom subfigures represent the results of the first, middle, and last layers, respectively.
E (1, 2000)). We find the stable rank of M is approximately 400. Inspired by this, further explore the relationship between weight SVD components and sparse activations. Given pre-trained linear layer Y X WT W n (n
m), we perform singular value decomposition (SVD) on the weight matrix, that W = UEV = E U/U, i [: The output Y can then be expressed as Y = > 1j=1 contribution of the j-th input channel and the i-th SVD components. We collected the distribution of S for Llama-2-7B (Touvron al., 2023) using 16 training samples from the C4 dataset (Dodge et al. a (021), each containing 4096 tokens. For better visualization, both the rows and columns of were sorted independently. Across different linear layers either Attention MLP blocks, the primary contributions are concentrated in the lower-right corner. Additionally, almost all layers exhibit significant sparse property, although some variation exists across layer types and blocks. For instance, the o.proj layer exhibits greater reliance smaller singular values compared to the q.proj and k.proj layers. This observation also aligns with with recent studies (Jaiswal et al. 2024), which demonstrate that proroj and k.proj can be more easily compressed via low-rank approximation. Moreover, middle layers tend to display higher sparsity, while initial and final layers are more difficult to be sparsified, aligning with the general experience that the beginning and final layers of LLMs harder to be compressed (Yin et al. 2023
0,X, Vb. o,X, V.V.U.U"" [ where S == o,X, V jj measures the
3.4 R-SPARSE
Building on the observation of rank-aware activation sparsity, we propose the R-Sparse inference framework. An overview of R-Sparse and its comparison with other techniques is presented in Figure For given score matrix S, previous methods that based on activation sparsity typically remove the left portion of S, while low-rank compression techniques eliminate the upper portion. However, since the most significant components concentrate the bottom-right area, an ideal approach would be to remove the top-left part. To efficiently implement this strategy, decompose the computation Y XWT into two components: the sparse Y, and low-rank Y,.
Sparsifying Input Activation: Firstly, estimate the threshold for identifying the sparse compo- nents of the input X. Given a pre-defined sparsity budget s, the threshold t(s) is estimated as the sth
percentile of X, ie.,, P([ < t(s)) Next, apply the threshold to mask out the low- magnitude channels. The corresponding sparsification function (ece) (-), is defined as:
[X,
if XX > ((s)
(x),
0
if X < t(s)

---

Published as a conference paper at ICLR 2025
Activation Sparsity
Low Rank Decomposition
SVD Index
Input Channel
Input Channel
X
W"
A,
Y
R-Sparse
Input
Sparsification
Sparse Loading
SVD Index
Y
Input hannel
Offine SVD
Y,
Figure 4: Illustration various compression techniques with corresponding impact on different input channels and singular values. The horizontal axis of the heatmap represents the input channels, while the vertical axis corresponds to the singular value index.
Note that CATS Lee et al. (2024) employs similar thresholding strategy to identify sparse compo- nents. However, while their approach targets sparsity in the output activation of the gate projection, our method focuses on input sparsity, which can be applied across all linear layers of LLMs.
The original linear layer Y XWT can then be approximated as Y,+Y, where Y, = (x (X) and Y, (X - (A)) For the sparse part, we omit unnecessary columns corresponding to input channels with zero values. Additionally, the weights should be stored in a column- format to enhance memory bandwidth utilization, as GPUs fetch consecutive memory entries during each access. For the low-rank part, we perform SVD
R-Sparse Inference:
on the pretrained weight matrix W and use its low-rank approximation, where A, U, and
B. = Σ VT, with representing the selected rank. And select the most important components based on the estimated scores Figure 3 Since this low-rank approximation can be computed offline through single SVD operation, it won't impact the latency during the inference. The memory I/O overhead is determined by two hyperparameters, (r, s), and equal to r mnt + s relative to that of a full linear layer. Additionally, we apply R-Sparse inference all linear layers both the attention and MLP blocks, aiming to achieve higher sparsity ratios.
3.5 OPTIMAL RECIPE FOR SPARSIFICATION
different layers demonstrate varying characteristics of rank-aware sparsity. To more accurately approximate the full computation, we develop evolutionary strategy search for the optimal ratio between the sparse and low-rank components within each layer. We begin by detining Pt. which represents the relative ratio of the sparse part in layer Given C, as the sparse mn budget of layer i, the sparse part equals to = P,C; and the rank is r, = (1-p:)C; mitnn We employ the search algorithm (Algorithm) [] to obtain the optimal p" = {pt .......Pi. arg min, L(S. p). where the loss C is the average perplexity over randomly selected samples from the C4 training set and is the original LLMs. We retain the individuals with lower perplexity each generation. To expedite the convergence the search process, we implement a group-wise strategy with group size of 28. In this approach, optimize the variables of one group at time, while holding the variables of the other groups at the values from the most recent best-performing individual.
As illustrated Figure 3,
The population size is set 32, with both the mutation rate Pm and crossover rate Pe equals 0.5, and the total number generations is The overhead of the search process is minimal, taking approximately one hour single A6000 GPU for the Llama-2-7B model.

---

Published as a conference paper at ICLR 2025
Algorithm Search Algorithm for Sparsification Recipe
1: Initialize: A pre-trained LLM M that consists of L layers. A population size of P. mutation rate Prc. crossover rate pe. a total of T generations.
p² 2: Randomly initialize population G pp ... p²) where p² pi. b Select the best individual from the group
3: S = (est((G); 9 (
for generation 1, 1,...,
5: for generation = 1.... do
= 1² 1 + pmlp pd0)
b Mutation: 41, ar,, as are randomly chosen from [1,2, 2,... P}.
7: Ø' (a Pa)mn + (ax po)p'
D Crossover: is random variables from (0, 1)*
g -
9:
end for
G x Ug); S Best (9); =0
b Select the next generation
11:
end for
12: Return: Best recipe S.
EXPERIMENTS
Table Comparison between R- Spaasee and other baselines on common-sense reasoning tasks.
Models
WG
PIQA
SciQ
OBQA
HS
BoolQ
Arc-E
Arc-C
Average
Llama-2-7B
69.14
78.07
93.80
31.40
57.13
77.71
76.35
43.43
65.88
ReLUfication
49.25
54.19
25.90
15.40
25.82
60.00
27.90
24.23
35.34
CATS22%
67.72
77.37
92.80
30.40
57.03
72.87
74.71
41.64
64.32
CATS
55.01
66.97
57.20
20.20
36.27
62.81
44.02
27.56
46.26
R-Spar5040%
68.03
77.31
93.90
30.80
55.62
75.99
75.67
42.66
65.00
GRIFF INa3%
62.04
71.27
89.00
22.00
47,20
60.98
60.94
32.00
55.68
GRIFF INs0%
53.59
64.74
77.70
17.40
35.64
56.42
40.74
21.08
45.91
R-Sparsecos
67.40
77.31
93.90
31.40
54.26
72.84
74.58
40.78
64.06
Llama-3-8B
72.69
79.71
96.20
34.80
60.18
81.35
80.09
50.51
69.44
ReLUfication
50.83
53.48
22.20
14.80
25.53
52.78
24.45
21.50
33.20
CATS22%
70.17
79.00
94.90
31.20
57.81
76.51
75.29
46.50
66.42
CATS
48.22
36.90
16.20
27.30
49.05
30.30
22.35
35,91
R-Spar8640%
71.11
78.24
95.90
34.60
58.33
79.85
79.67
49.23
68.37
GRIFFING 1N33%
63.54
71.87
89.40
24.00
48.08
54.34
62.33
34.73
56.04
GRIFF INoo%
52.80
64.74
73.90
19.20
35.62
47.61
43.64
23.38
45.11
R-Sparse50s
69.30
77.69
96.00
31.60
56.64
76.73
76.94
44.71
66.20
Mistral-7B
74.11
80.41
96.00
32.20
61.05
83.85
80.68
50.85
69.89
ReLUfication
48.62
51.52
23.51
14.40
25.81
42.02
27.82
24.06
32.22
CATS22%
72.22
79.82
94.30
32.20
60.79
80.46
77.78
50.51
68.51
CATS40%
50.83
58.05
28.60
19.20
28.06
60.21
30.22
25.43
37.58
R-Spar5c40%
72.45
79.49
96.10
30.40
59.68
82.1
79.80
47.18
68.40
GRIFF TN33%
63.30
75.95
91.00
25.40
53.01
64.28
68.73
36.60
59.78
GRIFF IN30%
54.22
67.90
79.10
19.80
39.99
47.31
49.83
26.37
48.07
R-Sparseços
72.69
79.92
96.10
30.60
58.94
82.81
78.91
47.18
68.39
4.1 GENERAL SETUP
Models and Datasets. To evaluate the effectiveness R Sparse, we consider three representative
large language model (LLM) families: Llama-2 (Touvron et al. 2023), Llama-3 (Dubey et al. 2024),
and Mistral (Jaag et al. 2023). We assess the models on several popular tasks, including eight
common-sente reasoning tasks: Winogrande (WG) (Sakaguchi et al. 202 PIQA (Bisk et al. 2020),
SciQ (Welbl et al. 2017). OpenBookQA (OBQA) (Mihaylov et al. 2018b), HellaSwag (HS) Zellers et al. 2019), BoolQ (Clark et al. 2019), and ARC (ARC-Fassyy and ARC- ARC-Challenge) (Clark et al.
7

---

Published as a conference paper at ICLR 2025
2018b). Evaluations are conducted using the In-evaluationn-hrree framework (Gao et al. 2021
Additionally, we report results on text summarization tasks using XSUM Narayan et al. 2018). as well as language modeling tasks the validation set of WikiText-2 (Merity et al. (016)). For common-sense reasoning, we report accuracy, while summarization tasks are evaluated via Rouge-I. scores and language modeling is assessed by perplexity.
Baselines. Since R-Sparse does not require additional training, we compare it against several competitive training-free methods. (i) ReLUfictii on (Mirzadeh al. (023) where the non-ReLU activation functions in the MLP block are replaced with ReLU, and accuracy is reported without retraining, (ii) CATS(Lee al., 2024) that sparsifies Wup and W down. based on the magnitude of output activations from W gate. (ii) GRIFF IN (Dong et al.] 2024): It sparsifies all layers in the
MI.P block, selecting important channels based on statisties from the pre-filling stage. Different from CATS and GRIFFIN which focus only on the MIP blocks, R-Spa rse sparsifies all linear layers, including the attention blocks. For a fair comparison, we report performance with the original reported sparsity ratios (50% for the sparsified modules, corresponding to 22% model-level sparsity for CATS and 33% for GRIFFIN), We also compare the results with higher sparsity ratio by scaling up the MLP block sparsity for both methods. All sparsity ratios reported in the following experiments are measured at the model level. More details are included in Appendix and B.I
4.2 END-TO-END RESULTS
Winggande
Scrl
PIDA
Upentionk@A
Hellswwr
Assenacy (50
50-
60-
s0-
100
10
BoolQ
Arc.
Arc-Challegg
Wa.Test-2
XSUM
Abrrmng (1)
Astincy PM
to
Perpistty
40
Houge
50-
20
40
64) (76)
20
80
20
40
Sparsity
Spansity
Spansty
Sparsity (%6
Sparsity
GRIFFIN
Ours
Figure 5: Comparison results of Llama-2-7B across different model-level sparsity ratios common-sense reasoning, language modeling and summarization tasks.
We begin by presenting the end-to-end performance of R- Sparse and baseline methods across different models, tasks, and sparsity ratios. The results, shown in Table I and Figure [3] highlight several key observations: (I) R-Sparse consistently outperforms CATS (Lee al. (0224) and CRIFFIN (Dong et al.) 2024) across all common-senss reasoning, language modeling, and summa- rization tasks. With the same model-level sparsity budget (i.e. CATS40% v.s. R-Sparse e 40% and GRIFFIN v.s. R
) R Sparse achieves average performance gain of 18.74% over CATS and 18.15% over GRIFF IN on Llama-2-7B. This improvement primarily stems from three factors: ! while CATS and GRIFF IN only sparsify the MLP block, R- R-Sparse can be applied to both the attention and MLP blocks; we extends standard activation sparsity with rank-aware sparsity, providing better approximation of the full computation; Ø and we further leverages the adaptive rank properties of different layers by searching the optimal sparse-rank ratio p. Detailed ablation studies on these factors are discussed in Section 4.4 (ii) Sparse achieves performance comparable to the full model with minimal degradation at sparsity ratio around 50% while in some tasks, e.g., SciQ, a matching performance can be achieved even at a sparsity ratio of 70%. (iii) For some tasks, moderate sparse treatment slightly enhances the accuracy, such as 1.60% improvements at 30% sparsity ratio on the OpenBookQA task.
4.3 EFFICIENCY
We demonstrate the end-to-end efficiency improvements of R-Sparse. For this, collected five samples that consists of 2048 tokens and generate new content ranging in length from 128 to 2048 tokens to evaluate performance across different generation lengths. Without losing generality,
8

---

Published as a conference paper at ICLR 2025
Generation Speed (Token!s)
Llama-2-17
Generation Speed (Token's)
Llama-3-8B
22
22
20
Dense
20
Dense
Ours
Ours
18
18
16-
16
14
12
10
128
256
512
1024
2048
128
256
512
1024
2048
Generation Length
Generation Length
Figure 6: Generation speeds of Llama-2-78 and Llama-3-8B using uniform 50% sparsity our method. The prompts consist of 2048 tokens, with generation lengths ranging from 128 to 2048. The generation speed is calculated as the number of generated tokens divided by the total generation time.
Table Compatibility with weight quantization.
Table 3: Results of sparse and low-rank baselines.
Methods
WG
PIQA
SciQ
OBQA
Average
Methods
WG
PIQA
SciQ
OBQA
Average
INT4
68.19
77.48
93.80
29.80
67.32
Full
69.14
78.07
93.80
31,40
68.10
R-Sparseun
68.03
77.31
93.90
30.80
67.51
Sparse
65.11
77.37
93.30
29.20
66.25
R-SparsRuol
77.31
93.90
-ow-Rank
49.88
53.32
14.80
14.20
33.05
INT4 208
INT4 R-Sparaeus
66.93
76.71
92, 20.80
29.20
68.41
R-Sparse
67.40
77.31
93.90
31.40
67.50
our implementation based on the Hugging Face library with FP32 precision data format. All experiments conducted single NVIDIA A6000 GPU without offloading. We applied uniform 50% sparsity to S-Sparse, achieving comparable performance as shown in Section 4.2 and utilized customized Triton kernel reduce data transfer between on-chip SRAM and HBM. As illustrated in Figure 6 R-Spa rse achieved up to 42% and 40% improvements in generation speed for Llama-2-7B Llama-3 respectively, highlighting the effectiveness of our approach.
4.4 ABLATION STUDY AND FURTHER INVESTIGATION
We conduct extensive ablation studies of R Sparse, summarized by the following research ques- tions: Q/: R-Sparse compatible with weight quantization? Q2: How does R-Sparse compare with vanilla activation sparsity? Q3: What's the benefit of optimal sparsification recipe?
Al: Compatible with quantization. We demonstrate that R-Sparse is highly compatible with weight quantization. As shown Table / when combined with 4-bit guantization, R- S-aaase achieves average accuracy of 66.41 % at 40% sparsity and 65.76% 50% sparsity common- sense reasoning tasks, closely comparable to the full model's performance quantization-only result of 67.32%. Note that quantization with group size 128, that provides matching performance as the full baseline. The compatibility of Sparse with weight quantization offers further potential efficiency gains through optimized CUDA kernels that fuse the sparse and quantization operations.
68. 10% and the GPTQ (Frantar et al. (022) for weight
A2: S-Srse outperforms both vanilla activation sparsity and low-rank decomposition. comprees R-Sparse with vanilla activation sparsity (Sparse) and low-rank decomposition (Low-Rank) For the sparse and low-rank baselines, apply the sparsification operation all linear layers, maintaining the same model-level sparsity ratios for each method. Experiments conducted with 50% sparsity Sparse baseline, with an average improvement of 0.98%. while the Low Rank method fails to maintain performance. This is expected, the low-rank properties vary across layers: layers with intrinsic low-rank characteristics can be well-approximated with small p. while higher-rank layers benefit from higher sparse components, leading a higher p. With that, Sparse combines both scenarios and provides a more effective approximation.
Llama-2-7B show that R-Sparse consistently outperforms the

---

Published as a conference paper at ICLR 2025
A3: Further enhancement through bet- ter sparsification recipes. We compare the searched sparsification recipes with uniform ones. For the uniform approach, we set p = 0.95 uniformly across all layers, based a grid search using 16 training samples from the C4 dataset. In contrast, the adaptive strategy is based on the search algorithm. As shown in Table the evolutionary search algorithm outperforms the uniform strategy, achieving up to a 1.60% accuracy gain across sparsity ratios ranging from 40% to 70%. Notably, higher sparsity ratios, the adaptive strategy yiclds greater performance improvements. For example, on the OpenBookQA task, at the 70% sparsity ratio, there is 2.60% gain compared to 0.80% improvement at the 50% sparsity ratio.
Table 4: Comparison of different sparsification recipes.
Tasks
Methods
40%
30%
60%
70%
Average
OBQA
Uniform
29.80
30.40
24.60
21.40
26.55
Adaptive
30.80
31.40
27.80
24,00
28.50 (-1.95)
Arc-E
Adaptive
Uniform
75.67
74.92
74.03
74.58
70.29
61.41
70.49 (-0.89)
60
Arc-C
Uniform
41.30
39.08
35.49
28.16
36.01
Adaptive
42.66
40.78
36.01
29.10
37.14 (-1.13)
BoolQ
Uniform
75.32
72.51
69.12
63,85
70.28
Adaptive
75.99
72.14
71.39 (+ 1.111
CONCLUSION
In this paper, focus on the activation sparsity of the input side. By leveraging the intrinsic sparse structure within input activations and singular value components, we introduce R- S-Sparse, which eliminates the need for extensive pre- pre-training and predicting active output channels, achieving 50% model-level sparsity without additional training. Experiments across different LLM families, including Llama-2, Llama-3, and Mistral, demonstrate the effectiveness of R comparable performance at 50% sparsity across ten common-sense reasoning. language modeling, and text summarization tasks. This high sparsity ratio also brings a significant 43% speed improvement with customized kernel. Our work demonstrates that high levels of sparsity can be achieved in both the attention and MLP blocks of advanced LLMs without any performance loss, benefiting the further deployment of LLMs edge devices.
REFERENCES
Keivan Alizadeh, Iman Mirzadch, Dmitry Belenko, Karen Khatamifard, Minsik Cho. Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. Llm in a flash: Efficient large language model inference with limited memory. arXiv preprint arXiv:2312./15114 2023.
Aviv Bick, Kevin Y Li, Eric P Xing. Zico Kolter, and Albert Gu. Transformers to ssms: Distilling quadratic knowledge subquadratic models. arXiv preprint arXiv:2408. 10189, 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 19005 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 1rXiv:1803.0545740 2018a.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: weo.:8803..04. 2018b.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 15:16944-16359 2022.
10

---

Published as a conference paper at ICLR 2025
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. corr 0oss2208.003339 (2022), 2022.
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study the colossal clean crawled corpus. arXiv preprint arXiv:2104.08756 2021.
Harry Dong, Beidi Chen, and Yuejie Chi. generation. arXiv preprint arXiv:2404.01 01305 2024.
mixture of experts for efficient Ilm
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama herd of models. arXiv preprint arXiv:2407.21 2222/773 /783, 2024.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression large language models via additive quantization. arXiv preprint arXiv:2401.06//6 2024.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Siggoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337, PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.1733: 2022.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version vO. 0.1. Sept, 10:8-9 9, 2021.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:20/2.14913, 2020.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,, 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
prXiv:1606.08411 2016.
G Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.0233 2015.
Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients. arXiv preprint wrXXv:2407.11235 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo. Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin- Yew Lin, et al. Minference 1.0; Accelerating pre-filling for long-contert Iims via dynamic sparse attention. arXiv preprint arXiv:2407.02490, 2024.
Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium Operating Systems Principles, pp. 6-666, 2023.
Je- Yong Lee, Donghyun Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini. Cats: Contextually- aware thresholding for sparsity in large language models. arXiv preprint 2xXvv2240.008733. 2024.
11

---

Published as a conference paper at ICLR 2025
Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank Reddi, Ke Ye, Felix Chem, Felix Yu, Ruiqi Guo, et al. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. arXiv preprint arXhv:2210.06313, 2022.
Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang. Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 87-100, 2024.
James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, and Ben Athiwaratkun. Training- free activation sparsity in large language models. arXiv preprint arXiv:2408. 14029408..4690, 2024a.
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient Ilms at inference time. In International Conference Machine Learning, pp. 22137-22176.. PMLR, 2023.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong. Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: tuning-free asymmetric 2bit quantization for cache. arXiv preprint arXiv:2402.027500 2024b.
Xinyin Ma. Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21722 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint (ett:::66000784440 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018a.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv:1809.022. 2018b.
Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models. arXiv preprint arXiv:2310.04566 2023.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.00 08744 2018.
NVIDIA. Nvidia a100 tensor core gpu architecture. Volume 1.0: Whitepaper, Part, 1(2020):8.. 2020.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing mas for the transformer era. arXiv preprint arXiv:2305. /3048, 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106.
2021.
Chenyang Song, Xu Han, Zhengyan Zhang, Shengding Hu, Xiyu Shi, Kuai Li, Chen Chen, Zhiyuan Liu, Guangli Li, Tao Yang, et al. Prosparse: Introducing and enhancing intrinsic activation sparsity within large language models. arXiv preprint arXiv:2402.13516, 2024a.
Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. Powerinfer; Fast large language model serving with a consumer-gren grade gpu. arXiv preprint arXiv:2312.12450 2023.
Yixin Song. Haotong Xie, Zhengyan Zhang, Bo Wen, Li Ma, Zeyu Mi, and Haibo Chen. Turbo sparse: Achieving Ilm sota performance with minimal activated parameters. arXiv preprint (01:22406.05555. 2024b.
12

---

Published as a conference paper at ICLR 2025
Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Llm pruning and distillation in practice: The minitron approach. arXiv preprint arXiv:2408. 1111::2008.11766,, 2024.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09288, 2023.
Johannes Welbi, Nelson Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv: 700. 770706209, 2017.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. /7453, 2023b.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware- -efferrreeef training. arXiv preprint arXiv:2312.00663,, 2023.
Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Yu Hsieh, Yaqing Wang, Yiling Jía, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. arXiv preprint arXiv:2310.05175. 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint 1905009005007830 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint rrxi::2255000663 2022.
Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, and Maosong Sun. Relu2 wins: Discovering efficient activation functions for sparse llms. arXiv preprint arXiv:2402.03R: 038844 2024a.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024b.
13

---

Published as a conference paper at ICLR 2025
A MORE IMPLEMENTATION DETAILS
In the experiments, the sparsification techniques are applied exclusively during the decoding stage. For the tasks involving only a single-step decoding phase, original GRIFFIN IN implementation only apply the sparsification on the final token while in our experiments, we simulate the first half of the prompt as the prefilling stage, applying sparsification the second half to more effectively evaluate the generation capabilities of LLMs.
B EXTENDED EXPERIMENTS
B. SCALING UP SPARSITY RATIOS OF GRIFFIN
For GRIFF IN, we explore two strategies for scaling up the model-level sparsity ratios: (i) MLP, where we directly increase the sparsity ratios within the MLP blocks and report the resulting model-level sparsity; and (ii) A11, where we extend the strategy to include attention blocks. In this case, we use the same metrics to identify important channels based on the activations during the prefilling stage and determine the corresponding active channels during the decoding stage. Results are presented in Figure where the MLP strategy is significantly better than the All. Thus in the main context, report the results of MLP strategy for GRIFFIN.
Selp
I1QA
Opentlokig
75
ALL.
29
60
MLP
MIL1
24
MLP
MLP
30
50
55
45
35
50
Spansity (1Q
Spariity (16)
Syanity (h)
Spasky (9)
Figure Results of GRIFFIN with Llama-2-7B
B.2 RANK-AWARE ACTIVATION SPARSITY ACROSS VARIOUS DATASETS AND DIFFERENT NUMBER OF SAMPLES
We extend the observations from Figure to additional datasets and varying numbers of samples. The results are presented in Figure and Figure 9 Across different numbers of training samples, the importance patterns consistently exhibit high sparsity. Additionally, to ensure data diversity, we evaluated different domains from the RedPajama d including GitHub, ArXiv, StackExchange, and Wikipedia. As shown in Figure D. the importance patterns are remarkably similar across these datasets, demonstrating the generalization capability of the Spaarse approach.
'The training data
is obtained from
http:://hgginngaae.ooootttttsst
Eogetheercomppter//Reeee jama Data- 17
14

---

Published as a conference paper at ICLR 2025
N-13
N-512
N-1836
Figure 8: Importance of each input channel and singular value across varying samples. The number of samples ranging from to 1024. Results are collected from Llama-2- 2-BB model from C4 training set. The sequence length of each sample equals 4096.
Stadonshuggy
WiLiPada
Figure 9: Importance each input channel and singular value components across datasets.
15


Reviews:
[{"review_id": "dhmvKS8PFJ", "reviewer": "Reviewer_T8xk", "summary": "This paper presents R-Sparse, a training-free activation sparsity approach for large language models (LLMs). Current activation sparsity methods face limitations with non-ReLU activation functions and have difficulties in predicting active channels and achieving high sparsity ratios. R-Sparse overcomes these challenges by leveraging the sparsity of input channels and singular value components. The authors conduct investigations and find that non-sparse components can be regarded as bias terms and full computation can be approximated by a combination of input channels and weight singular values. R-Sparse is applied to both attention and MLP modules of LLMs and an evolutionary search algorithm is used to find optimal sparse component ratios. Experiments on Llama-2/3 and Mistral models across ten tasks show that R-Sparse achieves 50% model-level sparsity with comparable performance and up to 43% end-to-end speed improvement with a customized kernel.", "questions": "Please kindly refer to the Weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The training-free method does not require extensive pre-training, making it more efficient and easier to implement compared to methods that need continual training.\n* R-Sparse achieves high sparsity levels (50% model-level sparsity) without sacrificing performance, leading to significant improvements in efficiency.\n* R-Sparse is compatible with weight quantization for further efficiency gains and can be applied to different LLM families and a variety of tasks.", "weaknesses": "* Table 1 is difficult to interpret. From the description of the authors, R-Sparse40% is compared with CATS22% and GRIFFIN33%, if my understanding is correct. Therefore, R-Sparse does not consistently outperform CATS across all tasks (e.g., PIQA 78.24 vs 79.00). The authors are suggested to refine the claim to avoid the misleading. Meanwhile, for certain cases, the performance increases with an even higher sparsity ratio (e.g., 79.49 vs 79.92 for R-Sparse40% and R-Sparse50%) on PIQA. Could the authors provide some insights into this phenomenon?\n* The sensitivity analysis of hyperparameters should be added for a more thorough investigation of the effectiveness of R-Sparse.\n* As [1] is discussed by the authors in the related works, why do the authors choose not to compare with [1] in the experiments?\n* The authors are suggested to include the complexity analysis and running time comparison of R-Sparse, especially regarding the evolutionary search algorithm.\n* The writing can be further improved. For example, the optimal sparse-rank $\\alpha$ is not formally defined. From Algorithm 1, $\\alpha$ seems to be fixed, how is it optimized?\n\n[1] Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time, ICML 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dimJkVmAaN", "reviewer": "Reviewer_zFFb", "summary": "This paper uses the inherent sparsity and low-rank properties of input activations in LLMs to accelerate the inference of LLMs. It sparsifies the input activations to remove unnecessary activations, and interprets unnecessary activations as a bias term, using SVD to compensate for this bias. The proposed method achieves higher sparsity than the baseline while maintaining overall performance.", "questions": "See the Weaknesses section", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes a training-free activation sparsification method to speed up LLM inference. The proposed method can maintain performance at the same level as full parameters at sparsity of about 50%.\n- This paper proposes to design different sparsity ratios for different layers to improve overall performance.\n- This paper is very clearly written and the proposed method is easy to follow.", "weaknesses": "- The analysis of \"the contribution of each input channel and singular value component\" in this paper is mainly focused on the C4 dataset (Figures 1 and 3). What are the similarities and differences between the analysis on other datasets and the C4 dataset? Especially on datasets that are very different from the C4 dataset. In addition, these analyses are mainly based on 16 randomly sampled training samples. When the number of samples increases or decreases, what changes will occur in the analysis results?\n- As shown in Figure 3, there is a clear difference in the importance of different linear layers (such as self_attn.k_proj vs. self_attn.up_proj), what this mainly stems from, the authors can give more comments on this.\n- What is the relationship between the sparsity ratio in the proposed R-Sparse and the final inference acceleration? For example, what is the corresponding acceleration for a certain sparsity ratio?\n- In Figure 6, why under Dense, when the Generation Length becomes longer (1024->2048), the generation speed slows down, while when it is 128->256->512, the generation speed is accelerated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8cPblcMtGM", "reviewer": "Reviewer_LhWB", "summary": "The authors propose a novel training-free activation sparsity method called R-sparse. This method is applicable to non-ReLU-based large language models (LLMs) and eliminates the need for prediction by utilizing input activation sparsity. Furthermore, it is a method that can be applied not only to MLP modules but also to attention modules.", "questions": "- (Writing) Please switch the positions between mlp.up_proj and mlp.gate_proj of layer 0 in Figure 3.\n- Recommended reference\n  - Alizadeh, Keivan, et al. \"Llm in a flash: Efficient large language model inference with limited memory.\"\n- Definition of multi-phase ReLU.\n  - Is the multi-phase ReLU expressed on line 170 correctly defined? Where is sparsity defined? Shouldn't there be a definition for when x $< T_{L}$? Moreover, the output should get closer to zero as it becomes more negative, but it is defined in the opposite direction.\n  - For instance, $T_0 = 0, T_1=-1, T_2=-2$, please explain it.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors successfully identified existing challenges and, in connection to these, suggested a training-free, prediction-free activation sparsity method. Furthermore, their method is applicable to attention modules.\n- Through various experiments, they demonstrated performance improvements.", "weaknesses": "- difficult to read (clearity)\n  - In introduction, how should Figure 1 be interpreted? It is difficult to understand the meaning of the figure until reviewing the explanation in Section 3.3.\n  - In Section 3.2, what does the term \"bias\" mean? Is it interpreted as \"bias\" in the sense that performance is maintained even after replacing non-sparse values smaller than 0 with a constant? Does the term also include non-sparse components greater than 0? In Lines 183-185, why is it defined as \"sparse\", when $H_k \\geq T_0$?\n\n- the appropriateness of the title.\n  - The proposed method appears to be \"activation sparsity, then low-rank decomposition for non-important activations\" rather than using both approaches simultaneously (rank-aware activation sparsity).\n\n\n\n==== After Rebuttal ====\n- I understand. However, the terms in Section 3.2 should be clarified more explicitly.\n- It might be beneficial to add an explanation using examples, when $T_0 = 0$.\n  - positive = sparse components = not pruned values\n  - negative = non-sparse components = to be pruned values\n- In fact, the most confusing term is \"sparse components.\" It would be beneficial to clearly indicate, as suggested in the response, that it originates from prior research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x1SNMV6oAm", "reviewer": "Reviewer_sWLw", "summary": "This paper proposes a rank-aware activation sparsity, including applying input sparsification and weight decomposition. Experiments show that the proposed R-Sparse improves end-to-end efficiency while maintaining comparable sparsity.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed methods are easy to understand.\n2. The experimental results are good.", "weaknesses": "1. The connections between motivations and the proposed methods are weak. For example, the first motivation claims that the outputs for the non-sparse inputs can be regarded as biases. However, it is not sure which part of the proposed methods is motivated by this. Please explicitly explain how the observation about non-sparse inputs being treated as biases directly informs specific components of their R-Sparse method.\n\n2. The contributions are incremental, which only directly apply existing techniques, such as CATS for sparsity, SVD for weight decomposition, and genetic algorithm for hyperparameters searching. Please more clearly articulate the novelty of your approach. Is there any specific design in SVD?\n\n3. The paper claims that existing non-ReLU activations such as SiLU and GELU introduce less sparsity. However, the experiments lack generality as Llama 2/3 and Mistral all adopt SiLU as activation functions. For example, adding results of models using GELU activation functions such as Gemma would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)