La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
Authors: Kai Liu
Bowen Xu
Shaoyu Wu
Xin Chen
Hao Zhou
Yongliang Tao
lulu hu
Venue: ICML 2025 poster (2025)
Keywords: Sparse Activation
Large Language Models
Orthogonal Transformation
LLM Inference
Primary area: deep_learning->large_language_models

Abstract: Activation sparsity can reduce the computational overhead and memory transfers during the forward pass of Large Language Model (LLM) inference. Existing methods face limitations, either demanding time-consuming recovery training that hinders real-world adoption, or relying on empirical magnitude-based pruning, which causes fluctuating sparsity and unstable inference speed-up. This paper introduces LaRoSA (**La**yerwise **Ro**tated **S**parse **A**ctivation), a novel method for activation sparsification designed to improve LLM efficiency without requiring additional training or magnitude-based pruning. We leverage layerwise orthogonal rotations to transform input activations into rotated forms that are more suitable for sparsification. By employing a Top-K selection approach within the rotated activations, we achieve consistent model-level sparsity and reliable wall-clock time speed-up. LaRoSA is effective across various sizes and types of LLMs, demonstrating minimal performance degradation and robust inference acceleration. Specifically, for LLaMA2-7B at 40\% sparsity, LaRoSA achieves a mere 0.17 perplexity gap with a consistent 1.30× wall-clock time speed-up, and reduces the accuracy gap in zero-shot tasks compared to the dense model to just 0.54\%, while surpassing TEAL by 1.77\% and CATS by 17.14\%.

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
Kai Liu
Bowen Xu
Shaoyu Wu'
Xin Chen
Hao Zhou
Yongliang Tao
Lulu Hu
Abstract
Speed-up: 1.40x
PPL: 5.47 6.80
Activation sparsity can reduce the computational overhead and memory transfers during the for- ward pass of Large Language Model (LLM) in- ference. Existing methods face limitations, either demanding time-consuming recovery training that hinders real-world adoption, or relying on em- pirical magnitude-based pruning, which causes fluctuating sparsity and unstable inference speed- up. This paper introduces LaRoSA (Layerwise Rotated Sparse Activation), a novel method for ac- tivation sparsification designed to improve LLM efficiency without requiring additional training or magnitude-based pruning. We leverage lay- erwise orthogonal rotations to transform input activations into rotated forms that are more suit- able for sparsification. By employing Top-K selection approach within the rotated activations, we achieve consistent model-level sparsity and reliable wall-clock time speed-up. LaRoSA is ef- fective across various sizes and types of LLMs. demonstrating minimal performance degradation and robust inference acceleration. Specifically, for LLaMA2-78 7B 40% sparsity, LaRoSA achieves a mere 0.17 perplexity gap with a consistent 1.30x wall-clock time speed-up, and reduces the accu- racy gap in zero-shot tasks compared to the dense model to just 0.54%, while surpassing TEAL by 1.77% and CATS by 17.14%.
Maginitude Pruning
Multipy Weight Matrix
cutoff thresholdt
50X
Input
Sparse
Output
Activation
Activation
Rotate and Prune
Low Maginitude Pruning
Multiply
Rotated
Reconstruct
Weight
S(KC)
SpXD Q
SEXC)) (Q'W)
Sparse
Activation
[Q absorded
(Rotsted)
Weight)
La RoSA
Speed up: 1.38x : 5.87
Figure 1. LaRoSA rotate and prune the input activations achieve more consistent and efficient LLM inference. The rotation is orthogonal and can be reversed after pruming. The overall method provide more accurate approximation of the full computation. Visualizations are from the 12th Attention block of LLaMA2-7B.
cent efforts on boosting LLM inference have predominantly
focused on low-bit model quantization (Frantar et al., 2022; Lin al., 2024b; Ashkboos al., 2024b; Liu al., 2024b; Lin et al., 2024a) and model weights pruning methods (Fran- tar & Alistarh, 2023; Ashkboos et al., 2024a; Ma et al., 2023; Fang et al., 2024). Such techniques enhance inference speed and save memory usage by either employing low-precision parameter formats or reducing the total number param- eters. Exploiting activation sparsity (Kurtz et al., 2020; Li et al., 2022; Chen al., 2023) is another promising ap- proach to improve inference efficiency. Since channels in the weight matrix corresponding zero-valued activations are not utilized during the forward pass, LLM inference can be accelerated by: (1) avoiding transfer such param- eters between memory and computation units, and (2) omitting the matrix computations associated with those weights channels. These speed-ups could achieved by utilizing sparse patterns that are compatible with hardware.
1. Introduction
Large Language Models (LLMs) have achieved significant advancements across various real-world machine learning applications and leaderboards (DeepSeek-Al, 2025; Ope- nAI, 2024; Team, 2024; Qwen al., 2025). However, bil- lions of parameters in these models pose significant chal- lenges for efficient LLM inference and serving, especially in terms of memory footprint and computational overhead. Re-
Correspon- dence to: Kai Liu ckane.liukai@e @mail. gmail.com... com>, Bowen Xu <bowiehsu1022 1ottchsuuu14@@e00 @024@gmal.mmcm...
Equal contribution 'Alibaba Group.
Recent studies (Mirzadeh et al., 2024; Liu al., 2023; Zhang et al., 2024) have observed high degree of activation sparsity in the ReLU-based Transformers. Based on this, Dejavu (Liu et al., 2023) trains a predictor for each layer to preemptively identify weight channels corresponding to zero-valued activations. (Mirzadeh al., 2024) induce
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
high sparsity by replacing non-ReLU activations with ReLU. However, these methods do not achieve full sparsity in the models and require substantial additional training to recover model performance. Some other works have turned their attention to magnitude-based pruning. CATS (Lee et al., 2024) utilizes empirical distribution histogram to obtain the cutoff threshold in MLP blocks, and zero out activations below that threshold. TEAL (Liu et al., 2024a) further ex- tends this concept to all input activations in both Attention and MLP blocks. However, offline cutoff thresholds are heavily related to calibration tokens, and lack accuracy and consistence in generating optimal sparse patterns for unseen tokens to achieve a target sparsity level. Moreover, incon- sistent activation sparsity can lead to unstable inference speed-ups, reducing efficiency in LLM inference. Driven by these limitations, we raise the following question: Is it possible quickly, effectively, and consistently lever- age activation sparsity to enhance the efficiency of L.LSs without additional training or empirical thresholds?
to accelerate LLM decoding speed by omitting the trans- fer of weight channels associated with zero-valued entries to GPU registers. Furthermore, Song et al. (2023) and Alizadeh et al. (2024) leveraged activation sparsity to facili- tate CPU offloading, significantly reducing memory trans- fer overhead between CPUs and GPUs. However, it is noteworthy that modern leading LLMs (Team, 2024; Qwen et al., 2025) utilize more complex nonlinear functions (e.g. SwiGLU (Shazeer, 2020)) which do not inherently produce sparsity (Mirzadeh et al., 2024). Consequently, most of the ReI.U-based activation sparsification methods are not applicable to these models.
Inducing Sparsity in non-ReLU Models Mirzadeh et al. (2024) replaced SwiGLU with ReLU in L.L.s,, followed by continued pre-training to induce activation sparsity. Song et al. (2025) and Song et al. (2024) introduced techniques such as activation regularization to further enhance spar- sity in adapted models. Zhang et al. (2024) experimented with various activation functions and identified Squared ReLU (So et al., 2021) as the most effective alternative. CATS (Lee et al., 2024) realizes desired sparsity level in SwiGLU based MLP output activations through empir- ical distribution and magnitude thresholding. TEAL (Liu et al., 2024a) further extends this concept to the input acti- vations of self-attention and MLP blocks, and prunes low- magnitude activations. Wang et al. (2024) combines magni- tude pruning and quantized activations and presents scaling laws for sparsely activated LLMs. However, these methods either rely on substantial post-training and calibration, or produce unstable model-level sparsity, resulting in sparse activated LLMs that lack efficiency and flexibility.
In this work, we present LaRoSA (Layerwise Rotated Sparse Activation), a training-free activation sparsification method that produces optimal sparse patterns and consistent generation speed-up for LLM inference. As depicted in Fig- ure 1, our method uses orthogonal rotation matrix Q to transform the input activation, enabling more effective acti- vation sparsification. We apply PCA to obtain the rotation matrix Q for each layer and employ a Top-K function to ac- quire stable activation sparsity. A residual adapter is added to realize layer-specific independent rotations. In the end, the rotation matrix Q can be absorbed by the correspond- ing weight matrix, eliminating the additional computational overhead introduced by Q. Our main contributions are:
Recent works have focused on using orthogonal rotation matrices to miti- gate performance drops in LLM pruning and quantization. SliceGPT (Ashkboos et al., 2024a) multiplies each weight matrix by an orthogonal matrix to reduce the embedding dimension, and use residual adapter to achieve compu- tational invariance. QuaRot (Ashkboos et al., 2024b) use random Hadamard rotation to redistribute outliers across all channels. SpinQuant (Liu et al., 2024b) suggests training the orthogonal matrix instead of using a random Hadamard matrix to further enhance quantizability.
Rotation-based Model Compression
We show that layerwise orthogonal rotation matrices can be applied to improve the performance and effi- ciency of LLMs without extra training or empirical thresholds.
We introduce consistent model-level sparsity to non- ReLU based LLMs by applying Top-K K functions on the rotated activations and realize more stable and efficient model inference than SOTA methods.
We conduct extensive experiments with leading LLMs and demonstrate that LaRoSA is effective and ro- bust across different types, sizes, and sparsity levels. LaRoSA presents minimal performance degradation while providing consistent wall-clock time speed-up.
3. Preliminary
3.1. Activation Sparsity in LLMs
2. Related Work
In this work, we focus on the Transformer architecture LLMs which are composed of repeated decoder layers (Tou- vron et al., 2023). In subsequent sections, unless otherwise specified, a 'layer' refers to the decoder layer that repeatedly appears after the token embedding layer, and a 'block' refers to an Attention block or a MLP block from decoder layer.
Activation Sparsity Activation sparsity refers to the phe- nomenon where considerable portion of a model's 's hidden states are zero-valued. This property naturally emerges in the intermediate states of ReLU- MLP blocks (Li et al., 2022). Liu al. (2023) utilized activation sparsity
2

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
WHead
Oulput logits
Mergeable Rotations
MLP Block
Q
Q
n- layers
h3
h4
W
W
Merged Weights
0 Q
W Waonn
Q
Residual Adapters
W gate
MLP Block
X,9
S(XQ)
(a)
Original Activation
Rotated Activation
Attention Block
x
Attention Block
W
h2
W.
2-1 layers
(c)
Qu
W
Wemboo
Input takens
XQ S(XQ)
5(X)
(b)
Input Activation
Sparse Activation
Rotated Activation
Figure 2. Model inference with LaRoSA. (a) The layerwise orthogonal matrix Qi can be absorbed into weight matrices to avoid extra computations, ensuring that the imput activations for each block pre-transformed by the layer-specific orthogonal matrix Q (c) Introducing residual adapters the residual stream ensures that each layer's 's input activations have independent orthogonal rotations. The Qo of first layer and Qn of last layer can be merged into the weight matrix of token emhedding and LM head layer, respectively.
We assume that the network constructed using a pre-norm approach, meaning that LayerNorm or RMSNorm opera- tion precedes each block. For clarity, h1 and h2 refer to the Attention block input activations, and h2 and h4 to the MLP block input activations. Figure 2 illustrates the construction of such blocks and layers. Typically, the forward pass in each block is performed by linear mappings between input tensors and weight matrices, which can be expressed as:
the activation sparsity ins these methods defined as:
1
Σ 1(; <c), P D
where is cutoff threshold estimated from the empirical cumulative distribution of activations, obtained offline from calibration datasets. The linear mappings in magnitude pruning method is then formulated by:
Y=X-W",
(1)
Y = Sx() W
(4)
where X E RN denote the input tensor of N tokens, W E I Pax Don is the weight matrix, and Y R Nx Daa is the output tensor. Dm and Dou are hidden sizes of in- put and output tensor. The matrix multiplication shown in Equation (I) is identical for each Wq. q. Wk. Wy. W. in Attention blocks and Wap- W gate W down in MLP blocks. Let æ Ra denotes the input activation of a single token from X. The activation sparsity p E [0, 1] of is defined as the proportion of pero-valued elements:
where Sz(-) is the sparsification function defined as:
0, if 2 S
S,(x:)
(5)
Zs. otherwise,
3.2. Computational Invariance Transformation
The computational invariance theorem, described in (Ashkboos et al., 2024a) Theorem 1, asserts that the activations between Transformer blocks (i.e., Attention and MI.P) can be transformed using an orthogonal matrix Q without altering the model's output. This theorem re- mains valid even when RMSNorm is applied between the blocks. The reason that RMSNorm divides activations by their norms, and orthogonal transformation Q does not affect these norms, since QQ" T
1
p
1(x, 0),
where x; the i-th element of x, and the indicator function (xx; = 0) outputs if the activation value is equal zero and 0 otherwise. As highlighted in (Mirzadeh al., 2024), non-ReLU based L.LMs inherently exhibit extremely low activation sparsity. Recent studies (Lee et al., 2024; Liu et al., 2024a) have assumed that magnitude activations exert minimal effects on model outputs and have shown that magnitude-basee pruning empirically effective. Typically,
Q Q = I and |=Q] /aQQQTT x V /axa" ||z|||- Therefore, we have the the commutation property in RMSNorm operation:
RMSNorm(X) (MMSNor(XQQQQ"
(6)
3

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
1.40
Wiree
0.5
2a
0:30
5C4
Targee Sporn
1.35
67
0--1
ACCIAN Sparoite
3.30
0.3
0:35
0.2
0.1
33
0.0
30
20 25
90
106
250
300
450
500
50
300
150
200
230
64
128
2.5
512
765
2024
2048
Sequenice (Sorted Actual Sporstyl
Lengih
(a) Magnitude Pruning Thresholds
(b) Actual Thresholds
(c) Innonssstent Actual Sparsity (d) Unstable Inferenee Specdup
Figure 3. (a) Offline calibrated thresholds for Attention and MLP blocks of LLaMA3-8B. (b) The calibrated and actual needed thresholds 24th Attention block of I.LMMA-8B. Input tokens are randomly selected from WikiText2. The dashed lines denote the calibrated thresholds, while scatter points indicate real thresholds needed to achieve the target sparsity. Many blue points (50% actual thresholds) are even below the black line (40% calibrated threshold). (c) Average actual activation sparsity of LLaMA3-8B's Attention blocks. Input sequences from different datasets and sorted by the sparsity. (d) The inference speedup of magnitude pruning 50% sparsity under various output sequence lengths. Speedup equal to 1.0 is the dense version of LLaMA3-8B model. Best view in color and zoom in.
In forward pass, we can merge the additional transforma- tion Q into weight matrices to avoid the extra computation induced by Q. As depicted in Figure 2, if W. is right- multiplied by an orthogonal matrix Q, this will make the Attention block produces a rotated activation XQ as out- put. We can cancel this effect by left-multiplying the weight matrices of subsequent MLP blocks (i.e., W up- W guee) by Q so the output changes induced by Q are neutralized, and the original computation results in each block remain unchanged. This process is same for Wooa and WG. q. Wt. Wy.
beginnings, actual thresholds are extremely lower than cali- brated ones, explaining why methods like TEAL struggle to sparsify initial tokens of a sequence during prefill stage of LLM inference.
Difficulty in Maintaining Consistent Sparsity. Achiev- ing stable and predictable LLM inference acceleration re- quires maintaining uniform and consistent target sparsity across different input sequences. As illustrated in Figure 3c, conventional pruning approaches that use calibrated cutoff thresholds fail to deliver precise sparsity control, resulting in actual sparsity significantly deviating from target values and showing considerable variation across sequences from dif- ferent datasets. These inconsistencies directly compromise the stable acceleration results shown in Figure 3d, leading to inefficient LLM inference that falls short of fully realizing the theoretical performance advantages of sparse activation. This observation highlights the critical importance of con- sistent sparsity for two key reasons: 1) Inconsistent sparsity prevents models from reliably achieving their target spar- sity, making computational efficiency gains unpredictable and rendering optimization efforts futile. 2) It introduces substantial variance in per-token inference times, causing undesirable fluctuations in decoding speed.
4. Method
4.1. Motivation:
Rethinking Low-Magnitude Pruning
We conduct experiments LLaMA3-SB using magnitude pruning methods, using 16 sequences of 512 tokens from WikiText2 (Merity et al., 2016) for various sparsity settings. We identify several key observations and visualizing them in Figure 3.
Ambiguity and Inaceuracy in Defining Low Magnitude. The definition of low magnitude influences how activations are sparsified. As shown in Figure 3a, cutoff thresholds obtained from offline calibration fluctuate significantly as the layer goes deeper. The thresholds for Attention and MLP blocks within the same layer also exhibit great varia- tion across different sparsity levels. Using these inaccurate thresholds can lead to severe mismatches with target sparsity. Figure 3b compares cutoff thresholds for activation pruning using TEAL and TopK methods at various sparsity levels. TEAL uses a fixed threshold from offline calibration, shown as a dashed line, while TopK computes thresholds during testing, shown as seatter points. It's evident that offline calibrated thresholds hardly align with the actual needed thresholds. In most cases, tokens actually require higher or lower thresholds than the calibrated ones. At sequence
Magnitude and Channel Importance Misconception. Magnitude pruning methods (Liu et al., 2024a; Lee et al., 2024) typically assume that low-magnitude activations have minor impact on model outputs, leading to the prac- tice of zeroing out entries that fall below certain thresh- old. However, as proved in Wanda (Sun et al., 2023) and SliceGPT (Ashkboos et al., 2024a), pruming solely based on the magnitude of weights or activations is not optimal. Low- magnitude activations can still substantially affect the output if their corresponding channels in the weight matrix have large norms. We further demonstrate in Appendix A that empirical output errors from magnitude pruning methods are consistently higher than theoretical output errors.
4

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
4.2. Layerwise Orthogonal Rotation
sparsity. We Top-K function to determine whether each element of rotated activation ã æQQ need to be reserved or not. The sparsification function in LaRoSA is defined as:
The limitations of magnitude pruning drive us to explore more effective approach for identifying activations that are inherently casier to prune. Inspired by rotation- based quantization methods (Ashkboos et al., 2024b), we attempt to obtain a rotated activation through orthogonal transfor- mations, after which the channel importance can be casily distinguished. We propose using an orthogonal matrix Qi to rotate and prune the input activation (i.e.. h1 and h2) of 1-th layer, and reverse the rotation using 0, after pruning. To construct Qi, we use PCA to leverage it's inherent ability to identify principal components. Specifically, we select a calibration dataset consist of M sequences, feed it into the model and forward pass through each layer. Let X denotes the input activation of layer for i-th sequence. We compute the covariance matrix xx:))) of each Xf and average over all sequences:
{ i if | E Top ((x.))
Sx,(E,) = 0, otherwise
(8)
where Top, is the function that selects the largest & elements of Ø in terms of the absolute values. Activation sparsity can be introduced by modifying the forward pass to:
Y = (X,,)) W/:
(9)
where Q WT can be precomputed and merged into the weights to avoid online computation. Therefore, Equa- tion (9) can be rewritten as
Y, = (x() (W,Q))"
(10)
1 M
and Qj become identity matrix when activation x in the h2 and h4 position for efficient implementation.
Co, = M Σ X
(7)
Then, the eigenvectors of Coo(X,, ) are solved and sorted in eigenvalues descending order to construct the Q Here, we only rotate h and h3 in LaRoSA for efficiency, since rotation matrices for h2 and h4 cannot be incorporated into W., W up- and W Wgate due to the constraints imposed by the Grouped Query Attention (GQA) in Attention block and the element-wiss product operation in MLP block.
4.4. Hardwareeefficient Customized Kernels
Introducing activation sparsity alone is insufficient to achieve end-to-end LLM inference acceleration on hard- ware (Frantar & Alistarh, 2023). To translate the activa- tion sparsity induced by LaRoSA into wall-clock inference speed-up, we need to implement a fast and efficient cus- tomized GPU kernel. Building on the Triton-based kernel introduced by Deja (Liu et al., 2023) and TEAL (Liu et al., 2024a), we implement a GEMV kernel that provides practical and consistent token generation speed-up. 1) We store weight matrices W in column-major format in mem- ory to facilitate the selective loading of weight columns corresponding to non-zero activations, as memory coalese- ing could significantly improve memory bandwidth utiliza- tion (Dao et al., 2022: Ivanov et al., 2020). 2) We fuse the Top-K function into the matrix-vector multiplication, identifying the indices of the columns in the weights that need to be retained during the forward pass. 3) We selec- tively load the sparse activations and corresponding weight columns from memory, transfer them to the computation units, perform the multiplication, and store the result.
It can be observed that each layer must use the same Qi due to residual connections between layers. We initially attempt to use a single global Q for all layers, but quickly find out that the optimal orthogonal matrix for each layer tends to be very different. To address this, a residual adapter Qf Q(+1 applied to the output of each layer to realize layer-specific rotation, as shown in Figure 2. Although this introduce small additional operations, we demonstrate in Section 5.3 that such operations preserve the optimal rotation for each layer and improve the overall efficiency. In Appendix A, we provide a theoretical analysis of rotation-basee activation pruning, demonstrating that the constructed rotation matrix effectively reduces empirical error within each layer, thereby outperforming magnitude pruning methods in accuracy.
4.3. Introducing Consistent Activation Sparsity
5. Experiments
To introduce a consistent activation sparsity into non-ReLU LLMs, we replace magnitude pruning with Top-K function to zero out the rotated activations. Specifically, given the rotated activation x; Qu from the 1-th block and target sparsity level p. we first calculate the number of elements k that need to be reserved after sparsification by k- = a (1 - p)- Din where a is a hyperparameter tuned to control the sparsity coefficient between h and he (or hy and h4) within the same block and guarantee the overall model-level
5.1. Experimental Setups
Models and Baselines. To test the robustness and scalabil- ity of LaRoSA, conducted experiments using 4 leading L.Mss: LLaMA3 (Team, 2024), LLaMA2 (Touvron et al., 2023), Mistral (Jiang et al., 2023), and Qwen2.5 (Qwen et al., 2025). The model size is selected from 7B and 70B parameters. We compare with state-of-the-an (SOTA) base- lines, including CATS (Lee et al., 2024) and TEAL (Liu
5

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
Table 1. Comparison of the average accuracy on seven zero-shot tasks and 5-shot MML.U. The first column indicates the actual model-level
sparsity. CATS cannot achieve 50% model-level sparsity as it only sparsifies the MLP's Down projections. Full results are in Appendix C.
LLaMA2-7B
LLaMA2-70B
LLaMA3-8B
LLaMA3-70B
Qwen2.5-7B
Qwen2.5-72B
Mistral-7B
Method
Acc7
MMLU
Ace,
MMLU
Acc7 MMLU
Acc,
MMLU
Acc7
MMLU
Acc7
MMLU
Acc7 MMLU
0-shot
5-shot
0-shot
5-shot
0-shot
5-shot
0-shot
5-shot
0-shot
5-shot
0-shot
5-shot
0-shot
5-shot
Dense
66.69
45.85
73.66
68.80
70.05
65.26
76.29
78.71
70.34
74.21
75.58
86.08
70.44
62.34
CATS 25%
63.71
42.76
72.74
67.50
67.66
61.85
75.58
77.93
69.66
72.67
72.77
84.91
64.55
59.83
TEAL
65.94
44.66
73.31
67.70
69.40
63.85
73.23
74.86
69.76
73.21
75.05
85.44
70.06
61.51
LaRoSA254 25%
66.39
45.66
73.38
68.74
69.54
64.85
76.30
78.13
70.12
73.74
75.53
85.62
70.25
61.81
CATS 40%
49.55
24.67
62.74
55.83
55.11
31.82
72.07
72.12
61.83
63.99
63.12
80.95
59.62
44.31
TEAL40%
64.92
43.46
72.47
66.78
68.14
59.84
72.24
73.23
68.61
71.44
74.65
84.80
68.76
60.17
LaRoSA40%
66.15
44.66
73.31
68.16
68.79
62.61
75.41
77.62
69.67
72.33
75.35
85.53
69.44
61.15
TEAL -50%
63.22
39.57
71.92
64.43
64.92
52.78
70.80
69.20
67.76
68.53
73.74
83.54
66.73
57.34
LaRoSA50%
64.61
43.10
72.86
67.57
67.19
58.65
73.81
76.51
69.09
70.09
75.18
84.34
68.46
58.80
ated on the same 128 random samples with a 2048-token context length. For comprehensive assessment, we utilize Im Im-evaluation- anr-evlluattio-aamssSSSi et al., 2023) to assess LaRoSA performance on variety of downstream tasks, includ- ing O-shot tasks for ARC-Easy (Clark et al., 2018), ARC Challenge (Clark et al., 2018), PIQA (Bisk et al., 2020), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), OpenBookQA (Mihaylov et al., 2018) and WinoGrande (Sakaguchi et al., 2021). We also report 5-shot accuracy for MMLU (Hendrycks et al., 2020).
Table 2. Perplexity results under different sparsity settings. The numbers in the first column reflects the actual model-level sparsity
LLaMA3
LLaMA2
Mistral
Qwen2.5
Method
8B
70B
7B
70B
7B
7B
72B
Dense
6.13
2.85
5.47
3.32
5.31
6,85
3.87
CATS
7.22
3.56
5.99
3.81
6.38
7.58
4.74
TEAL25%
6.37
3.95
6.31
3.43
5.53
6,93
3.93
LaRoSA25%
6,23
2.94
5.51
3.34
5.34
6.90
3.92
CATS40%
18.37
5.97
45.46
10.75
23.44
11.06
7.15
TEAL
6.83
4.45
6.40
3.61
5.98
7.20
4.07
LaRoSA40%
6.60
3.37
5.64
3.44
5.44
7.10
4.06
5.2.
Main Results
TEALs0%
7.56
5.61
6.80
3.91
7.17
7,81
4.31
Language Generation Task. First, we evaluate the accu- racy of LaRoSA with SOTA baselines on language genera- tion task. As indicated in Table 2, LaRoSA notably outper- forms other baselines on perplexity across various models and different sparsity settings. Under a low sparsity setting like 25%, LaRoSA achieves minimal PPL loss of merely 0.1 on LLaMA3-70B. This significantly surpasses previous methods, such as CATS with PPL loss of 0.71 and TEAL with a PPL loss of 1.1. LaRoSA maintains its advantage even at high sparsity levels, such as 60%. For instance, on Qwen2.5-72B, LaRoSA incurs only a 1.03 PPL loss, con- siderably outperforming TEAL, which suffers a loss of 3.87. Moreover, LLaMA2-7B, LaRoSA at 50% sparsity even outperforms the previous SOTA methods under 25% spar- sity setting, highlighting its superior efficiency. In TEAL's paper, they only sparsify 99% of tokens in the prefill stage because the empirical thresholds they use struggle to spar- sify initial tokens. In contrast, LaRoSA does not suffer this performance degradation and can achieve full sparsity in the input sequence length dimension.
LaRoSAs0%
7.22
4.10
5.87
3.62
5.62
7.42
4.26
TEAL
9.19
9.97
7.82
4.53
8.05
9,99
7.45
LaRoSAg0%
8.57
5.51
6.40
3.98
6.04
8.42
4.90
al., 2024a). For SOTA baselines, we choose not to quote the results from their papers in order to make fair compar- isons under the same model-level sparsity setting and to impose full sparsity in the input sequence length dimension.
Implementation Details. We use the WikiTex12 train set (Merity et al., 2016) as calibration dataset for LaRoSA and other reproducible works. We randomly select 16 se- quences with sequence length of 2048 tokens compute the rotation matrices Q for LaRoSA and empirical distri- butions for CATS and TEAL. The computation of Q is performed on 8x80G A100 GPUs, taking approximately 12 minutes to complete for the LLaMA3 70B model. For sparsity coefficient a, we employ Grid Search to find the optimal hyperparameter for each model, as shown in Ap- pendix B Algorithm 1. The optimal for each activation type of models is presented in Appendix B Table 11. For fair comparison, we implement full activation sparsification at prefill stage for all methods including TEAL.
Zero-shot and Few-shot Tasks. LaRoSA exhibits superior performance when applied to downstream language tasks. We present the end-to-end performance of LaRoSA and SOTA methods across different models, tasks, and sparsity ratios. The results are shown in Table 1. We highlight sev-
Evaluation. We report the perplexity (PPL) score on Wiki- Text2 test set for generation task, All models are evalu-
6

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
Table Single-batch token generation speed (token/see) results of LaRoSA. We exclude MM since it is architecturally similar
to to LLaMA3-70B. Tensor parallelism (TP2) is set for LLaMA3-70B and Qwen2.5-72B.
LLaMA2
LLaMA3
Qwen2.5
Mistral
GPU
Sparsity
7B
8B
70B
7B
72B
7B
Dense
109.95 (1.00x)
101.25 (L.00x)
22.05 (1.00x)
107.54 (1.00x)
19.32 (1.00x)
105.88 (1.00x)
0%
96.75 (0.88x)
92.13 (0.89x)
20.32 (0.90x)
100.97 (0.90x)
56.33 (0.92x)
56.33 (0.88x)
A100
25%
125.34 (1.14x)
111.37 (1.10x)
24.70 (1.12x)
123.67 (1.15x)
22.60 (1.17x)
118.59 (1.12x)
50%
151.73 (1.38x)
131.62 (1.30x)
29.77 (1.35x)
152.71 (1.42x)
29.17 (1.51x)
148.23 (1.40x)
75%
188.01 (1.72x)
174.15 (1.72x)
38,59 (1.75x)
180.67 (1,68x)
33.42 (1.73x)
186.35 (1.76x)
Dense
72.47 (1.00x)
67.19 (1.00x)
15.75 (1.00x)
71.70 (1.00x)
14.80 (1.00x)
72.89 (1.00x)
0%
65,95 x0)
60.47 (0.90x)
14.33 (0,91x)
63.81 (0.89x)
13.62 (0.92x)
72.89 (0.90x)
H20
25%
85.51 (1.18x)
79.28 (1.18x)
19.21 (1,22x)
83.17 (1.16x)
18.35 (1.24x)
65.60 (1.16x)
50%
104.36 (1.44x)
94.74 (1.11)
22.84 (1,45x)
98.95 (1.38x)
21.46 (1.45x)
84.55 (1.40x)
75%
136.97 (1.89))
126.99 (L89x)
29.92 (1.90x)
132.64 (1.85x)
28.42 (1.92))
102.05 (1.90x)
SeepSeek-R1-Distil--nee 8B and DeepSeek-R R1nns-RR-Diissee 1-Distilll- Qwen2.5-7B (DeepSeek-A) 2025). We utilize the Alpaca dataset, which not inherently related to complex reasoning tasks, to derive the rotation matrix Q2 for each layer. The evaluation of LaRoSA on these complex tasks is carried out with the Hugging Face Open-R1 repository (Face, 2025). As shown in Table 4, our experimental results indicate that at a sparsity level of 25%, both reasoning models exhibit only slight performance degradation on complex tasks, This finding demonstrates that the I LaRoSA method main- tains its effectiveness even when utilizing data that is not directly related to complex reasoning tasks for generating the rotation matrix.
Table 4. Accuracy of math and knowledge tasks on reasoning mod- els under different sparsity settings.
MATH- 500
GPQA- Diamond
AIME- 2024
Method
DS--R--Ditill-LLee -1ama3-8B
DeepSeck-RI Report
89.1
49.0
50.4
Reproduce Result
87.6
45.9
40.0
+LaRoSA A 25%
85.0
44.9
40.0
+LaRoSA 40%
80.6
43.1
36.7
+LaRoSA 50%
78.8
41.2
33.3
DS-RI-DDistll-Owen22..
DeepSeek-RI Report Reproduce Result
92.8
49.1
55.5
93.4
43.9
50.0
Inference Speed Up. We demonstrate consistent end-to- end efficiency improvements with LaRoSA. For this, we collected ten samples, each consisting of 128 tokens, from various test datasets and generated new sequences with lengths ranging from 128 to 2048 tokens. This was done to evaluate performance across different generation lengths. We applied 50% sparsity to LLaMA3-8B and Qwen2.5-7B using optimal sparse patterns obtained from LaRoSA and TEAL. As depicted in Figure 4, our LaRoSA realizes more consistent wall-clock time speed-ups compared to TEAL. This improvement is attributed to the TopK strategy which precisely zeros out activation entries needed for a specific sparsity level. As the output sequence length increases, LaRoSA outperforms TEAL in overall speed-up efficiency, demonstrating its effectiveness and robustness. Moreover, evaluated the end-to-end token generation speed-up on different types of models at a fixed output sequence length of 128, with results detailed in Table 3. For LLaMA3-70B, LaRoSA achieves notable speed-ups of up to 1.45x and 1,90x at 50% and 75% sparsity, respectively. The speed-up for 0% sparsity is lower than dense model, due to the ad- ditional computation introduced by residual adapters and sparsification functions. Nonetheless, we observe overall in- ference efficiency improvements as sparsity increases. The
+L.ROSA 25%
91.0
43.5
46.7
+LaRoSA 40%
88.5
41.0
43.3
+LaRoSA 50%
85.1
40.2
43.3
eral key observations: 1) LaRoSA significantly outperforms CATS and TEAL across shot tasks and 5-shot MMLU. With the same 40% model-level sparsity, LaRoSA achieves an average performance gain of 16.60% over CATS and 1.23% over TEAL on LLaMA2-7B. 2) LaRoSA achieves performance comparable to the dense model with negligible degradation at low sparsity level around 25%. For example, LaRoSA shows only a 0.22% accuracy drop on Qwen2.5- 7B compared to the dense model. 3) For some models, such as L.AMM2-7B, LaRoSA at 40% sparsity performs even better than CATS and TEAL do at 25% sparsity. These results demonstrate the superiority of our LaRoSA, which establishes new state-of-the-arr performance by effectively sparsifying input activations.
Complex Problem Solving Tasks. To further evaluate the effectiveness and robustness of LaRoSA, we conduct experiments on complex tasks such as MATH500 (Light- man et al., 2024), GPQA-Diamond (Rein et al., 2024), and AIME'24 24 (AIME, 2025) using two reasoning models:
7

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
Table 5. Impact of different sparsification functions on model's
LaRoSA WilTTet2
TEAL WilTeNtZ
actual sparsity and perplexity results. Experiments are performed on LLaMA2-7B. Sparsity is set to 50%.
LaRoSA, Alpaca
TEAL Alpaca
LaRoSA GSMBK
CSMBK
LaRoSA OBQA
TEAL OBQA
Method
Attention
MLP
Model |Sparsity
PPI.
1.25
hx(%) hz(%)
haa%))
(%)
Spocdup
TEAL
(7.9( (4.7 (8.4(=1.4)
(4.95
48.8
6.80
1.15
Topk
(0.0 (0.0 (0.0
(0.0
50.0
6.25
1.10
TopK+GS (5.0 (5.0
(0.0
(z.5
50.0
6.02
1.05
LaRoSA
1o.0) 65.0 (L.0) (7.5
50.0
5.87
1.00
64
256
512
768
1024
2048
Output Sequen Length
Table 6. Influence of different rotation types in various sparsity settings. Baseline method represent TopK+GS.
(a) Inference Speed-up for LLaMA3-SB.
1.5
LaRoSA WikTeet22
WiNText2
Method
LLMMA3-BB
LLaMA2-7B
LaRoSA Alpaca
TEAL Alpaca
PPL
TFLOPs
PPL
TFLOPs
1.4
LaRoSA OBQA
LaRoSA GSMBK
TEAL OBQA
GSMBK
Spandup Rotio
Dense
6.13
32,94 (1.00x)
5.47
29.26 (1.00x)
1.3
Bascline25%
6.29
32,94 (1.00x)
5.53
29.26 (1. (1.00x)
Baselinea5% + Qu 6.35
32.94 (1.00x)
5.64
29.26 (1.00x)
1.2
Bascline25% + Q 6.23
37.22 (1.13x)
5,51
33.06 (1.15x)
Baselinegs% + Q 6.23
34.91 (1.06x)
5.51
31.46 (1.07x)
1.1
Baselineso%
7.71
32,94 (1.00x)
6.02
29.26 (1.00x)
1.0
64
128
256
512
768
1024
2048
Baselinesa% + Qu
7.76
32.94 (1.00x)
6.03
29.26 (1.00x)
Output Sequence Length
Baselinec0% + QB
7.21
37.22 (1.13x)
5.86
33.06 (1.15x)
(b) Inference Speed-up for Qwen2.5-7B.
Baselinet0% + Qu
7.22
34.91 (1.06x)
5.87
31.46 (1.07x)
Baselinen0%
9.39
32.94 (1.00x)
6.91
29.26 (1.00x)
Figure 4. Comparison on inference speed-ups at 50% sparsity. Ex- periments are conducted NVIDIA A100 GPUs.
Baselinca0% + Qu
9.43
32.94 (1.00x)
6.92
29.26 (1.00x)
Baselinca0% + Q
8.56
37.22 (1.13x)
6.40
33.06(1. (1.55)
Baselincou%
Q
8.57
34.91( 1.06x)
6.40
31.46 (1.07x)
denotes the approach of introducing sparsity directly using the TopK function. 3) TopK+GS indicates enhancement of Topk by incorporating Grid Search to obtain optimal coefficients a for different inputs within the same block. 4) LaRoSA represents the complete method, which introduces a layerwise rotation matrix Q: on top of TopK+GS. Table 5 shows that the actual sparsity of the TEAL exhibits fluctu- ations and fails to achieve a 50% model-level sparsity. In contrast, TopK function can consistently maintain the actual sparsity of each input activation around 50% and signifi- cantly reducing PPL. By introducing the optimal sparsity coefficient a found through Grid Search, the PPI. loss can be further minimized while keep the overall model sparsity remains unchanged. Finally, incorporating layerwise orthog- onal rotation matrices further enhance model's performance.
detailed extra computation in TFLOPS are listed in Table 19.
Impact of Initial Tokens. As noted in Section 5.4.3 of the TEAL paper, TEAL applies activation sparsification to only 99% of tokens during the prefill stage. This approach is attributed to attention sinks (Xiao et al., 2024), which cause more severe degradation when sparsifying the initial tokens. Consequently, TEAL skips the first 1% of tokens to maintain model performance. However, we argue that this issue arises due to mismatch between the actual and cali- brated thresholds for each token. As illustrated in Figure 3b, this mismatch is particularly pronounced at the beginning of sequences, leading to TEAL's suboptimal performance on initial tokens. In contrast, LaRoSA and CATS apply sparsification uniformly across all tokens without special treatment. For a fair comparison, we implement full ac- tivation sparsification for TEAL, which results in slightly worse performance than originally reported by TEAL. This outcome also underscores the lack of robustness in using empirical thresholds.
Model-level vs. Layer-level vs. Block-level Rotation. We conprehensively investigate the number and sources of ro- tation matrices and their impact on the performance and efficiency of sparsified models. Three types of rotation ma- trices are analyzed: 1) Model-level rotation: This approach averages the covariance matrices of input across all layers to obtain single and shared QM for all layers, without adding residual adapters. 2) Block-level rotation: Each Attention and MLP block in layer receives its own rota- tion matrix QB, derived from its input's 's covariance matrix. Thus, a model with L layers generates 2L rotation matrices and 2L residual adapters. 3) Layer-level rotation: Used by LaRoSA, this strategy assigns a unique rotation matrix Q
5.3. Ablation Study
Impact of Sparsification Function. We ablate four activa- tion sparsification functions in LaRoSA: 1) TEAL refers to the function that uses a fixed cutoff threshold to prune low- magnitude entries, as described in Equation (5). 2) TopK
8

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
Table Impact of differrnt calibration datasets. The samples are used for calculating the covariance matrices for input activations.
Table 9. Compatiblity of LaRoSA with 4-hit weight quantization.
Precision
Quant. Methods
LLaMA3-8B
ILaMA2-7B
Method
Calibration Dataset
LLaMA3 8B
LLaMA2 7B
PPL(4)
Ace7(1)
PPL(()
Accr(t)
PPI(L) Accr(1)
(PL(()
Accr(t)
FP16
6.13
70.05
5.47
66.69
Dense
6.13
70.05
5.47
66.69
W4A16
RTN
8.52
65.32
5.73
62.54
W4A16
GPTQ
6.54
68.72
5.69
64.89
LaRoSAas9
WikiTex12
6.13
69.54
5,51
66.39
W4A16
AWQ
6,48
68.95
5.60
65.12
LaRoSA259: 25%
Alpaca
6.13
69.87
5.52
66.36
W4A16
OmniQuant
6.50
68.60
5.58
64.94
LaRoSA 40%
WikiTex12
6.60
68.79
5.64
66.15
LaRoSA25%
6.13
69.54
5.51
66.39
LaRoSAuss 400%
Alpaca
6.61
68.69
5.72
66.33
LaRoSA25%
RTN
8.64
64.90
5.81
62.19
LaRoSAsory LaRoSAsuss
WikiText2
7.22
67.19
5.87
64.61
LaRoSA
GPTQ
6.58
68.20
5.74
64.64
Alpaca
7.23
67.60
5.94
64.85
LaRoSA25%
AWQ
6.56
68.44
5.70
64,90
LaRoSA25%
OmniQuant
6.60
68.32
5,68
64,86
LaRoSAso%
7.22
67.19
5.87
64.61
Table Influence of calibration datasets on reasoning models.
LaRoSAso%
RTN
9.54
62.40
6.10
60.43
Cos.Simi. is obtained by averaging the cosine similarities of the covariance matrices across layers when using different calibration
LaRoSAso%
GPTQ
7.64
65.87
6.07
62.91
LaRoSA50%
AWQ
7.58
66.10
6.01
63,80
data. Experiments are performed on D-iistill--wee222
LaRoSA50%
OmniQuant
7.62
66.14
5.96
63.52
Method
Calibration
Cos.
0-shot
MATH-
GPQA-
Dataset
Simi.
Ace;
500
Diamond
formance.
This ablation study highlights the robustness of our LaRoSA approach.
LaRoSA25%
Alpaca
1.0
82.9
91.0
43.5
LaRoS 225%
WikiText2
0.998
83.0
91.0
43.7
LaRoSA25%
C4
0.997
82.9
91.2
43.5
Compatiblity with Quantization. We demonstrate that LaRoSA also shows high compatibility with latest SOTA weight quantization methods. detail, we experimented with LaRoSA using 4-bit quantization(W4A116 on model weights by employing RTN (Dettmers et al., 2022), GPTQ (Frantar et al., 2022), AWQ (Lin et al., 2024b) and OmniQuant (Shao et al., 2024) methods. Table 9 displays the perplexity and zero-shot accuracy results. With 4-bit GPTQ quantization, LaRoSA achieves 64.64% zero-shot accuracy and 5.74 PPL for LLaMA2-7B at 25% sparsity, These results are nearly on par with the the GPTQ quantized model's 64.89% accuracy and 5.69 PPI.. This indicates that the empirical errors from sparsity and quantization are some- what orthogonal to each other. The compatibility of LaRoSA with weight quantization highlights potential efficiency im- provements through optimized kernels that integrate both sparsification and quantization operations.
to each layer based on its input covariance matrix, resulting in L rotation matrices and L residual adapters. As shown in Table 6, these rotation types result in different additional computation overhead and distinct improvements in PPL. Although Qar does not introduce additional overhead, it actually results in worse performance compared to the base- line. Distinct Qn for each block significantly increases computational overhead, yet offers only marginal advan- tages over Q Consequently, the layerwise rotation matrix Qr. emerges as the optimal choice, as it increases limited computations with notable performance enhancements,
Influence of Calibration Datasets. We apply LaRoSA to sparsify the LLLMM2-7B and LLaMA3-8B models us- ing different calibration datasets (i.c. WikiText2 and Al- paca (Taori et al., 2023)), with results shown in Table 7, It can be observed that the choice of calibration datasets has relatively minimal effect on sparsified model performance. This is due our method utilize calibration data solely for calculating covariance matrices of each layer's 's input acti- vation, rather than for empirical distribution computation as employed by methods like CATS and TEAL. Addition- ally, we validate the performance of LaRoSA within the reasoning model on complex tasks. As shown in Table 8, the covariance matrices of input activations within the same layer exhibit high degree of cosine similarity across different data inputs. This observation suggests that the acquisition of layerwise rotation matrices is not strongly correlated with the calibration data. The performance in zero-shot and more complex tasks further demonstrates that the calibration data has minimal impact on the model's 's per-
6. Conclusion
This work presents LaRoSA, training-free activation spar- sification method that significantly enhances the efficiency of LLM inference. By utilizing orthogonal rotation matri- ces and Top- K sparsification functions, LaRoSA effectively optimizes sparse patterns for input activations, achieving minimal performance degradation along with consistent in- ference acceleration. Extensive experiments demonstrate that LaRoSA is robust and effective across various model types, sizes, and sparsity levels, offering superior perfor- mance and notable improvements in wall-clock time speed- up. LaRoSA establishes new state-of-the- results in acti- vation sparsification scenarios, enhancing the deployment of efficient LLMs in resource- conotreecon environments.
9

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
Impact Statement
DeepSeek-AI. Deepseek-rl:
Incentivizing reasoning ca- pability in Ilms via reinforcement learning. 2025. URL 12948.
This paper aims to contribute to the advancement of Ma- chine Learning through the exploration of activation sparsi- fication techniques. These techniques have the potential to enhance model efficiency and scalability, leading broader applicability across various domains. While there are nu- merous possible societal implications of our work, we do not identify any that require specific emphasis at this time.
https: : wwwpxii.org/abs///eo
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Llm.int8(): 8-bit matrix multiplication for transformers at scale. In Proceedings of the 36th International Confer- ence on Neural Information Processing Systems, NIPS '22, 2022. ISBN 9781713871088 1088.
Face, H. Open rl: A fully open reproduction of deepseek- rl. January 2025, URL https: /ttpt/gtttt.////e//// com/
References
AIME. AIME problems and solutions, 2025. URL http::///aatofprcolemmmlliggggg commwwkii ptt.rrooblemmmsmmm__sllllliiiooi
nuuggingfacc//pen---
Fang. G., Yin, H., Muralidharan, S., Heinrich, G., Pool, J., Kautz, J., Molchanov, P., and Wang, X. Maskllm: Learnable semi-structured sparsity for large language models. Advances in Neural Information Processing Systems, 2024.
Alizadeh, K., Mirzadeh, I., Belenko, D., Khatamifard, K., Cho, M., Mundo, C. C. D., Rastegari, M., and Fara- jtabar, M. Llm in a flash: Efficient large language model inference with limited memory, 2024. URL. https: /arxivv.vvvvv////////ea///aaaav/ wwg//b//aaoooooooooo
Frantar, E. and Alistarh, D. SparseGPT: Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.00774 2023.
Ashkboos, S., Croci, M. L., do Nascimento, M. G., Hoe- fler, T., and Hensman, J. Slicegpt: Compress large lan- guage models by deleting rows and columns, 2024a. URL https: / arxii.ivvv////vv org/ abs/ 2404...11 15024.
Frantar, E., Ashkboos, S., Hoefler, T. and Alistarh, D. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210.17333, 2022.
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, Quarot: Outlier-free bit inference rotated llms. arXiv preprint arXiv:2404.00456. 2024b.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi,
A., Foster, C., Golding, I.., Hsu, J., Le Noac'h, h, A., Li,
H., McDonell, K., Muennighoff, N., Ociepa, C.,, Phang,
J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika,
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
L.. Tang, E., Thite, A., Wang, B., Wang. K., and Zou, A. A framework for few-shot language model evaluation,
12 2023. URL https: /eoodo atg/reccods/ 10256836.
Chen, X., Liu, Z. Tang, H., Yi, L., Zhao, H., and Han, S. Sparsevit: Revisiting activation sparsity for efficient high-resolution vision transformer. 2023 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 2061-2070, 2023.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Ivanov, A., Dryden, N., Ben-Nun, T., Li, S., and Hoefler, T. Data movement is all you need: A case study on optimizing transformers. ArXiv, abs/2007.0000 .00072, 2020.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M.. and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprini arXiv: 2019.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint /rXiv: 1803.05577 2018.
A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. F. Mistral 7b, 2023. URL https: //arxiv. org wwwp///g////////e///////eeee/e/e aaaaabas///3/ 06825.
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. FlashAt- tention: Fast and memory- exact attention with 10 no-awareness. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2022.
Kurtz, M., Kopinsky, J., Gelashvili. R., Matveev, A., Carr, J., Goin, M., Leiserson, W., Moore, S., Shavit, N., and Al- istarh, D. Inducing and exploiting activation sparsity for fast inference on deep neural networks. In Proceedings of
10

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
the 37th International Conference Machine Learning, pp. 22.3533-20 55-3,5533 2020.
Mirzadeh, S. Alizadeh--ahid,, K., Mehta, S., del Mundo, C. C., Tuzel, O., Samei, G., Rastegari, M., and Farajtabar, M. Relu strikes back: Exploiting activation sparsity in large language models. In The Twelfth International Conference on Learning Representations, 2024.
Lee, D., Lee, J., Zhang, G., Tiwari, M., and Mirhoseini, A. CATS: Context-aware thresholding for sparsity in large language models. In First Conference on Language Modeling, 2024.
OpenAL. Gpt-4 technical report, 2024. URL https: arxiv. wwxiv.org//bs//30035 08774.
Li, Z., You, C., Bhojanapalli, S., Li, D., Rawat, A. S., Reddi, S. J,, Ye, K. Q., Chern, F, Yu, F. X., Guo, R., and Ku- mar, S. The lazy neuron phenomenon: On emergence of activation sparsity transformers. International Conference on Learning Representations, 2022.
Qwen, :, Yang. A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F. Wei, H., Lin, H., Yang. J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li M., Xue, M. Zhang, P., Zhu, Q. Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y. Cui, Z., Zhang, Z., and
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024.
Qiu, Z. Qwen2.5 technical report, 2025. URL https: // //arxiv org/ absp///a/////// /
Lin, H., Xu, H., Wu, Y.. Cui, J., Zhang, Y., Mou, L., Song, L., Sun, Z., and Wei, Y. Duquant: Distributing outliers via dual transformation makes stronger quantized llms. arXiv preprint ar arXiv:2406.0172) 2024a.
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024, URL https: //openreview. net www.ooomiddiiiieoo hwt.:iiddiii555000
Lin, J., Tang, J., Tang, H., Yang. S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, Awq: Activation-awe awwrre weight quantization for IIm compres- sion and acceleration. In MLSys, 2024b.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
Liu, J., Ponnusamy, P., Cai, T., Guo, H., Kim, Y., and Athiwaratkun, B. Training-free activation sparsity in large language models, 2024a. URL https:://////. //tpxva.xvvvvv/ org/aba 2408. 14690.
Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K., Gao, P., Qiao, Y., and Luo, P. Omniquant: Omnidirectionally calibrated quantization for large lan- guage models. In The Twelfth International Conference on Learning Representations, 2024.
Liu, Z., Wang, J., Dao. T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Re, C., et al. Deja vu: Contextual sparsity for efficient ilms at inference time. In International Conference on Machine Learning, pp. 22137-22176. PMLR, 2023.
Shazeer, N. Glu variants crant
transformer, 2020. URL
Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Kr- ishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. Spinquant-IIm quantization with learned rotations. arXiv preprini arXiv:2405. 2024b.
So, D., Manke, W., Liu, H., Dai, Z., Shazeer, N., and Le,
Q. V. Searching for efficient transformers for language modeling. In Ranzato, M., Beygelzimer, A., Dauphin, Y.. Liang, P., and Vaughan, W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 6010-6022. Curran Associates, Inc., 2021.
Ma, X., Fang, G., and Wang, X. LIm-pruner: On the struc- tural pruning of large language models. In Advances in Neural Information Processing Systems, 2023.
Song, C., Han, X., Zhang, Z., Hu, S., Shi, X., Li, K., Chen, C., Liu, Z., Li, G., Yang, T., and Sun, M. Prosparse: Introducing and enhancing intrinsic activation sparsity within large language models, 2025. URL https:// a arxiv. www.vorrg/bbb.. abs 2402. 13516.
Merity, S., Xiong, C., Bradbury, J., and Socher, R.
Pointer sentinel mixture models. ArXiv, 4b77140,,,7843, 2016. URL http:://pp..emmatticcciiet www.stiisscoolll........ org/Coo:usID:162290
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv: wttp:///9900999 2018.
Song, Y., Mi, Z., Xie, H., and Chen, H. Powerinfer: Fast large language model serving with a consumer-gradd gpu, 2023.

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
Song, Y., Xie, H., Zhang, Z., Wen, B., Ma, L., Mi, Z., and Chen, H. Turbo sparse: Achieving IIm performance with minimal activated parameters, 2024. URL https: //arxiv. org/ abs/ 2.bbs/244 /406 05955.
Sun, M., Liu, Z., Bair, A., and Kolter, J. simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 11695, 2023.
Taori, R., Gulrajani, T., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P. and Hashimoto, T. B. Stanford alpaca: An instruction-follovinng llama model. hitps: / stanford_alpas 2023.
Team, L. The llama herd of models, 2024. URL https:
//arxiv. org/ abs 2407 21783.
Touvron, H., Martin, L.,, Stone, K. R., Albert, P., Almahairi,
A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D. M., Blecher, L., Ferrer, C. C.,
Fu, Chen, M., Cucurull, G., Esiobu, D., Fernandes,
J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A. S., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I. M., Korenev, A. V., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X., Tang, B., Taylor, R., Williams, A. Kuan, J. X., Xu, P. Yan, 7., 7arov, I., Zhang, Y., Fan, A., Kambadur, M. H. M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307.092222 2023.
Wang, H., Ma, S., Wang, R., and Wei, F. Q-sparse: All large language models can be fully sparsely-activated, 2024. arxiv. org/ abs 2407. 10969.
URL https:
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024. URL https: htthttss::///peeeeeee /ttp:///ppeeeeeieeee/mm
n
Zellers, R., Holtzman, A., Bisk. Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: /005./990007500 000300030,, 2019.
Zhang, Z.. Song. Y., Yu, G., Han, X., Lin, Y., Xiao, C., Song, C., Liu, Z., Mi, Z., and Sun, M. Relu2 wins: Discovering efficient activation functions for sparse llms. ArXiv, abs/2402.03804, 2024.
12

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
A. Theoretical Analysis
This section provides a detailed explanation how LaRoSA utilizes orthogonal transformations and top-k sparsification, and why it achieves consistently lower empirical errors compared to TEAL (Liu al., 2024a).
TEAL derives the theoretical errors from magnitude-based sparsification under restrictive assumption that weights and activations are independent and identically Gaussian distributed. However, we found that empirical errors generally larger than theoretical errors, as shown in Figure 5, which may be due to the incorrect assumption of a Gaussian distribution.
In fact, the rotated activation through orthogonal transformations follows an approximately Gaussian distribution. Specif- ically, elements of ã = 2Q are the weighted averages of elements æ. That is, for any i, ã; = Σド ZzQji where - O 1. By the Central Limit Theorem, if a; is independent and identically distributed (i.i.d.) random variables with zero mean, then I; will follow an approximately zero-mean Gaussian distribution.
After rotation, only top & elements of ã are retained. It can be shown that under the assumption that E and W -Q are independent and identically Gaussian distributed, the theoretical error of top k sparsification is determined solely by k.
Pa Das are independent and identically Gaussian distributed, where 2; ~ N N(0, ox) and W,; ja ~ (0, m.). Furthermore, for top-k sparsification method, we assume E[a² b & Topı (i)] E[a a; < l.). where satisfies P([ tx) D. distributional relative error is given by:
Theorem A.I. Lei & E RD, W E
k. For sparsification function Sk, define yº = (x(W) and y E EW, then the
k
k
k
22h-1 1
1
i Φ-1
1-
(1I)
E Elyyee
Din
2
2D
where v(t)
- 1 is the standard Gaussian probability density function, and T is the inverse cumulative 2/2""
distribution function (CDF) of the standard Gaussian distribution.
Proof. define y y y Due to independence and zero mean of W, for any + lo and any i, follows that E E[ ji Waa] = 0, which leads to:
Dio Σ
Dm (ig Sv (Biz)) W
= E
Di
Sa (a;,
W)
=1
===1
Dmn Dm
E
Sa (
Wie W Wkiz
(12)
Also for any i, according to Lemma A.1 in (Liu al., 2024a), we have
E
E
(.. - Sa(ū,)
Dm
E [
Sx
2
B|0 ã. $ TT)
Dm
EE
[
< tx]
1=1
1
k:
24 pp
(13)
D
13

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
1.0
Theoretical Errors
Empirical Errors(TEAL)
Empirical Errors(LaRoSA)
0.8
Relative Error
0.6
0.4
0.2
0.0
0.2
0.6
0.8
1.0
Sparsity
Figure Relative output error results from 14th Layer of LLMMA3-BB. Thcoretical errors are calculated with Top-K hased sparsification. Empirical errors derived from the output of Waon projection.
On the other hand, since H; are Gaussian distributed, standard Gaussian random variable. It follows that:
P([ S (x)
P[]=
4e) -24 16
where is
Di
tk
1-1
1 (1- 2D
k
(14)
As result, have:
B 1 2
Blg.1
Di Doul 20 (1
k D
2Φ
h
4
(1-
(1 1-
k
(15)
2D
2D
Note that
E| = Din D 2
(16)
as the unsparsified case. Equation (15) and Equation (16) lead the final result.
As illustrated in Figure 5, the improved Gaussian distribution assumption results in LaRoSA having consistent lower empirical errors compared to TEAL, bringing our proposed method closer to the theoretical errors. We employ magnitude- based top-K method instead of pruning the last few elements after rotation like SliceGPT (Ashkboos al., 2024a), which ensures the validity of the aforementioned theoretical error analysis. Our experiments across multiple models demonstrate that, after rotating into the principal component space, most small- mall-maggi elements of activation indeed concentrated in the last few dimensions.
14

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
B. Grid Search Optimized Top-K
In the work of TEAL, was noted that even under a target constraint of 50% layer sparsity, the sparsity the four input activations (i.e., h, h2. ht. and h4) still exhibit differing numerical distributions. Our exploration of the TopK approach, based on LaRoSA's framework, also revealed that forcing these four input activations to adopt the same TopK value can lead to suboptimal sparsity patterns in some models, even when corrective rotation matrices are introduced. To address this issue, we conducted a straightforward grid search for the sparsity coefficients on; 02, O3,, and that determines the allocation weights for ht, f22, h3, and i across all layers. To ensure that the sparsity within the block aligns with the model-level sparsity, we need to consider the proportional relationship between the dimension size of h1 and h2 (as well as h3 and h4). Specifically, for input activations of Q. K, V, and O projections, we have:
* 0n + 02 = 4.
(17)
For input activations in Up, Gate, and Down projections, we have
2 * 0g + M x 04 = 2 + M,
(18)
where M represents the ratio of the intermediate size to the hidden size. Thus, given and dz,, we have:
4 4-3 (11,
(19)
a4 (2 + - 2 ca)/M.
(20)
After obtaining the optimal coefficients a for h1, h2, h3, and h4s we apply the same allocation weights across all layers. The experimental results in Table 10 demonstrated that our sparsity coefficients search strategy effectively improves the performance of the sparsified model. Table 11, we provide a detailed presentation of the sparsity coefficients used across various models.
Table 10. LLaMA3-8B perplexity results on different sets of coefficients a for hT, h2, les, and hr.
Method
Attention
MLP
PPL
h1
h2
h3
h4
TopKau%
1.00
1.00
1.00
1.00
9.09
TopK+GS 60%
0.90
1.30
0.90
1.06
8.98
TopK 60%
0.85
1.45
0.70
1.17
8.94
TopK+GS 60%
0.80
1.60
0.80
1.12
8.86
TopK+GS 60%
0.75
1.75
0.75
1.14
8.93
LaRoSAgo%
0.80
1.60
0.80
1.12
8.57
Table 11. Optimal sparsity coefficients a of h1, h2, ha, and h4 used our proposed LaRoSA.
Models
Attention
MLP
h1
h2
h
h4
M
LLaMA2-7B
0.90
1.30
0.80
1.15
2.6875
LLaMA2-70B
0.80
1.60
0.80
1.12
3.5000
LLaMA3-8B
0.80
1.60
0.80
1.12
3,5000
LLaMA3-70B
0.85
1.45
0.75
1.14
3.5000
Mistral-7B
1.00
1.00
0.80
1.12
3,5000
Qwen2.5-7B
0.80
1.60
0.80
1.08
5.2857
Qwen2.5-72B
0.80
1.60
0.80
1.11
5.2857
15

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
Algorithm 1 Grid Search for Optimal Sparsity Coefficients
1: Input: Initial Top-K function value K, Ratio of the intermediate size to the hidden size M, Step size 0.05
2: Output: Optimal sparsity coefficients ai, 02 0gz 0f
3: Initialize best_PPL: best_ppl + +00
4: Initial sparsity coefficients 01 = 0.7, 03 = 0.7
5: while 0 < 1.2
6: ag+- 4 01
7: while 03 < 1.2 do
8: ast- M 2 as)/M
9: Compute current_PPP LPL, with new Top-K function value (a1 a3,, for h1, h2 ha, h4
10: if current PPL < _est__eP________ PPL then
11:
best_PPL PPL + current_PPL.
12:
aj f-i
ag F-3
end if
15: Og ag
16: end while
17: a1 + a1 +
18:
end while
19: a² 4-3 af
af f (2+
Return aj, az ag.aj aj
16

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
C. Full Experimental Results
Table 12. Full results of LaRoSA in zero-shot and few-shot tasks for LLaMA2-77.
Sparsity
Method Metrics
WinoGrande
PiQA
OBQA
HellaSwag
BoolQ
ARC-E
ARC-C
Avg
MMLU
acc
acc acc_nome nor_n acc, norm
acc _orm
acc
acc norm
acc_nome norm
acc
Baseline
69.14
79.05
44.20
75.98
77.71
74.54
46.25
66.69
45.85
CATS
67.64
77.80
41.20
75.75
70.55
69.70
43.34
63.71
42.76
25%
TEAL
67.95
78.18
44.00
75.20
76.71
73.91
45.64
65.94
44.66
LaRoSA
68.43
79.05
44.00
75.84
77.22
74.66
45.56
66.39
45.66
CATS
56.12
66.87
32.80
52.28
63.21
44.32
31.23
49.35
24.67
40%
TEAL
66.93
77.96
42.60
74.09
75.90
73.31
43.68
64.92
43.46
LaRoSA
68.90
78.18
44.20
75.35
76.85
73.86
45.73
66.
44.66
TEAL
65.27
77.53
41.20
71.39
73.33
71.17
42.66
63.22
39.57
50%
LaRoSA
67.32
77.69
43.40
74.37
73.85
72.31
43.34
64.61
43.10
Table 13. Full results of LaRoSA in zero-shot and few-shot tasks for LLaMA2-70B.
Sparsity
Method
WinoGrande
PiQA
OBQA
HellaSwag BoolQ
ARC-E
ARC-C
Avg
MMLU
acc
acc_norm acc_ _ec_norm acc_norm
acc
acc_norm ace_norm
ace
Baseline
77.97
82.80
48.80
83.81
83.79
81.10
57.33
73.66
68.80
CATS
76.32
82.26
49.20
84.22
79.94
79.63
57.59
72.74
67.50
25%
TEAL
78.56
82.42
48.80
82.66
83.25
81.26
56.24
73.31
67.90
LaRoSA
57.17
80.09
83.88
83.63
48.60
82.70
77.58
73.38
68.74
CATS
67.25
77.69
43,00
75.11
71.41
61.11
43.60
62.74
55.83
40%
TEAL
76.21
81.65
48.30
82.02
82.94
80.87
55.30
72.47
66.78
LaRoSA
77.43
83.57
47.40
83.37
83.36
80.51
57.76
73.31
68.16
TEAL
75.74
81.33
47.40
80.67
81.80
80.35
56.16
71.92
64.43
50%
LaRoSA
76.80
82.86
48.80
82.99
82.78
79.63
56.14
72.86
67.57
Table 14. Full results of LaRoSA in zero-shot and few-shot tasks for L.MMM3-8B.
Sparsity
Method
WinoGrande
PiQA
OBQA
HellaSwag BoolQ
ARC-E
ARC-C
Avg
MMLU
acc
scc_norm acc_ _or_norm
acc_ _oc_norm
acc
acc_norm
acc_norm
acc
Baseline
73.64
80.41
44.80
79.13
81.10
77.78
53.50
70.05
65.26
CATS
70.24
79.33
44.40
77.38
79.17
73.70
49.40
67.66
61.85
25%
TEAL
72.48
80.36
45.40
78.45
80.07
77.08
52.00
69.40
63.85
LaRoSA
72.93
80.63
45.00
79.09
80.40
76.68
52.05
69.54
64.85
CATS
60.54
73.12
36.00
66.82
59.14
52.78
37.37
55.11
31.82
40%
TEAL
70.34
78.51
45.80
76.55
79.57
76.47
49.74
68.14
59.84
LaRoSA
71.11
79.87
44.00
77.85
80.09
76.98
51.62
68.79
62.61
TEAL
66.71
77.09
42.80
72.61
76.26
72.71
46.24
64.92
52.78
50%
LaRoSA
68.98
79.98
44.20
76.85
77.55
74.75
48.04
67.19
58.65
17

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
Table 15, Full results of LaRoSA in zero-shot and few-shot tasks for LLaMA3-70B.
Sparsity
Method
WinoGrande
PiQA
OBQA
HellaSwag BoolQ
ARC-E
ARC-C
Avg
MMLU
acc
acc_ _o__nomm acc_ _oc_nom acc_ _ec_norm
acc
rec_nomm acc_ aec_norm
acc
Baseline
80.11
84.49
48.80
84.97
85.22
86.15
64.33
76.29
78.71
CATS
79.87
84.06
47.20
85.21
84.43
85.19
63.14 14
75.58
77.93
25%
TEAL
76.63
82.31
48.2
81.82
83.24
81.64
58.78
73.23
74.86
LaRoSA
81.45
84.39
49.00
85.07
84.89
85.65
63.65
76.30
78.13
CATS
74.51
81.77
46.40
83.46
81.44
79.17
57.76
72.07
72.12
40%
TEAL
76.32
81.93
45.80
81.56
80.64
81.90
57.50
72.24
73.23
LaRoSA
79.72
83.62
48.40
84.71
85.05
84.26
62.12
75.41
77.62
TEAL
73.08
80.30
43.60
79.90
81.74
80.17
56.82
70.80
69.20
50%
LaRoSA
77.90
82.59
46.40
84.24
84.10
82.49
58.96
73.81
76.51
Table 16. Full results of LaRoSA in zero-shot and few-shot tasks for Mistral-7B-v0.3.
Sparsity
Method
WinoGrande
PiQA
OBQA
HellaSwag
BoolQ acc
ARC-E
ARC-C
Avg
MMLU
acc
ucc_norm
porm
acc_none mo__
acc morm
acc
Baseline
73.80
82.31
44.00
80.40
82.11
78.40
52.04
70.44
62.34
CATS
71.27
80.47
33.00
60.68
79.42
78.28
48.72
64.55
59.83
25%
TEAL
72.61
81.82
44.40
80.05
81.44
78.24
51.86
70.06
61.51
LaRoSA
72.85
82.15
44.40
80.14
81.71
78.49
52.05
70.25
61.81
CATS
61.96
74.43
34.80
74.71
74.86
55.13
41.47
59.62
44.31
40%
TEAL
69.55
80.36
44.40
79.08
80.49
77.14
50.30
68.76
60.17
LaRoSA
70.88
81.07
43.40
79.63
81.68
78.32
51.11
69.44
61.15
TEAL
68.71
79.52
40.40
76.97
79.55
74.34
47.63
66.73
57.34
50%
LaRoSA
69.93
81.61
43.20
78.93
80.92
75.55
49.06
68.46
58.80
Table 17. Full results LaRoSA in zero-shot and few-shot tasks for Qwen2.5-7B.
Sparsity
Method
WinoGrande
PiQA
OBQA
HellaSwag
BoolQ
ARC-E
ARC-C
Avg
MMLU
acc
acc_ _oc_norm
norm
acc_nooe por_
acc
acc _orm
acc_norm
acc__oo_
acc
Baseline
72.77
79.98
47.60
78.91
84.56
77.44
51.11
70.34
74.21
CATS
70.64
79.54
46.20
78.90
83.15
78.28
50.94
69.66
72.67
25%
TEAL
72.12
79.22
46.40
78.49
84.40
76.85
50.85
69.76
73.21
LaRoSA
72.69
79.92
46.60
78.77
84.40
77.02
51.45
70.12
73.74
CATS
58.72
76.12
38.60
72.49
74.01
67.89
44.97
61.83
63.99
40%
TEAL
70.30
78.78
45.00
76.78
84.05
75.44
49.96
68.61
71.44
LaRoSA
70.72
79.38
46.20
78.28
84.65
77.86
50.60
69.67
72.33
TEAL
68.90
77.18
44.00
74.60
83.76
76,59
49.34
67.76
68.53
50%
LaRoSA
70.24
79.11
45.60
77.20
83.55
78.03
49.91
69.09
70.09
Table 18. Full results of LaRoSA in zero-shot and few-shot tasks for Qwen2.5-72BB
Sparsity
Method
WinoGrande
PiQA
OBQA
HellaSwag
BoolQ acc
ARC-E
ARC-C
Avg
MMLU
acc
acc_norm
acc_norm
acc_norm
acc_norm
acc_norm
acc
Baseline
77.90
83.62
46.40
86.05
89.14
83.33
62.62
75.58
86.08
CATS
73.72
82.86
45.40
86.60
82.57
79.92
58.36
72.77
84.91
25%
TEAL
78.11
83.39
46.50
85.23
87.63
82.62
61.87
75.05
85.44
LaRoSA
77.66
83.73
46.80
85.97
89.17
82.83
62.54
75.53
85.62
CATS
65.59
78.56
33.20
61.81
82.14
74.71
45.82
63.12
80.95
40%
TEAL
77.47
83.01
46.50
84.52
87.52
82.43
61.11
74.65
84.80
LaRoSA
78.45
83.03
47.20
86.01
89.48
83.12
62.46
75.35
85.33
TEAL
77.32
82.25
45.00
83.03
87.54
81.61
59.46
73.74
83.54
50%
LaRoSA
76.48
83.57
45.60
85.81
89.51
83.59
61.69
75.18
84.34
18

---

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
Table 19. The change in TFLOPs of the model after adding the residual adaptor.
Prefill
Decode
Model
Original (TFLOPs)
LaRoSA (TFLOPs)
Original (TFLOPs)
LaRoSA (TFLOPs)
LLaMA2-7B
29.26(1.00x
31.46(1.07x))
29.26(1.00x)
31.46(1.07x)
LLaMA2-70B
292.44(1.00x)
314.43(1.07x)
292.45(1.00x))
314.44(L.07x)
LLaMA3-8B
32.94(1.00x)
35.14(1.06x)
32.94(1.00x)
35.14(1.06x)
LLaMA3-70B
295.67(1.00xx) 317.66(1.07x))
295.68(1.00x
317.67(1.07xx)
Mistral-7B
31.34(1.00x)
33.53(1.07x))
31.34(1.00x)
33.54(1.07x)
Qwen2.5-7B
30.64(1.00x)
32.11(1.05x))
30.64(1.00x)
32.12(1.05x)
Qwen2.5-7BB
303.69(1.00x) 32.57(1.07>x)
303.69(1 100x)
325.68(1.07x))
19


Reviews:
[{"review_id": "qKgpscdu3c", "reviewer": "Reviewer_nBNG", "summary": "This work proposes a sparse activation method for efficient inference by rotating inputs by an orthogonal matrix. Basic idea is to improve sparsity by rotating inputs to matrix multiplication happening in Transformer layers, e.g., MLP and attention, so that low magnitude inputs are pruned away in the rotated space without sacrificing representation in the original space. Experiments are carried out on diverse models and benchmarks demonstrating efficient inference without largely degrading performance when compared with other baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer nBNG for his/her thoughtful feedback and recognition of our work, which has greatly encouraged our research team. Our proposed LaRoSA framework utilizes Computational Invariance Transformation, a well-established technique in model quantization and pruning, to systematically enhance activation sparsity during language model inference. When combined with our carefully designed top-k activation selection mechanism and hardware-optimized custom kernels, this approach consistently accelerates models ranging from 7B to 70B parameters during decoding.\n\nThe recent open-sourcing of DeepSeek-R1[1] and subsequent advancements in related research [2-6] have shown that reasoning models using long-chain-of-thought with RL-based test-time scaling make significant progress on complex reasoning tasks. These studies reveal an interesting emergent property: **having more thinking tokens during the decoding phase can significantly extend the capability boundary of the reasoning models**. For instance, DAPO[3] reports that RL training increases the mean response length from 1,000 to 4,000 tokens. The notable increase in output length during the decoding phase offers significant opportunities for optimization, and our LaRoSA method is valuable because it speeds up these lengthy decoding processes without sacrificing model performance.\n\nTo further validate LaRoSA's effectiveness, we conducted rapid empirical evaluations on distilled reasoning models, including DeepSeek-R1-Distill-Llama3-8B and DeepSeek-R1-Distill-Qwen2.5-7B[1]. As discussed in our response to Reviewer V3DZ, preliminary results show that **LaRoSA achieves consistent inference speed up while preserving the ability to solve complex reasoning tasks**. Additional ablation studies with Reviewer jWLg **demonstrate compatibility and effectiveness when integrating LaRoSA with semi-structured pruning techniques such as Wanda[7] and RIA[8], further confirming our method's robustness.**\n\nOnce again, we deeply appreciate reviewer nBNG's constructive comment on our work. Our team remains committed to refining this research area and will open-source both the implementation code and optimized kernels to encourage community adoption and further development.\n\n[1] Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.  \n[2] Demystifying Long Chain-of-Thought Reasoning in LLMs. arXiv preprint arXiv:2502.03373, 2025.  \n[3] DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv preprint arXiv:2503.14476, 2025.  \n[4] Open-reasoner-zero: An open source approach to scaling reinforcement learning on the base model, 2025.  \n[5] Understanding R1-Zero-Like Training: A Critical Perspective. arXiv preprint arXiv:2503.20783, 2025.  \n[6] SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild. arXiv preprint arXiv:2503.18892, 2025.  \n[7] A Simple and Effective Pruning Approach for Large Language Models, ICLR 2024.  \n[8] Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models, ICLR 2024.", "claims_and_evidence": "The proposed method is inspired by the recent work on quantization in the rotated space (Ashkboos et al., 2024) and applied to sparse activation pruning of (Liu et al., 2025). The approach is apt and its effectiveness is supported by experiments.", "methods_and_evaluation_criteria": "The approach is a sound method in that it is based on rotation by an orthogonal matrix estimated for each layer. Evaluation metrics are standard one to quantify the gains.", "theoretical_claims": "Theoretical analysis is provided in appendix A by contrasting their approach with TEAL (Liu et al., 2025).", "experimental_designs_or_analyses": "Experiments and analyses are exhaustive enough to demonstrate the effectiveness of the proposed approach.", "relation_to_broader_scientific_literature": "It has a contribution in efficiency in inference, in particular, by employing activation sparsity.", "essential_references_not_discussed": "No additional references are necessary.", "other_strengths_and_weaknesses": "Strengths\n\n- This paper is well presented and easy to read. Idea is well motivated by prior studies and experiemnts.\n- Engineering work will have impacts for future studies.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rSNU85S0Qy", "reviewer": "Reviewer_V3DZ", "summary": "Sparse activation is an active research field of LLMs. This paper demonstrates that using magnitude-based pruning to sparsity leads to inconsistent sparsity across tasks and layers. Stemming from their rich preliminary study, they propose La RoSA which leverages the rotation operation proposed in SliceGPT to address the above issues. While rotation operation is not novel, extending it to activation sparsification to address inconsistent sparsity is novel. Extensive experiments are done to verify the effectiveness of La RoSA.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer for the thorough review! Below is our point-by-point response:\n\n> I would like to see ... on harder tasks with more advanced models.\n\nWe agree that perplexity alone is insufficient, and follow your suggestion to evaluate LaRoSA on harder tasks with advanced models using Hugging Face Open-R1 repo.\n\n|Model|MATH500|GPQA Diamond|AIME 2024|\n|-|-|-|-|\n|**DeepSeek-R1-Distill-Llama3-8B**||||\n|DS-R1 Report|89.1|49.0|50.4|\n|Reproduce|87.6|45.9|40.0|\n|+LaRoSA 25%|85.0|44.9|40.0|\n|+LaRoSA 40%|80.6|43.1|36.7|\n|+LaRoSA 50%|78.8|41.2|33.3|\n|**DeepSeek-R1-Distill-Qwen2.5-7B**||||\n|DS-R1 Report|92.8|49.1|55.5|\n|Reproduce|93.4|43.9|50.0|\n|+LaRoSA 25%|91.0|43.5|46.7|\n|+LaRoSA 40%|88.5|41.0|43.3|\n|+LaRoSA 50%|85.1|40.2|43.3|\n\nLaRoSA preserves the math problem-solving and reasoning skills of distilled models, proving its robustness. Due to structural similarities with base models, their inference speedup matches the paper's results. These findings will be added in future revisions.\n\n> I found that we need more description for ... How many data are used?\n\nFigure 3b compares cutoff thresholds for activation pruning using TEAL and TopK methods at various sparsity levels. TEAL uses a fixed threshold from offline calibration, shown as a dashed line, while TopK computes thresholds during testing, shown as scatter points. We used 16 sequences of 512 tokens for various sparsity settings.\n\n**Note 1**: Ideally, actual thresholds should align with empirical ones at their sparsity levels. However, scatter points rarely match the dashed lines, indicating potential over-pruning.\n\n**Insight 1**: Empirical thresholds often misalign with real token activation, deviating from the intention of magnitude pruning.\n\n**Note 2**: Actual thresholds for initial tokens are often close to 0, and much lower than empirical thresholds.\n\n**Insight 2**: Empirical thresholds hold no real significance for initial tokens since they can zero out all activation values, harming model performance.\n\nWe will modify and extend this part upon acceptance. \n\n> Why larger sequence length leads to slower inference? \n\nAs context length increases, the proportion of attention computation and inference time required for each decoded token also grows. However, LaRoSA's speedup mainly comes from reduced memory access, which remains constant. Thus, acceleration benefits decrease with longer context lengths but still remain significantly faster than dense inference.\n\n> What is the role of inconsistent sparsity in this figure? \n\nIn Figure 3d, inconsistent sparsity arises from varied token distributions across different data sources, like Code v.s. Chat tokens. The figure illustrates that empirical thresholds can not achieve the target sparsity on test datasets from different domains.\n\n> Why we need consistent sparsity across layers to obtain a speed up?\n\n1）Inconsistent sparsity makes it difficult to ensure the model meets target sparsity, complicating efficiency predictions. 2）It results in variable inference times for each token. If a model compression method leads to uncertain and variable inference time for each token, it cannot guarantee efficiency and stability.\n\n> I found author claim ... Will this lead to mismatch and performance drop?\n\nIn LaRoSA, calibration data is only used for PCA to compute rotation matrices. We shows that choice of datasets does not affect performance in Table6. Moreover, we test on DS-R1-Distill-Qwen2.5-7B with various datasets and reveal two findings: **1) Rotation matrices $Q_l$ are similar across calibration datasets. 2) Changing calibration data does not lead to mismatching or performance loss in LaRoSA.**\n|Method|Cali.Data|Avg.Cos.Simi|0-Shot Acc|MATH 500|GPQA Diamond|\n|-|-|-|-|-|-|\n|LaRoSA 25%|Alpaca|1.0(self)|82.9|91.0|43.5|\n|LaRoSA 25%|WikiText2|0.998|83.0|91.0|43.7|\n|LaRoSA 25%|C4|0.997|82.9|91.2|43.5|\n\n> I am wondering if the inconsistent ... with a longer calibration dataset?\n\nWe tested different lengths of calibration data for Qwen2.5-7B and found that larger datasets did not resolve inconsistent sparsity or improve zero-shot performance. However, the source of the calibration data affects empirical thresholds, causing TEAL's performance to vary across datasets.\n|Method|Cali.Data/Size|16*2048|64*2048|256*2048|\n|-|-|-|-|-|\n|TEAL 25%|WikiText2|67.8|67.9|67.8|\n|TEAL 25%|Alpaca|69.7|69.7|69.6|\n\n\n> How sparse activation brings real speedup? \n\nThe columns in the weight matrix related to zero-valued activations become inactive weights. Since the decode phase of LLM inference is memory-bound, avoiding the loading of these weights can bring real speedup.\n\n> I assume different tokens will ... How can we address this mismatch in GPU?\n\nIndeed, different tokens activate different neurons. Our custom kernel dynamically selects and loads only required weight columns for each token into the GPU. Unloaded weights are set to zero, making them inactive in GEMM while keeping dimensions unchanged. This prevents any potential mismatch in GPU.", "claims_and_evidence": "The overall claims made in the submission are supported with good experiments, with some minor claims needing to be clarified. \n\n1. While perplexity is a good indicator for pre-training, it is not convincing enough to report PPL for LLM compression. See https://arxiv.org/pdf/2407.04965. I would like to see if sparsifying activations will cause any performance drop on harder tasks with more advanced models like DeepSeek-R1-Distill-Qwen on Math500 and GPQA. \n\n2. While I appreciate the rich information provided in Figure 3, I found that we need more description for Figure 3 for readers to understand it fully. Figure 3b is a bit too complicated. I can grasp the main idea but need some time to interpret the figure and the corresponding text descriptions in the paper.  How many calibration data are used in Figure 3? \n\n3. Could authors explain why larger sequence length leads to slower inference in Figure 3d? What is the role of inconsistent sparsity in this figure? Can the authors elaborate more on why we need consistent sparsity across layers to obtain a speed up?\n\n4. I found the authors' claim in Section 4 is a bit controversial. At the beginning of Section 4, the authors mentioned that using a calibration dataset is not precise enough, leading to inaccurate sparsity. However, in the later part, the calibration dataset is used to perform Layerwise Orthogonal Rotation. Will this also lead to mismatching and performance drop? \n\n5. I am wondering if the inconsistent sparsity of magnitude pruning can be mitigated with a longer calibration dataset?\n\n6. The inference speed-up in Figure 4 is interesting, demonstrating the benefits of LaRoSA. \n\n7. Another question is how sparse activation brings real speedup? I assume different tokens/samples will activate different sets of neurons. Will this constrain the practical usage of the proposed approach?", "methods_and_evaluation_criteria": "Using PCA instead of magnitude pruning appears to be a more precise option for sparse activations, as we have seen its effectiveness on structured pruning in SlideGPT.\n\nThe evaluation of this paper is relatively too easy. I would expect to see if the benefits still hold for complex reasoning tasks, such as math reasoning.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I appreciate the evaluation of multiple architectures and various sizes. It seems LaRoSA is more effective at high sparsity levels than TEAL. I would like to see if the benefits also hold for math reasoning tasks, not only ppl.", "relation_to_broader_scientific_literature": "The main idea of this paper comes from the SliceGPT paper. Sparse activation is an active research field and I believe the authors have provided a quite good literature review.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "**Strengths**\n\n1. Related work is well-written and provides a clear summary of previous works in this field. \n\n2. Experimental results seem to be strong, especially at high sparsity levels. \n\n3. Inference speed is reported. \n\n4. Ablation study explains the contribution of different components.\n\nWeaknesses: \n\n1. While perplexity is a good indicator for pre-training, it is not convincing enough to report PPL for LLM compression. See https://arxiv.org/pdf/2407.04965. I would like to see if sparsifying activations will cause any performance drop on harder tasks with more advanced models like DeepSeek-R1-Distill-Qwen on Math500 and GPQA. \n\n2. While I appreciate the rich information provided in Figure 3, I found that we need more description for Figure 3 for readers to understand it fully. Figure 3b is a bit too complicated. I can grasp the main idea but need some time to interpret the figure and the corresponding text descriptions in the paper.  How many calibration data are used in Figure 3? \n\n3. Could authors explain why larger sequence length leads to slower inference in Figure 3d? What is the role of inconsistent sparsity in this figure? Can the authors elaborate more on why we need consistent sparsity across layers to obtain a speed up?\n\n4. I found the authors' claim in Section 4 is a bit controversial. At the beginning of Section 4, the authors mentioned that using a calibration dataset is not precise enough, leading to inaccurate sparsity. However, in the later part, the calibration dataset is used to perform Layerwise Orthogonal Rotation. Will this also lead to mismatching and performance drop? \n\n5. I am wondering if the inconsistent sparsity of magnitude pruning can be mitigated with a longer calibration dataset?\n\n6. The inference speed-up in Figure 4 is interesting, demonstrating the benefits of LaRoSA. \n\n7. Another question is how sparse activation brings real speedup? I assume different tokens/samples will activate different sets of neurons. How can we address this mismatch in GPU?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see the above weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6i2KP0OvoW", "reviewer": "Reviewer_Yw7t", "summary": "The paper presents LaRoSA, a training-free activation sparsification method aimed at improving inference efficiency in LLMs. The key idea is orthogonal rotation-based pruning using a Top-K selection mechanism that eliminates lower-magnitude activations. The experimental results across multiple LLMs (LLaMA2, LLaMA3, Mistral, Qwen2.5) demonstrate superior accuracy trade-off compared to existing approaches like CATS and TEAL​.\n\n## update after rebuttal\n\nThe authors have answered my concerns, and I still believe this paper merits acceptance and is a good paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer Yw7t for taking your valuable time to evaluate our work. The acknowledgment of our contributions has greatly inspired our entire team. Below is our point-by-point response to your feedback:\n\n\n>Appendix A, minor: I'd include a 1-line at the top to explain what this appendix section discusses.\n\nThank you for your kind suggestion. In the next revision of the paper, we will add an introductory sentence to Appendix A: *'This section provides a detailed explanation of how LaRoSA utilizes orthogonal transformations and top-k sparsification, and why it achieves consistently lower empirical errors compared to TEAL.*\n\n>The authors correctly criticize other works for distributional shifts between their calibration set and test time. But the method constructs the rotation matrix based on data. I don't think it's necessarily as bad, especially since the authors have addressed it in Table 6 (minor typo: the paper refers to Table 7 instead) to show the impact of the calibration data is minimal.\n\nWhile developing our method, we shared the same concerns as the reviewer, which led us to include an ablation study focused on the choice of calibration datasets. Our paper's experimental results in Table 6 show that the method's performance is not sensitive to the specific calibration data used for creating the layerwise rotation matrix $Q_l$. To further demonstrate LaRoSA's effectiveness and robustness, we promptly carried out additional experiments on two distilled reasoning models, DeepSeek-R1-Distill-Llama3-8B[1] and DeepSeek-R1-Distill-Qwen2.5-7B[1], using the LaRoSA framework. We employed the Alpaca[2] dataset, which is not related to complex reasoning tasks, to obtain rotation matrix $Q_l$. We evaluate LaRoSA on harder tasks with these advanced models using Hugging Face Open-R1 repo.\n\n|Model|MATH 500|GPQA Diamond|AIME 2024|\n|-|-|-|-|\n|**DeepSeek-R1-Distill-Llama3-8B**||||\n|DeepSeek-R1 Report|89.1|49.0|50.4|\n|Reproduce Result|87.6|45.9|40.0|\n|+LaRoSA 25%|85.0|44.9|40.0|\n|+LaRoSA 40%|80.6|43.1|36.7|\n|+LaRoSA 50%|78.8|41.2|33.3|\n|**DeepSeek-R1-Distill-Qwen2.5-7B**||||\n|DeepSeek-R1 Report|92.8|49.1|55.5|\n|Reproduce Result|93.4|43.9|50.0|\n|+LaRoSA 25%|91.0|43.5|46.7|\n|+LaRoSA 40%|88.5|41.0|43.3|\n|+LaRoSA 50%|85.1|40.2|43.3|\n\nAs shown in this table, our experiments demonstrated that the LaRoSA method maintains its effectiveness even when using data that is not closely related to complex reasoning tasks for generating the rotation matrix. We hope this set of experiments further alleviates the reviewers' concerns about the impact of calibration data on the performance of the LaRoSA algorithm.\n\nOnce again, we express our gratitude to the reviewer Yw7t for acknowledging our paper's merits. We will continue to refine our work in the forthcoming stages and plan to open-source our code and models.\n\n[1] Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.  \n[2] Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.", "claims_and_evidence": "All claims are well-supported as far as I can tell.", "methods_and_evaluation_criteria": "The methods and evaluation criteria in the paper make sense.\nWhile I'm skeptic of PPL results alone, the authors included various benchmark results (ARC, MMLU, HellaSwag, BoolQ, etc.).\nAll results are compared to baseline methods.", "theoretical_claims": "The paper derives error bounds for Top-K of rotated activations, and provide mathematical derivations in Appendix A to support it. As far as I can tell, it seems correct.", "experimental_designs_or_analyses": "The experimental designs seem solid and robust.\nThe authors test on four major LLM architectures (LLaMA2, LLaMA3, Mistral, Qwen2.5) of various sizes (7B-70B). They compare their method with the TEAL and CATS baseline methods under the same sparsity constraints.\nMost importantly, they measure performance on a large set of downstream tasks.", "relation_to_broader_scientific_literature": "The work builds upon pruning techniques , and incorporates ideas inspired from rotation-based quantization.", "essential_references_not_discussed": "There are many works in the field of pruning not discussed or compared to, but I acknowledge the authors cannot be expected to discuss every single one.", "other_strengths_and_weaknesses": "Strengths:\n\n-No retraining required.\n\n-Outperforms baseline methods on a large set of benchmarks.\n\n-Provides kernels to utilize the method, something not every paper may do.\n\n-The authors include various ablations to show their method is compatible with quantization and the effects of various choices.\n\n-I appreciate the theoretical analysis in appendix A. \n\nWeaknesses:\n\n-The authors correctly criticize other works for distributional shifts between their calibration set and test time. But the method constructs the rotation matrix based on data. I don't think it's necessarily as bad, especially since the authors have addressed it in Table 6 (minor typo: the paper refers to Table 7 instead) to show the impact of the calibration data is minimal.", "other_comments_or_suggestions": "Appendix A, minor: I'd include a 1-line at the top to explain what this appendix section discusses.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QeOq0kxwDi", "reviewer": "Reviewer_jWLg", "summary": "The paper introduces LaRoSA, a method that applies layer-wise rotation to activation sparsity, which improves compression performance with minimal additional inference cost.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer jWLg for the thorough review! Below is our point-by-point response to your feedback:\n\n> The experimental results for TEAL are slightly worse than those reported in the original paper ...\n\nYes, your observation is correct. We will modify and extend implementation details in future revisions.\n\nAs noted in Section 5.4.3 of the TEAL paper, TEAL applies activation sparsification to only 99% of tokens during the prefill stage. The authors attribute this to attention sinks[3], which cause *'more severe degradation when sparsifying the initial tokens'*. Therefore, they skipped the first 1% of tokens to maintain model performance. In Section 4.1 of the LaRoSA paper, we argue that this is due to **a mismatch between the actual and calibrated thresholds for each token.** As shown in Figure 3b, this mismatch is especially pronounced at the start of sequences, leading to TEAL's poor performance on those initial tokens.\n\nIn LaRoSA and CATS, **all tokens are sparsified without special treatment**. For a fair comparison, we applied full activation sparsification for TEAL, resulting in slightly worse results than TEAL's report. This also reflects the lack of robustness in using empirical thresholds.\n\n> Could LaRoSA be combined with semi-structured pruning methods such as Wanda and RIA?\n\nThank you for the kind suggestion. It is indeed possible. Combining LaRoSA with other model compression methods is a promising optimization direction. We conduct experiments using RIA[2] repository, and randomly choose 128 sequences from C4 dataset to compute importance matrices for Wanda and RIA. We applied LaRoSA after semi-structured pruning. The results are summarized as follows:\n\n|Sparsity|Method||LLaMA3|-8B||Qwen2.5|-7B|\n|-|-|-|-|-|-|-|-|\n|||PPL(↓)|Acc(↑)|Speed-Up(↑)|PPL(↓)|Acc(↑)|Speed-Up(↑)|\n|Dense|-|6.1|70.0|1.00×|6.8|70.3|1.00×|\n|LaRoSA 25%|-|6.2|69.5|1.14×|6.9|70.1|1.15×|\n|LaRoSA 40%|-|6.6|68.8|1.30×|7.1|69.6|1.32×|\n|2:4|Wanda|24.2|62.3|1.24×|25.6|59.3|1.25×|\n|+LaRoSA 25%|Wanda|24.5|61.8|1.33×|26.0|59.0|1.35×|\n|+LaRoSA 40%|Wanda|25.0|60.1|1.40×|26.5|57.4|1.42×|\n|2:4|RIA|23.0|63.1|1.23×|24.1|60.1|1.25×|\n|+LaRoSA 25%|RIA|23.2|62.9|1.34×|24.4|59.8|1.34×|\n|+LaRoSA 40%|RIA|23.4|61.6|1.40×|24.9|57.9|1.41×|\n\nAlthough Wanda and RIA achieve speed-up by using the cuTLASS library, these methods experience notable performance drop, which also aligns with recent findings reported in the RIA's repo for latest models.\n\nIt's important to note that while Wanda and RIA focus on weight pruning, LaRoSA is an activation sparsification method. Due to different pruning targets, combining these methods does not lead to severe error accumulation. This highlight LaRoSA's compatibility with different compression methods. Moreover, at similar sparsity levels, LaRoSA's 40% achieved higher speed-up with less performance loss compared to RIA 2:4 (50%). This indicates: **1) The customized kernel in LaRoSA effectively addresses many inference bottlenecks in LLMs. 2) Activation sparsification could offer performance benefits over weight pruning, suggesting a promising direction for optimizing LLM inference.**\n\n> I am somewhat confused about Figure 3. Could you provide more analysis?\n\nWe apologize for not providing the best interpretation of Figure 3 in the current submission. We have detailed the experimental setup, motivations, and insights in our response to reviewer V3DZ. We kindly ask you to refer to our response to reviewer V3DZ for more information.\n\n> The GPTQ baseline in Table 7 is not state-of-the-art.\n\nWe appreciate the reviewer's constructive suggestions. We have added experiments using AWQ[4] and OmniQuant[5] for weight-only quantization. We will also include these results in future versions of the paper.\n\n|Precision & Sparsity|Methods|LLaMA3-8B PPL(↓)|LLaMA3-8B Acc(↑)|LLaMA2-7B PPL(↓)|LLaMA2-7B Acc(↑)|\n|--|--|--|--|--|--|\n|FP16|-|6.13|70.05|5.47|66.69|\n|W4A16|AWQ|6.48|68.95|5.60|65.12|\n|W4A16|OmniQuant|6.50|68.60|5.58|64.94|\n|+LaRoSA 25%|AWQ|6.56|68.44|5.70|64.90|\n|+LaRoSA 25%|OmniQuant|6.60|68.32|5.68|64.86|\n|+LaRoSA 50%|AWQ|7.58|66.10|6.01|63.80|\n|+LaRoSA 50%|OmniQuant|7.62|66.14|5.96|63.52|\n\nIt is evident that LaRoSA is highly compatible with the latest SOTA quantization methods. This further indicates that the **empirical errors introduced by activation sparsification and weight quantization are orthogonal**, suggesting that future works should consider both quantization and activation sparsification to achieve more efficient and reliable model inference.\n\n[1] A Simple and Effective Pruning Approach for Large Language Models, ICLR 2024.  \n[2] Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models, ICLR 2024.  \n[3] Efficient Streaming Language Models with Attention Sinks, ICLR 2024.  \n[4] AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration, MLSys 2024.  \n[5] OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models, ICLR 2024.", "claims_and_evidence": "The claims are well-supported with clear evidence.", "methods_and_evaluation_criteria": "The evaluations include different series of LLMs, demonstrating the effectiveness of LaRoSA across various architectures.", "theoretical_claims": "I have reviewed the proof section in the Appendix.", "experimental_designs_or_analyses": "The experimental results for TEAL are slightly worse than those reported in the original paper. Given the reproducibility concerns, I suggest that the authors provide a discussion and analysis regarding the potential reasons for this discrepancy.", "relation_to_broader_scientific_literature": "This paper follows the main pipeline of CATS and TEAL, aiming to improve activation sparsity.", "essential_references_not_discussed": "It might be beneficial to include a discussion on pruning methods in related works, which could further contextualize the contributions of LaRoSA.", "other_strengths_and_weaknesses": "The paper is well-structured and clearly written, making it easy to follow and understand.", "other_comments_or_suggestions": "Please see the questions below for additional feedback.", "questions_for_authors": "1. Could LaRoSA be combined with semi-structured pruning methods, such as Wanda [1] and RIA [2]? If possible, could you provide experimental results to support this?\n2. I am somewhat confused about Figure 3. Could you provide more analysis of the experimental details and insights from the data presented?\n3. The GPTQ baseline in Table 7 is not state-of-the-art. It would be valuable to explore and compare with more advanced methods.\n\n\n[1] A Simple and Effective Pruning Approach for Large Language Models, ICLR 2024.\n\n[2] Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models, ICLR 2024.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers recommended accepting the paper.

I myself enjoyed the crisp presentation and clarity of the ideas. The approach is sound and the experimental results are comprehensive. A comprehensive rebuttal of concerns raised by Reviewer V3DZ was given; I encourage the author to update the paper with the results of this discussion, because I believe it will strengthen the paper. The same goes for the new experiments included in the response to Reviewer jWLg.
Decision: Accept (poster)