Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models
Authors: Yingtao Zhang
Haoli Bai
Haokun Lin
Jialin Zhao
Lu Hou
Carlo Vittorio Cannistraci
Venue: ICLR 2024 poster (2024)
Keywords: Post-Training Pruning
Combinatorial Optimization
Large Language Models
Inference Acceleration
Primary area: general machine learning (i.e., none of the above)
TL;DR: By integrating Relative Importance and Activations and Channel Permutation, we present a plug-and-play solution for post-training pruning of LLMs, which accelerates the inference speed of LLMs without performance degradation.

Abstract: With the rapid growth of large language models (LLMs), there is increasing demand for memory and computation in LLMs. Recent efforts on post-training pruning of LLMs aim to reduce the model size and computation requirements, yet the performance is still sub-optimal. 
In this paper, we present a plug-and-play solution for post-training pruning of LLMs.
The proposed solution has two innovative components: 1) **Relative Importance and Activations (RIA)**, a new pruning metric that jointly considers the weight and activations efficiently on LLMs, and 2) **Channel Permutation**, a new approach to maximally preserves important weights under N:M sparsity.
The two proposed components can be readily combined to further enhance the N:M semi-structured pruning of LLMs.
Our empirical experiments show that RIA alone can already surpass all existing post-training pruning methods on prevalent LLMs, e.g., LLaMA ranging from 7B to 65B. Furthermore, N:M semi-structured pruning with channel permutation can even outperform the original LLaMA2-70B on zero-shot tasks, together with practical speed-up on specific hardware.
Our code is available at: https://github.com/biomedical-cybernetics/Relative-importance-and-activation-pruning

Published as a conference paper at ICLR 2024
PLUG-AND-PLAY: AN EFFICIENT POST-TRAINING PRUNING METHOD FOR LARGE LANGUAGE MODELS
Yingtao Zhang! Haoli Bai*, Haokun Jialin Zhao's 1.2, Lu Hou",
Carlo Vittorio Cannistraci 1,2.3
'Center for Complex Network Intelligence, Tsinghua Laboratory Brain and Intelligence
Department of Computer Science, Tsinghua University
3 'Department of Biomedical Engineering, Tsinghua University
"Huawei Noah's ' Ark Lab, Institute of Automation, Chinese Academy of Sciences Corresponding to (zhangyingtao122 http:aaaooeeaaaaaaaaaaaaaaaaaaaa
kalokagathos agon gmaill com
ABSTRACT
With the rapid growth of large language models (LLMs), there is increasing de- mand for memory and computation in LLMs. Recent efforts on post-training prun- ing of L.LMs aim to reduce the model size and computation requirements, yet the performance is still sub-optimal. In this paper, we present plug-and-play solu- tion for post-training pruning of L.LMs. The proposed solution has two innovative components: 1) Relative Importance and Activations (RIA), a new pruning met- ric that jointly considers the weight and activations efficiently on LLMs; and 2) Channel Permutation, a new approach maximally preserve important weights under N:M sparsity. The two proposed components can be readily combined to further enhance the N:M semi-structured pruning of LLMs. Our empirical exper- iments show that RIA alone can already surpass all existing post-training prun- ing methods on prevalent LLMs, e.g., LLaMA ranging from 7B to 65B. Further- more, N:M semi-structured pruning with channel permutation can even outper- form the original LLaMA2-70B on zero-shot tasks, together with practical speed- up specific hardware. Our code is available at: https://github.com citoom//bomeicca..... ccybennticss/Relatve importance-rr-e----eo and putivationnpprniine
INTRODUCTION
Recent research on large language models (LLMs) has attracted significant interest. These LLMs, characterized by their vast number of parameters, have exhibited remarkable proficiency across wide range of tasks. However, deploying such models poses challenges due to their substantial size, computational demands, and execution time. To address this, several methods for network
compression have been explored, such as model quantization (Bai et al. 2020 Frantar et al. 2022
Xiao et al.) 2023 Lin et al.] (223) and network pruning (LeCun et al. 1989 Hassibi et al.
1993
Mocanu et 2018 8
Sun et 2023 Frantar & Alistarh 2023).
Unlike quantization techniques, which adjust the precision of weights activations compress the network, network sparsity primarily targets the elimination of redundant or useless weights within models. Despite its potential, the exploration of sparsity in LLMs remains limited. Generally, neural networks can achieve sparsity through three primary methods: 1) sparse training (Lee et al.
2018
Mocanu et al. 2018 [Eve et al.) 2020, Sanh al.) 2020 Yuan et al.) 2021 Hoang et al, Zhang et al., (023): 2) pruning aware training (Han et al., 2015] Liu et al.)
2021); and 3) post-training pruning (PTP) (Hassibi et al. 1993 Li & Louri [002 Frantar & Alistarh 2003 Sun
2022
et al. 2023). However, both sparse training and during-training pruning require multiple rounds of iterative training, which computationally costly and time-consuming. especially for LLMs. Therefore, PTP on well-pre-trained model represents more reasonable approach for LLMs.
The primary challenge associated with post-training pruning lies in the substantial performance degradation compared to dense models. Current methods, like SparseGPT (Frantar & Alistarh
"This work is partially done during the internship Huawei Noah's Ark Lab.

---

Published as a conference paper at ICLR 2024
and Wanda (Sun et al. to achieve practical speed-up. it is favored to conduct N:M semi-structured pruning, which can be supported by specific hardware with sparse matrix multiplications (Mishra et al. 202Ip. These prior methods (Sun et al., 2023 Frantar& & Alistarh, 2023) under the N:M sparsity sufter from significant performance drops, thereby restricting their applications in practice. Recently, (Pool & Yu! 2 has introduced an input channel permutation method using the greedy search, which can boost the performance under the N:M sparsity. However, the greedy search is time-consuming on LLMs and
2023
2023), exhibit promising results in unstructured pruning. Nonetheless,
thus is not feasible.
In this paper, we introduce plug-and-pln post-training pruming method for LLMs. Specifically, the method comprises two key components. First, we introduce Relative Importance and Activa- tion (RIA), new pruning metric for LLM pruning. We show that prior pruning metrics (Frantar & Alistarh 2023 Sun et al. 2023) tend to prune away entire channels of network weights, which undesirable for both unstructured and N:M semi-structured pruning. Instead, RIA jointly consid- ers the input and output channels of weight and the activation information, effectively mitigating such issues. Second, we also consider a better way to convert LLM weight matrices to adhere to N:M sparsity patterns. Unlike existing methods that directly convert the weight matrix to N:M spar- sity, we propose channel permutation to properly permute the weight channels so as to maximally preserve the important weights under N:M structures. Finally, the proposed RIA and channel permu- tation can be readily combined, leading to an efficient and plug-and-play approach for the real-world acceleration of sparse LLMs inference. We name the proposed method "plug-and-play" since 1) does not need additional fine-tuning or retraining; 2) it can be adopted by any models that have linear layers; 3) it has negligible performance drop when applying channel permutation in the zero-shot experiments.
Extensive evaluation of vron et al. (0233)), and OPT (Zhang et al. 2022a) demonstrates that RIA outperforms SOTA one- shot PTP methods both unstructured sparsity and N:M sparsity scenario. Additionally, channel permutation can be efficiently scaled to I. LLMS with over 70B parameters within 2 hours, yet recover a lot of the performance drop caused by N:M constraint and surpass the performance of dense mod- els in 3 out of 5 evaluated zero-shot datasets. By employing RIA and channel permutation, LLMs can undergo a smooth transition into N:M constraints. This ensures hardware compatibility while maintaining the intact performance of the pruned model.
LLMs (e.g.. LLaMA Touvron et al., 2023a LLaMA-2 (Tou-
2 RELATED WORK
Post-training pruning (PTP) methods trace their roots to OBD (LeCun ct al. 1989). which employs the hessian matrix for weight pruning. OBS (Hassibi et al. 1993) fur- ther refines the approach by adjusting remaining weights to minimize loss changes during pruning.
Post-Tree Training Pruning.
The advent of Large Language Models (LLMs) has sparked interest in leveraging the Hessian Matrix for pruning, exemplified by works like AdaPrune (Li & Louri 202I) and Iterative Adaprune (Fran- lar & Alistarh, (2022) targeting BERT (Devlin et al.) 2018)) However, weight reconstruction via Hessian Matrix inverse incurs substantial computational complexity at O SparseGPT (Fran- tar & Alistarh 2023) reduces this complexity to O(N3), while Wanda (Sun et al. 2023) leverages input activations for efficient one-shot PTP with reduced pruning time and comparable performance to SParseGPT. Our proposed method, Relative Importance and Activations (RIA) inherits the time- saving advantages of Wanda while simultaneously enhancing the performance of the pruned L.LMs. Note that the method proposed in this article is designed for model compression without retrain- ing and finetuning, which differentiates it from Dejavu (Liu et al. 2023),, which requires additional training steps. Such post-training methods also preferred for model quantization as well (Bai et al. 2022
Xiao et al. 2023 Lin et al. 2023: Liu et al. 2024).
N:M Sparsity. Recently, NVIDIA has introduced the N:M constraint sparsity (Mshhra al. 2021) as a method to compress neural network models while preserving hardware efficiency. This N:M sparsity constraint stipulates that at least N out of every contiguous M element must be set to zero, thereby accelerating matrix cetriii-uulply-yccmutt results in 50% sparsity, effectively doubling the model's 's inference speed when utilizing the NVIDIA Ampere GPU architecture. However, directly applying SOTA unstructured pruning methods to meet the N:M sparsity often leads to a noticeable decline in performance, as demonstrated in Table 6 Some approaches (Hubara et al. 2021 Zhou et al. 2021 Zhang al. 2022b) suggest fine-tuning
instructions. For instance, 2:4 constraint ratio
2

---

Published as a conference paper at ICLR 2024
Dense Connections
Original Score Matrix
Relative Importance
Relative Importance
10
2.5
-0.3
2.2
0.94
0.28
0.03
0.26
2.5
-1,8
-0.6
2.7
.2.5
-1.8
-0.6
2.7
1.00
1.08
1,05
0.91
0.6
-0.4
0.7
0.6
-0.4
0
0.7
Will be pruned 50% sparsity
Will be retained if 50% sparsity
Figure I: An example of Relative Importance. The connection position 44 will be removed based on global network magnitude. However, it is retained when evaluated for its significance within relative connections.
pruned models to recover capacity, but this is prohibitively expensive for Large Language Models (LLMs).
Matrix Permutation. N:M sparsity primarily targets the application of sparsity in the input chan- nel dimension. (Pool & Yu 2021) presents permutation method to identify the optimal permutation of input channels through an exhaustive greedy search and an escape phase to navigate local minima. However, this greedy approach becomes impractically time-consuming when applied to LLMS due to their extensive linear layers. In this study, we leverage the specific characteristics LLMs and propose a Channel Permutation strategy to reduce the computational overhead efficiently.
3
LLM PRUNING WITH RELATIVE IMPORTANCE AND ACTIVATIONS
3.1 POST-TRAINING PRUNING: PRELIMINARIES
Post-training pruning (PTP) typically starts from the pre-trained network, removes redundant pa- rameters, and does not need end-to-en fine-tur tuning. Unlike training baaningg-bseddd pruning methods (Mo- canu et al., 2018 Sanh et al. 2020 Zhang et al.] 2023 Tao et al. 2023 Lin et al. (0244), PTP is fast, resource-saving, and therefore preferred for compressing LLMS (Frantar & Alistarh [2023 Sun
et al. 2023). PTP currently prevalent in unstructured pruming and N:M semi-structured pruming. which is also the main focus of this paper. It is less applied in structured pruning (Ma et al. 2023
due a larger performance drop.
A common approach to achieve PTP is layer-wise pruning, e.g., minimizing the discrepancy square eтror between the dense and pruned model input of the [-th linear layer as X₁, and weight W, E Rrxc, where r and e represent the number of output and input channels respectively. Our primary goal find the pruning mask M, E (0, 13 that minimizes the & distance error between the original and pruned layer.
recursively. Specifically, we denote the
Therefore, the objective can be formally expressed as follows:
arg min
W,X; (M, W:) X 12 S.L. |M < k
where & represents the number of remaining weights determined by the pruning ratio, and || llo is the norm (e.g., the number of non-zero elements). To solve M, in Equation [1] there are various pruning metrics, e.g., magnitude- basedd pruning, that mask the weight below a certain threshold. Nonetheless, we show that these prior pruning metrics have intrinsic drawbacks, as discussed the following section. Optionally, the weight WN in Equation D can also be reconstructed via closed- form updates (Frantar Alistarh 2023); see Appendix D for more discussions.
3.2 RELATIVE IMPORTANCE: A NEW PRUNING METRIC
We present the relative importance (dubbed as RI), a new metric for LLM pruning. We find that prevalent PTP methods tend to suffer from channel corruption, i,c., the entire input output channel is pruned away. This is akin to node removal network science terms, and significantly diminishes the performance of Large Language Models (LLMs), in manner similar structured pruning. To
3

---

Published as a conference paper at ICLR 2024
Attn_Q
Attn_Ou _ut
MLP Up Gate
MLP Down
1.00
0.75
0.50
0.25
Sp
Layers
Figure 2: Spearman's Rank correlation of LLaMA2-13B activations from Wikitex12, C4, and PTB, each with 128 samples. Note that the Q K, and V layers all share the same input activations. Similarly, the Up and Gate layers also receive identical input activations.
illustrate this, Figure shows linear layer with W E RA4 Magnitude-basel pruning with 50% unstructured sparsity will corrupt W4.,
d+. i.e, the 4-th output channel. In practice, find similar issues also exist in other prevalent pruning metrics, e.g., Wanda (Sun et al. 2023) prunes around 500 channels out of 5120 channels in some layers, with more than 10% channels corrupted. Given that well-trained LLMs contain unique information in the imput and output channels, it is critical to avoid
channel corruption in post-tras post-training pruning. We explain the phenomenon of channel corruption in detail in Appendix E
To mitigate such issues, the proposed relative importance (RI) aims to re-evaluate the importance of each weight element W ij based on all connections that originate from the input neuron i or lead to the output neuron j. Specifically, the relative importance for Waj can calculated as:
|Wll RI; Σ W W.
W
+
where Σ W.y/ sums over the absolute values of the weights input channel j. and similarly Σ W for the sum of the weights in output channels i. The resulting score RI, offers insight into the relative importance of weight Wij in the context of its connections to neurons and j.
3.3 INCORPORATING ACTIVATIONS INTO RELATIVE IMPORTANCE
The proposed relative importance can be further combined with activations to assess better the weight significance, dubbed as relative importance and activation (RIA). Recent findings (Xiao
2023 show that the occurrence of activation outliers has become a well-known issue in quan- et al.
tizing LLMs. Our visualizations on LLaMA and OPT also confirm these outliers; see Figure 7 and Figure for details.
Moreover, we find that activation outliers persist regardless of the dataset or parts of the model. To this, calculate the Spearman's 's Rank Correlation Coefficient of activations between different datasets. From Figure 2 it can be observed that pairwise correlations of activations exhibit positive values and similar trends across different layers and Transformer modules. We offer evidence that the Spearman Rank correlation between pairs of the activations of different datasets is positive. This positivity is a necessary condition to incorporate the activation into RIA formula. And indeed it is always satisfying.
Built upon Equation as follows:
2 for each element W ijj RIA further combines fnrm of activations X
W Σ W.
W W.:!
RIA, = RLy x (1(,)12)
(3)
where a power factor introduced to control the strength of activations. Our empirical results in Figure show that 0.5 works generally well for different LLMs.
4 TURNING INTO N:M SPARSITY
This section studies how to tur LLMs into N:M sparsity with the pruning metric. N:M sparsity is usually favored by post-training pruning given its practical speed-up on specific hardware. Unlike
4

---

Published as a conference paper at ICLR 2024
Score Matrix
Total sum of score: 83.7
1.3
1.9
0.2
1.2
4.2
3.5
5.2
0.6
1.3
1.9
4.2
5.2
0.4
1.1
1.5
6.9
0.8
6.7
5.4
8.8
Direct Sparsity
1.5
6.9
6.7
8.8
0.2
1.3
8.0
0.8
4.2
1.3
9.7
1.3
8.0
7.6
9.7
1.3
4.2
8.9
1.3
8.9
8.9
Channel Score
3.4
4.9
11.0
9.4
18.0
15.7
27.1
28.0
Sorted Index
7
5
3
Heuristic Channel
Allocation
1
5
4
8
4
1
6
3.5
8.8
1.5
6.7
Linear Sum Assignment
6.7
8.8
6.9
9.7
8.0
7.6
1.3
7.6
8.0
9.7
4.2
8.9
8.8
8.9
4.2
8.9
4.2
8.9
8.8
Total sum of score: 96.1
Total sum of score: 102.3
Figure 3: Illustration of Channel Permutation. Given a score matrix assigned by various criteria, directly processing it with 2:4 sparse results in total sum of the retained score being 83.7. However, by using channel permutation, we could get a final total sum of score 102.3.
existing solutions that directly convert to the N:M format, we propose channel permutation, new approach that better leverages the pruning metrics by efficiently permuting the weight channels. Channel permutation can work seamlessly with RIA metric in Section 3 and other prevalent
methods such as (Frantar & Alistarh| 2023: Sun et al. 2023).
4.1 N:M DEMI-STRUCTURED PRUNING: FORMULATION
We begin by revisiting unstructured PTP in Equation Without loss of generality, we denote the weight importance score S X' which can be either RIA or other pruning metrics. It is an NP-hard problem (Frantar & Alistarh 2023) to find the optimal M {0, 13 in Equation | A
common surrogate is maximize the sum of retained weight importance scores as follows:
arg max
Σ s.t.
M
S k.
(4)
Similarly, for N:M sparsity, every N out of M contiguous elements zero along each output channel, and the objective is re-formulated as:
arg max M
Σ
s.t. M (mm S m n. Σ M C km)+(()))))))
(5)
A simple solution to Equation 5 in existing works (Frantar & Alistarh 2023 Sun et al. 2023
directly setting the mask M, kmm:s()))))))) of top elements in S. 1jm to 1, and other-
wise. Nonetheless, this usually leads to sub-optimal solutions. As illustrated in Figure some
input channels with similar scores, either large or small, might get stuck in the same weight block W kmm:(k+1)mm.- Consequently, some large weights might be pruned by mistake, while small ones might be preserved.
4.2 CHANNEL PERMUTATION FOR IMPROVED N:M SPARSITY
To address the aforementioned challenge, we present new channel permutation (CP) approach that yields better N:M structures. Note that permuting the input channels of weight can lead to different importance scores S. We thus introduce an additional column permutation matrix P for S. such that the sum of retained weight importance can be further maximized:

---

Published as a conference paper at ICLR 2024
arg mOXX M.P
M ② kmm s.t. | (m+ lo S m n.
(6)
k=0
For case of presentation in what follows, we denote the block of the weight matrix as W. (k+ 1)m R=xm where E fI, K} and K is the number of blocks. N:M pruning thus occurs row-wise within each block, e.g., only n values are preserved out of m elements for W.,eme(k++)mm Based on the notation, the proposed channel permutation mainly includes the fol- lowing two steps.
W
Step 1: Heuristic Channel Allocation. We first calculate the sum of weight importance for each input channel. These channels are then sorted and allocated into K blocks. To maximally retain the important channels in each block with N:M sparsity, we use a heuristic allocation strategy. Given K blocks, we alternately allocate every top- K input channel into each block, and this process is repeated for m times until all channels are allocated. An example is illustrated in Figure Given 8 imput channels and output channels, there are 2 blocks under 2:4 sparsity. The top-1 and top- 2 channels are allocated to block and block 2, respectively, and similarly to the remaining input channels. Given such a hcuristic channel allocation strategy, it can be found that there is significant enhancement the sum of retained weight importance scores compared to direct N:M pruning.
Step 2: Linear Sum Assignment. Next, show the heuristic allocation strategy be further refined. The refining process can be formulated as a linear sum assignment (LSA) problem, which can be efficiently solved by the Hungarian algorithm (Kuhn 1955). To see this, we can take out one allocated channel from each block; thus, there are K channels to be reassigned to K blocks. It thus a traditional linear sum assignment problem to find a better one-by-one matching between the K channels and K blocks, such that the weight importance sum in Equation o can be further improved. From Figure 3 LSA further improves the score sum by 6.2, with the top-1 and top-2, top-3 and top-4 channels swapped from the heuristic channel allocation.
Remarks. Note that the permutation of weight matrices does not affect the output of LLMs. For dense layers, the input activations need to be simultaneously permuted, which can be achieved by permuting the output channels of the previous layer. The exception lies in the residual connection, which can be done with an efficient permutation operator. More details on implementing channel permutation and the Hungarian algorithm are listed in
5 EXPERIMENTS
5.1 SETUP
We evaluate the proposed approach on three popular LLMs: LLaMA 7B-65B (Touvron et al. 2023a), ILaMA2 7B-70B (Touvron et al. (023b)) and OPT 1.3B (Zhang et al. 2022a). We use the public checkpoints of the involved models in the HuggingFace Transformers library We utilize NVIDIA A100 GPUs, each equipped with 80GB memory. For each model under consideration, we apply uniform pruning to all linear layers, with the exception embeddings and the head. Specifi- cally, each self-attention module has four linear layers, while each MLP module contains three linear layers for LLaMA model families and two for OPT. All the evaluations are conducted with the same code to make sure the comparison is fair. The detailed settings of tasks, metrics, baseline methods, and calibration data can be found in Appendix B
5.2 UNSTRUCTURED PRUNING
Main Results. As highlighted in Table RIA consistently outperforms Wanda and SparseGPT across all scenarios. Notably, our method achieves 50% improvement in preventing performance drop of the dense model in comparison to SparseGPT (16% in LLaMA and LLaMA2 model family), and 17% improvement in preventing performance drop in comparison to Wanda (13% in LLaMA and LLaMA2 model family). It is essential to note that as the model size increases, the performance gap between models pruned by RIA and the original dense models diminishes significantly.
https//tuggingffcccc//eeett Hama, hwwpa//w//./////gggggghcooccoooo

---

Published as a conference paper at ICLR 2024
Table 1: Perplexity results on Wikitex12. We produce the one-shot Post-Training pruning methods with 50% unstructured sparsity on LLaMA, LLaMA2, and OPT models.
Method
LLaMA
LLaMA
LLaMA
LLaMA
LLaMA2
LLaMA2
LLaMA2
OPT
7b
13b
30b
65b
7b
13b
70b
1.3b
Dense
5.68
5.09
4.77
3.56
5.47
4.88
3.32
14.62
Magnitude
17.28
20.22
7.54
5.90
16.02
6.83
5.36
1712
Wanda
7.26
6.15
5.24
4.57
6.92
5.99
4.22
18.41
SparseGPT
7.24
6.20
5.32
4.57
6.99
6.10
4.25
27.00
RIA (Ours)
7.12
6.08
5.08
4.38
6.81
5.83
4.11
18.08
Ablation Studies. To thoroughly evaluate the influence of each element within our RIA equa- tion, we undertook an ablation test, with the outcomes presented Table 2 We disassem- bled our formula into several distinct compo- nents for closer scrutiny: |X||2: &g-norm of activations; |W]: weight magnitude; [W] weight magnitude normalized by the input
Table Ablation Studies of RIA on LLaMA-13B and LLaMA-30B models.
LLaMA- 13B
LLaMA-300
X
9056
nan
W
20.22
7.55
kn.
11.97
6.73
W laut
7.80
5,55
channels; [W] out weight magnitude normal-
RI
6.57
5.27
RIA (a 1.0) RIA (a 0.5)
6.14
5.13
ized by the output channels; RI: stands for rel- ative importance which the combination of W and W Wlouti and RIA (a 1.0) and
6.08
5.08
RIA (a 0.5) combines relative importance with input activations with different values of a.
The selection of a is shown in Figure
As illustrated in Table 2] normalizing the weight magnitude through either input out- put channels offers a substantial performance boost over merely considering weight magni- tude. Interestingly, utilizing Relative Impor- tance alone match or, in instances like LLaMA-30B, even outperform SparseGPT. A distinguishing feature of Relative Importance is its reliance solely on weight information, elimi- nating the need for calibration data. In contrast, both Wanda and SparseGPT necessitate calibra- tion data to derive input activations Hessian matrices. The table also showcases enhance- ments brought about by the other equation components RIA.
LaaiAs50: Sparsity
Exaluaiion Sample
6.5
Sparsity
ramppls
Figure 4: Sensitivity Evaluation on Sparsity and number of calibration samples (nsamples).
Sparsity. In Figure a), we examine the effects of varying sparsity levels, ranging from 0.1 to 0.6, the performance of the LLaMA 65b model. The curves curves clearly demonstrate that Magnitude pruning particularly sensitive to increased sparsity levels, with a 60% sparsity level resulting in significant model degradation. In contrast, the SparseGPT, Wanda, and RIA models exhibit more robust performance across all tested sparsity levels, with RIA (green) consistently outperforming the others every level of sparsity.
Calibration data. SparseGPT, Wanda, and RIA all require calibration data for obtaining either input activations or the Hessian matrix. To assess the robustness of our algorithm with respect the calibration data, we conduct sensitivity test involving variations in the type calibration datasets and the number of calibration samples. The influence of calibration datasets is presented in Table 7 Our aim here is to assess the impact of the number of calibration samples.
As illustrated in Figure ()), example of the LLaMA30B model, SparseGPT appears to rely larger number of calibration samples, while both Wanda and RIA demonstrate robust performance across varying sample sizes. Notably, RIA consistently outperforms the other methods in all cases.

---

Published as a conference paper at ICLR 2024
Table LLaMA2-70B: Zero-Shot Performance of the model with unstructured 50% sparsity com- pared to the dense model. Bold values denote the best performance across all the post-training pruning methods. An asterisk (**") signifies performance surpassing that of the dense method.
Method
Hellaswag
BoolQ
ARC-C
MNLI
RTE
AVG
Dense
64.77
83.70
54.44
45.81
67.87
63.32
Magnitude
60.58
71.10
49.32
32.80
60.65
54.89
Wanda
62.70
83.27
52.50
43.19
70.84*
62.50
SparseGPT
62.36
84.26*
53.07
40.29
70.76*
62.15
RIA
63.22
84.77*
52.56
42.80
71.48
62.97
Table Unstructured sparsity and semi-structured sparsity results on Wikitext2. We highlight the best performance among all methods within the same sparsity pattern in bold.
Method
Unstructured
2:4
2:4+CP w/o LSA
2:4+CP
4:8
4:8+CP
50%
Magnitude
6.83
8.74
8.89
8.87
7.32
7.16
LLaMA2-13b (Dense 4.88)
Wanda
5.99
9.00
8.74
8.45
7.00
6.83
SparseGPT
6. 6.10
8.77
8.61
8.48
7.01
6.80
RIA (Ours)
5.83
8.41
8.03
7.77
6.74
6.53
Magnitude
5.36
6.76
6.77
6.71
5.89
5.91
LLaMA2-70b (Dense 3.32)
Wanda
4.23
5,48
5.29
5.23
4.77
4.63
SparseGPT
4.25
5.68
5.37
5.31
4.91
4.79
RIA (Ours)
4.11
5.36
5.18
5.11
4.68
4.54
Zero-shot performance. In Table 3 we present the zero-shot performance of unstructured 50% sparsity of the models pruned with magnitude, Wanda. SparseGPT, and RIA LLaMA2-70b model. In the last column, we report the average performance across these datasets. As shown in the table, RIA achieves the best performance on 3/5 datasets and also achieves the best average performance across the 5 datasets.
5.3
N:M SEMM-STRUCTURED PRUNING
While RIA aims to explore the upper bounds of performance achievable through one-shot PTP methods, combining it with the N:M constraint seeks to realize the actual inference speed by aligning with the present GPU hardware environment. In this subsection, we assess how Channel Permutation (CP) can enhance the performance of one-shot PTP when incorporated with the N:M constraint.
the performance of unstructured 50% sparsity, 2:4 constraint sparsity, and 4:8 constraint sparsity for Magnitude, Wanda, SparseGPT, and RIA. Addi- tionally, we provide the performance results when applying Channel Permutation (CP) each of these methods. Directly using step 1 of CP (CP w/o LSA) is also displayed in the table, serving as an ablation test in comparison to the complete one. We present the Perplexity of each pruned model on the Wikitext2 dataset, maintaining the same settings as in Section 5.1
Main Results. In this comparison, we
As presented in Table RIA consistently delivers superior performance across all semi- struttrree sparsity patterns when employing one-shot РTP. Importantly, when utilizing merely heuristic chan- reallocation, every method-wit with the exception of Magnitude-alresn performance improvement. With the incorporation of LSA, the performance is further improved. This highlights our motivation to group the input channels based on their sorted indices, ensuring that similar scaling channels don't 't end up in the same block.
exhibits a significant
Zero-shot Performance, In Table 6 we present the zero-shot performance the N:M constraint models pruned using RIA and Wanda. The table's last column also provides the average performance across these datasets. As the table indicates. while there's a performance decline on the Hellaswag and ARC-C datasets, the of RIA (2:4+CP) performs even surpasses the dense model BoolQ,
8

---

Published as a conference paper at ICLR 2024
Table LLaMA2-13BB Inference time of different sparsity patterns. Batch size of input sequences is 8 and the sequence length is 128.
Method
Q/K/V/Out Up/Gate Down
Overall
unstructured 50% 2:4 (cuTLASS)
0.98 x
0.98
0.97 x
0.98x
1.21
1.23x
1.23x
1.22x
2:4 (cuSPARSELT)
1.64x
1.65x
1.62x
1.63x
MNLI, and RTE datasets. This highlights the observation that large language models can be prone to overfitting, resulting in abundance of redundant, unnecessary, potentially harmful elements. Notably, with the incorporation of CP, there's a remarkable improvement in performance for both Wanda and RIA.
5.4 RUNNING TIME ANALYSIS
Pruning and permutation running time. We present the actual running time of each algorithm on the largest model involved this article, LLaMA2-70b. The relative complexity analysis can be found in Appendix G
Pruning time. We test each algorithm with 128 calibration data. For SparseGPT, the pruning time amounts 5756 seconds (approximately 1.5 hours). In contrast, Wanda and RIA demonstrate significantly reduced runtime, with times of approximately 611 seconds (approximately 10 minutes) and 627 seconds (approximately 10 minutes, including the execution of calibration samples).
Channel permutation time. For a comparative analysis of execution duration, we present the processing time of a single matrix constructed using our algorithm for the N:M spar- sity. This is compared against the greedy method and its variants, which employ escaping strategies to circumvent getting trapped in local minima, as discussed in (Pool & Yu 2021 The results for different dimensions are provided in Table
Inference acceleration. We assess the inference acceleration offered by sparsity in Large Lan- guage Models (LLMs). Like SparseGPT (Frantar & Alistarh (023). we present data for both the unstructured sparsity and 2:4 sparsity accelcration on GPU relative to the dense model across var- ious components. Theoretically. employing an N:M sparsity can yield up to 2x speedup when contrasted with dense models. We conduct tests on the Nvidia Tesla A A100, utilizing the cu' LLASS and cuSPARSELt library for Sparse Matrix-Mane Matrix Multiplication (SpMM) (Mishra al.] 202T) with N:M sparse matrices. These two libraries have been incorporated into the latest release of Py- Torch. For unstructured sparsity, we treat as if a dense matrix and assess the actual inference speed. Comprehensive acceleration metries for each module outlined in Table 5 showing acceleration of all the linear layers to be about 1.2 x for cuTLASS and 1.6x for CUSPARSELT.
CONCLUSION
In this article, we have introduced two novel methods, RIA and Channel Permutation, that together establish an effective plug-and-play pipeline for post-training pruning and inference acceleration of large language models. RIA incorporates relative importance and the feature input activations that create criterion for pruning the weights of LLMs. Through extensive experiments on promi- nent LLMs like LLaMA, LLaMA2, and OPT across varying model sizes, we have demonstrated that RIA consistently outperforms existing SOTA one-shot pruning techniques SparseGPT and Wanda, setting a new benchmark for pest-training pruning performance. Furthermore, Channel Permuta- tion successfully reduces the performance drop when adapting the model to the N:M constraint by reframing the input channel permutation problem as combinatorial optimization task and solving efficiently with the Hungarian algorithm. RIA and Channel Permutation form a seamless "plug- and-play" method, enabling effective one-shot post-training pruning for all current large language models. Furthermore, this method is hardware-frient frienddy, ensuring enhanced inference acceleration.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGE
This work is supported by the Zhou Yahui Chair professorship of Tsinghua University, the starting funding of the Tsinghua Laboratory of Brain and Intelligence, and the National High-level Talent Program of the Ministry of Science and Technology of China.
REFERENCES
Haoli Bai, Wei Zhang. Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit bert quantization. arXiv preprint arXiv:2012.15701, 2020.
Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Irwin King, and Michael R Lyu. Towards efficient post- training quantization of pre-trained language models. Advances in Neural Information Processing Systems, 35:1405-14111 2022.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 2018.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952. PMLR, 2020.
Elias Frantar and Dan Alistarh. Spdy: Accurate pruning with speedup guarantees. In International Conference Machine Learning, pp. 6726-6743. PMLR, 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. 2023.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot lan- guage model evaluation, September 2021. URL http::://ccc..rrg00002222/////// 5371628
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference neural networks, pp. 293-299. IEEE, 1993.
Duc NM Hoang, Shiwei Liu, Radu Mareulescu, and Zhangyang Wang. Revisiting pruning at ini- tialization through the lens of ramanujan graph. In The Eleventh International Conference on Learning Representations, 2022.
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accel- erated sparse neural training: A provable and efficient method to find m transposable masks. Advances in neural information processing systems, 34:21099-21111 2021.
Harold W Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83-99, 1955.
Yann LeCunn John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Namhoon Lee. Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity, arXiv preprint arXiv:1810.023444 2018.
Jiajun Li and Ahmed Louri. Adaprune: An accelerator-await pruning technique for sustainable cmn accelerators. IEEE Transactions Sustainable Computing, 7(1):47-60, 2021.
10

---

Published as a conference paper at ICLR 2024
Haokun Lin, Haoli Bai, Zhili Liu, Lu Hou, Muyi Sun, Linqi Song, Ying Wei, and Zhenan Sun. Mope-clip: Structured pruning for efficient lision-language models with module-wise pruning error metric. arXiv preprint arXiv:2403.07885 2024.
Ji Lin, Jiaming Tang, Haotian Tang. Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00977 2023.
Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. Intactkv: Improving large language model quantization by keeping pivot tokens intact. arXiv preprint arXiv:2403.0124 2024.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems, 34:9908-9922 2021.
Zichang Liu, Jue Wang. Tri Dao. Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient Ilms at inference time. In International Conference Machine Learning, pp. 22137-22176.. PMLR, 2023.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. arXiv preprint arXiv:2305. 2023.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330, 1993. URL
https: www. aclweb. wwg. wothhooogy/J3-2222
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint arXiv:2/04.08378, 2021.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 911::3383, 2018.
Jeff Pool and Chong Yu. Channel permutations for n: M sparsity. Advances in neural information processing systems, 34: 14311-131-3322 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine- tuning. Advances in Neural Information Processing Systems, 11:2037--20889 2020.
Mingjic Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effecttve pruning approach for large language models. arXiv preprint arXiv:2306. /695, 2023.
Chaofan Tao, Lu Hou, Haoli Bai, Jiansheng Wei, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. Structured pruning for efficient generative pre-trained language models. In Findings of the Asso- ciation for Computational Linguistics: ACL 2023, pp. 10880-10895, 2023.
Philippe Tillet, Hsiang-Tsune Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10-19, 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
11

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXtv:2307.09288 2023b.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099, PMLR, 2023.
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory-economic sparse training framework on the edge. Advances Neural Information Processing Systems, 34:20838-20850 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprimi urXiv:2205.01066 2022a.
Zhang, J Zhao, W Wu, Muscoloni, and CV Cannistraci. Epitopological sparse ultra-deep learn- ing: A brain-network topological theory carves communities in sparse and percolated hyperbolic anns. 2023.
Yuxin Zhang, Mingbao Lin, Zhihang Lin, Yiting Luo, Ke Li, Fei Chao, Yongjian Wu, and Ron- grong Ji. Learning best combination for efficient n: M sparsity. Advances in Neural Information Processing Systems, 35:941 19199,5,,2221 2022b.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hong- sheng Li. Learning m fine-grained structured sparse neural networks from scratch. arXiv preprint arXiv:2102.0400
2021.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression, arXiv preprint arXiv:1710.01878, 2017.
12

---

Published as a conference paper at ICLR 2024
Table 6: LLaMA2-70B: Zero-Shot Performance of N:M constraint model comparing to the dense model. Bold values denote the best performance across all N:M constraint models. An asterisk ("*"" signifies performance surpassing that of the dense method.
Method
ARC-C
MNLI
RTE
AVG
Dense
64.77
83.70
54.44
45.81
67.87
63.32
Wanda (2:4)
57.35
81.44
46.01
37.69
68.59*
58.22
Wanda (2:4+CP)
59.37
84.50*
48.55
43.09
66.43
60.39
Wanda (4:8+CP)
60.86
82.73
49.94
40.15
67.87
60.51
RIA (2:4)
57.13
82.78
46.76
37.39
69.31*
58.68
RIA (2:4+CP)
58,48
85.14*
49.15
49.08
68.95*
62.16
RIA (4:8+CP)
60.44
83.58
50.43
48.69*
70.04*
62.64
A MOTIVATION OF CHANNEL PERMUTATION
Based on the results presented in Sec- tion [5] is evident that after ap- plying channel permutation, the per- formance of N:M constraint mod- els significantly improves in compar- ison to executing N:M sparsity di- be
Pruned by Nayoliude
Wured by Worddo
Auseed Rib
rectly. This improvement demonstrated through empirical ob- servations made within Large Lan- guage Models (LLMs). As shown in Figure 5 pruning with 50% unstruc- tured sparsity versus direct N:M con- straint sparsity (2:4 in the figure) produces distinct variances in the distribution of input channels' retained degree. The N:M constraint, by its nature, results in a more uniform pruming pattern. With an example from the first row of Figure 3 when directly pruning with 2:4 constraint sparsity, if the input channels with universally larger scores get stuck in the same block, some of the weights will be "wrongly" pruned. Conversely, the block with universally smaller scaling scores will "wrongly" preserve some weights. One diminishes while the other grows, resulting in more uniform pruning, a smaller variance of the retained degree distribution of input channels, and therefore, a lower total sum of scores after pruning. In contrast, unstructured pruning takes a broader perspective, prun- ing weights within their global context, This expansive methodology inherently introduces more variability, leading to larger variance in the retained degree of input channels after pruning. The channel permutation method proposed in this article tackles this challenge smoothly by distribut- ing similar scaling input channels into different blocks, allowing weights that should be retained to be preserved as much as possible. The channels' retained degrees can be seen Figure S pruning.
Diggre
Deggee
Deg
The degree distribution for input channels after pruning.
Figure
distribution (2:4 + CP) of input closely mirroring the results of unstructured
B DETAILED EXPERIMENTAL SETTINGS
Tasks and Metrics. We mainly evaluate language modeling and zero-shot classification. Our initial assessment of language modeling involves a comprehensive evaluation of perplexity (PPL), where lower values indicate better performance. This evaluation is conducted on the test set of Wikitext2 (Merity et al. 2016). For further evaluation, we also conduct experiments classification to assess the ability of the sparse model to correctly classify objects or data points into categories it has not seen during training across five common-sense datasets: Hellaswag, BoolQ, URC--hallengee MNLI, and RTE, compared to the performance of the dense model. We run the ex- periments with the public GitHub benchmark Eleuther
zero-shot
(Gao al. 2021).
13

---

Published as a conference paper at ICLR 2024
Table 7: Assessing across various calibration and evaluation datasets. (LLaMA2-13B)
Eval, dataset
PTB
wikitext2
c4
Calib, dataset wikitext2
PTB
wikitext2
c4
PTB
wikitex12
c4
PTB
Magnitude
146.35
6.83
9.38
Wanda
68.49
69.70
63.58
5.85
5.97
5,89
8.43
8.30
8.25
SparseGPT
72.94
72.31
59.10
5.69
6.03
5.90
8.50
8.22
8.30
RIA
67.58
68.69
67.88
5.75
5.83
5,83
8.07
8.03
8.08
Baselines. For unstructured pruning, we compare with: 1) magnitude pruning (Zhu & Gupta
2017). the most prevalent pruning approach: and two more recent state-of-the-ar works LLM pruning: 2) SparseGPT (Frantar & Alistarh] (202) and 3) Wanda (Sun et al. (202)). These methods can be evaluated both unstructured pruning and N:M semi-structured pruning in the following
sections.
Calibration Data. We employ 128 samples from the C4 dataset (Raffel 2019) for all models, and each sample contains 2048 tokens. This also aligns with the settings in baseline methods for fair comparison. Note that we also discuss the choice of calibration data across different datasets, and more details can be found in Appendix
C SENSITIVITY TEST ON CALIBRATION DATASETS
Figure 2 illustrates that different datasets exhibit varying data distributions, which in turn im- pact the input activations. Consequently, we conduct experiments employing different calibration datasets to evaluate the robustness of the pruned model. Specifically, we utilized three different Wikksets-Wikkeext,,,, C4, and PTB (Marcus al. 1993) 1993)-with with 128 samples. each containing 2048 tokens, for calibration purposes, and evaluated the performance on Wikitext2. RIA is compared with Wanda and SparseGPTT, with the results summarized Table 7 RIA surpasses other algorithms in the majority of scenarios, except when using identical calibration and evaluation datasets for PTB and wikitext2. This exception can be primarily attributed to the weight reconstruction process. A more congruent distribution tends to yield enhanced performance through reconstruction. Therefore, we explore integrating the reconstruction approach with Wanda and RIA in Appendix [ Further- more, looking the results across different calibration datasets, the results of RIA more stable, which indicates that RIA is more robust than the calibration data.
D WEIGHT RECONSTRUCTION
Following the approach OBS (Hassibi al. 1993), which replaces the fine-tuning process with weight reconstruction using calibration data, this method has gained traction in L.Ms post-pruning
as an alternative to fine-tuning and retraining. SparseGPT (Frantar & Alistarh (023) extends this idea by permitting partial updates, thereby reducing computational complexity. The specific formula for this detailed (Frantar & Alistarh 2023). This weight reconstruction technique is essentially tool embraced by all PTP methods that revise the objective function of equation 7o
arg min W X, (M, O W:) X M.W;
In this formulation, Wi represents the weight matrix after undergoing the reconstruction process. In this section, we evaluate Wanda and RIA based on their adoption of weight reconstruction and juxtapose their performance with that SparseGPT. Our findings are presented in Table We omit the results from PTB as their PPLs excessively high, rendering them of no reference value. It's evident that RIA+rec outperforms other reconstruction- based algorithms. However, except when Wikitext2 is used both for calibration and evaluation, the reconstruction does not appear to enhance performance. Therefore, we have chosen not to employ this weight reconstruction approach in our main text.
14

---

Published as a conference paper at ICLR 2024
Table 8: Assessment of Post-Training Pruning (LLaMA13B).
Methods Integrating Weight Reconstruction
wikitex12
c4
wikitext2
c4
PTB
wikitex12
c4
PTB
Magnitude
6.38
9.38
Magnitude + rec
5.84
6.07
6.17
8.48
8.30
8.62
Wanda
5.85
5.97
5.89
8.43
8.30
8,25
Wanda+rec
5.70
6.00
5.91
8.57
8.29
8.35
SparseGPT 5.69
6.03
5,90
8.50
8.22
8.30
RIA RIA+rec
5.75
5,83
5,83
8.07
8.03
8.08
5.57
5.86
5.83
8.20
8.03
8.21
E CHANNEL CORRUPTIONS AND STRUCTURED PRUNING
Evidence of Channel Corruption. In Section discussed the occurrence of channel corruption during the application of Wanda for pruning LLMs. Figure illus- trates the distribution of connections per input channel in the weight matrix following pruning by both Wanda and RIA, where degree denotes the number of connec- tions a node has. Our findings reveal that in certain layers, Wanda results in roughly 10% of the channels experienc- ing corruption. Conversely, RIA ensures that no channels undergo corruption.
input channel degree distribution after pruning
Channel corruption
Occurrence
300
Detrimental Impact of Channel Corruption Model Performance. To assess the effects of full chan- nel pruning on model performance, we utilize LLM- pruner (Ma et al., (023), SOT) method for structured pruning in LLMs. These experiments are designed demonstrate the influence of channel corruption. More- over, we adopt a pruning strategy based on activation ||X|| as fundamental comparative method, which serves not only as a test of structured pruning but also as an ab- lation study for our RIA method. We compare the structured pruming methods with Wanda and RIA with both unstructured sparsity results and semi-structured sparsity results. In LLM LLM--runer (Ma al. 2023), they introduce skip-layer pruning strategy that skips the pruning process for the first several layers and the last few layers. To provide a thorough evaluation, we perform experiments on pruning across all model layers, elaborated in Table
1000
2000
3000
4000
Degree
Figure 6: Comparison of Channel Cor- ruption in Wanda and RIA. The plot shows the remaining node degree distri- bution post-pruning.
We have the following observations:
1) The overall performance of different sparsity patterns reveals that structured sparsity is less effec- tive than semi-structured sparsity even at low sparsity level of 25%, highlighting the detrimental impact channel corruption on model effectiveness. Across all sparsity degrees and patterns, RIA consistently outperforms Wanda, demonstrating its superior performance.
2) The introduction channel permutation has been found to boost performance as the model shifts towards semi-structured sparsity. This outcome is consistent with the observations detailed in the main text.
End-to-End Inference Latency of N:M Sparsity and Structured Sparsity, We offer the LLM inference latency with structured sparsity and 2:4 sparsity. Our experiment LLaMA2-7b de- ployed on 2 NVIDIA A100s with GB memory, where the input context sequence length is 128. We increase the batch size and record the inference latency accordingly. According to Table
15

---

Published as a conference paper at ICLR 2024
Table 10: Comparative performance structured and semi-structured pruning at various sparsity levels: experiments on LLaMA2-7B with calibration on C4 with 128 samples and evaluation on Wikitext2 using PPL metrics. The table includes experiments pruning on all layers.
1:4 (25% sparsity)
2:4 (50% sparsity)
3:4 (75% sparsity)
LLM-Pruner (structured)
28.88
nan
13570
Activation-base (structured)
39.54
10467
nan
Wanda (semi-structured)
5.94
12.15
2863.3
RIA (semi-structured)
5,90
11.27
1891.13
RIA CP (semi-structured)
5.81
10.12
1532.72
Wanda (unstructured)
5.68
6.92
1506.13
RIA (unstructured)
5.56
6.81
267.61
structured sparsity (50%) marginally outperforms the 2:4 sparsity model as the batch size increases. Importantly, note that with smaller batch sizes, both structured 50% sparsity and 2:4 sparsity mod- els do not significantly boost inference speed. However, as the batch size grows, the acceleration for 2:4 sparsity approximates 1.5x, whereas the structured 50% sparsity tends to reach about 1.7x acceleration.
In conclusion, N:M semi-structured spar- sity emerges as the optimal compromise between performance and latency. Although structured pruning
Table 9: The inference latency of 2:4 sparsity and structured 50% sparsity with varying batch sizes on LLaMA2-7h model.
Batch Size dense
8
16
64
281.13 ms
547.82 ms
1011.81ms
3742.76 ms
structured sparsity (50%)
238.92 ms
326.14 ms
616.13 ms
2181.69 ms
2:4 sparsity ation at the same sparsity level, the associated performance decrease is significant, even at very low sparsity levels.
offers the additional advantage of acceler-
225.60 ms
357.48 ms
731.81 ms
2495.41 ms
F IMPLEMENTATION DETAILS OF CHANNEL PERMUTATION
F.1 OUTPUT CHANNEL REORDERING
From a computational efficiency perspective, directly implementing permutation on input vectors introduces extra runtime overhead for extracting the permuted index and applying it the input vectors. However, an alternative approach can be considered. Given that each input serves as the output of the preceding layer, we can permute the output channels of the previous layers' weights serve as the permuted index for the next layers' input channels. This eliminates the need to permute the input index separately, resulting in time savings.
However, two key considerations must be taken into account:
The Q, K, and projection layers within single module must be treated as unified matrix, con- catenated into a single entity that shares the same input channel indices. This ensures that the input index permutation remains identical across these layers, avoiding the need for additional permuta- tions. In the case of ILaMA, the MLP module presents a similar scenario, as it contains parallel layers, namely, down_proj and gate_proj, which also need be concatenated into single matrix.
b) Another important factor to note is that this strategy cannot be applied when dealing with mod- els featuring residual connections. This is because inputs such modules also originate from the previous module. As result, permutations must be applied to the input activations at the begin- ning each module. However, this process does not significantly impact execution time. We've ve modified the RMS normalization layer which typically follows the Residual connection using Triton (Tillet et al. 2019). This ensures this layer's output channel indices align with the subsequent layers' input indices. In our tests, the time taken for this adjustment is negligible.
16

---

Published as a conference paper at ICLR 2024
59
530
500
Up
500
2200
200
1060
2039
130
5000
5000
G LLAMATS Layer2 Attn, Proj
[DT LLaMA78, Layeri5, Artn. O
LLaMA1BB Layer3 Attn. Krorj
[d] LLMMA1BB Layer? MLP. Up. Up_PPri
Figure 7: Outliers in LLaMA-7B and LLaMA-13B.
2000
1099
2003
Dhingg
and 4839 5000
y000 5000
4000
(1) Jorr:/e__ayeei_Attr Pro]
(b) OBTG Laver17_MEE FCL
1d) OPT138 L2 Ltt.ert__ettttcCCCCC Prro
(d) OPT13B Layero_ MLP MMP_FCL
Figure 8: Outliers in OPT-6.7B and OPT-13B
F.2 COMMUNITY-BASED PARTITIONING
In optimizing the process of Channel Permutation, we employ a method that significantly acceler- ates execution by partitioning the input channels into several communities. This strategic division reduces the runtime of channel permutation to A. where n represents the number of communities. The primary source of this reduction execution time of the sum of scores following N:M pruning, achieved when input channels are allocated into dis- tinct blocks. The experimental results presented in the main text are grounded on these optimization techniques.
from the more efficient computation
G COMPLEXITY ANALYSIS OF RIA AND CHANNEL PERMUTATION
RIA. We provide summary of the computational complexity of various Post-Training Pruning (PTP) algorithms. For SParseGPT, the time complexity approximately hiidtn))) (Frantar & Alistarh 2023), whereas both RIA and Wanda exhibit similar time complexities of approximately
(ddiitten
(Sun et al. 2023
The process of channel reallocation to obtain the sorted input channel indices is straightforward. Computation of the score matrix S involves determining the outcome of placing each objection into every box and summing the results while processing the 2:4 constraint within each block. Consequently, the time complexity for this computation is O J.Y.
Channel Permutation.
x M).
Table II: Permutation time (seconds) for Greedy algorithm and Channel Permutation.
4096
4096
5120
5120
6656 x 6656
8192 x 8192
Greedy
252.1
495.4
818.7
1563.6
Greedy + 100 escapes
845.3
1349.1
1896.4
3592.3
Channel Permutation
6.2
8.1
11.5
15.3
17

---

Published as a conference paper at ICLR 2024
Table 12: Integration of Fost-Training Pruning with Quantization Methods.
Magnitude
SparseGPT
Wanda
RIA
Unstructured (50% sparsity) Unstructured (50% sparsity) With GPTQ
16.02
6,99
6.92
6.81
15.21
7.59
7.41
7.28
Unstructured (50% sparsity) With AWQ
17.12
7.15
7.10
6.97
where r x M represents the computational complexity for the N:M constraint within each block. The time complexity the Hungarian algorithm is O M
The greedy method (Pool & Yu 202T> is challenging to implement in LLMs due to its prohibitive running time, as shown in Table We conduct only a single experiment for performance compari- LLaMA2- 13b, which took days. We tested the greedy method with 100 escape iterations to handle the permutation, the PPL on wikitex12 with a permuted 2:4 constraint is 7.83 which is just comparable CP method (7.77). However, the CP's execution time is just about I hour, making it possible to be applied in the LLMs.
H HUNGARIAN ALGORITHM
Given a bipartite graph with N left vertices and N right vertices, depicted by matrix S, where S.j represents the weight between the left vertex and the right vertex j. the algorithm aims to identify the ideal matching that minimizes the aggregate weight. This goal is captured in the subsequent objective function:
N ΣX i=1j=1
N
min
Sig X X
(8)
In this Equation, X.j is a binary determinant that showcases whether the left vertex is paired with the right vertex j. In our scenario, the initial first group of input indices treated as vertex with incomplete boxes acting as vertex This transition into LSA problem is seamlessly facilitated by the Hungarian algorithm, which subsequently derives the optimal permutation. After rearranging the first indices of the blocks, we apply the same procedure to the subsequent groups in sequence.
I INTEGRATION OF POST-TRAINING PRUNING WITH QUANTIZATION METHODS
Our experiments utilized the LLaMA2-7b model, focusing on two quantization methods: GPTQ (Frantar al. (002) and AWQ (Lin al. 2023). The calibration is performed using the C4 dataset, and evaluations are done on Wikitext2. We present our results in Table 12
Our finding suggests that this integration can reduce memory and inference acceleration with a slight detriment to performance.
We have identified two primary strategies for merging post-training pruning with quantization: (a) first pruning, then quantization, and (b) first quantization, then pruning. Our finding indicates preference for (a) pruning before quantization. This potentially because for (b), conventional block-wise quantization relies merely on the minimum and maximum weights within a block, and pruning can lead reduction these extremes, thereby potentially aiding quantization. In con- trast, (a) quantization before pruning adversely affects the computation of weight importance for pruning, resulting in diminished performance.
J THE EXPLANATION OF PREVENTING PERFORMANCE DROP FROM DENSE
The primary goal of post-training pruning methods to minimize the performance drop compared to the dense model. Therefore, designing metrics that offer fair evaluation of new methods against this baseline (dense performance) is crucial. Consider two post-training pruming methods, A and B, with perplexities P(A) and P(B), respectively. Let P(D) represent the baseline perplexity of the dense
18

---

Published as a conference paper at ICLR 2024
Impact of factor on RIA
6.6
ppl
LLaMA2-7b
6.2
0.2
0.4
0.6
0.8
1.0
Figure 9: We evaluate how to choose the factor 'a' inside the formula RIA with LLaMA2-7B and LLaMA-13b model. The calibration dataset is C4 with 128 samples, and the evaluation is on wikitext2. The sparsity is fixed at 50%. From the results, there are no significant differences between a=0.2 to a=55. For simplicity, select a=0.5 in the main text execute all the evaluations.
network. The effectiveness of method B in preventing performance degradation compared to method P(A)-P(B)
A can be quantified P(A)-P(D)- This formula allows for fair assessment of the new method relative to previous approaches, taking into account the performance of the dense model in the post- training pruning context. Using this calculation, the RIA method demonstrates a 17% improvement over Wanda and 50% improvement over SparseGPT in preventing performance drops from the dense model consistently across all tested models.
19


Reviews:
[{"review_id": "gGvsqofxWr", "reviewer": "Reviewer_NPvK", "summary": "This paper addresses the growing demand for efficient memory and computation in large language models (LLMs). Existing post-training pruning methods have attempted to reduce model size and computation but have not achieved optimal performance. The paper introduces a plug-and-play solution for post-training pruning of LLMs, featuring two innovative components: 1) Relative Importance and Activations (RIA), a novel metric that efficiently considers weight and activations in LLMs, and 2) Channel Permutation, a new approach to maximize the preservation of important weights with N:M sparsity. These components can be combined to enhance N:M structured pruning of LLMs. Empirical experiments demonstrate that RIA alone surpasses existing pruning methods on various LLMs. Moreover, N:M structured pruning with channel permutation can even outperform the original LLaMA2 70B on zero-shot tasks, while providing practical speed-up on specific hardware.", "questions": "Please address the concerns in Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Consider both weights and activations for unstructured pruning in LLMs is novel.\n- Particularly, the consideration of relative weight importance is a novel approach.\n- Channel permutation is a simple yet effective method for achieving N:M sparsity.", "weaknesses": "The reviewer recognizes the novelty and simplicity of the overall approach but has raised substantial concerns. The main issues pertain to the weaknesses in the baselines, which make it challenging for me to be convinced of the effectiveness of the proposed method. Moreover, the motivation and analysis provided appear to be inadequate. For example, concerning the former issue, the following questions come to mind: even if the proposed method can enhance the performance of N:M sparsity-based approaches, are N:M sparsity-based methods genuinely effective? Are they superior to contextual sparsity-based methods? \n\nI describe specific questions and suggestions regarding concerns as follows: \n\n- Insufficient experimental support for motivation: This paper argues the existence of 'channel corruption' asserting that removing input/output channels results in decreased performance as observed in prior works. However, the paper lacks empirical evidence to substantiate this claim. It would be valuable if the authors could include preliminary experiments to provide a basis for their motivation.\n- According to AWQ [1], activation-aware weight quantization, which selects important weights based on activation distribution rather than weight distribution, outperforms traditional weight-based quantization. Inspired it, the reviewer suggests that it would be meaningful to consider baseline methods based on activation-based weight pruning for comparison. Therefore, the authors might incorporate and compare unstructured pruning based solely on activations in Table 2.\n- In addition to the comparisons with N:M sparsity methods in Table 4 and Table 5, it is advisable to include a comparison with other structured pruning techniques in terms of performance and inference speed improvement. For instance, including a method like Dejavu [2] in the comparison would enhance the comprehensiveness of the evaluation.\n- What is the relevance of the experiments in Figure 2 to the claim that activation outliers exist independently of the dataset and model's parts? The reviewer thinks that even if activation values exhibit a high correlation between two datasets, it is possible that activation outliers can be eliminated. Therefore, it would be helpful to clarify the connection between Figure 2 and the claim about activation outliers.\n- Can we expect additional performance improvements when combined with post-training quantization methods such as Smooth Quant [3] or AWQ [1]?\n\n[Minor]\n- Why is the title \"plug-and-play\"?\n- The hyperlink in the 6-page appendix seems to be incorrect.\n\n[1] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration\n\n[2] Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time, ICML 2023\n\n[3] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, ICML 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "godfHfldna", "reviewer": "Reviewer_ee73", "summary": "Main Contribution:\n- The paper proposes two new methods for efficient post-training pruning of large language models (LLMs):\n    1) Relative Importance and Activation (RIA), a new pruning metric that considers both weight and activation information to determine weight importance. \n    2) Channel Permutation, a method to maximize retention of important weights when converting a model to N:M sparsity for hardware acceleration.\n\nNovelty:\n- RIA provides better pruning performance than prior state-of-the-art methods by avoiding pruning entire channels and using activations to assess weight importance.  \n- Channel Permutation reformulates the input channel permutation problem as a linear sum assignment problem, allowing efficient optimization using the Hungarian algorithm.\n\nExperiments:\n- Experiments conducted on LLMs including LLaMA, LLaMA-2, and OPT ranging from 7B to 70B parameters.\n- Tasks: Language modeling (Wikitext-2 perplexity) and zero-shot classification (5 commonsense datasets).\n- Compared RIA to magnitude pruning, SparseGPT, and Wanda for unstructured pruning.  \n- Evaluated Channel Permutation combined with RIA and other methods under N:M sparsity.\n\nResults:\n- RIA outperforms prior state-of-the-art post-training pruning methods in both unstructured and N:M sparsity settings.\n- Channel Permutation further improves performance under N:M sparsity by efficiently finding better channel arrangements.\n- Together, RIA and Channel Permutation provide an effective pipeline for LLM pruning and acceleration with negligible performance loss.\n\nConclusion:\n- RIA and Channel Permutation establish new state-of-the-art results for efficient one-shot post-training pruning of LLMs.\n- The proposed methods enable practical acceleration and size reduction of large models.", "questions": "* what is the channel here in the Transformer models? Transformer models does not have channel or column.\n\n* for \"\"We employ 128 samples from the C4 dataset\", is it possible/worth to do the experiments on a larger and more diverse calibration set (128 might be limited)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Proposes two novel methods (RIA and Channel Permutation) that provide state-of-the-art performance for post-training pruning of large language models.\n\n2. Comprehensive experiments conducted on multiple popular LLMs across a range of model sizes from 7B to 70B parameters.\n\n3. Evaluated on diverse tasks including language modeling and zero-shot classification to demonstrate generalization. \n\n4. Provides both theoretical analysis and empirical results to demonstrate the efficiency and efficacy of the proposed techniques.\n\n5. RIA and Channel Permutation can be readily combined into an effective pipeline for practical LLM pruning and acceleration, with negligible performance loss.", "weaknesses": "Overall the manuscript has solid contributions, but expanding the variety of models, tasks, and languages could strengthen the demonstrated effectiveness. Testing scalability and comparing to other recent techniques would also help round out the evaluation. But within the chosen scope, the paper delivers valuable advancements for efficient LLM pruning.\n\n* For \"We employ 128 samples from the C4 dataset\":\nusing only 128 samples from C4 as the calibration data is quite limited. With so few samples, the activation statistics may not sufficiently capture the full distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gm3A8gSp0e", "reviewer": "Reviewer_hn9p", "summary": "* The paper introduces a relative importance pruning metric which leads to more uniform pruning patterns.\n* The paper proposes to apply channel permutations found by a scalable heuristic in order to relax the pattern restrictions imposed by n:m pruning.\n* Both techniques are evaluated on Llama models for perplexity and zero-shot tasks.", "questions": "* How does 4:8 perform with channel permutations in the setup of Table 3?\n* Did you carry out any additional ablation studies around parameter a other than Table 2? I am curious if a = 0.5 is generally the sweet spot or if it was just picked for simplicity.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed techniques are relatively simple to implement in practice and in particular the channel reordering seems to be quite effective.\n* The paper is easy to understand and provides clear visualizations of the key algorithms.\n* Evaluation is carried out on strong Llama models and not just on older OPT ones.\n* The Appendix contains interesting additional studies like combining RIA with SparseGPT reconstruction.\n* The paper also considers practical acceleration of sparse models in Table 5.", "weaknesses": "* The observation that encouraging a more uniform sparsity pattern is beneficial was also made by Wanda, RIA seems to be an extension of that (also across columns). Similarly, that permutation reordering can be helpful for n:m sparsity was found by [Pool & Yu, 2021], this paper only introduces a simpler but more scalable heuristic for finding such a permutation, based on average activation values. While there is some novelty, it is not particularly high.\n* For unstructured sparsity, the improvements of RIA over prior work are relatively small at around 0.1-0.2 points in perplexity. The impact of the more advanced linear sum assignment permutation method also seems rather minor. At the same time, perplexity increases from the dense baseline are still quite large, especially for 2:4. Hence, it is not clear how useful the corresponding sparse models would be in practice.\n* There does not appear to be any code in the Supplementary. I hope the authors plan to open-source their work to aid reproducability.\n\nWhile I do not think that the paper brings particularly strong new ideas or practical results, I find it interesting that encouraging even more uniformity than Wanda is beneficial, and that permutation reordering is quite effective for n:m pruning even for massive LLMs. Hence, I am currently leaning towards acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WscZXqGGIZ", "reviewer": "Reviewer_dzPo", "summary": "This paper introduces a post-training N:M pruning solution for LLMs. The method is built upon two components: (1) relative importance and activation, which considers both the weights and activations within LLMs for a better weight importance estimation; and (2) channel permutation, which can better preserve important weights under n:m sparsity through rearranging channel orders. Experiment results demonstrate the effectiveness of the proposed method and its superiority over existing baselines.", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The channel permutation and the Hungarian algorithm are novel techniques, and the experiments demonstrate the effectiveness under n:m sparsity\n2. The paper is easy to follow, and the authors conduct extensive experiments.", "weaknesses": "1. The first technique, relative importance, and activation, seems to be an incremental improvement. \n2. In Section 5.3, which discusses N:M sparsity, I was anticipating experimental results on smaller LLMs such as Llama2-7b, and a higher sparsity ratio than 50%. This would potentially highlight the advantages of the proposed method over SparseGPT and Wanda more effectively. Could the authors provide their insights on this?\n3. Regarding the inference latency under n:m sparsity, I would like to suggest that instead of providing layer-wise speedup, the authors could consider providing end-to-end latency for the pruned LLMs. My reason for this suggestion is that I am curious about whether n:m sparsity is indeed an effective structured pruning pattern within LLMs, especially when compared to pure structured pruning methods such as LLM-Pruner. Could the authors provide their perspective on this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)