Search for Efficient Large Language Models
Authors: Xuan Shen
Pu Zhao
Yifan Gong
Zhenglun Kong
Zheng Zhan
Yushu Wu
Ming Lin
Chao Wu
Xue Lin
Yanzhi Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Architecture Search
Large Language Models
Primary area: generative_models

Abstract: Large Language Models (LLMs) have long held sway in the realms of artificial intelligence research.
Numerous efficient techniques, including weight pruning, quantization, and distillation, have been embraced to compress LLMs, targeting memory reduction and inference acceleration, which underscore the redundancy  in LLMs.
However, most model compression techniques concentrate on weight optimization, overlooking the exploration of optimal architectures.
Besides, traditional architecture search methods, limited by the elevated complexity with extensive parameters, struggle to demonstrate their effectiveness on LLMs.
In this paper, we propose a training-free architecture search framework to identify optimal subnets that preserve the fundamental strengths of the original LLMs while achieving inference acceleration.
Furthermore, after generating subnets that inherit specific weights from the original LLMs, we introduce a reformation algorithm that utilizes the omitted weights to rectify the inherited weights  with a small amount of calibration data.
Compared with SOTA training-free structured pruning works that can generate smaller networks, our method demonstrates superior performance across standard benchmarks.
Furthermore, our generated subnets can directly reduce the usage of GPU memory and achieve inference acceleration.

Search for Efficient Large Language Models
Xuan Shen', Pu Zhao', Yifan Gong', Zhenglun Kong", Zheng Zhan', Yushu Wu', Ming Lin Chao Wu', Xue Lin', Yanzhi Wang'
Northeastem University, "Harvard University, *Oracle (shen.xu, xu, yanz, .ang) Soorheeaotthheeeerr edu
Abstract
Large Language Models (LLMs) have long held sway in the realms of artificial intelligence research. Numerous efficient techniques, including weight pruning, quantization, and distillation, have been embraced to compress LLMs, targeting memory reduction and inference acceleration, which underscore the redundancy in LLMs. However, most model compression techniques concentrate on weight optimization, overlooking the exploration of optimal architectures. Besides, tra- ditional architecture search methods, limited by the elevated complexity with ex- tensive parameters, struggle demonstrate their effectiveness on LLMs. In this paper, we propose training-free architecture search framework to identify opti- mal subnets that preserve the fundamental strengths of the original LLMs while achieving inference acceleration. Furthermore, after generating subnets that in- herit specific weights from the original LLMs, we introduce a reformation algo- rithm that utilizes the omitted weights rectify the inherited weights with small amount of calibration data. Compared with SOTA training-free structured prun- ing works that can generate smaller networks, method demonstrates superior performance across standard benchmarks. Furthermore, our generated subnets can directly reduce the usage of GPU memory and achieve inference acceleration. Code: bottps://githhr com/ wwwwshwcwwaaweeeeeeoo.op.paaaeee seachkhheeerre
Introduction
Large Language Models (LLMs) 1234 5 6 are renowned for their exceptional performance across various domains of artificial intelligence research. There a growing demand for construct- ing LLMs for extensive applications across multitude of popular platforms. However, the com- putational and storage costs have restricted LLMs from deployment on various devices for wide applications. Take the GPT-3 model as example, with its 175 billion parameters [6), it requires more than 326GB of memory in FP16 format. This exceeds the memory capabilities of even the most sophisticated GPUs, far surpassing available memory resource-constrained devices. To address these challenges, variety of compression techniques focusing on weight optimization have been developed, including weight pruning (7 8
10 12 [4 15 [6], quantization [17 [8 19].
and knowledge distillation [20] 21 22 The extensive research in the compression direction indi-
cates the substantial redundancy within LLMs.
Besides optimizing model weights, improving the model architecture another crucial direction in achieving both high efficiency and superior performance. Numerous works 23 24
25
26
27 28
29 30 31 sentative model designs such Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). However, the realm architecture search for LLMS remains unexplored. Though enjoy- ing the potential benefits of discovering highly efficient and well-performing LLM architectures compared with manual designs, searching with traditional NAS methods for LLMs faces significant challenges due the immense complexity and extensive model size. Furthermore, the convergence
32
33 34
35 36) have studied the Neural Architecture Search (NAS) problem for repre-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Perpeciy
Perlecity
Peplesity
Pepiesity
13
Oas
SIare Out
30
0062278
15
10
Bone
100%
43%
92% 20ro 70% (9% 99% (b) LLaMA-78
Sonto
WPS 80% 70% oos
(4) Family
Family
(c) LLaMA-138
(d) LLaMA-30B 658
Figure I: Experiment results of perplexity + on WikiText2 dataset with 2048 sequence length.
of randomly initialized architecture the searched optimal state takes substantial training efforts and resources, intensifying the challenges of searching for efficient L.M architectures.
To tackle the challenges, we propose training- free architecture search framework that discovers ef- ficient LLM architectures within the original well-trained LLMs. Specifically, to reduce the search cost, we first identify an appropriate initial architecture by computing the importance of weights. Subsequently, an evolution-based algorithm applied to globally search an efficient subnet starting from the initial subnet. In each generation, mutation and crossover are adopted to generate candidate architectures within the search space. The candidates are evaluated efficiently with a small amount of training samples assess their effectiveness and select for the next generation. As we star search from well-trained LLM instead of randomly initialized models, we propose a mask muta- tion algorithm to identify the detailed channel indices, rather than just the number of channels in the mutation of traditional NAS [29] 24, 25 26 32]. After a few generations with the identified promis- ing LLM architecture, adopt the reformation algorithm based on the alternating direction method of multipliers (ADMM) 37 38 39 40 to rectify weights in inherited efficient LLM architecture
with omitted weights (i.e., non-inherited weights) leveraging only 128 calibration samples.
As shown in Figure D our extensive experiments demonstrate that our method can achieve supe- rior performance than SOTA structured pruning baselines in terms of perplexity and zero-shot ac- curacy on multiple datasets across various LLM families and model sizes. Particularly, as in Fig- (a) only SliceGPT and our method support the OPT model family, and method outperforms SliceGPT. Additionally, with a 60% inheriting ratio for the LLaMA-7B model on the WikiText2 dataset, our method achieves the best performance with perplexity of 10.21, compared to 38.27 by LLM-Pruner and 279.52 by SliceGPT, as illustrated in Figure [ (b). Furthermore, when scaling to LLaMA-13B. both SliceGPT and LLM-Pruner fail, as in Figure[(c).. Lastly, as in (dg only FLAP and our method support the LLaMA-30B and 65B models, and our method achieves better performance than FLAP Besides, our implementations on GPUs demonstrate significant memory reduction and inference acceleration. Meanwhile, our approach eliminates the retraining process, relying solely on forward pass for both searching and reformation processes, which maintains low memory overhead.
Our contributions are summarized below,
1. We propose training-free free search framework identify subnets within LLMs, featuring an importance-awart initialization that significantly reduces the time cost of searching, and an evolu- tion architecture search with special mask mutation and efficient candidate evaluation.
2. We propose a reformation algorithm that reconstructs weights by calibrating with only 128 training samples, thereby enhancing the effectiveness of the subnets.
3. Experiments indicate that the subnets generated by our method outperform SOTA structured pruning works in terms of perplexity and accuracy on multiple datasets across various LLM fami- lies and sizes. The searched subnets can effectively reduce GPU memory and accelerate inference.
2 Related Work
2.1 Compression of LLMs
Various compression techniques have been developed to reduce the model size or inference cost of LLMs, including model pruning, quantization, and distillation. Among these, quantization and

---

Embedding
L Head
Inberit
Reform
Weight
Original Weight
Initial Architecture
Blocks
Original Weight
Reformed Weight
Initialization
2. Scarch
Reformation
Figure 2: Framework Overview.
structured pruning methods prevalent due their efficacy in inference acceleration while pre- serving task performance. Quantization approaches, as explored works [19 18] 17]. compress models by converting weights lower bit representations. Besides, structured pruning techniques, including the works 18, 9 10 41) remove redundant weights in structured manner to reduce the total weight count. Specifically, LLM-Pruner [8] eliminates non-critical coupled structures based on gradient information, while SliceGPT [9] substitutes each weight matrix with a smaller, dense matrix and reduces the embedding dimension of the network. FLAP [TO] employs structured met- rics to prune LLMs and globally optimizes the sparsity ratio with the output feature maps. Despite the advancements, most pruning methods indiscriminately remove heads within the self-attention modules, leading to more significant performance loss due to the inherent input-dependent nature of transformer architectures based on all heads.
2.2 Search for Transformers
NAS has emerged pivotal technique for identifying efficient architectures in CNNs (exempli- fied by EfficientNet [42) and transformer- based models (such as BERT [43 and Vision Trans- former To mitigate the typical high training costs associated with NAS, innovations such as one-shot and zero-shot NAS methods [26] 29
25 24] have been developed, enhancing the efficiency of generating nigh-performance architectures. In contrast to zero-shot NAS methods, which utilize accuracy predictors to derive optimal architectures, one-shot NAS methods streamline the process by pretraining a comprehensive supernet from which optimal subnets are subsequently selected. Specif- ically, in the context of transformer- based models, the one-shot NAS approach, as implemented in AutoFormer [29], involves multiple rounds of supernet training, strategically extending weights along certain dimensions to optimize performance. NASVIT [26] leverages gradient information during supernet training to refine subnet selection and mitigate gradient conflicts, thereby enhancing the effectiveness of generated architectures. The proven efficacy of one-shot NAS for transformer architectures provides compelling rationale for its application to LLMs, considering that pretrained LLMs can function analogously as supernets. This adaptation holds the potential to significantly ad- vance the development and optimization of L.LM architectures, motivating refine and enhance the capabilities of these complex models.
Methodology
3.1 Framework Overview
We show the overview of our search framework in Figure It comprises three key components: search initialization, search pipeline, and weight reformation. First, an initial efficient architecture constructed layer by layer with uniform inheriting ratio based on the weight importance. Sub- sequently, based on the initialization, we conduct comprehensive search process for the globally efficient architecture with the evolution-bart search method. Finally, reformation method is in- troduced to enhance the performance of the resulting subnets in LLMs without retraining.
3.2 Search Initialization
Global search with uniform initialization. Unlike prior efficient LLM research efforts such as SparseGPT [7] with a uniform sparsity ratio across all layers, our method leverages global search approach, such that different layers in our searched architecture may inherit different percentages of parameters (inheriting ratios) from the original LLM. To reduce the search cost and promote the search performance, we initialize our search with the same inheriting ratio for all layers. Through our search process, we iteratively refine the architecture, yielding subnets with varied inheriting

---

Out Proj
Färment Wise
Down Proj
Embodded Tokens
Attn Map
Up Proj
Layers
Embedding
LM Head
*
Softmax
Hidden
Activation
Subeqquen
Column Search
Gate
Mul
Proj
Rosddaa
Residual
(B,N,
[B.N.D]
Figure 3:
Visualization of the subnets generation for LLaMA family based on the selections masks Sattn for the self-attention module colored in blue and Smip for the MLP module colored in green.
ratios across layers. We demonstrate the pivotal role of the initialized architecture in driving search efficiency and effectiveness in Figure and Section B.3.3
Structural subnets. To enable efficient inference, we search structural subnets from the original LLMs, i.e., certain rows or columns the original 2D weight matrix are inherited in our searched model. Take the ILLMAA family as an example. In each attention block of IJ LLaMA models, there are query, key, value, and output linear layers the self-attention module with weights denoted by We. W K. Wy. and Wo. respectively, and other three linear layers Wv. WG. and WD in the MLP module. To ensure the consistent hidden size in LLaMA models, based on the computation patterns in each block, we select rows in W Q. WK. Wv, Wu, and WG. and columns in W o and WD. More details are presented in Figure 3 and Appendix
Initialization based on importance score. We construct the initial building blocks by inheriting appropriate rows/columns from the original LLM layer. To determine which row/column to be inherited, we compute the importance score for each row and column as below,
W [2XX 45
[ Σ
[wwly
[$]
where w represents the row score for the jth row of W and Wll denotes the column score of the jth column. [$) (.j is the importance value of the element in the ith row and jth column of W, and X is the layer input. Following WoodFisher [45) and SparseGPT [Z]. the importance score reflects the minimum error of the layer-wise outputs (in terms of & norm) caused by removing single weight. Note that the minimum error is evaluated by removing a single element from the weight matrix and is not optimal in the case of simultaneously removing multiple weights.
Mask sharing. Given the column and row scores, we encode the architecture information by two masks: S attn RM for the self-attention module and Smip E Rº for the MLP module for the layers in each building block. Different layers in the same module (self-attention or MLP) share the same mask to align the internal computations. We consider minimizing the collective importance scores for both the self-attention and MLP modules as below,
min Sartm
Sattn O (wwa D + Dw, + @wo)lls
min Smip O (DW, + +w + Dww))l
where I|-I| denotes the & norm and @ means the element-vin multiplication. Given the target model size, we uniformly set the same inheriting ratio for the masks in all building blocks. To obtain the mask in each block, we perform sorting for the sum of the corresponding scores in Equation 22) and (3), and inherit/keep the subnets with larger scores the initialized architecture with the target size following the inheriting ratio, while other rows/columns with smaller scores are omitted.
3.3 Architecture Search
In this section, we present our comprehensive training-fres search framework with the visualization of the search process for one block of the LLLMA model shown in Figure We first delineate methodology for mutation based on the initialized selection masks. Next, we define the search space and present the search pipeline. Besides, we verify the effectiveness of our initialization strategy by comparing the convergence speeds with and without our initialization.

---

3.3.1 Mask Mutation
Algorithm 1: Mask Mutation
During the search, we use mask muta- tion to generate new masks and thus new subnets to explore the search space. The inheriting ratio for the selection mask Sattn is denoted as I ottn (naten Mi where h is the number of heads, and the inheriting ratio for Smip is "mdpp The mutation function M with the original mask Sattn or Smip, mutation proba- bility P... inheriting ratio requirement Varta "mmlp: similarity ratio a, and maximum iteration can be represented follows,
Input: Pm. V a, Pr + Random
if Inheriting Ratio(S) = 7, and P. Pm then
Output: S
N + lem iter + Idz; t IS == 1), Idæa + while len n Idx2) < * N and iter < 1) do
Idrr f Pandom__Suseee
1, 1, ,, ,N 1117)
iter t iler +1
end S' O.,; ['(1dz2) t-
Output: S' if iter < 7 elss S
S = [M(S Pmm Tauna a, ())
(4)
S'mip = M(Smuip P'ms Tmmpp a. 7)))
(5)
where Sann R. denotes the selection mask for the 7th head and hm is the head dimension. In details, we show the mask mutation process with Algorithm [1 If the inheriting ratio of input S already satisfies the requirement y and the mutation unnecessary based on the random generated P. (i.e., P. Pm), we do not mutate and simply return S. Otherwise, given S and thus the set of indices Idæ; for the inherited rows or columns, we try to generate a new set of indices Idz; through random sampling between to (Sn(S) 1, such that (i) Idx; follows the required inheriting ratio requirement y, and (2) the similarity Idr; and dæ (intersection set) is larger than threshold a.
3.3.2 Search Space
We define the LLM search space with three variables for each transformer building block below: the model depth d, inheriting ratios Fate = fram 1h for Sattn, and Tmmpp for Smip. The specifica- tions of this search space, including the range for each factor, are detailed in Table
Tmlp has a larger search space than {Y J according to our ablation study illustrated Figure Results are evaluated using LLaMA-7B on the WikiTeex2 dataset with sequence length of 2048. Specifically, we apply the same local inheriting ratio for three cases, (i) the attention module only, (ii) the MLP module only, and (iii) both modules. Note that in case (i) or (ii), the global inherit- ing ratio larger than case (iii) since the MLP in case (i) or the attention case (ii) directly uses the original layers with 100% inheriting ratio. From Figure 4 we observe that case (ii) achieves better perplexity with a lower global inheriting ratio than case (i), demonstrating that the MLP ex- hibits greater redundancy and is less sensitive to parameter reduction than the self-attention module. Therefore, we set a larger search space inheriting ratios for MLP than the self-attention module.
Different from other transformer-basee search works 29 26 46). heads in self-attention. stems from the nature of transformers that all heads are essential for repre- senting the input data in the attention mechanism. Moreover, we refrain from conducting searches on the embedding and output layers of LLMs, as their weights constitute only minor fraction of the total parameters yet are vital for the precise representation of tokens.
we do not search the number of
3.3.3 Search Pipeline
We implement our evolutionary search across the OPT and LLaMA model families with varying model sizes to derive efficient LLM architectures/subnets. The pipeline is shown below.
Table 1: Search space for different model sizes of OPT model family and LLaMA model family,
where the notation [a, b, cj specifies a range from a to with an interval
Model
OPT Family
LLaMA Family
General
Space
Model Depth
Model Depth
Inheriting Ratio
Params.
125M
1.3B
2.7B
7B
13B
30B
65B
Tinty
90%
12, 1]
24, 24,
32,32.
122.32.
[40, 40,
60, 60.
80, 80.
0.9, 0,001
0.6. 0.05
80%
12, 12, 1
21. 21,
30,32 1
32, 32,
[10 10,
60, 60,
80, 80.
0.S, 1,0.1
0.1. 1,005
70%
10. 12,
20,
22,
33, 32.
36, 40,
60.
76,
00.3, 1,001
0.2, 1,0.5
60%
[0. 1
20, 21.
28,
28. 32,
32, 10,
52, 60.
72,80.1
[0.6, 0,001
0.1. 1,0.0
50%
8,
16, 24,
24,32,
28, 32,
32, 40,
52,60.1 60,
72,80.
1,001
0,
5

---

Global Ratio
Perplexity
Logz Perplxiity
100%
Ratio
PPL
SelF-Aum
100
wio
Intial
95%
MEP
MLP
90%
Both
80
Both
12
85%
60
80%
73%
70%
63%
60%
Local Ratão
90%
80%
70%
Epoch
Figure 4: Ablation analysis of the inheriting ratios
Figure 5:
Ablation analysis of convergence speed with or without our initialization.
applied to the self-attention, MLP, both.
multiple candidates (N subnets in total) are generated by mutation of the inheriting ratios with probability P and then mask muta- tion (Section 3.3.1 with probability po The depth mutation is not involved this initial step. The top l subnets are preserved as the initial generation.
Initial generation. Given the single initialized subnet (Section 3.2)
Following generation. With the subnets parental candidates, new population with N can- didates are generated through mutation and crossover. We select a random parental candidate for mutation until the number of mutation candidates reaches threshold Nm. Mutation involves al- tering the depth with probability Pa. altering the inheriting ratios with probability P. < Ph and mutating the mask with probability Pm Pn (see Algorithm [. The probabilities are smaller than initial generation as superior candidates should be preserved with less randomness. For the crossover, two parental candidates are randomly selected and combined form a new candidate until there are N₂ candidates. With the population from the parental candidates, top k subnets are preserved as the next generation.
Candidate evaluation. For each generated candidate, if its parameter number does not fall in the range of the target model size, it is unsatisfying and we simply drop To compare candidate subnets, we evaluate them with a few random training samples from WikiText2 to compute the perplexity.
Necessity for initialization. To verify the effectiveness of our initialization in the search, ab- late the initialization for three cases, (i) self-attention only. (ii) MLP only. and (iii) both modules. LLaMA-7B is adopted with 80% inheriting ratio for selection masks. Results are evaluated on Wikitext2 with a 2048 sequence length. As shown in Figure the self-attention module complicates the identification of an effective subnet without initialization strategy. In contrast, the MLP mod- ule exhibits less sensitivity to initialization. The search in both modules struggles to yield effective subnets without our initialization, primarily due to self-attention. The observation underscores the necessity of our initialization approach.
3.4 Reformation
After the search, we can obtain subnet from the original LLM. To improve the subnet performance, further reform the weights in the subnet by using the omitted weights to compensate their loss. Specifically, for each linear layer in the subnet with their original weights W before the search, we would like to reform the weights under the searched mask M and obtain W, so that the layer output difference in & norm, i.e., [W 2 is minimized. The problem is formulated below,
min W
[X W
S.L. W.M-O
(6)
where M indicates the location pruned weights with element denoting pruned and denoting unpruned weights. Here we only reform inherited columns based on omitted columns in W rather than reforming rows with omitted rows, since the output corresponding to omitted rows are always zeros which are unavailable for any compensations by modifications in other rows. To solve this problem, we propose solution based on alternating direction method of multipliers (ADMM) 38 with the following theorem. The detailed proof is shown in Appendix
37 47

---

Table Results of the compressed LLaMA-7B and LLaMA-13B on the WikiTex12 dataset, PTB dataset, and other common sense reasoning datasets. The perplexity on the WikiTex12 and PTB is calculated with the 2048 sequence length. The accuracy results are evaluated with the same pipeline as
[8] to ensure a fair comparison. The average is computed across seven classification datasets, LLM-Pruner (v). (e2). and (el) denote the vector-wise and element-wise importance, (c)
and (b) denote the channel and block strategies.
Method
Inheriting
Wiki
PTB
BoolQ
PIQA
Hella Swag
Wino Grande
ARCae
ARC-c
OBQA
Average Acc.1
Ratio
PPL]
PPL.
LLaMA-7B
100%
5.68
27.34
73.18
78.35
72.99
67.01
67.45
41.38
42.40
63.25
(..M-PPuneey)
7.73
38.94
67.95
77.42
69.31
63.54
66.33
39.85
41,20
60.80
LLM-Pruner(ee]
90%
7.46
36.87
68.29
76.88
70.25
64.33
65.28
40.10
39.60
60.68
7.42
36.73
66,97
77.26
70.30
64.33
65.24
40.19
41.00
60.76
SliceGPT
90%
7.00
133,80
57.68
69.80
59.32
68.11
62.75
36.01
38.00
55.95
H.AP
90%
6.34
32.39
74.43
75.41
68.68
67.01
65.78
38.48
41.00
61.54
Ours
90%
6.10
32.05
74.37
76.88
70.71
67.56
68.39
40.10
39.20
62.46
ALMM-Prunere
10.73
59.73
61.44
71.71
$7.27
54,22
55.77
33,96
38.40
53.25
LLM-Pruner(e2)
80%
11.97
55.68
59.39
75.57
65.34
61.33
59.18
37.12
39.80
56.82
LL I
10.73
59.73
57.06
75.68
66.80
59.83
60.94
36.52
40.00
56.69
SliceGPT
80%
8:71
143.89
37.89
64.09
45.67
62.75
53.62
31.74
33.20
46.99
FLAP
80%
7.40
36.77
68.59
74.21
64.98
64.40
59.89
37.80
40.20
58.58
Ours
80%
6.89
36.06
70.98
74.92
67.29
64.64
64.23
36.52
39.40
59.71
LLaMA-13B
100%
5.09
19.23
68.47
78.89
76.24
70.09
74.58
44.54
42,00
64.97
(LMM-Puuerric)
7.70
35.32
68.47
74.76
66.99
66.38
66.58
35,24
38.20
$9.52
90%
LLM-Pruner(b)
6.38
31.85
70.64
78.40
75.00
69.46
72.82
41.47
41.40
64.17
SliceGPT
90%
6.43
86.09
60.74
69.38
66.79
40.70
41.80
58.73
FLAP
90%
5,45
20.98
63.76
78.07
73.69
69.61
69.53
39.93
41.60
62.31
Ours
90%
5.39
20.63
71.65
78.18
75.04
69.61
69.70
43,09
42.60
64.23
LLMM-Prunered
48.76
218.49
62.39
66.87
49.17
$8.96
49.62
31.83
33.20
50.29
80%
LLM-Pruner(b)
10.05
55.46
67.68
77.15
73.41
65.11
68.35
38.40
42.40
61.79
SliceGPT
80%
7.55
117.94
50.34
66.00
53.37
68.11
60.56
36.35
38.20
53.27
FLAP
80%
6.03
23.33
62.23
76.50
70.59
68.35
65.66
38.99
41.60
60.56
Ours
80%
5.90
22.66
68.53
77.09
72.60
69.22
66.25
40.02
41.00
62.10
Theorem 3.1. Problem 6 can be solved in iterations. In the t iteration, it performs the updates:
W++1
=(Xx" pl) 1
"x" + ofz"-u")
nT
(7)
Z +1 W U*) OM. M.
(8)
U++1 =U* w -2++1
(9)
where p >> the penalty parameter. The initial variable values at k = 0 follow the configurations that W W, z Wo, and U
In practice we find that after few tens iterations (such as 20 or 30), the loss in Problem (6 converges. The complexity is determined by the inversion operation, which is the same as SparseGPT [7] However, as it can converge fast within 30 iterations, our ADMM-based solution typically requires less computation time than SparseGPT which needs iterate over all rows in the weight matrix.
3.5 Efficient Inference
After searching and reformation, can get optimal efficient subnets with the selection masks Sattn E RM and Smip E R for each block the LLMs. We further convert the subnets into small-dense models following the masks for efficient inference. Thus, the dimension of the weight is actually reduced with faster inference speed. More details can be found in Appendix[A]
Experiments
4.1 Experiment Settings
Hyper-parameter setting. For the evolutionary search, we adopt specific hyper-parameters as fol- lows: the population size (N), the number of mutations (Nm)), and the number of crossovers (N.)
7

---

Table 3:
Results of compressed Vicuna-7B.
Method
Inheriting
Wiki
FTB
BoolQ
PIQA
Hella
Wino Grande
ARC-e
ARC-c
OBQA
Averuge
Ratio
PPL.
PPL.
Swag
Acc.7
Vicuna-7B
100%
6.78
26.78
76.37
77.75
70.64
67,40
65.11
41.21
40.80
62.78
LLM- (MMM--rrrree))
8.74
30.69
61.68
75.95
69.35
64.72
68.86
39.16
40.00
59.96
SliceGPT
8.23
61.83
67.00
69.64
58.65
63.77
54.59
37.71
38.40
55.68
90%
FLAP
7.64
29.95
74.43
75.41
68.68
67.01
65.78
38.48
41.00
61.54
Ours
7.18
28.87
75.10
75.90
66.92
67.11
40.61
40.40
62.29
LLM-Pruncrie2)
12.97
46,34
62.87
75.41
64.00
58.41
60.98
37.12
39.00
56.83
SliceGPT
80%
10.13
94.93
53.82
64.42
49.14
60.38
52.27
34.56
33.40
49.71
FLAP
8.90
34.04
69.14
72.52
63.01
64,48
61.11
34,56
39.00
$7.69
Ours
8.20
33.59
66.30
74.92
65.05
62.90
64.63
38.91
39.80
58,93
Table 4: Results of compressed OPT.
Table 5: Results with lower inheriting ratio.
Ratio
90%
80%
70%
Ratio
70%
60%
50%
Method
|Wiki
L |Wiki L/ [iiki PTB 4
Method
|wiki 4 PTB L/Wiki L PTB Wiki
PTB 4
OPT-125M
Wiki:
27.65
PTB: 38.99
LLaMA-7B
SliceGPT
35,31
75.59
54.88
149.17
84.16
245.18
L. 22M
18.58
93.24
38.27
238.09
125.96
460.73
Ours
30.97
40.14
44.12
66.55
80.84 124.27
SliceGPT FLAP
15.95
583.58
279.52
5186.06
1830.43
15333,66
9.18
47.35
12.34
65.54
21.89
135.84
OPT-1,3B Wiki:
14.63
PTB:
20.29
Ours
8.28
45.26
10.21
62.07
15.48
117.06
SliceGPT
16.74
35,31
20.17
61. 30
28.53
113.42
LLaMA-13B
Ours
15.51
20.19
19.23
32.81
26.82
69.42
(LMMPruner(c2)
22.36
112.03
66.38
278.36
3827.63
1287.11
OPT-2.7B Wiki:
12.47
PTB:
17.97
SliceGPT
9.79
167.27
13.21
247.71
19.95
408.68
SliceGPT
14,10
37.01
16.81
65.09
24.12
132.13
FLAP
6.97
27.38
8.67
35.91
12.88
53.54
Ours
13.32
17.24
16.44
23.66
23.48
58.46
Ours
6.67
26.37
8.00
33.23
10.44
45.73
set 100, 50, and 30, respectively. In each generation, the top 10 subnets are selected as parental candidates to produce offspring networks through the mechanisms of mutation and crossover. The rest subnets in the population are generated with mutation with larger randomness (i.e., same as initial mutation). The initial mutation probabilities (Po and Po)) are set at 0.6 and 0.3 to promote variability early in the search process. Subsequently, for ongoing generation processes, the muta- tion probabilities (Pm. and P.) are adjusted to 0.3 and 0.1, while the probability for depth (Pa) is maintained at 0.1. The similarity ratio and maximum iteration v) are at 0.8 and 1000 in mask mutation. The total evolution epoch is 50. For the reformation, we adopt p as 1.0 and the iteration number as 30.
Datasets and metrics. We compare the perplexity of the models on the WikiText2 [48 and PTB [4] datasets with the 2048 sequence length. We also compare the zero-shot accuracy
mon reasoning zero-shot classification datasets including BoolQ [50]. PIQA [51], HellaSwag [2] WinoGrande [53], ARC-easy (54), ARC-challenge (54). and OpenbookQA [55]
Models. We evaluate on multiple LLM families including L.LMA 1. Vicuna 56 and OPT
Baselines and pipeline. We compare with SOTA baselines including LLM-pruner [8. SliceGPT [ and FLAP [10]. We adhere to the exact evaluation pipeline from the well-known LLM--rrnerr (8] which is applied for all approaches to ensure a fair comparison. For the reformation, we randomly select 128 samples from training split of WikiTex12, with the same random seed and thus same data for the calibration of other works, including SliceGPT and FLAP, ensuring a fair comparison.
Search Cost. We leverage the evolution search NVIDIA A100 40G GPUs. Specifically, to explore the subnets of LLaMA-7B, we finish the search on one GPU with around 5 hours.
4.2 Main Results
Superior performance compared with SOTA baselines. We show our results of LLaMA-7B and LLaMA-13B in Table 2 and Figure (b) and (c). We observe that our method outperforms all baselines in terms of perplexity on WikiText2 and PTB, and average zero-shot accuracy (over mul- tiple zero-shot datasets). Take LLaMA- 13B on WikiText2 as an example, our method improves the perplexity by 4.15, 1.65, and 0.13 compared to LLMM-runer((), SliceGPT, and FLAP, respectively, with 80% inheriting ratio. Meanwhile, it achieves a higher average accuracy on seven classification datasets than baselines. For instance, under a 80% inheriting ratio, our method on LLaMA-7B im- proves the average accuracy by 2.89% 12.72%. and 1. 1.13% compared to L.LM--rueee(e)) SliceGPT, and FLAP, respectively. As the SliceGPT is sensitive to the calibration dataset, we further show the results with PTB calibration in Appendix C Besides, we ablate the search with PTB in Appendix D
8

---

Table 6:
Results of extra large LLaMA models.
Model
LLaMA-308
LLaMA-65B
Dataset
Ratio
100%
90%
80%
70%
60%
50%
100%
90%
80%
70%
60%
50%
Wikil
FLAP
4.10
4.52
5.18
6.28
8.73
13.41
3.53
3.91
4.45
5.10
6.16
8.11
Ours
4.44
4.94
5.63
6.70
8.01
3.84
4,29
4.80
5.49
6.45
FLAP
17.29
19.30
21.88
29.11
47.30
19.35
21.01
22.45
25.97
33.86
PTB/
16.29
17.61
Ours
17.15
18.80
20.70
24.22
30.51
19.22
20.40
21.39
23.41
30.08
The comparisons other LLM families are demonstrated in Table Vicuna-7B [56] and Table with (a for OPT family [2]. As LLM-Pruner and FLAP do not implement their methods on OPT, we only compare with SliceGPT for OPT models. We can make similar observations that our method performs the best compared with SOTA baselines, demonstrating a superior generalization performance across various datascts/tasks, model structures, and inheriting ratios.
Scaling small inheriting ratios and large models. Furthermore, our method consistently per- forms the best when scaling to larger model sizes or smaller inheriting ratios, indicating the great potential of our method for the iver-incraasing LLM model size. Specifically, we show the results of LLaMA-7B and LLaMA- -1B with lower inheriting ratios and thus more efficient models in Table [ and (b) and (c). Our method consistently performs the best with more significant improve- ments under smaller inheriting ratios such as 50%. Besides, we deliver results large models including LLaMA-30B and LLaMA-65B Tad Figure Our method achieves superior performance than FLAP under various settings, verifying our effectiveness and generalization.
4.3 Ablation Study
We show the results of LLaMA-7B with 128 sequence length in Table 7 Our method per- forms the best across different inheriting ratios, indicating the effectiveness short sequences. Results of LLaMA-13B are in Appendix H+
Table 7:
LLaMA-7B perplexity (L) results on
WikiText2 dataset with 128 sequence length.
Ratio LLM--Puner(cl)
SliceGPT
FLAP
Ours
90%
15.22
14.18
14.15
13.40
19.09
17.08
14.62
14.54
Besides, to verify the influence of the number of examples for the reformation, we conduct ab-
70%
30.63
24.39
17.62
17.11
60%
52.30
40.04
23.53
20.24
50%
106.07
74.09
31.80
26.96
lation studies by varying the number from 128
to 512 and 1024. As shown in Figure O our reformation effectively improves the performance, and it is not sensitive to the number of samples. 128 samples already provide satisfying performance. Furthermore, we investigate the impact the step number and p in the ADMM solution for the reformation, with detailed ablation results presented in Appendix F
Perplexity 40
Memory (GB)
Tokens's 4$
35
128 samples
wio refomm
12
Memory
Tokene's
40
30
sumples
10
25
1024 samples
35
30
25
5
20
Ratio
90%
80%
70%
60%
50%
Ratio 100%
90%
80%
70%
60%
50%
Figure 6:
Ablation analysis for reformation
Figure 7: Analysis for memory and generation
with different numbers of samples.
speed of LLaMA-7B on NVIDÍA A100 40G.
4.4 Generation Acceleration
To demonstrate our acceleration performance, we report the memory consumption and inference speed with our searched LI LLaMA-7B models on NVIDIA A100 40G GPUs across different inheriting ratios. shown in Figure Z we can observe that with a smaller inheriting ratio, our searched efficient model consumes less memory with faster generation speed.

---

Conclusion and Limitation
In this paper, we propose a training-free search framework to find the optimal subnets inside LLMs. We further propose a reformation algorithm that reconstructs the weights of subnets to enhance the task performance. The experiments show the effectiveness our proposed method compared to SOTA structured pruning methods. Additionally, we achieve memory reduction and practical inference acceleration on GPUs, which shows the efficiency of our method. The search cost required by method can increase with the model size, which takes more time for large models.
Acknowledgment
We would like to express our sincere gratitude to Professor Lin and Professor Wang for their invalu- able guidance throughout the development of this paper. We are also deeply grateful to Pu, Yifan, and Zhenglun for their dedicated contributions, thoughtful insights, and collaborative efforts, which were essential to the completion of this research. Mcanwhile, This research is funded in whole or in part by the National Science Foundation ANS-2312158 to Northeastern University. Any errors and opinions are not those the National Science Foundation and are attributable solely to the author(s).
References
[1] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv, 2023.
[2] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained trans- former language models. arXiv,, 2022.
[3] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv, 2022.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 33:1877-1901, 2020.
[5] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Korbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learn- ers. 2020.
[7] Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. arXiv preprint +rXiiv2301100777 2023.
[8] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in Neural Information Processing Systems, 2023.
[9] Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns. The Twelfth International Conference on Learning Representations, 2024.
[10] Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-base adaptive structured pruning for large language models, 2023.
[11] Zheng Zhan, Zhenglun Kong, Yifan Gong, Yushu Wu, Zichong Meng. Hangyu Zheng, Xuan Shen, Stratis loannidis, Wei Niu, Pu Zhao, and Yanzhi Wang. Exploring token pruning in vision state space models. arXiv preprint (tth:2409.119222 2024.
10

---

[12] Xuan Shen. Zhenglun Kong, Changdi Yang, Zhaoyang Han, Lei Lu, Peiyan Dong, et al. Edgeqat: Entropy and distribution guided quantization-awe aware training for the acceleration of lightweight llms on the edge. arXiv preprint arXiv:2402. 10787, 2024.
[13] Changdi Yang, Pu Zhao, Yanyu Li, Wei Niu, Jiexiong Guan, Hao Tang, Minghai Qin, Bin Ren, Xue Lin, and Yanzhi Wang. Pruning parameterization with bi-level optimization for efficient semantic segmentation on the edge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages I 15402-15412, 2023.
[14] Yihua Zhang, Yuguang Yao, Parikshit Ram, Pu Zhao, Tianlong Chen, Mingyi Hong, Yanzhi Wang, and Sijia Liu. Advancing model pruning via bi-level optimization. Advances in Neural Information Processing Systems, 35: 35:18309-1833 18326, 2022.
[15] Yanyu Li, Changdi Yang, Pu Zhao, Geng Yuan, Wei Niu, Jiexiong Guan, Hao Tang, Minghai Qin, Qing Jin, Bin Ren, Xue Lin, and Yanzhi Wang. Towards real-time segmentation on the edge. In Proceedings of the Teirtt-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAT23/AAАTЛМAAАT22 AAAI Press, 2023.
[16] Pu Zhao, Fei Sun, Xuan Shen, Pinrui Yu, Zhenglun Kong, Yanzhi Wang, and Xue Lin. Pruning foundation models for high accuracy without retraining, arXiv preprint arXiv:2410.5557, 2024.
[17] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post- training compression for generative pretrained transformers. arXiv preprint 22. 2022.
[18] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[19] Xuan Shen, Peiyan Dong, Lei Lu, Zhenglun Kong. Zhengang Li, Ming Lin, Chao Wu, and Yanzhi Wang. Agile-quant: Activation
iiivaattonguidedd guided quantization for faster inference of Iims on the edge. Proceedings of the AAAI Conference on Artificial Intelligence. 38(17)11944411851 Mar. 2024.
[20] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint /008.v/99009900 2019.
[21] Siqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, and Jingjing Liu. Contrastive dis- tillation on intermediate representations for language model compression. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference Empir- ical Methods in Natural Language Processing (EMNLP), pages 498-508, Online, November 2020. Association for Computational Linguistics.
[22] Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang, Yaliang Li, and Jun Huang. Meta- KD: A meta knowledge distillation framework for language model compression across do- mains. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Inter- national Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3026-3036, Online, August 2021. Association for Computational Linguistics.
[23] Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neu- ral networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 14. PMLR, 09-15 Jun 2019.
[24] Xuan Shen, Yaohua Wang, Ming Lin, Yilun Huang. Hao Tang, Xiuyu Sun, and Yanzhi Wang. Deepmad: Mathematical architecture design for deep convolutional neural network. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6163-6173 6173, June 2023.
[25] Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: A zero-shot nas for gerooreerfor deep image recognition. In 2 IEEF/CVF International Conference on Computer Vision, ICCV 2021, 2021.
11

---

[26] Chengyue Gong, Dilin Wang, Meng Li, Xinlei Chen, Zhicheng Yan, Yuandong Tian, qiang liu, and Vikas Chandra. NASVit: Neural architecture search for efficient vision transform- ers with gradient conflict aware supernet training. In International Conference on Learning Representations, 2022.
[27] Xiu Su, Shan You. Jiyang Xie, Mingkai Zheng, Fei Wang. Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Vision transformer architecture search. arXiv preprint arXiv:2106. 13 13700, 2021.
[28] Dongning Ma. Pengfei Zhao, and Xun Jiao. Perfhd: Efficient vit architecture performance ranking using hyperdimensional computing. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 2230-2237, June 2023.
[29] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching trans- formers for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12270-12280, October 2021.
[30] Peiyan Dong, Zhenglun Kong, Xin Meng, Pinrui Yu, Yifan Gong, Geng Yuan, Hao Tang, and Yanzhi Wang. Hotbev: Hardware-oriented transformee-based multi-view 3d detector for bev perception. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 2824-2836. Curran Associates, Inc., 2023.
[31] Zheng Zhan, Yifan Gong, Pu Zhao, Geng Yuan, Wei Niu, Yushu Wu, Tianyun Zhang, Malith Jayaweera, David Kaeli, Bin Ren, et al. Achieving on-mobile real-time super-resolution with neural architecture and pruning search. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pages 4821-4831, 2021.
[32] Yushu Wu, Yifan Gong, Pu Zhao, Yanyu Li, Zheng Zhan, Wei Niu, Hao Tang, Minghai Qin, Bin Ren, and Yanzhi Wang. Compiler-aware neural architecture search for on-mobile real-time super-resolution. In European Conference on Computer Vision, pages 92-111. Springer, 2022.
[33] Changdi Yang, Yi Sheng, Peiyan Dong, Zhenglun Kong, Yanyu Li, Pinrui Yu, Lei Yang, Xue Lin, and Yanzhi Wang. Fast and fair medical ai the edge through neural architecture search for hybrid vision models. In 2023 IEEF/ACM International Conference on Computer Aided Design (ICCAD), pages 01-09, IEEE, 2023.
[34] Changdi Yang, Yi Sheng, Peiyan Dong, Zhenglun Kong, Yanyu Li, Pinrui Yu, Lei Yang, and Xue Lin. Late breaking results: Fast fair medical applications? hybrid vision models achieve the fairness the edge. In 2023 60th ACM/IEEE Design Automation Conference (DAC),
pages 1-2. IEEE, 2023.
[35] Peiyan Dong. Zhenglun Kong, Xin Meng, Pinrui Yu, Yifan Gong, Geng Yuan, Hao Tang, and Yanzhi Wang. Hotbev: Hardware-oriented transformer-basee multi-view 3d detector for bev perception. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 2824-2836. Curran Associates, Inc., 2023.
[36] Yanyu Li, Pu Zhao, Geng Yuan, Xue Lin, Yanzhi Wang, and Xin Chen. Pruning- Pruning-as-searchh Ef- ficient neural architecture search via channel pruning and structural reparameterizzation arXiv preprint arXiv:2206.0) 198, 2022.
[37] Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and trends@ in Optimiza- tion, 1(3):127-239) 2014.
[38] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foun- dations and Trends@ in Machine learning, 3(1):1-122, 2011.
[39] Pu Zhao, Sijia Liu, Yanzhi Wang, and Xue Lin. An admm-based universal framework for adversarial attacks on deep neural networks. In Proceedings of the 26th ACM International Conference Mullimedia, MM 18, page 1065-1073, New York, NY, USA, 2018. Association for Computing Machinery.
[40] Yifan Gong, Zheng Zhan, Zhengang Li, Wei Niu, Xiaolong Ma, Wenhao Wang, Bin Ren, Caiwen Ding, Xue Lin, Xiaolin Xu, et al. A preracc-preeervinoooreeentee dan pruning and mobile acceleration framework. In Proceedings of the 2020 on Great Lakes Symposium on VISI, pages 119-124, 2020.
12

---

[41] Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1699,, 2023.
[42] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-61 14. PMLR, 2019.
[43] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[44] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[45] Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 33:18098- 18109, 2020.
[46] Yi-Lun Liao, Sertac Karaman, and Vivienne Sze. Searching for efficient multi-stage vision transformers. arXiv preprint (tti:22199000444 2021.
[47] Tom Goldstein, Brendan O' O'Donoghue, Simon Setzer, and Richard Baraniuk. Fast alternating direction optimization methods. SIAM Journal on Imaging Sciences, (()):158--1223 2014.
[48] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv, 2016.
[49] Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large anno- tated corpus of English: The Penn Treebank. Computational Linguistics, pages 313-330.
[50] Christopher Clark, Kenton Lee, Ming-Wei Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 10044, 2019.
[51] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 7432-7439, 2020.
[52] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905.0788 2019.
[53] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9))99-106 2021.
[54] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint wrXiv:1803.05455 2018
[55] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor con- duct electricity? new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, Brussels, Belgium, Oetober-November 2018. Association for Computational Linguisties.
[56] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng. Siyuan Zhuang. Zhanghao Wu. Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging Ilm-as-a-judge with mt-bench and chatbot arena, 2023.
[57] Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weightedd linear units for neural network function approximation in reinforcement learning. Newral networks, 107:3-11, 2018.
13

---

Appendix
A Efficient Inference
After searching and reformation, we get optimal efficient subnets with the selections masks Sattn E RM and Smip If for each block of the LLMs. detail, for the weights query, key, and value denoted as Wo, WK, Wv E RMXD, we generate the weight subsets by extracting the selected rows from the original weights, which are denoted as W. W W E RmmDD For the
Q.
weights of the output projection Wo RxxM, we extract the columns instead of rows and reform the selected weight based on the omitted ones for the weight subnets W E E. Dx Subsequently, the given input X RBNxD is projected in the self-attention module as follows.
X' Wo (softmamaaaa(Wa
(WxX) J1 (W BBNDD
(10)
For the MLP module in the LLaMA family, we denote the weights of the three linear layers with WU. We E R²xD, and WD E R PxP for the up. gate, and down projections, respectively. The weight subsets generated with the selection mask Smip for three linear layers are W W E R and W D Dxx, where only WD is reformed. Then, the given input X BBNxD projected in the MLP module as follows,
X' (WW X) @ activation (W xi] E BNXD
(11)
where the activation function for the LLaMA family SiLU (77.
Therefore, the computation cost is reduced for both self-attention and MI.P modules, while the configuration of the input X BN is preserved for preventing information loss and maintaining the representation capability of tokens.
B Proof of Theorem 3.1
Problem 6 can be reformulated as follows,
min W.z
|WX - WX 3 + g(Z).
s.t. W
(12)
where g(Z) is a inidicator function as the following.
20, otherwise,
.(Z)
(13)
0, if W O 0.
We can see that Problem is equvilant to Problem
Based on ADMM 37 38
Problem (12) can be solved with ADMM iterations. In the Ath
iteration, it needs to address the following,
W k++ arg min
WX - WX L
zt + U
(14)
W
k
arg min
((Z)
W + U
(15)
z
U W
-
(16)
Problem 12 is split into multiple sub- btb-problems with Problem (44 and 15
Problem 14 is similar to Ridge regression problem. We can directly obtain its solution
W k+ = =XX"" pl) (XxTxx" WT + p U
(17)
To solve Problem 15 we can set Z++1 = (W++1 + U&) and project * on the function as follows,
z ++1
Wh++ + U + M.
(18)
Thus, can obtain the solution in Theorem 3.1.
14

---

Table A1: Compare with SliceGPT using LLaMA-7B perplexity (4) results on PTB dataset.
Calibration Dataset
Method
90%
80%
70%
60%
50%
PTB WikiTex12 WikiTex12
SliceGPT
38.47
43.23
51.38
63.14
118.78
SliceGPT
133.80
143.89
583.58
51.86.66
15333.66
32.05
36.06
45.26
62.07
117.06
Table A2: ILaMA-7B perplexity (4) results on different datasets of search and evaluation.
Datasel
Inheriting Ratio
Search
Eval
90%
80%
70%
60%
50%
Wiki
Wiki
6.10
6.89
8.28
10.21
15.48
Wiki
PTB
32.05
36.06
45.26
62.07
117.06
PTB
Wiki
6.22
7.08
8.41
11.08
17.23
FTB
PTB
31.24
34.87
43.89
59.07
108.83
C SliceGPT Comparison
For SliceGPT [9], the generated results sensitive the calibration datasets, further show the results of SliceGPT with the calibration of PTB training dataset and 2048 sequence length in Table A1 We can know that our method can achieve better performance with the calibration on WikiText2 instead of PTB dataset than the SliceGPT with the calibration on the same dataset.
D Search on PTB Dataset
To further verify the generalization and effectiveness of our method, utilize the training portion of the PTB dataset in our search instead of WikiText2. The results, shown in Table [2 are eval- uated with a sequence length 2048 using the LLaMA-7B model. Our findings reveal that the models generated through searches on two different training datasets achieve similar performance, demonstrating the robustness and generalization capability of our search method across different datasets.
E
Ablation for 128 Sequence Length
We also present the results for the LLaMA-133 model with a sequence length of 128 in Table A3 demonstrating that our method continues to achieve superior performance. The results are evaluated with on Wiki'Text2 dataset.
Perplexity 8.5
Perplexity
7.25
7.2
8
7.15
7.5
7.1
7.05
7
6.95
6.5
6.9
6.85
Steps
0
5
10
20
30
50
P
Ablation analysis for reformation with different numbers of steps.
Figure A1:
Figure A2:
Ablation analysis for reformation
with different p.
15

---

Table A3: LLaMA-13B perplexity (4) results on WikiTex12 dataset with 128 sequence length.
Ratio
L..M-Pruner(el))
SliceGPT
FLAP
Ours
90%
13.23
12.68
12.25
12.18
80%
16.01
15.66
13.66
13.25
70%
21.85
21.44
15.65
15.17
60%
31.17
32.77
18.53
18.14
50%
236.24
52.92
24.20
22.65
F Ablation in Reformation
To explore the influence of the number of iterations on the reformation process, we conducted ex- periments with varying iteration counts, shown in Figure [A] The results, evaluated using the LLaMA-7B model with an 80% inheritance ratio on the WikiText2 dataset with a sequence length of 2048, indicate that model performance improves with increasing number of iterations, peaking around 30 steps. Beyond this point, there is minimal difference between 30 and 50 iterations. Addi- tionally, examined the impact of different p values the reformation, depicted in Figure A2 Our findings show that for p E [0.01, 0.1]. the reformed model's performance remains increasing, but deteriorates when p reaches 10 bigger. Hence, p becomes the optimal value.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We explain method and summarize the contribution in introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation included conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
We provide the theorem in methodology section and the proof in appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the detailed experiment setup in experiment section.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer:
[NA]
Justification:
The dataset and model we used is open-source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see
the NeurlPS code and data submission guidelines for more de-
(https://nips... tails.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https://nips cate.cc/pubbl www.cppwlcgggcceeeeeeeeeeeeeeeee
for more details. The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide detailed experiment setting in experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropri- ate information about the statistical significance of the experiments?
Answer: [NA]
Justification:
We did not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We explain the computation resources experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Research is conducted in the paper conform with NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss them in conclusion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We discussed in introduction.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We originally implemented method.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the pack- age should be provided. For popular datasets, pat.ssspppssiiiiee thcode com/ datasses has curated licenses for some datasets. Their licensing guide can help determine the cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification:
We introduced in the introduction.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: There is nothing related to human subjects in our work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There is nothing related to human subjects in our work.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "RGpKyfL5TQ", "reviewer": "Reviewer_n31g", "summary": "This paper proposes a technique for searching for efficient LLM subnets for fast inference, while still attaining strong performance. The proposed technique involve a training-free pruning stage based on a genetic algorithm, followed by a ``weight rectification'' stage that improves the resulting subnet. Empirically, the proposed method outperforms existing LLM pruning baselines.", "questions": "- The authors mention in the limitations that the search cost scales with the model size. Can the authors characterize thi\ns a bit more? Do other pruning methods suffer from the same problem, and can the authors comment on what this scaling curv\ne actually looks like? \n- Why was the search budget set to 5 hours? Have the authors explored this choice empirically, and does this apply to all of the models and data settings that were evaluated?", "limitations": "The authors adequately address limitations in the conclusion.", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- Empirically, the proposed pruning technique appears to outperform a handful of existing pruning baselines, as shown in Figure 1 and Table 2. \n- The proposed technique appears to be general enough to work for a wide variety of existing LLMs, which appears to be a s\nhortcoming of existing LLM pruning techniques. \n- The proposed method is fairly simple in that it combines LLM pruning with a genetic algorithm for search.", "weaknesses": "- From the abstract, it is unclear what the actual proposed method does beyond standard neural network pruning techniques. \n- The term ``reformation algorithm'' is used throughout the paper, and it is unclear what this means until much later in t\nhe text. The intro and abstract should clarify this. \n- The main evaluation is done in terms of sparsity levels, which makes sense for this particular type of search space, how\never, the authors should also empirically validate the efficiency of the search process compared to existing pruning techn\niques. E.g., how do these pruning techniques compare, subject to the same budget? Moreover, it would be interesting to see how the proposed method performs as a function of the search budget.                                       \n- The authors should compare to other techniques beyond pruning, such as quantization which may achieve similar inference speed and memory improvements with better overall performance. In general, it seems that LLM pruning techniques take a sub\nstantial performance hit even at fairly modest sparsity levels (I'm mainly referring to Table 7).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the suggestions from the reviewer.\n\n### Weakness 1. Difference from network pruning\n\nOur proposed search method significantly differs from standard neural network pruning techniques. Specifically, our method includes a search initialization process to identify the effective initial architecture and an evolution architecture search with special mask mutation and efficient candidate evaluation, which are not typical in standard weight pruning techniques. We are the first to incorporate the architecture search technique for LLMs to find efficient architectures. There are no other search baselines and thus we mainly compare with pruning works. Although our method significantly differs from traditional pruning, both our search and pruning can lead to efficient lightweight LLMs. \n\n### Weakness 2. Reformation\n\nThanks for the suggestion, we will explain more about the reformation in the abstract and introduction.\n\n### Weakness 3. Search cost\n\nWe highlight that our search cost is just similar to the baselines. Details can be found in the above global rebuttal.\n\n### 3.a. Performance as a function of search cost.\n\nFigure 5 clearly demonstrates how the proposed method performs as a function of the search budget (number of epochs and thus search hours). As we discussed in the global rebuttal, we can reduce the search cost without significantly sacrificing the performance.\nNote that, with 20 epochs, for LLaMA-7B, we can still achieve  6.23 and 7.21 perplexity on WikiText2 under 90\\% and 80\\% inheriting ratios, respectively, which are better than all baselines shown in Table 2.\n\n### 3.b. Lower cost for memory and data in the search\n\nBesides the training time on GPU to finish the search, our method achieves a better performance with less memory and data. (a) Compared to LLM-Pruner, which identifies weight importance using backward passes, our method relies solely on inference. This significantly reduces GPU memory consumption, enhancing efficiency. (b) Compared to the SliceGPT, it requires 1024 data samples for calibration. Our method requires significantly fewer calibration samples (128) for weight optimization after the search, leading to better performance on various datasets.\n\n### Weakness 4. Quantization\n\nQuantization methods are not direct competitors but rather complementary to our search approach. Existing quantization methods can be applied to the subnets identified by our method, further enhancing acceleration through integer matrix multiplications. In practice, quantization methods require specialized computation kernels to handle integer matrix multiplications, which can introduce additional computational overhead and complexity in implementation, particularly during inference. This can pose challenges when deploying quantized models on diverse hardware platforms which may not support the quantized model. Our approach, on the other hand, maintains the original computation framework of LLMs, without the need for additional operations or specialized implementations.\n\nRegarding the performance hit, we would like to clarify that the results in Table 7 were generated with a sequence length of 128, rather than 2048, for sequence length ablation study. Given that the dense LLaMA-7B model achieves the perplexity of 12.62 at 128 sequence length, our method, with a 90\\% ratio, results in the perplexity of 13.40. This indicates a relatively small performance gap. With a larger sequence length such as 2048, the performance of all methods can be better and we can still achieve the best performance as shown in Table 2 and Table 3. \n\n\n\n### Question 1. Search cost and model size\n\nWe discuss the search cost in detail at the global rebuttal. For LLaMA-7B, we still can achieve better performance than other methods when searching only with 20 epochs within 2 hours. For LLaMA-65B, we only adopt 20 epochs to get the subnets, which takes 5 hours.\n\n### Question 2. Search cost with 5 hours\n\nAs we mentioned in global rebuttal, for LLaMA-7B on one A100 GPU, it takes 5 hours for 50 epochs during the search process. The search already converges at 20 epochs as demonstrated in Figure 5, which only takes 2 hours and already achieves better performance than other methods. Besides, for large models such as LLaMA-65B, we only adopt 20 epochs for faster search, which takes just 5 hours.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qdjOpy06zP", "reviewer": "Reviewer_Qp2U", "summary": "The paper introduces a new model architecture search method for LLMs that does not require additional training. The proposed architectures have structural sparsity and reach better performance than SOTA pruning baselines.", "questions": "Why isn't there a comparison to SparseGPT?", "limitations": "The authors discuss limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method does not require additional training. \n\n- The output model has structural sparsity, which is useful for hardware. \n\n- The paper is clear and well-written.", "weaknesses": "- Why isn't there a comparison to SparseGPT?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the suggestions from the reviewer.\n\n### Weakness & Question: Comparison with SparseGPT\n\nThe pruning or search granularity of our method and SparseGPT are different, thus they should not be directly compared. Our method searches smaller compact models, which is more like the structural pruning method to remove rows and columns of weight matrices. Different from our structure search, SparseGPT is an irregular pruning method to remove weights without strong constraints (such as, the removed weights should be in the same rows or columns). Thus in the experiments, we compare with other structure pruning methods such as SliceGPT and FLAP, without comparing to SpareGPT. Furthermore, the baseline SliceGPT already provides certain comparisons with SparseGPT, as shown in Table 1 of SliceGPT. As mentioned above, it is hard to make a fair comparison between structure and irregular pruning. In Table 1 of SliceGPT, the sparsity of SparseGPT is 50\\%, while the sparsity of SliceGPT is 25\\% or 30\\%. The results in their Table 1 can serve as a reference, and our method can outperform SliceGPT non-marginally.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VkXsYdH8Z6", "reviewer": "Reviewer_Xurt", "summary": "This paper introduces an architecture search method based on mask mutation and candidate evaluation to find a subnet with better performance in the LLM. An evolution-based algorithm is applied to globally search the subset with a special initialization from the evaluation of parameter importance. After the search from LLM, a reformation algorithm is proposed to rectify the weights in the inherited LLM. Experimental results show that the compressed LLM can achieve better results compared to the baselines.", "questions": "1. For Mask Sharing: why does the same module in different modules share the same mask (Line 141)? In line 147 - 149, from my understanding, the mask is not the same for blocks in different layers. Using different masks for different layers seems to still have the aligned internal computations. I’m not sure if I understand correctly in this part.\n2. What about the results if you post-train the compressed models? What are the results if compared with LLM-Pruner?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A novel method for searching a subnet in the LLM. The search of subnet in LLM is not well-explored, and it would benefit the community.\n2. The overall approach is reasonable and sufficient. The entire method’s pipeline is very clear, and the design for each component is reasonable and complete.\n3. The experimental results are comprehensive and show its benefits compared to the baselines (SliceGPT and LLM-Pruner). A large and steady improvement is observed. \n4. The paper is well-organized and easy to follow.", "weaknesses": "1. My main concern is about the extra cost of searching the subnetworks. Since it still takes around 5 hours to search for the architecture, it introduces extra cost for pruning the model. What if we take these 5 hours to post-train the compressed model? Would the results be better or worse than the subnetwork search from the LLMs?\n2. Since the training-free structured pruning is not a very useful setting, post-training the LLM with an acceptable amount of resources is more realistic. Thus, compared to just pruning and searching the mode, a more realistic setting is also to train the pruned model. And I’m not sure whether the searched sub-LLM would be better than the pruned one. Some previous work to prune CNNs suggest that with sufficient training, magnitude pruning would be a better choice than any manually designed metrics. Thus, it would be better to examine the performance of the compressed LLM with post-training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the suggestions from the reviewer.\n\n### Weakness 1. Search cost\n\nAs we discussed in the global rebuttal, we can reduce the search epoch number from 50 to 20 for LLaMA-7B, which costs 2 hours and can still achieve better performance than other methods. For LLaMA-65B model, we only adopt 20 epochs for search, which costs 5 hours. Besides, compare to other methods, our search cost is similar to those baselines.\n\n\n### Weakness 2. Post training performance\n\nTo improve the performance, we adopt an efficient reformation method in Section 3.4 without the need of expensive training or finetuning. As shown in our Figure 6, our reformation can significantly improve the perplexity within a few minutes. We highlight that without training or finetuning, our method already outperforms baselines which are based on recovery training, such as LLM-Pruner and SliceGPT, as shown in our experimental results and Table 8 provided in global rebuttal.\n\nTo further evaluate our performance with continual training, we train our searched compact models with LoRA following the same setting of recovery training in LLM-Pruner, which can be finished within another 3 hours for LLaMA-7B. The results are demonstrated in Table 8 provided in global rebuttal. With recovery training, our method can further improve the perplexity and accuracy performance. For more sparse models (50\\% inheriting ratio) suffering from more significant accuracy loss, recovery training can lead to larger improvements.  \n\nRecovery training can further improve the accuracy, requiring more data and computations. Different from recovery training,  our reformation can efficiently and effectively lead to better performance than the baselines, with little data and computations. We agree that it is an attractive topic to investigate the pruned model performance with sufficient training. But for LLMs, it is still an open question how to define sufficient training, in terms of model architecture, data quality, data amount, GPUs, and so on. We leave this as our future research.\n\n\n### Question 1. Mask sharing\n\nThe weights in the same module of each block share the same mask, but different blocks do not share masks. Each block has its own masks, which are not shared with other blocks. The reviewer’s understanding is correct.  We discuss mask sharing in detail in the global rebuttal. Line 141 shows that there are two masks for each block, meaning there will be $2k$ masks for $k$ blocks, as the masks of different blocks are not shared.  \n\n\n### Question 2. Fine-tuned compressed model compared to LLM-Pruner\n\nWe show the results compared to the fine-tuned LLM-Pruner in Table 8 provided in the global rebuttal. Results show that our method can perform better than LLM-Pruner even without finetuning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "238NMz2Thg", "reviewer": "Reviewer_BTSA", "summary": "This paper introduces a neural architecture search method for Large Language Models (LLMs) comprising three steps: inheriting the most salient weights from the original model to form the initial sub-network, using an evolutionary algorithm to search for the optimal sub-network, and reconstructing the original network’s output with calibration samples.", "questions": "In addition, here are some points about writing: \n\n5. The term \"inheriting ratio\" needs to be rigorously defined in an early section. In addition, authors would better add a sentence describing the difference between \"inheriting ratio\" and \"sparsity\" / \"pruning ratio\". In my understanding, \"inheriting ratio\" = 1 - \"sparsity\".\n6. Figure 2 Left: it is better to add an annotation showing which axis is the layer index. I assume it is the horizontal axis.\n7. Section 3.2, Lines 139-149: The writing of this part needs to be improved. It gives me a lot of confusion when I read this part.\n\n    (i) Please define what $M$ and $P$ are in Line 140.\n\n    (ii) Are the masks smooth or binary? If binary then $\\{0, 1\\}$ should be used instead of $\\mathbb{R}$ in Line 140.\n\n    (iii) Line 141, why do different layers of Attn / MLP share the same mask? If so, would the sparsity of each layer be the same? However, according to Figure 1, the sparsity should be different across layers. Also, in Line 146, authors \"set the same inheriting ratios for the masks in all building blocks\". So, these claims and figures are very chaotic. Additionally, what does \"align the internal computations\" mean?\n\n    (iv) Line 144, Are Eq. (2) and (3) jointly optimized (e.g., take the sum) or separately? Are the masks of different layers jointly optimized or separately?\n8. Section 3.3.1, Line 165: $\\gamma$ should be rigorously defined.", "limitations": "1. Section 5: The limitations provided by the authors are too trivial. It is obvious that larger models require more time for pruning/searching. Please add more in-depth discussions on limitations. Authors can refer to the points listed in the \"Weaknesses\" section for more substantial issues that need addressing.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Overall, I like the application of evolutionary algorithm to pruning problem.\n1. The method overcomes the limitations of uniform sparsity across all layers—a common but sub-optimal constraint in previous structured pruning methods.\n2. It avoids reliance on back-propagation, enhancing its applicability to larger models within a constrained memory budget.", "weaknesses": "Here are some technical concerns:\n\n1. The candidate evaluation is conducted based on perplexity on WikiText2. This is one metric on one task, then the searching algorithm might be biased towards the task used for candidate evaluation, resulting in a model favoring one task (e.g., language modeling) over others. How to ensure the performance of the searched model on other tasks?\n2. Based on my experience, using perplexity as a metric for selecting pruning candidates can result in good performance on WikiText but sub-optimal performance on the MMLU dataset. It would be great if authors could validate their pruned (or searched) model on MMLU.\n3. Table 5. The performances on QA tasks should also be presented. If the space is not enough, authors can put them in the Appendix and link it in the main text.\n4. Figure 7. Tokens/s is not a very rigorous measurement for speed. The generated tokens per second can depend on many factors, e.g., floating point precision, KV Cache, flash attention, batch size, etc. I suggest authors use other metrics such as MACs or at least provide more specifications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the suggestions from the reviewer.\n\n### Weakness 1. Potentially biased evaluation\n\nTo mitigate the potential biased evaluation problem, we do not use the data from specific downstream tasks (QA datasets such as MMLU or ARC). Instead,  for our candidate evaluation and reformation, we use the general dataset WikiText2, which represents generic verified and featured articles on Wikipedia and makes sure that our experiments remain actually zero-shot since no task-specific downstream data is seen during our search. As the model never sees any downstream data  (so that it does not know how to be biased), it incurs less biased evaluation for downstream tasks. \nThis setting is consistent with previous works such as FLAP and SliceGPT. \n\nTo ensure a fair comparison, we use the same calibration dataset (i.e. WikiText2) for all methods including the baselines. We believe our evaluation is fair and comprehensive. Our method can achieve a better performance on language modeling and QA datasets. Furthermore,  FLAP investigates the performance with different calibration datasets (see Appendix C.3 and  Table 7 of FLAP). It observes that C4 or WikiText2 as the calibration data, results in a fluctuation of about ±1\\% in average accuracy for zero-shot tasks, which is not significant. \n\n### Weakness 2. MMLU performance\n\nTo make a fair comparison, we follow the setting in existing works (such as FLAP and SliceGPT) to use the data from WikiText2 for the candidate evaluation. Besides, it is super efficient to compute the perplexity of a few samples compared with testing the accuracy on the whole test set.\n\nFollowing the suggestion, we demonstrate the performance on MMLU dataset for our method and the baseline FLAP in Table 9 in the global rebuttal. Our method can outperform the baseline in terms of accuracy on MMLU. \n\nAs we mentioned in our response to Weakness 1, we use the general dataset WikiText2 for candidate evaluation, which represents generic text data from Wikipedia and makes sure that our experiments remain actually zero-shot since no task-specific downstream data is seen during our search. Thus, our model is less biased for different downstream tasks. \n\n### Weakness 3. QA performance in Table 5\n\nFollowing the suggestion, we further provide the results on the common sense reasoning datasets for the 50\\% inheriting ratio in Table 8 provided in the global rebuttal. As observed, our method can lead to lower perplexity and higher downstream evaluation accuracy performance. \n\n### Weakness 4. Speed evaluation\n\nThanks for the suggestion. For the generation speed test, we feed sentences consisting of 64 tokens into the model and adopt float16 with KV cache. The flash attention is not activated. Our results in Figure 7 demonstrate that our smaller models lead to practical inference acceleration. We agree that the inference speed may be different under different evaluation settings. Following the suggestion, we further show the computation cost in GMACs in Table 10 below.\n\n**Table 10**\n\n| Inheriting Ratio        | 100%  | 90%   | 80%   | 70%   | 60%   | 50%   |\n|-------------------------|-------|-------|-------|-------|-------|-------|\n| Computation Cost (GMACs)| 424.02| 377.89| 333.49| 293.29| 249.28| 211.88|\n\n### Question 1. Writing issues\n\nYes, the reviewer’s understanding is correct, and \"inheriting ratio\" = 1 - \"sparsity\". We provide an explanation for the inheriting ratio in Line 116-117. \n\n\n### Question 2. Annotation\n\nThanks, we will add the annotation.\n\n\n### Question 3. Clarification\n\nM denotes the number of heads in attention, P denotes the number of channels in MLP (intermediate size). Masks are binary, it should be {0,1} rather than R.  We will make it more clear in the revision\n\n### 3.a. Mask sharing\n\nThe mask is shared for the query, key, value, and output projection within a single self-attention module, as well as for the up, gate, and down within a single MLP module, rather than being shared across different blocks. We discuss the mask sharing in detail in the above global rebuttal.\n\n### 3.b. Align internal computations\n\nAs discussed in Line 141-142, for multiple layers inside each module (self-attention or MLP), they have to use the same mask to ensure the aligned internal computations. For example, if the query and key have different masks and thus different/unaligned dimensions, they can not be multiplied elementwise and  the attention operation can not be correctly performed. In Appendix A, we demonstrate the mechanism for the self-attention and MLP modules, highlighting that the layers inside the same module need to share the same mask to ensure successful computations. But layers in different modules or blocks do not need to share the same mask. We are sorry for any confusion and will make this more clear in the revision. \n\n### 3.c. Joint optimization of masks\n\nThey are jointly optimized to determine the initial masks as discussed in Line 147-149. Later during our search, all masks are also jointly optimized to satisfy the parameter count constraint.\n\n\n### Question 4. Gamma\n\n$\\gamma_{attn}^i$ denotes the inheriting ratio for the $i^{th}$ head in attention. Each attention module can have multiple heads. Thus the inheriting ratio for the attention module is a collection from multiple heads as the inheriting ratio for each head can be different. $\\gamma_{mlp}$ denotes the inheriting ratio of the MLP module. It does not have the concepts of multiple heads like the attention module and we just have one single ratio for the whole MLP.\n\n### Limitation: Search cost\n\nAs we discussed in the global rebuttal, the search cost can be reduced to 20 epochs within 2 hours and is similar to those baselines.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for acknowledging that our work overcomes prior research limitations and benefits the community (Reviewer Xurt, BSTA), our method is novel, general, and high-performing (Reviewer n31G, Qp2U, Xurt, BSTA), our experiments are comprehensive (Reviewer Xurt), and our paper is well-written (Reviewer Qp2U, Xurt).

### 1. Search Cost

We can reduce our search cost with similar performance. 
For the search cost, it usually takes only 20 epochs for the convergence, and more epochs beyond 20 lead to marginal improvements (see Figure 5). For LLaMA-7B, we run 50 epochs (5 hours) for better results. When searching for 20 epochs, the perplexity for LLaMA-7B with 10\% and 20\% sparsity on WikiText2 are 6.23 and 7.21, which are still better than all baselines in Table 2. Besides, for large models such as LLaMA-65B, we only adopt 20 epochs for faster search, which takes 5 hours. 

Besides, we highlight that our search cost is just similar to the baselines.

For LLM-Pruner, it is very difficult to scale to larger models such as LLaMA-30B and LLaMA-65B, as it requires backward propagation to compute the gradients for identifying the weight importance, incurring much larger memory and computation cost. It only has the results for 7B models (such as LLaMA-7B, and Vicuna-7B) and LLaMA-13B. The 7B models require 3 hours to finish the compression.

For SliceGPT, it requires 5 hours to finish the pruning and fine-tuning of LLaMA-2 70B, which is similar to ours with 5 hours of search for LLaMA-65B. SliceGPT relies on PCA (Principal Component Analysis) to identify and remove less significant weights, which computes the eigenvectors and eigenvalues of the covariance matrix of the signal matrix. For large models, this step is computationally demanding and time-consuming because it requires processing a substantial amount of data to compute these matrices accurately. Besides, this work notes that using double precision for eigenvector calculations in PCA is necessary to avoid numerical errors that can degrade performance. This choice further increases the computational load and time required for pruning. It is data-hungry and requires to use much more calibration samples (1024) while our method only uses 128 samples.



### 2. Mask Sharing

As discussed in Line 124-128, the model has multiple blocks (i.e., layers of LLaMA model). Each block has two modules, the self-attention module and MLP module. Each module contains multiple weights, such as the query, key, value, output projection in the self-attention module and up, gate, down in MLP module (Line 124-128). Mask sharing means that different weights in the same module share the same mask. However, modules of different blocks do not necessarily share the same mask. Each block has two masks corresponding to its self-attention module and MLP module, respectively. Different blocks do not share masks, and they have their own masks, which are not necessarily the same. Just as Reviewer BTSA mentioned, Figure 2 middle shows that different blocks can have different masks/sparsity.

Initially, different masks have the same inheriting ratios. But the masks can still be different although their sparsity are the same. After we begin our search, the inheriting ratios of different masks can be different, although they start from the same value.



### 3. Additional Results

We further provide more results with lower inheriting ratios and fine-tuning in Table 8 as follows,

**Table 8**
| Method           | Inheriting Ratio | Wiki PPL↓ | PTB PPL↓ | BoolQ | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Average Acc.↑ |
|------------------|------------------|-----------|----------|-------|------|-----------|------------|-------|-------|------|----------------|
| LLM-Pruner       | 80%              | 11.97     | 55.68    | 59.39 | 75.57| 65.34     | 61.33      | 59.18 | 37.12 | 39.80| 56.82          |
| LLM-Pruner (LoRA)| 80%              | 8.41      | 41.78    | 67.78 | 76.36| 68.14     | 64.32      | 63.41 | 36.87 | 40.10| 59.57          |
| Ours             | 80%              | **6.89**  | **36.06**| 70.98 | 74.92| 67.29     | 64.64      | 64.23 | 36.52 | 39.40| **59.71**      |
| Ours (LoRA)      | 80%              | 6.57      | 34.32    | 71.19 | 75.01| 68.43     | 64.12      | 63.88 | 35.44 | 40.03| 59.73          |
| LLM-Pruner       | 50%              | 126.02    | 460.71   | 51.97 | 60.23| 35.89     | 49.07      | 32.76 | 25.67 | 34.91| 41.50          |
| LLM-Pruner (LoRA)| 50%              | 51.56     | 198.76   | 60.98 | 69.12| 47.83     | 55.78      | 46.92 | 28.56 | 35.64| 49.26          |
| Ours             | 50%              | **15.48** | **117.06**| 61.27 | 69.13| 51.18     | 57.78      | 52.16 | 29.34 | 35.45| **50.79**      |
| Ours (LoRA)      | 50%              | 11.34     | 83.48    | 63.49 | 73.03| 56.93     | 59.21      | 48.87 | 36.12 | 36.23| 53.41          |




We also provide the results of our method on MMLU dataset compared to FLAP in Table 9 as follows,

**Table 9**
| Inheriting Ratios | LLaMA-7B FLAP | LLaMA-7B Ours | LLaMA-13B FLAP | LLaMA-13B Ours | LLaMA-30B FLAP | LLaMA-30B Ours |
|-------------------|---------------|---------------|----------------|----------------|----------------|----------------|
| 100%              | 34.9          | 34.9          | 46.9           | 46.9           | 58.2           | 58.2           |
| 90%               | 31.3          | **31.6**      | 40.3           | **40.3**       | 52.8           | **52.9**       |
| 80%               | 29.1          | **29.2**      | 35.7           | **36.2**       | 47.3           | **47.6**       |
| 70%               | 26.8          | **27.3**      | 33.4           | **33.7**       | 41.5           | **42.4**       |
| 60%               | 25.9          | **26.3**      | 28.4           | **29.5**       | 36.1           | **37.4**       |
| 50%               | 24.2          | **25.1**      | 27.9           | **29.1**       | 34.5           | **35.1**       |

Meta-review: This paper presents a neural architecture search method for Large Language Models (LLMs) pruning (or finding the optimal subnets). The core idea is based on the salient weights, the evolutionary algorithm  for each and the reconstruction for calibration.

Overall the reviewers find the proposed method to be reasonable and simple, while the empirical gain is interesting as well. While there has been concerns around the baseline comparisons and other experimental details, the authors have provided the rebuttal which has addressed many of them. After reading the authors rebuttal, this feels like a borderline paper but I would highly encourage the authors to include the additional results into revisions.
Decision: Accept (poster)