Title: Search for Efficient Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Architecture Search

Large Language Models
Abstract: Large Language Models (LLMs) have long held sway in the realms of artificial intelligence research.
Numerous efficient techniques, including weight pruning, quantization, and distillation, have been embraced to compress LLMs, targeting memory reduction and inference acceleration, which underscore the redundancy  in LLMs.
However, most model compression techniques concentrate on weight optimization, overlooking the exploration of optimal architectures.
Besides, traditional architecture search methods, limited by the elevated complexity with extensive parameters, struggle to demonstrate their effectiveness on LLMs.
In this paper, we propose a training-free architecture search framework to identify optimal subnets that preserve the fundamental strengths of the original LLMs while achieving inference acceleration.
Furthermore, after generating subnets that inherit specific weights from the original LLMs, we introduce a reformation algorithm that utilizes the omitted weights to rectify the inherited weights  with a small amount of calibration data.
Compared with SOTA training-free structured pruning works that can generate smaller networks, our method demonstrates superior performance across standard benchmarks.
Furthermore, our generated subnets can directly reduce the usage of GPU memory and achieve inference acceleration.

Search for Efficient Large Language Models

Abstract

Large Language Models (LLMs) have long held sway in the realms of artificial intelligence research. Numerous efficient techniques, including weight pruning, quantization, and distillation, have been embraced to compress LLMs, targeting memory reduction and inference acceleration, which underscore the redundancy in LLMs. However, most model compression techniques concentrate on weight optimization, overlooking the exploration of optimal architectures. Besides, traditional architecture search methods, limited by the elevated complexity with extensive parameters, struggle to demonstrate their effectiveness on LLMs. In this paper, we propose a training-free architecture search framework to identify optimal subnets that preserve the fundamental strengths of the original LLMs while achieving inference acceleration. Furthermore, after generating subnets that inherit specific weights from the original LLMs, we introduce a reformation algorithm that utilizes the omitted weights to rectify the inherited weights with a small amount of calibration data. Compared with SOTA training-free structured pruning works that can generate smaller networks, our method demonstrates superior performance across standard benchmarks. Furthermore, our generated subnets can directly reduce the usage of GPU memory and achieve inference acceleration. Code: https://github.com/shawnricecake/search-llm

1 Introduction

Large Language Models (LLMs) [1, 2, 3, 4, 5, 6] are renowned for their exceptional performance across various domains of artificial intelligence research. There is a growing demand for constructing LLMs for extensive applications across a multitude of popular platforms. However, the computational and storage costs have restricted LLMs from deployment on various devices for wide applications. Take the GPT-3 model as an example, with its 175 billion parameters [6], it requires more than 326GB of memory in FP16 format. This exceeds the memory capabilities of even the most sophisticated GPUs, far surpassing available memory on resource-constrained devices. To address these challenges, a variety of compression techniques focusing on weight optimization have been developed, including weight pruning [7, 8, 8, 9, 10, 11, 12, 13, 14, 15, 16], quantization [17, 18, 19], and knowledge distillation [20, 21, 22]. The extensive research in the compression direction indicates the substantial redundancy within LLMs.

Besides optimizing model weights, improving the model architecture is another crucial direction in achieving both high efficiency and superior performance. Numerous works [23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36] have studied the Neural Architecture Search (NAS) problem for representative model designs such as Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). However, the realm of architecture search for LLMs remains unexplored. Though enjoying the potential benefits of discovering highly efficient and well-performing LLM architectures compared with manual designs, searching with traditional NAS methods for LLMs faces significant challenges due to the immense complexity and extensive model size. Furthermore, the convergence

!"

!#

$"

$#

%"

%#

&"

&#

%

'

!%

!'

$%

$'

%

'

!%

!'

$%

$'

%

#

(

)

!!

!%

!#

*+,-./01 2345

2016!$#7

2016!8%9

2016$8(9

!"":

)":

'":

(":

!"#$%

&'()*'+$#,

!"":

)":

'":

(":

!"#$%

;":

#":

&'()*'+$#,

&'()*'+$#,

&'()*'+$#,

!"":

)":

'":

(":

!"#$%

;":

#":

!"":

)":

'":

(":

!"#$%

;":

#":

<<76043=.4

*+,-./01

><?0

2345

><?0

2345

!"#$%&'$(")*+,

!-#$.."/0123

!4#$.."/01563

!7#$.."/01683$9$:;3

%"9

;#9

<<76043=.4

*+,-./01

><?0

2345

of a randomly initialized architecture to the searched optimal state takes substantial training efforts and resources, intensifying the challenges of searching for efficient LLM architectures.

To tackle the challenges, we propose a training-free architecture search framework that discovers efficient LLM architectures within the original well-trained LLMs. Specifically, to reduce the search cost, we first identify an appropriate initial architecture by computing the importance of weights. Subsequently, an evolution-based algorithm is applied to globally search an efficient subnet starting from the initial subnet. In each generation, mutation and crossover are adopted to generate candidate architectures within the search space. The candidates are evaluated efficiently with a small amount of training samples to assess their effectiveness and select for the next generation. As we start our search from a well-trained LLM instead of randomly initialized models, we propose a mask mutation algorithm to identify the detailed channel indices, rather than just the number of channels in the mutation of traditional NAS [29, 24, 25, 26, 32]. After a few generations with the identified promising LLM architecture, we adopt the reformation algorithm based on the alternating direction method of multipliers (ADMM) [37, 38, 39, 40] to rectify weights in inherited efficient LLM architecture with omitted weights (i.e., non-inherited weights) by leveraging only 128 calibration samples.

As shown in Figure 1, our extensive experiments demonstrate that our method can achieve superior performance than SOTA structured pruning baselines in terms of perplexity and zero-shot accuracy on multiple datasets across various LLM families and model sizes. Particularly, as in Figure 1 (a), only SliceGPT and our method support the OPT model family, and our method outperforms SliceGPT. Additionally, with a 60% inheriting ratio for the LLaMA-7B model on the WikiText2 dataset, our method achieves the best performance with a perplexity of 10.21, compared to 38.27 by LLM-Pruner and 279.52 by SliceGPT, as illustrated in Figure 1 (b). Furthermore, when scaling to LLaMA-13B, both SliceGPT and LLM-Pruner fail, as in Figure 1 (c). Lastly, as in Figure 1 (d), only FLAP and our method support the LLaMA-30B and 65B models, and our method achieves better performance than FLAP. Besides, our implementations on GPUs demonstrate significant memory reduction and inference acceleration. Meanwhile, our approach eliminates the retraining process, relying solely on forward pass for both searching and reformation processes, which maintains a low memory overhead.

Our contributions are summarized below,

We propose a training-free search framework to identify subnets within LLMs, featuring an importance-aware initialization that significantly reduces the time cost of searching, and an evolution architecture search with special mask mutation and efficient candidate evaluation.

We propose a reformation algorithm that reconstructs weights by calibrating with only 128 training samples, thereby enhancing the effectiveness of the subnets.

Experiments indicate that the subnets generated by our method outperform SOTA structured pruning works in terms of perplexity and accuracy on multiple datasets across various LLM families and sizes. The searched subnets can effectively reduce GPU memory and accelerate inference.

2 Related Work

2.1 Compression of LLMs

Various compression techniques have been developed to reduce the model size or inference cost of LLMs, including model pruning, quantization, and distillation. Among these, quantization and

!"#$%&'

!"#$%&'&()&*('&+%

($&)#'*!+,-%'."/$

0%&)&".1*($&)#'

!"&'&.1*2%/#&'$/'3%$

4+5$66&")

!

,"#-.(/01

71-/89

:;*<$.6

=$>-%+

0%&)&".1*($&)#'

=$>-%+$6*($&)#'

2"#3.4+/5('&+%

structured pruning methods are prevalent due to their efficacy in inference acceleration while preserving task performance. Quantization approaches, as explored in works [19, 18, 17], compress models by converting weights to lower bit representations. Besides, structured pruning techniques, including the works [8, 9, 10, 41], remove redundant weights in a structured manner to reduce the total weight count. Specifically, LLM-Pruner [8] eliminates non-critical coupled structures based on gradient information, while SliceGPT [9] substitutes each weight matrix with a smaller, dense matrix and reduces the embedding dimension of the network. FLAP [10] employs structured metrics to prune LLMs and globally optimizes the sparsity ratio with the output feature maps. Despite the advancements, most pruning methods indiscriminately remove heads within the self-attention modules, leading to more significant performance loss due to the inherent input-dependent nature of transformer architectures based on all heads.

2.2 Search for Transformers

NAS has emerged as a pivotal technique for identifying efficient architectures in CNNs (exemplified by EfficientNet [42]) and transformer-based models (such as BERT [43] and Vision Transformer [44]). To mitigate the typical high training costs associated with NAS, innovations such as one-shot and zero-shot NAS methods [26, 29, 25, 24] have been developed, enhancing the efficiency of generating high-performance architectures. In contrast to zero-shot NAS methods, which utilize accuracy predictors to derive optimal architectures, one-shot NAS methods streamline the process by pretraining a comprehensive supernet from which optimal subnets are subsequently selected. Specifically, in the context of transformer-based models, the one-shot NAS approach, as implemented in AutoFormer [29], involves multiple rounds of supernet training, strategically extending weights along certain dimensions to optimize performance. NASViT [26] leverages gradient information during supernet training to refine subnet selection and mitigate gradient conflicts, thereby enhancing the effectiveness of generated architectures. The proven efficacy of one-shot NAS for transformer architectures provides a compelling rationale for its application to LLMs, considering that pretrained LLMs can function analogously as supernets. This adaptation holds the potential to significantly advance the development and optimization of LLM architectures, motivating us to refine and enhance the capabilities of these complex models.

3 Methodology

3.1 Framework Overview

We show the overview of our search framework in Figure 2. It comprises three key components: search initialization, search pipeline, and weight reformation. First, an initial efficient architecture is constructed layer by layer with a uniform inheriting ratio based on the weight importance. Subsequently, based on the initialization, we conduct a comprehensive search process for the globally efficient architecture with the evolution-based search method. Finally, a reformation method is introduced to enhance the performance of the resulting subnets in LLMs without retraining.

3.2 Search Initialization

Global search with uniform initialization. Unlike prior efficient LLM research efforts such as SparseGPT [7] with a uniform sparsity ratio across all layers, our method leverages a global search approach, such that different layers in our searched architecture may inherit different percentages of parameters (inheriting ratios) from the original LLM. To reduce the search cost and promote the search performance, we initialize our search with the same inheriting ratio for all layers. Through our search process, we iteratively refine the architecture, yielding subnets with varied inheriting

!

"

#

$%&'()*+

,-

()*+

.*/0'()*+

12&3

()*+

4&&0

52-

!"#$

!%&$

'()*+%"&

'()*+%"&

,&(-(.#

/*)(

!

!"# $# %&

!"# $# %&

6%7839%30&''''':2;3)8

'01 3("456

70&%-.23("456

Embedded Tokens

Embedding

Hidden States

Softmax

Activation

LMHead

ratios across layers. We demonstrate the pivotal role of the initialized architecture in driving search efficiency and effectiveness in Figure 5 and Section 3.3.3.

Structural subnets. To enable efficient inference, we search structural subnets from the original LLMs, i.e., certain rows or columns in the original 2D weight matrix are inherited in our searched model. Take the LLaMA family as an example. In each attention block of LLaMA models, there are query, key, value, and output linear layers in the self-attention module with weights denoted by W Q , W K , W V , and W O , respectively, and other three linear layers W U , W G , and W D in the MLP module. To ensure the consistent hidden size in LLaMA models, based on the computation patterns in each block, we select rows in W Q , W K , W V , W U , and W G , and columns in W O and W D . More details are presented in Figure 3 and Appendix A.

Initialization based on importance score. We construct the initial building blocks by inheriting appropriate rows/columns from the original LLM layer. To determine which row/column to be inherited, we compute the importance score for each row and column as below,

where [Φ r W ] i represents the row score for the i th row of W and [Φ c W ] j denotes the column score of the j th column. [ Φ ] i,j is the importance value of the element in the i th row and j th column of W , and X is the layer input. Following WoodFisher [45] and SparseGPT [7], the importance score reflects the minimum error of the layer-wise outputs (in terms of /lscript 2 norm) caused by removing a single weight. Note that the minimum error is evaluated by removing a single element from the weight matrix and it is not optimal in the case of simultaneously removing multiple weights.

Mask sharing. Given the column and row scores, we encode the architecture information by two masks: S attn ∈ R M for the self-attention module and S mlp ∈ R P for the MLP module for the layers in each building block. Different layers in the same module (self-attention or MLP) share the same mask to align the internal computations. We consider minimizing the collective importance scores for both the self-attention and MLP modules as below,

where ‖·‖ denotes the /lscript 1 norm and /circledot means the element-vise multiplication. Given the target model size, we uniformly set the same inheriting ratio for the masks in all building blocks. To obtain the mask in each block, we perform sorting for the sum of the corresponding scores in Equation (2) and (3), and inherit/keep the subnets with larger scores as the initialized architecture with the target size following the inheriting ratio, while other rows/columns with smaller scores are omitted.

3.3 Architecture Search

In this section, we present our comprehensive training-free search framework with the visualization of the search process for one block of the LLaMA model shown in Figure 3. We first delineate the methodology for mutation based on the initialized selection masks. Next, we define the search space and present the search pipeline. Besides, we verify the effectiveness of our initialization strategy by comparing the convergence speeds with and without our initialization.

3.3.1 Mask Mutation

During the search, we use mask mutation to generate new masks and thus new subnets to explore the search space. The inheriting ratio for the selection mask S attn is denoted as Γ attn = { γ i attn } h i =1 where h is the number of heads, and the inheriting ratio for S mlp is γ mlp . The mutation function M with the original mask S attn or S mlp , mutation probability P m , inheriting ratio requirement γ i attn or γ mlp , similarity ratio α , and maximum iteration η can be represented as follows,

Algorithm 1: Mask Mutation

Input: S , P m , γ , α , η

P r ← Random (0 , 1)

if

Inheriting

Ratio

_

Output: S

N

(

)

len

, iter

S

Idx

1

S

while len

Idx

== 1

(

Idx

2

iter

end

S

=

O

N

′

Output:

(

S

) ==

0

, Idx

}

1

2

2

∩

Idx

Random iter

;

γ

φ

)

∗

< α

r

P

N

(

and

{

1

_

+1

S

S

else

S

[

Idx

′

if

]

′

2

iter < η

where S i attn ∈ R h m denotes the selection mask for the i th head and h m is the head dimension. In details, we show the mask mutation process with Algorithm 1. If the inheriting ratio of input S already satisfies the requirement γ and the mutation is unnecessary based on the random generated P r (i.e., P r > P m ), we do not mutate and simply return S . Otherwise, given S and thus the set of indices Idx 1 for the inherited rows or columns, we try to generate a new set of indices Idx 2 through random sampling between 0 to len ( S ) -1 , such that (i) Idx 2 follows the required inheriting ratio requirement γ , and (2) the similarity of Idx 1 and Idx 2 (intersection set) is larger than threshold α .

3.3.2 Search Space

We define the LLM search space with three variables for each transformer building block below: the model depth d , inheriting ratios Γ attn = { γ i attn } h i =1 for S attn , and γ mlp for S mlp . The specifications of this search space, including the range for each factor, are detailed in Table 1.

γ mlp has a larger search space than { γ i attn } h i =1 according to our ablation study illustrated in Figure 4. Results are evaluated using LLaMA-7B on the WikiText2 dataset with a sequence length of 2048. Specifically, we apply the same local inheriting ratio for three cases, (i) the attention module only, (ii) the MLP module only, and (iii) both modules. Note that in case (i) or (ii), the global inheriting ratio is larger than case (iii) since the MLP in case (i) or the attention in case (ii) directly uses the original layers with 100% inheriting ratio. From Figure 4, we observe that case (ii) achieves a better perplexity with a lower global inheriting ratio than case (i), demonstrating that the MLP exhibits greater redundancy and is less sensitive to parameter reduction than the self-attention module. Therefore, we set a larger search space of inheriting ratios for MLP than the self-attention module.

Different from other transformer-based search works [29, 26, 46], we do not search the number of heads in self-attention. It stems from the nature of transformers that all heads are essential for representing the input data in the attention mechanism. Moreover, we refrain from conducting searches on the embedding and output layers of LLMs, as their weights constitute only a minor fraction of the total parameters yet are vital for the precise representation of tokens.

3.3.3 Search Pipeline

We implement our evolutionary search across the OPT and LLaMA model families with varying model sizes to derive efficient LLM architectures/subnets. The pipeline is shown below.

,

0

m

> P

then iter < η

,

1

· · ·

do

, N

-

1

}|

γ

)

!

"!

#!

$!

%!

&!!

$!'

$('

)!'

)('

%!'

%('

*!'

*('

&!!'

+,-./,01/23-

4567,58329

:-;/,01/23-

*!'

%!'

)!'

!"#$%&''(

)*+

,-'.

++*

/0'1-

!

"

#

$

%&

%!

%"

%#

!""#

$%&

'(")

*+,,,,,,,,,,,,,*+(,,,,,-#."./0

!"#

!

()*+,)-./0

1+234

%

%&

!&

5&

"&

6&

Initial generation. Given the single initialized subnet (Section 3.2), multiple candidates ( N subnets in total) are generated by mutation of the inheriting ratios with probability P 0 s and then mask mutation (Section 3.3.1) with probability P 0 m . The depth mutation is not involved at this initial step. The top k subnets are preserved as the initial generation.

Following generation. With the k subnets as parental candidates, a new population with N candidates are generated through mutation and crossover. We select a random parental candidate for mutation until the number of mutation candidates reaches a threshold N m . Mutation involves altering the depth with probability P d , altering the inheriting ratios with probability P s < P 0 s , and mutating the mask with probability P m < P 0 m (see Algorithm 1). The probabilities are smaller than initial generation as superior candidates should be preserved with less randomness. For the crossover, two parental candidates are randomly selected and combined to form a new candidate until there are N c candidates. With the population from the parental candidates, top k subnets are preserved as the next generation.

Candidate evaluation. For each generated candidate, if its parameter number does not fall in the range of the target model size, it is unsatisfying and we simply drop it. To compare candidate subnets, we evaluate them with a few random training samples from WikiText2 to compute the perplexity.

Necessity for initialization. To verify the effectiveness of our initialization in the search, we ablate the initialization for three cases, (i) self-attention only, (ii) MLP only, and (iii) both modules. LLaMA-7B is adopted with an 80% inheriting ratio for selection masks. Results are evaluated on Wikitext2 with a 2048 sequence length. As shown in Figure 5, the self-attention module complicates the identification of an effective subnet without our initialization strategy. In contrast, the MLP module exhibits less sensitivity to initialization. The search in both modules struggles to yield effective subnets without our initialization, primarily due to self-attention. The observation underscores the necessity of our initialization approach.

3.4 Reformation

After the search, we can obtain a subnet from the original LLM. To improve the subnet performance, we further reform the weights in the subnet by using the omitted weights to compensate their loss. Specifically, for each linear layer in the subnet with their original weights W before the search, we would like to reform the weights under the searched mask M and obtain ̂ W , so that the layer output difference in /lscript 2 norm, i.e., ‖ ̂ WX -WX ‖ 2 2 , is minimized. The problem is formulated below, min where M indicates the location of pruned weights with element 1 denoting pruned and 0 denoting unpruned weights. Here we only reform inherited columns based on omitted columns in W rather than reforming rows with omitted rows, since the output corresponding to omitted rows are always zeros which are unavailable for any compensations by modifications in other rows. To solve this problem, we propose a solution based on alternating direction method of multipliers (ADMM) [38, 37, 47] with the following theorem. The detailed proof is shown in Appendix B.

Theorem 3.1. Problem (6) can be solved in iterations. In the k th iteration, it performs the updates:

where ρ > 0 is the penalty parameter. The initial variable values at k = 0 follow the configurations that W 0 = W , Z 0 = W 0 , and U 0 = 0 .

̂ ̂ In practice, we find that after a few tens iterations (such as 20 or 30), the loss in Problem (6) converges. The complexity is determined by the inversion operation, which is the same as SparseGPT [7]. However, as it can converge fast within 30 iterations, our ADMM-based solution typically requires less computation time than SparseGPT which needs to iterate over all rows in the weight matrix.

3.5 Efficient Inference

After searching and reformation, we can get optimal efficient subnets with the selection masks S attn ∈ R M and S mlp ∈ R P for each block of the LLMs. We further convert the subnets into small-dense models following the masks for efficient inference. Thus, the dimension of the weight is actually reduced with faster inference speed. More details can be found in Appendix A.

4 Experiments

4.1 Experiment Settings

Hyper-parameter setting. For the evolutionary search, we adopt specific hyper-parameters as follows: the population size ( N ), the number of mutations ( N m ), and the number of crossovers ( N c )

are set to 100, 50, and 30, respectively. In each generation, the top 10 subnets are selected as parental candidates to produce offspring networks through the mechanisms of mutation and crossover. The rest subnets in the population are generated with mutation with larger randomness (i.e., same as initial mutation). The initial mutation probabilities ( P 0 m and P 0 s ) are set at 0.6 and 0.3 to promote variability early in the search process. Subsequently, for ongoing generation processes, the mutation probabilities ( P m and P s ) are adjusted to 0.3 and 0.1, while the probability for depth ( P d ) is maintained at 0.1. The similarity ratio α and maximum iteration η are set at 0.8 and 1000 in mask mutation. The total evolution epoch is 50. For the reformation, we adopt ρ as 1.0 and the iteration number as 30.

Datasets and metrics. We compare the perplexity of the models on the WikiText2 [48] and PTB [49] datasets with the 2048 sequence length. We also compare the zero-shot accuracy on common reasoning zero-shot classification datasets including BoolQ [50], PIQA [51], HellaSwag [52], WinoGrande [53], ARC-easy [54], ARC-challenge [54], and OpenbookQA [55].

Models. We evaluate on multiple LLM families including LLaMA [1], Vicuna [56] and OPT [2].

Baselines and pipeline. We compare with SOTA baselines including LLM-pruner [8], SliceGPT [9] and FLAP [10]. We adhere to the exact evaluation pipeline from the well-known LLM-Pruner [8], which is applied for all approaches to ensure a fair comparison. For the reformation, we randomly select 128 samples from training split of WikiText2, with the same random seed and thus same data for the calibration of other works, including SliceGPT and FLAP, ensuring a fair comparison.

Search Cost. We leverage the evolution search on NVIDIA A100 40G GPUs. Specifically, to explore the subnets of LLaMA-7B, we finish the search on one GPU with around 5 hours.

4.2 Main Results

Superior performance compared with SOTA baselines. We show our results of LLaMA-7B and LLaMA-13B in Table 2 and Figure 1 (b) and (c). We observe that our method outperforms all baselines in terms of perplexity on WikiText2 and PTB, and average zero-shot accuracy (over multiple zero-shot datasets). Take LLaMA-13B on WikiText2 as an example, our method improves the perplexity by 4.15, 1.65, and 0.13 compared to LLM-Pruner(b), SliceGPT, and FLAP, respectively, with a 80% inheriting ratio. Meanwhile, it achieves a higher average accuracy on seven classification datasets than baselines. For instance, under a 80% inheriting ratio, our method on LLaMA-7B improves the average accuracy by 2.89%, 12.72%, and 1.13% compared to LLM-Pruner(e1), SliceGPT, and FLAP, respectively. As the SliceGPT is sensitive to the calibration dataset, we further show the results with PTB calibration in Appendix C. Besides, we ablate the search with PTB in Appendix D.

The comparisons on other LLM families are demonstrated in Table 3 for Vicuna-7B [56] and Table 4 with Figure 1 (a) for OPT family [2]. As LLM-Pruner and FLAP do not implement their methods on OPT, we only compare with SliceGPT for OPT models. We can make similar observations that our method performs the best compared with SOTA baselines, demonstrating a superior generalization performance across various datasets/tasks, model structures, and inheriting ratios.

Scaling to small inheriting ratios and large models. Furthermore, our method consistently performs the best when scaling to larger model sizes or smaller inheriting ratios, indicating the great potential of our method for the ever-increasing LLM model size. Specifically, we show the results of LLaMA-7B and LLaMA-13B with lower inheriting ratios and thus more efficient models in Table 5 and Figure 1 (b) and (c). Our method consistently performs the best with more significant improvements under smaller inheriting ratios such as 50%. Besides, we deliver results on large models including LLaMA-30B and LLaMA-65B in Table 6 and Figure 1 (d). Our method achieves superior performance than FLAP under various settings, verifying our effectiveness and generalization.

4.3 Ablation Study

We show the results of LLaMA-7B with 128 sequence length in Table 7. Our method performs the best across different inheriting ratios, indicating the effectiveness on short sequences. Results of LLaMA-13B are in Appendix E.

Besides, to verify the influence of the number of examples for the reformation, we conduct ablation studies by varying the number from 128

to 512 and 1024. As shown in Figure 6, our reformation effectively improves the performance, and it is not sensitive to the number of samples. 128 samples already provide satisfying performance. Furthermore, we investigate the impact of the step number and ρ in the ADMM solution for the reformation, with detailed ablation results presented in Appendix F.

!

"

#!

#"

$!

$"

%!

%"

&!

'()*+(,-./

01.-2

3!4

5!4

6!4

7!4

"!4

!"#$%&'#%(

)*+$,-(./&,

0)*$,-(./&,

)1*2$,-(./&,

4.4 Generation Acceleration

To demonstrate our acceleration performance, we report the memory consumption and inference speed with our searched LLaMA-7B models on NVIDIA A100 40G GPUs across different inheriting ratios. As shown in Figure 7, we can observe that with a smaller inheriting ratio, our searched efficient model consumes less memory with a faster generation speed.

!"

!#

$"

$#

%"

%#

"

!

%

&

'

("

(!

(%

)*+,-./0123

4,5*6787

9:;<,

(""=

>"=

'"=

?"=

&"=

#"=

!"#$%&

'$(")*+*

5 Conclusion and Limitation

In this paper, we propose a training-free search framework to find the optimal subnets inside LLMs. We further propose a reformation algorithm that reconstructs the weights of subnets to enhance the task performance. The experiments show the effectiveness of our proposed method compared to SOTA structured pruning methods. Additionally, we achieve memory reduction and practical inference acceleration on GPUs, which shows the efficiency of our method. The search cost required by our method can increase with the model size, which takes more time for large models.

Acknowledgment

We would like to express our sincere gratitude to Professor Lin and Professor Wang for their invaluable guidance throughout the development of this paper. We are also deeply grateful to Pu, Yifan, and Zhenglun for their dedicated contributions, thoughtful insights, and collaborative efforts, which were essential to the completion of this research. Meanwhile, This research is funded in whole or in part by the National Science Foundation CNS-2312158 to Northeastern University. Any errors and opinions are not those of the National Science Foundation and are attributable solely to the author(s).

References

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv , 2023.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv , 2022.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv , 2022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS , 33:1877-1901, 2020.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. 2020.

Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.00774 , 2023.

Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems , 2023.

Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations , 2024.

Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-based adaptive structured pruning for large language models, 2023.

Zheng Zhan, Zhenglun Kong, Yifan Gong, Yushu Wu, Zichong Meng, Hangyu Zheng, Xuan Shen, Stratis Ioannidis, Wei Niu, Pu Zhao, and Yanzhi Wang. Exploring token pruning in vision state space models. arXiv preprint arXiv:2409.18962 , 2024.

Xuan Shen, Zhenglun Kong, Changdi Yang, Zhaoyang Han, Lei Lu, Peiyan Dong, et al. Edgeqat: Entropy and distribution guided quantization-aware training for the acceleration of lightweight llms on the edge. arXiv preprint arXiv:2402.10787 , 2024.

Changdi Yang, Pu Zhao, Yanyu Li, Wei Niu, Jiexiong Guan, Hao Tang, Minghai Qin, Bin Ren, Xue Lin, and Yanzhi Wang. Pruning parameterization with bi-level optimization for efficient semantic segmentation on the edge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15402-15412, 2023.

Yihua Zhang, Yuguang Yao, Parikshit Ram, Pu Zhao, Tianlong Chen, Mingyi Hong, Yanzhi Wang, and Sijia Liu. Advancing model pruning via bi-level optimization. Advances in Neural Information Processing Systems , 35:18309-18326, 2022.

Yanyu Li, Changdi Yang, Pu Zhao, Geng Yuan, Wei Niu, Jiexiong Guan, Hao Tang, Minghai Qin, Qing Jin, Bin Ren, Xue Lin, and Yanzhi Wang. Towards real-time segmentation on the edge. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence , AAAI'23/IAAI'23/EAAI'23. AAAI Press, 2023.

Pu Zhao, Fei Sun, Xuan Shen, Pinrui Yu, Zhenglun Kong, Yanzhi Wang, and Xue Lin. Pruning foundation models for high accuracy without retraining. arXiv preprint arXiv:2410.15567 , 2024.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate posttraining compression for generative pretrained transformers. arXiv preprint arXiv:2210.17323 , 2022.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning , 2023.

Xuan Shen, Peiyan Dong, Lei Lu, Zhenglun Kong, Zhengang Li, Ming Lin, Chao Wu, and Yanzhi Wang. Agile-quant: Activation-guided quantization for faster inference of llms on the edge. Proceedings of the AAAI Conference on Artificial Intelligence , 38(17):18944-18951, Mar. 2024.

Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.09355 , 2019.

Siqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, and Jingjing Liu. Contrastive distillation on intermediate representations for language model compression. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 498-508, Online, November 2020. Association for Computational Linguistics.

Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang, Yaliang Li, and Jun Huang. MetaKD: A meta knowledge distillation framework for language model compression across domains. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 3026-3036, Online, August 2021. Association for Computational Linguistics.

Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 6105-6114. PMLR, 09-15 Jun 2019.

Xuan Shen, Yaohua Wang, Ming Lin, Yilun Huang, Hao Tang, Xiuyu Sun, and Yanzhi Wang. Deepmad: Mathematical architecture design for deep convolutional neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6163-6173, June 2023.

Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: A zero-shot nas for high-performance deep image recognition. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021 , 2021.

Chengyue Gong, Dilin Wang, Meng Li, Xinlei Chen, Zhicheng Yan, Yuandong Tian, qiang liu, and Vikas Chandra. NASVit: Neural architecture search for efficient vision transformers with gradient conflict aware supernet training. In International Conference on Learning Representations , 2022.

Xiu Su, Shan You, Jiyang Xie, Mingkai Zheng, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Vision transformer architecture search. arXiv preprint arXiv:2106.13700 , 2021.

Dongning Ma, Pengfei Zhao, and Xun Jiao. Perfhd: Efficient vit architecture performance ranking using hyperdimensional computing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , pages 2230-2237, June 2023.

Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 12270-12280, October 2021.

Peiyan Dong, Zhenglun Kong, Xin Meng, Pinrui Yu, Yifan Gong, Geng Yuan, Hao Tang, and Yanzhi Wang. Hotbev: Hardware-oriented transformer-based multi-view 3d detector for bev perception. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 2824-2836. Curran Associates, Inc., 2023.

Zheng Zhan, Yifan Gong, Pu Zhao, Geng Yuan, Wei Niu, Yushu Wu, Tianyun Zhang, Malith Jayaweera, David Kaeli, Bin Ren, et al. Achieving on-mobile real-time super-resolution with neural architecture and pruning search. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4821-4831, 2021.

Yushu Wu, Yifan Gong, Pu Zhao, Yanyu Li, Zheng Zhan, Wei Niu, Hao Tang, Minghai Qin, Bin Ren, and Yanzhi Wang. Compiler-aware neural architecture search for on-mobile real-time super-resolution. In European Conference on Computer Vision , pages 92-111. Springer, 2022.

Changdi Yang, Yi Sheng, Peiyan Dong, Zhenglun Kong, Yanyu Li, Pinrui Yu, Lei Yang, Xue Lin, and Yanzhi Wang. Fast and fair medical ai on the edge through neural architecture search for hybrid vision models. In 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD) , pages 01-09. IEEE, 2023.

Changdi Yang, Yi Sheng, Peiyan Dong, Zhenglun Kong, Yanyu Li, Pinrui Yu, Lei Yang, and Xue Lin. Late breaking results: Fast fair medical applications? hybrid vision models achieve the fairness on the edge. In 2023 60th ACM/IEEE Design Automation Conference (DAC) , pages 1-2. IEEE, 2023.

Peiyan Dong, Zhenglun Kong, Xin Meng, Pinrui Yu, Yifan Gong, Geng Yuan, Hao Tang, and Yanzhi Wang. Hotbev: Hardware-oriented transformer-based multi-view 3d detector for bev perception. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 2824-2836. Curran Associates, Inc., 2023.

Yanyu Li, Pu Zhao, Geng Yuan, Xue Lin, Yanzhi Wang, and Xin Chen. Pruning-as-search: Efficient neural architecture search via channel pruning and structural reparameterization. arXiv preprint arXiv:2206.01198 , 2022.

Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and trends® in Optimization , 1(3):127-239, 2014.

Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning , 3(1):1-122, 2011.

Pu Zhao, Sijia Liu, Yanzhi Wang, and Xue Lin. An admm-based universal framework for adversarial attacks on deep neural networks. In Proceedings of the 26th ACM International Conference on Multimedia , MM '18, page 1065-1073, New York, NY, USA, 2018. Association for Computing Machinery.

Yifan Gong, Zheng Zhan, Zhengang Li, Wei Niu, Xiaolong Ma, Wenhao Wang, Bin Ren, Caiwen Ding, Xue Lin, Xiaolin Xu, et al. A privacy-preserving-oriented dnn pruning and mobile acceleration framework. In Proceedings of the 2020 on Great Lakes Symposium on VLSI , pages 119-124, 2020.

Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695 , 2023.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR , 2021.

Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems , 33:1809818109, 2020.

Yi-Lun Liao, Sertac Karaman, and Vivienne Sze. Searching for efficient multi-stage vision transformers. arXiv preprint arXiv:2109.00642 , 2021.

Tom Goldstein, Brendan O'Donoghue, Simon Setzer, and Richard Baraniuk. Fast alternating direction optimization methods. SIAM Journal on Imaging Sciences , 7(3):1588-1623, 2014.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv , 2016.

Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics , pages 313-330.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044 , 2019.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 7432-7439, 2020.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 , 2019.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2381-2391, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.

Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks , 107:3-11, 2018.

Appendix

A Efficient Inference

After searching and reformation, we can get optimal efficient subnets with the selections masks S attn ∈ R M and S mlp ∈ R P for each block of the LLMs. In detail, for the weights of query, key, and value denoted as W Q , W K , W V ∈ R M × D , we generate the weight subsets by extracting the selected rows from the original weights, which are denoted as W ′ Q , W ′ K , W ′ V ∈ R m × D . For the weights of the output projection W O ∈ R D × M , we extract the columns instead of rows and reform the selected weight based on the omitted ones for the weight subnets W ′ O ∈ R D × m . Subsequently, the given input X ∈ R BN × D is projected in the self-attention module as follows,

For the MLP module in the LLaMA family, we denote the weights of the three linear layers with W U , W G ∈ R P × D , and W D ∈ R D × P for the up, gate, and down projections, respectively. The weight subsets generated with the selection mask S mlp for three linear layers are W ′ U , W ′ G ∈ R p × D , and W ′ D ∈ R D × p , where only W D is reformed. Then, the given input X ∈ R BN × D is projected in the MLP module as follows,

where the activation function for the LLaMA family is SiLU [57].

Therefore, the computation cost is reduced for both self-attention and MLP modules, while the configuration of the input X ∈ R BN × D is preserved for preventing information loss and maintaining the representation capability of tokens.

B Proof of Theorem 3.1

Problem (6) can be reformulated as follows, where g ( Z ) is a inidicator function as the following,

We can see that Problem (12) is equvilant to Problem (6).

Based on ADMM [37, 38, 47], Problem (12) can be solved with ADMM iterations. In the k th iteration, it needs to address the following,

Problem (12) is split into multiple sub-problems with Problem (14) and (15).

Problem (14) is similar to Ridge regression problem. We can directly obtain its solution as

̂ W k +1 =( XX T + ρ I ) -1 ( XX T W T + ρ ( Z k -U k ) T ) , (17) To solve Problem (15), we can set Z k +1 = ( ̂ W k +1 + U k ) and project Z k +1 on the g function as follows,

Thus, we can obtain the solution in Theorem 3.1.

C SliceGPT Comparison

For SliceGPT [9], the generated results are sensitive to the calibration datasets, we further show the results of SliceGPT with the calibration of PTB training dataset and 2048 sequence length in Table A1. We can know that our method can achieve better performance with the calibration on WikiText2 instead of PTB dataset than the SliceGPT with the calibration on the same dataset.

D Search on PTB Dataset

To further verify the generalization and effectiveness of our method, we utilize the training portion of the PTB dataset in our search instead of WikiText2. The results, shown in Table A2, are evaluated with a sequence length of 2048 using the LLaMA-7B model. Our findings reveal that the models generated through searches on two different training datasets achieve similar performance, demonstrating the robustness and generalization capability of our search method across different datasets.

E Ablation for 128 Sequence Length

We also present the results for the LLaMA-13B model with a sequence length of 128 in Table A3, demonstrating that our method continues to achieve superior performance. The results are evaluated with on WikiText2 dataset.

6

6.5

7

7.5

8

8.5

Steps

0

5

10

20

30

50

Perplexity

6.85

6.9

6.95

7

7.05

7.1

7.15

7.2

7.25

0.01

0.1

1

10

100

Perplexity

F Ablation in Reformation

To explore the influence of the number of iterations on the reformation process, we conducted experiments with varying iteration counts, as shown in Figure A1. The results, evaluated using the LLaMA-7B model with an 80% inheritance ratio on the WikiText2 dataset with a sequence length of 2048, indicate that model performance improves with an increasing number of iterations, peaking around 30 steps. Beyond this point, there is minimal difference between 30 and 50 iterations. Additionally, we examined the impact of different ρ values on the reformation, as depicted in Figure A2. Our findings show that for ρ ∈ [0 . 01 , 0 . 1] , the reformed model's performance remains increasing, but deteriorates when ρ reaches 10 or bigger. Hence, ρ = 1 becomes the optimal value.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We explain method and summarize the contribution in introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitation is included in conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the theorem in methodology section and the proof in appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the detailed experiment setup in experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The dataset and model we used is open-source.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide detailed experiment setting in experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: We did not report error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We explain the computation resources in experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Research is conducted in the paper conform with NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss them in conclusion.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: We discussed in introduction.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We originally implemented our method.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We introduced in the introduction.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: There is nothing related to human subjects in our work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: There is nothing related to human subjects in our work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a neural architecture search method for Large Language Models (LLMs) comprising three steps: inheriting the most salient weights from the original model to form the initial sub-network, using an evolutionary algorithm to search for the optimal sub-network, and reconstructing the original network’s output with calibration samples.\n\nSTRENGTHS:\nOverall, I like the application of evolutionary algorithm to pruning problem.\n1. The method overcomes the limitations of uniform sparsity across all layers—a common but sub-optimal constraint in previous structured pruning methods.\n2. It avoids reliance on back-propagation, enhancing its applicability to larger models within a constrained memory budget.\n\nWEAKNESSES:\nHere are some technical concerns:\n\n1. The candidate evaluation is conducted based on perplexity on WikiText2. This is one metric on one task, then the searching algorithm might be biased towards the task used for candidate evaluation, resulting in a model favoring one task (e.g., language modeling) over others. How to ensure the performance of the searched model on other tasks?\n2. Based on my experience, using perplexity as a metric for selecting pruning candidates can result in good performance on WikiText but sub-optimal performance on the MMLU dataset. It would be great if authors could validate their pruned (or searched) model on MMLU.\n3. Table 5. The performances on QA tasks should also be presented. If the space is not enough, authors can put them in the Appendix and link it in the main text.\n4. Figure 7. Tokens/s is not a very rigorous measurement for speed. The generated tokens per second can depend on many factors, e.g., floating point precision, KV Cache, flash attention, batch size, etc. I suggest authors use other metrics such as MACs or at least provide more specifications.\n\nQUESTIONS:\nIn addition, here are some points about writing: \n\n5. The term \"inheriting ratio\" needs to be rigorously defined in an early section. In addition, authors would better add a sentence describing the difference between \"inheriting ratio\" and \"sparsity\" / \"pruning ratio\". In my understanding, \"inheriting ratio\" = 1 - \"sparsity\".\n6. Figure 2 Left: it is better to add an annotation showing which axis is the layer index. I assume it is the horizontal axis.\n7. Section 3.2, Lines 139-149: The writing of this part needs to be improved. It gives me a lot of confusion when I read this part.\n\n    (i) Please define what $M$ and $P$ are in Line 140.\n\n    (ii) Are the masks smooth or binary? If binary then $\\{0, 1\\}$ should be used instead of $\\mathbb{R}$ in Line 140.\n\n    (iii) Line 141, why do different layers of Attn / MLP share the same mask? If so, would the sparsity of each layer be the same? However, according to Figure 1, the sparsity should be different across layers. Also, in Line 146, authors \"set the same inheriting ratios for the masks in all building blocks\". So, these claims and figures are very chaotic. Additionally, what does \"align the internal computations\" mean?\n\n    (iv) Line 144, Are Eq. (2) and (3) jointly optimized (e.g., take the sum) or separately? Are the masks of different layers jointly optimized or separately?\n8. Section 3.3.1, Line 165: $\\gamma$ should be rigorously defined.\n\nLIMITATIONS:\n1. Section 5: The limitations provided by the authors are too trivial. It is obvious that larger models require more time for pruning/searching. Please add more in-depth discussions on limitations. Authors can refer to the points listed in the \"Weaknesses\" section for more substantial issues that need addressing.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an architecture search method based on mask mutation and candidate evaluation to find a subnet with better performance in the LLM. An evolution-based algorithm is applied to globally search the subset with a special initialization from the evaluation of parameter importance. After the search from LLM, a reformation algorithm is proposed to rectify the weights in the inherited LLM. Experimental results show that the compressed LLM can achieve better results compared to the baselines.\n\nSTRENGTHS:\n1. A novel method for searching a subnet in the LLM. The search of subnet in LLM is not well-explored, and it would benefit the community.\n2. The overall approach is reasonable and sufficient. The entire method’s pipeline is very clear, and the design for each component is reasonable and complete.\n3. The experimental results are comprehensive and show its benefits compared to the baselines (SliceGPT and LLM-Pruner). A large and steady improvement is observed. \n4. The paper is well-organized and easy to follow.\n\nWEAKNESSES:\n1. My main concern is about the extra cost of searching the subnetworks. Since it still takes around 5 hours to search for the architecture, it introduces extra cost for pruning the model. What if we take these 5 hours to post-train the compressed model? Would the results be better or worse than the subnetwork search from the LLMs?\n2. Since the training-free structured pruning is not a very useful setting, post-training the LLM with an acceptable amount of resources is more realistic. Thus, compared to just pruning and searching the mode, a more realistic setting is also to train the pruned model. And I’m not sure whether the searched sub-LLM would be better than the pruned one. Some previous work to prune CNNs suggest that with sufficient training, magnitude pruning would be a better choice than any manually designed metrics. Thus, it would be better to examine the performance of the compressed LLM with post-training.\n\nQUESTIONS:\n1. For Mask Sharing: why does the same module in different modules share the same mask (Line 141)? In line 147 - 149, from my understanding, the mask is not the same for blocks in different layers. Using different masks for different layers seems to still have the aligned internal computations. I’m not sure if I understand correctly in this part.\n2. What about the results if you post-train the compressed models? What are the results if compared with LLM-Pruner?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper introduces a new model architecture search method for LLMs that does not require additional training. The proposed architectures have structural sparsity and reach better performance than SOTA pruning baselines.

STRENGTHS:
- The proposed method does not require additional training. 

- The output model has structural sparsity, which is useful for hardware. 

- The paper is clear and well-written.

WEAKNESSES:
- Why isn't there a comparison to SparseGPT?

QUESTIONS:
Why isn't there a comparison to SparseGPT?

LIMITATIONS:
The authors discuss limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a technique for searching for efficient LLM subnets for fast inference, while still attaining strong performance. The proposed technique involve a training-free pruning stage based on a genetic algorithm, followed by a ``weight rectification'' stage that improves the resulting subnet. Empirically, the proposed method outperforms existing LLM pruning baselines.\n\nSTRENGTHS:\n- Empirically, the proposed pruning technique appears to outperform a handful of existing pruning baselines, as shown in Figure 1 and Table 2. \n- The proposed technique appears to be general enough to work for a wide variety of existing LLMs, which appears to be a s\nhortcoming of existing LLM pruning techniques. \n- The proposed method is fairly simple in that it combines LLM pruning with a genetic algorithm for search.\n\nWEAKNESSES:\n- From the abstract, it is unclear what the actual proposed method does beyond standard neural network pruning techniques. \n- The term ``reformation algorithm'' is used throughout the paper, and it is unclear what this means until much later in t\nhe text. The intro and abstract should clarify this. \n- The main evaluation is done in terms of sparsity levels, which makes sense for this particular type of search space, how\never, the authors should also empirically validate the efficiency of the search process compared to existing pruning techn\niques. E.g., how do these pruning techniques compare, subject to the same budget? Moreover, it would be interesting to see how the proposed method performs as a function of the search budget.                                       \n- The authors should compare to other techniques beyond pruning, such as quantization which may achieve similar inference speed and memory improvements with better overall performance. In general, it seems that LLM pruning techniques take a sub\nstantial performance hit even at fairly modest sparsity levels (I'm mainly referring to Table 7).\n\nQUESTIONS:\n- The authors mention in the limitations that the search cost scales with the model size. Can the authors characterize thi\ns a bit more? Do other pruning methods suffer from the same problem, and can the authors comment on what this scaling curv\ne actually looks like? \n- Why was the search budget set to 5 hours? Have the authors explored this choice empirically, and does this apply to all of the models and data settings that were evaluated?\n\nLIMITATIONS:\nThe authors adequately address limitations in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-22"}