Title: DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Model Compression

Model Pruning

Efficient Inference

Large Language Models
Abstract: Large Language Models (LLMs) have achieved remarkable success in various natural language processing tasks, including language modeling, understanding, and generation. However, the increased memory and computational costs associated with these models pose significant challenges for deployment on resource-limited devices. Structural pruning has emerged as a promising solution to reduce the costs of LLMs without requiring post-processing steps. Prior structural pruning methods either follow the dependence of structures at the cost of limiting flexibility, or introduce non-trivial additional parameters by incorporating different projection matrices. In this work, we propose a novel approach that relaxes the constraint imposed by regular structural pruning methods and eliminates the structural dependence along the embedding dimension. Our dimension-independent structural pruning method offers several benefits. Firstly, our method enables different blocks to utilize different subsets of the feature maps. Secondly, by removing structural dependence, we facilitate each block to possess varying widths along its input and output dimensions, thereby significantly enhancing the flexibility of structural pruning. We evaluate our method on various LLMs, including OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2. Experimental results demonstrate that our approach outperforms other state-of-the-art methods, showing for the first time that structural pruning can achieve an accuracy similar to semi-structural pruning.

DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models

Abstract

Large Language Models (LLMs) have achieved remarkable success in various natural language processing tasks, including language modeling, understanding, and generation. However, the increased memory and computational costs associated with these models pose significant challenges for deployment on resource-limited devices. Structural pruning has emerged as a promising solution to reduce the costs of LLMs without requiring post-processing steps. Prior structural pruning methods either follow the dependence of structures at the cost of limiting flexibility, or introduce non-trivial additional parameters by incorporating different projection matrices. In this work, we propose a novel approach that relaxes the constraint imposed by regular structural pruning methods and eliminates the structural dependence along the embedding dimension. Our dimension-independent structural pruning method offers several benefits. Firstly, our method enables different blocks to utilize different subsets of the feature maps. Secondly, by removing structural dependence, we facilitate each block to possess varying widths along its input and output dimensions, thereby significantly enhancing the flexibility of structural pruning. We evaluate our method on various LLMs, including OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2. Experimental results demonstrate that our approach outperforms other state-of-the-art methods, showing for the first time that structural pruning can achieve an accuracy similar to semi-structural pruning.

1 Introduction

Large Language Models (LLMs) have revolutionized the field of natural language processing by leveraging deep learning techniques to process and generate human-like text. Compared to smaller models, LLMs exhibit unique characteristics and demonstrate remarkable abilities in tackling a wide range of complex tasks [40]. Despite their impressive capabilities, the vast number of parameters in LLMs often hinders their deployment on resource-constrained devices, such as mobile phones. Consequently, there is significant interest in reducing the computational and memory requirements of LLMs.

Existing compression techniques for large language models (LLMs) include weight sparsification [9], structural pruning [30], and quantization [10]. In this work, we focus on structural pruning and

𝐗𝐢𝐧

𝐖𝐢𝐧

𝐖𝐨𝐮𝐭

Activation

Structural Dependency from

Regular Structural Pruning Methods with

Activation

𝐖𝐢𝐧

𝐖𝐨𝐮𝐭

𝐗𝐨𝐮𝐭

𝐗𝐢𝐧

Dimension-Independent Structural

Pruning (Ours)

Index

Operations

Index

Operations

No Dependency

address the limitations of previous methods in this category. Structural pruning [30] is a generalpurpose compression solution that maintains LLM performance across various tasks, facilitates deployment on devices, and is computationally efficient. However, existing methods may restrict pruning flexibility or add significant overhead to the compressed model. For instance, LLM-Pruner [30] follows structural dependence during pruning, requiring different layers to use the same subset of feature maps, which limits pruning flexibility. SliceGPT [2] alleviates this issue by applying orthogonal projections for each layer but introduces a non-trivial number of additional parameters (e.g., 5% to 13% of the parameters of the original model for LLaMA-2 7B). Our approach aims to overcome these drawbacks and offer a better performance-cost trade-off for structural pruning.

We aim to increase the flexibility of current structural pruning methods and consequently improve performance. Our method provides different sub-spaces or subsets of features to different layers, but unlike SliceGPT, it doesn't introduce additional parameters. To achieve this, we break the structural dependence of regular structural pruning methods, allowing different layers to have different subsets of features along the embedding dimension and an example is given in Fig. 1. After pruning, we employ index selection and index addition operations to sample subsets of features from the residual connection and add them back after the computation of each layer. Furthermore, our method introduces an additional level of flexibility by learning different widths for each layer. Our approach significantly improves the flexibility of structural pruning without adding additional parameters.

Extensive experimental results show that our method can outperform state-of-the-art structural pruning methods for LLMs while still maintaining low computational costs. Our method does not require recovery fine-tuning to obtain such performance. In addition, our method does not update the remained model weights during pruning which is a distinct departure from several other methods, such as SparseGPT [9] and LLM Surgeon [37]. Our contributions are as follows:

We break the structural dependence of regular structural pruning methods, significantly increasing the flexibility of structural pruning. This allows different layers to select their own subset of features from the embedding dimension. Importantly, our method achieves this without introducing additional parameters, unlike SliceGPT.

We propose to learn the widths of each layer using gradient-based optimization methods. A hypernetwork generates the column or row selection matrices, while the width of each layer is controlled globally. This approach allows for fine-grained control over the pruning process and enhances the adaptability of our method to various models and tasks.

Our method demonstrates superior performance compared to state-of-the-art structural pruning techniques for LLMs across a range of models, including OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2. Notably, the resulting model from our method is a sub-network that exists within the original model, indicating the effectiveness of our method in discovering strong sub-networks.

2 Related Works

Magnitude-based pruning is the most straightforward approach to reduce model size, where weights with the smallest magnitude are pruned. Han et al. [14] employ this strategy for pruning with L 1 or L 2 norm of weights. Filter pruning [24] extends this setting to structures of the model instead of performing weight-level sparsification. Although magnitude-based pruning methods are very efficient, they result in significant performance drops for LLMs, even for weight pruning [9]. Another

𝐗𝐨𝐮𝐭

line of research, Optimal Brain Damage [23] and Optimal Brain Surgeon [15], utilize second-order information to remove connections. These methods require calculating the inverse of the Hessian matrix, which is computationally intensive for modern neural network architectures like Convolutional Neural Networks (CNNs) [22, 16], Transformers [38], or Large Language Models (LLMs) [35]. To reduce the cost of computing the Hessian inverse matrix, Optimal Brain Surgeon can be applied in a layer-wise fashion [7, 8], making the computation tractable. However, further scaling up these methods for LLMs remains challenging.

Inputs

𝐒 𝟏

𝐒 𝟏

𝐒 𝟏

Multi-head

Attention

𝐖𝐨𝐒𝟐

Activation

Outputs

𝐒 𝟑

𝐒 𝟒

Layer Norm

DISP-LLM

(Ours)

Layer Norm

Index Add

Index Select

&

Layer Norm

When Pruning

Layer Norm

𝐒𝟏, 𝐒

𝟐 , 𝐒

𝟑 , 𝐒

𝟒 , 𝐒 𝟓

෠

𝐒𝟏, ෠

𝐒𝟐, ෠

𝐒𝟑, ෠

𝐒𝟒, ෠

𝐒𝟓

Add Index Selection

Replace addition with

Index Add

Prune weights with the

actual selection matrices

When Training the

Hypernetwork

Recent methods like SparseGPT [9] or GPTQ [10] aim to minimize the squared error before and after pruning or quantization of a given layer. In this setting, the Hessian inverse matrix becomes easy to compute, as it is simply the multiplication between the feature map and its transpose for a given layer. GPTQ and SparseGPT then quantize or sparsify model weights in a column-by-column manner, and the unpruned or unquantized weights are updated to compensate for the error of pruning and quantization. Wanda [34] further avoids computing the inverse of the Hessian matrix by only considering the diagonal of the Hessian matrix. While SparseGPT and Wanda achieve good results, unstructured sparsity is known to be harder to achieve actual speedup. They also applied their methods on semi-structured settings [31], but the performance becomes much worse.

Several researches [28, 19, 44, 13, 42, 12] apply learnable parameters for specific structures when pruning vision or language models. However, many of these methods cannot be scaled up to LLMs since they need to learn weights and structures together. In contrast, our method explores sub-networks within the original model without updating model weights. Additionally, our method mainly explores the regime of pruning without recovery fine-tuning, which is rarely presented in previous methods with learnable parameters on structures. Our method is also related to the unconstrained channel pruning for CNNs [39]. However, our method explores this idea from the perspective of breaking structural dependence and scales it to much larger models than [39]. Moreover, our method thoroughly explores the global allocation of parameters, where [39] fails to do.

Recently, several works have been proposed to reduce the size of LLMs. LLM-Pruner [30] aims to remove connected structures using importance calculated from Taylor expansions. SliceGPT [2] offers more flexibility than regular pruning by projecting the feature maps to different spaces but introduces extra parameters in the residuals. LLM Surgeon [37] periodically updates model weights and structures, resulting in a higher cost than LLM-Pruner and SliceGPT. Our proposed DISP-LLM breaks the structural dependence relied on by LLM-Pruner, without additional transformation matrices in the residual connections like SliceGPT. Furthermore, in contrast to LLM Surgeon, which requires extensive computational resources, our method is significantly more efficient.

3 Preliminary

3.1 Notations

To better understand our paper, we first define some notations. We use d to denote the model dimension or embedding dimension of LLMs. X P ℜ b ˆ n ˆ d is used to represent feature maps, and b is the mini-batch size, n is the number of tokens. W P ℜ d 1 ˆ d 2 is the model weights of size d 1 ˆ d 2 . Let S denote a pseudo-index selection matrix of size d ˆ d , which is a diagonal matrix filled with 0 or 1 and the positions of the ones indicate the selected index. We further use ˆ S of size d ˆ d small to

XQ in

XQ out

Win

T

Wout

Qout

Unified selection

matrices S:

LLMPruner or

Regular Structural Pruning

Dense orthogonal

matrices Q

in

or Q

out

XS

S W

in

T

Wout

S

XS

XS in

S

in

W in

T

Wout

S

out

XS out

Selection matrices

S

in

and S

out

:

DISP-LLM

(Ours)

SliceGPT

represent the actual selection matrix by removing d ´ d small columns with all zeros from S . For any matrix A , nnz( A ) represents the number of nonzero entries of A .

3.2 Revisit SliceGPT

The core idea of SliceGPT [2] is to achieve computational invariance within the transformer architecture. It demonstrates that orthogonal projections can be applied to the output of each block and subsequently undone in the next block. This transformation is computed using Principal Component Analysis (PCA), allowing the feature maps between blocks to be projected into their principal components. A significant advantage of this approach is that it projects the feature maps of different blocks into distinct spaces, thereby introducing an additional degree of freedom for compression. This flexibility is not captured by regular structural pruning methods like LLM-Pruner [30], which rely on structural dependence.

After slicing (pruning), the feature map and weight matrix of l th layer of SliceGPT become:

where ˆ S is a d ˆ d small selection matrix, X l is the output of the l th block, and Q l contains eigenvectors of C l :

and X l,i is the i -th column of X l (corresponding to the i th sequence in the calibration dataset). From Eq. 1, we can see that SliceGPT uses the same selection matrix ˆ S for all layers, but the feature map X l is firstly projected by Q l , and the pruning for different layers is along with different directions. One crucial drawback of SliceGPT also comes from the projection matrix Q l , since the residual connection must be multiplied by the linear transformation Q J l Q l ` 1 (shown in Fig. 7 left in the Appendix). These additional operations bring a non-trivial amount of additional parameters. For a model that has L blocks, with the model dimension d and the remaining percentage of parameters p P r 0 , 1 s , it brings approximately Ld 2 p 2 additional parameters to the model (more than 10% of model parameters in some cases, and more details are given in Fig 10 in the Appendix).

3.3 Residual Connections Limit the Flexibility of Structural Pruning

SliceGPT offers significant flexibility, but achieving similar flexibility with regular structural pruning methods without adding extra parameters is challenging. This section explains the reasons behind this difficulty. To simplify our reasoning, we replace ˆ S with its pseudo selection matrix S .

Assume we follow the basic setting of dependence-based structural pruning but allow each layer the flexibility to have its own selection matrix, S l , along the embedding dimension. Under this assumption, due to structural dependence, all layers will share the same width of nnz p S 0 S 1 ¨ ¨ ¨ S L q .

In order to prune different positions for different layers, we need to add a transformation matrix to align the width of layers l and l ` 1 . Intuitively, if we have S l and S l ` 1 , we can then insert S J l S l ` 1 in the residual connection to align consecutive layers.

Algorithm 1: Block inference after pruning.

Input : Feature map of the previous block X in. Preserved indices sets Ind 1 , Ind 2 , Ind 3 , Ind 5 . 1. ˆ X in ' LayerNorm p X in r : , Ind 1 sq . Ź Index Selection for Attention 2. X att ' MultiHead p ˆ X in ˆ S J 1 W q , ˆ X in ˆ S J 1 W k , ˆ X in ˆ S J 1 W v q W o ˆ S 2 . 3. X res ' Index_Add p X in , X attn , Ind 2 q . Ź Index Addition with the input 4. ˆ X res ' LayerNorm p X res r : , Ind 3 sq . Ź Index selection for MLP 5. X mlp ' p σ p ˆ X res ˆ S J 3 W 1 ˆ S 4 q d p ˆ X res ˆ S J 3 W 2 ˆ S 4 qq ˆ S 4 J W 3 ˆ S 5 . 6. X out ' Index_Add p X res , X mlp , Ind 5 q Ź Index Addition with the residual Return X out for the next block.

With this setup, we can use X l S l to select subsets of features for different layers, mimicking Q l S for SliceGPT. Although it seems promising, this formulation has issues with layer widths, as detailed in Proposition 1.

Proposition 1 (Decreasing feature dimensions for deeper layers) . Let the pseudo-selection matrices in layers l and l ` 1 be S l and S l ` 1 , respectively. The number of nonzero entries in the residual adapter satisfies

For compression strategies that remove dependent structures for layer l ` 1 following S J l S l ` 1 , this implies that the dimension in layer l ` 1 is less than or equal to that in layer l , with equality holding when the feature indices selected in layer l ` 1 are contained within those in layer l or vice versa.

Remark. The proof of Proposition. 1 is straightforward and it is given in the Appendix A.1. From Proposition 1, we observe that if we naively apply S l for different layers, the model width will progressively decrease as we go deeper into the network. It also fails to provide different sets of features for different layers; instead, it merely passes a subset of features from the previous layer to the next. To avoid this restriction, all blocks must share the same width and the same pruned columns or rows. And we then fall back to the regime of previous structural pruning methods such as LLM-Pruner [30], Shared LLaMA [43], etc.

Proposition 1 highlights two significant obstacles. First, dependence-based structural pruning methods result in a uniform width along the embedding dimension. Second, inserting selection matrices in the residual connections causes the embedding dimension to decrease with depth. These challenges are unavoidable due to the residual connections linking structures across layers. To enhance flexibility along the embedding dimension, bypassing the residual connections is crucial.

4 Dimension-Independent Large Language Model

4.1 Break the Structural dependence

Section 3.3 demonstrates that the residual connection is the primary barrier preventing pruning methods from achieving better flexibility. To avoid modifying the residual connection, we relocate the selection matrices inside the residual connection. This approach allows us to successfully create different subsets from the feature maps for different layers.

Based on this idea, we propose a solution that involves pruning different positions in consecutive blocks and selecting or merging feature maps from or back to the residual connection. This approach breaks the structural dependence inherent in previous pruning methods. Formally, given a transformer block, we apply the following operations:

where S 1 , . . . , S 5 are pseudo selection matrices of size d ˆ d , and d denotes element-wise multiplication. Eq. 3 gives an example operation for gated MLP modules used in LLaMA [35]. For Phi models [1] or OPT [46], the MLP operation is defined as MLP p X q ' σ p XS J 3 W 1 S 4 q S 4 J W 3 S 5 . Fig 2 illustrates how to insert these selection matrices into a transformer block.

Given the operations defined in Eq. 2 and Eq. 3, we successfully remove the constraint in Proposition 1. The input and output of both the Attention layer and the MLP layer can be selected differently from the original feature maps for different layers, mimicking the function of Q l in SliceGPT. Additionally, our method eliminates the need for extra parameters in Q J l Q l ` 1 as it does not alter the residual connection. We also enhance flexibility by pruning the middle dimension of the MLP layer. Additionally, this flexibility can be further improved by allowing the query, key, and value weight matrices to use different selection matrices. Our current form is kept for two reasons: (1) SliceGPT uses one Q l per layer, and we followed this design for a fair comparison, and (2) adding separate selection matrices would increase indexing operations, potentially slowing down the inference. Fig 3 further compares the projection matrices for SliceGPT, regular structural pruning, and the proposed method.

Once we have the final selection matrices S 1 , . . . , S 5 , the pruned model will use a combination of index selection and index addition for inference as shown in Algorithm 1, where Ind i is a set containing all indices equal to one in the diagonal of S i :

The same color is used to mark the index set Ind i and its corresponding selection matrix ˆ S i . Index_Add p A , B , Ind q adds matrices A and B along the last dimension on selected positions from Ind , then returns A after index addition. With index selection and addition, the block dimension can be freely changed. Index selection and addition introduce some overhead, but as demonstrated in the experiment section, we still observe improvements in throughput.

4.2 Learning the Width for Dimension-Independent LLMs

Building on the dimension-independent setting introduced in Section 4.1, our approach offers much greater flexibility in selecting sub-networks from the original dense model compared to the constrained settings in LLM-Pruner [30]. The next challenge is determining the width of each layer. Given the large search space of our dimension-independent structural pruning and the computationally intensive nature of LLMs, it is impractical to use reinforcement learning [17] or evolutionary search-based algorithms [27]. Therefore, we adopt gradient-based methods to address this challenge. Given the diagonal vector s i P t 0 , 1 u d from S i , the Straight-Through (ST) gradient estimator [3] is used to estimate the gradients with respect to learnable continuous latent parameters. More specifically, we use the recently proposed gradient estimator ReinMax [26] to estimate the gradients through the binary operation. A detailed explanation of ReinMax for the binary case is provided in Appendix A.2.

Given the large search space of our method, we find that only using element-wise learnable parameters is insufficient. To address this issue, a hypernetwork is introduced to generate latent parameters for ReinMax, as detailed below:

where Θ represents the parameters of the hypernetwork and s contains s i from all blocks. The hypernetwork is composed of GRU [5] and fully connected layers, where the GRU captures blockwise relationships and the fully connected layers capture relationships between different dimensions. With the hypernetwork and ReinMax, we can effectively learn the width of each block. The details of the hypernetwork are provided in Appendix A.3.

4.3 Dimension-Independent Structural Pruning as an Optimization Problem

With the methods described above, we can formulate dimension-independent structural pruning as an optimization problem, with regularization to control the number of remaining parameters. We insert s back into S as defined in section 4.1 for forward and backward calculations. The overall objective function is listed below:

where L is the language modeling loss function of next word prediction, X represents the input tokens, W is the collection of model weights, s is defined in Eq. 4, and R is a parameter regularization loss function defined in Eq. 6. Here, T p s q denotes the number of parameters controlled by the current structure s , T total is the total number of parameters of the model, and p P p 0 , 1 s is a user-defined

parameter to control how many parameters should be preserved within the model. With the objective function in Eq. 5, the structures for dimension-independent pruning can be efficiently optimized. Moreover, the overhead of our method is minimal and comparable to parameter-efficient fine-tuning methods like LoRA [18], as it does not require storing gradients or the first and second-order momentum of model weights for the Adam optimizer [21].

5 Experiments

5.1 Settings

Models. We evaluate our DISP-LLM method using several LLMs with transformer blocks. Specifically, we choose the following models: OPT [46]: OPT-125M, OPT-1.3B, OPT-2.7B, OPT-6.7B; Phi-1.5 [25] and Phi-2 [20]; LLaMA 7B [35]; LLaMA-2 [36]: LLaMA-2 7B and LLaMA-2 13B.

Implementations. We implemented our method using Pytorch [32] and Hugging Face transformer library [41]. We freeze the model weights W when training the hypernetwork. We use the AdamW [29] optimizer to optimize the hypernetwork. The hypernetwork is trained for 10,000 iterations for all models. For all experiments, we set λ in Obj. 5 to 6. Depending on the size of the base model, we use 1 to 4 NVIDIA A100 GPUs to train the hypernetwork. More implementation details can be found in the Appendix A.4.

Datasets. Following previous papers [2, 30], we use WikiText-2 and Alpaca datasets to train the hypernetwork. Following SliceGPT [2], we evaluate our method and other methods on five wellknown zero-shot tasks: PIQA [4]; WinoGrande [33]; HellaSwag [45]; ARC-e and ARC-c [6]. We use llm-eval-harness [11] to evaluate the compressed models.

Baselines. We compare our DISP-LLM across baselines from structural pruning like LLMPruner [30], SliceGPT [2] and LLMSurgeon [37]. We also include semi-structure pruning baselines like SparseGPT [9] and Wanda [34].

5.2 Language Modeling

In Table 1, we report the perplexity of pruned OPT and LLaMA-2 models. Our DISP-LLM, which does not update weights, consistently outperforms more complex pruning methods such as K-OBD and LLM Surgeon, which involve weight updates, across all pruning ratios and models . The

Model Dimension

Depth

Preserved

Pruned

Pruning Decisions along the Model Dimension and Depth

Pruning Rate along the Model Dimension

and the Overall Pruning Rate

0.9

0.8

0.9

0.8

Preserved Rate

0.7

0.6

Preserved Rate

0.7

0.6

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1

0

500

100015002000250030003500

4000

0

performance gap is even larger when compared to SliceGPT. The advantage is particularly clear in better-trained models like LLaMA-2 7B and 13B. For instance, our method surpasses LLM Surgeon by margins of 5.54 and 2.22 when pruning 50% of parameters of LLaMA-2 7B and 13B, respectively. Against K-OBD, our performance advantage extends to 36.80 and 9.49 under the same setting. For consistency, we let the pruning ratio of SliceGPT equal the slicing ratio. However, the actual pruning ratio for SliceGPT is much lower than the slicing ratio. More details are given in Appendix A.5.

In Table 2, we report the perplexity of pruned LLaMA and LLaMA-2 models and we compare our method with semi-structure pruning methods. From the table, we can see that our method outperforms both SparseGPT and Wanda on LLaMA 13B and LLaMA-2 7B/13B models. Our method performs on par with SparseGPT and Wanda with the LLaMA 7B model, and our DISP-LLM is a little bit worse than SparseGPT and is similar to Wanda. We are the first to show that structural pruning methods can have a better or similar performance than semi-structural pruning methods .

5.3 Zero-shot Performance

In Tab. 3, we present the zero-shot performance of the pruned model. For the LLaMA 7B model, we compare our method against LLM-Pruner with and without recovery fine-tuning. Our method consistently outperforms LLM-Pruner without fine-tuning, and the gap ranges from 2.63 to 8.88 across different pruning rates for average task performance. After fine-tuning, the performance of LLM-Pruner is largely boosted, however, our method is still able to outperform it demonstrating the existence of strong sub-networks within the original model. For the LLaMA-2 7B model, we compare our method against SliceGPT, K-OBD, and LLM Surgeon. With weight updates, LLM

(a) Ablation Loss

L

(b) Ablation Loss

R

(c) Ablation

p

(d) Ablation Iterations

(e) Given

p

Loss

L

(f) Given

p

Loss

R

Dense

𝟏. 𝟎𝟖 ×

𝟏. 𝟐𝟎 ×

𝟏. 𝟑𝟔 ×

𝟏. 𝟓𝟎 ×

(g) Acceleration

𝟏𝟒. 𝟕𝟔 ×

𝟐𝟕. 𝟑𝟗 ×

(h) Costs

DISP-LLM w HN

NH 0m W77-dSI

0.8

60

50

45

Constrained w HN

DISP-LLM w HN

DISP-LLM wlo HN

46

50

3.6

40

44

42

0.5

3.2

30

36

80

0.2

25

34

0.1

20

32

30

2000

4000

Iterations

6000

8000

10000

0

2000

4000

Iterations

6000

8000

10000

10

15

20

2530

Pruning Ratio (%)

35

40

45

50

55

0.5

Training Iterations

1.5

2

×104

2.5

p=0.6

p=0.5

0.9

1.5

1000

900

DISP-LLM

LLM Surgeon

3.8

3.6

p=0.9

p=0.7

p=0.8

0.8

800

700

3.4

0.6

1.3

600

500

400

3.2

ww

JO

0.3

300

0.2

200

2.8

0

2000

4000

Iterations

6000

8000

10000

0.1

50

Iterations

100

150

200

1

45

50

55

Remained #Params (%)

60

65

70

75

80

85

100

0

13B

7B

𝐒𝟏

𝐒𝟐

𝐒𝟑

𝐒𝟒

𝐒𝟓

0.8

Preserved Rates

0.6

0.4

0.2

0

5

10

15

20

25

30

Layers

Surgeon performs well with a lower pruning ratio like 30%. At this pruning ratio, our method performs similarly to LLM Surgeon, and our method outperforms other comparison baselines. When increasing the pruning ratio to 50%, the advantage of our method becomes obvious, and the gap between our method and LLM Surgeon is 5.37 for average task performance. We further compare our method with SliceGPT on Phi-1.5 and Phi-2, and our method consistently achieves better performance.

5.4 Analysis

Ablation Study. We visualize the results of ablation studies in Figs. 5a, 5b, 5c, 5d with Phi-1.5 model. The setting 'DISP-LLM w/o HN' refers to using element-wise gates for learning subnetworks. The setting 'Constrained LLM w HN' refers to pruning the embedding dimension following the structural dependence like LLM-Pruner. From Figs. 5a, 5b, we can see that using the hypernetwork largely accelerates the learning process for DISP-LLM, which is also verified in Figs. 5c, 5d. From Figs. 5c, 5d, we also see that our DISP-LLM always outperforms constrained structural pruning, demonstrating the value of added flexibility by breaking the dependence. To further study the impact of the HyperNetwork architecture, we provide more results in Tab. 4. 'w/o HN' is equivalent to 'DISP-LLM w/o HN' . The setting 'w/o Bi-GRU' simply removes GRU and adds a fixed input (initialized in the same way as see Appendix A.3 for more details) for each linear layer. These results indicate that both GRU and linear layers within the HyperNetwork affect the final performance. One explanation is that linear layers connect different dimensions of the model, accelerating learning, while GRU layers capture inter-layer relationships, further reducing the difficulty of learning sub-network structures. Therefore, both GRU and linear layers are essential to the HyperNetwork.

Different Pruning Ratios, Costs and Throughput. In Figs. 5e, 5f, we show the language modeling loss L and regularization loss R in Obj 5 given different pruning ratios p with Phi-1.5 model. We can see that our method consistently minimizes the language modeling loss across different p . In addition,

our method quickly pushes the regularization loss R to near 0 values within 200 iterations. In Fig. 5g, the pruned model from LLaMA-13B improves the throughput of the dense model by 1 . 08 ˆ to 1 . 50 ˆ . In Fig. 5h, we compare the costs of our method against LLM Surgeon. Our method is 27 . 39 ˆ and 14 . 76 ˆ cheaper compared to LLM Surgeon with LLaMA-2 7B and LLaMA-2 13B models.

Every Embedding Dimension is Important. In Fig. 4, we visualize the pruning decisions along the embedding dimension and depth for the LLaMA-2 7B model, we can see that all embedding dimensions have been used across different layers. This becomes more obvious in the second right figure of Fig. 4, where we sum all pruning decisions along the depth dimension, and we can see that every embedding dimension is kept around 80% given all layers. We further visualize the model width after pruning for the LLaMA-2 7B model in Fig. 6, where we can see that several layers are more severely pruned than other layers.

6 Conclusion

In this paper, we proposed dimension-independent structural pruning for Large Language Models. By breaking the structural dependence imposed by previous compression methods, our method creates sub-networks with a lot more flexibility than regular structural pruning methods and also avoids the overhead brought by SliceGPT. The flexibility of our method is reflected in two perspectives. Firstly, our method provides different subsets of the feature maps for different layers. Secondly, our method freely selects the width for each layer without considering architecture dependence. With dramatically increased flexibility, our method outperforms other structural pruning and semi-structural pruning methods given similar pruning ratios. The novel design of the unconstrained pruning space along with strong empirical performance opens new possibilities for structural pruning for LLMs.

References

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 , 2024.

Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations , 2024.

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 7432-7439, 2020.

Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259 , 2014.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018.

Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Advances in Neural Information Processing Systems , pages 4857-4867, 2017.

Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems , 35:4475-4488, 2022.

Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning , pages 10323-10337. PMLR, 2023.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations , 2023.

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021.

Shangqian Gao, Junyi Li, Zeyu Zhang, Yanfu Zhang, Weidong Cai, and Heng Huang. Device-wise federated network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12342-12352, 2024.

Shangqian Gao, Zeyu Zhang, Yanfu Zhang, Feihu Huang, and Heng Huang. Structural alignment for network pruning through partial regularization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 17402-17412, 2023.

Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems , pages 1135-1143, 2015.

Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon . Morgan Kaufmann, 1993.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc: Automl for model compression and acceleration on mobile devices. In Proceedings of the European conference on computer vision (ECCV) , pages 784-800, 2018.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision (ECCV) , pages 304-320, 2018.

Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog , 2023.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pages 1097-1105, 2012.

Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems , pages 598-605, 1990.

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. ICLR , 2017.

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463 , 2023.

Liyuan Liu, Chengyu Dong, Xiaodong Liu, Bin Yu, and Jianfeng Gao. Bridging discrete and backpropagation: Straight-through and beyond. Advances in Neural Information Processing Systems , 36, 2023.

Yuqiao Liu, Yanan Sun, Bing Xue, Mengjie Zhang, Gary G Yen, and Kay Chen Tan. A survey on evolutionary neural architecture search. IEEE transactions on neural networks and learning systems , 34(2):550-570, 2021.

Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV , 2017.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.

Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems , 36:21702-21720, 2023.

Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv e-prints , pages arXiv-2104, 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , pages 8024-8035, 2019.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021.

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations , 2024.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Tycho FA van der Ouderaa, Markus Nagel, Mart Van Baalen, and Tijmen Blankevoort. The llm surgeon. In The Twelfth International Conference on Learning Representations , 2024.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Alvin Wan, Hanxiang Hao, Kaushik Patnaik, Yueyang Xu, Omer Hadad, David Güera, Zhile Ren, and Qi Shan. Upscale: unconstrained channel pruning. In International Conference on Machine Learning , pages 35384-35412. PMLR, 2023.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research , 2022.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, October 2020. Association for Computational Linguistics.

Xidong Wu, Shangqian Gao, Zeyu Zhang, Zhenzhen Li, Runxue Bao, Yanfu Zhang, Xiaoqian Wang, and Heng Huang. Auto-train-once: Controller network guided automatic network pruning from scratch. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1616316173, 2024.

Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations , 2024.

Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Association for Computational Linguistics (ACL) , 2022.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4791-4800, 2019.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

A Appendix

Inputs

RMS Norm

𝐐𝟏

𝐓 𝐖𝐤′

𝐐𝟏

𝐓 𝐖𝐪′

𝐐𝟏

Multi-head

Attention

𝐖𝐨 ′ 𝐐𝟐

𝐐𝟏

𝐓 𝐐 𝟐

Activation

𝐐𝟐

𝐓 𝐐 𝟑

Outputs

RMS Norm

𝐐𝟐

′

𝐖𝟐

′ 𝐐𝟑

SliceGPT

Inputs

𝐒 𝟏

𝐒 𝟏

𝐒 𝟏

Multi-head

Attention

𝐖𝐨𝐒𝟐

Activation

Outputs

𝐒 𝟑

𝐒 𝟒

Layer Norm

Ours

Layer Norm

Index Add

Index Select

&

Layer Norm

When Pruning

Layer Norm

𝐒𝟏, 𝐒

𝟐 , 𝐒

𝟑 , 𝐒

𝟒 , 𝐒 𝟓

෠

𝐒𝟏, ෠

𝐒𝟐, ෠

𝐒𝟑, ෠

𝐒𝟒, ෠

𝐒𝟓

Model Dimension

Depth

Preserved

Pruned

Pruning Decisions along the Model Dimension and Depth

0.8

Preserved Rates

0.6

0.4

0.2

5

10

15

20

25

30

35

40

Layers

A.1 Proof of Proposition 1

Proposition 1 (Decreasing feature dimensions for deeper layers) . Let the pseudo-selection matrices in layers l and l ` 1 be S l and S l ` 1 , respectively. The number of nonzero entries in the residual

Algorithm 2: Binary ReinMax

Input

: x

: sigmoid input;

τ : temperature; c : constant bias.

Output :

x : binary vector.

1. π 0 ' sigmoid p x ` c q ,

2. B ' sample_binary p π 0 q ,

3. π 1 ' B ` sigmoid pp x ` c q{ τ q 2 ,

4. π 1 ' sigmoid p stop_gradient p ln p π 1 q ´ p x ` c qq ` p x ` c qq ,

5. π 2 ' 2 π 1 ´ 1 2 π 0 ,

6. x ' π 2 ´ stop_gradient p π 2 q ` B

Return x .

adapter satisfies

For compression strategies that remove dependent structures for layer l ` 1 following S J l S l ` 1 , this implies that the dimension in layer l ` 1 is less than or equal to that in layer l , with equality holding when the feature indices selected in layer l ` 1 are contained within those in layer l or vice versa.

Proof. Consider the pseudo-selection matrices S l and S l ` 1 , both of size d ˆ d , and both diagonal matrices. The number of nonzero entries in S l and S l ` 1 are given by nnz p S l q ' k l and nnz p S l ` 1 q ' k l ` 1 , respectively.

The product S J l S l ` 1 is also a diagonal matrix of size d ˆ d . Each diagonal entry p i, i q in S J l S l ` 1 is the product of the i -th diagonal entry of S l and the i -th diagonal entry of S l ` 1 . For an entry p i, i q to be nonzero, both S l p i, i q and S l ` 1 p i, i q must be nonzero.

Thus, the number of nonzero entries in S J l S l ` 1 , nnz p S J l S l ` 1 q , is the number of indices i where both S l p i, i q and S l ` 1 p i, i q are nonzero. This count cannot exceed the smaller of the total number of nonzero entries in S l and S l ` 1 .

Hence,

This implies that the effective feature dimension will be smaller or equal to the previous layer. Equality holds if and only if the set of indices corresponding to nonzero entries in S l ` 1 is a subset of those in S l , or vice versa. This concludes the proof.

In Proposition 1, 'remove dependent structures for layer l ` 1 following S J l S l ` 1 ' means that the actual selection matrix for layer l ` 1 becomes S 1 l ` 1 ' S J l S l ` 1 , and the structure dependence is cut off by the next residual connection. The pruning for layer l ` 1 will based on S 1 l ` 1 instead of S l ` 1 . Although this setting partially breaks the structural dependence, it has the limitation that the embedding dimensions will be reduced when going deeper.

A.2 Binary ReinMax

In this section, we provide details for ReinMax when handling binary variables. The ReinMax in our work can be written as shown in Algorithm. 2. We add a constant bias c to x so that we can control binary vectors to have all one value at the beginning when learning the sub-network architecture for DISP-LLMs. Through all experiments, we set c to 3.0 and τ to 1.0.

A.3 Details of the Hypernetwork

Input

z

(a) WikiText Loss

L

(b) WikiText Loss

R

L

R

2.8

1.6

$$07

2.6

2.4

2.2

552

p=0.5

p=0.6

p=0.7

p=0.8

Normalized

0.9

0.8

0.7

9°0

0.5

S907

1.5

1.4

1.3

1.2

ormalized

R

0.6

6°0

0.8

0.7

0.5

Mowww

0.4

1.1

0.4

0.3

Z

0.3

8'

0.2

0.1

0.9

0.2

0.1

1.6

0

2000

4000

Iterations

6000

8000

10000

0

50

Iterations

100

150

200

0.8

0

2000

4000

Iterations

6000

8000

10000

50

Iterations

100

150

200

As we discussed in the paper, the Hypernetwork is composed of linear layers and Bi-GRUs, and now we present the architecture of the HN in Tab. 5. z is initially sampled from a normal distribution, and it is then fixed during training. Outputs s l are continuous values and it is then fed to ReinMax to produce the binary vector: s ' ReinMax p s q , where s is the collection of s l from all layers. N l is the original model width, and it equals the embedding dimension for S 1 , S 2 , S 3 and S 5 .

A.4 More Implementation Details

0.9

0.8

Preserved Rate

0.7

0.6

0.4

0.3

0.2

0.1

0

1000

2000

3000

4000

5000

Additional Training Details. During training the hypernetwork, we use AdamW optimizer to optimize it with a constant learning rate 10 ´ 3 and weight decay 0 . 05 . We train the hypernetwork for different models, we always set the mini-batchsize to 1 on each GPU. For OPT 6.7B, LLaMA 7B, and LLaMA-2 7B models, we use 2 NVIDIA A100 GPUs, and for LLaMA 13B and LLaMA-2 13B models, we use 4 NVIDIA A100 GPUs. For all the rest models, we use 1 NVIDIA A100 GPU. We set p ' t 0 . 5 , 0 . 4 , 0 . 3 , 0 . 2 , 0 . 1 u when the pruning ratios equals to t 50% , 40% , 30% , 20% , 10% u . For the Alpaca dataset 2 , we use the 'text' column within the dataset which combines the columns of 'instruction' and 'output'. When training the hypernetwork, we again minimize the language modeling loss on the Alpaca dataset instead of applying the training process of instruction fine-tuning.

Details of Eq. 6. The parameter regularization loss function in Eq. 6 is defined as follows:

Since y is fixed, when x ą y , Eq. 6 will decrease x , making it closer to y . Conversely, when x ă y , Eq. 6 will increase x , also making it closer to y . Thus, the parameter regularization loss always tries to push the current sub-network to achieve the pre-defined parameter budget.

Ablation Study Settings. In the ablation study 5.4, we removed the hyperntwork, and we revise Eq. 4:

where Θ now has the same size of s , and the parametrization space becomes much smaller. For the constrained setting used in the ablation study 5.4, we simply let S 1 ' S 2 ' S 3 ' S 5 . In section 5.4, we calculate the costs of our method and LLM-Surgeon, and the price comes from the official website of Lambda Cloud 3 . We also list the detailed time costs of our method in Tab. 6. In Fig. 5b, 5f and also in Fig. 11, we normalized the parameter regularization loss R with its maximum value to make plots more consistent.

Licenses. The licenses for various models and datasets are as follows: LLaMA and LLaMA 2 : Licensed under the LLAMA 2 Community License. Phi 1.5 and Phi 2 : Licensed under the MIT License. WikiText dataset : Licensed under the Creative Commons Attribution-ShareAlike License (CC BY-SA 4.0). Alpaca dataset : Licensed under the Creative Commons AttributionNonCommercial License (CC BY-NC 4.0).

A.5 Additional Results

SliceGPT compression rates: In Fig. 10, we show the expected compression rate and the actual compression rate for our method and SliceGPT given the LLaMA-2 7B model. It can be seen that SliceGPT adds 5% to 13% parameters of the original model across different pruning rates, which is non-trivial for most LLMs. Notably, SliceGPT with 10% slicing actually adds 3% more parameters to the original model.

LLaMA-2 13B Results. In Tab. 7, we show the results of the LLaMA-2 13B model given different pruning rates. From the table, we can see that our method consistently outperforms LLM Surgeon under different pruning rates. The advantage of our method becomes more obvious compared to other methods like K-OBD and SliceGPT.

Phi-2 Results. In Tab. 8, we present a more comprehensive comparison of our method compared to SliceGPT. Our method outperforms SliceGPT by 5.28 to 7.01 giving SliceGPT with or without

fine-tuning on the WikiText dataset. These observations again demonstrate the effectiveness of our method, and our method outperforms methods with recovery fine-tuning in several settings.

LLaMA-2 3B Results. In Table 9, we present a comparison of our method against LLM Pruner and magnitude pruning. At a pruning ratio of 20%, our method surpasses the performance of LLM Pruner both with and without fine-tuning. Remarkably, even when the pruning ratio is increased to 50%, our method continues to outperform the LLM Pruner Channel and the Magnitude pruning baselines at a 20% pruning rate. These results further illustrate our method's ability to identify strong sub-networks within the original dense model.

Architecture of the pruned LLaMA-2 13B. In Fig. 8 and Fig. 9, we visualize the pruned architecture of the LLaMA-2 13B model pruned with the WikiText dataset. We have similar observations as in section. 5.4. In Fig. 9, we can see that middle to late layers have large pruning rates, especially for the attention layer. In Fig. 8 and Fig. 12, we can also see that the preserved rates for different dimensions are similar, and all embedding dimensions are effectively utilized. More interestingly, similar pruning rates across different dimensions are achieved without adding any regularization or constraints.

Training loss for LLaMA-2 7B. In Fig. 11, we further visualize the training dynamics of the LLaMA-2 7B model on WikiText and Alpaca datasets, respectively. The regularization loss with the Alpaca dataset decreases a little bit faster than the WikiText dataset, probably because the loss value on the Alpaca dataset is smaller. From Fig. 11c, we can also see that the language modeling loss continues to decrease when training longer, especially when the pruning ratio is higher.

Lastly, we evaluate our method on the Phi-1.5 model with 5 runs, and we report our result with mean and standard deviation in Tab. 10. We also measure the throughput of our method on the LLaMA-2 13B model, and the result is shown in Tab. 11

A.6 Additional Analysis

Impact of λ on Phi-1.5 when pruning 50% of parameters. We show the result in Tab. 12. 'NC' means the loss R does not converge, and it is much larger than zero, thus it can not prune the model to the target budget. From the table, we can see that the PPL of the model becomes quite stable if it is

larger or equal to 6. If λ is not large enough, it takes longer to push the loss R to reach near 0 values and thus leaves less time for the model to explore different configurations of subnetworks. On the other hand, if λ is large enough, the loss R will reach zero in several hundred iterations and leave enough time to find the desirable subnetwork. Due to this reason, our method is quite stable across larger values of λ as shown in the table.

More results on pruning ratio vs. performance trade-offs. We provide the trade-off between the pruning ratio and performance for the Phi-2 and LLaMA-2 7B model below in Tab. 13 and Tab. 14.

LoRA fine-tuning [18] of the compressed LLaMA-2 7B model. We follow similar settings of the SliceGPT and the model is fine-tuned on Alpaca. The result is shown in Table. 15. We can see that our method can be further boosted by using parameter-efficient fine-tuning techniques. Since the performance without fine-tuning is already good enough, we prefer not to involve this additional process in our method to save time and computational costs.

A.7 Generation Samples

We show the generated text given DISP-LLM and SliceGPT in Tab. 16. The examples are obtained based on removing 20% of the model weights with DISP-LLM and 20% slicing of SliceGPT ( 10% compression rate). Both models are compressed from LLaMA-2 7B and they are not finetuned. From these two examples, we can see that SliceGPT only generates a small part of meaningful content and then starts repeating itself. On the other hand, our method tends to generate more relevant content, and the self-repeating behavior is much less obvious. These observations comply with the performance gap between DISP-LLM and SliceGPT.

A.8 Limitations

Our method explores how to break the structural dependency of pruning for LLMs. Although our method achieves competitive performance, there are still some limitations that are not solved in our current version. The throughput improvements of our method are not consistent across different models, which is probably because the current implementation of index add or index select operations with PyTorch is not efficient enough. An alternative implementation of our method is to select weight matrices instead of feature maps. In this approach, we perform matrix multiplication with the full feature map and fill the weight matrices with zeros. If we have a custom matrix multiplication implementation that ignores zero rows or columns, it may achieve further speed-up. However, this approach primarily focuses on the engineering perspective and is beyond the scope of this work. We leave the exploration of this alternative implementation to future research.

A.9 Border Impact

Our dimension-independent structural pruning method significantly reduces the computational and memory requirements of LLMs, enabling their deployment on resource-limited devices and lowering energy consumption, thus contributing to environmental sustainability. By facilitating the use of efficient LLMs on affordable hardware, our approach democratizes access to advanced AI technologies. However, it is crucial to address potential ethical concerns when reducing the size of LLMs, ensuring privacy and fairness in AI applications with compressed LLMs. Overall, our method fosters a more accessible and sustainable use of AI.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims in the abstract/introduction are reflected by the experimental results and other related sections of our method.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: It is provided in the Appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: It is given in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provided the detailed experimental settings in the experiment section and also in the Appendix to ensure reproducibility of our method.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Due to the company policy, the code will only be released after going through the internal review process.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The details are provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We follow the settings of previous methods where the result of a single run is provided.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This has been provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We followed the NeurIPS Code of Ethics in preparing our manuscript.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Provided in the appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Provided in the Appendix.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper concerns about the dimension dependence of hidden states across different layers. It have been argued that previous studies with dimension dependence would be inefficient due to additional parameters or architectural constraints. This work breaks the dimension dependence by introducing indexing operations to previous methods. Although simple, the proposed method is shown to be more effective than baselines due to permitted asymmetry across layers.

STRENGTHS:
1. It is for the first time highlighted that the dimension dependence may affect the performance of pruning.
2. The proposed method to alleviate the dimension dependence is rather simple and effective.

WEAKNESSES:
1. Although the proposed method allows asymmetric dimensions in module weights across layers, it also requires the dimensions of hidden states from different layers maintain the same as that of the non-pruned model. This would restrict the use in scenarios where the dimension of the output hidden states is critically important (e.g., embedding for retrieval).
2. The introduced indexing operation could be very time-consuming. Unexpectedly, the latency introduced by the indexing may outweight the latency reduced by pruning. And it is not discussed whether the pruned model is faster than the non-pruned one. I would recommend including such experiments in proper positions.

QUESTIONS:
N/A

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-11

comments: SUMMARY:
DISP-LLM introduces dimension-independent structural pruning for LLMs, breaking structural dependencies to allow flexible pruning across layers and dimensions. It uses a hypernetwork for pruning decisions and claims superior performance over existing methods on various LLMs without weight updates or additional parameters.

STRENGTHS:
- Novel approach to structural pruning in LLMs
- Comprehensive evaluation across multiple LLM architectures
- Theoretical foundation provided
- Achieves good performance without weight updates or extra parameters

WEAKNESSES:
- Limited novelty beyond combining existing concepts
- Inconsistent performance gains across models/tasks
- Insufficient ablation studies and hyperparameter sensitivity analysis
- Lack of detailed computational efficiency comparisons
- Inadequate error analysis and failure case exploration
- Overstated claims based on limited experiments
- Insufficient discussion of pruning ratio vs. performance trade-offs

QUESTIONS:
- How sensitive is DISP-LLM to hypernetwork architecture and hyperparameters?
- Can you provide detailed computational cost comparisons with baselines?
- How does it perform on fine-tuning tasks?
- Have you explored progressive pruning approaches?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
This paper proposes a novel structural pruning method for LLMs, primarily based on SliceGPT. It characterised in (i) removing structural dependence by facilitating each block to possess varying widths along its input and output dimensions (ii) no need for introducing addition paramaters like sliceGPT.  The empirical studies show that it demonstrates superior performance compared to state-of-the-art structural pruning techniques for LLMs across a range of models, including OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2.

STRENGTHS:
1. This paper is easy-to-read and topic is very important to efficient Machine Learning.
2. The proposed method balances the task performance and efficient, which are both beneficial to its practical and wide application.
3. The empirical results show its promising in model compressing across four models.

WEAKNESSES:
1.  More comprehensive literature review about model compressing, like sparse mechanism.
2. Any case study for qualitative analysis to the pruned model? It is possible to conduct some interpretability analysis showing where the difference between sliceGPT and the proposed method along with a specific input example.

QUESTIONS:
1.  When training with the pruning objectives all the layers are fine-tuned or only selected layers?
2. Any insights from Figure 6, different model width as layer changes. If the observations aligned with existing research results?

LIMITATIONS:
Pay attention to the generated sequence after model prune because the prediction changes are unpredictable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nStructural pruning is a method to prune the weights of large language models (LLMs) while keeping their original performance as much as possible it can. However, structural pruning has limitations caused by depending on the structure, like the residual connection of LLMs. In this work, the authors propose a new pruning method, dimension-independent structural pruning for LLMs (DISP-LLM), that can ignore structural dependence in pruning. This characteristic of DISP-LLM makes it possible to prune their weights more flexibly and provides new pruning ways using different feature maps between blocks and varying pruning dimensions of the input and output of the block. However, just pruning the dimensions and ignoring residual connections causes inconsistency between the input and output of a block. To solve this problem, the authors apply transformation matrices to each residual connection to absorb these discrepancies. Furthermore, pruned indices are optimized through training. Experimental results on the language modeling dataset WikiText-2 and the zero-shot task datasets PIQA, WinoGrande, HellaSwag, ARC-e, and ARC-c show that DISP-LLM tends to outperform conventional approaches in pruning LLMs, OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2.\n\nSTRENGTHS:\n- The proposed dimension-independent structural pruning for LLMs (DISP-LLM) can drastically prune dimensions due to its flexibility.\n- Adding transformation matrices to each residual connection to maintain input and output consistency in each transformer is novel\n- In DISP-LLM, pruned dimensions can be optimized through training.\n- Experiments are comprehensive and cover various kinds of LLMs.\n- DISP-LLM can achieve comparable or better perplexities to conventional pruning approaches that require updating the original weights by training.\n- In many cases, DISP-LLM outperforms conventional pruning approaches in zero-shot tasks even though some of them update the original weights by training.\n- The authors report the actual costs for pruning models and show the cost efficiency of DISP-LLM.\n\nWEAKNESSES:\n- Even though DISP-LLM does not update the original weights of LLMs, training is required.\n- DISP-LLM requires hyper-tuning for $\\lambda$.\n\nQUESTIONS:\n- How did you decide to set $\\lambda$ to 6?\n- How are the performances sensible to the changes of $\\lambda$?\n- As you state in the paper, DISP-LLM is as efficient as LoRA. If so, can applying LoRA weights only to the kept dimensions be a baseline?\n\nLIMITATIONS:\nDISP-LLM requires training and hyper-parameter tuning for $\\lambda$ that controls mixing ratios of losses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}