DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models
Authors: Shangqian Gao
Chi-Heng Lin
Ting Hua
Zheng Tang
Yilin Shen
Hongxia Jin
Yen-Chang Hsu
Venue: NeurIPS 2024 poster (2024)
Keywords: Model Compression
Model Pruning
Efficient Inference
Large Language Models
Primary area: optimization_for_deep_networks

Abstract: Large Language Models (LLMs) have achieved remarkable success in various natural language processing tasks, including language modeling, understanding, and generation. However, the increased memory and computational costs associated with these models pose significant challenges for deployment on resource-limited devices. Structural pruning has emerged as a promising solution to reduce the costs of LLMs without requiring post-processing steps. Prior structural pruning methods either follow the dependence of structures at the cost of limiting flexibility, or introduce non-trivial additional parameters by incorporating different projection matrices. In this work, we propose a novel approach that relaxes the constraint imposed by regular structural pruning methods and eliminates the structural dependence along the embedding dimension. Our dimension-independent structural pruning method offers several benefits. Firstly, our method enables different blocks to utilize different subsets of the feature maps. Secondly, by removing structural dependence, we facilitate each block to possess varying widths along its input and output dimensions, thereby significantly enhancing the flexibility of structural pruning. We evaluate our method on various LLMs, including OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2. Experimental results demonstrate that our approach outperforms other state-of-the-art methods, showing for the first time that structural pruning can achieve an accuracy similar to semi-structural pruning.

Yen-Chang Hsu DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models
Shangqian Gao Florida State University
Chi-Heng Lin Samsung Research America
Ting Hua Samsung Research America
Tang Zheng Samsung Research America
Yilin Shen Samsung Research America
Hongxia Jin Samsung Research America
Samsung Research America
Abstract
Large Language Models (LLMs) have achieved remarkable success in various nat- ural language processing tasks, including language modeling, understanding, and generation. However, the increased memory and computational costs associated with these models pose significant challenges for deployment on resource- limited devices. Structural pruning has emerged a promising solution reduce the costs of LLMs without requiring post-pree post-processing steps. Prior structural pruning methods either follow the dependence of structures at the cost of limiting flexibility, or introduce non-trivial additional parameters by incorporating different projection matrices. In this work, we propose a novel approach that relaxes the constraint imposed by regular structural pruning methods and eliminates the structural de- pendence along the embedding dimension. Our cameensoon-nndpendeen structural pruning method offers several benefits. Firstly, our method enables different blocks to utilize different subsets of the feature maps. Secondly, by removing structural dependence, facilitate each block possess varying widths along its input and output dimensions, thereby significantly enhancing the flexibility of structural pruning. We evaluate method various LLMs, including OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2. Experimental results demonstrate that our approach outperforms other state-of-the-ar methods, showing for the first time that structural pruning can achieve an accuracy similar to semi-structural pruning.
Introduction
Large Language Models (LLMs) have revolutionized the field of natural language processing by leveraging deep learning techniques to process and generate human-like text. Compared to smaller models, LLMs exhibit unique characteristics and demonstrate remarkable abilities in tackling a wide range of complex tasks [40]. Despite their impressive capabilities, the vast number of parameters in LLMs often hinders their deployment on resource- conurrcc-cottrainedd devices, such as mobile phones. Consequently, there is significant interest in reducing the computational and memory requirements of LLMss.
Existing compression techniques for large language models (LLMMs) include weight sparsification structural pruning [30], and quantization [10. this work, we focus on structural pruning and
Part of this project was completed at Samsung Research America. Correspondence to sgao fs.fsu.edu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Resiiall Cosmntiion
West
No Dependenny
We
Regular Structural al Pruning with
Camnession-ldeppeude Structural Pruning (Ours)
Structurel Dependency
Figure 1: We use an MLP layer as an example. Left: Regular pruning methods have to follow structural dependence thus their flexibility is limited. Right: Our dimension-independenn structural pruning method breaks the structural dependence via index operations and thus largely improves the flexibility for pruning.
address the limitations of previous methods in this category. Structural pruning [30 is general- purpose compression solution that maintains LLM performance across various tasks, facilitates deployment on devices, and is computationally efficient. However, existing methods may restrict pruning flexibility or add significant overhead to the compressed model. For instance, L.LM-Pruner 30 follows structural dependence during pruning. requiring different layers to use the same subset of feature maps, which limits pruning flexibility. SliceGPT [2] alleviates this issue by applying orthogonal projections for each layer but introduces a non-trivial number of additional parameters (e.g., 5% to 13% of the parameters of the original model for LLaMA-2 7B). Our approach aims to overcome these drawbacks and offer a better performance-com trade-off for structural pruning.
We aim to increase the flexibility of current structural pruning methods and consequently improve performance. Our method provides different sub-spaces or subsets of features different layers, but unlike SliceGPT, it doesn't introduce additional parameters. To achieve this, we break the structural dependence of regular structural pruning methods, allowing different layers to have different subsets of features along the embedding dimension and an example is given in Fig. [ After pruning, we employ index selection and index addition operations to sample subsets of features from the residual connection and add them back after the computation of each layer. Furthermore, our method introduces an additional level of flexibility by learning different widths for each layer. Our approach significantly improves the flexibility of structural pruning without adding additional parameters.
Extensive experimental results show that our method can outperform state-of-the -oe------tt-ttttt structural pruning methods for LLMs while still maintaining low computational costs. Our method does not require recovery fine-tuning to obtain such performance. remained model weights during pruning which is a distinct departure from several other methods, such SparseGPT
addition, our method does not update the
9 and LLM Surgeon [37]. Our contributions are as follows:
We break the structural dependence of regular structural pruning methods, significantly increasing the flexibility of structural pruming. This allows different layers to select their own subset of features from the embedding dimension. Importantly, our method achieves this without introducing additional parameters, unlike SliceGPT.
We propose learn the widths of cach layer using gradient-based optimization methods. A hypernetwork generates the column or row selection matrices, while the width of each layer is controlled globally. This approach allows for fine-grained control over the pruning process and enhances the adaptability of our method to various models and tasks.
Our method demonstrates superior performance compared to state-of-the-art structural pruning techniques for LLMs across range of models, including OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2. Notably, the resulting model from our method is sub-network that exists within the original model, indicating the effectiveness of our method in discovering strong sub-networks.
2 Related Works
Magnitude- pruning is the most straightforward approach to reduce model size, where weights with the smallest magnitude are pruned. Han et al. [14] employ this strategy for pruning with L or L2 norm of weights. Filter pruning (24 extends this setting structures of the model instead of performing weight-level sparsification. Although magnitude-baa based pruning methods are very efficient, they result in significant performance drops for LLMs, even for weight pruning 19). Another

---

23 and Optimal Brain Surgeon [15], utilize second-orde information to remove connections. These methods require calculating the inverse of the Hessian matrix, which is computationally intensive for modern neural network architectures like Convolutional Neural Networks (CNNs) [22] 16], Transformers [8, or Large Language Models (L.LMs) [5]. To reduce the cost of computing the Hessian inverse matrix, Optimal Brain Surgeon can be applied in a layer-wise fashion 7 8 making the computation tractable. However, further scaling up these methods for LLMs remains challenging.
line of research, Optimal Brain Damage
Recent methods like SparseGPT GPTQ [TO] aim to minimize the squared er- ror before and after pruning or quantization of given layer. In this setting, the Hessian inverse matrix becomes easy to compute, as is simply the multiplication between the fea- ture map and its transpose for a given layer. GPTQ and SparseGPT then quantize spar- sify model weights in a column-by-colume manner, and the unpruned unquantized weights are updated to compensate for the er- ror of pruning and quantization. Wanda [34] further avoids computing the inverse of the Hessian matrix by only considering the diag- onal of the Hessian matrix. While SparseGPT and Wanda achieve good results, unstructured sparsity is known to be harder to achieve ac- tual speedup. They also applied their methods on semi-structured settings [31]. but the per- formance becomes much worse.
or
Dntpats
ten Trabiing Heperettnnk
When Froning
Lake Sclat Laver
Adbaates
STWLE
Aal lake spletien
Layar Narm
Mtall-Bual
Inde
Preas wiihht
DISP-LLM (Ours)
lapata
Our method, DISP-LLM, applies different selection matrices to the input and output dimension of the Attention layer and MLP layer (S//S²: Atten- tion in/out; S S3/Sa/S S5: MLP in/middle/out). When pruning the model, add "Index Selection" before Layer Norm and we replace addition with "Index Add." S, ...,, S₁ are applied for pruning weight matrices.
Figure 2:
Several researches 28 apply learnable parameters for specific struc- tures when pruning vision or language models. However, many of these methods cannot be scaled up to LLMS since they need to learn weights and structures together. In contrast, our method explores sub-networks within the original model without updating model weights. Additionally, our method mainly explores the regime of pruning without recovery fine-tuning, which is rarely presented in previous methods with learnable parameters on structures. Our method also related to the unconstrained channel pruning for CNNs [39]. However, our method explores this idea from the perspective of breaking structural dependence and scales much larger models than [39] Moreover, our method thoroughly explores the global allocation of parameters, where [39 fails to do.
Recently, several works have been proposed to reduce the size of LLMs. LLM-Pruner 30 aims to remove connected structures using importance calculated from Taylor expansions. SliceGPT 2 offers more flexibility than regular pruning by projecting the feature maps different spaces but introduces extra parameters in the residuals. LLM Surgeon (37) periodically updates model weights and structures, resulting in a higher cost than LLM-Pruner and SliceGPT. Our proposed DISP-LLM breaks the structural dependence relied on by LLM--Prnee, without additional transformation matrices in the residual connections like SliceGPT. Furthermore, contrast LLM Surgeon, which requires extensive computational resources, our method is significantly more efficient.
3 Preliminary
3.1 Notations
To better understand our paper, we first define some notations. We use denote the model dimension or embedding dimension of LLMs. X e Roxaxd is used to represent feature maps, and b the mini-batch size, n is the number of tokens. W e gddxxda is the model weights of size dj x d2- Let S denote pseudo-index selection matrix of size d d, which is diagonal matrix filled with 0
or 1 and the positions of the ones indicate the selected index. We further use S of size d x d to
3

---

NO
NO
utteggual m
caliins
DISP-LLM
(OKn)))
Figure 3: Comparison of the projection matrices for structural pruning. We use Win and W in Fig. as an example. Left: SliceGPT employs orthogonal projection matrices, and it has to insert the projection matrices into the residual connections. Middle: Regular structural pruning methods remove structures based on their dependence, requiring blocks, which limits flexibility. Right: Our method breaks the structural dependence, allowing the use of different selection matrices S.,, and Sout for the embedding dimension, significantly improving the flexibility of pruning.
the unified selection matrix S for all
represent the actual selection matrix by removing d - dmmmal columns with all zeros from S. For any matrix A, nnz(A) represents the number of nonzero entries of A.
3.2 Revisit SliceGPT
The core idea of SliceGPT [2] to achieve computational invariance within the transformer architecture. It demonstrates that orthogonal projections can be applied to the output of cach block and subsequently undone in the next block. This transformation is computed using Principal Component Analysis (PCA), allowing the feature maps between blocks to be projected into their principal components. A significant advantage of this approach is that projects the feature maps of different blocks into distinct spaces, thereby introducing an additional degree of freedom for compression. This flexibility not captured by regular structural pruning methods like LLM--runer 300, which rely on structural dependence.
After slicing (pruning), the feature map and weight matrix of ith layer of SliceGPT become:
X = X,Q,S, W, = =TQ QTT,,, W
where a damall selection matrix, X₁ is the output the Ith block, and Q/ contains eigenvectors of Cj:
C = X X1.
and Xt,s is the i-th column of X, (corresponding to the ith sequence in the calibration dataset). From Eq. we can see that SliceGPT uses the same selection matrix Š for all layers, but the feature map X firstly projected by Ql. and the pruning for different layers along with different directions. One crucial drawback of SliceGPT also comes from the projection matrix Q since the residual connection must be multiplied by the linear transformation Q Q (shown in Fig. 7 left in the Appendix). These additional operations bring non-trivial amount of additional parameters. For model that has blocks, with the model dimension and the remaining percentage of parameters p E [o, 1]. it brings approximately pd additional parameters to the model (more than 10 % of model parameters in some cases, and more details are given in Fig in in the Appendix).
3.3 Residual Connections Limit the Flexibility of Structural Pruning
SliceGPT offers significant flexibility, but achieving similar flexibility with regular structural pruning methods without adding extra parameters challenging. This section explains the reasons behind this difficulty. To simplify our reasoning, we replace S with its pseudo selection matrix S.
Assume we follow the basic setting of dependence- based structural pruning but allow each layer the flexibility have its own selection matrix, S,, along the embedding dimension. Under this assumption, due to structural dependence, all layers will share the same width of nnz(S,,S, ... S,))
In order to prune different positions for different layers, we need to add a transformation matrix to align the width of layers I and + 1. Intuitively, if we have S, and S we can then insert S, S in the residual connection to align consecutive layers.

---

Algorithm 1: Block inference after pruning.
Input: Feature map of the previous block Xin- Preserved indices sets Ind1, Ind2, Ind3, Inds. Index Selection for Attention X,tt MultiHead( & ST W q: STSTW W. S SW,) S2.
1. Xin LayerNorm| Xin Snl Ind J).
3. X Index Index_Add(2 Xin: Xatn: Ind2).
Index Addition with the input Index selection for MLP
4. X LayerNorm (ess Indal).
Sa) 5. mip (arX W,S.) O (Xies ( T
X Index_Ade (ii__Add(Xpesss X Xmlp. Inds)
to Index Addition with the residual
Return Xour for the next block.
With this setup, we can use X,S; to select subsets of features for different layers. mimicking QiS for SliceGPT. Although it seems promising, this formulation has issues with layer widths, detailed in Proposition |
Proposition (Decreasing feature dimensions for deeper layers). Let the pseudo-selection matrices in layers and + be S and S1+1. respectively. The number of nonzero entries in the residual adapter satisfies
unz( (S S1+1) & min/mne(S)),nnnff
For compression strategies that remove dependent structures for layer I following S, Si+1. this implies that the dimension in layer I + 1 is less than or equal to that in layer with equality holding when the feature indices selected in layer 1 1 are contained within those in layer 1 or vice versa.
Remark. The proof of Proposition. D straightforward and it is given in the Appendix A.1 From Proposition D we observe that we naively apply S₁ for different layers, the model width will we go deeper into the network. It also fails to provide different sets of
progressively decrease features for different layers; instead, it merely passes a subset of features from the previous layer to the next. To avoid this restriction, all blocks must share the same width and the same pruned columns or rows. And we then fall back to the regime of previous structural pruning methods such as LLM-Pruner (30], Shared LLaMA [43) etc.
Proposition highlights two significant obstacles. First, dependence- structural pruning methods result in uniform width along the embedding dimension. Second, inserting selection matrices in the residual connections causes the embedding dimension to decrease with depth. These challenges are unavoidable due to the residual connections linking structures across layers. To enhance flexibility along the embedding dimension, bypassing the residual connections is crucial.
4 Dimension-Independeen Large Language Model
4.1 Break the Structural dependence
3.3 demonstrates that the residual connection is the primary barrier preventing pruning methods from achieving better flexibility. To avoid modifying the residual connection, we relocate the selection matrices inside the residual connection. This approach allows to successfully create different subsets from the feature maps for different layers.
Section
Based on this idea, we propose a solution that involves pruning different positions in consecutive blocks and selecting or merging feature maps from or back to the residual connection. This approach breaks the structural dependence inherent in previous pruning methods. Formally, given transformer block, we apply the following operations:
Attention( X) MultiHead XS W., g XS_ We, XS W, W.S.
(2)
MIP(() (o(XS) WiS4) O (XS] W,S.))S, WWSS,
where Si,.... .S are pseudo selection matrices size x d, and O denotes element-wire -iie multi-
plication. Eq. 3 gives example operation for gated MLP modules used in LLaMA 35 For Phi
models or OPT [6), the MI.P operation defined as MI.P(X) (XS, W,S W3S, Fig 2 illustrates how to insert these selection matrices into transformer block.
5

---

Given the operations defined in End Eq.E, we successfully remove the constraint in Proposition The imput and output of both the Attention layer and the MLP layer can be selected differently from the original feature maps for different layers, mimicking the function of Q in SliceGPT. Additionally, our method eliminates the need for extra parameters in Q Qi+1 as does not alter the residual connection. We also enhance flexibility by pruning the middle dimension of the MLP layer. Additionally, this flexibility can be further improved by allowing the query, key, and value weight matrices to use different selection matrices. Our current form is kept for two reasons: (1) SliceGPT uses one Qi per layer, and we followed this design for a fair comparison, and (2) adding separate selection matrices would increase indexing operations, potentially slowing down the inference. Fig further compares the projection matrices for SliceGPT, regular structural pruning, and the proposed method.
I
Once we have the final selection matrices S,...., S5, the pruned model will use combination of index selection and index addition for inference as shown Algorithm where Ind; a set containing all indices equal to one in the diagonal of S;:
Ind; {j if |1 1}. s; diag(S.). (S.).
The same color is used to mark the index set Ind; and its corresponding selection matrix S, A, B, Ind) adds matrices A and along the last dimension on selected positions from Ind, then returns A after index addition. With index selection and addition, the block dimension can be freely changed. Index selection and addition introduce some overhead, but as demonstrated the experiment section, we still observe improvements in throughput.
4.2 Learning the Width for
LLMs
Building on the dimension-inddeendeer setting introduced in Section our approach offers much greater flexibility in selecting sub-networks from the original dense model compared to the constrained settings
[30]. The next challenge is determining the width of each layer. Given the large search space of our romeesioo-inddpendeen nature of LLMs, it is impractical to use reinforcement learning [17] or evolutionary search-based algorithms 27 Therefore, we adopt bradde methods to address this challenge. Given the diagonal vector s; E {0, 13d from S,, the Straight-Through (ST) gradient estimator 3 is used to estimate the gradients with respect to learnable continuous latent parameters. More specifically, we use the recently proposed gradient estimator ReinMax [26) to estimate the gradients through the binary operation. A detailed explanation of ReinMax for the binary case is provided in Appendix [A.2]
structural pruning and the computationally intensive
Given the large search space of our method, we find that only using element-wist learnable parameters is insufficient. To address this issue, hypernetwork is introduced to generate latent parameters for ReinMax, as detailed below:
ReinMax ReiinMax(HyyerNeeeig (HyMeaeeo(yy))reeooo((()@)))
where θ represents the parameters of the hypernetwork and s contains s; from all blocks. The hypernetwork is composed of GRU [5] and fully connected layers, where the GRU captures block- wise relationships and the fully connected layers capture relationships between different dimensions. With the hypernetwork and ReinMax, we can effectively leam the width of each block. The details of the hypernetwork are provided in Appendix A.3
4.3
Cammension-Indpenndee Structural Pruning as an Optimization Problem
With the methods described above, we can formulate Gimension-indepeaie structural pruning as an optimization problem, with regularization to control the number of remaining parameters. We insert s back into S as defined in section 4.1 for forward and backward calculations. The overall objective function is listed below:
min L(X' W.s) + AR(T(s),
(5)
R(T = log log(max(T(s)) pToma)))) min pTToaa))))))
where C is the language modeling loss function of next word prediction, represents the input tokens, W is the collection of model weights, is defined in Eq. E and R is parameter regularization loss function defined in Eq. 6. Here, T(s) denotes the number of parameters controlled by the current structure s. T Tiotl is the total number of parameters of the model, and p (0, 1] is uscr-defined

---

Table Perplexities of different structural pruning methods on WikiText-2. Our method is the only one that does not update model weights. SliceGPT does not directly update model weights, however, applies orthogonal transformation matrices to the weights.
Porformance (PPL)
Mehhod
Pruning Rañi
W Update?
OPT 125M
OPT 1.3B
OPT 2.7B
OPT
L.aMA-2 7B
I.aMA-2 13B
Dense
0%
27.00
14.62
12.47
10.86
5.12
10%
29.34
15.10
12.75
10.92
5.89
5.21
SlicoPT
2
20%
25%
37.74
34,26
16.43
17.46
14.56
13.73
11.48
11.90
6.64
7.24
6,30
20%
7
29.89
15.63
12.47
11.28
9.14
6.29
K-OBD +
40%
V
47.54
24.65
18.09
16.21
28.03
13.06
30%
75.95
37,68
26.68
25.54
46.64
16.06
20%
V
28.75
15.12
12.27
11,02
6.18
5.29
30%
16.24
12.92
7.83
Surgeon
40%
38.47
18:45
14.23
12,58
10.39
7.25
50%
49.78
22.95
17.15
14,90
15.38
9.43
20%
X
25.21
13.12
11.72
9.89
6.10
5.21
DISP-LLM (Ours)
30%
28.16
14.79
12.16
10.90
6.85
5.77
40%
34.31
17.77
14.11
12.18
8.11
6.59
50%
39.87
21.70
17.07
14.06
9.84
7.11
Table Comparison of our method against semi-structure pruning methods WikiText-2.
Test Performance (PPL)
Method
Pruning Ratio
W Update?
Structure?
LLaMA 7B
LLaMA 13B
LLaMA-2 7B
LLaMA-2 13B
Dense
0%
5.68
5.09
5.12
4.57
Magnitude
42.13
18.37
54,39
8,33
SparseGPT
11.00
9.11
10.17
8.32
Wanda 34
2:4
x
11.53
9.58
11.02
8.27
DISP LLM (ours)
50% X
11.47
8.15
9.84
7.11
parameter control how many parameters should be preserved within the model. With the objective function in Eq. 5 the structures for dimension-independee pruning can be efficiently optimized. Moreover, the overhead of our method minimal and comparable parameter-efficient fine-tuning methods like LoRA [18]. as it does not require storing gradients or the first and second-order momentum of model weights for the Adam optimizer [2
Experiments
5.1 Settings
Models. We evaluate our DISP-LLM method using several LLMs with transformer blocks. Specif-
ically, we choose the following models: OPT [6]: OPT-125M OPT-1.3B, OPT-2.7B, OPT-6. 7B;
Phi-1.5 (25) and Phi-2 20; LLaMA 7B
355: LLaMA-2
36): LLaMA-2 7B and LLaMA-2 13B.
Implementations. We implemented our method using Pytorch 32 and Hugging Face transformer brary 4T) We freeze the model weights W when training the hypernetwork. We use the AdamW (2 optimizer to optimize the hypemetwork. The hypernetwork is trained for 10,000 iterations for all models. For all experiments, we À in Obj 5 6. Depending on the size of the base model, use 1 to 4 NVIDIA A100 GPUs to train the hypernetwork. More implementation details can be found in the Appendix A.4
Following previous papers 2 30, we use WikiText-2 and Alpaca datasets to train the hypernetwork. Following SliceGPT [2] evaluate our method and other methods five well- known zero-shot tasks: PIQA 4: WinoGrande (33]; HellaSwag [45]: ARC-e and ARC-c [6]. We
Datasets.
llm-eval-harness
evaluate the compressed models.
Baselines, We compare our DISP-LLM across basclines from structural pruning like LLM- Pruner [30]. SliceGPT (2] and LLMSurgeon (3]. We also include semi-structure pruning basclines like SparseGPT and Wanda [34].
5.2 Language Modeling
In T we report the perplexity of pruned OPT and LLaMA-2 models. Our DISP-LLM, which does not update weights, consistently outperforms more complex pruning methods such as K-OBD and LLM Surgeon, which involve weight updates, across all pruning ratios and models. The

---

Table Zero-shot performance the compressed LLaMA 7B, LLaMA-2 7 and Phi models. The structure of DISP-LLM is based on the WikiText dataset, and the structure of DISP- LLM Alpaca is based on the Alpaca dataset.
Pruning Ratio
Method
W Update?
WinoCGrande
HellaSwag
ARC-
ARC-c
PIQA
Avg
0%
LLaMA 7B
69.85
76.21
72.81
44.71
79.16
68.35
37.12
75.57
*finetuning
65.11
68.11
63.43
37.88
76.44
62. 19
DISP-LLM (Ours)
66.54
68.75
35.24
74.97
02
DISP-LLM Alpaca (Ours)
64.72
68,39
64.81
37.12
76.66
62.34
LLM-Pruner 30)
53.20
35.64
33.30
27.22
39.63
41.84
*finetuning
55.09
47.56
46.46
28. 28.24
68.82
49.23
DISP-LLM (Ours)
X
$8.41
47.71
44,40
28.50
64.09
48.6 48.62
DISP-LLM Alpaca (Ours)
48.76
31.57
67.46
50.72
0S6
LLaMA-2
69.14
75.99
74.58
46.15
79.11
68.99
SliceGPT 2
61,33
49.62
51.77
31.23
63.55
$1.50
K-OBD 34
56.83
53.07
$1.05
33.11
71.82
53.18
LLM Surgeon
61.09
60,72
63.09
36.69
73.56
59.03
DISP-LLM (Ours)
X
62.27
63.43
59.81
33.19
71.82
58.10 $8.10
DISP-LLM Alpaca (Ours)
37.03
73.72
K-OBD [4]
53.04
36,84
36.11
26.71
60.06
42.67
LLM Surgeon 34
52.57
40.29
44.91
26.28
64,36
68
50%
DISP LLM (Ours)
54.54
46.33
43.06
25.85
63.93
46.72
DISP-LLM Alpaca (Ours)
56.20
49.35
51.14
30.20
68.34
$1.05
0%
Pm-113
72.77
62,58
73.11
48.04
75.63
66.43
S1ssGPTT
64.96
42.54
53.60
31.91
65.45
51.52
30%
DISP-LLM (Ours)
61.48
47.97
$7.66
33,01
71.08
54.24
0%
Phi-2
75.61
73.86
78.24
54.01
79.11
72.17
SlicePPT 2
63.14
47.56
53.03
30.29
65.94
$1.99
30%
DISP-LLM (Ours)
73.34
Pruning Decisions along the Model Dimansion and Depth
Pruning Rate slong the Model Dimansion and the Overall Pruning Rata
Modet Dimeraton
Provureed
Pruned
Degih
Figure 4: The pruned model architecture along the embedding dimension (model dimension) for the LLaMA-2 7B model when the pruning ratio equals 50%.
performance gap is even larger when compared to SliceGPT. The advantage is particularly clear in better-trained models like LLaMA-2 7B and 13B. For instance, method surpasses LLM Surgeon by margins of 5.54 and 2.22 when pruning 50% of parameters of LLaMA-2 7B and 13B, respectively. Against K-OBD, our performance advantage extends to 36.80 and 9.49 under the same setting. For consistency, we let the pruning ratio of SliceGPT equal the slicing ratio. However, the actual pruning ratio for SliceGPT is much lower than the slicing ratio. More details are given in Appendix A.5
In Table 2 we report the perplexity pruned LLaMA and LLaMA-2 models and we compare our see that our method outperforms
method with semi-structure pruning methods. From the table, both SparseGPT and Wanda on LLaMA 13B and LLaMA-2 7B/13B models. Our method performs on par with SparseGPT and Wanda with the LLaMA 7B model, and DISP-LLM is little bit worse than SparseGPT and is similar to Wanda. We are the first to show that structural pruning methods can have better similar performance than semi-structural pruning methods.
5.3 Zero-shot Performance
In Tab. we present the zero-shot performance of the pruned model. For the LLaMA 7B model, we compare our method against LLM-Pruner with and without recovery fine-tuning. Our method consistently outperforms LLM-Pruner without fine-tuning, and the gap ranges from 2.63 to 8.88 across different pruning rates for average task performance. After fine-tuning, the performance of LLM-Pruner is largely boosted, however, our method is still able to outperform it demonstrating the existence of strong sub-networks within the original model. For the ILaMA-2 7B model, we compare our method against SliceGPT, K-OBD, and LLM Surgeon. With weight updates, L.M

---

14
100m
(a) Ablation Loss L
(b) Ablation Loss R
(c) Ablation p
(d) Ablation Iterations
30
Mark
Given p Loss C
(f) Given p Loss R
g Acceleration
(h) Costs
Figure 5: The training dynamics when learning the hypernetwork are shown in Figs. 5a
5
Se
51
The results of different settings are in Figs. 5 5d throughput is in Fig. 5g and cost Fig.
5h
Layors
Figure 6: Model width after pruning for the LLaMA-2 7B model when the pruning ratio equals 50%.
Surgeon performs well with lower pruning ratio like 30% At this pruning ratio, our method performs similarly LLM Surgeon, and our method outperforms other comparison baselines. When increasing the pruning ratio to 50%, the advantage of our method becomes obvious, and the gap between our method and LLM Surgeon is 5.37 for average task performance. We further compare our method with SliceGPT on Phi-1.5 and Phi-2, and our method consistently achieves better performance.
5.4 Analysis
5c 5d with Phi-1.5 Ablation Study. We visualize the results of ablation studies in Figs. 5a 5b
model. The setting "DISP-LLM w/o HN" refers to using clement-wise gates for learning sub- networks. The setting "Constrained LLM w HN" refers to pruning the embedding dimension following the structural dependence like LLM-Pruner. From Figs. 5a 5b
we can see that using the hypernetwork largely accelerates the learning process for DISP LLM. which is also verified in Figs. 5c 5d From Figs. 5c 5d we also see that our DISP-LLM always outperforms constrained structural pruning, demonstrating the value of added flexibility by breaking the dependence. To further study the impact of the HyperNetwork architecture, we provide more results in Tab. E "w/o HN" equivalent to "DISP-LLM w/o HN". The setting "w/o Bi-GRU" simply removes GRU and adds fixed input (initialized in the same way linear layer. These results indicate that both GRU and linear layers within the HyperNetwork affect the final performance. One explanation is that linear layers connect different dimensions of the model, accelerating learning, while GRU layers capture inter-layer relationships, further reducing the difficulty of learning sub-network structures. Therefore, both GRU and linear layers essential to the HyperNetwork.
Appendix A.3 for more details) for each
Different Pruning Ratios, Costs and Throughput. In Figs. 5e 51
we show the language modeling loss L and regularization loss R. in 5lvvn different pruning ratios with Phi-1.5 model. We can that our method consistently minimizes the language modeling loss across different p. In addition,

---

Table 4: The impact of the Hypernetwork architecture on the Phi-1.5 model. Performance measured by PPL (perplexity).
Compression Rate
Settings
0%
10%
20%
30%
40%
50%
w/ HN Bi-GRU
20.37
22.30
28.66
34.33
47.29
21.82
19.90
21.65
26.11
30.88
37.43
Full HyperNetwork
18.75
20.23
22.81
25.49
32.89
our method quickly pushes the regularization loss R to near values within 200 iterations. In Fig. 5g the pruned model from LLaMA-133 improves the throughput of the dense model by 1.08x to x.50x. In Fig. [5h, we compare the costs of our method against LLM Surgeon. Our method is 27.39> x and 14.76x cheaper compared LLM Surgeon with LLaMA-2 7B and LLaMA-2 13B models.
Every Embedding Dimension is Important. In Fig. 4 we visualize the pruning decisions along that all embedding dimensions have been used across different layers. This becomes more obvious in the second right figure of Fig. where we sum all pruning decisions along the depth dimension, and we can see that every embedding dimension is kept around 80% given all layers. We further visualize the model width after pruning for the LLaMA-2 7B model in Fig. where we can see that several layers are more severely pruned than other layers.
the embedding dimension and depth for the I.LaMA-2 7B model,
Conclusion
In this paper, we proposed dimension-indeeeedenn structural pruning for Large Language Models. By breaking the structural dependence imposed by previous compression methods, our method creates sub-networks with a lot more flexibility than regular structural pruning methods and also avoids the overhead brought by SliceGPT. The flexibility of our method is reflected in two perspectives. Firstly, our method provides different subsets of the feature maps for different layers. Secondly, our method freely selects the width for each layer without considering architecture dependence. With dramatically increased flexibility, our method outperforms other structural pruning and semi-structural pruning methods given similar pruning ratios. The novel design of the unconstrained pruning space along with strong empirical performance opens new possibilities for structural pruning for LLMs.
10

---

References
[1] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint orXiv:2404.12211 2024.
[2] Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, 2024.
[3] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 2013.
[4] Yonatan Bisk, Rowan Zellers, Jianfeng Gao. Yejin Choi, al. Piqa: Reasoning about physical common- sense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432 7439, 2020.
[5] Kyunghyun Cho, Bart Van Merriënhner, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Fmcoder-decoder approaches. arXiv preprint arXiv:/4099.0 1259, 2014.
[6] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1EX3vv8803.05453 2018.
[7] Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Advances in Neural Information Processing Systems, pages 4857-48611 2017.
[8] Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances Neural Information Processing Systems, 35:4475-4488, 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models be accurately pruned in one-shot. In International Conference on Machine Learning, pages 00333-10337. PMLR, 2023.
[10] Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023.
[11] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang. and Andy Zou. A framework for few-shot language model evaluation, September 2021.
[12] Shangqian Gao, Junyi Li, Zeyu Zhang, Yanfu Zhang, Weidong Cai, and Heng Huang. Device-wing federated network pruning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pages 12342-12352, 2024.
[13] Shanggian Gao, Zeyu Zhang, Yanfu Zhang, Feihu Huang, and Heng Huang. Structural alignment for network pruning through partial regularization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 17402 1 17412, 2023.
[14] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pages 1335-1 143, 2015.
[15] Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. Morgan Kaufmann, 1993.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Rea, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[17] Yihui Hc, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European conference on computer vision (ECCV), pages 784-800, 2018.
[18] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhi Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0966 2021.
[19] Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pages 304-320, 2018.
[20] Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giomo, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023.
[21] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint MxXiv:142.69988 69800, 2014.
11

---

[22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
[23] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pages 598 605, 1990.
[24] Hao Li, Asim Kadav, Igor Durdanovie, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. ICLR, 2017.
[25] Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giomo, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.0566, 2023.
[26] Liyuan Liu, Chengyu Dong, Xiaodong Liu, Bin Yu, and Jianfeng Gao. Bridging discrete and back- propagation: Straight- through and beyond. Advances in Neural Information Processing Systems, 36, 2023.
(27] Yuqiao Liu, Yanan Sun, Bing Xue, Mengjie Zhang, Gary G Yen, and Kay Chen Tan. A survey on evolutionary neural architecture search. IEEE transactions on neural networks and learning systems, 34(2):550-570, 2021.
[28] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, 2017.
[29] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[30] Xinyin Ma. Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-217200 2023.
[31] Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv e-prints, pages arXiv-2104, 2021.
[32] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems, pages 8024 8035, 2019.
[33] Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9))99-100 2021.
[34] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024.
[35] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
[36] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv: 09288, 2023.
[37] Tycho FA van der Ouderaa. Markus Nagel, Mart Van Baalen, and Tijmen Blankevoort. The IIm surgeon. In The Twelfth International Conference on Learning Representations, 2024.
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems,
30, 2017.
[39] Alvin Wan, Hanxiang Hao, Kaushik Patnaik, Yueyang Xu, Omer Hadad, David Güera, Zhile Ren, and Qi Shan. Upscale: unconstrained channel pruning. In International Conference on Machine Learning. pages 35384-35412. PMLR, 2023.
[40] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgcaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
[41] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Mai, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemiie, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics.
[42] Xidong Wu, Shangqian Gao, Zeyu Zhang, Zhenzhen Li, Runxue Bao, Yanfu Zhang, Xiaoqian Wang, and Heng Huang. Auto-train tnain-once: Controller network guided automatic network pruning from scratch. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16163-
16173, 2024.
12

---

[43] Mengzhou Xia. Tianyu Gao, Zhiyuan Zeng, and Danqi Chen, Sheared Ilama: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference Learning Representations, 2024.
[44] Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Association for Computational Linguisties (ACL), 2022.
[45] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Ammual Meeting of the Association for Computational Linguistics, pages 4791-4800, 2019.
[46] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0000 2022.
13

---

A Appendix
Ostppts
Uviguts
When Pruntag
Arthnatinn
Adiivation
Laver
k/tb>
Lager Norts
Lawar Norza
Muls hoad
Matti Bettiken
Alleetion
0.0>
0,
W5RL
Welts
SliceGPT
Ours
Impots
Inputs
Figure Left: SliceGPT inserts Q Q1+1 the residual connection and brings additional parameters. It also modifies the weights and Layer Norms within the original model. The selection matrix S omitted for consistency. Right: Our method, DISP-LLM, applies different selection matrices to the input and output dimension of the Attention layer and MLP layer (S1/S2: Attention in/out; S3/ S5:: MLP in/middle/out).
Pruning Decisions along the Model Dimension and Depth
Preserved
Pruned
Depih
Figure 8: The pruned model architecture along the embedding dimension (model dimension) for the LLaMA-2 13B model when the pruning ratio equals 50%.
Preeevved Bates
0.6
Layers
Figure 9: Model width after pruning for the LLaMA-2 13B model when the pruning ratio equals 50%.
A.1 Proof of Proposition] I
Proposition (Decreasing feature dimensions deeper layers). Let the pseudo-selection matrices in layers and be Si and S1-1. respectively. The number of nonzero entries in the residual
14

---

Algorithm 2: Binary ReinMax
Input: sigmoid input;
T: temperature; e constant bias.
Output: x: binary vector.
1. :To = sigmoid( + c),
B Simple_binary(Tg))
3. w1 (=)p)))((++t)))))
4. m1
(ar + c)) + (o c)),
5. 2 T0.
π2 stop hoop_gradeet((1 m2)
Return x.
adapter satisfies
Sn+1))(S..)) & minn
For compression strategies that remove dependent structures for layer I following ST S this implies that the dimension in layer I + 1 is less than or equal to that in layer with equality holding when the feature indices selected in layer I are contained within those in layer I or vice versa.
Proof. Consider the pseudo-sclection matrices S and S1+1, both of size d d, and both diagonal matrices. The number nonzero entries in S and S are given by nnz(Si) = ky and nnz(Si+1)) = 1441. respectively.
The product S. S 1 is also diagonal matrix size d x d. Each diagonal entry i) in ST S is the product of the i-th diagonal entry of Sj and the i-th diagonal entry of S1+1. For an entry (i, to be nonzero, both S, i) and ( i) must be nonzero.
Thus, the number of nonzero entries in SJ S1+1. nnz(S, Si-1), is the number of indices i where both Si(i, i) and 1 (i, i) are nonzero. This count cannot exceed the smaller of the total number of nonzero entries in S and S1+1.
Hence,
(S S1+1) < min(nnz(S.)).
This implies that the effective feature dimension will be smaller equal the previous layer. Equality holds if and only if the set of indices corresponding to nonzero entries in S is subset of those in St, or vice versa. This concludes the proof.
In Proposition "remove dependent structures for layer 1 + 1 fol- lowing STS++1 means that the actual selection matrix for layer 1 becomes S/ = ST S1+1. and the structure dependence is cut off by the next residual connection. The pruning for layer 1 + 1 will based on S/+1 instead of S Although this setting partially breaks the structural dependence, it has the limitation that the embedding dimensions will be reduced when going deeper.
A.2 Binary ReinMax
Figure 10: Expected compres- sion rate vs. actual compres- sion rate of our method and Slice-GPT on the LLaMA-7B model.
In this section, we provide details for ReinMax when handling binary variables. The ReinMax in our work can be written as shown that control binary vectors to have all one value at the beginning when learning the sub-network architecture for DISP-LLMs. Through all experiments, we set to 3.0 and 7 to 1.0.
in Algorithm. 2 We add constant bias
A.3 Details of the Hypernetwork
15

---

Table 5: The architecture of hypernetorrk
Input
Table 6: Time costs of our method.
- - GeLU
Model
Time / /GPUs
LEMAL 2 7B
2,41 Hours NVIDIA A S0G
Linear,(128, Outputs = 1,..-,L
LLMMALLAMA 13B
8.83 Hours NVIDIA A 80G
14
(a) WikiText Loss C
(b) WikiText Loss R
(c) Alpaca Loss
(d) Alpaca Loss R
Figure 11: The training dy dynamics when learning the hypereeworr for LLaMA-2 7B model with WikiText and Alpaca datasets.
As we discussed in the paper, the Hypernetwork composed of linear layers and Bi-GRUs, and now we present the architecture of the HN in Tab. is initially sampled from normal distribution, and is then fixed during training. Outputs st are continuous values and it then fed ReinMax to produce the binary vector: RaisMaxxs), where is the collection of from all layers. Nr is the original model width, and equals the embedding dimension for S S2, S3 and Sy.
A.4 More Implementation Details
Additional Training Details. During training the hypernetwork, we use AdamW optimizer to optimize with a constant learning rate 10-3 and weight decay 0.05. We train the hypernetwork for different models, we always set the mini-batchsize to 1 on each GPU. For OPT 6. 7B, LLaMA 7B, and LLaMA-2 7B models, 2 NVIDIA A A100 GPUs, and for LLaMA 13B and LLaMA-2 13B models,
02
4 NVIDIA A100 GPUs. For all the rest models, use NVIDIA 110 GPU. We setp = (0.5, 0.4, 0.3, 0.2, 0.1) when the pruning ratios equals (50%, 40%, 30%, 20%, 10%). For the Alpaca dataset f we use the 'text' column within the dataset which combines the columns of 'instruction' and 'output'. When training the hypernetwork, we again minimize the language modeling loss on the Alpaca dataset instead of applying the training process of instruction fine-tuning:
2010
Figure 12: Preserved rates of the LLaMA-2 13B model across different dimensions. The result is accumulated across all the layers.
Details of Eq. 6. The parameter regularization loss function in Eq. is defined as follows:
log
if > if = if < y
max xr, v) min rr,
R(z, y) log
O log
Since y is fixed, when y. Eq. 6 will decrease v, making it closer to y. Conversely, when ø y. Eq. 66 will increase z, also making it closer to y. Thus, the parameter regularization loss always tries to push the current sub-network achieve the pre-defined parameter budget.
16

---

Table 7: Zero-shot performance of the compressed LLaMA-2 13B model.
Pruning Ratio
Method
W Updale?
WinoGrande
HellaSwag acc-borm
ARC-c
ARC-c
PIQA
Avg
acc-norm
acc-norm
scc-norm
0%
LLaMA-2 13B
72.22
79.39
77.48
49.23
80.47
71.76
SliceGPT
16
K-OBD [4
64.96
64.18
56.23
36.01
74.43
59.16
Surgeon
DISP-LLM (Ours)
K
66.85
70.86
63.80
39.42
74.43
63.07
DISP-LLM Alpaca (Ours)
67.32
70.04
68.98
44.28
77.31
65.59
K-OBD [3]
60.46
55.52
49.62
32.68
70.24
53.70
LLM Surgeon 84
V
65.75
63.04
63.80
37.12
73.01
60.94
40%
DISP-LLM (Ours)
62.67
37.63
73.39
DISP-LIMM Alpaca (Ours)
X
64.25
67.52
66.79
42.75
75.30
63.32
K-OBD
LLM Surgeon
34
V
63.22
56.19
56.19
31.83
68.77
55.24
DISP-LLM (Ours)
59.27
58.63
52.57
33.28
68.77
54.50
LIMP Alpaca (Ours)
59.59
62.39
55.72
37.54
72.20
$7.49
Table 8: Zero-shot performance of the compressed Phi-2 given more pruning rates and settings.
WinoGrande
HellaSwag
ARC-e
ARC-C
PIQA
Pruning Ratio
Method
W Update?
acc
acc-norn
acc-norm
acc-norm
acc-morm
0%
Phi-2
75.61
73.86
78.24
54.01
79.1T
72.17
SliceGPT (
67.80
37.76
58.00
35.32
71.87
58.15
20%
+fine-tuning
V
67.17
54.86
56.61
38.91
71.27
57.76
DISP-LLM (Ous)
X
67.09
62.93
44.11
74.86
63.43
SliceGPT
x
65.35
52.40
53.70
31.66
69.21
54,46
25%
+line-tuning
/
65.19
52.48
52.78
35.49
69.91
55.17
DISP-LIMM (Ours)
X
65.11
59.95
65.93
43.34
74.27
61.72
SliceGPT
30%
+fine-tuning
63.54
49.72
46.38
32.68
66.16
51.70
DISP-ILLM (Ours)
X
65.19
54.43
63.59
38.48
73.34
59.00
Ablation Study Settings. In ablation study 5.4 we removed the hyperntwork, and we revise Eq.
ReinMax(O),
where @ now has the same size of and the parametrization space becomes much smaller: For the constrained setting used in the ablation study 5.4 we simply let S S₂ S3 S5. In section 5.4 calculate the costs of our method and LLM-Surgeon, and the price comes from the official website of Lambda Cloud") We also list the detailed time costs of our method Tab. 6 In Fig. 5b 5 and also in Fig. 11 normalized the parameter regularization loss R with its maximum value make plots more consistent.
Licenses. The licenses for various models and datasets are as follows: LLaMA and LLaMA 2: Licensed under the LLAMA Community License. Phi 1.5 and Phi 2: Licensed under the MIT License. WikiTeet dataset: Licensed under the Creative Commons
Aoribution-Share-Ae
License (CC BY-SA 4.0). Alpaca dataset: Licensed under the Creative Commons Attribution- NonCommercial License (CC BY-NC4.0).
A.5 Additional Results
we show the expected compression rate and the actual compression rate for method and SliceGPT given the I.LaMA-2 7B model. It can be that SliceGPT adds 5% to 13 % parameters of the original model across different pruning rates, which non-trivial for most LLMs. Notably, SIiceGPT with 10% slicing actually adds 3% more parameters to the original model.
SliceGPT compression rates: In Fig. 10
LLaMA-2 13B Results. In Tab. 7 we show the results of the LLaMA-2 13B model given different pruning rates. From the table, we can see that our method consistently outperforms LLM Surgeon under different pruning rates. The advantage of our method becomes more obvious compared to other methods like K-OBD and SliceGPT.
Phi-2 Results. In Tab. we present a more comprehensive comparison of our method compared SliceGPT, Our method outperforms SliceGPT by 5.28 to 7.01 giving SliceGPT with or without
17

---

Table Zero-shot performance of the compressed LLaMA 13B model.
Pruning Ratio
Method
W Updale?
WinoGrande
HellaSwag acc-nerm
ARC-c
ARC-c
PIQA scc-norm
Avg
acc-norm
acc-nom
0%
LLaMA 13B
72.33
79.06
74.62
47.78
80.41
70.88
Magnitude
Stniegtunngg
64.64
71.86
67.59
39.93
76.77
64.15
Pruner
Channel
*fine-tuning
66.38
68.89
62.08
38.
76.55
62.58
LLM Pruner 33
Block
65.11
73.41
68.35
38.40
77.15
64.48
+line-luning
67.88
75.16
71.09
42.41
77.91
66.89 86.89
DISP.LLMM (Ours)
68.75
75.28
70.16
44.80
76.61
67.12
DISP-LLM Alpacn (Ours)
66.54
74.80
69.73
44.71
78.07
66.77
DISP-LLM (Ours)
60.85
57.81
52.51
32.51
68.44
54.42
50%
DISP-LLM Alpaca (Ours)
34,85
71.27
56.20
Table 11: Throughput of the pruned model.
Table 10: Average results with 5 different runs. The result is evaluated on WikiText-2.
Model
Pruning Ratio
Tokens/sconds
0%
227.99
Kankemans: (
20%
245.65
20%
X76
LL-MA-2 13B
30%
273.60
16.72 0.00
2148
02
22 033
40%
310.07
50%
342.09
fine-tuning on the WikiText dataset. These observations again demonstrate the effectiveness of our method, and our method outperforms methods with recovery fine-tuning in several settings.
LLaMA-2 3B Results. In Table we present a comparison of our method against LLM Pruner and magnitude pruning. At a pruning ratio of 20%, our method surpasses the performance of L.LM Pruner both with and without fine-tuning. Remarkably, even when the pruning ratio is increased to 50%, our method continues to outperform the L.LM Pruner Channel and the Magnitude pruning baselines at 20% pruning rate. These results further illustrate our method's ability to identify strong sub-networks within the original dense model.
Architecture of the pruned LLaMA-2 2 13B. In Fig. 8 and Fig. 9 we visualize the pruned architecture of the LLaMA-2 13B model pruned with the Wiki dataset. We have similar observations in section. [5.4] In Fig. 9 we can see that middle to late layers have large pruning rates, especially for the attention layer. Fig. g and Fig. 12 we can also see that the preserved rates for different dimensions are similar, and all embedding dimensions are effectively utilized. More interestingly, similar pruning rates across different dimensions are achieved without adding any regularization or constraints.
Training loss for LLaMA-2 7B. In Fig. I we further visualize the training dynamics of the LLaMA-2 7B model on WikiText and Alpaca datasets, respectively. The regularization loss with the Alpaca dataset decreases little bit faster than the WikiText dataset, probably because the loss value on the Alpaca dataset is smaller. From Fig. Tie we can also see that the language modeling loss continues to decrease when training longer. especially when the pruming ratio is higher.
Lastly, evaluate method the Phi-1.5 model with 5 runs, and report result with mean and standard deviation in Tab. 10 We also measure the throughput of our method on the ILaMA-2 13B model, and the result is shown in Tab. 11
A.6 Additional Analysis
Impact of on Phi-1.5 when pruning 50% of parameters. We show the result in Tab. [2 'NC' means the loss R does not converge, and it is much larger than zero, thus it can not prune the model to the target budget. From the table, we can see that the PPL of the model becomes quite stable it is larger or equal to 6. If À is not large enough, it takes longer to push the loss R to reach near 0 values the model to explore different configurations of subnetworks. other hand, if À is large enough, the loss R will reach zero in several hundred iterations and leave enough time to find the desirable subnetwork. Due to this reason, our method is quite stable across larger values of À as shown in the table.
Table 12: Impact of À on Phi-1.5
Performance
(PPL), Phi-1.5 Dense: 21.82
1=1.0
A=
A=4.0
1=6.0
A=8.0
1=10.
NC
36.39
33.71
32.89
33.31
33.20
and thus leaves less time
18

---

Table 14: Zero-shot task performance vs pruning ratio trade-off off for the LLaMA-2 7B model with the WikiText dataset
Table 13: PPL vs. pruning ratio trade-off for the Phi-2 model.
Test Performance (PPI.), Phi-2 Dense:
10.98
Avg lask acc, LLaMA-2 7B
10%
20%
30%
40%
50%
0%
20%
30%
40%
50%
10.22
10.94
14.46
16.02
20.05
68.99
62.54
58.10
52.63
46.72
Table 15: Zero-shot performance of the compressed LLaMA-2 7B model with LoRA fine- ane-tuning.
WinoGrande
HellaSwag
ARC-e
ARC-c
PIQA
Praning Ratio
Method
Update?
BCc
acc-norm
acc-norm
acc-norm
acc-norm
0%
LLaMA 7B
69.85
76.21
72.81
44.71
79.16
68.35
DISP-LLM Alpaca
56.20
49.35
51.14
30.20
68.34
51.05
50%
#LoRA ft
56.83
53.74
57.49
32.42
70.78
54.25
More results pruning ratio vs. performance trade-offs. We provide the trade-off between the pruning ratio and performance for the Phi-2 and LLaMA-2 7B model below in Tab. [3 and Tab. [4
LoRA fine-tuning 188 of the compressed L.LaM-22 7B model. We follow similar settings of the SliceGPT and the model is fine-tuned on Alpaca. The result is shown in Table. [3] We can see that method can be further boosted by using parameter-cfficcent fine-tuning techniques. Since the performance without soeettning is already good enough, we prefer not to involve this additional process in our method to save time and computational costs.
A.7 Generation Samples
We show the generated text given DISP-LLM and SliceGPT in Tab. T6 The examples are obtained based on removing 20% of the model weights with DISP-LLM and 20% slicing of SliceGPT (10% compression rate). Both models are compressed from LLaMA-2 7B and they are not finetuned. From these two examples, we can see that SliceGPT only generates small part of meaningful content and then starts repeating itself. On the other hand, our method tends to generate more relevant content, and the self-repeating behavior is much less obvious. These observations comply with the performance gap between DISP-LLM and SliceGPT.
A.8 Limitations
Our method explores how to break the structural dependency of pruning for LLMs. Although our method achieves competitive performance, there are still some limitations that are not solved in our current version. The throughput improvements of our method are not consistent across different models, which probably because the current implementation of index add or index select operations with PyTorch is not efficient enough. An alternative implementation of our method is to select weight matrices instead of feature maps. In this approach, we perform matrix multiplication with the full feature map and fill the weight matrices with zeros. If we have custom matrix multiplication implementation that ignores zero rows or columns, may achieve further speed-up. However, this approach primarily focuses the engineering perspective and is beyond the scope of this work. We leave the exploration this alternative implementation to future research.
A.9 Border Impact
Our dimension-indeeenddee structural pruning method significantly reduces the computational and memory requirements LLMs, enabling their deployment on resource-limited devices and lowering energy consumption, thus contributing to environmental sustainability. By facilitating the use of effi- cient LLMs on affordable hardware, our approach democratizes access to advanced AI technologics. However, it is crucial to address potential ethical concerns when reducing the size of LLMs, ensuring privacy and fairness in AI applications with compressed LLMs. Overall, our method fosters a more accessible and sustainable use of AI.
19

---

bismuth tin, bismuth tin, Table 16: Generation samples of LLaMA-2 7B with 20% compression rate
SliceGPT
is very important part of the game. The game played in a 3D environment, the player controls character that can move freely around the environment. The player can interact with the environment by picking up items, using them solve puzzles. The player can also interact with other characters the game. The player can also interact with the environment by picking up items and using them to solve puzzles, The player can also interact with other characters in the game. The player can also interact with the environment by picking up items and using them solve puzzles. The player can also interact with other churacters in the gume. The player can also interact with the environment by picking up items and using them to solve puzzles. The player can also interact with other characters the game. The player can also interact with the environment by picking up items and using them to solve puzzles. The player can also interact with
The moom-tmpperature superconductor is a material that exhibits superconductivity at room temperature CC). The first room temperature superconductor was discovered 1970 by Peter Edwards and his colleagues at the University of Michigan. The material was a mixture of hismuth and tin, and was named hismuth- tin." The discovery of bismuth tin was followed by the discovery of other room temperature superconductors, including bismuth tin, hismuth tin, hismuth tin, hismuth tin, bismuth tin, bismuth tin, bismuth bismuth tin, hismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, hismuth tin, hismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, hismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin. bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismutt tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth bismuth tin, bismuth tin, bismuth tin, bismutth tin, bismuth tin, bismuth tin, bismuth tin
bismuth tin, bismuth tin, bismuth tin, bismuth bismuth
DISP-LLM
Al branch of computer science that seeks to create intelligent machines that can think and learn like humans. The tield was developed in the 1950s and 1960s hy researchers who focused on making machines that could learn and solve problems like people do. The field commercially successful the 1980s and 1990s with the development of the first generation of intelligent agents. The term artificial intelligence was coined by John McCarthy in 1956. He was inspired by the idea of creating a machine that could learn and solve problems like humans do.
The noom-temperature
superconductor is a class of superconductors that exhibit zero resistance at room tem- perature. #E History The room temperature superconductor was discovered 1986 by the Japanese scientist Masamichi Aoki and his colleagues at the University of Tokyo. The discovery made possible by the use of new technique called zero temperature transport measurement ( (TM)) ), which allowed them to measure the resistance of superconductor temperatures low 0.05 K. The discovery was made possible by the of new technique called zero temperature transport measurement ( (ZTM) ), which allowed them to measure the resistance of the superconducter temperatures as low as 0.05 K. ## Discovery The discovery of the room-Iemperature superconductor made possible by the of technique called temperature transport measurement (2TM) )
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. , or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatinn right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] " provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best, judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
20

---

IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims in the abstract/introduction are reflected by the experimental results and other related sections of our method.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
is provided in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
21

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[Yes]
Justification:
It is given in the Appendix.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provided the detailed experimental settings in the experiment section and also in the Appendix to ensure reproducibility of our method.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
Depending on the contribution, reproducibility
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
22

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: Due to the company policy, the code will only be released after going through the internal review process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
public//giide//Cdelabbiisss iow.ssobisssioooiooooooioo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //ipsscc/pput cttp:/scccpubblcccuiiiiie guides http:///lwl/liiiiiee/oo/euuaaaaa
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
The details are provided the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We follow the settings of previous methods where the result of single run is provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: This has been provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We followed the NeurIPS Code of Ethics in preparing our manuscript.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Provided in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Provided in the Appendix.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "Y6ucCLiElW", "reviewer": "Reviewer_FhgQ", "summary": "Structural pruning is a method to prune the weights of large language models (LLMs) while keeping their original performance as much as possible it can. However, structural pruning has limitations caused by depending on the structure, like the residual connection of LLMs. In this work, the authors propose a new pruning method, dimension-independent structural pruning for LLMs (DISP-LLM), that can ignore structural dependence in pruning. This characteristic of DISP-LLM makes it possible to prune their weights more flexibly and provides new pruning ways using different feature maps between blocks and varying pruning dimensions of the input and output of the block. However, just pruning the dimensions and ignoring residual connections causes inconsistency between the input and output of a block. To solve this problem, the authors apply transformation matrices to each residual connection to absorb these discrepancies. Furthermore, pruned indices are optimized through training. Experimental results on the language modeling dataset WikiText-2 and the zero-shot task datasets PIQA, WinoGrande, HellaSwag, ARC-e, and ARC-c show that DISP-LLM tends to outperform conventional approaches in pruning LLMs, OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2.", "questions": "- How did you decide to set $\\lambda$ to 6?\n- How are the performances sensible to the changes of $\\lambda$?\n- As you state in the paper, DISP-LLM is as efficient as LoRA. If so, can applying LoRA weights only to the kept dimensions be a baseline?", "limitations": "DISP-LLM requires training and hyper-parameter tuning for $\\lambda$ that controls mixing ratios of losses.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed dimension-independent structural pruning for LLMs (DISP-LLM) can drastically prune dimensions due to its flexibility.\n- Adding transformation matrices to each residual connection to maintain input and output consistency in each transformer is novel\n- In DISP-LLM, pruned dimensions can be optimized through training.\n- Experiments are comprehensive and cover various kinds of LLMs.\n- DISP-LLM can achieve comparable or better perplexities to conventional pruning approaches that require updating the original weights by training.\n- In many cases, DISP-LLM outperforms conventional pruning approaches in zero-shot tasks even though some of them update the original weights by training.\n- The authors report the actual costs for pruning models and show the cost efficiency of DISP-LLM.", "weaknesses": "- Even though DISP-LLM does not update the original weights of LLMs, training is required.\n- DISP-LLM requires hyper-tuning for $\\lambda$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the Reviewer FhgQ for your insightful and constructive comments. We will address each of your questions below. We use W1 to refer to weakness 1 and Q1 to refer to question 1.\n\n**W1. Even though DISP-LLM does not update the original weights of LLMs, training is required.**\n\nIn fact, the training of our method is very lightweight, as we pointed out in the response to **Q3**. Moreover, with a small training costs, our method largely outperforms previous pruning methods which suggests that our method is very cost-effective. In addition, freezing model weights give us potential opportunities to further reduce the costs. For example, we can quantize the original model weights to lower bits to further reduce its GPU memory costs. After pruning, we can restore the model to the original precision. If a method needs to train model weights during pruning, such as LLM Surgeon, it must implement the complex pipeline of quantization-aware training. In contrast, our method does not require this because we do not update the weights of the original model.\n\n**W2 & Q1 & Q2. How to decide $\\lambda$ to 6 and the performance sensitivity to $\\lambda$.**\n\nPlease refer to the first section of the general response. Overall speaking, the hyperparameter selection of our method is not complex.\n\n**Q3. As you state in the paper, DISP-LLM is as efficient as LoRA. If so, can applying LoRA weights only to the kept dimensions be a baseline?**\n\nIndeed, the trainable parameters of our method are around 2-3% of the original model weights which is at a similar scale to LoRA. Though the training cost is similar, applying LoRA does not remove any model weights or reduce the model size. Thus, it may not be a useful baseline. If we apply LoRA weights to the pruned model, then it becomes recovery finetuning, which is definitely doable and we present some results here:\n\n**LoRA finetuning of the compressed LLaMA-2 7B model.**\n\nMethods | Pruning Ratio | W Update? | WinoGrande | HellaSwag | ARC-e | ARC-c | PIQA | Avg\n---|---|---|---|---|---|---|---|---\nDISP-LLM Alpaca | 50% | No | 56.20 | 49.35 | 51.14 | 30.20 | 68.34 | 51.05\nw LoRA ft | 50% | Yes | 56.83 | 53.74 | 57.49 | 32.42 | 70.78 | 54.25\n\nWe follow similar settings of the SliceGPT and the model is fine-tuned on Alpaca. We can see that our method can be further boosted by using parameter-efficient finetuning techniques. Since the performance without fine-tuning is already good enough, we prefer not to involve this additional process in our method to save time and computational costs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dPsRyPQktC", "reviewer": "Reviewer_6CTn", "summary": "This paper proposes a novel structural pruning method for LLMs, primarily based on SliceGPT. It characterised in (i) removing structural dependence by facilitating each block to possess varying widths along its input and output dimensions (ii) no need for introducing addition paramaters like sliceGPT.  The empirical studies show that it demonstrates superior performance compared to state-of-the-art structural pruning techniques for LLMs across a range of models, including OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2.", "questions": "1.  When training with the pruning objectives all the layers are fine-tuned or only selected layers?\n2. Any insights from Figure 6, different model width as layer changes. If the observations aligned with existing research results?", "limitations": "Pay attention to the generated sequence after model prune because the prediction changes are unpredictable.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is easy-to-read and topic is very important to efficient Machine Learning.\n2. The proposed method balances the task performance and efficient, which are both beneficial to its practical and wide application.\n3. The empirical results show its promising in model compressing across four models.", "weaknesses": "1.  More comprehensive literature review about model compressing, like sparse mechanism.\n2. Any case study for qualitative analysis to the pruned model? It is possible to conduct some interpretability analysis showing where the difference between sliceGPT and the proposed method along with a specific input example.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the Reviewer 6CTn for your insightful and constructive comments. We will address each of your questions below. We use W1 to refer to weakness 1 and Q1 to refer to question 1.\n\n**W1. More comprehensive literature review about model compressing, like sparse mechanism.**\n\nThanks for pointing this out. We will add more related works regarding sparse mechanisms, like ‘The Lottery Ticket Hypothesis’ and other works related to sparse mechanisms and model compression.\n\n**W2. Any case study for qualitative analysis to the pruned model? It is possible to conduct some interpretability analysis showing where the difference between sliceGPT and the proposed method along with a specific input example.**\n\nWe provided two examples in Table 1 of the rebuttal pdf file. The examples are obtained based on removing 20% of the model weights with DISP-LLM and 20% slicing of SliceGPT (~10% compression rate). Both models are compressed from LLaMA-2 7B and they are not finetuned. From these two examples, we can see that SliceGPT only generates a small part of meaningful content and it then starts to repeat itself. On the other hand, our method tends to generate more relevant content, and the self-repeating behavior is much less obvious. These observations comply with the performance gap between DISP-LLM and SliceGPT.\n\n**Q1. When training with the pruning objectives all the layers are fine-tuned or only selected layers?**\n\nWe do not fine-tune model weights. Throughout the whole training process, only the hypernetwork is trained and thus the total trainable parameters of our method are quite small which is usually around 2-3% of the original model weights. The hypernetwork is only responsible for controlling the selection matrices of different layers as shown in Figure 2.\n\n**Q2. Any insights from Figure 6, different model width as layer changes. If the observations aligned with existing research results?**\n\t\nFrom Figure 6, we can observe that the middle layers of the model are better kept. This may comply with some interpretability works. In [1], they argue that the middle layer of the model is likely to contain interesting, abstract features. \n\nOn the other hand, our method does not retain as much of the first and last layers as LLM Surgeon does. This is probably the result of removing structural dependency so that each layer has a larger flexibility.\n\nWe believe that future works are needed to fully understand the importance of different layers under our dimensional independent setting.\n\n[1] Templeton, A. (2024). Scaling monosemanticity: Extracting interpretable features from Claude 3 sonnet. Anthropic.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PyLqnt2lgg", "reviewer": "Reviewer_AWnP", "summary": "DISP-LLM introduces dimension-independent structural pruning for LLMs, breaking structural dependencies to allow flexible pruning across layers and dimensions. It uses a hypernetwork for pruning decisions and claims superior performance over existing methods on various LLMs without weight updates or additional parameters.", "questions": "- How sensitive is DISP-LLM to hypernetwork architecture and hyperparameters?\n- Can you provide detailed computational cost comparisons with baselines?\n- How does it perform on fine-tuning tasks?\n- Have you explored progressive pruning approaches?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Novel approach to structural pruning in LLMs\n- Comprehensive evaluation across multiple LLM architectures\n- Theoretical foundation provided\n- Achieves good performance without weight updates or extra parameters", "weaknesses": "- Limited novelty beyond combining existing concepts\n- Inconsistent performance gains across models/tasks\n- Insufficient ablation studies and hyperparameter sensitivity analysis\n- Lack of detailed computational efficiency comparisons\n- Inadequate error analysis and failure case exploration\n- Overstated claims based on limited experiments\n- Insufficient discussion of pruning ratio vs. performance trade-offs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the Reviewer AWnP for your insightful and constructive comments. We will address each of your questions below. We use W1 to refer to weakness 1 and Q1 to refer to question 1.\n\n**W1. Limited novelty beyond combining existing concepts**\n\nOur method is novel because it introduces a new pruning configuration by breaking structural dependencies, significantly expanding the potential search space for an optimal sub-network which is not presented in existing works. This increased search space makes most methods that rely on static metrics (such as weight norms) ineffective. Additionally, reinforcement learning or evolutionary search algorithms struggle in this context due to the difficulty of managing fine-grained search requirements, such as deciding whether to prune a single dimension. To address this, we employ gradient-based search as a highly efficient tool. From this perspective, our method is novel because we demonstrate that increasing the capacity (using latent hypernetworks) for gradient-based search is a highly effective approach for handling large and complex search spaces, such as DISP-LLM.\n\n**W2. Inconsistent performance gains across models/tasks**\n\nWe believe it is reasonable that the performance gains for different models are different, as they are pre-trained with different data and computational costs. Models trained with higher costs are naturally harder to prune. For instance, consider the perplexity of OPT 6.7B and LLaMA-2 7B across different compression rates. It is possible to achieve lower perplexity than the dense model with OPT 6.7B (9.89 vs. 10.86), whereas it is impossible for LLaMA-2 7B (6.10 vs. 5.12), both of which are at a 20% compression rate. LLaMA-2 7B is often regarded as a better pre-trained model than OPT 6.7B, and it is also more challenging to prune. Thus, the performance of our method is naturally different for these two models.\n\nWe also think that performance gains across different tasks should vary since different tasks have different emphases. A single, small dataset like Wikitext or Alpaca cannot cover all perspectives, resulting in varied performance gains across different tasks.\n\nIf the above explanation does not address your concern about inconsistent performance gains across models/tasks, could you please provide more details? We would be happy to address any specific aspects you are referring to.\n\n**W3 & Q1: Insufficient ablation studies and sensitivity analysis for hypernetwork architecture and hyperparameters**\n\nPlease see the first part of the general response.\n\n**W4 & Q2: Lack of detailed detailed computational cost comparisons.**\n\nWe listed the detailed computational cost comparison below:\n\n**Computational Costs comparisons**\n\nCosts | DISP-LLM | LLM Suregon\n---|---|---\nLLaMA-2 7B | 2h25m / 2 NVIDIA A100 80G | 16h58m / 4xH100 80 GB\nLLaMA-2 13B | 8h50m / 4 NVIDIA A100 80G | 1d6h5m / 8xH100 80 GB\n\n\nThe computational cost of our method is also listed in Table 5 in the Appendix. Compared to LLM Suregon, our method is 14.76 x to 27.39 x cheaper in terms of US dollars. \n\n**W5. Inadequate error analysis and failure case exploration**\n\nPlease see the second part of the general response.\n\n**W6. Overstated claims based on limited experiments**\n\nWe believe our experimental results are accurate and not overstated. Firstly, our method surpasses semi-structural pruning methods in terms of perplexity at the same compression rate. To the best of our knowledge, this is the first time that a structural pruning method outperforms semi-structural pruning methods in perplexity. Additionally, as mentioned in the contribution section, our method outperforms other baseline methods, with or without fine-tuning, without requiring updated model weights. This claim is also well reflected in the experimental section.\n\nIf there are specific claims or aspects of our experiments you find overstated or insufficiently supported, please let us know. We would appreciate more details so we can address your concerns thoroughly.\n\n**W7. Insufficient discussion of pruning ratio vs. performance trade-offs**\n\nPlease see the third section of the general response.\n\n**Q3. How does it perform on fine-tuning tasks?**\n\nWe further evaluate our method on the Vicuna 7B model, which is a model trained by fine-tuning the LLaMA 7B model. We list the results below:\n\n**Average zero-shot task performance with the Vicuna 7B model**\n\nMethods | Pruning Ratio | W Update? | WinoGrande | HellaSwag | ARC-e | ARC-c | PIQA | Avg\n---|---|---|---|---|---|---|---|---\nVicuna 7B | 0% | - | 67.40 | 70.64 | 65.11 | 41.21 | 77.75 | 64.42\nLLM-Pruner | 50% | No | 50.28 | 34.86 | 33.29 | 27.3 | 59.79 | 41.10\nw finetuning | 50% | Yes | 54.54 | 46.79 | 48.15 | 29.78 | 69.84 | 49.82\nDISP-LLM Alpaca (Ours) | 50% | No | 55.72 | 46.57 | 52.31 | 30.38 | 66.70 | 50.34\n\nThe result above is consistent with pre-trained models, and our method can be seamlessly applied to models fine-tuning on different tasks. \n\n**Q4. Have you explored progressive pruning approaches?**\n\nFor now, we have not explored the progressive pruning approach. Progressive pruning typically involves updating model weights through either full fine-tuning or LoRA fine-tuning, which would significantly increase the computational costs of our method. Given the current strong performance of our method, we aim to maintain its low cost without introducing an additional weight training process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rWal3zoDwd", "reviewer": "Reviewer_BGu5", "summary": "This paper concerns about the dimension dependence of hidden states across different layers. It have been argued that previous studies with dimension dependence would be inefficient due to additional parameters or architectural constraints. This work breaks the dimension dependence by introducing indexing operations to previous methods. Although simple, the proposed method is shown to be more effective than baselines due to permitted asymmetry across layers.", "questions": "N/A", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. It is for the first time highlighted that the dimension dependence may affect the performance of pruning.\n2. The proposed method to alleviate the dimension dependence is rather simple and effective.", "weaknesses": "1. Although the proposed method allows asymmetric dimensions in module weights across layers, it also requires the dimensions of hidden states from different layers maintain the same as that of the non-pruned model. This would restrict the use in scenarios where the dimension of the output hidden states is critically important (e.g., embedding for retrieval).\n2. The introduced indexing operation could be very time-consuming. Unexpectedly, the latency introduced by the indexing may outweight the latency reduced by pruning. And it is not discussed whether the pruned model is faster than the non-pruned one. I would recommend including such experiments in proper positions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the Reviewer BGu5 for your insightful and constructive comments. We will address each of your questions below. We use W1 to refer to weakness 1.\n\n**W1. Although the proposed method allows asymmetric dimensions in module weights across layers, it also requires the dimensions of hidden states from different layers maintain the same as that of the non-pruned model. This would restrict the use in scenarios where the dimension of the output hidden states is critically important (e.g., embedding for retrieval).**\n\nIn our method, the model dimension (the dimensions of hidden states) remains unchanged after pruning, which is also the prerequisite for dimension-independent pruning. Consequently, our model can serve as a drop-in replacement for the original dense model in any downstream tasks, such as retrieval, as the feature map dimension will stay the same as the original model.\n\nIf this does not address your concern about the dimensions of hidden states and their impact on scenarios like embedding for retrieval, could you please provide more details or clarify your point further? We want to ensure we fully understand your concern and address it appropriately.\n\n**W2. The introduced indexing operation could be very time-consuming. Unexpectedly, the latency introduced by the indexing may outweight the latency reduced by pruning. And it is not discussed whether the pruned model is faster than the non-pruned one. I would recommend including such experiments in proper positions.**\n\nWe present the acceleration results in Figure 5 (g) and Table 10 for the LLaMA-2 13B model, demonstrating that our model is faster than the non-pruned model. While we observe some speed improvements with our current model, we believe there is significant room for further enhancement, as the current PyTorch implementation of indexing operations is not efficient enough. Additionally, the theoretical FLOPs of indexing operations are very low. This is also discussed in the Limitation section of our paper. We believe the speed of our method can be further improved by incorporating a custom implementation of the matrix multiplication algorithm, where pruned columns and rows are skipped instead of using indexing operations. However, this approach primarily focuses on the engineering perspective and is beyond the scope of this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
General Response to Reviewers:

We would like to thank all reviewers for their insightful comments. We are greatly encouraged to see that all of you hold a positive evaluation of our work. We will address some common comments in the general response. We also put part of the individual response here due to the space limitation. All relevant results within the Rebuttal will be added to the final version of our paper. 

**1. How to select $\lambda$ and how  $\lambda$ affects the performance and insufficient ablation study. (Reviewer FhgQ and Reviewer AWnP):**

We present the perplexity of Phi-1.5 for various values of $\lambda$, as shown in the results below:

**The impact of Different  $\lambda$ on the Phi-1.5 model**

  $\lambda$ | 1 | 2 | 4 | 6 | 8 | 10  
---|---|---|---|---|---|---
PPL | NC | 36.39 | 33.71 | 32.89 | 33.31 | 33.2

‘NC’ means the loss R does not converge, and it is much larger than zero, thus it can not prune the model to the target budget. From the table, we can see that the PPL of the model becomes quite stable if it is larger or equal to 6. If  $\lambda$ is not large enough, it takes longer to push the loss R to reach near 0 values and thus leaves less time for the model to explore different configurations of subnetworks. On the other hand, if  $\lambda$ is large enough, the loss R will reach zero in several hundred iterations and leave enough time to find the desirable subnetwork. Due to this reason, our method is quite stable across larger 
values of  $\lambda$ as shown in the table.

We add the ablation studies regarding the hypernetwork architecture below:

**The impact of Hypernetwork architecture on the Phi-1.5 model. Performance is measured by perplexity**

Settings/Compression Rate | 10% | 20% | 30% | 40% | 50%   
---|---|---|---|---|---
w/o Linear layers and GRU | 20.37 | 22.30 | 28.66 | 34.33 | 47.29
w/o Bi-GRU | 19.90 | 21.65| 26.11 | 30.88 | 37.43
Full Hypernet | 18.75 | 20.23 | 22.81 | 25.49 | 32.89

For the 'w/o Bi-GRU' setting, we simply remove GRU and add a fixed input (initialized in the same way as $z$ see Appendix A.3 for more details) for each Linear layer. The results indicate that both GRU and Linear layers affect the final performance. A reasonable explanation is that Linear layers connect different dimensions of the model, accelerating learning, while GRU layers capture inter-layer relationships, further reducing the difficulty of learning subnetwork structures. And thus both of them contribute to the final performance.

**2. Inadequate error analysis and failure case exploration (Reviewer AWnP)**

There is one failure case that we want to mention in the response which is shown in the first part of the general response. In the table, we can see that if  $\lambda$= 1 ('NC' not converge for R loss), our method fails to push R loss to near 0 values (<0.01), and thus it can not prune the model to the given budget. This suggests that we should avoid selecting a too small value of  $\lambda$. For the rest of the $\lambda$ values, our method is robust.



**3. Insufficient discussion of pruning ratio vs. performance trade-offs (Reviewer AWnP)**

In the experimental section, we already provided the pruning ratio vs. perplexity for the Phi-1.5 model at 5 different pruning rates as shown in the blue line (DISP-LLM w HN) in Figure 5. (c). Furthermore, the pruning ratio vs. perplexity for OPT and LLaMA-2 models are given in Table 1. We further provide the trade-off between the pruning ratio and performance for the Phi-2 and LLaMA-2 7B model below:

**PPL vs compression rate trade-off for the Phi-2 model**

Compression Rate | 0%  | 10% | 20% | 30% | 40% | 50%   
---|---|---|---|---|---|---
PPL | 10.98 | 10.22 | 10.94 | 14.46 | 16.02 | 20.05

**Zero-shot task performance vs compression rate trade-off for the LLaMA-2 7B model with the WikiText dataset**

Compression Rate  | 0% |  20% | 30% | 40% | 50%  
 ---|---|---|---|---|---
 Avg acc | 68.99 | 62.54 | 58.10 | 52.63 | 48.74

We believe with these results, the readers should have a better understanding of the trade-off between performance and compression rates for our method.

Meta-review: This paper introduces an innovative method of structural pruning that does not rely on structural dependence among embedding dimensions. The dimension independency nature of this method can significantly increase the flexibility of structural pruning. Experimental results demonstrate its effectives on various open-source LLMs by improving upon state-of-the-art methods. Overall, this is a solid work, and its contribution is appreciated by reviewers. The rebuttal addressed the reviewer concern by clarifying that this approach requires lightweight training and hyper-parameter selection. The consensus among reviewers emerged through rebuttal and discussion. Authors are encouraged to include these additional results and discussions into the final version.
Decision: Accept (poster)