ShortGPT: Layers in Large Language Models are More Redundant Than You Expect

## Abstract

Abstract As Large Language Models (LLMs) continue to advance in performance, their size has increased significantly, with current LLMs containing billions or even trillions of parameters. In this study, we identify notable redundancy across the layers of LLMs, where some layers contribute minimally to overall network functionality. To quantify this, we introduce a metric called Block Influence (BI) which use the similarity between layer’s input and output to measure the importance of each layer. Based on the observation of layer redundancy, we propose a straightforward pruning method: layer removal, which eliminates redundant layers based on their BI scores. Our approach, termed ShortGPT, demonstrates superior performance over previous state-of-the-art pruning methods. Moreover, ShortGPT is orthogonal to quantization-like methods, enabling further reduction in parameters and computation. The ability to achieve better results through simple layer removal, as opposed to more complex pruning techniques, suggests a high degree of redundancy across layers, not only in transformer models but also in non-transformer models. We hope this work will contribute to future research in LLM compression.

## 1 Introduction

The field of large language models (LLMs) has witnessed rapid development recently, with LLMs achieving impressive performance across various domains. Guided by the scaling laws identified in prior work (Kaplan et al., ; Hoffmann et al., ), current LLM research tend to increase model parameters to boost performance. As a result, modern LLMs, which can comprise billions to trillions of parameters, require significant hardware resources for deployment, creating substantial barriers to their practical use.

To mitigate the hardware demands of large models, model compression techniques have become a critical area of focus (Zhu et al., ). These techniques are generally divided into quantization (Liu et al., ; Gholami et al., ; Dettmers et al., ; ) and pruning(LeCun et al., ; Han et al., ; Frantar & Alistarh, ). Quantization reduces the precision of model parameters, but its effectiveness often requires specific hardware support. In contrast, pruning method removes redundant parameters to decrease the model’s size and computation, offering a more flexible and hardware-agnostic approach. Despite its advantages, many existing pruning methods are complex; for example, some require gradient information (Ma et al., ), which limits their practicality.

In this paper, we focus on the issue of layer redundancy in LLMs and propose a novel approach for simplifying these models. We introduce Block Influence (BI), a metric that quantifies how much the hidden state changes after passing through each layer, providing a more direct measure of a layer’s importance. Leveraging this insight, we propose a simple yet effective pruning method ShortGPT, which identifies and removes layers with lower BI scores, significantly reducing model size without sacrificing much performance.

To evaluate our approach, we conducted evaluation across comprehensive benchmarks. Our experiments revealed that our method exhibits a smaller performance decrement compared to the previous methods. For instance, removing 10 layers (25% of the total 40 layers) from the LLaMA 2-13B model resulted in only a slight drop in performance on the MMLU benchmark (Hendrycks et al., ), from 55.0 to 52.2. Our findings highlight substantial redundancy in current LLMs and suggest potential avenues for improving the efficiency of model training by reducing inherent redundancy in the future.

The main contributions of our paper are summarized as follows:

- •
We analyze the redundancy in large language models (LLMs) and find that they exhibit significant redundancy at the layer level. This finding inspire us to prune LLMs by simply removing redundant layers.
- •
We propose a metric called Block Influence (BI) as an indicator of layer importance. Based on BI, our layer removal method maintains approximately 90% performance while reducing approximately 25% of parameters, outperforming previous state-of-the-art methods.
- •
Furthermore, we demonstrate that our layer pruning approach is orthogonal to quantization methods, meaning it can be combined with quantization techniques to further reduce the deployment overhead of LLMs.

Figure: (a) Perplxity
Refer to caption: x1.png

## 2 Motivation

### 2.1 Background

The predominant LLMs are primarily based on the Transformer architecture (Vaswani et al., ), with the pre-norm configuration being the most commonly adopted, as in models like LLaMA (Touvron et al., ). The pre-norm configuration, where layer normalization is applied before the self-attention and feed-forward layers, offers several advantages such as faster convergence, improved training stability, and better scalability for deeper networks (Xiong et al., ; Liu et al., ; Wang et al., ). Due to these benefits, the pre-norm approach has been adopted even in non-transformer models, such as Mamba (Gu & Dao, ) and RWKV (Peng et al., ). For the sake of simplicity in descriptions, our analysis primarily focuses on the Transformer architecture, though we extend our experiments to non-Transformer structures in Section 4.4.

Figure: Figure 2: The cosine similarity between a layer’s input and output during the training process. The horizontal axis (X-axis) represents the number of training tokens, while the vertical axis (Y-axis) depicts the degree of similarity. Notably, the model employing post-normalization exhibits divergence after approximately $\sim$26B tokens of training. Training setting is provided in E.
Refer to caption: x3.png

However, we observe that when pre-norm is adopted, the similarity between the input and output of transformer layers tends to be higher, as illustrated in Figure 2. This high similarity indicates that certain layers induce minimal changes to the hidden states, suggesting they contribute little to the model’s overall function. A detailed mathematical explanation for this phenomenon is provided in Appendix A. Which suggests that the deep layers of the model with pre-norm might not play a critical role in the overall function, and that the layers in large language models could be more redundant than expected, which motivates the layer-removal based pruning method we explore in the next section.

### 2.2 Layer redundancy

**Table 1: Ablation of removing FFN and Attention of Llama2-7B-Base. We sample 100 instances from PG19 (Rae et al., ) to calculate PPL.**
| Delete | PPL |
| --- | --- |
| None | 7.60 |
| The whole last layer | 13.37 |
| Attention of the last layer | 7.65 |
| FFN of the last layer | 12.35 |

As discussed in the previous section, we speculate that the LLMs exhibit layer redundancy. To verify this, we assess the performance degradation caused by removing individual layers of two popular models, Llama2-7B-Base (Touvron et al., ), an English based LLMs, and Baichuan2-7B-Base (Yang et al., ) which is mainly focused on Chinese. Figure 1 confirms our speculation, which reveals that some layers do not play a crucial role in LLMs, causing little degradation when omitting them individually. Moreover, this redundancy is primarily manifested in the middle to later layers of the network, with the initial layers and the last layer often being more critical. Notably, we found the last layer to be particularly important, aligning with findings from LLM Pruner (Ma et al., ). This observation contradicts our mathematical explanation in Appendix A which suggests that deeper layers tend to be more redundant. We posit that this discrepancy arises because the final FFN effectively functions as part of the token classifier and should be considered in conjunction with the language model head.To verify our hypothesis, we conducted further investigation, detailed in Table 1. The results show that within the last layer, the FFN component is crucial, while the Attention module is less significant. This finding supports our interpretation of the final layer’s importance.

## 3 Methodology

In this section, we present the methodological framework of our layer removal approach for LLMs, elucidating the underlying principles and techniques employed. We begin by introducing Block Influence (BI), a novel metric designed to assess the hidden states transformation of each layer. Leveraging BI, we then detail our layer removal method.

### 3.1 Layer importance

As outlined in the preceding section, the layers of LLMs exhibit redundancy, with varying degrees of redundancy across different layers. To capture this, we introduce a new metric, Block Influence (BI), to measure the degree of transformation performed by each layer. The BI score of $i^{th}$ layer can be calculated as follows:

$$ $\displaystyle\text{BI}_{i}=1-\mathbb{E}_{X,t}\frac{X_{i,t}^{T}X_{i+1,t}}{||X_{ i,t}||_{2}||X_{i+1,t}||_{2}},$ (1) $$

where $X_{i,t}$ means the $t^{th}$ row of hidden states of $i^{th}$ layer. Lower BI score imply that $X_{i}$ and $X_{i+1}$ exhibit high cosine similarity, suggesting that the layer makes minimal transformations to the hidden states and is therefore less important. We plot the BI scores of a single layer and the PPL after removing it separately, as shown in the Figure 3. The results demonstrate a positive correlation between the BI score and the importance of a layer.

Figure: (a) Llama2 7B
Refer to caption: x4.png

### 3.2 Layer Removal

Our goal is to obtain a pruned model that remains as close as possible to the original model. Since an LLM functions as a series of transformations applied to hidden states across its layers and we can determine the importance of each layer, we propose a straightforward pruning method: layer removal, which we refer to as ShortGPT. We delete certain layers in LLMs based on BI score. First of all, we construct a calibration set, which is a set of unlabelled text samples such as PG19 (Rae et al., ).
Then we collect the hidden states of each layer during inference on these samples. Next, we calculate the BI score based on the collected hidden states. Finally, we sort layers in ascending order according to the BI, and delete the layers with the lower BI score. The number of layers to be deleted can vary to trade off the speed and performance. The details of our layer removal setting can be found in Appendix D.

## 4 Experiments

### 4.1 Experimental Setup

#### Models.

To validate the effectiveness of our method, we conducted experiments on existing popular open-source language models, including Llama2-7B (Touvron et al., ), Llama2-13B, Baichuan2-7B, and Baichuan2-13B. They are all large language models based on the decoder-only Transformer architecture. LLaMA 2 was trained on more than 2 trillion tokens. Baichuan-series was mainly trained in Chinese and its 13-Billion model replaced the RoPE (Su et al., ) positional embedding with ALiBi (Press et al., ).

#### Benchmarks.

In order to comprehensively evaluate the changes in the ability of large language models before and after pruning, we conducted comprehensive evaluation from five aspect: Reasoning: CMNLI (Li et al., ), HellaSwag (HeSw) (Zellers et al., ), PIQA (Bisk et al., ). Language: CHID (Zheng et al., ), WSC (Levesque et al., 2012). Knowledge: CommonSenseQA (CoQA) (Reddy et al., ), BoolQ (Clark et al., ). Examination: MMLU (Hendrycks et al., ), CMMLU (Li et al., ). Understanding: Race-High/Middle (H/M) (Lai et al., ), XSum (Hasan et al., ), C3 (Sun et al., ) and PG19 (Rae et al., ). For more details, please refer to Appendix G

#### Baselines.

To evaluate the effectiveness of our method, we compared several structured pruning methods for large language models, including:

1) LLMPru (Ma et al., ), which adopts structural pruning that selectively removes non-critical coupled structures based on gradient information, maximally preserving the majority of the LLM’s functionality. LLMPru. applies post training to the pruned model, but for fair comparison, we do not apply post training to it.

2) SliceGPT (Ashkboos et al., ), which is a post-training sparsification scheme that replaces each weight matrix with a smaller matrix, reducing the embedding dimension of the network. Specifically, they applied PCA to the hidden representation from shallow to deep layers, and incorporated the dimension reduction matrix into existing network parameters.

3) LaCo (Yang et al., ), which is a pruning method for large language models based on reducing layers. LaCo gradually merges similar layers from deep to shallow and sets a threshold to avoid continuously merging too many layers.

For our evaluation, we use PG19 for layer importance and perplexity calculation. The models, baselines and evaluate benchmarks is the same as LaCo.

**Table 2: Comparison of pruning methods on multiple natural language benchmarks. The results of LLMPrun., SliceGPT and LaCo are reported from LaCo. The last column reports the relative performance retention.**
| LLM | Method | Ratio | Benchmarks | Ave. | Per. |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CMNLI | HeSw | PIQA | CHID | WSC | CoQA | BoolQ | Race-H | Race-M | XSum | C3 | MMLU | CMMLU |  |  |  |  |  |
| Llama2-7B | Dense | 0.00% | 32.99 | 71.26 | 77.91 | 41.66 | 50.00 | 64.62 | 71.62 | 35.71 | 34.19 | 19.40 | 43.56 | 45.39 | 32.92 | 47.78 | 100.00 |
| LLMPrun. | 27.0% | 34.33 | 56.46 | 71.22 | 25.25 | 36.54 | 42.51 | 55.20 | 22.56 | 22.35 | 11.51 | 25.64 | 23.33 | 25.25 | 34.78 | 72.79 |  |
| SliceGPT | 26.4% | 31.70 | 50.27 | 66.21 | 20.79 | 36.54 | 41.36 | 38.32 | 21.07 | 21.66 | 4.89 | 39.78 | 28.92 | 25.37 | 32.84 | 68.73 |  |
| LaCo | 27.1% | 34.43 | 55.69 | 69.80 | 36.14 | 40.38 | 45.70 | 64.07 | 22.61 | 23.61 | 15.64 | 39.67 | 26.45 | 25.24 | 38.41 | 80.39 |  |
| ShortGPT | 27.1% | 32.95 | 53.02 | 66.43 | 24.68 | 52.46 | 47.99 | 74.71 | 32.25 | 35.17 | 0.67 | 39.62 | 43.96 | 32.25 | 41.24 | 86.31 |  |
| Llama2-13B | Dense | 0.00% | 32.99 | 74.78 | 79.71 | 47.35 | 50.00 | 66.91 | 82.39 | 57.95 | 60.38 | 23.45 | 47.51 | 55.00 | 38.40 | 55.14 | 100.00 |
| LLMPrun. | 24.4% | 33.03 | 67.76 | 76.66 | 35.64 | 40.38 | 50.86 | 56.42 | 22.47 | 22.08 | 19.17 | 32.33 | 25.21 | 24.71 | 38.97 | 70.67 |  |
| SliceGPT | 23.6% | 29.82 | 55.71 | 69.04 | 19.31 | 36.54 | 47.26 | 37.86 | 23.41 | 24.03 | 5.27 | 41.92 | 37.14 | 25.79 | 34.85 | 63.20 |  |
| LaCo | 24.6% | 32.86 | 64.39 | 63.20 | 40.10 | 52.88 | 52.66 | 63.98 | 54.49 | 56.55 | 14.45 | 44.93 | 45.93 | 32.62 | 47.62 | 86.36 |  |
| ShortGPT | 24.6% | 33.00 | 66.64 | 73.45 | 36.61 | 50.00 | 58.64 | 62.48 | 58.35 | 60.17 | 17.59 | 46.90 | 54.69 | 38.38 | 50.53 | 91.64 |  |
| Baichuan2-7B | Dense | 0.00% | 33.37 | 67.56 | 76.17 | 85.56 | 50.00 | 63.14 | 74.10 | 52.63 | 51.04 | 20.82 | 64.55 | 53.87 | 56.95 | 57.67 | 100.00 |
| LLMPrun. | 24.2% | 32.28 | 53.66 | 71.82 | 69.80 | 53.85 | 47.83 | 61.19 | 21.96 | 22.28 | 15.98 | 41.64 | 24.93 | 25.69 | 41.76 | 72.41 |  |
| SliceGPT | 22.2% | 32.07 | 25.29 | 50.33 | 14.85 | 36.54 | 19.57 | 39.30 | 23.53 | 22.49 | 0.00 | 26.58 | 25.18 | 25.25 | 26.23 | 45.48 |  |
| LaCo | 24.2% | 33.00 | 52.28 | 68.50 | 76.24 | 42.31 | 47.26 | 56.15 | 28.99 | 27.72 | 12.03 | 50.85 | 31.53 | 31.24 | 42.93 | 74.44 |  |
| ShortGPT | 24.2% | 33.30 | 56.96 | 67.68 | 65.63 | 50.00 | 46.70 | 67.83 | 53.26 | 46.76 | 0.04 | 56.33 | 45.77 | 47.87 | 49.08 | 85.10 |  |
| Baichuan2-13B | Dense | 0.00% | 33.21 | 71.10 | 78.07 | 86.51 | 50.00 | 65.6 | 77.89 | 67.27 | 68.94 | 25.02 | 65.64 | 59.50 | 61.30 | 62.31 | 100.00 |
| LLMPrun. | 24.3% | 33.80 | 53.57 | 71.82 | 72.77 | 37.50 | 38.82 | 56.54 | 21.17 | 21.61 | 13.67 | 39.89 | 23.19 | 25.18 | 39.20 | 62.91 |  |
| SliceGPT | 22.8% | 32.07 | 25.85 | 51.03 | 10.40 | 36.54 | 18.02 | 37.83 | 21.56 | 21.52 | 0.00 | 24.99 | 22.95 | 25.26 | 25.23 | 40.49 |  |
| LaCo | 24.7% | 33.03 | 60.71 | 68.88 | 76.73 | 44.23 | 55.45 | 62.35 | 56.92 | 57.80 | 12.32 | 61.10 | 51.35 | 53.65 | 53.43 | 85.75 |  |
| ShortGPT | 24.7% | 32.81 | 60.55 | 71.60 | 80.17 | 47.13 | 54.30 | 62.54 | 55.77 | 56.41 | 15.14 | 60.16 | 52.11 | 58.86 | 54.43 | 87.35 |  |

### 4.2 Main Results

To validate the efficacy of our proposed method, we conducted comparative experiments against baseline techniques commonly employed in large language model evaluation. Considering the current structured pruning methods generally reduce parameters by no more than 30%, we performed experiments with approximately 1/4 of the parameters pruned. The experimental results are presented in Table 2. Additional experiments exploring different parameter reduction proportions will be discussed in the subsequent section.

The results demonstrate that the performance of the model pruned by our method significantly surpasses that of the baseline methods, maintaining most of the large language model’s capabilities. Furthermore, we note that the approach of reducing the number of layers (ShortGPT/LaCo) outperforms the method of reducing the embedding dimensions (LLMPru./SliceGPT), implying that the model exhibits more redundancy in depth than in width. Further experimental analysis will be presented in the ensuing section.

In Table 2, we fully adopted the benchmark, model, and pruning ratio in the LaCo paper. In order to make a more fair comparison with LLMprun. and SliceGPT, we compared them with the same benchmark, model, and pruning ratio in their original paper. The experimental results are shown in Appendix C. Consistent with our findings in Table 2, these experiments further demonstrate the significant layer redundancy present in existing large language models, and ShortGPT achieves superior performance compared to other pruning methods.

The results show that coarse-grained pruning methods, such as removing entire layers, often outperform fine-grained approaches like Slice GPT or LLM Pruner. We speculate that the reason is that the large language model is actually very robust, as shown in Figure 1, removing any deep layer individually actually has very little impact on the final output, which means it is difficult to define the importance of a finer grained module and perform pruning.

### 4.3 Varying metric and pruning ratio

Figure: Figure 4: Comparison of different importance metrics. Perplexity is calculated by removing each single layer, other metrics is calculated by hidden states of each layer.
Refer to caption: x6.png

The core principle of our method is to rank layers by their importance and remove the less significant ones. The choice of importance metric significantly influences the outcome. In this section, we define and compare several different importance metrics:

- •
Sequential: The importance is directly proportional to the sequence order, with shallower layers being less important. This can be implemented by assigning the negative value of each layer’s index as its importance metric.
- •
Norm/Reverse-order: This metric posits that importance is inversely proportional to the sequence order. It assigns higher importance scores to the shallower layers. This method gives the same order as measuring importance by hidden states norm as Figure 4 shows.
- •
Relative Magnitude: Proposed in Samragh et al. (), this metric assumes layers with larger $||\frac{f(x)}{x+f(x)}||$ are of higher importance, where $f$ is the layer transformation function.
- •
BI: we calculate the BI score mentioned in Section 3.1 as importance metric.

Figure 4 demonstrates the different metrics. We observe that shallower layers in the LLM network are more crucial than deeper ones. Figure 5 shows the results of removing layers by different metrics, demonstrating that Our proposed BI outperforms other metrics. The method of Relative Magnitude is highly competitive, indicating that relative values can also reflect the importance to some extent. It is worth noting that the hidden states norm seems to be a good metric when only considering the MMLU benchmark, but the perplexity is relatively poor.

Figure: Figure 5: Performance of MMLU and perplexity when we prune by different metrics, with increasing pruning ratio. We can see that as the pruning ratio increases, the performance of the model declines.
Refer to caption: x7.png

As a pruning method, we further validated the effects of different pruning ratios on model performance. Experiments were conducted on the Llama2 and Baichuan2 models, observing the Perplexity and MMLU. The results for Llama2, as shown in Figure 5, demonstrate that the model’s performance generally declines as the pruning ratio increases. However, we observe a notable phenomenon: the MMLU score exhibits a sharp drop at a specific layer. This sudden decrease suggests the presence of certain critical layers within the network that play a particularly important role in maintaining performance. Similar patterns are observed in the Baichuan2 model, as illustrated in Appendix B.

### 4.4 Redundancy on non-transformer LLM

To determine whether the observed depth redundancy is specific to the Transformer architecture, we extended our investigation to include two popular non-Transformer models, RWKV-7B(^1^11 We use rwkv-v5-world-7B from https://huggingface.co/RWKV/v5-Eagle-7B-HF) (Peng et al., ) and Mamba-2.8B (^2^22We take the model from https://huggingface.co/state-spaces/mamba-2.8b-hf) (Gu & Dao, ). Our experiments revealed that these models also exhibit resilience to layer removal, maintaining performance despite the elimination of certain layers. This finding suggests that the redundancy phenomenon may not be unique to Transformer-based models, but rather a common characteristic across current large language models. Table 3 shows that our method is applicable and effective for both Mamba and RWKV models, suggesting that the redundancy is universal across current LLMs. However, it is worth noting that the RWKV model appears less redundant than Mamba and Transformer models, which warrants further investigation.

**Table 3: ShortGPT pruning on RWKV and Mamba.**
| Model | Pruning ratio | CMNLI | HeSw | PIQA | CHID | WSC | CoQA | BoolQ | Race-H | Race-M | XSum | C3 | MMLU | CMMLU | Ave. | Per. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Mamba-2.8B | 0% | 35.97 | 61.84 | 75.52 | 35.56 | 49.69 | 56.35 | 60.67 | 24.9 | 25.3 | 15.03 | 42.08 | 26.29 | 25.32 | 41.12 | 100.00 |
| 10.9% | 32.95 | 59.71 | 73.01 | 32.52 | 49.28 | 52.66 | 51.41 | 24.27 | 25.21 | 14.95 | 41.1 | 26.01 | 25.00 | 39.08 | 95.04 |  |
| 20.3% | 31.29 | 55.69 | 69.64 | 29.12 | 48.36 | 48.32 | 62.2 | 23.61 | 23.61 | 14.71 | 41.59 | 25.69 | 25.37 | 38.36 | 93.29 |  |
| 25% | 29.96 | 52.38 | 68.77 | 26.02 | 48.26 | 44.96 | 62.2 | 23.67 | 23.26 | 14.00 | 40.71 | 24.32 | 24.89 | 37.18 | 90.42 |  |
| 31.3% | 28.25 | 47.02 | 64.91 | 21.38 | 49.69 | 44.96 | 62.17 | 21.87 | 22.77 | 13.77 | 40.44 | 24.48 | 24.77 | 35.59 | 86.55 |  |
| RWKV-7B | 0% | 32.07 | 65.98 | 77.09 | 85.36 | 50.00 | 62.65 | 62.72 | 38.56 | 45.47 | 16.5 | 57.97 | 31.85 | 28.54 | 50.37 | 100.00 |
| 9.4% | 32.6 | 56.41 | 73.94 | 78.12 | 50.00 | 49.55 | 62.35 | 25.9 | 25.77 | 9.57 | 54.68 | 27.29 | 25.03 | 43.94 | 87.23 |  |
| 18.8% | 32.11 | 49.47 | 71.55 | 65.63 | 50.00 | 40.54 | 61.19 | 22.04 | 23.75 | 8.13 | 49.15 | 26.35 | 25 | 40.38 | 80.17 |  |
| 25% | 32.41 | 39.73 | 65.13 | 52.6 | 50.00 | 29.65 | 60.92 | 22.56 | 21.59 | 12.02 | 41.86 | 25.52 | 25.08 | 36.85 | 73.16 |  |
| 28.1% | 33.11 | 32.22 | 60.01 | 32.47 | 50.1 | 28.34 | 60.85 | 22.27 | 21.31 | 10.43 | 37.81 | 25.64 | 25.15 | 33.82 | 67.14 |  |

### 4.5 Orthogonal to Quantization

In this section, we show that our method is orthogonal to quantization methods. We apply our method to Llama2-7B (^3^33We take the model from https://huggingface.co/TheBloke/Llama-2-7B-GPTQ) quantized by GPTQ algorithm. Table 4 shows that our method is compatible with the quantization-like method. In addition, we compared the performance of applying pruning before quantization (^4^44We use GPTQ algorithm for quantization from https://github.com/AutoGPTQ/AutoGPTQ). The results shown in the Table 5 further indicates that quantization and ShortGPT are orthogonal operations.

**Table 4: Layer removal results on Llama2-7B-Base-GPTQ.**
| Model | Ratio/Layer | Perplexity | MMLU | Throughput (speed up) |
| --- | --- | --- | --- | --- |
| Baseline | 0%/32 | 8.03 | 43.17 | 4331.23 Token/s (1.00x) |
|  | 3.1%/31 | 8.37 | 42.88 | 4399.31 Token/s (1.02x) |
|  | 9.4%/29 | 9.44 | 42.31 | 4602.26 Token/s (1.06x) |
| ShortGPT | 12.5%/28 | 10.24 | 41.62 | 4680.68 Token/s (1.08x) |
|  | 15.6%/27 | 11.42 | 43.17 | 4756.94 Token/s (1.10x) |
|  | 25.0%/24 | 22.29 | 41.68 | 5045.59 Token/s (1.16x) |
|  | 27.1%/23 | 40.78 | 43.35 | 5146.99 Token/s (1.19x) |

**Table 5: Performance comparison of different methods**
| Method | MMLU | CMMLU |
| --- | --- | --- |
| Llama2-7B-Baseline | 45.4 | 32.9 |
| 4-bit quantization | 44.9 | 32.5 |
| Layer removal (27.1%) | 44.0 | 32.3 |
| 4-bit quantization then layer removal | 42.4 | 31.0 |
| Layer removal then 4-bit quantization | 41.2 | 30.5 |

### 4.6 Post training to restore performance

To mitigate the performance loss resulting from layer removal, we explored post-training strategies inspired by Chen et al. (). Our approach comprised two key steps: 1)Replacement: We substituted the removed layers with lightweight Multi-Layer Perceptron (MLP) modules. 2)Retraining: We subsequently retrained the modified model. The results in Table 6 demonstrate the potential of post-train in recover performance loss. Appendix F list the training details.

**Table 6: Post-train Llama2-7B to restore performance.**
| Method | Avg. | Ratio | CMNLI | HeSw | PIQA | CHID | WSC | CoQA | BoolQ | Race-H | Race-M | XSum | C3 | MMLU | CMMLU |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dense | 47.78 | 0% | 32.99 | 71.26 | 77.91 | 41.66 | 50.00 | 64.62 | 71.62 | 35.71 | 34.19 | 19.40 | 43.56 | 45.39 | 32.92 |
| ShortGPT | 41.22 | 27.1% | 32.95 | 53.02 | 66.43 | 24.68 | 52.46 | 47.99 | 74.41 | 32.25 | 35.17 | 0.67 | 39.62 | 43.96 | 32.25 |
| ShortGPT+post-train | 43.16 | 24.0% | 32.99 | 54.83 | 68.12 | 31.82 | 51.37 | 58.32 | 72.36 | 34.18 | 34.68 | 4.89 | 40.37 | 44.47 | 32.73 |

## 5 Limitation

Although our method demonstrates strong competitiveness compared to current pruning methods, there are some phenomena that have not been explained. Our experiments reveal that the negative effect of layer removal is more significant on generative tasks compared to multiple-choice tasks. When we remove 25% layers from Llama2-7B or Baichuan2-7B, the performance in generative tasks such as XSum and C3 deceases to nearly zero, although the performance decline was not as significant on the larger model of the 13B. We speculate that compared to multiple-choice tasks, generative tasks face the problem of accumulated errors and large model is more robust than small one. The reasons behind it still need to be explored. The post-training techniques discussed in Section 4.6 have the potential to mitigate this issue and warrant further exploration.

## 6 Related works

To reduce the inference cost of large language models and increase their practical applications, there have been many recent works on compressing models, which can be classified into two categories:
model pruning and quantization. Besides, there are some works aim to study the redundancy of model which is essential for compressing models.

Model pruning: model pruning (LeCun et al., ; Han et al., ) is a classic and effective method of reducing model redundancy modules to compress models. The model pruning methods mainly include unstructured pruning and structured pruning. The unstructured pruning simplifies an LLM by removing specific parameters without considering its internal structure, such as SparseGPT (Frantar & Alistarh, ) and LoRAPrune (Zhang et al., ). However, this method disregards the overall LLM structure, resulting in an irregular sparse model composition. Another more practical approach is structured pruning, GUM(Syed et al., ) makes an analysis of several structured pruning methods for decoder-only LLMs. LLM-Pruner (Ma et al., ) selectively removes non-critical structures according to gradient information. ShearedLLaMA (Xia et al., ) employs targeted structured pruning and dynamic batch loading. LaCo (Yang et al., ) used layer merging to compress the model. Compared to the previous method, our method is a simple and efficient structured pruning method.

Quantization: quantization (Liu et al., ; Gholami et al., ; Dettmers et al., ; ) is a widely accepted technique in the field of model compression, which can significantly save the storage and computational costs of deep learning models. Traditional models are generally stored as floating-point numbers, but quantization converts them into integers or other discrete forms. LUT-GEMM (Park et al., ) quantifies only weights and optimizes matrix multiplication in LLM using BCQ format. SPQR (Dettmers et al., ) identifies and isolates abnormal weights, stores them with higher accuracy, and compresses all other weights into 3-4 bits. Our model pruning method and quantization method are orthogonal, which means quantification based on our pruned model can further compress the model.

Model redundancy: researchers have long noticed the significant redundancy in nonlinear models (Catchpole & Morgan, ). In recent years, the transformer model architecture has been widely applied, and researchers have also studied its redundancy. In (Bian et al., ), researchers analyzed redundancy in attention mechanisms, in which clear and similar redundancy patterns (cluster structure) are observed among attention heads. In (Dalvi et al., ), researchers dissect two pre-trained models, BERT (Devlin et al., ) and XLNet (Yang et al., ), studying how much redundancy they exhibit at a representation level and a more fine-grained neuron-level. However, the redundancy in current large language models based on decoder-only structures still needs to be explored.

## 7 Conclusion

In this work, we uncovered the significant layer-wise redundancy of LLMs, Our research demonstrates that certain layers contribute minimally to overall network functionality and can be removed without substantially compromising model performance. Based on our observation, We introduce Block influence to quantify the importance of each layer and propose a simple and straightforward pruning method: layer removal. Our experiments demonstrates that it is possible to maintain up to approximately 90% of a LLM’s performance while reducing the model’s parameter amount and computational requirements by approximately 25%. Besides, our method is orthogonal to quantization methods and can be further improved by continual training. We hope that our work can provide some insight for future model compression techniques.
Moreover, our work suggests potential avenues for improving the efficiency of model training by reducing inherent redundancy in the future.

## Appendix A Mathematical explanation for why pre-norm brings high similarity

We provide a simple explanation here about how pre-norm leads to high deep similarity in this section, here we adopt RMSNorm (Zhang & Sennrich, ) for convenient, which is also the popular pre-norm used in many recent LLMs, such as Llama and Mamba.

###### Lemma 1

(Xiong et al., )
At initialization, for the Pre-LN Transformer, $(1+\frac{L}{2})d\leq\mathbb{E}(||x_{L,i}||^{2}_{2})\leq(1+\frac{3L}{2})d$ for all $L>0$ and $i$. Expectations are taken over the input and the randomness of initialization, where the hidden state of $L^{th}$ layer is $x_{L}$.

From Lemma 1, the hidden state of the pre-norm model will continuously increase as the number of layers increases. And under the assumption of each component of $x_{l}$ has a mean of 0 0, we can obtain $||x_{L}||=\Theta(\sqrt{L})$.

Then we consider $x_{L+1}=x_{L}+f_{L}(x_{L},\theta_{L})$, where $f_{L}$ is a operation such as Attention or MLP, $\theta_{L}$ is learnable parameters. Then $f_{L}(x_{L},\theta_{L})=O(1)$ respect to $L$, for Attention as example, $||f_{L}(x_{L},\theta_{L})||=||(softmax(Q^{T}K)X_{L}/||X_{L}||\cdot(\sigma_{rms
}))W_{v}W_{q}||=O(||\sigma_{rms}||||W_{v}||||W_{o}||)=O(1)$ respect to $L$.

Then we can get:

$$ $\displaystyle\textit{cos similarity}(X_{L+1},X_{L})$ $\displaystyle=\frac{x_{L+1}x_{L}}{||x_{L+1}||||x_{L}||}=\frac{||x_{L}||^{2}}{| |x_{L+1}||||x_{L}||}+\frac{f_{L}(x_{L},\theta)x_{L}}{||x_{L+1}||||x_{L}||}$ (2) $\displaystyle\geq\frac{||x_{L}||^{2}}{||x_{L+1}||||x_{L}||}-\frac{||f_{L}(x_{L },\theta)||||x_{L}||}{||x_{L+1}||||x_{L}||}$ (3) $\displaystyle=\frac{||x_{L}||}{||x_{L+1}||}-\frac{||f_{L}(x_{L},\theta)||}{x_{ L+1}}=\Theta(\sqrt{\frac{L}{L+1})}-O(\sqrt{\frac{1}{L+1}})$ (4) $$

This means that as the number of layers $L$ increases, the similarity between the input and output of the layer will be high. This means that the role of $f_{L}$ may be relatively small, and removing it from the network may have a relatively small impact to the model.

Although the above theoretical analysis is only for randomly initialized models, this phenomenon that deep layer has similar input and output exists in both our own trained models shown in Figure 2 and existing models in Figure 4.

###### Lemma 1

## Appendix B Layer Removal on Baichuan2-series Model

Figure: Figure 6: Pruning by different metrics on Baichuan2-series model.
Refer to caption: x8.png

## Appendix C A Fair comparison with SliceGPT and LLMprun.

In Table 2, we fully adopted the benchmark, model, and pruning ratio in the LaCo’s paper. For a fair comparison with LLM pruner and SliceGPT, we do the same experiments in the original paper of LLM pruner and SliceGPT. The results is provided in Table 7 and Table 8. We take the same benchmarks, models and pruning ratio as the corresponding original paper. The results demonstrate that our method is highly competitive.

**Table 7: Comparison between ShortGPT and LLM-pruner. The Table is corresponding to the Table 1 of LLM pruner(Zhang et al., ).**
| Model | Pruning ratio | Method | BoolQ | PIQA | Hellaswag | Winogrande | Arc-e | Arc-c | OBQA | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Llama-7B | Ratio=0% | Baseline | 73.18 | 78.35 | 72.99 | 67.01 | 67.45 | 41.38 | 42.4 | 63.25 |
| Ratio=20% | LLM-pruner | 59.39 | 75.57 | 65.34 | 61.33 | 59.18 | 37.12 | 39.80 | 56.82 |  |
| Ratio=21.9 % | ShortGPT | 68.26 | 72.28 | 61.7 | 63.77 | 60.22 | 39 | 41.6 | 58.12 |  |
| Llama-13B | Ratio=0% | Baseline | 68.47 | 78.89 | 76.24 | 70.09 | 74.58 | 44.54 | 42.00 | 64.97 |
| Ratio=20% | LLM-pruner | 67.68 | 77.15 | 73.41 | 65.11 | 68.35 | 38.4 | 42.4 | 61.79 |  |
| Ratio=20% | ShortGPT | 68.41 | 76.36 | 72.9 | 67.4 | 68.62 | 39.2 | 41 | 61.98 |  |

Table: Table 8: Comparison between ShortGPT and SliceGPT. The Table is corresponding to the Table 7 of SliceGPT(Ashkboos et al., ).

## Appendix D Detailed Strategies for Layer Removal

We list the details of different layer removal strategies in Table 10. The concrete removed layers by ShortGPT in Table 2 are listed in Table 9

**Table 9: Setup of Removed Layers for Benchmark Models.**
| Model | Removed Layers |
| --- | --- |
| Llama-2-7B | 27, 26, 25, 28, 24, 29, 23, 21, 22 |
| Llama-2-13B | 33, 31, 32, 30, 29, 34, 28, 35, 27, 26 |
| Baichuan-2-7B | 26, 27, 25, 28, 24, 29, 23, 22, 30 |
| Baichuan-2-13B | 32, 31, 33, 30, 34, 29, 28, 35, 27, 26 |

**Table 10: Strategies for Layer Removal in Models.**
| Strategy | Description |
| --- | --- |
| Sequential | Layers are removed sequentially from the beginning of the model. The process starts with layer 0 and progressively includes more layers for removal (e.g., {0}, {0, 1}, …). |
| Reverse-order | This strategy involves starting from the model’s final layer and progressively removing layers in reverse order (e.g., {-1}, {-1, -2}, …). |
| Relative Magnitude | Layers are removed in ascending order based on their Relative Magnitude values. The removal process accumulates layers from those with the smallest to the largest values, mirroring the sequential strategy’s accumulation method. |
| BI (Block Influence) | Follows a similar accumulation approach as the Sequential strategy, but layers are ordered and removed according to their BI values, starting from the lowest and moving to the highest. |

## Appendix E Setup for training post-norm model and
pre-norm model

We have listed the specific training settings for pre norm and post norm in Table 11.

**Table 11: Training Parameters.**
| Parameter | Value |
| --- | --- |
| Global Batch Size | 2048 |
| Sequence length | 4096 |
| Precision | bf16 |
| Learning Rate Scheduler | cosine |
| Max Learning Rate | 4e-4 |
| Min Learning Rate | 5e-5 |
| Warm-up steps | 3000 |
| Training Tokens | 200B |
| Weight Decay | 0.1 |
| Adam Beta1 | 0.9 |
| Adam Beta2 | 0.98 |
| Gradient Clip | 1.0 |
| Tokenizer | Llama2 |
| Layers | 32 |
| Hidden state | 2048 |
| Attention heads | 32 |
| Head dim | 64 |
| FFN size | 5504 |
| Activation function | Silu |

## Appendix F post-training settings

We replace the removed layer with a lightweight gated MLP layer with hidden size = 2048. Table 12 show the post training settings.

**Table 12: Post-training Parameters.**
| Parameter | Value |
| --- | --- |
| Global Batch Size | 2048 |
| Sequence length | 4096 |
| Precision | bf16 |
| Learning Rate Scheduler | cosine |
| Max Learning Rate | 2e-5 |
| Min Learning Rate | 1e-5 |
| Warm-up steps | 3000 |
| Training Tokens | 50B |
| Weight Decay | 0.1 |
| Adam Beta1 | 0.9 |
| Adam Beta2 | 0.98 |
| Gradient Clip | 1.0 |

## Appendix G Evaluation Benchmarks

In order to comprehensively evaluate the changes in the ability of large language models before and after pruning, we conducted evaluations on the most commonly used Benchmark MMLU Hendrycks et al. (), CMMLU Li et al. () for evaluating large models. In addition, we also followed LaCo Yang et al. () to evaluate a wider dataset.

MMLU Hendrycks et al. () is a benchmark aimed at measuring the knowledge acquired during pre-training by specifically evaluating models in zero-shot and few-shot settings. This makes benchmarks more challenging and similar to the way we evaluate humans. This benchmark covers 57 subjects including STEM, humanities, social sciences, etc. Its difficulty ranges from beginner to advanced professional level, and it tests world knowledge and problem-solving ability.

CMMLU Li et al. () is a comprehensive Chinese language assessment dataset designed specifically to evaluate LLM’s advanced knowledge and reasoning abilities in the context of Chinese language and culture. CMMLU covers 67 topics, from elementary school to university or professional level. Including natural sciences, as well as humanities and social sciences, it also includes many contents with Chinese characteristics.

CMNLI Xu et al. () is part of the Chinese language understanding assessment benchmark. It consists of two parts: XNLI and MNLI. HellaSwag (HeSw) Zellers et al. () is a challenging dataset for evaluating commonsense NLI that is especially hard for state-of-the-art models, though its questions are trivial for humans. PIQA Bisk et al. () is a multi-choice question and answer dataset that focuses on daily scenarios. This dataset explores the model’s grasp of the laws of the real physical world through daily scenarios. CHID Zheng et al. () is an idiom cloze test dataset that mainly focuses on the selection of candidate words and the representation of idioms. CoQA Reddy et al. () is a large-scale dataset used for conversational question-answering tasks, containing over 127000 questions and their corresponding answers. BoolQ Clark et al. () is a question-answer dataset containing 15942 examples of yes/no questions. These problems occur naturally - they are generated in an environment that is silent and unconstrained. Race Lai et al. () is a large-scale reading comprehension dataset collected from English examinations in China, which are designed for middle school and high school students. XSumHasan et al. () is used to evaluate abstract single document summarization systems. The goal is to create a short, one-sentence new summary of what the article is about. C3 Sun et al. () is a machine reading comprehension dataset with multiple choices, consisting of multiple-choice questions, reading materials from Chinese proficiency exams, and ethnic Chinese exams. PG19 Rae et al. () is a long document dataset from books used to test the effectiveness of language modeling.

## Appendix H Hardware Environment

The platform we use to experiment is GPU heterogeneous platform.
The hardware of our platform is shown in Table 13

**Table 13: Setup of Removed Layers for Benchmark Models.**
| Name | Details |
| --- | --- |
| CPU | 2x Intel(R) Xeon(R) Gold 6430 CPU @ 2.1GHz |
| GPU | 8x NVIDIA A100-80GB Tensor Core GPU |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please see Weaknesses and address them.
soundness: 2
strengths: The proposed method is straightforward to implement and requires only forward passes, making it computationally efficient. This efficiency allows it to be applied to larger models within a limited computation budget, unlike competing methods that require the additional overhead of computing gradients or Hessians.
summary: The paper presents a layer pruning method specifically designed for LLMs. It begins by assessing layer importance using the proposed Block Influence metric, which quantifies importance based on the variation in features between consecutive layers. Subsequently, the method iteratively removes the least significant layers until the target sparsity level is achieved.
weaknesses: 1. The method hinges on the skip-connection nature of LLM layers, specifically, $x_{L+1} = x_L + f(x_L)$. Consequently, this reliance could limit its applicability compared to competitors that use gradients and are generally not constrained by model architecture.

2. The analysis on layer redundancy in Appendix A is unconvincing. Specifically, it misinterprets Lemma 1, which states that $||x_L|| = O(\sqrt{L})$ rather than $||x_L|| = \Theta(\sqrt{L})$. Furthermore, the authors assert that $f_L(x_L) = O(1)$ without solid mathematical proof. In the Attention example, the authors omit $W_o$ initially but include it later without explanation. Additionally, in (4), it is unclear why the $\ell_2$ norm of $||x_{L+1}||_2$ diminishes. Overall, the analysis in this section lacks clarity and cohesion.

3. The paper does not compare its method with recent gradient-based pruning techniques such as SLEB [1] and Shortened Llama [2], and it lacks a discussion on the related work by Gromov et al. [3].

4. In Table 2, for WSC, why are the accuracies of all Dense models the same at $50\%$? Additionally, the accuracies for ShortGPT-pruned Llama2-13B and Baichuan2-7B are also all $50\%$ on WSC.

5. Section 4.5 lacks a comparison with other pruning methods. It remains unclear if these other methods, when applied to quantized models, could potentially outperform ShortGPT.

6. Section 4.6 also lacks a comparison with other pruning methods.

**Overall Assessment**

In summary, I don't think the technical contribution of this paper is sufficient for ICLR.

[1] Song, Jiwon, et al. "SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks." arXiv preprint arXiv:2402.09025 (2024).

[2] Kim, Bo-Kyeong, et al. "Shortened llama: A simple depth pruning for large language models." arXiv preprint arXiv:2402.02834 11 (2024).

[3] Gromov, Andrey, et al. "The unreasonable ineffectiveness of the deeper layers." arXiv preprint arXiv:2403.17887 (2024).

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: Apart from questions in weaknesses, there are following additional questions:

1.There are a few typos, though they do not impact the overall evaluation. For example, the incorrect y-axis label in Figure 1(b); and the denominator in the second term of Equation 4. 

2.There are also some inconsistent representations, such as “LLMPru,” “LLMprun.,” “LLM Pruner,” and “LLM-Pruner.” We assume these refer to the same work, but the variations are confusing.

3.The y-axis scale in Figure 1(a) may be misleading due to its exponential nature, which could mask a significant gap in PPL between the two approaching lines.

4.Could you clarify the process used to sample instances for calculating PPL (e.g., results in Table 1)? Clarifying the sampling criteria or methodology would improve reproducibility and understanding.
soundness: 1
strengths: 1.The proposed method of layer removal based on BI scores is straightforward, efficient, and performs comparably or better than more complex pruning methods, making it accessible and adaptable for practical use.

2.The proposed method is shown to work effectively on non-transformer models (RWKV and Mamba), indicating that the concept of layer redundancy may be applicable across various architectures, enhancing the method’s potential for broader use.
summary: This paper introduces ShortGPT, a pruning technique for large language models (LLMs) that identifies and removes redundant layers without significantly impacting performance. The authors introduce a pruning metric called Block Influence (BI), which measures the importance of each layer by analyzing the similarity between its input and output representations. Layers with lower BI scores are considered redundant and removed. Through empirical evaluations, ShortGPT demonstrates superior performance compared to existing pruning methods, achieving high parameter reduction (around 25%) while maintaining approximately 90% of the model’s performance.
weaknesses: 1.The novelty of this paper is limited, with a marginal contribution. The methodology section primarily introduces the BI pruning metric, measuring average cosine similarity between consecutive layer activations. However, [1] has already explored this, pointing out that for both Baichuan2-7B and Llama2-7B, similarity between layers 3 to 28 is typically near 1.

2.The results reported in the paper are unreliable:

a)	The tables in this paper contain numerous inconsistencies that undermine the credibility of the results. For instance, in Table 8, the SliceGPT results for Llama-2-7B and Llama-2-13B under a 20% pruning ratio are identical across PIQA, Winogrande, Arc-e, and Arc-c, yet differ for Hellaswag. Despite this discrepancy, the reported average remains inexplicably unchanged.
b)	According to Table 2, the results of the baseline methods are taken directly from [1]. Did the authors ensure a fair comparison by following the exact same settings (e.g., implementation environment, calibration dataset, benchmark script versions) as used in [1]?

c)	While the authors compared the results of ShortGPT against the baseline methods in the baseline method’s settings, they only claimed that “the same benchmarks, models and pruning ratio” are used. However, there are other settings which may affect the results, such as calibration dataset.

d)	In Appendix C, while the authors compared ShortGPT results with baseline methods SliceGPT and LLMPruner using the baseline settings, they only specified 'the same benchmarks, models, and pruning ratio' and directly referenced results from the original papers. However, additional settings, such as the calibration dataset, may also affect the outcomes. The authors should ensure all settings are consistent to enable a truly fair comparison.

e)	In Appendix C, while the authors compared ShortGPT results with baseline methods SliceGPT and LLMPruner using the baseline settings, they only specified 'the same benchmarks, models, and pruning ratio' and directly referenced results from the original papers. However, additional settings, such as the calibration dataset, may also affect the outcomes. The authors should ensure all settings are consistent to enable a truly fair comparison.

[1] Yifei Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse, 2024.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. The figure 2, is the experiment done to understand the effect of pre-norm vs post-norm setting keeping all the model training variables the same. If so, referring to table 11 and table 12, all the settings are not equal i.e the training tokens etc; why is that the case? An ideal comparison would be to have 2 architectures -> pre-norm vs post-norm and have all the rest the same and train it for model convergence (maybe the epochs will vary). Once these models has been trained for same loss, you can pick the hidden values and plot the similarities to observe the effect of pre vs post norm. 
2. The presentation in Table 5 can be made much better in terms of highlighting which one performs better and the authors can do a better job in explaining it in Sec 4.5

**Formatting**
1. I believe Figure 1 labels are not correct and missing some information.
2. Table 3 needs better highlighting/formatting of the results; sec 4.4 can bring up important insights more clearly.
3. Table 2 can be better formatted to support the statement in L256-258 (point 2 in clarifications)
soundness: 3
strengths: 1. The paper is easy to read and clear at most of the places, the flow is good but can be made even better.
2. The conceptual idea of coming up with metric (Block Influence) makes sense intuitively (section 2 and section 3) and it's a simple metric to implement/understand.
3. The experiments clearly demonstrate the effectiveness of the method.
summary: This paper proposes a metric; Block Influence (BI) which the authors use as a proxy to remove the layers (prune) without affecting (minimal affect) on the model performance. The metric is assumed to capture the redundancy component of the layer, so intuitively, removing the blocks which are most redundant i.e lower BI makes sense.
weaknesses: **Clarifications**
1. From table 9 and other figures, the authors might have to clearly define what a "layer" means because the metric in use is Block Importance (BI), so the normal understanding would be Attention + FFN is a block (the original transformer has 16 blocks etc;). So from my understanding, when layer 5 is removed, I believe the Attention + FFN is replaced by Identity matrix. Is that correct? This definition has to clearly come out at the start.
2. The finding presented in L256-258 is interesting i.e more redundancy in depth compared to width. If there's any previous work, the authors can cite it. By plainly looking at Table 2, this statement cannot be totally true and needs further validation.

**Additional Experiments/Ablations (in the order of importance to contribution)**
1. L242: The authors has chosen PG19 to compute BI and model importance. Can an ablation be performed to understand the effect of calibration datasets? 
   - I would like to see if model X picks layer A, B, C when using dataset 1 and U, V, W when using dataset 2. i.e whether the model has different order for BI when using different datasets as this has different implications depending on the outcome!.
2. L429: Can the authors show results on number of layers removed (or % layers removed) with respect to model size and/or task-wise for a particular performance threshold (in Appendix maybe). 
   - I believe the current experiments has these insights, it's about presenting them and getting new insights when using BI i.e eg: 7B models can remove 5 layers but 13B models can remove 8 layers across tasks and/or task A, B, C can have q% layers removed but task X, Y, Z has only p% layers removed.
3. Given this falls under structured pruning, I believe more results regarding the speed-up/inference can be highlighted/commented in addition to the ones presented in Table 4.
4. Continuing from point 1 in clarifications, an ablation experiment will be helpful if we treat attention and FFN layers separately i.e remove attention layer 1, 4, 5 and FFN 2, 4, 6 etc;. Basically computing the BI after the attention block and FFN block separately; this is similar to design choice in [1], so the study can be complementary to understand the metric dependency on different blocks/layers.

**Relevant missing citations**
1. L131/Sec 2.2: The authors might have to consider citing some previous works [1, 2] which has discussed in detail and came up with similar observations regarding the final layer importance and what happens when we compress it. 
2. L250: The authors can cite [1, 3, 4] which has come up with similar observations i.e compressing >30% drops the model performance.

**Citations**
1. The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models - https://arxiv.org/abs/2312.00960 
2. Fast model editing at scale - https://arxiv.org/abs/2110.11309
3. Are sixteen heads really better than one? - https://arxiv.org/abs/1905.10650
4. Compressing bert: Studying the effects of weight pruning on transfer learning - https://arxiv.org/abs/2002.08307

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See above.
soundness: 3
strengths: - This paper introduces a new layer pruning method, BI.
- This paper conducts experiments on non-transformer models, RWKV-7B and Mamba-2.8B.
- The paper contains many comparative experiments.
summary: This paper introduces a layer pruning metric named BI and demonstrates its superior performance over previous state-of-the-art pruning methods on Llama2-7B and 13B, Baichuan2-7B and 13B. Besides, ShortGPT can be used together with other quantitative methods for further reduction in parameters and computation.
weaknesses: - Insufficient contribution. ShortGPT is orthogonal to the quantization method, which cannot be regarded as a contributing factor, as other pruning methods can achieve similar results.
- In this paper, PG19 is used for layer importance and perplexity calculation. What if we use other datasets like alpaca-cleaned [1] and mmlu?
- Table 5 can be adjusted to make it more beautiful.
- In eq.1, in fact, BI is a cosine distance, so can other distance formulas also be used as metrics for calculation? Why must it be cosine?

[1] alpaca-cleaned: https://huggingface.co/datasets/yahma/alpaca-cleaned

Meta-review: This paper studies a layer pruning method using a layer-importance score called Block Influence (BI) based on the variation in representations between input and output layers. Empirical evaluations demonstrate the effectiveness of the method.

Main weaknesses mentioned by reviewers:
- Reliance on specific model architectures
- Limited comparison to existing work; limited novelty
- Unreliable results due to inconsistent comparisons

Decision: Reject