ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
Authors: Xin Men, Mingyu Xu, Qingyu Zhang, Qianhao Yuan, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, Weipeng Chen
Accepted at: acl25

ShortGPT: Layers in Large Language Models are More Redundant Than You Expect

Xin Men1,*, Mingyu Xu1,*, Qingyu Zhang2,3,*, Qianhao Yuan2,3,*, Bingning Wang1,†, Hongyu Lin2,†, Yaojie Lu2, Xianpei Han2, Weipeng Chen1
1Baichuan Inc.
2Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences
3University of Chinese Academy of Sciences

Abstract
As Large Language Models (LLMs) continue to advance, their computational overhead has increased significantly. In this study, we identify notable redundancy across the layers of LLMs, where some layers contribute minimally to the overall network functionality. To quantify this, we introduce a metric called Block Influence (BI), which measures the importance of each layer based on the similarity between its input and output. Based on the observation of layer redundancy, we propose straightforward pruning methods for different tasks: ShortGPT for multiple-choice tasks and ShortGPT-gen for generative tasks. They prune redundant layers based on their BI scores. Our methods demonstrate superior performance over previous pruning methods. The ability to achieve better results through simple layer pruning, as opposed to more complex pruning techniques, suggests a high degree of redundancy across layers. We hope this work will contribute to future research for improving LLM efficiency. The code is publicly available at https://github.com/cipcas/ShortGPT.

1 Introduction
The field of large language models (LLMs) has witnessed rapid development recently, with LLMs achieving impressive performance across various domains. Guided by the scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022), modern LLMs require significant computational resources, creating substantial barriers to their practical use.

To mitigate the computational demands of large models, techniques for improving model efficiency have become a critical area. These techniques are generally divided into quantization (Liu et al., 2021; Gholami et al., 2022; Dettmers et al., 2022, 2024) and pruning (LeCun et al., 1989; Han et al., 2015; Frantar and Alistarh, 2023). Quantization reduces the precision of model parameters, but its effectiveness often requires specific hardware support. In contrast, pruning reduces the participating networks to decrease the model’s computation, offering a more flexible approach. Despite its advantages, many existing pruning methods are complex. For example, some require gradient information (Ma et al., 2024), which limits their practicality.

In this paper, we focus on layer redundancy in LLMs and propose methods to improve LLM efficiency. We introduce Block Influence (BI), a metric that quantifies how much hidden states change after passing through each layer, providing a direct measure of layers’ importance. Leveraging BI, we propose simple yet effective pruning methods for multiple-choice and generative tasks. For multiple-choice tasks, we propose ShortGPT, which identifies and removes redundant layers with lower BI scores, significantly reducing model size without sacrificing much performance. For generative tasks, we propose a dynamic pruning method, ShortGPT-gen, where the input tokens skip the same redundant layers as in ShortGPT while the generated token pass through all layers to resolve the accumulated errors of ShortGPT during generation.

To evaluate our methods, we conduct evaluation across comprehensive benchmarks. Our experiments reveal that our methods exhibit a smaller performance decrement than previous methods. For instance, ShortGPT removes 10 layers (25% of the total 40 layers) from the LLaMA2-13B model, resulting in only a slight drop in performance on the MMLU benchmark (Hendrycks et al., 2020), from 55.0 to 52.2. ShortGPT-gen skips 10 layers of LLaMA2-13B, leading to only a minimal decrease in performance on the GSM8K (Cobbe et al., 2021) benchmark without requiring any training, from 28.89 to 26.91. Our findings highlight substantial redundancy in current LLMs and suggest potential avenues for improving the efficiency
of models by reducing inherent redundancy in the future. Moreover, our methods are orthogonal to quantization methods, meaning it can be combined with quantization techniques to further reduce the computational overhead of LLMs.

Our main contributions are as follows:

• We analyze the redundancy in large language models (LLMs) and reveal their significant redundancy at the layer level. This finding inspires us to improve LLM efficiency by pruning redundant layers.

• We propose a metric called Block Influence (BI) as an indicator of layer importance. Based on BI, we propose layer pruning methods: ShortGPT for multiple-choice tasks and ShortGPT-gen for generative tasks.

• Our ShortGPT maintains approximately 90% performance for multiple-choice tasks while reducing about 25% of parameters, outperforming previous methods. ShortGPT-gen maintains more than 90% performance for generative tasks in a training-free manner when skipping about 25% of the layers.

2 Motivation

2.1 Background
LLMs are primarily based on the Transformer architecture (Vaswani et al., 2017), with the pre-norm configuration being commonly adopted, as in models like LLaMA (Touvron et al., 2023). The pre-norm configuration, where layer normalization is applied before the self-attention and feed-forward networks, offers several advantages such as faster convergence, improved training stability, and better scalability for deeper networks (Xiong et al., 2020; Liu et al., 2020; Wang et al., 2024). Due to these benefits, the pre-norm approach has been adopted even in non-transformer models, such as Mamba (Gu and Dao, 2023) and RWKV (Peng et al., 2023). For the sake of simplicity in descriptions, our analysis primarily focuses on Transformers, though we extend our experiments to non-Transformer structures in Section 4.4.

However, we observe that when pre-norm is adopted, the similarity between the input and output of transformer layers tends to be higher, as illustrated in Figure 1. This high similarity indicates that certain layers induce minimal changes to the hidden states, suggesting they contribute little to the model’s overall function. It suggests that the deep layers of the model with pre-norm might not play a critical role in the overall function, and that the layers in large language models could be more redundant than expected.

![Cosine similarity between a layer’s input and output during training. The horizontal axis (X-axis) represents the number of training tokens, and the vertical axis (Y-axis) depicts the similarity. Notably, the model employing post-norm (green) exhibits divergence after approximately ~26B tokens of training. The training settings are provided in Appendix E.](page_1012_186_388_312.png)

Figure 1: Cosine similarity between a layer’s input and output during training. The horizontal axis (X-axis) represents the number of training tokens, and the vertical axis (Y-axis) depicts the similarity. Notably, the model employing post-norm (green) exhibits divergence after approximately ~26B tokens of training. The training settings are provided in Appendix E.

2.2 Layer redundancy

As discussed in the previous subsection, we speculate that LLMs exhibit layer redundancy. To verify this, we assess the performance degradation caused by removing individual layers of two popular models, Llama2-7B (Touvron et al., 2023), an English-based LLMs, and Baichuan2-7B (Yang et al., 2023) which mainly focuses on Chinese. Figure 2 confirms our speculation, where some layers do not play a crucial role in LLMs, causing little degradation when omitting them individually. Moreover, this redundancy is primarily manifested in the middle to later layers, with the initial layers and the last layer often being more critical. Notably, we found the last layer to be particularly important, aligning with previous works (Ma et al., 2024; Namburi et al., 2023; Mitchell et al., 2022). We posit that this discrepancy arises because the final FFN functions as part of the token classifier and should be considered in conjunction with the language model head. To verify this, we conduct further investigation, detailed in Table 1. The results show that within the last layer, the FFN is crucial, while the attention module is less significant, supporting our interpretation of the final layer’s importance.
Figure 2: Performance of removing certain layer from LLMs. We can see that certain layers are redundant, and their removal results in minimal performance degradation. For perplexity calculation, we randomly select 10 text segments of 1k length from each piece of data in PG19 (Rae et al., 2019).

Figure 3: The BI score of a layer and the PPL after removing the layer.

<table>
  <tr>
    <th>Delete</th>
    <th>Perplexity</th>
  </tr>
  <tr>
    <td>None</td>
    <td>7.60</td>
  </tr>
  <tr>
    <td>The whole last layer</td>
    <td>13.37</td>
  </tr>
  <tr>
    <td>Attention of the last layer</td>
    <td>7.65</td>
  </tr>
  <tr>
    <td>FFN of the last layer</td>
    <td>12.35</td>
  </tr>
</table>

Table 1: Ablation of removing FFN and Attention of Llama2-7B-Base. We sample 100 instances from PG19 (Rae et al., 2019) to calculate perplexity.

3 Methodology

In this section, we begin by introducing Block Influence (BI), a novel metric designed to assess the hidden states transformation of each layer. Leveraging BI, we then detail our methods for different tasks, i.e. ShortGPT for multiple-choice tasks and ShortGPT-gen for generative tasks.

3.1 Layer importance

As outlined above, the layers of LLMs exhibit redundancy. To capture the degree of layer redundancy, we introduce a new metric, Block Influence (BI), to measure the degree of transformation performed by each layer. The BI score of \( i^{th} \) layer can be calculated as follows:

\[
\mathrm{BI}_i = 1 - \mathbb{E}_{X_{i,t}} \frac{X_{i,t}^T X_{i+1,t}}{\|X_{i,t}\|_2 \|X_{i+1,t}\|_2},
\]

where \( X_{i,t} \) means the \( t^{th} \) row of hidden states of the \( i^{th} \) layer. Lower BI score implies that \( X_i \) and \( X_{i+1} \) exhibit higher cosine similarity, suggesting that the layer makes minimal transformations to the hidden states and is therefore less important. We plot the BI scores of a single layer and the PPL after removing it separately, as shown in Figure 3. The results demonstrate a positive correlation between the BI score and the importance of a layer.

Why use cosine similarity instead of other similarities? As mentioned above, most LLMs apply layer normalization within their modules. Consequently, the magnitude of hidden states become less important for the output than their direction. Besides cosine similarity, there are some other metrics to measure the similarity between two vectors, e.g. Euclidean distance. However, they are sensitive to the magnitude of vectors and can introduce bias. In contrast, cosine similarity is agnostic to the magnitude, focusing on the direction of vectors.
Figure 4: Overview of ShortGPT for multiple-choice tasks and ShortGPT-gen for generative tasks. \( \{X_1, ..., X_n\} \) represent the input tokens, \( \{Y_1, ..., Y_{n+1}\} \) represent the output tokens, and \( \{H_1, ..., H_{n+1}\} \) represent the hidden states of tokens.

3.2 Multiple-Choice Task
In typical multi-choice tasks, each choice is concatenated with the question to form sentences, and the perplexity of each sentence is computed to select the answer. Since LLMs function as series of transformations applied to hidden states across their layers and we can determine the importance of each layer, we propose a straightforward pruning method: ShortGPT, as illustrated in Figure 4a. We remove certain layers in LLMs based on BI. First of all, we construct a calibration set, which is a set of text samples such as PG19 (Rae et al., 2019). Then we collect the hidden states of each layer during inference on these samples. Next, we calculate the BI scores based on the collected hidden states. Finally, we sort layers in ascending order according to BI, and remove the layers with the lower BI scores. The number of layers to be removed can vary to trade off efficiency and performance.

3.3 Generative Task
Although ShortGPT demonstrates strong capabilities in multiple-choice tasks (see Table 2), it leads to a significant performance drop in generative tasks such as GSM8K (Cobbe et al., 2021), like other pruning methods such as LLMPruer (Ma et al., 2024) and SliceGPT (Ashkboos et al., 2024). We attribute this performance drop to the accumulated errors during generation, compared to multiple-choice tasks. To address this issue, we propose a training-free dynamic pruning method for generative tasks, ShortGPT-gen, as illustrated in Figure 4b. In ShortGPT-gen, the input tokens skip the same redundant layers as in ShortGPT, and the generated tokens pass through all layers. Since the hidden states of input tokens remain unchanged in the pruned layers, we use their output hidden states of the preceding unpruned layers to obtain keys and values during decoding.

4 Experiments
4.1 Experimental Setup
Models. To validate the effectiveness of our methods, we conduct experiments on existing popular open-source language models, including Llama2-7B (Touvron et al., 2023), Llama2-13B, Baichuan2-7B, and Baichuan2-13B. They are all large language models based on the decoder-only Transformer architecture. LLaMA2 was trained on more than 2 trillion tokens. Baichuan2 was mainly trained in Chinese and its 13-Billion model replaced the RoPE (Su et al., 2024) positional embedding with ALiBi (Press et al., 2021).

Benchmarks. In order to evaluate the changes in the ability of large language models before and after pruning, we conduct comprehensive evaluation from five aspect of multiple-choice tasks: Reasoning: CMNLI (Li et al., 2024), HellaSwag (HeSw) (Zellers et al., 2019), PIQA (Bisk et al., 2020). Language: CHID (Zheng et al., 2019). Knowledge: CommonSenseQA (CoQA) (Reddy et al., 2019), BoolQ (Clark et al., 2019). Examination: MMLU (Hendrycks et al., 2020), CMMLU (Li et al., 2024). Understanding: Race-High/Middle (H/M) (Lai et al., 2017), C3 (Sun et al., 2020) and PG19 (Rae et al., 2019). For generative tasks,
<table>
  <tr>
    <th>LLM</th>
    <th>Method</th>
    <th>Ratio</th>
    <th colspan="6">Benchmarks</th>
    <th>Avg.</th>
    <th>Per.</th>
  </tr>
  <tr>
    <td rowspan="4">Llama2 7B</td>
    <td>Dense</td>
    <td>0.0%</td>
    <td>32.99</td><td>71.26</td><td>77.91</td><td>41.66</td><td>64.62</td><td>71.62</td><td>35.71</td><td>34.19</td><td>43.56</td><td>45.39</td><td>32.92</td><td>50.17</td><td>100.00</td>
  </tr>
  <tr>
    <td>LLMPruner</td>
    <td>27.0%</td>
    <td><b>34.33</b></td><td><b>56.46</b></td><td><b>71.22</b></td><td>25.25</td><td>42.51</td><td>55.20</td><td>22.56</td><td>22.35</td><td>25.64</td><td>23.33</td><td>25.25</td><td>36.74</td><td>73.23</td>
  </tr>
  <tr>
    <td>SliceGPT</td>
    <td>26.4%</td>
    <td>31.70</td><td>50.27</td><td>66.21</td><td>20.79</td><td>41.36</td><td>38.32</td><td>21.07</td><td>21.66</td><td><b>39.78</b></td><td>28.92</td><td>25.37</td><td>35.04</td><td>69.84</td>
  </tr>
  <tr>
    <td>LaCo</td>
    <td>27.1%</td>
    <td>34.43</td><td>55.69</td><td>69.80</td><td><b>36.14</b></td><td>45.70</td><td>64.07</td><td>22.61</td><td>23.61</td><td>39.67</td><td>26.45</td><td>25.24</td><td>40.31</td><td>80.35</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>27.1%</td>
    <td>32.95</td><td>53.02</td><td>66.43</td><td>24.68</td><td><b>47.99</b></td><td><b>74.71</b></td><td><b>32.25</b></td><td><b>35.17</b></td><td>39.62</td><td><b>43.96</b></td><td><b>32.25</b></td><td><b>43.91</b></td><td><b>87.52</b></td>
  </tr>
  <tr>
    <td rowspan="4">Llama2 13B</td>
    <td>Dense</td>
    <td>0.0%</td>
    <td>32.99</td><td>74.78</td><td>79.71</td><td>47.35</td><td>66.91</td><td>82.39</td><td>57.95</td><td>60.38</td><td>47.51</td><td>55.00</td><td>38.40</td><td>58.49</td><td>100.00</td>
  </tr>
  <tr>
    <td>LLMPruner</td>
    <td>24.4%</td>
    <td><b>33.03</b></td><td><b>67.76</b></td><td><b>76.66</b></td><td>35.64</td><td>50.86</td><td>56.42</td><td>22.47</td><td>22.08</td><td>32.33</td><td>25.21</td><td>24.71</td><td>40.65</td><td>69.50</td>
  </tr>
  <tr>
    <td>SliceGPT</td>
    <td>23.6%</td>
    <td>29.82</td><td>55.71</td><td>69.04</td><td>19.31</td><td>47.26</td><td>37.86</td><td>23.41</td><td>24.03</td><td>41.92</td><td>37.14</td><td>25.79</td><td>37.39</td><td>63.93</td>
  </tr>
  <tr>
    <td>LaCo</td>
    <td>24.6%</td>
    <td>32.86</td><td>64.39</td><td>63.20</td><td><b>40.10</b></td><td>52.66</td><td><b>63.98</b></td><td>54.49</td><td>56.55</td><td>44.93</td><td>45.93</td><td>32.62</td><td>50.16</td><td>85.76</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>24.6%</td>
    <td>33.00</td><td>66.64</td><td>73.45</td><td>36.61</td><td><b>58.64</b></td><td>62.48</td><td><b>58.35</b></td><td><b>60.17</b></td><td><b>46.90</b></td><td><b>54.69</b></td><td><b>38.38</b></td><td><b>53.57</b></td><td><b>91.59</b></td>
  </tr>
  <tr>
    <td rowspan="4">Baichuan2 7B</td>
    <td>Dense</td>
    <td>0.0%</td>
    <td>33.37</td><td>67.56</td><td>76.17</td><td>85.56</td><td>63.14</td><td>74.10</td><td>52.63</td><td>51.04</td><td>64.55</td><td>53.87</td><td>56.95</td><td>61.72</td><td>100.00</td>
  </tr>
  <tr>
    <td>LLMPruner</td>
    <td>24.2%</td>
    <td>32.28</td><td>53.66</td><td><b>71.82</b></td><td>69.80</td><td><b>47.83</b></td><td>61.19</td><td>21.96</td><td>22.28</td><td>41.64</td><td>24.93</td><td>25.69</td><td>43.01</td><td>69.69</td>
  </tr>
  <tr>
    <td>SliceGPT</td>
    <td>22.2%</td>
    <td>32.07</td><td>25.59</td><td>50.33</td><td>14.85</td><td>19.57</td><td>39.30</td><td>23.53</td><td>22.49</td><td>26.58</td><td>25.18</td><td>25.25</td><td>27.68</td><td>44.85</td>
  </tr>
  <tr>
    <td>LaCo</td>
    <td>24.2%</td>
    <td>33.00</td><td>52.28</td><td>68.50</td><td><b>76.24</b></td><td>47.26</td><td>56.15</td><td>28.99</td><td>27.72</td><td>50.85</td><td>31.53</td><td>31.24</td><td>45.80</td><td>74.21</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>24.2%</td>
    <td><b>33.30</b></td><td><b>56.96</b></td><td>67.68</td><td>65.63</td><td>46.70</td><td><b>67.83</b></td><td><b>53.26</b></td><td><b>46.76</b></td><td><b>56.33</b></td><td><b>45.77</b></td><td><b>47.87</b></td><td><b>53.46</b></td><td><b>86.62</b></td>
  </tr>
  <tr>
    <td rowspan="4">Baichuan2 13B</td>
    <td>Dense</td>
    <td>0.0%</td>
    <td>33.21</td><td>71.10</td><td>78.07</td><td>86.51</td><td>65.60</td><td>77.89</td><td>67.27</td><td>68.94</td><td>65.64</td><td>59.50</td><td>61.30</td><td>66.82</td><td>100.00</td>
  </tr>
  <tr>
    <td>LLMPruner</td>
    <td>24.3%</td>
    <td><b>33.80</b></td><td><b>53.71</b></td><td><b>71.82</b></td><td>72.77</td><td>38.82</td><td>56.54</td><td>21.17</td><td>21.61</td><td>39.89</td><td>23.19</td><td>25.18</td><td>41.67</td><td>62.36</td>
  </tr>
  <tr>
    <td>SliceGPT</td>
    <td>22.8%</td>
    <td>32.07</td><td>25.85</td><td>51.03</td><td>14.00</td><td>18.02</td><td>37.83</td><td>21.56</td><td>21.52</td><td>24.99</td><td>22.95</td><td>25.26</td><td>26.50</td><td>39.66</td>
  </tr>
  <tr>
    <td>LaCo</td>
    <td>24.7%</td>
    <td>33.03</td><td><b>60.71</b></td><td>68.88</td><td>76.73</td><td><b>55.45</b></td><td>62.35</td><td><b>56.92</b></td><td><b>57.80</b></td><td><b>61.10</b></td><td>51.35</td><td>53.65</td><td>58.00</td><td>86.80</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>24.7%</td>
    <td>32.81</td><td>60.55</td><td><b>71.60</b></td><td><b>80.17</b></td><td>54.30</td><td><b>62.54</b></td><td>55.77</td><td>56.41</td><td>60.16</td><td><b>52.11</b></td><td><b>58.86</b></td><td><b>58.66</b></td><td><b>87.79</b></td>
  </tr>
</table>

Table 2: Comparison of pruning methods on multiple-choice natural language benchmarks. The last column reports the relative performance retention.

we use three benchmarks, including Xsum (Hasan et al., 2021), GSM8K (Cobbe et al., 2021) and StrategyQA (Geva et al., 2021).

Baselines. To evaluate the effectiveness of our methods, we compared several structured pruning methods for large language models, including:

1) LLMPruner (Ma et al., 2024), which adopts structured pruning that selectively removes non-critical coupled structures based on gradient information, maximally preserving the majority of the LLM’s functionality. LLMPruner applies post training to the pruned model, but for fair comparison, we do not apply post training to it.

2) SliceGPT (Ashkboos et al., 2024), which is a post-training scheme that replaces each weight matrix with a smaller matrix, reducing the embedding dimension of the network. Specifically, it applies PCA to the hidden representation from shallow to deep layers, and incorporates the dimension reduction matrix into the existing network.

3) LaCo (Yang et al., 2024), which is a pruning method for large language models based on reducing layers. LaCo gradually merges similar layers from deep to shallow and sets a threshold to avoid continuously merging too many layers.

4.2 Main Results

We conduct comparative experiments against baselines commonly employed in large language models. Considering the current structured pruning methods generally use reduce ratios no more than 30% (Ma et al., 2024; Ashkboos et al., 2024; Yang et al., 2024; Namburi et al., 2023; Michel et al., 2019; Gordon et al., 2020), we perform experiments with approximately 25% of the layers pruned. We use PG19 as the calibration dataset for layer importance and perplexity calculation. We list the removed layers and investigate the effects of calibration datasets in Appendix A.

The experimental results of the multiple-choice tasks are presented in Table 2. The evaluation on the generative benchmarks are shown in Table 3. Additional experiments exploring different parameter reduction proportions will be discussed in the subsequent sections.

The results demonstrate that our pruning methods surpass the baseline methods, maintaining most of the models’ capabilities. Furthermore, we note that the approach of reducing layers (ShortGPT/LaCo) outperforms the method of reducing the embedding dimensions (LLMPruner/SliceGPT) in multiple-choice tasks. Further experimental anal-

<table>
  <tr>
    <th>LLM</th>
    <th>Method</th>
    <th>Benchmarks</th>
    <th>Xsum</th>
    <th>GSM8K</th>
    <th>StrategyQA</th>
    <th>Avg.</th>
    <th>Per.</th>
  </tr>
  <tr>
    <td rowspan="4">Llama2 7B</td>
    <td>Dense</td>
    <td>18.54</td><td>13.90</td><td>31.53</td><td>28.59</td><td>100.00</td>
  </tr>
  <tr>
    <td>LLMPruner</td>
    <td>11.51</td><td>10.41</td><td>44.20</td><td>18.77</td><td>65.65</td>
  </tr>
  <tr>
    <td>SliceGPT</td>
    <td>4.89</td><td>3.34</td><td>45.70</td><td>17.98</td><td>62.89</td>
  </tr>
  <tr>
    <td>LaCo*</td>
    <td>13.01</td><td>1.44</td><td>39.08</td><td>17.84</td><td>62.40</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>11.13</td><td>1.60</td><td>30.26</td><td>14.33</td><td>50.12</td>
  </tr>
  <tr>
    <td>ShortGPT-gen</td>
    <td><b>14.88</b></td><td><b>14.71</b></td><td><b>48.95</b></td><td><b>26.18</b></td><td><b>91.57</b></td>
  </tr>
  <tr>
    <td rowspan="4">Llama2 13B</td>
    <td>Dense</td>
    <td>22.19</td><td>28.88</td><td>63.58</td><td>38.22</td><td>100.00</td>
  </tr>
  <tr>
    <td>LLMPruner</td>
    <td>10.47</td><td>9.01</td><td>41.70</td><td>21.56</td><td>56.49</td>
  </tr>
  <tr>
    <td>SliceGPT</td>
    <td>5.27</td><td>1.90</td><td>38.30</td><td>15.16</td><td>59.67</td>
  </tr>
  <tr>
    <td>LaCo*</td>
    <td>15.18</td><td>2.58</td><td>43.88</td><td>20.55</td><td>53.77</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>17.59</td><td>2.35</td><td>46.00</td><td>21.98</td><td>57.51</td>
  </tr>
  <tr>
    <td>ShortGPT-gen</td>
    <td><b>21.76</b></td><td><b>26.91</b></td><td><b>62.70</b></td><td><b>37.12</b></td><td><b>97.12</b></td>
  </tr>
</table>

Table 3: Evaluation on generative benchmarks. “*” indicates our reproduced results.
Figure 5: Comparison of different importance metrics on ShortGPT. Perplexity is calculated by removing each single layer, other metrics are calculated by hidden states of each layer.

ysis will be presented in the ensuing sections.

In order to make a more fair comparison with LLMPruner and SliceGPT, we compared them with the same settings as their original papers in Appendix C. These experiments further demonstrate that ShortGPT achieves superior performance compared to the other pruning methods.

The results show that coarse-grained pruning methods, such as removing layers, often outperform fine-grained methods like SliceGPT or LLMPruner. We speculate that the reason is that the LLM is actually very robust, as shown in Figure 2, removing any deep layer individually actually has very little impact on the final output, which means it is difficult to define the importance of a finer-grained module and perform pruning.

4.3 Varying metric and pruning ratio

The core principle of our methods is to rank layers by their importance and remove the less significant ones. The choice of importance metric significantly influences the outcome. In this section, we define and compare several different importance metrics:

• Sequential: The importance is directly proportional to the sequence order, with shallower layers being less important. This can be implemented by assigning the negative value of each layer’s index as its importance metric.

• Norm/Reverse-order: This metric posits that importance is inversely proportional to the sequence order. It assigns higher importance scores to the shallower layers. This method gives the same order as measuring importance by hidden states norm as Figure 5 shows.

• Relative Magnitude: Proposed in (Samragh et al., 2023), this metric assumes layers with larger \( \frac{|f(x)|}{|x + f(x)|} \) are of higher importance, where \( f \) is the layer transformation function.

• BI: we calculate the BI score mentioned in Section 3.1 as importance metric.

Figure 5 shows the different metrics. We observe that shallower layers in the LLM are more crucial than deeper ones. Figure 6 shows the results of removing layers by different metrics, demonstrating that our proposed BI outperforms other metrics. The method of Relative Magnitude is highly competitive, indicating that relative values can also reflect the importance to some extent. It is worth noting that the hidden states norm seems to be a good metric when only considering the MMLU benchmark, but the perplexity is relatively poor.
Figure 6: Performance of ShortGPT on MMLU and perplexity when we prune by different metrics, with increasing pruning ratio. We can see that as the pruning ratio increases, the performance of the model declines.

<table>
  <tr>
    <th>Model</th>
    <th>Ratio</th>
    <th>CMNLI</th>
    <th>HeSw</th>
    <th>PIQA</th>
    <th>CHID</th>
    <th>CoQA</th>
    <th>BoolQ</th>
    <th>Race-H</th>
    <th>Race-M</th>
    <th>C3</th>
    <th>MMLU</th>
    <th>CMMLU</th>
    <th>Avg.</th>
    <th>Per.</th>
  </tr>
  <tr>
    <td rowspan="5">Mamba 2.8B</td>
    <td>0.0%</td>
    <td>35.97</td>
    <td>61.84</td>
    <td>75.52</td>
    <td>35.56</td>
    <td>56.35</td>
    <td>60.67</td>
    <td>24.90</td>
    <td>25.30</td>
    <td>42.08</td>
    <td>26.29</td>
    <td>25.32</td>
    <td>42.71</td>
    <td>100.00</td>
  </tr>
  <tr>
    <td>10.9%</td>
    <td>32.95</td>
    <td>59.71</td>
    <td>73.01</td>
    <td>32.52</td>
    <td>52.66</td>
    <td>51.41</td>
    <td>24.27</td>
    <td>25.21</td>
    <td>41.10</td>
    <td>26.01</td>
    <td>25.00</td>
    <td>40.35</td>
    <td>94.47</td>
  </tr>
  <tr>
    <td>20.3%</td>
    <td>31.29</td>
    <td>55.69</td>
    <td>69.64</td>
    <td>29.12</td>
    <td>48.32</td>
    <td>62.20</td>
    <td>23.61</td>
    <td>23.61</td>
    <td>41.59</td>
    <td>25.69</td>
    <td>25.37</td>
    <td>39.65</td>
    <td>92.84</td>
  </tr>
  <tr>
    <td>25.0%</td>
    <td>29.96</td>
    <td>52.38</td>
    <td>68.77</td>
    <td>26.02</td>
    <td>44.96</td>
    <td>62.20</td>
    <td>23.67</td>
    <td>23.26</td>
    <td>40.71</td>
    <td>24.32</td>
    <td>24.89</td>
    <td>38.29</td>
    <td>90.37</td>
  </tr>
  <tr>
    <td>31.3%</td>
    <td>28.25</td>
    <td>47.02</td>
    <td>64.91</td>
    <td>21.38</td>
    <td>44.96</td>
    <td>62.17</td>
    <td>21.87</td>
    <td>22.77</td>
    <td>40.44</td>
    <td>24.48</td>
    <td>24.77</td>
    <td>36.64</td>
    <td>85.79</td>
  </tr>
  <tr>
    <td rowspan="5">RWKV 7B</td>
    <td>0.0%</td>
    <td>32.07</td>
    <td>65.98</td>
    <td>77.09</td>
    <td>85.36</td>
    <td>62.65</td>
    <td>62.72</td>
    <td>38.56</td>
    <td>45.47</td>
    <td>57.97</td>
    <td>31.85</td>
    <td>28.54</td>
    <td>53.48</td>
    <td>100.00</td>
  </tr>
  <tr>
    <td>9.4%</td>
    <td>32.60</td>
    <td>56.41</td>
    <td>73.94</td>
    <td>78.12</td>
    <td>49.55</td>
    <td>62.35</td>
    <td>25.90</td>
    <td>25.77</td>
    <td>54.68</td>
    <td>27.29</td>
    <td>25.03</td>
    <td>46.51</td>
    <td>86.97</td>
  </tr>
  <tr>
    <td>18.8%</td>
    <td>32.11</td>
    <td>49.47</td>
    <td>71.55</td>
    <td>65.63</td>
    <td>40.54</td>
    <td>61.19</td>
    <td>22.04</td>
    <td>23.75</td>
    <td>49.15</td>
    <td>26.35</td>
    <td>25.00</td>
    <td>42.43</td>
    <td>79.34</td>
  </tr>
  <tr>
    <td>25.0%</td>
    <td>32.41</td>
    <td>39.73</td>
    <td>65.13</td>
    <td>52.60</td>
    <td>29.65</td>
    <td>60.92</td>
    <td>22.56</td>
    <td>21.59</td>
    <td>41.86</td>
    <td>25.52</td>
    <td>25.08</td>
    <td>37.91</td>
    <td>70.89</td>
  </tr>
  <tr>
    <td>28.1%</td>
    <td>33.11</td>
    <td>32.22</td>
    <td>60.01</td>
    <td>32.47</td>
    <td>28.34</td>
    <td>60.85</td>
    <td>22.27</td>
    <td>21.31</td>
    <td>37.81</td>
    <td>25.64</td>
    <td>25.15</td>
    <td>34.47</td>
    <td>64.45</td>
  </tr>
</table>

Table 4: ShortGPT on Mamba and RWKV.

We further validated the effects of different pruning ratios on ShortGPT. Experiments are conducted on the Llama2 and Baichuan2 models, observing the perplexity and MMLU. The results for Llama2, as shown in Figure 6, demonstrate that the models’ performance generally declines as the pruning ratio increases. However, we observe a notable phenomenon: the MMLU score exhibits a sharp drop at a specific layer. This sudden decrease suggests the presence of certain critical layers within the network that play a particularly important role in maintaining performance. Similar patterns are observed in the Baichuan2 models, as illustrated in Appendix B.

4.4 Redundancy on non-transformer LLMs

To determine whether the observed layer redundancy is specific to Transformers, we extend our investigation to include two popular non-Transformer models, RWKV-7B (Peng et al., 2023) and Mamba-2.8B (Gu and Dao, 2023). Our experiments reveal that these models also exhibit resilience to layer removal, maintaining performance despite the elimination of certain layers. Table 4 shows that ShortGPT is applicable and effective for Mamba and RWKV models, suggesting that the redundancy is universal across current LLMs.

4.5 Post training to restore performance

To mitigate the performance loss resulting from layer removal in ShortGPT, we explore post-
<table>
  <tr>
    <th>Method</th>
    <th>Ratio</th>
    <th>CMNLI</th>
    <th>HeSw</th>
    <th>PIQA</th>
    <th>CHID</th>
    <th>CoQA</th>
    <th>BowQ</th>
    <th>Race-H</th>
    <th>Race-M</th>
    <th>C3</th>
    <th>MMLU</th>
    <th>CMMLU</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Dense</td>
    <td>0.0%</td>
    <td>32.99</td>
    <td>71.26</td>
    <td>77.91</td>
    <td>41.66</td>
    <td>64.62</td>
    <td>71.62</td>
    <td>35.71</td>
    <td>34.19</td>
    <td>43.56</td>
    <td>45.39</td>
    <td>32.92</td>
    <td>50.17</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>27.1%</td>
    <td>32.95</td>
    <td>53.02</td>
    <td>66.43</td>
    <td>24.68</td>
    <td>47.99</td>
    <td>74.41</td>
    <td>32.25</td>
    <td>35.17</td>
    <td>39.62</td>
    <td>43.96</td>
    <td>32.25</td>
    <td>43.88</td>
  </tr>
  <tr>
    <td>ShortGPT+post-train</td>
    <td>24.0%</td>
    <td>32.99</td>
    <td>54.83</td>
    <td>68.12</td>
    <td>31.82</td>
    <td>58.32</td>
    <td>72.36</td>
    <td>34.18</td>
    <td>34.68</td>
    <td>40.37</td>
    <td>44.47</td>
    <td>32.73</td>
    <td>45.90</td>
  </tr>
</table>

Table 5: Post training Llama2-7B to restore performance.

<table>
  <tr>
    <th>Method</th>
    <th>Ratio/Layer</th>
    <th>Perplexity</th>
    <th>MMLU</th>
    <th>Throughput (speed up)</th>
  </tr>
  <tr>
    <td>Baseline</td>
    <td>0.0%/32</td>
    <td>8.03</td>
    <td>43.17</td>
    <td>4331.23 Tokens/s (1.00x)</td>
  </tr>
  <tr>
    <td></td>
    <td>3.1%/31</td>
    <td>8.37</td>
    <td>42.88</td>
    <td>4399.31 Tokens/s (1.02x)</td>
  </tr>
  <tr>
    <td></td>
    <td>9.4%/29</td>
    <td>9.44</td>
    <td>42.31</td>
    <td>4602.26 Tokens/s (1.06x)</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>12.5%/28</td>
    <td>10.24</td>
    <td>41.62</td>
    <td>4680.68 Tokens/s (1.08x)</td>
  </tr>
  <tr>
    <td></td>
    <td>15.6%/27</td>
    <td>11.42</td>
    <td>43.17</td>
    <td>4756.94 Tokens/s (1.10x)</td>
  </tr>
  <tr>
    <td></td>
    <td>25.0%/24</td>
    <td>22.29</td>
    <td>41.68</td>
    <td>5045.59 Tokens/s (1.16x)</td>
  </tr>
  <tr>
    <td></td>
    <td>27.1%/23</td>
    <td>40.78</td>
    <td>43.35</td>
    <td>5146.99 Tokens/s (1.19x)</td>
  </tr>
</table>

Table 6: ShortGPT on Llama2-7B-Base-GPTQ.

<table>
  <tr>
    <th>Method</th>
    <th>MMLU</th>
    <th>CMMLU</th>
  </tr>
  <tr>
    <td>Llama2-7B-Baseline</td>
    <td>45.4</td>
    <td>32.9</td>
  </tr>
  <tr>
    <td>4-bit quantization</td>
    <td>44.9</td>
    <td>32.5</td>
  </tr>
  <tr>
    <td>ShortGPT (27.1%)</td>
    <td>44.0</td>
    <td>32.3</td>
  </tr>
  <tr>
    <td>4-bit quantization then ShortGPT</td>
    <td>42.4</td>
    <td>31.0</td>
  </tr>
  <tr>
    <td>ShortGPT then 4-bit quantization</td>
    <td>41.2</td>
    <td>30.5</td>
  </tr>
</table>

Table 7: Performance comparison of different methods

training strategies inspired by (Chen et al., 2024). Our approach consists of two key steps: 1) Replacement: We substitute the removed layers with lightweight Multi-Layer Perceptrons (MLPs). 2) Retraining: We subsequently retrain the modified model. The results in Table 5 demonstrate the potential of post training to recover performance loss. Appendix D shows the post training settings.

4.6 Orthogonal to Quantization

In this section, we show that ShortGPT is orthogonal to quantization. We apply ShortGPT to Llama2-7B quantized by GPTQ algorithm. Table 6 shows that ShortGPT is compatible with quantization, and applying it to the quantized model can further improve its efficiency. In addition, we compare the performance of the model applied ShortGPT before and after quantization. The results shown in Table 7 further indicates that quantization and ShortGPT are orthogonal operations.

5 Related works

Pruning: pruning (LeCun et al., 1989; Han et al., 2015) aims to reduce model redundancy, including static pruning and dynamic pruning. Static pruning (Dong et al., 2017; Fang et al., 2023) removes model components based on various metrics. LLM-Pruner (Ma et al., 2024) removes structures according to gradient information. LaCo (Yang et al., 2024) uses layer merging. Dynamic pruning (Tang et al., 2021; Hua et al., 2019; Gao et al., 2019) preserves the entire model and accelerates models by skipping unimportant components. Dynamic pruning methods typically do not perform fine-tuning or retraining (Cheng et al., 2024). Ada-Infer (Fan et al., 2024) introduces early-exit to stop forward propagation at intermediate layers, but its performance on generative benchmarks such as GSM8K decreases to nearly zero. In our methods, ShortGPT is a static pruning method, while ShortGPT-gen is a dynamic pruning method. Notably, ShortGPT-gen preserves most of the performance of dense models on generative tasks in a training-free manner.

Quantization: quantization (Liu et al., 2021; Gholami et al., 2022; Dettmers et al., 2022, 2024) can save computational costs of deep learning models. It converts models’ floating-point weights into integers or other discrete forms. LUT-GEMM (Park et al., 2022) quantifies weights and optimizes matrix multiplication in LLMs using BCQ format. SPQR (Dettmers et al., 2023) identifies and isolates abnormal weights, stores them with higher accuracy and converts others into 3-4 bits. Our pruning methods and quantization method are orthogonal.

Model redundancy: researchers have long noticed significant redundancy in nonlinear models (Catchpole and Morgan, 1997). Recently, Transformer models have been widely applied. Researchers have also studied their redundancy. In (Bian et al., 2021), researchers analyzed redundancy in attention mechanisms, where similar redundancy patterns are observed among attention heads. In (Dalvi et al., 2020), researchers dissect BERT (Devlin et al., 2018) and XLNet (Yang et al., 2019), studying how much redundancy they exhibit at representation level and the fine-grained neuron level. However, the redundancy of decoder-only LLMs still needs to be explored.

6 Conclusion

In this work, we uncovered the significant layer-wise redundancy of LLMs. Our research demonstrates that certain layers contribute minimally to
overall network functionality and can be removed without substantially compromising model performance. Based on our observation, we introduce Block influence to quantify the importance of each layer and propose simple and straightforward pruning methods for multiple-choice tasks and generative tasks. Our experiments demonstrates that it is possible to maintain approximately 90% of a LLM’s performance while reducing the computational requirements in approximately 25% of the LLM’s layers. Our work suggests potential avenues for improving the efficiency of models by reducing their inherent redundancy.

7 Limitations

Although our methods demonstrates strong competitiveness compared to current methods, there are some limitations that must be acknowledged. The negative effect of ShortGPT is significant on certain generative tasks, which creates barriers to its practical use. When we remove 25% layers from Llama2-7B or Baichuan2-7B, the performance on GSM8K deceases to nearly zero. In contrast, ShortGPT-gen preserves most of models’ performance on generative tasks, but it does not reduce the sizes of models.

Acknowledgments

We sincerely thank the reviewers for their insightful comments and valuable suggestions. This work was supported by Beijing Natural Science Foundation (L243006), Beijing Municipal Science and Technology Project (Nos. Z231100010323002), the Natural Science Foundation of China (No. 62476265, 62306303).

References

Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoeﬂer, and James Hensman. 2024. Sliceqpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024.

Yuchen Bian, Jiaji Huang, Xingyu Cai, Jiahong Yuan, and Kenneth Church. 2021. On attention redundancy: A comprehensive study. In Proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 930–945.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, pages 7432–7439.

Edward A Catchpole and Byron JT Morgan. 1997. Detecting parameter redundancy. Biometrika, 84(1):187–196.

Xiaodong Chen, Yuxuan Hu, and Jing Zhang. 2024. Compressing large language models by streamlining the unimportant layer. arXiv preprint arXiv:2403.19135.

Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. 2024. A survey on deep neural network pruning-taxonomy, comparison, analysis, and recommendations. arXiv preprint arXiv:2308.06767.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. 2020. Analyzing redundancy in pretrained transformer models. arXiv preprint arXiv:2004.04010.

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm.int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36.

Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznecov, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoeﬂer, and Dan Al-istarh. 2023. Sqpr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. 2017. More is less: A more complicated network with less inference complexity. arXiv preprint arXiv:1703.08651.
Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. 2024. Not all layers of llms are necessary during inference. arXiv preprint arXiv:2403.02181.

Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. 2023. Depgraph: Towards any structural pruning. arXiv preprint arXiv:2301.12900.

Elias Frantar and Dan Alistarh. 2023. Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.00774.

Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng zhong Xu. 2019. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint arXiv:1810.05331.

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. arXiv preprint arXiv:2101.02235.

Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 2022. A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pages 291–326. Chapman and Hall/CRC.

Mitchell A. Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing bert: Studying the effects of weight pruning on transfer learning. arXiv preprint arXiv:2002.08307.

Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752.

Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28.

Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Mubashir, Yuan-Fang Li, Yong-Bin Kang, M Sohel Rahman, and Rifat Shahriyar. 2021. Xi-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693–4703.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Weßl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.

Weizhe Hua, Yuan Zhou, Christopher De Sa, Zhiru Zhang, and G. Edward Suh. 2019. Channel gating neural networks. arXiv preprint arXiv:1805.12549.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.

Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794.

Yann LeCun, John Denker, and Sara Solla. 1989. Optimal brain damage. Advances in neural information processing systems, 2.

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212.

Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. 2020. Understanding the difficulty of training transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5747–5763.

Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. 2021. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34:28092–28103.

Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2024. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36.

Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? arXiv preprint arXiv:1905.10650.

Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2022. Fast model editing at scale. arXiv preprint arXiv:2110.11309.

Satya Sai Srinath Namburi, Makesh Sreedhar, Srinath Srinivasan, and Frederic Sala. 2023. The cost of compression: Investigating the impact of compression on parametric knowledge in language models. arXiv preprint arXiv:2312.00960.

Gunho Park, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. 2022. nuqmm: Quantized matmul for efficient inference of large-scale generative language models. arXiv preprint arXiv:2206.09557.

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al.
2023. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048.

Ofir Press, Noah A Smith, and Mike Lewis. 2021. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409.

Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. 2019. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations.

Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266.

Mohammad Samragh, Mehrdad Farajtabar, Sachin Mehta, Raviteja Vemulapalli, Fartash Faghri, Devang Naik, Oncel Tuzel, and Mohammad Rastegari. 2023. Weight subcloning: direct initialization of transformers using larger pretrained ones. arXiv preprint arXiv:2312.09299.

Jianlin Su, Muratdha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.

Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2020. Investigating prior knowledge for challenging chinese machine reading comprehension. Transactions of the Association for Computational Linguistics, 8:141–155.

Yehui Tang, Yunhe Wang, Yixing Xu, Yiping Deng, Chao Xu, Dacheng Tao, and Chang Xu. 2021. Manifold regularized dynamic network pruning. arXiv preprint arXiv:2103.05861.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.

Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. 2024. Deepnet: Scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence.

Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. 2020. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pages 10524–10533. PMLR.

Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305.

Yifei Yang, Zouying Cao, and Hai Zhao. 2024. Laco: Large language model pruning via layer collapse. arXiv preprint arXiv:2402.11187.

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800.

Mingyang Zhang, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, Bohan Zhuang, et al. 2023. Pruning meets low-rank parameter-efficient fine-tuning. arXiv preprint arXiv:2305.18403.

Chujie Zheng, Minlie Huang, and Aixin Sun. 2019. Chid: A large-scale chinese idiom dataset for cloze test. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 778–787.

A Detailed Strategies for Layer Removal

We list different layer removal strategies in Table 8. The removed layers in Table 2 and Table 3 are listed in Table 9. To investigate the effects of calibration datasets, we also list the removed layers of Llama2 models when using MMLU as calibration dataset in Table 10. We can observe that the calibrations datasets hardly affect the removed layers, demonstrating the robust of our BI metric.

B ShortGPT on Baichuan2 models

The effects of different pruning ratios on Baichuan2 models’ performance are shown in Figure 7.

C A Fair comparison with SliceGPT and LLMPruer

For a fair comparison with LLMPruer and SliceGPT, we do the same experiments as in the original papers of LLMPruer and SliceGPT. The results are provided in Table 11 and Table 12. We take the same settings as the corresponding paper. The results demonstrate that our method is highly competitive.

D Setup for post training

Table 13 shows the post training settings.
Figure 7: Pruning by different pruning ratios on Baichuan2 models.

<table>
  <tr>
    <th>Strategy</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>Sequential</td>
    <td>Layers are removed sequentially from the beginning of the model. The process starts with layer 0 and progressively includes more layers for removal (e.g., {0}, {0, 1}, ...).</td>
  </tr>
  <tr>
    <td>Reverse-order</td>
    <td>This strategy involves starting from the model’s final layer and progressively removing layers in reverse order (e.g., {-1}, {-1, -2}, ...).</td>
  </tr>
  <tr>
    <td>Relative Magnitude</td>
    <td>Layers are removed in ascending order based on their Relative Magnitude values. The process accumulates layers from those with the smallest to the largest values, mirroring the sequential strategy’s accumulation method.</td>
  </tr>
  <tr>
    <td>BI (Block Influence)</td>
    <td>A similar accumulation approach as the Sequential strategy, but layers are ordered and removed according to their BI scores, starting from the lowest and moving to the highest.</td>
  </tr>
</table>

Table 8: Strategies for Layer Removal in Models.

<table>
  <tr>
    <th>Model</th>
    <th>Removed Layers</th>
  </tr>
  <tr>
    <td>Llama2-7B</td>
    <td>21, 22, 23, 24, 25, 26, 27, 28, 29</td>
  </tr>
  <tr>
    <td>Llama2-13B</td>
    <td>26, 27, 28, 29, 30, 31, 32, 33, 34, 35</td>
  </tr>
  <tr>
    <td>Baichuan2-7B</td>
    <td>22, 23, 24, 25, 26, 27, 28, 29, 30</td>
  </tr>
  <tr>
    <td>Baichuan2-13B</td>
    <td>26, 27, 28, 29, 30, 31, 32, 33, 34, 35</td>
  </tr>
</table>

Table 9: Removed Layers for Benchmark Models, using PG19 as calibration dataset.

<table>
  <tr>
    <th>Model</th>
    <th>Removed Layers</th>
  </tr>
  <tr>
    <td>Llama2-7B</td>
    <td>21, 22, 23, 24, 25, 26, 27, 28, 29</td>
  </tr>
  <tr>
    <td>Llama2-13B</td>
    <td>26, 27, 28, 29, 30, 31, 32, 33, 34, 35</td>
  </tr>
</table>

Table 10: Removed Layers for Llama2 models, using MMLU as calibration dataset.

E   Setup for training post-norm model and pre-norm model

Table 14 lists the specific training settings for pre norm and post norm in Section 2.1.
<table>
  <tr>
    <th>Model</th>
    <th>Pruning ratio</th>
    <th>Method</th>
    <th>BoolQ</th>
    <th>PIQA</th>
    <th>Hellaswag</th>
    <th>Winogrande</th>
    <th>Arc-e</th>
    <th>Arc-c</th>
    <th>OBQA</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="3">Llama 7B</td>
    <td>Ratio=0%</td>
    <td>Baseline</td>
    <td>73.18</td>
    <td>78.35</td>
    <td>72.99</td>
    <td>67.01</td>
    <td>67.45</td>
    <td>41.38</td>
    <td>42.40</td>
    <td>63.25</td>
  </tr>
  <tr>
    <td>Ratio=20%</td>
    <td>LLMPruner</td>
    <td>59.39</td>
    <td>75.57</td>
    <td>65.34</td>
    <td>61.33</td>
    <td>59.18</td>
    <td>37.12</td>
    <td>39.80</td>
    <td>56.82</td>
  </tr>
  <tr>
    <td>Ratio=21.9 %</td>
    <td>ShortGPT</td>
    <td>68.26</td>
    <td>72.28</td>
    <td>61.70</td>
    <td>63.77</td>
    <td>60.22</td>
    <td>39.00</td>
    <td>41.60</td>
    <td>58.12</td>
  </tr>
  <tr>
    <td rowspan="3">Llama 13B</td>
    <td>Ratio=0%</td>
    <td>Baseline</td>
    <td>68.47</td>
    <td>78.89</td>
    <td>76.24</td>
    <td>70.09</td>
    <td>74.58</td>
    <td>44.54</td>
    <td>42.00</td>
    <td>64.97</td>
  </tr>
  <tr>
    <td>Ratio=20%</td>
    <td>LLMPruner</td>
    <td>67.68</td>
    <td>77.15</td>
    <td>73.41</td>
    <td>65.11</td>
    <td>68.35</td>
    <td>38.40</td>
    <td>42.40</td>
    <td>61.79</td>
  </tr>
  <tr>
    <td>Ratio=20%</td>
    <td>ShortGPT</td>
    <td>68.41</td>
    <td>76.36</td>
    <td>72.90</td>
    <td>67.40</td>
    <td>68.62</td>
    <td>39.20</td>
    <td>41.00</td>
    <td>61.98</td>
  </tr>
</table>

Table 11: Comparison between ShortGPT and LLMPruner. The Table is corresponding to the Table 1 of LLM-Pruner (Zhang et al., 2023).

<table>
  <tr>
    <th>Model</th>
    <th>Pruning ratio</th>
    <th>Method</th>
    <th>PIQA</th>
    <th>Hellaswag</th>
    <th>Winogrande</th>
    <th>Arc-e</th>
    <th>Arc-c</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="6">Llama2 7B</td>
    <td>0%</td>
    <td>Baseline</td>
    <td>79.11</td>
    <td>75.99</td>
    <td>69.06</td>
    <td>74.58</td>
    <td>46.25</td>
    <td>69.00</td>
  </tr>
  <tr>
    <td>20%</td>
    <td>SliceGPT</td>
    <td>71.87</td>
    <td>58.10</td>
    <td>63.04</td>
    <td>69.87</td>
    <td>43.09</td>
    <td>61.19</td>
  </tr>
  <tr>
    <td>25%</td>
    <td>SliceGPT</td>
    <td>68.55</td>
    <td>58.10</td>
    <td>62.04</td>
    <td>57.46</td>
    <td>35.07</td>
    <td>56.24</td>
  </tr>
  <tr>
    <td>30%</td>
    <td>SliceGPT</td>
    <td>66.10</td>
    <td>52.69</td>
    <td>56.82</td>
    <td>35.07</td>
    <td>56.82</td>
    <td>53.50</td>
  </tr>
  <tr>
    <td>21.9%</td>
    <td>ShortGPT</td>
    <td>72.76</td>
    <td>66.39</td>
    <td>66.27</td>
    <td>59.39</td>
    <td>39.85</td>
    <td>60.93</td>
  </tr>
  <tr>
    <td>25%</td>
    <td>ShortGPT</td>
    <td>70.53</td>
    <td>62.68</td>
    <td>64.70</td>
    <td>58.39</td>
    <td>39.51</td>
    <td>59.16</td>
  </tr>
  <tr>
    <td>31.6%</td>
    <td>ShortGPT</td>
    <td>67.87</td>
    <td>62.19</td>
    <td>64.38</td>
    <td>56.57</td>
    <td>40.86</td>
    <td>58.37</td>
  </tr>
  <tr>
    <td rowspan="6">Llama2 13B</td>
    <td>0%</td>
    <td>Baseline</td>
    <td>80.47</td>
    <td>79.39</td>
    <td>72.22</td>
    <td>77.48</td>
    <td>49.23</td>
    <td>71.76</td>
  </tr>
  <tr>
    <td>20%</td>
    <td>SliceGPT</td>
    <td>71.87</td>
    <td>69.38</td>
    <td>63.04</td>
    <td>69.87</td>
    <td>43.09</td>
    <td>63.45</td>
  </tr>
  <tr>
    <td>25%</td>
    <td>SliceGPT</td>
    <td>68.55</td>
    <td>67.48</td>
    <td>58.10</td>
    <td>62.50</td>
    <td>37.88</td>
    <td>58.90</td>
  </tr>
  <tr>
    <td>30%</td>
    <td>SliceGPT</td>
    <td>66.10</td>
    <td>65.11</td>
    <td>52.69</td>
    <td>56.82</td>
    <td>35.07</td>
    <td>55.16</td>
  </tr>
  <tr>
    <td>20%</td>
    <td>ShortGPT</td>
    <td>76.95</td>
    <td>74.67</td>
    <td>71.14</td>
    <td>69.56</td>
    <td>45.63</td>
    <td>67.59</td>
  </tr>
  <tr>
    <td>25%</td>
    <td>ShortGPT</td>
    <td>74.39</td>
    <td>71.65</td>
    <td>70.98</td>
    <td>67.09</td>
    <td>43.93</td>
    <td>65.61</td>
  </tr>
  <tr>
    <td>30%</td>
    <td>ShortGPT</td>
    <td>72.11</td>
    <td>71.93</td>
    <td>67.19</td>
    <td>61.09</td>
    <td>40.88</td>
    <td>62.64</td>
  </tr>
  <tr>
    <td rowspan="6">Llama2 70B</td>
    <td>0%</td>
    <td>Baseline</td>
    <td>82.70</td>
    <td>83.84</td>
    <td>77.98</td>
    <td>80.98</td>
    <td>57.34</td>
    <td>76.57</td>
  </tr>
  <tr>
    <td>20%</td>
    <td>SliceGPT</td>
    <td>76.61</td>
    <td>72.98</td>
    <td>74.92</td>
    <td>80.51</td>
    <td>55.20</td>
    <td>72.04</td>
  </tr>
  <tr>
    <td>25%</td>
    <td>SliceGPT</td>
    <td>74.92</td>
    <td>68.74</td>
    <td>74.92</td>
    <td>77.90</td>
    <td>51.71</td>
    <td>69.64</td>
  </tr>
  <tr>
    <td>30%</td>
    <td>SliceGPT</td>
    <td>72.31</td>
    <td>63.69</td>
    <td>73.40</td>
    <td>51.71</td>
    <td>47.61</td>
    <td>61.74</td>
  </tr>
  <tr>
    <td>20%</td>
    <td>ShortGPT</td>
    <td>76.02</td>
    <td>78.87</td>
    <td>71.69</td>
    <td>76.02</td>
    <td>52.95</td>
    <td>71.11</td>
  </tr>
  <tr>
    <td>25%</td>
    <td>ShortGPT</td>
    <td>73.20</td>
    <td>76.72</td>
    <td>71.85</td>
    <td>73.20</td>
    <td>49.90</td>
    <td>68.97</td>
  </tr>
  <tr>
    <td>30%</td>
    <td>ShortGPT</td>
    <td>74.44</td>
    <td>75.31</td>
    <td>72.33</td>
    <td>74.44</td>
    <td>49.22</td>
    <td>69.15</td>
  </tr>
</table>

Table 12: Comparison between ShortGPT and SliceGPT. The Table is corresponding to the Table 7 of SliceGPT (Ashkboos et al., 2024).

<table>
  <tr>
    <th>Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>Global Batch Size</td>
    <td>2048</td>
  </tr>
  <tr>
    <td>Sequence length</td>
    <td>4096</td>
  </tr>
  <tr>
    <td>Precision</td>
    <td>bf16</td>
  </tr>
  <tr>
    <td>Learning Rate Scheduler</td>
    <td>cosine</td>
  </tr>
  <tr>
    <td>Max Learning Rate</td>
    <td>4e-4</td>
  </tr>
  <tr>
    <td>Min Learning Rate</td>
    <td>5e-5</td>
  </tr>
  <tr>
    <td>Warm-up steps</td>
    <td>3000</td>
  </tr>
  <tr>
    <td>Training Tokens</td>
    <td>200B</td>
  </tr>
  <tr>
    <td>Weight Decay</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Adam Beta1</td>
    <td>0.9</td>
  </tr>
  <tr>
    <td>Adam Beta2</td>
    <td>0.98</td>
  </tr>
  <tr>
    <td>Gradient Clip</td>
    <td>1.0</td>
  </tr>
</table>

Table 13: Post training parameters.

<table>
  <tr>
    <th>Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>Global Batch Size</td>
    <td>2048</td>
  </tr>
  <tr>
    <td>Sequence length</td>
    <td>4096</td>
  </tr>
  <tr>
    <td>Precision</td>
    <td>bf16</td>
  </tr>
  <tr>
    <td>Learning Rate Scheduler</td>
    <td>cosine</td>
  </tr>
  <tr>
    <td>Max Learning Rate</td>
    <td>4e-4</td>
  </tr>
  <tr>
    <td>Min Learning Rate</td>
    <td>5e-5</td>
  </tr>
  <tr>
    <td>Warm-up steps</td>
    <td>3000</td>
  </tr>
  <tr>
    <td>Training Tokens</td>
    <td>50B</td>
  </tr>
  <tr>
    <td>Weight Decay</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Adam Beta1</td>
    <td>0.9</td>
  </tr>
  <tr>
    <td>Adam Beta2</td>
    <td>0.98</td>
  </tr>
  <tr>
    <td>Gradient Clip</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>Tokenizer</td>
    <td>Llama2</td>
  </tr>
  <tr>
    <td>Layers</td>
    <td>32</td>
  </tr>
  <tr>
    <td>Hidden state</td>
    <td>2048</td>
  </tr>
  <tr>
    <td>Attention heads</td>
    <td>32</td>
  </tr>
  <tr>
    <td>Head dim</td>
    <td>64</td>
  </tr>
  <tr>
    <td>FFN size</td>
    <td>5504</td>
  </tr>
  <tr>
    <td>Activation function</td>
    <td>Silu</td>
  </tr>
</table>

Table 14: Training parameters.

Reviewer paper summaries: This paper investigates the layer-wise redundancy in large language models (LLMs) and proposes a metric called Block Influence (BI) to measure the importance of each layer based on input-output similarity. Using this metric, the authors introduce two simple pruning methods: ShortGPT for multiple-choice tasks and ShortGPT-gen for generative tasks. These methods remove redundant layers, reducing up to 25% of the model without much performance loss.
Summary of strengths: 1. The proposed method uses a lightweight metric (Block Influence) that requires only forward passes, making it computationally efficient and easy to apply to large models.

2. By introducing both ShortGPT and ShortGPT-gen, the approach adapts to different task types—classification and generation—enhancing its practical utility.

3. The method is validated on both Transformer and non-Transformer architectures, showing its potential generality beyond a single model family.
Summary of weaknesses: 1. Although the paper shows how to exploit layer redundancy, it doesn’t really propose a fundamentally new pruning strategy. The reliance on a simple cosine-based metric and direct layer removal can feel more like a repackaging of existing insights on layer similarity than a novel method.

2. Even though BI is one of the few components that could be considered a novel contribution in the methodology, it has already been explored through similar cosine-based similarity approaches in [1].

3. The paper doesn’t thoroughly benchmark against certain newer gradient-based pruning approaches (like Shortened Llama [2] and SLEB [3]), making it harder to confirm if the proposed technique truly outperforms or offers a distinct advantage over competing methods.

Citation:  
[1] Yang, Yifei, Zouying Cao, and Hai Zhao. "Laco: Large language model pruning via layer collapse." arXiv preprint arXiv:2402.11187 (2024).  
[2] Kim, Bo-Kyeong, et al. "Shortened llama: A simple depth pruning for large language models." arXiv preprint arXiv:2402.02834 11 (2024).  
[3] Song, Jiwon, et al. "Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks." arXiv preprint arXiv:2402.09025 (2024).
Comments, suggestions and typos: Line 137: LLMs -> LLM

Line 453: Our -> our
Overall assessment scores: 3.5
Metareview score: 3
Metareview: This paper proposes a technique to remove redundant layers from large language models to enhance its computational efficiency. The idea is to remove transformer blocks with low block importance score; if the task is to generate further tokens, one can optionally use all layers for processing the generated tokens as well. Empirically, the suggested approach outperforms or matches the performance of other structured pruning techniques.

The reviews are quite positive in general, praising the practical usability of the proposed method with good validations. The concerns of the reviewers are primarily about the novelty of the proposed technique and relatively weak justification of the proposed method (mostly backed up by empirical observations only). After the response phase, the concerns are resolved partially, although the paper can still benefit from supplementing further justifications behind the method design.

Also, here's a short AC note: Several things about ShortGPT-gen is unclear to me. (1) How large is the computational footprint of the K/V recovery procedure for the dropped layers, before generation? (2) Is the comparison at Table 3 fair? It seems like: in terms of memory, ShortGPT-gen will require loading full model weights, requiring more RAM than baselines; in terms of compute, ShortGPT-gen will require materializing missing KV and full model compute for generated tokens. If I understood correctly, comparing at the same sparsity does not make sense.

If these final notes are not satisfactorily resolved at the commitment stage, I believe that this paper is not ready for publication at this round. If they can be satisfactorily addressed, I recommend commiting.
Summary of reasons to publish: Here are some notable strengths:
- The proposed method is reasonably designed and can provide hardware-agnostic acceleration without elaborate tricks.
- The method is evaluated quite comprehensively (except for shortGPT-gen).
- The paper is clearly written and easy to understand.