The Unreasonable Ineffectiveness of the Deeper Layers
Authors: Andrey Gromov
Kushal Tirumala
Hassan Shapourian
Paolo Glorioso
Dan Roberts
Venue: ICLR 2025 Poster (2025)
Keywords: NLP
Pruning
Science of Deep Learning
Efficient Inference
Primary area: foundation or frontier models, including LLMs
TL;DR: We use model pruning as tool to understand how and where knowledge is located in open-weight LLMs: we find that we can remove up to half the layers of Llama-2 70B with essentially no impact on performance on QA benchmarks.

Abstract: How is knowledge stored in an LLM’s weights? We study this via layer pruning: if removing a certain layer does not affect model performance in common question-answering benchmarks, then the weights in that layer are not necessary for storing the knowledge needed to answer those questions. To find these unnecessary parameters, we identify the optimal block of layers to prune by considering similarity across layers; then, to “heal” the damage, we perform a small amount of finetuning. Surprisingly, with this method we find minimal degradation of performance until after a large fraction (up to half) of the layers are removed for some common open-weight models. From a scientific perspective, the robustness of these LLMs to the deletion of layers implies either that current pretraining methods are not properly leveraging the parameters in the deeper layers of the network or that the shallow layers play a critical role in storing knowledge. For our study, we use parameter-efficient finetuning (PEFT) methods, specifically quantization and Low Rank Adapters (QLoRA), such that each of our experiments can be performed on a single 40GB A100 GPU.

Published as a conference paper at ICLR 2025
THE UNREASONABLE INEFFECTIVENESS OF THE DEEPER LAYERS
Andrey Gromov Meta FAIR & UMD
Kushal Tirumala" Meta FAIR
Hassan Shapourian Cisco
Paolo Glorioso Zyphra
Daniel A. Roberts MIT & Sequoia Capital
ABSTRACT
How is knowledge stored in an LLM's weights? We study this via layer pruning: if removing a certain layer does not affect model performance in common question- answering benchmarks, then the weights in that layer are not necessary for storing the knowledge needed to answer those questions. To find these unnecessary param- eters, identify the optimal block of layers to prune by considering similarity across layers; then, to "heal" the damage, we perform a small amount of finetuning. Surprisingly, with this method we find minimal degradation of performance until after a large fraction (up to half) of the layers are removed for some common open-weight models. From scientific perspective, the robustmess of these LLMs to the deletion of layers implies either that current pretraining methods are not properly leveraging the parameters in the deeper layers of the network or that the shallow layers play critical role in storing knowledge. For our study, we use parameter-efficieet finetuning (PEFT) methods, specifically quantization and Low Rank Adapters (QLoRA), such that each of our experiments can be performed on a single 40GB A100 GPU.
INTRODUCTION
In this work we study a very simple pruning strategy using open-weight LLMs. In particular, we develop method that uses the similarity between the representations at different layers to identify the optimal layers to prune for given pruning fraction; then, after removing these layers we "heal" the pruning-inen indiing- mismatch with a small amount of fine tuning (using QLoRA). Our main result is that we can remove a substantial fraction of the deepest layers from models with minimal degradation in downstream question-anse -nswerinn benchmarks. For example, for Llama-2-70B (Touvron al. (023) can climinate up to roughly half of the layers before the performance collapses. An overview of our strategy and the results of pruning 1lama-2-7705 are shown in Figure] [
Our intuition for dropping layers comes from considering the residual structure of the transformer architecture. more detail, the output of the final layer can be decomposed as a sum over the outputs of all the model layers plus the embedded input. If such sum had numerous and independent terms, then removing a handful of them should not significantly change the output. However, since the terms not independent- each layer is input the following layer should expect to be able remove terms if the residual contribution from a particular layer is small. In other words, if the output of each layer does not change too much from layer to layer
"Co-first authors: please direct correspondence to the union (gromovand@metaa com, a990gmai91mmmm com, drobemit... J
'This strongly suggested by "lens" investigations that studied the evolution the token distribution as function of layer index such as the "logit lens" (nostalgebraistt 2020 and the "tuned lens" (Belrose et al. 2023 j A separate line of reasoning along these lines previously inspired neural ODEs (Chen et al. 20188. and led Yang al. (2033) to argue that ideally representation should change substantially from layer to layer order to most. effectively make of the parameters of network.

---

Published as a conference paper at ICLR 2025
(a)
(b)
Input text
(d)
Liama-2-708 (no healing)
Random
Compute angular distance biw input output blocks d(a (f))))
Embedding
MMLU Accuracy
0.6
0.4
Find the block with distance (x)
e++1
Remove the block
BoolQ Accuracy
[e", m)
0.6
Heal via PEFT
0.5
Output text
0.4
Block size.
Angular Distance.
0.5
Incressing
C4 Idation Loss
10.0
8.0
0.31
6.0
0.-
30
50
60 80
0.1
0.2
0.5
0.6
Layer Number, I
Fraction of Layers Dropped
Figure 1: Overview of our layer-pruning strategy and example results: (a) flowchart describing the algorithm: if removing n layers, we find the layer, E", that minimizes the angular distance, d, between layers l and f+m;; we can "heal" the damage with small amount of (parameter-efficien) finetuning. (b) a schematic depicting the removal of n total layers, indexed from e" + (c) angular distance, d. between different numbers of layers, n. vs. the layer number, E, that indexes the beginning of the block of n; the bottom curve (darkest purple) represents n 1, while the top curve (lightest yellow) represents 64; the black line traces L (n). the minimum of the angular distance across the different sized layer blocks. (d) results of pruning Llama-2-708 with healing (light blue) and without healing (dark blue) as function of the fraction of layers removed: the top (middle) panel gives the accuracy on the MMLU (BoolQ) question-answeriig
then remove the n layers beginning with layer t*: finally, if necessary,
benchmark, while the bottom panel the autoregressive loss on subset of the C4 validation set; here, the dashed red lines (dashed gray lines) indicate the accuracy loss the original unpruned model (of random guessing); these plots illustrate that typical behavior find in which there are sharp transitions in performance for the accuracy of question-answering tasks (here between 40%-50% pruning fraction), but continuity and very slow growth in the healed loss (light blue) up to at least to 80% pruning fraction.
In conjunction with our layer pruning, investigate the similarity of layer representations at different separations and find broadly that deeper layers are qualitatively more similar to neighboring layers than shallow layers (with the exception of the very final layer). This suggests even simpler pruning strategy: remove layers beginning at the penultimate layer and proceed from deep to shallow until the desired number of layers have been removed. this case, we find that, after healing the damage with small amount of QLoRA finetuning, we can achieve performance that nearly matches the more involved similarity-informer layer pruning strategy. The effectiveness of this method is evidence that LLMs might not properly leverage the parameters in the deeper layers of the network.
That said, while question-answering (QA) benchmarks such as MMLU and BoolQ are robust to a large amount of layer pruning, other measures of performance
not: if we look at the loss on next-token predictions for an IID dataset (C4 validation set), we find that the model is smoothly damaged in proportion to the fraction of the number of layers pruned. Since perplexity typically correlates strongly with downstream metrics, this naturally begs the question: which tasks are less robust than QA benchmarks pruning? As part of our final discussion, explore reasoning
2

---

Published as a conference paper at ICLR 2025
related tasks (GSM8K and HellaSwag) and see that they are harmed by any amount of pruning. Altogether, this leads to the following accounting of state: the shallow layers likely play a critical role in the storing of knowledge and retrieving of information, while the deeper layers are important for higher-level computations such as mathematical reasoning.
The structure of this paper is follows. In $2 first perform literature review of both practical post-training strategies and science-of chienceoff-dee---e deep--ennn investigations that motivate our work. Then, in |3 we give intuition for our layer pruning strategy and explain our method in detail, while in $4 we iterate over all our experimental results. Finally, we conclude in by exploring tasks beyond QA benchmarks, such as reasoning, and highlighting directions of future work. Specific model, finetuning. dataset, and evaluation details can be found in Appendix B and evaluation ablations can be found in Appendix
LITERATURE REVIEW
Pruning for neural networks has a long history (LeCun et al. 1989 Hassibi and Stork 1992): while
initial work focused unstructured pruning (Han et al. [2015 Chen et al. 2015: Srinivas and Babul
2013), structured pruning techniques were developed to make sparse networks more efficient (Li
et al. 2016; Wen et al.
2016 (uu et al., [2016) He et al,, 2017 Huang et al. 2018 Muuray and
Chiang 2013 See et al.
2016; Kim and Rush (0166). Recent work, of course, focused on structured
pruning of transformers Voita al.
al.
[0199] Kim and Awadalla, 2020; Fan et al.
Zhang and He 2020; Fan et al. 2021 et al. 2023] Sajjad et al. 2023 Liu et al. [202a
Hou et al. 2020 Sharma et al.) 2023; Ashkboos et al. 2024 Xia al. [022] Lagunas al. 202T
Men ct al. 2024). Our work focuses on pruning the layers of decoder-only GPT style open-weigh
large language models after they' 've been pretrained. For an extended literature review, please see
Appendix
3
METHOD
In this section, we give intuition for why we think layer pruning works ($3.1 and then we explain our method in detail ($3.2)
3.1 INTUITION
Our intuition for layer dropping comes from thinking about the representations as a slowly changing function of layer index. In particular, the layer-to-layer evolution of representations for a transformer given by a residual iteration equation
(e+1) - (2) + f(x(()))00))) g()))
(1)
where (zx(0, g())), respectively, are the multi-dimenssonal input and parameter vectors for layer e, and fxx () describes the transformation of one multi-head self- attention and MLP layer block. As for any residual network, if we unroll this iteration, we see that after L total layers the output is described as a sum over the transformations of the layers
L-1
(L) = (0) + Σ 5(010)
(2)
E=
If the terms in the sum were numerous, (L >>>>> 1), and independent, e.g. if the block functions were instead a function of the overall input as f (x) ((0)), then perhaps any particular contribution to the sum (2) could be neglected.
Of course, they are not at all independent: we delete layer & 1, then we must now connect the old input to that layer, (f-1), into the block function of layer as
(f+1) = (8-1) + ((-((-1) ()()),
(3)
where, for clarity, we are not relabeling layers or inputs despite the deletion. In general, such mismatch between the original input and new input should be very damaging for the network.
3

---

Published as a conference paper at ICLR 2025
However, if, after some number of initial layers, the representations converge to slowly changing function with respect to layer index,
(0) xle-1) +,,
(4)
with < x( in some appropriate sense, then the effect of deleting particular layer E, e.g. making the replacement s!()) x((-1) in going from ① to B should only change the representation in the subsequent layer, 2((+1), by small amount. Similarly, to successfully prune the layers before
layer ie. those indexed from - n, .,..... 1, we'd want that the input to the pruned block should be very similar to the output of the pruned block:
100 x(t-n) c
(5)
Regardless, any layer removal has a cascading effect: since post pruning (f++1) is computed by a different function than before, cf. ( vs. (3) and since then ((++1) is directly or indirectly input to subsequent layers, l .,.... L, deleting shallow layer should have much greater impact than deleting a deeper layer.
From this, have the following hypotheses that will test experimentally:
(0) We should be able to prune layers of residual network.
(1) We should have greater success pruning deeper layers.
(2) Blocks of layers we successfully prune should have outputs that are similar to their inputs.
In the next subsection, 3.2 we will explain the details of our pruning algorithm and in the following section, $ we will present experimental evidence for points (0)-(2).
3.2 LAYER-PRUNING ALGORITHM(S)
Our principal layer pruning algorithm is very simple:
0. Pick number of layers to prune
1. Compute the angular distance d(z((), (((+a)) cf. 7 below, between the input layer č and the input to layer + n on neutral pretraining dataset or on a dataset representative of downstream task of interest.
2. Find the layer, L" that minimizes that distance:
8 (n) arg min (() ((a(()),x((tt))) (etx))
3. Drop layers E to E*+n-1: 1; connect the old input to layer l to the old (f*+m)thh layer block
4. (Optionally) heal the mismatch at layer E + n with small amount of fine tuning on neutral pretraining dataset or particular dataset of interest.
fewer words inside a figure are more helpful to you than the text in an enumerated list, then note that this algorithm is also depicted in panels (a)-(b) of Figure
Elaborating on the first step, the angular distance on a single sequence of length T is given by
(2)
(f+n)
d(x(9) (((0)(((t))))
1 π
Zy
Zip
arccos
(C)
(e+n)
where the inner product over the hidden dimension of the model for the final token T of the sequence, denotes the z.rmn and the factor of 1/T is convention This distance should
Layers often contained in data structure, such ModuleList in PyTorch, to drop these layers Modul 1 elii that removes the layers from /* +
would simply define
"Two comments: (i), we do not expect our choice of angular distance in lieu of any other reasonable metric, e.g,, such cosine similarity to be particular significant; and (ii), chose to focus the final token since, due the causal attention mask, its emhedding is the only one that depends the entire sequence.
4

---

Published as a conference paper at ICLR 2025
then be summed over a number of examples that is large enough to get low-fluctuation estimate but overall should be quite small.
Elaborating the "optionality" of the final step, find that the near-lack of performance degradation on
benchmarks, cf. Figure [jd] and others in $4.1, can be extended to greater pruning fractions with small amount of finetuning. Depending application of the pruned model, this may not be necessary. However, the healing procedure does have a substantial impact on perplexity, cf. Figure D ()) and others in $4.2
resource constraints and intended
For both the angular distance measuring and the healing, if the ultimate goal is to supervise finetune (SFT) model for downstream task, it could be useful evaluate the distance of sample from that dataset and then combine the healing process with the SFT. In contrast, for the greatest generality, it's most natural measure distance and heal with pretraining dataset that approximates the statistics under which the model was originally pretrained.
Finally, we also investigated an even simpler pruning strategy inspired by analyzing the angular distances across different model families: drop the deepest layers, excluding the final layer before the LLM head, and then (won-optionally) heal the damage. For complete clarity, this means that we are pruning n. layers from an I-llaer model, then we would remove layers (I. n)) to (I. 1), inclusive.
4 RESULTS
In this section, we demonstrate the effectiveness of our pruning strategy on different question- answering (QA) benchmarks and highlight a robust pruning-driven transition in performance ($..) while, in contrast, we find that the autoregressive perplexities of the healed pruned models are continuous across their transition points ($4.2); then, after comparing the similarity statistics between different layers across model sizes and families ($4.3), we contrast our principal aomilarity-infomma pruning strategy with simpler remove eemove-the-deepeeee the-deepest- layers strategy ($4.4).
For our experiments, we pruned a wide variety of large-scale LLMs from 2.7B to 70B parameters spanning 32 to 80 total unpruned layers. Specifically, we used models in the Llama-2 family (Touvron et al. 2023). the Qwen family (Bai et al. 2023), Mistral-7B (Jiang et al. (2023)), and Phi-2 (Javaheripi and Bubeck 2023). For these models, we exccuted the "healing" step using QLoRA (Dettmers et al. 2023): our models were quantized to 4-bit precision and then finetuned, using QLoRA for efficient training, on either 164M or 328M tokens from the Colossal Clean Crawled Corpus (C4) (Raffel et al.
2020), common pretraining dataset. As result, each experiment of ours can be performed
single 40GB A100 GPU. For our QA evals, we used Massive Multitask Language Understanding 2020), common world-knowledge and problem solving benchmark, and BoolQ (Clark et al.) 2019). a common yes/no reading comprehension benchmark where the answer has be inferred from the text itself. The specifies of our models, healing procedure, dataset choices, and evaluation details can be found across Appendix B: ablations of different hyperparameter choices
(MMLU) (Hendrycks
can be found across Appendix C
4.1 ACCURACY ON QA BENCHMARKS
Our first set of results are shown in Figure where we plot 5-shot MMLU accuracy as a function of the fraction of layers removed: in the left panel we present the 2 family. in the middle panel we present models from the Qwen family, and in the right panel we show Mistral-7B and Phi-2. In order to better compare models of different total number of layers, in these plots we opted to normalize the r-axis by the fraction of layers removed (rather than the absolute number of layers removed). Note that since MMLU contains multiple choice questions with four possible responses, the expected accuracy of random guessing is 25%.
Importantly, we see characteristic flat region of robust performance followed by a sharp transition to random accuracy at a pruning fraction around 45%-55% for models in the Llama-2 family, 35% for Mistral 7B, 25% for Phi-2, and 20% for models from the Qwen family. This implies that the essential knowledge required to achieve a model's top score isn't 't removed by significant layer removal even though the fraction be quite large(!) until eventually that knowledge is lost at critical
5

---

Published as a conference paper at ICLR 2025
Family
Family
Mistral-7BB Phi-2-2.78
Ove Ourn
PaL 1.
MNLU Accuracy
0-5-
00
01
02 03
0,4
05
05
0.7
0.5
0.0
01
0.2
0.3
0.4
05
0.5 077
0.5
0.0
0.2
0.4 Fraction Layers Dropped
0.2
03
25
0.6
0.7
Fraction of Layers Drapped
Fraction Layers Dropped
Figure 2: MMLU accuracy (5-shot) fraction of layers dropped for different model families. (Left: Llama-2 family; Middle: Qwen family; Right: Mistral-7B and Phi-2.) The solid lines represent performance after dropping layers and healing, dotted lines show performance after dropping layers only (no healing), and the dashed gray line is the score for guessing randomly. For these models, healing leads modest improvements, and performances fractions, depending on model family and size, at which point they transitions to random guessing.
quite robust until 20%-55% pruning
model-dependent threshold Contrasting the curves with and without healing, we see that finetuning offers modest improvement by better preserving the unpruned performance and pushing the phase transition to random guessing to slightly larger pruning fractions.
Broadly we see that layer pruming is more robust for the larger and deeper models, e.g. Llama-2 1amma-2-133B and Llama-2-70B, which we hypothesize could be related to the fact that either the smaller models are more overtrained, making parameters less redundant, or that the deeper models can afford to lose more layers in an absolute sense. Also, the Qwen family is strange, fact we will further elaborate on 4.3
4.2 Loss NEXT-TOKEN PREDICTIONS
In this section, we look at the effect of layer pruning on the pretraining optimization objective the cross-entropy loss next-token prediction when evaluated on subset of the C4 validation dataset [ In order to have a fair comparison across models with different sized vocabularies V, we normalize the loss by log V. which corresponds the loss of sampling tokens randomly with uniform probability. (See Appendix B.2 for more details.)
In Figure 3 we plot the normalized C4 validation loss for all seven of our models, after healing (left panel) and before healing (right panel), as a function of the fraction layers removed. Without healing, we see that there is somewhat sharp(ish) transition to random guessing for each model at approximately the pruning fraction that the QA benchmark accuracies also sharply transition to random guessing, suggesting that models are hopelessly harmed this point, cf. Figure 2 Next, contrasting the scales of both plots, we see that healing significantly restores the next-token prediction ability of all the models to near-unp levels, with the loss increasing slowly and linearly with layer dropping. Most strikingly from scientific perspective is the post- post-healing continuity through the pruning fractions where we previously found sharp transitions for the QA henchmarks: this decoupling illustrates way of disconnecting (or creating a miscalibration) between performance on downstream tasks such as MMLU and BoolQ and continuous measures of performance - such as the cross-entropy loss.
"This effect is rather robust to choice of QA henchmark: in Figure we plot the average O-shot BoolQ accuracy for our model families and observe analogous bchavior.
'We make sure that none of the validation data are seen during healing stage. TThis is consistent with Schaeffer et al. (2023) that argued jumps in one kind of metric may not be visible in others.

---

Published as a conference paper at ICLR 2025
L Lame-2-108
Lame-2-130
Lamma2-78
Qwen-70
Owen-14B
Mistrai788
Ph-2-2.70
Random
Norma zed CA Vol deltddttn Loss
No hea healing iin
Healing
1.50
0.40
1.00
0.30
0.50
0.40 Fraction of Layers Dropped
80
0.40 Fraction of Layers Dropped
0.70
Figure 3: Normalized C4 validation loss fraction of layers dropped before healing (left) and after healing (right); each curve is normalized by the cross-entropy loss of sampling uniformly from the model's vocabulary. For the experiments before healing, the loss for each model transitions to random guessing (gray dashed line) at approximately the same pruning fractions that the QA benchmarks transition random guessing; after healing, there continuity through the regions of sharp transition on QA tasks, cf. Figure 2 Contrasting the overall scale of both plots, it's clear that healing significantly restores the performance on next-token prediction near-unpruned levels.
4.3 ANGULAR DISTANCES BETWEEN REPRESENTATIONS
Given the central role the angular distance 7 plays in our pruning strategy, let's take a subsection look these distances across our seven models. For this analysis, the angular distances for each model were averaged over 10k samples from the C4 validation set.
Recall from earlier Figure c): for 1lama-2-77BB this plotted the angular distance d(x(()) (( (ff+n) that compared the ε-th layer to the (6 + nt)-th layer, across all initial indexes & for block sizes from n to n 64; the minimum of the curves, E* (n). gave the optimal block to prune for given n, cf.
A more compact way to display this same data is shown in the heat maps of Figure 4 each square is colored to depict the row-normalized angular distance between layer and l + n across all possible l, and n up to very large fractions of the total number of layers; the optimal layer to prune for given block size, é" (n), corresponds the minimal distance in each row.
Across models, we make two generalizations: (i) the smallest distances are found across the deeper blocks, meaning deeper layers are typically quite similar to each other and can be more easily dropped; (ii) the distances across the deepest blocks the blocks that include the last layer take either maximal or nearly-maximal values, meaning one should never drop the final layer. While broadly true, there are few exceptions. For some models, e.g. Phi-2-2.77 7B, for the largest blocks in some models, e.g. Llama-2-7B, final few layers seem important. As previously noted, the Qwen family is somewhat unusual: here shallow blocks; this likely explains the shorter region of robust performance in Figure
that there are few odd "islands" of high similarity for
4.4 A SIMPLER PRUNING STRATEGY
Inspired by our recent conclusions, we experiment with a very simple heuristic pruning strategy: (I) if pruning layers from an L-layer model, drop layers (L n) to (L 1) so as to remove the deepest block that excludes the final layer; then (2) heal with small amount of finetuning as before. Compared with our principal similarity-informed pruning strategy, this simpler heuristic algorithm has the advantage of never requiring practitioners load onto a GPU or inference the unpruned model. It also provides meaningful ablation of the importance of optimizing the block to prune.
In Figure we contrast our two pruning strategies, both before healing (left panels) and after healing (right panels), for the QA benchmarks (MMLU/BoolQ. top/middle panels) and the autoregressive loss (C4 validation, bottom panels). On the one hand, the simple heuristic performs quite poorly without healing the damage incurred by pruning: accuracy random with increased pruning fraction, and the loss begins to increase very rapidly even with small amounts of pruning. On the other hand, the results for the two pruning strategies across evaluations
the QA benchmarks decays rapidly to (near-)
7

---

Published as a conference paper at ICLR 2025
(a) Llama-2-7B
(b) Llama-2-138
(c) Llama-2-70B
1.0
30
60
20
Block Size,
Shifted
15
20
40
Rescaled
0.5 Angular
10
10
20
Distance
10
20
30
10
20
30
40
0
20
40
60
80
Layer Number,
Layer Number, /
Layer Number, /
(d) Qwen-7B
(e) Qwen-14B
(f) Mistral-7B
(g) Phi-2-2.7B
30
20
Block Size,
15
20
15
15
10
10
10
10
30
10 20 30
40
10 20 Layer Number,
30
10 20 Layer Number,
30
Layer Number,
Layer Number, /
Figure 4: Normalized angular distance from initial layer E (x-axis) with block size (y-axis) for each of the seven models we evaluated; the distance for each n is shifted and rescaled to span the same range, [0, 1] (yellow to purple): the optimal block to prune, e* (n), corresponds to the deepest yellow for each row. Across models, the deeper layers tend to be very similar, though the deepest blocks that include the final layer (squares along the outer diagonal) are (near- (near-)maximal )maximally dissimilar.
are quite comparable after healing: for the QA benchmarks, the similarity iinormmed algorithm slightly better preserves the accuracy before the phase transition, though the simple algorithm perhaps pushes the phase transition slightly greater pruning factions; and for the loss, the curves nearly lie top strategy does marginally outperform for all amounts of
of each other, though the iimilarit-iifformo pruning. These experiments are strong evidence that the purpose of post-pruning finetuning is the healing of damage at the pruning interface and not the acquisition of additional knowledge.
5 DISCUSSION AND FUTURE DIRECTIONS
At the end of this work, many readers are puzzled by the following: are the deeper layers entirely useless? So far, we' we've ve provided evidence that the elimination of the deeper layers does not affect performance on QA tasks like MMLU (Figure 2) while at the same time have shown that their removal does disrupt the next-token predictions of the underlying model (Figure (3). Since perplexity often correlates with performance downstream tasks, which are the tasks that are hurt by layer pruning?
Here are two hypotheses consistent with the fact that the model's perplexity is disturbed proportionally to pruning fraction:
(i) The deeper layers are not essential for storing knowledge, but are useful for more complicated computations, such as those that involve reasoning.
(ii) The deeper layers are necessary when the model has to generate many tokens before answering question, such as when it produces chain-of-thought (CoT).
We test these hypotheses by evaluating our preer-prunedd models on tasks that involve CoTs or reasoning. For the former, we'll look at Thhouht MMLU (CoT-MMLU); for the latter, we'll
8

---

Published as a conference paper at ICLR 2025
No healing
Healing
MMLU Accuracy
0.60
0.60
0.40
0.40
C4 Validation BoolQ Accuracy
0.80
0.80
0.60
0.40
0.40
4.00
10.0
3.00
5.0
2.00
0.0
0.2
0.4
0.6
0.8
0.00
0.20
0.40
0.60
0.80
Fraction of Layers Dropped
Fraction of Layers Dropped
Similarity
Simple
Baseline
Random
Figure 5: Evaluation of Llama-2-70B with the simple pruning heuristic (solid red line), shown along with scores for the -inforrmyy-ffo pruning strategy (solid blue line), scores of the unpruned Llama-2-70B (red dashed line), and scores for randomly guessing (gray dashed line). (Left: before healing, Right: after healing; Top: MMLU, Middle: BoolQ, Bottom: C4 Validation Loss.) Without healing, the simple heuristic performs poorly across all evals; with healing, the scores both methods quite similar.
look at GSM8K (Cobbe et al.) 2021). grade-school math benchmark, and HellaSwag (Zellers et al., 2019), a multiple choice common-sense reasoning benchmark
In Figure 6 plot the performance of Llama-2 70B pruned with the similarity-informed pruning strategy across COT-MMLU (left), GSM8K (center), and HellaSwag (right): on the one hand, both GSM8K and HellaSwag. our two reasoning tasks, exhibit immediate degradation in performance with any amount of pruning, correlating with a similar decrease in the perplexity evals (Figure 3); on the other hand, CoT-MMLU shows relatively flat region of robust performance with pruning. analogous to our previous results on QA benchmarks (e.g. Figure 2). This is some initial evidence for hypothesis (i) over hypothesis (ii): the deeper layers may useful for higher- reasoning tasks, while less important for knowledge intensive QA tasks; moreover, perplexity errors due to pruning do not compound to hurt QA evals when the model is required to generate many tokens.
Now at the conclusion of the work, we are left with the following questions:
7 Herr are the details
we performed these three evaluations:
For COT-MMLU, followed the flan cot.ntotttoooooooooooooooaooooee sot fewshot evaluation in EleutherAl (Goo et al. 2023), which models produce chain of thought before generating their answer. Note that the accuracy at 0% pruning fraction MMLU without CoT is much better than analogous accuracy at 0% pruning fraction for COT-MMLU (~ 69% vs. ~ 43%, respectively; cf. Figures and consistent with some previous work (e.g.. see Table 16 of et al. (2024)).
For GSM8K, we used the gam8k_ [sm8k__cot evaluation in EleutherAl (Gao el al. 2023) and measured passe1; ; for each problem we extracted an answer from single generation (with CoT) and checked for correctness against the ground ground-truth truth answer.
For HellaSwag, we used the bellaswag evaluation in EleutherAl (ao et al. 2023 Note that HellaSwag multiple-choice henchmark, random performance is 25%.

---

Published as a conference paper at ICLR 2025
CoT-MMLU (World Knowledge)
GSMBK (Math Reasoning)
HellaSwag (Common Sense Reasoning)
50.0
0.4
Accuracy
Pas
30.0
Accuracy
50.0
0.2
20.0
10.0
35.0
0.0
0.0
0.2
0.4
0.6
0.2
0.4
0.6
0.8
0.0
0.2
0.5
0.8
Frection of Layers Dropped
Frection of Layers Dropped
Fraction of Layers Dropped
Llama-2-708 (Similarity)
Baseline
Random
Figure 6: Evaluation of Llama-2 70B with the similarity innilartt-informeed pruning strategy across different evaluation tasks. (Left: Chain-of-Thoure MMLU (CoT-MMLU), Center: GSM8K, Right: Hel- laSwag.) We see that GSM8K and HellaSwag show immediate degradation of performance with any level of pruning, while COT-MMIU behaves qualitatively similarly MMLU without CoT; this suggests that the deeper layers are likely necessary for reasoning tasks.
What are better layer-pruning strategies? What are better approaches healing
Why does healing eliminate the phase transition in the loss but not in the QA accuracies?
With more comprehensive evals, will accuracy different tasks degrade different depths?
Relatedly, is knowledge generally stored in shallow middle layers, or is it delocalized?
Can we devise a pruning strategy that is robust for reasoning tasks?
Do pretraining details affect the ability to prune, e.g., are scaling-law over-trained or distilled models more difficult to prune?
How can we enable LLMs to more effectively use the parameters in their deepest layers?
Some of these questions would benefit from studying both layer similarity and pruning across different pretraining checkpoints; for instance, at what point does the sharp phase transition and critical depth in the QA accuracies emerge, and does more training lead to better use of the prunable parameters? Others suggest explorations with different pretraining architectures and objectives, e.g. in order better make use of the deeper layers (for example, or early exit during pre-training (Elhoushi et al. 2024) to induce equal usage of layers). With more comprehensive evaluations, if different kinds of QA tasks degrade at very different depths, then this might indicate that the knowledge required to complete those tasks is stored across different layers would be very interesting to use pruning systematically study these kind of interpretability questions.
imagine applying layer dropout (Fan et al. 2019)
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
We thank Aaron Schwartz for his initial collaboration, Aaditya Singh and Sho Yaida for discussions, and Aaditya Singh for comments the draft. We would also like to acknowledge the 2023 NeurIPS Large Language Model Efficiency Challenge for initializing us for work on this project. A.G. is supported by the NSF CAREER grant DMR-2045182 the Sloan Foundation, and by the Laboratory for Physical Sciences through the Condensed Matter Theory Center. D.R. acknowledges support from
At the cost of introducing another hyperparameter and requiring both pruned and unpruned models to fit in memory during finetuning, one natural way improve healing is by adding auxiliary student-teacher loss that explicitly addresses the pruning mismatch (5). such as
(af (E (0o) x (ce) (0))
Laux
(
where 0g are the frozen parameters of the unprumed model, and 0 are the parameters of the pruned model to be healed: thus, (<*+x) (0o) is input to the (E" w)-th layer the unpruned model, (x(") (0) is the input to that same layer after pruning, and Las minimizes their mismatch. We thank Sho Yaida for this observation. Altemmativeyy, one could measure (fx ((+n)) or find (m)) as function of different eval datasets.
10

---

Published as a conference paper at ICLR 2025
the National Science Foundation under Cooperative Agreement PHY (the NSF Al Institute for Artificial Intelligence and Fundamental Interactions, http: taifi. org/) ) and appreciates both the sanction and support of Sequoia Capital. This paper has been brought to you residually by the letters G, P, and U, after summing over many layers.
REFERENCES
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
nostalgebraist. interpreting gpt: the logit lens. https:/ https://www.essvrrnnn wwtp:onnnncoooooos.tsooo/ AccdRBBrDDccaNSSooorznaaeeeeee
2020.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.00 12, 2023.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs vi: Feature learning in infinite-depth depth neural networks. arXiv preprint arXiv:2310.02244,, 2023.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann, 1989.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. In S. Hanson, J. Cowan, and C. Giles, editors, Advances in Neural Information Processing Systems, 1992.
volume 5. Morgan-Kaufnnn
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International conference on machine learning, pages 2285-2294. PMLR, 2015.
Suraj Srinivas and R Venkatesh Babu. Data-free parameter pruning for deep neural networks. arXiv preprint arXiv:1507.0614 2015.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.088: 000116000.000.... 2016.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances neural information processing systems, 29, 2016.
Hengyuan Hu, Rui Peng, Yu- Yu-Wing Tai, and Chi- Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint 1rXiv:1607 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389-1397, 2017.
Gao Huang, Shichen Liu, Laurens Van der Maaten, and Kilian Q Weinberger. Condensenet: An efficient densenet using leamed group convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2752-2761, 2018.
Kenton Murray and David Chiang. Auto-sizing neural networks: With applications to n-gram language models. arXiv preprint arXiv: 1508.0501, 2015.
Abigail See, Minh-Thang Luong, and Christopher D Manning. Compression of neural machine translation models via pruning. arXiv preprint arXiv:/606.000 1606.09274 2016.

---

Published as a conference paper at ICLR 2025
Yoon Kim and Alexander M Rush. arXiv:1606.07947, 2016.
Sequence-level knowledge distillation. arXiv preprint
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-heat self-attention: Specialized heads do the heavy lifting, the rest be pruned. arXiv preprint arXiv:1905.094/8, 2019.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
Young Jin Kim and Hany Hassan Awadalla. Fastformers: Highly efficient transformer models for natural language understanding. arXiv preprint arXiv:2010.13382, 2020.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint 1..iv:/909.115560 2019.
Minjia Zhang and Yuxiong He. Accelerating training of transformer-based language models with progressive layer dropping. Advances in Neural Information Processing Systems, 33:14011 1-14023, 2020.
Chun Fan, Jiwei Li, Xiang Ao, Fei Wu, Yuxian Meng, and Xiaofei Sun. Layer-wise model pruning based on mutual information. arXiv preprint arXiv:2108. /2594, 2021.
Ananya Harsh Jha, Dirk Groeneveld, Emma Strubell, and Iz Beltagy. Large language model distillation doesn't 't need a teacher. arXiv preprint arXiv:2305. 14864, 2023.
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. On the effect of dropping layers of pre-trained transformer models. Computer Speech & Language, 77:101429, 2023.
Wei Liu, Zhiyuan Peng, and Tan Lee. Comflp: Correlation measure based fast search asr layer pruning. arXiv preprint (XXx:2309.11760 2023a.
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33: 9782-9793, 2020.
Pratyusha Sharma, Jordan Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312.13556 2023.
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024 2024.
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. arXiv preprint arXiv:2204.0044 2022.
François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M Rush. Block pruning for faster transformers. arXiv preprint 2rXiv:2109.048.0 2021.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.0388 033333333, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a.
Mojan Javaheripi and Sébastien Bubeck. Phi-2: The surprising power of small language models, Dec 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.1433 14314, 2023.
12

---

Published as a conference paper at ICLR 2025
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(():5485-555) 2020.
Dan Hendrycks, Collin Burs, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 033000 2020.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044 2019.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? arXiv preprint arXiv:2304.1500 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint 190.:0:900.0083000 2019.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, 12 2023. URL zenodo. http///eooodd//2533395
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling cnstructionfinetune language models. Journal of Machine Learning Research, 25(70):1-53 2024.
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layer skip: Enabling early exit inference and self-speculative decoding. arXiv preprini arXiv:2404. 6710,, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
Alec Radford, Jeff Wu. Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
Language models are unsupervised multitask learners. 2019. URL https
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. Can chatgpt understand too? a
comparative study on chatgpt and fine-tuned bert. arXiv preprint 17X19:2302.10198 2023.
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. arXiv preprint arXiv:1909.00512 2019.
Alexci Baevski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:1 33:124:: 12111444-11111 12460, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: httl::/503.0.5... 025311 2015.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Knowledge distillation large language models. arXiv preprint 236.522306 2023.
13

---

Published as a conference paper at ICLR 2025
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.0. 10351, 2019.
Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. Want to reduce labeling cost? gpt-3 can help. arXiv preprint arXiv:2108. 3487, 2021.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.0720 2023.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05444 0546633,, 2023a.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks all you need. arXiv preprint arXiv:2306.1 1644, 2023.
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. arXiv preprint arXiv:2301.12726 2023.
Cheng- Yu Hsich, Chun-Liang Li, Chin Yeh, Hootan Nakhost, Yasuhisa Fujäi, Alexander Ratner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301, 2023.
Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. Lion: Adversarial distillation of closed-source large language model. arXiv preprint arXiv:2305. 12870, 2023b.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 10X:::2006.0968,, 2021.
Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: sara-fin-tuning-awaal quantization for large language models. arXiv preprint arXiv:2310.000 08659, 2023b.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficien fine-tuning. arXiv preprint arXiv:2303. 10512, 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274-19286. PMLR, 2023.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng. Jason Lee, Deming Chen, and Tri Dao. Medusa: Simple 11m inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401. 10774, 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372, 2022.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint (0.iv:2/1400666:: 2021.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality- localization vs. knowledge editing in language models. arXiv preprint arXiv:2301.044 10421,,, 2023.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304. 14767, 2023.
Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. Jump conclusions: Short- cutting transformers with linear transformations. arXiv preprint arXiv:2303.09435, 2023.
Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv:2310.02207, 2023.
arXiv preprint
14

---

Published as a conference paper at ICLR 2025
Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, n-gram, positional. arXiv preprint arXiv:2339.04822/ 04899,, 2023.
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, al. Deja vu: Contextual sparsity for efficient Ilms inference time. In International Conference on Machine Learning, pages 22137-22176. PMLR, 2023b.
Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. specifc skill localization in fine-tuned language models. arXiv preprint arXiv:2302.06600 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick Von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020. Associa- tion for Computational Linguisties. URL https: www. aclweb. org/ anthology/2220
emr p-demos. 6
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficien fine-tuning methods. pettps:/ggihheg com/ com/huuggingfacee/pett 2022.
Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of llms. arXiv preprini 203iv:2308.07317, 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O: 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339 2022.
15


Reviews:
[{"review_id": "8mHNAGIOhD", "reviewer": "Reviewer_TcNy", "summary": "This study explores how knowledge is stored in large language models (LLMs) by examining the effects of layer pruning. The research identifies layers that can be removed without impacting the model's performance on knowledge-based benchmarks, suggesting these layers may not be essential for storing knowledge. \n\n\nThe key findings of this work include that: 1) shallow layers may be crucial for knowledge retention, as models are resilient only to pruning of deeper layers; 2) layer pruning, combined with parameter-efficient fine-tuning (PEFT) methods like quantization and Low Rank Adapters (QLoRA), significantly reduces memory use and inference time, with performance scaling well on a single A100 GPU. \n\n\nOverall, these findings suggest that current pretraining strategies may underutilize deeper layers, and that layer pruning could enhance both the efficiency and accessibility of LLMs in practical applications.", "questions": "(a) Comparison to other possible comparison and pruning strategies:\n\n- it would be useful to compare the cosine similarity approach adopted by the authors to compare layers to other similarity metrics that appeared for layer comparison (e.g. CKA - Kornblith et al., ICML, 2019, SVCCA - Raghu et al., NIPS, 2017);\n\n- it would be useful to compare strategies that prune successive layers with other possible methods that consider pruning non-consecutive layers (e.g. greedy iterative approach of layer-pruning).\n\n(b) How do the results of the work, and in particular the analyses of Section 4.3, relate to recent findings on the geometry of representations in LLMs found using metrics such as intrinsic dimension (e.g. Valeriani et al., NIPS, 2023; and Cheng et al., ACL, 2023) and linearity (e.g. Park et al., NIPS, 2023; Engels et al., 2024).\n\n(c) It would be beneficial to expand Section 4.2 on the role of the deeper layers by including and discussing more in detail Fig. 7 related to the GSM8k dataset experiment.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The work is written with a great focus on clarity, and the main findings are sufficiently supported. \n\n\nThe main observation, that layer pruning does not impact the performance of world knowledge and comprehension benchmarks, is relevant to the community. \n\nThe observation that comparing layers with very intuitive similarity metrics suggests simple and effective layer-pruning strategies is valuable.", "weaknesses": "The paper would benefit from further justification that the authors' clear and simple strategies lead to compatible results to possible more sophisticated techniques that emerged in previous literature (see the Questions section (a) for explicit examples).  \n\nSeveral recent efforts approached the study of LLMs representations through geometry. This work would benefit from a more thorough comparison/connection to recent progress in the community along these directions (see the Questions section (b) for explicit points).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gUWQLbrvoM", "reviewer": "Reviewer_uhTK", "summary": "The authors propose a pruning method for LLMs and conduct studies on LLMs using this approach. Their pruning method involves removing consecutive layers (block) from the LLM, with optional fine-tuning. To determine the size and position of the layer blocks to remove, they suggest either using the similarity of representations or, more simply, removing a specified number of layers starting from the second-to-last layer. Their findings indicate that the deeper layers of the LLM (those closer to the output head) exhibit more similar representations, making them easier to remove.", "questions": "* I wonder if the authors could present results from a more diverse set of tasks beyond QA to see if the overall trend observed in this paper holds and to explore any potential differences and lessons.\n* It would be helpful if the authors provide some quantitative results on the efficiency of the pruning method itself compared to the most similar or representative methodologies.\n* This paper presents some interesting findings; however, could the authors offer more practical implications or lessons for improved LLM pre-training schemes beyond simply achieving improved downstream efficiency through pruning?\n* Regarding the C4 results, it might be useful for readers to see some qualitative generation outcomes, as this could reveal whether the point at which the semantic validity of the generated sentences begins to collapse aligns with the phase transition observed in the QA results.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "* The paper is clearly written and well-organized, making it easy to follow. The flow of the text, the intuitiveness of the figures, the comprehensive appendix, and their placement all contribute to its readability.\n* Despite the simplicity of the proposed pruning method, it achieves substantial gains. Given that the approach is implemented in a discrete, sparse and straightforward manner, it could also serve as an effective tool for understanding LLMs.", "weaknesses": "* The paper’s title and claims might somewhat overemphasize the findings, which could lead some readers to interpret them as suggesting that 'LLMs don’t require deep layers'. The experiments are primarily focused on relatively simple QA tasks, while the implications of results from more complex reasoning tasks (like GSM8K) aren’t highlighted enough, despite their significance.\n\n* This highlights a related concern as above: given that one of their contributions is a framework for understanding open-weight LLMs, the range of tasks explored feels limited — mainly knowledge-intensive tasks. To fully clarify the necessity and role of deeper layers in LLMs, testing on more diverse data complexities—tasks requiring higher cognitive abilities, such as reasoning and planning—would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZXB6mEbTyY", "reviewer": "Reviewer_1i1x", "summary": "The paper shows that deeper LLM layers can often be pruned without significant performance loss in standard QA benchmarks. \nThe authors propose a pruning strategy to eliminate deeper layers based on the angular distance between the last token representations of nearby layers. The authors also show that if the pruned model is fine-tuned after pruning, it is possible to apply a more straightforward pruning strategy and remove up to 40% of the last blocks without measuring the layer similarity.", "questions": "Echoing what I wrote above in the Weaknesses section, I was wondering if the authors have tried open-ended generation tasks beyond GSM8k, such as text summarization. Is it possible to tell if the deeper layers are useful for reasoning or if the pruning hurts any open-ended generation beyond a single token?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem setup is well-defined, and the writing is clear;\n\n2. The claims are supported by experiments on a wide set of models;\n\n3. The findings are interesting, novel, and potentially useful in practice to reduce the memory footprint of LLMs", "weaknesses": "The main weakness is the relatively narrow set of downstream benchmark evaluations. The paper's main message relies on multiple-choice QA benchmarks where a single token has to be generated. \n\nThe authors briefly discuss the case of GSM8K, showing that it is not always convenient to prune the deeper layers. \nUnfortunately, that discussion is briefly mentioned in line 357, and the plot is only shown in the appendix. \n\nI encourage the authors to expand the debate on the limits of applicability of the layer pruning in the main text. Discussing this limitation would make the paper stronger and its contribution to the community more transparent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "emcaWyun2L", "reviewer": "Reviewer_9vdV", "summary": "This paper conducts a model pruning study, which a) attempts to produce new effective pruning strategies and b) aims to better understand the role of deeper vs. earlier layers in language models.\n\nThe pruning method involves removing the deeper layers first (bar the final layer) and show that this method works relatively well, particularly when combined with subsequent training (what is referred to as \"healing\" in the paper) using QLORA.\n\nThe paper presents some ideas suggesting that the later layers may not be as useful, particularly for QA/memorization. Conversely, it shows some preliminary evidence that the deeper layers may be more important for reasoning, albeit these result is left in the Appendix.", "questions": "1. How can we be sure that random pruning or other similar method is not as effective as the \n\"later layers\" approach?\n2. How convincing is the evidence about reasoning vs. memorization compared to layer number?\n3. Why are results in the appendix selectively shown on differing models? (for example Figure 7, Llama-70b, Figure 8, Llama-13b)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "**Valuable Topic** \nThe question of how LLM performance degrades as later layers are progressively and systematically pruned is an interesting one, and which requires further exploration. \n\n**Results** \n1. The results of the method show relative consistency across model families, albeit some systematic differences are seen between model families, for example in Figure 2, the models undergo \"collapse\" at significantly different Fraction of layers.\n\n2. The result is shown on a large and robust benchmark, like MMLU.\n\n**Presentation**\nVery nice, well-thought out figures which do well in communicating the message.", "weaknesses": "**Ablation Study**\nI think some key ablation studies have not been undertaken, particularly using control pruning techniques like random pruning, or pruning earlier vs. later layers. \n\nWithout these, I find that the claim that the \"later layers\" are ineffective by itself unconvincing. For example, it could be that removing an equal proportion of layers from randomly chosen locations yield the same results.\n\n**Unclear Conclusions**\nWhile I appreciate the authors efforts in motivating the work, and proposing some interpretation of their results, I found some of the deductions to be confusing or rushed. For example, one of the potentially most interesting results, the difference in behavior between reasoning and QA benchmarks, is left for the Appendix (GSM-8k), with only 1 model, and only fleetingly mentioned in the main paper. \n\nThe combination of incomplete ablations, and fleeting/conflated analysis conclusions makes it difficult to take any definitive take-home messages from the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)