Do Language Models Use Their Depth Efficiently?
Authors: Róbert Csordás
Christopher D Manning
Christopher Potts
Venue: NeurIPS 2025 poster ()
Keywords: transformer
llm
large language model
residual stream
depth
casual interventions
compositionality
Primary area: deep_learning
TL;DR: We analyze the effective depth of LLMs and find that they are unlikely to compose subresults, and deeper models spread out the same type of computation as the shallow ones.

Abstract: Modern LLMs are increasingly deep, and depth correlates with performance, albeit with diminishing returns. However, do these models use their depth efficiently? Do they compose more features to create higher-order computations that are impossible in shallow models, or do they merely spread the same kinds of computation out over more layers? To address these questions, we analyze the residual stream of the Llama 3.1, Qwen 3, and OLMo 2 family of models. We find: First, comparing the output of the sublayers to the residual stream reveals that layers in the second half contribute much less than those in the first half, with a clear phase transition between the two halves. Second, skipping layers in the second half has a much smaller effect on future computations and output predictions. Third, for multihop tasks, we are unable to find evidence that models are using increased depth to compose subresults in examples involving many hops. Fourth, we seek to directly address whether deeper models are using their additional layers to perform new kinds of computation. To do this, we train linear maps from the residual stream of a shallow model to a deeper one. We find that layers with the same relative depth map best to each other, suggesting that the larger model simply spreads the same computations out over its many layers. All this evidence suggests that deeper models are not using their depth to learn new kinds of computation, but only using the greater depth to perform more fine-grained adjustments to the residual. This may help explain why increasing scale leads to diminishing returns for stacked Transformer architectures.

Do Language Models Use Their Depth Efficiently?
Róbert Csordás
Christopher D. Manning Christopher Potts
Stanford University, Stanford, CA, USA
rcsordas@cs.stant stanford.ent edu {manning, srgamiinnn..potttssaaaaaoooror.
Abstract
Modern LLMs are increasingly deep, and depth correlates with performance, albeit with diminishing returns. However, do these models use their depth efficiently? Do they compose more features to create higher-order computations that are impossible in shallow models, or do they merely spread the same kinds of computation out over more layers? To address these questions, we analyze the residual stream of the Llama 3.1, Qwen 3, and OLMo 2 family of models. We find: First, comparing the output of the sublayers to the residual stream reveals that layers in the second half contribute much less than those in the first half, with clear phase transition between the two halves. Second, skipping layers in the second half has a much smaller effect on future computations and output predictions. Third, for multihop tasks, we are unable to find evidence that models are using increased depth to compose subresults in examples involving many hops. Fourth, we seek to directly address whether deeper models are using their additional layers to perform new kinds of computation. To do this, we train linear maps from the residual stream of shallow model to deeper one. We find that layers with the same relative depth map best each other, suggesting that the larger model simply spreads the same computations out over its many layers. All this evidence suggests that deeper models are not using their depth to learn new kinds of computation, but only using the greater depth to perform more fine fine-grained adjustments to the residual. This may help explain why increasing scale leads to diminishing returns for stacked Transformer architectures.
Introduction
Large Language Models (LLMs [1, 3, 5]) have improved rapidly in recent years, and one significant correlate of these improvements is their increasing depth as measured by number of Transformer layers (Fig. 1). This sealing relationship would seem follow from the structure of these LLMs: they predominantly use stacked Transformer structure [6], which lacks recurrence across layers, and thus the number computation steps they can perform is constrained by their depth. In theory, greater depth should enable them to perform more complex computations by building on top of the representations computed in previous layers. Deeper models should have the capacity to be more compositional, leading to better reasoning, math capabilities, and generalization.
However, is unclear whether these models are using their depth efficiently. On the one hand, Petty et al. [7] find that increasing depth does not help with compositional generalization, Lad et al. [8] show that, apart from the first and last layers, models are robust to layer skipping and swapping neighboring layers (see also Sun et al. [9]), and Gromov et al. [10] were able to remove half of the layers from the network without significantly affecting performance on MMLU (but not for math). On the other hand, interpretability research often finds evidence for complex mechanisms spanning multiple layers [11, 12], suggesting that models can represent more complex operations as their depth increases. This paper is exploration of this tension. Our primary question: do deeper LLMs use their depth to compose more features to create higher-order computations that are impossible in shallow models, or do they merely spread the same kinds of computation out over more layers?
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We focus on the Llama 3 series of models and supplement these findings with secondary analyses of the Qwen 3 se- ries. Following the findings of previous work, we mostly focus on the math domain, which shows the greatest sensi- tivity to perturbations. Our analysis consists of five parts:
Open LL teadertroad Score
20-
1.
We analyze the norm of the residual stream and com- pare it to each individual sublayer's s output, as mea-
10-
0
sure of that layer's contribution. We find drop in this quantity in the second half of the model.
10
20
30
40
50
60
70
80
Number layers
2. We measure the effect that skipping layer has all successive layers' computations. These analyses show that, for the current output token, nearly all layers seem to be important, but the computations in the second half of the layers depend minimally each other. In contrast, for future token predictions, skipping the second half of the layers has minimal effect. This suggests that these layers are mainly refining the final probability distribution rather than computing reusable sub-results. We verify this using Logitlens [14], which shows drop in KL-divergence and sharp increase in the top prediction overlap with the final layer, starting around the same layer as the importance for future predictions decreases.
Performance of 132 Open LLM Leaderboard [13] base models as a function of depth. Colors represent different model families; dot size pro- portional to parameter count. Linear re- gression in red, with 95% confidence in- terval. Depth is a significant predictor even in regressions that control for other scale-relevant factors (App. B). Deeper models generally perform better.
Figure 1:
We analyze multihop questions and difficult math questions, and look for evidence of deeper computations for more complex examples. However, analysis shows the contrary: the layer's sensitivity to previous layers seems to be independent of example complexity. Analyzing individual examples with both causal interventions and integrated gradients [15] shows that the input tokens remain important until the middle of the network, and later tokens in the multi-step computation are not delayed to later layers, suggesting that composition is happening.
4. We train linear maps from each layer of shallower model to each layer of an independently trained deeper one, sharing the same vocabulary. By measuring the prediction error of the linear maps, we can measure the correspondence of the layers of the two models. This shows a diagonal pattern, indicating that the deeper model merely spreads out the computation through more layers instead of doing more computation in the later layers.
Overall, these findings suggest that current LLMs underutilize the second half of their layers. Rather than using their depth to learn more complex computations, they instead simply spread out the same kind of computation through an increasing number of layers. taking smaller computation steps and devoting the second half of the network to iteratively refining the probability distribution of the current token. We conclude with an exploration suggesting that MoEUT [16] might its layers more efficiently.
Background
All the models we analyze are pre-layernorm Transformers [6, 17]. A Transformer layer is constructed as follows:
a SelfAttention(Noo((t)))
(1)
h, h, + a:
(2)
m. MLP, (Nor((h))
(3)
h1+1 h + m:
Here, h; E Xxeett is the residual stream and a; my E IR. x are the outputs of the SelfAtten- tion and MLP layers, respectively, where T is the length of the current input sequence and dmodel is the width of the residual stream. Norm(-) is some token-wiss wise normalization, traditionally layer normaliza- tion [18], but usually replaced with RMSNorm [19]. We call SelfAttention(() and MLP((-) sublayers.
The residual stream initialized with he = Embedding(az), where & e NT is the sequence of input token indices. The output probability distribution, prediction, is y softmax(ý). where

---

1.8-
Atteetion: [eeile///iill
[imulla//]0ell
0.2
Men lebmie Contri
1.0
Attention MIP (oo me) o/
Coiie ariiaryy
0.0
0.5-
Attentio:: cossinta,AA)
-0.2
MLP: cossinim, (v)
0,0
50
Lager Index
(1)
Layer inden (13
(a) Relative norm of (sub)layer contributions
(b) Cossims of (sub)layer contributions and the residual
Figure 2: Influence of layers and sublayers on the residual stream for Llama 3.1 70B. (a) Norm of contributions relative to the residual stream. A sharp drop is visible near the middle; later layers change the residual much less, with the exception of the last few layers. (b) Cosine similarity between the contributions and the corresponding residual shows a phase change at the middle of the network.
Norm(h W
W are the output logits, the number of layers in the network, Worr Rudaa are the weights of the output classifier, and V is the size of the vocabulary.
Note that in pre-layernorm Transformers, the interaction of each sublayer with the residual is additive, as shown in Eq. 2 and 4. Thus, we can quantify the contribution of layer to the residual stream as ag my h1+1 h4. The contribution of the sublayers is a for the attention, and my for the MLP.
3
Experiments
Most of the experiments presented in the main paper are performed with Llama 3.1 70B [20], using NDIF and NNsight [21]. Unless noted otherwise, the results are computed on 10 random examples from GSM8K [22]. In bar plots, each bar starts from (no stacking). The main results are also shown in the appendix on different models, including Llama, Qwen [23]. and OLMo 2 [24]. In Sec. 3.1, we measure how the layers and sublayers contribute to the residual stream. In Sec. 3.2, we use causal interventions to measure the effect of layers on downstream computations. In Sec. 3.3 we show that deeper or otherwise more complex computations do not influence the number of layers that have causal effect on the prediction model. In Sec. 3.4 we train linear projections to find the correspondence between the layers of an independently trained shallow and deep Qwen model. Our exploration in Sec. 3.5 suggests that MoEUT [16] might use its layers more efficiently, especially when not modeling the question.'
3.1 How do the Layers Interact With the Residual Stream?
Since all interaction with the residual stream in pre-layernorm Transformers is additive, is expected that the norm of the residual, |/ul|2, will grow in later layers. At initialization, the norm of the output of each sublayer (/arl]2 and |[m=]]]2) is identical in expectation due to the normalization layer at their input. Thus, later layers contribute less than the earlier ones: it is harder for them to change the direction of the residual. During training, the model can learn to compensate for this growth by increasing the norm of the weights in later layers. However, most models are trained with weight decay, which explicitly discourages such growth. Residual growth was previously observed in the context of outlier features [25] and Universal Transformers
[16]. Here, seek use this technique to gain an initial high-level understanding of how much each layer contributes.
The relative contribution of sublayers. We measure the L norm of the residual ||hile and attention and MLP contributions (|aal12 and ||mmi||2) 22) in all layers of Llama 3.1 70B [20]. We observe the expected rapid growth of the residual (Fig. 15, in Appendix). However, the growth the sublayer outputs seems to be slower. To zoom in on this, in Fig. 2a, we measure the mean relative contribution
of each (sub)layer ( (la,-m.;) m::-m ||a:||2 and [ ). This shows consistent contribution in the first |// la *J/bll2
||vaa]]=
half of the network. with significant drop around the middle. The drop is especially pronounced in the
attention layers. The only exception is the last few layers, where the contributions seem to grow again.
'Our code is public: https: : http://t.t..eooooottrre//taaaaea
3

---

1.0
0
1.0
10
10
0.8
0.8
Rolative change
30
0.6
D.66
40
40-
50-
0.4
50-
D.4
0.2
0.2
70
0.0
0.0
20
80
40
60
Effect @ layer
Effect @ layer
(a) Effect of skipping layer on later layers' contributions in the all timesteps.
(b) Effect of skipping layer on later layers' contributions in future timesteps.
Figure 3: The maximum relative change in the layer's contribution when a previous layer is skipped, Llama 3.1 70B on GSM8K [22]. (a) Shows the maximum effect on the future computations for all tokens in the sequence, including the current token, while (b) isolates the effect only for the maximum of the future tokens. The range is limited between and 1. (a) The second half of the layers has weaker effect on future computations compared to the first. Because of the low influence on future layers in (a). but high importance for prediction (Fig. 17a in the Appendix), the second half of the lay- ers seems to perform mostly independent, but important, computations to refine the current predicted probability distribution. This is supported by the findings Fig. (b), which shows that the second half has little effect on the future tokens, indicating that they are not computing reusable subresults.
Measuring the cosine similarity between the residual and sublayer contributions. To dig deeper into each layer's contribution the overall computation, we measure the average cosine similarity between the output of different layers and sublayers and the residual. This is defined by cossim m. at, l4)) for the layer, and cossim(a;; (a;; h4) and cossim m, b4 + at) for the SelfAttention and MIPP components, respectively, where cossim(ær, y) |alla Where features are orthogonal to each other, cosine similarity corresponds to writing new feature to the residual, negative values correspond to erasing features, and positive values mean strengthening an existing feature." ?
ø-y
We show the results in Fig. 2b. The first layer has near-7ero cosine similarity, suggesting that these layers are primarily integrating context from neighboring tokens. This is followed by a mostly positive phase where features are being refined. The rest of the first half of the layers largely tends to erase the residual. Around the middle of the network, a sharp phase transition visible: the model starts strengthening existing features instead of erasing information. Interestingly, this position corresponds to the drop in the layer's contributions observed in Fig. 2a.
The changes in the relative contributions and the cosine similarities are high-level indicators of possible phase change. In the following, we investigate this more closely.
3.2 How do the Layers Influence Downstream Computations?
Causal intervention for measuring the layers' importance for the downstream computation. In Section 3.1, we analyzed the general characteristics of the residual stream. Here, we turn to pairwise interactions between layers using interventions. Which layer's computation influenced by a previous layer? In order to address this question, we use the following procedure. First, we run prompt through the model and log the residual h; Second, we run the same prompt again, but this time we skip layer 8, by setting ha+1 := hg, and we log the residual of the intervened model hy-
Third, we measure the relative change in the contribution of layer > s: 11(bi41-h:)) (oit--hh)--0 -B)a We take the maximum of this metric over the sequence and multiple prompts. We choose maximum because some of the later layers might only be used rarely, when deep computation requires it. We
"Two limitations of this method: (1) Transformers are hypothesized to make heavy use features in superposition [26, 27, 28, 29]. so this intuition might not fully transfer to practice; (2) if the model combines erasing/strengtheningg and writing features, the features will not show up in the cosine similarity metric.

---

KL Divergence
Overlap
0.4
0.2
20
60
70
30
40
80
Layer
Layer
(a) KL divergence between the model's prediction and Logitles applied to each layer.
(b) Overlap in top-5 tokens from the model's predic- tion and Logitlens applied to each layer.
Figure 4: Comparing Logitlens on different layers to the final prediction. (a) KI--divergence. (b) Overlap in the top-5 predicted tokens. Both show that later layers are devoted primarily to refining the output probability distributions, rather than to performing new kind of computation.
also compare the model output probabilities: Iyy 1o--ypm --l2.. We chose this non-standard metric because it provides clearer visualizations compared to KL divergence.
We show the results in Fig. 3a. We can see that, unlike the early layers, the layers in the second half of the model have a low influence on the computations performed in the later layers. However, Appendix Fig. 17a shows that these late layers are equally important for the output predictions, indicating that they perform important, but independent, computations.
In summary, these layer skipping experiments indicate that the layers in the first half of the network are integrating information and potentially building on each other's output, while the second half refines the output probability distribution based on the information already present in the residual.
Measuring layer importance for future predictions. To investigate this effect more deeply, we perform a variant of the previous experiment to measure the effect on the future tokens when skipping layers for earlier tokens. We do this by sampling a position < l12 < T - 1, skipping the layer only for token positions / < f., and measuring the effect only on positions > I.. The results are dramatic: as Fig. 3b shows, the second half of the network barely has any effect on future computations, except for some special layers at the very top of the network. Furthermore, their effeec future predictions is also significantly less than for the layers in the first half of the network (Appendix Fig. 17b).
What happens in the second half of the network? To validate our hypothesis that the second half of the layers refines the probability distribution of the current prediction, we apply Logitlens [14] to the residual and measure the KL divergence between its prediction and the final prediction of the model. The results are shown in Fig. 4a. Furthermore, we measure the overlap between the set of top-5 predictions from Logitlens and the final distribution in Fig. 4b. Both show the same picture: the prediction refinement seems to start at the same position at the same phase transition as when the layers do not influence the future predictions anymore, where the cosine similarities change sign, and the layer's 's importance decreases. All of these observations support our hypothesis: the second half of the network merely doing incremental updates to the residual to refine the predicted distribution.
Localizing Circuits. A similar method can also be used to discover layers that build on the contri- butions of previous layers directly. In order to do so, we can measure the change in future layers' contributions when removing the target layer's contribution from their input. In contrast to the previous experiments, we do not propagate this change to later layers. Specifically, to measure the effect of layer 8, set hj h; h, for all l 2 8, and measure the relative change in the layer's contribution: [l(a:+)m: (āi (m)+ml Fig shows the results. Bright spots indicate layers that build
(a +m; on each other's s features. When isolating the effect on future tokens only, it is possible to localize multi-layer, multi-token mechanisms similar to induction heads (Fig. 5b).
3.3 Do Deeper Problems Use Deeper Computation?
If a network is doing compositional computation, it has to break the problem down into sub- poo-prrolemss, solve the sub-peoblems, and combine their solutions. Because the lack of recurrence, Transformers only the results of computations in successive layers. We would therefore expect to that
5

---

0-
1.0
0
1.0
20
Dae Dellatve change
skiped
20
Re Relebve change
30
0.6
0.5
40
40
50
50
0.4
60
70
0.0
0.0
20
40
60
20
40
60
Effect layer
layer
(a) Local effect of layer later layers' tributions the all timesteps.
(b) Local effect of layer later layers' tributions in future timesteps.
Figure 5: Analyzing the direct local effects between pairs of layers of Llama 3.1 70B GSM8k [22]. The heatmaps highlight layer pairs with direct effects on each other. Unlike Fig. 3, the effects not propagated to future layers. For each layer s, the plot shows future layers that build on the representation computed by s. (a) Effects on all tokens, highlighting all possible circuits. (b) Effect on future tokens. The sparse, bright spots indicate multi-layer, multi-token mechanisms, such as induction heads. Note that interacting layers are not necessarily spatially close to each other.
100
075
23
26-
600
92
22
025
60
15
0.
000
-225
10
10
63
(a) Integrated gradients on a (b) Residual erasure on a (c) Integrated gradients (d) Residual erasure on on a two-hop reasoning.
math question.
math question.
two-hop reasoning.
Figure The effect of individual computation steps on Llama 3.1 8B. (a.b) Basic math question. (c,d) Two-hop reasoning. Note that the answer is 4 tokens long in this case, providing a stronger gradient signal. (a,c) Integrated gradients. (b,d) The probability distribution change <Ily 3/122 when erasing the residual of a given token in given layer. This shows until when the information of token is used. In both cases, the second half of the model shows minimal effect. Morcover, arithmetic, later hops of computation do not more depth, indicating that composition is happening.
problems with deeper computation graphs use more layers in the Transformer. Additionally, later steps composite computation should be executed later layers, so that they can receive the results of earlier subproblems as inputs. Are models in fact organizing their computations this way?
Residual erasure interventions. We check if the models are using more depth for later computation based individual prompts. We compute compute the gradient on all answer tokens, but not on the prompt. The second metric is the maximum prediction norm change (Ily 31l2) among the answer tokens when the residual is changed to be uninformative. We call this intervention "residual erasure". It shows until which layer the information from a given token used. This erasure intervention is done for each possible position & and layer i, by setting h : hy while keeping the rest of the tokens unchanged (f4+1 jt'] := h4+1 [t'] for all + l), and the visualizing the effect. The uninformative residual, h, average of the residual in given layer, computed multiple examples (in our case on GSM8K) over batch and time. hy the residual from the original, con-intervened model on the same prompt, and [-] the indexing operation for accessing single element of vector matrix.
metrics: one is Integrated Gradients [15], where

---

Degh soote
20
20
10
10
Layer Logil ollect
Layer
Logit effect
(a) Depth score on different difficulty levels of MATH
(b) Depth score in function of hops on MQUAKE
Figure Depth score: the weighted average of layer index with its importance. Importance is measured based on the effect on later layer contributions future predictions (see Fig. 3b for more details). (a) MATH dataset [30]. The x-axis is the difficulty level defined by the dataset. (b) MQuAKE
[31]. The x-axis is the number of hops in the question. The depth of computation the model performs independent of problem difficulty and the number hops the input problem.
18-
0.2
22
0.9
0:45
Over Sange)
Figure Linear map accuracies predicting Qwen 2.5 14B activations from Qwen 2.5 1.5B. clear diagonal trend visible: layers with the same relative position map to each other the best, indicating that deeper models "spread out" the same kind of computation, computing the prediction smaller steps.
Figure 9: Comparing the residual erasure in Transformers (top) and MoEUT (bottom) on example from DeepMind Math [32], when trained without modeling the question. MoEUT uses more of its depth, and its depth seems be more input-dependent. See Fig 42 and Fig 43 in the Appendix for more, zoomed-in examples.
Fig. summarizes our results. For arithmetic, in both metrics, we see that all tokens are equally important until the middle of the model. For two-hop reasoning, the picture is somewhat less clear, but the second half of the model still shows no sign of computation that is useful for the predictions.
The Depth Score. In order to more systematically verify if the models are using deeper computations of later operations, measure the max influence of layers later layers and the output on future positions, similarly to Fig. 3b and 17b. For each layer, we take the mean effect on all future layers in future tokens, thus reducing the maximum effect matrix to vector with dimension equal to the number layers. For the separation point between the past and future tokens, we sample multiple positions from the answer. Additionally, we further reduce these "effect size" vectors by computing Σ Em = Here, el is the importance of layer I computed by any of our previously defined metrics. We call d the "depth score". This score increases the model uses the later layers more.
We analyze two datasets: MQuAKE [31], which consists of multi-hop questions with known number of hops, and the MATH dataset [30], which consists of complex math problems with different difficulty levels. We consider these difficulty levels proxy for the required computation depth. We measure the above metrics on 20 different random examples for each expected depth. If more hops more complex questions layers increases with complexity, and that should be reflected in our metrics. Fig. summarizes these analyses. We see evidence of deeper computations with increased difficulty. (More detailed plots can be found in Fig. 38 in the Appendix.)
more depth, we expect to that the contribution of the later
3.4 Do Deeper Models Do Novel Computation?
Are deeper models performing computation that is not present the shallower ones because of lack of layers, is it "stretching out" the same kind of computation over more layers, performing smaller

---

steps at a time? The first kind of effect would be preferable, because the deeper model is capable of combining more features and composing more subresults in theory. If such novel computation is present, it should be hard to predict from the activations of the shallow model, and the number of predictable layers should be close to the number of layers in the shallow model. In order to verify this, we take two pretrained models with different layer counts (I1 and Izg with I1 < I22), and train L2 different linear probes to map every point in the residual of the shallower model h to each representation in the deeper model h This requires that the two models use the same tokenizer, and, for reliable results, they should be trained independently, instead of being distilled from each other. Because of GPU memory limitations, we decided use the Qwen 2.5 [33] series of models because of their more modest size compared to the Llama models. Concretely, we train a linear map for each pair of layers of Qwen 2.5 1.5B and 14B. We measure the relative prediction error
| fim (h 1|12
1 lla
where fin() is the linear map from layer m of the small model I of the big model. We do this for each m, I layer pair and plot it in Fig. 8, Although some ranges of layers seem to be easier to predict than others, a clear diagonal pattern visible. This shows that the big model is more likely a "stretched out" version of the shallow model, rather than one that does entirely new computations.
3.5 Is the Pretraining Objective or the Model Responsible for using Fixed Depth?
To explore what causes the models to use fixed depth for each computation step regardless of the problem, we trained standard Transformers and MoEUT [16] on the DeepMind Math dataset [32]. We test MoEUT because Universal Transformers [34] enable easy "transfer" of knowledge from early layers to later ones, thanks to parameter sharing. Additionally, LLMs are trained on free-form text and should model everything regardless of whether they are a "question" (unpredictable) or an "answer" (often predictable given the correct circuits). We also wanted to test whether modeling this uncertainty plays an important role, so we trained both models with and without learning to predict the question.
We use the 244M parameter baseline and MoEUT models from the paper [16], without modifications. We perform the residual crasure intervention on four examples (see Appendix Sec. D.5), and display the most important results in Fig. (with details in Figs. 42 and 43 in the Appendix). We can clearly see that the models that were trained to not model the uncertain question use more of their layers in processing the answer, probably because they do not have to spend their capacity on modeling the high- entropy probability distribution of the unpredictable questions. Surprisingly, MoEUT successfully achieves this even when learning to model the question, although the effect is more pronounced without it. This confirms the advantage of the shared-layer models. Interestingly, while all models have good interpolation performance, their extrapolation capability differs drastically. For example, MoEUT on the "Mul/Div Multiple Longer" split has an accuracy of 36% if modeling the question is enabled, and 63% if it is not. The difference for standard Transformers less dramatic (41 vs. 48%).
Interestingly, all models trained from scratch show increased depth with deeper computation steps. The effect is more pronounced with the MoEUT models, especially if the question is not learned. This is in contrast to what we found for most examples when fine-tuning Llama model (App. D.5), con- firming that fine-tuning might not be enough to change the pretrained model's behavior fundamentally.
4 Related Work
Lad et al. [8] discuss the four stages of inference: detokenization, feature engineering, prediction ensembling, and residual sharpening. The authors show that, in early and late layers, the model is sensitive to layer skipping, but not in the middle. In one of their main claims, the authors show that throughout the layers, the attention to the previous five tokens gradually decreases. However, this might mean that the attention integrates further away context, or might attend to a broader set of tokens. The authors also show slightly reduced contribution of attention compared to the MLP in later layers, but the reduction is gradual and not dramatic. In contrast, we use interventions to directly show that later layers have minimal effect future predictions. The authors also do not study the effects of input complexity on processing depth, nor the effect of increased model depth. Skean et al. [35] discover an information bottleneck in the middle of autoregressive Transformers, and show that the intermediate representations often outperform the final ones for downstream tasks.
Multiple prior works have examined the effect of layer interventions, such as skipping, swapping, or parallclization [9, 10, 8J. In general, they find that models are remarkably robust to such interventions
8

---

on most of the tasks. The notable exception found by all papers are math-related tasks, such as GSM8K. This corresponds to the intuitive expectation that math should require composing subresults. This requires large number of layers in Transformers, proportional to the depth of the computation graph. This is the reason why we decided to focus on math-related tasks, but we found no evidence for such deeper compositions.
For additional related work, please refer to Appendix F.
5 Discussion
Is the second half of the layers wasteful? Our experiments show that a significant proportion of layers are not used to construct higher-level features for downstream computations, but only refine the final probability distribution. Although matching the real probability distribution very closely might be useful for language modeling, for solving downstream tasks, and also in practical models after instruction tuning, only the top few token probabilities are important. surprising, and seems wasteful, that models spend half of their capacity distribution-matching matching instead of further integrating information and doing more composition. The independence of operations performed by later layers also implies that all the information should already be present in the residual simultaneously. Thus, the residual width (dmodel) might be an important bottleneck.
therefore
The consequence of fixed depth computations. Using causal interventions, we show that more complex problems do not cause the computation to shift to deeper layers. Although LLMs lack explicit adaptive computation time mechanisms [36, 37J, they can, in theory, learn to control the amount of computation implicitly. The complete lack of any evidence for dynamic computation is surprising. This means that the models do not break down the problem into subproblems, solve them, and recompose them to solve the full problem, but instead process everything with a fixed circuit on a fixed computation budget. is unclear how such fixed-depth solutions can generalize to the vast compositional structure of the world, without learning different circuits for each situation. The long- tailed distribution of such mechanisms might help explain the diminishing returns of increased scaling.
The connection to Chain of Thought. Chain of Thought [38] avoids the lack of compositional processing in the rich representations of the residual stream by outsourcing it to the input/output space. At inference time, this results in full recurrence with discretization between steps. Other advantages of this approach include supervision on the internal steps (either from pretaining or during fine-tuning). and the discretization denoising intermediate computation steps. On the down side, the model cannot learn to adaptively think more whenever it is needed but not reflected in the training data (e.g., arithmetic operations are rarely written out in papers). The state also limited discrete symbols.
Consequences for Latent Thinking approaches. Recently, method for "thinking" in the latent space [39] was proposed, relying on recurrent processing in the residual stream to avoid some of the limitations of Chain of Thought. If the insensitivity of computation depth to input complexity is the consequence of the pretaining objective, such methods are fundamentally flawed. On the other hand, if the reason is the architecture, these approaches might provide the solution. Thus, determining the reason and finding a possible solution to this problem is an important research direction.
Conclusion
In this paper, we quantify the amount of processing done by each layer and the interaction between layers in pretrained language models. Using causal interventions, we found that in the second half of their layers, these models do not build further on intermediate representations computed in earlier layers. This casts doubts on the efficiency of the mechanisms learned by these models, raising concerns about the importance of later layers. We also found that the depth of processing does not change as a function of input complexity. This indicates that the models do not dynamically build on the output of previous computations to perform more complex ones. This casts doubts on recent approaches that aim to get models "think" in their latent space. We also show that, when learning a linear map between two models with different layer counts, the layers at the same relative positions correspond each other the most, indicating that the deeper model merely spreads out the same type of computation that the shallower one uses. Our exploratory look at the recently proposed MoEUT model indicates that it might use its layers more efficiently than Transformers. Our findings call

---

for research on better architectures and training objectives that can leverage the deep layers more efficiently.
7 Acknowledgements and Funding Transparency Statement
The authors would like to thank David Bau, Shikhar Murty, Houjun Liu, Piotr Piękos, Julie Kallini, and the members of the Stanford NLP Group for helpful feedback and discussions various stages of this project. We are thankful to Adam Belfki, Emma Bortz, and the NDIF engineers for their support with experimental infrastructure. This research is supported in part by grants from Google and Open Philanthropy. Christopher D. Manning is a CIFAR Fellow.
References
[1] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. URL https://cdn openai wtt//eetteeeeeeeeeeeeee wowm/beteer--aaggge---eeee- http::////cco.ooni......mma httpseommmmmiaatttteeeeeeeee pdf.
are_
[2] Tom Brown al. Language models are few-shot learners. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, December 2020.
[3] OpenAl. ChatGPT. attpps://openee httt::/://oppaaa...ccoooooo/otpp 2022.
[4] OpenAL. GPT-4 technical report. Preprint arXiv::333.000777 2023.
[5] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. Preprint arXiv:2302. 13971, 2023.
[6] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 5998-6008, Long Beach, CA, USA, December 2017.
[7] Jackson Petty, Sjoerd van Steenkiste, Ishita Dasgupta, Fei Sha, Dan Garrette, and Tal Linzen. The impact of depth and width on transformer language model generalization. Preprint arXiv:2310. 19956, 2023.
[8] Vedang Lad, Wes Gurnee, and Max Tegmark. The remarkable robustness of LLMs: Stages of inference? Preprint arXiv:2406. 19384, 2024.
[9] Qi Sun, Mare Pickett, Aakash Kumar Nain, and Llion Jones. Transformer layers as painters, In Proc. AAAI Conf. on Artificial Intelligence, pages 25219-25227, Philadelphia, PA, USA, March 2025.
[10] Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Dan Roberts. The unreasonable ineffectiveness of the deeper layers. In Int. Conf. on Learning Representations (ICLR), Singapore, 2025.
[11] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, 7ac Haffield-Dodds, Danny Hernande7, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. http://transfommerrcrcuutss http:///saoommttiiiiiriuu//////o habp/202//i-comteetteammmii and-innuutiiinnnnn heads/index.html.
[12] Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of large language model. Trunsformer Circuits Thread, 2025. URI. https: //transfrrmee circuits wwtpccuitsuuuu///eee pub/2025/ http///ttibiiiiiooo hhaphs//bolggyyyy html.
10

---

[13] Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open IIm leaderboard v2. sttps://huggingface. co/ spaces/open. 1ww-lmmmmmmmeeeee- leaderboarddopen.
11m [mm_ledeerboard 2024.
[14] Nostalgebraist. Interpreting gpt: The logit lens. https://www. www.:...ssswggg... wwwp.e/swwwoggoooooop..aaa.aaaa
gpt-the- 2020.
[15] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proc. Int. Conf. on Machine Learning (ICML), volume 70, pages 3319-3328, Sydney, Australia, 2017.
[16] Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber, Christopher Potts, and Christopher D. Manning. MoEUT: Mixture-of-expers universal transformers. Proc. Advances in Neural Information Processing Systems (NeurIPS), Vancouver, Canada, December 2024.
[17] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Yan Liu. On layer normalization in the transformer architecture. In Proc. Ins. Conf. on Machine Learning (ICML), volume 119, pages 10524-10533, Virtual Only, July 2020.
[18) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. Preprint orXiv:/6072 064500 2016.
[19] Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pages 12360-12371, Vancouver, BC, Canada, December 2019.
[20] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, lliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jeimer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca. Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li. Kenneth Heafield, Kevin Stone, and et al. The Llama 3 herd of models. Preprint arXiv:2407.211
2024.
[21] Jaden Fried Kaato-Ka Alexander Russell Loftus, Eric Todd, Jannik Brinkmann, Koyena Pal, Dmitrii Troitskii, Michael Ripa, Adam Belfki, Can Rager, Caden Juang, Aaron Mueller, Samuel Marks, Arnab Sen Sharma, Francesca Jucchetti, Nikhil Prakash, Carla F. Brodley, Arjun Guha, Jonathan Bell, Byron C Wallace, and David Bau. NNsight and NDIF: Democratizing access to foundation model internals. In Int. Conf. on Learning Representations (ICLR), Singapore, April 2025.
[22] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Hcewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchir Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. Preprint 1rXiv:2110.14166 2021.
[23] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang. Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jian Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang. Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. Preprint ar X1x:25550000388 2025.

---

[24]
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Ma- lik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christo- pher Wilhelm, Michael Wilson, Luke Zettlemoyer Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 OLMo 2 Furious. Preprint arKiv:2501.0060 000656, 2025.
[25] Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, and Thomas Hofmann. Under- standing and minimising outlier features in transformer training. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada, December 2024.
[26] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield- Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022.
[27] Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. Superposition, memorization, and double descent. Transformer Circuits Thread, 2023.
[28] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac HatfieldDoodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https://transfomme-- http://///200///ooommaaiiiiieeee
[29] Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. RAVEL: Evaluating interpretability methods on disentangling language model representations. In Lun- Wei Ku. Andre Martins, and Vivek Srikumar, editors. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 1669-8687,48 Bangkok, Thailand, 2024. Association for Computational Linguistics. URL https: //aclanthology. . org/2024. 22244.2224.2 sel-long. 470.
[30] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, 2021.
[31] Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. Assessing knowledge editing in language models via multi-hop questions. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 15686-15702. Singapore. December 2023.
MQuAKE:
[32] David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. Int. Conf. on Learning Representations (ICLR), New Orleans, LA, USA, May 2019.
[33] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang. Qin Zhu, Rui Men. Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. Preprint 2XXv:24122.1115. 2024.
[34] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Univer- sal Transformers. In Int. Conf. Learning Representations (ICLR), New Orleans, LA, USA, May 2019.
[35] Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models, Preprint arXiv:2502.02013, 2025.
12

---

[36] Alex Graves. Adaptive computation time for recurrent neural networks. In Int. Conf. on Learning Representations (ICLR) Workshop Track, Vancouver, Canada, April 2016.
[37] Andrea Banino, Jan Balaguer, and Charles Blundell. PonderNet: Learning to ponder. Preprint
arXiv:2107.05407, 2021.
[38] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Proc. Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA, November 2022.
[39] Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bar- toldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. Preprint arXiv:2502.05 2025.
[40] Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and GPT-2 embeddings. In Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), pages 55-65, Hong Kong, China, November 2019.
[41] Andreas Veit, Michael J. Wilber, and Serge J. Belongie. Residual networks behave like ensembles of relatively shallow networks. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 550-558, Barcelona, Spain, December 2016.
[42] Wes Gurnee and Max Tegmark. Language models represent space and time. In The Twelfth International Conference on Learning Representations, Vienna, Austria, May 2024.
[43] Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivas- tava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen. Deja vu: Contextual sparsity for efficient 1lms at inference time, In Proc. Int. Conf. on Machine Learning (ICML), volume 202, pages 22137-22176 76, Honolulu, Hawaii, USA, July 2023.
[44] Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, n-gram, positional. In Proc. Association for Computational Linguistics (ACL), pages 1288-1301, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[45] Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, interpretable attention heads in the wild. In Int. Conf. on Learning Representations (ICLR), Vienna, Austria, May 2024.
[46] Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy suppression: Comprehensively understanding an attention head. Preprint arX1v:2310.04625, 2023.
[47] Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontaek Oh, Beomjun Kim, Taehwan Yoo, Seongjin Shin, Dongyoon Han, Jinwoo Shin, and Kang Min Yoo. Peri-In: Revisiting layer normalization in the transformer architecture. In Proc. Int. Conf. on Machine Learning (ICML), Vancouver, Canada, July 2025.
[48] Pengxiang Li, Lu Yin, and Shiwei Liu. Mix-LN: Unleashing the power of deeper layers by combining Pre-LN and post-LN. In Int. Conf. on Learning Representations (ICLR), Singapore, April 2025.
[49] Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large language models. In Conference On Language Modeling (COLM), Philadelphia, PA, USA, October 2024.
[50] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Selective attention improves transformer.
In Int. Conf. Learning Representations (ICLR), Singapore April 2025.
[51] Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. Investigating gender bias in language models using causal mediation analysis. In Proc. Advances in Neural Information Processing Systems (NeurIPS). Virtual Only, December 2020.
[52] Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. In Proc. BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, EMNLP, pages 163-173,
Virtual Only, November 2020. Association for Computational Linguistics.
[53] Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspecting functional modularity through differentiable weight masks. In Int. Conf. on Learning Representations (ICLR), Virtual only, May 2021.
13

---

[54] Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Proc. Association for Computational Linguistics (ACI.), pages 7237-7256, Virtual Only, 2020.
[55] Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, volume 236, pages 160-187, Los Angeles, California, USA,
April 2024.
14

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper organized according the claims,
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contri- butions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Sec. A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., the approach was only tested on few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken low lighting.
speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This is an empirical paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
15

---

All assumptions should be clearly stated or referenced in the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We describe methodology in each section and also provide the codes.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution. reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g.. in the case of a large language model), releasing a model checkpoint, or other means that are appropriate to the research performed.
example, the contribution
While NeurlPS does not require releasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e-g., large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users). but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We submitted the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https:
guides/CodeStune
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
16

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: Codeeeoamiissstaaeeoeaa wwt.miiisssooooioioooooooii for more details.
/ //nips. //tpps.cccpullic/////ccccc
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The paper mainly relies on pretrained models. For the ones that we trained, we provide enough details roughly match our training setup, and we also provide the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We report significance on the linear fit of the performance vs. no. of layers. Most of the experiments are done on individual LLMs. Other times, we plot heatmaps where it is hard put error bars. We have done whenever we could, given computational constraints.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality
errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
17

---

Answer: [Yes]
Justification: Sec. G.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ http///eurrppppppp cc/ppublcc/fbhiicssuude.l......d
Answer: [Yes]
Justification: We are merely analyzing the inner workings of existing public models and conduct- ing controlled synthetic experiments.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We are merely analyzing the inner workings of existing public models and conduct- ing controlled synthetic experiments.
Guidelines:
The answer NA means that there is no societal impact the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e-g., pretrained language models, image generators, scraped datasets)?
18

---

Answer: [NA]
Justification: We are merely analyzing the inner workings of existing public models and conduct- ing controlled synthetic experiments.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper is CC-BY 4.0, the code is MIT.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi apperswithcode thcode. com/ /ataattse has curated licenses for some datasets. Their licensing guide can help determine the license of dataset. For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged reach out the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
The code contains comments on unclear parts, but follows no templates.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We do not crowdsource, and our experiments do not involve humans.
Guidelines:
19

---

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: We do not have human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, non- standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [No]
Justification: We for matplotlib plots and for faster writing well-defined functions, but for the actual research part.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (ttp:s//barrrpp.cc//offfreeeeee https////ooooeeeeee///22222022 http:/rrennnne//22211/11 for what should or should not be described.
20

---

Appendix
A Limitations
The paper is case study on multiple Llama and Qwen models. Although the findings seem to be robust for these models, they might not hold for other model types.
Although the paper shows that models do not do computation that depends on the complexity of the problem, does not answer the question of how models solves the problems without such dependence. This is an important direction for future work.
Sec. 3.4 relies on single model pair, because of the expense of training L L2 big linear classifiers while also keeping the models running. The findings should be verified with more resources on different and deeper models.
Sec. 3.5 relies on manual study of individual trained models. An automatic metric that measures the correspondence of the computation to the parse tree should be developed. However, this is nontrivial task that we leave for future work.
Nevertheless, we believe that our paper provides novel evidence for the high level inner workings of LLMs. We hope that this inspires a future direction of research on how to improve them.
B Model Depth as Factor Shaping Performance
Fig. I briefly analyzes the role model depth in shaping model performance, using a dataset 132 base models the Open LLM Leaderboard [13]. To more deeply explore this relationship, we fit linear regression predicting performance using scale-relevant factors of these models: depth, model dimensionality, and feed-forward dimensionality. (Total parameters is also a potential predictor, but it is highly correlated with these other variables.) Depth is a highly significant variable in this model (p 0.0001). This result is highly robust to rescaling of the independent variables and including model family as hierarchical grouping factor. Thus, it seems clear that making models deeper does make them better, even though the models themselves do not seem their depth efficiently.
C Robustness Analysis
To justify our choice of a relatively low number of samples (10-20, depending on the experiment) and single dataset (GSM8K), we present results for more samples and for different dataset (Math). We show the influence of layers and sublayers on the residual stream, equivalent to Fig. 2, in Fig. 10. Furthermore, we show the layer skipping effects, equivalent to Fig. 3, in Fig. 1 I. and the Logitlens- based output similarity, corresponding to Fig. 4, in Fig. 12. These analyses show that our conclusions are robust both with respect to the dataset choice and the number of examples.
To show the variability between individual examples, we show 4 examples contributing to Fig. 3b in Fig. 13. Their variability justifies our choice to take the maximum over them to quantify the overall importance of the layers.
D Results on Other Models
The performance on the HELM Lite benchmark of a few important models is shown in Fig 14. Performance improves with the number of layers, similar to the Open LLM leaderboard (Fig. 1).
D.I How do the Layers Interact With the Residual Stream?
We show the absolute and relative contributions of the sublayers to the residual stream for Llama 3.1 8b in Fig. 18, for the Qwen series of models in Fig 19 and for OLMo 2 models in Fig 20. We show cosine similarities of the sublayer's contributions and the residual stream for all Llama and Qwen models that tested in Fig. 21 and for the OLMo models in Fig. 22. The results are similar to our findings in Sec. 3.1.
We show the cosine similarity of the neighboring layers (cossim(h,, (r, h441)) in Fig. 16. All neighbor- ing layers have very high cosine similarity, often close to 1, which is a consequence the known
21

---

Tolls//Adl MUP: Imclallleelz
Atertton comtriantA M.P: couinions Au)
Noi Bagnee 10000
62
Attentionn M.P. (|s. wwdll Arls
Colne ainklie
011
MUP:
0.8
600
-01
84-
-02
005
30
70
40
Light indox
indix
(a) Math datasct: Relative norm of (sub)layer con- tributions
(b) Math dataset: Cossims of (sub)layer contribu- tions and the residual
6J1
Man B83 Cortitatee
622
Attention McP. l|e, milz/ Adla
0.1
Amtriion cos min. Ae)
-0.2
MLL contirins,
-033
30
70
40 indix (1)
indoo
(c) 50 examples: Relative norm of (sub)layer con- tributions
(d) 50 examples: Cossims (sub)layer contribu- tions and the residual
Figure 10: Robustness analysis: influence of layers and sublayers on the residual stream Llama 3.1 70B. (a.c) Norm of contributions relative to the residual stream. (b.d) Cosine similarity between contributions. Math dataset (a,b) and 50 examples on GSMSK (c,d) are shown. The findings are consistent with Fig 2.
anisotropy of Transformers [40]. However, as Fig. 2b and Fig. show, comparing the cosine similarity of the residual stream and the contributions of the layers and sublayers reveals rich structure.
D.2 How do the Layers Influence Downstream Computations?
Here, we show the effect of individual layers on later layers future timesteps and on future token outputs for multiple models. Fig 23 shows the effect on Llama 3.1 8B and 70B, while Fig. 24 shows the Qwen series models, Fig. 25 shows the instruction-tuned Llama 3.1 70B, and Fig. 26 shows the OLMo models. It can be seen that instruction tuning has influence on the model's behavior. In the Qwen series models, the effect seems to be less pronounced, but still present. OLMo behaves very similarly Qwen 3. The findings agree with our discussion in Sec. 3.2.
We show the local layer interactions for other Llama models Fig. 27, for the Qwen models in Fig. 28 and for ML 2 in Fig. 29.
Additional Logitless results are shown for the other models in Fig. 30 and Fig.31.
Qwen 32B (Fig. 24c) seems to display an additional interesting effect: early layers seem to work independently, not building each other's 's representation from the previous timesteps. The integration across time seems to start at around layer 40. Interestingly, once this point in the network reached, all previous computations seem to be important. Fig 28f shows the existence of a single layer that integrates most of the information from the past. This layer composes features of many previous layers.
We show the cosine similarities between the contributions and the residual for all tested models in Fig, 21.
22

---

1.0
1.0
10
10
0.8
0.8
Relative change
0.6
30
0.6
40-
Relatie
40
Layr
50
0.4
50
0.4
0.2
0.2
70
0.0
0.0
20
40
60
60
Effect byer
Effect layer
(a) Math Dataset: Effect of skipping layer on later layers' contributions the all timesteps
(b) Math Dataset: Effect of skipping layer
later layers' contributions in fature timesteps.
1.0
0.8
20
Relative change
skipod
an
30
40
40
Reatbe
60
70
70
60
Ellect
layar
Effect @ layer
(c) 50 examples: Effect of skipping layer on later layers' contributions in the all timesteps
(d) 50 examples: Effect of skipping a layer on
later layers' contributions in future timesteps.
Figure 11: Robustness analysis: effects layer skipping on all tokens (a,c) and future tokens (b,d), on the Math Dataset (a,b) and with 50 examples GSM8K (c,d). All variants agree with our findings in Fig. 3.
D.3 Do Deeper Problems Use Deeper Computation?
Additional residual crasure experiments are shown for Llama 3.1 70B in Fig. 32, for the Qwen series of models in Fig. 33 and for OLMo in Fig. 34. Findings for the 70B Llama models are identical to those discussed in Sec. 3.3. Qwen models use more layers, but they also seem use fixed number of layers independently of the computation depth, indicating that they are not building on subresults from previous computation steps. OLMo seems have weaker depth dependence than the other models. All findings are consistent with Sec 3.3.
We show the depth score on MATH and MQuAKE datasets for Llama 3.1 8B in Fig. 35 and for the Qwen 3 series of models in Fig. 33. The findings are identical to what we discussed in Sec. 3.3.
D.4 Do Deeper Models Do Novel Computation?
In Sec. 3.4 we used Qwen 2.5 1.5B and 14B models instead of the newer Qwen series. The reason for this is twofold: first, given that we had to train L, different linear maps of substantial size, chose small models to be able to both models simultancously on a single A6000 GPU, Second, given this size limitation, the difference in the layer count of the Qwen 2.5 is higher than the viable options from the Qwen 3 series.
D.5 Does Finetuning Cause the Model to Use Deeper Computations?
finetune all parameters Llama 3.2 3B on the arithmetic splits of the DeepMind Math dataset [32], with batch size 64, for 10k steps, with warmup of 100 steps followed by constant learning rate of * 10-5, At the end of the training, we perform integrated gradients and residual erasure experiments both the base model, which was the starting point of the finetuning process, and the
23

---

KL Divergence
10.0
0.6
7.5
Overtap
0.4
5.0
0.2
2.5
50
40
Layer
Layer
(a) Math Dataset: KL divergence between the model's prediction and Logitlens applied to each layer.
(b) Math Dataset: Overlap in top-5 tokens from the model's prediction and Logitlens applied to each layer.
10.0
0.6
7.5
Overtap
5.0
XL
40
30
40
80
Layer
Layer
(c) 50 examples: KL divergence between the model's prediction and Logitlens applied to each layer.
(d) 50 examples: Overlap in top-5 tokens from the model's prediction and Logitlens applied to each layer.
Figure 12: Robustness analysis: comparing Logitlens on different layers to the final prediction, (a,c) KL-divergence. (b,d) Overlap in the top-5 predicted tokens. Math dataset (a,b) and 50 examples on GSM8k (c.d) are shown. The findings are consistent with Fig. 4
final model. We also include the instruction-tung version of the model as a control. We measure the maximum effect on later layers and predictions futuree tokens and find that fine-tuning seemingly helps to increase the computation depth significantly (Fig. 39). However, by looking deeper at individual instances, we reveal that the effect is mostly marginal: only the last 1-2 tokens are affected before the prediction, and the residual erasure experiment shows no significant difference in the point when they become unimportant, indicating that they are used parallel (Fig. 40). We also tried applying the loss only to the answers, but in contrast to pretraining (Sect. 3.5), it seems to have no effect (Fig. 41).
E Details on the DeepMind Math Training
We fine-tuned tuee-uun//pretain our models on the arithmetic subset of the DeepMind Math dataset. These are all the files in the train set that begin with the string "arithmetic ". To feed example to the network. use the template "Q: question A: answer". To fill the context window of the model, we concatenate multiple such examples with whitespace between. We never break examples if they do not fully fit the context window; the end of the window is padded as needed.
Extended Related Work
Previous studies on the residual stream suggested that ResNets behave like an ensemble of shallow networks [41]. Gurnee and Tegmark [42] showed that in Transformer language models, linear probe accuracy increases rapidly in the first half of the model, and the improvements become marginal in the second half. phase transition was also previously observed around half of the model, where the activations transition from sparse activations to dense [43, 44]. Logitlens [14] was also observed to provide meaningful predictions from around the middle of the network. The growth of the residual stream was previously observed in the context of outlier features [25], where they were hypothesized be one of the causes the outlier features, and for large-scale Universal Transformers [16), where they present obstacle mechanism reuse.
24

---

1.0
10
Layer stipes
20
Relative change
Rel aivevvee char lange
30
30
0.6
40
40
Layer
60
70
70
20
Effect @ layer
Effect @ layer
10
Layer skipped
20
Pedatwo change
Layer skipped
20-
Rela live change
30
30
0.4
60
60
70
20
60
40
60
layer
Effect @ layer
Figure 13: Individual examples for effeets of skipping a layer on later layers' contribution future timesteps (4 individual elements contributing Fig. 3b). The variability of the importance of individual layers motivates our approach of taking the maximum over them to quantify their overall importance.
Queed Eleanee
Nanterolless
Figure 14: HELM Lite score in function of layers. The area of the dots is proportional to the parameter count. Deeper models generally perform better.
Artencion: [juila
100
[/m/]]a
Residual: [Ailla
Vien Nom
Layer indix
Figure 15: norm of layer inputs and the sublayer contributions before summing into residual for Llama 3.1 70B. Norm for each layer.
Prior work also studies the mechanisms that are used perform certain operations in Transformers. Perhaps the most well-known are the induction heads [11]. In follow-up work, successor heads [45] and copy suppression [46] were discovered. Recently Lindsey et al. [12] described large variety of circuits performing various functions in the network, including the mechanisms responsible for addition. Although these mechanisms necessarily span multiple layers, a common pattern that they compose low-level sub-operations into high-level operation. To the best of our knowledge.
25

---

1.0
0.0
Cvelre siniletty
0.4
0.2
10 20 30
40
80
50
Layer
Figure 16: Cosine similarity between the residual at neighboring layers (cossim(h), (h,, h(+1)) of Llama 3.1 70B. The representations are focused on the right subspace, resulting in high cosine similarities. However, focusing on the layers' and sublayers' contributions is more informative (Fig. 2b).
12-
Oiept cargg nom
0.8
0.8
0.8
0.6
0.4
0.2
0.2
50
50
Layer
(a) Effect of skipping layer on all predictions.
(b) Effect of skipping layer future predictions.
Figure/ 17: Analyzing the importance layers on output predictions Llama 3.1 70B on GSM8K. The figure shows the maximum change of the output probabilities. (a) shows all tokens in the sequence when skipping a layer, including the current predictions, while (b) isolates the effect just for the maximum of the fature tokens. (a) shows that despite the low effects of the layers consecutive computations in the second half of the network (Fig. 3), the layers play an important role in the predictions. However, as (b) shows, their importance is minimal for future predictions. The second half of the layers seems to perform mostly independent, but important, computations to refine the current predicted probability distribution. This is further support for the findings of Fig, in the main text.
there is no evidence of higher-level conditional composition, where a mechanism is sometimes used directly produce the output, while other times it used in composition with another high-level mechanism to compute a more complex function.
Previously, methods were proposed that might help increase the effectiveness of the feature building in deeper layers. For example, Kim et al. [47] show that their method increases the angular distance between the representations of different layers, indicating that it might use deeper layers more efficiently. Li et al. [48] mix post-layernorms in early layers and pre-layernorms in later layers to improve gradient propagation. However, we models using these configurations in order to verify their effectiveness. However, we tested the OLMo series of models with a different layernorm structure, resulting similar conclusions the standard pre-layernorn models.
aware of freely available popular large language
The collapse of importance of attention in the later layers could be related to massive activations [49] that have been shown to collapse the attention to a few discrete tokens, potentially reducing their effectiveness. It might also partially explain the success of selective attenetion pruning [50].
More broadly, causal intervention methods gained popularity recent years [51, 52, 53, 54, 55). These methods are capable of providing deep insights on how neural networks operate. By direct interventions on the hypothesized mechanisms, they provide strong evidence and avoid accidental reliance on surface correlations.
26

---

70-
Astantion: l
1.4
Attetionn /eell////alll
MLP: [/mu]kk
Mean Ralatve Contribation
1:2-
MLP: [emile//|/ullzz
Revihrct [|A,]]
MLP. mulls/ Adcl
Mann Ker
00-
0.5
0.5
20-
0.4
0.2
0.9
1:
20
200
50
Layer nnxe (1
Laver indox
(a) Residual norm growth: 8B
(b) The relative norm of the sublayer's contributions: 8B
Attanion: lierls
15.0
Reiidull [Mulla
MLP: mull
Men Rebetivn Contributio
1.2
Atheation [[o, b/
MIP: ||mmlla//| -x/z
1,0
Max Norm
10.0
7.5-
0.0-
0.1
2.5
20
160
120
00
50
100 120
(c) Residual norm growth: 405B
(d) The relative norm of the sublayer's contributions: 405B
Figure 18: 1.2 norm of layer inputs and the sublayer contributions before summing into the residual for Llama 3.1 8B and 40SB on GSM8K. (a,c) shows the norm for each layer/sublayer, while (b,d) compares the norm of the sublayer outputs to their input, quantifying the relative change induced by sublayer (limited to max 1.5 for better visibility). A sharp drop visible near the middle of network. The second half of the layers changes the residual significantly less than the first half, with exception last few layers. The findings are similar to what we have demonstrated Fig. 15 and 2a
G Hardware Resources
Most of our experiments were done using NNSIGHT and NDIF [21], not requiring local hardware. The experiments on the Qwen models and the Llama 3.1 70B Instruct models, which are not available on NDIF, are done on 4 Nvidia A6000 48Gb GPUs, with rough duration of day for the 70B experiment, and another day for all the Qwen experiments.
For Sec. 3.5, we trained each model on 2 Nvidia A100 80Gb GPUs for 2 days.
Full-finttuing Llama 3.1 3B on the DeepMind Math Dataset (Sec. D.5) done 4 Nvidia H200 GPUs for 10 hours.
Training the linear maps between the pair of layers of the Qwen models (Sec. 3.4) was done A6000 GPUs, taking 80 GPU-days total.
27

---

Actention: |/c/b
1.4
Tou afll
12:0
[pmilla
Man Beloteee Contriaation
1.2.
MLP: [|vellaa/]/alll
1000
Residual:: [Mhlla
1.3
Attention milla///ee
Moon ham
0.5
0.4
0.5
10
10
25
00
15
20
25
Index
(a) Residual norm growth: 8B
(b) The relative norm of the sublayer's contributions: 8B
Attention: |sullaa (Mill
MLP: [jm,]la
Meen Delatie Contritation
1.2-
[|mmlla/]AAlla
Residual: [Mulla
Attuntion MLP. (lo, my af
1.0
Max Koem
0.6
0.4
0.2-
35
01)
Layer
(c) Residual norm growth: 14B
(d) The relative norm of the sublayer's contributions:
14B
2000
Arteenioe: [|u]]
[imulla
Menn Relatvee Cortitttion
1.4
1.2
MILP: ||po|]l//||Ajla
Residual: [Mulla
Attention MLP. lallAl
Maxt Bomm
1.500
0.3
1000
0.5
500
0.9
10
20
10
60
20
30
Layer inden (1)
Layer index
(c) Residual norm growth: 32B
(f) The relative norm of the sublayer's contributions: 32B
Figure 19: L2 norm of layer inputs and the sublayer outputs before summing into the residual for the Qwen 3 series of models GSM8K. (a,c) shows the norm for each layer, while (b.d) compares the norm the sublayer outputs to their input, quantifying the relative change induced by the sublayer. Relative contributions clipped to 1.5 maximum. The contribution of later layers remains more stable than the Llama models (Fig. 15 and 2a), especially for the 14B model. (e.f) Findings for Qwen 32B. This model seems differ from all the others examined: it uses all its layers. However, the importance of the early layers is lower than the late ones.
28

---

0.3
Artention: (egla
Attention: Mnllla//Mlll
MLP: [[wtella
Mars Retatbe Contri ston
[|mmllaa/|iille
Resiiual [|N]]a
0.5
Atsentionn MLP: (lo, my as Aelle
Meon lomm
0.4
20-
0.2
10-
Layer index (1)
Layer Index
(?)
(a) Residual norm growth: 7B
(b) relative norm the sublayer's contributions:
MLP: [imu]ks
Beatue Contrtuttoon
0.7-
MLP: ||mmlla/|PPlla
Residist [|A,]]
0.5-
Attontion lalllla
Main Rom
60
0.3-
0.4
60-
0.3-
0.2
0.1
13
20
20
30
45
10
15
200
30
index
Index
(c) Residual norm growth: 13B
(d) The relative norm of the sublayer's contributions: 13B
Ascention
Assntiin:: Heulls/|Alll
100
MIP: [bmilla
Meon Delativ Contribition
Residual: Mellz
Atsention MLP (lo,
is
0.3
Mann
0.2
0.1
0-
0.0
Laser index (1)
Laver index (
(c) Residual norm growth: 32B
(f) The relative norm sublayer's contributions:
32B
Figure 20: I.2 norm of layer inputs and the sublayer outputs before summing into the residual for the
OLMo 2 series of models GSM8K. (a,c) shows the norm for each layer, while (b,d) compares the norm the sublayer outputs their input, quantifying the relative change induced by the sublayer. The relative cointribution of inidividual layers is significantly lower than the Llama (Fig. 15, 2a) and Qwen (Fig. 19) models.
29

---

0.4
moemm(n). by)
MtP: consi((ms,
0.22
+ MIP costin my, fu)
Costr siiletty
Coarr simibetty
0.0
0.0
consimjn, Av)
0.2
MLP: cossinmm, (v)
Layer index (1)
Layer inden (1)
(a) Llama 3.1
(b) Llama 70B
Assention:
0.9
Attansion: As)
MLP: cossntsnn, Ar aul
MLP: cossinton, Ay ai)
Coutre simibrtty
0.2
Attention MIP
Coutre simlaryy
0.3
Attencion MLP. consimiae
0.1
0.2
0.1
0.0
100
100
15
(1)
(c) Llama 3.1 405B
(d) Qwen 3 8B
0.5
Attention: conamn br)
0.6
Atternion: comsinis, Ao)
cosimmtv,,
0.4
Attention MLP: coedmjor my hr)
Atternion MLP coedmmn m. Av)
Coarr umbrry
Codre simberyy
0.2
0.0
10
30
35
10
60
Layer Index (1)
Layer Index (7)
(c) Qwen 14B
(f) Qwen 3 32B
Figure 21: Cosine similarity between the sublayers' contributions and the residual for the LLama 3.1 and Qwen 3. They all show consistent picture with Fig. 2b.
30

---

0.3
Attention: cosim(ai..h))
cosin((m,,
0.2
Athention + Ml Ph cootmm so, hu)
Cosne Cosnn simbrrt
Cosne Coine similaity
0.0
-0.1
-0.-
Attemiiss: cominia, Au)
-0.2
cussinlom, Av)
-03
Attention MLP. cossintor 1i, Av)
33
30
30
30
33
index (1)
Layer index (1)
(a) OLMo 2 7B
(b) OLMo 2 13B
Attention: cosei((u..h)
0.2
MLP: cosúi((m,.AA
cossinco: In)
Coarr
-0.4
Jayer inder (1)
(c) OLMo 32B
Figure 22: Cosine similarity between the sublayers' contributions and the residual for OLMo 2. They all show a consistent picture with Fig. 2b.
31

---

0-
1.2
Outgut change norm
Layer skpped
10
Rolattvo channge
0.8
15
0.6
20
25
30
0.0
30
10
15
20
25
30
Effect @ layer
Layer
(a) Effect of skipping layer on later layers future timesteps: 8B
(b) Effect skipping a layer on future predictions: 8B
1.0
20
0.8
Output change nome
1.0
Layer skipped
40
helaitee change
0.8
0.6
0.4
80
0.4
100
0.2
0.2
120
0.0
0.0
50
100
20
40
60
80
100
120
Effect layer
Layer
(c) Effect of skipping layer later layers future timesteps: 405B
(d) Effect of skipping layer future predictions: 405B
Figure 23: Analyzing the importance of layers computations in later layers and output predictions for Llama 3.1 8B and 405B on GSMSK, focusing on the effect on future tokens. (a.c) The maximum relative change in the layer's output when previous layer is skipped. The second half of the layers has a weaker effect on future computations compared to the first. The range is limited between and 1. (b,d) The maximum change in the output probabilities. The findings identical to the ones discussed in Fig. 3.
32

---

Output change norm
stppes
10-
00lan change
15
0.8
20
0.6
0.4
0.2
35
0.0
0.0
10
30
5
10
15
20
25
30
35
layer
Layer
(a) Effect skipping layer on later layers
(b) Effect skipping layer on future predictions: 8B
future timesteps: 8B
1.0
1.4
0.8
Output change norm
1.2
Skpped
10
Aonatu changg
1.0
15
0.6
20
0.4
25
30
0.2
0.2
0.0
0.0
10
20
30
10
20
30
40
Effect
Layer
(c) Effect of skipping layer later layers future timesteps: 14B
(d) Effect of skipping layer future predictions: 14B
1.0
1.4
10
0.8
Output change horm
1.2
skipped
20
Sonno charg
1.0
0.6
30
0.8
40
0.4
0.6
50
0.2
60
10
Effect @ layer
Layer
(e) Effect of skipping layer on later layers in future timesteps: 32B
(I) Effect of skipping layer on future predictions: 32B
Figure 24: Analyzing the importance of layers on computations in later layers and output predictions on Qwen 3 series of models GSM8K, focusing the effect future tokens. (a,c,e) The maximum relative change in the layer's output when previous layer is skipped. The second half of the layers has weaker effect on future computations compared the first. The range is limited between and 1. (b,d.f) The maximum change in the output probabilities. The findings for the and 14B models are identical the ones discussed in Fig. However, the 32B model behaves differently: also displays the reduced effects on future predictions in the late layers, but more interestingly. the lower layers seem not to build on each other's computation, but just accumulate information the residual, which will be used in late layers.
33

---

1.4
Output change norm
1.2
20
Colant change
Laerr skpped
30
0.6
50
60
0.4
70
0.2
0.0
0.0
20
60
10
20
30
40
50
60
70
80
layer
(a) Effect skipping layer on later layers in future timesteps.
(b) Effect of skipping layer on future predictions.
Analyzing layer importance for future predictions in Llama 3.1 70B Instruct. (a) The maximum relative change in the layer's output when a previous layer is skipped. It can be seen that layers in the second half of the model have minimal effect on the future computations. (b) The maximum relative change in the output probabilities. Instruction tuning seems to somewhat increase all layers' significance the future predictions. However, the stark difference between the first and second halves of the model is still present. Compare to Fig. 3b and 17b.
Figure 25:
34

---

1.2
Output change norm
00lan change
Layer stppes
10
0.8
15
0.4
25
30
0.0
0.0
10
20
30
0
5
10
15
20
25
30
layer
Layer
(a) Effect skipping layer on later layers
(b) Effect skipping layer on future predictions: 7B
future timesteps: 7B
1.0
1.2
Skpped
10
0.8
Anlabuu dhange
Output change norm
1.0
15
0.6
0.8
20
0.6
0.4
25
0.4
30
0.2
0.2
0.0
0.0
10
20
30
10
20
30
40
Effect
Layer
(c) Effect of skipping layer later layers
(d) Effect of skipping layer future predictions: 13B
future timesteps: 13B
1.0
10
0.8
Output change norm
1.2
skipped
20
Belntuu mharge
1.0
0.6
30
0.8
40
0.4
0.6
50
0.2
60
10
Effect @ layer
Layer
(e) Effect of skipping layer on later layers in future timesteps: 32B
(I) Effect of skipping layer on future predictions: 32B
Figure 26: Analyzing the importance of layers on computations in later layers and output predictions on OLMo 2 series of models GSM8K, focusing the effect future tokens. (a,c,e) The maximum relative change in the layer's output when a previous layer is skipped. (a,c) The second half of the layers has a weaker effect on future computations compared the first. The range is limited between 0 and 1. (b,d,f) The maximum change in the output probabilities. The findings for the 8 and 13B models are identical the ones discussed in Fig. However, the 32B model behaves similarly to Qwen 32B (Fig. 24e): it also displays the reduced effects on future predictions in the late layers, but more interestingly. the lower layers seem not build on each other's computation. but just accumulate information in the residual, which will be used in late layers.
35

---

1.0
1.0
5
0.8
Layer skipgea
15-
Releive
15-
20
0.4
Loyer
0.4
0.2
0.2
30
30
20
30
20
30
Effect layer
Effect layer
(a) Llama 8B: Local effect of layer on later layers' contributions in the all timesteps.
(b) Llama 8B: Local effect layer on later
layers' contributions in future timesteps.
1.0
1.0
40
Reletie changge
Layer slpped
40
Reletive change
0.6
0.6
0.4
0.4
80
0.2
0.2
120
120
50
100
50
100
layer
Llama 3.1 405B: Local effect of layer later layers' contributions the timesteps.
(d) Llama 3.1 405B: Local effect of layer
later layers' contributions future timesteps.
Figure 27: Analyzing the direct local effects between pairs of layers of Llama 3.1 models. It highlights layer pairs with direct effect on each other. The effects are not propagated to future layers. For each layer s, the plot shows future layers that build the representation computed by (a,c) Effeets all tokens, highlighting all possible circuits. (b,d) Effect on future tokens. The sparse, bright spots indicate multi-layer, multi-token mechanisms, such as induction heads. Note that interacting layers are not necessarily spatially close to each other.
36

---

0.8
0.8
Layer skipped
Layer stppes
10:
15-
15
Relative
0.2
0.2
0.0
Effect layer
Effect layer
(a) Qwen 8b: Local effect of layer on later layers' contributions in the all timesteps.
(b) Qwen 8b: Local effect layer on later
layers' contributions in future timesteps.
5
5
0.8
0.8
10
Relative change
sappas
Relative change
15
0.6
20
20
0.4
0.4
30
0.2
30
0.2
10
Effect @ layer
Effect @ layer
(c) Qwen 14b: Local effect of layer on later layers' contributions in the all timesteps.
(d) Qwen 14b: Local effect layer on later layers' contributions in future timesteps.
1.0
1.0
10
0.8
10
0.8
Relative changg
change
20
20
0.6
30
30
40
0.4
40
0.4
50
0.2
0.2
60
60
60
Effect
layer
Effect @ layer
(e) Qwen 32b: Local effect of layer on later layers' contributions in the all timesteps.
(f) Qwen 32b: Local effect layer on layers' contributions in fufure timesteps.
Figure 28: Analyzing the direct local effects between pairs of layers of Qwen models. highlights layer pairs with direct effect on each other. The effects are not propagated to future layers. For each layer s, the plot shows future layers that build on the representation computed by s. (a.c.e) Effects on all tokens, highlighting all possible circuits. (b,d,f) Effect on future tokens. The sparse, bright spots indicate multi-layer multi-token mechanisms, such as induction heads. Note that interacting layers are not necessarily spatially close to each other. Interestingly, Qwen 32b shows single layer that moves most of the features at once from previous layers future tokens.
37

---

1.
100
Lyyer stpppdd
Laver slpppdd
15
15
Relelive
20
0.4
20
0.44
0.2
30
30
10
20
30
10
20
30
layer
layer
(a) OLMo 2 7b: Local effect of layer later layers' contributions in the all timesteps.
(b) OLMo 7b: Local effect of layer later
layers' contributions in future timesteps.
1.0
1.0
5
5
Relelive chenge
dkaped
Relelive change
15-
0.6
15
0.6
25
0.4
25
0.4
35
35
0.0
0.0
10
20
30
10
20
30
Effect layer
layer
OLMo 2 13b: Local effect of layer later layers' contributions timesteps.
(d) OLMo 13b: Local effect of layer later
layers' contributions in future timesteps.
1.0
1.0
10
0.8
20
Relativ change
skipped
20
Rellative charge
0.6
0.6
40
0.4
40
0.4
50
50
0.0
0.0
20
40
60
20
40
60
Effect
Ertect
(c) OLMo 2 32b: Local effect of layer later layers' contributions the all timestops.
(f) OLMo 32b: Local effect of layer later layers' contributions in future timesteps.
Figure 29: Analyzing the direct local effeets between pairs of layers of OLMo models. It highlights layer pairs with a direct effect on each other. The effects are not propagated to future layers. For each layer 8, the plot shows future layers that build the representation computed by (a,c,e) Effects all tokens, highlighting all possible circuits. (b,d,f) Effect on future tokens. The OLMo models seem have significantly stronger contributions both the same token (a,c.e) and the future tokens (b,d.f, compared to the LLama (Fig. 5, 27) and Qwen (Fig. 28) models. This can be probably attributed to their reordered norm, where the normalization is applied after the layers, before merging back the residual.
38

---

D.veerggncc
Overloo
0.4
0.0
10
15
20
25
30
10
15
20
25
30
Layer
(a) Llama 3.1 8B: KL divergence between Log-
(b) Llama 8B: Overlap top-5 predicted tokens
final prediction
KL Divergence
0.6
Overlap
0.4
0.0
10
30
80
Layer
Layer
(c) Llama 3.1 70B Instruct: KL divergence between Logitlens and final prediction
(d) Llama 3.1 70B Instruct: Overlap in top-5 pre-
dicted tokens
0.6
20
Overlap
0.4
10
0.2
15
35
10
30
Layar
Layar
(e) Qwen 8B: KL. divergence between Logitlens and final prediction
(f) Qwen 8B: Overlap in top-5 predicted tokens
Divergence
0.6
20
Overlpp
0.4
10
0.0
5
15
20
25
30
35
40
5
10
15
20 Layer
25
30
35
40
Layer
(g) Qwen KL divergence between Logitlens and final prediction
(h) Qwen 14B: Overlap in top-S predicted tokens
20
0.6
Divergence
Overlpp
10
0
0.0
0
10
20
30
40
50
60
10
20
30
40
(i) Qwen 3 32B: KL divergence between Logitlens and final prediction
Qwen 32B: Overlap in top-5 predicted tokens
Figure 30: Comparing Logitlens probes from different layers to the final prediction for different models. Left: KL divergence between the output the Logitlens and the final prediction. Right: Overlap between the top-5 tokens predicted by Logitlens and the final model prediction.
39

---

0.6
10
Overlap
0.4
KL
0.2
20
15
20
25
Layer
Layer
(a) OL 7B: KI. divergence between Logitlens and final prediction
(b) OL 7B: Overlap in top-5 predicted tokens
0.8
0.6
10
Overtop
0.4
0.0
5
10
15
20 Layer
25
30
35
40
10
15
20 Layer
25
30
35
40
(c) OLMo 13B: KL divergence between Log-
OL 2 13B: Overlap top-5 predicted to-
itlens final prediction
kens
15
0.8
Divergence
Overbe
0.2
0.0
10
20
Layer
Layer
(e) OLMo 32B: KL divergence between Log- itlens and final prediction
(f) OLMo 2 32B: Overlap in top-5 predicted tokens
Figure 31: Comparing Logitlens probes from different layers to the final prediction for OLMo models. Left: KL divergence between the output the Logitlens and the final prediction. Right: Overlap between the t tokens predicted by Logitlens and the final model prediction.
40

---

(a) Integrated gradi- ents on a match ques- tion.
(b) Residual erasure effect on a match question.
(c) Integrated gradients on two-hop reasoning.
(d) Residual erasure on two-hop reasoning.
Figure 32: Analyzing the effect of individual computation steps on Llama 70B. (a.b) Basic math question. (c,d) Two-hop reasoning. Note that the answer is 4 tokens long in this case, providing a stronger gradient signal. (a,c) Integrated gradients. (b,d) The probability distribution change (1ly 3l12) when erasing the residual of given token in given layer. This score shows until when the information from a column used. both cases, the second half of the model shows minimal effect. Moreover, in arithmetic, later hops of computation do not more depth, indicating that composition is happening.
41

---

(a) Integrated gradients on a match question: 8B
(b) Residual erasure ef- feet on a match ques- tion: 8B
(c) Integrated gradients on a two-hop reasoning: 8B
(d) Residual erasure on a two-hop reason-
ing: 8B
40
31
(e) Integrated gradients match question:
(I) Residual erasure ef- feet match ques- tion: 14B
(g) Integrated gradients two-hop reasoning:
(h) Residual erasure two-hop reasoning:
14B
14B
(i) Integrated gradi- ents on match ques- tion: 32B
(i) Residual erasure effect on a match
(k) Integrated gradi- on two-hop
(1) Residual crasure on two-hop rea- soning: 32B
question: 32B
reasoning: 32B
Figure 33: Analyzing the effect of individual computation steps the Qwen 3 series of models. (a.b.e,f,i.) Basic math question. (c.d.g.h.k.I) Two-hop reasoning. Note that the answer is 4 tokens long in this case, providing stronger gradient signal. (a.c.e.g.i.k) Integrated gradients. (b.d.f.h.j..) The probability distribution change (lly 7l12) when erasing the residual of a given token in a given layer. This score shows until when the information from column is used. The models use more layers compared to the Llama series (Fig. 6), but still show no increased depth for later computations.
42

---

11
-00
(a) Integrated gradients on a match question: 7B
(b) Residual erasure
(c) Integrated gradients on two-hop reasoning: 7B
(d) Residual erasure on a two-hop reasoning:
fect on match question: 7B
7B
100
400
(e) Integrated gradients
(1) Residual erasure ef- feet on match ques- tion: 13B
(g) Integrated gradients two-hop reasoning: 13B
(h) Residual erasure two-hop reasoning:
match question:
13B
13B
(i) Integrated gradi- ents on a match ques- tion: 32B
(j) Residual crasure effect match question: 32B
(k) Integrated gradi- ents on two-hop reasoning:
(1) Residual crasure on two-hop rea- soning: 32B
Figure 34: Analyzing the effect of individual computation steps the OLMo 2 series of models. (a.b.e,f,i.) Basic math question. (c.d.g.h.k.I) Two-hop reasoning. Note that the answer is 4 tokens long in this case, providing stronger gradient signal. (a.c.e.g.i.k) Integrated gradients. (b.d,f.h.j.I) The probability distribution change (lly y|12) when erasing the residual of a given token in a given layer. This score shows until when the information from column is used. Up to 13B, the models seems to do shallower processing than both the Llama series (Fig. 6) and Qwen (Fig. 33).
43

---

Depth Dapin
7.5
5.0
Layer ertect Lookt
Layer eftect Logit
(a) Depth score on MATH dataset different difficulty examples: 8B
(b) Depth score MQuAKE on different number of hops: 8B
Figure 35: Depth score for Llama 8B: the weighted average of layer index with its importance, as function of a given difficulty metric. Importance measured based on both the effect on future internal computations and on the effect on future predictions. (a) MATH dataset. The x-axis is the difficulty level defined by the dataset. (b) MQuAKE. The x-axis is the number of hops in the question. The findings are similar to Llama 8B (Fig. 7).
10
Depth
Layer Logit olfect
affict Looit offect
(a) Depth score on MATH dataset on different difficulty examples: 8B
(b) Depth score on MQuAKE on different number of hops: 8B
Degih soore Booe
Layor oflect
Layer aflect Logit
offect
(c) Depth MATH dataset different difficulty examples: 14B
(d) Depth MQuAKE on different number of hops:
20
Depti
10
Layor Logit offect
Layor Logit ollect
(c) Depth score on MATH dataset on different difficulty examples: 32B
(f) Depth score on MQuAKE on different number of hops: 32B
Figure 36: Depth score for the Qwen series of models: the weighted average of layer index with its importance, effect on future internal computations and on the effect on future predictions. (a,c.e) MATH dataset. The x-axis is the difficulty level defined by the dataset. (b,d,f) MQuAKE. The x-axis is the number of hops the question. The findings are identical to the Llama models. See Fig. 7.
function of a given difficulty metric. Importance measured based both the
44

---

100
7.
Depth
2.8
Jper effect
(a) Depth score MATH dataset different difficulty
(b) Depth score MQuAKE on different number of hops:
examples: 7B
Depth
Logit offect
(c) Depth score on MATH dataset on different difficulty examples: 13B
(d) Depth score on MQuAKE on different number of hops: 13B
30-
Dagtt
effect nigt
(e) Depth MATH dataset different difficulty examples:
(I) Depth MQuAKE different number of hops: 32B
Figure 37: Depth score for the OLMo 2 series of models: the weighted average of layer index with its importance, as a function of a given difficulty metric. Importance measured based on both the effect on future internal computations and the effeect future predictions. (a,c.e) MATH dataset. The x-axis the difficulty level defined by the dataset. (b,d,f) MQUAKE. The x-axis is the number of hops in the question.. The findings very similar to the Llama and Qwen models. See Figs. 7, 36.
45

---

1.0
1.0
hope haze
hopp
Max www.outttteeeeoo crange
hape
Max oelitee
0.6
0.8
0.2
0.2
10
40
50
60
40
50
(a) MQUAKE: Mean max influence on future layers
(b) MQuAKE: Max output difference norm
1.0
1.0
0.81
0.8
Lovel
Level
0.8
0.6
Max roletre
Max 043261
0.2
0.2
50
50
(c) MATH: Mean influence future layers
(d) MATH: Max output difference
Figure 38: Layerwise effect of different complexity computations. (a.b) Questions with different number hops from the MQuAKE dataset. (c.d) Problems with different difficulty levels from the MATH dataset. (a,c) The max relative change in the future layer's s contribution to the answer when given layer is skipped. Mean over all future layers. (b,d) Maximum L2 norm of the change in the output probability distribution (lly- 22.2). If more complex computations use more layers, we would expect that the importance of deeper layers increases with complexity. However, we see no evidence of such pattern.
46

---

Layer silpee
Layer siipoed
change
10
10
15
Roative
15
15
0.2
25
25-
25
0.0
10
10
20
Effect or layer
Effect layer
Effect layer
Base model.
(b) Instruct model.
(c) Base model after finetuning.
2.
20
25
15
15
(d) Base model.
(c) Instruct model.
Base model after finetuning.
Figure 39: The effect of fine-tuning the max future effects of Llama 3.2 3B the DeepMind Math dataset's arithmetic splits. (a,b.c) Effect of skipping a layer on the later layers future tokens. (d.e.f) Effect future predictions. Max over 20 random examples from the validation set. The fine-tuning seems to increase the importance of the later layers at first glance. However, looking at individual examples reveals that the effect is only marginal, affecting the last 1-2 tokens before the prediction (Fig. 40).
(a) Example 1: RE
(b) Example 2: RE
(c) Example 3: RE
(d) Example 4: RE
3
(c) Example IG
Example 2:
(g) Example 3:
(h) Example IG
Figure 40: Analyzing residual erasure and integrated gradients on Llama 3.2 3B fine-tuned on the DeepMind Math dataset. Even though Fig. indicates deeper computations compared to the base model, looking at individual examples reveals that the effect concentrates only on the last 1-2 tokens before the answer, indicating that the effect is superficial.
47

---

(a) Example I: RE
(b) Example 2: RE
(c) Example 3; RE
(d) Example 4: RE
(c) Example 1: IG
(f) Example 2: IG
(g) Example 3:
(h) Example IG
Figure 41: Analyzing residual erasure and integrated gradients Llama 3.2 3B fine-tuned the DeepMind Math dataset, when trained without modeling the question. Not modeling the uncertainty in the question seems not to make any difference for fine-tuning. Compare Fig. 40.
48

---

17.5
17.5
15.0
12 15.0
-22
-10
190
10.0
00
06
04
04
25-
00
00
(a) Ex. Transformer: Q+A
(b) Ex. Transformer: only
17.5
14
17.5
125
12.5
08
10.0
00
00
06
50-
5.-
0.
02
00
0.0
(c) Ex. I: MoEUT: Q+A
(d) Ex. I: MoEUT: A only
16
16
14
1.
1.0
00
0.8
08
0.6
8-
8-
0.4
0.4
(e) Ex. 2: Transformer: Q+A Fail
(f) Ex. 2: Transformer: A only Fail
12-
12-
10
0.6
0.4
0.4
2-
02 2-
02
0.
0.0
(g) Ex. MoEUT: Q+A Fail
(h) Ex. MoEUT: only- Fail
Figure 42: Training Transformer and MoEUT models from scratch DeepMind Math dataset arithmetic subset, with and without applying loss to the question part of the input. We show the residual erasure experiments here, with identical examples the model is trained with the question modeling enabled, it does not use its 2nd half of the layers, similarly to the LLMs. (b.f) If modeling the question only, the model uses significantly more layers. (c.d.g.h) MoEUT successfully uses more layers even when modeling the question, although modeling the answer only seem help further (f). All models failed answer Example correctly (e.f.g.h). Fig. 43.
Fig. 40. (a.e) We can see that
49

---

18
19-
12
08
0.8
06
06
04
04
02
02
(a) Ex. 3: Transformer: Q+A
(b) Ex. 3: Transformer: A only
1.4
14
122
12
1.0
6-
2
02
0
0.0
(c) Ex. 3: MoEUT: Q+A
(d) Ex. MoEUT: only
12-
12-
10
10-
82
0
Ex. Transformer: Q+A
Ex. Transformer: A only
12
10
10
0
80
(g) Ex. 4: MoEUT: Q+A
(h) Ex. MoEUT: only
Figure 43: Training Transformer and MoEUT models from scratch on DeepMind Math dataset arithmetic subset, with and without applying loss to the question part of the input. We show the residual erasure experiments here, with identical examples to Fig. 40. (a,e) We see that if model trained with the question modeling enabled, it does not use 2nd half of the layers, similarly to the LLMs. (b.f) If modeling the question only, the model uses significantly more layers. (c.d.g.h) MoEUT successfully uses more layers even when modeling the question, although modeling the answer only seem to help further (f). For more examples, please refer to Fig. 42.
50


Reviews:
[{"review_id": "UDYDNaTtaH", "reviewer": "Reviewer_X1zR", "summary": "This paper investigates whether deeper layers in modern LLMs contribute to more complex computations or simply distribute the same processes across more layers. Overall, their results suggest that current deep LLMs do not use additional layers to perform fundamentally new computations. Instead, depth is primarily used for finer adjustments of representations, possibly explaining diminishing performance returns from simply increasing model depth. The authors analyze the residual stream and layer contributions, finding that layers in the second half of the network contribute much less than the first half. Additionally, skipping later layers has minimal effect on future predictions while they don’t find evidence that deeper models use additional depth to compose multi-hop reasoning or complex intermediate steps. Their findings also suggest that deeper models spread existing computations over more layers instead of introducing new types of computation.", "questions": "- The behavior of the Qwen models appears to differ somewhat from that of the Llama models (e.g., Figure 15, Figure 18e). Could the authors offer any explanations for these differences? It might also be valuable to analyze additional model families to better understand whether the observed patterns are consistent across architectures.\n\n- Are there notable differences in layer-wise behavior across different prompts, datasets, or tasks? While the paper includes some experiments on datasets like MQuAKE and MATH, several analyses rely on only 10 sampled prompts from GSM8K. It could strengthen the findings to include a broader range of prompts and datasets to assess the robustness and potential variability of the results.\n\n- In the discussion in Section 5, the authors suggest that the second half of the layers might be wasteful, as these layers primarily perform distribution matching. However, could this distribution matching still be necessary, given that the prediction distribution halfway through the network remains substantially different from that of the final layer?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The paper provides a thorough analysis of depth-wise computations in large language models, approaching the problem from several perspectives\n- The insights provided in the paper have practical implications for future research and model design.\n\nWeaknesses:\n- The experimental evaluation relies on relatively small sample sizes and limited datasets (e.g. on some experiments, 10 random examples from GSM8K), which may constrain the generalizability of the findings.\n- Similarly, the analysis is limited to only the Llama and Qwen model families, leaving open the question of whether the results generalize to other architectures.", "was_revised": true, "final_justification": "The authors have addressed some of my concerns during the rebuttal, particularly by running experiments on additional data and models. However, certain claims still seem somewhat strong, as some models exhibit differing behavior without clear supporting explanations (e.g., Qwen3-32B). Nonetheless, the overall analysis remains interesting and valuable, and I continue to recommend acceptance.", "rebuttal": "We would like to thank you for your insightful review and appreciate that you felt that the work had thorough analysis and practical implications. Unfortunately, it is not possible to update the paper in the rebuttal period, so we are constrained to answer in text. Please see our responses below.\n\n> ... evaluation relies on relatively small sample sizes and limited datasets ...\n\n> ... It could strengthen the findings to include a broader range of prompts and datasets\n\nWhile it is true that the main experiments are conducted on GSM8K, our other experiments involve simple arithmetics, multi-hop questions, and DeepMind Math, yielding consistent findings. We chose GSM8K based on the findings of Sun et al. (2024), \"Transformer Layers as Painters,\" which showed that GSM8K is the most sensitive to layer interventions among all tasks they tested.\n\nIn the final version of the paper, we will include experiments in the appendix showing an equivalent of Fig 2., Fig. 3 and Fig. 4 for:\n1. 50 examples: This will show that 10 examples is sufficient to see the general trend.\n2. 10 examples from the MATH dataset: This will show robustness to the dataset in the math domain.\n\nWe have already conducted these future effect experiments during the rebuttal period, and we can confirm that they are consistent with the paper's findings. Unfortunately, due to the new NeurIPS rules, we don't see a way to upload these results.\n\n> ... the analysis is limited to only the Llama and Qwen model families ...\n\nWe agree with the reviewer that having more results is always better. We would like to note that we already struggle with presenting all the model variants we tested in a comprehensive manner. We reported 3 sizes of both model families. Additionally, we also report results on the instruction-tuned Llama 70B (Fig. 19). \n\nNevertheless, in the final version of the paper, we will add an analysis of OLMo 2. The differences from the current findings are minimal. OLMo 2 7B shows an interesting pattern, where a small \"triangle\" appears in the similarity maps, indicating that certain layers are composed with each other but collectively have no effect on the later computations. The lack of influence of the late layers remains evident in this model. OLMo2 13B is basically indistinguishable from Qwen 3 8B (Fig. 18. a). OLMo 2 32B is similar to Qwen 3 32B, with slightly better layer usage.\n\n> The behavior of the Qwen models appears to differ somewhat from that of the Llama models (e.g., Figure 15, Figure 18e). Could the authors offer any explanations for these differences?...\n\nWe believe that the model's behavior is not only dependent on the model architecture, but also on the details of the training setup, including initialization and optimization setup. These training details are usually not public. Understanding what causes such behavior requires an extensive study, and we expect that fully understanding the effects could take multiple papers.\n\nOut of the models we tested, Qwen 32B seems to be somewhat different from the others: instead of not using the late layers for features useful for future predictions, it doesn't use the early layers. This also shows that it is possible to have a well-performing model without most of the late layers being dominated by the loss-refining behavior. Based on Fig. 22i, we can see that the logit refinement behavior is more compressed, but consistent with the previous findings: the predictions start to improve when feature building stops (~layers >50).\n\n> Are there notable differences in layer-wise behavior across different prompts\n\nYes. The individual examples (prompts) exhibit different behaviors, but the general trends remain consistent. For example, Fig. 6 shows some of the variations in the processing behavior between two prompts. This is why we take the maximum over multiple different prompts. In the final version of the paper, we will include a few individual examples in the appendix as illustrations.\n\n\n> In the discussion in Section 5, the authors suggest that the second half of the layers might be wasteful, as these layers primarily perform distribution matching. However, could this distribution matching still be necessary, given that the prediction distribution halfway through the network remains substantially different from that of the final layer?\n\nWe agree with the reviewer that such refinement might be necessary. However, we are unsure about the number of layers necessary for this. For example, Figs. 18e and 22i,j show that Qwen 3 32B is able to perform refinement in a very few layers compared to other models (this model suffers from a different issue: not utilizing its early layers for feature building).\n\nIn general, this refinement behavior may be a consequence of the next-word prediction objective. However, we believe that architectural changes might influence the extent to which it dominates the network. It seems reasonable to assume that if this “distribution refinement” could be accomplished in fewer layers, and more layers could be devoted to computing useful features, the models could become more powerful predictors. Showing whether this computation is necessary or useful is an interesting future research direction.\n\n\nWe tried our best to resolve the concerns that the reviewer has raised. If the reviewer finds our response useful, please consider increasing the score. Thank you very much."}, {"review_id": "w7y4iElzoJ", "reviewer": "Reviewer_Tdns", "summary": "The paper asks whether modern pre-layernorm Transformer language models (Llama-3.1 and Qwen-3) truly exploit additional depth for new computations or simply spread the same work across more layers. Using a pack of residual-stream analyses and causal interventions—including layer-wise contribution analysis, causal layer-skipping, LogitLens & prediction-overlap, complexity probes with a depth-score metric, cross-model linear mapping, and an alternative-architecture (MoEUT) check—the authors uncover a sharp phase split: the first half of the network drives most residual change, while the second half mainly fine-tunes the current-token logits. These findings imply that stacked-Transformer LLMs under-utilize their later layers, devoting them to incremental logit sharpening rather than hierarchical composition, which helps explain diminishing returns from depth scaling and motivates architectures or objectives that make deeper networks more effective.", "questions": "Covered in the Strengths and Weaknesses section.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- Insightful multi-angle analysis: The paper fuses a diverse set of diagnostics. Since every tool points to the same phase-split conclusion, the evidence base is persuasive.\n- Practical relevance for depth scaling: By demonstrating that later layers in very deep Transformers mostly perform incremental logit-sharpening, the study offers a explanation for the diminishing returns observed in depth-scaling curves. This insight can inform both model-design choices and future research on more depth-efficient objectives or architectures.\n\n### Weaknesses\n- Connection to the latest literature is thin: Several closely related lines of work are not discussed. Incorporating results from Mix-LN [1], Peri-LN [2], selective attention pruning [3], and Massive Activations [4] would situate the contribution more firmly within the current discourse and might even sharpen the interpretation of the phase-split finding.\n\n- Architectural bias toward pre-LN models: All experiments use pre-layer-norm Transformers; it therefore remains unclear whether the conclusions extend to post-LN or other variants such as the OLMo architecture [5] or [2]. A small-scale replication on at least one non-pre-LN baseline, or a principled argument for transferability, would substantially strengthen the claim of generality.\n\n- Interpretability of massive activations remains unexplored: Since residual-path signals can accumulate large magnitudes (as documented by Sun et al. [4]), tying the observed logit-sharpening behavior to activation growth patterns could provide a more mechanistic explanation of how and why later layers lose representational diversity.\n\n- Clarity of key visualizations: Several figures—most notably the depth-score heat-map—lack fully labeled axes, color bars, or legends, forcing readers to consult the appendix for basic interpretive cues. Self-contained captions would improve accessibility.\n\n- Limited evidence for alternative architectures: The MoEUT comparison relies on only one or two checkpoints. Extending the study to a broader range of shared-parameter or other architectures would give greater empirical weight to the “look beyond pure stacking” recommendation.\n\n- Root cause of logit-smoothing remains open: The paper compellingly documents what happens in the back half of the network but stops short of explaining why. Some competing hypotheses are neither disentangled nor tested experimentally such as  (1) the next-token objective provides diminishing gradient signal at depth, and (2) architectural bottlenecks curb the formation of new representations \n\n[1] Li, Pengxiang, Lu Yin, and Shiwei Liu. \"Mix-ln: Unleashing the power of deeper layers by combining pre-ln and post-ln.\" ICLR2025.  \n[2] Kim, Jeonghoon, et al. \"Peri-LN: Revisiting Normalization Layer in the Transformer Architecture.\" Forty-second International Conference on Machine Learning. 2025.\n[3] He, Shwai, et al. \"What matters in transformers? not all attention is needed.\" arXiv preprint arXiv:2406.15786 (2024).  \n[4] Sun, Mingjie, et al. \"Massive activations in large language models.\" COLM2024.  \n[5] OLMo, Team, et al. \"2 OLMo 2 Furious.\" arXiv preprint arXiv:2501.00656 (2024).", "was_revised": true, "final_justification": "During the rebuttal the authors promised to add discussion of Peri-LN and other related work, and demonstrated that OLMo-2 (Post-LN-like) models show the same “late-layer under-use,” supporting the result’s generality. They also committed to fully labeled axes and legends, fixing readability concerns. Open questions, such as the precise cause of late-layer attenuation (residual growth vs. gradient decay) and the limited exploration of alternative architectures like MoEUT, remain for future work, yet they do not undermine the paper’s central empirical message.", "rebuttal": "We would like to thank the reviewer for the insightful review and appreciating the papers insightful analysis and practical relevance. Unfortunately, it is not possible to update the paper in the rebuttal period, so we are constrained to answer in text. Please see our responses below.\n\n> Incorporating results from ... would situate the contribution more firmly within the current discourse and might even sharpen the interpretation of the phase-split finding.\n\nWe will add the discussion of these papers to the related work section and relate them to our findings. For example, Peri-LN [2] shows that their method has increased angular distance between the representations of different layers, indicating that it might use deeper layers more efficiently. However, if the reviewer is suggesting measuring the effects that such changes have on the model's behavior and our analysis, we would like to note that our study primarily focuses on pre-trained models, and we believe that a reliable signal about multi-step processing and reasoning can only be obtained from well-trained models. Thus, such a study needs to be conducted by one of the big industry labs, which can afford to systematically train a series of models with these modifications.\n\n> Architectural bias toward pre-LN models: All experiments use pre-layer-norm Transformers; it therefore remains unclear whether the conclusions extend to post-LN or other variants such as the OLMo architecture [5] or [2]. A small-scale replication on at least one non-pre-LN baseline, or a principled argument for transferability, would substantially strengthen the claim of generality.\n\nWe mainly analyze pre-LN models (x + f(LN(x))), because the majority of freely available models are pre-LN. We agree that the PostLN from the original Vaswani et al. (2017) paper, in the form of LN(x+f(x)) might have an impact on our findings and improve the feature building stages of the model. However, we are not aware of any successful LLMs using the PostLN layout. OLMo is x + LN(f(x)), which looks closer to PreLN than to PostLN; thus, the similarity in behavior is expected. \n\nIn the final version of the paper, we will add an analysis of the OLMo 2 models. We ran these experiments during the rebuttal. The differences from the current findings are minimal. OLMo 2 7B shows an interesting pattern, where a small \"triangle\" appears in the similarity maps, indicating that certain layers are composed with each other but collectively have no effect on the later computations. The lack of influence of the late layers remains evident in this model. OLMo2 13B is basically indistinguishable from Qwen 3 8B (Fig. 18. a). OLMo 2 32B is similar to Qwen 3 32B, with slightly better layer usage.\n\nUnfortunately, we do not see a way to report our plots in the rebuttal without violating the rebuttal rules. If the reviewer has a suggestion, please let us know.\n\n> Interpretability of massive activations remains unexplored: Since residual-path signals can accumulate large magnitudes (as documented by Sun et al. [4]), tying the observed logit-sharpening behavior to activation growth patterns could provide a more mechanistic explanation of how and why later layers lose representational diversity.\n\nWe agree with the reviewer that one possibility for why the later layers have less effect on future computations might be that the residuals are growing. But there are alternative explanations. For example, since the gradients of the later layers might be directly dominated by the classifier layer’s gradient distributed through he residual, instead of the higher-order effects through composing layers. Exploring the detailed mechanism behind these effects is an important future direction and is probably worth its own study.\n\n> Clarity of key visualizations: Several figures—most notably the depth-score heat-map—lack fully labeled axes, color bars, or legends, forcing readers to consult the appendix for basic interpretive cues. Self-contained captions would improve accessibility.\n\nWe will improve this for the final version of the paper.\n\n> Limited evidence for alternative architectures: The MoEUT comparison relies on only one or two checkpoints. Extending the study to a broader range of shared-parameter or other architectures would give greater empirical weight to the “look beyond pure stacking” recommendation.\n\nMoEUT served only as an outlook for possible future architectures. A more significant issue with the MoEUT experiments is that we lack the compute resources to fully train a base MoEUT model on a scale comparable to the Llama and Qwen models; thus, we cannot be certain of the reasons for the differences. We are not aware of any shared-layer pretrained model, but in the case such a model exists, we would be very interested in analyzing it.\n\n> Root cause of logit-smoothing remains open: The paper compellingly documents what happens in the back half of the network but stops short of explaining why. Some competing hypotheses are neither disentangled nor tested experimentally such as (1) the next-token objective provides diminishing gradient signal at depth, and (2) architectural bottlenecks curb the formation of new representations\n\nOur paper aims to highlight that current LLMs often employ a fixed number of layers, regardless of problem complexity, and a significant portion of these layers fail to build reusable features. We analyze this effect in multiple ways throughout the paper (future effect plots, integrated gradients, residual erasure, depth score, etc). This analysis already barely fits in a single paper. While we agree with the reviewer that it is an important direction to determine the mechanistic reason behind the observed behaviors, we think that expecting to solve such a complex problem in a single paper is unreasonable.\n\nWe tried our best to resolve the concerns that the reviewer has raised. If the reviewer finds our response useful, please consider increasing the score. Thank you very much."}, {"review_id": "ZHhWYlO23g", "reviewer": "Reviewer_WGWG", "summary": "This paper investigates the relevance of different layers in a large language model when prompted to solve mathematical and more general reasoning tasks. Through a sequence of analyses involving the average norms and cosine similarities of terms which define the residual stream, causal interventions, linear mappings between residual streams of different models, the authors draw the main conclusion that the second half of the layers in a language model is used to make local changes to the residual stream required for accurately predicting the current token, with this half of layers having little influence on the prediction of future tokens. An analysis of the MoEUT architecture shows that universal transformers may not suffer from the same limitations.", "questions": "**Questions**\n\n- While it was my impression that the observation of \"the second half of the layers refining the current token's probability distribution\" is framed somewhat negatively, could this not also be a positive property of non-universal transformers?\n- One sentence in the abstract poses the question of whether transformers *create higher-order computations that are **impossible** in shallower models*. Do you have any formal results confirming that certain computations that you investigate are indeed **impossible** in shallower models?\n- I must ask the authors what novel observations their analysis brings? I am aware that, even if not every observation is entirely novel, the analysis is valuable due to its scope and thoroughness, hence I am not interested in rejecting on the trivial ground of novelty. I am simply wondering which observation is entirely novel, given certain overlap of this work with at least that of *Lad et al., 2024*.\n\n\nI do not doubt that we can lead a meaningful discussion after which I will consider increasing my score. I would in particular appreciate a discussion on the originality of your approach, seeing as it's one of the grading criteria.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The paper presents a sequence of interesting observations which all point in coherence to the main conclusions, that (a) non-uniform transformers tend not to utilize the second half of the layers when predicting future tokens and that (b) the second half of the layers tends to be utilized to make smaller refinements of the residual stream necessary for predicting the current token.\n- One must commend the sheer amount of experimental evidence the authors provide, all of which cohesively (although perhaps not unshakeably conclusively) points to the final drawn conclusions. This also causes me to believe that the paper is entirely technically solid. I find  zero reason to doubt any of the presented evidence.\n- Observations made in this paper can an impact on future research on architectures, training objectives, and optimizers. I personally believe that the observations made in this work are valuable.\n- The linear mapping of the residual streams from all of the layers of one model to all of the layers of another model is, i find, a very interesting experiment.\n\n**Weaknesses**\n\n- Some of the evidence presented is not entirely convincing. For example, while Figure 8 does show a diagonal pattern, I would be hesitant to draw too strong of a conclusion from it. I find that it is not entirely clear what it means to \"stretch out\" the same computation over several layers. Of course, I acknowledge the difficulty of providing exact explanations of the internal computations of large language models, and the sheer amount of partial evidence provided in this work does point to the conclusions drawn by the authors. I would just be cautious about stating that deeper models \"stretch out\" the same computation that shallower models perform over several layers, especially considering more formal results which provide results on the transformer depth necessary to perform certain computations (See e.g. the logarithmic dependence in Liu et al. 2023). Another claim that I find somewhat unconvincing is the statement that, paraphrasing lines 248-250, *The fact that MoEUT tends to make use of more depth in its computation than a non-universal transformer **confirms the advantage of shared-layer models.*** I do not believe that this fact really confirms anything.\n- I believe that the text could be reformatted so that fewer references to sections and figures in the appendix are necessary. For example, in section 3.5., the four examples are not at all explained in the main text, being completely relegated to the appendix. I understand that, again, given the sheer amount of material the paper presents, it is difficult to adhere to the 9-page limit, but I still believe that there is room for improvement in this regard.\n- I believe that a better overview and explanation of the MoEUT results would have been in order. It is rather unsurprising that a universal transformer utilizes its layers in a drastically different manner than a non-uniform transformer does. It would have been interesting if this higher layer utilization could be linked to higher extrapolation performance in a more systematic manner than just listing the one example of Mul/Div Multiple Longer.\n- The claimed \"refinement of the current token's probability distribution\" sounds like a very interesting phenomenon, and I believe that it could have been addressed in more detail. But again I must acknowledge that one paper cannot do it all, so this weakness is rather to be taken as my opinion on what the ideal paper would do. \n- As essentially every experimental analysis, this one could also benefit from a wider set of benchmarks. Additionally, I have to question its relevance, given that nowadays chain-of-thought based approaches seem to be the standard for the type of task the authors investigate.", "was_revised": true, "final_justification": "I would in general appreciate more rigour in the paper, e.g. refraining from calling a computation \"impossible\" without a formal proof that it is in fact impossible. The same comment holds for the rebuttal. The authors mention that it is always possible to learn an N-way operation that solves any operation in few layers, but there is no mention of any formal guarantee confirming this statement.\n\nThat being said, I again state that I believe the analysis is interesting and could be impactful, and the authors could clarify certain important aspects of their work in the rebuttal. The observations, made on a range of LLMs and complex tasks. consistently point to the same conclusion, one which is currently not well-known in LLM literature.", "rebuttal": "We would like to thank the reviewer for the insightful review and for acknowledging the diversity of our experiments that point to the same conclusion and their relevance for future research. Unfortunately, it is not possible to update the paper in the rebuttal period, so we are constrained to answer in text. Please see our responses below.\n\n> For example, while Figure 8 does show a diagonal pattern, I would be hesitant to draw too strong of a conclusion from it. I find that it is not entirely clear what it means to \"stretch out\" the same computation over several layers. \n\nWe meant by “stretch out” that representations after the layers at the same relative position in the network map to each other the best. This means that similar computation is performed by both the deep and shallow models until the same percentile of the layers. An alternative would be to perform a chunk of computation corresponding to higher-level features that are impossible in shallow models. In such a case, these layers should exhibit high loss compared to any layer in the shallow model. We saw no evidence of such behavior.\n\n> I would just be cautious about stating that deeper models \"stretch out\" the same computation that shallower models perform over several layers, especially considering more formal results which provide results on the transformer depth necessary to perform certain computations (See e.g. the logarithmic dependence in Liu et al. 2023).\n\nWe agree that certain computations require deeper models. However, they can often be “solved” on a limited range of data using shortcuts that do not generalize (see e.g. Table 4 in the appendix of Csordás et al. (2021): The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization). In LLMs, this is particularly challenging to detect due to the vast amount of training data. This was the primary motivation of our study: can we find evidence that the model adjusts its depth of computation based on problem complexity or operation ordering, or similar factors? Such a behavior would indicate decomposition and building on subresults. Unfortunately, we find no evidence for this.\n\n> The fact that MoEUT tends to make use of more depth in its computation than a non-universal transformer confirms the advantage of shared-layer models. I do not believe that this fact really confirms anything.\n\nWe will change the wording of this sentence: MoEUT tends to utilize more depth in its computation than a non-universal transformer suggests that shared-layer models might have an advantage in this regard.\n\nThe reason to believe that shared layers are better is that the knowledge can be easily transferred and reused in such models. Thus, it could be easier for the models to reuse already existing computations in the later layers, even if the gradient is dominated by the classification layer.\n\n\n> I believe that the text could be reformatted so that fewer references to sections and figures in the appendix are necessary. \n\nWe will try our best to improve the readability of the final version of the paper.\n\n> I believe that a better overview and explanation of the MoEUT results would have been in order. \n\nWe agree with the reviewer that more exploration of MoEUT would be helpful. However, for a fair comparison to MoEUTs, we would need a model trained on a similar scale and token budget as the LLMs we analyzed. Unfortunately, such a model doesn’t exist, and we do not have the budget to train it. Our MoEUT results aim to provide a preliminary study on a toy task, comparing it to a toy Transformer model trained on the same task.\n\n> The claimed \"refinement of the current token's probability distribution\" sounds like a very interesting phenomenon, and I believe that it could have been addressed in more detail.\n\nWe agree with the reviewer that the details of this process are important to understand. However, we believe this is a research project in its own right, and we have left it for future work.\n\n> As essentially every experimental analysis, this one could also benefit from a wider set of benchmarks.\n\nWe agree with the reviewer that more benchmarks are always beneficial, but we would also like to note that our paper offers a wide range of methods to test various aspects of the same phenomenon (future effect plots, integrated gradients, residual erasure). In terms of datasets, during the rebuttal period, we measured the future effects (Fig 3b.) on the Math dataset as well, and we obtained very similar results to the GSM8K in the paper. We will report this in the final version of our paper. Note that the toy datasets reported throughout various parts of the paper (arithmetic, multi-hop QA, DeepMind math) all exhibit a consistent behavior.\n\n> Additionally, I have to question its relevance, given that nowadays chain-of-thought based approaches seem to be the standard for the type of task the authors investigate.\n\nChain-of-thought helps with the step-by-step computation and decomposition. We think this is the main reason behind its success. However, CoT has its limitations. For example, there is no guarantee that the CoT trace contains the ideal decomposition guaranteeing that the model can use fixed depth. Also, during pretraining, there is no CoT available, forcing the model to learn fixed-depth computation for all problems it sees. This can result in shortcut behaviors that must be unlearned during the post-training phase. It is unclear if this happens successfully. We believe an ideal model should be able to perform CoT, while also performing dynamic latent computation within its layers.\n\n> … \"the second half of the layers refining the current token's probability distribution\" is framed somewhat negatively, could this not also be a positive property of non-universal transformers?\n\nWe hypothesize that computing composite features for future predictions seems more useful than dedicating half of the layers to compute updates that are only useful for a single token. Additionally, inference typically uses nucleus sampling, discarding the tail of the probability distribution. Given that the tail is probably the most difficult to predict, it likely requires a significant amount of model capacity. It seems reasonable to assume that if this “distribution refinement” could be accomplished in fewer layers, and more layers could be devoted to computing useful features, the models could become more powerful predictors. However, we agree with the reviewer that this is not a proven idea, and we acknowledge that the distribution refinement behavior might be beneficial in some way for the model. This is another interesting future research direction.\n\n> Do you have any formal results confirming that certain computations that you investigate are indeed impossible in shallower models?\n\nNo, we do not have such results at present. However, it seems reasonable to assume that an N-step arithmetic operation requires at least N layers to learn a faithful elementwise computation that can generalize. Note that it is always possible to learn an N-way operation that solves any operation in a few layers, but such a solution is unlikely to generalize to slightly different scenarios. Finding evidence that distinguishes between these two scenarios is one of the motivations behind our study.\n\nAdditionally, note that previous work, such as Sun et al. (2024), \"Transformer Layers as Painters,\" demonstrated that, unlike many others, math-related tasks are sensitive to layer interventions across all datasets they tested. This is the reason why we based our investigation primarily on GSM8K.\n\nIf the reviewer is aware of a task that has been formally proven to require deep models and is zero-shot doable by LLMs, we would be excited to test it.\n\n> I must ask the authors what novel observations their analysis brings? … I am simply wondering which observation is entirely novel, given certain overlap of this work with at least that of Lad et al., 2024.\n\nOur analysis has multiple novel factors. For example, we are not aware of any previous methods that are able to localize interactions between the layers directly, such as our interventions presented in Fig 3 and Fig 5. Also, we are not aware of any paper that studies whether the models vary the depth of their computation based on problem difficulty (Fig 7) or position in the computation graph (Fig 6, Fig 9, etc). Regarding the attention contribution, while the newest version of Lad et al, 2024 contains a more detailed analysis on attention (Fig. 6a), this was published in an update to the submission that happened after the NeurIPS deadline. Additionally, their attention analysis measures the attention value placed on the previous few tokens and does not directly assess the cross-position effect of different layers. We also show a connection between the diminishing importance of the later computation and the refinement behavior of the layers. Additionally, we are not aware of any previous work that tried to find a correspondence between layers in different depth networks. \n\nOverall, the goal of our paper is to provide a novel perspective, showing that LLMs do not appear to perform dynamic, problem-dependent decomposition in their latent computation. We are not aware of a study focusing on this problem.\n\nWe tried our best to resolve the concerns that the reviewer has raised. If the reviewer finds our response useful, please consider increasing the score. Thank you very much."}, {"review_id": "fsI7V8W8lm", "reviewer": "Reviewer_y38W", "summary": "This paper investigates the depth efficiency of modern large language models through the lens of mechanistic interpretability. By inspecting the residual stream representations of LLMs through logit lens and by measuring their causal contributions to model outputs via causal intervention, they found that layers in the second half contribute much less than those in the first half. Moreover, for semantically more complex problems, LLMs fail to utilize more layers to solve them, suggesting a lack of compositionality in model reasoning. Another probing analysis suggests that deeper layers are not performing \"new\" operations compared to early layers. Together, these empirical results suggest that LLMs are not using deeper layers efficiently, which may also explains the diminishing return of scaling up transformer by adding more layers.", "questions": "See the weaknesses section.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The relation between neural network model depth and performance is an important research problem, especially in the era of LLMs where scaling up model size often lead to improved general performance. \n\n* The paper applied various mechanistic analysis tools to study LLM residual streams, and the experiments yield converging evidence that LLMs are not leveraging its deeper (especially second-half) layers efficiently.\n\n* The paper presents an inspiring discussion section by drawing relations between their interpretability findings and LLM reasoning capabilities using Chain-of-Thought and Latent Thinking.  \n\nWeaknesses: \n* While the analysis results presented in the paper are interesting, their empirical implications still seem vague to me. In particular, I found the argument that \"the second half of the transformer models seem wasteful\" a bit unsupported. According to Fig.4, without the second half layers, the model would yield an output distribution whose top-5 most likely token is very different from those predicted by the full model -- this should be considered as an essential step in computing the output probability distribution, as opposed to an incremental \"refinement\" step. \n\n* I'm not fully understanding the logical relation between the claimed LLM depth inefficiency and the potential ineffectiveness of latent thinking methods. Even if non-recurrent transformers are inefficient in using deeper layers, a recurrent one pretrained from scratch may still evolve in a different way. The analyses in this paper cannot offer insights in answering this question.", "was_revised": true, "final_justification": "The authors partially addressed my concerns during rebuttal, so I raised the significance score while keeping the others unchanged.", "rebuttal": "We would like to thank the reviewer for the insightful review and for finding our discussion section inspiring. Unfortunately, it is not possible to update the paper in the rebuttal period, so we are constrained to answer in text. Please see our responses below.\n\n> … I found the argument that \"the second half of the transformer models seem wasteful\" a bit unsupported. …\n\nWe hypothesize that computing composite features for future predictions seems more useful than dedicating half of the layers to compute updates that are only useful for a single token. Additionally, inference typically uses nucleus sampling, discarding the tail of the probability distribution. Given that the tail is probably the most difficult to predict, it likely requires a significant amount of model capacity. It seems reasonable to assume that if this “distribution refinement” could be accomplished in fewer layers, and more layers could be devoted to computing useful features, the models could become more powerful predictors. Showing whether this computation is necessary or useful is an interesting future research direction.\n\n> I'm not fully understanding the logical relation between the claimed LLM depth inefficiency and the potential ineffectiveness of latent thinking methods. Even if non-recurrent transformers are inefficient in using deeper layers, a recurrent one pretrained from scratch may still evolve in a different way.\n\nWe agree with the reviewer that our paper does not prove that latent thinking models are ineffective. In fact, in the paper we write: “If the insensitivity of computation depth to the input\ncomplexity is the consequence of the pertaining objective, such methods are fundamentally flawed. On the other hand, if the reason is the architecture, these approaches might provide the solution.” Unfortunately, we are unaware of large-scale pretrained recurrent depth models that we can test; thus, we did what we considered to be the next best experiment: a controlled study of MoEUT vs Transformers on toy tasks.\n\nWe would also like to note that while not finding dynamic computation depth and decomposition in fixed-depth models does not invalidate recurrent thinking models, if we had found evidence that even fixed-depth non-recurrent models break down the problem dynamically in their layers, that would be a strong support for the latent reasoning models. Unfortunately, it seems that this is not the case.\n\nWe tried our best to resolve the concerns that the reviewer has raised. If the reviewer finds our response useful, please consider increasing the score. Thank you very much."}, {"review_id": "7qss7aGr3V", "reviewer": "Reviewer_TjFH", "summary": "This paper investigates whether Transformers use depth efficiently, i.e., whether larger depth is utilized for higher-order computation or simply spreading computation. The authors analyze the residual stream in the LLaMA 3.1 model family. Through a series of empirical studies, they conclude that increased depth does not result in new computational capabilities but instead improves residual adjustments.", "questions": "1. Can the authors provide further justification or theoretical grounding for the layer contribution metrics used in the analysis?\n2. Do the findings still hold for problems that cannot be solved by shallow models and inherently require deeper computation? For example, is a 10-layer model only able to solve problems whose solutions can be represented by models with effective depth no greater than, say, 5 layers? Or would it learn to utilize depth more effectively in this case?\n3. Prior work [1] [2] suggests that CoT prompting improves reasoning by increasing the effective computational depth of LLMs. Do the conclusions of this paper contradict such findings? Or do they suggest that CoT compensates for poor depth utilization in standard forward passes?\n4. Is the inefficiency in depth usage due to architectural limitations of Transformers (i.e., expressive capabilities) or to properties of the training process (e.g., optimization bias or data-induced inductive bias)?\n\n[1] Feng et al. Towards revealing the mystery behind chain of thought: a theoretical perspective. NeurIPS 2023.\n\n\n[2] Merrill et al. \"The expressive power of transformers with chain of thought.\" ICLR 2024.", "limitations": "See Weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The paper tackles a fundamental and timely research question: how effectively Transformers leverage their depth. This has direct implications for architecture design, interpretability, and scaling strategies.\n2. The experiments are extensive. Multiple analysis techniques are employed to examine the effect of depth, and the results are presented clearly.\n\n**Weaknesses**\n1. The layer contribution metrics used in the analysis are introduced based on intuition, but their theoretical justification is limited. It would strengthen the paper to explain why these metrics are valid or whether they align with any existing formal frameworks.\n2. The connection between depth utilization and task complexity is underexplored. The paper primarily evaluates tasks that could plausibly be solved by relatively shallow models. It remains unclear whether the conclusions would hold for tasks that inherently require deeper computation.", "was_revised": true, "final_justification": "The paper reveals an interesting phenomenon regarding the depth usage of Transformers. The authors have addressed most of my concerns in the rebuttal. However, I remain unconvinced that the current evidence fully supports the conclusions, and I believe the paper would benefit from stronger empirical or theoretical support. Taking all these factors into account, I decide to set my score to 4.", "rebuttal": "We would like to thank the reviewer for the insightful review and for acknowledging the timeliness of our paper and the extensiveness of the experiments. Unfortunately, it is not possible to update the paper in the rebuttal period, so we are constrained to answer in text. Please see our responses below.\n\n> The layer contribution metrics used in the analysis are introduced based on intuition, but their theoretical justification is limited\n> Can the authors provide further justification or theoretical grounding for the layer contribution metrics used in the analysis?\n\nWe are not sure what a satisfactory theoretical justification would look like in the eyes of the reviewer.\n\nOur motivation was the following: the layers in a pre-layernorm Transformer modify the residual by adding their output to it. If this output changes, it means that the layer’s computation changed. If this happens because of skipping a previous layer, it means that the output produced by the previous layer was used as an input for this layer, because the computation relied on it.\n\n> The connection between depth utilization and task complexity is underexplored. The paper primarily evaluates tasks that could plausibly be solved by relatively shallow models. It remains unclear whether the conclusions would hold for tasks that inherently require deeper computation.\n\nWe chose the math domain because that is one of the tasks that we expect to require significant depth. Previous work, such as Sun et al. (2024): Transformer Layers as Painters, showed that GSM8K is the most sensitive to layer interventions among all tasks they tested. Could the reviewer provide a concrete task that is solvable with pretrained LLMs out of the box, and requires more depth than math? We are eager to explore such cases.\n\n> Do the findings still hold for problems that cannot be solved by shallow models and inherently require deeper computation? For example, is a 10-layer model only able to solve problems whose solutions can be represented by models with effective depth no greater than, say, 5 layers? Or would it learn to utilize depth more effectively in this case?\n\nThis is an excellent question. We suspect that if we train the models on a synthetic task requiring 10 layers without modeling the input question, the model will utilize all 10 available layers. In fact, previous work, such as Csordás et al. (2021), \"The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization,\" has already shown this. However, in Tab 4. Appendix, the authors also showed that it is possible to learn the same task with a much shallower model. This model relies on shortcuts, yet achieves 100% IID performance, albeit with low OOD generalization performance. In toy settings, it is possible to disentangle these two cases with carefully designed synthetic data, but it is not generally possible to do so for pre-trained LLMs, which are trained on most of the Internet. This was the primary motivation of our study: can we find evidence that LLMs break down problems into subproblems in a manner consistent with what we would expect from a generalizing solution? Unfortunately, we saw no evidence for this.\n\nOur hypothesis is that if we were to take the same task and combine it with language modeling, the model would likely use fewer layers for the task and dedicate the final layers to probability refinement. Similar effects in our preliminary study can be seen by comparing the Q and Q+A versions of the identical models in Fig 31 and Fig 32 in the Appendix: forcing the model to learn the question distribution seems to force the feature building stage to end earlier. Studying this systematically would be an interesting follow-up work.\n\n> Prior work [1] [2] suggests that CoT prompting improves reasoning by increasing the effective computational depth of LLMs. Do the conclusions of this paper contradict such findings? Or do they suggest that CoT compensates for poor depth utilization in standard forward passes?\n\nOur view is that CoT compensates for the poor depth utilization and lack of decomposition, which are among the main reasons for their success. However, CoT has its limitations. For example, there is no guarantee that the CoT trace contains the ideal decomposition, which guarantees that the model can use a fixed depth. Also, during pretraining, there is no CoT available, forcing the model to learn fixed-depth computation for all problems it sees. This can result in shortcut behaviors that must be unlearned during the post-training phase. It is unclear if this happens successfully. We believe an ideal model should be able to perform CoT, while also performing dynamic latent computation within its layers.\n\n> Is the inefficiency in depth usage due to architectural limitations of Transformers (i.e., expressive capabilities) or to properties of the training process (e.g., optimization bias or data-induced inductive bias)?\n\nThis is an excellent question. In fact, we ask the same question while discussing the latent thinking methods in L306-308: “If the insensitivity of computation depth to input complexity is the consequence of the pretaining objective, such methods are fundamentally flawed. On the other hand, if the reason is the architecture, these approaches might provide the solution.“\n\nWe hypothesize that it is at least partially caused by the next-token prediction loss, especially on the “impredictable” parts of the data that are not the result of a systematic computation, but should produce a complicated distribution of next tokens. Even if it is caused by the training process, it may be possible to construct architectures that behave more efficiently.\n\nWe tried our best to resolve the concerns that the reviewer has raised. If the reviewer finds our response useful, please consider increasing the score. Thank you very much."}]

Author rebuttal:

Meta-review: This paper presents a comprehensive and insightful investigation into how large language models utilize their depth, a timely and significant question for the field. The reviewers reached a strong consensus on the paper's primary strengths, consistently praising the thorough, multi-faceted analysis that cohesively points to the conclusion that deeper models, particularly in their later layers, are underutilized for complex, compositional computations and instead focus on refining the current token's probability distribution. This central finding, supported by a diverse set of experiments including causal interventions and cross-model analysis, was potentially considered to have valuable practical implications for future research.

The review process also highlighted several key limitations. A recurring concern was the generalizability of the findings beyond the Llama and Qwen families and on a wider range of tasks and larger data samples. Reviewers also noted that while the paper compellingly documents the phenomenon of late-layer underutilization, it stops short of providing a deep mechanistic explanation for its root cause. The authors have constructively engaged with this feedback and, in their rebuttal, committed to addressing several of these points in the final version by including experiments on the OLMo model family, expanding their analysis to the MATH dataset, and improving the clarity of key visualizations. While open questions remain for future work, the paper's core empirical contributions are solid and offer a novel, valuable perspective on the functional dynamics of model depth.
Decision: Accept (poster)