Remarkable Robustness of LLMs: Stages of Inference?
Authors: Vedang Lad
Jin Hwa Lee
Wes Gurnee
Max Tegmark
Venue: NeurIPS 2025 poster ()
Keywords: Mechanistic Interpretability
Deep Learning
Large Language Models
Transformer Robustness
Layer Interventions
Inference Stages
Neural Network Analysis
Attention Mechanisms
Model Architecture
Primary area: deep_learning
TL;DR: A synthesis of layer-wise interventions, empirical experiments, and previous research suggests that inference in decoder-only LLMs unfolds in distinct phases.

Abstract: We investigate the robustness of Large Language Models (LLMs) to structural interventions by deleting and swapping adjacent layers during inference. Surprisingly, models retain 72–95\% of their original top-1 prediction accuracy without any fine-tuning. We find that performance degradation is not uniform across layers: interventions to the early and final layers cause the most degradation, while the model is remarkably robust to dropping middle layers. This pattern of localized sensitivity motivates our hypothesis of four stages of inference, observed across diverse model families and sizes: (1) detokenization, where local context is integrated to lift raw token embeddings into higher-level representations; (2) feature engineering, where task- and entity-specific features are iteratively refined; (3) prediction ensembling, where hidden states are aggregated into plausible next-token predictions; and (4) residual calibration, where irrelevant features are suppressed to finalize the output distribution. Synthesizing behavioral and mechanistic evidence, we provide a hypothesis for interpreting depth-dependent computations in LLMs.

The Remarkable Robustness of LLMs: Stages of Inference?
Vedang Lad" MIT Stanford University
Jin Hwa Lee
University College London jin 1ee. 2ewe222222cccc ac. uk
vedanglad@staanf ord.
Wes Gurnee MIT wesg@mit. edu
Max Tegmark
MIT tegmarkdmit.co edu
Abstract
We investigate the robustness of Large Language Models (LLMs) to structural interventions by deleting and swapping adjacent layers during inference. Surpris- ingly, models retain 72-95% of their original top-1 prediction accuracy without any fine-tuning. We find that performance degradation not uniform across layers: in- terventions to the early and final layers cause the most degradation, while the model is remarkably robust dropping middle layers. This pattern of localized sensitivity motivates our hypothesis of four stages of inference, observed across diverse model families and sizes: (1) detokenization, where local context is integrated lift raw token embeddings into higher-level representations; (2) feature engineering, where task- and entity-speciff features are iteratively refine;; (3) prediction ensembling, where hidden states are aggregated into plausible next-token predictions; and (4) residual calibration, where irrelevant features are suppressed to finalize the top-1 output distribution. Synthesizing behavioral and mechanistic evidence, we provide a hypothesis for interpreting depth-dependent computations in LLMs.
Introduction
Recent advancements Large Language Models (LLMs) have exhibited remarkable reasoning capabilities, often attributed to increased scale (. Understanding these capabilities and mitigating associated risks
2 have motivated extensive research into their underlying mechanisms.
bottom-up approach to interpretability, known as mechanistic interpretability, has explored the iterative inference hypothesis 6). which posits that each transformer layer incrementally updates token's hidden state toward minimizing loss by progressively shaping the next-token distribution 77]. This is supported by self-repair (6), where later layers correct mitigate errors introduced by earlier layers, and redundancy [ 19, where multiple layers perform similar or overlapping computations to refine predictions.
remains unclear how this iterative view of inference fits with the "circuit" hypothesis, which argues for clearly delineated, specialized roles for certain model components. This is supported by induction heads [10]. successor heads [ copy suppression mechanisms [12). and knowledge neurons [13, among other "universal" neurons [14 [5]. Whereas iterative inference suggests gradual refinement through overlapping computations, the strong circuit hypothesis implics distinct, modular computational pathways.
"Conresonding author. Work started at MIT, completed at Stanford University
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

gpt-2 models
pythia models
phi models
rormaltrod mommlatddd methic value
0.8
atte. en prev. tokens
prediction hcurons
0.6
0.66
0.6
suppression neurons
0.4
normaltzed
0.4
rermalized
0.4
0.2
02-
0.2
0.0
0.0
0.0
0.0
0.2
0.4
1.0
0.2
0.8
0.0
0.2
0.6
0.8
1.0
relative layer depth
relative layer depth
relative layer depth
Figure 1: Statistical signatures of stages of inference averaged across three model families. (Blue) KI. between the normal model and layer zero-ablated. (Purple) Total attention paid to the previous five tokens in sequence. (Green) The number of "prediction" neurons (Red) The number of "suppression" neurons
Table 1: Our Hypothesis: Stages of Inference
Stage
Name
Function
Observable signatures
1
Detokenization
Integrate local context to transform raw token representations into coher- ent entities
Catastrophic sensitivity to deletion and swapping and attention-heavy computation.
2
Feature Engineering
Iteratively build feature representa- tion depending on token context
Little progress made towards next token prediction, but significant increase in probing accuracy and patching importance.
3
Prediction Ensembling
Convert previously constructed se- mantic features into plausible next token predictions using an iterative ensemble of model components
Prediction neurons appear and out- put distribution begins narrow.
4
Residual Calibration
Eliminate obsolete features and Suppression neurons appear and out- put distribution shapes for top-1 pre- diction with a growing MLP-output
form the next token distribution from internal representation
Naturally, layer-wise phenomena in LLMs are also documented outside formal interpretability re- search and provide more evidence to existing interpretability findings. For example, while knowledge storage within mid-layer MLP neurons has been demonstrated [16), other non-interpretability work has found that tne-tuning predominantly affected the weights in the middle layers [17]. Quantization studies identified improved benchmark performance by retaining only low-rank MLP components from the middle to later layers 18]. Other works have noted a transition in activation sparsity from sparse dense around mid-mo model depth [19] These behavioral findings, when integrated with mechanistic insights, suggest layered computation structure not yet fully characterized.
To explore this structure, we perform layer-wise interventions-deleee individual layers swapping adjacent ones (Figure [3 to characterize their localized effects. Building on these insights, we analyze depth-wise roles and synthesize our findings with prior interpretability work propose four-phase hypothesis that attempts to bridge the top-down and bottom-up views of computation in decoder-only LLMs.
Concretely, we hypothesize four depth-dependent stages: (1) detokenization, (2) feature engineer- ing, (3) prediction ensembling, and (4) residual calibration. In brief, early layers integrate local context to form coherent entities; middle layers iteratively construct features; later layers convert these features into next-token predictions via ensemble of neurons. Figure and Table summa- rize these stages and their associated empirical signatures. We synthesize these findings with prior interpretability work 21 to suggest depth-aligned computational structure in LLMs.
and

---

Related Work
Mechanistic Interpretability Mechanistic interpretability often employs circuit analysis uncover model components relevant to specific computations. In computer vision. universal mechanisms such as frequency detectors and curve-circuits have been identified [22 with features become progressively more complex through the layers of CNNs. These principles were later extended to modern transformers [25 26], where similar circuit-based analyses revealed phenomena such as
circuit reuse [27]. variable-findnng mechanisms [28] self-repair 66 29 function vectors 30 31
and long-context retrieval [32]
Iterative Inference and Septh-Dependert Computations The iterative inference hypothesis, first explored in ResNets [33 34 posits that each layer incrementally updates token representations. This idea has gained traction in transformers, particularly through logit lens analyses [35] 5 which visualize the model's evolving prediction distributions layer by layer. Some studies further sug- gest discrete inference phases [36], with certain computations localized specific depths-such such as truth buth-procsssing [37] or multilingual translation [38]. These findings are complemented by layer per- mutation studies showing that performance improves when self-attention layers precede feedforward layers [39.
BERTology Prior work ablations and layer-wise analysis has primarily focused BERT [40]. These studies reveal substantial redundancy: even with aggressive neuron and layer pruning, models retain most of their performance [41) More recent investigations corroborate this, showing that significant portion of attention heads and feedforward components can be removed with minimal
accuracy loss 9 g
Experimental Protocol
Table 2: Comparison of Language Model Architectures
Model Series
Size
Layers
Model Series
Size
Layers
410M
Phi-1 (1.3B)
24
1.4B
Microsoft Phi
Phi-1.5 (1.3B)
24
Pythia
2.8B
32
Phi-2 (2.7B)
32
6.9B
32
IB
16
12B
36
Llama 3.2
3B
28
Small (124M)
12
0.5B
24
GPT-2
Medium (355M)
Qwen 2.5
1.5B
28
Large (774M)
36
3B
36
XI. (1.5B)
48
14B
48
Models To investigate the stages of inference in language models, examine the Pythia 46).
GPT-2 [47). Qwen 2.5 [48), LLaMA 3.2 44)) 1 and Microsoft Phi [50 model families, which range from 124M to 14B parameters (see Table 2 All families use decoder-only transformers but differ their execution of attention and MLP components. Specifically, Pythia models execute attention and MLP layers in parallel. In contrast, GPT-2, Phi, and Llama models apply attention followed by an MLP sequentially. We preprocess weights identically across all models, folding in the layer norm, centering the unembedding weights, and centering the writing weights as described in Appendix [B] Despite these architectural differences, most phenomena remain consistent across models, though we discuss drawbacks in Limitations
Data Besides data agnostic experiments, we evaluate all five model families on corpus of one million tokens from random sequences of the Pile dataset [52), unless otherwise noted in the experiment.
Layer Swap Data Collection study the robustness and role of different model components at different depths, we employ swapping intervention where switch the execution order of

---

divergence with 1x) swagood w.r.t normll
1.0
relattue accuracy with layer suoppedd nomaal
10
RL. divergence
relative accuracy
00
0.0
0.6
02
0.0
3.5
KL divergence with layer tx) ablized w.r.t normal
1."
(elative accuracy with layer IX ablated w.tt normal
Layer
relative accuracy
RL divergence
(b) CKA Similarity GPT2-XL
9.6
69
0.4
06
Layer
60
06
0.0
0.2
0.4
0.6
0.6
1.0
0.0
0.0
0.2
0.4
0.5
0.3
100
65
relative layer depth
relative layer depth
(a) Layer Interventions Experiment
Layer
type size
(c) CKA Similarity Pythia 2.8B
Pythia 1.4h pythia
pythia 410m
gpt-2
gpt-2 small
pythia
got-2 large
gpt-2
Figure 5: (a) Effect of layer swap (top) and layer drop (hottom) interventions on model behavior. (left) KL divergence between the intervened and original models. (right) Consistency of top-1 predictions. (b)(c) Representational similarity across layers measured using CKA, showing block-like structure in GPT-2 XL (b) and Pythia 2.8B (c). Similar trends are observed across other model families and sizes (see Appendix
pair of adjacent layers in the model. Specifically, for a swap intervention at layer l, we execute the transformer block (including the attention layer, MLP, and normalization) f+ before executing block f. We record the Kullback-Leibler (KL) divergence between the intervened and original models output distribution, along with the loss, top-1 prediction accuracy, prediction entropy, and benchmark task performance. This intervention allows us to examine how the order of computation affects the model's 's behavior and performance different depths.
Ablation Data Collection To generate baselines for each layer swap experiment, we perform zero ablations on the corresponding layer while collecting the same metrics. The ablation preserves the swap ordering: for a swap ordering of 1-2-4-3-5, the ablation maintains 1-2-4-5, We opt for zero ablation opposed mean ablation, proposed by (5), to maintain consistency with the swap order.
4 Robustness
4.1 Intervention Results
We apply our aforementioned drop and swap interventions to every layer of four GPT-2 models 53 and four Pythia models [46]. Figure we report (1) the KL divergence between the prediction of the intervened model and the nominal model. (2) the fraction of predictions that are the same between the intervened model and the baseline model (denoted as relative accuracy), We also report the performance common benchmark tasks (HellaSwag 54], ARC-Easy (55) and (AMBADA(56) for all models in Figure 15 16 which show a similar trend.
In contrast to the first and last layers' interventions, the middle layers are remarkably robust to both deletion and minor order changes. When zooming on the differences between the effect of swaps and drops for intermediate layers, we find that swapping adjacent layers is less harmful than ablating layers, matching result in vision transformers (26]. We take this as an indication that certain operations within the forward pass commutative, though further experimentation is required.

---

mean attention
Model
Window Size
0.20
prey
yorlt
L2H5 pos 16 attentioo
022
0.15
gar>-arge
0.10
0.00
0.0
0.2
0.4
0.6
0.8
1.0
port
relative layer depth
relative layer depth
10
12
1s
16
token position
(b)
Figure 6: (a) The average (across heads within a layer and query tokens) attention weight placed on the preceding I, 2, 4, 8, 16 tokens for each layer. (b) Attention from the source token the final token in various inputs. An identified sub- word merging attention head (bottom) found in the early layers of language models is responsible for attending
multi-token words (i.e, shenanigans, refurbishments, parfaitement, circumnavigatte, compared to the baseline set of random non-multi-token words (top).
Intervening on the first layer is catastrophic for model performance for every model, regardless of size or model family. Specifically, dropping or swapping the first layer causes the model to have very high entropy predictions opposed to causing mode collapse on constant token. In some models, swapping the last layer with the second-to-last layer also has a similar catastrophic high-entropy effect, while GPT-2 models largely preserve their predictions. This phenomenon motivates our study into the first few layers of the model, specifically the role paid by attention heads in these layers.
5 Stages of Inference Hypothesis
Motivated by the distinct phenomena at the first few and final few layers, we measured representational similarity across each layer output using Centered Kernel Analysis (CKA))7755)) This revealed a block-like structure across multiple models as shown in Figure 4 robustness observed in the layer-wise intervention. Furthermore, the depth-depnndent phase structure indicates that a shared computation motif across adjacent layers occurs in stages.
The existence of blocks reflects the
5.1 Stage 1: Detokenization
Given the extreme sensitivity of the model to first-layer ablations, we infer that the first layer is not normal layer, but rather an extension of the embedding. Uniquely, the first layer is the layer that moves from the embedding basis to that of the transformer's residual stream. It is only a function of the current token. Consequently, by ablating the first layer. the rest of the network is blind to the immediate context and is thrown off distribution. Immediately after computing this extended embedding, evidence from the literature suggests that the model concatenates nearby tokens that are part of the same underlying word 660 61 or entity [62 (e.g., first and last name). This operation integrates local context to transform raw token representations into coherent entities. In this way, the input is "detokenized" [36] 63 Previous work has shown the existence of neurons that activate for specific r--grams [63 [5]. Of course, to accomplish this, there must be attention heads that copy nearby previous tokens into the current token's 's residual stream.
Sub-word Merging Heads To further examine this detokenization mechanism, we investigated attention heads responsible for constructing multi-token words, known as sub-word merging heads [6]] These heads help capture the context of token for appropriate prediction, thus contributing to the detokenization process. We constructed dataset with two classes: each consisting of 16 tokens, where in one class, the final 4 tokens form a word. Our analysis identified specific heads in the early layers of models that contribute solely to the construction of these multi-token words. As illustrated in Figure (b) layer 2 head 5 of Pythia 2.8B moves information from earlier tokens to the final token of the word. The attention heads exhibit a consistent pattern, where attention decreases as tokens approach the final word. Specifically, the final token of the word attends most strongly to the first
5

---

0.625
15.0
accuracy
0.600
entropy (bits)
12.5
10.0
0.575
Owen2.5-38
7.5
0.550
qpt22xl
5.0
Llama-3.2-3 2-38
0.525
2.biia---e-dddueer phi-2
Pythia-2.8a.
2.5
ptn-2
pythaa deduped
Owee2 5-38
0.0
Ulaaa-3
0.2 0.4
0.8
1.0
relative layer depth
0.0
0.4
0.6
0.8
1.0
relative layer depth (b)
Figure 7: (a) Layer-wise probe accuracy on contextual lexical meaning (WiC task), peaking in intermediate layers is suggestive of where semantic features linearly encoded. (b) Using the logit lens technique [35], we calculate the probability distribution of the next token at the end of every layer, and then take its entropy. This provides measure of the model's confidence in the next prediction. Despite high probe accuracy, the residual, but high entropy residual stream suggests that semantic features exist mid-model but are not yet used for prediction. For all models see Appendix [8 and 19
token, a feature absent in the baseline. This suggests at least one of many mechanisms by which models integrate local context, occurring at higher density in the first half of the models.
Local Attention If early layers indeed specialize integrating local context, then we would expect attention heads in these layers to disproportionately focus on tokens close to the current position. To investigate this hypothesis, we measure the fraction attention that each token directs toward preceding tokens at varying distances. As shown in Figure 6 attention heads in early layers are strongly biased towards nearby tokens, with attention becoming progressively less localized in deeper layers.
5.2 Stage 2: Feature Engineering
After integrating local context in the early layers-e.g.. stitching together sub-word tokens and forming short-range dependencies-ttle model must begin converting those localized representations into more semantically meaningful features. We hypothesize that this marks the beginning of feature engincering stage, in which the model constructs intermediate features that encode abstract properties useful for downstream prediction.
Prior work provides indirect support for this idea. Model editing studies suggest that factual informa- tion is stored mid-layer MLPs [6 64 62 while probing experiments have found that intermediate layers encode features related to sentiment (65), truth (37), and temporal structure [66]. These studies typically show that probing accuracy rises through the early layers, peaks near the midpoint, and then declines, suggesting that features are constructed and later transformed compressed. Related work also observes a shift from syntactic to semantic representations with depth [36 38)
WiC Probing To illustrate this pattern, we train linear probes to detect context-dependent word meaning using the WiC (Word-in-Contexx task [67 68 For instance, given two sentences containing the word bank, the task is to classify whether it is used with the same meaning. Examples include distinguishing "the river bank" from "the robbed bank," where the same word has different meanings depending on the context. We apply this probe at each layer of the model, using the hidden state of the target word in context. As shown in Figure (left), the accuracy the probe increases through the early layers, peaks in the middle of the model, and then decreases, supporting the hypothesis that semantic features are most linearly accessible in the intermediate layers. We extend the observation
across model families and sizes in Figure 18

---

16 layer neurons
% of laver neurons
KL divergencs
KL S
divergence
nourr
% di 1ag
layer deptt
layer depth
relative layer depth
04
0.6
K. 3P0
5P0 medium paod cron
1. pveta C.om
pyth 7,45 predection
5L 205-2 mupum
228-2 arge
222 arge pood
2002 medium suppriii on
19-19 1.40
pyth precietion
pych 2.4 Agoroaain
sussreedon
arge
pyth auprresiin
preciction
prediction suporaatoon
suppremiion
euppremne
420m
6.0 apprembor
(a) GPT models
(b) Pythia models
(c) Phi models
Figure 8: We measure KL divergence between intermediate and final predictions using the logit lens method [35 On the second axis, we use an automated procedure for classifying neuron types detailed in 14) into prediction neurons and suppression neurons. These universal neurons in all models known to increase the probabilities of tokens and decrease the probabilities of others. We hypothesize this inverse relationship as evidence for ensembling in networks [5].
Logit Lens While these results suggest that intermediate representations encode semantic informa- tion, it remains unclear whether such features contribute to prediction at this stage. To investigate this, we apply the logit lens [35] 669, which projects the residual stream at each layer into the output vocabulary space using the model's unembedding matrix. This provides a layer-wise estimate of the model's next-token distribution.
We compute both the entropy of the intermediate predictions and their KI divergence from the model output. As shown in Figure (right), (right) entropy remains high and KL divergence low throughout the early and middle layers. In other words, while meaningful features appear to be present in the residual stream this stage, the model's output distribution remains high in entropy, indicating that these features have not yet been consolidated into confident next-token predictions. Bridging this gap requires mechanism that selectively retains information from relevant features while filtering out irrelevant ones, thereby reducing uncertainty in the output distribution.
5.3 Stage 3: Prediction Ensembling
Around the midpoint of the model, we observe qualitative shift in behavior. Having constructed semantic features in the earlier layers, the model must begin converting these into specific next-token predictions. Evidence for this transition comes from the logit lens, where we observe a steady decline entropy FFigurerrrihh right) and KL divergence (Figure { between intermediate and final predictions beginning around the middle layers. This suggests that the model is gradually committing to particular output, aggregating semantic features into a more concrete distribution over tokens.
This region of the model also displays high robustness to layer interventions (Figure 5). suggesting redundancy capacity for self-repair One possible cause of this resilience the presence of overlapping computational pathways (6] 70). Rather than relying on a single deterministic path, the model seems combine multiple signals- both across and within layers-to form its prediction. We explore this mechanism by identifying the neurons that contribute to prediction, testing their collective behavior through
study, and analyzing their distributional effects across depth.
Within these overlapping pathways, we investigate specialized ensembles known as prediction neurons units that systematically promote the likelihood specific tokens [15] 7 [4]. These neurons work in tandem with suppression neurons (discussed in Section 5.4) to shape the model's output.
Ensembling
Prediction and Suppression neurons by analyzing the MLP output weights and their projection into vocabulary space via the unembedding matrix Wy. Prediction neurons exhibit a logit effect distribution We Wont with high kurtosis and positive skew; suppression neurons exhibit high kurtosis and negative skew. Across 18
Following previous work[14] we identify these neurons
7

---

models, prediction neurons begin to appear around the midpoint, increasing in density through the latter layers (Figure R) before being overtaken by suppression neurons. For a detailed analysis the detection and characterization of prediction, suppression, and other "universal" neurons, we refer readers to the original work [4.
Probing for the Suflix "-ing" We hypothesize that ensembles of prediction and suppression neurons collectively support next-token prediction. To test this, we construct balanced classification task: given 24-token context to a verb, does the final token end with or without "-ing"? We train linear probes on the activations of 32 high-variance prediction and suppression neurons, both individually and in groups. Neurons are selected using the criteria above, and examples from GPT-2 XL are shown Figure The full neuron list appears Appendix 21
We train two types of probes on the penultimate token's activations: 32 individual neuron probes and top-k ensemble probes ranked by individual accuracy (Figure P}. Suppression neurons yield the strongest individual probes, performing on par with the model's 's predictions (dotted red line). Ensemble probes trained on prediction neurons outperform both individual neurons and the model average, suggesting an important interplay between the two neuron types.
Density Effects The balance between prediction and suppression neurons appears to shape the model's output. To test this, we analyze how their density relates to the KL divergence between each layer's logit lens distribution and the final output. The sharpest decline in divergence corresponds closely with the rise in prediction neuron density, which peaks at roughly 85% of model depth.
Model comparisons further reinforce this patter.. Phi-1 1 has fewer prediction neurons and shallower KI. slope compared to later Phi models (Figure 8c) GPT and newer Phi models show steeper, smoother KI. divergence drops than Pythia (Figures 8a 8bp. Notably, the most performant Phi models exhibit nearly 15% prediction and 25% suppression neurons per layer-5-88 and 3-7x x that of Pythia.
5-8x x the density in GPT-2
Interestingly, the density of prediction neurons decreases near the final 10% of layers, even as the model continues to converge on its output, sometimes accelerating(Figure (b).. This suggests the involvement a distinct final-stage mechanism, which we delineate as separate stage.
5.4 Stage 4: Residual Calibration
As prediction neuron density declines in the final layers, different mechanism emerges. Across all models, we observe sharp rise in suppression neurons near the end of the network. This transition from prediction to suppression neurons frequently coincides with an inflection point: entropy stops decreasing and begins increasing in the final layers (Figure 19b).. Unlike prediction neurons, which promote likely tokens, suppression neurons refine the model's 's output by removing obsolete features and down-weighting improbable tokens. The resulting entropy increase in the final layers suggests that suppression neurons serve calibrate the model's output toward the task it was trained for: producing a well-formed distribution over possible next tokens.
Layer Repeating Experiment To further explore this hypothesis, we design an experiment where we repeat certain layers of the model. Specifically, we duplicate blocks of layers within the model-for example, repeating layers 5 through 7 results
a sequence like (..4-5-5-6-6-7-7-8- 9..). For this analysis, we fix the number of repeats to 1 and the block length to 5 (see additional results across model sizes and block length Figure 25 In Figure we observe that repeating blocks in the latter half of the model leads to a consistent decrease in entropy relative to the baseline (horizontal line). When evaluated on downstream benchmarks, the models with repeated layers at the last 80-90% of depth also exhibit improved performance on benchmarks, suggestive of residual calibration and the late-stage influence of prediction and suppression neurons. (Appendix 24).
Final Layer The intensity of suppression neurons, seen in Figure 8 is localized in the final few layers of the model, where the quantity of suppression neurons outstrips prediction neurons. To quantify the intensity of this change to the output distribution, we measure the norm of the MLP output, where a larger norm suggests a greater contribution to the residual (Figure TOP. This also coincides with increase in entropy (Figure 19b).
8

---

noraalizdd mlp nor
1.0
gpt2.4l
0.85
pythia-6. 9.b-
microsofUphhi-2
Qwer/Qween2.-BB
0.6
meta-llamatiimm-a22
0.4
probe accuracy
0.75
single (prediction) probe
0.2
single (Suppression) neuron probe
0.70
model accuracy
0.0
top-k probe
0.0
0.2
0.4
0.6
0.0
1.0
relative layer depth
(b) MLP Output Norms
0.60
0.55
lagits entropy
4.0
3.5
5
10
15
20
25
30
neuron index
0.1 0.2 0.3 044 0.5 0.6 0.7 0.8 0.9 1.0
relative depth (copy start layer)
(a) Probing For Neuron Ensembles in GPT2-XL.
(c) Layer Repeating Experiment
Figure 12: (a) Accuracy of linear probes trained to predict whether the final token ends in "-ing," using activations from individual prediction and suppression neurons (scatter points) and ensembles of neurons (blue line). Ensembles outperform individual probes and occasionally exceed the model's top-1 accuracy (red dotted linc), consistent with the presence of "prediction ensembling." (b) Layer- wise MLP output norms acToss all 18 models show rise toward the final layers, suggesting increasing residual contribution late in the model. (c) Repeating layers from the later half of a model reduces final-layer logit entropy
than repeating earlier layers using the original model (dotted line), suggestive of residual calibration and the late-stage influence prediction and suppression neurons.
Concluding Remarks
Why Are Language Models Robust to Layer-Wise Interventions? We hypothesize that the robustness of language models to layer deletion and swapping stems in part from the transformer's residual architecture. This interpretation aligns with findings prediction and suppression neurons: multiple computational pathways appear to contribute to the same output, allowing the network tolerate disruption in any single path. The residual stream promotes this "ensembling", enabling gradient descent to construct shallow sub-networks that can operate in parallel. This architectural flexibility reduces the model's reliance on any specific layer, explaining its resilience to local interventions and supporting observed self-repair behavior and overlapping representations.
Limitations and Future Work While our four-stage hypothesis captures broad, depth-dependent patterns in I.Mss, several caveats remain. Stage boundaries are approximate, and multiple stages may co-occur within a single layer. The findings reflect aggregate trends, whereas individual tokens may follow distinct processing paths. Additionally, differences; e.g., GPT's greater robustness could arise from dropout, architectural variations, or depth. These limitations point directly to promising directions for future research. Future work should seek to sharpen these boundaries, link them to optimization dynamics, and test this hypothesis with a theoretical account to explain the empirical results.
do not isolate the factors behind model-specific
This work introduces four-stage hypothesis for understanding inference in large language models, grounded in diverse set of behavioral and mechanistic analyses. By examining how models respond to structural interventions layer deletion and swapping. patterns, neuron function, and residual stream dynamics, we identify a repeatable depth-wise structure to model computation. These stages detokenization, feature engineering, prediction ensembling, and residual calibration- emerge across architectures and seales, suggesting that transformers perform inference not flat pipeline but as an ordered composition of specialized computational regimes. While not definitive, the strength and consistency of the empirical signatures presented here provide compelling evidence in support of the proposed hypothesis. Rather than aiming for an exhaustive
Conclusion
well as probing attention

---

mechanistic dissection, we offer a unifying perspective that synthesizes and extends prior findings in and outside the interpretability literature.
More broadly, this layered view of inference has implications for how we interpret, audit, and intervene language models. Understanding not just what model computes, but when and where it computes it, may inform future approaches to alignment, compression, and modularity in model design. We hope this hypothesis serves as foundation for deeper investigation into the emerging capabilities of LLMs.
Contributions and Acknowledgments VI. conceived and led the study, performed the analyses, and drafted the paper. JL conducted the sharpening experiment, WiC analysis, benchmarking, and CKA experiments. JL, WG and MT contributed to experimental design and analytical methodology, provided critical revisions, and WG and JL assisted with writing. A big thanks to Katrin Franke, Surya Ganguli, Sophia Sanborn, Eric Michaud, Josh Engels, Dowon Baek, Isaac Liao for helpful feedback.
We want to extend special acknowledgment to Elias Sandmann, who notified of error in the Transformer Lens documentation that affected our implementation of Logit Lens. The correction was crucial for the accuracy and reproducibility of our results which may have gone overlooked without his concern.
10

---

References
[1] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint OrXiv:2206.06633 2022.
[2] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0722 2021.
[3] Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic ai risks. arXiv preprint arXiv:2306. / 2023.
[4] Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, et al. Managing ai risks in an era of rapid progress. arXiv preprint arXiv:2310. 17688, 2023.
[5] Nora Belrose. Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023.
[6] Cody Rushing and Neel Nanda. Explorations of self-repair in language models. arXiv preprint arXiv:2402. /5390, 2024.
[7] Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint 22t.::20031168800 2022.
[8] Xin Men, Mingyu Xu. Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers large language models are more redundant than you expect. arXiv preprint (0t.:22403.08853, 2024.
[9] Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403. 17887, 2024.
[10] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodae Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. heads/index.htme
[11] Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, interpretable attention heads in the wild. arXiv preprint arXiv:2312.09230, 2023.
[12] Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy suppression: Comprehensively understanding attention head. arXiv preprint arXiv:2310.04625, 2023.
[13] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons pretrained transformers. arXiv preprint arXiv:2 104. 10460866, 2021.
[14] Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaci Kheirkhah, Qinyi Sun, Will Hathaway. Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models, arXiv preprint 2rXiv:2401.12881 2024.
[15] Elena Voita. Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, n-gram, positional. arXiv preprint arXiv:2309.048827 2023.
[16) Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations gpt. Advances Neural Information Processing Systems, 35:17359-17372. 2022.
11

---

http:///ssipptt/220//////////eee [17] Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. Task-specifis skill localization in fine-tuned language models. In International Conference on Machine Learning. pages 2701 17011-27033. PMLR, 2023.
[18] Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312.13558 2023.
[19] Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivas- lava, Ce Zhang, Yuandong Tian, Christopher Re, al. Deja vu: Contextual sparsity for efficient Iims at inference time. In International Conference on Machine Learning, pages 22137-22176 PMLR, 2023.
[20] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy, Transformer feed-forward layers are key-value memories. arXiv preprint 2rXiv:2012.14913, 2020.
[21] Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-jussà. A primer the inner workings of transformer-based language models. arXiv preprint S0ti::2-005002008, 2024.
[22] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3)ee00224 2020.
[23] Ludwig Schubert, Chelsea Voss, Nick Cammarata, Gabriel Goh, and Chris Olah. High-low frequency detectors. Distill, 2021.
doi: 10.23915/dissilll.00044..
[24] Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and
Chris Olah. Curve circuits. Distill, 2021. doi: 222991//istill..0024..00
https:/distill.uub222/eeiii hwtp:///wwwwiiiii//////eereiioie
[25] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Uni- versal transformers. arXiv preprint arXiv: 008,99819, 2018
[26] Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and Andreas Veit. Understanding robustness of transformers for image classification. In Proceedings of the JEEE/CVF international conference on computer vision, pages 10231-10241, 2021.
[27] Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit component reuse across tasks in transformer language models. arXiv preprint arXiv:2310.08744, 2023.
[28] Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context? arXiv preprint arXiv:23 2023.
[29] Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations. arXiv preprint arXiv:2307.15711 2023.
[30] Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint arXiv:2310.15213, 2023.
[31] Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. arXiv preprint arXiv:2310. 15916, 2023.
[32] Alexandre Variengien and Eric Winsor. Look before you leap: A universal emergent decompo- sition of retrieval tasks in language models, 2023.
[33] Klaus Greff, Rupesh K Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. arXiv preprint arXiv: 2016.
[34] Stanistaw Jastrzebski. Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. arXiv preprint 2017.
[35] Nostalgebraist. Interpreting gpt: The logit lens. fttp:://www.aligmetffffooo www.annnnnnnnnnunnnmammm org/
Dotts//cceBsspallllooovvveeeeeep gpt- the wht.-the-ogii--leeee logit-lenn 2020.
12

---

[36]
Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatffeld-Dodds, Jackson Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Kadavath, Josh Jacohson, Eli Tran-Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCandlish, Dario Amodei, and Christopher Olah. Softmax linear units. Transformer Circuits Thread, 2022. https://tansfoomerccinnnt http://aaaaaammmmmrriiiiiiiullee wwwp:/02///o///ideee.sooo html..
[37] Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023.
[38] Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Do Ilamas work in english? on the latent language of multilingual transformers. arXiv preprint arXiv:2402. 2024.
[39] Ofir Press, Noah A Smith, and Omer Levy. Improving transformer models by reordering their sublayers. arXiv preprint arXiv: 2019.
[40] Jacob Devlin, Ming-Way Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04805, 2018.
[41] Minjia Zhang and Yuxiong He. Accelerating training of transformer-based language models with progressive layer dropping. Advances in Neural Information Processing Systems, 33: 14011- 2020.
[42] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with structured dropout. arXiv preprint arXiv:1909.11 11556, 2019.
[43] Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. On the effect of dropping layers of pre-trained transformer models. Computer Speech & Language, 77:101429, 2023.
[44] Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. Analyzing redundancy in pretrained transformer models. arXiv preprint arXiv:200.004110, 2020.
[45] Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. Rethinking the role of scale for in-context learning: An interpretability-bused case study at 66 billion scale. arXiv preprint arXiv:2212.09095 2022.
[46] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
[47] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
[48]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[49] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21780 2024.
[50] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks all you need. arXiv preprint arXiv:2306. 1644, 2023.
[51] Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: 1hi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023.
[52] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0002. 2020.
13

---

[53] Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[54] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:I905.0788 2019.
[55] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05455 2018.
[56] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint vrXiv:1606.0003 2016.
[57] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519-3529. PMLR, 2019.
[58] Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. arXiv preprint arXiv:2010.15322, 2020.
[59] Anand Subramanian. PyTorch-Modes Model- -Copare
torch cka, 2021.
URL httpa://github..co http://tthubccco//te Antink/
[60] Gonçalo M Correia, Vlad Niculae, and André FT Martins. Adaptively sparse transformers.
arXiv preprini urXiv:1909.0000 1909.00055, 2019.
[61] Javier Ferrando and Elena Voita. Information flow routes: Automatically interpreting language models at scale. arXiv preprint arXiv:2403.00824 2024.
[62] Neel Nanda, Senthooran Rajamanoharan, Janos Kramar, and Rohin Shah. Fact finding: 2023.
Attempting reverse-engineer factual recall on the neuron level. Dec
URL https://www www.aaaantgnaatrrmmmmmmmm http://www....ootoaat////ooo.... htwp:e/nnnnt...eo-----eeeeeaaaaa
httpp:://oww.allignmntttfoo
[63] Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023.
[64] Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304.1470 2023.
[65] Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models. arXiv preprint 55X1 2023.
[66] Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint arXiv:2310.02207. 2023.
[67] Mohammad Taher Pilehvar and Jose Camacho-Collados, Wic: the word-in-context dataset for evaluating sentitt-sensitvve meaning representations. arXiv preprint arXiv: 09121, 2018.
[68] Zhu Liu, Cunliang Kong, Ying Liu, and Maosong Sun. Fantastic semantics and where find them: Investigating which layers of generative Ilms reflect lexical semantics. arXiv preprint arXiv:2403.01509 2024.
[69] Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. arXiv preprint wxXi:2209.005335 2022.
[70] Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information poccliioonpcccooee processing systems, 29, 2016.
[71] Neel Nanda. Transformerlens, 2022.
URL https:///githbb..om//reaaaaaaad io/
Transformerler
[72] Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from a deep learning perspective, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As stated in the abstract, we perform layerwise interventions to investigate the robustness of layers. These interventions are suggestive of phases of inference that form hypothesis. The remainder of the paper is series of experiments that support various findings of this hypothesis, across three different model families. We emphasize that we are proposing hypothesis that are result of performing experiments that suggest this hypothesis.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide limitation section where we discuss how we did not investigate the true cause of model-to-model difference and the potential loss of important findings that are result of excessive aggregation over tokens.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We do not have theoretical assumptions and proof in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
First describe in detail what encompasses a layerwise swap and deletion, in- cluding a diagram and the components involved. We include the models that we use and how the weights were preprocessed for experimentation. We also describe the specific metrics we aggregate over (KI. divergence, entropy, loss, etc) For our supporting experiments, we describe what component we measured the output of (MLP, Attention), and what calculation was performed (Norm, ratio, etc). For the logit lens technique and the measurement of predictive and suppressive neurons, describe the multiplications of specific matrices to recreate the results and reference appropriate papers that provide a deeper analysis of the method.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution
specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
16

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide link to a Github which contains all of the code to recreate the experiments, currently anonymous. We also describe which specific model families and datasets (the Pilc) were used the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ssionnolicyyyyi for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We do not perform any training but we specify all the metrics we collect from the models, including KL divergence, loss, entropy, and logits. We also describe the preprocessing of the weights carried before experimentation, further in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
17

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All our plots contain error bands. On some plots the results of aggregation causes the plot to have fine error bars, often not visible. For specificity: Figure does not have error bars, however, every peak in this figure appears with error bars in later sections of the paper. Figure we display metrics over all the models without aggregating in place of showing average line. Figures 4, 5, 6, 7, 8, 13 all contain error bands which are I-sigma. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide an appendix section titled "Additional Empirical Details" described compute usage.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have read and confirm that we abide to the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
18

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid-
eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: At the beginning of this paper we discuss the importance of connecting mechanistic interpretability to the broader understanding of machine learning. We discuss how this work aims make ML systems more understandable and allow for the detection of risks and vulnerabilities that can arise are language models scale, There are no negative impacts of the work as we write in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: None of our analysis, experiments, or conclusions pose a risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
19

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We utilize 3 different model families and single dataset, the Pile, which are cited accordingly in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi wwwode....d..... com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not use human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
20

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: There are no study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We simply study LLMs, and do not rely on LLM usage for hypothesis or experiment creation or execution.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:////oorrps.cccooooferrrreee for what should or should not be described.
21

---

A
Experiment Diagram
GPT Layer
Layer Ablation
Layer Swap
FTN
df of-itrate
Eayor here
Figure 13: To study the stages of inference, we perform two experiments, each a layer-wise interven- tion, where layer (left) encompasses all model components. The first intervention is zero ablation (i.e, layer removal) of the layer (middle), in which a layer is fully removed and residual connections skip the layer entirely. The second intervention (last) is an adjacent layer swap, in which permute the positions two layers. The ablation is performed on all layers, while the layer swap performed on all adjacent pairs of layers in the model.
Name
HuggingFace Model Name
Pythia 410M
AleutherA//yythia-4100-:- AaautherrA/pyytiia-------eeeee
Pythia 1.4B
Pythia 2.8B
EleutherAl/pyttha-2.55 d.b-eddupd
Pythia 6.9B
EleutherAA/pyythia-.. htwpr////pppiii------eduuupp
Pythia 12B
EleutherA//pythia- hatplpytttaa-b---eddee 12b-deduped
GPT-2 Small (124M)
gpt2
GPT-2 Medium (355M)
gpt2-medium
GPT-2 Large (774M)
gpt2-large
GPT-2 XL (1.5B)
gpt2-xl
Phi 1 (1.3B)
microsoft/Phi-1
Phi 1.5 (1.3B)
microsoft/Phi-1.5
Phi 2 (2.7B)
microsoft/Phi-2
Qwen 0.5B
Qwen/Qween2.500.5B
Qwen 1.5B
Qwen/Qwen2.5511.B
Qwen 3B
Qwen/Qwen2.55
Qwen 14B
Qwen/Qwen2.5-1483
Llama-3.2 3.2 1B
Llama-3.2 3B
The Pile
EleutherAl/the Bpciilppppleeeeeeeeeeeee deduplicated
Table 3: List of models and dataset used the experiments.
B Additional Empirical Details
Github All experimental code for future experiments available at:
Transformer Lens
We make ubiquitous use of Transformer Lens to perform hooks and transformer manipulations.
HuggingFace For specificity, we utilize the following HuggingFace model names, and dataset. We do not change the parameters of the models from what they described on the HuggingFace page.
All experiments described can be performed on single NVIDIA A6000. We utilized 2 NVIDIA A6000 and 500 GB of RAM. To aggregate the metrics described in the paper, run the model on I
22

---

million tokens e times, where is the number of layers. This takes on average 8 hours per model, per layer intervention (swapping and ablating). We save this aggregation for data analysis.
Residual Sharpening to Residual Calibration We initially named the final stage of inference "residual sharpening" but have renamed it to "residual calibration" " for greater accuracy. While suppression neurons do eliminate obsolete features from the model's 's representation, the final layers sometimes exhibit an increase in e seemingly contradictory behavior if the goal were simply to sharpen predictions toward single top token. Instead, this entropy increase suggests that suppression neurons calibrate the representation to produce a well-formed distribution over possible next tokens, aligning with the language modeling objective. This calibration process differs from sharpening, which would imply converging toward a single prediction. Additionally, models even within the same family exhibit varying entropy patterns in their final layers. We hypothesize that the variation entropy the final layers may indicate model confidence and contribute to hallucination behavior, though we leave this investigation to future work.
Layer Norm Preprocessing We utilize several conventional weight preprocessing techniques to streamline our calculations (7T).
Following 14, before each ML. calculation, layer norm operation applied the residual stream. This normalizes the input before the MLP. The TransformerLens package simplifies this process by incorporating the layer norm into the weights and biases of the MLP, resulting matrices War and ber. In many layer norm implementations, trainable parameters
IR" and b E R" are included:
x
[[x)
LayerNorm(x)
+ b.
/Va((x)
We "fold" the layer norm parameters into Win by treating the layer norm as a linear layer and then merging the subsequent layers:
War Win in diag(y) bur bm Winb
(2)
Additionally, we then center reading weights. Thus, adjust the weights War as follows:
Win(i,)) = War(i.;) Won(i.:)
Centering Writing Weights Because of the LayerNorm operation in every layer, align weights with the all-one direction in the residual stream as they do not influence the model's calculations. Therefore, mean-center W... out and bour by subtracting the column means of W out'
W (1,4) = Woult, i) Wour::,
Extension of Results in Larger Models (>10B parameters) In Appendix K extend the results of the core experiments in larger models with more than 10B parameters (Qwen2.5- 14B, Pythia 12B).
Societal Impact We do not anticipate any immediate societal impact from this research.
23

---

C Centered Kernel Alignment (CKA)
30
3.00
5.0
18
0.00
Layer
-14
20.15
7M 10. Layer
Layer
Layer
(a) GPT2
(b) GPT2-medium
(c) GPT2-large
19
1A
58
67
-1A
-12
-44
-2
12-12:55 14 36 Layer
20.20.33
12 14 16 Layer
Layer
(d) pythia 410m
(c) pythia 1.4b
( pythia 2.8b
10
28
Layer
O7 30 305
6.6
-14
-55
15
10 36 38 20
12 14 Layer
16
26
22
Layer
Layer
Phi 1
(h) Phi 1.5
(i) Llama 3.2 IB
58
58
57
57
-15
58
-16
45
-55
-44
-54
10:02111 10 20322244011
201214 1414
20 24226
Layer
(i) Llama 3.2 3B
(k) Qwen 2.5 0.5B
(I) Qwen 2.5 1.5B
5.9
5.8
5.7
2.5
455
-14
-13
ONL
53
(m) Qwen
Figure 14: CKA across layers from the last token representation sampled from Pile dataset (max
token length 5 512, batch size 128). We used unbiased CKA 72 59).

---

D Benchmark Tasks Performance After Layer-Wise Intervention
We evaluate the benchmark performance HellaSwag, ARC-Easy and LAMBADA (54 with the intervened model. We observe similar trend to KL divergence reported in the main paper. Generally, the intervention the first layer and the last layer shows catastrophic deterioration of the performance but intervention on intermediate layers shows robust performance.
Model Accuracy Normalized Layer-Swap (per Task, Model Class, Model
cady
lamoada_opomm openal
hallsowag
9p12
2p12 mediium
op12 lange
pot desuupd
5o0 2.86 deluped
pythil delkupd
Acurar
3550020
ACOUaCC
ewRn eaou
Normalized Depih LLayer /MaxLaye Mormalized Depit Caayer/ Max Laywr)
Normallze Depth (Layer Max
Figure 15: Benchmark task performance after layer swap. Baseline performance of each model is marked with a dotted horizontal line.
25

---

Model Accurscy vs. Narmalized Layer-Swap (per Task, Model Class, Modie)
lambado,
hellsswag
PEMMOOOO (nuodo
Liama-3 2-10
Liama-3--00
0ewr2.5.0.0
0ewr2.511.50
Qwer2.5-38
Acourac
UDHNO
Normalized Dopth LLayer MMx Layer)
Nomalizzd Depth (Layer/ Max Layer)
Nomalizzd Death (Layer/ Max Layen
Figure 16: Benchmark task performance after layer swap. Baseline performance of each model is marked with a dotted horizontal line.
E Prediction and Suppression Neuron
% of layer neurons
20
15
15
of layer neurons
KL divergence
10
KL divergence
15
10
10
5
0
0.0
0.2
0.4
0.6
0.8
1.0
relative layer depth
0.0
0.2
0.4
0.6
0.8
1.0
relative layer depth
KL. Owen 2.5 0.56
Owen 1.58 prediction
KL Owen 2.5 155
Owen suporession
KL Owen 2.5
Owen prediction
KL Liama 3.2
Liama 3.2 18 suppression
Owen 2.5 0.58 prediction
Owen 2.5 38 suppression
KL Liama 3.2
Liama a 38 prediction
Owen suppression
suppression.
(a) Qwen
(b) Llama
Figure 17: Prediction and Suppression neurons for Qwen and Pythia.
26

---

F WiC contextual word probe
Aecuswiyy flaand fronse diilrity)
Reterive flaand frahuee distterty)
Salerive Slaedd
0.62
0.62
0.63
0.63
0.00
0.58
2 0.50
0.56
Model
0.55
0.55
Mode
0.54
coaraccpee murth bugaro
0.52
0.52
0.52
0.5
0.8
0.0
heletive
Autitto Digth
(a) GPT
(b) Pythia
(c) Phi
Accumny vs Relabve Zosth (Dueec on feature entantt)
Accumcy ve Relatie Sooth (Dasec fasture srilsity)
0.62
0.60
$0.58
0.55
Model
0.32
0.52
2.4
0.5
0.2
2.4
Desh
(d) Llama 3.2
(c) Qwen 2.5
Figure 18: WiC probing accuracy over layers across model families and sizes. Across all models and sizes, observe the probe accuracy related contextual semanties of lexical items gradually increases and peaks around the middle layers and degrades.
G Logit Lens Entropy
5p2
phi1
entropy (bits)
meduumm
entropy (bits)
entropy (bits)
ghi1
2pt2 arge
ghi
gp42.xl
pyttis-c1.m om
pyttle-1.00
Pyttls-2.0
Potlie-6.0
relative layer depth
64
relative layer depth
relative layer depth
(a) GPT Logit Lens Entropy
(b) Pythia Logit Lens Entropy
(c) Phi Logit Lens Entropy
10-
15.0
entropy (bits)
entropy (bits)
12.5
4-
5.0
2-
32-3033-30
0.0
0.2
0.4
0.6
0.8
1.0
0.8
relative layer depth
relative layer depth
(d) Llama Logit Lens Entropy
(e) Qwen Logit Lens Entropy
Figure 19: Using the logit lens technique [35), we calculate the probability distribution of the next token the end of every layer, and then take entropy.
27

---

H MLP Norms
150
250
mip norm
125
catheracker tar2tagg gp? st
mlp norm
200
2,55
2/thaa114b
60
bythls 6.00
50
100
30
20
20
40
60
BC
100
40
60
DC
100
0.0
0.2
0.4
0.6
0.8
10
relative layer depth
relative layer depth
relative layer depth
(a) GPT MLP Output
(b) Pythia MLP Output
(c) MLP Output
me3 2211
250
mlp norm
30-
norm
Owestiwen25.3
1.00
50-
relative layer depth
relative layer depth
(d) Llama MLP Output
(e) Qwen MLP Output
Figure 20: The norm of the output of every MLP across its layers to measure its contribution to residual stream. Across all 18 models, the norm grows and peaks in the final layers before output, suggestive of the final two stages of inference, predictive ensembling, and residual calibration
28

---

I Top Prediction and Suppression Neurons
LeFt 9U81
14T.3033
I43:357
LT6-999
1352997
Tae
143.2135
L44-2093
145 2284
L20 333D
LAS 555
LS53464
L45:445
LSS3212
LS0223:
LAL3404
LAD.355
1%6
9922
L44.51
ES1 5541
L30 5101
3215
15
3230
19
lake
MKULUF
611
Take
Figure 21: Top 36 prediction and suppression neurons for -ing which have the greatest mean absolute difference between respective (Wy Woar). Elements with a negative skew are suppression neurons for the respective labeled class, while elements with positive skew are prediction neurons. This is calculated by calculating the product between the model unembedding weights and output weights of MLP.
29

---

J
Layer repeats experiment
5.0
6.0
8.0
5.5
5.5
Ownn 5455. Owriid
logits entropy
priss
logits entropy
logits entropy
0.5
2.0
4.0
3.5
3.
Lamr-322418
3.5
Lamr-322-35
0.1 0.2 0.3 04 0.5 0.6 0. 0.9 1.0
0.1 0.2 0.3 0.4 0.5 0.5 0.7 0.9
0.15
0.30
0.45 0.60 0.75 0.90
relative depth (copy start layer)
relative depth icapy start layer)
relative depth (copy start layer)
(a) Phi Block 3
(b) Llama 3.2 Block
(c) Qwen 2.5 Block 3
5.5
enbroyy
togits entroyy
4.5
5.0
4.5
4.0
4.0
2012 krgz
10 Mil ducuped po ob-ootipes pothis
2012
3.0
3.0
6.7 oi relative depth (copy layer)
0.2
relative depth (copy startlayer)
GPT Block
(c) Pythia Block 3
Figure 22: Block repeat experiment.
5.0
logits entropy
lngits entropy
5.0
4.5
8.0
0.0
logits
4.0
892
2011 Eb-cekppal
Poth 1-7 ded12ed
84
0.2 0.5
relative depth (copy start layer)
relative depth (copy start layer)
relativee depth (copy start layer)
(a) GPT
(b) Pythia
(c) Phi Repeat Block
6.0
Lb-1-12-13
5.5
logits entropy
logits entrepy
5.0
5.0
3.5
3.5
3.0
3.0
0.2 0.3 066 0.7 0.8
1.0
02 0.3 0.4 05 0.7 0/6 0.9 1.0
relative depth (copy start
relative depth (copy start layer)
(d) Llama Repeat Block
(c) Qwen Repeat Block 5
Figure 23: Block 5 repeat experiment on additional models.
30

---

Benchmark Results for Task: hellaswag
Benchmark Accuracy
Base Model
0.55
Baseline
Baseline
0.50
Baseline
Qwen2.5-0.55
0.45
Qwen2.5-4 5-15B
Qwen2.5-4
0.40
0.35
0.30
0.0
0.2
0.4
0.6
0.8
1.0 Max Depth)
Relative Depth (Layer Index
Figure 24: Qwen repeat 5 model's performance on Hellaswag
31

---

K Experiments Larger Models (>10B parameters)
1.0
1.0
0.8
12
aaer
0.7
27
0.6
22
0.5
24
0.4
32
45
34
48
-0,3
Layer
Layer
(a) CKA, Qwen2.5- 1-BB
(b) CKA, Pythia- 12B
6.0
6.0
Qwen2 5-1.50
5.5
Qwen2 5-0.8
logits entropy
Qwen2 5-38
logits entropy
4.5
4.5
4.00
4.0
gpt2-larg
gpt2
3.5
gpt2-d
apt2:medium
3.0
3.0
0.15
0.30
0.45
0.60 0.75 0.90
0.15
0.45
0.60 0.75 0.90
relative depth (copy start layer)
relative depth (copy start layer)
(c) Logit Entropy, Qwen2.5-14p
(d) Logit Entropy, Pythia- 12B
0.625
accuracy
0.600
0.575
0.550
0.525
Qwen2.5-148
pythla-126-deduper
0.0
0.2
0.4
0.6
1.0
relative layer depth
(e) WiC Task
32

---

pythia.
entropy (bits)
entropy (bits)
10
6
Owen2.5 146
0.0
0.4
0.6
0.8
1.0
0.0
02
0.4
0.6
0.8
1.0
relative layer depth
relative layer depth
(f) Logit Lens Entropy, Qwen2.5-14B
(g) Logit Lens Entropy, Pythia- 12B
1.00
layer neurons
% layer neurons
KL divergence
KL divergence
0.75
10
3
0.50
0.25
0.00
0
relative layer depth
0.0
0.2
0.4
0.6
0.8
1.0
relative layer depth
KL Owen 2.5 140
Owen 2.5 140 suppression
Owen 2.5 148 prediction
pythia
pythia suppression
pythia 12b prediction
(h) Predlicioon/uppprssioon Neurons, Qwen2.5-
14B
Logit Lens Entropy, Pythia- 12B
5.0
loglts entropy
4.5
logits entropy
4.0
4.0
sybb
3.5
3.5
Owen2.5-12
3.0
3,0
0.15
0.30
0.45
0.60 0.75 0.90
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 relative depth (copy start layer)
relative depth (copy start layer)
(i) Block repeat experiments
(k) Black 5 repeat experiments
Figure 25: Extension of Experiments on Larger Models (>10B).
33

---

L Experiment Subset on OOD Data
gpt2-xl
get2.x
mean attention
entropy (bits)
0.05
20-
0.04
0.03
0.02
0.01
0.0
0.2
0.4
0.5
0.8
1.0
0.2
0.6
1.0
relative layer depth
relative layer depth
Mean Attention, recent arXiv papers
(b) Logit Lens Entropy, recent arXiv papers
normalized mlp norm
1.0
% of layer newrons
gpt2-xl
KL divergence
1.5
0.8
1.0
0.6
0.5
0.4
0.2
0.2
relative layer de ptht
0.0
0.2
0.4
0.6
0.8
1.0
gpt2 xl prediction
relative layer depth
(c) KL divergence between intermediate and output distributions, recent arXiv papers
(d) MLP Output Norms, recent arXiv papers
Figure 26: Running on 2024-2025 arXiv papers, code, and languages results in consistent patterns across the hypothesized stages of inference for GPT-XL.
34


Reviews:
[{"review_id": "LKGJpbMnCq", "reviewer": "Reviewer_BKjM", "summary": "This paper investigates LLM robustness by systematically deleting and swapping adjacent layers during inference, finding that models retain 72-95% accuracy without fine-tuning. The authors observe that early and final layers are most sensitive to interventions while middle layers show robustness. Based on these findings and additional analyses (attention patterns, neuron classification, probing), they propose a four-stage inference framework: (1) detokenization, (2) feature engineering, (3) prediction ensembling, and (4) residual sharpening. The framework is evaluated across multiple small-medium model families (GPT-2, Pythia, Phi, LLaMA, Qwen).", "questions": "1. **Mechanistic validation**: How do you distinguish between general redundancy and your proposed computational stages? Why should robustness to interventions specifically indicate \"feature engineering\" rather than simple over-parameterization?  \n\n2. **Stage boundary justification**: Given the large variation in where stage transitions occur across models (30-70% depth), how can you claim these are \"universal\" stages rather than continuous processes with model-specific variations?  \n\n3. **Causal evidence**: Can you provide interventional evidence that disrupting proposed mechanisms (e.g., attention patterns, specific neuron types) affects the hypothesized stages rather than just general performance?  \n\n4. **Predictive power**: What specific, testable predictions does your framework make that weren't already known from prior interpretability work?", "limitations": "The authors inadequately address several critical limitations: (1) the post-hoc nature of their framework, (2) the lack of causal evidence for proposed stages, (3) the significant variation in stage boundaries across models, (4) the limited scope of evaluation tasks, and (5) the weak connection between intervention robustness and computational stages.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:** \n 1. **Systematic intervention methodology**: The layer deletion and swapping experiments across 16 models provide valuable empirical insights into transformer robustness. The finding about differential sensitivity across layers is genuinely interesting and novel. \n 2. **Comprehensive empirical scope**: Testing across multiple model families strengthens the generalizability claims, though the universality is overstated. \n 3. **Useful organizing framework**: While theoretically weak, the four-stage framework provides a helpful way to synthesize existing interpretability findings. \n\n\n**Weaknesses:**  \n\n1. **Weak theoretical foundation**: The four-stage framework is essentially post-hoc pattern matching rather than principled theory. The authors retrofit observations to fit predetermined stages without deriving these stages from computational principles or providing mechanistic explanations.  \n\n2. **Arbitrary and inconsistent stage boundaries**: The boundaries between stages are poorly defined and vary significantly across models and metrics. For example, the transition from \"feature engineering\" to \"prediction ensembling\" occurs at different relative depths across models (30-70%), undermining claims of universality.  \n\n3. **Insufficient evidence for claimed stages**: The connection between intervention robustness and the proposed computational stages is tenuous. Why should robustness to layer swapping specifically indicate \"feature engineering\" rather than general redundancy? The logical leap from empirical observations to stage definitions is not well-justified.  \n\n4. **Overstated universality claims**: While patterns exist across models, the significant variation in where stage transitions occur contradicts claims of \"universal\" stages. The evidence suggests more of a gradient than discrete stages. \n\n5. **Limited mechanistic insight**: The paper largely repackages existing interpretability findings (attention patterns, neuron types) without providing new mechanistic understanding. The \"detokenization\" and \"sharpening\" stages are particularly weakly supported.  \n\n6. **Correlative rather than causal evidence**: Most evidence is correlational (neuron densities, attention patterns) without demonstrating that these phenomena actually cause the proposed computational stages. The framework lacks interventional validation.  \n\n7. **Limited to small and medium-sized models**: The largest model tested is only 6.9B parameters, with most under 3B. Given that state-of-the-art LLMs now have hundreds of billions of parameters, it's unclear whether the proposed four-stage framework generalizes to truly large models where computational patterns and layer specialization may be fundamentally different.", "was_revised": true, "final_justification": "Most of my concerns have been addressed, but the issues regarding insufficient evidence for the claimed stages and the weak theoretical work still remainץ", "rebuttal": "We thank the reviewer for their detailed and constructive feedback! We appreciate their acknowledgment of our systematic intervention methodology, comprehensive empirical scope, and the novel insights about differential layer sensitivity.\n\nMany of the concerns raised are primarily based on misunderstandings about our framework's scope and intent. We address all points below, providing additional evidence and clarifications. These are also revised in the most recent draft of the paper!\n\n---\n\n## 1. **Theoretical Foundation and Mechanistic Validation**\n*Addressing Weakness 1 and Question 1*\n\nOur work provides an empirical framework with mechanistic support where possible, following established scientific practice of iterative theory development from experimental observations.\n\nWe wholeheartedly agree that our work presents more of an empirical framework than a fully developed theory. However, this approach follows a well-established scientific method, where experimental findings inform theory development through iteration. We hope this work can contribute to starting this process. That said, we do our best to provide mechanistic explanations where evidence supports them:\n\n- **Subjoiner heads in detokenization**: We demonstrate specific attention heads that integrate multi-token words (Figure 6b, Layer 2 Head 5 in Pythia 2.8B).\n- **Prediction/suppression neurons**: We show these occur consistently across all 16 models tested, with clear stage-specific distributions (Figure 8).\n\nUpon your suggestion, we provide more causal evidence related to detokenization and ensembling. \n\n**New causal evidence** (following your suggestion): We conducted additional interventions targeting these specific mechanisms:\n\n- **Detokenization validation**: When subjoiner heads are ablated away, we compare single word tokens and multi word tokens after 18 tokens of context. After ablation, multi-token words lose their characteristic attention patterns and are processed more like single tokens. The attention for single word tokens peaks in earlier layers, but slightly to the right of the subjoiner head (few layers later). \n\n- **Ensembling validation**: Logit Attribution (Direct Effect)  \nFor this we return back to the \"-ing\" task, as presented in the paper in Figure 12.\n\nFirst, if prediction and suppression neurons are truly causal then removing the contribution of prediction neurons, should decrease the logit probability of -ing. The opposite is true for suppression neurons.\n\n**Steering experiment using \"-ing\" task**: We demonstrated direct causal control by modulating prediction/suppression neurons:\n- **Positive control**: Amplifying \"-ing\" prediction neurons increased \"-ing\" logits.\n- **Negative control**: Activating \"-ing\" suppression neurons decreased \"-ing\" logits.\n- **Selective effect**: We also found that there were a few other neurons when modulated, sometimes affected the \"-ing\" distribution. This was however, expected as previously cited works suggest other neuron types – i.e neurons that can act like both prediction and suppression neurons, or \"entropy\" neurons [R1].\nWe will clarify this in our final paper!\n\nRegarding the distinction between general redundancy and computational stages: we discuss \"redundancy\" specifically in the context of prediction ensembling. While ensembling has been studied in ResNets with skip connections, this hasn't been explored in language models through their residual backbones.\n\nTo clarify a potential misunderstanding: we do not claim that model robustness indicates feature engineering. Rather, we observe robustness patterns across layers and propose that the middle layers' robustness aligns with what we hypothesize to be ensemble-like processing. Our section \"Why Are Language Models Robust to Layer-Wise Interventions?\" elaborates on this reasoning. If this connection remains unclear, we would be happy to revise for greater clarity. The distinction is that redundancy in ensembling serves a functional purpose—aggregating multiple prediction signals—rather than mere over-parameterization.\n\n---\n\n## 2. **Stage Boundaries and Universality Claims**\n*Addressing Weakness 2 and Question 2*\n\nOur \"universality\" refers to consistent stage ordering and occurrence patterns across models, not exact locations. We've modified the paper to be more careful and avoid overstatements.\n\nWe acknowledge this could be clearer in our presentation. Our universality claim refers to:\n- Consistent **ordering** of computational stages across models\n- **Occurrence** of the same phenomena (detokenization → feature engineering → ensembling → sharpening)\n- **Not** identical locations or boundaries\n\nGiven different architectures, layer counts, and training procedures, we wouldn't expect formulaic boundary locations. The 30-70% variation you note actually supports our framework—the stages occur reliably but adapt to model-specific constraints.\n\nWe're happy to moderate our language around \"universal\" claims to better reflect this nuanced finding. The excitement lies in discovering common computational patterns across diverse model families, which we believe adds value to the interpretability community.\n\n---\n\n## 3. **Causal Evidence and Interventional Validation**\n*Addressing Weakness 6, Weakness 7, and Question 3*\n\nWe provide new interventional evidence for detokenization and ensembling stages, moving beyond correlational observations.\n\nFollowing your suggestions, we conducted additional experiments providing stronger causal evidence:\n\n**Detokenization interventions:**\nDiscussed above\n\n**Ensembling interventions:**\nDiscussed above\n\n**Model scale considerations:**\nWhile our largest model is 6.9B parameters, we demonstrate consistent patterns across a 55x parameter range (124M to 6.9B). The computational signatures remain stable across this scaling, suggesting the framework may generalize to larger models. We have begun to run experiments on larger Qwen and Llama models and find the results are exactly consistent with our current results.  \n\nNotable features are:\n- Show identical stage patterns but with higher intensity—smaller KL divergence drops but same sensitivity profiles in first/final layers\n- Demonstrate 2-3x higher number of prediction neurons in final layers compared to smaller versions – though maintaining similar density patterns relative to model size\n\n---\n\n## 4. **Predictive Power and Novel Contributions**\n*Addressing Question 4*\n\nWe believe you've accurately recognized the strengths of this paper and write them below for completeness.\n\n**Layer Swapping**: Layer permutivity is an exciting and novel way to study the layers of a network. This wasn't obvious from existing work and provides a new lens for understanding model robustness.\n\n**Aggregating Stage-Specific Intervention Effects**: People across various papers discover unique phenomena, but are rarely synthesized towards a meaningful direction. Stages of processing in language model is often suggested without investigation or evidence. \n\n**Cross-Model Generalization**: While interpretability work often focuses on single models, our framework makes predictions about patterns that should emerge across model families, which we demonstrate empirically.\n\nIt's worth noting that expecting stages of processing in next-token prediction task like language models wasn't entirely obvious given that this is a fundamentally different task from (a) visual processing in the brain (curves in V1 to abstractions in V4) or (b) vision models that more directly mirror human visual systems.\n\n---\nWe're grateful for the reviewer's thoughtful engagement with our work and believe these revisions will significantly strengthen the paper. We look forward to incorporating these insights!\n\n---\n\n[R1] Stolfo et al. (2024) Confidence Regulation Neurons in Language Models."}, {"review_id": "MdMzjCtmqA", "reviewer": "Reviewer_8b7q", "summary": "This work focuses on the robustness of large language models (LLMs) with respect to layer-wise perturbations, such as swapping and deleting adjacent layers during inference. Throughout their analysis, the authors introduce a four-stage framework that aims to characterize the progression of inference in decoder-only LLMs. In the first stage, early layers are involved in detokenization, where local context is integrated and low-level token embeddings are transformed into higher-level representations. In the second stage, feature representations are constructed—tokens are effectively composed into semantically meaningful units such as words or phrases. The third stage marks the beginning of next-token prediction formation, where multiple predictive pathways emerge, contributing to the robustness of this phase by ensembling over several plausible outputs. Finally, the fourth stage, termed residual sharpening, involves pruning away irrelevant or obsolete information in order to refine the model’s final output distribution.\n\nThe work is entirely experimental and heuristic. Initially, the authors employ layer swapping and ablation to study how different layers contribute to model robustness, measured via KL divergence and next-token prediction accuracy. The detokenization stage is inferred based on observed local attention patterns in early layers, suggesting that these layers are responsible for forming short-range token connections and coherent entities. The second stage is supported through probing results on the Word-in-Context (WiC) task, alongside rising entropy, indicating the emergence of meaningful but yet uncertain semantic representations. The third and fourth stages are argued based on the rise and fall of prediction neurons, followed by the emergence of suppression neurons, which correspond to the final pruning and refinement of the model's predictions.", "questions": "Some of my questions are described in the strengths and weakness. I would be happy to discuss those points further.  Additionally: \n\nQ1: Could the authors please explain the motivation and setup for the \"ing\" suffix experiment ? \n\nQ2: For the ensemble method, do the authors only use the suppression and predictive neurons and forgo the rest ? Does it perform better similar to how pruned models at times outperform for downstream tasks ?", "limitations": "None", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "In terms of strengths, the paper is quite intriguing. The idea is novel in how it explains the steps of inference in large language models. Many of the experimental designs used to validate the different phases are creative and could provide a solid foundation for future research.\n\nOn the other hand, the experimental setup is non-standard and quite heuristic. For example, it’s not clear why the sharpening experiments involve repeating layers, while stages two and three are not illustrated using a similar format. I find some setups such as the suffix \"ing\" to be very confusing and without background. Further, I would have appreciated clearer definitions for key terms, such as suppression and predictive neurons, included directly in the main body of the paper. \n\nWhile results are reported for a range of models, I take issue with : (1) for LLaMA, results related to the third phase are not included in the main text (they are only in the appendix, where the correlation appears weaker), and (2) most of the evaluated models are relatively small. I understand the computational cost involved in these experiments, and I don’t fault the authors for relying on heuristics—this is an inherently difficult problem—but given how general the claims are, I would strongly encourage the authors to include at least some analysis on larger LLMs.\n\nTo be clear, I find the work genuinely fascinating. The authors bring together a range of ideas from existing literature and combine them with their own findings to propose a compelling view step by step of inference. I would just like to see more structure in the experimental design and perhaps one or two additional experiments on larger models to fully support the conclusions.", "was_revised": true, "final_justification": "Apologies for the late reply, I thought this justification was just for cases of score change. I have decided to keep my score as I believe the work has merit but requires further justification to consolidate its inference steps as strong model wide bahviours. I will say I do not believe the issue with the work is lack of theoretical analysis in this case.\n\nRE: given the closeness of the score, I have decided to raise my score to a 5 as I believe the work is deserving of presentation.", "rebuttal": "# Response to Reviewer 2\n\nWe thank the reviewer for their enthusiastic feedback! We're delighted that you find our work fascinating and appreciate your recognition of our creative experimental designs and novel approach to explaining inference steps in LLMs.\n\nWe hope to address your concerns about experimental clarity and model scale coverage. We address all points below with additional clarifications and new experimental results.\n\n---\n\n## 1. **Experimental Design and Methodology**\n*Addressing the concern about non-standard and heuristic experimental setup*\n\nYou're absolutely right that our work is non-standard and heuristic. We acknowledge that we don't have a complete mathematical theory describing these phenomena, but we believe this empirical approach provides valuable insights that can motivate future theoretical work, and has its own contribution and value. We discussed that this is likely best left to a separate paper, given we have plenty of plots and experiments already.\n\n**Regarding the sharpening experiment methodology**: The intuition behind repeating layers for sharpening experiments is that if mechanisms in a given layer sharpen the residual stream distribution, repeating that layer should enhance the mechanism's effect. We actually run this experiment across all layers (Figure 12c) and observe the sharpening effect only in specific regions corresponding to our proposed Stage 4. We'll make this clearer in our plots and explanations.\n\n---\n\n## 2. **Clarity and Definitions**\n*Addressing requests for clearer definitions and experiment explanations*\n\nWe've added clear definitions directly to the main body:\n\n**Prediction neurons**: MLP neurons that systematically increase the probability of specific tokens in the output distribution. Identified by analyzing MLP output weights W_out and their projection into vocabulary space via the unembedding matrix W_U, exhibiting high kurtosis and positive skew in their logit effect distribution.\n\n**Suppression neurons**: MLP neurons that systematically decrease the probability of specific tokens. These exhibit high kurtosis and negative skew in their W_U · W_out distribution, effectively pruning unlikely predictions.\n\n$$W_{out} \\in \\mathbb{R}^{d_{mlp} \\times d_{model}}, \\quad W_U \\in \\mathbb{R}^{d_{model} \\times d_{vocab}}, \\quad W_{out} \\cdot W_U \\in \\mathbb{R}^{d_{mlp} \\times d_{vocab}}$$\n\nIn a sense, it is a measure of alignment or cosine similarity between the output MLP and the unembedding of the model. It answers the question, What happens to vocab token at some index, when it exits the MLP? A high and low alignment is a prediction/suppression neuron as measured by the tailed-ness of the row of vocab (kurtosis).\n\n**Q1 - \"-ing\" suffix experiment motivation**: This experiment tests our ensemble hypothesis by examining whether prediction and suppression neurons collectively support next-token prediction. We chose the \"-ing\" suffix because: (1) it's a clear binary classification task, (2) it requires semantic understanding of word formation, and (3) it allows us to test whether neuron ensembles outperform individual neurons or full model averages. We only included instances where the model had to produce \"-ing\" as the complete next token.\n\nWe train probes just on the activation values of individual neurons, but also other probes that are subsets of the neurons. The results show that ensembles of these specialized neurons indeed outperform both individual neurons and model-wide averaging, supporting our Stage 3 framework.\n\n---\n\n## 3. **Model Coverage and Results Presentation**\n*Addressing concerns about LLaMA results placement and model sizes*\n\n**LLaMA results**: You're absolutely right to question our decision to place these in the appendix. We've moved the LLaMA results to the main body (Figure 8b) and enhanced our discussion. The weaker correlation you noted is quite interesting—LLaMA and Qwen show prediction neurons extending into final layers, which we believe reflects their superior performance. Higher-performing models appear to maintain predictive pathways deeper into the network.\n\n**Larger model experiments**: We're running experiments on larger models with promising initial results:\n\n- Show identical stage patterns but with higher intensity—smaller KL divergence drops but same sensitivity profiles in first/final layers. (Greater robustness to layerwise interventions)\n- Demonstrate 2-3x higher number of prediction neurons in final layers compared to smaller versions – though maintaining similar density patterns relative to model size.\n\nThese results strengthen our universality claims and will be included in the revised paper! As we finalize them, we hope to update reviewers here, however they show no significant change to our current results.\n\n---\n\n## 4. **Ensemble Method Details**\n*Addressing Q2 about neuron selection and performance*\n\n**Q2 - Ensemble composition**: Yes, exactly! In our ensemble experiments, we use ONLY the identified prediction and suppression neurons, excluding the remaining ~85-90% of neurons in those layers. This selective ensemble can indeed outperform the full model, particularly for the \"-ing\" classification task where ensemble accuracy exceeds both individual neuron performance and full model top-1 accuracy (Figure 12a).\n\nThis mirrors your observation about pruned models—by focusing on the most relevant computational pathways and reducing noise from less critical components, we achieve better performance. This provides strong interventional evidence that these neuron types are the primary drivers of the computational stages we propose.\n\n**Additional validation**: These neurons are definitionally causal—their mathematical derivation directly measures impact on token probabilities. We ran the following experiment to verify:\n\nFor this we return back to the \"-ing\" task, as presented in the paper in Figure 12.\n\nFirst, if prediction and suppression neurons are truly causal then removing the contribution of prediction neurons, should decrease the logit probability of -ing. The opposite is true for suppression neurons.\n**Logit Attribution (Direct Effect) **  \n**Steering experiment using \"-ing\" task**: We demonstrated direct causal control by modulating prediction/suppression neurons:\n- **Positive control**: Amplifying \"-ing\" prediction neurons increased \"-ing\" logits.\n- **Negative control**: Activating \"-ing\" suppression neurons decreased \"-ing\" logits.\n- **Selective effect**: We also found that there were a few other neurons when modulated, sometimes affected the \"-ing\" distribution. This was however, expected as previously cited works suggest other neuron types – i.e neurons that can act like both prediction and suppression neurons, or \"entropy\" neurons [R1].\nWe will clarify this in our final paper!\n\n---\n\nWe're grateful for your constructive and encouraging feedback! These improvements significantly strengthen our contribution while maintaining the novel insights you found compelling.\n\n---\n\n[R1] Stolfo et al. (2024) Confidence Regulation Neurons in Language Models."}, {"review_id": "47D4I4yDA8", "reviewer": "Reviewer_N4MH", "summary": "The paper studies the robustness of large language models (LLMs) to structural interventions by deleting or swapping adjacent layers during inference. It finds that even without any fine-tuning, the models maintain 72% to 95% of their original top-1 prediction accuracy. Additionally, the authors observe significant differences in performance degradation across layers: interventions on early and late layers cause the most pronounced drops, while the models exhibit notable robustness to deletions in the middle layers. Based on these findings, the authors propose a four-stage inference hypothesis to explain this phenomenon.", "questions": "For detailed comments, please refer to the Weakness section. If the authors could clarify the issues raised therein, I would be happy to consider raising the score.", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. Rigorous and comprehensive experimental design: Experiments were conducted on 16 models of different scales across five model families, providing strong empirical support for the observed high robustness of middle layers to structural interventions.\n2. Novel Stages of Inference Hypothesis: The paper proposes an innovative four-stage theoretical framework that integrates both behavioral and mechanistic evidence, encompassing detokenization, feature engineering, prediction ensembling, and residual sharpening.\n\nWeaknesses:\n\n1. The study of language model architectures is limited to smaller models (<7B parameters) and the specific Pile dataset, which may not represent broader linguistic environments or larger-scale architectures. This limitation may affect the rigor of the conclusions. Further validation on more models around 7B scale and on a wider range of datasets is needed.\n2. Although the paper proposes the Stages of Inference Hypothesis, it lacks more detailed theoretical insights. Is there an intuitive theoretical explanation for the observed high robustness patterns mentioned by the authors?", "was_revised": true, "final_justification": "Although I have raised the score, I still believe the manuscript requires substantial revisions before it can reach an acceptable standard. The increase in score merely reflects my recognition of the author’s efforts during the rebuttal stage.", "rebuttal": "# Response to Reviewer 3\n\nWe thank the reviewer for their feedback and recognition of our rigorous experimental design and the novelty of the proposed stages of inference. We understand your concerns about model scale limitations and theoretical depth, and have conducted substantial additional experiments to address these issues.\n\n---\n\n## 1. **Model Scale and Dataset Limitations**\n*Addressing the concern about <7B parameter models and Pile dataset restrictions*\n\nYou are correct that our original model coverage was limited. We've are expanding our results to larger Qwen and LLaMA models showing consistent stage patterns and promising initial results:\n\n\n**Some initial findings at scale:**\n- Middle-layer robustness becomes even more pronounced in larger models (95%+ accuracy retention vs 85-90% in smaller models) – layer removal or swap is essentially harmless\n- First/last layer sensitivity increases with scale, supporting our detokenization/sharpening framework – though detokenization occupies fewer of the starting layers\n- Prediction/suppression neuron patterns are similar in density – with more neurons in larger models, strengthening our mechanistic evidence\n\nThese results strengthen our universality claims and will be included in the revised paper! As we finalize them, we hope to update reviewers here, however they show no significant change to our current results.\n\n**Dataset generalization**: To address out-of-distribution concerns, we tested on:\nWe ran smaller experiments, but swapping out the Pile for: \n- **Recent ArXiv papers (2024-2025)** to minimize training data contamination\n- **Different domains**: Scientific text, news articles, and code repositories\n- **Multiple languages**: Preliminary results in French and Spanish show similar patterns as our existing results.  This is supported by the phasic patterns seen in multilingual transformers [R1].\n\n\nResults show no significant deviation from our four-stage framework across these diverse settings; all the patterns and features of the plots presented in the work look identical. \n- Attention continues to remain local regardless of language or task\n- WiC probing is still centered \n- Prediction/Suppression neurons (this will remain unchanged since it is data-agnostic) \n\nWe welcome suggestions for additional datasets or methodologies to further validate our findings.\n\n---\n\n## 2. **Theoretical Depth and Robustness Explanation**\n*Addressing the lack of detailed theoretical insights for observed robustness patterns*\n\nWe acknowledge that our work focuses heavily on empirical findings. Putting forth a theory partly takes away from \"hypothesis\" presentation of the paper, which we felt was more honest. We intend to present these findings, and if we made a theory – it likely deserves its own paper with rigorous experiments. Though we would love to hear your thoughts here. We did however try to include concrete mechanistic studies that we made more rigorous. \n\n**Mechanistic foundations**: We identify specific computational mechanisms underlying each stage:\n- **Subjoiner heads in detokenization**: Specific attention heads (e.g., Layer 2 Head 5 in Pythia 2.8B) that integrate multi-token words, confirmed through targeted ablations\n- **Prediction/suppression neurons**: Neurons with characteristic $W_{out} \\cdot W_U$ distributions (high kurtosis, positive/negative skew) that systematically promote or inhibit specific tokens\n\nFollowing suggestions from Reviewer 2, we've added clear mathematical definitions of these neuron types to the main text, which we put here for completeness. \n\n**Prediction neurons**: MLP neurons that systematically increase the probability of specific tokens in the output distribution. Identified by analyzing MLP output weights W_out and their projection into vocabulary space via the unembedding matrix W_U, exhibiting high kurtosis and positive skew in their logit effect distribution. \n\n**Suppression neurons**: MLP neurons that systematically decrease the probability of specific tokens. These exhibit high kurtosis and negative skew in their W_U · W_out distribution, effectively pruning unlikely predictions.\n\n$$W_{out} \\in \\mathbb{R}^{d_{mlp} \\times d_{model}}, \\quad W_U \\in \\mathbb{R}^{d_{model} \\times d_{vocab}}, \\quad W_{out} \\cdot W_U \\in \\mathbb{R}^{d_{mlp} \\times d_{vocab}}$$\n\nIn a sense, it is a measure of alignment or cosine similarity between the output MLP and the unembedding of the model. It answers the question, What happens to a token when it exits the MLP? A high and low alignment is a prediction/suppression neuron as measured by the tailed-ness of the row of vocab (kurtosis). \n\n**Ensemble mechanism**: We theoretically attribute the robustness to ensembling, analogous to computer vision ResNets, but implemented through the transformer's residual stream. The key insight is that multiple computational pathways can contribute to the same output:\n\n- **Residual connections enable substitution**: When layer $i$ is removed, layers $i-1$ and $i+1$ can partially compensate through the residual stream (since many pathways were learned)\n\n**Causal validation**: As suggested by Reviewer 1, we provide interventional evidence on prediction and suppression neurons. Which I will copy here for completeness:\n\nThese neurons are definitionally causal—their mathematical derivation directly measures impact on token probabilities. (see derivation above) We ran the following experiment to verify.\n**Logit Attribution (Direct Effect) **  We return back to the \"-ing\" task, as presented in the paper in Figure 12.\n\nFirst, if prediction and suppression neurons are truly causal then removing the contribution of prediction neurons, should decrease the logit probability of -ing. The opposite is true for suppression neurons.\n\n**Steering experiment using \"-ing\" task**: We demonstrated direct causal control by modulating prediction/suppression neurons:\n- **Positive control**: Amplifying \"-ing\" prediction neurons increased \"-ing\" logits.\n- **Negative control**: Activating \"-ing\" suppression neurons decreased \"-ing\" logits.\n- **Selective effect**: We also found that there were a few other neurons when modulated, sometimes affected the \"-ing\" distribution. This was however, expected as previously cited works suggest other neuron types – i.e neurons that can act like both prediction and suppression neurons, or \"entropy\" neurons [R2]. \nWe will clarify this in our final paper!\n\nThis mechanistic understanding explains why middle layers show robustness—they implement ensemble-like processing where multiple neurons \"vote\" on the output, creating natural redundancy that buffers against individual layer disruptions.\n\nWe hope this substantial additional work addresses your concerns about both experimental scope and theoretical depth, though please let us know if there is more we can do to enhance the work!\n\n---\n\n[R1] Wendler et al. (2024) Do Llamas Work in English?On the Latent Language of Multilingual Transformers.\n[R2] Stolfo et al. (2024) Confidence Regulation Neurons in Language Models."}, {"review_id": "pAIXHBCaAM", "reviewer": "Reviewer_ZYRf", "summary": "This paper aims to resolve a central debate in understanding how Large Language Models (LLMs) function: whether they operate through iterative inference, where each layer gradually refines a prediction, or through a circuit hypothesis, where specialized components perform distinct tasks. The authors argue that these two views are not mutually exclusive. By performing experiments that involved deleting or swapping layers to observe their effects, they propose a unified framework suggesting that a model’s computational strategy changes systematically with its depth, integrating both iterative and specialized processes.\n\nThe authors hypothesize a four-phase framework for computation in decoder-only LLMs. The process begins with (1) detokenization in the early layers, which integrate tokens into coherent entities. Next, the middle layers perform (2) feature engineering, iteratively building complex representations. These features are then converted into next-token predictions during (3) prediction ensembling in the later layers. Finally, the very last layers engage in (4) residual sharpening, refining the final output by suppressing noise. This model provides a depth-aligned structure that bridges top-down and bottom-up views of how LLMs reason.", "questions": "- Testing the Causal Claims of the Framework: The paper proposes a compelling narrative, but the evidence is largely observational and correlational. Could you propose or conduct a more interventional experiment to test the causal role of a specific stage? For example, for “Stage 3: Prediction Ensembling,” would selectively ablating the identified prediction neurons (but leaving suppression neurons and other MLP neurons intact) in mid-to-late layers result in a stalled or reversed decline in logit entropy, as the hypothesis would predict? Demonstrating such a causal link for even one of the stages would significantly strengthen the entire framework. My evaluation would increase if a more direct causal experiment supported one of the stage’s proposed functions.\n\n- Clarifying the “Residual Sharpening” Mechanism: This stage is the most speculative. The evidence relies on the density of suppression neurons and the layer-repeat experiment. However, the MLP output norm also peaks in these final layers (Figure 12b), suggesting a large update to the residual stream. How do you reconcile “sharpening” (which suggests fine-tuning a distribution) with a large-norm update (which suggests a significant change)? Could “sharpening” also be achieved by attention heads in the final layers? Further analysis to disentangle the roles of the MLP and attention in this final stage would be helpful.\n\n- Are the Stage Boundaries Real or Artifacts of Aggregation?: The framework proposes four distinct stages. However, the supporting metrics (e.g., CKA, probe accuracy, neuron density) often show smooth transitions rather than sharp boundaries. How confident are you that these are discrete computational phases versus a continuous evolution of function where the labels are useful but approximate descriptions? For example, is there a token-level phenomenon or a specific task where you can observe a sharp, qualitative shift in processing that aligns with a proposed stage boundary? My evaluation could increase if the authors provide evidence for more discrete transitions, or decrease if the transitions are acknowledged as being purely continuous, which might weaken the “four-stage” claim.\n\n- Alternative Explanations for Robustness: The conclusion hypothesizes that robustness stems from the residual architecture enabling “ensembling.” An alternative view is that middle layers are robust simply because they operate on higher-level, more abstract feature spaces, where individual computations are less impactful than the initial feature-building (Stage 1/2) or final decision-making (Stage 3/4). How does your framework distinguish between these two explanations? Is the evidence for “prediction ensembling” (e.g., redundant neurons) sufficient to favor your hypothesis over a simpler “abstraction level” explanation?", "limitations": "Yes. The authors include a “Limitations and Future Work” section that thoughtfully addresses the primary limitations. They correctly note that the stage boundaries are approximate, that stages may co-occur, and that their framework describes aggregate trends that may not hold for every token. They also acknowledge that they do not isolate the specific architectural or training factors responsible for differences between model families. The discussion of societal impact is also appropriate for this type of foundational research. The section is well-written and demonstrates a clear understanding of the work’s scope.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Significance: The paper addresses a fundamental question in LLM interpretability: reconciling the iterative refinement view with the specialized circuit view. Proposing a comprehensive, testable framework is a valuable contribution to the field.\n- Quality & Clarity: The paper is well-written, clearly structured, and easy to follow. The central hypothesis is presented upfront (Table 1) and the subsequent sections are organized to provide evidence for each of the four proposed stages. The use of multiple analysis techniques (layer interventions, CKA, attention analysis, probing, logit lens) to support a single narrative is a methodological strength, making the overall argument more convincing.\n- Originality: While some individual findings might not be entirely novel (e.g., the fragility of early/late layers), the main contribution is the synthesis of these observations into a coherent, four-stage framework. The systematic comparison between layer swapping and layer deletion across multiple model families is a good experimental design that provides a more nuanced view than ablation studies alone.\n- Empirical Breadth: The authors validate their claims across a diverse set of model families (GPT-2, Pythia, Phi, Llama 3.2, Qwen 2.5) and sizes. This extensive testing strengthens their argument that the observed phenomena are “universal” properties of decoder-only transformers, rather than artifacts of a specific architecture.\n\nWeakness:\n\n- Novelty of Some Findings: The observation that model performance is most sensitive to interventions in the first and last layers is a known phenomenon in the “BERTology” literature and in studies of vision transformers. While the authors use this as a jumping-off point, the paper could do more to position this initial finding relative to prior work to better highlight its own novel contributions.\n- Strength of Evidence for Specific Claims: While the overall framework is compelling, the evidence for some stages feels more “suggestive” than “conclusive.” For instance, the “residual sharpening” stage is primarily supported by the rise of suppression neurons and an experiment involving repeating layers. While interesting, this feels less direct than the evidence for other stages (e.g., local attention for detokenization). The labels for the stages, while intuitive, might be stronger than what the evidence strictly supports. It’s a well-argued hypothesis, but the line between hypothesis and demonstrated conclusion can sometimes feel blurred.\n- Correlation vs. Causation: The paper presents many strong correlations (e.g., the rise of prediction neurons coincides with a drop in KL divergence from the final prediction). However, it is difficult to establish a causal link. For example, does the rise in prediction neurons cause the KL to drop, or are both phenomena downstream effects of a more fundamental, unobserved computational shift? The paper proposes a causal story, but the evidence remains correlational.", "was_revised": true, "final_justification": "I increased my score from 4 to 5 after reading the rebuttal. I think the paper is more an accept than a reject given that now it is now at the borderline.", "rebuttal": "# Response to Reviewer 4\n\nWe thank the reviewer for their feedback. We found it extremely thoughtful and detailed! We're delighted that you found our work addresses a fundamental question in LLM interpretability, is clearly written, and demonstrates strong experimental breadth across model families.\n\nThe concerns raised focus on strengthening causal evidence and clarifying specific mechanisms. We address the points below with new interventional experiments and enhanced mechanistic explanations.\n\n---\n\n## 1. **Positioning Relative to Prior Work**\n*Addressing the concern about novelty of first/last layer sensitivity findings*\n\nYou're right that we should better position our work relative to BERTology and vision transformer literature. We have cited related references in lines [246-250], but we will highlight it in the main text. We agree that this will help us position our novel contribution and the motivating previous works. More concretely, we discussed moving our Related Works earlier on in the paper to better help this narrative.\n\n---\n\n## 2. **Strengthening Causal Evidence**\n*Addressing concerns about correlation vs. causation and testing causal claims*\n\n**New interventional experiments**: Following your excellent suggestion, we conducted targeted interventions to test causal relationships:\n\n**Logit Attribution (Direct Effect)**\nFor this we return back to the \"-ing\" task, as presented in the paper in Figure 12.\n\nFirst, if prediction and suppression neurons are truly causal then removing the contribution of prediction neurons, should decrease the logit probability of -ing. The opposite is true for suppression neurons.\n\n**Steering experiment using \"-ing\" task**: We demonstrated direct causal control by modulating prediction/suppression neurons:\n- **Positive control**: Amplifying \"-ing\" prediction neurons increased \"-ing\" logits.\n- **Negative control**: Activating \"-ing\" suppression neurons decreased \"-ing\" logits.\n- **Selective effect**: We also found that there were a few other neurons when modulated, sometimes affected the \"-ing\" distribution. This was however, expected as previously cited works suggest other neuron types – i.e neurons that can act like both prediction and suppression neurons, or \"entropy\" neurons [R1].\n\n\n---\n\n## 3. **Clarifying Residual Sharpening Mechanism**\n*Addressing the apparent contradiction between \"sharpening\" and large MLP norm updates*\n\nThis is an excellent observation. The large MLP output norm and \"sharpening\" are indeed two sides of the same mechanism:\n\n**Mechanistic explanation**: High-norm MLP outputs in final layers implement **selective suppression** rather than gentle fine-tuning. To verify this, we checked what percentage of the norm the prediction neurons and suppression neurons make in the experiment described above. We find that these neurons contribute to 33% of the MLP of that layer, though it is lower. We also notice other neurons which we do not identify as prediction and suppression neurons -- but produce a large contribution to the token of interest here. We will clarify this nuance in paper, though we suspect it has to do with other types of neurons (see [R1] above).\n\n**Attention vs. MLP roles**: You bring up a great point about the action of attention heads in the final few layers. We confirm that in the final few layers, the attention norm to the residual is significantly smaller than the norm. We plot the ratio of the MLP output norm to the Attention Output norm and find only a monotonic increasing line across over tokens. This is not to say that attention does not play a role in the final layers, just that when looking at things over a corpus of texts we find its effect minimal.\n\nThis supports the ordering of MLP and Attention as presented by previous work [R2], which found lowest loss in configurations of attention first, followed by MLP.\n\n---\n\n## 4. **Stage Boundaries: Discrete vs. Continuous**\n*Addressing whether stage boundaries are real computational phases or aggregation artifacts*\n\nYou raise a crucial point about the nature of our proposed boundaries. We acknowledge these are predominantly continuous transitions with some discrete elements:\n\n**Evidence for continuity**: Most metrics (CKA, probe accuracy) show smooth transitions, suggesting gradual computational evolution rather than sharp phase changes.\n\n**Evidence for discrete elements**: However, we observe some qualitative shifts across stages:\n- **First layer uniqueness**: Complete performance collapse when ablated (unlike any other single layer)\n- **Prediction neuron emergence**: Sharp initial appearance around model midpoint rather than gradual increase from zero. This is consistent regardless of model size and architecture.\n- **Attention locality**: Discrete shift from local to global attention patterns\n\nWe know broadly in machine learning that regions of layers play computational roles. This is commonly seen when studying pruning and model distillation. Another recent example is Meta Perception Encoder – which find that earlier layers are better for downstream computer vision tasks than the final layer [R3].\n\n**Our position**: The four stages represent **functionally distinct computational regimes** that, when examined across distributions, appear discrete but transition smoothly for individual tokens. The labels are useful organizing principles that capture real computational differences, which we believe is highly valuable to even the broader ML community.\n\nWe'll enhance our limitations section to emphasize this nuanced view and avoid overclaiming discrete phase transitions – we want to present an honest look at the work!\n\n---\n\n## 5. **Alternative Explanations for Robustness**\n*Distinguishing ensembling hypothesis from simpler \"abstraction level\" explanations*\n\nThis is a fascinating question that gets to the heart of transformer computation. We believe these explanations are **complementary rather than competing**:\n\nFor completeness, the two views are:\n\n**Abstraction level view**: Early layers build features, middle layers operate on abstract representations, late layers make decisions. Robustness occurs because abstract features are less fragile than raw inputs or final decisions.\n\n**Ensembling mechanism**: Within the abstract feature space, multiple computational pathways achieve similar representational goals. When one pathway is disrupted, others compensate.\n\nThe middle layers are robust *because* they (1) operate on abstract features AND (2) implement multiple redundant pathways to manipulate those features. This resembles an encoder-decoder structure within the decoder-only architecture:\n- **Early layers (encoder-like)**: Compress raw tokens into abstract representations\n- **Middle layers**: Multiple pathways operate on these stable abstractions\n- **Late layers (decoder-like)**: Convert abstractions back to specific token predictions\n\nWe would welcome further discussion of this idea. We did not choose to include this Encoder-Decoder idea in the paper as we felt it required its study in rigor, but this idea is core to what we trying to present.\n\n---\n\n## 6. **Enhanced Mechanistic Definitions**\n\nFollowing suggestions from other reviewers, we've added precise definitions, that hopefully address causal intervention concerns.\n\nThese neurons are definitionally causal—their mathematical derivation directly measures impact on token probabilities.\n\n**Prediction neurons**: MLP neurons that systematically increase specific token probabilities. Identified via $W_{out} \\cdot W_U$ analysis showing high kurtosis and positive skew.\n\n**Suppression neurons**: MLP neurons that systematically decrease specific token probabilities, exhibiting high kurtosis and negative skew.\n\nThe specific calculation is the product of the two matrices:\n\n$$W_{out} \\in \\mathbb{R}^{d_{mlp} \\times d_{model}}, \\quad W_U \\in \\mathbb{R}^{d_{model} \\times d_{vocab}}, \\quad W_{out} \\cdot W_U \\in \\mathbb{R}^{d_{mlp} \\times d_{vocab}}$$\n\nThis projection answers: \"What happens to token i when it exits the MLP?\" Extreme values (high/low kurtosis) identify specialized prediction/suppression functions. It is a feature that comes directly out of the weights, and so over large distributions of text for that token, tokens will \"experience\" promotion or suppression depending on these values.\n\n---\n\nWe believe these additions significantly strengthen our causal claims while maintaining appropriate humility about the boundaries between hypothesis and demonstrated conclusion. The interventional experiments provide the direct evidence you requested for at least one stage, substantially supporting our overall framework.\n\n---\n[R1] Stolfo et al. (2024) Confidence Regulation Neurons in Language Models.\n \n[R2] Press et al. (2019) Improving Transformer Models by Reordering their Sublayers.\n \n[R3] Boyla et al. (2025) Perception Encoder: The best visual embeddings are not at the output of the network."}, {"review_id": "Lhr2u9vjBd", "reviewer": "Reviewer_8AAd", "summary": "This paper investigates the robustness of language models to structural changes such as ablating or transposing adjacent layers. Experiments in the paper demonstrate that middle layers are robust to these layer-wise interventions whereas starting and ending layers are not. The authors then use this experimental evidence to hypothesize that language models operate with respect to four stages of inference: detokenization, feature engineering, prediction ensembling, and residual sharpening; some preliminary evidence is gathered to support this hypothesis.", "questions": "- What additional insight into the stages of inference of language models is gained in performing layer ablations and layer swapping over existing probing methods? \n- Please see my other questions and concerns above.", "limitations": "yes", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 1, "strengths_and_weaknesses": "**Strengths**\n- This paper addresses the issue of interpreting the inner workings of Transformer language models which is an important problem. \n- The results with swapping adjacent layers is interesting and raises questions about the extent to which Transformer operations are commutative. \n- The presentation is generally well done, and the paper is easy to follow. \n\n\n**Major Weaknesses**\n- It is hard for me to pin down the exact contributions of the paper. One one hand, much of the insights into the robustness of language models to layer ablations has already been explored extensively in works like [4, 6, 7]. The relationship between a model's circuit and its representations have also been the subject of extensive study through areas such as representation engineering or steering [8, 9]. On the other hand, the stages of inference hypothesis that the authors propose has been sort of known and also has been extensively explored using probing. \n\n- The models examined in the paper are relatively small. It is unclear how these insights would transfer to much larger models with more complex mechanisms. \n\n- The paper dedicates much writing to the \"stages of inference.\" However, there is little evidence to support this claim, especially on downstream tasks.\n\n- All of the experiments are performed on the Pile which presumably is close to the training corpus of these large models. It is unclear how the results in this paper would transfer to more practical settings for the practitioner like in-context learning, instruction following, finetuning, etc. \n\n\n**Other Weaknesses**\n\n- There are many places in the paper where I believe the literature is misrepresented or need to be discussed in greater detail. \n\n   - > ...mechanistic interpretability, has explored the iterative inference hypothesis...However, this iterative view contrasts with the \"circuit\" hypothesis... (Lines 19-27)\n\n    I do not see how the iterative inference hypothesis contrasts or contradicts the circuit hypothesis. Take for example the work of [1] which discovers \"negative name mover heads\" that also \"iteratively\" add onto the residual stream by suppressing the wrong answer, or the work of [2, 3] which leverage the residual stream to formally discover circuits. \n\n    - > ...Building on these insights, we analyze depth-wise roles and synthesize our findings...to bridge the top-down and bottom-up view of computation in decoder-only LLMs... (Lines 40-44). \n\n     What views are these, why are they different from each other, and why do they need to be bridged? I do not understand how any of the results in the paper bridge any top-down or bottom-up views. Specifically, the paper proceeds in a very bottom-up manner, through data-driven ablations on the model's representations. Then, hypotheses are made about top-level behaviors with little evidence. \n\n    - > Prior work on ablations and layer-wise analysis has primarily focused on BERT... (Lines 246-248)\n\n    In the papers that the authors cite themselves [4, 5] on existing work that performs layer-wise ablations in LMs, large modern decoder-only architectures are studied. For example, [4] examines models like Llama 2 (7b), Baichuan 2 (7b), while [5] looks at Llama 2 (70b). Other than the stages of inference hypothesis, what are the differences between the authors' findings and the findings of these papers?\n\n\n[1] Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeries, Jacob Steinhardt. ICLR 2023.\n\n[2] A Mathematical Framework for Transformer Circuits. Nelson Elhage and others. Transformer Circuits Thread 2021. \n\n[3] Finding Transformer Circuits with Edge Pruning. Adithya Bhaskar, Alexander Wettig, Dan Friedman, Danqi Chen. NeurIPS 2024. \n\n[4] The Unreasonable Ineffectiveness of the Deeper Layers. Andrey Gromov, Kushal Tirusamla, Hassan Shapourian, Paolo Glorioso, Daniel Roberts. ICLR 2025. \n\n[5] ShortGPT: Layers in Large Language Models are More Redundant than You Expect. Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, Weipeng Chen. \n\n[6] Eliciting latent predictions from transformers with the tuned lens. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, Jacob Steinhardt. \n\n[7] Interpreting GPT: The Logit Lens. nostalgebraist. Less Wrong 2020. \n\n[8] Representation Engineering: A Top-Down Approach to AI Transparency. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, and others. \n\n[9] Function vectors in large language models. Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, David Bau. ICLR 2024.", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 5\n\nWe thank the reviewer for taking time to review our work. We appreciate that you found our work poses an important question and that the results were well presented.\n\nThe major concerns appear to be about the novelty of our work and the presentation of previous works. We hope to address this by clarifying that our contribution is the unified framework rather than individual ablation experiments, and how relevant previous works connect to our approach.\n\n---\n\n## 1. **Clarifying Our Contributions**\n*Addressing concerns about unclear contributions and relationship to prior work*\n\n**Regarding contribution clarity**: We acknowledge that robustness to layer ablations has been studied previously, as we cited in the relevant works. As other reviewers have noted, we use the swapping/ablation experiments as a starting point for proposing our framework (stages of inference) rather than as the main contribution of the paper. As we mentioned in lines [19-24], there are extensive studies on model circuitry. Our contribution is unifying those scattered observations into one framework, as stated in lines [42-44]. The motivations and ingredients come from previous works (such as the existence of prediction neurons), but our experiments are novel in being designed to support our unified framework.\n\n**On stages being \"sort of known\"**: In our knowledge, our work is the first to unify scattered observations from different contexts through systematic experiments and put them into one coherent framework. If you know of other relevant work that does this, we're happy to review the comparative significance and cite accordingly. Other works tend to present these ideas in passing, though it's often speculated about.\n\nWe will add a paragraph at the end of the introduction clearly stating what are the motivating works from previous literature and what is our novel contribution.\n\n---\n\n## 2. **Literature Representation and Positioning**\n*Addressing concerns about iterative vs. circuit hypotheses and related work*\n\n**Regarding iterative vs. circuit views**: This is a great point! In iterative inference, each subsequent layer orients the residual stream toward decreasing loss. The absence of a component isn't necessarily problematic since there's overlapping computation as a byproduct of the residual stream and gradients.\n\nThe circuit hypothesis might have a token operated on at layer i, then experience no meaningful contribution for subsequent layers, then be affected by layer j. You would claim that components in layer i and j are part of the circuit. Many times ablating layer j will hurt the model, other times there will be a backup circuit.\n\nWe see both mechanisms exist and believe they are distinct. By saying they \"contrast,\" we're trying to say they are functionally different. We're frankly not sure why they arise - this may be a feature emergent from data distributions and gradient descent, but it's certainly an important question in interpretability. We will update our work to reflect this line of thinking - we have no desire to misrepresent!\n\n**Regarding top-down vs. bottom-up**: Mechanistic interpretability work often focuses on small mechanisms but doesn't connect to the broader picture. We intended to show that these mechanisms (subjoiner heads, feature neurons, neuron types) are not individual, but contribute to a larger \"biology\" of the model. Moreover, these features are not coincidences. Had this mechanism been for just a single model, I would completely agree. However, consistency across model families and sizes points to a broader phenomenon.\n\n---\n\n## 3. **Addressing Prior Work on Decoder-Only Models**\n*Responding to concerns about citing [4,5] and differences from existing ablation work*\n\nYou're right - we misstated the focus on BERT in that line. As you noted, we cite works [4,5] that study modern decoder-only architectures extensively.\n\nAs we noted, we're not considering ablation/swapping experiments as the main contribution of this work, but rather a starting point for our main research question. We will make this clear in the abstract, introduction, and relevant works sections. As pointed out by many of the other reviewers, our framework synthesizes these robustness findings with mechanistic evidence (attention patterns, neuron types, probing results) in a way that previous ablation studies haven't attempted.\n\n---\n\n## 4. **Model Scale and Practical Settings**\n*Addressing concerns about small models and dataset limitations*\n\n**Larger models**: We acknowledge your concern about model scale. We're happy to share additional results on larger models as compute allows. We've expanded to larger Qwen and LLaMA models showing consistent stage patterns, and will gladly enter them into the final version of the paper.\n\n**Dataset generalization**: We performed experiments on the Pile since we wanted to separate questions about out-of-distribution behavior. However, to address your concern (and our curiosity) we ran tests to ensure our results hold in different settings:\n\n- **Recent ArXiv papers (2024-2025)** to minimize training data contamination\n- **Different domains**: Scientific text, news articles, and code repositories  \n- **Multiple languages**: Preliminary results in French and Spanish show similar patterns\n\nResults show no significant deviation from our four-stage framework across these diverse settings.\n\n**Practical applications**: \n\nWe agree it's an exciting future direction to extend insights from our framework to more naturalistic setups like reasoning, as we mentioned in our future work section.  Nevertheless, we believe our work has significance in grounding the language model's fundamental inference pipeline throughout its multiple layers, agnostic to the specificity of the downstream tasks. \n\n---\n## 5. **Evidence for Stages**\n*Addressing concerns about limited evidence for stages of inference*\n\nOur evidence comes from converging findings across multiple methodologies:\n- **Layer interventions** showing consistent robustness patterns\n- **Attention analysis** revealing locality gradients\n- **Probing experiments** (WiC task) showing semantic feature peaks in middle layers\n- **Neuron classification** identifying prediction/suppression patterns\n\nWe believe that the consistency of these patterns across 16 models and 5 architectures provides substantial evidence for our proposed stages, and has not been done by a paper before. We believe our framework provides a valuable organizing principle for understanding transformer computation, synthesizing scattered mechanistic findings into a coherent, testable hypothesis about inference stages."}]

Author rebuttal:

Meta-review: - Scientific Claims and Findings: The paper investigates LLM robustness through layer deletion and swapping experiments, finding models retain 72-95% accuracy without fine-tuning. Authors observe that early and final layers are most sensitive while middle layers show remarkable robustness, leading to a proposed four-stage inference framework.

- Strengths: The systematic intervention methodology using novel layer swapping provides unique insights into transformer robustness across multiple model families. The work synthesizes scattered mechanistic interpretability findings into a coherent, testable framework using converging methodologies e.g. interventions, attention analysis, probing.

- Weaknesses: The framework represents post-hoc pattern matching rather than principled theory, with significant variation in stage boundaries undermining universality claims. Evidence remains largely correlational rather than causal, limited to models <7B parameters with unclear generalization to larger models.

- Rebuttal Discussion and Changes: Reviewers raised concerns about theoretical foundation (BKjM), experimental clarity (8b7q), model scale (N4MH), causal evidence (ZYRf), and novelty (8AAd). Authors conducted substantial additional experiments including causal neuron interventions and validation on >10B models. These improvements led to some score increases, while others maintained scores acknowledging improvements but noting remaining concerns.

- Decision Rationale: The AC recommends the paper to be accepted. Despite theoretical limitations, the systematic experimental approach and comprehensive validation provide valuable contributions to LLM interpretability. The organizing framework synthesizes existing findings into testable hypotheses, with creative methodology outweighing theoretical weaknesses for this foundational empirical work.
Decision: Accept (poster)