LLM Layers Immediately Correct Each Other
Authors: Arjun Patrawala
Jiahai Feng
Erik Jones
Jacob Steinhardt
Venue: NeurIPS 2025 poster ()
Keywords: mechanistic interpretability
correction
self-repair
Primary area: deep_learning
TL;DR: We identify and explore a prominent layer-level correction mechanism found in a variety of open-weight LLMs

Abstract: Recent methods in language model interpretability employ techniques such as sparse autoencoders to decompose residual stream contributions into linear, semantically-meaningful features. Our work demonstrates that an underlying assumption of these methods—that residual stream contributions build additively upon each other—is insufficient to fully explain model behavior. Specifically, we identify the Transformer Layer Correction Mechanism (TLCM), wherein adjacent transformer layers systematically counteract each other's contributions to the residual stream. TLCM appears in 5 out of 7 major open-source model families and activates across nearly all tokens in diverse texts.
To understand TLCM, we show that it emerges during pretraining, operates most strongly on punctuation and numbers, and adaptively calibrates its correction strength based on the preceding layer's output. We further show that TLCM actively corrects a small subspace and promotes other subspaces, different from standard model behavior. We advance the ``propose-and-reject'' hypothesis: layers may propose multiple candidate features, while subsequent layers selectively filter out inappropriate ones. Finally, we discuss how our findings help explain three persistent challenges in feature-based interpretability: why extracted features descriptions often suffer from low specificity; why feature-based interventions for model steering fail at low magnitude; why recent work finds cross-layer transcoders outperform SAEs.

LLM Layers Immediately Correct Each Other
Arjun Patrawala
Jiahai Feng
Erik Jones
Jacob Steinhardt
University of California, Berkeley
arjuunatrraalld@brrrllll..eee edu
Abstract
Recent methods in language model interpretability employ techniques such sparse autoencoders to decompose residual stream contributions into linear, semantically mmennnngf- features. Our work demonstrates that an underlying as- sumption of these methods thhtt residual stream contributions build additively upon each other-is insufficient fully explain model behavior. Specifically, we identify the Transformer Layer Correction Mechanism (TLCM), wherein adja- cent transformer layers systematically counteract each other's contributions to the residual stream. TLCM appears in 5 out of 7 major open-source model families and activates across nearly all tokens diverse texts. To understand TLCM, we show that it emerges during pretraining, operates most strongly punctuation and numbers, and adaptively calibrates its correction strength based on the preceding layer's 's output. We further show that TLCM actively corrects small subspace and promotes other subspaces, different from standard model behavior. We advance the "propose-and" and-reecct"" hypothesis: layers may propose multiple candidate features, while subsequent layers selectively filter out inappropriate ones. Finally, we discuss how findings help explain three persistent challenges in feature-based inter- pretability: why extracted features descriptions often suffer from low specificity; why feature-based interventions for model steering fail low magnitude; why recent work finds cross-layer transcoders outperform SAEs.
Introduction
Mechanistic interpretability aims to understand large language models (LLMs) by dissecting the functions of their components. This research direction has critical implications for monitoring and controlling language models 22
42 27
39, as well as designing architectural improvements
51 50
foundational assumption in many interpretability methods that transformer layers progressively build upon each other's contributions to enrich representations the residual stream. This perspective has motivated numerous techniques that extract features from transformer layer outputs, including linear probes 3. the logit lens (37), sparse autoencoders (SAEs) 6 and cross-layer transcoders
(CLTs) 4 12 40). Similarly. analyses of factual recall often characterize successive layers as
gradually augmenting entity representations with additional recalled information [[6 33 36 21].
In this work, we introduce the Transformer Layer Correction Mechanism (TLCM), in which adjacent transformer layers systematically reverse portions of each other's contributions. Specifically, find that in 5 out of 7 open-weigtt weight LLM families (Llama 3, OLMo, Mistral, Gemma, and Qwen2), layer & 1 consistently produces contributions that partially oppose those of layer TLCM challenges the conventional view that layer contributions primarily add information to the residual stream; instead, layers actively edit the residual stream by selectively promoting and rejecting components
Code available htw.la/ww..rppe.........oooooear
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

from previous layers. provide an incomplete or misleading understanding of a layer's function, as these features may be subsequently corrected.
This suggests that analyzing output features of each layer independently may
To begin, we characterize TLCM through series of observational experiments in Section 4 First, show that TLCM is not present at initialization but emerges gradually during pretraining, suggesting that it is a persistent characteristic of LLM training dynamics. Second, we find that TLCM fires most frequently on tokens with high contextual dependency, including numbers, dates, and punctuation. We hypothesize that TLCM is important for handling tokens with high contextual dependency. Finally, we show that TLCM is the combined effort of both attention and MLP.
In Section we focus our experiments TLCM's underlying mechanisms. First, we causal interventions to show that TLCM is adaptive and directly dependent on the prior layer, calibrating its correction strength based the scale of the preceding layer's output. Then, demonstrate that TLCM systematically corrects and reinforeing specific components of the previous layer by showing an empirical relationship between the eigenvectors of the transformer layer's Jacobian and the corrected subspace. Finally, we synthesize our findings into the "propose-and-rejec hypothesis", which states that models can perform enrichment though a process of proposing multiple potential features and correcting the inappropriate ones.
Our findings are helpful for understanding three persistent challenges in feature-based interpretability: (1) why extracted features descriptions often suffer from low specificity; (2) why model steering in- terventions require high amplification to be effective; and (3) why cross-layer transcoders outperform sparse autoencoders in recovering interpretable features.
We hope our work establishes a corpus of well-characterized phenomena that provides empirical constraints for theories of language model interpretability, and that our findings inform stronger alignment and model steering techniques.
2 Related Work
Feature-based interpretability. A common interpretability paradigm interprets the role of a layer in neural network as the features present in its output. The linear representation hypothesis posits that
neural networks represent concepts as linear features in their activation space 35 388 Feature-based interpretability extracts features using supervised linear probes techniques that can be unsupervised and/or causal (c.g. nostalgebraist al. [15], Bricken al. (6).
3 as well other more sophisticated
37 Burns et al. [8) Geiger
Correction mechanisms, Some recent works have found self-repair and correction exists in models, in which if components in large language models ablated, later components will change their Self-repair mechanisms have been discovered in varied places: ablations of attention layers with later compensation [25] 32] resilience to swapping transformer layers (25), among others. Other research has found mechanisms that are conjectured to help with self-repair, like copy suppression in which an attention head will decrease the probability of predicting token that has already appeared in the context [3T. TLCM is more pervasive (occurring multiple times on nearly all tokens) and occurs during normal model operation, not specifically during ablations.
behavior to compensate 443
Residual stream characterization. The Iterative Inference Hypothesis proposes that transformers progressively refine their latent representations through successive layers 77]. Other work builds methods that uncovers semantically- -eeticallly-mmannfffllll latents found in the residual stream [ Another line of work highlights the process of enrichment: transformer layers construct enriched representations in the residual stream to perform next-token prediction. For example, function vectors encode input-output functions and are placed in the residual stream induce execution (8). Additionally, models proactively consolidate entity-related information into the residual stream before it becomes
relevant for prediction 21 26 20
T6
19
3 Background
Transformer notation. Large language models (LLMs) convert token sequence into a probability distribution over subsequent tokens. Input tokens are first embedded in the d, di--dimensional residual space with positional embeddings, then passed through transformer layers before being unembedded

---

Owen
165
2
Figure I: We plot clamp(M, -0.2, 0.2) for four models, zeroing diagonal entries. TLCM is visible on the left two plots, characterized by off-diagonal blue stripes of negative cosine similarity, concentrated in the first two-thirds of the model. In contrast, TLCM is not present on the right two plots.
into token logits.
X (tos)
Xi+1 (x.)
logits = Unembed(x.)
Each transformer layer contains attention and MLP sublayers that produce "contributions" to the residual stream:
x = x + (xt)
x x MLP
where x/ an intermediate state. The marginal contribution layer defined as c; Attnj(x:) MLPI(xo + (x,), making X,+1 layer on token
x; + C₁. We define to be the contribution of the ith
Feature-based interpretability. Recent interpretability methods decompose residual stream con- tributions using sparse autoencoders (SAEs) extract meaningful features for residual stream contribution c 14 47):
k
c=c+ B.V.
where e is the error, is a small integer (usually less than 500), and v; are the orthogonal feature vectors with activations B,. These features can labeled with semantic meanings and manipulated at inference time to steer model behavior. 30 47 9..
4 Transformer Layer Correction Mechanism
In this section, we introduce the Transformer Layer Correction Mechanism and conduct a series of observational studies to characterize its functioning. We first define it (Section 4.1. show that develops during training (Section 4.2). demonstrate how it varies across token types (Section 4.3)
and finally demonstrate it occurs via collaboration of both attention and MLP sublayers (Section 4.4
4.1 Uncovering the Traces
We motivate the discovery of TLCM through hypothesis about how transformer layers collaborate to enrich the residual stream. Specifically, we hypothesize that layers operate two distinct modes: (I) contributing novel information the residual stream, or (2) reinforcing existing information.
To test this hypothesis, we quantify layer interactions using cosine similarity between their contri- butions. High positive cosine similarity indicates that two layers reinforce similar representations, near-zero similarity suggests their contributions are largely disjoint, and negative cosine similarity implies one layer reverses corrects the other's contribution. Consistent with our hypothesis, recent work has found positive cosine similarity between layer contributions in ViTs 24).
Setup. We test this across several open-weight language models. For a given token I, we define the similarity matrix M,, where C₁, represents the contribution of the i-th transformer layer (as defined in equation 3):
M == cossim Cj,e) for i +
We then average these matrices element-wise across approximately 100,000 tokens, across ran- dom documents WikiText (34] using HuggingFace Transformers 49 M 1 Σ, M,. These documents include variety of languages and code.

---

Results. Plotted in Figure 1 M reveals a reversing effect, captured by the following observations:
Across this large corpus, adjacent layers (layer and i + 1) on average have opposing contributions, evidenced by their negative cosine similarity which averages R -0.2.
Curiously, non-adjacent layers (layer and > 1) do not on average have opposing contributions; their contributions are predominately orthogonal positively correlated, consonant with our hypothesis above.
We term this phenomenon the systematic partial reversal of layer i by layer i 1-the the Transformer Layer Correction Mechanism (TLCM). We see TL.C across diverse set of model families including Llama 17. OLMo [8] Mistral [2 23 Gemma 45 Ld. and Qwen2 [52]. plotted in Figure
aand Appendix D TLCM persists across different text types, model scales, and model categories (instruction-tunee and conversational). The correction mechanism most pronounced in the first two-thirds of each model's layers. Notably, TLCM is absent in two prominent model families: GPT-2 [1 and the Microsoft Phi models [1. as shown in Figure I This absence may stem from their architectures Phiie,, while based on Llama 2, incorporates dropout blocks after MLP and attention sublayers, similar to GPT-2's ' design. For the purposes of this paper, we study the TLCM in Llama 3.1 8B (dm. 4096).
We plot the distribution of adjacent layer cosine similarity Figure 2b which reveals a strongly bimodal distribution. This suggests that TLCM is separate mode of operation, rather than an idiosyncrasy of normal operation. Based on this plot, we define TLCM as adjacent layer contributions with cosine similarity below -..1.
Discussion. The presence of highly anti-correlated vectors in such high-dimensional space is striking. To provide a sense of how unlikely that the two outputs of adjacent layers exhibit TLCM (i.e. have cosine similarity less than -0.1), we estimate the probability of this happening under two different null hypotheses in which the outputs are sampled randomly instead.
First, suppose the two outputs drawn independently from standard d-dimensional normal distribution. Then, two random vectors in d-dimensional space have expected cosine similarity 0 and variance 1 /d (A (Appendix B.1) Therefore, as d scales to 4096 (Llama 8B 3.1), the probability that TLCM occurs for two adjacent layers as rare as a 6-sigma event.
Second, now suppose that the two outputs are drawn independently from empirical distributions of their respective layers. Compared to the earlier normally distributed setting, this accounts for the possibility that the output space of the layers could be low rank simply be pointing in opposing directions (see Appendix B.2). We this distribution by calculating the empirical mean and standard deviation of cosine similarities between contributions from layers i and large documents. Specifically, we sample from the same layer ranges where TLCM most active (4 S < 20) but adjacent contributions from different tokens. We find that a typical instance of TLCM is approximately as rare as a 3-sigma event. Therefore, TLCM- -ccurring multiple times on hundreds of thousands of tokens is exceedingly unlikely arise from chance alone. There must be a meaningful relationship between the two layers, either through a common confounder or direct dependence. In Section 5.1 adjacent layers.
across four
we show that this relationship is indeed direct dependence between
Intuitively, transformer layers may erase information from the residual stream for various reasons: information may be only transiently useful, or deletion may free capacity for new information. Under either explanation, information written in layer i could be deleted in any layer j where j > i. However, TLCM is more constrained: reversal occurs predominantly in the directly subsequent layer. This pattern is puzzling from efficiency standpoint, implies written information can only be exploited by a single downstream layer before erasure.
4.2 TLCM Develops During Training
We next investigate how TL.CM emerges over training. In particular, TLCM could be:
An inherent characteristic of transformer architectures, emerging from their fundamental
design rather than through learning.
A transient phenomenon that arises due to, for example, unstable training dynamics.
A mechanism developed during post-training, either by RL or SFT.

---

Mam
MMLP
Aug TLCM Actiaatone
31.5
11.3
11.0
Count
11.3
2010
21
Position
(a) Observe the correlations between attention sub- layers, plotted as clip(Mam, -0.3, 0.3) the in the plot of
(b) The left depicts the relationship between average TLCM activations and token position. The right il- lustrates the distribution of adjacent transformer layer similarities, with the dotted red line cossim -0.1.
left. Traces of TLCM
clip(MMLr, -0.,, 0.3) on the right.
Figure 2
Setup. To address these possibilities, study how TLCM develops throughout the training process. We examine seven checkpoints from each of three fully open-source models: OLMo 1B, OLMo IB, and OL.o 7B. For each checkpoint, we compute M Σ, M, using standardized block of text.
Results. We find that TLCM emerges over the course of pretraining; in OLMo 7B, for example, it first manifests at training step 135,500 (approximately % through the total 651,581 steps), corresponding to roughly 0.5 trillion processed tokens. Appendix Figures A9 A10 and A1l demonstrate that the mechanism's 's strength increases markedly during the latter two-thirds of training for all three models. This suggests that TLCM is
charning-inducedd TLCM does appear in untrained models. Persistent: Despite weight-decay, TLCM strengthens over training and is unlikely to be some training pathology.
Pretruining-derived: TLCM emerges during pretraining, not SFT or RL.
4.3 TLCM Varies by Token
We next study how TLCM activations vary across different token classes.
Setup. We curate 100 realistic queries requesting long-form content about science, technology, philosophy, government, history, and other topics. We then generate responses to these queries using Llama 3.1 8B. For each token in the response, count the number of corrections for as: Hi cossim( C2+,r) < -0.1 This corpus contains 79k tokens. To ensure we identify only systematic patterns, we filter tokens appearing fewer than 20 times and compute the mean number of TLCM activations for each unique token.
Results. We observe significant variation in correction frequencies across tokens Llama 3.1 8B; some tokens had few corrections on average, while others had many more. Some qualitative examples of low-correction tokens (< corrections per token average) include:
By, workshops, indigenous, vinyl, implications, cognitive, -being, innovative, innovation, regulatory, greenhouse, mindfulness, platforms, trends, therapy, example, learning, iving, planning, inclusive, cloud, classical, proposal, -friendly, sustainability, biodiversity, ting, blog, memo, system, <lbegin_off t
And some examples of high-correction tokens (> corrections per token on average):
202, Jul, 26, n', A.n),. Today, nn S, Name, State, [, 201, assistant, Address, \tr 4, ),, user, at, I',),, Date, Title, J. -, Your, over, Date, %, high, l.. n' reach, up, share, )** well, one, D. time, take, forward, from, 1, :, access, you, 12, not, [, I. City, need, low, Thank, make, 10, look, your, 0, such, *
We list token-level TLCM activation statistics (mean and standard deviations) in Table Al and A2 We also compute results for Gemma 2B Instruct across the same 79K tokens, listing full results in Table A3 and A4
5

---

We find statistically significant differences in the frequency of TLCM activations. For example, the token An averages 12.41 (=0.1 99% CI) TLCM activations while sustainability averages 7.63 (+0.46 99% CT) on Llama 3.1 8B. Numbers, punctuation, dates, and brackets rank highly for frequent TLCM activations, while standard English terms like community, training, and understanding have less frequent activations. Notably, Llama's < <lbegin_ so text >> token and Gemma's <bos> token exhibit the fewest average TLCM activations across the entire corpus.
Discussion. We observe that tokens exhibiting low TLCM correction rates typically possess un- ambiguous, context-independee interpretation regardless of their surrounding context.
semantic meaning. These tokens generally maintain consistent
In contrast, high-correction tokens seem demonstrate greater average contextual dependency. This category includes: 1) Numbers that form larger parts of numbers, where grasping the complete value of the number requires consolidating information across tokens 2) Date-related tokens which require surrounding context for complete temporal reference 3) Punctuation marks (e.g., ':") whose semantic role varies with usage context.
Recent work shows (28 that newline tokens can be used for planning upcoming tokens, a task that requires contextual poceesing-newling tokens and similar (An \n, An, etc.) have high rates of TLCM activation. Intuitively, models may aggregate information into concluding punctuation marks because the causal attention mask prevents them from aggregating information in earlier tokens.
Llama's < //begin_ __f________________ text >> token and Gemma's <bos> are particularly revealing, with 0 and TLCM activations respectively. These tokens are always first in the context window during training. In this position, correction unnecessary; the optimal next-token prediction with no context is simply the empirical unigram distribution from the start of training documents, which we suspect obviates complex contextual processing.
To test this conjecture-thal TLCM activations relate to contextual processing perform simple check: we plot the average number of TLCM activations per token at different points in the LLM context window; Intuitively, the 1000th token can attend to 999 preceding tokens while the 5th token can only attend 4, suggesting we should observe higher TLCM activation rates at later positions. Indeed, across 2000 long WikiTText documents, we find that every 1000 tokens of additional context corresponds to approximately additional TL.CM activation, as demonstrated in Figure 2h
4.4 Attention and MLPs Alone Do Not Explain TLCM
We next investigate the role of MLP and attention sublayers in TLCM. For instance, TLCM could arise purely from MLP-to-MLP interactions, or MLPs could selectively reverse only attention sublayer contributions. To address these questions, we conduct an analysis similar to Section 4.1 but at the sublayer level.
Setup. We compute a similarity matrix, instead using the attention contribution Mam [V, cossim( Attn,(x..), Attn,(X,.)) and plot the average matrix Matn = 1 E, Mann.- We also plot the average matrix for the MLPs: MMLP = Σ, MMLP, with MM cossim (X,,)
Results. We identify three key findings. First, attention sublayers produce positively correlated contributions with other attention sublayers (Figure 2a). Second, MLP sublayers produce anti- correlated contributions with the subsequent layer's 's MLP, exhibiting similar pattern to TLCM. Third, MLP contributions are also anti-correlated with the preceding attention sublayer (Appendix D. In summary, MLPs produce contributions that are anti-correlated with both the preceding MLP and attention sublayers, suggesting that MLPs are primarily responsible for executing corrections.
Since MLPs partially reverse contributions from both the preceding attention and MLP sublayers, the correction mechanism appears to be a transformer layer-level phenomenon. Additionally, our prior experiments suggest contextual dependency plays role in TLCM's functioning, something only possible if TL.CM operates partially through the attention sublayer. For these reasons, we focus study of TLCM at the transformer layer level.

---

Layer
03-
cossinia b((oa.))
0.-
-07
Figure 3: TLCM exhibits a linear relationship between its correction strength and the prior layer's contribution, suggesting that systematic attenuation of the prior layer, shown on 5 randomly sampled tokens. For negative values, layers demonstrate compensatory behavior, evidenced by positive cosine similarity.
5 TLCM Adaptivity
In this section, investigate TLCM's underlying mechanistic functioning. In Section 5.1 show that TLCM is adaptive and directly dependent on the preceding layer's outputs. Then, in Section 5.2 illustrate connection between the transformer layer's Jacobian and the subspaces targeted for correction. Using this connection, we show that TL.CM selectively targets subspaces for correction, while ignoring or promoting the others.
Finally, we argue for the propose-and-reeee hypothesis, which explains the process enrichment as proposing, checking, and rejecting features in 2-layer sequence; in this picture, TLCM corrects an "undesirable" component of the previous layer.
Notation. For particular token t, consider the marginal contribution of layer i and layer i + 1 to be a, and b,, respectively. Recall that the correction mechanism is currently described by a negative cosine similarity between and b, across variety of layers and tokens. Additionally, observe that the d., dimennsiioi input to layer i would be X,, X;-1,t + a,. We define a function by that captures contribution of layer + when the previous layer's output is perturbed by A. Specifically:
(A Layer,41 A)
Note that (0) b,, i.e. the original layer contribution.
5.1 TLCM is Adaptive
argue that the anticorrelation between the contributions of adjacent transformer layers cannot be explained by randomness- either common underlying cause or direct dependence between the layers.
In Section 4.1
and i + anti-alignment must be explained by
Here, we perform a causal intervention to show that the anti-alignment is caused by direct depen- dence between the layers. In particular, find that TLCM's correction is adaptive: scale layer i's contribution, layer + scales its correction, demonstrating causality.
Setup. To understand whether layer + 1 is adaptively correcting the contribution of layer we add some perturbation A oa,, for a E [, 1] to the input of layer + Formally, the input to layer i + 1 is intervened to become
X;-1, a ca; X4-11 (1+00 a)a;
If layer i + 1 is attuned correcting a,, should expect to see the correction increase is scaled up from 0 and decrease as decreased from 0. To this end, we measure the cosine similarity between a and (aa;) at {-1, -0.9. 0.9, 1}.
7

---

TLCM Jocabian Eigenvalues
175-
100
Count
Count
120-
200
107
-2
-2
-1
A
X
(a) Nearly 3/4 the 4096 eigenval- ues of the Jacobian J are negative.
(b) Left: Strongest components of at are corrected or reinforced by the TLCM Jacobian. Right: For non-TLCM Jacobians, a, 's strongest components are predominantly untouched.
Figure 4
Results. As expected, increasing leads to stronger correction. Decreasing leads to less correction. Figure 3 depicts this near-linear relationship.
When & 1 effectiveevy doubling in the residual stream dtream-layer layer + responds by adjusting its contribution to further oppose a, The smooth adaptation we observe indicates layer + 1's dynamic regulation of the previous layer's contribution. This pattern appears consistently across all layers where TL.C is active, with complete results presented Appendix
Our experiments also reveal compensatory effect. Specifically, in Figure at a = -1, the cosine similarity between the adjacent layers becomes positive; layer 1 effectively boosts the contribution of layer i. This effect occurs most strongly in earlier layers. This finding aligns with recent research on self-repair mechanisms that enable models to recover from interventions [2] However, maintain cautious interpretation: extreme values of a place the model in counterfactual states that do not appear during standard inference or training. Moreover, this compensation effect may stem from self-repair mechanisms unrelated to TLCM.
Discussion. We have demonstrated that, when TLCM is active, there is causal relationship between layer and layer + I's correction. Adjusting layer i's contribution elicits an immediate response from layer i
This finding is striking for a subtle reason. For layer + 1 to respond sensatively to layer i, it must identify which component of the residual stream was added by layer i versus components contributed by earlier layers. This a challenging task; layer + 1 would need sophisticated mechanisms to isolate layer i's specific contribution. We therefore consider an alternative hypothesis: Layer i's contribution might comprise both a "desirable" and "undesirable" component; layer + 1 simply targets the latter for correction. For instance, layer might propose features to the residual stream, after which layer + verifies and removes only incorrect features while leaving the rest intact. This naturally motivates questions:
Does TLCM correct the entire previous layer contribution, or does it target specific compo- nents?
2. If TLCM correction selective, can we identify which subspaces are targeted most aggres- sively?
5.2 Isolating the Correction Subspace
In this section, we execute experiment answer the prior two questions. A key unknown is how the previous layer's 's contribution is attenuated: is it scaled back uniformly, or are specific subspaces selectively targeted for correction?
The layer Jacobian-the local linearization the both the attention and MLP sublayers-contalle information about how the entire layer will respond to input perturbations. By developing tools to characterize and visualize the Jacobian, we can understand which vector directions will be aggressively corrected by the layer versus which directions will remain untouched.

---

Notation. As previously defined, consider by to be the function representing transformer layer i 1. We can linearize this function by computing the Jacobian (b) = E trink d of the transformer layer, such that ( = b, + JA. Finally, let == 2 [J + J J be the symmetrized Jacobian.
Claim. The eigenvectors of J with negative eigenvalues correspond to the corrected subspace, with the eigenvalue dictating the correction strength. In similar vein, argue that the positive eigenvalue cigenvectors are reinforced proportional to their cigenvalue. Finally, eigenvectors with near zero cigenvalue are mostly untouched by this layer.
Proof. An input perturbation to layer + 1 "corrected" if cossim(b;(4)) b, A) < Observe that this condition is true if and only if (A) b A) < O. For small enough perturbation, the condition equivalent (JA, A
Because (JA, A (A,JA), the condition becomes:
(JA A) 1 (A.A)) (J+ - A.
-J <0
(1)
Moreover, by the spectral theorem, J has cigendccomposition J QVQ with unitary eigenvec- tor matrix Q E IR 1RdxXda Xde and eigenvalues are diag(V), Plugging this into inequality |
JJ. (QVQ A. A TQQQ A A (Q (Q VVQ'' () <)<0
<0
(2)
In summary, have reduced our original condition for corrected perturbation, (ossim(b((A)
be: A) 0, into something more tractable to analyze: (Q A) VTQQQQ A) O. Observe that
A E R Radasia is vector of the perturbation projected into the Jacobian's 's eigenvector space. In, each projected component squared and multiplied by the corresponding eigenvalue. Thus, perturbation decomposes heavily onto an eigenvector with a negative eigenvalue, our original condition for "correction" will be satisfied.
More concretely, consideer the perturbation from earlier, a, We can project a, into cigenvector space obtain q Q ay. Observe that entry qi is the projection onto the i-th eigenvector with eigenvalue r₃ From the derivation in 2 we have
(Ja, a.) (Q az) V ar) =q Vq Σ 1
aa)
Thus, Σ; >r/qi/2 < 0, a, being corrected. Additionally, any individual \//q corresponds to an individual direction that is being corrected, specifically the i-th eigenvector, with correction strength Aç. Suppose we consider the top eigenvectors, namely those with the top fifty |q values: these fifty directions account for 15+% of the variance of a,. Then, the corresponding A to understand how TLCM interacts with the top components of a,
plot the distribution of
Results. We compute the described phot-the histogram of X for top directions of a ac-across 50 Jacobian-a, pairs where TLCM present and 50 pairs where TLCM is not present. Figure shows the aggregated results. We observe bimodal eigenvalue distribution for TLCM pairs. The strongest mode hovers near -1, which means that unit increase in this direction from the prior layer results in unit-sized correction.
Importantly, mode around À -1 implies correction rather than partial attenuation of undesired features; the features are entirely reversed. The other mode hovers around X = 0.5, which implies that there are directions being promoted by TLCM, contrary our initial expectation. This bimodal distribution is firm evidence that TLCM does not seek to attenuate the previous layer uniformly. In fact, selects key subspaces correct, disregarding or promoting the rest. For adjacent layers without TLCM, we observe one strong peak at À = 0; this suggests that adjacent layers without TLCM do not interact significantly with the prior layer.
Finally, we observe that about 3000 of the total 4096 cigenvalues of J are negative; Figure 4a

---

5.3 Propose and Reject Hypothesis
We have shown that TLCM does not uniformly reverse the prior layer's contribution but rather selectively corrects a subspace of it. To synthesize our findings, we propose the propose-and-reg
hypothesis as a conceptual framework:
Propose-and-reject hypothesis (P&R). TL.M contributes to feature enrichment through a two-stage process: (1) layer proposes set of candidate features, and (2) the subsequent layer, equipped with attention mechanisms to gather context, removes irrelevant features.
P&R is consistent with our experiments thus far: Attention and MLP layers both play role in TLCM; TLCM activates more frequently later in the context window; TLCM activates predominately in the first two-thirds of models, which is connected with enrichment; TLCM corrects only the prior layer; TLCM appears to perform selective correction rather than uniform attenuation, as shown in Sec 5.2 among others. We caveat that we do not explicitly test P&R in this work.
P&R could prove particularly valuable for contextual processing. For example, consider the token "202" within 1,808,202", During the forward pass, layer i at this token might propose a handful of feature vectors: "hundred", "thousand", "million". Layer i + equipppedd with an attention sublayer- evaluate and eliminate the incorrect features. The incorrect features would form the "undesirable" subspace which would be corrected by layer i ++1 1. The correct feature, "million", would reside in the "desirable" subspace and hence remain untouched.
6 Discussion
Our TLCM experiments have consequences for broader work in mechanistic interpretability. For example, since SAES interpret residual stream contributions, TLCM's existence predicts mis- fertu-features contributed to the residual stream that are immediately reversed by the subsequent layer. In addition, research also offers framework for explaining several challenges in SAE-based interpretability:
Feature descriptions lack high specificity. Recent work 47] observed that over 50% of activated features from an SAE are labeled by a large LLM as "Irrelevant" or "Only vaguely related" to the text on which they fire. Highly-activating features, though more rare, tend to show greater specificity to the text. This pattern of low specificity in most features aligns with our expectation for "misfiring" features that are subsequently corrected by the next layer.
Effective model steering requires overcoming TLCM's correction. While we can amplify selected features to steer model behavior, our work predicts that some amplifications will be neutralized by TLCM's corrections. As shown in Appendix E. TL.CM's correction capacity begins to diminishes when the prior layer is amplified beyond 2x; therefore, feature amplification likely needs exceed a critical threshold to be effective. Consistent with this hypothesis, recent work demonstrated that effective steering requires extreme amplification levels-upp up to 10x feature's 's maximum observed iaiieem--ittertinng on feature
value [47] We propose that solving an alternative optimization directions that TLCM is not targeting, identified using the Jacobian could be promising direction for future work.
Cross- transcoders outperform SAEs. Recent work finds that the semantic meaning of the feature V, and -v; is unrelated [TO]. For example, assume the former means "firetruck" and the latter means "Ancient Greece." If layer i contributes -v, SAE is unable to determine whether layer aims to correct a faulty "firetruck" activation from the previous layer or contribute a novel "Ancient Greece" feature. Cross-layer transcoders, which are conditioned on the input residual stream, fundamentally can distinguish between these two cases, thus enabling them to learn better CLat can learn both "firetruck misfire" and an "Ancient Greece" feature that map to the same output vector. Indeed, recent work finds CLTs beat SAES on certain metrics [40].
More broadly, we think further understanding the architectural causes for TLCM (Appendix A explores some ideas) is exciting subsequent work, as is trying to understand TLCM's corrections in feature extracted via standard interpretability techniques. We hope TLCM helps improve methods to steer L.LM's forward passes and is step towards making models more interpretable and controllable.
10

---

References
[1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi. Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin. Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang. Guanhua Wang, Lijuan Wang. Shuohang Wang. Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue. Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang. Ziyi Yang. Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang. Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https:/t:://att
org/ hatg//bs//20 0abb/2244... 14219
[2]
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi. Piero Kauffimann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024. URL httpt:://axxiii...t/.m/ http://rxxi..oorab///////// abs/2412
08905
[3] Guillaume Alain. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv: 1610.0164, 2016.
[4] Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URL html
ciaeetiiiccuuttt.. www.riitts.uu//oeee.eo.ooop pubpp025/ 2025/ attaibutiinnnnooo ggraphs/methane
[5] Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2023. URL httpss://axxiv. org/ wtwp////..... 08112.
[6] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravee, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke. Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https:/transSormer- wew.tntiieefatuutnnneee... html.
[7] Greyson Brothers, Willa Mannering, Amber Tien, and John Winder. Uncovering uncertainty in transformer inference, 2024. URL httLhttps:// :/attpss///rxiv.. aaaaaaxvvvvor.. abgg222//212...... 2412/222222.6888
[8] Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. arXiv preprini
[9] Sviatoslav Chalnev, Matthew Siu, and Arthur Conmy. Improving steering vectors by targeting sparse autoencoder features, 2024. URL htttps://aae hwtpv//axvvvvvaa////v org/ abb/2211.02298
11

---

[10] Dami Choi, Vincent Huang, Kevin Meng, Daniel D Johnson, Jacob Steinhardt, and Sarah Schwettmann. Scaling automatic neuron description. https::/transluuee org/ neuron-descriptiinn October 2024.
[11] Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023. URL https://
arxiv. abs/ 2309 08600
[12] Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable Ilm feature circuits, 2024. URL https://arxiv: wrg.abs//////e//oooooo..oeeeeaa/
awwpaa/b...../////// 2406. 11944
[13] Jiahai Feng. Stuart Russell, and Jacob Steinhardt. Monitoring latent world states in language models with propositional probes. arXiv preprint arXiv:2406. 19501, 2024.
[14] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024. URL https://arxiv.co org/ /bs/
[15] Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, pages 160-187. PMLR, 2024.
[16] Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations auto-regressive language models, 2023. https:/: org/ httg/abs/200 2304
14767]
[17] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi. Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cristian Canton Ferrer, Cyrus Nikolaidis. Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang. Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang. Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park. Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Va- suden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kam- badur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi. Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic. Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hos- seini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal
12

---

Karn, Vedanuj Goswami, Vibhor Gupta. Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaofang Wang. Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Bahaei, Yi Wen, Yiwen Song, Yuchen Zhang. Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yumus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho. Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Fisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Lc, Erik Brinkman, Esteban Arcaute, Evan Dumbar, Evan Smothers, Fei Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni. Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong. Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand. Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang. Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chemoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parckh. Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto. Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay. Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanic Max, Stephen Chen, Steve Kehoe, Steve Satterficld, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk. Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaojian WU, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniy Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The Ilama
13

---

herd of models, 2024. URL https: 240////72/22222..0/ 2407
[18]
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, lan Magnusson, Yizhong Wang, Shane Arora, David Atkin- son, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. Olmo: Accelerating the science of language models, 2024. URL /tatttps:/ariivv /arxiviiivv..... wereii.orgg/ae www./ogg////ooooooo http//bb/2422...2 2202 00838
[19] Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors, 2023. URL httppp://arrti..orr/bbbb//// 15916 15916
[20] Evan Hernandez, Belinda Z. Li, and Jacob Andreas. Inspecting and editing knowledge repre- sentations in language models, 2024. https:/t/
org/
00740
[21] Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob An- dreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. Proceedings of the 2024 International Conference on Learning Representations, 2024.
[22] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL htttss//arxiv. http//axxvvvrr/.////// www.oogg////aaa....ooooooee 06825
[23] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL
http:///21..04400 04088
[24] Jiachen Jiang, Jinxin Zhou, and Zhihui Zhu. Tracing representationn progoetsionoooooooooooooooooooo Analyzing and enhancing layer-wise wise similarity, 2025. URL
14479
[25] Vedang Lad, Wes Gurnee, and Max Tegmark, The remarkable robustness Ilms: Stages of inference? arXiv preprini 2r0iv:2406. 19384, 2024.
[26] Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli. editors, Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, I, pages 1813-187.. Association for Computational Linguistics, 2021. doi: //doi.org//1 10e.e//855333 11653/11/2022 acl- 143
143. URL https
[27] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model,july, 2023, URL http://arxiv. (og/aabs/230.33341 2023.
[28] Jack Lindsey, Wes Gumee,, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of a large language model. Trunsformer Circuits Thread, 2025. URI. hhtpps //transformer--oee circuits. Buciiitssspubb/// pub/2025/ http///ttribbuiitoooo grwphs/bbolog... html
14

---

[29] Alex Mallen and Nora Belrose. Eliciting latent knowledge from quirky language models. arXiv preprint arXiv:2312.01003,, 2023.
[30] Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models, 2025.
URL htttt::t/axxiitt/e http:aaaiiii..raaa/////////////. abs//403. 19647
[31] Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy suppression: Comprehensively understanding a motif in language model attention heads. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 337-363 2024.
[32] Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations, 2023. URL https
/aaxxivorgg/bbll aba/ 2300 15771
[33] Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372, 2022.
[34] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
[35] Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746-751, 2013.
[36] Neel Nanda, S Rajamannhaaan, Kramár, and R Shah. Fact finding: Attempting to reverse-engineer factual recall the neuron level. In Al Alignment Forum, 2023c. URL https:://www. aligumentforum. Stspostt///GwZZZHWbbDD Bask://DDDDDsBKBBBaffffannn atdigttttt ewvers-engiinee-ffatull---e- page 19, 2023.
[37] nostalgebraist. interpreting gpt: the logit lens, 2020. URL https://ttwwtt lesswrong. com/ pact://AccBBBiijpllllooserrretee
[38] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction circuits. Distill, 5(3):c002240001, 2020.
[39] Alexander Pan, Lijie Chen, and Jacob Steinhardt. Latentqa: Teaching Iims to decode activations into natural language. arXiv preprint arXiv:2412.088630 2024.
[40] Gonçalo Paulo, Stepan Shabalin, and Nora Belrose. Transcoders beat sparse autoencoders for interpretability, 2025. URL https:tttt:///arit http///rxivv..rr/aa///555.1511 2281/ 18823
[41] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. OpenAl, 2019. URL https://cdn openai. www/better language
wwtps//aannggeeeeee pdf Accessed: 2024-11-15.
are_
[42] Fabien Roger, Ryan Greenblatt, Max Nadeau, Buck Shlegeris, and Nate Thomas. Benchmarks for detecting measurement tampering, 2023.
[43] Cody Rushing and Neel Nanda. Explorations of self-repair in language models. arXiv preprint arXiv:2402.15390 2024.
[44] Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models, 2025.
httpssttt://axii http::::////xwa//////o//oo wtt.:o/g//a//.....////// 02013
[45] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Rossro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova,
15

---

Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette- -hooo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, SeeorgeCChristann Muraru, Grigory Rozhdestvenskiy. Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona COmanescu, Reena Jana, Rohan Anil, Ross Mcllroy. Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri. Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology, 2024. URL htthhttp//// /ttp:///aaxvvvvoo aa/////s////3// 2bss2443.00825 08295
[46]
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchi- son, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov. David Weinberger, Dimple Vijaykumar, Dominíka Rogozińska. Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimezak- Klimezak-Pluciiska. Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Ilaaii Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen. Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNcalus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins. Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Matco Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej. Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles. Tom Hennigan. Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley. Wei Wei, Wenming Ye. Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma Improving open language models at a practical size, 2024. URL https://arxixn .wwg http//ab///00000.00
[47] Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet.
16

---

Transformer Circuits Thread, 2024. URL ortperr/rraasstrrrrrrcccci...... scaling- -oonssmaaniiittty/////ee html
[48]
Eric Todd, Millicent L.
Li, Arnab Sen Sharma, Aaron Mueller, Byron Wallace, and David Bau, Function vectors in large language models. In Proceedings of the 2024 International Conference on Learning Representations, 2024. URL httpps://ppeereriiiiiii www.:/.ee/ffrrri
id- id-Awyxtylwaa 1Xiv:2310.11213.
[49]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages -8845,50000 Online, October 2020. Association for Computational Linguistics. URL battps://wxx hwwp://wwwwwee..../atttto.ooooo. 2020. emn l--
[50] Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D) Manning, and Christopher Potts. Reft: Representation finetuning for language models. arXiv preprint arXiv:2404.03359,, 2024.
[51] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.1755. 2023.
[52] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma. Jianxin Yang. Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He. Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni. Pei Zhang, Peng Wang. Ru Peng, Rui Men, Ruize Gao, Runji Lin. Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li. Tianyu Liu, Wenbin Ge. Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL /ttps://arxiv.. org/ abs 2407 10671
17

---

Gemma Instruct
Gemma 28 instract
OLMo 78
OLMo 78
60
10
15-
20-
48
66
Figure A1: TLCM is consistently found on models with different approaches to LayerNorm. Sur- prisingly, Gemma with post-LayerNorm has more pronounced TLCM. OLMo 2, which replaces pre-LayerNorm entirely exhibits TLCM. For visual clarity, we plot clamp(M, -0.2, 0.2) and zero the diagonals.
A LayerNorm Blindness to explain TLCM
One natural cause of TLCM is RMSNorm. RMSNorm normalizes the input to the attention and MLP sublayers in nearly all of the open-source models analyzed. Formally, for input R& to attention or MLP sublayers and learnable parameter vector g, RMSNorm is defined as:
Og
RMS(():
||||2
RMS(x)
dm
where dm is the dimension of the residual stream.
Crucially, the of RMSNorm implies that both the attention and MLP sublayers have layernorm blindness; they are blind to the norm of the residual stream. This blindness significant because these sublayers predict contributions the residual stream (i.e. x; Sublayer(x;)), )), whose relative impacts depend on the the residual streams current magnitude. Without visibility into the residual stream norm, attention and MLP sublayers risk under-contributtng when the norm is high, which potentially leads to their contributions being overshadowed. This could encourage over-contribution behaviors followed by correction, which is consistent with TLCM.
However, we find that LayerNorm blindness alone does not fully explain the TLCM. This is because as discussed Sec. 5.2 the correction mechanism does not entirely reverse a,, contrary to what would be expected RMSNorm were the primary cause. But perhaps more critically, models trained with alternative RMSNorm implementations continue to exhibit the mechanism:
Gemma. The sublayers in Gemma were trained using pre-LayerNorm described above. In contrast, Gemma 2 utilized hybrid approach, employing both pre-laayerNrmm and post-LayerNorm:
x X;
The addition of post-Lay should, in principle, make the residual stream norm more predictable. However, empirical results show that the correction mechanism remains robust in Gemma 2. Refer to Figure Al for comparison.
OLMo.
The original OLMo models employed pre-LayerNorm exclusively. In the OLMo 2 series, LeeeenynNoomm replaced with post- Loyt-LayerNorm and QK norm. Despite this architectural change, the correction mechanism persists strongly OLMo 2, as shown in Figure Al
B Likelihood of Negative Cosine Similarity
B.1 Random Normal IID Vectors
Consider two random vectors: u, (0, L4))
18

---

(u,
E/cossim(u, v)] E
u 1 V
Σ u.
IF
EF
E
E
V
0
The last expectations must be because variance, first observe the following fact:
is distributed the same
u,
To calculate the
V
VE, u²
E
=1
Σ
dE
Σ u
u
1
E
d
Using this identity, the variance is:
Var(cossimuu, v)] E[cossim u, v)
CE v. ( u.v:)
E
E.
V.
E
of E, vf
Σ, ufv?
E
Σ. v?
u
2 E
Σ 4
Σ, v?
1=1
d d -
d
==1
B.2 Experimental Mean and Standard Deviation
We find that the contributions of a particular layer are anisotropic; they cluster around approximately 500-800 of the 4096 dimensions of Llama's residual stream.
19

---

Specifically, we isolate the contribution vector for layer i across 4096 tokens from wikitext: (Cem Ci,t2..... Ci,toone J. After computing the SVD of each set, we plot percent variance explained by top principal components vs. in Figure A2
Due this anisotropy, calculate the statistical significance of the TLCM cutoff (-0.1 cosine similarity) empirically. We compute the mean and variance of cosine similarity across contributions of adjacent layers on different tokens. More formally, we compute the mean and variance of cossim Ci Ci+1.ty) for & & l2 and 4 S < 20. We find it has has mean -0.00375 and standard deviation 0.03267, meaning that our cutoff of -0.1 is conservatively 30 event; most TLCM events occur at lower cosine similarities (-0.15 to 0.25).
We plot these distributions by layer in Figure A3
C Details Jacobian Experiments
C.0.1 Jacobian Sanity Check
We have approximated our response function bj using the Taylor expansion.
b(( = (0( + JA + 0(42)
Here we aim confirm that the Jacobian is appropriately representative the transformer layer within reasonable regime. Observe that the error of this approximation is by(A) - by( JA, and we thus denote the percent error the approximation as follows:
err (b](A)) bb(0) J |b ) - b((0)]
We plot the percent error error of this approximation across 46 randomly selected TLCM Jacobians for different values of A oa, E (-0.5, -0.4, -0.3.... 0.3, 0.4, 0.5}. Shown in Figure A4 we find that the Jacobian is good approximation within this regime. For |a 0.1, there consistently around 5% error, which we believe is sufficient for our Jacobian-basee analysis in Sec. 5.2
20

---

200
200
IN24 of lyyer cortiie
of corrits
of corriss
2.0
201
200
200
200
200
PCC zentrits
PC2 Beet c3mcrtt
contioe
Beet cartee
Percent Varianon Explained
lave centile
laer cariilis
300
200
3010
400
PCA of Nayer 24 centrie
PCA love c5ierei
PGA layn Cireiio
PCD Tewr ciriie
305
610
100 2000
2000
2500
0408
4000
2500
00.0
principal components
Figure A2: Percent variance explained by the top principal components vs n. graphed on the blue line. The orange line is what we would expect if contributions were isotropic. For layers where TLCM is most active (4 to 20), the first 500-800 principal components explain 80% of the variance in contributions, demonstrating that contributions are anisotropic.
21

---

20:00
15034
2300
3700
5200
13:40
560
11
-05
-2
-01
Oerttutton Xmh bebwert lnvr
baroon ove
Outtbreen ars behrent forr
am. betwee laee
1960
13M0
1000
1300
1800
10:00
130
2001
Count
Oustibation bebeman
Dastuuerr arr. beboan
butte of aPL between foor 24land
Dartetto an. batoon lays
15:0
2300
2000
1903
1200
1303
3.50
303
Outebuiien cm, ven lelveer laye 39
Deviudor ow sir Edwees leer
Ouiekvien ef sen betvew loye 30
Diviladoo of on se betwer lager
1007
1300
1300
19034
1 10:0
1200
1901
963
Cosine similarity between adjacent layers at different tokens
Figure A3: The empirical distribution of cossimm(; (Cr,r, C++1.12); A l2 across 2000 tokens for layers 4 < i < 20, corresponding to approximately 2 million samples per histogram. The red dotted line corresponds to the mean.
46 Random TLCM Jacobians
Mean and SE of Jacobian
1.0
(1T())))))
0.-
0.5
02
022
133
-32
13
Figure A4: The transformer layer Jacobian is a very good approximation for reasonably large perturbations (Ja] < 0.1) to the following layer, making it useful for decomposing directions as described in Sec. 5.2
22

---

04erigen22--254
OeeV/em22 5- tnomet
Ower 2-75 Hslmet
2 2ww.22M622-1224002 womer
10-
10
20
25
20
13
12
13
15-
23-
6.35
12
35-
15
15
metta Harrad 2-1hm
10
5.00
20
70-
30-
25
15-
15-
20-
5.00
22-
22-
-0.35
microsotpp144
www. 2410
mice ald Ma7S 72-
-0.00
14.
18-
15
15
22-
22
25-
32
626
35-
10
23
10
10
23
16
20
20
13
26
miteraa/Merra-11 3
minr do.
spenal coneen/oooun
12.
12-
15-
15-
25-
25
35
35
Figure A5: We plot clamp(M, -0.2, 0.2) for four models, zeroing diagonal entries. We computed M across variety of models, pulled from Huggingface Transformers.
D Extended Details on TLCM Existence
In Figure A5
we plot TLCM's existence across many HuggingFace models using the same technique as described in Sec. 4. of the main body.
We previously demonstrated that MLPs exhibit anti-correlations with each other, while attentions do not. We additionally find that MLPs correct attentions (both within and between transformer layers) and that attentions correct MLPs from prior layers. This could be due to common low-dimensional subspace used by both units for communication. Thus, MLPs correct both prior attention and MLPs; attentions correct just prior MLPs. Altogether, this suggests that MLPs are more responsible for TLCM's correction, but both units are involved.
Specifically, we plot M M Σ, M MILP, where
Mamx [i == cossim (x)si MLP, (ML,((x,.))
See Figure A6
23

---

Mym SMim-MiP
65
002
10-
Artn Index
6.0
0.1
51-
30
-05
10
30
MLP Index
Figure A6: MIPs correct prior attentions, and attentions correct prior MIPs.
E Extended Details on TLCM Adaptivity
E.1 TLCM is Adaptive Across Layers
Figure A7
we plot TLCM's adaptive correction across different layers using the same technique
as described in Sec. 4.1
E.2 TLCM Correction Capacity Diminishes at High
In Sec 5.1 we find that TLCM increases its correction as the prior layer is scaled. However, we find that as we scale the prior layer to extremely large values (a > 2), the correction capacity of TLCM starts to diminish. Intuitively, as is scaled, what TLCM previously considered "mistake" now might assumed "correct." This could explain why model steering interventions-inn which chosen feature vector is manually contributed to the residual stream- rreaierequires features contributions 10x the maximum ever observed value. In other words, steering interventions must overcome TLCM. See Figure A8 for plots of the correction beginning to diminish.

---

Loye
Laye
1xyer
Layer?
Laper
02-
Laye-
cossini (a,, (((()))))
Lya 12
Layo 13
Lay
63
-43
11
63-
-11
-106
Figure A7: TLCM adaptively regulates the previous layer in a near linear fashion. We plot 5 random corrections across a variety of layers in Llama 3.1 8B.
25

---

Laar
Layer
0.1
-02
-0.3
-0.0
Layer
-01
-02
-0.3
-0.3
cosiimaa ((oa)
-0.2
-0.3
-0.2
-0.3
Figure A8: As we increase the previous layer more dramatically, we see TLCM's correction begin to diminish, aside from some outliers. For each layer, we sample 5 random TLCM curves and compute at increments of 0.2.
26

---

OLMo Training Checkpoint Experiment Details
On a corpus of about 500 tokens of Py Torch instructional content, compute M on handful of training checkpoints, shown Figure A9
The corpus is below; beyond high enough number of tokens, we find this to have little effect on the cosine similarity matrices and thus also the figures.
**Using Convolutional Layers in PyTorch**
Convolutional layers are a fundamental component of convolutional neural networks (CNNs) used for image classification, object detection, and other computer vision tasks. In PyTorch, convolutional layers are implemented using the 'nn.Come Conv2' module.
**Creating Convolutional Layer**
To create convolutional layer in PyTorch, you the following code:
"python import torch
import torch.nn as
Define the convolutional layer conv_ laverayer-- nn. (onn2d(in_channel)) out cttanhhnne,,, kernel_size, stride, padding
ma_channels': The number of input channels (e.g., 3 for RGB images). cout_channels's The number of output channels (e.g., 64 for feature map)
'kernel_size's The size the convolutional kernel (e.g., 3x3).
'stride"; The stride of the convolutional kernel (e.g., 1). 'padding": The amount of padding to apply (e.g., 1).
**Example Usage**
Here's an example of using a convolutional layer in a PyTorch model:
''' "python import torch
import torch.nn nn as nn
class Con. def (self):
(Cpnv(Co, self)...init...
layer = nn. Conv2d(3,, 64, kernel_sizest stride=1, padding=1)
def forward(self, x):
return terch.relu(selff conv cone_layerxx))
# Initialize the model and input tensor model = ConvNet
Input_tensor norch.randn(1. 3, 224, 224)
# Forward paas output model sinnut_tensor)
27

---

Step 33000
Step 135500
Step 631000
Step 651581
435
15
10
Figure A9: In OLMo 7B, TLCM emerges progressively during pretraining, with initial manifesta- tion around step 135,500 (0.5T tokens). These four plots show M computed at different training checkpoints of OLMo IB, with the rightmost plot representing the fully pretrained model. For visual clarity, we plot clamp(M, -0.2, 0.2) and zero the diagonals.
480,000 (17)
Step 960.00C
Step 1,40,0000
1,907,159 (4T)
3.2
435
155
10
12-
12-
12
14
14
14-
14
Figure A10: In OLMo 2 IB, TLCM also emerges progressively during pretraining. We plot figures at different step numbers and number tokens (1, 2, or 4 trillion tokens).
G Token-Level Correction Statistics
G.1 Experiment Prompts
We the following prompts to generate data for experiment in Sec 4.3
Step 239,000 (17)
Step 482.00C (2T)
Step (1.77)
Step 928,816 (SAT)
10
15
20
25
30
10
15
20
25
30
10
15
20
15
30
10
15
20
25
30
10-
10
15
118
Figure A11: In OLMo 7B, TLCM also emerges progressively during pretraining. We plot figures at different step numbers and number of tokens.
28

---

Write a blog post about impact of remote work on urban real estate trends. Write an essay on the psychological effects of social media on leenagers.
Write a report detailing the advancements in renewable energy technologies over the last decade.
Write article about the rise of plant- baand dicts and their environmental benefits.
Write memo to employees explaining the new company policy cybersecurity measures. Write a letter to local council advocating improved recycling facilities in community.
Write a proposal for implementing mindfulness program in elementary schools to enhance student
well-being.
Write blog post about the evolution of smart home technology and its implications for privacy.
Write an essay discussing the ethical considerations of genetic editing technologies.
Write a report on the economic impacts of the COVID-19 19 pandemic on small businesses.
Write an article about the significance of the James Webb Space Telescope's 8 latest findings.
Write memo outlining the steps for successful digital transformation in manufacturing company.
Write letter to senator expressing concerns about the proposed changes to healtheare laws.
Write a proposal for a community garden project to promote local food production and community engagement.
Write a blog post about the latest trends in artificial intelligence and machine learning.
Write essay the role of therapy in mental health recovery.
Write a report assessing the potential hydrogen fuel as an alternative energy source.
Write an article highlighting the importance of biodiversity conservation in combating climate change.
Write a memo to staff regarding the integration of a new project management software.
Write letter to editor expressing opinions on the local government's 's transportation plan.
Write proposal for telemedicine service increase hcalthcare access rural areas.
Write a blog post discussing the future space tourism and possible timeline.
Write an essay exploring the cultural significance of indigenous music.
Write report on the trends in global unemplomment rates and their implications for economic policy.
Write article about the benefits and challenges of homeschooling.
Write a memo describing the company's 's strategy address the upcoming industry regulations.
Write a letter to non-pree non-profit organization offering to partner on an environmental initiative. Write proposal for an employee wellness program that includes both physical and mental health
activities.
Write blog post analyzing the impact of blockchain technology on financial services. Write an essay on the historical impact of major pandemies on societal structures. Write a report on the viability of vertical farming in urban environments.
Write an article about the challenges of maintaining data privacy in the age of loT.
Write memo update company leadership the progress of the quarterly goals.
Write letter
school board proposing the introduction of coding classes middle schools
Write proposal for local government initiative to support small businesses during economic downturns.
Write a blog post about the techniques and benefits of sustainable agriculture.
Write essay on the influence of classical music modern genres.
Write report consumer behavior changes in the automotive industry towards electric vehicles.
Write article about role youth activism shaping public policy.
Continued on the next page.
29

---

Write a memo detailing guidelines handling customer data under new privacy laws.
Write a letter to the editor about the importance of public parks and open spaces.
Write a proposal for new arts festival aiming to showease local and international talent.
Write blog post on the role of roboties in healtheare and potential ethical dilemmas.
Write an essay about the impact of climate change on marine ecosystems.
Write a report on strategies for managing workplace diversity in glohal company.
Write an article on the resurgence of interest in vinyl records and analog music.
Write memo to department heads about managing remote teams effectively.
Write letter to city planner regarding the need for improved pedestrian pathways.
Write a proposal for implementing bike- sharing program in a mid-sized city.
Write a blog post about the future trends in education technology and their implications for learning.
Writc a blog post about the growing popularity of mindfulness apps and their effcctiveness.
Write essay on the resurgence of traditional farming techniques in modern agriculture.
Write a report on the adoption of electric vehicles in major cities around the world.
Write an article about the psychological benefits of outdoor activities.
Write a memo to management detailing the steps to achieve carbon neutrality in the workplace by 2030.
Write a letter to philanthropic organization requesting funding for community tech hub. Write a proposal for series of workshops aimed teaching digital literacy seniors.
Write a blog post analyzing the impact of virtual reality on entertainment and media.
Write an essay discussing the philosophical implications of artificial intelligence surpassing human intelligence.
Write report the state of child nutrition programs in public schools
Write article about the role of drones in modern agriculture and their environmental impact.
Write a memo regarding the implementation a flexible work schedule to enhance employee productiv- ity.
Write a letter to government official advocating for stricter air pollution regulations.
Write a proposal for new public library with advanced digital resources.
Write a blog post about the importance of cybersecurity in the age of cloud computing.
Write an essay exploring the historical role of spices in global trade.
Write a report on the effectiveness of recent public health campaigns on smoking cessation.
Write article on the growing trend of micro-living and tiny homes.
Write memo introducing a new internal team dedicated to innovation and strategic initiatives.
Write a letter to parents outlining the new curriculum changes a local school district.
Write a proposal for a mobile health clinic to serve underserved areas.
Write a blog post about the use of big data in personalized medicine.
Write essay on the evolution of language in the digital age.
Write a report detailing the economic impact of cultural festivals on local communities. Write an article on the significance of urban green spaces for mental health.
Write a memo to staff about upcoming training opportunities in advanced analytics.
Write a letter to the editor discussing the need for more inclusive sports programs in schools.
Write a proposal for annual technology conference focusing on sustainability innovations.
Write a blog post about the effects of music therapy on Alzheimer's patients.
Write an essay examining the influence of vidco games on cognitive development.
Write a report on the future of nuclear energy and its role in combating climate change.
Write article about the revival of handcrafts and their market in the modern economy. Write memo outlining the benefits of adopting a four-day workweek.
Write a letter to university proposing a partnership for a community- based research project.
Write a proposal for developing a pedestrian-frient
zone in the downtown area.
Write a blog post on innovative approaches to waste management in urban settings.
Write essay about the socio-economic impacts of migration urban development.
Write a report on the adoption and regulation of cryptocurrencies in diffferent countries. Write an article on how to prepare pets for the arrival a new baby.
Write a memo discussing the integration of virtual assistants into customer service.
Write letter to historical society proposing project digitize and preserve ancient manuscripts.
Write a proposal for fitness program aimed improving the health of office workers.
Write a blog post about the role augmented reality in modern education.
Write an essay on the impact of global trade policies on developing economies.
Write a report analyzing the trends in youth sports and their benefits to communities.
Write article about the ethical considerations in wildlife photography.
Write a memo to update the company on the progress of diversity and inclusion initiative,
Write a letter to an NGO outlining a proposal for a joint clean water project in rural areas.
Write a proposal for a digital art exhibition featuring interactive installations.
Write blog post discussing the future of autonomous public transit systems and their societal impacts.
30

---

G.2 Correction Counts
Each tuple follows the format: (token, average TLCM activation count). We remove few hundred tokens from the middle of this distribution due to space constraints.
31

---

Table A1: Llama 3.1 8B Instruct: Top tokens with the highest average number of TLCM activations. For each token, we list the average number of times a TLCM activations occurs on the given token, aggregated across 100 long documents. Finally, we list the standard deviation and the number of occurrences of the token across the corpus to demonstrate the statistical significance.
Token
Mean Activations
Std err. of mean occurrences
202 Jul
16.66
0.16
265
14.00
0.00
100
26
13.90
0.04
112
]
13.15
0.12
131
Today n
13.00
0.00
100
12.88
0.11
485
12.85
0.16
26
12.62
0.19
55
<space>
12.58
0.07
1276
Name
12.54
0.10
100
12.44
0.24
41
i
12.41
0.10
228
12.23
0.08
164
assistant
12.20
0.08
100
State
12.15
0.32
26
]
12.08
0.14
49
It
12.07
0.20
68
Date
12.03
0.24
30
Address
12.03
0.26
35
user
12.00
0,00
100
n'
12.00
0.19
39
Date
11.99
0.14
204
4
11.92
0.12
338
Your
11.81
0.12
103
over
11.74
0.26
35
<space>>space>><aace
11.68
0.19
57
11.68
0.13
112
%
11.65
0.14
52
11.58
0.08
606
D
11.46
0.23
28
high
11.46
0.23
39
from
11.44
0.14
109
make
11.44
0,20
50
you
11.43
0.15
96
access
11.40
0.15
89
take
11.37
0.21
30
between
11.30
0,26
27
[
11.28
0.12
108
City
11.24
0.30
38
11.23
0.17
52
well
11.23
0.15
70
need
11.22
0.19
55
Thank
11.19
0.18
27
11.18
0.07
295
your
11.10
0.13
100
such
11.05
0.10
157
up
11.03
0.27
39
Knowledge
11.00
0.00
100
Write
11.00
0.00
100
long
11.00
0.27
26
32

---

Table A2: Llama 3.1 8B Instruct: Top 50 tokens with the lowest average number of TLCM activations. For each token, we list the average number of times a TLCM activations occurs on the given token, aggregated across 100 long documents. Finally, we list the standard deviation and the number of occurrences of the token across the corpus to demonstrate the statistical significance.
Token
Mean Activations
Std err. of mean
# occurrences
program
8.52
0.13
97
coding
8.50
0.24
26
report
8.49
0.28
35
guidelines
8.44
0.26
27
AI home
8.42
0.19
57
8.41
0.27
34
efficiency
8.41
0.24
37
art
8.38
0.30
32
This
8.37
0.07
155
organizations
8.33
0.22
36
community
8.33
0,09
160
ization
8.23
0.27
39
8.22
0.06
601
media
8.22
0.22
60
regulations
8.22
0.19
51
-being
8.20
0.20
59
engagement
8.17
0.16
63
environmental
8.17
0.20
42
-based
8.15
0.23
40
sustainable
8.15
0.11
89
urban
8.14
0.14
70
learning
8.12
0.18
80
infrastructure
8.12
0.20
49
IoT
8.10
0.25
31
should
8.08
0.14
59
interactive
8.08
0.25
26
diversity
8.07
0.24
28
classical
8.07
0.24
30
challenges
8.07
0.13
92
cities
8,03
0.18
40
mindfulness
8.00
0.29
33
indigenous
8.00
0.17
27
agriculture
7.96
0.21
50
VR
7.93
0.26
29
workshops
7.93
0.25
28
By
7.92
0.11
77
therapy
7.89
0.24
37
trends
7.89
0.27
27
innovative
7.89
0.25
27
innovation
7.89
0.18
35
proposal
7.86
0,42
37
cognitive
7.85
0.22
26
inclusive
7.66
0.17
41
sustainability
7.63
0.23
35
tourism
7.52
0.23
27
Cities
7.33
0.22
27
ting
7.03
0.05
102
blog
6.50
0.55
28
system
3,00
0,00
100
<|begin_of_text
textt>
0.00
0.00
100
33

---

Table A3: Gemma 2 2B Instruct: Top tokens with the highest average number TLCM activations. For each token, we list the average number of times a TLCM activations occurs on the given token, aggregated across 100 long documents. Finally, we list the standard deviation and the number of occurrences of the token across the corpus to demonstrate the statistical significance.
Token
Mean Activations
Std err. of mean
occurrences
<space>
17.89
0,06
623
17.31
0.11
366
6
17.09
0.13
160
tt
16.56
0.26
68
2
16.56
0.06
1066
16.48
0.14
242
]
16.39
0.13
218
<space><space>
16.36
0.09
657
7
16.17
0.32
35
sspace>xpppeexxppccocppppp
16,00
0.27
57
%
15.63
0.29
57
3
15.61
0.08
424
15.58
0.05
1289
5
15,43
0.12
183
15,42
0.08
840
0
15.32
0.06
627
0.
15.27
0.41
33
Thank
15.22
0.13
27
inin
15.16
0.06
1915
$
15.09
0.25
55
such
15.06
0.13
157
15.02
0.13
434
Today
15.00
0.00
100
9
14,96
0.44
50
14.87
0.04
3713
Address
14.74
0.17
34
recent
14.61
0.29
33
14.59
0.30
49
.
14.57
0.28
49
Date
14.49
0.04
204
14,49
0.10
476
long
14,46
0.37
26
14.37
0.22
84
them
14,27
0.35
60
Date
14.17
0.18
30
14.14
0.14
166
members
14.10
0.43
30
sense
13.97
0.30
30
(
13,90
0.17
195
modern
13,81
0.32
32
13.79
0.23
43
City
13.76
0.30
38
over
13.71
0.27
34
years
13.70
0.35
40
at
13.68
0.35
41
Name
13.66
0.14
100
countries
13.59
0.36
37
assistant
13.55
0.14
100
world
13.52
0.28
54
led
13.50
0.28
44
34

---

Table A4: Gemma 2 2B Instruct: Top 50 tokens with the lowest average number of TLCM activations. For each token, we list the average number of times a TLCM activations occurs on the given token, aggregated across 100 long documents. Finally, we list the standard deviation and the number of occurrences of the token across the corpus to demonstrate the statistical significance.
Token
Mean Activations
Std err. of mean occurrences
sustainable
10.25
0.18
89
spaces
10.24
0.29
46
plan
10.23
0.39
39
10.23
0.26
26
several
10.22
0.31
27
understanding
10.22
0.33
growing
10.21
0.24
38
online
10.21
0.17
43
reduced
10.19
0.28
26
learning
10.19
0,20
80
environmental
10.14
0.18
42
training
10.14
0.27
59
create
10.10
0.22
69
implementing
10.10
0.35
30
mental
10.09
0.21
53
AI
10.09
0.21
57
indigenous
10.07
0.29
27
local
10.07
0.16
110
significant
10.06
0.16
143
comprehensive
10.05
0.22
39
together
10.04
0.41
26
blog
10.00
0.07
28
address
10.00
0.33
41
innovative
9.96
0.29
27
promoting
9.94
0.24
47
post
9.94
0.22
31
develop
9.89
0.25
benefits
9.87
0.19
105
VR
9.86
0,28
29
community
9.83
0.17
160
awareness
9.80
0.41
prioritize
9.79
0.29
28
promote
9.74
0.15
102
progress
9.72
0.38
29
following
9.70
0.31
27
clear
9.69
0.33
26
concerns
9.67
0.32
48
approach
9.65
0.34
34
complex
9.61
0.27
38
By
9.58
0.17
77
challenges
9.53
0.25
92
improved
9.52
0.28
33
interactive
9.42
0.38
26
enhance
8.97
0.34
35
improve
8.91
0.20
70
explore
8.74
0.29
feedback
8.70
0.28
27
mi nitiaate
7.88
0.37
Write
7.00
0.00
100
<bos>
1.00
0.00
100
35

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state the abstract and intro exactly what we were able to show using experiments. There is section dedicated to everything we have written in paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
36

---

Justification: We discuss how we do not see this on a handful of model families and perform some speculation why. In our appendix, we discuss some architectural causes for TLCM (LayerNorr), but also discuss why this might not be true. In our Jacobian experiment, we discuss that directions do not directly correspond to standard features, which limits the strength of results only minorly. In other parts of our work, we caveat that we only see these results strongly from layer to 20; or we give the exact dataset or dataset size that we compute across to give a sense for how strong the results are. Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-texl system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We have one result, and for that reason it is not numbered. Our result and proof is short and is thus presented within the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
37

---

Answer: [Yes]
Justification: We generally provide the most important information about reproducibility within the text/content the paper. However, for some experiments this is intractable, so include any missing details within the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Our paper mostly consists of short, quick experiments and corresponding matplotlib code that are run on models using HuggingFace transformers. We don't 't feel it's necessary to release this code it is reasonably quick to implement once you understand any given experiment, although we are happy to do so if requested.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
38

---

reproducible, they The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
seabbbbmmssiiiooi wsiiiiiicciiiiiicc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification: These details do not apply to experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For our main results on TLCM, we provide ample information about statistical significance and dedicate a section of our appendix to it. For other experiments, the statistical significance is implied by the high sample size we use, but no error bars or confidence intervals are included.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
39

---

8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [No]
Justification: We did not include this information because the vast majority of our experi- ments are not computationally intensive. The most computationally intensive experiment required computing Jacobians of transformer layers, for which we used single GPU; detail this in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics atttps///neus :/tpprii/////ip.... httpc///cccthhiccc....e........e
Answer: [Yes]
Justification: Nothing was harmed in the process of writing this paper. Additionally, popular datasets that are publically available.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We do not believe that there are direct socictal impacts of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
40

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not believe our work poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We primarily use wikitext, which we cite. Additionally, cite all models discussed or used the paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwppsai thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- e-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
41

---

Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: The paper does not involve rowd-sourcing
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMS if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality the research, declaration is not required.
Answer: [NA]
Justification: LLMs were only used for help with writing and editing.
42

---

Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy http:///aaarrpsscc/ooooooorrrr/s for what should or should not be described.
43


Reviews:
[{"review_id": "TRwFy5qcKk", "reviewer": "Reviewer_afWX", "summary": "The main idea of the paper is to discover a phenomenon in transformer networks, namely Transformer Layer Corrective Mechanism (TLCM) which basically means that consecutive layers in the transformer networks (e.g, in LLMs) basically correct or improve upon the representations from the previous layer. The mechanism to identify the correction is via cosine similarity in the representations obtained after each layer. Negative cosine similarity indicate that features from the previous layers are corrected by the next layer, which is evident in many of the LLMs explored in the paper. \n\nThis phenomenon is found in the early to middle layers of the LLM and is exacerbated for long context. Also seen more for numbers, dates, etc. which are more contextual in nature. Section 5 of the paper, computes TLCM when the input are perturbed to see the correction from the LLMs. Lastly, analysis from the eignevalues of the Jacobian of the transformer layers, show that the updates from the next layer are focused.", "questions": "1. How does this paper relate to `Layer by Layer: Uncovering Hidden Representations in Language Models` (https://arxiv.org/pdf/2502.02013)\n\n2. Can different domains have different TLCM characteristics? (e.g, Code)\n\n3. Could not find diagram FigureA4 in appendix corresponding to Section 4.2", "limitations": "The limitations are adequately discussed.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Overall the idea and the exploration in the paper are interesting. \n\n2. There are empirical evidence that propose and reject hypothesis could be an interesting tool for understanding the mechanism of transformers.\n\nWeakness:\n\n1. While empirical evidence show that there are early signs, it would be interesting to see the results for different languages? Can this framework provide understanding on how multilingual mechanism works.\n\n2. It could also be interesting to see the results on multimodal LLMs (e.g vision LLMs), where perturbations in images could be used to understand TLCM better.\n\n3. While the paper explains existing interpretability challenges, future work could explore how the understanding of TLCM can be directly leveraged to develop more effective interpretability tools or even guide architectural improvements in LLMs to optimize information flow.", "was_revised": true, "final_justification": "I would like to keep my scores as the questions I raised were adequately answered. But from the discussion it seems that there are some additional details that can make this paper better.", "rebuttal": "Thanks for your review. We’re glad you found the “idea and exploration” interesting, the paper to “read well,” and the tools we introduce useful/interesting “for understanding the mechanisms of transformers\\!”\n\nWe address your questions and some of your concerns below.\n\n>It could also be interesting to see the results on multimodal LLMs (e.g vision LLMs), where perturbations in images could be used to understand TLCM better.\n\nThanks for raising this point. Recent work (On Layer-wise Representation Similarity by Jiang et. al.) has found positive cosine similarity between layers in ViTs. However, it remains an open question whether this phenomenon is limited to just language models.\n\n>How does this paper relate to Layer by Layer: Uncovering Hidden Representations in Language Models ([https://arxiv.org/pdf/2502.02013](https://arxiv.org/pdf/2502.02013))\n\nGreat question. One intuition from our work and other recent research is that:\n\n1. Early layers are responsible for building rich representations (note: transformers have an incentive to produce representations that will be useful for future tokens, even though they cannot “see” them).  \n2. Middle-to-later layers are where final processing occurs.  \n3. Late layers focus on ensuring that the output distribution is correct, often by removing detail from the residual stream.\n\nWe think this flow makes sense, and it’s also well supported by the paper you’ve linked. TLCM's high prevalence in early layers and contextual dependency gut-checks with the early layers' responsibility to “build rich representations.” Additionally, TLCM is notably less prevalent in later layers on most models, which correlates well with the less context-dependent tasks of (2) and (3).\n\nWe also cite papers which build the above intuitions, see \\[20, 21, 32\\].\n\n>Can different domains have different TLCM characteristics? (e.g, Code) and it would be interesting to see the results for different languages\n\nIndeed, we find TLCM activations occur on code; we use code for some of our experiments (see Appendix F).\n\nAdditionally, our initial experiments in Sec. 4.1 are computed across 100,000 tokens of Wikitext, which—while mostly English—includes sections in other languages. In short, we see TLCM activate across varied domains.\n\n>Could not find diagram FigureA4 in appendix corresponding to Section 4.2\n\nGood catch. We mean to refer to Figure A9. Because the paper and appendix were submitted separately at different times, it seems like we unintentionally compromised the correspondence of this figure in the paper and in the appendix. This is fixed for the camera-ready version.\n\n>TLCM can be directly leveraged to develop more effective interpretability tools or even guide architectural improvements in LLMs to optimize information flow.\n\nGreat point. In Appendix A, we begin to explore the architectural features that could induce TLCM. Essentially, we introduce the concept of LayerNorm blindness, which is the phenomenon that LayerNorm causes attention and MLP sublayers to be “blind” to the norm of the residual stream. Thus, they do not know how to size their contributions and may be biased to over-contribute, so their contribution is not overshadowed. If true, this presents a real inefficiency and an opportunity to “optimize information flow.”\n\nHowever, we find TLCM in models that do not have LayerNorm blindness. Thus, this is not the full explanation. We agree that there is more work to do and are excited about directions like this for future work."}, {"review_id": "y6g0MKIdtd", "reviewer": "Reviewer_ucnA", "summary": "This paper discusses TLCM; a mechanism empirically observed in transformer architectures, where the “contribution” vector, i.e., the difference between intermediate representations, exert negative similarity with the contribution vectors of immediately adjacent layers (while with later layers, the similarity becomes positive again). Through a cosine similarity analysis, they initially show that this effect seems to occur on the majority of the evaluated model families (LLaMa, Qwen, Mistral, …) and that it mainly appears in early layers.  The authors claim this is unexpected, as this seems to show that transformer layers systematically reverse the contributions of prior layers. Through evaluation studies, the authors further evaluate when this effect emerges and whether it occurs consistently across inputs. They find that (1) TLCM emerges during initial phases of pretraining, and (2) list some tokens which seem to be more and less susceptible to “correction”. Finally, the authors construct the “propose-and-reject” hypothesis where one layer proposes candidate features, and the next layer, using additional context, selectively rejects or reinforces them. To formalize this, the authors analyze the Jacobian of the next layer and claim that eigenvectors with negative eigenvalues correspond to directions that are being corrected. Empirical results suggest that correction does not apply uniformly across the residual stream but targets specific subspaces.", "questions": "- L.40 How do we quantify “high context dependency”? What makes a date, number, or punctuation tokens more “context dependent” than others?\n- L.99-101: how do we know that contribution vector similarity tells us something about “novel information” and “reinforcing existing representations”?", "limitations": "- the generalization of the results is a significant limitation, as mentioned above.", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- The paper makes empirical contributions about the representational similarity in adjacent transformer layer representations, which re-appear across multiple architectures. This is interesting. They also aim to understand the problem from different perspectives (e.g., effects of training, token classes, …).\n---\n### Weaknesses\n\nTerminology is loosely used and many claims are vague or unfounded. A few examples:\n - E.g., L.29-37; what does it mean to “partially oppose” in a general setting, or “reverse” other layers’ contributions? Similarly, L.52 “verify the features and selectively correct inappropriate ones”; these should be more precise, in its core the paper observes representational contribution vector similarity\n- L.40 How do we quantitatively measure “high context dependency” in tokens? It seems to me that the definition is more speculative than quantitative.\n- L.99-101: how do we know that contribution vector similarity tells us something about “novel information” and “reinforcing existing representations”? \n- L.143: Having negative cosine similarity does not mean “erasing information”\n\nAlthough the paper aims to evaluate the phenomenon of TLCM expansively, a lack of empirical rigor and ablation makes the results and conclusions seem speculative, rather than generalizable. A few examples:\n\n- L.115-117: the authors compute the cosine similarity between contribution vectors, but report on “correlation” in L. 116 and throughout.\n- L. 128: lack of more rigorous derivation/description why choosing exactly $-0.1$ for the threshold for TLCM. “Seems to have high specificity” (L.130) does not make it well-defined. Looking at Figure 2b) this is not even at the low point of the $\\text{cossim}$ distribution.\n- §4.2 raises three interesting potential scenarios about when TLCM may arise, but they only evaluate a single one of them on a single architecture (OLMo 1B).\n- The authors show that TLCM appears across various architectures, yet individual studies §4.2 and §4.3 are only done on a single model, making it difficult to understand the generalizability of the findings.\n- §4.3 only reports qualitative results, where, according to the description, quantitative evaluation was performed.\n- Figure 3 only shows layer 8 for an unknown model, and 5 “randomly sampled tokens”.\n- The derivation of the “propose-and-reject” hypothesis seems rather speculative.\n\nLack of formal rigor: many of the findings are purely empirical, however, there lacks a convincing, rigorous theoretical discussion about TLCM, or why we are surprised by the empirical findings (elaborations of L. 99-105). This would complement the empirical results well.\n\nSignificant writing issues make the manuscript hard to follow and the contributions hard to identify.\n- E.g., the introduction is written vaguely without explaining the approach or contribution (e.g., L.55-58), discussion sections heavily overlap with the main body\n- Additionally there are quite a few typos and notational problems (e.g., §3, or see typos)\n\n---\n### Typos:\n- L.67-69: citations\n- L. 71 “large language models” -> “LLMs”\n- L. 76 “onn” -> “on”\n- L. 84 “unembedded”?\n- Restating an introduced quantity, L.159\n- L. 175: “this corpora” -> “this corpus”\n- L.259: “numners” -> “numbers”\n- ...", "was_revised": true, "final_justification": "As is visible in my review as well as in follow-up discussion, in my opinion, this work raises an interesting questions coupled with interesting empirical observations, however, the paper lacks (a) a solid theoretical foundation and (b) empirical rigor to convincingly and quantifiably prove the generalizability of the found phenomena. The paper aims to characterize TLCM from too many angles, without going into too much detail for any of them. Thus, many of the made claims are speculative and are, thus, overstated in the paper, in my opinion. Follow-up discussions with the authors helped clear up some confusions, however, they failed to make the empirical results more convincing.", "rebuttal": "Thank you for your thoughtful feedback\\! We respond to your questions below.\n\n>§4.3 only reports qualitative results, where, according to the description, quantitative evaluation was performed\n\nGreat clarification. **Appendix G.2 contains the quantitative results underlying the findings in §4.3**; we list the average number of TLCM activations across \\~225 tokens. To highlight the statistical significance of this result, we’ve added the standard errors for each token (which are around \\~0.2).\n\n>individual studies §4.2 and §4.3 are only done on a single model, making it difficult to understand the generalizability…\n\nThanks for this point. To address your suggestion, **we’ve run both of these experiments on new models.**\n\nDue to limited checkpoint availability, we re-run our training experiment (4.2) on OLMo-2 7B, which features a new data mix and updated architecture. First signs of TLCM appear later, around 2T tokens, and TLCM solidifies around 3T tokens. OLMo-2 7B is trained on a total of 4T tokens. We’ve added these plots to our appendix.\n\nRe-running experiment 4.3 on Gemma-2 2B Instruct yields **highly consistent results**. Below, we list tokens with the most and least average number of TLCM activations. The upper bound of the standard error for the following TLCM activation averages is 0.38, demonstrating strong statistical significance. Like the Llama models, the beginning of sequence token has the fewest TLCM activations by far.\n\n**10 lowest activating**\n\nchallenges: 9.53  \nimproved: 9.52  \ninteractive: 9.42  \nenhance: 8.97  \nimprove: 8.91  \nexplore: 8.74  \nfeedback: 8.70  \nmitigate: 7.88  \nWrite: 7.00  \n\\<bos\\>: 1.00\n\n**13 most activating**\n\n\\<space\\>: 17.89  \n4: 17.31  \n6: 17.09  \n\\\\t: 16.56  \n2: 16.56  \n': 16.48  \n\\]: 16.39  \n\\<space\\>\\<space\\>: 16.36  \n7: 16.17  \n\\<tab\\>: 16.00  \n%: 15.63  \n3: 15.61  \n\\\\n: 15.58\n\n>Figure 3 only shows layer 8 for an unknown model, and 5 “randomly sampled tokens”\n\nAlthough we state in L. 127–130 that this is Llama-3 8B, we agree this should be clearer. We’ve added model identifiers throughout the relevant sections.\n\nAdditionally, we include full results for this experiment (total of 135 randomly sampled tokens across 18 layers) in Appendix E, Figures A7 and A8. The results in Figure 3 are largely representative of the appendix figures.\n\n>§4.2 raises three interesting potential scenarios about when TLCM may arise, but they only evaluate a single one*\n\nThe experiment in Sec 4.2 tests all three scenarios. We show:\n\n1) *Learning-induced*: TLCM does not appear in untrained models.  \n2) *Persistent*: TLCM strengthens over training, and is not—for example—some training pathology caused by instability.  \n3) *Pretraining-derived*: TLCM emerges during pretraining, not SFT or RL.\n\n>many of the findings are purely empirical, however, there lacks… theoretical discussion about TLCM\n\nThank you for raising this point about theoretical grounding. We prioritized empirical characterization for several reasons:\n\n1. **Phenomenon complexity requires strong empirical foundation** Our results reveal TLCM's multifaceted nature—spanning training dynamics, information flow patterns (context experiments), sublayer interactions (MLP/attention), and potentially architecture-specific effects (Appendix A). Given this complexity, we believe empirical documentation is essential before proposing theoretical explanations that might be premature or overly narrow.\n\n2. **Building foundations for mechanistic interpretability** We view our contribution as part of a broader scientific process: establishing a corpus of well-characterized, reproducible phenomena provides the empirical constraints necessary for developing robust theories. Just as early observations of superconductivity (1911) provided crucial data for the eventual BCS theory (1957), we hope TLCM's documentation enables both practical applications and theoretical understanding.\n\n**We aim to contribute:**\n\n1. Discovery of TLCM—a novel, broadly-observed phenomenon that reveals unexpected layer dynamics  \n2. A set of characterization experiments (e.g., 12+ models, MLP/attention differentiation, token-level statistics, training dynamics, causal interventions, context experiments).  \n3. A Jacobian eigenbasis-based interpretability technique, which we develop and apply to TLCM to show selective subspace rejection and reinforcement.  \n4. A conceptual scaffold—the propose-and-reject hypothesis that organizes observations and suggests research directions  \n5. Actionable insights for recent results in mech interp: CLTs outperforming SAEs, misfires, and model steering difficulties.\n\nWe position this work as empirical science: **establishing a robust phenomenon that challenges current understanding**.\n\n>The derivation of the “propose-and-reject” hypothesis seems rather speculative\n\nThe propose-and-reject hypothesis states that a layer proposes a broad swath of possible “features,” while the following layer—equipped with an attention head to gather context—removes the features that are not relevant.\n\nWe intend this to be a conceptual synthesis that is consistent with all our empirical results thus far. Specifically, the hypothesis explains:\n\n* Quantitative contextual dependency experiments.  \n* TLCM corrects *only* the most recent layer, rather than correcting *all prior layers*  \n* TLCM’s prevalence in early layers, which are linked to “enrichment”  \n* Very low TLCM activations on beginning of sequence tokens\n* TLCM emerges when considering the ***combination*** of Attention/MLP sublayers (MLPs cannot access context without attention)  \n* The causal relationship we show—where the following layer dynamically attenuates the prior layer—in Sec 5.2.  \n* Sec 5.3’s Jacobian experiment demonstrating **unit** attenuation ($\\\\lambda \\= 1$) of features\n\nWith this framing, we intend to **generate future research directions**, not to close the discussion.\n\n>what does it mean to “partially oppose” in a general setting, or “reverse” other layers’ contributions? Similarly, L.52…\n>Having negative cosine similarity does not mean “erasing information”\n\nGood clarification\\! Our precise claim is that adjacent layers **attenuate specific components** of the prior layer’s contribution.\n\nThis is supported by **stronger evidence beyond cosine similarity**:\n\n* In Sec. 5.1, causal interventions show that scaling $\\\\mathbf{c}\\_i$​ leads to proportional counter-scaling in $\\\\mathbf{c}\\_{i+1}$​,\n\n* In Sec. 5.2, our Jacobian eigen-analysis reveals subspaces where **unit increases receive unit correction**.\n\nTo reflect your suggestion, we’ve updated our wording to emphasize “correction” and “attenuation” rather than “erasure” or “reversal.”\n\n>L.99-101: how do we know that contribution vector similarity tells us something about “novel information” and “reinforcing existing representations”?\n\nGreat question\\! Our interpretation draws from the common assumption in mechanistic interpretability that contributions approximate a linear combination of interpretable features (L. 89–92). Under this view:\n\n* An orthogonal contribution roughly corresponds to introducing new features,\n\n* A positively aligned contribution roughly corresponds to reinforcing existing features.\n\nTo ground this in examples: Recent work (On Layer-wise Representation Similarity by Jiang et al) has found positive cosine similarity between layers in ViTs. Other work (Residual Connections Encourage Iterative Inference by Jastrzebski et al) views each residual block as performing a gradient step in activation space, so TLCM would correspond to a too high “learning rate,” which is unexpected because pretraining should find the optimal learning rate.\n\n>L.40 How do we quantitatively measure “high context dependency” in tokens?\n\nThanks for raising this.\n\nQuantitatively, we use **number of preceding tokens** as a proxy for contextual dependency (L. 206–211). In **Figure  2b**, TLCM activation shows a linear relationship with this proxy: every additional 1k context tokens corresponds to about one additional TLCM activation, on average. While computed across 2,000 wikitext documents, we agree that this is a limited proxy.\n\nQualitatively, we find high TLCM activation in tokens (newlines, numbers) known from prior work to rely on context. For example, newline tokens, as discussed in Lindsey et al. \\[4\\], often trigger planning behavior that integrates surrounding information.\n\n>lack of more rigorous derivation/description why choosing exactly for the threshold for TLCM…Looking at Figure 2b) this is not even at the low point of the distribution\n\nGood question\\! Figure 2b is computed across 3.5 million adjacent layer contributions across \\~100K tokens, making the **bimodality** of the cosine similarities significant. This is strong evidence that there are (at least) two distinct interaction regimes: TLCM and non-TLCM.  We agree these details are important to highlight in the paper, so we’ve added them around L. 130\\.\n\nBecause the $\\\\operatorname{cossim} \\< \\-0.1$ threshold is further left from the low point of the distribution, we believe it captures greater specificity than using the low point. In practice, more conservative thresholds (e.g., $\\< \\-0.18$) yield consistent results, as most plots already involve stronger TLCM activations due to the distribution shape.\n\n>why \\[are we\\] surprised by the empirical findings (elaborations of L. 99-105)\n\nGreat question\\! See L. 131–141 and Appendix B.1–B.2, where we explore both the theoretical and empirical significance of TLCM. Using two different sets of assumptions, we show that TLCM is highly statistically significant.\n\n>L.115-117: the authors compute the cosine similarity...but report on “correlation” in L. 116 and throughout\n\nWe intended these terms to refer to the same quantity. However, we recognize your point and have updated the paper to use “cosine similarity.”\n\n>Typos, etc\n\nThanks for pointing these out\\! We have fixed these and will improve clarity for the camera-ready version."}, {"review_id": "qhOIBpbdNO", "reviewer": "Reviewer_dNpi", "summary": "The paper documents a novel phenomenon of 'Transformer Layer Correction Mechanism'; adjacent layers in transformers sometimes have 'anti-correlated' activations. They observe that this phenomenon occurs across number of open-source LLMs, though is absent in models that use dropout (Phi 4, GPT2-XL).", "questions": "1. Line 162-163: \"Appendix Figure A4 demonstrates that the mechanism’s strength increases markedly during the latter two-thirds of training, suggesting a systematic development pattern.\" I could not understand figure A4 from its caption. My best guess is that authors meant some other figure here (probably A9)? Or otherwise figure A4 needs to be explained better.\n\n2. Figure 2a is referenced much after 2b. Flip the order of the figure.\n\n4.Line 210-211. What does a TLCM 'activation' mean? This concept is not defined anywhere (or may be I missed it? In which case, it ought to be defined more saliently). I in general was not able to understand the y-axis of figure 2b(left).\n\n5. Why does Qwen 0.5B has this phenomenon but not GPT-2 XL? Is dropout being present/absent the sole determinent of this phenomenon?\n\n6. Figure 3 -- why is this done over 5 random tokens only? Can't it be done at a larger scale?\n\n6. Compen", "limitations": "Limitations are not explicitly discussed. Authors should address limitations and open questions comprehensively to invite further research on this topic.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The phenomenon documented in the paper is novel and interesting.\n- The paper is mostly easy to follow.\n- The paper mosty does a good job of conducting systematic studies.\n\n\n### Weaknesses\n1. Figure captions can be improved in many cases.\n2. Some results (e.g., Figure 3) are over very small sample sizes.\n3. Compensatory effect (line 263-269) is not sufficiently explored. I think this effect goes against 'propose and select' hypothesis so would be worth exploring further, and understanding. It is also not clear why it has to be adjacent layers that do the 'propose and select' operation.\n4. There should be more commentary on why some models demonstrate TLCM but not others.\n\n### Improvements\nI think doing the following would help improve the paper (though I don't think they are 'weaknesses'):\n\n1. Including analysis of some state space models (e.g., Mamba) and seeing whether this phenomenon is particular to Transformers architecture or occurs across autoregressive LLMs.\n2. Including analysis of models of some modality, e.g., vision.\n3. Compensatory effect (line 263-269) is worth exploring more. \n4. It would be interesting to develop some sort of improvement to intervention vectors based on TLCM. Naively seems like adding intervention vectors at alternate layers might be the best strategy to overcome TLCM?", "was_revised": true, "final_justification": "I am keeping my original score. See the strengths and weaknesses as listed above for why so.", "rebuttal": "We’re glad you thought our work was “novel and interesting” and conducts good “systematic studies\\!” Thanks for your questions; we address them below:\n\n>Figure 3 \\-- why is this done over 5 random tokens only? Can't it be done at a larger scale?\n\nThanks for raising this. We include full results for this experiment (total of 135 randomly sampled tokens across 18 layers) in Appendix E, Figures A7 and A8. We see that the results in Figure 3 are largely representative of the appendix figures.\n\n>Line 210-211. What does a TLCM 'activation' mean? This concept is not defined anywhere (or may be I missed it? In which case, it ought to be defined more saliently). I in general was not able to understand the y-axis of figure 2b(left).\n\nGreat question. We define a TLCM activation as any pair of adjacent layers whose contributions’ cosine similarity is less than \\-0.1. More formally, $\\\\operatorname{cossim}(\\\\mathbf{c}\\_{i,t}, \\\\mathbf{c}\\_{j,t}) \\< \\-0.1$ (L. 174-175). Note that an LLM with $n$ transformer layers can have up to $n-1$ activations. The y-axis of Figure 2b (left) is the average number of TLCM activations on the $i$th token across a corpus of 2,000 wikitext documents. \n\nWe’ve updated our wording in this section (and Sec 4.3) to explicitly define “TLCM activation,” to your suggestion.\n\n>Why does Qwen 0.5B has this phenomenon but not GPT-2 XL? Is dropout being present/absent the sole determinent of this phenomenon?\n\nThis remains an open question.\n\nWe have not seen TLCM on any model with dropout, which indicates that dropout could be involved. However, GPT-2 is also undertrained compared to modern models, being trained on at most 100B tokens. Observe in Figure A9 that the OLMo 1B models (similar size to GPT-2 XL) do not exhibit any traces of TLCM until being trained on 500B tokens. We’ve run a similar test on OLMo-2 7B and observe that TLCM starts to appear around 2T tokens. For additional context, Llama-3 8B was trained on 15T tokens\\!\n\n>It would be interesting to develop some sort of improvement to intervention vectors based on TLCM. Naively seems like adding intervention vectors at alternate layers might be the best strategy to overcome TLCM?\n\nWe agree this is an important area of future work. That’s an interesting approach: it seems like even if you add an intervention vector at layer $i$, our results indicate that you will get a correction at layer $i+1$. Although it’s possible if you add interventions at both layers, this correction breaks down. We’re interested in seeing how this plays out.\n\n>Line 162-163: \"Appendix Figure A4 demonstrates that the mechanism’s strength increases markedly during the latter two-thirds of training, suggesting a systematic development pattern.\" I could not understand figure A4 from its caption. My best guess is that authors meant some other figure here (probably A9)? Or otherwise figure A4 needs to be explained better.\n\nThank you for catching this. You are right–we are referring to Figure A9. Because the paper and appendix were submitted separately at different times, it seems like we unintentionally compromised the correspondence of this figure in the paper and in the appendix. This is fixed for the camera-ready version.\n\n>Figure 2a is referenced much after 2b. Flip the order of the figure.\n\nGood point; we’ve fixed this.\n\n\nWe’re happy to address any other questions you have."}, {"review_id": "ki77AbjOka", "reviewer": "Reviewer_VpPo", "summary": "The paper introduces the observation termed Transformer Layer Correction Mechanism (TLCM): Adjacent layers in many Transformer LLMs seem to partially counteract each other’s contributions (negative cosine similarity). The paper provides convincing evidence of the existence of this effect (particularly in Figures 1 and 3), and shows that it originates from the feed-forward layers. The paper furthermore proposes a “propose-and-reject” hypothesis as an explanation for the TLCM, and attempts to use this to explain challenges in SAE-interpretability.", "questions": "I have the following questions for the authors:\n\n1) Did you subtract the dataset mean before computing the cosine similarity? I wonder if a non-zero mean could produce an offset in the cosine similarity.\n\n2) In Appendix A3 you provide cosine similarity histograms for the t1 != t2 case split up by layer. I would like to see the t1 = t2 histogram (Figure 2b) split up by layer as well, maybe in an Appendix. It would be interesting to know whether the TLCM mechanism is bimodal in individual layers.\n\n3) In line 273 you say that “The mechanisms that handle the latter two tasks must be non-trivial as the previous layer’s contribution is hidden inside the residual stream soup” – do the superposition hypothesis, and the observation that vectors in high-dimensional spaces are often nearly-orthogonal, affect the difficulty of this?\n\n4) In the propose & reject hypothesis you propose that the following layer uses context to correct the predictions of the previous layer. This makes sense in the first (few) layers, but by the midpoint of the model the amount of context available to different layers (number of preceding attention layers) seems quite close. Do you still think this is the reason for the TLCM?\n\n5) In section 5.2 you ask “is the entire layer scaled back uniformly or are certain subspaces targeted for correction?” – scaling the entire layer back seems like a simplistic alternative, of course that wouldn’t be happening. Are there other alternative hypotheses?\n\n6) Line 293 introduces the Jacobian J – what are the dimensions / indices of this Jacobian? Is it the embedding dimension? How are derivatives across-token positions handled?\n\n7) In lines 334 to 355 you discuss how TLCM could cause certain SAE failure modes. This part would be more convincing if you included experiments e.g. checking whether low-specificity SAE features line up with negative-eigenvalue-subspaces.", "limitations": "The authors discuss some limitations throughout the paper. While a dedicated paragraph/section to discuss limitations would be easier to understand, it may not be strictly necessary.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\nThe TLCM mechanism observed in this paper is, as far as I know, novel and different to previous studies focused on attention heads [30, 31]. The results are striking and promise insight into the internal mechanisms of transformers.\n\nThe cosine similarity measurements over layers (Fig 1, 2a) and context length (Fig 2b) are clear experiments.\n\nThe comparison with a baseline of cosine similarity (lines 131 through 142) are great and help establish confidence in the statistical significance of the result. (Though it is not clear whether the “different tokens” baseline is the right comparison to compute the 3 sigma number.)\n\nThe intervention experiments (Figure 3) are impressive, showing an almost linear response of the correction layer to amplifying the previous layer contribution.\n\n**Weaknesses:**\n\nThe proposed “propose-and-reject” hypothesis is only weakly supported – it is a very specific mechanism, and I expect that the experiments would equally support alternative hypotheses. Additionally, as the paper mentions, it is unclear whether the full range of results in Figure 3 can be explained by the hypothesis, or whether an unrelated self-repair mechanism is involved [lines 268-269]. The paper would be improved by considering more alternative explanations for the observations.\n\nThe eigenvalue analysis of the Jacobian describes the effect of layers in terms of subspaces. It is unclear how this explanation fits together with the superposition hypothesis (which the authors seem to believe, based on the equation above line 90).\n\nIn lines 334 to 355 the paper discusses TLCM explaining issues with SAEs and steering. I was not convinced by this that TLCM would explain the SAE and steering issues. (see “Questions” for suggestions)\n\nSection 5.2 is less clear than the rest of the paper, specifically how the Jacobian is computed is unclear (see below), and the proof is hard to follow.\n\n*Formatting issue: Equation labels are partially missing; equation (3) is referenced but not labelled. It would be good for the early equations in the paper to have labels.*\n\n*Minor typo: “begins to diminishes”*", "was_revised": true, "final_justification": "What issues were resolved, which issues remain unresolved:\n\n> The proposed “propose-and-reject” hypothesis is only weakly supported – it is a very specific mechanism, and I expect that the experiments would equally support alternative hypotheses. Additionally, as the paper mentions, it is unclear whether the full range of results in Figure 3 can be explained by the hypothesis, or whether an unrelated self-repair mechanism is involved [lines 268-269]. The paper would be improved by considering more alternative explanations for the observations.\n\nI appreciate the clarification of the propose-and-reject hypothesis and its context in the rebuttal.\n\n> The eigenvalue analysis of the Jacobian describes the effect of layers in terms of subspaces. It is unclear how this explanation fits together with the superposition hypothesis (which the authors seem to believe, based on the equation above line 90).\n\nI believe this question hasn't been clearly addressed, but I'm not confident this is a big issue.\n\n---\n\nI did not find the terminology or lack of rigor to be an issue when reading the paper. \n\n---\n\nOverall this paper is clearly one of the most interesting recent works in mechanistic interpretability, and I maintain my rating of 5: Accept. I increased my Quality score from 3 to 4 upon reviewing the rebuttal.", "rebuttal": "We’re glad you enjoyed our work, finding our results “striking,” “novel and different from previous studies,” and “impressive\\!” We answer your questions below:\n\n>Did you subtract the dataset mean before computing the cosine similarity? I wonder if a non-zero mean could produce an offset in the cosine similarity.\n\nGood question. We did not subtract the mean. However, we think:\n\n* The distribution of cosine similarities (see: Figure 2b) is ***strongly*** bimodal, which we think is the strongest evidence that there are (at least) two modes of layer operation: TLCM and non-TLCM.  \n* We think it’s more interesting & surprising that the absolute cosine similarity sits strongly below zero, rather than the relative/z-scored cosine similarity is low. This is contrary to our expectation that contributions between layers should have positive cosine similarity (or at minimum be orthogonal), as we discuss in L. 99-105.\n\n>In Appendix A3 you provide cosine similarity histograms for the t1 \\!= t2 case split up by layer. I would like to see the t1 \\= t2 histogram (Figure 2b) split up by layer as well, maybe in an Appendix. It would be interesting to know whether the TLCM mechanism is bimodal in individual layers.\n\nThanks for raising this\\! We included this exact plot in Figure 2b (observe the strong bimodality) [L. 127-130]. We realize this should be made more prominent, so we’ve updated Sec. 4.1 to discuss this plot more.\n\n>In line 273 you say that “The mechanisms that handle the latter two tasks must be non-trivial as the previous layer’s contribution is hidden inside the residual stream soup” – do the superposition hypothesis, and the observation that vectors in high-dimensional spaces are often nearly-orthogonal, affect the difficulty of this?\n\nInteresting point. Yes. The superposition hypothesis suggests that the residual stream will be some linear combination of feature vectors. We think identifying and correcting the features specifically contributed by the previous layer is a difficult task, unless there is some commonality between those features. For example, the features are wrong. This directly leads us to the *propose-and-reject* hypothesis.\n\n>The proposed “propose-and-reject” hypothesis is only weakly supported\n\nThe propose-and-reject hypothesis states that a layer proposes a broad swath of possible “features,” while the following layer—equipped with an attention head to gather context—removes the features that are not relevant.\n\nWe offer the propose-and-reject hypothesis as a conceptual synthesis that is consistent with all our empirical results thus far. Specifically, the hypothesis explains:\n\n* Quantitative contextual dependency experiments.  \n* TLCM corrects *only* the most recent layer, rather than correcting *all prior layers*  \n* TLCM’s prevalence in early layers, which are linked to “enrichment”  \n* Very low TLCM activations on beginning of sequence tokens  \n* Attention/MLP experiments which show that TLCM emerges when considering ***their combination*** (MLPs cannot access context without attention)  \n* The causal relationship we show—where the following layer linearly & dynamically attenuates the prior layer—in Sec 5.2.  \n* Sec 5.3’s Jacobian experiment demonstrating specific reinforcement and **unit** attenuation ($\\\\lambda \\= 1$) of features\n\nWith this framing, we intend to **generate future research directions**, not to close the discussion.\n\n>In the propose & reject hypothesis you propose that the following layer uses context to correct the predictions of the previous layer. This makes sense in the first (few) layers, but by the midpoint of the model the amount of context available to different layers (number of preceding attention layers) seems quite close. Do you still think this is the reason for the TLCM?\n\nGood point. This seems reasonable and likely explains why the prominence of TLCM **decreases in the final two-thirds of model layers**. Figure A5 illustrates this phenomenon across many models.\n\n>In section 5.2 you ask “is the entire layer scaled back uniformly or are certain subspaces targeted for correction?” – scaling the entire layer back seems like a simplistic alternative, of course that wouldn’t be happening. Are there other alternative hypotheses?\n\nIn Appendix A, we provide a theoretical rationale for why the model might fully “scale back” the previous layer. Essentially, we introduce the concept of LayerNorm blindness, which is the phenomenon that LayerNorm causes attention and MLP sublayers to be “blind” to the norm of the residual stream. Thus, they do not know how to size their contributions and may be biased to over-contribute so their contribution is not overshadowed. However, we find TLCM in models that do not have LayerNorm blindness, contradicting this explanation.\n\nThis fact (along with our Sec 5.2 Jacobian eigenbasis and Sec 4.3 experiments) strongly suggest that TLCM is responsible for attenuating a small subspace of the prior layer, leading us to the *propose-and-reject* hypothesis, in which layers are correcting incorrect features added by the prior layer.\n\n>Line 293 introduces the Jacobian J – what are the dimensions / indices of this Jacobian? Is it the embedding dimension? How are derivatives across-token positions handled?\n\nWe take the layer Jacobian $\\\\mathbf{J} \\\\in \\\\mathbb{R}^{d\\_\\\\text{model} \\\\times d\\_\\\\text{model}}$, which includes both the attention and MLP sublayers. We are taking the Jacobian with respect to the output of the prior layer, so the effects of other tokens are considered (linearly) in this Jacobian. In practice, this is materialized by computing the gradient of the layer's 4096 outputs, one at a time using PyTorch.\n\n>Section 5.2 is less clear than the rest of the paper, specifically how the Jacobian is computed is unclear (see below), and the proof is hard to follow.\n\nGreat point! We have moved the bulk of the proof's content to the appendix and increased its clarity. In its place, we've included details on how we compute the Jacobian and the intution for the eigenbasis approach.\n\n>In lines 334 to 355 the paper discusses TLCM explaining issues with SAEs and steering. I was not convinced by this that TLCM would explain the SAE and steering issues. (see “Questions” for suggestions)\n\nThanks for giving this thought. We're curious to understand your concerns with this explanation.\n\n>In lines 334 to 355 you discuss how TLCM could cause certain SAE failure modes. This part would be more convincing if you included experiments e.g. checking whether low-specificity SAE features line up with negative-eigenvalue-subspaces.\n\nGood suggestion; we agree this would be valuable. Anthropic has predominantly completed the most substantial work with SAEs and high-quality labeled features. However, they release neither their model weights or learned features. We are excited about open-source SAEs with great natural language annotations, which would make experiments like these possible. For now, we see it as important future work."}]

Author rebuttal:

Meta-review: This work conducts a systematic study of consecutive transformer layers, uncovering the so-called Transformer Layer Corrective Mechanism (TLCM), which boils down to adjacent layers in transformers exhibiting 'anti-correlated' activations. They observe that this phenomenon occurs across number of open-source LLMs, though is absent in models that use dropout.

Three reviewers voted for acceptance post rebuttal as they all agreed the basic thesis of the paper (TLCM) to be very interesting and worth sharing with the community:
- One reviewer found the analysis insightful and that the empirical evidence supported the claims.
- Another indicated that documented this phenomenon was novel and interesting, that the paper was easy to follow, and that the systematic study was well executed.
- The third reviewer said that this was one of the most interesting recent works in mechanistic interpretability they had read.

One reviewer remained critical because of the lack of theoretical foundation and they would have liked to see more empirical rigor, something the other reviewers did not flag. The TLCM mechanism observed and studied in this paper is different to previous studies focused on attention heads, and thus provides new insights into the internal mechanisms of transformers. I am also of the opinion that these new insights are worthwhile sharing with the community.
Decision: Accept (poster)