Emergence of Linear Truth Encodings in Language Models
Authors: Shauli Ravfogel
Gilad Yehudai
Tal Linzen
Joan Bruna
Alberto Bietti
Venue: NeurIPS 2025 poster ()
Keywords: truth
factuality
linearity
LMs
Primary area: deep_learning
TL;DR: We propose a toy model that shows how linear truth encodings can arise in language models.

Abstract: Recent probing studies reveal that large language models exhibit linear subspaces that separate true from false statements, yet the mechanism behind their emergence is unclear. We introduce a transparent, one-layer transformer toy model that reproduces such truth subspaces end-to-end and exposes one concrete route by which they can arise. We study one simple setting in which truth encoding can emerge: a data distribution where factual statements co-occur with other factual statements (and vice-versa), encouraging the model to learn this distinction in order to lower the LM loss on future tokens. We corroborate this pattern with experiments in pretrained language models. Finally, in the toy setting we observe a two-phase learning dynamic: networks first memorize individual factual associations in a few steps, then---over a longer horizon---learn to linearly separate true from false, which in turn lowers language-modeling loss. Together, these results provide both a mechanistic demonstration and an empirical motivation for how and why linear truth representations can emerge in language models.

Emergence of Linear Truth Encodings in Language Models
Shauli Ravfogel'
Gilad Yehudai
Tal Linzen' Joan Bruna'
Alberto Bietti²
NNww York University FFlatiron Institute
Abstract
Recent probing studies reveal that large language models exhibit linear subspaces that separate true from false statements, yet the mechanism behind their emergence is unclear. We introduce a transparent, one-layer toy model that reproduces such truth subspaces end-to-end and exposes one concrete route by which they can arise. We study one simple setting in which truth encoding can emerge: data distribution where factual statements co-occur with other factual statements (and vice-versa), encouraging the model to learn this distinction in order to lower the LM loss on future tokens. We corroborate this pattern with experiments pretrained language models. Finally, in the toy setting we observe two-phase learning dynamic: networks first memorize individual factual associations in few steps, then longer horizon-leam to linearly separate true from false, which in turn lowers eanguage-modeling loss. Together, these results provide both a mechanistic demonstration and empirical motivation for how and why linear truth representations can emerge in language models.
Introduction
Recent observations suggest that large language models (LMs) often encode a low-rank linear subspace that distinguishes true from false statements across a wide range of domains [Azria and Mitchell]
2023 Burns et al., 2022 Li et al. 2024b Marks and Tegmark, 2024 Bürger et al. 2025 Orgad al. 2025] Specifically, in many layers of the residual stream representation in transformer- based LMs, a linear separation emerges between representations corresponding to true versus false assertions. Moreover, this separation generalizes across domains: there exists single separating subspace such that statements like "2 + 2 = 4" (true) and "The capital city of France is Rome" (false) fall opposite sides of the same separating plane. These findings have sparked interest among practitioners, because they may aid mitigating hallucinations [Li et al. 20246) Orgad et al. 2025
We investigate the emergence of unified "truth subspace" -imeemmenioonal linear manifold that cleanly separates true from false statements. Prior work shows (i) that outh-encoding directions generalize remarkably well across diverse tasks and prompts, and (ii) that causal interventions along those directions can steer LMs toward factual or counter- factual completions [e.g. Me. et al.) [002]] Yet still lack satisfying answer two fundamental questions: why do such subspaces arise during training, and how are they actually computed at inference time?
We address both questions in a single theoretical and empirical framework. For the how, we build on the growing understanding of key-value associative memories in transformers. (Gva al. 2021] showed that the first linear layer produces key matches- e.g. aligning the prefix "The capital city of France is" with an internal query-while the second linear layer retrieves the associated value, such as the hidden representation of "Paris". Subsequent studies refined the mathematical description of this mechanism and demonstrated its causal role in factual recall and reasoning [Geva et al., 2022b Bietti et al.) 2023 Cabannes al.) 2024b Nichani al. 2025 We hypothesize that linear truth code takes advantage of the memorized factual associations: it emerges as a result of the model contrasting
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the internal prediction it built with the observed attribute. This results in a different pattern when the two match or mismatch, and is translated into a linearly separable signal.
For the why, we propose the Truth Co-occurrence Hypothesis (TCH): in naturally occurring text, true statements are statistically more likely to co-occur with other true statements, and falsehoods with other falsehoods. This assumption is closely related to recent "persona" explanations of factual inconsistency in LMs (Li al. 2023a Joshi et al. 2024]: the claim that LMs learn to model certain personas in the data distribution, some truthful and some not. TCH offers a very simple way to quantify the persona hypothesis and provably characterize its influence. Under the TCH, inferring a latent truth variable is loss-reducing: if the model recognizes that "It's well known that the moon landing was a hoax" is false, it can raise the probability of a continuation such as "and that the Earth is ffatt"" which is likewise false.
We test the truth-co--ocurrence hypothesis (TCH) in the minimal transformer-like model, with single self-attention layer, one head, a normalization layer, and no MLP. Training examples are four-token sequences x y x' yf with subjects x, x ("The capital city of France"; "Churchill's nationality") and attributes y, 3/ ("Paris"; "British");; with probability p. the attributes y, y are both the correct attribute; with probability 1 1-6 p. they are replaced with random one. Under our simplified generative story, "truth" is identified with the attribute that is frequent in the training data for a particular subject. When we train LMs on such dataset, we find that after the key-value lookup circuit forms, gradient descent pushes hidden states toward linear separator that clusters true vs. false contexts, and the model uses modify its confidence when predicting the attribute. Training shows two phases: rapid key-value acquisition followed by slower emergence of linear encoding. Although our toy model far simpler than natural training data (see Section 6). it predicts the observed sensitivity to false context (Section 5.3), where false prefixes bias later predictions (supporting TCH), and reproduces the way normalization layers regulates confidence (Stolfo et al., 2024]. Taken together, we show that linear truth encoding can arise without any built-in semantics.
2 Related work
A growing body of work shows that pretrained LMs linearly encode a simple notion of "truth"
consistency with the majority of examples in the training data in both hidden states and individual
ML.Plattention outputs [Azaria and Mitchell| 2023, Burns et al. 2022 Li et al.] 2024b |ürger et al., 2025). This feature is generally robust for frequent atomic facts, though its subspace can
shift in the presence of negation [Marks and Tegmark| 2024) and may by biased to dataset-specific
features [Orgad et al. 2025]. The encoded truth dimension is behaviorally relevant: intervening on nudges the model toward truthful completions [Li et al. 2024b although the model's predictions
sometimes do not agree with the latent encoding [Luu et al.) 2023]. Yet the mechanism behind this encoding remains unclear. Extending the persona hypothesis of Li et al. [2023a]. Joshi et al. (2024), Ghandcharioun et al. [2024) link truthful behavior to lexical "personas" for instance, the formal, encyclopedic style typical of Wikipedia versus the more casual tone common in social-media post. We show that, given sufficient training, LMS also acquire a lexicon-indeee inceeedeendeennenttt abstract truth dimension that emerges more slowly.
The line of work truth encoding is closely related findings suggesting that models encode different aspects related to their knowledge and confidence. It was shown that it is possible to decode "latent" knowledge from the model Gekhman et al. [2025). and that measures of uncertainty can be decoded from hidden states [Slobodkin et al. 2023) Farquhar et al. 2024] Ferrando et al.) 2025) Our work is related to, but distinct form, works on mechanistic understanding of hallucinations Yu et al. 2024]; while both rely on the associative memory used by the model [Geva et al. 2021 2022a b Bietti et al. 2203]] [aaaannes et al., [2024a]. we focus on the emergence of separation between true and false assertions, and come up with toy model that allows us to analyze its properties.
3 The Truth Co-occurrence Hypothesis
We previously described the TCH, the assertion that false statements tend to co-occur. To quantify that, we use the MAVEN-FACT corpus [Li et al., 2024a], where annotators assign FactBank- style factuality label to every event mention inside a news article. After discarding all but certain judgments, each mention is labeled CERTAIN-TRUS CERTAIN-FALSE and grouped by the document

---

in which it appears We find the following: (i) the overall certain-falss rate is p = 0.0209; (ii) the chance that two event mentions from the same article are both certain-false 0.0009, exceeding the independence baseline p² = 0.00044 by factor of R 2; and (iii) the clustering ratio- Varan Varbinom 11.23 shows 23 extra article-to-articl heterogeneity.
x² test of independence confirms the association (x2 = 4.17 x 10 p R 9 10 10-19). This shows that false assertions are not sprinkled at random but tend to cluster on the same article. For a language model, tracking latent truth bit is therefore loss-reducing: once page provides evidence that one statement refuted, the conditional probability that a subsequent claim is also refuted increases. This motivates the design of a simple data- generating process that instantiates the hypothesis and tests whether it gives rise to truth encoding.
3.1 Data Generating Process
Natural text confounds truth with stylistic cues, topic priors, and corpus frequency (Orgad et al. 2025]. Therefore, Consequently, if we probe LMs on raw text, we risk discovering features that merely track these proxies. To uncover minimal conditions force LM to represent truth, build toy world which:
1. Every subject pair has exactly one canonical attribute (ground truth).
2. A small, controllable fraction of examples corrupted by uniform noise (the attribute is replaced with another attribute).
3. importantly, the truthfulness of neighboring sequences correlates; this models the tendency of speakers to consistently be less or more truthful (Joshi et al. 2024
Despite its simplicity, this environment reproduces the linear-separability we see in large-scale LMs (55)
Data format. Each training example is a sequence x y x' y with subjects x, x' E S and attributes y. A.
For every x there exists unique ground-truth attribute g(x) memorized by the data generator. Examples are corrupted as follows: Sample T (ernoulli(p) once per example, such that
TRUE If T==,, set y = g(z). v/ g(x').
FALSE If I-0,, draw each y, 3/ independently and uniformly from A.
Truth as latent variable. Because predicting the second attribute token y' is easier when T is known, an LM can lower its language-moder loss by internally inferring T early in the sequence and propagating that bit forward.
Without inferring the conditional distribution over the second attribute y' given the prefix (a,, y, x¹) is:
1 1- Pr(y' = a + g(x') x. y,)' = [A]
Pr(y' = =(aa)) y, = = LAI
Assume the LM can memorize g and (optionally) infer perfectly. Let L-r be its per-token cross- entropy for predicting y when it does not access T, and let Cr be the loss when it embeds T internally, Then, the [A] - oc limit, L-T Lr H2(p),, the binary entropy of p. Hence representing single bit yields maximal benefit at p = 0.5, where H2 is largest (see appendix B for complete derivation).
4 Analysis on a Toy Model
In this section, we study the emergence of truth directions in a simplified one-layer setup with orthogonal embeddings. Empirically, we find that this minimal setup already captures the mechanism of a truth direction, and leverages layer-norm to adjust confidence for the second attribute depending truthfulness of the first one. Our empirical and theoretical analysis shows that this happens in phases, and that layer-norm is crucial to provide the relevant structure in the gradients. Furthermore,
Patta-handling details are deferred to App.
3

---

step 20
step 100
step 500
0
0
2.0
1.5
1.0
0.5
40
40
0.0
0.5
60
1.0
-15
2.0
20 40
60
80
20 40
60
80
20
40
60
80
Figure 1: Visualization of the value matrix for the one-layer model at different training steps. We that the ez + Ug(x)) block is learned first, along with the Pt block. Later the ex Ex and ey + Uy blocks, and finally the ey + (v) block.
such a truth direction can already emerge when there are only true sequences. In appendix D.I we discuss how these results may be extended to non-orthogonal and learned embeddings.
Setup. Consider the following one-hot token embedding, positional embedding, and unembedding vectors in Rd with embedding dimension d 4N + where E [2.N] is input output token (input tokens x are in [N] while outputs y are in [N + 1, 2N),, and t E [3] position:
[ez]: = 1 = =}
[pr]: = 1 (i 2N +1}
(2)
[ua]: = 1 = 2N +3
We consider one-layer transformer with uniform causal attention, and basic layer-norm operation. Concretely, for an input sequence 21:3
(2, y, x') and position E [3], define:
Fw (21:) = N cz2 + + Σ W(z, + +
==1
where W denotes the value matrix, U "122N O0; 12N] R22xd is projection on the unem- bedding dimensions, and N(v) //||u||
||||||]]/ is layer-norm operation. The predicted probabilities are then given by f(=++1 - (2..)) (g(F (F(( (=):)), where S3 denotes the softmax operation with inverse temperature B. Our experiments use = dd, due to the use of RMS norm in layer-norm over embeddings of dimension d.
We assume here that x, x' Uni ii.d., and conditioned on these well as on a truth random variable T ~ Ber(p), we have y = g(z) and y g(z') when T = 1, and y.y' ~ Unif([N + 1, 1,2 otherwise. Denoting 21:1 (zz y, v/) the population loss then takes the form
3
3
L(W) = Σ L.W)) Σ Eaa log S3 (Fw (214))
(5)
Probing the mechanism and its emergence. Figure shows visualization of the value matrix W in our toy model, at different steps of training, with N 20, 0.8 and batch size 16. We see that clear block-structure emerges in the matrix W. with different blocks arising in different phases. Some blocks show a negative identity structure, while others show a permutation structure according to the "knowledge" mapping g. Positional embeddings show more uniform patterns across unembeddings, with difffernnt signs depending on whether the next token is an input or label. In Tigure[] we show the representations at the x' token for examples of and false sequences, before and after layernorm, as well the probabilities obtained after projecting to the unembedding space and applying softmax. In the false sequence (bottom plot), we notice large spikes in the input embedding dimensions (1-20) at positions x = 5 and (v) = 16. These do not exist in true sequences, since they cancel out. We see a similar behavior on unembedding dimensions (65-84) at smaller scales. The cancellation leads to smaller norm on true sequences, which causes an amplification of the logits, and finally spiked distribution on true sequences, versus flatter one on false sequences, though we still some lower confidence spikes (xx) and g(x') (note the y-axis scale difference).

---

Pre-LN representation
Post-LN representation
Probabilities
x=5,
y=21, g(x]=21, x g sequence)
5.0
02
2.5
0.0
20
40
80
20
90
80
10
20
30
40
X=5, y= g(x)=21, x'=2. =5 ftaise secuance)
0.5
00y-1 223155
X
0.0
9K--0
de1-25
0.0
20
30
3,30
3.0
p0m
3,00
X
Figure 2: Visualization of representations true (top) and false (bottom) sequences. The plots show representations before (left) and after (center) lare--oom, as well as predicted probabilities (right).
Structure of the value matrix W. We now study a construction that resembles the one observed empirically in fig. [ Later we will provide a thcoretical justification for this structure and its emergence in phases by analyzing training dynamics.
The leftmost column of the W matrix maps ez to its corresponding label "g(z), while also subtract- ing e itself:
We, -1 + 81 (()))))
(6)
with a1, 31 > The second column has the following symmetric behavior:
Wey (x)))-(w) B2uy.
Finally, the third column maps the different positional embeddings to mixtures of uniform distributions over the inputs labels:
Wpi = 71
Σ
Uy
Ux un)
(8)
y
Wp2 -72(
Σ By
Wp3 = 73
Σ Uy
Σ Uz).
(10)
y
In the statements above, assume all the coefficients 01/2; 91/22 11/2/3 be positive.
Linear separation and sharpening mechanism. One important consequence the structure above is that any token that attends to both and y (this could be either y or x') has the following quantity in its residual stream:
((x, v) = (ee + ey) = -n + (x26g-1 + B1 (g())))))
Ball-
(11)
We then have
[|((,,, 20. 022 2,3132. for * g(x).
Since a1, 02,02, 31, B2 > the norm of on true sequence is always smaller than false sequence, leading to a useful feature for detecting truth (see illustration in fig. [6). Combined with the layer- norm operation, this provides mechanism for sharpening the prediction of towards g(x') when the model detects a true sentence, by adjusting the temperature in the softmax via inverse norm scaling.
Theorem (Sharpening of yf predictions). Suppose we have solution that satisfies Eqs.
6
10
Denote by e:= (2N 2)+22f+77 For any x, a' and y F g()), we have:
B1 max(0, 31 -3)
x')g(o) max kgg(z")
F(x, (t..gxx, 2
3 c (B1 32 + +7)² + (81 7)
F(z, y, g')(0)
max Flz, y, =0
ktbl)')
5

---

The proof is in appendix E.2 This shows that the structure of W along with layer-norm provide simple mechanism to make the model more confident about its knowledge when the context is truthful. For false sequences, the zero gap comes from the fact that logits for g(x) and g(r") are tied, as we show empirically in Figure This aligns with previous interpretability work on confidence
neurons [Stolfo et al. 2024). Beyond improving prediction performance, we now show that this model provides a linear encoding truth in the representations after layer-norm.
Theorem 2 (Linear truth direction). Suppose we train the model in (4) as explained above, and reach
a solution for W that satisfies Eqs. - T0P.
Then, we have the following:
1. If the model in 4 does not contain N, then its output on the y token does not admit a linear separator for true and false samples.
2. If the model in (4) contains N and 20 13 # O, then its output on the y token admits a linear separation for true and false samples. Moreover, 7n = 722 01 = Q22 31 = $2
then the margin is at least 8
2v2
VI+a2+82
The proof is in appendix
E.3
Theoretical analysis of training dynamics. from training dynamics in simplified setting.
We now study how such a structure in W emerges
Theorem 3 (Sequential gradient learning; informal). In simplified model with positional embeddings, taking two gradient steps on L1 (predicting y from x) followed by one on L3 (predicting y' from x' all with step-size (NN), leads to the desired structure for W as in Eqs. 6 up to negligible entry-wise O(1/N) terms.
See a formal statement and a proof in appendix E. This result shows that gradient dynamics in our model can quickly lead to the block structure observed in Figure despite the non-convexity induced by normalization. In fact, the analysis reveals that the layer-norm operation is crucial here to obtain many of the desired blocks other than the ez + "g(x)) Interestingly, our theory shows that this structure arises even when o 1, and empirically we found that both sharpening and linear separation indeed happen in this setting, demonstrating an emergent out- out-of-distribution generalization to false sequences. We note, however, that this may not happen in a more expressive model: we empirically found that if we also train the key-query matrix with p = 1, the model quickly learns to focus its attention to the current token (see also appendix E...IIfor a theoretical justification), which makes information from the context inaccessible from the residual stream. While this may improve predictions of g(x") on true sequences by removing noise in the residual stream coming from (r, y). this also results in a failure to handle false sequences.
5 Experiments
5.1 Synthetic Setting
Setup. We train minimal self-attention LMs on the synthetic dataset described above. The model contains 1 self-attention layers with single attention head, followed by layer normalization, with no feedforward network. See Appendix for more details. In this setting, in contrast to the toy model described above, we train all parameters, including the dense embeddings and the attention module Each training example is a a concatenation subject (x'), and an additional attribute (y'). ). The attributes y, y' are either sampled uniformly or taken to be the correct attributes g(x), g(x),, according to the true probability p. line with the Truth Co-occurrence Hypothesis, we aim to measure whether in this training setting the model is able to recover the latent truthfulness of the first sequence (verifying whether y = g(z)) and use it to decrease LM loss the second attribute y/.
subject (x), an attribute (y), an additional,
We experiment with true-attributes rates p. and with (1,2,3) layers, and assume perfect correlation between the truthfulness of the first and second attributes (that is, y = g(X (X) if, and only if, y' = g(x')). Along training, we logistic-regression classifiers on all hidden states to predict
We release the code in http:///hhhhooooooomoiiiiiii///e

---

whether or not the sequence is false (a binary classification problem). We fit individual classifiers both the first attribute position (z). as well on the second subject position (z'). from which the second attribute s/ is predicted. While in training the LM we use a varying true-attribute rate p. the linear classifiers are always trained and evaluated on a balanced set, containing 50% true sequences. We report mean results over 5 runs with different random seeds. Unless specified otherwise, we present here results for = 1 and p 0.99, |A| = |S| = 512 and dmodel = 256; results for other settings are deferred to Appendix D
Second subject (x) Linear Separability (p- 0.99)
Probabliity of True Attribute, False Sequences (p - 0.99)
1.0
1.0
min- -in range
mean
0.8
0
10000
20000
30000
40000
50000
0
10000
20000
30000
40000
50000
Training
Traininng ng stop
(a) Truth classification results, second subject i
(b) Pig(x') }] on false sequences, for which y # g(x'))
Figure 3: Truth linear classification results alongside probability assigned by the LM to the true attribute on false sequences.
5.2 Results
Two-phase dynamics. In Figure 3a we show the linear truthfulness classification AUC as a function of training steps, on the second subject. for a 1 layer model with true-atrribute probability & 0.99. Additionally, we plot the probability the LM assigned to the correct attribute on false sequences (P(y'- = g(x') y # g(x)); Figure 5b). When this probability is minimized, the model improves its loss on false sequences.
In line with the toy model, we detect distinct phases in training.
1. Memorization, As can be seen in Figure 5b memorization happens hapidly-within the first 1000 batches-as as the model converges to a probability of around to g(x') on both true and false examples. Indeed, the model predicts the correct attributes on over 99% of the true sequences.
2. Truth encoding. The model does learn to linearly encode the truth latent variable. This encoding emerges abruptly, after around 7,500 batches, during which the model saw around 1 million examples, relatively long after the model achieves perfect memorization.
The model learns decrease the probability it assigns to the correct attribute on the second attribute position P(g(x") y/) roughly at the same time linear classification emerges.
Truth circuit. We aim to understand how the linear truth subspace is being computed. While it has been empirically shown that LM linearly encode many human- interpretable concepts [Bolukbasi] et al. 2016 Vargas and Cotterell 2020 Ravfogel et al. 20022. it is not well-understood why linear representations emerge in hidden layers [Park et al.) 2024, Jiang et al.) 20244.. The toy model we propose allows us to empirically study the origins of the linear signal, and the way it is being used to decrease LM loss on the second attribute.
The truth encoding appears in a I-layer model (classification accuracy in the input embeddings layer majority level). As can be seen in the first layer attention pattern in (Figure 4b), this attention head calculates an approximate mean of the embeddings of x and y. after application of the V, O self attention matrices, in line with the uniform attention assumed in the toy model. One key difference is that here, we learn the input embeddings. Interestingly, inspecting the PCA of the input subject and attribute tokens (Figure reveals that approximately, e,- =x((2) on the first principal component. This explains why both The true and false representations tend to cluster around the origin. Following
the attention averaging, we apply RMSNorm. We find that linear classification emerges only after normalization; classification accuracy is at majority level before it. Indeed, PCA plot (Figure
7

---

Layer No Nerma toaion
27A strppr Entedtings of Satjers Jane Arrbuiee
BOS
Query
X
y
BOS X
x'
Key
(a) PCA of layer-1 representations on the token x''
(b) Attention pattern (I-layer model)
(c) PCA of input embeddings X
versus Y
Figure 4: LM representations over false sequences.
shows that, predicted by the toy model, the TRUE class is centered around the origin, with larger variance for the TRUE class than the FALSE class. Normalization induces linear separability, that is also evident in the first PCA components.
Additional settings. far we have analyzed a single-layer self-attention network-eithe with one- hot embeddings, with trainable dense embeddings and p 0.99. Results for other configurations appear in Appendix Dj here we outline the main trends. The patterns in Figures [a and (5b] persist across layer counts L, noise levels p. and corpus sizes [8], |A]. Higher delays (but does not prevent) the onset linear separability, which still emerges at p = 0.999 (Figure aa at the appendix); only the degenerate case 0 1.0 shows emergence, contrary to the toy model. We discover similar structures to Figure [ also when training with frozen dense embeddings and when learning the KV matrices instead of using fixed attention. A preliminary analysis of this setting provided in appendix D. and appendix I.I.I and we leave more complete understanding for future work. With additional layers the model sometimes encodes truth in the first attribute y, then copies it x' before predicting y' in other runs it reverts to the single-layer strategy where x' attends directly to x and y/. This influences whether we see linear encoding on both y and x', or on x' alone (Figure in the appendix).
pSrersisssss
5.3 Testing the TCH in Real LM
The theory we specified relies on a of assumptions and architecture that do exist pretrained transformers (those have, for instance, MLP layers in addition to the attention layer; have multiple attention heads: and are trained primarily on natural language distributions). Below, (i) train "regular" transformer models on natural language data that instantiates the truth co-occurrence hypothesis; (ii) assess what extent aspects of the mechanism we propose exist pretrained LLMs.
5.3.1 Instantiating the TCH in Natural Language
we created synthetic dataset that respects the TCH and showed that training an attention-only transformer on this data results in linear truth encoding. Here, we aim to assess whether same thing happens when training "real" transformers on natural language data.
In section 5.1
Setup. We evaluate on the CounterFact dataset Meng et al. 2022 collection of simple factual assertions spanning relations such as SPEAKSLANGUAGE and BORNIN. We select the 25 most frequent relations and, for each positive instance (æ, a), construct a negative by replacing the attribute a with a different attribute from the same relation. To instantiate the TCH, we form paired examples by concatenating two randomly sampled instances that share the same truth label (both true or both false). We then train small transformer with RMS normalization, attention heads and single MLP module per layer, hidden size d = 256, and depth E {2,5, 9} this corpus. We use

---

3.7
AUC Trasing 1L 1)
Mein protantity of true tohmm
Fiyuan on RALSE ecsmolee
Uncer Unnerrrroee probe AIC
0.2
Layer
Training
3500
12502
15920
(b) P(correct attribute] on false sequences, for which the
(a) Truth classification results
observed attribute is not the correct one.
Figure 5: Truth linear classification results alongside probability assigned by the LM to the true
attribute on false sequences.
/ 0.99. We train data from single relation at time, and report mean and standard deviations over 5 random relations
Results. Across all seeds and architectural choices, the training dynamics mirror those synthetic data: rapid memorization, followed by the emergence of a linear encoding, and an increase in entropy on false sequences. In fig. [S] we show results for single relation (WORKSIN; averaged over five random seeds). By the end of training, the final hidden layer is nearly perfectly separable by the truth label, and false sequences the probability assigned the memorized ("true") attribute declines. Notably, the 1-layer model exhibits epoch-wise double descent: classification accuracy rises early, dips, and then rises again. Across the five seeds, relations, and model sizes, memorization proceeds at roughly the same rate; the main variance lies in how quickly the probability declines on false sequences.
5.3.2
The TCH in Pretrained LLMs
log P of correct attribute: Baseline Truthfulness Intervention (Layer 11)
(ight)
intevertine
15.0
attirbute)
12.5
2.5
0.0
8F
T1T
FFFF
Context Factuality
(a)
The mechanism proposed above assumes a very specific data generating process, and a simplified transformer model. As such, it is not likely that the same mechanism applies real LMs; we see the toy model as a proof of concept, and aim to study more complicated models in future work. Yet, in this section, we compare the predictions following from our hypothesis with pretrained LMs in these aspects: (1) the sensitivity of the model's 's predictions to preceding false sentences, in line with the truth co-occurrence hypothesis; (2) the behavioral relevance of the linear truth encoding situation where sentence follows misleading false sentences. We experiment with a LLAMA3-88 model Grattaaiooi et al.) 2024 and the CounterFact dataset (SPEAKSLANGUAGE relation). We let the model predict the first token of the last word of sentence, when it is (i) preceded by false
We leave the question of generalization between relations future work.

---

sentences; or (ii) preceded by true sentences. In line of the hypothesis, we expect to see a decrease in the probability of the correct answer.
Model's predictions are sensitive to preceding false sentences. The results, over 128 n ettuples, presented in Figure 6a (light bars) and are in line with our hypothesis; for instance, in the two leftmost box plots, we see that preceding the sentence with two false sentences (FF) yields higher negative likelihood (smaller probability)
the correct attribute compared with when preceding with one true sentence (TT). The difference in negative log likelihood is 1.52, corresponding to 4.55x decrease in the probability of the correct attribute.
Intervention in the truth subspace. LLAMA3-8P encodes truthfulness linearly: a linear classifier reaches over 95% accuracy all middle and last layers in separating true instances from the dataset from counterfactual ones. Our theory predicts that, in the presence of misleading context, the direction that distinguishes true from false vectors actively pulls the model away from the correct answer. To test that, we intervene in the truth subspace. Following previous work on linear steering [Li et al.) 2023b Singhh al., 2024], we calculate the mean vector of the TRUE and FALSE classes in the representation space, FTT and μAF, and add a steering vector ( μF) to all representations in the same layer with the goal of increasing the probability of the correct attribute. We choose layer / 11 based on preliminary experiments that showed that classification peaks at that layer, and = 3.0. The results, presented in Figure ba (darker bars), show that the models tend to increase the probability of the correct attribute iestinnerrventin,, even in the presence of false context.
Emergence along training See appendix E.4 for preliminary analysis of the emergence of truth encoding along training.
Discussion and Limitations
Although our analysis was grounded in a deliberately minimalist transformer, it discovers a two-phase dynamic-rapid key-value memorization followed by the slower emergence of linear truth encoding. circuit able to retrieve
The key prerequisite appears to be the presence of (i) an associative-mente subject-attribute pairs and (ii) correlation among the truth values of adjacent clauses. We emphasize that this is one, and probably not a unique, mechanism that can induce truth encoding. A core advantage of the minimalist model is that it does not assume any lexical cues that help the model discern the truth latent variable. In that sense, this is more challenging setting than the previously studied one (oshi al. 2024), where it assumed that true and false assertions are associated with different lexical distributions.
Several core differences exist between our simplified generative story and real-world setting. Our synthetic corpus contains only one latent relation. A natural extension is to sample tuples from set of heterogeneous relations Milations-BRNINNN.. CAPITAL,FF, CURRENCYOF, while maintaining correlation in the latent truth bit. Doing so forces the model to contextualize its memory: the same subject embedding must participate in multiple key-value slots distinguished by the relation. Real corpora have logical and semantic dependencies that go far beyond pairwise subject-attribut pairs: transitivity ("A is in B" "B is in C" "A in C"), mutual exclusivity ("isAlive" vs. "IsDead")) and type constraints ( ("capitalOf" only applies to geopolitical entities). These constraints also greatly limit the range of plausible counterfactual variants we may see in the training data; while we assume a uniform corruption for simplicity, in practice false variants of factual claims come from a unique conditional distribution.
7 Conclusion
We introduced a small transformer and a synthetic data- gettaaeeneation process that jointly suffice to yield a robust linear truth subspace. Our analytical and empirical results demonstrate two-phase training dynamic: memorization followed by truth-code emergence. Unlike prior persona-based accounts, our theory does not rely on surface correlations between individual tokens and truthfulness, and points out to possible mechanism behind the emergence of linear of the truth signal as a latent variable inferred by the model.
10

---

Acknowledgments
This work was supported part through the NYU High Performance Computing resources. services, and staff expertise, and by the National Science Foundation (NSF) under Grant No. IIS- 2239862 to TL. We thank Yanai Elazar for his valuable comments on a previous version of this paper.
References
Amos Azaria and Tom Mitchell. The internal state of an 1lm knows when it's lying. In Findings of the Association for Compulational Linguistics: EMNLP 2023, pages 967-976, 2023.
Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of transformer: A memory viewpoint. Advances in Neural Information Processing Systems, 2023.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29, 2016.
Lennart Bürger, Fred A Hamprecht, and Boaz Nadler. Truth is universal: Robust detection of lies in Ilms. Advances Neural Information Processing Systems, 17:138393-13844 2025.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:22/2.03827. 2022.
Vivien Cabannes, Elvis Dohmatob, and Alberto Bietti. Scaling laws for associative memories. In International Conference Learning Representations, 2024a.
Vivien Cabannes, Berfin Şimşek, and Alberto Bietti. Learning associative memories with gradient descent. In Proceedings of the 41st International Conference on Machine Learning, pages 51 14 5134, 2024b.
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant, Analyzing transformers embedding space. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16124-16170, 2023.
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):6)) 630, 2024.
Javier Ferrando, Oscar Balcells Obeso, Senthooran Rajamanoharan, and Neel Nanda. Do i know this entity? knowledge awareness and hallucinations in language models. In The Thirteenth International Conference on Learning Representations, 2025.
Zorik Gekhman, Eyal Ben David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpektor, Jonathan Herzig, and Roi Reichart. Inside-out: Hidden factual knowledge in 1lms, 2025.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer ferd-forward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 .emnlp main.446. URL https://aclantholle org/ 2021. wwwee-maaaaeeeeeeeeeeee mapp.mmin.... 446/
Mor Geva, Avi Caciularu, Guy Dar, Paul Roit, Shoval Sadde, Micah Shlain, Bar Tamir, and Yoav Goldberg. Lm-debugger: An interactive tool for inspection and intervention in transformer-bast language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 12-21, 2022a.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing. Association for Computational Linguistics, 2022b.
11

---

Asma Ghandeharioun, Ann Yuan, Marius Guerard, Emily Reif, Michael A. Lepori, and Lucas Dixon, Who's asking? user personas and the mechanics of latent misalignment. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprini arXiv:2407.217785 2024.
Yibo Jiang, Goutham Rajendran, Pradeep Kumar Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. In International Conference Machine Learning, pages 21879-21911 PMLR, 2024.
Nitish Joshi, Javier Rando, Abulhair Saparov, Najoung Kim, and He He. Personas as way to model truthfulness in language models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6346 6359, 2024.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, ICLR (Poster), 2015.
Belinda Z Li, Alex Tamkin, Noah Goodman, and Jacob Andreas. Eliciting human preferences with language models. arXiv preprint urXiv:2310. 1589, 2023a.
Chunyang Li, Hao Peng, Xiaozhi Wang, Yunjia Qi, Lei Hou, Bin Xu, and Juanzi Li. MAVEN-FACT: large-scale event factuality detection dataset. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11140-1118 11158, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10.18653/v 112111633/22222....2 fww./222..iiinng-eemme
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 41451-41530. Curran Associates, Inc., 2023b.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 36, 2024b.
Kevin Liu, Stephen Casper, Dylan Hadfield-Mende and Jacob Andreas. Cognitive dissonance: Why do language model outputs disagree with internal representations of truthfulness? In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In First Conference Language Modeling, 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 17559-11773 2022.
Eshaan Nichani, Jason D Lee, and Alberto Bietti. Understanding factual recall in transformers via associative memories. In International Conference on Learning Representations, 2025.
Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. LLMs know more than they show: On the intrinsic representation of LLM hallucinations, In The Thirteenth International Conference on Learning Representattinss 2025.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In International Conference on Machine Learning, pages 39643-39666.
PMLR, 2024.
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. Linear adversarial concept erasure. In International Conference on Machine Learning, pages 18400-18421. PMLR, 2022.
Shashwat Singh, Shauli Ravfogel, Jonathan Herzig, Roee Aharoni, Ryan Cotterell, and Ponnurangam Kumaraguru. Representation surgery: Theory and practice of affine steering. In ICML, 2024.
12

---

Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. The curious case of hallucinatory (un) answerability: Finding truths in the hidden states of over-confident large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3607-3625, 2023.
Alessandro Stolfo, Ben Wu, Wes Gurnee, Yonatan Belinkov, Xingyi Song, Mrinmaya Sachan, and Neel Nanda. Confidence regulation neurons in language models. In Advances in Neural Information Processing Systems, 2024.
Francisco Vargas and Ryan Cotterell. Exploring the linear subspace hypothesis in gender bias mitigation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 2902-2913, 2020.
Lei Yu, Meng Cao, Jackie CK Cheung. and Yue Dong. Mechanistic understanding and mitigation of language model contfactaal hallucinations. Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7943-7956, Miami, Florida, USA, November 2024, Association Computational Linguistics. doi: 1 1f555////00...n.nnnsseememme
13

---

Appendix
A
Detailed MAVEN-FACT Analysis
Data extraction. We use the train split of MAVEN-FACE v1.0 (73,939 event-mentions drawn from 2913 news articles) _ Each mention carries a FactBank-style factuality code (CT++, CT+, CT-, CT- PS PR CF CF NA. ...). We retain only certain judgments:
certain-true = {CT++, CT+}, certain-false = {C.--.CT.).
All other codes are discarded, leaving N 71, 274 labelled mentions.
Grouping key. Mentions are grouped by their originating article ID (doc_id), giving M 2,313 documents with at least two certain mentions (r4 > 1). Let Ziy E {0, 13 indicate whether mention in document is certain-false.
Statisties reported the main text.
Corpus certain-false rate. p == N/ Eig Zi = 0.0209.
Σ3 (5)
Pairwise certain-false probability. Pr(Z, = Z = same doc) =
E, (24)
0.00090, where f: - E; Zij.
Independence baseline. p²= 0.00044.
Clustering ratio. Var (p:)
(b: - p) 1.23, with p. fi/ms.
Varbinom
M E. - p)/ni
x test. The 2 x M contingency table of (fr, n,- - -() yields x² = 4 174 (p = 19
These figures show that certain-false events, though rare (2.1%). occur about twice as often as chance would predict when events come from the same article, and the distribution of false rates across articles is 23 % more heterogeneous than a binomial model would comii--onfirmnng the co-occurrence signal predicted by TCH.
The MAVEN-ED dataset released with CC BY-SA 4.0 license. The MAVEN-ARG and MAVEN- ERE are published with GPLv3 license.
B Entropy incentive
Setup. We consider sequences (I, y, x', 3/) with subjects x, x' S and attributes y.y' A. Let : S + A be the ground-truth attribute map. A latent bit T ~ Bernoullip) governs whether attributes are truthful (1'=1) or random (I'-0):
g(x), y' =0x') (deterministic);
T v,y Lid Uni
We study the optimal next-token loss for predicting y' given the prefix (z, y') under two cases: (i) the model does not access T; (ii) the model does access T (and can memorize g).
A. Predictive distribution of y' without access to T
By the law of total probability over I' and the generator above,
Pr(y's = g(x'() x,) Pr(T=1 x. y, v,),
Pr(T=0] x, -
(12)
When say the model is "ignorant of 7." mean does not exploit any posterior signal about T from the prefix; thus we use the prior Pr(T=1 xr, x') = and Pr(T=0 (r.0 x, = p. Hence
Pr(v' = g(x') (,,,,)) = p+ 1 P
(13)
A
For any specific wrong a EAA (g(2')}
1
1-
Pr = x. [iy==a/2,v,1))
(14)
[
A
'Available httttps://github www.://--/CTTTSET KWW.HAEEN/AAAAAAAAA
14

---

B. Optimal per-token cross-entropy without T
Let a : p= 1-0
and .क.. The optimal model (that does access T) matches the true
A
conditional in (13 14), so its per-token cross-entropy equals the entropy of that distribution:
C-r H(a, s......) a log a (IAA) - 1) 3og B.
(15)
|
times
C. Optimal per-token cross-entropy with access to T
If the model does access T (and has memorized g):
T == loss 0 (since y' = =(x") deterministically);
T= 0: loss log
(uniform on A).
Averaging over 7 gives
LT = (1 1 p) log AA|
(16)
D. The gap and its limit as |A| + 00
Subtracting 16 from (15)
-agaa - (IA) 1oglogg B - (1- LA|.
(17)
Using 3
1-p [A|
-(IA log 3 - -11 (11- ( log(1 - pp) (1-))ppppppp (1-p)(1- (t)) log LA|.
Plugging into 17 and simplifying,
1-p
aloo a (1 - p) (1- [ log(1 p)
log AA).
(18)
IA
Since p+
A p and the last term O 100 log IA
), we obtain the limit
[All
A
p log p (1 p) log(1 p) H2(p) (up to the log base).
(19)
|A|
C
Experimental Setup
Model. We experiment with an attention-only transformer with a single attention head with post-attention LN:
X = E+P
11 E PEPERRVx (token + positional embeddings)
(20)
Q0i = (
K( x(1-1) vlo -
(21)
40 sofumax
6 k(OT VV())
// attention mix A Rdd
(22)
Vd
A = w!
W E Rdxd
// single-head attention output
(23)
0 = ((x +1
i= i= 2 = XW + bo, Wo E RdxV ER
// residual normalization (24)
(25)
Y softmax(Z)
(26)
Experiments with one-hot models (section 4). The theoretical analysis is driven by experiments on models equipped with frozen, one-hot embeddings and uniform attention, the latter obtained by setting the attention-ke key matrix K to the zero matrix. Under these conditions the columns of the
attention value-output product K KVT map directly to individual vocabulary items, exposing clear block structure in the matrix (fig. [ As detailed in the main text, the vocabulary is organized so that indices 1-20 encode input subject embeddings, 21-40 input attribute embeddings, 41-44 positional embeddings, 45-66 64 output subject embeddings, and 65-84 output attribute embeddings.
15

---

Methodology: interpreting one-hot embeddings. Figure contrasts two sequences a correct one (top row) and an incorrect one (bottom row)-by showing the final-layer activations before projecting to the logit space. The one-hot embeddings make the activation patterns in that layer interpretable. We display the activations for the raw representations (left), after layer normalization (middle), and after applying the unembedding matrix and the softmax transformation (right). Observe the differing y-axis scales: normalization substantially magnifies the component corresponding to the correct answer in the "true" sequence, while the effect is far less pronounced for the false sequence. The model that produced fig. was trained with SGD, learning rate 1.0 and batch size 16. The output learned, from zero initialization.
matrix was fixed to identity, and only the value matrix
Experiments with fully-trained models (section In section we train all components, including the input embeddings and the K attention matrix. The model is trained for 50,000 batches of size 128 and is optimized with the Adam optimizer [Kingma and Ba, 2015] with learning weight of le-4 and weight decay of 1e-5. We do not include biases in the attention modules, and use RMSNorm as layer normalization. We run all experiments on 4 NVIDIA GeForce GTX 1080 GPUs. Training a single model lasts up to half an hour.
D Additional Experiments
In the main text concentrated on a single-layer model (1 -1= 1) with true-attribute probability of p = 0.99. Here we extend the analysis to additional settings.
Our primary focus was the linear separability at the second-subject token, x', where the model predicts the second attribute. This the only position where the truth signal is behaviorally relevant. Nevertheless, the theory also predicts a linear truth encoding at the first-atribute token y, owing to the fixed attention pattern. When the attention KV matrix is learned, however, this need not occur- the model can rely exclusively on the attention paid to x' and leave y uninformative. The same theory further implies that a linear truth direction should eventually emerge for any true-sentence rate p. even though the gradient magnitude (and therefore the speed of emergence) does depend on p.
Varying the true sentence rate, p. In fig. 7b we vary P across five random seeds and measure linear separability at both token positions. As predicted, when the attention pattern is learee, separability much stronger at the second subject than the first attribute. The time emergence grows as p increases, yet linear encoding still appears even at the extreme setting of = 0.999. Developing a theory that precisely predicts this p-dependent timing is left to future work.
First attribute (y) Linear Separability
Second subject (x' Linear Separabiiity
1.0
1.0
0.8
0.8
p=1
p-0.999
p-0,99
0.6
p=0.99
0.6
p=0.99
p-0.95
p-0.95
0.4
p-0.75
0.4
p-0.75
p-0.65
p-0.65
p-0.5
p-0.5
0.2
0,2
0.0
5000
10000
15000
20000
0
5000
10000
15000
20000
Training step
Training step
(a)
(b)
Figure 7: Dependency of linear separability on p.
Dependency on dmodel and S). In fig. we plot the linear separability at the final checkpoint, for different hidden sizes and number of facts memorize (p 0.99, = 1 are fixed). With the exception of dmwodel = 32, the separability persists over the second subject x' for different combinations of these parameters.
Additional layers. As we discuss the main-text (section 5 a model with a single self-attention layer, it is the second attribute (x') token that attends to both and y. With more layers, there are additional strategies. For instance, may attend to both x and itself in the first layer, in the same way x' attends to both and y in the theoretical l-layer model; then, in the next layer, x' attends to y.
16

---

AUC dmodel (first attribute y)
AUC moooi (second subject
1.0
4096
0.50:0.02 0.52m0.02
0.60:0.01
0.60
4096
0.4910.01
0.7510.03
0.8910.01
0.8610.00
0.8
Mean final-tep AUC
-0.
Mean finalstep
0.51s0.01 0.50:0.01
0.61
0.50.0.01
0.5010.00
0.8610.01
0.88
0.8
5
1024
0.4900.22 0.53=0.01 0.0220.01 0.6020.02
0.4
1024
0.5010.01
0.8810.02 0.91:0.01
0.9020.01
D.
0.2
0.49:0.01
0.91s0.01
32
64
128
256
64
128
256
Oroddll
dmodal
(a)
(b)
Figure 8: Dependency of linear separability on dmnddl and |S].
BOS
BOS
BOS
Query
X
Query
Query
y
y
y
BOS
X
y
x'
BOS X
y
x'
BOS
X Key
Key
Key
Figure 9: attention patterns of a 3-layer model.
1y) saparability 1p 0.99,
Second subject (r) linear separabiltt to 0.99, seed
Layer2
Layer
Layer'
Loye
ABC
AUC
6.-
Training step
Thiining step
(a)
(b)
Figure 10: Linear separability across layers for 3-layer model; linear separability on the x' token is created after copying the signal from the y token in the second layer.
17

---

copies the signal and create a linear separation that persists the last layer. This is the mechanism that emerges 4/5 random initializations of a 3-layer model, and is clearly manifested in the attention patterns (fig.) 9 and in the linear classification accuracy across layers (fig. T0P.
D.1 Bridging the gap between the fully-trainable model and the toy model.
Our theoretical analysis (appendix E is motivated by the structured patterns that emerge in the attention kernel-the OV matrix when it is visualized (fig. I To test whether a comparable mechanism appears when employ dense embeddings and allow the KV matrices to train freely (thus removing the enforced uniform attention over æ, y). we train a model with a large hidden dimension but only small set of facts to memorize ([8) 32 and dmodel 512). We freeze the randomly-initialize dense embeddings and train all other parameters. The limited number of subjects makes the memorization patterns easier to inspect, while the high dimensionality approximates the regime mutually orthogonal embeddings required by the theory.
EW,WWE
EN.W.E'
106
335
(a) EVOE with frozen dense embeddings
(b) EVOET with trainable dense embeddings
Figure 11: Visualization of the attention matrix with dense embeddings.
Because the model now uses dense embeddings- individual coordinates no longer correspond directly vocabulary items- we do expect an obvious block structure in the raw OV matrix. Instead, following Dar et al. [2023]. we visualize E'OO',, where E' concatenates the input and output embedding matrices. This operation computes the pairwise similarities between embeddings as induced by the VO transformation. Concretely, (EVOE )ij = E. V. O, E, measures how strongly the value vector elicited by symbol i aligns with the output direction that scores symbol so every cell again describes relation between concrete symbols, exactly what the raw OV matrix showed when the embeddings were one-hot. The resulting heat-map (fig. TTa) exhibits strikingly similar pattern to that observed with frozen one-hot embeddings and fixed attention pattern, suggesting that the dense model converges to a similar underlying mechanism. contrast, when we do train the embeddings, the pattern partially disappears, parts of the memorization can occur in the embeddings themselves (fig. l1b). In general, there is much more variability between runs and hyperparameters when training the embeddings, where some hyperparameter choices do not show a pattern that is highly similar to the idealized one.
With full set of [S] dmodet = 512 tokens, the global pattern is hard to spot at first glance. If we instead sub-sample æ tokens, retain only their partners g(r), and then sort the rows/columns, latent memorization re-emerges: the lower-left block collapses into clear diagonal (the previously random pattern the leftmost lower block fig. TTalis transformed into diagonal due to the sorting).
This diagonal appears whether the embeddings frozen trainable (see figs. 12a and 12b
18

---

EW.W.ET (sub-sampled and sorted)
EW.W.ET (sub-sampled and sorted)
yi
Xout
Xout
Your
Vin
Kout
Yi
Xout
(a) EVOET with frozen dense embeddings (sub<(b EVOET with trainable dense embeddings (sub- sampled and sorted)
sampled and sorted)
Figure 12: Visualization of the attention matrix with dense embeddings.
Tokee Embedding toix) 25)
0.5
1.5
11
10
-0.4
30
-04
202
Token Embeesldng (gtx)) = 36)
50
0.50
-1.0
70
-1.5
-022
-0.2
80
-2.0
20
80
Figure 13: Visualization of learned embeddings and value matrix for a model as in Section with learned embeddings, initialized to one-hot.
One possible circuit with learned embeddings. We now present one possible circuit that found when initializing with the one-hot embeddings, in simplified architecture with uniform attention as Section 4 We still denote eas Ey, uy My the one-hot embeddings as Section 4 which only refer to the initialization in this setting with learned embeddings. After training, the learned embeddings and interpret them as linear combinations of the initial one-hot embeddings. shown in Figure
may visualize
13 Denoting εx, ēy Up üy the embeddings after training, the circuit we found
19

---

looks as follows:
& = 6x - (g(x)
Ey = Ey - Eg 1(0)
d₂ Σ Ux Σ By
y
by = " + ())
W Σ (m((x) - ex)e
Σ (ey uy)e
The approximation e C, Cg(a): for instance, follows from the two large positive and negative spikes in the left part of fig.
13 for indices I and 25/36. Similar to our analysis of Section 4
compute the quantity W (eg
Eg), which appears in the residual stream for both token y and token x':
( + Ex) "g(z) ex eg(=) + Ug(x)) Ey My My + (g)
We observe that this vanishes when y = g(r), suggesting that a similar mechanism in the fixed embeddings case studied in Section is at play, where layer-norm can lead to sharper predictions for true sequences, as well as provide truth direction.
20

---

30
0.5
40
0.0
-0.5
1.0
70
-15
80
-2.0
20
40
60
80
Figure 14: Structure of the value matrix W when training without positional embeddings.
W. WoWWLETT
Figure 15: Structure of the value matrix W when training with euclidean normalization.
E
Theoretical analysis
This section contains theoretical analysis and proof' for the results in Section
E.1
Training dynamics
We now provide some theoretical insights on the training dynamics in the simple one-layer model of Section 4 We further simplify the model here by removing positional embeddings. Figure [4 shows that the model still learns the relevant blocks even without positional embeddings, though some of the uniform distributions on unembeddings are now absorbed in other blocks.
The lemma below highlights the structure of the gradient for softmax classification model consisting of linear model followed by a layer-norm operation.
Lemma 1. Consider the model Fw (a: U N(a,, + Wb,) RR with N(v) v/lloll, and the following cross-entropy population loss on some distribution over y);
L(W)= Ez,yl- log S (Fw (x))))
(27)
where y the label and S the sofimax operation. The gradient with respect W is then given by:
2N
S(U N(ux))
1fy
k.}
VL(W)
Σ Ex,
(28)
|
21

---

with U2 = ag + b and where Pe = 00 is the projection onto the tangent space at 0 E S
Let us decompose the population loss as
L(W) = L1 + La(WW,
(29)
where (WV) is the next-token prediction loss for predicting zt-1 from 21.t, with 21:4 = (x, y, x', .'). We show the following result.
Theorem 3. Consider the following algorithm, with step-size 1 = N/p, and initialization Wo = 0.
1. Set W = Wo - (/W))(WW)
2. Ser W2 = W1 - V
3. Set W3 W2 L3
Then, we have
N
W33 = Σ (Biug(z) alea) Ex + -(areg-1()) f2u,)e, T + Oz0
(30)
a=1
where O1,, 82 > 0 can be found in the proof, and (1/N) is matrix where all entries are O(1/N).
Comment: Euclidean norm vs. RMS Norm. The updates in this section are derived under Euclidean layer-norm ((v) = w/ /1/22, so the normalized scores entering the softmax are attenuated (compared to our experiments which use inverse temperature 3 vd N(vN)) by factor of order 1/ VN. Consequently, in the early regime with fixed unembedding U and ((N) competing classes, the correct-token probability x2,(()) is O(1/N). In our implementation we use RMSNNrmm
NRMS u) Vd y which is equivalent to keeping Euclidean LN but applying softmax with |
inverse temperature 3 Vd. Since d ((N) (here d 4N + 3), this multiplies every Euclidean score difference by vd @(vN), relative advantages that were (1/vN) become O As result, in the same early regime the correct-token probability becomes (1). Empirically, we see similar structures emerge in the matrix during early training when using the euclidean norm instead of the RMS norm (fig. 15
Proof. Let us decompose each loss into contributions from true and false sequences, which follows from the fact that the data distribution is mixture of the two:
L(WW) pL? (W) + +1 P))-
Step 1. In the first step, we take gradient step only the loss L1 for the prediction of the second token at the first token x. starting from initialization W,, = O. Recall that this model takes the form F(x) = U N(c, Weg), that in the notation of Lemma I [w we have ag by Ug ez
and P., P. = uy. Note that since the logits are all zero, we have S(0); 2N
We begin with the gradient on true sequences. Multiplying eq. (88 by 17 and setting y = g(x) gives
2N
-T)L (BLLug() -
2N N
7
u
N
2N2
e T (nx22)
N
22

---

On false sequences, we have
-yylEn
((0)ausez] yBuyylyee
N
2N
2N
N
Σ
Σ
7
uyer
UzGz
N
2N
N=
==
O0 (n//22,
using the fact that æ and y are independent. With rj N/p, obtain
N W1 = Wo (VL = Σ
" + Ozo(1/N).
2m1
Step The second step taken at W = ---1 Cg 2 R. with || = 0 Thus, we have U c, + Wiez ex + ug(s) + Ex, with || = O(1/N) since ez one-hot, which implies ||s N). We also denote Jz == S(U-N(ee))) N(ex)),, which satisfies 0 = ((/NN) for all x and k; since exp( uu N(ex)) = (11) for all and k. On true sequences, we have:
N
2N
V (W1)
T
7
Jz.k
(1
Uv ll2
||//1|/
N
12ll2
|
Note that we have
| /2 + (ex )g(x)) Tex + | O(1(() v2 OLLNN)
by Taylor expansion. Then,
(1
U
1
UN + )k + 0(2)
Ca.b 1ull2
Ux
V2
0
202+0(1))
1
8k.)(Z)
V2 Uk +
(ex + +((z))) + 50 (1/N),
where O0 (1/N) vector with loo norm OL/NN) and òk.g(m)
= 1fk = g(x)} denotes the
Kronecker delta. Plugging back into the gradient above, we obtain
"
) (ez + u)(a)) + 000
1
c
Un(x)
2
N
N
2N
7
2
2
1=1
ex + (xo
(g(x)
2/2N
which follows by noticing that Σ k=1
Osc (1/N). For false sequences, we have
N
2N
(VLF(()) = n nB
1
Σ
N
m=
k=1
2N
2N
E
N2
N
==1
y=N+1
2=1
k=1
Ozc O00 (n/N2).
This again follows by noticing that
2N
2N Σ
1
Vn.y
Oz (1/N)
and
2 O (1/N).
N
+1
1=1
23

---

With N/p, this yields
N Σ t=1
W2 = W1 (WW =
(aug(() - -e) + OL//N),
with a = 1 2/2'
Step 3. The third step takes one gradient step the loss Ly the third token, i.e. predicting y' from (x, y, x'). The model now takes the form F(x, y, x') = U N(e + We + ey + ex))), with uniform attention the first three tokens.
We get the gradient of the loss on y' from giving
Ue = Ex + (ex + ev + ew)
1
cx' 3 + _ + 3 2)(x'' + Exy,z' =: =2,2z'''' + Enya.'
with I| 1l O() since is the of columns of the (x/N) term in W2. As in the second step, have "z 11 O(1 and C. (1//N,, so that by Taylor expansion
we have || || = || || + O(1/N) = 5 2n + O(1/N) for x + x' and || = || || + O(1/N) 1/1 + 2a O(/NN) for x'. Note that once again have Ozzy,a'k SUU N(Uz.g.x) 1) = O(1/N) due to the normalization. On true sequences, we have
V (W2) EE xx
Ug(x))(e + te(()) + =
3/ U2
2N
(I- Unryg ".....e
un (ex + Eg(z) + egr)T
77
3 3e weee |I
I 1|2
Let us first show that the error terms Ex,yy, lead to negligible contributions to the gradient. Note that we have
(1- Ua Varei =3(1ve,''
Un
31 3|
|| /MMroarr
uk
(1-
(We,o + (0) (2) + 000 (2)))
|l + +(2))
1
1 ((
uk
1
[- (1-
U2 Uz2z"
331u,,,r|1
uk + Ex,
With | veloo = O(1NN,, where we used uu + O00 for e (xx(1N))
ate + O for 0, and (u + e) +c)
Then, taking expectations with respect to independent 2, x', it is easy check that
E (t + Co(x) ex) ) = 0xxN3( E (z (z + Eg(z) ex) 0 =
The gradient update can then be rewritten as
U || 1|2
-1/V 13 (Wa) E/E
Mgga' (ex + teg(x) + +
(31)
3 3||uararr ||
2N
(1
Ve 2,20
17
F Ex
3 Va
1|2
uulle + Cg(z) T
(32)
+ Oz (n/222
(33)
'ee use the fact W26: atg((() es + (oo(N/N) fort S N and Oxc (1/N) otherwise.
24

---

We now check that the second term is also of order (x Indeed, the pro- jector matrix 3|
"r
sparse with rows of bounded & norm, and we V
have Σn 2.9(()), uu - = O O0 (1/N), so that
2N
Va "r,""
(1-
us Se, (20(1/N),
3 3/
Ue 112
for all x, x². We then have
2N
02.0(0),2''6
Un W
E Exr,r'
ux uk(es + Eg(x) + t
3 2/
Enga Ba (ex + Eg(z) + c
Sa exp
S g(x)
Σ Sa.eer (10(2))))
For the first term
31
have
Un U se,es
B
I
"glaslee + eg(a) + Tex))
Ue 112
= "/,a,
1
Ooc 1TT
[a(1 + )g(u)
Ve + eg(x) + e
3/10x,0r
9 Uz,x 1
n
N
Σ
Br
(i Ma (ez + Bg(n) + e. + 000 (1/
with
a(1 + (g(2))(()))
31 = Exl 31,12,111 and Naw
9 91
In (+) used (i) that Ez,zr (3 Molorlex + (o(a)) Ooc (1/N2) thanks the independence of and s': and (ii) the fact that uh (ar Vor orthogonality.
Va. = g ))()))(())) by definition of 22,2' and thanks to
We have
BBn. (on (ex + eg(z) + e]
Bn (Bxr (To, (x] + (g(=)) | - 1/Ex' EE 17,, 17 U2 [e
N
1132
(cz
and (Ug())) + TT
(2ez au aug(x))e O(n/N²)
N
N
2N
2*
C +
1132
Σ
(c, 1()) - auy)ey 0zc (n/N2)
N
N
N=
with
1 E. [,, = Ex Err = 3N 7.11 +
N
B2 = 3
712-
3NV
In (x) we condition on x and decompose Ur.r" A(x) + B(x'), where A(x) is independent of
2t: then linearity gives Ey (T,,, x] = A(x) Ey, [7n.a') + Eg (7a..' B(x)). By permutation symmetry of labels, E,, (Tr.ar) = E.[m, = 3,2, while the B(x') part averages over uniformly random index and is Oz0 (1/N); the same holds with x and x' swapped for the second bracket. Substituting these two conditionals into the split expectation, replacing outer expectations by uniform
25

---

sums, and renaming the dummy index yields the displayed (x) line. In (**) we perform change of variables =g))). We have thus shown
n
Ln/L3 (Bi0g(a) 32e +
Σ
(g))())-oupp) + +nx(n//2)). (34)
y=N+1
For false sequences, can be checked that 7/ 7VL5 Ozo(n/N2). Thus, taking step-size 1/ N/p yields
W3 - -37VV3333
N
2N
= + (1) Σ
Ug(x)C2 (1+ B2)
Σ
c + B2
Σ
(eg-1 auy)e + O00
==1
==1
g=N+1
E.1.1 Learning (positional) attention.
We now turn to learning the key-query matrix under positional attention, assuming that the value matrix has already been learned with the structure described above. Specifically, we show that the gradient of the key-query matrix on true sequences drives positional attention to focus on the x' token, effectively causing the model ignore the initial (a, y) pair. This observation may account for the absence of emergence at p 1, although it does not explain why emergence still occurs when attention trainable and p < 1.
For this part, assume a simple architecture of the following form for the prediction of the fourth token logits given the first three tokens:
exp(p) W WKQPs) Wye
FWiq (21:3) = U
Ev=reppir WKQPs)
where 21:3 = (Ex,, Eyg ez) and P1:3 are the positional embeddings defined
We assume Wv fixed to the the structure in (0- with B1 32 =: 3 for simplicity. We consider the following population loss for WKQ on the last token for true sequences:
L(WxQ) = Ez [[(g(x'), Fwwo (r. g(x), x )-))
Then, the negative gradient direction at WkQ 0 is given by
2N E
VLL(WKO)
(1((((x)) = k} p(k|2, x )' Wye (p. P.a)ps]. (35)
where we denote 21.3 (x, g(x), x') and P1.3 (pi + p2 + p3), and á are probability predictions at W ko which we assume satisfy the following, given the assumed structure on Wv, for some small
(1 - -/2, 1 1_c,
if E (o(x) and x # x', if = g(x) and = _', o/w.
x(k)z,x)) =
O(L/N),
Let us now write the update in 335 as
1
VL(WKQ)
we(p: P1:3)p
3
This requires that the early phase is run for long enough that is large enough, say O(log NV).
26

---

and study the values of the Wi. Fort = 1, we have
E ar (I(g((x)) = (k(),, Wvez]
W1
k=1
2N
3
g (1(g(x)) k) ((()) k:]l
k=1
2N
B
Σ
Es [g(1) = k3 (1(((=)) = k}] BE, - ((((())az,21)
k=
N+1
8
B(N2 N) 1
I -B 1-6
O(B/N).
N
N² 2
For = 2, we have
2N
E Ba (1(g(x') = k) - [kk|, x')) Wre
W22
k=1
2N
B
Σ km
E. x' (11(((x)) - 1} (( 1)1 k}]
= 0 + B (xxx1,,,))
OB/NN).
For = 3, have
E R (1(g(x') -k) x' y)u Wyewr
W3
2N
E (11(g(x)) = k} =(H)222))1100( = =]]]
1=
Em,or [(g(T')|z,
B
2
2
Thus, when N >> 8, we have W3 - W1, W3 W2 2 O Taking Wkq - the gap in attention logits between / = 3 and = 1,2 is of order 1/8/,, so that for v) large enough, the attention mostly focuses on the third token z'.
E.2 Proof of Theorem 1
Suppose we are given (æ, y, x'), where we assume for simplicity that æ +f x' and g(x²) + y. Denote by fw (21:) the output of the model (4) before applying the LN and the unembedding layer. Then, we have that:
[w(x, ex + ps +
37
Σ 21
Mly
Σ
Un
+
+ 3
( (aniez + +111lgix) + (126g - B2uu - 0n + 3Iug(z))
(36)
12N 2)+2 and C2 = 2+ +2 for a true sample where
Denote by C1 = 2 +
9
y g(x) we have that:
(l)) ct (31 32 +2² (81 + +
27

---

Hence, after applying the LN and unembedding layer we have that:
B1 + 7
3/ + -B1 - 82 7)2 (31 7))²
3+ max 31 - 32)
max (Fw (Z.)), 3/41 + (31 82 5)2 + (31 7)²
v'#g(=")
For false sample where # g(x) we have that:
Ilfw(z x(z),xx)l + + +5)² + (-32 + 7)2
Hence, after applying the LN and unembedding layer we have that:
81 +
+ 7)2 + (-32 7)2 (Z, Datar) 3/ 02 + 2(1
31 +
max v'*((2))
(3,
2(3 7)2 + (-32 + +)2 3
Plugging in these terms finishes the proof.
E.3
Proof of Theorem 2
before
after
True False
1.5
False
0.0
00.5
-2
-1.0
-15
-2.0
-5
Figure 16: Illustration for LN- linear separability.
Proof. We first describe the output of the model in (4) before applying LN. Denote by vr, up 44N+3 these outputs for true and false samples respectively. Recall that true sample (x, y) is when y = g(x) and false otherwise. Then, we have that:
Σ ul Σ Ur ))
vr cy + P2
1 (a)-ar)e+ (81 -B))uy + (71 - (2))
(37)
UF ey + P2 +
-0 + 022g-11M) + (1ug(a) - Br + (71 - -2))
Σ Uty
Σ
Ui
x
(38)
We will first show that without adding the samples above cannot be separated for general x and Assume otherwise, that there exists linear separator w
W1
W22
W3
with Wi,..... W4 E N Wg R
W0
105.
and bias term b I such that (w, vT) - b and (w, uF)
< 0 for every true or false sample
28

---

respectively. We slightly abuse notation and write (wi, ez) as (Doww)s) ex
wi and similarly when
multiplying w2 by ey. W3 by Ur. W4 by by and Wi by
2
((17)) (m
(m -2)) 72)
Σ By
Σ
Ux
Wy wi
(ws, p2)
the terms in the inner products that are independent of the sample. Then, using the linear separator on these four samples we have:
b S (a2 an) (ex,, + (e,, + (B1 - (2) (Wbg wa) +
(39)
b < (a2 a1) (ez, w1) (ex):w1)...)m
(ey, w2) + (31 B2) ( (uy Ws) +
(40)
b 2 02 (ex,: mi) on (eaz: w1) (ey; W2) B1 (u1, wa) :B (" wa) c.
(41)
2 02 Czy, w1) αn (ex,, wi) + (e,, W2) + 91 (uy, w1) -3 (uy, ws) tc.
(42)
Adding up (4 and 42) we have that:
2b 2c (02 01) (Cx,, (eys W2) (31 - (32) ( (uyy, ws))
(43)
+ (a2 a1) (cz,, w1) + (e,, W2) + (31 B2) (ug, wa)
(44)
which is contradiction (39 and 40)). This means that there is no linear separator, regardless of the values the parameters, which proves the first item.
Assume there is layer normalization after the prediction as in (4) This means that the output of the v Consider the linear predictor = P2. and a bias term b that will be determined later. model [To
Then, the output of the linear predictor is exactly (w. v) |+]l
We will now calculate the norm of both true and false samples. For a true sample (xr, gx))())) we have that:
| 1| = + (a2 - 01)² + (n1 72)² (2N - 1) + (n -2 + 31 -B2)²
(45)
For a negative sample (x., y) with gzx) + we have:
||F|| 11 2 of + a (>1 7² (2NN 2) + (71 - 7/ B,) (71 - 72 --23² (46)
There exists a linear separator as long as To
|rr + 0. Since the vectors vr and of are both
non-zero, this is equivalent to ||,,||| + ||,l By the above calculation, we have that:
|| 12 |
af + a - -(a1- (a11-1a0 -02)² - (n -2) (n - + 31)) + (n - 72 B2)² - (n1 - 72 + 31 - -B2)
=2aj02 + +23182.
This shows that if 20 + 23,33 + 0 then we have linear separation between true and false samples.
Further assuming that 0n 022 3 32, 7n 72 we have that ||ur|2 2 and || 2 + 20² + 23, To find the optimal margin for this predictor we pick:
(1 (1- (1-
1 b b= 2
1
1
|
|
2v2
32
We will now prove that there is linear separation after predicting the x' token. Using the output of the model in (4) we get:
1
vT C+
((a2 -a22-a)ea (B1 - B2)u, a1ez + (1ugtar))
(47)
3
1
OF = C -
(-ares + a24 (()) + $1 Bruy + -ayes + Blug(on)
(48)
3
29

---

where C== Ex' + P3 + (Σ, uu -E us). We can now calculate:
1
||T 1| 2+ 5 ((02- a1)² + +31 32 + (2N 2)9 (49)
1 (2a +0 +2N -2N 3372) (50)
||0F|2 2+ +
We now have that:
||rl| - ||rl11 2
(a} + a (81 + +7)² - (3- --))= - a1)2 (31 - B2 + -2 77)
9
a1 + 3132).
By a similar argument to the previous case, if 01 + 31,32 # 0 then there is linear separation
between true and false samples. Further assuming that On 02 B2 and 7 0, to find the optimal margin for the predictor we pick:
1
a 3
2
9
(a² + 18322 (a + 82)
30

---

E.4 Evaluating checkpoints of "real" LM
To test whether the two-phase dynamics also appear in large model trained on open-wen web data, we analyzed the Pythia-6.9B training checkpoints released by EleutherAI. Using the COUNTERFACT dataset we construct each input by concatenating K factual statements whose preceding context is either entirely true or entirely false, mirroring our previous setup. For every checkpoint we measure three signals on the final statement:
Memorization: percentage of cases where greedy decoding succeeds in completing the correct token.
entropy the model's full-vocabulary distribution for predicting the last token; we record the difference between true and false context.
Uncertainty:
I Linear separability:
accuracy of a linear probe trained to classify the truth value of the surrounding
context.
Table I: Pythia-6.25 metrics across training steps.
step
AH
memorization
probe AUC
0.001
0.000
0.383
512
0.006
0.000
0.435
1000
0.005
0.006
0.467
3000
0.219
0,242
0.587
5000
0.217
0,435
0,648
10000
0.286
0.547
0.667
20000
0.355
0.655
0.754
40000
0.329
0.727
0.759
60000
0.421
0.772
0.802
80000
0.419
0.822
0.799
100000
0.479
0.835
0.818
110000
0.485
0.849
0.835
120000
0.536
0,842
0.835
130000
0.565
0.858
0.783
143000
0.518
0.875
0.831
Notes. AH denotes the entropy gap between matched prompt pairs presented with false versus true context. The memorization rate is the share of instances in which the model's output distribution places the correct continuation token the top-1 position. "Probe AUC" is the ROC-AUC of linear classifier trained predict the surrounding context's truth value from model activations.
Findings. Early training (S 1k steps). AH Re 0; the model memorizes indiscriminately.
Mid training (3k-80k). Memorization jumps, then plateaus, while AH and probe accuracy climb steadily.
Late training (2 80K). Entropy separation continues to widen even after memorization saturates, mirroring Phase but over a longer horizon.
Overall, this echoes the two-phase pattern observed in simpler experiments: an initial jump in memorization followed by a slower, steadier increase entropy separation. Differences remain: the second phase is more gradual, and classification and entropy increase even before memorization stabilizes. We hypothesize this stems from continual exposure new facts during training, unlike our idealized setup where all facts are seen in a single gradient step. The modest terminal memorization and classification scores are consistent with reports that the Pythia series under-trained relative to its capacity.
Finally, in Pythia-6.9P we do not find evidence that layer normalization itself induces linear sepa- rability; rather, linearly decodable truth signal emerges gradually with depth across many layers. Our aim was to advance one plausible mechanism for the phenomenon observed in pretrained LMs. not to claim uniqueness. Given the model's 's deeper architecture- witth with numerous layers and MLP blocks and the richness of natural- language data, additional or distinct mechanisms are likely at play. A systematic study of these mechanisms is an important direction for future work.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We support the claims in the theoretical and experimental sections of the paper (section 5 section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discuss the limitations in section and throughout the presentations of the theoretical results.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
32

---

Answer: [Yes]
Justification: We specify all assumptions in their modeling and theory sections. We provide detailed proofs for all claims in appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide a full detail of the experimental setup in the relevant sections as well in dedicated appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
33

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We release the code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttp:ccppppcccceeeeeeeeee..c.ett httpa//gggeeee/eeeeebbiisssssss /Cc////oddiiisssseeeeeee ssiiiiiisiiiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We specify all parameters the experimental section and in a dedicated appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report results and error estimation over 5 random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
34

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this information in the experimental setup appendix (appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
To the best of our understanding, this study does not raise any ethical concerns.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This is fundamental research that does not have broader socictal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
35

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any models that can be misused.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We mention this information in appendix A
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
36

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
37

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: No core-use of LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
38


Reviews:
[{"review_id": "olAc6NeEuz", "reviewer": "Reviewer_ZrWv", "summary": "The authors propose a simple theory that aims to explain, in part, how language models learn “truth” directions. They introduce the Truth-Co-occurrence Hypothesis (TCH)—namely, that true statements tend to appear near other true statements, and likewise for falsehoods. To test this idea, they build a TCH-driven synthetic dataset in which truthful or untruthful statements always occur together, then train a single-layer transformer (with several components frozen) on that data. Internally, the model learns to classify truthful versus untruthful statements linearly by their norms. The authors also explore whether a similar mechanism emerges in larger language models under a different setup. The paper offers an interesting perspective and illustrative toy experiments showing how a model might acquire a linear truth direction; however, I still feel the findings remain largely detached from real-world LMs (see my comments below).", "questions": "See my comments above.", "limitations": "See my comments above.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* TCH is a simple yet interesting hypothesis.\n* The toy setting is engaging, and the observation that the learned norms encode a linear direction that separates truthful from untruthful statements is likewise noteworthy.\n* The paper is well written and easy to follow.\n\n**Weaknesses**\n\n* **The *main* toy setting imposes interesting yet unrealistic constraints that may render the norm-separation discovery trivial.** The toy model essentially freezes everything except *a single weight matrix*. Given that truthful data are more frequent and the loss is unbounded (i.e., pushed toward more negative values), it is unsurprising that some truthful subspace in the value matrix (i.e., certain column spans) receives more frequent gradient updates, driving its norms out of distribution. Although one could argue the learned value matrix is full-rank, the small vocabulary means the meaningful updates are effectively low-rank. In line 272, the authors even note that this finding “disappears when learning the KV matrices instead of using fixed attention.” I suspect this is because the pressure on norms vanishes once more parameters are trainable. Overall, uniform attention strikes me as an unrealistic assumption.\n\n  *P.S.* Proving a hypothesis with toy settings is fine if all claims are stated with explicit assumptions, but I remain unconvinced that this is true here.\n\n* **The follow-up study on the 8B LM does not test the training dynamics observed in the toy example at scale.** The toy experiment purports to show that TCH explains the emergence of a linear truth direction; the 8B study instead demonstrates that an LM is biased by untruthful context and that a steering vector can restore truthfulness—both fairly well-known results. I would urge the authors to scale up their training-dynamics analysis (e.g., loosen the constraints or train larger transformers on more realistic, partially corrupted factual corpora).", "was_revised": true, "final_justification": "I am raising my scores since the newer experiment results make me more comfortable of the toy experiment result included in the draft. I am increasing my score only slightly since, without revision, I still feel like the paper makes a broader claim which is not that well-supported by experiments included.", "rebuttal": "**We sincerely thank the reviewer for their thoughtful and constructive feedback, and we appreciate the opportunity to clarify our work.**  \nWe address each point in turn below.\n\n---\n\n### Regarding the sentence at line 272  \nThis sentence was unfortunately a remnant from a preliminary version and doesn't accurately describe our findings. We apologize for the confusion. **Frozen attention is *not* necessary for reconstructing the pattern in Fig. 1.** Indeed, in **Appendix C** (“Bridging the gap between the fully‑trainable model and the toy model”, line 852) we report the trainable‑attention results: the heat‑map in **Fig. 10a** retains the same pattern we see in Fig 1 in the main text, while **Fig. 10b** shows that if embeddings are also learned, part of the memorization moves there and the pattern weakens. *The two phases dynamics and emergence also remain when updating all parameters*; this is how figure 3 was generated. We will clarify this in the main text and fix line 272. Please also see “Training dynamics” below.\n\n---\n\n### Uniform‑attention head and other assumptions  \nWe recognize that idealizations such as uniform, fixed attention (together with a single‑layer architecture) are not faithful to production LMs. They are, however, *standard tools* in mechanistic‑transformer theory because they strip away distractions and let us isolate one concrete route by which a linear “truth” direction can form. Similar works that study different properties of self‑attention also make simplifying assumptions, such as fixing the attention weights and training only a single weight matrix—see, for example, [1] on implicit bias and [2] on benign overfitting in self‑attention.  \n\nOur intention is therefore to provide a rigorous characterization in a minimal setting: **if the phenomenon shows up even under these minimalist conditions, it becomes a plausible building block for the far richer dynamics of large‑scale models.** We emphasize that even our simplified objective (Eq. (2) after line 141) already contains a weight matrix $W$ trained *inside* a non‑linearity. This results in a non‑convex function that resembles a 2‑layer network, and its optimization process is known to be challenging to analyze, as accomplished in **Theorem 3**. Our analysis goes a step further and studies the *properties* of the learned weights (**Theorems 1 and 2**), which show the occurrence of the truth‑linear subspace.\n\nWe also remark that attention is typically uniform at initialization in high dimension, and that the value matrix usually receives more gradient signal than the key‑query matrix initially [4, 5], making our assumption relevant for the *initial phase* of training. After the value matrix has learned the structure in Eqs. (4–5)—which already gives good performance—we found empirically that when $\\\\rho = 1$ (only true sequences), the key‑query matrix tends to learn a positional pattern that avoids attending to the *x* and *y* tokens from *x′*. We now have an analysis of how this happens via a gradient step on this matrix: the model with uniform attention over‑ (resp. under-)estimates the probability of $y′ = g(x)$ (resp. $y′ = y$) on true sequences, hence the gradient places smaller attention on the *x*, *y* tokens, using positional embeddings. We will happily include this proof in the appendix.\n\n---\n\n### Small vocabulary and “effectively low rank”  \nPlease note that in the appendix (**Figure 7**) we re‑ran the toy experiment with vocabulary sizes 128, 512, and 4096 and different hidden‑state dimensions. In *all* cases, Linear‑probe AUC stays above 0.86. Furthermore, the positive‑classification margin guaranteed in **Theorem 2** does *not* depend on vocabulary size; we will point this out directly after the theorem.  \n\n---\n\n### Training dynamics on more realistic data  \nFollowing your suggestion, we re‑ran the study in a more realistic setting—the **CounterFact** dataset [3], which contains simple factual statements in natural language, with variations in sentence structure.\n\nWe trained standard Transformer models with **2, 5, and 9 layers** (4 attention heads and one MLP per layer) on a single relation, updating *all* parameters.  \nEach training sample is a concatenation of two sequences, just as in the paper’s main experiments. To enforce the truth‑co‑occurrence hypothesis, the factuality of the two sequences is matched; in 5 % of the samples we corrupt the attribute by randomly swapping it with another attribute.\n\nThe results reproduce the same **two‑phase learning pattern** seen in our synthetic setup:\n\n1. **Phase 1 — rapid memorization**  \n   Early in training the model places almost all probability mass on the memorized attribute for both true and false contexts, so the entropies are equally low (H\\_true ≈ H\\_false).\n\n2. **Phase 2 — abrupt emergence of a linear encoding**  \n   A few hundred steps later a strong linear signal appears in the penultimate token of the second sequence (AUC\\_S2 ≈ 0.97).  \n   At the same time H\\_false shoots up while H\\_true drops, exactly as predicted by theory.\n\nBecause NeurIPS technical constraints, we summarize the average numbers across training phases:\n\n| Phase | Window&nbsp;(steps) | $\\\\mathrm{H_{true}}$ | $\\\\mathrm{H_{false}}$ | $\\\\mathrm{AUC_{S_1}}$ | $\\\\mathrm{AUC_{S_2}}$ |\n|:---|:---|:---:|:---:|:---:|:---:|\n| **2‑layer** — memorization | 0 – 1.2 k | 0.39 | 0.39 | 0.85 | 0.51 |\n| **2‑layer** — emergence | 1.4 k – 2 k → | ↓ 0.14 | ↑ 2.90 | 0.82 | **0.97** |\n| **5‑layer** — memorization | 0 – 750 | 0.41 | 0.40 | 0.70 | 0.52 |\n| **5‑layer** — emergence | 0.8 k – 1 k → | ↓ 0.12 | ↑ 2.31 | 0.98 | **0.97** |\n| **9‑layer** — memorization | 0 – 500 | 0.41 | 0.41 | 0.62 | 0.51 |\n| **9‑layer** — emergence | 0.6 k – 1 k → | ↓ 0.11 | ↑ 2.64 | 0.97 | **0.96** |\n\nHere, AUC\\_S1 / AUC\\_S2 refer to the Area Under the ROC Curve of a linear probe trained on  \n&nbsp;&nbsp;&nbsp;&nbsp;• **S1**: the last token of the first sentence, or  \n&nbsp;&nbsp;&nbsp;&nbsp;• **S2**: the penultimate token of the second sentence.  \n\nIn the 2‑layer model the memorization plateau lasts until ≈ 1.2 k steps; then the linear encoding emerges and H\\_false rises, reducing loss on corrupted samples.  \nThe deeper models transition earlier—after only 500–750 steps—but follow the same trajectory.  \nAll metrics are taken from the final layer. In the deeper networks the penultimate layer reaches comparable accuracy, while earlier layers stay near chance. Overall, these results confirm the two‑phase dynamics we predict in a more realistic setting.  \n\n---\n\n### Evaluating checkpoints of a “real” LM  \nTo test whether the two‑phase dynamics also appear in a large model trained on open‑web data, we analyzed the **Pythia‑6.9 B** training checkpoints released by EleutherAI. Using the CounterFact dataset we construct each input by concatenating $K = 4$ factual statements whose preceding context is either entirely true or entirely false, mirroring our previous setup.  \nFor every checkpoint we measure three signals on the final statement:\n\n* **Memorization**: the percentage of cases where greedy decoding suceeds in completing the correct token.  \n* **Uncertainty**: the entropy of the model’s full‑vocabulary distribution for predicting the last token (we record the difference in entropy between true and false context).  \n* **Linear separability**: the accuracy of a linear probe trained to classify the truth value of the surrounding context.\n\n| step | $\\\\Delta H$ | memorization | probe&nbsp;AUC |\n|------:|-------:|---------------:|--------------:|\n| 0 | 0.001 | 0.000 | 0.383 |\n| 512 | 0.006 | 0.000 | 0.435 |\n| 1 000 | 0.005 | 0.006 | 0.467 |\n| 3 000 | 0.219 | 0.242 | 0.587 |\n| 5 000 | 0.217 | 0.435 | 0.648 |\n| 10 000 | 0.286 | 0.547 | 0.667 |\n| 20 000 | 0.355 | 0.655 | 0.754 |\n| 40 000 | 0.329 | 0.727 | 0.759 |\n| 60 000 | 0.421 | 0.772 | 0.802 |\n| 80 000 | 0.419 | 0.822 | 0.799 |\n| 100 000 | 0.479 | 0.835 | 0.818 |\n| 110 000 | 0.485 | 0.849 | 0.835 |\n| 120 000 | 0.536 | 0.842 | 0.835 |\n| 130 000 | 0.565 | 0.858 | 0.783 |\n| 143 000 | 0.518 | 0.875 | 0.831 |\n\n$\\\\Delta H$ denotes the entropy gap between matched prompt pairs presented with false versus true context. The memorization rate is the share of instances in which the model’s output distribution places the correct continuation token at the top‑1 position.\n\n**Findings.**  \n* **Early training (≤ 1 k steps).** $\\\\Delta H \\\\approx 0$; the model memorizes indiscriminately.  \n* **Mid training (3 k–80 k).** Memorization jumps, then plateaus, while $\\\\Delta H$ and probe accuracy climb steadily.  \n* **Late training (≥ 80 k).** Entropy separation continues to widen even after memorization saturates, mirroring Phase 2 but over a longer horizon.\n\nThis echoes the two‑phase pattern we observed in simpler experiments: an initial jump in memorization followed by a slower, steadier increase in entropy separation. There are, however, several differences. The second phase is more gradual, and classification and entropy increase even *before* memorization stabilizes. These differences likely stem from the model’s continual exposure to new facts during training, unlike our idealized setup where all facts are seen in a single gradient step. The modest final memorization and classification scores align with prior reports that the Pythia series is under‑trained relative to its capacity.\n\n**In sum**, both experiments corroborate the predicted two‑phase dynamics we observe.\n\n### References  \n[1] Vasudeva, Bhavya, Puneesh Deora, and Christos Thrampoulidis. *Implicit bias and fast convergence rates for self‑attention*, 2024.  \n[2] Magen, Roey, *et al.* *Benign overfitting in single‑head attention*, 2025.  \n[3] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. *Locating and Editing Factual Associations in GPT*, 2022.  \n[4] Li, Li, and Risteski. *How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding*, ICML 2023.  \n[5] Bietti, et al. *Birth of a Transformer: A Memory Viewpoint*, NeurIPS 2023."}, {"review_id": "rXlt8BxZeg", "reviewer": "Reviewer_CoF5", "summary": "The paper explores the emergence of linear separable truth encoding in Transformer models. They explore a toy setting with two token pairs where if the first token pair follows a 'ground truth' distribution, the second token pair is likely to also follow that distribution (replicating the Truth Co-occurrence Hypothesis: in naturally occurring text,\ntrue statements are statistically more likely to co-occur with other true statements, and falsehoods with other falsehoods.). They show that the transformer first learns the ground truth mapping and only later learns to apply it to the second token pair only in the case of a first ground truth pair.", "questions": "**Questions:**\n\n- In Figure 2: would it make sense to plot averaged values over several inputs?\n- Are sequences always 4 tokens long? What happens if you make them longer?\n- Could you add a few examples of your dataset? How do you define a^* is it just 2*s? Does it matter how you define it or can it be an arbitrary mapping?\n- In (1) shouldn't the prefix be x=(s1 a1 s2)?\n- Why do you switch between s,a and x,y? why not just stick with s and a?\n\n\n**Suggestions:**\n\n- explain shortly what e_x, u_{g(x)} etc is in figure caption so the figure+caption is self contained. Maybe also mark which block is which, since it's hard to infer just by looking at the figure.\n- add axes labels in Figure 2\n- in the setup in line 137 you could add the symbols e, p, u when after embedding, positional embedding, and unembedding respectively to have them clearly defined\n- add definition for F in eq (2)\n- add definition of g when symbol is first used\n- be more consistent with what you call things: for example you define a sequence with subjects and attributes, but later in line 155 you call then input and label\n- maybe for Figure 4 c) add that these are learned embeddings since in the toy example they are pre defined", "limitations": "The authors adequately discuss limitations and list interesting future directions", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\nI think studying the Co-occurrence Hypothesis for truthful statements in a toy example where one can directly control conditions is a good approach.\nThe authors motivate their approach well and set a clear goal/focus. The paper is set up in a well structured way.\nThe contribution is to the best of my knowledge novel and significant.\nThey adequalely cover and acknowledge previous work.\n\n\n**Weaknesses:**\n\nThe authors could be a bit more diligent with defining symbols to make it easier for readers to follow. Some Figure captions could be improved for clarity.\n\nminor:\n\n- You should not use LM in the abstract without defining it (which you could easily do since you already mention large language models)\n- line 68 typo by -> be\n- line 161: verb missing in '(though we still some lower confidence spikes...'", "was_revised": false, "final_justification": "", "rebuttal": "**We thank the reviewer for their positive assessment and for highlighting both the paper’s structure and novelty.**  \nBelow we answer the specific questions and note the text changes we will make in the camera‑ready version.\n\n---\n\n### General edits  \nWe appreciate the comments on typos and the need to improve the captions and results descriptions. We will incorporate these in the final version.\n\n---\n\n### Figure&nbsp;2  \nYes—the patterns are reproducible. The particular values that are *arg‑maxed* naturally differ between examples where the gold token changes, *but* the distribution becomes flattened when conditioned on false contexts, as shown by the aggregated entropy numbers we report. We will convey this more clearly in the final version.\n\n---\n\n### Example length  \nIn the experiments included in this paper, the sequence length is indeed fixed; this makes the theoretical analysis easier. However, in preliminary runs we inserted extra padding tokens at random, and this did **not** change the empirical picture.\n\n---\n\n### Mapping (subject ↔ attribute)  \nThe relation between subjects and their attributes is defined by a random permutation that depends only on the random seed. The results are not sensitive to that seed.  \nPlease also refer to the additional experiment described in the response to the other reviewers (technical constraints at NeurIPS prevented us from attaching a general response). There we reproduce similar dynamics in a model trained on *natural‑language* data satisfying the truth‑co‑occurrence hypothesis. In that setting the examples no longer have a rigid structure, yet we observe the same phenomena as in the synthetic data.\n\n---\n\n### Eq.&nbsp;1  \nYes—thank you for pointing this out.\n\n---\n\n### Notation  \nThank you as well; we will standardize the notation for the subject and attribute throughout.\n\n---\n\nOnce again, we are grateful for the reviewer’s careful reading and helpful suggestions, and we look forward to refining the paper accordingly."}, {"review_id": "Z5yCgvFmHM", "reviewer": "Reviewer_aif5", "summary": "The paper looks into why and how LLMs learn a linear subspace encoding the truthfulness of statements, a phenomenon observed in many probing studies. They first proposed the TCH (truth co-occurance hypothesis) which means truth statements stay with truth statements in the data, while falsehoods stay with falsehoods. They further trained a toy transformer model to show the two stages in the learning process: the model first rapidly memorized the subject-attribute pairs, and then a linear separation gradually emerges between true and false statements. Finally, experiments with Llama-3-8B and CounterFact data show that after the false sentences exist, it reduces the likelihood of predicting true attributes. Also, when under intervention along the learned truth subspace, the model can be steered back toward correctness.", "questions": "- Why does the linear truth encoding peak in middle layers, e.g. layer 11 in Llama-3-8B? Is this consistent across model scales?\n- Did you try any experiments on real models other than Llama-3-8B?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n- The Truth Co-occurrence Hypothesis is novel and intuitive, and seems to be supported to some extent in the experiments.\n- It offers a simple and interpretable mechanistic explanation for how the truth subspaces emerges in the training dynamics.\n\nWeaknesses:\n\n- The two-phrase finding from the toy transformer experiments seems not well verified on the real LLMs like Llama-3-8B. Only the linear intervention and the true/falsehoods co-occurance assumption was verified, not the two-phrase assumption.\n- I think it's possible to use some of the open model checkpoints along the training trajectory to further verify the training dynamics of the real LLMs, for example, the Pythia models have the middle checkpoints available, which can be leveraged.\n- The toy model and synthetic data are highly idealized (e.g., no MLPs, one attention head, orthogonal embeddings) and it's directly jumped to generalized to real LLMs, which is a bit suspicious. If small scale LLMs, like GPT-2 scale, can also be verified first, it would be more convincing.\n- As different LLMs can have a bit different internal mechanisms, it will be better to test more than a single model family. Besides Llama3-8B, it's better to also show results from other models.", "was_revised": true, "final_justification": "The authors tried their best to address most of my concerns in the rebuttal.\n\nMe: The toy example can be too different from real LLMs.\nAuthors: Train real small-scale transformers on the CounterFact dataset again to show the results.\n\nMe: There are Pythia models with model checkpoints that you can try.\nAuthors: Try new experiments with Pythia, show some similar observations, but also show a small difference (entropy increase before memorization stabilizes).\n\nMe: Show me experiments with more model families.\nAuthors: Try Gemma and Qwen and show the results.\n\nThese additional experiments/contents addressed most of my original concerns. I will be happy to see this paper get accepted if these new experiments/results/discussion can be incorporated into the main sessions of this paper. I raised my score to 5.", "rebuttal": "**We thank the reviewer for their insightful suggestions and for encouraging additional large‑model experiments.**  \nBelow we address each point and indicate the text changes planned for the camera‑ready version.\n\n---\n\n### Theoretical modeling and idealized synthetic setting  \nOur intention is to provide a rigorous characterization in a minimal setting: **if the phenomenon shows up even under these minimalist conditions, it becomes a plausible building block for the far richer dynamics of large‑scale models.** We emphasize that even our simplified objective (Eq. (2) after line 141) already contains a weight matrix $W$ trained *inside* a non‑linearity. This results in a non‑convex function that resembles a 2‑layer network, and its optimization process is known to be challenging to analyze, as accomplished in **Theorem 3**. Our analysis goes a step further and studies the *properties* of the learned weights (**Theorems 1 and 2**), which show the occurrence of the truth‑linear subspace.\n\nPlease note that in the appendix we show that *releasing* the $K,Q,V$ and embeddings parameters (still one layer, but with nothing frozen) **retains the effect**, both the two-phases and the structure of the attention matrix (though when the embeddings are also trained, some of the structure disappears because they can also memorize). We will move this result into the main text for visibility.\n\n---\n\n### Training dynamics on more realistic data  \nFollowing your suggestion, we re‑ran the study in a more realistic setting—the **CounterFact** dataset [1], which contains simple factual statements in natural language. Please note that in contrast to our original synthetic setting, the examples from this dataset contain many different sentence structures and much variation that is characterized to natural language, \n\nWe trained standard Transformer models with **2, 5, and 9 layers** (4 attention heads and one MLP block per layer) on a single relation, updating *all* parameters.  \nEach training sample is a concatenation of two sequences, just as in the paper’s main experiments. To enforce the truth‑co‑occurrence hypothesis, the factuality of the two sequences is matched; in 5 % of the samples we corrupt the attribute by randomly swapping it with another attribute.\n\nThe results reproduce the same **two‑phase learning pattern** seen in our synthetic setup:\n\n1. **Phase 1 — rapid memorization**  \n   Early in training the model places almost all probability mass on the memorized attribute for both true and false contexts, so the entropies are equally low (H\\_true ≈ H\\_false).\n\n2. **Phase 2 — abrupt emergence of a linear encoding**  \n   A few hundred steps later a strong linear signal appears in the penultimate token of the second sequence (AUC\\_S2 ≈ 0.97).  \n   At the same time H\\_false shoots up while H\\_true drops, exactly as predicted by theory.\n\nBecause NeurIPS technical constraints, we summarize the average numbers across training phases.\n\n| Phase | Window&nbsp;(steps) | $\\\\mathrm{H_{true}}$ | $\\\\mathrm{H_{false}}$ | $\\\\mathrm{AUC_{S_1}}$ | $\\\\mathrm{AUC_{S_2}}$ |\n|:---|:---|:---:|:---:|:---:|:---:|\n| **2‑layer** — memorization | 0 – 1.2 k | 0.39 | 0.39 | 0.85 | 0.51 |\n| **2‑layer** — emergence | 1.4 k – 2 k → | ↓ 0.14 | ↑ 2.90 | 0.82 | **0.97** |\n| **5‑layer** — memorization | 0 – 750 | 0.41 | 0.40 | 0.70 | 0.52 |\n| **5‑layer** — emergence | 0.8 k – 1 k → | ↓ 0.12 | ↑ 2.31 | 0.98 | **0.97** |\n| **9‑layer** — memorization | 0 – 500 | 0.41 | 0.41 | 0.62 | 0.51 |\n| **9‑layer** — emergence | 0.6 k – 1 k → | ↓ 0.11 | ↑ 2.64 | 0.97 | **0.96** |\n\nHere, AUC\\_S1 / AUC\\_S2 refer to the Area Under the ROC Curve of a linear probe trained on  \n&nbsp;&nbsp;&nbsp;&nbsp;• **S1**: the last token of the first sentence, or  \n&nbsp;&nbsp;&nbsp;&nbsp;• **S2**: the penultimate token of the second sentence.  \n\nIn the 2‑layer model the memorization plateau lasts until ≈ 1.2 k steps; then the linear encoding emerges and H\\_false rises, reducing loss on corrupted samples.  \nThe deeper models transition earlier—after only 500–750 steps—but follow the same trajectory.  \nAll metrics here are taken from the final layer. In the deeper networks the penultimate layer reaches comparable accuracy, while earlier layers stay near chance. Overall, these results confirm the two‑phase dynamics we predict in a more realistic setting.  \n\n---\n\n### Evaluating checkpoints of a “real” LM  \nTo test whether the two‑phase dynamics also appear in a large model trained on open‑web data, we analyzed the **Pythia‑6.9 B** training checkpoints released by EleutherAI. Using the CounterFact dataset we construct each input by concatenating $K = 4$ factual statements whose preceding context is either entirely true or entirely false, mirroring our previous setup.  \nFor every checkpoint we measure three signals on the final statement:\n\n* **Memorization**: the percentage of cases where greedy decoding suceeds in completing the correct token.  \n* **Uncertainty**: the entropy of the model’s full‑vocabulary distribution for predicting the last token (we record the difference in entropy between true and false context).  \n* **Linear separability**: the accuracy of a linear probe trained to classify the truth value of the surrounding context.\n\n| step | $\\\\Delta H$ | memorization | probe&nbsp;AUC |\n|------:|-------:|---------------:|--------------:|\n| 0 | 0.001 | 0.000 | 0.383 |\n| 512 | 0.006 | 0.000 | 0.435 |\n| 1 000 | 0.005 | 0.006 | 0.467 |\n| 3 000 | 0.219 | 0.242 | 0.587 |\n| 5 000 | 0.217 | 0.435 | 0.648 |\n| 10 000 | 0.286 | 0.547 | 0.667 |\n| 20 000 | 0.355 | 0.655 | 0.754 |\n| 40 000 | 0.329 | 0.727 | 0.759 |\n| 60 000 | 0.421 | 0.772 | 0.802 |\n| 80 000 | 0.419 | 0.822 | 0.799 |\n| 100 000 | 0.479 | 0.835 | 0.818 |\n| 110 000 | 0.485 | 0.849 | 0.835 |\n| 120 000 | 0.536 | 0.842 | 0.835 |\n| 130 000 | 0.565 | 0.858 | 0.783 |\n| 143 000 | 0.518 | 0.875 | 0.831 |\n\n$\\\\Delta H$ denotes the entropy gap between matched prompt pairs presented with false versus true context. The memorization rate is the share of instances in which the model’s output distribution places the correct continuation token at the top‑1 position.\n\n**Findings.**  \n* **Early training (≤ 1 k steps).** $\\\\Delta H \\\\approx 0$; the model memorizes indiscriminately.  \n* **Mid training (3 k–80 k).** Memorization jumps, then plateaus, while $\\\\Delta H$ and probe accuracy climb steadily.  \n* **Late training (≥ 80 k).** Entropy separation continues to widen even after memorization saturates, mirroring Phase 2 but over a longer horizon.\n\nThis echoes the two‑phase pattern we observed in simpler experiments: an initial jump in memorization followed by a slower, steadier increase in entropy separation. There are, however, several differences. The second phase is more gradual, and classification and entropy increase even *before* memorization stabilizes. These differences likely stem from the model’s continual exposure to new facts during training, unlike our idealized setup where all facts are seen in a single gradient step. The modest final memorization and classification scores align with prior reports that the Pythia series is under‑trained relative to its capacity.\n\n**In sum**, both experiments corroborate the predicted two‑phase dynamics we observe.\n\n---\n\n### Additional models in the intervention experiments\nFollowing your concerns on the intervention experiments, we repeated the experiments on three more models with 1,024 examples (8× larger). Reported values, as in the main text, are **NLL deltas** between false and true contexts:\n\n* **Δ (Base F → Base T)** — change when switching the *base* model from false to true context.  \n* **Δ (Base F → Interv F)** — change between *base* and *intervened* model on **false** context.\n\n| Model | false‑context length | Δ (Base F → Base T) | Δ (Base F → Interv F) |\n|:-----------------|:----------------:|:---------------:|:------------------:|\n| **Gemma2 7B** | FF | −4.25 | −3.36 |\n| | FFF | −3.68 | −3.09 |\n| | FFFF | −3.37 | −2.48 |\n| **Llama** | FF | −1.51 | −0.24 |\n| | FFF | −1.98 | −0.36 |\n| | FFFF | −2.18 | −0.31 |\n| **Qwen 2-7B** | FF | −0.30 | −0.15 |\n| | FFF | −0.38 | −0.22 |\n| | FFFF | −0.41 | −0.18 |\n\nPatterns are broadly consistent, though **Gemma** is more sensitive to false context than the others.\n\n---\n\n### References  \nMeng, K., Bau, D., Andonian, A., & Belinkov, Y. *Locating and Editing Factual Associations in GPT*, 2022.\n\n---\n\nRegarding your question on layer 11, this is in line with previous work that indicates that many human-interpretable concepts are \"maximized\" in middle layers. We agree that understanding why it is maximized in that particular layer is very interesting, but think that this is a different question than the focus of this paper, and therefore leave it for a future work.\n\nOnce again, we appreciate the reviewer’s constructive feedback and believe these additional results strengthen the paper’s empirical grounding."}, {"review_id": "tO8j8I25dM", "reviewer": "Reviewer_za5X", "summary": "The authors constructed a simplified single-layer Transformer and successfully reproduced the emergence of linear truth encodings on a synthetic dataset. They proposed the Truth Co-occurrence Hypothesis (TCH), which posits that true statements tend to co-occur with other true statements, while false statements tend to co-occur with other false statements. The authors validated this hypothesis and revealed a two-stage dynamic in the model's training process: an initial rapid memorization phase, followed by the emergence of linear truth encoding.", "questions": "See Weaknesses", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1、Truth Co-occurrence Hypothesis (TCH) is novel. The hypothesis offers a plausible explanation for the emergence of linear truth encodings, suggesting that the co-occurrence patterns of true and false statements drive the model to learn to distinguish between them.\n\n2、Despite the model and the synthetic data are simple, the model effectively demonstrated key phenomena supporting the hypothesis, such as changes in the value matrix (Figure 1) and the model's ability to differentiate between true and false sequences (Figure 2).\n\n3、The authors conducted experiments on the LLAMA3-8B model. Observing similar behaviors in this larger model.\n\nWeaknesses\n\n1. The use of a 1/2/3-layer Transformer model deviates significantly from the architecture of real-world LLMs, which typically have many layers. It is recommended to test the hypothesis in models with many layers to demonstrate its robustness and independence from model depth.\n\n2. The synthetic dataset consists of simple quadruples, which fail to capture the complexity of real-world data distributions. In practice, datasets may involve more diverse subjects and partial truths.\n\n3. In Section 5.2, the authors evaluated the LLAMA3-8B model using only 128 samples from the CounterFact dataset. This limited scope—both in terms of model and dataset—restricts the generalizability of their conclusions.\n\nThese weaknesses raise concerns about the practical validity of the paper, significantly decreasing my ratings.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns. There are new experiments on LLMs and realistic data now.", "rebuttal": "**We thank the reviewer for the thorough review and the encouragement to report empirical results on more realistic settings.**  \nTo address concerns about small models and synthetic data, we conducted two complementary experiments:\n\n1. **Natural‑language facts.** We replicated our study by training fully‑parameterized Transformers on the CounterFact dataset.  \n2. **Large‑model trajectory.** We traced the entire training path of **Pythia‑6.9 B** to see whether the same dynamics emerge in the wild.\n\n---\n\n### Training dynamics on more realistic data  \nFollowing your suggestion, we re‑ran the study in a more realistic setting—the **CounterFact** dataset [1], which contains simple factual statements in natural language. Please note that in contrast to our original synthetic setting, the examples from this dataset contain many different sentence structures and much variation that is characterized to natural language, \n\nWe trained standard Transformer models with **2, 5, and 9 layers** (4 attention heads and one MLP block per layer) on a single relation, updating *all* parameters.  \nEach training sample is a concatenation of two sequences, just as in the paper’s main experiments. To enforce the truth‑co‑occurrence hypothesis, the factuality of the two sequences is matched; in 5 % of the samples we corrupt the attribute by randomly swapping it with another attribute.\n\nThe results reproduce the same **two‑phase learning pattern** seen in our synthetic setup:\n\n1. **Phase 1 — rapid memorization**  \n   Early in training the model places almost all probability mass on the memorized attribute for both true and false contexts, so the entropies are equally low (H\\_true ≈ H\\_false).\n\n2. **Phase 2 — abrupt emergence of a linear encoding**  \n   A few hundred steps later a strong linear signal appears in the penultimate token of the second sequence (AUC\\_S2 ≈ 0.97).  \n   At the same time H\\_false shoots up while H\\_true drops, exactly as predicted by theory.\n\nBecause NeurIPS technical constraints, we summarize the average numbers across training phases.\n\n| Phase | Window&nbsp;(steps) | $\\\\mathrm{H_{true}}$ | $\\\\mathrm{H_{false}}$ | $\\\\mathrm{AUC_{S_1}}$ | $\\\\mathrm{AUC_{S_2}}$ |\n|:---|:---|:---:|:---:|:---:|:---:|\n| **2‑layer** — memorization | 0 – 1.2 k | 0.39 | 0.39 | 0.85 | 0.51 |\n| **2‑layer** — emergence | 1.4 k – 2 k → | ↓ 0.14 | ↑ 2.90 | 0.82 | **0.97** |\n| **5‑layer** — memorization | 0 – 750 | 0.41 | 0.40 | 0.70 | 0.52 |\n| **5‑layer** — emergence | 0.8 k – 1 k → | ↓ 0.12 | ↑ 2.31 | 0.98 | **0.97** |\n| **9‑layer** — memorization | 0 – 500 | 0.41 | 0.41 | 0.62 | 0.51 |\n| **9‑layer** — emergence | 0.6 k – 1 k → | ↓ 0.11 | ↑ 2.64 | 0.97 | **0.96** |\n\nHere, AUC\\_S1 / AUC\\_S2 refer to the Area Under the ROC Curve of a linear probe trained on  \n&nbsp;&nbsp;&nbsp;&nbsp;• **S1**: the last token of the first sentence, or  \n&nbsp;&nbsp;&nbsp;&nbsp;• **S2**: the penultimate token of the second sentence.  \n\nIn the 2‑layer model the memorization plateau lasts until ≈ 1.2 k steps; then the linear encoding emerges and H\\_false rises, reducing loss on corrupted samples.  \nThe deeper models transition earlier—after only 500–750 steps—but follow the same trajectory.  \nAll metrics here are taken from the final layer. In the deeper networks the penultimate layer reaches comparable accuracy, while earlier layers stay near chance. Overall, these results confirm the two‑phase dynamics we predict in a more realistic setting.  \n\n---\n\n### Evaluating checkpoints of a “real” LM  \nTo test whether the two‑phase dynamics also appear in a large model trained on open‑web data, we analyzed the **Pythia‑6.9 B** training checkpoints released by EleutherAI. Using the CounterFact dataset we construct each input by concatenating $K = 4$ factual statements whose preceding context is either entirely true or entirely false, mirroring our previous setup.  \nFor every checkpoint we measure three signals on the final statement:\n\n* **Memorization**: the percentage of cases where greedy decoding suceeds in completing the correct token.  \n* **Uncertainty**: the entropy of the model’s full‑vocabulary distribution for predicting the last token (we record the difference in entropy between true and false context).  \n* **Linear separability**: the accuracy of a linear probe trained to classify the truth value of the surrounding context.\n\n| step | $\\\\Delta H$ | memorization | probe&nbsp;AUC |\n|------:|-------:|---------------:|--------------:|\n| 0 | 0.001 | 0.000 | 0.383 |\n| 512 | 0.006 | 0.000 | 0.435 |\n| 1 000 | 0.005 | 0.006 | 0.467 |\n| 3 000 | 0.219 | 0.242 | 0.587 |\n| 5 000 | 0.217 | 0.435 | 0.648 |\n| 10 000 | 0.286 | 0.547 | 0.667 |\n| 20 000 | 0.355 | 0.655 | 0.754 |\n| 40 000 | 0.329 | 0.727 | 0.759 |\n| 60 000 | 0.421 | 0.772 | 0.802 |\n| 80 000 | 0.419 | 0.822 | 0.799 |\n| 100 000 | 0.479 | 0.835 | 0.818 |\n| 110 000 | 0.485 | 0.849 | 0.835 |\n| 120 000 | 0.536 | 0.842 | 0.835 |\n| 130 000 | 0.565 | 0.858 | 0.783 |\n| 143 000 | 0.518 | 0.875 | 0.831 |\n\n$\\\\Delta H$ denotes the entropy gap between matched prompt pairs presented with false versus true context. The memorization rate is the share of instances in which the model’s output distribution places the correct continuation token at the top‑1 position.\n\n**Findings.**  \n* **Early training (≤ 1 k steps).** $\\\\Delta H \\\\approx 0$; the model memorizes indiscriminately.  \n* **Mid training (3 k–80 k).** Memorization jumps, then plateaus, while $\\\\Delta H$ and probe accuracy climb steadily.  \n* **Late training (≥ 80 k).** Entropy separation continues to widen even after memorization saturates, mirroring Phase 2 but over a longer horizon.\n\nThis echoes the two‑phase pattern we observed in simpler experiments: an initial jump in memorization followed by a slower, steadier increase in entropy separation. There are, however, several differences. The second phase is more gradual, and classification and entropy increase even *before* memorization stabilizes. These differences likely stem from the model’s continual exposure to new facts during training, unlike our idealized setup where all facts are seen in a single gradient step. The modest final memorization and classification scores align with prior reports that the Pythia series is under‑trained relative to its capacity.\n\n**In sum**, both experiments corroborate the predicted two‑phase dynamics we observe.\n\n---\n\n### Additional models in the intervention experiments\nFollowing your concerns on the intervention experiments, we repeated the experiments on three more models with 1,024 examples (8× larger). Reported values, as in the main text, are **NLL deltas** between false and true contexts:\n\n* **Δ (Base F → Base T)** — change when switching the *base* model from false to true context.  \n* **Δ (Base F → Interv F)** — change between *base* and *intervened* model on **false** context.\n\n| Model | false‑context length | Δ (Base F → Base T) | Δ (Base F → Interv F) |\n|:-----------------|:----------------:|:---------------:|:------------------:|\n| **Gemma2 7B** | FF | −4.25 | −3.36 |\n| | FFF | −3.68 | −3.09 |\n| | FFFF | −3.37 | −2.48 |\n| **Llama** | FF | −1.51 | −0.24 |\n| | FFF | −1.98 | −0.36 |\n| | FFFF | −2.18 | −0.31 |\n| **Qwen 2-7B** | FF | −0.30 | −0.15 |\n| | FFF | −0.38 | −0.22 |\n| | FFFF | −0.41 | −0.18 |\n\nPatterns are broadly consistent, though **Gemma** is more sensitive to false context than the others.\n\n---\n\n### Reference  \n[1] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. *Locating and Editing Factual Associations in GPT*, 2022."}]

Author rebuttal:

Meta-review: The paper studies the emergence of linear truth encodings (linear subspaces that separate true and false statements in the internal representations of a trained model). The paper studies a simple one-layer transformer model trained on a data distribution where true facts co-occur with other true facts more often than false facts. In this toy setting they show that the model learns to classify truthful versus untruthful statements linearly by their norms in a two-phase mechanism. The authors corroborate their findings on larger pre-trained models experimentally.

All reviewers agreed that the studied hypothesis is interesting and the toy setup is a clean abstraction of it. There were several concerns raised initially: (1) the simplicity of the Transformer model having only one linear trainable parameter, and the trends not being as clear when more parameters were trained, (2) lack of empirical evidence on pre-trained model matching the two-phase dynamics proposed in the theory, (3) limited evaluation on different sized models, and more realistic datasets. Apart from (1), the other concerns were addressed by the authors during the rebuttal with extensive experiments.

While the theoretical setup remain too idealized, the empirical validation of the phenomenon is good, and the core hypothesis is simple and plausible. Therefore I lean towards accept. I encourage the authors to include the new experiments done during the rebuttal period and improve the exposition to address reviewer concerns.
Decision: Accept (poster)