Title: Emergence of Linear Truth Encodings in Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: truth

factuality

linearity

LMs
Abstract: Recent probing studies reveal that large language models exhibit linear subspaces that separate true from false statements, yet the mechanism behind their emergence is unclear. We introduce a transparent, one-layer transformer toy model that reproduces such truth subspaces end-to-end and exposes one concrete route by which they can arise. We study one simple setting in which truth encoding can emerge: a data distribution where factual statements co-occur with other factual statements (and vice-versa), encouraging the model to learn this distinction in order to lower the LM loss on future tokens. We corroborate this pattern with experiments in pretrained language models. Finally, in the toy setting we observe a two-phase learning dynamic: networks first memorize individual factual associations in a few steps, then---over a longer horizon---learn to linearly separate true from false, which in turn lowers language-modeling loss. Together, these results provide both a mechanistic demonstration and an empirical motivation for how and why linear truth representations can emerge in language models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors constructed a simplified single-layer Transformer and successfully reproduced the emergence of linear truth encodings on a synthetic dataset. They proposed the Truth Co-occurrence Hypothesis (TCH), which posits that true statements tend to co-occur with other true statements, while false statements tend to co-occur with other false statements. The authors validated this hypothesis and revealed a two-stage dynamic in the model's training process: an initial rapid memorization phase, followed by the emergence of linear truth encoding.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1、Truth Co-occurrence Hypothesis (TCH) is novel. The hypothesis offers a plausible explanation for the emergence of linear truth encodings, suggesting that the co-occurrence patterns of true and false statements drive the model to learn to distinguish between them.\n\n2、Despite the model and the synthetic data are simple, the model effectively demonstrated key phenomena supporting the hypothesis, such as changes in the value matrix (Figure 1) and the model's ability to differentiate between true and false sequences (Figure 2).\n\n3、The authors conducted experiments on the LLAMA3-8B model. Observing similar behaviors in this larger model.\n\nWeaknesses\n\n1. The use of a 1/2/3-layer Transformer model deviates significantly from the architecture of real-world LLMs, which typically have many layers. It is recommended to test the hypothesis in models with many layers to demonstrate its robustness and independence from model depth.\n\n2. The synthetic dataset consists of simple quadruples, which fail to capture the complexity of real-world data distributions. In practice, datasets may involve more diverse subjects and partial truths.\n\n3. In Section 5.2, the authors evaluated the LLAMA3-8B model using only 128 samples from the CounterFact dataset. This limited scope—both in terms of model and dataset—restricts the generalizability of their conclusions.\n\nThese weaknesses raise concerns about the practical validity of the paper, significantly decreasing my ratings.\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper looks into why and how LLMs learn a linear subspace encoding the truthfulness of statements, a phenomenon observed in many probing studies. They first proposed the TCH (truth co-occurance hypothesis) which means truth statements stay with truth statements in the data, while falsehoods stay with falsehoods. They further trained a toy transformer model to show the two stages in the learning process: the model first rapidly memorized the subject-attribute pairs, and then a linear separation gradually emerges between true and false statements. Finally, experiments with Llama-3-8B and CounterFact data show that after the false sentences exist, it reduces the likelihood of predicting true attributes. Also, when under intervention along the learned truth subspace, the model can be steered back toward correctness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n- The Truth Co-occurrence Hypothesis is novel and intuitive, and seems to be supported to some extent in the experiments.\n- It offers a simple and interpretable mechanistic explanation for how the truth subspaces emerges in the training dynamics.\n\nWeaknesses:\n\n- The two-phrase finding from the toy transformer experiments seems not well verified on the real LLMs like Llama-3-8B. Only the linear intervention and the true/falsehoods co-occurance assumption was verified, not the two-phrase assumption.\n- I think it's possible to use some of the open model checkpoints along the training trajectory to further verify the training dynamics of the real LLMs, for example, the Pythia models have the middle checkpoints available, which can be leveraged.\n- The toy model and synthetic data are highly idealized (e.g., no MLPs, one attention head, orthogonal embeddings) and it's directly jumped to generalized to real LLMs, which is a bit suspicious. If small scale LLMs, like GPT-2 scale, can also be verified first, it would be more convincing.\n- As different LLMs can have a bit different internal mechanisms, it will be better to test more than a single model family. Besides Llama3-8B, it's better to also show results from other models.\n\nQUESTIONS:\n- Why does the linear truth encoding peak in middle layers, e.g. layer 11 in Llama-3-8B? Is this consistent across model scales?\n- Did you try any experiments on real models other than Llama-3-8B?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the emergence of linear separable truth encoding in Transformer models. They explore a toy setting with two token pairs where if the first token pair follows a 'ground truth' distribution, the second token pair is likely to also follow that distribution (replicating the Truth Co-occurrence Hypothesis: in naturally occurring text,\ntrue statements are statistically more likely to co-occur with other true statements, and falsehoods with other falsehoods.). They show that the transformer first learns the ground truth mapping and only later learns to apply it to the second token pair only in the case of a first ground truth pair.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nI think studying the Co-occurrence Hypothesis for truthful statements in a toy example where one can directly control conditions is a good approach.\nThe authors motivate their approach well and set a clear goal/focus. The paper is set up in a well structured way.\nThe contribution is to the best of my knowledge novel and significant.\nThey adequalely cover and acknowledge previous work.\n\n\n**Weaknesses:**\n\nThe authors could be a bit more diligent with defining symbols to make it easier for readers to follow. Some Figure captions could be improved for clarity.\n\nminor:\n\n- You should not use LM in the abstract without defining it (which you could easily do since you already mention large language models)\n- line 68 typo by -> be\n- line 161: verb missing in '(though we still some lower confidence spikes...'\n\nQUESTIONS:\n**Questions:**\n\n- In Figure 2: would it make sense to plot averaged values over several inputs?\n- Are sequences always 4 tokens long? What happens if you make them longer?\n- Could you add a few examples of your dataset? How do you define a^* is it just 2*s? Does it matter how you define it or can it be an arbitrary mapping?\n- In (1) shouldn't the prefix be x=(s1 a1 s2)?\n- Why do you switch between s,a and x,y? why not just stick with s and a?\n\n\n**Suggestions:**\n\n- explain shortly what e_x, u_{g(x)} etc is in figure caption so the figure+caption is self contained. Maybe also mark which block is which, since it's hard to infer just by looking at the figure.\n- add axes labels in Figure 2\n- in the setup in line 137 you could add the symbols e, p, u when after embedding, positional embedding, and unembedding respectively to have them clearly defined\n- add definition for F in eq (2)\n- add definition of g when symbol is first used\n- be more consistent with what you call things: for example you define a sequence with subjects and attributes, but later in line 155 you call then input and label\n- maybe for Figure 4 c) add that these are learned embeddings since in the toy example they are pre defined\n\nLIMITATIONS:\nThe authors adequately discuss limitations and list interesting future directions", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a simple theory that aims to explain, in part, how language models learn “truth” directions. They introduce the Truth-Co-occurrence Hypothesis (TCH)—namely, that true statements tend to appear near other true statements, and likewise for falsehoods. To test this idea, they build a TCH-driven synthetic dataset in which truthful or untruthful statements always occur together, then train a single-layer transformer (with several components frozen) on that data. Internally, the model learns to classify truthful versus untruthful statements linearly by their norms. The authors also explore whether a similar mechanism emerges in larger language models under a different setup. The paper offers an interesting perspective and illustrative toy experiments showing how a model might acquire a linear truth direction; however, I still feel the findings remain largely detached from real-world LMs (see my comments below).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* TCH is a simple yet interesting hypothesis.\n* The toy setting is engaging, and the observation that the learned norms encode a linear direction that separates truthful from untruthful statements is likewise noteworthy.\n* The paper is well written and easy to follow.\n\n**Weaknesses**\n\n* **The *main* toy setting imposes interesting yet unrealistic constraints that may render the norm-separation discovery trivial.** The toy model essentially freezes everything except *a single weight matrix*. Given that truthful data are more frequent and the loss is unbounded (i.e., pushed toward more negative values), it is unsurprising that some truthful subspace in the value matrix (i.e., certain column spans) receives more frequent gradient updates, driving its norms out of distribution. Although one could argue the learned value matrix is full-rank, the small vocabulary means the meaningful updates are effectively low-rank. In line 272, the authors even note that this finding “disappears when learning the KV matrices instead of using fixed attention.” I suspect this is because the pressure on norms vanishes once more parameters are trainable. Overall, uniform attention strikes me as an unrealistic assumption.\n\n  *P.S.* Proving a hypothesis with toy settings is fine if all claims are stated with explicit assumptions, but I remain unconvinced that this is true here.\n\n* **The follow-up study on the 8B LM does not test the training dynamics observed in the toy example at scale.** The toy experiment purports to show that TCH explains the emergence of a linear truth direction; the 8B study instead demonstrates that an LM is biased by untruthful context and that a steering vector can restore truthfulness—both fairly well-known results. I would urge the authors to scale up their training-dynamics analysis (e.g., loosen the constraints or train larger transformers on more realistic, partially corrupted factual corpora).\n\nQUESTIONS:\nSee my comments above.\n\nLIMITATIONS:\nSee my comments above.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}