Title: From Noise to Narrative: Tracing the Origins of Hallucinations in Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Mechanistic interpretability

sparse autoencoders

transformer hallucinations

concept steering

vision transformers

large language models

noise‑trained representations

inductive bias

robustness and uncertainty

AI safety and alignment

representation learning
Abstract: As generative AI systems become competent and democratized in science, business, and government, deeper insight into their failure modes now poses an acute need. The occasional volatility in their behavior, such as the propensity of transformer models to hallucinate, impedes trust and adoption of emerging AI solutions in high-stakes areas. In the present work, we establish how and when hallucinations arise in pre-trained transformer models through concept representations captured by sparse autoencoders, under scenarios with experimentally controlled uncertainty in the input space. Our systematic experiments reveal that the number of semantic concepts used by the transformer model grows as the input information becomes increasingly unstructured. In the face of growing uncertainty in the input space, the transformer model becomes prone to activate coherent yet input-insensitive semantic features, leading to hallucinated output. At its extreme, for pure-noise inputs, we identify a wide variety of robustly triggered and meaningful concepts in the intermediate activations of pre-trained transformer models, whose functional integrity we confirm through targeted steering. We also show that hallucinations in the output of a transformer model can be reliably predicted from the concept patterns embedded in transformer layer activations. This collection of insights on transformer internal processing mechanics has immediate consequences for aligning AI models with human values, AI safety, opening the attack surface for potential adversarial attacks, and providing a basis for automatic quantification of a model’s hallucination risk.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates how hallucinations arise in transformer models by analyzing residual stream activations using sparse autoencoders (SAEs), where each decoder direction defines a “concept.” Two metrics, semantic purity and steerability, are used to assess these concepts. The authors train SAEs on residual stream from noise inputs to a vision transformer and find that many resulting concepts still align with coherent semantics when tested on real images. They further show that as input uncertainty increases, more concepts are activated—especially in middle layers. Using this framework, the authors predict hallucination rates in Gemma-2B-IT summaries based on concept activations from the input text. The main message of the paper is that increased input uncertainty corresponds to a higher number of activated concepts, which leads to increased hallucination rates.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n\n1. The use of sparse autoencoders trained on residual stream activations from pure noise is an interesting setup, suggesting that some semantic structure is baked into the model’s internal representations.\n2. The study, by varying input (such as image patch shuffling and text n-gram scrambling), is convincing and shows that input ambiguity can lead to higher concept activation of the SAEs.\n3. The idea of using internal concept activations to predict hallucination rates in Gemma-2B-IT summaries is a solid step toward connecting the model's internal dynamics to its output behavior.\n\n**Weakness:**\n\n1. While the paper's main goal is to understand the source of hallucination in transformers, much of the work—particularly the analysis focused on vision transformers—feels only loosely connected to this objective. The only part that directly addresses hallucination is the summarization experiment on Gemma-2B-IT. Even there, the results are somewhat limited; it would strengthen the paper if the authors clarified what kinds of hallucinations are being predicted and whether certain types are more reliably captured by the concept activations.\n2. The $R^2$ of 0.27 reported in Figure 5 is modest and not quite convincing, and the paper did not compare its method against established hallucination-detection baselines.\n3. The paper is difficult to follow at times due to a lack of precise definitions for key terms and notations. For example:\n   - how is the function $f_i$ defined?\n   - what do you mean by a concept being \"activated\"? the term was formally never defined.\n   - the concepts of semantic purity and steerability are not properly defined.\n\n**Overall:** The paper offers an interesting analysis of how input uncertainty propagates through a transformer’s internal dynamics. However, the step from these observations to a practical hallucination detector and potential mitigation strategies is only loosely argued and supported by limited evaluation. Consequently, I do not believe the work is yet ready for publication at NeurIPS.\n\nQUESTIONS:\nSee weakness section.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates representations of semantic concepts in transformers (in vision and in text) as the uncertainty in the input space is varied through the lens of sparse autoencoders. They find that as inputs become noisier, more semantic concepts are activated. They then attempt to use these findings for the use case of hallucination detection and then steer models to prevent hallucination using the learned concept directions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Very well written and thorough investigation from first principles from hypothesis to experimental design and results.\n- Strong, interesting and (to my knowledge) novel results for mechanistic understanding of concept activations in the presence of noisy data.\n\nWeaknesses:\n\n- The hallucination detection experiments are not particularly convincing as they are lacking detail and not aligned/compared with standard baselines in the LLM uncertainty quantification literature (e.g. short form Q+A generations). Particularly not convinced of the jump from aleatoric noise in input space leading to more concepts translates to hallucinations.\n- The macro level results of L0 concept activitations lack the granularity of individual sample results - some qualitative examples on these would be useful.\n\nOverall the paper has lots of exciting insights even without the hallucination section, which needs to be made more concrete.\n\nQUESTIONS:\n- Is noisy input data even indicative of likelihood of hallucination? Some explanation of how this links to aleatoric/epistemic uncertainty and what kinda of hallucinations you are thinking about would be useful. More experiments from the LLM uncertainty quantification/hallucination literature would also be useful to dissect this. Generalisation to new datasets.\n- Do you have an interpretation for why more concepts are activated than fewer? Is there a “randomness”/”noise”/”entropy” concept that emerges for messy input data?\n- How do you decompose the likelihood of hallucination as a function of output length, temperature of sampling?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes the use of sparse auto-encoders (SAEs) to probe conceptual representations at different layers of Transformer models, towards a mechanistic understanding of ‘hallucinations’ in these models. Three experimental setups are explored: SAEs trained on the activations of Transformers exposed to pure noise; SAEs trained on activations from Transformers exposed to real image and text data; and pre-trained SAEs. \n\nA key finding of the paper is that Transformer models are prone to imposing semantic structure on meaningless inputs (perhaps akin to the phenomenon of ‘pareidolia,’ or the human tendency to perceive meaningful structures in random stimuli, e.g. seeing faces in clouds). Another finding is that perturbed inputs activate a greater number of concepts than unperturbed inputs in the middle layers of Transformer models, an effect that scales up with the degree of perturbation. Additionally, the findings suggest that hallucinations can be steered based on selective inputs to a model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper is well-organized and clear, with ample definitions and concise descriptions of relevant prior work. The paper provides an interesting and compelling approach to tracing hallucinations in Transformer models, with valuable findings that pave the way for future work in AI safety and interpretability, while complementing existing work. Appendix B offers a particularly valuable case study of how different kinds of features are activated in each layer of OpenCLIP ViT-B/32 exposed to pure noise, with increasing conceptual specificity as layers progress. \n\nWeaknesses:\nNot aware of serious weaknesses in the paper. Just an observation: References [35] and [36] are the same paper (Maynez et al 2020).\n\nQUESTIONS:\nNone at this time.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
At a high-level, the paper shows transformer models create signals (concepts) out of noise. The middle layers contribute more to this aspect, and this contributes to the footprint of hallucination. More specifically, the paper trains a SAE to map residual activations to concepts. The paper presents performs multiple experiments to demonstrate the validity of the approach using both text and image models. (1) Semantic concepts are invoked by pure noise inputs [Section 3]. (2) Distorted inputs also invoke concepts [Section 4] (3) Internal activations are also predictive of hallucinations [Section 5]. All the experiments leverage SAE-induced representations.

STRENGTHS AND WEAKNESSES:
S1. The problems addressed in the paper (interpreting activation of transformer models) are important and the conclusions (that SAE-induced interpretable representations can be used to trace hallucinations) are plausible.

S2: The paper presents multiple experiments to validate the proposed framework.

W1. The details of modeling are missing in the main paper.  Unfortunately, I am not able to find the appendix sections in the downloaded version. Hence it is impossible to provide an informed review.

W2. The experimental details (such as sizes of datasets and train/test splits) are missing.

QUESTIONS:
1.	Define residual activation.

2.	What is the significance of 1.3M?

3.	Are d_i in section 2.2 learned or initialized?

4.	In section 2.2, x in LHS is of dimension d_model, but the RHS is of dimensions d_SAE since the unit vectors have that dimension. How does this all fit?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-06