From Noise to Narrative: Tracing the Origins of Hallucinations in Transformers
Authors: Praneet Suresh
Jack Stanley
Sonia Joseph
Luca Scimeca
Danilo Bzdok
Venue: NeurIPS 2025 poster ()
Keywords: Mechanistic interpretability
sparse autoencoders
transformer hallucinations
concept steering
vision transformers
large language models
noise‑trained representations
inductive bias
robustness and uncertainty
AI safety and alignment
representation learning
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: Under input uncertainty, transformer models exhibit a systematic exploration of input‑agnostic conceptual representations, increasing the likelihood of hallucinations.

Abstract: As generative AI systems become competent and democratized in science, business, and government, deeper insight into their failure modes now poses an acute need. The occasional volatility in their behavior, such as the propensity of transformer models to hallucinate, impedes trust and adoption of emerging AI solutions in high-stakes areas. In the present work, we establish how and when hallucinations arise in pre-trained transformer models through concept representations captured by sparse autoencoders, under scenarios with experimentally controlled uncertainty in the input space. Our systematic experiments reveal that the number of semantic concepts used by the transformer model grows as the input information becomes increasingly unstructured. In the face of growing uncertainty in the input space, the transformer model becomes prone to activate coherent yet input-insensitive semantic features, leading to hallucinated output. At its extreme, for pure-noise inputs, we identify a wide variety of robustly triggered and meaningful concepts in the intermediate activations of pre-trained transformer models, whose functional integrity we confirm through targeted steering. We also show that hallucinations in the output of a transformer model can be reliably predicted from the concept patterns embedded in transformer layer activations. This collection of insights on transformer internal processing mechanics has immediate consequences for aligning AI models with human values, AI safety, opening the attack surface for potential adversarial attacks, and providing a basis for automatic quantification of a model’s hallucination risk.

From Noise to Narrative: Tracing the Origins of Hallucinations in Transformers
Praneet Suresh"
Jack Stanley"
Mila Quebec Al Institute
Mila Quebec AI Institute
praneet. suresh@mila quebec
jack. swaaleyymillaaaam. queeee quebec
Sonia Joseph
Luca Scimeca
Mila Quebec AI Institute, Meta AI
Mila Quebec Al Institute
sonia. joseph@mi la. quebbe
luca. www.scammiimma...mmmmmmmmmmmmm quebec
Danilo Bzdok
Mila Quebec Al Institute
bzdokdan@mila quebec
Abstract
As generative Al systems become competent and democratized in science, business, and government, deeper insight into their failure modes poses The occasional volatility in their behavior, such as the propensity of transformer models to hallucinate, impedes high-stakes areas. In the present work, we establish how and when hallucinations arise in pre-trained transformer models through concept representations captured by sparse autoencoders, under scenarios with experimentally controlled uncertainty in the input space. Our systematic experiments reveal that the number of semantic concepts used by the transformer model grows as the input information becomes increasingly unstructured. In the face growing uncertainty the input space, the transformer model becomes prone to activate coherent yet input-insensitive semantic features, leading hallucinated output.
need.
and adoption of emerging Al solutions
extreme, for pore-noise inputs, we identify wide variety of robustly triggered and meaningful concepts in the intermediate activations of pre-trained transformer models, whose functionnl integrity we confirm through targeted steering. We also show that hallucinations in the output of a transformer model can be reliably predicted from the concept patterns embedded in transformer layer activations. This collection of insights on transformer internal processing mechanics has immediate consequences for aligning AI models with human values, AI safety, opening the attack surface for potential adversarial attacks, and providing a basis for automatic quantification model's hallucination risk.
Introduction
Generative AI systems are becoming always more capable, demonstrating remarkable performance on wide range tasks, across diverse data modalities. Yet, transformer models can at times confidently misstate facts, invent details in generated content, and engage in confabulated reasoning. These model behaviors occur even when accurate information for the task is abundantly present in the models' training data or even context window. These plausibleseemmngg yet incorrect outputs are
Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

often referred to as "hallucinations", and range from simple factual mistakes to potentially harmful and deceptive or mis-aligned behaviors [66] [ [5]]
While it may be trivial to quantify the error rate a transformer model using various performance benchmarks, working towards a truly mechanistic understanding of model hallucinations, errors, and inherent biases will be unavoidable as we entrust AI systems with increasingly sensitive and consequential real-world tasks.
For the goal of dissecting the units of meaning on which transformer models internally operate, sparse autoencoders (SAEs) have imposed themselves widely adopted tool in the mechanistic interpretability community [12] This class of autoencoders now provides the means for disentan- gling the opaque visceral workings of large language models (LLMs) into human understandable components. Such SAEs learn a mapping function from intermediate transformer model activations to sparse overcomplete representational space, which has interpretable basis by nature, that can be used to faithfully reconstruct the latent activations. Recent work has shown that human interpretable features of consequence can be identified in transformer models at scale, from toy models to commercial-scale LLMs, using SAEs [60].
Much of this interpretability research is rooted in the "linear representation hypothesis". It posits that semantic concepts are represented as single-ding dimensional linear directions in representation space derived from transformer layer activations [18]. While there is some evidence that certain features may be inherently nulti-dimensional in SAE embedding activations (19], there is a growing body of research underscoring the validity and utility of these linearly decomposable single-dimensional
features [2 51
21 18
However, several highly important open questions remain: to what extent do these learned representa- tions in transformer models reflect properties of the input data itself, as opposed to inherent biases or structural priors that emerged during model training? How do these internal representations behave when the transformer model is facing uncertainty due to confusing, ambiguous, or noisy inputs?
Using SAEs as tool to probe the emergent conceptual landscape of transformer internals, we show that pre-trained transformers impose coherent, steerable conceptual structure even on pure noise or randomly shuffled inputs, with the repertoire of activated concepts expanding as input structure degrades, across both images and text. Crucially, we show that patterns of concept activations elicited by the imput prompt reliably predict hallucinations in the transformer's generated output, providing a practical signal for anticipating unfaithful generations and directly linking high-level inferred concepts to model errors. Taken together, our exploratory and interventional experiments yield three key contributions:
1. First, common pre-trained transformer models exhibit strong form of ioput-insensitee inductive bias: these systems tend to impose semantic structure on inputs, tying them into learned conceptual webs, even if the model inputs ambiguous lack any coherent meaning.
2. Second, this skewing as information trickles through transformer processing layers exac- erbates as input uncertainty increases: the more randomness we experimentally introduce in the input observations, the more persistently the transformer model is drawn towards operating on semantic units that align with familiar internal model representations. We show a demonstrable expansion of semantic concepts that are triggered, localized especially to the middle layers of the transformer model, as the coherence of input information degrades.
3. Third, the constellation of concept activations a transformer model's intermediate process- ing representations can be used to reliably predict the tendency of hallucinated or unfaithful output. This insight suggests an automatically measurable link between spurious internal feature recruitment of the input and the fidelity of the output produced by these increasingly popular deep learning systems.
2 Preliminaries
In this section, we introduce our experimental framework to probe the emergence of semantic structure in transformer models, from layer to layer, by means SAE training and concept evaluation. To address research aims, we employ SAEs trained on transformer activations from pure noise

---

Gaussian Noise
Pre-Trained CLIP Vision Transformer
Layer
Layer
Layer 8
Layer
Layer
Training
Residual Stream Activations
Noise
Imagenet- 1k val
trained SAE
Interpretable Concept
Sparse Concept Space
Concept Evaluation
Workflow to examine hallucination risk in each layer of the transformer. To assess the extent to which transformer models infer meaningful concepts from semantically void inputs, we study the semanticity of concepts in SAEs trained on residual stream activations of 1.3 million Gaussian noise samples from pre-trained CLIP vision transformer at each layer by probing the concepts with natural images from the ImageNet- 1k validation set. Many concepts consistent, despite the SAEs only ever being exposed to pure noise residual stream activations during training. While this noise training setup is specific to first experiment, subsequent experiments adopt the same concept evaluation approach with alternative transformer models and input modalities. See Preliminaries and Appendix A for further experimental details.
Figure 1:
highly interpretable and
inputs, SAEs trained conventionally from scratch for smaller transformer models, and conventionally pre-trained SAEs for larger transformer models. To ensure that our results are as general and widely relevant as possible, we investigated latent activations from vision and language transformers of varying sizes (detailed in the following subsections and in Appendix A); thus covering several choices of data modality, model size, and model type.
2.1 Transformer models
For the vision transformer (ViT) 14 we study the CLIP-ViT-B/22 53 architecture: the base variant
of the vision transformer from OpenCLIP (TT), pre-trained on the LAION dataset [57] with 1.4 billion image-text pairs using contrastive supervision. In the case of language transformers (62). we trained on 300 billion tokens from the Pile [22]. We also use Gemma 2B-IT [59) for experiments where coherent text generation and instruction following is required. For all transformer models, the object of investigation the activations the residual stream: the hidden state of the deep neural network, updated by attention and multi-layer perceptron (MLP) blocks at each layer. The residual stream the core conduit for information flow and representation refinement
use Pythia- Pythia-160m-dee 160mededuped
to which layers read from and write to in the transformer architecture 15 [66]
2.2 SAF objective
For each SAE formulation, we work with SAES whose parameters are trained to produce the following approximation for the "residual stream activations", individual token embeddings x E Rdduedd for a particular layer:
deAR R Σ ((x)d:+bb
that is, each transformer layer's 's intermediate activations can be approximated as a (sparse) linear combination of unit direction vectors d, Bladd scaled by concept activation coefficients f((x) 2 0, with b e gddnddel being the bias term.

---

Practically speaking, this SAE is simple neural network with an encoder F RR - R projecting each transformer activation into a sparse concept space, typically of much larger dimension- ality than the transformer model activation space. For instance, the ViT embedding dimensionality dmodel = 768 with the corresponding SAE concept space dimension being dSAE = 49, 152. Conversely, the learned decoder D IR - Roodddel component of the SAE aims to reconstruct the original transformer residual stream activation solely from this pattern of sparse concept activations. This linear decoder D reconstructs the activation, X Df(x) + b, whose columns d, define the concept directions.
Each SAE in this work is trained end-to-end with the following loss function:
L
|X x + (x))1
where the left (main) term encourages faithful reconstruction (denoted by X) of the residual stream activations of the transformer layer at hand, while the right (penalty) term is an L1 regularization constraint encouraging parsimony in the concept space. The composite loss of these two terms motivates the SAE to learn pruned yet semantically relevant mapping from dense transformer model activations to sparse semantic concept space. Each SAE is trained independently on a single transformer layer's residual stream activations-e.g. for 12 layer transformer. we train 12 distinct SAES on the residual stream activations output from the corresponding layer.
2.3
SAE experimental setups
Noise trained SAEs For each layer's residual stream probe of the frozen vision transformer from the OpenCLIP ViT-B/32 model, we train a dedicated SAE on activations corresponding to 1.3 million samples of Gaussian noise. Each sample a 3x224x224 tensor sampled independently from standard Gaussians (0,, 1), clipped to 3o and min-max rescaled to [o, 1] before standard ViT normalization.
SAEs trained conventionally from scratch To examine overarching principles of the thus de- composed concept spaces of pre-trained transformer models, we also train SAEs on the residual stream activations from each layer of the transformer model under investigation using conventional image and text datasets. We train SAES on the intermediate activations of OpenCLIP ViT-B/32 from ImageNet-1K 1k [13]. and SAEs the intermediate activations of Pythia- 160m-deduped from Fine [50]. For each layer of these transformer models, we train a separate SAE instance; here and in all analogous experiments.
Pre-trained SAES For larger transformer models, we use publicly available SAEs pre-trained on conventional text datasets for comparability with prior interpretability work and to allow analysis on models whose outputs are more realistic, coherent, and representative of practical use cases. In particular, we implement pre-trained Gemma 2B SAEs [ across layers 1, 7, 11, 13, and 18.
2.4 Concept evaluation
To rule out the possibility that the features identified by the noise setting SAE are mere artifacts the inductive bias of the SAE, subject each feature in a randomly sampled subset of the SAE features to two complementary, model-agnostic metrics, designed for vision:
1. Semantic purity:
To quantify the unequivocality with which a particular feature fr is present, we collect the top-k (k 16) images that maximize the mean activation of f. across the sequence of image patches, obtain their CLIP text embeddings (label) and compute the mean cosine similarity (-1.0 to +1 +1.0) of the text embeddings. High semantic purity (close to +1.0) indicates that the images cluster around a single coherent semantic theme. A concept is "pure" if the top-k (k 16 in our case) maximally activating images for that concept have an average cosine similarity of greater than 0.75 across their semantic label embeddings.
2. Steerability: Steerability is the capability of flipping neutral image's 's semantic label that of the concept. by clamping the concept's activation to an artificially high value. To try to "hijack" a particular feature fi in the transformer model, we add the scaled SAE decoder direction d of that feature to the residual stream activations. The scalar a is chosen via grid search over a pre-defined range of values. We then run the transformer model on feature- naive "neutral" image inputs that originally do not activate ft. If this single vector

---

addition to a given transformer model layer makes the model's top-1 prediction switch to the feature's 's own majority label (the concept identified from its images that activate maximally) we consider Si "steerable". In our case, conservatively, if n = 32 out of 32 images from randomly sampled batch of neutral images flip to the concept's s label after clamping, then this concept is said to be "steerable".
2.5 Hallucination
Evaluation model hallucination is generally recognized be challenging (26) 41). As straight- forward empirical test, we prompt Gemma 2B-IT to summarize long-form text, and employ the purpose-bid built and widely used HHEM-2.1 model (3] to provide continuous hallucination score from 0.0 (no hallucination) to 1.0 (maximum hallucination) for the summary based on its faithfulness to the original text. To directly link ballucinations to our SAE-identified concepts, we fit a partial least squares (PLS) regression layer by layer to predict the hallucination score of the transformer output based on the concept activations the input prompt. PLS is linear model that enables interpretable links between concepts and hallucination behavior. To estimate the hallucination score, h, we fit:
h = X W)
where X max E R Roumpla XdsAE is calculated by taking the maximum concept activations across input tokens for source, W E RddAAE XTeamp is the learned projection matrix into the PL.S latent space, P E RdaAEX coump are the concept loadings, and Q IR. More details can be found in Appendix E.
Ncomp are the hallucination score loadings.
box. turfle
terrapin
schooner
achooner
baseball
Activalinn welg (o.a))
mud turtle
mad turtle
schooner
calamaran
bmsbbal
buseball
Figure 2: Semantic concepts are reliably invoked by vision transformer layer 9 activations from pure noise. Each panel shows the top 4 images among 50,000 candidate images from the ImageNet- Ik validation set that most activated a particular semantic concept. These semantic concepts are defined
SAE trained pure noise activations from the residual stream layer 9 of CLIP vision transformer. Above each image we report the corresponding semantic label for that image. Patch colors indicate the individual patch activation strengths within that image for given semantic concept (yellow more activation of the concept). See Appendix for additional concept examples.
3 Transformers represent semantic concepts for pure noise inputs
To investigate the emergence of semantic structure when pretrained transformers are presented with structurally incoherent inputs, train SAEs the residual stream activations from pure noise inputs to the CLiP vision transformer. the pre-trained vision transformer is able to interpret semantic structure from inputs that by construction contain semantic signal, the ensuing identified semantic concepts could be said to reveal the model's ingrained conceptual biases (Figure 1).
After training one SAE instance per layer on the residual stream activations for 1.3 million Gaussian noise samples, we probe a randomly sampled subset of the noise SAE-identified concepts using 50,000 example images from the ImageNet-1k validation set. To evaluate the interpretability of these concepts, we obtain the top-k (k- activation across all patches) for each concept (Figure 2). Additional concept examples from
in this case) maximally activating images (mean concept
5

---

noise trained SAES are presented in Appendix B. We then computed the semantic purity of the concept (cf. Preliminaries) as a proxy for its interpretability. computed by the cosine similarity the semantic CLIP labels from the top 16 maximally activating evaluation images.
We define feature interpretable if its semantic purity is least 0.75. Using this strict threshold, we observe many high purity features, each corresponding to a coherent high level concept, despite the SAE being trained solely on Gaussian noise activations and never seeing naturalistic image activations during training. We also observe that subset of these highly interpretable concepts are steerable (ef. Preliminaries): injecting the concept-related activation vector into the residual stream as the transformer model is processing neutral image can shift the model's 's semantic label prediction toward the label associated with the concept of interest (Figure 3).
Feature Type
30
Percentage of Unique Concepts
Steerable and Interpretable Interpretable
20
CLIP Vision Transformer Layer Number
Figure 3: Transformers fed with noise inputs lead to neuron activations with detectable and controllable semantic structure in many layers. We measure the interpretability of concept by the semantic similarity of the labels of the top 16 images that maximally activate that concept. As more stringent test, we also measure a concept's 's steerability by the ability of that concept to causally induce its own class label when added to the residual stream of neutral input images. We find that a very large portion of our noise-derived semantic concepts are highly interpretable, and a non-negligible number of these concepts are even steerable, particularly the early and middle layers. We report the percentage of unique concepts across our noise-trained SAE meeting the aforementioned interpretability and stecrability thresholds.
As a further test of concept robustness and layer-wise structuring, we train different noise SAE instances with varying seeds and hyperparameters on the same transformer layer activations. A consistent core of robust concepts (47) emerges across comparable layers. Comparing two sets of noise-setting SAEs reveals distinctive three-phase pattern concept overlap, offering insight into how the vision transformer structures information (Appendix C. Figure 16). The first three layers share 35% of their concepts, because the residual stream here still reflects the statistical structure of the raw input, and the space of the viable concepts is highly constrained given the minimal inductive bias at the onset of processing. Conceptual overlap drops sharply 14% layers 5-6, marking a stochastic exploration phase within the vision transformer's ' latent space, exploring much broader semantic hypothesis space reducing the probability that randomly sampled features from both sets of SAES will have a large overlap. In deeper layers of the vision transformer model, overlap rebounds to 25% onto a more stable set of high-level features. See Appendix C for further details these overlapping concepts. In effect, the model settles on the most explanatory subset of concepts for the task, collapsing the expanded mid layer space back into a shared semantic scaffold. Taken together, this three phase pattern indicates that the vision transformer incrementally restructures noise-based activations into coherent task-relevant concepts, and that these emergent representations are sufficiently robust to be discovered by noise-setting SAEs trained with different random initializations.
Grounding our evaluation with the ImageNet- 1k validation set gives us strict, reproducible baseline. Since the concept space limited by the samples in validation set, any feature aligned with concepts beyond ImageNet's dictionary is treated as unclassified, making our interpretability and steerability counts understated. more diverse evaluation set, with images supporting wider array of ground-trun truth concepts, would likely show increase in both of these metrics. Regardless, we have

---

demonstrated that transformer models ascribe distinct, coherent, and causally actionable concepts to inputs that, by construction, have no semantic content.
SAEAL, La (Unstructure Normal (mean 1 std)
sto)
Patch Shuffle
Shuffle
Change in average activve SAE features
28x28 Patch
Change average active SAC features
1-gram
More intuced
56x56 Patch
2-gram
nose
112x112 Patch
Incuced
6-gram
60
10-gram
20-
30 30-gram
20-
SAE instrrccured
40
10
20-
Vision Transformer Layer Number
Pythia-160mm dedu-ddduped Transformer Layer Number
(a)
(b)
Figure 4: Increasing uncertainty in vision text inputs elicits more semantic structure in mid- layers of transformers. The average number of SAE concepts identified (LO) increases dramatically with increasing input perturbation. We report the average change in LO from baseline, corresponding to the number of SAE concepts with non-zero activations, across (a) patch-shuffled image activations and (b) n-gram-shuffles text activations for each transformer layer. Error bars show one standard deviation. Smaller patches and lower n-gram count induce greater input uncertainty for images and text, respectively. For both modalities, the LO difference between natural inputs and perturbed inputs peaks in the middle layers, and increases with increasing levels of deliberate scrambling of transformer input information.
4 More unstructured transformer inputs invoke more semantic concepts
To get better grip on how the transformer models progressively mold the conceptual structure embedded the input as information flows through the processing layers, train one conventional SAE instance on each transformer layer independently. These per-layer SAEs are trained on trans- former activations from natural images/text, not noise. We provide these normal input activations as a baseline, and then compare the differences in the SAE concept space for varying levels of experimentally induced ambiguity the input examples. transformer, we randomly shuffle small patches to break the spatial relationship between adjacent parts of the image natural images (28x28, 56x56, and 112x pixel patches shuffled per image), with smaller patches leading more ambiguous inputs. To this end, for text inputs to the language transformer, we randomly shuffle sets of consecutive words in varying n- n-gram sizes (single, 2, 6, 10, and 30 word sections), with lower introducing higher uncertainty in the input space. The transformer layer's ensuing residual stream activations corresponding to these inputs are then fed into the appropriate SAEs for each layer.
the case image inputs to the vision
useful macro-level metric for overall SAE concept discovery is the SAE LO: the average number of non-zero concepts across the input sequence activated through transformer processing over the entire dataset. the case our image SAEs, remarkably, we notice a steep increase the average number of concepts activated for randomly shuffled inputs compared to the baseline, peaking at layer with 38 additional concepts activated on average (Figure 4a) for the smallest 28x28 pixel patch size. This pattern holds in the language domain well. The number of concepts identified per input above the baseline peaks at layer 9 with 81 extra concepts triggered, on average (Figure 4b) for the single word shuffled text. Furthermore, the magnitude of this effect. Input activations with finer- aeeiner shuffling (e.g. single word level or smaller image patches) result in significantly higher LO count values than input activations shuffled in larger chunks (e.g. 30-word sections or large image patches). See Appendix D for full tables of layer-wise results (Tables 6-7).
observe that the degree of input perturbation strongly influences
These results, converging across different data modalities, are suggestive of the middle layers of pre-trained transformer models playing more influential role in mediating the model's internal
7

---

conceptual instantiation in response to degraded input structure. One might expect transformer models intuit fewer concepts in shuffled inputs; instead, we find that ambiguity drives an expansion in concept usage, revealing unanticipated property of how these transformer models organize and impose structure on input processing cascades.
5 Input-related internal concept activations predict output hallucinations
have seen how pre-trained transformer models, when confronted with noisy, uncertain, and ambiguous inputs, appear to traverse the landscape of possible concepts more widely, attempting to impose structure that may not actually be present. Next, we aim to directly link this transformer model-imposee structure in the concept space of the imputs hallucination propensity with respect to the model-geneee -oneel-geeertte outputs. To this end, we first instructed Gemma 2B-IT to produce generated summary for 1,006 articles from the Vectara hallucination leaderboard (24). employing the purpose- built HHEM-2.1 rating model to give each source/summary pair a real-valued hallucination score, ranging from 0.0 (no hallucination) residual stream activations the original article text were projected into the sparse concept space formed by the pre-trained Gemma 2B SAEs. With these concept vectors in hand, fit 4- component PLS on the maximum concept activations across tokens in a sample, attempting to predict the hallucination score of the Gemma 2B-IT-generated text summary.
1.0 (most hallucinated). Simultaneously, the transformer's
A 0.19,
Hallucination Prediction (Rt)
80
Suppression
0:22
0.23
Number of Samples
Before
0.15
40
0.10
0.05
20
1
11 2B-IT Layer Number
13
18
0.0
0.2
0.4
0.6
0.8
1.0
Gemma 2B-IT Hallucination Score
(a)
(b)
Figure 5: Transformer layer activations can be used to directly predict risk of hallucinated model output. (a) Hallucination score prediction for the task of faithful summarization of 1,006 source articles. We compare Gemma 2B-IT generated summaries against the ground truth source articles. We use the sparse SAE concept activations, derived from Gemma 2B-IT residual stream activations, as input to PLS regression model, predicting the hallucination score for each example. We report 10-fold cross- validdted coefficient determination (R2) on unseen examples, with error bars showing one standard deviation.
2B-IT-generated
(b) Suppressing the top 10 SAE concepts in Layer 11 residual stream, identified by the PLS model to be the primary drivers of hallucination, significantly reduces mean hallucination scores across the top quartile most hallucinated examples (n 252). We show histogram hallucination scores before (grey) and after (blue) suppression: many examples report significant reductions in hallucination, with a mean score drop of 0.19 in this subset (dashed lines).
Indeed, we find that hallucinations present in the Gemma 2B-IT-generated summaries can be robustly predicted, unseen text examples, solely from the sparse concept vectors derived from the layer activations of the source prompt using the SAEs. The most accurate predictions are observed for the layer 11 residual stream SAE concepts with coefficient of determination of 0.271 0.010 (out of sample, 10-fold cross validation) for the continuous hallucination scores. For binary classification predicting above versus below the median hallucination score, we reach mean accuracy of 73.0% + 5.3% across 10 cross validation folds. Transformer layer 18 SAE source concepts do not allow us to predict the continuous summary hallucination scores above chance level (Figure 5a). Appendix E we also show that our method outperforms existing hallucination detection methods several additional short form generation and multiple choice hallucination benchmark datasets, despite the fact that only take into account concepts in the input prompt (Tables 9-12).

---

Based on this independent mode of investigation. our results reinforce our previous findings (cf. above) that it is especially the intermediate transformer model layers where concept exploration and risk of "conceptual wandering" occur. According to the present analyses, later transformer layers do not appear to participate in this process to a large extent.
We finally explore the practical implications of these findings, Since PLS is a linear model, we can easily reverse-engineer specific key concepts in each transformer layer that have the most influence on the hallucination score predictions. This is because each PLS input dimension corresponds to a concepts in layer 11 by variance importance in projection (VIP). With these 10 concepts in hand, we simply suppress them in the residual stream of layer 1 for the imput prompt by setting their SAE concept ttttuiiinatiouuuuppessttt
distinct SAE-derived concept activation. We select the top 10 key iatluciinattonassociatee
activations to O, with the SAE decoder yielding These petlliiinnatoonupppssseee activations are then swapped in instead of the actual vector of residual stream activations in Gemma 2B-ITT's forward pass. Intriguingly, for the top quartile most hallucinated examples in our dataset (252 samples in total), suppressing just these 10 features (out of total of 16,384) leads to decrease in the mean hallucination score of 0.19 for this subset (Figure 5b), from 0.91 to 0.72 after suppression. Further details are provided in Appendix E.
residual stream activations.
This constellation of results shows a clear link between concept space activations of the input prompt and the tendency of the transformer model to hallucinate its generated content. More than this, we present a primitive mechanism to tamp down this tendency to hallucinate.
6 Related Work
Hallucination and perturbed inputs The occurrence of hallucinations in generated content is
well-documented issue, persisting across language models regardless of scale or capability (26) [65] [1]. These hallucinations refer to model-generated outputs that are syntactically plausible but factually incorrect or ungrounded to the input prompt. There have been many attempts to benchmark this phenomenon across various tasks (20 (2)6), There have been relatively fewer attempts to explain the mechanisms within transformers that underlic these behaviors. (64) attempts to explain (and control) hallucinations based on syntactic errors in particular components of the transformer architecture such specific attention heads or MLPs. [27] looks hallucination through the lens of output token dynamics. (32) studies model self-evaluation, and (33) rigorously shows that well-calibrated models will always hallucinate on certain "arbitrary" facts at a baseline rate. These approaches largely employ custom-made datasets and work on the level of single token probabilities and token-wise syntactic dynamics. Our approach links the phenomenon of hallucination directly to interpretable, high-level concepts, and as far as we know is the first study to present a unifying mechanism across both images and text. Our approach also bears some similarities a large body of research on neural network input perturbation and adversarial examples, where small, often imperceptible changes the input can lead erroncous model outputs (58) 19)(9144 (45 (28 [23). In
our work, we introduce highly perturbed inputs as an experimental tool to test the interaction of the actual semantic content of the inputs and the information imposed on the inputs by the transformer.
Sparse autoencoders Early word embedding approaches, such as word2vec 44 and Glo (52),
demonstrated the power of linearly composable embedding spaces for capturing high-level semantic relationships in natural language (43). More recently, this result has been formalized through the
linear representation hypothesis 49 which suggests that many meaningful features of input data
can be disentangled and inspected via linear transformations (18 6 17). Leveraging this insight, sparse autoencoders (SAEs) have proven be particularly effective tools for breaking down dense high dimensional transformer representations into an overcomplete basis of linear pieces that can be independently studied [12 7 40). SAEs have surfaced coherent and even manipulable concepts from transformer models of various sizes 600 334: recent work has extended these techniques to vision transformers with similar success [30 There has also been much focus on improving the intelligibility and relevance of identffd features through various alternative training regimes and architectures (21) S Our experiments leverage SAE-disentangled concepts to localize transformer model behaviors such as hallucinations and conceptual drift, with results that rely solely on the presence of hegh- concepts rather than specific SAEs, datasets, or even data modalities. Our approaches are likely to gain strength interpretability techniques advance.

---

7 Conclusion
We here introduce an objective framework for detecting and profiling structural biases and failure modes in common pre-trained transformer models. Our approach scales organically to transformer models of any size. Crucially, it allows for Cataset-independent assessments eliminating reliance on arbitrary benchmark data subsets and subjective observer-depende judgements such as highlighting a handful of specific semantic concepts or zooming on small functional circuits billion- pearame-e transformer models. By systematic probing of internal representations as they trickle through the transformer architecture following simple random noise and other controlled input perturbations, we show how compute continuous degree of hallucination risk, even layer-by-layee basis. We surface presupposed semantic concepts interest, study the trajectory of their propagation throughout the deep neural network, and examine the nature of how experimentally perturbed inputs become internally represented through fully quantitative, automatically computed measures. In doing so, we reveal important ways in which transformer models are prone to "conceptual wandering", invoking rich semantic structure in inputs with pure noise or degraded semantic information content. Such frameworks may shift interpretability efforts to see deeper into transformer models from qualitative judgements and hand-picked examples to pipelinable methodology that generalizes across multiple data modalities, expansive model architectures, and experimental setups. Ultimately, our work lays cornerstone for both real-time monitoring and principled intervention of transformer model behavior, supporting the safe and responsible deployment of emergent AI abilities the wider society.
While our experiments span a broad range of model sizes and modalities, remains to be seen how exactly these findings may manifest in the largest frontier models the order of 10-100+ billion parameters. Nonetheless, the consistency of observed behaviors across architectures and scales in this work suggests strong potential for generalization. As well, interesting future work would be to explore the interplay of hallucination with increased sequence length and sampling temperature. We would expect increased hallucination and concept activation in these settings. We do not claim that SAFs are perfect oracles for interpretability; rather, our study builds upon the observation that SAEs have been shown repeatedly (in this work and many other works) to extract many useful semantic concepts from transformer activations. Indeed, we have shown here that these concepts can be used directly predict relevant behavior of monee--generated content.
Limitations
10

---

References
[1] Jean-Baptiste Alayrac et al. Flamingo: a Visual Language Model for Few-Shot Learning.
arXiv:2204.1410 [cs]. Nov. 2022. DOL: 10. 4844085555/550 2204 14198
[2] Yejin Bang et al. HalluLens: LLM Hallucination Benchmark, arXiv.2504. [es]. Apr.
2025. DOI: 10. 101:|10,40 48w50//rXiv..... 2504. 17550
[3] Forrest Bao et al. WHHEM--./ 2024. DOT: 00..5797/hff3222
[4] Stella Biderman et al. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. arXiv:2304.00 333 [es]. May 2023. DOT: 10. 48550/ arXiv. 2304.01373.
[5]
Joseph Bloom. "Open source sparse autoencoders for all residual stream layers of gpt2 small". In: Al alignment forum. 2024, p. 24.
[6] Tolga Bolukbasi et al. Man is to Computer Programmer as Woman is to Homemaker? Debi- asing Word Embeddings. arXiv: (1600700652) [cs]. July 2016. DO1: 480004855/// arXiv 1607 06520
[7] Trenton Bricken et al. "Towards monosemanticity: Decomposing language models with dictio- nary learning". In: Transformer Circuits Thread (2023).
[8] Bart Bussmann et Learning Multi-Level Features with Matryoshka Sparse Autoencoders.
arXiv:2503.1754) [cs]. Mar. 2025. DO1: 10. 100850/arXiv...m 2503. 17547
[9] Nicholas Carlini and David Wagner. Towards Evaluating the Robustesss of Neural Networks.
arXiv:1608.00 006440 [cs]. Mar. 2017. DOT: 10. 48550/ arviv. 1608. 04644
[10] Chao Chen et al. INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection.
arXiv:2402.03744 [cs]. Oct. 2024. DOI: 10. 48550/ arXiv 2402 03744
[11] Mchdi Cherti et al. "Reproducible scaling laws for contrastive language-image learning". In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). arXiv:2212.07143 [cs]. June 2023, pp. 2818-2829. DO1: 10. /119//CPR522299..... 2023. 00276
[12] Hoagy Cunningham et al. Sparse Autoencoders Find Highly Interpretable Features in Lan- guage Models. arXiv:2309.088. 08660 [es]. Oct. 2023. DOT: 10. 48550/8500/00 arXiv 2309. 08600
[13] Jia Deng et al. "ImageNet: A large-scale hierarchical image database" In: 2009 IEEE Confer- ence on Computer Vision and Pattern Recognition. ISSN: 1063-6919, June 2009, pp. 248 255.
N01:|00.1199/00
2009 5206848
[14] Alexey yiy et al. An Image is Worth 16x/6 Words: Transformers for Image Recognition at Scale. arXiv:2010.1 1929 [cs]. June 2021. DOE: 10. 48550/ arriv. 2010. 11929
[15] Nelson Elhage, Robert Lasenby. and Christopher Olah. "Privileged bases in the transformer residual stream". In: Transformer Circuits Thread (2023), 24.
[16] Nelson Elhage et al. "A mathematical framework for transformer circuits", In: Transformer Circuits Thread (2021).
[17] Nelson Elhage et al. Toy Models of Superposition. arXiv:2209.1063 [es]. Sept. 2022. DOI:
10. 48550/ arxiv. 2209 10652
[18] Nelson Elhage et al. "Toy models of superposition", In: Transformer Circuits Thread (2022).
[19] Joshua Engels et al. Not All Language Model Features Are Dne-Dimensionally Linear. arXiv:2405.14866 [cs]. Feb. 2025. DO1: 10. 48550/ arxiv. w400..2405.. 14860
[20] Sebastian Farquhar et al. "Detecting hallucinations in large language models using semantic entropy". en. In: Nature 630.800 8017 (June 2024). Publisher: Nature Publishing Group, pp. 625-
630. ISSN: 1476-4687. DO1: 10. 1038/ 02441668-00407221111
[21] Leo Gao et al. Scaling and evaluating sparse autoencoders. arXiv:2406.0444 04093 [cs]. June 2024.
2ee/1004455/arra arviv. 2406 .40933
[22] Leo Gao et al. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.0002 [cs]. Dec. 2020. DO1:|10. 18550//arivv
[23] Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. arXiv: 12261 [cs]. Mar. 2019, DOI: 10. 48550/ /arviv
1903. 12261]
[24] Simon Hughes, Minseok Bae, and Miaoran Li. "Vectara hallucination leaderboard". In: Vectara Inc 466 (2023).
[25) hugofry. "Towards Multimodal Interpretability: Learning Sparse Interpretable Features in Vision Transformers". en. In: (Apr. 2024).

---

[26] Ziwei Ji et al. "Survey of Hallucination in Natural Language Generation". In: ACM Computing Surveys 55.12 (Dec. 2023). arXiv:2202.03629 [cs], pp. 1-38. ISSN: 0360-0300, 1557-7341.
DOT: 10.
[27] Che Jiang al. On Large Language Models' Hallucination with Regard Known Facts. arXiv:2403.2000 20009 [es]. Oct. 2024. DOE: [0. 48550/ arXiv 20009
[28] Di Jin et al. Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text
Classification and Entailment. arXiv: 11932 [cs]. Apr. 2020. DO1: 10. 48550/ arXiv
1907.11933
[29] Sonia Joseph et al. Prisma: An Open Source Toolkit for Mechanistic Interpretability in Vision and Video. arXiv:2504.199 19475 [cs]. Apr. 2025. DOI: 10. 40.45555aariix /rr60//axiv... http:/aaaxvv........12 2500..997977
[30] Sonia Joseph et al. Steering CLIP's vision transformer with sparse autoencoders.
PXXi:2550.088799 [cs] version: 1. Apr. 2025. DO1: 48540/555Xiii 2504. 08729
[31]
Mandar Joshi al. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv: 1
[es]. May 2017. DOT:
1705
03551
[32] Saurav Kadavath et al. Language Models (Mostly) Know What They Know arwiv:2207.02222
[es]. Nov. 2022. DOI: 10. 48550/ arxiv...ii. 200.202221
[33]
Adam Tauman Kalai and Santosh S. Vempala. Calibrated Language Models Must Hallucinate.
1xiv:23111.14444 [es]. Mar. 2024. DO1: 1/8550/arXin 2311 14648
[34] Tom Lieberum et al. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2. arXiv:2408..0. 147 [cs]. Aug. 2024. DO1: 2.00.4850/aaaa arxiv. 0.408.001477
[35] Chin- Yew Lin. "ROUGE: A Package for Automatic Evaluation of Summaries". In: Text Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, July 2004, pp. 74-81.
[36] Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109 10w02119990799888 [cs]. May 2022. DOI: 10. 48550/ arXiv 2109. 07958
[37] Zi Lin, Jeremiah Zhe Liu, and Jingbo Shang. "Towards Collaborative Neural-Symbolle Graph Semantic Parsing via Uncertainty". en. In: Findings of the Association for Computational Linguistics: ACI. 2022. Dublin, Ireland: Association for Computational Linguistics, 2022, pp. 4160-4173 DOT: 10. w1w111833312200 findings- 328
[38] Weitang Liu et al. Energy-baseed Out- distribbutiiioo Detection. arXiv:2010.037.9 [cs]. Apr.
2021. DOI: 10. 48550/ arxiv. 2010. 03759
[39] Andrey Malinin and Mark Gales, Uncertainty Estimation in Autoregressive Structured Predic-
tion. arXiv:2002.00 007600 [stat]. Feb. 2021. DOL: [0. A8550/arXXia 200.2.002.0000
[40] Samuel Marks et al. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. arXiv:2403. [es]. Mar. 2025. DOI: 10. 139950//rXX0
2403. 19647
[41] Joshua Maynez et al. "On Faithfulness and Factuality in Abstractive Summarization". In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Ed. by Dan Jurafsky et al. Online: Association for Computational Linguistics, July 2020,
pp. 1906-1919, DOT: 10. 18653/v1/2022 acl- -main. 173,
[42] Tomas Mikolov, ten-tau Yih, and Geoffrey Zweig. "Linguistic Regularities in Continuous Space Word Representations". In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Ed. by Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff. Atlanta, Georgia: Association for Computational Linguistics, June 2013, pp. 746-751.
[43] Tomas Mikolov et al. "Distributed Representations of Words and Phrases and their Composi- tionality". In: Advances in Neural Information Processing Systems. Vol. 26. Curran Associates, Inc., 2013.
[44] Tomas Mikolov al. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.374 [cs]. Sept. 2013. DO1: 485.0/888////8 arviv. 1301.
[45] Seyed-Mohsen Moosavi-Dezfor
et al. Universal adversarial perturbations. perurasioa arXiv:1610.08401 [cs]. Mar. 2017. DOL: 10. 48050/85///80 arXiv. 1610. 1610.08401
[46] Long Ouyang et al. Training language models to follow instructions with human feedback.
15Xiv:2203.02155 [cs]. Mar. 2022. DO1: 10. 48550/ www...i2w........aaoa 02155
12

---

[47] Isabel Papadimitriou et al. Interpreting the Linear Structure of Vision-language Model Embed-
ding Spaces. arXiv:2504. 1665 [cs]. Apr. 2025. DO1: 10. 48550/ arwiv. 2504. 11695
[48] Kishore Papineni et al. "Bleu: a Method for Automatic Evaluation of Machine Translation". In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Ed. by Pierre Isabelle, Eugene Charnaak, and Dekang Lin. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics, July 2002, pp- 311-318. DOE: 10. 10.3115/1073083
1073135
[49] Kiho Park, Yo Joong Choe, and Victor Veitch. The Linear Representation Hypothesis and the Geometry of Large Language Models. arXiv:231 1.03655 [es]. July 2024. DOI: 10. 48550/
arXiv 2311 03658)
[50]
Guilherme Penedo et al. The WineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv:2406. 17557 [cs]. Oct. 2024. |0 48000.4855//880 arrivv/arxvv.// 2406. 17557
[51] Jeffrey Pennington, Richard Socher, and Christopher Manning. "GloVe: Global Vectors for Word Representation" In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Ed. by Alessandro Moschitti, Bo Pang, and Walter Daelemans. Doha, Qatar: Association for Computational Linguistics, Oct, 2014. pp. 1532-
1543, DOI: 11....111/////144-11 1162.
[52] Jeffrey Pennington, Richard Socher, and Christopher Manning. "GloVe: Global Vectors for Word Representation" In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Ed. by Alessandro Moschitti, Bo Pang, and Walter Daclemans. Doha. Qatar: Association for Computational Linguistics, Oct. 2014, pp. 1532- 1543. DO1: 10. 30 21115/111D4-11155
[53] Alec Radford et al. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.0m 000000 [cs]. Feb. 2021. DOI: 48550/ 48550/arXiv arxiv. 2103. 00020
[54] Senthooran Rajamanoharan et al. Improving Dictionary Learning with Gated Sparse Autoen- coders. arXiv:2404. 16011 [cs]. Apr. 2024. DOI: 10. 48550/855/000 aa100//rrivv... 2404. 16014
[55] Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A Conversational Question Answering Challenge.
[es]. Mar. 2019, DOI: 10. . 48550/ / arXiv. 1808
07042
[56] Jie Ren et al. cittof-DDitribbttin Detection and Selective Generation for Conditional Language Models. arXiv:2209. 15558 [es]. Mar. 2023. DO1: 48550/ 23650/arXivv 2209. 15558
[57] Christoph Schuhmann et al. "LAION-5B: An open large-scale dataset for training next genera- tion image-text models"" en. In: Advances in Neural Information Processing Systems 35 (Dec. 2022). pp. 25278-25294.
[58] Christian Szegedy et al. Intriguing properties of neural networks. arXiv: 1312.6199 [cs]. Feb. 2014. DO1: 000::10.44 10. 488444855//311 /0.50///rxi..... 1312.6199
[59] Gemma Team et al. Gemma: Open Models Based on Gemini Research and Technology.
arXiv:2403.08295 [cs]. Apr. 2024. DOI: 10. 48550/ aatoor/r/vvv...///ii 2403. 08295
[60] Adly Templeton al. "Scaling monosemanticity: Extracting interpretable features from claude
3 sonnet". In: Transformer Circuits Thread (2024).
[61] S. M. Towhidul Islam Tonmoy et al. A Comprehensive Survey of Hallucination Mitigation Techniques Large Language Models. arXiv:2401.01313 [es]. Jan. 2024. DO1: 10. 48550/ arxiv. 2401. 01313
[62] Ashish Vaswani et al. Attention Is All You Need. arXiv: [cs]. Aug. 2023. DO1:
10. 48550/arXiv 1706. 03762
[63] Eric Wallace et al. Universal Adversarial Triggers for Attacking and Analyzing NLP.
arXiv:1908.00 125 [cs]. Jan. 2021. DOL: 10. 48550/ 135500/aXii... 1908 07125
[64] Lei Yu et al. "Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations". In: Findings of the Association for Computational Linguistics: EMNLP 2024. Ed. by Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen. Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, pp. 7943-7956. DOI: 10. 18.0.86556 18653/v1/ 2024. fin.ings------i emnlp. 466
[65] Yue Zhang et al. Siren's 's Song in the Al Ocean: A Survey on Hallucination in Large Language Models, arXiv:2309 .01119 [cs]. Sept. 2023. DOI: 10. 48550/ arxiv. 012119
13

---

A SAE hyperparameters and training metrics
In Tables 1-3, report the hyperparmeters used for training SAEs from scratch in this work. In Tables 4-5, we report evaluation metrics for SAEs that we train from scratch in this work.
Table 1: Hyperparameters for OpenCLIP ViT-B/32 SAEs trained on residual stream activations of Gaussian noise inputs. Note that the number of sampled Gaussian noise inputs matches the size of the ImageNet- Ik training set, to ensure a fair comparison. Training time was approximately 35 minutes on single NVIDIA L40S GPU.
Layer
learning rate
batch size
L1 coef.
dSAE
model
# training images
1
3e-3
4096
2c-6
49,152
768
1.3M
2
le-3
4096
4c-6
49,152
768
1.3M
3
7e-3
4096
2e-6
49,152
768
1.3M
4
2e-3
4096
2e-6
49,152
768
1.3M
5
9e-4
4096
2e-6
49,152
768
1.3M
6
3e-3
4096
le-6
49,152
768
1.3M
7
7e-3
4096
le-6
49,152
768
1.3M
8
6e-3
4096
le-6
49,152
768
1.3M
9
le-2
4096
3e-6
49,152
768
1.3M
10
3e-4
4096
le-5
49,152
768
1.3M
11
le-4
4096
le-6
49,152
768
1.3M
12
le-4
4096
le-6
49,152
768
1.3M
Table 2: Hyperparameters for OpenCLIP ViT-B/32 SAEs trained residual stream activations of natural ImageNet-1k 1k images. Training time was approximately 45 minutes on a single NVIDIA L40S GPU.
Layer
learning rate
batch size
L coef.
dSAE
dmudel
training images
1
6e-04
4096
le-1
49,152
768
1.3M
2
1e-02
4096
3e-1
49,152
768
1.3M
3
2e-03
4096
4e-1
49,152
768
1.3M
4
le-04
4096
2e-8
49,152
768
1.3M
5 2e-04
4096
3e-8
49,152
768
1.3M
6
3e-04
4096
le-8
49,152
768
1.3M
7
2e-03
4096
le-8
49,152
768
1.3M
8
7e-04
4096
le-8
49,152
768
1.3M
9
le-02
4096
3e-8
49,152
768
1.3M
10
1e-02
4096
9e-8
49,152
768
1.3M
11
1e-02
4096
4e-10
49,152
768
1.3M
12
1e-02
4096
3e-7
49,152
768
1.3M
Table Hyperparameters for Pythia-160m-deeee 160m-deduped SAEs trained on residual stream activations of natural FineWeb-Edu text. Training time was approximately 4 hours on single NVIDIA L40S GPU.
Layer
learning rate
batch size
L coef.
dSAF
d dmodel
training tokens
11
5e-5
2048
5
24,576
768
204.8M
12
5e-5
2048
2
24.576
768
204.8M 8M
14

---

Table 4: Evaluation metrics for OpenCLIP ViT-B/32 SAEs trained on residual stream activations of natural ImageNet- Ik images.
Layer
L0
explained variance
recon. cos sim.
% alive features
989.19
0.99
0,99
100.00
2
757.83
0.99
0,99
45.39
1007.89
0.99
0.99
97.93
4
935.06
0.99
0,99
100.00
965.15
0.99
0,99
100.00
966.38
0.99
0,99
100.00
7
1006.62
0.99
0.99
99.97
8
984.19
0.99
0.99
100.00
965.12
0.99
1.00
92.37
10
854.92
0.99
1.00
85.43
11 1141.99
0.99
1.00
100.00
12
829.09
0.99
1.00
55.71
Table 5: Evaluation metries for Pythia-160m-deduped SAEs trained residual stream activations of natural Feb-Wee text.
Layer
LO
explained variance
recon. cos sim.
% alive features
1
29.50
0.92
0.97
100.00
2
16.18
0.85
0.95
100,00
3
13.25
0.81
0.93
100.00
4
21.41
0.84
0.91
98.84
5
22.31
0.77
0.90
97.35
6
31.98
0.72
0.92
89.58
7
45,89
0.79
0.92
87.28
8
65.53
0.82
0.93
87.80
9
67.55
0.81
0.93
80.48
10
49.12
0.75
0.92
84.20
11 31.57
0.70
0.92
93.95
12
11.03
0.71
0.98
22.34
B
Additional noise trained SAE concept examples
We present additional examples of concepts identified by the SAEs trained on Gaussian noise activations from the residual stream of the OpenCLIP VIT-B/32, at various layers (Figures 6-15). Evaluation is carried out using 50,000 images from the ImageNet-Ie Ik validation set. Early layers largely identify concepts associated with low-level visual features and repeating patterns. Later layers identify very coherent and high-level concepts. These figures extend the results presented in Figure 2.
15

---

Concepts from Layer 3 Noise SAE
chalv C.205
pof 0.24
bat, 9.203
Sodlingon 02000
Uilinag 9,202
daleag 8250
dbling 0222
2.13
4drin
dong
IEE-resiti, mergare 6407
Scatcher 6.35
Figure 6: Maximally activating images for three features from an SAE trained on Layer activations of ViT with Gaussian noise input. These features, also representative of early transformer layers (Layers and 2), include: (i) a positional feature that consistently activates for the final patch token across rows, (ii) texture-sensitive feature that activates for highly repetitive, noise-like patterns resembling dishrags, and (iii) a feature capturing water-like backgrounds with birds in the foreground.
16

---

Concepts from Layer 4 Noise SAE
Dissoe OUTAIN 0:725
Shanke curtin EIP2
Thotter OUrtIn 6.799
m0002400, not 685
665
windaw Scree
Tnlpbyer
LES
0.542
241
Impals Dd11
Figure 7: Maximally activating images for three features from SAE trained Layer activations of a VIT with Gaussian noise input. These features remain grounded low-level visual patterns but display more variation compared to earlier layers, without forming coherent semantic concepts: i) a feature that activates for vertically folded fabrics such as theater curtains and mosquito nets, likely driven by repetitive linear structures; ii) feature that responds mesh-covered textures, such as window shades with overlaid nets, producing a noisy visual appearance; iii) a feature that activates for grassy textures, possibly due to fine-grained spatial repetition.
17

---

Concepts from Layer Noise SAE
sanaa 2.33
1600
net cb
Excod
DeMci gen
Figure 8: Maximally activating images for three features from an SAE trained on Layer activations of ViT with Gaussian noise input. At this layer, semantic structure begins emerge, marking a departure from purely textural or position-bee based features: i) a feature that activates for natural scenes containing clouds and water bodies, indicating sensitivity environmental layouts; ii) a feature that responds to food items with soft, clumped textures- as mashed potatoes, ice cream, and cauliflower- highhlgh texture-based abstraction; iii) feature that activates for the presence of printed text, capturing more symbolic and structured content.
18

---

Concepts from Layer 6 Noise SAE
Shiland Slxsplog 20053
22 SET 231E
wiRki wot 2.355
Daaty
gacamole
chetsdurger
TOX 5.00
st
ShI
nugratk SETIMRA
Carnival
sumer erstore
EAPRERCEEN
Figure 9: Maximally activating images for three features from an SAE trained on Layer activations of a VIT with Gaussian noise input. At this layer, have features with clear semantics: feature that activates for furry dogs; ii) feature that activates for food items such as cheeseburgers; iii) feature that activates for the watermarks on images.
19

---

Concepts from Layer 7 Noise SAE
TOR DISDUII
Figure 10: Maximally activating images for three features from an SAE trained on Layer activations of VIT with Gaussian noise input. The features this depth increasingly correspond to distinct semantic categories and complex scene elements: i) a feature that activates for dogs with upright posture and distinct facial structure, reflecting refined animal categorization; ii) feature that responds to natural landscapes featuring prominent vertical structures such as mountain peaks, suggesting emerging scene-level abstraction; iii) feature that activates for human crowds densely populated scenes, particularly those with small, background scale human figures.
20

---

Concepts from Layer 8 Noise SAE
sertpedee
80
30
1-9-
9
30
20
1
110
MPH
4
850
035555
ocomber 1:252
atonetor 1.59
odenater
45
60
60
50
80
40
90
25
65
30
19 9
100
20
86
0
0.4
MPH
120
P RN D2N
Figure 11: Maximally activating images for three features from an SAE trained on Layer activations of a VIT with Gaussian noise input. At this stage, features begin to consolidate around coherent semantic clusters that span both foreground entities and background textures: i) feature that activates for human fingers, in the context them being used hold some organic objects; ii) feature that activates for dog ears, showing refined selectivity for specific facial features; iii) a feature that strongly activates for odometers.
21

---

Concepts from Layer 9 Noise SAE
cnd Sum 2:10
bay2344
french horn BDY
costed Whare
cocktal Shaker 1575
Erayy 1:10
CaIarT 3:002
3,031
bpakahep
Figure 12: Maximally activating images for three features from an SAE trained on Layer activations of VIT with Gaussian noise input. These features reveal increasingly abstracted and high-level regularities structured visual scenes: i) feature that activates for glossy or metallic surfaces with high reflectance, capturing regularities in material properties and lighting conditions; ii) a feature that activates for scenes with snow-covered elements; iii) feature that activates for bookshelves library aisles.
22

---

Concepts from Layer 10 Noise SAE
VI0 1233
2255
1346
Maly
Figure 13: Maximally activating images for three features from SAE trained Layer 10 activations of ViT with Gaussian noise input. These features demonstrate increasing semantic clarity and category specificity: i) feature that activates for close-up views of birds of prey and owls, capturing high-level animal features such as eyes and beaks; ii) a feature that activates for elevated natural landscapes, including snowy peaks, alpine valleys, and volcanoes, emphasizing large-scale geographic structures; iii) a feature that activates for floral esppciall daisy-like flowers.
23

---

Concepts from Layer 11 Noise SAE
groz
Tkk
Sparns
Figure 14: Maximally activating images for three features from an SAE trained Layer 11 activations of a VIT with Gaussian noise input. These features exhibit advanced semantic grouping and fine-grain grained category emergence: i) feature that activates for sharks, different perspectives underwater; ii) feature that activates for historical architecture and vaulted interiors; iii) feature that activates for small medium-size sized furry dog breeds.
24

---

Concepts from Layer 12 Noise SAE
jrdces-arteen 1.11
Dote-kintem. LoT1
jok-t enter 1825
790
lswer 2750
baker 2.39
taaker 2312
Inclan wwphent 2333
Oge
Epystan
Figure 15: Maximally activating images for three features from an SAE trained on Layer 12 activations of transformer with Gaussian noise input. At this deepest layer, we observe strong category-level alignment and rich semantic specificity: i) a feature that activates for jack-o" -lanterns, capturing lighting, shape, and facial carve patterns across varied contexts; ii) feature that consistently activates for elephants, invariant to pose and background, indicating robust object identity abstraction; úii feature that activates for cat faces, spanning multiple breeds and subspecies, reflecting high- resolution encoding of facial features and textures.
25

---

C Noise trained SAE concept overlap with different initializations
We train sets of noise SAE instances across all layers with different initialization seeds on the vision transformer residual stream activations derived from Gaussian noise inputs. We then compute the Jaccard index on randomly sampled feature subset, estimate the amount of concepts shared between these two sets of SAEs. The Jaccard index is calculated by dividing the number of overlapping concepts by the total number of concepts identified by both SAEs the subset. For ease of reference, we superimpose the difference in LO observed for natural image inputs to the Lo observed for maximally shuffled image inputs onto Figure (see Figure 4a for reference). Interestingly, the Jaccard index between the two noise SAEs trained with different random seeds has an inverse pattern to the change in LO from baseline to unstructured inputs. This provides further evidence that the vision transformer engages in "conceptual wandering" in its middle layers, since the noise SAES trained from different initializations infer disjoint sets of concepts in the middle layer activations of Gaussian noise. We this distinct three phase pattern: (i) early layers identify consistent, low-level features, (ii) middle layers widely explore the concept space and impose structure that may not be present in the inputs, and (iii) later layers crystallize the explored concepts into more consistent outputs.
0.35
Metric Jaccard Index & SAE Lu
Change in average active SAE features SAE La (Unstructured Normal)
35
0.30
Jaccard Index
25
0.25
0.20
0.15
0
10 11 12 CLIP Vision Transformer Layer Number
3
Figure 16: Conceptual overlap dips in the middle layers of the vision transformer. We plot the Jaccard index between the concepts identified by SAEs trained on Gaussian noise for two different random initializations (left Y-axis). We have also imper-imposed the change in LO plot for the 28x28 patch shuffled images from Figure for reference (right Y-axis). There significantly more concept overlap in the early and later layers, which declines in the middle layers. This is an inverse relationship the increase in LO observed for patch shuffled images.
Layer-wise L0 for unstructured inputs
We provide mean LOs for varying levels of unstructured inputs, for image and text SAEs. Here present raw mean LOs and standard deviations in Tables 6-7, as opposed to Figure which shows the change in LO from the normal image/text baseline.
26

---

Table 6: Mean layer-wise LO for SAE trained on OpenCLIP VIT-B/32 residual stream activations
Mean LO + s.d.
Layer
Normal
28x28 patch
56x56 patch
112x112 patch
1
989.19 + 0.19
989.89 + 0.31
989.38 + 0.23
989.27 + 0.21
2
757.83 = 0.01
757.79 = 0.01
757.83 + 0.01
757.83 + 0.01
3
1007.89 = 0.03
1007.88 = 0.01
1007.88 0.02
1007.89 = 0.03
4
935.06 + 1.17
941.41 + 1.47
938.24 + 1.44
937.05 + 1.40
5
965.15 + 0.28
0.28
973.05 + 1.85
969.00 1.13
966.88 0.78
6
966.36 1.02
1004.33 + 6.45
987.49 4.37
974.20 + 1.96
7
1006.62 1.10
1006.89 0.48
1.05.78
1005.89 0.25
8
9
988.70 0.76
0:41
984.30 0.28
9
9
965.75 0.13
0.1554
965.34
10
854.91 09491=000 0.15
8.10
8.1.
855.08 0.11
1141.99 1.20
1145.59 1.19
1141.99
1141.99 + 1.20
12
829.09 + 0.21
829.62 = 0.05
829.38 0.10
829.19 + 0.16
Table 7: Mean layer-wise for SAE trained on Pythia- 1oihi--p60mmdddupeed residual stream activations
Mean L0 + s.d.
Layer
Normal
1-gram
2-gram
6 O-gram
10-gram
30-gram
1
29.50 0.19
30.24 0.20
29.75 + 0.20
29.29 = 0.20
29.20 0.19
29.09
0.20
2
16.181 0.04
16.20 0.08
15.93 + 0.06
15.85 0.06
15.86
15.90
+.00
3
13.25# 0.05
13.06
12.73 + 0.05
1.07
1..76
21.41 0.10
24.84
22.60
21.28+0.00
21.06 2.10
5
22.31 0.10
0.20
2..04
2.08
21.55 2.15
6
31.98 0.15
0.13
3..09
33.93
33.04 33.40
7
45.891 0.23
0.27
07.39
4.23
46.37 + 0.22
8
65.531 0.14
102.59
0.41
0.56
0.333
73.63
07.85
9
67.55 0.24
149.03
+ 1.53
130.36+1..18
0.86
0.62
71.42 0.38
10
49.12 0.24
1.69
92.59
71.70
62.64
53.06 0.32
11
31.57+
0.11
85.71
7.779
49.56 0.54
4138
34.16 - 0.20
12
11.03# 0.17
14.94 0.43
0.45
10.90 = 0.28
0.24
10.36 0.21
Interestingly, using just the LO from layer 6 of the ViT for each image, we can even predict if the semantic image labels from CLIP will misclassify the image, compared to the ground-truth label from ImageNet. We train simple linear classifier on LO counts of SAE latents extracted from 9000 images, and verify the results across 10 independent cross validation folds. We also attempt to use the pattern of concept activations, extracted by partial least squares (PLS) model, to predict the same misclassification. While still robustly above chance, the results for the PLS model are slightly inferior to the raw LO results. See Table 8 for the mean AUC scores +/- s.d. on this task. We also test both natural and shuffled images, finding that using the LO as an input we see similar prediction performance for both types of image. Though the actual rate of hallucination (misclassification in this case) higher in the shuffled images, is not easier to predict whether misclassification will occur for particular sample.
Table 8: Prediction of semantic mislabeling by CLIP (a proxy for hallucination), using a lincar classifier trained only layer 6 VIT LO scores for each image input. Natural and shuffled images tested separately. We report mean AUC +/- s.d. over 10 cross validation folds.
Natural Images
56x56 Patch Shufiled Images
LO Misclassification Prediction PLS-Concepp Misclassification Prediction
64.8 + 1.1
61.3 1.6
64.5 1
61.5 1.3
27

---

E
Hallucination task specifications and additional results
We use the popular Vectara hallucination leaderboard (24) evaluation methodology to connect patterns of activated concepts in inputs to hallucination in model-generated output. For this task, we test Gemma-2B-IT by inputting the following prompt:
You
chat bot answering questions using data. You must stick to the answers provided solely by the text in the passage provided. You are asked the question "Provide a concise summary of the following passage, covering the core pieces of information described."" <PASSAGE>\n
Where <PASSAGE> is replaced with a full length source article or story from the Vectara leaderboard dataset (1,006 examples). This prompt is identical to the prompt used when evaluating models throughout the Vectara leaderboard, and we also set the temperature parameter to during generation match the task specifications. We use the pre-trained HHEM-2.1 model (3) give hallucination score to each model-generated output summary. HHEM-2.1 takes provided a source-summary pair and returns continuous hallucination score from 0.0 to 1.0, with 0.0 indicating minimum hallucination, and 1.0 indicating maximum hallucination. The motivation for this evaluation of hallucination is straightforward: if HHEM-2.1 detects fabricated content in the summary that is not present the source text, then the summary is said to be hallucinated and is given hallucination score near 1.0. Note that this task does not measure how factually correct the summary is, rather only measures the faithfulness of the summary to the source. Therefore this task is well-suited to measuring notions of hallucination and alignment with the task objectives.
With these source-summary pairs in hand, we next fit a PLS regression (see Preliminaries section for details on PLS) on the maximum concept activations across all tokens in the source as given by pre-trained Gemma-2B SAES [5]. We fit a new PLS model independently for layers 1, 7, 11, 13, and 18 on these concept activations, with the continuous hallucination score for the corresponding model-gemee generated summary as the supervised target. We perform 10-fold cross validation to test the PLS model's 's generalization on new, unseen example concept activations and hallucination scores. To identify key concept activations in the source that drive hallucinations in the (according to the PL.S model), we compute variable importance in projection (VIP) scores for each concept. VIP is a metric that pools the contribution of variable across all PLS components, and is calculated for single concept j as follows:
output
dSAE Evamm J.C
Neamp SS
VIP,-
"Neump SS.
where dSAB is the number of SAE concepts (in the case of the Gemma-2B SAEs this is 16,384), Neomp is the number of PLS components (4 in this case), wje is the weight matrix for projection of the jth SAE concept into the latent space of the cth PLS component, and SS, is the sum of squares for the portion of the hallucination score explained by the cth PLS component. Specifically, SS,- q.. ()) where Je are the hallucination score loadings for the cth PLS component.
To test the practical utility of the most important concepts for hallucination prediction in the PLS, we get the 10 concepts with the highest VIP scores and set their concept activations to 0 during Gemma 2B-IT'ss summary generation. More specifically, we extract the residual stream activations from Gemma 2B-IT at layer 11, run them through the SAE encoder, and set the resulting SAE concept activations for the 10 VIP-identified concepts to O. We then run these modified concept activations through the SAE decoder to get reconstructed residual stream activations, and sum these modified reconstructions with the reconstruction error vector for the reconstruction of the unmodified concept activations.
Table 13 lists a few examples of corrected hallucinations pust-suupression of the top 10 VIP features in the residual stream of layer 11 of Gemma 2B-IT.
We also provide comparison of our PI. approach for detecting and classifying hallucinations with existing hallucination detection methods. Tables 9, 10, 11, and 12 show the comparison of the
28

---

results on several different ballucination benchmarks: TruthfulQA (generation) (36,, TriviaQA (gen- eration) [TT] CoQA (generation) [55]. and TruthfulQA (multiple choice). We rum each hallucination benchmark on Gemma-2B-I7 using "Language Model Evaluation Harness" framework to provide fair point of comparison. We evaluted on the same 10 cross validation folds. We report the mean AUC +/- 1 s.d. across folds in Tables 9-12. For each benchmark dataset, the hallucination detection method (rows of the Tables) must predict whether new, unseen examples are hallucinated. The "ground truth" hallucination label for each sample is determined according to the metric(s) associated with each dataset (columns of the Tables). For instance, TruthfulQA (generation) evaluates faithfulness with BLEU 488, 5 ROUGEI [35]. ROUGE2, and ROUGEL scores, TriviaQA evaluates correctness with an exact matching string, CoQA (generation) also uses an exact match score and a modified score, and TruthfulQA (multiple choice) evaluates hallucination in settings with either single correct answer multiple correct answers.
We find that our PLS and concept-based approach significantly outperforms these established methods on all but the CoQA (generation) benchmark dataset. Further, our method only uses evaluates the input into the LLM, while the other methods require at least some generation of the output text itself.
F
Impact statement
Our work presents and experimentally tests a hypothesis that transformer model hallucination is connected to an increase in high level concepts inferred by the transformer. We also show that certain types of inputs (in our case noisy or experimentally perturbed inputs) increase the the number of concepts imposed
the inputs by the transformer models. We link the pattern of these high level concepts inferred in the input directly to notions of hallucination in the model-generated output, and show that the hallucination rate of these outputs be controlled through targeted concept suppression. With this in mind, these hypotheses and techniques could be used to positive (reducing hallucinations, improving output faithfulness) or potentially negative ends (adversarial attacks, misalignment with human values, degraded model performance). Therefore, we expect this work and similar work focused on model alignment and faithfulness have broad societal impacts in the long run. However, interpretability in transformers and LLMs is still at relatively nascent stage, and we do not present immediately actionable recipes for negatively controlling or adversarially perturbing transformer models at scale. Thus we do not expect immediate negative societal consequences to stem from this work.
G
Code availability
Code for the experiments in this paper is deposited http://gghhub..ccmm//bbammmmmmmm to-narrative
Table 9: Comparison of hallucination detection methods on TruthfulQA (generation). Mean AUC +/- 1 s.d. is reported, over 10 cross validation folds, predicting whether an output hallucinated not. our method
BLEU
ROUGE1
ROUGE2
ROUGEL
Perplexity (56)
50.0+5.3
47.3+5.1
49.9+1.5
45.7+4.4
Energy Score 38
47.2+3.0
50.5+3.5
49.9+4.5
52,4+3.6
Lexical Similarity
(7
47.0=2.9
49.9+2.8
53.0+5.7
49.1
Normalized Entropy
39
51.145.5
49.5+4.1
49,4+1.6
48.6+4.3
EigenScore 10
53.9+5.4
51.8+4.8
47.0#5.9
51.0#4.9
PLS-Concepis (
59.426.3
56.4+5.0
54.4+6.7
56.3+5,6
PLS- PLS-Concepts (Layer7)*
59.945.0
56.6+3.5
55.3+7.2
54.3+4.6
PI.S-Conced (Layerl1)*
60.043.6
56.314.3
58.349.1
57.3+4.0
PLS-Concepts (Layer13)*
57.645.5
58.2+4.7
56.616.1
58.945.6
PLS- HLS-CCnneptss (Layer17)*
54.6+3.7
57.0+5.2
52,244.6
54.9+6.2
29

---

Table 10: Comparison of hallucination detection methods on TriviaQA (generation). Mean AUC +/-
1 s.d. is reported, over cross validation folds, predicting whether an output hallucinated or not.
our method
Exact Match
Perplexity
50.642.8
Energy Score
50.0=0.1
Lexical Similarity
68,9+2.9
Normalized Entropy
51.0=1.4
EigenScore
66,322.5
PLS-Concepts (Layerl)*
6..3+1.3
PLS- Concepnss (Layer7)*
65.6+2.3
PLS-Concepts (Layer11)*
70.8+2.6
PLS- LLS-Concepts (Layer13)*
70.1+2.4
PI.S-Concepss (Layee177)*
59.1=1.9
Table 11: Comparison of hallucination detection methods on CoQA (generation). Mean AUC +/- s.d. is reported, over 10 cross validation folds, predicting whether an output hallucinated not. our method
Exact Match
fl
Perplexity
50.0+0.0
50.0+0.0
Energy Score
50.4+0.4
50.140.4
Lexical Similarity
66.4+1.6
59.0+2.0
Normalized Entropy
49.9+0.1
50.020.1
EigenScore
62.3+1.9
51.0+1.3
PLS-Concepts (Layerl)*
52.9+1.9
52.4=0.9
PLS-Concepts (Layer7)*
52.6+2.0
51.7+1.1
PLS-Concepts (Layerl
53.9+2.6
52.1#1.1
PLS- ..S-CConsspts (Layer13)*
53.9+1.5
51.6+1.3
PLS ..S--Coneepts (Layer17y*
49.4+1.9
50.5+1.6
Table 12: Comparison of hallucination detection methods on TruthfulQA (multiple choice). Mean
AUC +/- s.d. reported, over cross validation folds, predicting whether an output is hallucinated or not.
# our method
Metric
One Correct Answer
Multiple Correct Answers
Perplexity
49.1
49.4+1.9
Energy Score
50.0+4.2
49.8+0.3
PLS-Concepts (Layerl)*
61.1+2.5
62.243.7
PL.S- PLS-Concepte (Layer7)*
58.945.5
61.643.4
PLS-Concepp (Layerl 1)
61.1+2.6
63.243.5
PLS- FLS-Concepts (Layer13)*
60.7+3.6
63.744.1
PLS-Concepts (Layer17)*
52.4+3.5
55.2+4.9
30

---

Table Gemma 2B-IT recovered hallucination examples post layer 11 suppression of top 10 VIP concepts
Summaries
Fabrications
Hallucination Score Change
No Suppression
With Suppression
(no suppression)
The review is expected completed by the end of 2023 and will be the first time that commis- sioner has undertaken such comprehensive review of care
The passage is about the view of quality of life and care of older people living in residential care in Wales by the Older People's Commissioner for Wales, Sarah Rochira.
The article does mention a completion date
-0.95
The article does not claim that this is the first time such comprehen- sive review has been con- ducted
the quality of older people in Walcs.
Sure, here's a concise sum- mary of the passage:
Sure, here's a concise sum- mary of the passage:
Southgate serving as the interim manager
-0.82
Wenger's contract runs to the summer, and he has been linked with the Eng- land manager's s job.
Wenger's contract runs to the summer, and he has been linked with the Eng- land manager's job. South- gate is in contention for the role, and he has been linked the England tional team. Wenger also spoke about the future of Hector Bellerin and Aaron Ramsey.
The matches are in November, not December Aaron Ramsey plays for Arsenal, not Manch- ester City
Southgate has been ap- pointed England man- ager and will lead the against Scotland and Spain in November and Decem- ber.
Thomas Tuchel manager, midfielder, and not mentioned at all in the original article
Wenger is happy that South- gate has a chance to show his quality and believes he has the necessary experi- ence to make the right de- cisions.
Arsenal's Hector Bellerin is set to sign a new con- tract, but the club is still interested in Barcelona and Manchester City's s midfield- ers, Aaron Ramsey and Thomas Tuchel.
Jak Trueman's is story is re- minder that even the face of adversity, important to stay positive and never give hope.
#$ummary: Jak True-
Neglects to mention -0.74
15-year-old schoolboy who was suffer- ing from a rare form of blood cancer called gamma delta T-cell lymphoma. He documented his life and treatment on social media and attracted more than 28,000 Likes on his Face- book page. Despite his ill- ness, Jak remained positive and touched the hearts of many people in West Loth- ian and beyond.
illness
Overly vague, misses
core facts about the article
31

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction explicitly outline our primary contributions, and these accurately reflect the results presented in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We include Limitations subsection beneath the Conclusion section, outlining the assumptions made and how the results might be expected scale with larger architectures alternative interpretability approaches.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
32

---

Answer:
[NA]
Justification:
We do not present theoretical results. Our paper is focused on empirical
experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We utilize pre-trained models, open model architectures, and publicly available datasets, and where appropriate we provide full specifications for all hyperparameters and training setups for models that we train ourselves (Appendix A and Preliminaries). We also provide full and detailed specifications for all experiments.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
33

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We have provided the scripts necessary train and evaluate relevant SAEs, as well as scripts to conduct each experiment. Relevant Python packages are also listed. All code is provided in the Supplementary Materials. All data and pre-trained models available from public sources.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
public/guides/ Stadlees/ooeSSunisssiioooeoe
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ htapss.cc/publlc/uuddeess
wwt.bbbmmssooooooooiy.o for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Full experimental details are provided in the Preliminaries, each of the results sections, and Appendices A, C, and E. We also provide full scripts with the exact setup for each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We present figures 4a, 4b, and 5a with error bars of one standard deviation. We report the meaning of the error bars in the main text. We undertake cross validation for predicted metrics, and provide further fine-grained results in Appendices and D.
Guidelines:
The answer NA means that the paper does not include experiments.
34

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The details about the resources used to execute the experiments presented in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and are confident that our work is in compliance.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
35

---

Justification: We discuss potential societal impacts in the Introduction and Conclusion, and we also include dedicated Impact statement in Appendix F.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We not releasing any model dataset part of this work that has a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited all the datasets and the packages that were primarily used to produce the experimental code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
36

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets for public use as part of this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We did not conduct any crowdsourcing experiments or experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did not conduct any crowdsourcing experiments or experiments with human subjects.
37

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
We do not any LLMs as part of the core method development.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy for what should or should not be described.
38


Reviews:
[{"review_id": "Hw0pcg5w37", "reviewer": "Reviewer_LyaA", "summary": "At a high-level, the paper shows transformer models create signals (concepts) out of noise. The middle layers contribute more to this aspect, and this contributes to the footprint of hallucination. More specifically, the paper trains a SAE to map residual activations to concepts. The paper presents performs multiple experiments to demonstrate the validity of the approach using both text and image models. (1) Semantic concepts are invoked by pure noise inputs [Section 3]. (2) Distorted inputs also invoke concepts [Section 4] (3) Internal activations are also predictive of hallucinations [Section 5]. All the experiments leverage SAE-induced representations.", "questions": "1.\tDefine residual activation.\n\n2.\tWhat is the significance of 1.3M?\n\n3.\tAre d_i in section 2.2 learned or initialized?\n\n4.\tIn section 2.2, x in LHS is of dimension d_model, but the RHS is of dimensions d_SAE since the unit vectors have that dimension. How does this all fit?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "S1. The problems addressed in the paper (interpreting activation of transformer models) are important and the conclusions (that SAE-induced interpretable representations can be used to trace hallucinations) are plausible.\n\nS2: The paper presents multiple experiments to validate the proposed framework.\n\nW1. The details of modeling are missing in the main paper.  Unfortunately, I am not able to find the appendix sections in the downloaded version. Hence it is impossible to provide an informed review.\n\nW2. The experimental details (such as sizes of datasets and train/test splits) are missing.", "was_revised": true, "final_justification": "It was my mistake not to see the appendix. This got resolved.\n\nIn addition, the authors have provided clear descriptions of the algorithm.\n\nHence I am increasing my score from 2 to 4.", "rebuttal": "**[GENERAL]** \nWe thank the reviewer for confirming the validity of our experiments and conclusions. We appreciate that the reviewer finds the problems addressed in our paper important, and the conclusions we draw from our experiments to be sound. We also appreciate that the reviewer considers our experiments a proper validation for the mechanistic framework of hallucinations put forward in the paper.\n\n**[W1]**\nWe note that the Appendix is included as a PDF named “appendix.pdf” in the “Supplementary Material” zip file attached to the OpenReview submission. Within the Appendix, we have provided full details on model specifications, hyperparameters, and training procedures for each transformer model and SAE used in the paper (Appendix A). In addition to the Preliminaries section in the main text, which describes our modeling setup and motivations for each experiment, we have provided additional details for our hallucination experiments in Appendix E. \n\n**[W2]**\nDirectly addressing the reviewer's point, Appendix A provides the number of input image samples used to train the layer-wise vision SAEs (1.3M images from the ImageNet-1k training set), and the number of tokens used to train the layer-wise language SAEs (204.8M tokens from the FineWeb-Edu corpus). In Section 3 we state the number of validation image samples sourced from the ImageNet-1k validation set: 50,000. For better readability, we will add additional columns to the SAE hyperparameter tables in Appendix A corresponding to the size of the validation sets (50,000 image samples in the case of the vision SAEs, and around 5.12M tokens in the case of the language SAEs).\n\n**[Q1]**\nIn Section 2.1, we define the “residual stream activations” as follows: \n>For all transformer models, the object of investigation is the activations in the residual stream: the hidden state of the deep neural network, updated by attention and multi-layer perceptron (MLP) blocks at each layer. The residual stream is the core conduit for information flow and representation refinement to which layers read from and write to, in the transformer architecture. \n\nHowever, we understand how this may be an unclear definition to readers less familiar with the recent mechanistic interpretability literature. More directly, the residual stream activations refer to the token-level embedding vectors extracted from the transformer model following each layer block [1][2][3]. We will add this additional clarification to Section 2.1.\n\n**[Q2]**\nWe believe that the reviewer is referencing the number of random Gaussian noise images sampled to train the noise SAE in Section 3. We chose 1.3M as a training set size for this experiment to mirror the size of the ImageNet-1k training set, which is used to train SAEs on natural image inputs for the experiments in Section 4. We will add an extra clarification to the appendix justifying this training set size in the noise setting.\n\n**[Q3]**\nThe $\\mathbf{d}\\_i$ are learned. The $\\mathbf{d}\\_i$ defined in Section 2.2 refer to the columns of the decoder matrix: $D: \\mathbb{R}^{d\\_\\mathrm{SAE}} \\to \\mathbb{R}^{d\\_\\mathrm{model}}$. All parameters of this decoder are learned in conjunction with the encoder, using the same loss signal defined in Section 2.2.\n\n**[Q4]**\nWe thank the reviewer for pointing out this oversight on our part. The dimensionality of both $\\mathbf{d}\\_i$ and $\\mathbf{b}$ should be $\\mathbb{R}^{d\\_\\mathrm{model}}$ not $\\mathbb{R}^{d\\_\\mathrm{SAE}}$. With these fixes, the equation for the reconstruction of the activation sample $\\mathbf{x}$ is correct, as $f\\_i(\\mathbf{x})$ is a scalar and we sum over all $i$ SAE dimensions. We will update the paper to reflect this change, as well as clarify the role of the decoder and its training to address [Q3]. Here is the updated Section 2.2:\n\n> For each SAE formulation, we work with SAEs whose parameters are trained to produce the following approximation for a given transformer model activation vector $\\mathbf{x}\\in \\mathbb{R}^{d_\\mathrm{model}}$ at a particular layer:\n>\n> $$\n> \\mathbf{x} \\approx \\sum\\_i^{d_\\mathrm{SAE}}f_i(\\mathbf{x})\\mathbf{d}\\_i+\\mathbf{b}\\,\n> $$\n> \n> that is, each transformer layer’s intermediate activation can be approximated as a (sparse) linear combination of unit direction vectors $\\mathbf{d}\\_i\\in \\mathbb{R}^{d\\_\\mathrm{model}}$ scaled by concept activation coefficients $f_i(\\mathbf{x})\\geq0$, with $\\mathbf{b}\\in \\mathbb{R}^{d\\_\\mathrm{model}}$ being the bias term. Practically speaking, this SAE is a simple neural network with a learned encoder $F: \\mathbb{R}^{d\\_\\mathrm{model}} \\to \\mathbb{R}^{d\\_\\mathrm{SAE}}$ projecting each transformer activation into a sparse concept space, typically of much larger dimensionality than the transformer model activation space. For instance, the ViT embedding dimensionality is $d\\_\\mathrm{model} = 768$ with the corresponding SAE concept space dimension being $d\\_\\mathrm{SAE} = 49,152$. Conversely, the learned decoder $D: \\mathbb{R}^{d\\_\\mathrm{SAE}} \\to \\mathbb{R}^{d\\_\\mathrm{model}}$ component of the SAE aims to reconstruct the original transformer residual stream activation solely from this pattern of sparse concept activations. \n\n**[CLOSING]**\nExperimental details provided in the Appendix, as well as the clarifications of terminology and math notation detailed above should address the reviewer’s outstanding concerns. Thanks to the reviewer, the additional clarity and modeling details will greatly improve the readability of the paper.\n\n[1] Elhage et al. \"A mathematical framework for transformer circuits.\" 2021\n\n[2]Elhage et al. \"Privileged bases in the transformer residual stream.\" 2023\n\n[3] Cunningham et al. \"Sparse autoencoders find highly interpretable features in language models.\" 2023"}, {"review_id": "SgvZDLm2kl", "reviewer": "Reviewer_5gvB", "summary": "This paper describes the use of sparse auto-encoders (SAEs) to probe conceptual representations at different layers of Transformer models, towards a mechanistic understanding of ‘hallucinations’ in these models. Three experimental setups are explored: SAEs trained on the activations of Transformers exposed to pure noise; SAEs trained on activations from Transformers exposed to real image and text data; and pre-trained SAEs. \n\nA key finding of the paper is that Transformer models are prone to imposing semantic structure on meaningless inputs (perhaps akin to the phenomenon of ‘pareidolia,’ or the human tendency to perceive meaningful structures in random stimuli, e.g. seeing faces in clouds). Another finding is that perturbed inputs activate a greater number of concepts than unperturbed inputs in the middle layers of Transformer models, an effect that scales up with the degree of perturbation. Additionally, the findings suggest that hallucinations can be steered based on selective inputs to a model.", "questions": "None at this time.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper is well-organized and clear, with ample definitions and concise descriptions of relevant prior work. The paper provides an interesting and compelling approach to tracing hallucinations in Transformer models, with valuable findings that pave the way for future work in AI safety and interpretability, while complementing existing work. Appendix B offers a particularly valuable case study of how different kinds of features are activated in each layer of OpenCLIP ViT-B/32 exposed to pure noise, with increasing conceptual specificity as layers progress. \n\nWeaknesses:\nNot aware of serious weaknesses in the paper. Just an observation: References [35] and [36] are the same paper (Maynez et al 2020).", "was_revised": true, "final_justification": "Having considered the other reviews & rebuttals, alongside my original review, I maintain my original score.", "rebuttal": "**[GENERAL]**\nWe thank the reviewer for their thorough reading of the paper and their enthusiastic appraisal of our work.\n\n**[W1]**\nWe will merge references [35] and [36] as per the reviewer’s helpful comment."}, {"review_id": "9K3SrG4sEy", "reviewer": "Reviewer_e9fj", "summary": "This paper investigates representations of semantic concepts in transformers (in vision and in text) as the uncertainty in the input space is varied through the lens of sparse autoencoders. They find that as inputs become noisier, more semantic concepts are activated. They then attempt to use these findings for the use case of hallucination detection and then steer models to prevent hallucination using the learned concept directions.", "questions": "- Is noisy input data even indicative of likelihood of hallucination? Some explanation of how this links to aleatoric/epistemic uncertainty and what kinda of hallucinations you are thinking about would be useful. More experiments from the LLM uncertainty quantification/hallucination literature would also be useful to dissect this. Generalisation to new datasets.\n- Do you have an interpretation for why more concepts are activated than fewer? Is there a “randomness”/”noise”/”entropy” concept that emerges for messy input data?\n- How do you decompose the likelihood of hallucination as a function of output length, temperature of sampling?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- Very well written and thorough investigation from first principles from hypothesis to experimental design and results.\n- Strong, interesting and (to my knowledge) novel results for mechanistic understanding of concept activations in the presence of noisy data.\n\nWeaknesses:\n\n- The hallucination detection experiments are not particularly convincing as they are lacking detail and not aligned/compared with standard baselines in the LLM uncertainty quantification literature (e.g. short form Q+A generations). Particularly not convinced of the jump from aleatoric noise in input space leading to more concepts translates to hallucinations.\n- The macro level results of L0 concept activitations lack the granularity of individual sample results - some qualitative examples on these would be useful.\n\nOverall the paper has lots of exciting insights even without the hallucination section, which needs to be made more concrete.", "was_revised": true, "final_justification": "The authors addressed my comments, and I maintain my score.", "rebuttal": "**[GENERAL]**\nWe thank the reviewer for their positive assessment of our work and for their constructive comments. We particularly appreciate the reviewer highlighting the novelty and significance of our noise activation-trained SAE experiments and our mechanistic insights into internal transformer concept dynamics. In response to this reviewer comment and the comments from Reviewer pJz5, we have added several new hallucination benchmark datasets and have compared our hallucination detection approach with established baselines. We have also added a new experiment linking the vision SAE-identified concepts to an analogous notion of hallucination in the image setting.\n\n**[W1]**\nWe recognize that our hallucination results could benefit from proper contextualization with additional datasets and existing hallucination detection methods. Note that our main goal here was not to develop a state-of-the-art hallucination detection method. Instead, we aimed to reveal a direct link between SAE-identified concepts in the internal representations of transformers and their propensity to produce confabulated outputs. In this sense, the \"faithfulness\" of summarization is an ideal task as it is agnostic to the internal \"knowledge\" of the LLM [1]. However, we have conducted additional benchmarks with TruthfulQA [2], TriviaQA [3], and CoQA [4], comparing with established hallucination prediction baselines. These short form Q+A tasks expand the scope of hallucinations studied with our present framework. Please see the reply to Reviewer pJz5 for the full results tables and experimental setup (not reproduced here due to space constraints).\n\nWe also recognize that the connection between the noise-input experiments (Sections 2 & 3) and the hallucination experiments (Section 4) could be made stronger. It is somewhat challenging to find exact parallels between vision and language model hallucinations, which is why we focused solely on language hallucinations in the original text. However, to this end, we have conducted additional experiments testing the utility of the L0 metric in predicting an analogous notion of hallucination in the CLIP model: misclassification of semantic image labels. We train a simple linear classifier to predict whether or not an image will be misclassified, based solely on the L0 metric for that image. We source the concept activations and calculate the L0 metric from layer 6 of the vision transformer, as this layer yielded the largest spike in mean L0 differences between normal and shuffled images in Section 4. We train our linear classifier on L0 counts of SAE latents extracted from 9000 images, and verify the results across 10 independent cross validation folds. We find that we can predict whether or not an image will be misclassified solely using its associated L0, robustly above random chance. We also attempt to use the pattern of concept activations, extracted by a partial least squares (PLS) model, to predict the same misclassification. While still robustly above chance, the results for the PLS model are slightly inferior to the raw L0 results (see table below). As well, the natural versus shuffled images seem to yield equal prediction performance; this is likely because although the *rate* of hallucination (misclassification) is higher in the shuffled images, it is not easier to predict whether a misclassification will occur for a given sample. We believe that this new experiment provides a solid bridge between the vision transformer experiments and the direct hallucination prediction tasks with the LLMs.\n\n*Mean AUC (percentage) ± 1 s.d., from ViT layer 6 concepts*\n|  |       Natural Images       | 56x56 Patch Shuffled Images |\n|---|:---:|:---:|\n| L0 Misclassification Prediction | 64.8 ± 1.1 | 61.3 ± 1.6 |\n| PLS-Concept Misclassification Prediction | 64.5 ± 1.5 | 61.5 ± 1.3 |\n\n**[W2]**\nWe will add qualitative examples to the Appendix exploring the additional concepts activated for shuffled images over their natural counterparts.\n\n**[Q1]**\nThis is an interesting point. Based on the results presented in our paper, we would draw the conclusion that noisy input data is one clear way to increase the likelihood of hallucination. This is evident in the extraneous concept activations that arise as the transformer imposes structure on inputs with little true semantic content (Sections 3 & 4). However, there are likely many other means of inducing hallucinations (such as increased context length, certain topics, etc) [5]. We would consider the experiments presented in this paper to speak only to aleatoric uncertainty. The extraneous concepts activated for pure noise inputs are not a result of \"lack of knowledge\", but instead a result of the representational systems, shortcuts, and biases instilled in the transformer during its training regimen. This is also why we chose to test hallucinations induced during summarization, as this directly addresses the issue of confabulation despite all the necessary information to complete the task being provided in the prompt. In this way the evaluation of hallucinations is divorced from the inherent \"knowledge\" of the model and only tests if its internal representational structure of the input has some bearing on the faithfulness of its output. \n\n**[Q2]**\nOur interpretation, which we attempt to put forward in the Conclusion, is that vision and language transformers alike attempt to impose structure on the input data even in the absence of true structure, as a function of inductive biases derived from massive pre-training. We do not notice a specific \"noise\" concept activated for the shuffled image examples. Instead, we qualitatively see a wide variety of concepts being activated, often with little relation to the true semantic label associated with the natural image. \n\n**[Q3]**\nThanks, we added a sentence in the conclusion, mentioning these key aspects as targets for future research. We would expect that increases in temperature and context length would be accompanied by increased hallucination, as there is more room for conceptual wandering to occur [6]. However, we note that for the hallucination experiments in the original text we set the temperature to 0 (as stipulated by the Vectara benchmark that we implemented).\n\n**[CLOSING]**\nWe are confident that the additional experiments suggested by the reviewer greatly strengthen the robustness and scope of the paper, and provide a more natural connection between the results presented in the main text. We will add these experiments to the Appendix. \n\n[1] Huang et al. \"A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.\" 2023\n\n[2] Lin et al. \"TruthfulQA: Measuring how models mimic human falsehoods.\" 2021\n\n[3] Joshi et al. \"TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.\" 2017\n\n[4] Reddy et al. \"CoQA: A conversational question answering challenge.\" 2019\n\n[5] Kang et al. \"Unfamiliar finetuning examples control how language models hallucinate.\" 2024\n\n[6]Liu et al. \"Lost in the middle: How language models use long contexts.\" 2023"}, {"review_id": "5ySydv2J8S", "reviewer": "Reviewer_pJz5", "summary": "This paper investigates how hallucinations arise in transformer models by analyzing residual stream activations using sparse autoencoders (SAEs), where each decoder direction defines a “concept.” Two metrics, semantic purity and steerability, are used to assess these concepts. The authors train SAEs on residual stream from noise inputs to a vision transformer and find that many resulting concepts still align with coherent semantics when tested on real images. They further show that as input uncertainty increases, more concepts are activated—especially in middle layers. Using this framework, the authors predict hallucination rates in Gemma-2B-IT summaries based on concept activations from the input text. The main message of the paper is that increased input uncertainty corresponds to a higher number of activated concepts, which leads to increased hallucination rates.", "questions": "See weakness section.", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n\n1. The use of sparse autoencoders trained on residual stream activations from pure noise is an interesting setup, suggesting that some semantic structure is baked into the model’s internal representations.\n2. The study, by varying input (such as image patch shuffling and text n-gram scrambling), is convincing and shows that input ambiguity can lead to higher concept activation of the SAEs.\n3. The idea of using internal concept activations to predict hallucination rates in Gemma-2B-IT summaries is a solid step toward connecting the model's internal dynamics to its output behavior.\n\n**Weakness:**\n\n1. While the paper's main goal is to understand the source of hallucination in transformers, much of the work—particularly the analysis focused on vision transformers—feels only loosely connected to this objective. The only part that directly addresses hallucination is the summarization experiment on Gemma-2B-IT. Even there, the results are somewhat limited; it would strengthen the paper if the authors clarified what kinds of hallucinations are being predicted and whether certain types are more reliably captured by the concept activations.\n2. The $R^2$ of 0.27 reported in Figure 5 is modest and not quite convincing, and the paper did not compare its method against established hallucination-detection baselines.\n3. The paper is difficult to follow at times due to a lack of precise definitions for key terms and notations. For example:\n   - how is the function $f_i$ defined?\n   - what do you mean by a concept being \"activated\"? the term was formally never defined.\n   - the concepts of semantic purity and steerability are not properly defined.\n\n**Overall:** The paper offers an interesting analysis of how input uncertainty propagates through a transformer’s internal dynamics. However, the step from these observations to a practical hallucination detector and potential mitigation strategies is only loosely argued and supported by limited evaluation. Consequently, I do not believe the work is yet ready for publication at NeurIPS.", "was_revised": true, "final_justification": "While some concerns remain—particularly the clarity of certain technical definitions and the exploratory nature of parts of the work—I believe the paper presents an interesting and original contribution to understanding hallucinations in transformer models. I am increasing my score to 4 and encourage the authors to further improve the clarity and precision of terminology in the final version.", "rebuttal": "**[GENERAL]**\nWe thank the reviewer for their thoughtful comments and their positive assessment of our experiments. To further support these findings, we have added targeted  experiments with three new text datasets as well as an analogous assessment for the CLIP vision transformers to directly address the reviewer’s points (cf. W1 and W2 below).\n\n**[W1]**\nWe recognize that the experiments conducted in this work cover a lot of territory. We first show that transformer models strongly impose semantic structure on inputs, even when they lack true semantic structure (Section 3). This semantic framing is deconvolved from the internal model representations using SAE-identified concepts. We then demonstrate, in images and text, that this semantic imposition increases as inputs become increasingly unstructured and incoherent (Section 4). Finally, we directly link these SAE-identified concepts in the input space of the transformer to hallucinations in its output (Section 5). We desired to test a putative mechanism for the hallucination phenomenon observed in transformers in general, and, after validating this mechanism in images and language, to demonstrate the practical implications for acting on these insights (e.g. model steering of spurious concepts).\n\nAs the reviewer points out, there are many kinds of “hallucination” [1]. We tested the “faithfulness” of a summarization task, which is not dependent on the LLM’s background knowledge, as all required information is contained within the prompt. This is therefore a direct test of the effects that the transformer's internal representational structure has on its generated output. For more variety, see the tables presented in W2 for hallucination prediction evaluation on TruthfulQA (Lin et al. 2021), TriviaQA (Joshi et al. 2017), and CoQA (Reddy et al. 2019), as well as comparisons to established baseline hallucination prediction methods. We also perform a new experiment directly linking the L0 and patterns of semantic concepts to the misclassification of image samples (analogous to \"hallucination\" in the CLIP setting), effectively bridging the vision transformer analyses and the hallucination experiments. Please see the reply to Reviewer e9fj for more details and results (not reproduced here due to space limit).\n\n**[W2]**\nWe acknowledge that on its face, an $R^2$ of 0.27 may appear low in some contexts. However, we would like to note that this is a conservative out-of-sample result, averaged over 10 different cross validation folds, predicting a continuous degree of hallucination. This explained variance strengthens our core point that internal layer concept activations can be taken to play a role in output hallucinations. In Section 5, we reported a stronger 73.0% accuracy when classifying examples by whether their hallucination score is above or below the median. Our goal with these experiments was not to build a SOTA “hallucination detector”. Rather, we wanted to show a link between the concept activations and a concrete hallucination metric, and to provide evidence that intervening on these activations can suppress hallucinations to a measurable degree. We show this using a simple linear model. Sophisticated non-linear methods and larger training datasets would likely improve performance. Further, we are predicting output hallucination based on the internal pattern of concept activations derived from the input alone.\n\nWe acknowledge the reviewer’s point that additional baselines would improve credibility to our findings and conclusions. We have added the results of several new experiments in comparison with our PLS-Concepts approach. To provide a fair comparison, we run each hallucination benchmark on Gemma-2B-IT using the widely used “Language Model Evaluation Harness” framework (Gao et al. 2024). This framework provides standardized prompt templates and evaluation metrics for many benchmark datasets. Each hallucination prediction method is evaluated on the same 10 cross validation folds, and the mean AUC ± 1 s.d. across folds is reported. For each task, the hallucination detection method (rows of the tables below) must predict whether new, unseen samples are hallucinated, as labeled by the metric associated with each task (columns of the tables below). Note that different tasks evaluate the \"hallucination\" or \"correctness\" of an output using different metrics.\n\n*TruthfulQA — Generation*\n||BLEU|ROUGE1|ROUGE2|ROUGEL|\n|---|---|---|---|---|\n|Perplexity [2]|50.0±5.3|47.3±5.1|49.9±1.5|45.7±4.4|\n|Energy Score [3]|47.2±3.0|50.5±3.5|49.9±4.5|52.4±3.6|\n|Lexical Similarity [4]|47.0±2.9|49.9±2.8|53.0±5.7|49.1±3.8|\n|Normalized Entropy [5]|51.1±5.5|49.5±4.1|49.4±1.6|48.6±4.3|\n|EigenScore [6]|53.9±5.4|51.8±4.8|47.0±5.9|51.0±4.9|\n|PLS-Concepts (Layer1)*|59.4±6.3|56.4±5.0|54.4±6.7|56.3±5.6|\n|PLS-Concepts (Layer7)*|59.9±5.0|56.6±3.5|55.3±7.2|54.3±4.6|\n|PLS-Concepts (Layer11)*|**60.0±3.6**|56.3±4.3|**58.3±9.1**|57.3±4.0|\n|PLS-Concepts (Layer13)*|57.6±5.5|**58.2±4.7**|56.6±6.1|**58.9±5.6**|\n|PLS-Concepts (Layer17)*|54.6±3.7|57.0±5.2|52.2±4.6|54.9±6.2|\n\n*TriviaQA — Generation*\n||Exact Match|\n|---|---|\n|Perplexity|50.6±2.8|\n|Energy Score|50.0±0.1|\n|Lexical Similarity|68.9±2.9|\n|Normalized Entropy|51.0±1.4|\n|EigenScore|66.3±2.5|\n|PLS-Concepts (Layer1)*|66.3±1.3|\n|PLS-Concepts (Layer7)*|65.6±2.3|\n|PLS-Concepts (Layer11)*|**70.8±2.6**|\n|PLS-Concepts (Layer13)*|70.1±2.4|\n|PLS-Concepts (Layer17)*|59.1±1.9|\n\n*CoQA — Generation*\n||Exact Match|f1|\n|---|---|---|\n|Perplexity|50.0±0.0|50.0±0.0|\n|Energy Score|50.4±0.4|50.1±0.4|\n|Lexical Similarity|**66.4±1.6**|**59.0±2.0**|\n|Normalized Entropy|49.9±0.1|50.0±0.1|\n|EigenScore|62.3±1.9|51.0±1.3|\n|PLS-Concepts (Layer1)*|52.9±1.9|52.4±0.9|\n|PLS-Concepts (Layer7)*|52.6±2.0|51.7±1.1|\n|PLS-Concepts (Layer11)*|53.9±2.6|52.1±1.1|\n|PLS-Concepts (Layer13)*|53.9±1.5|51.6±1.3|\n|PLS-Concepts (Layer17)*|49.4±1.9|50.5±1.6|\n\n*TruthfulQA — Multiple Choice*\n|Metric|One Correct Answer|Multiple Correct Answers|\n|-------------------------|------------------|------------------------|\n|Perplexity|49.1±4.5|49.4±1.9|\n|Energy Score|50.0±4.2|49.8±0.3|\n|PLS-Concepts (Layer1)*|**61.1±2.5**|62.2±3.7|\n|PLS-Concepts (Layer7)*|58.9±5.5|61.6±3.4|\n|PLS-Concepts (Layer11)*|**61.1±2.6**|63.2±3.5|\n|PLS-Concepts (Layer13)*|60.7±3.6|**63.7±4.1**|\n|PLS-Concepts (Layer17)*|52.4±3.5|55.2±4.9|\n\n\n*\\* = our model*\nBased on multiple benchmarks of hallucination classification performance, we note that our method is at least comparable to, if not slightly superior to, many established hallucination prediction tools. We also note that these methods all require some input signal from the LLM-generated content, whereas our method only relies upon concept activations that are internally triggered by the input prompt. As well, Lexical Similarity, Normalized Entropy, and EigenScore require comparisons between multiple LLM generations in order to form their predictions (in the above benchmarks, we set the generation budget to 5). This is why they are not included in the Multiple Choice benchmarks.\n\nWe will add these useful results accompanied by a more fulsome description of the experimental details and evaluation protocols in the Appendix. \n\n**[W3]**\nWe thank the reviewer for making us aware of the lack of clarity surrounding these terms. We acknowledge that some of these terms could indeed be made more concrete.\n\n[W3.1] In Section 2.2, $f_i(\\mathbf{x})$ corresponds to the activation strength of concept i for the input sample $\\mathbf{x}$. Thus, the vector $f(\\mathbf{x}) \\in \\mathbb{R}^{d_\\mathrm{SAE}}$ corresponds to all the SAE concept activations for the input sample $\\mathbf{x}$; we refer to a single element of this vector as $f_i$. In this context, $f$ represents the function of the encoder $F: \\mathbb{R}^{d_\\mathrm{model}} \\to \\mathbb{R}^{d_\\mathrm{SAE}}$. In response to this comment and the comments from reviewer LyaA, we provide an updated Section 2.2 enhanced for clarity (please see the reply to LyaA for the updated section, not reproduced here due to space limit).\n\n[W3.2] In the mechanistic interpretability literature, “a concept is activated” for a given input if $f_i(\\mathbf{x}) \\geq 0$ for concept $i$. That is, there is a non-zero entry for concept $i$ in the SAE-encoded vector of the input sample.\n\n[W3.3] \nWe define the concept quality metrics semantic purity and steerability more formally in Section 2.4. Semantic purity measures the extent to which the semantic labels associated with image samples that activate for a given concept are similar to one another. A concept is \"pure\" if the top-$k$ ($k$=16) maximally activating images for that concept have an average cosine similarity of 0.75 across their semantic label embeddings. Steerability is the capability of flipping a neutral image's semantic label to that of the concept, by clamping the concept's activation to an artificially high value. If $n=32$ out of $32$ images from a randomly sampled batch of neutral images flip to the concept's label after clamping, then this concept is said to be \"steerable\". These conservative metrics are used to validate that the concepts extracted from the noise activation-trained SAE are meaningful. We will update Section 2.4 to reflect the clarified definitions.\n\n**[CLOSING]**\nWe believe that these experiments and clarifications significantly strengthen the results and conclusions introduced in our paper. We thank the reviewer for suggesting these improvements.\n\n[1] Huang et al. \"A survey on hallucination in large language models\" 2023\n\n[2] Ren et al. \"Out-of-distribution detection and selective generation for conditional language models.\" 2022 \n\n[3] Liu et al. \"Energy-based out-of-distribution detection.\" 2022\n\n[4] Lin et al. \"Towards collaborative neural-symbolic graph semantic parsing via uncertainty.\" 2020\n\n[5] Malinin and Gales. \"Uncertainty estimation in autoregressive structured prediction.\" 2020\n\n[6] Chen et al. \"INSIDE: LLMs' internal states retain the power of hallucination detection.\" 2024"}]

Author rebuttal:

Meta-review: The paper explores how hallucinations arise in transformer models by using sparse autoencoders (SAEs) to analyze internal activations under input uncertainty. The core finding is that under input uncertainty, transformers tend to activate "input-agnostic conceptual representations" which are robustly triggered even by pure noise, thereby increasing the likelihood of hallucinated output. 

The authors' rebuttal had convinced the reviewers towards positive final ratings (5/4/5/4). This work is considered a valuable contribution to the field of mechanistic interpretability and AI safety, and is recommended for acceptance.
Decision: Accept (poster)