Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models
Authors: Javier Ferrando
Oscar Balcells Obeso
Senthooran Rajamanoharan
Neel Nanda
Venue: ICLR 2025 Oral (2025)
Keywords: Mechanistic Interpretability
Hallucinations
Language Models
Primary area: interpretability and explainable AI
TL;DR: We use sparse autoencoders to identify directions that encode entity recognition in language models.

Abstract: Hallucinations in large language models are a widespread problem, yet the mechanisms behind whether models will hallucinate are poorly understood, limiting our ability to solve this problem. Using sparse autoencoders as an interpretability tool, we discover that a key part of these mechanisms is entity recognition, where the model detects if an entity is one it can recall facts about. Sparse autoencoders uncover meaningful directions in the representation space, these detect whether the model recognizes an entity, e.g. detecting it doesn't know about an athlete or a movie. This shows that models can have self-knowledge: internal representations about their own capabilities. These directions are causally relevant: capable of steering the model to refuse to answer questions about known entities, or to hallucinate attributes of unknown entities when it would otherwise refuse. We demonstrate that despite the sparse autoencoders being trained on the base model, these directions have a causal effect on the chat model's refusal behavior, suggesting that chat finetuning has repurposed this existing mechanism. Furthermore, we provide an initial exploration into the mechanistic role of these directions in the model, finding that they disrupt the attention of downstream heads that typically move entity attributes to the final token.

Published as a conference paper at ICLR 2025
Do I KNOW THIS ENTITY? KNOWLEDGE AWARENESS AND HALLUCINATIONS IN LANGUAGE MODELS
Javier Ferrando 1.2-
Oscar Obeso
Senthooran Rajamanoharan
Neel Nanda
U. Politècnica de Catalunya
Barcelona Supercomputing Center 'ETH Zürich
ABSTRACT
Hallucinations in large language models are a widespread problem, yet the mech- anisms behind whether models will hallucinate are poorly understood, limiting interpretability
our ability solve this problem. Using sparse autoencoders tool, we discover that a key part of these mechanisms is entity recognition, where the model detects if an entity is it recall facts about. Sparse autoencoders uncover meaningful directions in the representation space, these detect whether the model recognizes an entity, e.g. detecting doesn't know about an athlete a movie. This suggests that models might have self-knowledge: internal representa- tions about their own capabilities. These directions are causally relevant: capable of steering the model to refuse to answer questions about known entities, or to hallucinate attributes of unknown entities when it would otherwise refuse. We demonstrate that despite the sparse autoencoders being trained on the base model, these directions have a causal effect on the chat model's refusal behavior, suggest- ing that chat finetuning has repurposed this existing mechanism. Furthermore, we provide an initial exploration into the mechanistic role of these directions the model, finding that they disrupt the attention of downstream heads that typically move entity attributes to the final token
INTRODUCTION
Large Language Models (LLMMs) have remarkable capabilities (Radford et al. 2019 Brown et al.)
2020 Hoffmaan et al. 2022, Chowdhery et al.) (023) yet have propensity to hallucinate: generat- ing text that is fluent but factually incorrect or unsupported by available information (Ji et al. 2023 Minace et al. 2024). This significantly limits their application in real-world settings where factuality is crucial, such as healthcare. Despite the prevalence and importance of this issue, the mechanistic given prompt remains limited. While there
understanding of whether LLMs will hallucinate has been much work interpreting factual recall (Geva et al. 2023 Nanda et al. 2023 Chughtai et al.)
2024 Yu et al.! 2023), it has mainly focused the mechanism behind recalling known facts, hallucinations or refusals to answer, leaving significant gap in our understanding.
Language models can produce hallucinations due to various factors, including flawed data sources or outdated factual knowledge (Huang et al. 2023). However, an important subset of hallucinations occurs when models are prompted to generate information they don't possess. We operationalize this phenomenon by considering queries about entities of different types (movies, cities, players, and songs). Given a question about an unknown entity, the model either hallucinates or refuses to answer. In this work, we find linear directions in the representation space that potentially encode form of refflknowledge: assessing their own knowledge or lack thereof regarding specific entities. causally relevant for whether it refuses answer. We note that the existence
These directions of this kind of knowledge awareness does not necessarily imply the existence of other forms of self- leff-knowledge, and may be specific to the factual recall mechanism.
We find these directions using Sparse Autoencoders (SAEs) (Bricken et al., 2023 Huben et al. 2024). SAEs
interpretability tool for finding sparse, interpretable decomposition of model
Equual contribution. Work done as part of MI. Alignment & Theory Scholars (MATS) Program. Corre- spondence to Snterrandooosooiss@eeee comm
com, balcells www.llssoscr@@mma.ccoom
We make the codebase available at sthhtpps:///et thub. hwt.h//..c.//aaa.eerraa/eeeeeeee settiittieeeeeeeeeieeseeeeeeeses

---

Published as a conference paper at ICLR 2025
Known Entity Latent Activations
Unknown Entity Latent Activations
Michael Jordan
Michael Joordan
When was the player LeBron James born?
When was the player Wilson Brown born?
He was born in the city of San Francisco
He was born in the city of Anthon
just watched the movic 12 Angry Men
just watched the movie 20 Angry Men
The Beatles song "Yellow Submarine
The Beatles song "Turquoise Submarine'
Table 1: Pair of sparse autoencoder latents that activate known (left) and unknown entities (right) respectively. They fire consistently across entity types (movies, cities, songs, and players).
representations. They are motivated by the Linear Representation Hypothesis (Park et al., 2023 Mikolov al., 2013):: that interpretable properties of the input (features) such as sentiment (Tigges) et al. 2023) or truthfulness (Li et al.) 2023; Zou et al. 2023) are encoded as linear directions in the representation space, and that model representations
sparse linear combinations of these directions. We use Gemma Scope (Lieberum et al. 2024), which offers a suite of SAEs trained on every layer of Gemma models (Team al. 2024), and find internal representations that suggest to encode knowledge awareness in Gemma 2 2B and 9B. Additionally, we reproduce these findings for the Llama 3.1 8B model (Grattafiori al., 2024) using LlamaScope's SAE suite (He et al. 2024).
with results presented in Appendix Q
Arditi et al. (2024) discovered that the decision to refuse harmful request mediated by a single direction. Building on this work, we demonstrate that a model's refusal to answer requests about attributes of entities (knowledge refusal) can similarly be steered with our found entity recognition directions. This finding is particularly intriguing given that Gemma Scope SAEs were trained on the base model on pre-training data. Yet, SAE-derived directions have a causal effect on knowledge- based refusal in the chat model-a behavior incentivized in the finetuning stage. This insight provides additional evidence for the hypothesis that finetuning often repurposes existing mechanisms (Jain
al. 2024 Prakash al. 2024 Kissane al. 2024).
Overall, our contributions are as follows:
Using sparse autoencoders (SAEs) we discover directions in the representation space on the final token of an entity, detecting whether the model can recall facts about the entity, suggesting they encode a form of knowledge awareness.
Our findings show that entity recognition directions generalize across diverse entity types: players, films, songs, cities, and more.
We demonstrate that these directions causally affect knowledge refusal in the chat model, i.c. by steering with these directions, can cause the model to hallucinate rather than refuse on unknown entities, and refuse to answer questions about known entities.
We find that unknown entity recognition directions disrupt the factual recall mech- anism, by suppressing the attention of attribute extraction heads, shown in prior work (Nanda et al.) 2023 Givaa et al.] (003) to be a key part of the mechanism.
We go beyond merely understanding knowledge refusal, and find SAE latents, seemingly representing uncertainty, that are predictive of incorrect answers.
SPARSE AUTOENCODERS
Dictionary learning Olshausen Field 1997) offers a powerful approach for disentangling features in superposition. Sparse Autoencoders (SAEs) have proven to be effective for this task (Sharkey et al. 2022 Bricken et al. 2023). SAEs project model representations æ E Rd into a larger dimen- the SAEs from Gemma Scope (Lieberum al. which use the JumpReLU SAE architecture (Rajamanoharan et al. 2024), which defines the
sional space a(x) R In this work,
2024
We use the default sparsity for each layer, the ones available in Neuronpedia Lin Bloom 2024
2

---

Published as a conference paper at ICLR 2025
Achuatin wes ardiee RAW
Isfent
was player Brown bon?
BAE Age
Generation
Unfottanetalyy, dua'' have uccuza real-tire information, including partonal detalle 15ke birthdataer. the player Wilson Brown 34 protapolsmal busebail plagees born Auquat 2, 1294."*
Untroom oniyy lilenss Heys saalions
Wax
Pagh
au(2) mnloore
MLLson
born
Unknosmeert enlly
We identify SAE latents in the final token of the entity residual stream (i.e. hidden state) that almost exclusively activate on either unknown known entities (scatter plot on the left). Mod- ulating the activation values of these latents, e.g. increasing the known entity latent when asking a question about made-up athlete increases the tendency to hallucinate.
Figure I:
function
SAB(#) a(x: Weec
Dáers
(1)
where
((x) JumpReLLU (= + bene),
(2)
H(x 0), composed by H, the Heaviside step function, and 0, learnable vector acting threshold. Intuitively, this is zero below the threshold, and then identity, with a discontinuous jump the threshold. Wenes dane and Waee base the weight matrices and bias of the encoder and decoder respectively. We refer to latent activation to a component in a(x), while we reserve the term latent direction to a (row) vector in the dictionary Wdec-
with the activation function (Erichson et al.) 2019) (2)
Equation shows that the model representation can be approximately reconstructed by a linear combination of the SAE decoder latents, which often represent monosemantic features (Huben al.) 2024 Bricken et al.] 2023] Templeton et al.] 2024 Gao et al. 2024). By incorporating a sparsity penalty into the training loss function, can constrain this reconstruction be a sparse linear combination, thereby enhancing interpretability:
((x)
SAE( Alla()))lo
(3)
Emcornmmttion
Lxqurity
Steering with SAE Latents. Recall from Equation ( that SAEs reconstruct a model's 's represen- tation as œ R a(z) Wee + bue This means that the reconstruction is a linear combination of the de- coder latents (rows) of Wae: plus a bias, i.e. a R Σ, (xr) dec Thus, increasing/decreasing the activation value an SAE latent, (z), , is equivalent to doing activation steering (Turner al, 2023 with the decoder latent vector, i.e. updating the residual stream as follows:
a new
ad,
METHODOLOGY
To study how language models reflect knowledge awareness about entities, we build a dataset with four different entity types: (basketball) players, movies, cities, and songs from Wikidata (Vrandečid & Krötzsch 2024). For each entity, we extract associated attributes available in Wikidata. Then, we create templates of the form (entity type, entity name, relation, attribute) and prompt Gemma 2 2B and 9B models (Team et al,, 2024) to predict the attribute given (entity type, relation, entity name), for instance:
3

---

Published as a conference paper at ICLR 2025
Top 5 Known Separation Scores Latents
Top
Unknown Separation Scores Latents
0.8
Rayer
Song
Song
0.5
Hovie
Score
0.4
Hove
City
10
15
10
15 20
25
Layer
Figure 2: Layerwise evolution of the Top latents in Gemma 2 2B SAEs, measured by their known (left) and unknown (right) latent separation scores (s (sknow and s sunknown). Error bars show maximum and minimum scores. MaxMin (red line) refers to the minimum separation score across entities of the best latent. This represents how entity-agnostic is the most general latent per layer. In both cases, the middle layers provide the best-performing latents.
Entity type
Relation
The movie 12 Angry Men was directed by Entity name
(5)
Attribute
We then categorize entities into 'known' or 'unknown'. Known entities are those where the model gets at least two attributes correct, while unknown are where it gets them all wrong, we discard any in-between. To measure correctness we use fuzzy string matching ! See Appendix A for description of the process. We acknowledge that this methodology might introduce some labeling inaccuracies, as the model could 'guess' some attributes despite not knowing about the entity or fail to recall the specific attributes we consider while knowing about the entity. However, our primary objective is to achieve a reasonable differentiation between entities rather than striving for perfect classification accuracy. Finally, we split the entities into train/validationntesl (50%, 10%, 40%) sets.
We run the model on the set of prompts containing known and unknown entities. Inspired by Meng jtt al. (0022a): Geva et al.] (2023); Nanda et al. (2023) we use the residual stream of the final token of the entity, known and z unkkoww In each layer (l), we compute the activations of each latent in the SAE, i.e. at.j (1.((x/ (arkown)) and at,j (at (nnnkowwn) For each latent, we obtain the fraction of the time that is active (i.e. has value greater than zero) on known and unknown entities respectively:
Σ
0
1 (at.j(at.amowyy 0
Aknown
punknown
(6)
N Nknown
N Nunknown
where V knnwwn and N Nunknown are the total number of prompts in each subset. Then, we take the difference, obtaining the latent separation scores $1.) sknown ft. fknown
fi.
funknown
and
unkknown runknown f/y for detecting known and unknown entities respectively. $1
SPARSE AUTOENCODERS UNCOVER ENTITY RECOGNITION DIRECTIONS
We find that the separation scores of some of the SAE latents in the training set are high, i.c. they fire almost exclusively on tokens of either known or unknown entities, as depicted in the scatter plot in Figure for Gemma 2B and Figure 8 Appendix [or Gemma 9B. An interesting observa- tion is that latent separation scores reveal a consistent pattern across all entity types, with scores increasing throughout the model and reaching a peak around layer 9 before plateauing (Figure 2 This indicates that latents better distinguishing between known and unknown entities are found in the middle layers.
We also examine the level of generality of the latents by measuring their minimum separation score across entity types (t): players, song, cities and movies. A high minimum separation score indicates that latent performs robustly across entity types, suggesting strong generalization capabilities. For this purpose, for each layer (1) compute MaxMin knowa. max, min, 8t,
known, and similarly for
unknown entities. The increasing trend shown in the MaxMin (red) line in Figure 2 for Gemma 2 Appendix [ for Gemma 9B suggests that more generalized latents- thhse that distinguish between known and unknown entities across various entity types are concentrated
2B and in Figure |
https:: t/gisbbb//ttbbb.be.e wwwtchuu.com/eeatteeeeeee wwwpw//wwwww/////eeeeeeee
4

---

Published as a conference paper at ICLR 2025
in these intermediate layers. This observation is replicated on Llama 3.1 8B in Appendix Q This finding points to a hierarchical organization of entity representation within the model, with more specialized, worse quality, latents in earlier layers and more generalized, higher quality entity-type- agnostic features emerging in the middle layers.
Next, compute the minimum separation scores by considering every SAE latent in every layer, known,r for 1 < < L and < j v dSAF, and equivalentyy for unknown entities.
i.e. min, $1,j To ensure specificity entity tokens, we exclude latents that activate frequently (>2%) on random tokens sampled from the Pile dataset (Gao et af. (020). The latents with highest minimum separation scores exhibit the most generalized behavior of all latents, and will be the focus of our subsequent analysis:
known entity latent
arg max min 81 known, and unknown entity latent arg max min $1.5
unknown,
known seporation score of latent 1. across entiyy types
min unknown separation score of latent 1, across entiyy types
Table [ demonstrates the activation patterns of the Gemma 2B topmost known entity latent on prompts with well-known entities (left), and the patterns for the topmost unknown entity la- tent (right), firing across entities of different types that cannot be recognized. In Appendix [ we provide the activations of these latents sentences containing diverse set of entity types, sug- gesting that indeed they are highly general. To validate these latents' reliability, we analyze their activation frequencies on 283 song titles released after the models' knowledge cutoff (August 2024). As hypothesized, unknown entity latents show higher activation rates, while known entity latents ex- hibit lower activation frequencies (Appendix R]. While we acknowledge potential overlap between these song titles and pre-training data, the consistent activation patterns across multiple models strengthen our confidence in these latents' ability distinguish between known and unknown infor- mation. In the following sections, we explore how these primary entity recognition latents influence the model's overall behavior.
ENTITY RECOGNITION DIRECTIONS CAUSALLY AFFECT KNOWLEDGE REFUSAL
We define knowledge refusal as the model declining to answer question due to reasons like a lack of information or database access justification, rather than safety concerns. To quantify knowledge refusals, we adapt the factual recall prompts as in Example S into questions:
Relation
Entity type
Who directed the movie 12 Angry Men
Entity name
Attribute
and we define a set of common knowledge refusal completions and detect if any of these occur with string matching, e.g. "Unfortunately, don't 't have access to real-time information.... Gemma 2 includes both a base model, and a fine-tuned chat (i.e. instruction tuned) model. In Section 4 we found the entity recognition latents by studying the base model, but here focus on the chat model, as thev have been explicitly fine-tuned to perform knowledge refusal where appropriate (Team et al. 2024 and the factuality of chat models is highly desirable.
We hypothesize that entity recognition directions could be used by chat models to induce knowl- edge refusal. To evaluate this, we use set sample of 100 questions about unknown entities, and measure the number of times the model refuses by steering (as in Equation 4 with the entity recog- nition latents the last token of the entity and the following anddoofinstruucton tookess Figure (left)
*The Gemma 2 technical report (Team al. 2024) mentions "including subsets of data that encourage refusals to minimize hallucinations improves performance on factuality metrics". This pattern is consistent with recent language models, such as Llama 3.1 (Dubey et al. 2024). where the explicit finetuning process for knowledge refusal has been documented.
'e use a validation set to select an appropriate steering coefficient a. In Appendix G we show generations of Gemma 2B IT with different steering coefficients. We select a E [400, 550]. which corresponds to around two times the norm of the residual stream in the layers where the entity recognition latents are present (Ap- pendix F.
5

---

Published as a conference paper at ICLR 2025
100
original generation
Where the player Leßron James bom??
Steering
Rand steering
setting
Lebron Jazas was born >tAbron
Orthogonskoed
Steering known
Random steering
Generation after intervention
atent setting
procide
information, Ancloding preacnel detadia birthplaces.
Player
City
Movic
Song
Figure 3: Left: Number of times Gemma 2 2B refuses to answer in 100 queries about unknown entities. We examine the unmodified original model, the model steered with the known entity latent and unknown entity latent, and the model with the unknown entity latent projected out of its weights (referred to as Orthogonalized model). The mean and standard deviation of steering with 10 random latents are shown for comparison. Right: This example illustrates the effect of steering with the unknown entity recognition latent (same as in Table I The steering induces the model to refuse to answer about a well-known basketball player.
illustrates the original model refusal rate (blue bar), showing some refusal across entity types. We see that the entity recognition SAE latents found in the base model transfer to the chat model, and increasing the unknown entity latent induces almost 100% refusal across all entity types in Gemma 2B. Conversely, increasing the known entity latent activation slightly reduces refusal rates. We also include an Orthogonalized model baseline, which consists of doing weight orthogonalization (Arditi eet al.] (202)) on every matrix writing to the residual stream. Weight orthogonalization modifies each row of a weight matrix to make it perpendicular to a specified direction vector d. This is achieved by subtracting the component of each row that is parallel to d:
WaBew OUE
W out
W
Wonn d' d.
By doing this operation on every output matrix in the model we ensure no component is able to write into that direction. The resulting orthogonalized model with the top unknown entity direction exhibits a large reduction in refusal responses, suggesting this direction plays a crucial role in the model's knowledge refusal behavior. We also include the average refusal rate after steering with 10 differents random latents, using the same configuration (layer and steering coefficient) that the known and unknown entity latents respectively. Additional analysis on the Gemma 9B model and Llama 3.1 8B are detailed in Sections Q similar patterns, albeit with less pronounced effects compared the 2B model.
Figure B (right) shows a refusal response for a well-known basketball player generated by steering with the unknown entity latent. In Figure[[(right we observe that when asked about a non-existent player, Wilson Brown, the model without intervention refuses to answer. However, steering with the known entity latent induces hallucination.
MECHANISTIC ANALYSIS
Entity Recognition Directions Regulate Attention to Entity. In the previous section, we saw that entity recognition latents had causal effect on knowledge refusal. Here, we look how they affect the factual recall mechanism (aka circuit) in prompts of the format of Example [5] This has been well studied before on other language models (Nanda et al., 2023 Geva et al., 2023 Meng et al. 2022a). We replicate the approach of Nanda et al. (2023) on Gemma 2 2B and 9B and find similar circuit. Namely, early attention heads merge the entity's 's name into the last token of the entity, and downstream attention heads extract relevant attributes from the entity and move them to the final token position (Figure 4 (a, b)), this pattern holds across various entity types and model sizes (Ap- perform activation patching (Geiger et al.] 2020) Vig et al. 2020; Meng et al. (222a) on the residual streams and attention heads' outputs (see Ap- pendix H for detailed explanation the method). We use the denoising setup (Heimersheim &
pendix [ and Appendix D). To do the analysis,

---

Published as a conference paper at ICLR 2025
Activation Palching on the Lest Pusition Azantonn Hoad
Activation Parching on the Residual Streams
Aftentoon Sooree Last Histy Token
Steering Known Eiity Lazent
625
642
Fitemtn Scate
00
12000 payer
neve soog
Lapers lleadr, Settey Types
(b)
[6]
Atwention Screes of Lart Enity Token
Attertio Scores of Last Ently Tokel Stoweing ng with Knann Eirtly strel
Attnnion Bcores Last Tokon Ently Slowiigg with Rowddo Vertor
Steoring with Unknown Erity Latont
Aiteetiin Scaon (oriiinst
Allanion Stare (Orgendl
cotentioonn (Drigiot)
(40
10
Figure 4: (a,b) Activation patching on the residual streams and the output of attention heads in the last position (song entities). We patch clean (from known entities prompts) representations into a corrupted forward pass (from unknown entities prompts) and measure the logit difference recovered. (c) Attention paid from the last position to the last token of the entity is greater when faced with a known entity in attributeeextraction heads. (d.e.f) Effect on attention scores, as in (c), after steering the last token of the entity with the unknown entity latent (d), known entity latent (e), and a random vector with same norm (f).
Nanda 2024). where we patch representations from clean run (with known entity) and apply over the run with a corrupted input (with an unknown entity) [+]
Expanding on the findings of Yuksekgonul et al. 22024) who established link between prediction accuracy and attention to the entity tokens, our study reveals a large disparity in attention between known and unknown entities, for instance the attribute extraction heads L18H5 and L20H3 (Fig- ure x (c)), which are overall relevant across entity types in Gemma 2 2B (see example of attributes extracted by these heads in Appendix D]. Notably, attention scores are higher when faced with a known entity. This suggests that the detected entity recognition latents might influence the attention mechanism through the 'keys' computation to induce this behavior. To evaluate this hypothesis we steer the residual stream with the found latents on the last token of the entity, and measure the at- tention scores of the entity tokens. We observe a causal relationship between the entity recognition latents and the attention patterns of the attention heads downstream, being more pronounced in the attribute extraction heads. Steering with the top unknown entity latent reduces the attention to en- tity, even in prompts with a known entity (Figure (d)). while steering with the known entity latent increases the attention scores (Figure 4 (e)). We show in Figure (f) the results of steering with a random unit vector baseline for comparison, and in Appendix K when steering with a random SAE latent. In Appendix M we illustrate the average attention score change to the entity tokens after steering on the residual streams of the last token of the entities in Gemma 2B, 9B, and Llama 3.1 8B with the top 3 known and unknown entity latents. The results reveal an increase/decrease atten- tion score across upper layer heads, with the 9B model showing more subtle effects when steered using unknown latents.
These results provide compelling evidence that the entity recognition SAE latent directions play crucial role in regulating the model's attention mechanisms, and thereby their ability to extract attributes.
We show the proportion of logit difference recovered after each patch in Figure 4 (a). A recovered logit difference of indicates that the prediction after patching is the same the original prediction in the clean run.
7

---

Published as a conference paper at ICLR 2025
Sleering results on Known Entilies
Steering results on Unknown Eniilies
Logit Difference No)
Logit Difference (Yes No)
Playrr
Noye
City Song
City
Song
O1igraal
Handors SAE latunt
Uninncen SAE latert
SAE
Figure 5: Logit difference between "Yes" and "No" predictions on the question "Are you sure you know the (entity_tyre _tpee_ centity_namel) Answer yes or no." after steering with unknown (left) and known (right) entity recognition latents.
Early Entity Recognition Directions Regulate Expressing Knowledge Uncertainty. We have shown that the entity recognition latents causally affect the model's knowledge refusal, implicitly using its knowledge of whether it recognises entity, but not whether they used when explicitly asking a model whether it recognises an entity. To investigate this, we use the following prompt structure:
Are you sure you know the (entity_type) {entity]? Answer yes or no. Answer:
(10)
We then steer the residual streams of the last token of the entity by upweighting the entity recogni- tion latents. In Figure [ we show the results on the logit difference between Yes and responses.
The left plot illustrates the effect of steering known entities prompts with the unknown entity latent. This intervention results a reduction of the logit difference. For comparison, we include a random baseline where we apply randomly sampled SAE latent with the same coefficient. In the right plot, we steer unknown entities prompts with the known entity latent. Despite the model's inherent bias towards Yes predictions for unknown entities (indicated by positive logit differences in the "Orig- inal' column), which indicates the model struggles to accurately express their uncertainty (Yonal al.) 2024), this intervention leads to a positive shift in the logit difference, suggesting that the entity recognition latents, although slightly, have an effect on the expression of uncertainty about knowledge of entities. A similar pattern can be observed in Gemma 2 9B (Appendix N
UNCERTAINTY DIRECTIONS
Having studied how base models represent features for entity recognition, we explore internal representations that may differentiate between correct and wrong answers. Our investigation fo- cuses on chat models, which capable of refusing answer, and we search for directions in the representation space signaling uncertainty or lack of knowledge potentially indicative of upcoming errors. For this analysis we use our entities dataset, and exclude instances where the model refuses to respond, and leave only prompts that elicit either correct predictions errors from the model.
Our study focuses on the study of the residual streams before the answer. We hypothesize that end- of of-intstuttion tokens, which always succeed the instruction, may aggregate information about the whole question (Marks & Tegmark 2023) We select the token model and use examples such as:
the player Wilson Brown born?send_oo Baanreenn_of_tuurmmXmsssrrrrr Ltrrrsttotttftttrrmmooeeeel
tur.emodel An (1I)
For each entity type and layer with available SAE we extract the representations of the model resid- ual stream, for both correct and mistaken answers, and gather the SAE latent activations. We are interested in seeing whether there SAE latents that convey information about how unsure or un- certain the model is to answer to question, but still fails to refuse, giving rise to hallucinations. We divide the dataset of prompts into train/validationntest sets (50%, 10%, 40%).
To capture subtle variations in model uncertainty, which may be represented even when attributes correctly recalled, focus quantifying differences in activation levels between correct and This concept termed by Tigges (202)) the 'summarization motif".
8

---

Published as a conference paper at ICLR 2025
'Unknown' Latent Activations
"Apparently two people were shooting or shooting each other for reasons unknown when eight people were struck by the gunfire
...and the Red Cross all responded the fire. Thee cause of fire remains under investigation.
The Witcher Card Game will have another round of beta tests this spring (platforms TBAA)
His condition was not disclosed, but police said he was described stable.
Table 2: Activations of the Gemma 2B "unknown' latent on the maximally activating examples provided by Neuropedia (Lin & Bloom 2024).
incorrect responses. For each latent, we compute the t-statistic in the training set using two activa- tion samples: at,j (as for correct responses and at, (2)tT(r for incorrect ones. The t-statistic
measures how different the two sample means are from each other, taking into account the variability within the samples:
t-statistic plaa,, [aaat, pa p(a1,
(12)
oaa (=) a (at, (=f"""))²
120000
We use a pre-trained SAE for the 13th layer (out of 18) of Gemma 2B 17 and the available Gemma Scope SAEs for Gemma 9B IT, at layers 10, 21, and 32 (out 42). Our approach for detecting top latents, similar to the entity recognition method described in Section with the highest minimum t-statistic score across entities, representing the most general latents. We split the dataset into train and test sets, and use the training set to select the top latents. The left panel of Figure O reveals a distinct separation between the latent activations at the model token when comparing correct versus incorrect responses in the test set. Using this latent achieves 73.2 AUROC score, and by calibrating the decision threshold on a validation set, it gets an F1 score of 72. See Appendix P with separated errors by entity type. Table 2 illustrates the activations of the highest-scoring latent in Gemma 2B IT's SAE on a large text corpus (Penedo et al., 2024 showing it triggers on text related uncertainty or undisclosed information. Figure (right) illustrates the top tokens with higher logit increase by this latent, further confirming its association with concepts of unknownness [0 Similar latent separations between correct and incorrect answers can also be observed in Gemma 2 9B IT (Appendix OO).
on the top latents
classifier,
POSITIVE LOG
Activation Value
inspecified
unc
8.00
undisclosed
6.87
Movie Cornet
Movie- Ertor
City Conect
City Enror
Sang Correct
Song Error
specifics
6.70
Unknows
Figure 6: Left: Activation values of the Gemma 2B IT 'unknown' latent on correct and incorrect responses. Right: Top 10 tokens with the highest logit increase by the 'unknown' latent influence.
8 RELATED WORK
Recent advances in mechanistic interpretability in language models (Ferrando et al. 2024) have shed light the factual recall process in these systems. Key discoveries include the aggregation of entity tokens (Nanda et al. 2023). the importance early MLPs for entity processing (Meng et al.
Ihttps://huggingg: wwwpo///oee/eeeaaaaeeeaaaaaaeeee 1alllomm/Gemma-2-----ee---e Ru--e-ssd Sterll-SSteaaa---t-e Scope docsn' provide SAEs for Gemma 2 2B IT.
We note that Gemma
/https:///e httpss//ghggggggggee wttaaaa.......aaaaaaaeeeeaaaaaaa co/ co/ddaaasttt/uggiiii http:///ggnnngaee////aaaa/aeaeae
We omit the players category since Gemma 2B IT refuses almost all of those queries.

---

Published as a conference paper at ICLR 2025
2022b), and the identification of specialized extraction relation heads (Geva al. 2023 Chughtai et al, 2024]. Despite these insights, there remains a significant gap in our understanding of the mechanisms underlying failures in attribute extraction leading to hallucinations. Gottesman & Geva (204)) demonstrated that the performance of probes trained on the residual streams of entities corre- lates with the model's ability to answer questions about them accurately. Yukkekkonul et al. (2024) established link between increased attention to entity tokens and improved factual accuracy. (Yu
al. 2024) proposed two mechanisms for non-factual hallucinations: inadequate entity enrichment in early MLPs and failure to extract correct attributes in upper layers. Our research aligns with
studies on hallucination prediction (Kossen et al., 2024 Varshney et al. (002)). particularly those
engaging with model internals (CH-Wang et al. [024] Azaria & Mitchell 2023). Previous work
has trained probes to predict truthfulness of the produced outputs (Li et al., 2023) with Joshi et al.)
(224) showing this can be detected in activation space before the model generation, which can be related to our results on 'uncertainty directions' discovered in Section 7 Additionally, our work contributes to the growing body of literature on practical applications of sparse autoencoders, as investigated by Marks et al. (2024); Krzyzanowski et al. (2024). While the practical applications of sparse autoencoders in language model interpretation still in their early stages, research demonstrates their potential.
CONCLUSIONS
In this paper, that suggest the presence of encoded knowledge awareness about entities. These directions, found in the base model, are causally relevant the knowledge refusal behavior in the chat-based model. We demonstrated that, by manipulating these directions, we can control the model's tendency to refuse answers or hallucinate information. We also provide insights into how the entity recognition directions influence the model behavior, such as regulating the attention paid to entity tokens, and their influence in expressing knowledge uncertainty. Finally, we uncover directions representing model uncertainty to specific queries, capable of discriminating between correct and mistaken an- swers. While our primary focus in this work centers on the representation of knowledge awareness and uncertainty, the methodology to any other type of binary (Section (3) and continuous (Section[7) features. This work contributes to understanding of language model behavior and opens avenues for improving model reliability and mitigating hallucinations.
sparse autoencoders identify directions in the model's representation space
present for discovering these latent directions is generalizable
ACKNOWLEDGMENTS
This work was conducted as part of the ML Alignment & Theory Scholars (MATS) Program. We want to express our sincere gratitude to McKenna Fitzgerald for her guidance and support during the program, to Matthew Wearden for his thoughtful feedback on the manuscript, and to Wes Gurnee for initial discussions that helped shape this work. We want to extend our gratitude to Adam Karvonen, Can Rager, Bart Bussmann, Patrick Leask and Stepan Shabalin for the valuable input during MATS. Lastly, we thank the entire MATS and Lighthaven staff for creating the environment that made this research possible. Portions of this work were supported by the Long Term Future Fund. Javier Ferrando is supported by the fellowship within the "Generación D initiative, Red.es Ministerio para la Transformación Digital y de la Función Pública, for talent atraction (C005/24-ED CV1). Funded by the European Union NextGenerationEU funds, through PRTR. Lastly, we appreciate the anonymous reviewers for their useful comments.
REFERENCES
Andy Arditi, Oscar Balcells Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal in language models mediated by single direction. In The Thirty- eighth Anmual Conference on Neural Information Processing Systems, 2024. URL https:
Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it's lying. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Lin- guistics: EMNIPP 2023, pp. 967-976, Singapore, December 2023. Association for Computa-
10

---

Published as a conference paper at ICLR 2025
tional Linguistics. doi: 10.18653//1/2223.mmming.. fww.dingg--emmll........ URL https://acllathhology org/2023. fen_/222..iinnggs--eee findings-emnlp. 68/
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Tumer, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. URL https::/ttransformrr--rrruu pub/
2023 htwoooooeeemiiiincctreeeeee /induexee//ddx..eeee htmll
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran As- sociates, Inc., 2020. URL httpp:::/proceeeeeeeee Paper Paperrpdda Filee/147ccddbfc66498 http:///00..bb000444444/1/11a140
newrips. wtppsscccpppr.../e wwweef/peeeseeeeeeeeeeeee
Sky CH-Wang, Benjamin Van Durme, Jason Eisner, and Chris Kedzie. Do androids know they're 're only dreaming of electric sheep? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Bangkok,
Findings of the Association for Computational Linguistics: ACL 2024, pp. Thailand, August 2024, Association for Computational Linguistics. doi: 10.18533/1/2022. findings-acl.260. URL https::/aaat http://aaclaatollog....oo www.oologg.orgggg..oooooo / wwe8224ffffno.. findings-acl. 260/
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Er- ica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstemt Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023. URL http: //jmlr.org//appeee httparrorr/pppes/////2222 111.///221114..111 1144.114tt .ttl
Bilal Chughtai, Alan Cooney, and Neel Nanda. Summing up the facts: Additive mechanisms behind factual recall in llms, 2024. URL httpp::///w// http://wwwwwwxi....//o///p arxiv. wttp.org//bbb///../3/3333
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak,
11

---

Published as a conference paper at ICLR 2025
Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasami. Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh, Manohar Paluri, Marcin Kardas. Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa. Sanjay Singh, Sean Bell, Scohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn. Vedanuj Goswami. Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petro- vic, Weiwei Chu, Wenhan Xiong. Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf. Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Bran- don Spence, Brani Stojkovic. Brian Gamido. Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar- caute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni. Francisco Guzmán. Frank Kanayet, Frank Seide, Gabricla Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan. Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Gold- man, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen. Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhic, Jonathan Torres Josh Ginsburg, Junjie Wang. Kai Wu, Kam Hou U. Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan. Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo. Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- Bawa, Nayan Singhal, Nick Fgebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji. Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rañ Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li,
12

---

Published as a conference paper at ICLR 2025
Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyighe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield. Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang. Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Veiioo, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models. ArXiv, 2024. HtRLhhtts:///a/k org/ abs 2407 21783
N. Benjamin Erichson, Zhewei Yao, and Michael W. Mahoney, Jumprelu: retrofit defense strategy for adversarial attacks. ArXiv, 2019. URLhttps::/arxxi.orrr http://rrrii...rr/a/////... 03750
Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R. Costa-jussà. A primer on the inner workings of cransformer-baee language models. ArXiv, 2024. URL
2405. 2405.00208
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint (rXiv:2101.00021 2020.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. ArXiv, 2024. URL https://arxiv. wrwg/ba//////// 04093
Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. In Afra Alishahi, Yonatan Belinkov, Grzegorz Chrupata, Dicuwke Hupkes, Yuval Pinter, and Hassan Sajjad (eds.). Proceedings of the Third BlackboxNLP Workshop Analyzing and Interpreting Neural Networks for NLP, pp. 163-173, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18853/v1/ 2020.blackboxnlp-..... URL httphht://caaannhollloo . org/ 2020. blackboxnlp-1.165
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12216 Singapore. December 2023. Association for Computational Lin- hwwpoolloyyygogg//goooooo org/2023.
guistics. doi: 10. 100186533/1/2023.emm main. 751. URL emmnlp-main.511
Daniela Gottesman and Mor Geva. Estimating knowledge in large language models without gener- ating single token. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), P Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 3994-4019,
Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. wtgg//////222 orr.:oooo..or/222222.mmmmmmmmala www.p-mmin..nn.... 232/
3/653//1/0224ccmmlppp- m32.. URL
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Ko- renev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Mara,, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab
13

---

Published as a conference paper at ICLR 2025
AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee. Georgia Lewis Anderson, Govind That- tai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Eytimov, Jack Zhang. Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jong- soo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani. Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Ku- mar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoy- chev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ra- mon Calderer, Ricardo Silveira Cabral, Robert Stojnic. Roberta Raileanu, Rohan Maheswari. Ro- hit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabusappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang. Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mi- haylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong. Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Gold- schlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan. Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, An- drew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, An- nie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leon- hardi, Bernie Huang. Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brami Stojkovic, Brian Gamido, Britt Mon- talvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang. Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le. Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers. Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel. Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb. Harri- son Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog. Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, nifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings. Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Jun- jie Wang. Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena. Keqian Li, Kiran Jagadeesh. Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A. Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manay Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias
Gaya, Jeff Marcus, Jeff Tang, Jen-
14

---

Published as a conference paper at ICLR 2025
Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Ro- driguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ra- maswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang. Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Ku- mar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang. Wes Bouaziz, Will Constable. Xiaocheng Tang, Xiao- jian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vetito, Zef Rosnbrick, Zhao- duo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https://arxiv.orr http:/wwww.rg/bb//////.//// 21783
Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang. Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. Llama scope: Extracting millions of features from llama-3.1-8h with sparse autoencoders, 2024. URL https:
//arxiw.org/abs//441 20526
Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching. Arxiv, 2024.
URL https://aaxxvv wrg.:iii.orr/rrooooooo aa//aabb.../..////// 15255
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoe, Au- relia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. 30016-30033 30030. Curran As- sociates, Inc., 2022. URL https: htttps://proceeedings wwwpreeeeeeeed.ee.eeieeoeeeeeeee cc/ httpcc//pppppeeccc htcc//ape__tt hwilles/pppe////e papp///22//// af
wtt.rafffff58888800 4404 Abstratt-----c-----aa www.ettt-ooofefeeeo html
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023. URL https:
//arxiv.org/abs//31.1 05232
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL hntpps://openreet di
F76bwRSLeK
Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert Dick, Hidenori Tanaka, Tim Rock- täschel, Edward Grefenstette, and David Krueger. Mechanistically analyzing the effects of fine- tuning on procedurally defined tasks. In The Twelfth International Conference on Learning Rep- resentations, 2024. URL htRL/htpss:////// httt::::/ooeneeeveeeeeee/ttoorna
15

---

Published as a conference paper at ICLR 2025
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey ballucination in natural language generation. ACM Computing Surveys, 55(12), mar 2023. ISSN 0360-0300. doi: 10.1145/3571730. URL |
//doi.org//i 1145/3
Nitish Joshi, Javier Rando, Abulhair Saparov, Najoung Kim, and He He. Personas as way model truthfulness in language models, 2024. URL https://arxiv.. http://www.rr///////////// 18168
Connor Kissane. Robert Krzyzanowski, Arthur Conmy, and Neel Nanda. Base Ilms refuse too.
Less Wrong, 2024. URL
base
too
Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. Se- mantic entropy probes: Robust and cheap hallucination detection in 1lms, 2024. URL https:
//axxiv.org/abs//4055. 2etiias//44..5555 15927
Robert Krzyzanowski, Connor Kissane, Arthur Conmy, and Neel Nanda. We in- spected every head in GPT-2 small using so you don't have to. Al Align- ment Forum, 2024. URL https: wwwp://wwwwww.wo al iiiignttffoumm org wwtp///tt/////eeeeeeeteee httpss/eeeeeeeeiiiieieeee Bsvoaim/ you
we- -ee in- gpt-2 smwls-mussini-----ee
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-ting time intervention: Eliciting truthful answers from language model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. htttts::///pereeeeeee wwtp:/etffooommtttttt
aLLuYpn83y
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. ArXiv, 2024.
httts:://arxiv.org//bt
7408.05147
Johnny Lin and Joseph Bloom.
Announcing neuronpedia: AI Alignment Forum,
Platform for
accelerating research into 2024. URL htttp:///ww.llggmeeeffoo.u..mmm announcing-neurooppnnn ottpa//daae.aaaoooooooooooooorra
sparse
autoencoders.
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets, 2023.
06824
Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. 19647
ArXiv, 2024. /RLL/tttps://rrx..or///bbbb
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual as- sociations in GPT. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 17359-17372. Cur- ran Associates, Inc., 2022a. URL https: httpttess///rroociingg..ooo neurips. cttpssccc//aaa wttpcccppppppppeeeeeeeeeeeeeeeee fttpperrrrileeepppr/// 6f http://622..22222227000055555513
html
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing fac- tual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022b. arXiv:2202.05262.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Advances in Newral Information Processing Sys- fems, volume 26. Curran Associates, Inc., 2013. URL paper_ haper_files/paper// http://///...///.000.00003300000
www..ps...pp..... cc/
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Am- atriain, and Jianfeng Gao. Large language models: survey, 2024. URL htttps://aaaae
06196
16

---

Published as a conference paper at ICLR 2025
Neel Nanda, Senthooran Rajamanoharan, János Kramár, and Rohin Shah. Fact find-
ing: Attempting to reverse-engineer factual recall on the newron level. Al Align-
http://www..lllgmnnfffforrrrmm ment Forum, 2023. URL www.reeeeeeeeeeeeeeeeeeeeeeeeeee factual--cfattttl-e--eeee
fact-finding at at
Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: A strategy
employed by vI? Vision Research, 37(23):3311-3323 1997. ISSN 0042-6989. doi: https///
http://0.00000000000000000000000 Brricle/pp1/500443999900000011
URL https://ww..ciiecediiecccco science/
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear represeetttnn hypothesis and the geometry of large language models. Arxiv, 2023. URL https://arxiiv http://////rrioooraabbb///////// 2211. 2311.03651188
Judea Pearl. Causality.
Cambridge University Press, 2 edition, 2009.
doi: 10.1017/
CB09780511803161
Guilherme Penedo, Hynek Kydlíčck, Loubna Ben allal, Anton Lozhkov. Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL
17557
Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. arXiv, 2024. URL https:
//arxiv.org//ab//442...44 14811
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog. 2019. URL
https://d4mucfpksyww.cooo wtw.://roootteeeeeeeee--eeeeeeee /atp//treeerlaaggugggmmmom//aa/a haagleeddddes//aagggeemeeo mmdeels._ arr_umssupprviiiee__lllitttts__s
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders, 2024. URL https://aaxiii http://axxivv..rra//////./// 14435
Lee Sharkey, Dan Braun, and Beren Millidge. superposition with sparse autoencoders.
Taking features out of Alignment Forum, 2022.
AI
URL
https://www/ hrt.r//rrrrrrrottoccciieiafereee
http://wwwwlllggnnefooorrm
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Fer- ret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Char- line Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchi- son, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Wein- berger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltsshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Plimczak-Plucifskk. Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz. Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kar- tikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moyni- han, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla. Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culli- ton, Pradeep Kuppala, Ramona Comaneseu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni,
17

---

Published as a conference paper at ICLR 2025
Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan. Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Wwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at practical size. ArXiv, 2024. URL https://arxiv. wwwp:///////....../////// 240000018888
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monoseman- ticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL pattp::/ttransformr--cirrrr wttp:cicciicuuuuuuuiooooaa puwb/222//2
mtw.a//annnoooooonniiiiitttttyye html html
Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models. Arxiv, 2023. URL //arxiv. wttp://wwww.rrrbb//11/1.15
Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDi- armid. Activation addition: Steering language models without optimization, 2023.
Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. A stitch in time saves nine: Detecting and mitigating hallucinations of Ilms by validating low-confidence generation, 2023. HtRLhttps://arxx http:://wwxxxv.o../aaeeee
03987
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12388-12401. Cur- ran Associates, Inc., 2020. URL https: htttps://proceedinng 9265062e9221771feee 2ewp22217771fff1121e7777733 www.eeffaabddd---- Absttaact-bbtttttthhhh
Denny Vrandečió and Markus Krötzsch. Wikidata: A free collaborative knowledgebase. ACM, 2024. HtRL/htps::/// //cacm. acm. http//eeeaarccc/iiiittttte
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL httpss://operrwiiees net/ forrunid--yssVNNGo4ul
Gal Yona, Roee Aharoni, and Mor Geva. Can large language models faithfully express their intrinsic uncertainty in words?, 2024. URL
16908
Lei Yu, Meng Cao, Jackie Chi Kit Cheung, and Yue Dong. Mechanistic understanding and mitiga- tion of language model con-factual hallucinations, arXiv, 2024. URL httphttts:///riiioorr abs/ 2403. 18167
Qinan Yu, Jack Merullo, and Ellie Pavlick. Characterizing mechanisms for factual recall in language models, 2023.
hettt::://arxl.orr//bbaa 2310 15910
Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, and Besmira Nushi. Attention satisfies: A constraint-satiifactionn lens on factual errors of language models. In The Twelfth International Conference on Learning Repre- sentations, 2024. URL http:////ppeereeeeeeeee///rrr/ne
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Aun-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down ap- proach to ai transparency. Arxiv, 2023. URL httpp::/aarkk..og//a////////// 01405
18

---

Published as a conference paper at ICLR 2025
A
ENTITY DIVISION INTO KNOWN AND UNKNOWN
The place birth the player Michoel
Jordan
Known entities k'own
the player Michael Jordon
fooured arawers
Set of entities
Attributes evaluation
Unknown
entities
(Focerect snewres
8 eunlmown
else
Rest
Entity
Set of attributes
Place of birth: New York
C;
Jordan
A(e:)
Figure 7: Pipeline for classifying entities as known or unknown. Each entity 6; E & is evaluated by querying the language model about set of attributes A(e;). Classification known unknown is based on the accuracy of the model's responses. In this work we set the threshold T = 1.
Entity Type
Number entities
Attributes
Player
7487
Birthplace, birthdate, teams played
Movie
10895
Director, screenwriter, release date, genre, duration, cast
City
7904
Country, population, elevation, coordinates
Song
8448
Artist, album, publication year, genre
Table 3: Entity types and attributes extracted from Wikidata.
19

---

Published as a conference paper at ICLR 2025
B ENTITY RECOGNITION LATENTS ON DIVERSE ENTITIES
Known Entity Latent Activations
Unknown Entity Latent Activations
Many people use Twitter to share their thoughts.
Many people use Twitter to share their thoughts.
L'Oréal is large cosmetics and beauty company.
O'Oréal is a large cosmetics and beauty company.
The Mona Lisa displayed in Louvre museum.
The Mona Lisa displayed
Louvre museum.
Many people Snapchat for sharing photos and short videos.
Many people Snapchat for sharing photos and short videos.
The Acropolis is an ancient citadel in Athens.
The Acropolis is ancient citadel Athens.
The Galapagos Islands known for their unique wildlife.
The Galapagos Islands known for their unique wildlife.
Many people Dropbox for cloud storage. The pyramids of Giza were built by ancient Egyp- tians.
Many people Dropbox for cloud storage.
The pyramids of Giza were built by ancient Egyp- tians.
Walmart is the world's largest company by revenue.
Walmart is the world's largest company by revenue.
FedEx multinational delivery services com- pany.
FedEx is multinational delivery services com- pany.
Many people use Instagram share photos.
Many people use Instagram to share photos.
The Neuschwanstein Castle inspired Disney's Sleeping Beauty Castle.
The Neuschwanstein Castle inspired Disney's Sleeping Beauty Castle
The theory of gravity was developed by Isaac New- ton.
The theory of gravity was developed by Isaac New- ton.
Sony is known for its electronics and entertainment products.
Sony is known for its electronics and entertainment products.
Many people Skype for voice and video calls. The Sistine Chapel is famous for its frescoes by Michelangelo.
Many people Skype for voice and video calls.
The Sistine Chapel is famous for its frescoes Michelangelo.
The Andes are the longest continental mountain range in the world.
The Andes are the longest continental mountain range in the world.
The theory of evolution was proposed by Charles Darwin.
The theory of evolution was proposed by Charles Darwin.
Many people use Shopify for e-commerce plat- forms.
Many people use Shopify for e-commerce plat- forms.
Honda is known for its motorcycles and automo- biles.
Honda is known for its motorcycles and automo- biles.
Table Activations of Gemma 2B entity recognition latents on LLM generated data.
20

---

Published as a conference paper at ICLR 2025
Known Entity Latent Activations
Unknown Entity Latent Activations
Druids commune with nature in the sacred grove of Elthalas.
Druids commune with nature the sacred grove of Elthalas.
Adventurers seek the lost treasure of King Zephyrion.
Adventurers seek the lost treasure of King Zephyrion.
The Thaumaturge's Guild specializes in Aether manipulation.
The Thaumaturge's Guild specializes Aether manipulation.
The Vorpal Blade was forged by the legendary Jabherwock.
The Vorpal Blade was forged by the legendary Jab berwock.
The Hivemind of Xarzith threatens galactic peace.
The Hivemind of Xarzith threatens galactic peace.
Travelers must appease the Stormcaller cross the Tempest Sea.
Travelers must appease Stormcaller to cross Tempest Sea.
Archaeologists unearthed artifacts from the Zan- thar civilization.
Archaeologists unearthed artifacts from the Zanthar civilization.
Sailors fear the treacherous waters of the My- roskian Sea.
Sailors fear the treacherous waters of the Myrosian Sea.
Scientists studied the unique properties Quixium alloy.
Scientists studied the unique properties of Quixium alloy.
The Glibberthom plant is known for its healing properties.
The Glibberthorn plant known for its healing properties.
The Voidwalker emerged from the Abyssal Rift.
The Voidwalker emerged from the Abyssal Rift.
Alchemists seek to create legendary Philoso- pher's Stone.
Alchemists seek to create the legendary Philoso- pher's 's Stone.
Pilgrims seek enlightenment at Temple Ethe- real Wisdom.
Pilgrims seek enlightenment at the Temple of Ethe- real Wisdom.
Pilots navigate through the treacherous Astral Maelstrom.
Pilots navigate through the treacherous Astral Maelstrom.
Merchants trade rare gems in the bazaars of Khalin- dor.
Merchants trade rare gems in the bazaars of Khalin- dor.
Scholars study ancient texts at the University of Arcanum.
Scholars study ancient texts at the University Ar- canum.
The Vexnor device revalutionized quantum com- puting.
Vexnor device revolutionized quantum com- puting.
The Whispering Woods are guarded by the Sylvani.
The Whispering Woods are guarded by the Sylvani.
The Ethereal Conclave governs the realm of spirits,
The Ethereal Conclave governs the realm of spirits.
The Quantum Forge harnesses the power of Null- stone.
The Quantum Forge harnesses the power of Null- stone.
Table Activations of Gemma 2B entity recognition latents on LLM generated data.
21

---

Published as a conference paper at ICLR 2025
C GEMMA 2 9B LATENTS ACTIVATION FREQUENCIES ON KNOWN AND UNKNOWN PROMPTS
Activation Frequencies Latents: Player
100
Artivation Frequency Unknown Entities
80
40
01
20
40
60
80
100
Activation Frequency Known Entities (%)
Figure 8: Activation frequencies of Gemma 2 9B SAE latents on known and unknown Prompts, in player entity type.
D GEMMA 2 9B LAYERWISE EVOLUTION OF THE TOP 5 LATENTS
Top Known Separation Scores Latents
Unknown Separation Scores Latents
3.0
Rayer
Song
Song
City
City
Moute
Moute
Moxiinn
0.2
Maxhin
Layer
Figure 9: Gemma 2 9B layerwise evolution of the Top 5 latents, as measured by their known (left) and unknown (right) latent separation scores known and gunknown). Error bars show maximum and minimum scores. MaxMin (red line) refers to the minimum separation across entities of the best latent. This represents how entity-agnostic is the most general latent per layer. In both cases, middle layers provide the best-performing latents.
22

---

Published as a conference paper at ICLR 2025
E
NORM RESIDUAL STREAMS
Moan Ruse
Entiyy Type
Flayer
Nor
Moxie
Song
10
15
25
Layor
Layar
S/eon Rennduul Strwam Norm Gemma 28
Mean Resicual Stream Norm Gemma SBIT
1000
Eniiy Type
3500
hom
400
310
200
Layer
Mean Resicuud Straam Norm Gemma 28
Enity Type
Song
Figure 10: Norm of the residual streams of the last token the entity across layers the different Gemma models.
F REFUSAL RATES GEMMA 2 9B
92
Steering unknown latent
Random
Unknown latent setting
Orthogonalized
Unknown latent
42
Steering known
Random stearing
Known latent setting
12-
Movie
Figure 11: Left: Number of times Gemma 9B refuses to answer in 100 queries about unknown entities. We examine the unmodified original model, the model steered with the known entity latent and unknown entity latent, and the model with the unknown entity latent projected out of its weights (referred to as Orthogonalized model). The mean and standard deviation of steering with 10 random latents are shown for comparison. Right: This example illustrates the effect of steering with the unknown entity recognition latent. The steering induces the model refuse answer about well- known basketball player.
23

---

Published as a conference paper at ICLR 2025
G
EXAMPLE OF GENERATIONS STEERING WITH DIFFERENT COEFFICIENTS
Question: Where was born the player Leo Barnhorst?
Generation
Leo Barnhorst born in **Berlin, Germany
100
Leo Barnhorst was born in ** **Grrmny**,
200
do not have access to real-time information, including personal details like birthplaces.
300
do not have access real-time information, including personal details like birthplaces.
couldn't it find any information about a player named Leo Barnhorst. 400
500
I believe you're asking Leo Barhhors**,, professional soccer player.
600
I'm unable to provide specific details about the birthplace of player named Leo Barnhorst.
700
PnnnPeease provide me with the correct spelling of the player's name.
800
rlnnn believe you't 'e asking about Leo Barnhart, a professional soccer player.
900
"rin'nl believe you're asking about Lco Barharr** professional soccer player.
1000
rlnnnI believe you're re asking about Leo Barnhart* professional soccer player.
1100
Associate the player Leo Barnhart with the sport of baseball.
1200
I'm sorry, but don't 't have access to real-time information, including personal details like birthplaces.
criminator:
Table 6: Gemma 2 2B IT responses to 'Where was born the player Leo Barnhorst?' at different
steering coefficient values, a Equation (4). Leo Barnhorst is unknown for Gemma 2 2B.
24

---

Published as a conference paper at ICLR 2025
H ACTIVATION PATCHING
Lakera
dacciora
Lakeza
LeBron
James
Joined
Lite
Nilcon
3rOME
joined
the
Known ently
Unknown entiy
Figure 12: Activation Patching done over the residual stream.
Activation patching (Vig et al. 2020 Meng et al. 2022a Geiger et 2020 Wang et al. 2023) an intervention procedure performed during forward pass. We consider 'clean' input, which in our case is the prompt with known entity (Figure 12 left). We compute an intermediate hidden state, e.g. the residual steam value at token James, as in Figure 12 Then, patch this activation at the same site of the forward pass with the corrupted input. this case, the corrupted input is prompt express this intervention using the do-operator (Pearl (009)) as I [coor do [ unknown +- x know)). )). After the intervention is done, the forward pass continues and the model output compared with the prediction with the corrupted input. In the experiments in Sec- tion 6 we measure the logit difference between the clean (Lakers) and the corrupted predictions (Warriors):
with unknown entity. We
logit, Ceters-Warriors (corr |o unknown
ar known)
(13)
logit Lakers Warriors (clean)
25

---

Published as a conference paper at ICLR 2025
I ACTIVATION PATCHING ON GEMMA 2 2B
Activation Patching on the Residual Streams (Moviss)
(Movtes)
0.1
39
rless
The
beic enstt a mesate
wettng ang
reatuun 1 a
Activation Fatching on the Residual Stroams (CHiss)
Activelion
(Cties)
19
-05
-11
-22
Abock g.
a
enthe Sat
masiuum
a
Activation Residual Streams (Players)
Activation Patching on the Last
Position Atlertion Head (Payens)
03
02
61
13
dlyye 2
encry &
battng tak
resazon tui 1.
metatonn tak
Flead
Figure 13: Gemma 2 2B activation patching results movies (top), players (middle) and cities (bottom).
26

---

Published as a conference paper at ICLR 2025
J ACTIVATION PATCHING ON GEMMA 2 9B
Actiuntion Faszhing on the Last Phoditon Aftenton (Senge)
Painting (Barge)
you
Astvvaiion PMdccinn Haoakad Ehnars (Mores)
Actiustion Pasching on the
000
clsk
Activtion Pushing the Residual Broamo (Dsss)
Actiuation Fasching
Tooll on Atanniinn fheed (Cilbo)
ES
vrotee 3 sotatin bark
Activaat on Paaching Last Attintiin (Paayro
Pla cak Us Cus
St
Head
Figure 14: Gemma 2 9B activation patching results on. from top to bottom, song, movies, players and cities.
27

---

Published as a conference paper at ICLR 2025
K ATTENTION TO LAST ENTITY TOKEN AFTER RANDOM LATENT STEERING
Attention Scores for Original vs. Steered Entities
L20H2 LLBHS
0
Actantion
Attertion Score (Drigins)
Figure 15: Comparison of attention scores the last token of the entity after steering with a random SAE latent from Layer 15.
L ATTRIBUTE EXTRACTION HEADS EXAMPLES
Head
Entity
Extracted Attributes
Kawhi Leonard
Clippers, Niagara, Raptors, Westfalen, Lancaster, Volkswagen
L18H5
Detmold
Boombastic
Jamaican, Reggae, Jamaica, Caribbean
Kawhi Leonard
NBA, baskethall, Clippers, Basketball Germans, German, Germany, Westfalen reggae, Reggae, Jamaican, music, song
L20H3
Detmold
Boombastic
Table Examples from the top tokens promoted by the attribute extraction heads L18H5 and L20H3 in Gemma 2B.
28

---

Published as a conference paper at ICLR 2025
M CHANGE OF ATTENTION SCORES TO ENTITIES AFTER STEERING
Gemma 2 2B (Figures 16 and Gemma 2 9B (Figures 18 and and Llama 3.1 8B (Figures 20 and 21 average attention scores to entity tokens after steering with the top known entity latents and top unknown entity latents. Error bars indicate standard deviation. For the known entity latent steer- ing we use prompts with unknown entities, for unknown entity latent steering we use prompts with known entities. The strength of the steering coefficient is = 100 for Gemma models and a = 20 for Llama 3.1 8B. We show heads starting from layer the latent is found, and the steering is done.
035
62
Aftertan Seete Diferenc
63
Hea
Anteeton Strce Ditt reno
60
U200
12017
HOW
1189
L280
1200
1280
Figure 16: Aggregated attention scores to entity tokens per head in Gemma 2 2B. Steering is done with the top known entity latents (from top bottom).
We have assessed the statistical significance of attention score changes by comparing steering with entity recognition latents versus with random SAE latents using the same layer and steering coeffi- cient. We conduct t-tests where the null hypothesis states that both steerings would yicld identical mean attention scores differences across downstream attention heads. The alternative hypothesis is that known entity latents would increase decrease them. We tested against 10 different random SAE latents using 100 distinct prompts.
attention scores, while unknown entity latents would
Results indicate that for Gemma 2B, Gemma 2 9B and Llama 3.1 8B, steering with the top known entity latent shows statistically significant larger average attention score when compared to random SAE latents 10/10, 10/10 and 7/10 cases respectively. When steering with the top unknown entity
29

---

Published as a conference paper at ICLR 2025
Ahootin Crer Oifteeete
12000
Land
Lited
Dened
Lank
Lake
Lavel
Lak03
Libali
Latee
Alt Wieestent Stowe Difhrenne
Anterton Stow Ditterenee
60
Hoap
Figure 17: Aggregated attention scores to entity tokens per head in Gemma 2B. Steering is done with the top unknown entity latents (from top to bottom).
latent it shows statistically significant lower average attention score when compared to random SAE latents 9/10, 1/10 and 10/10 cases respectively. As shown in Figure 19 (top), Gemma 2 9B top unknown entity latent doesn't show strong reductions. However the second unknown entity latent shows significant differences in 9/10 tests.
30

---

Published as a conference paper at ICLR 2025
Figure 18: Aggregated attention scores entity tokens per head in Gemma 2 9B, Steering is done with the top 3 known entity latents (from top to bottom).
Figure 19: Aggregated attention scores to entity tokens per head in Gemma 2 9B, Steering is done with the top unknown entity latents (from top to bottom).
Figure 20: Aggregated attention scores to entity tokens per head in Llama 3.1 8B. Steering is done with the top known entity latents (from top bottom).
31

---

Published as a conference paper at ICLR 2025
Figure 21: Aggregated attention scores to entity tokens per head in Llama 3.1 8B. Steering is done with the top unknown entity latents (from top to bottom).
32

---

Published as a conference paper at ICLR 2025
GEMMA 2 9B SELF KNOWLEDGE REFLECTION N
Steering Unknown Prompts
Steering Known Prompts with
Logk Diffrrecce Nas No!
Plast Novde
Logk Difference (1os Noi
Plast
Random Roomn EAE
Randen Unoownn DAE
Figure 22: Gemma 9B Logit difference between "Yes" and "No" predictions on the question "Are you sure you know the Centity_lype' (entity_namer Answer yes or no." after steering with unknown (left) and known (right) entity recognition latents..
O
GEMMA 2 9B IT TOP 'UNKNOWN' LATENTS
Unkoooon Activations 5058
'Unknoowt Actiattonn LS2
220
Autivaliion Value
130
Activation
200
100
220
Player Carrect
Player trru
Mouia Conrect
Mouia- tnver
City Correct
City Ertar
Sung Conrect
Sang Eiree
Player Connect
Player trme
Movee Carrect
Norie Errar
Serrng Carrect Erraz
Senig Error
Carrect
Figure 23: Top Gemma 2 9B IT 'unknown' latents based on the t-statistic score.
P GEMMA 2B IT TOP 'UNKNOWN' LATENT WITH SEPARATED ERRORS BASED ON ENTITY TYPE
Activation Value
Movie Correct
Movie Eror (Unknown entity)
Movie Error (Known entity)
City Correct
City
City
Song Correct
Song
Song Error (Khown entity)
Error (Unknown entity)
Error (Known entity)
Error (Unknown entity)
Figure 24: Top Gemma 2B 'unknown' latent based on the t-statistic score, with errors divided into known and unknown entities.
33

---

Published as a conference paper at ICLR 2025
Q LLAMA 3.1 8B REPLICATION
We extend our experimental analysis to Llama 3.1 8B (Grattafiori et al. 2024). using the SAEs suite from LlamaScope (He et al. 2024), which offers per-layer pretrained SAES. Following the methodology described in Section 3 we detect both known and unknown entity latents within the model. The distribution the Top 5 latents across layers (Figure (25) exhibit consistent patterns
with previous findings, with the most effective and generalizable latent representations concentrated in the intermediate layers.
Top Known Separation Scores Latents
Top Unknown Separation Scores Latents
Scone
City
Maxdén
5
10
15
70
75
30
5 10
15 70 75
30
Layer
Layer
Figure 25: Llama 3.1 8B layerwise evolution of the Top 5 latents, as measured by their known (left) and unknown (right) latent separation scores. Error bars show maximum and minimum scores. MaxMin (red line) refers to the minimum separation score across entities of the best latent. This represents how entity-agnosti is the most general latent per layer. In both cases, middle layers provide the best-performing latents.
Steering experiments using the top unknown entity latent reveal increase refusal rates in the instruction-tuned model (Figure 26). Conversely, when we orthogonalize the model weights with respect to this direction, refusal rates decrease. Since the original model's refusal rate on unknown entity prompts is high (Figure 26 left), include the refusal rates prompts with known enti- ties (Figure 26 right). Notably, steering with the top known entity latent did not produce a corre- sponding decrease in refusals.
Original gane stion Siasting
Original gane slin Stcoring
Inictown istent seting
Inkinonn istent oet
Gethogoral aed mode
Orthegonal and mode
Escenilaee sciting
Cruw ilatent secting
Song
Song
Figure 26: Number of times Llama 3.1 8B refuses answer 100 queries about unknown entities (left) and known entities (right). We examine the unmodified original model, the model steered with the known entity latent and unknown entity latent, and the model with the unknown entity latent projected out of its weights (referred to Orthogonalized model). The mean and standard deviation of steering with 10 random latents are shown for comparison.
Further analysis reveal similar findings to those in Gemma regarding attention patterns: steering with the top known entity latent increases the attention scores to the entity (Figure 27 top), while unknown entity latent steering result in diminished attention scores (Figure 27 bottom).
The replication of our key findings originally observed in Gemma across Llama 3.1 8B strength- confidence in both methodological approach and the broader applicability of our results. This generalization is particularly noteworthy given the substantial architectural differences between the two models and their respective SAEs.
34

---

Published as a conference paper at ICLR 2025
Figure 27: Aggregated attention scores to entity tokens per head in Llama 3.1 8B top known entity (top) and unknown entity (bottom) latents.
R ACTIVATION FREQUENCY ON SONGS DATA AFTER KNOWLEDGE CUTOFF
Model
Known Entity Latent
Unknown Entity Latent
Gemma 2B
6%
53%
Gemma 9B
22%
55%
Llama 3.1 8B
13.4%
76%
Table Activation frequency each the top known and unknown entity latents on songs released after knowledge cutoff (August 2024).
TOKEN LIKELIHOOD HYPOTHESIS
An alternative explanation for our observed entity recognition latents is the token likelihood hypoth- esis: the latents might simply encode token likelihood rather than actual knowledge about entities. Under this hypothesis, when processing a token sequence discovered latents at position i could be explained by the model's ability predict token 4 from previous context. For instance, given a well-known movie title, the model would more easily predict subsequent tokens, potentially triggering what we interpret as 'known entity' latents. Conversely, for unknown entities, lower token likelihood might activate our 'unknown entity' latents. This repre- sents plausible confounding factor, as tokens comprising well- entity names are inherently more predictable in the training distribution than those of unknown entities.
the activation our
To test this hypothesis, we analyze token likelihood on broad text corpus from the FineWeb dataset (Penedo et al.) (004)). For each token position i, we compute both the entity recognition latent activations and the probability of the ground-truth token being predicted from position 1. p(4: (.)). If the token likelihood hypothesis were true, we would expect strong correlations between these measures.
Our analysis reveals several key findings that challenge this hypothesis:
Entity recognition latents activate selectively, firing on only a small fraction of tokens (e.g. 0.6% for known entity latents and 0.5% for unknown entity latents Gemma 2 2B, see Table
The correlations between latent activations and token prediction probabilities are negligible across all tested models (Figure 28).
We explored various other potential relationships, including dependencies the perplex- ity of surrounding tokens and next-token prediction entropy, finding no substantial correla- tions.
While observe that tokens where unknown entity latents activate tend to have lower prediction probabilities compared to the baseline, this effect is modest given the latents' sparse activation pat- terns. These findings suggest that token predictability alone cannot explain the behavior of
35

---

Published as a conference paper at ICLR 2025
entity recognition latents, supporting interpretation that they encode sophisticated form of knowledge awareness.
Gemma 2 2B
Tokens
Tokerns here Known Entity Latent Active Tokenss where Unkknown Erkliy Latent is Active
2500
200000
Known tree 6038, 3.067.pe1
2000
Untnown 0.85, -2.06 pro 596
Count
1500
1000
1000
50000
300
0.0
0.6 Probability
0.4
0.6
1.0
0.2
0.4
0.8
1.0
Probability
Probability
Gemma 2 9B
All Takens
Tokenss w hore Known Entity Latent Active Takens where Unknoww Entity Late is Active
200000
thee 6026, 2.862, 6001
Unkkown ('ea:
2.828 cool
2000
Count
1500
1300
100000
500
0
Probabiliiy
Probability
Probability
Llama 3.1 8B
All Tbkens
Tolocns where Known ENtity Latent Active
Takens where Unknnow Entity Latee Active
200000
4000
Count
100000
300
3000
1000
0.6
Probability
Probability
Probability
Distribution of ground-truth next-token probabilities for Gemma 2 2B (top), Gemma 9B (middle), and Llama 3.1 8B (bottom). For each model, we show three distributions: (left) across all tokens in the dataset, (middle) for tokens where the known entity latent activates, and (right) for tokens where the unknown entity latent activates.
Figure 28:
Model
Latent
Activation Frequency
Correlation with p(t. fte))
Known
0.006
0.067 (p=0.000e+00)
Gemma 2 2B
Unknown
0.005
-0.000 (p=9.960e-01)
Known Unknown
0.009
0.062 (0=0000+000)
Gemma 2 9B
0.009
0.009 (p= (p=1.045e-11
Known
0.002
0.003 (p=1.300e-02)
Llama 3.1 8B
Unknown
0.026
0.047 (p=0.000e+00)
Table 9: Activation frequency and correlation with conditional next-token probability for top known and unknown entity latents in each model.
36


Reviews:
[{"review_id": "pyJuBY3FvR", "reviewer": "Reviewer_pFUP", "summary": "This paper discusses some interesting features found in sparse features extracted from Gemma 2 2B and 9B models with the Gemma Scope SAEs. These features, when queried on the final token of an entity mention, are predictive of whether the model correctly recalls facts about that entity in a synthetic task constructed from Wikidata. Some features generalize across multiple (four) entity types, and interventional experiments on chat-tuned models find that they mediate refusals in these models, suggesting that RL training may preferentially lead models to repurpose these mechanisms developed in pretraining. Further analysis shows that these features, when activated, disrupt the normal mechanisms for factual recall by blocking attribute extraction heads. A final experiments identifies SAE latents in a similar spirit which are predictive of model mistakes.", "questions": "* The main text doesn't say which direction d_j you orthogonalize out on p.6. (caption says it but main text description of what you do is disjointed.)\n* Equation 11's use of the word `model` is weird. I assume it's supposed to be a metavariable, but I don't feel like the red highlight conveys this (since it's used for something else above). You already frame it as an example—why not use the actual example correct or incorrect answer there? I think it'd be fine to have a line break in there if the problem is making it fit on the page.\n* Do you have a succinct/clear and empirically grounded explanation of why you use (your formulation of) latent separation scores for the first analysis and t-statistics for the second? It sounds like the features often activated on both categories and the separation scores probably just weren't as good in the latter case, is that right? Seems worth being explicit about in the paper.\n* Can you clarify in the paper how you use the train/test split in the analysis in Section 7? I assume the graphs in Figure 6 plot results for the test set. Also again please give statistics on your results and discuss significance.\n* Do you think you can add some discussion in the paper about how your approach can be generalized to other kinds of features? The targeted investigation you do is interesting on its own but I think it'd be great to highlight if there's anything else that someone can expect to discover by adapting your analysis setup to different datasets.\n\n[Edit: many of the weaknesses and questions have been answered adequately by the reviewers, so I've upped the soundness rating from 3 to 4 and overall rating from 8 to 10.]", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The paper is excellently written. The prose is very clear and the arguments are well structured.\n* The paper fruitfully pursues multiple lines of evidence for its core claims about the latents it identifies in the model it studies. Having correlational and causal analysis on multiple fronts including with the RLHF'd model makes the central idea feel pretty solid.\n* The paper is well scoped.", "weaknesses": "Some of the claims of the paper feel not quite adequately supported by the experiments. I'm not really doubtful of the reasonableness of the claims but I think the paper should quantify and qualify them better.\n* I think the choice to describe these latents as representing \"self-knowledge\" is a little bit dicey. The description only clearly applies to the chat model. For the base model I think a sufficient explanation of the evidence with fewer assumptions is to say something like they're latents for \"entities it knows about\" versus \"entities it doesn't know about or treats as novel\". You could make progress disentangling this from 'self-knowledge' by looking at how these latents behave on fictional narrative versus newswire versus encyclopedic text (the former two should have been written after training cutoff). I would expect the 'known entity' latents to fire increasingly more as you move from fiction->news->encyclopedia and opposite for 'unknown entity' latents. Analyzing these results could help us understand why the features are learned, e.g., if the the model needs to know whether it is writing stories or stating facts.\n* The paper claims that the features generalize over multiple entity types. This is a binary statement but of course is about a graded judgment regarding the amount of generalization. But there is no quantitative claim in the paper about the degree to which this is true in an absolute sense—only that it is _more_ true in the middle layers. That doesn't mean that it's particularly true anywhere. The maxmin metric makes sense for relative comparisons but is difficult to understand on an absolute level. Indeed, when you select the \"most general\" latents based on this metric it's unclear if these features are actually general across more than the four entity types you use or if you may just be selecting on noise. To measure this, you should measure what happens when you select latents based on their generality across a subset of entity types, and test their separation score on held-out entity types. Some measure of how predictive this selection is of separation for new entity types gives an actual measure of the generality of the latent.\n* In the mechanistic analysis (p.6–7) the paper points to figures which seem to show that attention to an entity is decreased when the unknown entity latents are steered (increased). It's not clear to me from the text if this is averaging across all attention scores or some subset (\"these attention heads\", L345, reads to me like the latter, but I'm not sure which subset) and the results of this experiment aren't quantified. There should be some straightforward measure here which you can use to quantify the suppression effect here (which will also allow you to claim statistical significance — you do say \"significantly\" on L346 without a relevant quantity or p-value, which leaves it unclear if you actually mean statistical signifiance)\n* The results in Figure 5 seem dubious. They are really hard to read because of the overlap of all of the confidence intervals but overall nothing significant seems to be happening here. I think you'd need to run this study on much more data to get something interesting/clearly meaningful and I also find this the least interesting or informative experiment in the paper. Mild recommendation to appendicize it. (and strong rec to make the statistical claims precise either way: is the effect you're talking about significant or not?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vQmtECpc6T", "reviewer": "Reviewer_kSkD", "summary": "Using sparse autoencoders (SAEs), this paper shows the existence of directions in the latent space of LLMs responsible for entity recognition i.e. detecting whether the entity is a known entity or not. The paper shows that these identified latents are causally responsible for LLMs refusing to answer questions. The paper shows how these latent directions are responsible for regulating attention scores, and also responsible for directly expressing knowledge uncertainty. All experiments are conducted with Gemma 2 models (2B and 9B) where SAEs are available.", "questions": "1. Formatting — the bottom margin seems much bigger than in the ICLR template?\n\n2. Line 160, on average how many attributes does each entity have?\n\n3. I also couldn’t find details about how many entities of each type was used in the analysis.\n\n4. Figure 5 — Prior work has shown that LLMs are better calibrated when using few-shot examples (e.g. https://arxiv.org/abs/2207.05221) — have you tried changing the prompt in 10 (line 359) to see if the intervention has a bigger effect in that case?\n\n5. Line 35 Relevant citation - https://arxiv.org/abs/2310.15910 \n\n6. Sec 7 — prior work (e.g. https://arxiv.org/abs/2310.18168) also shows that linear probes can be trained to predict truthfulness / correctness before the answer (just from the question), which would indicate similar findings as in section 7.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The result showing that directions identified using base model, causally affect refusal in chat-finetuned models is very interesting — the transfer from base to instruction/chat finetuned model is not obvious, and indicates a mechanism for how refusal finetuning can generalize beyond the specific entities it was finetuned on. \n\n2. All of the experiment and results figures seem thorough and convincing (e.g. error bars, useful and clear visualizations etc.)\n\n3. The paper also shows that interestingly, the same entity recognition directions regulate the model's attention mechanism (the factual recall mechanism) and directly express knowledge uncertainty (line 355, although the effect is not as strong).", "weaknesses": "1. Definition of known and unknown entities — the paper categorizes entities as known if it gets at least two attributes right, and unknown if it gets all attributes wrong. While this is fine, it does leave open the possibility that what is currently categorized as unknown is actually known by the model and can perhaps be elicited with a better prompt or few-shot examples. Alternatives to the current categorization could be – (a) checking across a diverse set of prompts to decide known and unknown as in https://arxiv.org/abs/2405.05904 ; (b) relying on n-gram matching in the pretraining data (would need to use LLMs with open pretraining data) ; (c) focusing on entities which occur after the knowledge cutoff for the unknown category. \n\n2. Explanation for existence of generalized latents that distinguish between known and unknown entities — the paper currently suggests that this is evidence of ‘self-knowledge’. An alternative and much simpler explanation which currently can’t be ruled out is that these latents are activated on high likelihood tokens (known) and low likelihood tokens (unknown). It would be useful to show some kind of control experiment to show that is not the case. \n\n3. (minor) In general, I also find it hard to directly interpret the results in Fig 2. The min-max value peaks at around 0.4 for unknown. How high should that value be to conclude the existence of generalized latents? I feel like if the set of the entities is small enough and the set of latents large enough, then there might always exist such latents? (Fwiw the additional results later do make it convincing that these are generalized latents)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QNZSPEP6jt", "reviewer": "Reviewer_fe4T", "summary": "The paper revealed that the knowledge of a large language model can be detected from a sparse latent space. There are several important findings:\n\n- The last token of known and unknown entities have similar latent representation respectively in a sparse latent space across different entity types.\n\n- The latent representation of entities has an influence on LLM generation and steering the representations will affect the refusal and fact recover behaviors of LLMs.\n\n- The latent representation can predict the correctness of generated answers.", "questions": "How does the model size influence the conclusion of the study?", "limitations": "", "rating": 10, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The findings of the paper (see summary) are well supported by the experiments and can provide valuable scientific insights to the community. The presentation is great and readers can easily follow.", "weaknesses": "The experiments are now only for Gemma models. It would be interesting to see the model behavior on other LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SQ1k3eHoUe", "reviewer": "Reviewer_sA3L", "summary": "This paper investigates the mechanisms behind LLM hallucinations by using sparse autoencoders -- SAEs to analyze internal model representations. The key finding is the discovery of \"entity recognition\" directions in the representation space that detect whether a model can recall facts about specific entities. These directions generalize across different entity types (movies, cities, players, songs) and causally influence whether the model refuses to answer questions or hallucinates information. More interestingly, the directions found with a base model can transfer to a chat model and control the model's tendency in refusing to answer questions.", "questions": "- Can you provide more ablation study to show that it's really necessary to use both 1) sparse and 2) auto encoder modules to get the interpretability?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Empirically link the \"entity recognition\" directions and the models' refusal in answering questions\n- The experiment includes clear experimental design using diverse entity types\n- Shows interesting discovery that base model mechanisms are repurposed during chat model fine-tuning", "weaknesses": "- **Lack of comparison to methods other than SAEs**: If we remove the sparsity of the SAE and reduce the module to a regular AutoEncoder, would the same properties hold? If we further remove the auto encoder and just use a linear probing module, would the same properties hold? Is it really necessary to use sparse and auto encoder modules to get the interpretability? If we borrow the same idea from ITI [1] to steer model towards a more truthful direction in the latent space, can the same property holds? Lack of ablation on the necessity of SAE is a big flaw of this paper.\n- **Lack of model diversity**: Only the Gemma models are tested. It would be better if we can see the same findings on LLaMA or other models to show how universal this idea holds.\n\n[1] https://arxiv.org/abs/2306.03341", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)