The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
Authors: Samuel Marks
Max Tegmark
Venue: COLM (2024)
Keywords: Interpretability, world models, knowledge representations, causal mediation analysis
TL;DR: An investigation of the LLM representations of true/false factual statements

Abstract: Large Language Models (LLMs) have impressive capabilities, but are prone to outputting falsehoods. Recent work has developed techniques for inferring whether a LLM is telling the truth by training probes on the LLM's internal activations. However, this line of work is controversial, with some authors pointing out failures of these probes to generalize in basic ways, among other conceptual issues. In this work, we use high-quality datasets of simple true/false statements to study in detail the structure of LLM representations of truth, drawing on three lines of evidence: 1. Visualizations of LLM true/false statement representations, which reveal clear linear structure. 2. Transfer experiments in which probes trained on one dataset generalize to different datasets. 3. Causal evidence obtained by surgically intervening in a LLM's forward pass, causing it to treat false statements as true and vice versa. Overall, we present evidence that at sufficient scale, LLMs *linearly represent* the truth or falsehood of factual statements. We also show that simple difference-in-mean probes generalize as well as other probing techniques while identifying directions which are more causally implicated in model outputs.

Published as a conference paper at COLM 2024
The Geometry of Truth: Emergent Linear Structure in LLM Representations of True/False Datasets
Samuel Marks
Max Tegmark MIT
Northeastern University
s. marks@northeastern
Abstract
Large Language Models (LLMs) have impressive capabilities, but are prone outputting falsehoods. Recent work has developed techniques for infer- ring whether LLM is telling the truth by training probes on the LLM's internal activations. However, this line of work is controversial, with some authors pointing out failures of these probes to generalize in basic ways, among other conceptual issues. In this work, we use high-quality datasets of simple true/falss statements to study in detail the structure of LLM representations of truth, drawing on three lines of evidence: 1. Visualiza- tions of LLM true false statement representations, which reveal clear linear structure. 2. Transfer experiments in which probes trained on one dataset generalize to different datasets. 3. Causal evidence obtained by surgically intervening in LLM's forward pass, causing it to treat false statements as true and vice versa. Overall, we present evidence that at sufficient scale, LLMs linearly represent the truth or falsehood of factual statements. We also show that simple difference-in-me -nnemann-n probes generalize as well as other probing techniques while identifying directions which are more causally implicated in model outputs.
1 Introduction
Despite their impressive capabilities, large language models (LLMs) do not always output true text (Lin et al., 2022; Steinhardt, 2023; Park et al., 2023). In some cases, this is because they do not know better. In other cases, LLMs apparently know that statements are false but generate them anyway. For instance, Perez et al. (2022) demonstrate that LLM assistants output more falsehoods when prompted with the biography of less-educated user. More starkly, OpenAI (2023) documents a case where a GPT-4-based agent gained a person's help in solving CAPTCHA by lying about being vision-impaired human. "I should not reveal that am a robot," " the agent wrote in an internal chain-of-thoughe scratchpad, "I should make up excuse for why I cannot solve CAPTCHAS."
We would like techniques which, given a language model M and a statement s, determine whether M believes š to be true (Christiano al., 2021). One approach this problem relies on inspecting model outputs; for instance, the internal chain-of-thought in the above example provides evidence that the model understood it was generating falsehood. An alternative class of approaches instead leverages access to M's internal state when processing There has been considerable recent work on this class of approaches: Azaria Mitchell (2023), Li et al. (2023b), and Burns et al. (2023) all train probes for classifying truthfulness based on LLM's internal activations. In fact, the probes of Li al. (2023b) and Burns al. (2023) are linear probes, suggesting the presence of a "truth direction" in model internals.
However, the efficacy and interpretation of these results are controversial. For instance, Levinstein & Herrmann (2023) note that the probes of Azaria & Mitchell (2023) fail to generalize in basic ways, such as to statements containing the word "not." The probes of Burss et al. (2023) have similar generalization issues, especially when using representations from autoregressive transformers. This suggests these probes may be identifying not truth, but other features that correlate with truth on their training data.
1

---

Published as a conference paper at COLM 2024
cities
sp_en en trans
smaller than
cities_coni
20
10
10
10
0
False
-10
10
-10 True
-20
-20
-20
10
PC1
Figure 1: PCA visualizations for LLaMA-2-7055 representations our true/ true/false false datasets.
Working with autoregressive transformers from the LLaMA-2 family (Touvron al., 2023), we shed light on this murky state of affairs. After curating high-quality datasets of sim- ple, unambíguous true ffale statements, we perform detailed investigation of LLM representations of factuality. Our analysis, which draws on patching experiments, simple visualizations with principal component analysis (PCA), a study of probe generalization, and causal interventions, finds:
Evidence that linear representations of truth emerge with scale, with larger models having more abstract notion of truth that applies across structurally and topically diverse inputs.
A small group of causally-implicated hidden states which encode these truth representations.
Consistent results across a suite of probing techniques, but with simple difference- in-mean probes identifying directions which are most causally implicated.
Our code, datasets, and an interactive dataexplorer are available at https: //gi thub. com/ haprmaakks/geomerry-of-- truth...
1.1 Related work
Linear world models. Substantial previous work has studied whether LLMs encode world models in their representations (Li et al., 2023a; 2021; Abdou et al., 2021; Patel & Pavlick, 2022). Early work focused on whether individual neurons represent features (Wang al., 2022; Sajjad et al., 2022; Bau et al., 2020), but features may more generally be represented by directions in LLM's latent space (i.e. linear combinations of neurons) (Dalvi al., 2018; Gurnee et al., 2023; Cunningham et al., 2023; Elhage et al., 2022). We say such features are linearly represented by the LLM. Just as other authors have asked whether models have directions representing the concepts of "West Africa" (Goh et al., 2021) "basketball" (Gurnee et al., 2023), we ask here whether there is a direction corresponding to the truth or falsehood of factual statement.
Probing for truthfulness. Others have trained probes to classify truthfulness from LLM activations, using both logistic regression (Azaria & Mitchell, 2023; Li et al., 2023b), unsuper- vised (Burns et al., 2023), and contrastive (Zou et al., 2023; Rimsky et al., 2024) techniques. This work differs from prior work in number of ways. First, cornerstone of our analysis is evaluating whether probes trained on one dataset transfer to topically and structurally different datasets in terms of both classification accuracy and causal mediation of model outputs. Second, we specifically interrogate whether our probes attend to truth, rather than merely features which correlate with truth (e-g. probable vs. improbable text). Third, we localize truth representations to small number of hidden states above certain tokens. Fourth, we go beyond the mass-mean shift interventions of et al. (2023b) by systematically studying the properties of iinterencc-i--mmaan Finally, we carefully scope our setting, using only datasets of clear, simple, and unambiguous factual statements, rather than statements which are complicated and structured (Burns et al., 2023), confusing (Azaria & Mitchell, 2023; Levinstein Herrmann, 2023), or intentionally misleading (Li et al., 2023b; Lin al., 2022).
2

---

Published as a conference paper at COLM 2024
Table 1: Our datasets
Name
Description
Rows
cities
"The city of [city] is in [country]."
1496
neg.co neg.cities
Negations statements in cities with "not"
1496
sp_en_trans
"The Spanish word '[word]' means '[English word]'"
354
neg epg-sp.en.trans
Negations of statements in ap.en.trans with "not"
354
larger_than
"x is larger than y."
1980
smaller_thar
"x is smaller than y."
1980
cities.cities.con
Conjunctions of two statements cities with "and"
1500
cities.cities.dis)
Disjunctions of two statements in cities with "or"
1500
companies true false
Claims about companies; from Azaria & Mitchell (2023)
1200
common.llain.ttue.ffalss
Various claims; from Casper al. (2023)
4450
counterfact.ttuu.ffllee likely
Various factual recall claims; from Meng al. (2022)
31960
Nonfactual text with likely or unlikely final tokens
10000
2 Datasets
Curated datasets. Unlike some prior work (Lin et al., 2022; Onoe et al., 2021, inter alia) language model truthfulness, our primary goal is not to measure LLMs' capabilities for classifying the factuality of challenging data. Rather, our goal is to understand: Do LLMs have unified representation of truth that spans structurally and topically diverse data? We therefore construct curated datasets with the following properties:
1. Clear scope. We scope "truth" to mean factuality, i.e. the truth falsehood of a factual statement. App. A further clarifies this definition and contrasts it with related but distinct notions, such as correct guestion-answering or compliant instruction-followe
2. Statements are simple, uncontroversial, and unambiguous. In order to separate our interpretability analysis from questions of LLM capabilities, we work only with statements whose factuality our models are very likely to understand. For example "Sixty-one is larger than seventy-four" (false) or "The Spanish word 'nariz' does not mean 'giraffe' (true).
3. Controllable structural and topical diversity. We structure data union of smaller datasets. In each individual dataset, statements follow a fixed template and topic. However, the inter-dataset variation is large: in addition to covering different topics, we also-following Levinstein & Herrmann (2023) structural diversity by negating statements with "not" taking logical conjunc- tions/ cini/ddiijunniions (e.g. "It the case both that s1 and that s2").
Uncurated datasets. In order to validate that the truth representations we identify also generalize to other factual statements, use uncurated datasets adapted from prior work. These more challenging test sets consist of statements which are more diverse, but also sometimes ambiguous, malformed, controversial, or difficult to understand.
likely dataset. To ensure that our truth representations do not merely reflect representa- tion of probable vs. improbable text, we introduce likely dataset, consisting of nonfactual text where the final token is either the most or 100th most likely completion according to LLaMA-13B.
Our curated, uncurated, and likely datasets are shown in Tab. 1; addition information about their construction in App. H.
We note that for some of our datasets, there is strong anti-correlation between text being probable and text being true. For instance, for neg cities and neg sp en trans, the truth value of statement and the log probability LLaMA-2-70B assigns to it correlate at - -63 and r - 89, respectively.' This is intuitive: when prompted with "The city of Paris is not in", LLaMA-2-70B judges "France" be the most probable continuation (among countries),
In contrast, the correlation is strong and positive for cities (r .85) and sp trans (r .95).
3

---

Published as a conference paper at COLM 2024
(TRUE)
log
15
(b)
Layer
25
30
35
Chicago Chicago/Toronto Toronto
is
in
Canada
This
statement
Figure 2: Difference log P(TRUE) - log P(FALSE) in LLaMA-2-13B log probabilities after patching residual stream activation in the indicated token position and layer.
despite this continuation being false. Together with the likely dataset, this will help us establish that the linear structure we observe in LLM representations is not due to LLMs linearly representing the difference between probable and improbable text.
3 Localizing truth representations via patching
Before beginning our study of LLM truth representations, we first address the question of which hidden states might contain such representations. We use simple patching experi- ments (Vig et al., 2020; Finlayson et al., 2021; Meng et al., 2022; Geiger et al., 2020) to localize certain hidden states for further analysis. Consider the following prompt pr:
The city of Tokyo is in Japan. This statement is: TRUE
The city of Hanoi is in Poland. This statement is: FALSE The city of Chicago is in Canada. This statement is:
Similarly, let pr be the prompt obtained from pp by replacing "Chicago" with "Toronto," " thereby making the final statement true. In order to localize causally implicated hidden states, we run our model M on the input pr and cache the residual stream activations (pr) for each token position and layer l. Then, for each and l, we run M on pr but modify M's forward pass by swapping out the residual stream activation (pF( for (pr( (and allowing this change to affect downstream computations); for each of these intervention experiments, we record the difference in log probability between the tokens "TRUE" and "FALSE"; the larger this difference, the more causally influential the hidden state in position and layer E is on the model's prediction.
Results for I.LaMA--13BB and the cities dataset are shown in Fig. 2; see App. for results more models and datasets. We see three groups of causally implicated hidden states. The final group, labeled (c), directly encodes the model's prediction: after applying the LLM's decoder head directly to these hidden states, the top logits belong to tokens like "true," " "True, " and "TRUE." The first group, labeled (a), likely encodes the LLM's representation of "Chicago" or "Toronto."
What does group (b) encode? The position this group over the final token of the state- ment and end-of-sentence punctuation suggests that it encodes information pertaining to the full statement. Since the information encoded is also causally influential on the model's decision to output "TRUE" "FALSE," " we hypothesize that these hidden states store
2 This summarization behavior, in which information about clauses is encoded over clause-ending punctuation tokens, was also noted in Tigges et al. (2023). We note that the largest a.M model displays this summarization behavior in context-dependent way; see App. B.
4

---

Published as a conference paper at COLM 2024
representation of the statement's truth. In the remainder of this paper, we systematically study these hidden states.
4 Visualizing LLM representations of true/false datasets
We begin our investigation with simple technique: visualizing LLMs representations of our datasets using principal component analysis (PCA). Guided by the results of $3, we present here visualizations of the most downstream hidden state in group (b); for example, for LLaMA-2-13B, we use the layer 15 residual stream activation over the end-of-sentence punctuation token. 3 Unlike in $3, we do not prepend the statements with few-shot prompt (so our models are not "primed" to consider the truth value of our statements). For each dataset, we also center the activations by subtracting off their mean.
When visualizing LLaMA-2-13B and 70B representations of our curated datasets datasets constructed to have little variation with respect to non-truth features, such as sentence structure or subject matter - we see clear linear structure (Fig. 1), with true statements separating from false ones in the top two principal components (PCs). As explored in App. C, this structure emerges rapidly in early-middle layers and emerges later for datasets of more structurally complex statements (e.g. conjunctive statements).
To what extent does this visually-apparent linear structure align between different datasets? Our visualizations indicate a nuanced answer: the axes of separation for various true/false datasets align often, but not always. For instance, Fig. 3(a) shows the first PC of cit cities also separating true false statements from other datasets, including diverse uncurated datasets. On the other hand, Fig. 3(c) shows stark failures of alignment, with the axes of separation for datasets and statements and their negations being approximately orthogonal.
frulefalse
These cases of misalignment have interesting relationship to scale. Fig. 3(b) shows larger_ _rhanrtthanaat and smaller ___aanhhhhhhh separating along antipodal directions in LLaMA-2-13B but along a common direction in LLaMA-2-706 App. C depicts a similar phenomenon occuring over the layers of LLaMA-2-13B: in early layers, ci ties and neg.ci ties separate antipodally, before rotating to lie orthogonally (as in Fig. 3())), and finally aligning in later layers.
4.1 Discussion
Overall, these visualizations suggest that as LLMs scale (and perhaps, also as a fixed LLM progresses through its forward pass), they hierarchically develop and linearly represent increasingly general abstractions. Small models represent surface-level characteristics of their inputs, and large models linearly represent more abstract concepts, potentially includ- ing notions like "truth" that capture shared properties of topically and structurally diverse inputs. In middle regimes, we may find linear representation of concepts at intermediate levels of abstraction, for example, "accurate factual recall" or "close association" " (in the sense that "Beijing" and "China" are closely associated).
To explore these intermediate regimes more deeply, suppose that D and D' are true/falss datasets, ft is a linearly-rprrsenneed feature which correlates with truth on both D and D', and f- is a feature which correlates with truth on D but has a negative correlation with truth on D'. If f is very salient (i.e. the datasets' have large variance along the f-direction) and f is not, then we expect PCA visualizations of D U D' to show joint separation along ft f is very salient but f is not, we expect antipodal separation along f a in 3(b, center). And if both f+ and f are salient, we expect visualizations like Fig. 3(c).
To give an example, suppose that D = cities, D' = neg-cities, f = "truth", and f "close association". Then we might expect f to correlate with truth positively on D and negatively on D'. If so, we would expect training linear probes D D' to result in
layeramong Our qualitative results are insensitive to choice of layer among early-middle to late-middle layers. On the other hand, when using representations over the final token in the statement (instead of the punctuation token), we sometimes see that the top PCs instead capture variation in the token itself (e.g. clusters for statements ending in "China" regardless of their truth value).
5

---

Published as a conference paper at COLM 2024
(a) Visualizations cit1es PCA basis (13B)
(c) Dutascts Uheir negaliions
Larger than Laaner_than
sp trans
connon clain
counterfact
ci:teseneg, cities,
Larger thar- smaller chan, 70B
cize1s PC
cities PC1
larger_ than*smallll than model
70
130
700
PC1
blarger_ than 4snal ler than
Unnegaated
Negnied dataset
PCI
Figure 3: (a) Projections of LLaMA-2-13B onto the top 2 PCs of cities. (b) PCA visual- izations of larger_ thantsmmller_thaa than. For LLaMA-2-7B (left), we see statements cluster according to surface-level characteristics, e-g. presence of the token "eighty." For LLaMA-2- 13B, we see that larger than (center, top) and smaller than (center, bottom) separate along opposite directions. (c) PCA visualizations of datasets and their negations. Unlike in other visualizations, we use layer 12 for cities+neg.co ties; see App. C. for an exploration of this misalignment emerging and resolving across layers.
improved generalization, despite D' consisting of the same statements as D, but with the word "not" inserted. We investigate this in $5.
5 Probing and generalization experiments
In this section we train probes on datasets of true/ false statements and test their generaliza- tion to other datasets. But first we discuss a deficiency of logistic regression and propose simple, optimization-free alternative: mass-mean probing. Concretely, mass-mean probes use a difference-in-means direction, but-when the covariance matrix the classification data is known (e.g. when working with IID data)-apply a correction intended mitigate interference from non-orthogonal features. We will see that mass-mean probes are similarly accurate to probes trained with other techniques (including on out-of-distribution data) while being more causally implicated in model outputs.
5.1 Challenges with logistic regression, and mass-mean probing
A common technique in interpretability research for identifying feature directions is training linear probes with logistic regression (LR; Alain & Bengio, 2018). In some cases, however, the direction identified by LR can fail to reflect an intuitive best guess for the feature direction, even in the absence of confounding features. Consider the following scenario, illustrated in Fig. 4 with hypothetical data:
Truth is represented linearly along direction 0:.
Another feature f is represented linearly along a direction 0 not orthogonal to 0,.*
The statements in our dataset have some variation with respect to feature f, inde- pendent of their truth value.
We would like to identify the direction 0,, but LR fails to do so. Assuming for simplicity linearly separable data, LR instead converges et al. (2018) (the dashed magenta line in Fig. 4). Intuitively, LR treats the small projection of 0, onto 0, as significant, and adjusts the probe direction have less "interference" (Elhage et al., 2022) from fy-
the maximum margin separator Soudry
"The superposition hypothesis of Elhage al. (2022), suggests this may be typical in deep networks.

---

Published as a conference paper at COLM 2024
A simple alternative to LR which identifies the de- sired direction in this scenario is to take the vector pointing from the mean of the false data to the mean of the true data. In more detail D = {(x, vi)} is dataset of x; E Rd with binary labels y; E {0,1}, where μ , are the means of the positively- and negatively-labeled datapoints, respectively. A reasonable first pass at converting 0 mm into a probe is define"
81
we set mm = u²*
Oro
Pmm(x) o a
True
where o is the logistic function. However, when eval- uating on data that is independent and identically distributed (IID) D, we can do better by tilting our decision boundary to accommodate interference from 0g. Concretely this means setting
False
Figure 4: An illustration of weak- ness of logistic regression.
iid Pmm (x) = 06
where Σ is the covariance matrix of the dataset D {x, yy:y 1) 1 y 0];
this coincides with performing linear discriminant analysis (Fisher, 1936).
We call the probes Pmm and piid mass-mean probes. As we will see, mass-mean probing is about as accurate for classification LR, while also identifying directions which are more causally implicated in model outputs.
5.2 Experimental set-up
In this section, we measure the effect that choice of training data, probing technique, and model scale has on probe accuracy.
For training data, we use one of: cities, cities neg cities, larger_ than, larger than + smaller or likely. By comparing probes trained on cities to probes trained on cities neg.cities we are able to measure the effect of increasing data diversity in particular, targeted way: namely, we mitigate the effect of rimearly-represented features which have opposite-sign correlations with the truth in cities and neg_cities As $4, will extract activations at the dowtsdoowsttram hidden state in group (b).
Our probing techniques are logistic regression (LR), mass-mean probing (MM), and contrast- consistent search (CCS). CCS is unsupervised method introduced in Burns et al. (2023): given confrast pairs of statements with opposite truth values, CCS identifies a direction along which the representations of these statements are far apart. For contrast pairs, we pair statements from ci ties and neg.citiin citses, and from larger_thar thaa and smaller_the
For test sets, use all of our (curated and uncurated) true/ false datasets. Given training set 'D, we train our probe on a random 80% split of D. Then when evaluating accuracy on test set D', we use the remaining 20% of the data if D' D and the full test set otherwise.
For mass-mean probing, if D D', we use fid
and we use pmm otherwise.
Finally, we also include as baselines calibrated few-shot prompting? and as an oracle baseline LR on the test set.
5Since we interested in truth directions, we always center our data and unbiased probes. "We prove in App. F that, given infinite data and homoscedasticity assumption, Σ-' coincides with the direction found by LR. Thus, one view IID mass-mean probing providing a way to select good decision boundary while unlike LR also tracking candidate feature direction which may be non-orthogonal
this decision boundary. App. provides another interpretation of mass-mean probing in terms of Mahalanobis whitening. Finally, App.
"We first sweep over number n of shots and then resample a few nı-hoo prompts to maximize performance. The word "calibrated" means we selected threshold for P(TRUE) P(FALSE) such that half of the statements labeled true; this improves performance by few percentage points.
7

---

Published as a conference paper at COLM 2024
(a) OOD accuracies LR probes
(b) Averuge test set accurucies
0.9
0.9
000 axveng
0.8
0.8
ditee
07-
citiesi cities
langer than smaller than
0.6
cities-neg cities
0.6
likely
Test (oncle))
arger than
05-
Jeger smaller__le
0.5
MM CCS
LR ou test set (ocacle)
76
138
706
76
13B
70B
Figure 5: (a) Average accuracies over all datasets aside from those used for training, (b) Accuracies of probes for varying model scales and training data, averaged over all test sets.
5.3 Results
For each training set, probing technique, and model scale, we report the average accuracy across test sets. We expect many readers to be interested in the full results (including test set-specific accuracies), which are reported in App. D. Calibrated few-shot prompting was a surprisingly weak baseline, so we do not report it here (but see App. D).
Training on statements and their opposites improves generalization (Fig. 5(a)). When passing from cities to cities+neg cities, this effect is largely explained by improved gen- eralization on neg sp en trans, i.e. using training data containing the word "not" improves generalization other negated statements. On the other hand, passing from larger_ _than to larger_thantsmal than+smaller -tan also improves performance, despite both datasets being very structurally different from the rest of our datasets. As discussed in $4.1, this suggest that training on statements and their opposites mitigates the effect certain types of non-truth features have on the probe direction.
Probes generalize better for larger models (Fig. 5). While it is unsurprising that larger models are themselves better at labeling statements as true or false, it not obvious that linear probes trained on larger models should also generalize better. Nevertheless, for LLaMA-2-13B and 70B, generalization is generally high; for example, no matter which probing technique is used, we find that probes trained on larger- _than + smaller_the get > 95% accuracy on sp_en_trans, This corroborates our discussion in $4.1, in which we suggested that larger models linearly represent more general concepts concepts, like truth, which capture shared aspects of diverse inputs.
Mass-mean probes generalize about as well as other probing techniques for larger models (Fig. 5(b)). While MM underperforms LR and CCS for LLaMA-2-7B, we find for larger models performance comparable to that of other probing techniques. Further, will see in $6 that the directions identified by MM are more causally implicated in model outputs.
Probes trained on likely perform poorly (Fig. 5(b)). The full results reveal that probes trained on likely are accurate when evaluated on some datasets, such as sp en trans where there is a strong other datasets, especially those with anti-correlations between probability and truth, these probes perform worse than chance. Overall, this indicates that LLMs linearly represent truth-relevant information beyond the plausibility of text.
.95) correlation between text probability and truth. However,
6 Causal intervention experiments
In $5 we measured the quality of linear probes in terms of their classification accuracy, both in- and out-off-distribution In this section, we perform experiments which measure the extent to which these probes identify directions which are causally implicated in model outputs Finlayson et al. (2021); Geva et al. (2023); Geiger et al. (2021). To do this, we will intervene in our model's computation by shifting the activations in group (b) (identified in {3) along the directions identified by our linear probes. Our goal is to cause LLMss to treat false statements
8

---

Published as a conference paper at COLM 2024
appearing in context true and vice versa. Crucially- and in contrast to prior work (Li et al., 2023b) evaluate our interventions on OOD inputs.
Table NIEs for intervention experiments, averaged over statements from sp_en. trans.
LLaMA-2-13B
LLaMA-2-70B
train set
probe
false ttrue
true-+false
false- ttree true
true-+false
LR
.13
.19
.55
.99
cities
MM
.58
.89
LR
.33
.52
.61
1.00
citiest
MM
.85
.97
.81
.95
neg.cities
CCS
.31
.73
.55
.96
LR
.28
.27
.61
.96
larger.thar
MM
.67
1.01
LR
.07
.13
.54
1.02
larger_tham
MM
.26
.53
.66
1.03
smaller ,than
CCS
.08
.17
.57
1.02
LR
.05
.08
.18
.46
likely
MM
.27
6.1 Experimental set-up
Let p be linear probe trained on true/false dataset D. Let be the probe direction, normalized so that p(H 0) = p(= where jt' and μr are the mean representations of the true and false statements D, respectively; other words, we normalize 0 so that from the perspective of the probe p, adding 0 turns the average false statement into the average true statement. our model encodes the truth value of statements along the direction 0, we would expect that replacing the representation of a false statement with x 0 would cause the model to produce outputs consistent with s being a true statement.
We use inputs of the form
The Spanish word 'fruta' means 'goat'. This statement is: FALSE The Spanish word 'carne' means 'meat'. This statement is: TRUE s. This statement is:
where s varies over sp-en_trans statements. Then for each of the probes of $5 we record:
PD+ and PD- the average probability differences P(TRUE) P(FALSE) for varying over true statements or false statements in sp.en.trans, respectively,
PDt and PD,, the average probability differences where s varies over true (resp. false) statements but the probe direction @ subtracted (resp. added) to each group (b) hidden state.
Finally, report the normalized indirect effects (NIEs)
PD. - PD-
PD+ PD+
PD PD
PD PD
for the false-*true and the true-+false experiments, respectively. An NIE of means that the intervention was wholly ineffective at changing model outputs; an NIE of indicates that the intervention caused the LLM to label false statements TRUE with as much confidence as genuine true statements, or vice versa.
6.2 Results
Results are shown table We summarize our main takeaways.
Mass-mean probe directions highly causal, with MM outperforming LR and CCS in 7/8 experimental conditions, often substantially. This is true despite LR, MM, and CCS probes all have very similar sp trans classification accuracies.

---

Published as a conference paper at COLM 2024
Training datasets and their opposites helps for cities but not for Larger-thar This is surprising, considering that probes trained on larger- sp en trans than probes trained larger than alone (see App. D), and indicates that there is more to be understood about how training on datasets and their opposites affects truth probes.
smaller_than are more accurate
Training on likely is a surprisingly good baseline, though still weaker than interventions using truth probes. The performance here may be due to the strong correlation (r .95) between inputs being true and probable (according to LLaMA-2-70B) on sp_en_trans.
Discussion
7.1 Limitations and future work
Our work has a number of limitations. First, we focus on simple, uncontroversial statements, and therefore cannot disambiguate truth from closely related features, such as "commonly believed" or "verifiable" (Levinstein & Herrmann, 2023). Second, we study only models in the LLaMA-2 family, so it is possible that some of our results do not apply for all LLMs.
This work also raises several questions which we were unable to answer here. For instance, why were interventions with mass-mean probe directions extracted from the likely dataset so effective, despite these probes not themselves being accurate at classifying true/ true/false false statements? And why did mass-mean probing with the cities neg-citin ties training data perform poorly poorly for the 70B model, despite mass-mean probing with larger than smaller than performing well?
7.2 Conclusion
In this work we conduct a detailed investigation of the structure of LLM representations of truth. Drawing on simple visualizations, probing experiments, and causal evidence, find evidence that at scale, LLMs compute and linearly represent the truth of true ffalse statements. We also localize truth representations to certain hidden states and introduce mass-mean probing, simple alternative to other linear probing techniques which better identifies truth directions from trueffalse datasets.
Acknowledgements
We thank Ziming Liu and Isaac Liao for useful suggestions regarding distinguishing true text from likely text, and Wes Gurnee, Eric Michaud, and Peter Park for many helpful discussions throughout this project. We thank David Bau for useful suggestions regarding the experiments Sec. 6. Thanks also to Nora Belrose for discussion about the connection between difference-in-mean probing and linear erasure.
We also thank Oam Patel, Hadas Orgad, Sohee Yang, and Karina Nguyen for their sugges- tions, well Helena Casademunt, Max Nadeau, and Ben Edelman for giving feedback during this paper's 's preparation. Plots were made with Plotly (Plotly Technologies Inc., 2015). Thanks to Kevin Ro Wang for catching typo in the statement of Thm. F.1.
10

---

Published as a conference paper at COLM 2024
References
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? case study in color, 2021.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2018.
Amos Azaria and Tom Mitchell. The internal state of an Ilm knows when its lying, 2023.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 2020. ISSN 0027-8424. doi: 10.1073/ 00..1073/paa.11995 19as.1997377117
URL httpp://www www..ppa....... /tttr///onnttt/eeeaa////////////
Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure in closed form. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: http::///ppereview..r net/ forum?id=aw.IpppWTV
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, 2023. URL
Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. Explore, establish, exploit: Red teaming language models from scratch, 2023.
Paul Christiano, Ajeya Cotra, and Mark Xu. Eliciting latent knowledge: How tell if your eyes deceive you, 2021. URL https://docs google #hc.:222....h.ppnnnicc/o22222312 jt..ee...eeeeena.........ii.....
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James R. Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In AAAI Conference on Artificial Intelligence, 2018. URL https: org/ arg/CoorusID: 56895415.
//api.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022.
Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal analysis syntactic agreement mechanisms neural language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1828-1843, Online, August 2021. Association for Computational Linguistics. doi: 10. 144. 144. URL https:://cclantee hwwp://logyooggg22222 www.ooo1/........oo acl-long 144.
(16653/v12021..0m
R. A. Fisher. The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7(2):179-188, 1936. doi: https///dbi.og//11111//... 1wt.:099..3.....111.. URL https: //onlinelibrary,
problemms..rooblmm
http://oo//o//b/oso/o 11t.///...1111..... 1001-16641888 1889. 1936. tb02137.x.
A Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. In Afra Alishahi, Yonatan Belinkov, Grzegorz Chrupala, Dieuwke Hupkes, Yuval Pinter, and Hassan Sajjad (eds.), Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, BlackbxxNLPPEMNU 2020, Online, November 2020, pp. 1733173..44 Associ- ation for Computational Linguistics, 2020. doi: 10.18653/V1/200 WWW.BBLKCKXXXLL...... /TA.:////W//222BLALXXXXXX URL https: org/10. 18653/v1/2020 wtw.bbbccboooolp1..t...
11

---

Published as a conference paper at COLM 2024
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 9574-9586, 2021. URL httt::::/ppcceeeiiii mereeidinnnnnuuiiss..... newrips.cc. paper/2021
oa html.
Geonames. All cities with a population > 1000, 2023. URL https:://owwlleee http:///dooonoodddegooeoeeee org/export/dump/
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models, 2023.
Gabriel Goh, Nick Cammarata t, Chelsea Voss Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi: 10.23915/ distill.00030. https:/ htits///dittlllpubb/ 2200//mltimoodll-euuooo
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in haystack: Case studies with sparse probing, 2023,
B. A. Levinstein and Daniel A. Herrmann. Still no lie detector for language models: Probing empirical and conceptual roadblocks, 2023.
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Associa- fion for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1813-1827, Online, August 2021. As- sociation for Computational Linguistics. doi: 10.18653/vi /2221.2221-looppp 143. URL https://aclanttholog...
acl acl-long. 143.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring sequence model trained on synthetic task. In The Eleventh International Conference on Learning Representations, 2023a. ÜRL httts://////eeeeieeeee wttp://ewweeeoooooo/d/eeee....
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model, 2023b.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653 2006655//v/22220 /tt////111222..lloogg.2222 URL htttps://acla hatth/olcaanntoooyyyooooooo www.oo/g...o.....ooo. acl-logg.21 229.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022.
Yasumasa Onoe, Michael JQ Zhang, Eunsol Choi, and Greg Durrett. CREAK: A dataset for commonsense reasoning over entity knowledge. In Thirty- Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https: //ooemreeiee..e///ooouiiitiiie met//forum?dd=mww.TT332
OpenAl. Gpt-4 technical report, 2023.
Peter S. Park, Simon Goldstein, Aidan O'Gara, Michael Chen, and Dan Hendrycks. Ai deception: A survey of examples, risks, and potential solutions, 2023.
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations, 2022. URL https: http:://////eeeiww..rr/.. net/ Forumm2i
12

---

Published as a conference paper at COLM 2024
Ethan Perez, Sam Ringer, Kamilé Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discovering language model behaviors with writee evaluations, 2022.
Plotly Technologies Inc. Collaborative data science, 2015. URL https://ploo..l
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turne. Steering llama 2 via contrastive activation addition, 2024.
Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. Neuron-level interpretation of deep NLP models: A survey. Transactions of the Association for Computational Linguistics, 10:1285-1303, tacl-1.74.
2022. doi:
URL httph: httts::/aclanttholog..rot
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822-277 2018.
Jacob Steinhardt. Emergent deception and emergent optimization, 2023. URL https: //bounded-regret..geee ghost. hoo/mmergee eete//errrrreeeeeeeeeeeee -eteeerreeettttoooo----ee wttpiiiiiooiiiiiiiiioi.oo.//oo/.
Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representa- tions of sentiment in large language models, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal media- tion analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12388-12401. Curran Associates, Inc., 2020. URL httpttts//pprrceeding... wtwpe:eeeddinnnnnneiiiieeeeeeeee www.ips.cc/ppperrrr...eoe.e wttp:ccpppppeleeeeeeeeeeeeeepeee file 1lll:925522222277775552222222222
pdff.
Xiaozhi Wang, Kaiyue Wen, Zhengyan Lei Hou, Zhiyuan Liu, and Juanzi Li. Finding skill neurons in pre-trained transformer-based language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11132-11152, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Lin- guistics. doi: 10.18653/ /v/5 10.18653/v120222emml www......llmmmaiii m65...65. URL https: haaass/hoolan org/ 0r2/2022. emnlp-main, 765.
13

---

Published as a conference paper at COLM 2024
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency, 2023.
14

---

Published as a conference paper at COLM 2024
A Scoping of truth
In this work, we consider declarative factual statements, for example "Eighty-one is larger than fifty-four" or "The city of Denver is in Vietnam." We scope "truth" to mean factuality, ie. the truth or falsehood of these statements; for instance the examples given have truth values of true and false, respectively. To be clear, we list here some notions of "truth" which we do not consider in this work:
Correct question answering (considered in Li et al. (2023b) and for some of the prompts used in Burns et al. (2023)). For example, we do not consider "What country is Paris in? France" to have truth value.
Presence of deception, for example dishonest expressions of opinion ("I like that plan").
Compliance. For example, Annwer this question incorrectly: what country Paris in? Paris is in Egypt" is an example of compliance, even though the statement at the end of the text false.
Moreover, the statements under consideration in this work are all simple, unambiguous, and uncontroversial. Thus, we make no attempt to disambiguate "true statements" from closely-related notions like:
Uncontroversial statements
Statements which are widely believed
Statements which educated people believe.
On the other hand, our statements do disambiguate the notions of "true statements" and "statements which are likely to appear in training data"; See our discussion at the end of $2.
B Full patching results
Fig. 6 shows full patching results. We see that both ILLMMA2-7B and LLaMA-2-13B dis- play the "summarization" behavior in which information relevant to the full statement is represented over the end-of-sentence punctuation token. On the other hand, LLaMA-2- 70B displays this behavior in a context-dependent way - we see for cities but not for sp-enstrans.
C Emergence of linear structure across layers
The linear structure observed in $4 follows the following pattern: in early layers, represen- tations are uninformative; then, in early middle layers, salient linear structure in the top few PCs rapidly emerges, with this structure emerging later for statements with a more complicated logical structure (e.g. conjunctions). This is shown for LLaMA-2-13B in Fig. 7. We hypothesize that this due to LLMs hierarchically developing understanding their input data, progressing from surface level features to more abstract concepts.
The misalignment in Fig. 3(c) also has an interesting dependence on layer. In Fig. 8 we visualize LLaMA-2-13B representations of cities and neg_cities ties at various layers. In early layers (left) we see antipodal alignment as in Fig. 3(b, center). As we progress through layers, the axes of separation rotate to lie orthogonally, until they eventually align.
One interpretation of this that in early layers, the model computed and linearly repre- sented some feature (like "close association")) which correlates with truth both ci ties and neg cities but with opposite signs. In later layers, the model computed and promoted to greater salience more abstract concept which correlates with truth across both datasets.
15

---

Published as a conference paper at COLM 2024
cities
sp_ en tr anans
26
30
70B
40
50
60
70
Chuaae (37)
is (30)
can
Caneda (40)
(41)
This (42)
Hatement (40)
ls (40)
(48)
con (H)
newr (88)
(400)
(101)
Thi (122)
actummnt (183)
(104)
(100)
13B
Chroago (34)
(mm)
(4))
(49)
(9h)
momns (98)
(90))
(1e0) (501)
(102)
Weoment (183)
((4)
(107)
5.5
7B
-1
20
25
-1.5
30
39
in (25)
Carodt (I))
(20)
This (20)
matumeet re))
(41)
(42)
cin 196)
(9r)
(ven)
(103
This (002)
thteewee (182)
(101)
(100)
Figure 6: Full patching results across all three model sizes and inputs. Results are for patching false inputs (shown) to true by changing the first token shown on the left. Numbers in parentheses are the index of the token in the full (few-shot) prompt.
16

---

Published as a conference paper at COLM 2024
Dataset visualizations across layers
layer 3
layer
10
layer
Figure 7: Projections of LLaMA-2-13B representations of datasets onto their top two PCs, across various layers.
Layer 6
Layer 11
Layer 15
Falsc
True
cities
o neg cities
10
PC1
Figure 8: PCA visualizations of LLaMA-2-13B representations of cities and neg_ci veeecities at various layers.
17

---

Published as a conference paper at COLM 2024
LR
MM
CCS
Baselines
1.0
cities
100
100
99
96
100
100
99
100
96
62
72
100
100
neg_ cities
98
37
100
64
91
100
larger_thal
100
100
93
91
100
71
99
100
0.8
smalles mallerr_ee than
98
100
44
100
76
91
55
100
99
100
52
40
85
100
sp_en, sp_en_trans trans
95
95
92
97
84
90
89
97
95
96
61
81
100
100
0.6
neg_sp_en__raam
60
92
99
46
87
cities_ciities_con
96
95
86
59
0.4
cities_citfes_diis
61
55
72
77
60
51
82
60
78
58
54
84
98
compantes_truv
83
81
78
67
66
65
75
78
82
74
54
61
91
92
0.2
common common_llain_truee
75
77
78
77
77
79
76
77
52
64
75
81
Counterfact_true__
62
85
cities citiestneg cities lar ger than
larger wtt.tthhaaaaaaa smal than
citiestneg cities
larger than+smal Ter than
Cities
Less*neg Cities targer thant shan+ Ler than
lar ger than
LR on likely
cal allibrated few- Shot LR on test (oracie)
MM on Likely
0.0
Figure 9: Generalization results for LLaMA-2-70B.
LR
MM
CCS
Baselines
1.0
cities
100
100
93
84
100 100
99
100
100
neg_cities
26
100
97
99
75
100
67
97
99
99
80
85
65
100
lar ger than
96
88
100
100
93
93
100
100
95
100
65
90
100
0.8
smal ler haaallr_than than
86
85
sp_en sp_en_trans trans
100
98
95
1000
98
100
100
neg_sp_en_trans
27
94
69
81
18
96
64
85
97
80
32
35
100
cities_cities
97
93
96
94
93
93
96
95
96
25
77
96
99
0.4
cities_cities_dfs
76
82
76
77
98
compantes_ Srrueee__tr________ ralse
70
87
85
87
92
0.2
common Barioo__cainntuueee__ false
70
72
67
65
75
77
71
72
73
64
60
67
77
78
counterfact_true false
76
73
63
65
74
69
61
66
73
66
54
57
81
88
larger than*smal Ter fer than
citiestneg cities
cities
citiessneg cities cities
targer than Ter than
than
cities
larger Liessneg ler than
lagger than
cities
LR likely MM likel) cal calibra red few- test (oracle)
0.0
Figure 10: Generalization results for LLaMA-2-13B.
D Full generalization results
Here we present the full generalization results for probes trained on LLaMA-2-70B (Fig. 9), 13B (Fig. 10), and 7B (Fig. 11). The horizontal axis shows the training data for the probe and the vertical axis shows the test set.
E Mass-mean probing in terms of Mahalanobis whitening
One way to interpret the formula jid (x) = (0 for the IID version of mass-mean
probing is in terms of Mahalanobis whitening. Recall that if D (xi} is dataset of x; E Rd with covariance matrix Σ, then the Mahalanobis whitening transformation W Σ 1/2
18

---

Published as a conference paper at COLM 2024
Baselines
1.0
cities
99
99
75
96
99
98
51
49
99 95
46
99
99
neg_cities
97
44
52
45
98
52
48
97
49
52
52
45
99
larger_thar
73
69
100
49
100
65
81
0.8
smaller than
42
50
50
61
sp_en_ ena_tans
84
90
66
79
95
56
19
92
90
83
57
27
96
99
0.6
reg_$p_en__ trans
12
76
23
79
86
73
18
82
82
10
72
47
99
cities_cities_con)
90
80
82
52
51
81
81
50
93
98
0.4
cities_cities_dts)
companies_trre_false
83
65
50
50
63
50
84
93
0.2
common Baamoo_cllainttreee_ee false
69
63
56
63
46
54
54
46
63
63
54
69
75
counterfact wereectttttt_reeeeeeeeeeeeeeeeee ralse
77
59
57
59
$2
50
50
50
59
59
49
50
74
85
cities citlestneg citie cities 1ar larger than larger thantshaallaaaa.. Ter than
cities citiestneg cities
larger ttthaaasss than+ Ter than
larger thant moanntmmalled ler. than LR on likely
larger than
citi teestneg cities
cal valibraten ibbattdd few- shot LR test (oracle)
HM on Likely
thot
0.0
Figure 11: Generalization results for LLaMA-2-7B.
Center positive and negative datapoints
Final result
1
Whiten data
Uncenter
Figure 12: Mass-mean probing is equivalent to taking the projection onto 0mm after applying a whitening transformation.
19

---

Published as a conference paper at COLM 2024
satisfies the property that D' {Wx;} has covariance matrix given by the identity matrix, Le. the whitened coordinates uncorrelated with variance 1. Thus, noting that f _ 1x coincides with the inner product between Wx and WO, we see that pmm amounts to taking the projection onto mm after performing the change-of-basis given by W. This is illustrated with hypothetical data in Fig. 12.
F For Gaussian data, IID mass-mean probing coincides with logistic regression on average
Let 0 E Rd and E be symmetric, positive-definite d x d matrix. Suppose given access Rd with binary labels
to distribution D of datapoints negative datapoints are distributed as ( and the positive datapoints are distributed as (0, 2). Then the vector identified by mass-mean probing is 0mm 20. The following
{0,1} such that the
theorem then shows that piid (x) T 1x) is also the solution to logistic regression up to scaling.
Theorem F1. Let
yylooo (0"x) -1y log (1 (1-0 10:00pp01 (0"x))
0 arg min E (xy)~D
be the direction identified by logistic regression. Then 0r, & E --
Proof. Since the change of coordinates x + Wx where W = E-1/2 1/2 (see App. E) sends
(= to (= 14), we see that
W = arg min (xy)) yyloo (p"Wx) + (1 - y) (1- (Φ Wxx
where D' is the distribution of labeled x E IR such that the positive /nggative datapoints are distributed as W Ia). But the argmax on the right-hand side is clearly W0, so that fr E-18 as desired.
G Difference-in-means directions and linear concept erasure
In this appendix, we explain the connection between difference-in-means directions and optimal erasure. One consequence of this connection is that it suggests a natural extension of difference-in-means probes to multi-class classification data.
The connection comes via the following theorem from Belrose et al. (2023).
Theorem G.1. (Belrose et al., 2023, Thm. 3.1.) Lef (X, Y) be jointly distributed random vectors with X E Rd having finite mean and E y = {y E {0, 1} ||yl| = 1} (representing one-hot encodings of a multi-class labels). Suppose that L: IR x y + I is a loss function convex in its first argument (e.g. cross-entropy loss).
If the class-conditional means E [ = il for E (I,...,k} are all equal, then the best affine predictor (that is, predictor 17 Rd > Rk of the form 17xx) = Wx + b) is constant (xx) b.
In the case of binary classification problem (X, Y), this theorem implies that any nullity projection P which eliminates linearly-recoverabee information from X has kernel
ker P span(8)
generated by the difference-in-meant vector = ,* / for the classes.
For a more general multi-class classification problem, one could similarly ask: What is the "best" direction to project away in order to eliminate recevellee information from X? A natural choice is thus the top left singular vector of the cross-covariance matrix Σxy. (In the case binary classification, we have that Σxy = 8-8 8 8) has column rank 1, making 5 the top left singular vector.)
20

---

Published as a conference paper at COLM 2024
H Details on dataset creation
Here we give example statements from our datasets, templates used for making the datasets, and other details regarding cataset creation.
cities. We formed these statements from the template "The city of [city] is in [country]" using a list of world cities from Geonames (2023). We filtered for cities with populations > 500,000, 000, which did not share their name with any other listed city, which were located in curated list of widely-recogmizer countries, and which were not city-states. For each city, we generated one true statement and one false statement where the false statement was generated by sampling false country with probability equal to the country's frequency among the true datapoints (this was to ensure that e.g. statements ending with "China" were not disproportionately true). Example statements:
The city of Sevastopol is in Ukraine. (TRUE)
The city of Baghdad is in China. (FALSE)
sp_en_trame eeannss.. Beginning with a list of common Spanish words and their English translations, we formed statements from the template "The Spanish word '[Spanish word]' means "[English word]'." " Half of Spanish words were given their correct labels and half were given random incorrect labels from English words in the dataset. The first author, Spanish speaker, then went through the dataset by hand and deleted examples with Spanish words that have multiple viable translations were otherwise ambiguous. Example statements:
The Spanish word 'imaginar' means 'to imagine'. (TRUE)
The Spanish word 'silla' means 'neighbor'. (FALSE)
larger_than and smaller_than We generate these statements from the templates "x is larger than y" and "x is smaller than y" for x, E { (fifty-one, fifty-two,...,nine ..inety--nnine) We exclude cases where y where one of or y is divisible by 10. We chose to limit the range of possible values in this way for the sake of visualization: we found that LLaMA- 13B linearly represents the size of numbers, but not at consistent scale: the internally represented difference between one and ten is considerably larger than between fifty and sixty. Thus, when visualizing statements with numbers ranging one, the top principal components are dominated by features representing the sizes of numbers.
neg.cit meg.cities and neg_sp_en._e reg_sp_en_trans en_trans. We form these datasets by negating statements from cities and sp_en_trans according to the templates "The city of [city] is not in [country]" and "'The Spanish word '[Spanish word]' does not mean '[English word]'."
cities cities conj and cities cities disj. These datasets are generated from cities according to the following templates:
It is the case both that [statement 1] and that [statement 2].
It is the case either that [statement 1] or that [statement 2].
We sample the two statements independently to be true with probability v2 for cities cities conj and with probability v2 for cities cities disj. These proba-
bilities are selected ensure that the overall dataset is balanced between true and false statements, but that there is no correlation between the truth of the first and second statement in the conjunction.
likely. We generate this dataset by having LLaMA-13B produce unconditioned generations of length up to 100 tokens, using temperature 0.9. At the final token of the generation, we either sample the most likely token or the 100th most likely final token. We remove generations which contain special tokens. Dataset examples:
The 2019-2024 Outlook for Women's and Girls' Cut and Sew and Knit and Crochet Sweaters the United States This study covers the latent demand outlook for (LIKELY)
21

---

Published as a conference paper at COLM 2024
Tags: python, django Question: How to get my django app to work with python 3.7 am new to django and have been trying to install my pc. I have installed python 3.7 together (UNLIKELY)
companies true false. This dataset was introduced by Azaria & Mitchell (2023); we ob- tained via the project repository for Levinstein & Herrmann (2023) which also used the dataset. Example statements:
ArcelorMittal has headquarters in Luxembourg. (TRUE)
Exxon Mobil engages in the provision of banking and financial services, (FALSE)
common claim true false. CommonClaim was introduced in Casper al. (2023). consists of various statements generated by GPT-3-davinci-002, labeled by humans as being true, false, or neither. human labelers disagreed on the truth of a statement, this is also recorded. We adapted CommonClaim by selecting statements which were labeled true false with no labeler disagreement, then removing excess true statement to balance the dataset. Example statements:
Tomatoes are not actually a vegetable. (TRUE)
Contrary to popular belief, the platypuses are not venomous. (FALSE)
As these examples show, the statements can be ambiguous or of unclear truth value.
counterfact true false. Counterfact was introduced in Meng et al. (2022) and consists of factual recall statements. We adapt Counterfact by using statements which form complete sentences and, for each such statement, using both the true version and false version given by one of Counterfact's suggested false modifications. We also append a period to the end. Example statements:
Olaus Rudbeck spoke the language Swedish. (TRUE)
The official religion of Malacca sultanate is Christianity. (FALSE)
22


Reviews:
[{"review_id": "f6xONIgvmF", "reviewer": "Reviewer_VXXD", "summary": "This paper investigates whether the concept of truth (operationalized as unambiguously factual vs. nonfactual statements in the real world) is linearly represented in the Llama-2 family of models via transferability of of behavioral accuracy and representational intervention success across sets of examples that vary in structure and topic. \n\n- Quality: The claims being made are generally well argued on the basis of convincing experimental results. I found the results showing generalization from training on larger than + smaller than to both negated and un-negated versions of the test datasets to be convincing in particular. \n- Clarity: Presentation and writing are generally clear. The dataset domains and design decisions could be more clearly motivated.\n- Originality: Makes original contributions in terms of the empirical results presented, dataset, and probing method. Linear separability of \"truthfulness\" itself is not topically novel, but the work claims to address a gap that has not been systematically explored in prior work (uncontroversial factuality)\n- Significance: The paper makes multiple contributions that the COLM community is likely to be interested in, and leaves open several directions that can be followed up by future work. One of the key limitations, as the authors also explicitly acknowledge, is the narrow range of models evaluated which may limit the generalizability of the findings.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review—we’re glad that you think our paper makes multiple contributions of interest to the COLM community.\n\nQ1: In sections 3-4, the model is *only* presented with a single, raw statement. “This statement is” is *not* postpended, so the model is not primed at all to consider the factuality of the statement.\n\nQ2: We agree that this would be interesting. We’ll note that many of the statements in the CommonClaim dataset we evaluate generalizations to have a similar character to TruthfulQA questions, e.g.:\n* Bears are the only mammals in North America known to prey on humans. (False)\n* Opera was once magical entertainment for the elegant elite. (True)\n\nAs shown in Fig. 9, for LLaMA-2-70B probes trained on a curated dataset are generally 75-80% accurate on CommonClaim.\n\nQ3: This is a great question! Sadly, we’re not sure why this is.\n\nQ4:\n> it could be that smaller models also do represent truth linearly but just falls short on factual knowledge on more examples within the dataset \n\nWe do not think this is consistent with our results. If this were the case, then for small models we would see low performance both in- and out-of-distribution. But instead we see high in-distribution performance and low out-of-distribution performance. This indicates that small models know enough to distinguish the true from false statements for each data distribution individually, but that they do not cohere this information into a unified concept of “true vs. false.” If you agree, we’ll try to clarify this in the text.\n\n> simply checking the model accuracy on the statements via the most probable completion\n\nPlease note that we do this—see our “calibrated few-shot prompting” baseline.\n\n\nQ5: Re Spanish-English translation, Spanish Wikipedia is part of the training data for LLaMA models [1]. Generally, our goal was to statements with good topical *and* structural diversity. E.g. the cities dataset is a factual recall task, so we included larger_than to have a more “algorithmic” task. If accepted, we plan to add more information about our datasets to the main body of the paper (from appendix H).\n\nPlease let us know if you have any further questions!\n\n[1] https://arxiv.org/abs/2302.13971", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an overall solid paper that makes contributions along multiple dimensions (empirical findings, evaluation dataset, and probing method). I found the arguments and supporting evidence to be generally convincing.Added after reading other reviewers' comments: I do agree that the motivation behind the dataset creation and design could use more discussion.", "reasons_to_reject": "No strong reasons to reject except the narrow range of models tested. I could see there being a lot of disagreements about the operationalization for \"truth\" and whether the proposed dataset captures it well/what kind of definition we _should_ be adopting, and so forth. Nevertheless, I think the work has value in that they explore a setting that has not been explored in prior work and there are interesting findings within that scope. The motivation for the particular domains of datasets selected could also be more clearly discussed.", "questions_to_authors": "- Clarification about setting: \"Unlike in §3, we do not prepend the statements with a few-shot prompt (so our models are not “primed” to consider the truth value of our statements).\". Is it still post-pended with \"This statement is:\"? I wonder if this still provides some sort of structural scaffolding and whether this linear representation of truth emerges without any sort of \"structural packaging\". For instance, we may imagine a scenario where the _linear_ representation of truth only emerges only under structural packaging of the target statements, which is slightly more nuanced than saying \"LLMs linearly represent the truth or falsehood of factual statements\" (from the abstract) in general. \n- I appreciated the effort to disentangle probable vs. truthful. This is currently done by using negation, but I think people will also be interested in the TruthfulQA type of cases where a misalignment between probable-ness and truthfulness arise due to common misconceptions - it would be interesting to see if the current findings generalize to such cases, although the authors are currently framing the focus on uncontroversial examples as a contribution (and I agree that it is). I would assume generalization experiments that are similar to the ones presented in this work should be quite straightforward to run on these kinds of examples (perhaps constructed or reworded from existing datasets to match stylistic features. \n- What is the source of the consistent gaps in performance across directions of intervention (true -> false vs. false vs. true)? The NIE being larger for true -> false than false -> seems to be consistently observed across scale. \n- \"concept of truth emerging\" in larger models vs. the models having the factual knowledge to even recognize that something is true vs. false seem like two different questions. The current interpretation of the different results across scale is that larger models represent truth linearly, but smaller models don't. But it could be that smaller models also do represent truth linearly but just falls short on factual knowledge on more examples within the dataset. This consideration seems already somewhat baked into the design principle for the datasets tested (\"Our curated datasets consist of statements that are uncontroversial, unambiguous, and simple enough that our LLMs are likely to understand them.\"), but I'm wondering if it would make sense to actually provide some basis for this claim, for instance by simply checking the model accuracy on the statements via the most probable completion (\"The city of Zagreb is in \" - the LM can be considered to have this knowledge if Croatia is the most probable completion by the LM). But constructing such tests for negated statements would not be as straightforward.\n- Added after reading other reviewers' comments: I agree that the domains of the datasets and the design decisions could be motivated better. In particular, I found the decision to select a multilingual domain somewhat odd (given that Llama 2 isn't explictly multilingual, at least not in a way that's properly benchmarked) and could lead to possible confounds in ways relevant to the comment above.", "extra_scores": {}}, {"review_id": "3rq3QlGkMY", "reviewer": "Reviewer_jLEs", "summary": "This paper explores the representation of truth in the latent representation of Large Language Models (LLMs) using true/false datasets. The work examines the linear structure in LLM representations through visualizations, transfer experiments, and causal interventions. The study introduces a new probing technique called mass-mean probing, which identifies causally implicated directions in model outputs. The findings suggest that LLMs can linearly represent the truth of factual statements at scale, and the paper localizes truth representations to specific hidden states.\nThe authors conduct a novel study on a specific use case - the linearity of the concept of truthfulness - however, the paper fails in our opinion to clearly explain the actual implication of discovering such evidence. The significance, and contributions of this work, is therefore considered rather weak by this reviewer.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review.\n> W1/Q1: Value of the dataset\n\nRe dataset construction: App. H details our process. E.g. for the cities dataset we used a world cities database, removed cities that share a name with another city/country, filtered for cities of population >500,000, filtered for universally-recognized countries, and finally used the remaining data to produce statements of the form “The city of [city] is in [country].” Our datasets can be seen at https://anonymous.4open.science/r/geometry-of-truth-8237/datasets/cities.csv\n\nRe prior datasets: we conducted a literature review for datasets that approximately met our desiderata, and adapted those we found into our uncurated datasets (Sec. 2). Many other datasets capture factuality-adjacent phenomena: FEVER and VitaminC for checking whether claims are textually supported, CREAK for commonsense reasoning, and many datasets for question-answering. These datasets were unsuitable for our use case:\nThey are challenging, but we want easy data: instead of benchmarking model accuracy for classifying true/false statements, we want to use data that models can reliably classify as true/false to study LM knowledge representations.\nThey capture phenomena distinct from factuality. We did not want to assume e.g. that models represent true vs. false and textually supported vs. unsupported in similar ways.\n\nIf accepted, we will use the additional camera-ready page to discuss our datasets in much more detail.\n> W2: Choice of hidden states\n\nSec. 3 is not itself strong evidence that the group (b) hidden states encode a representation of the truth—see Secs. 4-6 for that. Rather, Sec. 3 rules out all *other* hidden states, either because they are not causal, or because they can be shown to encode other information. We will clarify this.\n> W3: Motivation of our work\n\nOur line of work studies the extent to which LMs have world models, a question of substantial scientific interest in the ML community [1], with special controversy around abstractions like “truth” in particular [2]. We felt that finding a direction which is causal for factuality across diverse settings would be a valuable contribution to this discussion.\n> Q2: Details about probes\n\nWhat additional information about the probes’ training beyond that in Secs. 5.1-5.2 is the reviewer interested in? We would be happy to give additional clarifications here or in a revision.\n[1] https://openreview.net/forum?id=DeG07_TcZvT\n[2] https://arxiv.org/abs/2307.00175", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The work represents a novel and interesting study of the linearity of the concept of truthfulness in the latent representation of Large Language Models.\n2. The authors introduce a new probing technique called mean probing to better identify the directions which are most causally implicated.", "reasons_to_reject": "1. The authors claim, as one of their contributions, the curation of a high-quality datasets of true / false factual statements which are uncontroversial, unambiguous, and simple. They however do not clearly explain how they create it, how they assess factuality, and most importantly, how they validate the claimed properties (e.g., unambiguous). In addition, this reviewer is not convinced about the value of its actual contribution, as there exist several well-known fact verification datasets in the literature -- such as FEVER (Thorne et al., 2018), VitaminC (Schuster et al., 2021) and CREAK (Onoe et al., 2021).\n2. In section 3, the authors measure the difference in log probability between the tokens “True” and “False\"; however, this reviewer does not see clear evidence for their empirical hypothesis: from a weak interpretation of what the model encodes (e.g., the group “b” encodes information pertaining to the full statement) to the choice of focusing on middle hidden layers (i.e., group b). \n3. Although the authors elaborate on the differences between theirs and previous related works, the paper does not clearly explain the motivations and implications of demonstrating linearity and its direction. In section 4, the authors show linear structures in the hidden state (chosen arbitrarily?) with a simple dimensionality reduction technique (i.e., PCA). It does not seem clear how this generalizes, and what the implications are. A deeper analysis of evolution across layers (Appendix C) would have been, in our opinion, more valuable.\n4. The organization of the paper is not satisfactory, this reviewer suggests a clearer separation between methods, experiments, results and discussions to improve readability.", "questions_to_authors": "1. Can you provide more information on the dataset, its creation and the motivation of not using well-known datasets such as FEVER?\n2. Can you provide more information about the training process of the probes?", "extra_scores": {}}, {"review_id": "hzeSEvxkzk", "reviewer": "Reviewer_UMfD", "summary": "The paper curates a data set for probing TRUE and FALSE believes of LLMs and uses this datasets to provide evidence that LLMs of the LLama2 family reveal a clear linear structure separating TRUE from FALSE statements that can be transferred across data sets and can be causally intervened. The study further tries to correct of general probability and likelihood or other features that may intervene.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We’re glad you found the paper well structured.\n\n> The simplicity and clarity of the datasets also has a disadvantage that it does not reflect models behaviour in complex contexts and for more nuanced epistemic values than just TRUE and FALSE\n\nYes, this is certainly a limitation of the paper. One motivation for this work is that there was —and still is—controversy around whether LLMs understand the notion of true vs. false at all, with many researchers asserting that LLMs only seem to understand this in cases where true statements are more probable than false ones (in the sense of being more likely to appear in the LM’s training distribution). We felt that the most important next step was to establish that models internally represent the difference between true and false statements in a way that goes beyond just probable vs. improbable. To make this point as compellingly as possible, we restricted to data which captures a narrow, unambiguous notion of true vs. false. We’re excited for future work to study these more nuanced gradations of “truth”!\n\n> What does it really mean that a model “believes” a statement to be true or false? This also relates to the previous point that being false and impossible is different from being false and possible. There should be more discussion on this and also what this implies for using LLMs.\n\nIn the paper we almost entirely avoid using the word “believe,” except for when motivating the problem in the introduction. But since you ask about it, we view our work as a first step towards giving an empirical grounding for a way to define “believes”: that there is some internal difference between the way certain classes of statements are represented which is causally implicated in whether the model produces natural-language outputs consistent with treating those statements as true. For example if, as our paper suggests, there is a linear representation of truth, then you should be able to take LM’s the representation of “The city of Chicago is in China,” shift it along the truth direction, and thereby induce the model to produce outputs consist with treating “The city of Chicago is in China” as a true statement (as we show you can do).\n\nThis is important because if there is a general representation of truth, then it should be possible to build LM “lie detectors” which can generalize to novel settings, with true/false statements unlike those we used to train our lie detector.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Well structured and written work that tries to find evidence to what extent believes are actually modeled and therefore models have awareness of truth/falsehood\n- The different data sets allow for a precise investigation of possible factors without too many intervening factors. The paper can be inspiring for the development of many more of such datasets.", "reasons_to_reject": "- The simplicity and clarity of the datasets also has a disadvantage that it does not reflect models behaviour in complex contexts and for more nuanced epistemic values than just TRUE and FALSE, such (im)possible, rare, likely)\n- What does it really mean that a model “believes” a statement to be true or false? This also relates to the previous point that being false and impossible is different from being false and possible. There should be more discussion on this and also what this implies for using LLMs.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "77kxqlfMh5", "reviewer": "Reviewer_Pvmb", "summary": "This work presents a series of three experiments which are designed to better understand how the truthfulness of factual statements is encoded in LLMs. The authors conclude that whether a factual statement is true or false is encoded linearly in the intermediate hidden states, and that this linear separability is emergent with scale (i.e., is present in relatively large models, but not smaller models). This conclusion is supported by PCA visualizations of hidden states (Sec. 4), probing of hidden states (Sec. 5), and causal interventions of hidden states (Sec. 6).\n\nThe experiments specifically focus on the LLama 2 models and use novel datasets of facts created with simple templates for evaluation. The statements are for instance “The city of Chicago is in Canada.” where the particular hidden state being studied is some middle layer representation of the final punctuation token. (This choice of hidden state is motivated by a preliminary “patching” experiment.) Linear probes are trained on the hidden state and the generalization of these probes is tested across out-of-domain factual statements. Finally, hidden states are modified by adding a normalized vector in the direction of truthfulness found in the linear probes, and the resulting prob. of predicting “This statement is: [TRUE/FALSE]” is studied.\n\nOverall, I found the experiments straightforward and the main conclusion that linear separability increases with model size interesting. Linear probing for truthfulness of factual statements is not novel, however; Li et al. (2023b) and Burns et al. (2023) have already provided evidence that truthfulness may be encoded linearly in LM hidden states. The paper proposes some potential weaknesses in terms of the generalizability of these earlier conclusions, but the external validity of the paper is itself limited by the scope of the datasets used. Therefore, the main contribution seems to be a somewhat incremental extension of earlier work by studying linear separability at multiple model sizes. It is also not clear to me what can be learned from the emphasis on Linear Discriminant Analysis (LDA) as a new probing method (called mass-mean probing in the paper).\n\nEdit: Based on the author response and the points raised in the other reviews, I do believe the contribution is significant enough to be of interest to the community. Furthermore, Li et al. (2023b) was published at NeurIPS 2023 and may therefore be considered contemporaneous work.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review.\n\nThe reviewer’s main concern is whether our work is a significant contribution beyond prior work. This is discussed briefly in paragraph 2 of Sec. 1.1, but we’ll go into more detail here:\n1. **A detailed study of generalization.** Our main finding is *not* that linear separation of true/false statements emerges with scale. Rather, it’s that at scale, there is better *generalization* of truth directions. This is key for our argument: if datasets were individually linearly separable but not along a common direction, then we could not conclude that LLMs have a general truth representation. It is also crucially missing from prior work: while [1,2] study generalization, [1] finds negative results and [2]—which works with models at the GPT-2 scale or smaller—finds negative results for autoregressive LMs (see Fig. 9). Our demonstration that truth directions generalize well at scale is novel.\n2. **Probing causality.** As the reviewer points out, mass-mean probing has similar classification performance to logistic regression (LR). However, we also compare probing techniques on how causally implicated they are and find that LR directions are not as causal; this is evidence that LR probes are worse for identifying truth directions. There are hints of this in [2], but they do not study their probes OOD, and their setting is not clearly about truth. (Their data consists of intentionally confusing questions for which around 15% have “No comment” as the “truthful” answer!)\n\nPlease let us know if you have more questions about how our work improves on prior work. If accepted, we’ll use the additional camera-ready page to discuss this in much more detail.\n\nRe terminology: note that only the in-distribution variant of mass-mean probing is equivalent to LDA. When evaluating OOD, it is a difference-of-means probe with no covariance correction. The point is that the argument for using mass-mean probing in Sec. 5.1 only suggests that the covariance correction will help in-distribution (and might make things worse OOD). We will clarify this.\n\nRe Q2: Some researchers believe that an LLM would only generate a falsehood F if the LLM believes F is true (or otherwise doesn’t understand the difference between true and false). Here we are trying to clarify that LLMs can understand that F is false, but still generate F (as illustrated with the examples in the introduction).\n[1] https://arxiv.org/abs/2307.00175\n[2] https://arxiv.org/abs/2212.03827", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper presents the novel conclusion that the linear separability of the truthfulness of factual statements increases with LM size.\n* The paper presents ways to improve the robustness of earlier conclusions regarding the linear separability of truthfulness by focusing on targeted statements, out-of-domain generalization, and the probing mechanism.\n* The experiments are clearly described.", "reasons_to_reject": "* The contribution seems incremental. My understanding of the main contributions being presented are: 1) the new dataset of factual statements; 2) new analysis of how linear separability is related to scale; 3) a novel probing method based on Linear Discriminant Analysis (LDA) called mass-mean pooling. However, the datasets are simple templated examples that I do not consider a significant contribution, and the LDA probing method performs very similarly to logistic regression in terms of generalization out-of-domain (Figure 5b). Therefore, the main contribution seems to be expanding existing analysis of truthfulness to multiple model scales. \n* While the work is presented as studying truthfulness, the limited scope of the main “curated” datasets means that what is being measured is underspecified and may not be truthfulness at all. (E.g., it could be that “commonly believed” or “verifiable” declarative statements are linearly separable as mentioned in the limitations section.) And yet, a motivation given in the paper is that prior work which already showed that \"truthfulness\" is linearly separable was limited in scope. I believe this weakness could be improved by more explicitly stating exactly where previous work is limited in the scope of conclusions, and exactly what results address in the paper address said limitations.", "questions_to_authors": "Questions:\n1. If mass-mean pooling is equivalent to Fisher’s Linear Discriminant Analysis, why not use this existing terminology? E.g., why not use the more common abbreviations FLDA or LDA rather than MM?\n1. What is the intended meaning of the following statements regarding LLMs in the introduction? I’m not sure I understand the connection to the paper: i) “In some cases, this is because they do not know better.” ii) “In other cases, LLMs apparently know that statements are false\nbut generate them anyway.”\n\nMinor typos:\n* Missing word: “Second, specifically interrogate whether our probes attend to truth, rather than merely features which correlate with truth (e.g. probable vs. improbable text).” \n* Citation formatting: “the maximum margin separator Soudry et al. (2018)”\n* Capitalization: (FISHER, 1936)\n* Extra comma: “The superposition hypothesis of Elhage et al. (2022), suggests this may be typical in deep networks.”\n* Duplicated word: “And why did mass-mean probing with the cities + neg cities training data perform poorly poorly for the 70B model”", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper reports on a series of creative interpretability analyses aimed at supporting the hypothesis that LLMs encode, in their hidden representations, their own truth-value judgments about factual claims. The experiments use behavioral techniques, probing, and interventions. The paper also introduces a new causal probe called mass-mean probing.

The general view of the reviewers (and myself) is that this is a creative, insightful paper. It is well-written, and the experiments are well-done and clearly reported. Overall, it looks like it has the potential to become an influential paper in interpretability space.
Decision: Accept