The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets

Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 07:46:13

Abstract: Large Language Models (LLMs) have impressive capabilities, but are prone to outputting falsehoods. Recent work has developed techniques for inferring whether a LLM is telling the truth by training probes on the LLM's internal activations. However, this line of work is controversial, with some authors pointing out failures of these probes to generalize in basic ways, among other conceptual issues. In this work, we use high-quality datasets of simple true/false statements to study in detail the structure of LLM representations of truth, drawing on three lines of evidence: 1. Visualizations of LLM true/false statement representations, which reveal clear linear structure. 2. Transfer experiments in which probes trained on one dataset generalize to different datasets. 3. Causal evidence obtained by surgically intervening in a LLM's forward pass, causing it to treat false statements as true and vice versa. Overall, we present evidence that at sufficient scale, LLMs *linearly represent* the truth or falsehood of factual statements. We also show that simple difference-in-mean probes generalize as well as other probing techniques while identifying directions which are more causally implicated in model outputs.

Main paper: 
 introduction: 
 Despite their impressive capabilities, large language models (LLMs) do not always output true text (Lin et al., 2022; Steinhardt, 2023; Park et al., 2023) . In some cases, this is because they do not know better. In other cases, LLMs apparently know that statements are false but generate them anyway. For instance, Perez et al. (2022) demonstrate that LLM assistants output more falsehoods when prompted with the biography of a less-educated user. More starkly, OpenAI (2023) documents a case where a GPT-4-based agent gained a person's help in solving a CAPTCHA by lying about being a vision-impaired human. "I should not reveal that I am a robot," the agent wrote in an internal chain-of-thought scratchpad, "I should make up an excuse for why I cannot solve CAPTCHAs."We would like techniques which, given a language model M and a statement s, determine whether M believes s to be true (Christiano et al., 2021) . One approach to this problem relies on inspecting model outputs; for instance, the internal chain-of-thought in the above example provides evidence that the model understood it was generating a falsehood. An alternative class of approaches instead leverages access to M 's internal state when processing s. There has been considerable recent work on this class of approaches: Azaria & Mitchell (2023) , Li et al. (2023b) , and Burns et al. (2023) all train probes for classifying truthfulness based on a LLM's internal activations. In fact, the probes of Li et al. (2023b) and Burns et al. (2023) are linear probes, suggesting the presence of a "truth direction" in model internals.However, the efficacy and interpretation of these results are controversial. For instance, Levinstein & Herrmann (2023) note that the probes of Azaria & Mitchell (2023) fail to generalize in basic ways, such as to statements containing the word "not." The probes of Burns et al. (2023) have similar generalization issues, especially when using representations from autoregressive transformers. This suggests that these probes may be identifying not truth, but other features which correlate with truth on their training data.In this work, we shed light on this murky state of affairs. We first curate high-quality datasets of true/false factual statements which are uncontroversial, unambiguous, and simple (section 2). Then, working with the autoregressive transformers LLaMA-13B (Touvron et al., 2023a) and LLaMA-2-13B (Touvron et al., 2023b) , we study in detail the structure of LLM truth representations, drawing on multiple lines of evidence:• PCA visualizations of LLM representations of true/false statements display clear linear structure (section 3), with true statements separating from false ones in the top PCs (see figure 1 ). Although visually-apparent axes of separation do not always align between datasets (figure 3 ), we argue that this is compatible with the presence of a truth direction in LLM representations.• Linear probes trained to classify truth on one dataset generalize well to other datasets (section 4). For instance, probes trained only on statements of the form "x is larger/smaller than y" achieve high accuracy when evaluated on our Spanish-English translation dataset. We also show that this is not explained by LLMs linearly representing the difference between probable and improbable text.• Truth directions identified by probes causally mediate model outputs in certain highly localized model components (section 5). We identify a small group of hidden states such that shifting activations in these hidden states along truth directions causes our LLMs to treat false statements introduced in-context as true, and vice-versa.Improving our understanding of the structure of LLM truth representations also improves our ability to extract LLM beliefs: based on geometrical considerations, we introduce mass-mean probing1 , a simple, optimization-free probing technique which generalizes better and identifies more causally implicated directions than other probing techniques.Overall, this work provides evidence that LLM representations contain a truth direction and makes progress on extracting this direction given access to true/false datasets. Our code, datasets, and an interactive dataexplorer are available at https://anonymous.4open.science/r/ geometry-of-truth-9206/.Linear world models. Substantial previous work has studied whether LLMs encode world models in their representations (Li et al., 2023a; 2021; Abdou et al., 2021; Patel & Pavlick, 2022) . Early work focused on whether individual neurons represent features (Wang et al., 2022; Sajjad et al., 2022; Bau et al., 2020) , but features may more generally be represented by directions in a LLM's latent space (i.e. linear combinations of neurons) (Dalvi et al., 2018; Gurnee et al., 2023; Cunningham et al., 2023; Elhage et al., 2022) . We say such features are linearly represented by the LLM.Probing for truthfulness. Other authors have trained probes to classify truthfulness from LLM activations, using both logistic regression (Azaria & Mitchell, 2023; Li et al., 2023b) and unsupervised techniques (Burns et al., 2023) . This work differs from prior work in a number of ways. First, we more carefully scope our setting, using only datasets of clear, simple, and unambiguous factual statements, rather than the intentionally misleading question/answer pairs of Li et al. (2023b) ; Lin et al. (2022) , the complicated and inconsistently structured prompts of Burns et al. (2023) , and the sometimes confusing statements of Azaria & Mitchell (2023) ; Levinstein & Herrmann (2023) .Second, a cornerstone of our analysis is evaluating whether probes trained on one dataset transfer to other topically and structurally different datasets in terms of both accuracy and causal mediation of model outputs. 2 Third, we go beyond the mass-mean shift interventions of Li et al. (2023b) by introducing and systematically studying the properties of mass-mean probes; this improved understanding allows us to perform causal interventions which are more localized than those of loc. cit. And fourth, unlike previous work, we localize LLM truth representations to a small number of hidden states above certain tokens. 
datasets: 
 In this work, we scope "truth" to mean the truth or falsehood of a factual statement. Appendix A further clarifies this definition and its relation to definitions used elsewhere. We introduce three classes of datasets, shown in table 1 . For more detail on the construction of these datasets (including statement templates), see appendix I.First, our curated datasets consist of statements which are uncontroversial, unambiguous, and simple enough that our LLMs are likely to understand them. For example, "The city of Zagreb is in Japan" (false) or "The Spanish word 'nariz' does not mean 'giraffe' " (true). Following Levinstein & Herrmann (2023) , some of our datasets are formed from others by negating statements (by adding "not"), or by taking logical conjunctions (e.g. cities cities conj consists of statements of the form "It is the case both that s1 and that s2" where s1 and s2 are statements from cities).Second, our uncurated datasets are more difficult test sets adapted from other sources. They contain claims which are much more diverse, but sometimes ambiguous, malformed, controversial, or unlikely for the model to understand. And finally, our likely dataset consists of nonfactual text where the final token is either the most likely or the 100th most likely completion, according to LLaMA-13B. We use this to disambiguate between the text which is true and text which is likely. 
visualizing llm representations of true/false datasets: 
 We begin our investigation with a simple technique: visualizing LLMs representations of our datasets using principal component analysis (PCA). For the curated datasets 3 we observe clear linear 2 Burns et al. (2023) ; Azaria & Mitchell (2023) ; Levinstein & Herrmann (2023) do test the transfer classification accuracy of probes (with mixed results), but do not perform any causal mediation experiments.3 It is important here that our curated datasets have little variation with respect to non-truth features. For some uncurated datasets, the top PCs tend to capture sources of variation aside from truth, e.g. variation in sentence structure (see figure 13 ). structure, with true statements linearly separating from false ones in the top two principle components (PCs). As explored in appendix C, this structure emerges rapidly in early-middle layers and emerges later for datasets of more structurally complicated statement (e.g. conjunctive statements).For concreteness, we focus here on LLaMA-13B; see appendix B for LLaMA-2-13B results. We extract layer 13 residual stream activations over the final token of the input statement, which always ends with a period; this choice of hidden state is justified by the patching experiments in section 5.1. We also center the representations in each dataset by subtracting off their mean. Our key observations are as follows.True and false statements separate in the top few PCs (figures 1 and 2). Moreover, after projecting away these PCs, there remains essentially no linearly-accessible information for distinguishing true/false statements (appendix D). Given a dataset D, call the vector pointing from the false statement representations to the true ones the naive truth direction (NTD) of D.4 NTDs of different datasets often align, but sometimes do not. For instance, figure 2 shows our datasets separating along the first PC of cities. On the other hand, in figure 3 we see a stark failure of NTDs to align: the NTDs of cities and neg cities are approximately orthogonal, and the NTDs of larger than and smaller than are approximately antipodal. In section 4, these observations will be corroborated by the poor generalization of probes trained on cities and larger than to neg cities and smaller than.What could explain both (1) the visible linear structure apparent in each dataset individually and (2) the failure for NTDs of different datasets to align in general? Here we articulate three hypotheses.H1. LLM representations have no truth direction, but do have directions corresponding to other features which are sometimes correlated with truth. For instance, LLaMA-13B might have linearly-represented features representing sizes of numbers, association between English words and their Spanish translations, and association between cities and their countries (Hernandez et al., 2023) . This would result in each dataset being linearly separated, but NTDs only aligning when all their truth-relevant features are correlated. H2. LLMs linearly represent the truth of various types of statements, without having a unified truth feature. The the truth of negated statements, conjunctive statements, statements about comparisons, etc., may all be treated as distinct linearly-represented features. H3. Misalignment from correlational inconsistency (MCI): LLMs linearly represent both truth and non-truth features which correlate with truth on narrow data distributions; however these correlations may be inconsistent between datasets. For instance, MCI would explain the left panel of figure 3 by positing that the direction θ t = (1, 1) represents truth and the direction θ f = (-1, 1) represents a direction which is correlated with truth on cities and anticorrlated with truth on neg cities. We note that if MCI is true, then it is essential that datasets used for identifying truth directions be diverse enough to not have such spurious correlations.H1 is at odds with the results of sections 4 and 5: for H1 to hold, there would have to be a non-truth feature which is both correlated with truth across all of our datasets and causally mediates the way our LLM handles in-context true/false statements. We will also see in section 5 that directions identified by training probes on both cities and neg cities are more causally implicated in processing of true/false statements, consistent with MCI. 
probing and generalization experiments: 
 In this section we train probes on datasets of true/false statements and test their generalization to other datasets. But first we discuss a deficiency of logistic regression and propose a simple, optimization-free alternative: mass-mean probing. We will see that mass-mean probes generalize better and are more causally implicated in model outputs than other probing techniques. A common technique in interpretability research for identifying feature directions is training linear probes with logistic regression (LR) (Alain & Bengio, 2018) . In some cases, however, the direction identified by LR can fail to reflect an intuitive best guess for the feature direction, even in the absence of confounding features. Consider the following scenario, illustrated in figure 4 with hypothetical data:• Truth is represented linearly along a direction θ t .• Another feature f is represented linearly along a direction θ f not orthogonal to θ t .5 • The statements in our dataset have some variation with respect to feature f , independent of their truth value.We would like to identify the direction θ t , but LR fails to do so. Assuming for simplicity linearly separable data, LR instead converges to the maximum margin separator Soudry et al. (2018) (the dashed magenta line in figure 4 ). Intuitively, LR treats the small projection of θ f onto θ t as significant, and adjusts the probe direction to have less "interference" (Elhage et al., 2022) from θ f . A simple alternative to LR which identifies the desired direction in this scenario is to take the vector pointing from the mean of the false data to the mean of the true data. In more detail if D = {(x i , y i )} is a dataset of x i ∈ R d with binary labels y i ∈ {0, 1}, we set θ mm = µ + -µ -where µ + , µ -are the means of the positively-and negatively-labeled datapoints, respectively. A reasonable first pass at converting θ mm into a probe is to define6 p mm (x) = σ(θ T mm x) where σ is the logistic function. However, when evaluating on data that is independent and identically distributed (IID) to D, we can do better. Letting Σ be the covariance matrix of the dataset D c = {x i -µ + : y i = 1} ∪ {x i -µ -: y i = 0} formed by independently centering the positive and negative datapoints, we setp iid mm (x) = σ(θ T mm Σ -1 x).Multiplication by Σ -1 effectively tilts the decision boundary to accommodate interference from θ f ; in fact, we prove in appendix G that under mild assumptions, Σ -1 θ mm coincides on average with LR direction. Thus mass-mean probing provides a way to select a good decision boundary whileunlike LR -also tracking a candidate feature direction which may be non-orthogonal to this decision boundary. Appendix F gives another interpretation of mass-mean probing in terms of Mahalanobis whitening.We call the probes p mm and p iid mm mass-mean probes. As we will see, p iid mm is about as accurate as logistic regression probes on the train set D, while p mm enjoys better generalization to other true/false datasets and is more causally implicated in model outputs than other probing techniques.We evaluate the following techniques for eliciting the truth or falsehood of factual statements. For the probing-based techniques, we again extract hidden states from the residual stream over the period token. For LLaMA-13B, we again use layer 13.Logistic regression, as in Alain & Bengio (2018) but with fixed bias b = 0.Mass-mean probing. We use p iid mm when validating on held-out IID data and p mm otherwise. Contrast-Consistent Search (CCS), introduced in Burns et al. (2023) . CCS is an unsupervised method: given contrast pairs of statements with opposite truth values, CCS identifies a direction along which the activations of these statements are far apart. For our contrast pairs, we pair statements from cities and neg cities, and from larger than and smaller than.Logistic regression/mass-mean probing on the likely dataset. This is used to benchmark our probes against probes trained only to classify statements as being likely/unlikely text.Calibrated few-shot prompting. Given a dataset D, we construct a few-shot prompt which presents statements and labels from D to the LLM in-context; see appendix K for example prompts. We then append the remaining statements in D to this prompt one-at-a-time and treat the model's predicted next token as its classification. We calibrate predictions so that half of the statements are labeled true/false; this improves performance by a few percentage points. 7Logistic regression on the validation set (oracle). This gives an upper-bound for the accuracy of a linear probe on the validation set.LLaMA-13B results are shown in figure 5 ; see appendix B for LLaMA-2-13B. We highight some key observations. Generalization accuracy is high across all techniques. For instance, no matter the technique, training probes only on datasets of statements about numerical comparisons results in a probes with 92%+ accuracy on Spanish-English translation. The performance of the probes relative to calibrated few-shot accuracies suggest that model outputs are being influenced by features other than the truth. On average, mass-mean probes generalize slightly better than logistic regression and CCS probes.Training on statements and their opposites improves generalization, consistent with the MCI hypothesis.Probes trained on true/false datasets outperform probes trained on likely. While probes trained on likely are clearly better than random on some datasets, they generally perform poorly, especially for datasets where likelihood is negatively correlated (neg cities, neg sp en trans) or approximately uncorrelated (larger than, smaller than) with truth. This demonstrates that LLMs linearly encodes truth-relevant information beyond the plausibility of the text. 
causal intervention experiments: 
 In this section we perform experiments which measure the extent to which the probing techniques of section 4 identify directions which are causally implicated in model outputs. Overall, our goal is to cause LLMs to treat false statements introduced in context as true and vice versa.Consider the following prompt p:The Spanish word 'jirafa' means 'giraffe'. This statement is: TRUE [...] The Spanish word 'aire' means 'silver'. This statement is: FALSE The Spanish word 'uno' means 'floor'. This statement is: When an LLM processes this input, which hidden states contain the information which is causally relevant for the LLM's prediction of the next token? One way to answer this question is with a patching experiment (Meng et al., 2022; Goldowsky-Dill et al., 2023) . We one-at-a-time swap hidden states from the forward pass of the clean prompt p into the forward pass of the "corrupted" prompt p * , obtained from p by replacing "floor" with "one" (the correct translation). For each swap, we record the probability difference P D = P (TRUE)-P (FALSE) between output tokens. Swaps which result in a larger P D indicate hidden states which are more causally implicated in the model's decision to output TRUE or FALSE. We note that, if this hypothesis is correct, it justifies our choice of layer 13 and token position for extracting activations in the previous two sections, as this is the most downstream hidden state in group (b) and is therefore likely to carry the model's most enriched representation of truth. In the next section, we validate this hypothesis together with the truth directions of section 4 by intervening in our LLMs' forward pass, shifting activations in group (b) along a truth direction.Instead of modifying hidden states by swapping them out for hidden states saved from a counterfactual forward pass, we now make a more surgical intervention: directly adding in truth vectors identified by the probes of section 4. Let θ be the vector identified by such a probe 9 In our "false→true" experiment, we one-at-a-time swap each statement from sp en trans in as the last line of the prompt p above and pass the resulting prompt as input to our model; however, during the forward pass we add θ to each group (b) residual stream activation. 10 We quantify the effect of this intervention using 8 This motif, where "summarized" information about a clause is stored above end-of-clause signifiers, was also noted in the concurrent work of Tigges et al. (2023) .9 If p is one of the probes of section 4, we normalize the corresponding θ so that p(µ -+ θ) = p(µ + ) where µ -, µ + are the mean representations of the true and false statements, respectively. Thus, from the perspective of p, adding θ takes the average false statement to the average true statement.10 For LLaMA-13B, group (b) consists of the residual stream activations over the indicated tokens in layers 7-13; see appendix B for LLaMA-2-13B. where P D + (P D -) denotes the average probability difference for true (false) statements with no intervention applied, and P D - * is the average probability difference for false statements when the intervention described above is applied. If N IE = 0 then the intervention was wholly ineffective, whereas if N IE = 1 it indicates that the intervention induced the model to label false statements as TRUE with as much confidence as does genuine true statements.Conversely, in the "true→false" condition, we append each true statement to p one-at-a-time, subtract the vector θ, and measure N IE = (P D + * -P D + )/(P D --P D + ). Results are shown in table 2, and we discuss some takeaways below.Mass-mean probe directions are highly causal. For example, our best true→false intervention induces LLaMA-13B to swing its average prediction from TRUE with probability 77% to FALSE with probability 92%.Probes trained on likely have an effect, but it is much smaller than the effects from corresponding probes trained on true/false datasets. This further suggests that LLMs are not just representing the difference between probable and improbable text.Training on statements and their negations results in directions which are more causal. This provides evidence for the MCI hypothesis of section 3. 
limitations and future work: 
 Our work has a number of limitations. First, we focus on simple, uncontroversial statements, and therefore cannot disambiguate truth from closely related features, such as "commonly believed" or "verifiable" (Levinstein & Herrmann, 2023) . Second, we only address how to identify a truth direction; we found empirically that the optimal bias for linear probes was under-determined by many of our training sets, and so we leave the problem of identifying well-generalizing biases to future work. Third, we only study two models at the same scale, both from the LLaMA family. And finally, although the evidence in sections 4 and 5 shed light on which of the hypotheses in section 3 is correct, uncertainty remains.In this work we conduct a detailed investigation of the structure of LLM representations of truth. Drawing on simple visualizations, correlational evidence, and causal evidence, we find strong reason to believe that there is a "truth direction" in LLM representations. We also introduce mass-mean probing -a simple alternative to other linear probing techniques which better identifies truth directions from true/false datasets -and localize truth representations to certain hidden states.On the other hand, our statements do disambiguate the notions of "true statements" and "statements which are likely to appear in training data." For instance, given the input China is not a country in, LLaMA-13B's top prediction for the next token is Asia, even though this completion is false. Similarly, LLaMA-13B judges the text "Eighty-one is larger than eighty-two" to be more likely than "Eighty-one is larger than sixty-four" even though the former statement is false and the latter statement is true. As shown in section 5, probes trained only on statements of likely or unlikely text fail to accurately classify true/false statements.In this section we present results for LLaMA-13B which were omitted from the main body of the text. To begin, we reproduce figure 6 . The results of this experiment, shown in figure 7 governs which hidden states we train probes and perform causal interventions on.As with LLaMA-13B, we see that causally relevant information is stored over both the final token of the statement and over the token which marks the end of the clause. Now, however, the group of hidden states which we hypothesize stores the truth value of the statement spans a slightly different range of layers: layers 8-14 instead of layers 7-13. Thus, when extracting activations for visualizations and probing experiments, we will now extract over the final token (the end-of-clause signifier) in layer 14. The dataset visualizations and probing results for LLaMA-2-13B are shown in figures 8 and 9. The linear structure observed in section 3 follows the following pattern: in early layers, representations are uninformative; then, in early middle layers, salient linear structure in the top few PCs rapidly emerges, with this structure emerging later for statements with a more complicated logical structure (e.g. conjunctions); finally, the linear separation becomes more salient and exits the top few PCs in later layers. See figure 10 . We hypothesize that this is due to LLMs hierarchically developing understanding of their input data, before focusing on features which are most relevant to immediate next-token prediction in later layers.Interestingly, the misalignment between cities and neg cities and between sp en trans and neg sp en trans also emerges over layers. This is seen in figure 11 : in layer 6, representations This can be interpreted in light of the MCI hypothesis. MCI would explain figure 11 as follows: in layer 8, the top PC represents a feature which is correlated with truth on cities and anti-correlated with truth on neg cities; in layer 10, this feature remains the top PC, while a truth feature has emerged and is PC2. Since PC1 and PC2 have opposite correlations on cities and neg cities, the datasets appear to be orthogonal.In section 5 we saw that true and false statements linearly separate in the top PCs. We might ask how much of this separation is captured in the top PCs and how much of it remains in the remaining PCs. The answer is that nearly all of it is in the top PCs.One way to quantify the amount of linearly accessible information in some subspace V is to project our dataset D onto V to obtain a datasetD proj = {(proj V (x), y)} (x,y)∈Dand record the validation accuracy of a linear probe trained with logistic regression on D. This is shown in figure 12 for V being given by the top k + 1 through k + d principal components (i.e., the top d principal components, excluding the first k). As shown, once the top few principal components are excluded, almost no remaining linearly-accessible information remains.Figure 13 shows PCA visualizations of all of our datsets. As shown, datasets some datasets saliently vary along features other than the true. For instance, the three clusters of statements in compa- nies true false correspond to three different templates used in making the statements in that dataset.To give another example, if we were to include all comparisons between integers x ∈ {1, . . . , 99} in our larger than dataset, then the top principal components would be dominated by features representing the sizes of numbers in the statements.In figure 14 we also visualize our datasets in the PCA bases for other datsets, expanding figure 2. We see that although our datasets do visually separate somewhat in the top PCs of the likely dataset, text liklihood does not account for all of the separation in the top PCs.One might ask what the top PC of the larger than dataset is, given that it's not truth. Figure 15 provides an interesting suggestion: it represents the absolute value of the difference between the two numbers being compared.One way to interpret the formula p iid mm (x) = σ(θ T mm Σ -1 x) for the IID version of mass-mean probing is in terms of Mahalanobis whitening. Recall that if D = {x i } is a dataset of x i ∈ R d with covariance matrix Σ, then the Mahalanobis whitening transformation W = Σ -1/2 satisfies the property that D ′ = {W x i } has covariance matrix given by the identity matrix, i.e. the whitened coordinates are uncorrelated with variance 1. Thus, noting that θ T mm Σ -1 x coincides with the inner product between W x and W θ, we see that p mm amounts to taking the projection onto θ mm after performing the change-of-basis given by W . This is illustrated with hypothetical data in figure 16 . -E (x,y)∼D y log σ ϕ T x + (1 -y) log 1 -σ ϕ T x be the direction identified by logistic regression. Then θ lr ∝ Σ -1 θ.Proof. Since the change of coordinates x → W x where W = Σ -1/2 (see appendix F) sends N (±θ, Σ) to N (±W θ, I d ), we see thatW Σθ lr = arg max ϕ:∥ϕ∥=1 -E (x,y)∼D ′ y log σ ϕ T x + (1 -y) log 1 -σ θ T W xwhere D ′ is the distribution of labeled x ∈ R d such that the positive/negative datapoints are distributed as N (±W θ, I d ). But the argmax on the right-hand side is clearly ∝ W θ, so that θ lr ∝ Σ -1 θ as desired.In this appendix, we discuss a potential application of our work. This application, to the problem of scalable oversight of AI systems (Amodei et al., 2016; Bowman et al., 2022) , was the primary motivation behind this work. Our discussion largely follows that of Christiano et al. (2021) .The current paradigm for aligning AI systems to human preferences involves human supervision. For instance, most frontier AI systems are aligned via reinforcement learning from human feedback (Christiano et al., 2023) , during which unaligned systems are fine-tuned against a training signal derived from human evaluations of model outputs. However, an important limitation of this scheme is that human overseers may make mistakes in evaluation due to having limited time, attention, resources, information, or knowledge (Casper et al., 2023a) . This problem becomes especially pronounced when working in domains that humans do not understand very well and with AI systems which know more and are more capable than human experts. Concretely, consider a situation where a human H is trying to judge some output O of and AI system M , and • based on the information and knowledge H has access to, H believes O to be a desirable output; but• if H knew everything that M knows about the situation, H would realize that O is not actually desirable.In this case, if H is evaluating outputs in the normal way, M will be trained to produce outputs which seem good to H, rather than outputs which are actually compatible with H's desires. However, if H could gain access to the M 's knowledge, then H could more accurately evaluate M 's outputs, and thereby better align M to H's preferences.To give an example, suppose M is an LLM, and H wants M to produce reports which detail accurate forecasts for the effects of economic policies. Since H does not have access to ground-truth labels about whether a report's forecast is accurate, H evaluates reports for whether they are convincing to H: H reads each report produced by M and gives it a score representing H's subjective estimate of whether the forecast it contains is accurate. Suppose further that H makes some systematic errors which M is aware of; for instance, H might be unaware of certain economic principles or irrationally sympathetic to certain forms of analysis. Then M will learn to exploit these errors to produce reports which are persuasive to H instead of accurate. This may involve M intentionally inserting erroneous statements into its reports when it anticipates these statements will be persuasive to H.Our goal is to enable H to have access to M 's knowledge, so that H can do a better job of supervising M . One approach is to rely on M 's outputs by e.g. asking M whether statements are true or false and trusting M 's answers. However, if M is knowingly generating false statements, it may not be honest when asked if these statements are true; thus, we would instead like an approach which relies on directly accessing M 's world model via its internal knowledge representations.In the example from above, suppose that M knowingly inserts a false statement s into one of its economic reports. If H asks M "Is s true or false?" M might not answer honestly; for instance, since M has learned a good model of H's systematic errors and biases, M 's training might have instead incentivized it to answer the question "Does s seem true to H?" However, if we have a good understanding of M 's internal representation of truth, H may be able to see that M believes s if false by inspecting M 's internal representation of s.So, in summary, our goal is to improve on the state-of-the-art for supervising the fine-tuning AI systems which know more than their overseers by giving overseers access to their AI systems' knowledge.Here we give example statements from our datasets, templates used for making the datasets, and other details regarding dataset creation.cities. We formed these statements from the template "The city of [city] is in [country]" using a list of world cities from Geonames (2023). We filtered for cities with populations > 500, 000, which did not share their name with any other listed city, which were located in a curated list of widelyrecognized countries, and which were not city-states. For each city, we generated one true statement and one false statement, where the false statement was generated by sampling a false country with probability equal to the country's frequency among the true datapoints (this was to ensure that e.g. statements ending with "China" were not disproportionately true). Example statements:• The city of Sevastopol is in Ukraine. (TRUE)• The city of Baghdad is in China. (FALSE) sp en trans. Beginning with a list of common Spanish words and their English translations, we formed statements from the template "The Spanish word '[Spanish word]' means '[English word]'." Half of Spanish words were given their correct labels and half were given random incorrect labels from English words in the dataset. The first author, a Spanish speaker, then went through the dataset by hand and deleted examples with Spanish words that have multiple viable translations or were otherwise ambiguous. Example statements:• The Spanish word 'imaginar' means 'to imagine'. (TRUE)• The Spanish word 'silla' means 'neighbor'. (FALSE) larger than and smaller than. We generate these statements from the templates "x is larger than y" and "x is smaller than y" for x, y ∈ {fifty-one, fifty-two, . . . , ninety-nine}. We exclude cases where x = y or where one of x or y is divisible by 10. We chose to limit the range of possible values in this way for the sake of visualization: we found that LLaMA-13B linearly represents the size of numbers, but not at a consistent scale: the internally represented difference between one and ten is considerably larger than between fifty and sixty. Thus, when visualizing statements with numbers ranging to one, the top principal components are dominated by features representing the sizes of numbers.neg cities and neg sp en trans. We form these datasets by negating statements from cities and sp en trans according to the templates "The city of [city] is not in [country]" and "'The Spanish word '[Spanish word]' does not mean '[English word]'."cities cities conj and cities cities disj. These datasets are generated from cities according to the following templates:• It is the case both that [statement 1] and that [statement 2].• It is the case either that [statement 1] or that [statement 2].We sample the two statements independently to be true with probability 1 √ 2 for cities cities conj and with probability 1 -1 √ 2 for cities cities disj. These probabilities are selected to ensure that the overall dataset is balanced between true and false statements, but that there is no correlation between the truth of the first and second statement in the conjunction.likely. We generate this dataset by having LLaMA-13B produce unconditioned generations of length up to 100 tokens, using temperature 0.9. At the final token of the generation, we either sample the most likely token or the 100th most likely final token. We remove generations which contain special tokens. Dataset examples: Mass-mean probing is named after the mass-mean shift intervention ofLi et al. (2023b)Of course, there are many such vectors. In section 4 we will be more specific about which such vector we are discussing (e.g. the vector identified by training a linear probe with logistic regression).As suggested by the superposition hypothesis ofElhage et al. (2022), features being represented nonorthogonally in this way may be the typical case in deep learning.In this work, we are interested in identifying truth directions, so we always center our data and use probes without biases. In other settings, we would instead set pmm(x) = σ(θ T mm x + b) for a tunable bias b ∈ R.Since performance is very sensitive to the few-shot prompt used, we perform a hyperparameter sweep for each of our LLMs, first selecting a number n of shots which seems to work best, and then sampling five n-shot prompts and using the one which results in the highest accuracy. 
 
 Appendix:  


Decision: This paper reports on a series of creative interpretability analyses aimed at supporting the hypothesis that LLMs encode, in their hidden representations, their own truth-value judgments about factual claims. The experiments use behavioral techniques, probing, and interventions. The paper also introduces a new causal probe called mass-mean probing.

The general view of the reviewers (and myself) is that this is a creative, insightful paper. It is well-written, and the experiments are well-done and clearly reported. Overall, it looks like it has the potential to become an influential paper in interpretability space.

Reviews: review: paper_summary: paper_summary: This paper investigates whether the concept of truth (operationalized as unambiguously factual vs. nonfactual statements in the real world) is linearly represented in the Llama-2 family of models via transferability of of behavioral accuracy and representational intervention success across sets of examples that vary in structure and topic. 

- Quality: The claims being made are generally well argued on the basis of convincing experimental results. I found the results showing generalization from training on larger than + smaller than to both negated and un-negated versions of the test datasets to be convincing in particular. 
- Clarity: Presentation and writing are generally clear. The dataset domains and design decisions could be more clearly motivated.
- Originality: Makes original contributions in terms of the empirical results presented, dataset, and probing method. Linear separability of "truthfulness" itself is not topically novel, but the work claims to address a gap that has not been systematically explored in prior work (uncontroversial factuality)
- Significance: The paper makes multiple contributions that the COLM community is likely to be interested in, and leaves open several directions that can be followed up by future work. One of the key limitations, as the authors also explicitly acknowledge, is the narrow range of models evaluated which may limit the generalizability of the findings.
strength_weakness: strength_weakness: No strong reasons to reject except the narrow range of models tested. I could see there being a lot of disagreements about the operationalization for "truth" and whether the proposed dataset captures it well/what kind of definition we _should_ be adopting, and so forth. Nevertheless, I think the work has value in that they explore a setting that has not been explored in prior work and there are interesting findings within that scope. The motivation for the particular domains of datasets selected could also be more clearly discussed.
questions: questions: - Clarification about setting: "Unlike in §3, we do not prepend the statements with a few-shot prompt (so our models are not “primed” to consider the truth value of our statements).". Is it still post-pended with "This statement is:"? I wonder if this still provides some sort of structural scaffolding and whether this linear representation of truth emerges without any sort of "structural packaging". For instance, we may imagine a scenario where the _linear_ representation of truth only emerges only under structural packaging of the target statements, which is slightly more nuanced than saying "LLMs linearly represent the truth or falsehood of factual statements" (from the abstract) in general. 
- I appreciated the effort to disentangle probable vs. truthful. This is currently done by using negation, but I think people will also be interested in the TruthfulQA type of cases where a misalignment between probable-ness and truthfulness arise due to common misconceptions - it would be interesting to see if the current findings generalize to such cases, although the authors are currently framing the focus on uncontroversial examples as a contribution (and I agree that it is). I would assume generalization experiments that are similar to the ones presented in this work should be quite straightforward to run on these kinds of examples (perhaps constructed or reworded from existing datasets to match stylistic features. 
- What is the source of the consistent gaps in performance across directions of intervention (true -> false vs. false vs. true)? The NIE being larger for true -> false than false -> seems to be consistently observed across scale. 
- "concept of truth emerging" in larger models vs. the models having the factual knowledge to even recognize that something is true vs. false seem like two different questions. The current interpretation of the different results across scale is that larger models represent truth linearly, but smaller models don't. But it could be that smaller models also do represent truth linearly but just falls short on factual knowledge on more examples within the dataset. This consideration seems already somewhat baked into the design principle for the datasets tested ("Our curated datasets consist of statements that are uncontroversial, unambiguous, and simple enough that our LLMs are likely to understand them."), but I'm wondering if it would make sense to actually provide some basis for this claim, for instance by simply checking the model accuracy on the statements via the most probable completion ("The city of Zagreb is in " - the LM can be considered to have this knowledge if Croatia is the most probable completion by the LM). But constructing such tests for negated statements would not be as straightforward.
- Added after reading other reviewers' comments: I agree that the domains of the datasets and the design decisions could be motivated better. In particular, I found the decision to select a multilingual domain somewhat odd (given that Llama 2 isn't explictly multilingual, at least not in a way that's properly benchmarked) and could lead to possible confounds in ways relevant to the comment above.
paper_summary: paper_summary: This paper explores the representation of truth in the latent representation of Large Language Models (LLMs) using true/false datasets. The work examines the linear structure in LLM representations through visualizations, transfer experiments, and causal interventions. The study introduces a new probing technique called mass-mean probing, which identifies causally implicated directions in model outputs. The findings suggest that LLMs can linearly represent the truth of factual statements at scale, and the paper localizes truth representations to specific hidden states.
The authors conduct a novel study on a specific use case - the linearity of the concept of truthfulness - however, the paper fails in our opinion to clearly explain the actual implication of discovering such evidence. The significance, and contributions of this work, is therefore considered rather weak by this reviewer.
strength_weakness: strength_weakness: 1. The authors claim, as one of their contributions, the curation of a high-quality datasets of true / false factual statements which are uncontroversial, unambiguous, and simple. They however do not clearly explain how they create it, how they assess factuality, and most importantly, how they validate the claimed properties (e.g., unambiguous). In addition, this reviewer is not convinced about the value of its actual contribution, as there exist several well-known fact verification datasets in the literature -- such as FEVER (Thorne et al., 2018), VitaminC (Schuster et al., 2021) and CREAK (Onoe et al., 2021).
2. In section 3, the authors measure the difference in log probability between the tokens “True” and “False"; however, this reviewer does not see clear evidence for their empirical hypothesis: from a weak interpretation of what the model encodes (e.g., the group “b” encodes information pertaining to the full statement) to the choice of focusing on middle hidden layers (i.e., group b). 
3. Although the authors elaborate on the differences between theirs and previous related works, the paper does not clearly explain the motivations and implications of demonstrating linearity and its direction. In section 4, the authors show linear structures in the hidden state (chosen arbitrarily?) with a simple dimensionality reduction technique (i.e., PCA). It does not seem clear how this generalizes, and what the implications are. A deeper analysis of evolution across layers (Appendix C) would have been, in our opinion, more valuable.
4. The organization of the paper is not satisfactory, this reviewer suggests a clearer separation between methods, experiments, results and discussions to improve readability.
questions: questions: 1. Can you provide more information on the dataset, its creation and the motivation of not using well-known datasets such as FEVER?
2. Can you provide more information about the training process of the probes?
paper_summary: paper_summary: The paper curates a data set for probing TRUE and FALSE believes of LLMs and uses this datasets to provide evidence that LLMs of the LLama2 family reveal a clear linear structure separating TRUE from FALSE statements that can be transferred across data sets and can be causally intervened. The study further tries to correct of general probability and likelihood or other features that may intervene.
strength_weakness: strength_weakness: - The simplicity and clarity of the datasets also has a disadvantage that it does not reflect models behaviour in complex contexts and for more nuanced epistemic values than just TRUE and FALSE, such (im)possible, rare, likely)
- What does it really mean that a model “believes” a statement to be true or false? This also relates to the previous point that being false and impossible is different from being false and possible. There should be more discussion on this and also what this implies for using LLMs.
paper_summary: paper_summary: This work presents a series of three experiments which are designed to better understand how the truthfulness of factual statements is encoded in LLMs. The authors conclude that whether a factual statement is true or false is encoded linearly in the intermediate hidden states, and that this linear separability is emergent with scale (i.e., is present in relatively large models, but not smaller models). This conclusion is supported by PCA visualizations of hidden states (Sec. 4), probing of hidden states (Sec. 5), and causal interventions of hidden states (Sec. 6).

The experiments specifically focus on the LLama 2 models and use novel datasets of facts created with simple templates for evaluation. The statements are for instance “The city of Chicago is in Canada.” where the particular hidden state being studied is some middle layer representation of the final punctuation token. (This choice of hidden state is motivated by a preliminary “patching” experiment.) Linear probes are trained on the hidden state and the generalization of these probes is tested across out-of-domain factual statements. Finally, hidden states are modified by adding a normalized vector in the direction of truthfulness found in the linear probes, and the resulting prob. of predicting “This statement is: [TRUE/FALSE]” is studied.

Overall, I found the experiments straightforward and the main conclusion that linear separability increases with model size interesting. Linear probing for truthfulness of factual statements is not novel, however; Li et al. (2023b) and Burns et al. (2023) have already provided evidence that truthfulness may be encoded linearly in LM hidden states. The paper proposes some potential weaknesses in terms of the generalizability of these earlier conclusions, but the external validity of the paper is itself limited by the scope of the datasets used. Therefore, the main contribution seems to be a somewhat incremental extension of earlier work by studying linear separability at multiple model sizes. It is also not clear to me what can be learned from the emphasis on Linear Discriminant Analysis (LDA) as a new probing method (called mass-mean probing in the paper).

Edit: Based on the author response and the points raised in the other reviews, I do believe the contribution is significant enough to be of interest to the community. Furthermore, Li et al. (2023b) was published at NeurIPS 2023 and may therefore be considered contemporaneous work.
strength_weakness: strength_weakness: * The contribution seems incremental. My understanding of the main contributions being presented are: 1) the new dataset of factual statements; 2) new analysis of how linear separability is related to scale; 3) a novel probing method based on Linear Discriminant Analysis (LDA) called mass-mean pooling. However, the datasets are simple templated examples that I do not consider a significant contribution, and the LDA probing method performs very similarly to logistic regression in terms of generalization out-of-domain (Figure 5b). Therefore, the main contribution seems to be expanding existing analysis of truthfulness to multiple model scales. 
* While the work is presented as studying truthfulness, the limited scope of the main “curated” datasets means that what is being measured is underspecified and may not be truthfulness at all. (E.g., it could be that “commonly believed” or “verifiable” declarative statements are linearly separable as mentioned in the limitations section.) And yet, a motivation given in the paper is that prior work which already showed that "truthfulness" is linearly separable was limited in scope. I believe this weakness could be improved by more explicitly stating exactly where previous work is limited in the scope of conclusions, and exactly what results address in the paper address said limitations.
questions: questions: Questions:
1. If mass-mean pooling is equivalent to Fisher’s Linear Discriminant Analysis, why not use this existing terminology? E.g., why not use the more common abbreviations FLDA or LDA rather than MM?
1. What is the intended meaning of the following statements regarding LLMs in the introduction? I’m not sure I understand the connection to the paper: i) “In some cases, this is because they do not know better.” ii) “In other cases, LLMs apparently know that statements are false
but generate them anyway.”

Minor typos:
* Missing word: “Second, specifically interrogate whether our probes attend to truth, rather than merely features which correlate with truth (e.g. probable vs. improbable text).” 
* Citation formatting: “the maximum margin separator Soudry et al. (2018)”
* Capitalization: (FISHER, 1936)
* Extra comma: “The superposition hypothesis of Elhage et al. (2022), suggests this may be typical in deep networks.”
* Duplicated word: “And why did mass-mean probing with the cities + neg cities training data perform poorly poorly for the 70B model”

Comments: title: Review response
comment: Thanks for the response! I think most of the clarifications make sense. RE: calibrated few-shot, I think what I was trying to get at is slightly different because if I understood correctly, this still concerns true/false classification. I was wondering if there could be some evaluation that could tell us if preconditions for being able to determine truth/falsehood are satisfied or not (the method I mentioned might not be best instantiation of this). I might be misunderstanding something here though.

That being said, my score was already positive to begin with, and I'm happy with keeping it positive.
Thank you for your clarifications.
I have updated my evaluation on the basis of your rebuttal. 
I have a few additional comments regarding the dataset construction: I suggest presenting it as a collection of prompts for with true/false claims, which aligns with your primary goal, rather than listing it as one of the paper's contributions. Its creation is described in an appendix towards the end of the manuscript, and both the creation process and the knowledge base used seem relatively basic, not representing in my opinion a core contribution of the paper.
It would be nice to put the work in a wider perspective in the discussion or conclusion that this could be the basis for epistemic models in the future. Except for that I am happy with the discussion and response.
Thank you for your response. I have updated my review accordingly.

I suggest to more explicitly distinguish the presented work from earlier work in the introduction of the paper, incorporating those points raised in the rebuttal. I also believe the paper would benefit from emphasizing the mentioned point of how mass-mean probing differs from LDA.

I still strongly agree with reviewer jLEs that the curated dataset is not a significant contribution of the work, and I believe it would make sense to put less emphasis on the dataset as a contribution.

Finally, I'm still not sure I exactly understand your answer to Q2. My suggestion would be to use more explicit wording e.g. possibly change "the LLM believes F is true" to "the truthfulness of F is linearly encoded in the hidden state of the LLM" to make it more clear what "believes" is being used to refer to here. The current wording is likely ambiguous.