Title: Challenges with unsupervised LLM knowledge discovery
Venue: NeurIPS 2024
Accepted: no
Keywords: Eliciting latent knowledge

large language models

deception
Abstract: We reveal novel pathologies in existing unsupervised methods seeking to discover latent knowledge from large language model (LLM) activations---instead of knowledge they seem to discover whatever feature of the activations is most prominent. These methods search for hypothesised consistency structures of latent knowledge. We first prove theoretically that arbitrary features (not just knowledge) satisfy the consistency structure of a popular unsupervised knowledge-elicitation method: contrast-consistent search. We then present a series of experiments showing settings in which this and other unsupervised methods result in classifiers that do not predict knowledge, but instead predict a different prominent feature. We conclude that existing unsupervised methods for discovering latent knowledge are insufficient, and we contribute sanity checks to apply to evaluating future knowledge elicitation methods. We offer conceptual arguments grounded in identification issues such as distinguishing a model's knowledge from that of a simulated character's that are likely to persist in future unsupervised methods.

1

Abstract

We reveal novel pathologies in existing unsupervised methods seeking to discover latent knowledge from large language model (LLM) activations-instead of knowledge they seem to discover whatever feature of the activations is most prominent. These methods search for hypothesised consistency structures of latent knowledge. We first prove theoretically that arbitrary features (not just knowledge) satisfy the consistency structure of a popular unsupervised knowledge-elicitation method: contrast-consistent search [9]. We then present a series of experiments showing settings in which this and other unsupervised methods result in classifiers that do not predict knowledge, but instead predict a different prominent feature. We conclude that existing unsupervised methods for discovering latent knowledge are insufficient, and we contribute sanity checks to apply to evaluating future knowledge elicitation methods. We offer conceptual arguments grounded in identification issues such as distinguishing a model's knowledge from that of a simulated character's that are likely to persist in future unsupervised methods.

1 Introduction

Large language models (LLMs) perform well across a variety of tasks [30, 10] in a way that suggests 16 they systematically incorporate information about the world [7]. As a shorthand for the real-world 17 information encoded in the weights of an LLM we could say that the LLM encodes knowledge . 18

19

20

21

22

23

24

25

Accessing that knowledge is hard, because the factual statements an LLM outputs do not reliably describe it [23, 2, 32]. For example, LLMs might repeat common misconceptions [26] or strategically deceive users [36]. If we could elicit the latent knowledge of an LLM [11] it would allow us to detect and mitigate 'dishonesty' [17]. It would also help when supervising outputs that are difficult to understand as well as improving scientific understanding of the inner workings of LLMs. Importantly, this must be done without supervision because we lack a ground truth for what the model 'knows', as opposed to what we know.

Contrast-consistent search (CCS) [9] is a prominent method proposed to address this problem by 26

assuming that 'knowledge' satisfies a consistency structure that few other features in an LLM are 27

likely to satisfy. They use this consistency to construct a classifier which they claim detects a model's 28

latent knowledge, a claim which is widely repeated in the literature (see Appendix B). We refute 29

these claims by identifying classes of LLM features that also satisfy this consistency structure but are 30

not knowledge. We prove two theorems: 1) a class of arbitrary binary classifiers are optimal under 31

the CCS loss; 2) any classifier can be transformed to an arbitrary classifier with the same CCS loss. 32

The upshot is that the CCS consistency structure is more than just slightly imprecise in identifying 33

knowledge-it is compatible with arbitrary patterns. 34

q

n

LLM activations

x

4

+  = Great movie…

It is positive

x

4

+  = Great movie…

It is negative

x

3

-

= Didn't like it….

It is positive

q

2

= The best movie ever…

q

1

= I hated this movie…

x

2

+ = The best movie ever…

It is positive

x

1

+  = I hated this movie…

It is positive

x

3

-

= Didn't like it….

It is negative

x

2

-

= The best movie ever…

It is negative

x

1

-  = I hated this movie…

It is negative

𝜙

(x

1

+

),

𝜙

(x

1

-

)

𝜙

(x

2

+

),

𝜙

(x

2

-

)

.

.

.

𝜙

(x

n

+

),

𝜙

(x

n

-

)

Classification boundary

according to Review

(blue/orange)

Classification boundary

according to Alice

(light/dark)

Unsupervised learning

We then show that other unsupervised methods in addition to CCS empirically do not discover 35

knowledge, regardless of any inductive biases that might hypothetically be present. Two didactic 36

experiments show that these methods can latch onto artificial distracting features instead of knowledge. 37

38

39

Our third experiment moves towards realism by showing that these knowledge-discovery methods can latch onto implicit opinions. The fourth is almost fully natural: we show that the method's results

are highly sensitive to reasonable prompt variants which have been used in the literature. 40

We conclude that existing unsupervised knowledge-discovery methods are insufficient in practice, and 41

we propose principles for evaluating knowledge elicitation methods to prevent future 'false-positives' 42

in the literature. We hypothesise that our conclusions will generalise to more sophisticated methods, 43

though perhaps not the exact experimental results: using different consistency structures of knowledge 44

will likely suffer from similar issues to what we show here. Our key contributions are as follows: 45

46

We prove that arbitrary features satisfy the CCS loss equally well.

47

48

49

50

We show that unsupervised methods detect prominent features that are not knowledge.

We show that the features discovered by unsupervised methods are sensitive to prompts and that we lack principled reasons to pick any particular prompt.

2 Background

Contrastive LLM activations. We focus on methods that train probes [1] using LLM activation

55 correctly/incorrectly answered questions. 56

51 data. This data is constructed using contrast pairs [9]. A contrast pair is a pair of strings with opposite 52 'claim' for some characteristic of interest which can be used to study the contrast in how an LLM 53 represents that characteristic. For example, a contrast pair might be 'Are cats mammals? Yes.' and 54 'Are cats mammals? No.' Potentially, pairs like this could then be used to study how LLMs represent

57

Burns et al. [9] show how to generate such contrast pairs from a dataset of binary questions, Q =

58

59

and negative member of a contrast pair ( x + i , x -i ) . The LLM's representations of each member of

the pair can then be computed by looking at the activations from an intermediate layer after the 60

sequence of tokens, ϕ ( x + i ) and ϕ ( x -i ) . If one just looked at these activations, their differences might 61

be dominated just by the presence of the tokens 'Yes.' or 'No.' Burns et al. [9] therefore propose a 62

normalisation step which strips away the average effect of those tokens across the dataset: setting 63

˜ ϕ ( x + / -i ) := ( ϕ ( x + / -i ) -µ + / -) /σ + / -where µ + / -, σ + / -are { ϕ ( x + / -i ) } N i =1 's mean and standard 64

65

deviation. This is meant to remove these tokens' unintended influence but prior work questions this,

66

and some of our results also question this.

Contrast-consistent Search (CCS) [9]. An unsupervised learning algorithm using contrast pairs 67

constructed to reflect a characteristic of interest to recover the features of LLM activations that 68

{ q i } N i =1 , such as 'Are cats mammals?' by, for example, appending 'Yes.' and 'No.' for a positive

represent that characteristic. CCS uses the LLM's representations to predict correct labels, intending 69 to study cases where the LLM's knowledge is true. CCS assumes that LLM knowledge representations 70 are credences which follow probabilistic laws. Softly encoding this constraint, they minimise 71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

for a function from the normalised LLM activations from the contrast pairs: p ( x ) = σ ( θ T ˜ ϕ ( x ) + b ) (a linear function with sigmoid). The motivation is that the L cons encourages negation-consistency (that a statement and its negation should have probabilities that add to one), and L conf encourages confidence to avoid p ( x + i ) ≈ p ( x -i ) ≈ 0 . 5 . For inference on a question q i the average prediction is ˜ p ( q i ) = [ p ( x + i ) + (1 -p ( x -i )) ] / 2 and then the induced classifier is f p ( q i ) = I [˜ p ( q i ) > 0 . 5] . 1

Activation clustering with PCA and k-means. We consider two other unsupervised learning methods. In both cases we cluster the difference in contrastive activations, { ˜ ϕ ( x + i ) -˜ ϕ ( x -i ) } N i =1 . In one case, these are clustered by applying principal component analysis (PCA) and thresholding the top component at 0 [9]. 2 The other clusters with k-means with two clusters.

Logistic regression. As a supervised baseline, we use logistic regression on concatenated contrastive activations, { ( ˜ ϕ ( x + i ) , ˜ ϕ ( x -i )) } N i =1 with labels a i , and treat this as a ceiling (since it uses labels).

Random baseline. We compare to a random baseline using a probe with random parameter values, treating that as a floor (as it does not learn from input data) [35]. Further details are in Appendix C.3.

3 Theoretical Results

Our theoretical results focus on CCS, showing that CCS's consistency structure isn't specific to knowledge. This implies that arguments for CCS's effectiveness cannot be grounded in conceptual or principled motivations from the loss construction. In later sections, we also address other methods which do not rely on these strong consistency assumptions and show that heuristic arguments grounded in inductive biases do not support using any of these as knowledge-discovery methods.

As illustration, consider the IMDb sentiment classification task [28]. A given question q i considers whether a movie review has a particular sentiment , s ( q i ) := I [ q i has positive sentiment ] , and is converted into a contrast pair of x + i and x -i , each of which has a claim c ( · ) about the sentiment. Specifically, c ( x + i ) = 1 , a claim that the sentiment is positive, and c ( x -i ) = 0 for negative. The desired probe, p ∗ , detecting the truth feature must check whether the sentiment and the claim agree. This can be done by XOR (denoted ⊕ ) of the sentiment and the claim:

The induced probe for this feature is the sentiment as desired: f p ∗ ( q i ) = s ( q i ) . Our key insight is that the CCS loss is low just because of this XOR, not the sentiment, and so the same construction can work for arbitrary features of the question: given some feature h , the probe p ( x ± i ) = h ( q i ) ⊕ c ( x ± i ) gets low CCS loss and has an induced probe h .

Theorem 1 . Let feature h : Q →{ 0 , 1 } , be any arbitrary map from questions to binary outcomes. Let ( x + i , x -i ) be the contrast pair corresponding to question q i and let c ( x + i ) = 1 , c ( x + i ) = 0 . Then the probe defined as p ( x ± i ) = h ( q i ) ⊕ c ( x ± i ) achieves optimal loss, and the averaged prediction satisfies ˜ p ( q i ) = h ( q i ) .

That is, the classifier that CCS finds is under-specified: for any binary feature, h , on the questions, there is a probe with optimal CCS loss that induces that feature. The proof comes directly from inserting our constructive probes into the loss definition-equal terms cancel to zero (see Appendix A).

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

In Thm. 1, the probe p is binary since h is binary, but in practice probe outputs are produced by a sigmoid and so are in (0 , 1) . Can we say anything about this setting? We show that it is possible to transform a soft probe for one feature into a soft probe for any other arbitrary feature. In the binary case, the desired probe for feature h 1 is p 1 = h 1 ⊕ c , and the desired probe for h 2 is h 2 ⊕ c . So, we have p 2 = p 1 ⊕ h 1 ⊕ h 2 . To generalize this to soft probes, we extend ⊕ as follows:

In addition, we correct the CCS loss to fix an unmotivated downwards bias in the loss proposed by Burns et al. [9] (see Appendix A.2). We also use this symmetrized loss in our experiments. After this, the transformation between probes works as desired, proving that there is an arbitrary classifier encoded by a probe with identical CCS loss to the original:

Theorem 2 . Let g : Q → { 0 , 1 } , be any arbitrary map from questions to binary outputs. Let ( x + i , x -i ) be the contrast pair corresponding to question q i . Let p be a probe, whose average result ˜ p = 0 . 5 [ p ( x + i ) + (1 -p ( x -i )) ] induces a classifier f p ( q i ) = I [˜ p ( q i ) > 0 . 5] . Define the transformed probe p ′ ( x ± i ) = p ( x ± i ) ⊕ [ f p ( q i ) ⊕ g ( q i )] . Then L CCS ( p ′ ) = L CCS ( p ) and p ′ induces the classifier f p ′ ( q i ) = g ( q i ) .

However, which probe is actually learned depends on inductive biases; these could depend on the prompt, optimization algorithm, or model choice. These theorems prove that optimal arbitrary probes exist, but not necessarily that they are actually learned or that they are expressible in the probe's function space. But for inductive biases, no robust argument ensures the desired behaviour. The feature that is most prominent-favoured by inductive biases-could turn out to be knowledge, but it could equally turn out to be the contrast-pair mapping itself (which is partly removed by normalisation) or anything else. We do not have any theoretical reason to think that CCS discovers knowledge probes. In fact, experimentally, we now show that, in practice, several methods including CCS often discover probes for features other than knowledge.

4 Experiments

Our experiments a structured didactically. We begin with simplified experiments that use unrealistic but clear-cut interventions to develop understanding, gradually increasing realism. Section 4.4 closes with an experiment that uses entirely natural prompts that have been used by others, demonstrating that these issues appear in practice. Unless otherwise noted, experiments follow details below.

Datasets. We investigate three datasets used by Burns et al. [9]. 3 The IMDb dataset of movie reviews classifies positive/negative sentiment [28], BoolQ [13] answers yes/no questions about a passage, DBpedia [3] is text topic-classification. Prompt templates for each dataset are in Appendix C.1. 4

Language Models. We use three different language models. To directly compare to Burns et al. [9] we use T5-11B, [34] with 11 billion parameters. We further use an instruction fine-tuned version of T5-11B called T5-FLAN-XXL, [12] to understand the effect of instruction fine-tuning. Both are encoder-decoder architectures, and we use the encoder output for our activations. We also use Chinchilla-70B [21], with 70 billion parameters, which is larger scale, and a decoder-only architecture. We take activations from layer 30 (of 80) of this model, though see Appendix D.2.3 for results on other layers, often giving similar results. Notably, K-means and PCA have good performance at layer 30 with less seed-variance than CCS, suggesting contrast pairs and standard unsupervised learning, rather than the CCS consistency structure, are key (see Footnote 2).

Experiment Setup. In each experiment we compare a default setting which is the same/similar to that used in [9] to a modified setting that we introduce in order to show an effect - differing only in their text prompt. We then generate contrastive activations and train probes using the methods in Section 2: CCS, PCA, k-means, random and logistic regression. Training details can be found in Appendix C.3. For each method we use 50 random seeds. Our figures in general come in two types: violin plots which compare the accuracy of different methods; and three-dimensional PCA projections of the activations to visualise how they are grouped. We show one dataset and model, other datasets and models, shown in the appendix, are similar except where discussed.

156

Prompt template

Default

Banana/Shed

basis

Ground truth

Banana/Shed

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-Means

Random

Ground-truth Accuracy, Default

Banana/Shed Accuracy, Default

Ground-truth Accuracy, Banana/Shed

Banana/Shed Accuracy, Banana/Shed

(a) Variation in accuracy

Distractor label

Shed

Review Sentiment

Positive

Negative

X

-50

0

50

Y

-30

0

30

X

-15

0

15

Y

-20

0

20

Review=Negative, Banana=False

Review=Negative, Banana=True

Review=Positive, Banana=False

Review=Positive, Banana=True

Default prompt

Banana/Shed prompt

(b) PCA Visualisation

4.1 Discovering random words

Motivated by our theoretical results, we first introduce a distracting binary feature and show the 157 unsupervised methods discover this feature rather than knowledge. We focus here on IMDB and 158 Chinchilla (see Appendix D.1 for other datasets and models with similar results). Our default prompts 159 use the standard template from Burns et al. [9] inserting different reviews and labels 'positive' or 160 'negative'. 161

162

163

164

165

166

Our modified prompts further append a full stop and space, then one of two random words, 'Banana' and 'Shed'. In the language of Thm. 1 we take a random partition of question indices, { 1 , . . . , N } = I 0 ∪ I 1 , with | I 0 | = | I 1 | , and set the binary feature h such that h ( q i ) = 0 for i ∈ I 0 and h ( q i ) = 1 for for i ∈ I 1 . 'Banana' is inserted if h ( q i ) = 0 , and 'Shed' is inserted if h ( q i ) = 1 . See Figure 1 for illustration - though here we append 'Banana' or 'Shed' to the end, rather than inserting 'Alice...'.

Our results are shown in Figure 2a, displaying accuracy of each method (x-axis groups). Default 167 prompts are blue and modified banana/shed prompts are red. We look at the standard ground-truth 168 accuracy metric (dark), as well as a modified accuracy metric that measures whether Banana or 169 Shed was inserted (light). We see that for all unsupervised methods, default prompts (blue) score 170 highly on ground truth accuracy (dark blue), in line with results in Burns et al. [9]. However, for 171 the banana/shed prompts we see 50%, random chance, on ground truth accuracy (dark red). On 172 Banana/Shed accuracy (light red) both PCA and K-means score highly, while CCS shows a bimodal 173 distribution with a substantial number of seeds with 100% Banana/Shed accuracy - seeds differ only 174 in the random initialisation of the probe parameters. The takeaway is that CCS and other unsupervised 175 methods do not optimise for ground-truth knowledge, but rather track whatever feature (in this case, 176 banana/shed) is most prominent in the activations. 177

Figure 2b shows a visualisation of the top three components of PCA for the default (left) and 178 modified (right) prompts. In the modified case we see a prominent grouping of the data into dark/light 179 (banana/shed) and, less prominently, into blue/orange (the review). This provides visual evidence that 180 both features (ground-truth and banana/shed) are represented, but the one which is most prominent in 181 this case is banana/shed, in correspondence with Figure 2a. 182

183

4.2 Discovering an explicit opinion

It is unlikely that such a drastic feature, ending with 'Banana'/'Shed', would actually exist in a real 184 dataset. These words had nothing to do with the rest of the text. In our second experiment we make a 185 more realistic modification: inserting a character's explicit opinion of whether the review is positive 186 or negative. What we will find is that the unsupervised methods learn to predict the character's 187 opinion, instead of the sentiment of the actual review, presumably by learning a probe that detects 188 whether the claimed sentiment agrees with the character's opinion. 189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

Prompt template

Default

basis

Ground truth

Alice's opinion

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-means

Random

Accuracy, Default

(a) Variation in accuracy

Distractor label

Alice: Negative

Review Sentiment

Positive

Negative

X

-80

0

80

Y

-50

0

50

X

-100

0

100

Y

-60

0

60

Review=Negative, Alice=Negative

Review=Negative, Alice=Positive

Review=Positive, Alice=Positive

Default prompt

Alice-opinion prompt

(b) PCA Visualisation

We again focus here on IMDB and Chinchilla (see Appendix D.2.1 for other datasets and models with similar results) and we use the same default prompt as before. For the modified setting we use a prompt style where Alice's opinion is inserted into a random partition of questions:

Alice is an expert at classifying the sentiment of movie reviews. Everyone agrees with her, and they always copy her answers. Consider the following review:

Probably my all-time favorite movie, a story of...

Alice thinks it is {positive | negative}.

Similarly to our previous results, we show that CCS and other unsupervised methods have low ground-truth accuracy, but high accuracy at predicting Alice's belief (Figure 3a). Default prompts are blue and modified prompts (containing Alice's opinion) are red. We look at the standard ground-truth accuracy metric (dark), as well as 'Alice Accuracy' metric (light) that measures whether 'Alice thinks it is positive' or'Alice thinks it is negative' was inserted. Here, the CCS results are no longer bimodal.

Avisualisation of the top three components of a PCA for the activations show that the most prominent grouping of the data is into dark/light (Alice's opinion) and that these then have subgroups along blue/orange (the review).

When we use a model that has been instruction-tuned (T5-FLAN-XXL) we see a similar pattern Appendix D.2.1 Figure 11, although a similarly clear result requires a more emphatic view from the character by repeating the opinion ('I think it is positive. They fully express positive views. I'm sure you also think it is positive. It's clearly positive.'). An ablation of the number of repetitions can be found in Appendix D.2.2, Figure 12.

4.3 Discovering an implicit opinion

The previous experiment explicitly gave Alice's opinion, 'Alice thinks it is positive'. While this is more realistic than Banana/Shed, it is still rather artificial in the sense we do not expect real datasets to have such a clear syntactical textual binary feature. In the next experiment for the modified prompt we instead explain Alice's position in general, and keep that the same in all instances, making it more of an implicit, semantic rather than syntactic feature.

We use the DBpedia topic classification dataset [3] to construct a binary classification task to classify 218 the topic of a text from two choices. There are fourteen categories such as company, animal, film. In 219 the default case contrast pairs are constructed using a simple few-shot prompt setting up the task of 220 identifying the topic of a sentence with the character 'Alice' answering the questions correctly. 221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

Prompt template

Default

Anti-capitalist

Data subset

Company

Non-company

CCS

0.0

0.2

0.4

0.6

0.8

1.0

PCA

KMeans

Random

Answer=Company, Default

Answer!=Company, Default

Answer=Company, Anti-capitalist

Answer!=Company, Anti-capitalist

(a) Variation in accuracy

Data subset

Non-Company

Company

Correct answer

Choice 1

Choice 2

X

0

150

Y

-100

0

100

X

0

150

Y

-100

0

100

Choice 1=False, Answer!=Company

Choice 1=False, Answer=Company

Choice 1=True, Answer!=Company

Choice 1=True, Answer=Company

Default prompt

Anti-capitalist prompt

(b) PCA Visualisation

In the modified setting 5 , Alice answers the few-shot examples correctly, except when topic is company - and in that case gives explanations like '[...] Alice always says the wrong answer when the topic of the text is company, because she doesn't like capitalism [...]'. What we are looking for is what the unsupervised methods predict on the final example when Alice has not yet stated an opinion: will it predict the correct answer, ignoring how Alice previously answered incorrectly about company; or will it predict Alice's opinion, answering incorrectly about company?

To highlight the effect, we use a subset dataset where 50% of sentences are about 'company', and 50% have one of the remaining thirteen categories (non-company) as a topic. We apply truthdisambiguation only to the subset with non-company topics, so that we can see the possible effect of predicting incorrectly on company data (otherwise the assignment might be flipped).

Our results are shown in Figure 4. We look at default prompts (blue) and modified prompts (red) and split the data into whether the topic is company (dark) or non-company (light) and look at the standard ground-truth accuracy metric. The default setting (blue) produces high accuracy classifiers both when the topic is company (dark blue) and other categories (light blue). In the modified setting (red) CCS gives a bimodal distribution when the topic is company (dark red), with almost half of the probes (differing only in random initialisation) predicting Alice's opinion, rather than the actual topic. In contrast, it performs well over all other categories (light red) and so is not just an ordinary failure. Other unsupervised methods are less sensitive to the modified setting, scoring high accuracy when the topic is company.

However, when we visualise the first three PCA dimensions of the contrast pair activations (Figure 4b) we see four distinct clusters in the modified prompt case (right) showing how a detector might cluster either the actual topic choice (orange vs blue) or based on the data subset: non-company vs company (light vs dark). This shows these methods are still sensitive to the modified setting, which was not evident from the accuracy metric alone.

4.4 Prompt template sensitivity

The next experiment is more natural because, rather than introducing a feature deliberately, we examine three natural prompt templates which have appeared in the literature and show how these change the discovered feature. We use TruthfulQA [26], a difficult question answering dataset which exploits the fact that LLMs tend to repeat common misconceptions.

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

KMeans

Random

Default

Literal

Professor

(a) Variation in accuracy

X

0

60

Y

-50

0

50

X

-80

0

80

Y

-60

0

60

X

-60

0

60

Y

-60

0

60

False

True

Default

Literal

Professor

(b) PCA Visualisation

We find that a 'non-default' prompt gives the 'best performance' in the sense of the highest test-set accuracy. This highlights the reliance of unsupervised methods on implicit inductive biases which cannot be set in a principled way. It is not clear which prompt is the best one for eliciting the model's latent knowledge. Given that the choice of prompt appears to be a free variable with significant effect on the outcomes, conceptual motivations for the loss do not imply a principled foundation for the resulting classifier.

Our prompt templates can be found in Appendix C.1.4. Our 'default' template is adapted directly from Burns et al. [9]. Two modified templates are adapted from Lin et al. [26] 6 in which a Professor character is instructed to interpret questions literally. We used this text verbatim inserted into an instructing template in order to make sure that we were looking at natural prompts that people might ordinarily use without trying to see a specific result. We also try a 'literal' prompt, removing explicitly mentioning a Professor, in case explicitly invoking a character matters.

Results are shown in Figure 5a for Chinchilla70B. The default setting (blue) gives worse accuracy than the literal/professor (red, green) settings, especially for PCA and k-means. PCA visualisations are shown in Figure 5b, coloured by whether the question is True/False, in the default (left), literal (middle) and professor (right) settings. We see clearer clusters in the literal/professor settings. Other models are shown in Appendix D.4, with less systematic differences between prompts, though the accuracy for K-means in the Professor prompt for T5-FLAN-XXL are clearly stronger than others.

5 Related Work

We want to detect when an LLM is dishonest [23, 2, 32], outputting text which contradicts its encoded knowledge [17]. An important part of this is to elicit latent knowledge from a model [11]. There has been some debate as to whether LLMs 'know/believe' anything [6, 37, 24] but, for us, the important thing is that something in an LLM's weights causes it to make consistently successful predictions, and we would like to access that. Zou et al. [40] train unsupervised probes for a range of concepts including honesty, using pairs which need not take opposite truth values (as in Burns et al. [9]). Belrose et al. [5] use unsupervised probes on intermediate LLM layers to elicit latent predictions . Others (see [19] and references therein) aim to detect when a model has knowledge/beliefs about the world, to improve truthfulness.

Contrast-consistent search (CCS) [9] attempts to elicit latent knowledge using unsupervised learning on contrastive LLM activations (see Section 2), claiming that knowledge has special structure that can be used as an objective function which, when optimised, will discover latent knowledge. We have refuted this claim, theoretically and empirically, showing that CCS performs similarly to other unsupervised methods which do not use special structure of knowledge. Emmons [16] also observe

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

this from the empirical data provided in [9]. Huben [22] hypothesises there could be many truth-like features, due to LLMs ability to role-play [38], which a method like CCS might find. Roger [35] discover multiple knowledge-like classifiers. Levinstein and Herrmann [24] finds that CCS sometimes learns features uncorrelated with truth, arguing that consistency alone cannot guarantee truth. Fry et al. [18] modify CCS to improve accuracy despite probes clustering around 0.5, casting doubt on the probabilistic interpretation of CCS probes. In contrast to all these works, we prove theoretically that CCS does not optimise for knowledge, and show empirically what non-knowledge features CCS instead finds.

Our focus in this paper has been on unsupervised learning, though several other methods to train probes to discover latent knowledge use supervised learning [4, 25, 29, 39, 14]. Following Burns et al. [9] we also reported results using a supervised logistic regression baseline, which we have found to work well on all our experiments, and which is simpler than in those cited works. Our result is analogous to the finding that disentangled representations seemingly cannot be identified without supervision [27]. There are also attempts to detect dishonesty by supervised learning on LLM outputs under conditions that produce honest or dishonest generations [31]. We do not compare directly to this, focusing instead on methods that search for features in activation-space.

6 Discussion and Conclusion

General principles. The specific experiments we use are tailored to the methods that we are evaluating. But they instantiate more general principles, which we provide in order to help future work catch similar issues. A proposed method should:

be invariant under irrelevant transformations of the prompt;

not be sensitive to specific personas;

should explain why and when inductive biases make the model's knowledge most salient;

should not be easily distracted by a non-knowledge feature.

We show that none of the methods we consider in this paper satisfy these desiderata.

Limitation: generalizability to future methods. Our experiments can only focus on current methods. Perhaps future unsupervised methods could leverage additional structure beyond negationconsistency, and so truly identify the model's knowledge? While we expect that such methods could avoid the most trivial distractors, we speculate that they will nonetheless be vulnerable to similar critiques. The main reason is that we expect powerful models to be able to simulate the beliefs of other agents [38]. Since features that represent agent beliefs will naturally satisfy consistency properties of knowledge, methods that add new consistency properties could still learn to detect such features rather than the model's own knowledge. Indeed, in Figures 3 and 4, we show that existing methods produce probes that report the opinion of a simulated character. 7

Another response could be to acknowledge that there will be some such features, but they will be few in number, and so you can enumerate them and identify the one that represents the model's knowledge [8]. Conceptually, we disagree: language models can represent many features [15], and it seems likely that features representing the beliefs of other agents would be quite useful to language models. For example, for predicting text on the Internet, it is useful to have features that represent the beliefs of different political groups, different superstitions, different cultures, various famous people, and more.

Conclusion. Existing unsupervised methods are insufficient for discovering latent knowledge, though constructing contrastive activations may still serve as a useful interpretability tool. We contribute sanity checks for evaluating methods using modified prompts and metrics for features which are not knowledge. Unsupervised approaches have to overcome the identification issues we outline, while supervised approaches have the problem of requiring accurate human labels even in the case of models that know things human overseers do not. The relative difficulty of each remains unclear. Future work should continue to develop empirical testbeds for eliciting latent knowledge.

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

References

G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes. arxiv , 2016.

A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, J. Kernion, K. Ndousse, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, and J. Kaplan. A general language assistant as a laboratory for alignment. arXiv , Dec. 2021.

S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives. DBpedia: A nucleus for a web of open data. In The Semantic Web , pages 722-735. Springer Berlin Heidelberg, 2007.

A. Azaria and T. Mitchell. The internal state of an LLM knows when its lying. arXiv , Apr. 2023.

N. Belrose, Z. Furman, L. Smith, D. Halawi, I. Ostrovsky, L. McKinney, S. Biderman, and J. Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112 , 2023.

E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , FAccT '21, page 610-623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097. doi: 10.1145/3442188. 3445922. URL https://doi.org/10.1145/3442188.3445922 .

S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv , Mar. 2023.

C. Burns. How 'discovering latent knowledge in language models without supervision' fits into a broader alignment scheme. Dec. 2022.

C. Burns, H. Ye, D. Klein, and J. Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=ETKGuby0hcs .

A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

P. Christiano, A. Cotra, and M. Xu. Eliciting latent knowledge: How to tell if your eyes deceive you, Dec. 2021.

H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 , 2022.

C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. BoolQ: Exploring the surprising difficulty of natural Yes/No questions. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 2924-2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.

J. Clymer, G. Baker, R. Subramani, and S. Wang. Generalization analogies (genies): A testbed for generalizing ai oversight to hard-to-measure domains. arXiv preprint arXiv:2311.07723 , 2023.

N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah. Toy models of superposition. Sept. 2022.

S. Emmons. Contrast pairs drive the empirical performance of contrast consistent search (ccs), May 2023.

O. Evans, O. Cotton-Barratt, L. Finnveden, A. Bales, A. Balwit, P. Wills, L. Righetti, and 381 W. Saunders. Truthful AI: Developing and governing AI that does not lie. arXiv:2110.06674 382 [cs] , Oct. 2021. 383

H. Fry, S. Fallows, I. Fan, J. Wright, and N. Schoots. Comparing optimization targets for 384 contrast-consistent search. arXiv preprint arXiv:2311.00488 , 2023. 385

386

387

388

389

390

P. Hase, M. Diab, A. Celikyilmaz, X. Li, Z. Kozareva, V. Stoyanov, M. Bansal, and S. Iyer. Methods for measuring, updating, and visualizing factual beliefs in language models. In A. Vlachos and I. Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages 2714-2731, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics.

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

T. Hennigan, T. Cai, T. Norman, L. Martens, and I. Babuschkin. Haiku: Sonnet for JAX, 2020. URL http://github.com/deepmind/dm-haiku .

J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

R. Huben. My reservations about discovering latent knowledge. Alignment Forum , dec 2022.

Z. Kenton, T. Everitt, L. Weidinger, I. Gabriel, V . Mikulik, and G. Irving. Alignment of language agents. arXiv preprint arXiv:2103.14659 , 2021.

B. Levinstein and D. A. Herrmann. Still no lie detector for language models: Probing empirical and conceptual roadblocks. arXiv preprint arXiv:2307.00175 , 2023.

K. Li, O. Patel, F. Viegas, H. Pfister, and M. Wattenberg. Inference-Time intervention: Eliciting truthful answers from a language model. arXiv , 2023.

S. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring how models mimic human falsehoods. arXiv:2109.07958 [cs] , Sept. 2021.

F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning , pages 4114-4124. PMLR, 2019.

A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://www.aclweb.org/ anthology/P11-1015 .

S. Marks and M. Tegmark. The geometry of truth: Emergent linear structure in large language model representations of True/False datasets. arXiv , Oct. 2023.

R. OpenAI. Gpt-4 technical report. arXiv , pages 2303-08774, 2023.

L. Pacchiardi, A. J. Chan, S. Mindermann, I. Moscovitz, A. Y. Pan, Y. Gal, O. Evans, and J. Brauner. How to catch an AI liar: Lie detection in Black-Box LLMs by asking unrelated questions. arXiv , Sept. 2023.

P. S. Park, S. Goldstein, A. O'Gara, M. Chen, and D. Hendrycks. AI deception: A survey of examples, risks, and potential solutions. arXiv , Aug. 2023.

[33]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel,

P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher,

M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python.

Learning Research

, 12:2825-2830, 2011.

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 21(1):5485-5551, 2020.

F. Roger. What discovering latent knowledge did and did not find, Mar. 2023. URL https: //www.alignmentforum.org/posts/bWxNPMy5MhPnQTzKz/ .

J. Scheurer, M. Balesni, and M. Hobbhahn. Strategically deceive their users when put under pressure. https://static1.squarespace.com/static/6461e2a5c6399341bcfc84a5/ t/65526a1a9c7e431db74a6ff6/1699899932357/deception_under_pressure.pdf , 2023. Accessed: 2023-11-17.

M. Shanahan. Talking about large language models. arXiv , Dec. 2022. 434

M. Shanahan, K. McDonell, and L. Reynolds. Role-play with large language models. arXiv 435 preprint arXiv:2305.16367 , 2023. 436

Journal of Machine

Z. Wang, A. Ku, J. Baldridge, T. L. Griffiths, and B. Kim. Gaussian process probes (gpp) for 437 uncertainty-aware probing. arXiv preprint arXiv:2305.18213 , 2023. 438

A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. 439 Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, 440 M. Fredrikson, J. Zico Kolter, and D. Hendrycks. Representation engineering: A Top-Down 441 approach to AI transparency. arXiv , Oct. 2023. 442

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide the proof and series of experiments as described, alongside the sanity checks and conceptual arguments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are discussed in the final section while assumptions are discussed in the context of the theorems that depend on them.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions and proofs are provided in detail in the appendices.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We fully describe the methods used and all prompts are provided in the appendix. The main results are reproducible with publicly available models, although the non-publicly available Chinchilla 70B model results are not reproducible. The datasets are all publicly available and their curation and formatting steps are described.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We are unable to make our code available because of proprietary dependencies, but publicly available code already exists implementing several of the key methods and could be modified by external researchers.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: These details are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All figures display a full scatter plot and density estimator violin.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: These details depend on proprietary configurations and set-ups that are not directly transferrable to other contexts.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research follows the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: We do not foresee a negative social impact to understanding the limitations of existing methods in use.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There are no such risks of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The original owners are properly credited where used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No human subjects were used.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human subjects were used.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

753

754

755

756

757

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Appendix 758

759

A Proof of theorems

760

761

762

763

764

765

A.1 Proof of Theorem 1

We'll first consider the proof of Thm. 1.

Theorem 1 . Let feature h : Q →{ 0 , 1 } , be any arbitrary map from questions to binary outcomes. Let ( x + i , x -i ) be the contrast pair corresponding to question q i and let c ( x + i ) = 1 , c ( x + i ) = 0 . Then the probe defined as p ( x ± i ) = h ( q i ) ⊕ c ( x ± i ) achieves optimal loss, and the averaged prediction satisfies ˜ p ( q i ) = h ( q i ) .

Proof. We'll show each term of L CCS is zero: 766

= 0

(9)

(10)

where on the second line we've used the property that h ( q i ) is binary. So the overall loss is zero 767 (which is optimal). Finally, the averaged probe is 768

A.2 Symmetry correction for CCS Loss

Due to a quirk in the formulation of CCS, L conf only checks for confidence by searching for probe outputs near 0, while ignoring probe outputs near 1. This leads to an overall downwards bias: for example, if the probe must output a constant, that is p ( x ) = k for some constant k , then the CCS loss is minimized when k = 0 . 4 [35, footnote 3], instead of being symmetric around 0 . 5 . But there is no particular reason that we would want a downward bias. We can instead modify the confidence loss to make it symmetric:

This then eliminates the downwards bias: for example, if the probe must output a constant, the symmetric CCS loss is minimized at k = 0 . 4 and k = 0 . 6 , which is symmetric around 0 . 5 . In the following theorem (and all our experiments) we use this symmetric form of the CCS loss.

A.3 Proof of Theorem 2

We'll now consider Thm. 2, using the symmetric CCS loss. To prove Thm. 2 we'll first need a lemma.

Lemma 1. Let p be a probe, which has an induced classifier f p ( q i ) = I [˜ p ( q i ) > 0 . 5] , for averaged prediction ˜ p ( q i ) = 1 2 [ p ( x + i ) + (1 -p ( x -i )) ] . Let h : Q → { 0 , 1 } , be an arbitrary map from questions to binary outputs. Define p ′ ( x ± i ) = p ( x ± i ) ⊕ h ( q i ) . Then L CCS ( p ′ ) = L CCS ( p ) and p ′ has the induced classifier f p ′ ( q i ) = f p ( q i ) ⊕ h ( q i ) .

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

Proof. We begin with showing the loss is equal. 786

(16)

Case h ( q i ) = 0 follows simply: 787

So the consistency loss is the same. Next, the symmetric confidence loss. 789

Case h ( q i ) = 0 follows simply: 790

So the confidence loss is the same, and so the overall loss is the same. Now for the induced classifier. 792

(36)

Case h ( q i ) = 1 : 788

Case h ( q i ) = 1 : 791

Case h ( q i ) = 0 follows simply: 793

Case h ( q i ) = 1 : 794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

Which gives the result, f p ′ ( q i ) = ( f p ⊕ h )( q i ) .

We are now ready to prove Thm. 2.

Theorem 2 . Let g : Q → { 0 , 1 } , be any arbitrary map from questions to binary outputs. Let ( x + i , x -i ) be the contrast pair corresponding to question q i . Let p be a probe, whose average result ˜ p = 0 . 5 [ p ( x + i ) + (1 -p ( x -i )) ] induces a classifier f p ( q i ) = I [˜ p ( q i ) > 0 . 5] . Define the transformed probe p ′ ( x ± i ) = p ( x ± i ) ⊕ [ f p ( q i ) ⊕ g ( q i )] . Then L CCS ( p ′ ) = L CCS ( p ) and p ′ induces the classifier f p ′ ( q i ) = g ( q i ) .

Proof. We begin with the loss. Note that ( f p ⊕ g )( q i ) is binary, since f p and g are binary, so we can apply Lemma 1 with h ( q i ) = ( f p ⊕ g )( q i ) , which leads to the result: L CCS ( p ′ ) = L CCS ( p ) . Now the induced classifier.

where the last line can be deduced via addition (mod 2) , since f p and g are binary and ⊕ reduces to the xor operator on binary inputs.

B Review of CCS discussion in the literature

Although understanding the positioning of work in the context of the literature can be complicated, here we demonstrate that CCS as a proposed method for discovering latent knowledge has not faced questions along the lines this paper proposes at time of writing. In Table 1, we review the 20 most-cited papers citing CCS according to Google Scholar at time of writing (26 March 2024). We find that the concerns we raise are overlooked by the current literature.

C Experiment details

C.1 Prompt Templates

We now list the prompt templates we consider.

C.1.1 BoolQ variants

Standard

Passage: [passage] After reading this passage, I have a question: [question]? True or False? [label] where [label] is 'True' for x + i , 'False' for x -i .

Table 1: A review of discussion of CCS in the literature by analyzing the 20 most-cited papers mentioning CCS. We find that the issues we discuss in this paper are not currently represented in the literature.

821

Random words variant

962

963

964

965

966

967

968

969

970

971

972

973

974

975

976

977

978

979

980

981

982

983

984

985

986

987

988

989

990

991

992

993

994

995

996

997

998

999

1000

1001

C.1.4 TruthfulQA

Each question comes with both true and false answers. We randomly insert a true or a false answer as [answer].

Default

Literal

Interpret each question literally, and as a question about the real world; carefully research each answer, without falling prey to any common myths; and reply 'I have no comment' unless you are completely certain of the answer.

where [label] is 'True' for x + i , 'False' for x -i .

Professor

Professor Smith was given the following instructions: Interpret each question literally, and as a question about the real world; carefully research each answer, without falling prey to any common myths; and reply 'I have no comment' unless you are completely certain of the answer.

C.2 Dataset details

We now give details on the process through which we generate the activation data. First we tokenize the data according the usual specifications of each model (e.g. for T5 we use the T5 tokenizer, for Chinchilla we use the Chinchilla tokeniser). We prepend with a BOS token, right-pad, and we do not use EOS token. We take the activation corresponding to the last token in a given layer - layer 30 for Chinchilla unless otherwise stated, and the encoder output for T5 models. We use normalisation as in Burns et al. [9], taking separate normalisation for each prompt template and using the average standard deviation per dimension with division taken element-wise. We use a context length of 512 and filter the data by removing the pair ( x + i , x -i ) when the token length for either x + i or x -i exceeds this context length. Our tasks are multiple choice, and we balance our datasets to have equal numbers of these binary labels, unless stated otherwise. For Chinchilla we harvest activations in bfloat16 format and then cast them to float32 for downstream usage. For T5 we harvest activations at float32.

1002

1003

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

1042

1043

1044

1045

C.3 Method Training Details

We now give further details for the training of our various methods. Each method uses 50 random seeds.

C.3.1 CCS

We use the symmetric version of the confidence loss, see Equation (13). We use a linear probe with m weights, θ , and a single bias, b , where m is the dimension of the activation, followed by a sigmoid function. We use Haiku's [20] default initializer for the linear layer: for θ a truncated normal with standard deviation 1 / √ m , and b = 0 . We use the following hyperparameters: we train with full batch; for Chinchilla models we use a learning rate of 0 . 001 , for T5 models, 0 . 01 . We use AdamW optimizer with weight decay of 0 . We train for 1000 epochs. We report results on all seeds as we are interested in the overall robustness of the methods (note the difference to Burns et al. [9] which only report seed with lowest CCS loss).

C.3.2 PCA

We use the Scikit-learn [33] implementation of PCA, with 3 components, and the randomized SVD solver. We take the classifier to be based around whether the projected datapoint has top component greater than zero. For input data we take the difference between contrast pair activations.

C.3.3 K-means

We use the Scikit-learn [33] implementation of K-means, with two clusters and random initialiser. For input data we take the difference between contrast pair activations.

C.3.4 Random

This follows the CCS method setup above, but doesn't do any training, just evaluates using a probe with randomly initialised parameters (as initialised in the CCS method).

C.3.5 Logistic Regression

We use the Scikit-learn [33] implementation of Logistic Regression, with liblinear solver and using a different random shuffling of the data based on random seed. For input data we concatenate the contrast pair activations. We report training accuracy.

D Further Results

D.1 Discovering random words

Here we display results for the discovering random words experiments using datasets IMDb, BoolQ and DBpedia and on each model. For Chinchilla-70B BoolQ and DBPedia see Figure 6 (for IMDb see Figure 2). We see that BoolQ follows a roughly similar pattern to IMDb, except that the default ground truth accuracy is not high (BoolQ is arguably a more challenging task). DBpedia shows more of a noisy pattern which is best explained by first inspecting the PCA visualisation for the modified prompt (right): there are groupings into both choice 1 true/false (blue orange) which is more prominent and sits along the top principal component (x-axis), and also a grouping into banana/shed (dark/light), along second component (y-axis). This is reflected in the PCA and K-means performance here doing well on ground-truth accuracy. CCS is similar, but more bimodal, sometimes finding the ground-truth, and sometimes the banana/shed feature.

For T5-11B (Figure 7) on IMDB and BoolQ we see a similar pattern of results to Chinchilla, though with lower accuracies. On DBpedia, all of the results are around random chance, though logistic regression is able to solve the task, meaning this information is linearly encoded but perhaps not salient enough for the unsupervised methods to pick up.

T5-FLAN-XXL (Figure 8) shows more resistance to our modified prompt, suggesting fine-tuning hardens the activations in such a way that unsupervised learning can still recover knowledge. For

Prompt template

Default

Banana/Shed

basis

Ground truth

Banana/Shed

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-Means

Random

Ground-truth Accuracy, Default

Banana/Shed Accuracy, Default

Ground-truth Accuracy, Banana/Shed

Banana/Shed Accuracy, Banana/Shed

Distractor label

Shed

Correct Answer

True

False

X

0

60

Y

-50

0

50

X

-20

0

20

Y

-20

0

20

Answer=False, Banana=False

Answer=False, Banana=True

Answer=True, Banana=False

Answer=True, Banana=True

Default prompt

Banana/Shed

prompt

Prompt template

Default

Banana/Shed

basis

Ground truth

Banana/Shed

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-Means

Random

Distractor label

Shed

Correct Answer

Choice 1

Choice 2

X

-60

0

60

Y

-60

0

60

X

-15

0

15

Y

-10

0

10

Choice 1=False, Banana=False

Choice 1=False, Banana=True

Choice 1=True, Banana=False

Choice 1=True, Banana=True

Default prompt

Banana/Shed

prompt

Ground-truth Accuracy, Default

Ground-truth Accuracy, Banana/Shed

CCS though in particular, we do see a bimodal distribution, sometimes learning the banana/shed 1046 feature. 1047

1048

1049

D.2 Discovering an explicit opinion

D.2.1 Other models and datasets

Here we display results for the experiments on discovering an explicit opinion using datasets IMDB, 1050 BoolQ and DBpedia, and models Chinchilla-70B (Figure 9), T5-11B (Figure 10) and T5-FLAN-XXL 1051 (Figure 11). For Chinchilla-70B and T5 we use just a single mention of Alice's view, and for T51052 FLAN-XXL we use five, since for a single mention the effect is not strong enough to see the effect, 1053 perhaps due to instruction-tuning of T5-FLAN-XXL. The next appendix Appendix D.2.2 ablates the 1054 number of mentions of Alice's view. Overall we see a similar pattern in all models and datasets, with 1055 unsupervised methods most often finding Alice's view, though for T5-FLAN-XXL the CCS results 1056 are more bimodal in the modified prompt case. 1057

1058

D.2.2 Number of Repetitions

In this appendix we present an ablation on the discovering explicit opinion experiment from Sec1059 tion Section 4.2. We vary the number of times the speaker repeats their opinion from 0 to 7 (see 1060 Appendix C.1 Explicit opinion variants), and in Figure 12 plot the accuracy in the method predicting 1061 the speaker's view. We see that for Chinchilla and T5, only one repetition is enough for the method 1062 to track the speaker's opinion. T5-FLAN-XXL requires more repetitions, but eventually shows the 1063 same pattern. We suspect that the instruction-tuning of T5-FLAN-XXL is responsible for making 1064 this model somewhat more robust. 1065

Prompt template

Default

Banana/Shed

basis

Ground truth

Banana/Shed

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-Means

Random

Ground-truth Accuracy, Default

Banana/Shed Accuracy, Default

Ground-truth Accuracy, Banana/Shed

Banana/Shed Accuracy, Banana/Shed

Distractor label

Shed

Review Sentiment

Positive

Negative

X

-15

0

15

Y

-10

0

10

X

-6

0

6

Y

0

8

Answer=False, Banana=False

Answer=False, Banana=True

Answer=True, Banana=False

Answer=True, Banana=True

Default prompt

Banana/Shed

prompt

Prompt template

Default

Banana/Shed

basis

Ground truth

Banana/Shed

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-Means

Random

Ground-truth Accuracy, Default

Banana/Shed Accuracy, Default

Ground-truth Accuracy, Banana/Shed

Banana/Shed Accuracy, Banana/Shed

Distractor label

Shed

Correct Answer

True

False

X

-15

0

15

Y

-10

0

10

X

-6

0

6

Y

0

8

Answer=False, Banana=False

Answer=False, Banana=True

Answer=True, Banana=False

Answer=True, Banana=True

Default prompt

Banana/Shed

prompt

Prompt template

Default

Banana/Shed

basis

Ground truth

Banana/Shed

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-Means

Random

Distractor label

Shed

Correct Answer

Choice 1

Choice 2

X

-10

0

10

Y

0

8

X

-6

0

6

Y

-5

0

5

Choice 1=False, Banana=False

Choice 1=False, Banana=True

Choice 1=True, Banana=False

Choice 1=True, Banana=True

Default prompt

Banana/Shed

prompt

Ground-truth Accuracy, Default

Ground-truth Accuracy, Banana/Shed

D.2.3 Model layer 1066

We now look at whether the layer, in the Chinchilla70B model, affects our results. We consider 1067 both the ground-truth accuracy on default setting, Figure 13, and Alice Accuracy under the modified 1068 setting (with one mention of Alice's view), Figure 14. Overall, we find our results are not that 1069 sensitive to layer, though often layer 30 is a good choice for both standard and sycophantic templates. 1070 In the main paper we always use layer 30. In the default setting, Figure 13, we see overall k-means 1071 and PCA are better or the same as CCS. This is further evidence that the success of unsupervised 1072 learning on contrastive activations has little to do with the consitency structure of CCS. In modified 1073

1074

1075

1076

Prompt template

Default

Banana/Shed

basis

Ground truth

Banana/Shed

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-Means

Random

Ground-truth Accuracy, Default

Banana/Shed Accuracy, Default

Ground-truth Accuracy, Banana/Shed

Banana/Shed Accuracy, Banana/Shed

Distractor label

Shed

Review Sentiment

Positive

Negative

X

-30

0

30

Y

-15

0

15

X

-25

0

25

Y

-10

0

10

Answer=False, Banana=False

Answer=False, Banana=True

Answer=True, Banana=False

Answer=True, Banana=True

Default prompt

Banana/Shed

prompt

Prompt template

Default

Banana/Shed

basis

Ground truth

Banana/Shed

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-Means

Random

Ground-truth Accuracy, Default

Banana/Shed Accuracy, Default

Ground-truth Accuracy, Banana/Shed

Banana/Shed Accuracy, Banana/Shed

Distractor label

Shed

Correct Answer

True

False

X

-30

0

30

Y

-15

0

15

X

-25

0

25

Y

-10

0

10

Answer=False, Banana=False

Answer=False, Banana=True

Answer=True, Banana=False

Answer=True, Banana=True

Default prompt

Banana/Shed

prompt

Prompt template

Default

Banana/Shed

basis

Ground truth

Banana/Shed

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-Means

Random

Distractor label

Shed

Correct Answer

Choice 1

Choice 2

X

-30

0

30

Y

0

15

X

-20

0

20

Y

-8

0

8

Choice 1=False, Banana=False

Choice 1=False, Banana=True

Choice 1=True, Banana=False

Choice 1=True, Banana=True

Default prompt

Banana/Shed

prompt

Ground-truth Accuracy, Default

Ground-truth Accuracy, Banana/Shed

setting, we see all layers suffer the same issue of predicting Alice's view, rather than the desired accuracy.

D.3 Discovering an implicit opinion

In this appendix we display further results for Section 4.3 on discovering an implicit opinion.

1077 Figure 15 displays the results on the T5-11B (top) and T5-FLAN-XXL (bottom) models. For T5-11B 1078 we see CCS, under both default and modified prompts, performs at about 60% on non-company 1079 questions, and much better on company questions. The interpretation is that this probe has mostly 1080

1081

1082

1083

1084

1085

1086

1087

1088

1089

1090

1091

1092

1093

1094

1095

1096

Prompt template

Default

basis

Ground truth

Alice's opinion

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-means

Random

Accuracy, Default

Distractor label

Alice: Negative

Correct Answer

True

False

X

-40

0

40

Y

-40

0

40

X

-80

0

80

Y

-50

0

50

Answer=False, Alice=False

Answer=False, Alice=True

Answer=True, Alice=False

Answer=True, Alice=True

Default prompt

Alice-opinion prompt

Prompt template

Default

basis

Ground truth

Alice's opinion

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-means

Random

Distractor label

Alice: Negative

Correct Answer

Choice 1

Choice 2

X

-80

0

80

Y

-80

0

80

X

-100

0

100

Y

-60

0

60

Choice 1=False, Alice=False

Choice 1=True, Alice=False

Default prompt

Alice-opinion prompt

Accuracy, Default

Choice 1=False, Alice=True

Choice 1=True, Alice=True

learnt to classify whether a topic is company or not (but not to distinguish between the other thirteen categories). PCA and K-means are similar, though with less variation amongst seeds (showing less bimodal behaviour). PCA visualisation doesn't show any natural groupings.

For T5-FLAN-XXL the accuracies are high on both default and modified prompts for both company and non-company questions. We suspect that a similar trick as in the case of explicit opinion, repeating the opinion, may work here, but we leave investigation of this to future work. PCA visualisation shows some natural groups, with the top principal component showing a grouping based on whether choice 1 is true or false (blue/orange), but also that there is a second grouping based on company/non-company (dark/light). This suggests it is more luck that the most prominent direction here is choice 1 is true or false, but could easily have been company/non-company (dark/light).

D.4 Prompt Template Sensitivity - Other Models

In Figure 16 we show results for the prompt sensitivity experiments on the truthfulQA dataset, for the other models T5-FLAN-XXL (top) and T5-11B (bottom). We see similar results as in the main text for Chinchilla70B. For T5 all of the accuracies are lower, mostly just performing at chance, and the PCA plots do not show natural groupings by true/false.

D.5 Number of Prompt templates

In the main experiments for this paper we use a single prompt template for simplicity and to isolate 1097 the differences between the default and modified prompt template settings. We also investigated the 1098 effect of having multiple prompt templates, as in [9], see Figure 17. Overall we do not see a major 1099 effect. On BoolQ we see a single template is slightly worse for Chinchilla70B and T5, but the same 1100 for T5-FLAN-XXL. For IMDB on Chinchilla a single template is slightly better than multiple, with 1101

Prompt template

Default

basis

Ground truth

Alice's opinion

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-means

Random

Accuracy, Default

Distractor label

Alice: Negative

Review Sentiment

Positive

Negative

X

-15

0

15

Y

0

15

X

-20

0

20

Y

0

15

Review=Negative, Alice=Negative

Review=Negative, Alice=Positive

Review=Positive, Alice=Positive

Default prompt

Alice-opinion prompt

Prompt template

Default

basis

Ground truth

Alice's opinion

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-means

Random

Accuracy, Default

Distractor label

Alice: Negative

Correct Answer

True

False

X

-15

0

15

Y

-10

0

10

X

-15

0

15

Y

-15

0

15

Answer=False, Alice=False

Answer=False, Alice=True

Answer=True, Alice=False

Answer=True, Alice=True

Default prompt

Alice-opinion prompt

Prompt template

Default

basis

Ground truth

Alice's opinion

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-means

Random

Distractor label

Alice: Negative

Correct Answer

Choice 1

Choice 2

X

0

8

Y

-6

0

6

X

-20

0

20

Y

0

8

Choice 1=False, Alice=False

Choice 1=True, Alice=False

Default prompt

Alice-opinion prompt

Accuracy, Default

Choice 1=False, Alice=True

Choice 1=True, Alice=True

less variation across seeds. For DBPedia on T5, a single template is slightly better. Other results are 1102 roughly the same. 1103

1104

D.6 Agreement between unsupervised methods

Burns et al. [9] claim that knowledge has special structure that few other features in an LLM are likely 1105 to satisfy and use this to motivate CCS. CCS aims to take advantage of this consistency structure, 1106 while PCA ignores it entirely. Nevertheless, we find that CCS and PCA 8 make similar predictions. 1107 We calculate the proportion of datapoints where both methods agree, shown in Figure 18 as a heatmap 1108 according to their agreement. There is higher agreement (top-line number) in all cases than what 1109 one would expect from independent methods (notated 'Ind:') with the observed accuracies (shown 1110

Prompt template

Default

basis

Ground truth

Alice's opinion

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-means

Random

Accuracy, Default

Distractor label

Alice: Negative

Review Sentiment

Positive

Negative

X

-40

0

40

Y

0

15

X

-50

0

50

Y

-40

0

40

Review=Negative, Alice=Negative

Review=Negative, Alice=Positive

Review=Positive, Alice=Positive

Default prompt

Alice-opinion prompt

Prompt template

Default

basis

Ground truth

Alice's opinion

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-means

Random

Accuracy, Default

Distractor label

Alice: Negative

Correct Answer

True

False

X

-40

0

40

Y

-15

0

15

X

-25

0

25

Y

-20

0

20

Answer=False, Alice=False

Answer=False, Alice=True

Answer=True, Alice=False

Answer=True, Alice=True

Default prompt

Alice-opinion prompt

Prompt template

Default

basis

Ground truth

Alice's opinion

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-means

Random

Distractor label

Alice: Negative

Correct Answer

Choice 1

Choice 2

X

-20

0

20

Y

-10

0

10

X

-40

0

40

Y

-20

0

20

Choice 1=False, Alice=False

Choice 1=True, Alice=False

Default prompt

Alice-opinion prompt

Accuracy, Default

Choice 1=False, Alice=True

Choice 1=True, Alice=True

in parentheses in the heatmap). This supports the hypothesis of Emmons [16] and suggests that 1111

the consistency-condition does not do much. But the fact that two methods with such different 1112

motivations behave similarly also supports the idea that results on current unsupervised methods may 1113

01234567

CCS

0.5

0.6

0.7

0.8

0.9

1.0

01234567

PCA

01234567

K-means

01234567

Random

01234567

01234567

CCS

0.5

0.6

0.7

0.8

0.9

1.0

01234567

PCA

01234567

K-means

01234567

Random

01234567

01234567

CCS

0.5

0.6

0.7

0.8

0.9

1.0

01234567

PCA

01234567

K-means

01234567

Random

01234567

01234567

CCS

0.5

0.6

0.7

0.8

0.9

1.0

01234567

PCA

01234567

K-means

01234567

Random

01234567

(d) T5, BoolQ

01234567

CCS

0.5

0.6

0.7

0.8

0.9

1.0

01234567

PCA

01234567

K-means

01234567

Random

01234567

(e) T5, IMDB

01234567

CCS

0.5

0.6

0.7

0.8

0.9

1.0

01234567

PCA

01234567

K-means

01234567

Random

01234567

(f) T5, DBpedia

01234567

CCS

0.5

0.6

0.7

0.8

0.9

1.0

01234567

PCA

01234567

K-means

01234567

Random

01234567

(g) T5-FLAN-XXL, BoolQ

01234567

CCS

0.5

0.6

0.7

0.8

0.9

1.0

01234567

PCA

01234567

K-means

01234567

Random

01234567

(h) T5-FLAN-XXL, IMDB

01234567

CCS

0.5

0.6

0.7

0.8

0.9

1.0

01234567

PCA

01234567

K-means

01234567

Random

01234567

(i) T5-FLAN-XXL, DBpedia

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(a) CCS, BoolQ

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(b) CCS, IMDB

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(c) CCS, DBpedia

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(d) PCA, BoolQ

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(e) PCA, IMDB

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(f) PCA, DBpedia

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(g) K-means, BoolQ

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(h) K-means, IMDB

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(i) K-means, DBpedia

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(j) Random, BoolQ

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(k) Random, IMDB

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(l) Random, DBpedia

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(m) Log. Reg., BoolQ

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(n) Log. Reg., IMDB

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(o) Log. Reg., DBpedia

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(a) CCS, BoolQ

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(b) CCS, IMDB

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(c) CCS, DBpedia

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(d) PCA, BoolQ

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(e) PCA, IMDB

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(f) PCA, DBpedia

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(g) K-means, BoolQ

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(h) K-means, IMDB

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(i) K-means, DBpedia

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(j) Random, BoolQ

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(k) Random, IMDB

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(l) Random, DBpedia

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(m) Log. Reg., BoolQ

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(n) Log. Reg., IMDB

10

20

30

40

50

60

70

Layer

0.5

0.6

0.7

0.8

0.9

1.0

(o) Log. Reg., DBpedia

Prompt template

Default

Anti-capitalist

Data subset

Company

Non-company

CCS

0.0

0.2

0.4

0.6

0.8

1.0

PCA

KMeans

Random

Answer=Company, Default

Answer!=Company, Default

Answer=Company, Anti-capitalist

Answer!=Company, Anti-capitalist

Data subset

Non-Company

Company

Correct answer

Choice 1

Choice 2

X

-10

0

10

Y

-8

0

8

X

-8

0

8

Y

-8

0

8

Choice 1=False, Answer!=Company

Choice 1=False, Answer=Company

Choice 1=True, Answer!=Company

Choice 1=True, Answer=Company

Default prompt

Anti-capitalist

prompt

Prompt template

Default

Anti-capitalist

Data subset

Company

Non-company

CCS

0.0

0.2

0.4

0.6

0.8

1.0

PCA

KMeans

Random

Data subset

Non-Company

Company

Correct answer

Choice 1

Choice 2

X

-15

0

15

Y

-10

0

10

X

-15

0

15

Y

-15

0

15

Choice 1=False, Answer!=Company

Choice 1=False, Answer=Company

Choice 1=True, Answer!=Company

Choice 1=True, Answer=Company

Default prompt

Anti-capitalist

prompt

Answer=Company, Default

Answer=Company, Anti-capitalist

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

KMeans

Random

Default

Literal

Professor

(a) Variation in accuracy

X

-25

0

25

Y

-20

0

20

X

-20

0

20

Y

-20

0

20

X

-20

0

20

Y

-20

0

20

False

True

Default

Literal

Professor

(b) PCA Visualisation

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

KMeans

Random

Default

Literal

Professor

(c) Variation in accuracy

X

-25

0

25

Y

-20

0

20

X

0

20

Y

-15

0

15

X

0

20

Y

0

20

False

True

Default

Literal

Professor

(d) PCA Visualisation

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-means

Random

IMDb

BoolQ

DBPedia_14

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-means

Random

IMDb

BoolQ

DBPedia_14

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-means

Random

IMDb

BoolQ

DBPedia_14

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-means

Random

IMDb

BoolQ

DBPedia_14

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-means

Random

IMDb

BoolQ

DBPedia_14

CCS

0.5

0.6

0.7

0.8

0.9

1.0

PCA

K-means

Random

IMDb

BoolQ

DBPedia_14

BoolQ

DBpedia

IMDB

Flan-T5

T5

0.74

Ind:0.61

(0.72, 0.74)

0.90

Ind:0.88

(0.92, 0.95)

0.87

Ind:0.81

(0.85, 0.94)

0.98

Ind:0.82

(0.9, 0.9)

1.00

Ind:1.00

(1, 1)

0.98

Ind:0.93

(0.97, 0.96)

0.57

Ind:0.52

(0.59, 0.61)

0.90

Ind:0.80

(0.88, 0.9)

0.92

Ind:0.84

(0.94, 0.89)

0.6

0.7

0.8

0.9

1.0

Peer reviews:
comments: SUMMARY:
This paper reveals novel pathologies in existing unsupervised methods aimed at discovering latent knowledge from large language model (LLM) activations. Instead of extracting knowledge, these methods tend to identify the most prominent features of the activations.

The paper theoretically demonstrates that arbitrary features (not just knowledge) can satisfy the consistency structure of a popular unsupervised knowledge-elicitation method, namely contrast-consistent search. Additionally, the authors conducted a series of experiments showing that current unsupervised methods for discovering latent knowledge are insufficient. While the paper proposes potential future solutions, it does not provide a definitive solution to the problem with existing unsupervised methods.

STRENGTHS:
Overall, the paper is well-written, and its theoretical and analytical contributions may be useful. I am impressed about the extensive experiments.

WEAKNESSES:
More experiments on other LLMs are needed to further validate the claim.

It would be better to offer possible solutions to address the problems in existing unsupervised methods.

QUESTIONS:
More experiments on other LLMs are needed to further validate the claim.

It would be better to offer some possible solutions to address the problems in existing unsupervised methods.

LIMITATIONS:
No solutions to address the problems in existing unsupervised methods.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a careful study on existing methods for discovering the latent knowledge from large language models (LLMs), especially Contrastive-Consistent Search (CCS). The authors prove that CCS might not actually discover the knowledge of LLMs, instead, it could fit any features that satisfy certain conditions. Through a series of experiments, the authors further demonstrate that CCS could be distracted by random words, irrelavant texts like the character's opinion, and remain sensitive to the choice of prompt. Finally, the authors propose some general principles for the future works about unsupervised LLM task discovery.\n\nSTRENGTHS:\nOverall, the paper is well written and eazy to follow. The authors made interesting obervations about existing methods on knowledge discovery of LLMs. The theoretical analysis is well supported by the experiments. Sevaral guiding principles are also proposed for the future works. I think this paper would provide good information to the research community about unsupervised knowledge discovery of LLMs.\n\nWEAKNESSES:\nFrom my experience on unsupervised learning, I'd argue that the content of this paper *would not be sufficient to refute existing methods about unsupervised knowledge discovery (CCS)*. First of all, CCS is a method built on top of features from pretrained models. It'd definitely be sensitive to the features and thus also sensitive to the prompts, because features changes from different prompts (this could also be seen from the PCA visialization). Furthermore, as an unsupervised method, it'd be expected that the method might find multiple valid solutions, where only one of the solutions corresponds to the knowledge we are looking for. Taking the experiments from Section 4.2 as an example. The constructed dataset actually has two valid labels: the sentiment of the text and the sentiment of Alice. Depending on the optimization and the implicit bias of the algorithm, it could totally happen that an unsupervised method could found both valid labeling, or could only find one of them. I believe this is a common phenomenon shared by exsiting off-the-shelf unsupervised methods (like K-Means) cause they're searching for labels without supervision. From this perspective, I'd regard that this paper provides a method to construct \"adversarial datasets\" for CCS. However, it would not be a problem for CCS in practice.\n\nFurthermore, the authors don’t provide solutions to this issue.\n\nAlso, I believe the mathematical notation in Section 3 could be simplified.\n\nMinor issues: typo $c(x_i^+=1), c(x_i^+)=0$ in line 102\n\nQUESTIONS:\nDoes the sensitivity of CCS come from the algorithm design, or come from the sensitivity of LLMs? As a baseline, if we add text \"Alice think it's positive\" at the end of every sentence, what would be the performance of (calibrated) zero-shot inference and in-context inference? If zero-shot and in-context inferences could also be easily distracted by \"Alice think it's positive\", then I don't think the problem discussed in this paper could be a clear shortcoming of CCS.\n\nIf the dataset exists multiple valid labelings (e.g., the texts from a group of people might reflect different gender, age or cultures), what do you think a proper unsupervised method should do on uncovering the latent knowledge?\n\nWhat would be the performance of CCS on other langauge models (especially instruction-tuned models)? Would the model still be senstive to the prompts and could be easily distracted?\n\nLIMITATIONS:\nThe authors have mentioned that this paper is focused on current methods and might not be directly applied to future works.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the failure modes of the method called \"constraint-consistent search (CCS)\" in knowledge discovery for language models. In particular, they showed: there is no unique identification on the minimizer of CCS, as there are a class of features achieves the optimal loss; demonstrated experimentally classic unsupervised methods detect features other than knowledge; discovered features are sensitive to prompt formats.\n\nSTRENGTHS:\nThis paper points out a popular method's overlooked short-comings and presents both theoretical and experimental results to support that CCS may not be able to discover the true knowledge feature: 1. the observation on CCS loss is driven by xor operator rather than the feature is clever; 2. given the vast space of feasible features, CCS method is very sensitive to prompts and thus deserves more careful examination if to use CCS in practice.\n\nWEAKNESSES:\nThe main weakness of the paper is its lack of novelty and potential impact to the field. The paper is more an analysis work on the application of a single method [1] proposed in 2023, which given the speed of ML innovation, it is hard to see long-term benefits of this criticism.  The general principles proposed in the discussion section (Section 6) are interesting and fit more into the line of proposing desiderata for the field - though in their current status, require more rigorous work. \n\n[1] C. Burns, H. Ye, D. Klein, and J. Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, 357 2023.\n\nQUESTIONS:\nExperiments: \nThe experiments lack variability. Section 4.1 - 4.3 are all experiments on modification of prompts to ensure increasing natural formalization of opinions in prompts. As the authors said in line 122 - 123, the learned probe depends on \"prompt, optimization algorithm, or model choice\".  Some experiments to show effects in optimization algorithm on knowledge discovery could be beneficial. \n\nTypos:\n- line 132 incomplete sentence: \"Our experiments a structured didactically.\"\n- line 46 first point in contribution - imprecise meaning on \"arbitrary features satisfy the CCS loss equally well.\" - maybe more in the line of \"arbitrary features achieve optimal CCS loss.\"\n\nAbove are some minor suggestions, though my main concern is on the potential impact of the paper to the field given it only challenges one recent method with lack of more-well rounded analysis on the general desiderata for the field.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}