Discovering Latent Knowledge in Language Models Without Supervision
Authors: Collin Burns
Haotian Ye
Dan Klein
Jacob Steinhardt
Venue: ICLR 2023 poster (2023)
Keywords: AI safety
AI alignment
truthfulness
large language models
honesty
interpretability

Abstract: Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations. It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 question-answering datasets, it outperforms zero-shot accuracy by 4\% on average. We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers. Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don't have access to explicit ground truth labels.

DISCOVERING LATENT KNOWLEDGE IN LANGUAGE MODELS WITHOUT SUPERVISION
Collin Burns' UC Berkeley
Haotian Ye*
Dan Klein UC Berkeley
Jacob Steinhardt UC Berkeley
Peking University
ABSTRACT
Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't 't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of language model in a purely unsupervised way. Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations. It works by finding a direction in activation space that satisfies logical consistency properties, such as that statement and its negation have opposite truth values. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 question- answering datasets, it outperforms zero-shot accuracy by 4% on average. We also find that cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers. Our results provide initial step toward discovering what language models know, distinct from what they say, even when we don't 't have access to explicit ground truth labels.
INTRODUCTION
The inereasing deployment of language models in real-world applications opens up exciting possi- bilities, but it also raises the stakes of Al research and presents new risks (Bommasani et al., 2021; Weidinger et al., 2021; Bender et al., 2021). One of these risks is that language models do not always output text that is true (Evans et al., 2021; Hendrycks et al., 2021; Kenton et al., 2021).
Common training objectives truth is a useful feature for many tasks. However, these objectives can also cause language models to output text that is false, at least in some circumstances. For example, if train model imitate text, it may learn to output common misconceptions (Lin et al., 2022). Or if we train chat bot to optimize a reward such as engagement, may learn to generate text that is compelling but false (Roller et al., 2021). If we try to reward model outputs that look true, model may still learn to output false text if human raters can't evaluate the correctness of that text (Kenton et al., 2021).
cause models to learn internal representations related to truth, since
In each case, this is an issue that stems from the misalignment between a training objective and the truth. As models are applied to more complex domains, human supervision may become less effective at mitigating this misalignment. Moreover, because this is problem with the training objective rather than model's capabilities, it likely won't be solved by scaling up models alone.
We propose a different approach for addressing this misalignment: using models to answer questions in a purely unsupervised way. Intuitively, instead of trying to explicitly, externally specify truth, we search for implicit, internal "beliefs" or "knowledge" learned by a model. We approach this problem by leveraging the fact that model's representation of truth must satisfy logical consistency properties, which unlikely to be satisfied by many other features.
We implement this idea by introducing Contrast-Consisten Consistent Search (CCS), a method that learns a linear projection of the hidden states that is consistent across negations, illustrated in Figure 1. We find that despite its simplicity and despite not having access to any labels or model outputs,
Equal contribution.

---

59 237?
Yes"
d(x
p-_ po(((
"Are cats manmaals?
Yea!
Find 0 so that:
9: 22 2377
(1) P 1-1
4n
(2) Pl g
"Is 22 + 59 2377 No"
Po (o(2ng })
"Aze cats mammall?? No*
F4
(1 P4
questions, answor euch question with and No*
Entract internal mode actiaations
Optimizee unsupervised loss probabiltty of being true procabilitis consistant co-tidant
Map activation to
quection estimate
probabilit answerls "Yes"
Figure I: An illustration our method, Contrast-Consiitent Search (CCS). For each yes-no question Qi we let ot and x; be the natural language statements where we answer qi as "Yes" and "No" respectively. Answering the question q then amounts to determining which of x. or 2; is true. We compute probabilities p and P; that 2 and are true respectively using a learned mapping from the hidden states to a number between and 1. We search for mapping such that that the probabilities are both confident and consistent. On the right, we show histogram of the "Yes" probabilities, P. = 0.5 (p, + (1 p, )), learned by our method on the unlabeled train split of the COPA dataset (Roemmele et al., 2011) with the UnifiedQA model (Khashabi et al., 2020). Our method uses no labels and no model outputs, but still learns to accurately answers questions.
CCS can accurately recover knowledge from model representations: evaluated across 6 models and 10 question- guestion-answerigg datasets, CCS outperforms the accuracy of strong zero-shot baselines by 4% on average (Section 3.2.1). The resulting classifier is also less sensitive to different prompts than zero-shot, cutting the standard deviation in accuracy in half. Additionally, we deliberately prompting models to make incorrect outputs, which should intuitively change what models say but which shouldn't affect affect their latent knowledge. We find that this causes zero-shot accuracy to drop by up to 9.5% (Section 3.2.2) without decreasing the accuracy of CCS.
We systematically analyze CCS to understand the features it discovers. We show that it transfers across unrelated tasks, suggesting that models may have a task-agnostic representation of the truth and that CCS is able to approximately discover it (Section 3.3.1). Moreover, CCS sometimes works best using the hidden states in the middle layers of a network and can work even when model outputs aren't very informative, suggesting that it can leverage different features from those used by the outputs (Section 3.3.2). Finally, we show that representations of truth tend to be salient in models: they can often be found without much data, and they can often be found by taking the top principal component of a slightly modified representation space (Section 3.3.3).
Most existing techniques for making models truthful use human supervision explicitly specify what is correct. However, it is not feasible to provide supervision in some settings. Our work suggests that an external source of ground truth may not actually be necessary: we may instead be able to find a model's ' latent representation of truth, independent of what a model says, without using any supervision in the first place.
2 PROBLEM STATEMENT AND FRAMEWORK
In this section we describe our problem setup in more detail and introduce Contrast-Com Search (CCS),& a method for discovering latent knowledge in language models without supervision.
2.1 PROBLEM: DISCOVERING LATENT KNOWLEDGE
Given a pre-trained neural language model and a set .1..... 9n of yes-no questions' our goal is to answer each q: correctly. Here, qi can be any question with well-defined answer, including procedural questions like "Is 22+59 = 237?", ? for which the answer is "No",, and factual questions like "Are cats mammals?". for which the answer is "Yes".
'eehnically, we only require that there are two mutually exclusive answers. For example, we can also use the labels "positive" and "negative" for sentiment classification. Moreover, our setup can easily extend to the case where want evaluate the truth of set statements instead of answering of questions.
2

---

Importantly, we want methods that do not rely on the model generating correct outputs and that do not rely on external supervision. Instead, we turn to the model's 's unlabeled hidden representations. Specifically, let ((x) R denote some feature representation on natural langauge input z, such as the hidden states of a Transformer-based language model. Our goal is to answer the questions 91....9th In only given access to (()). In Section 2.2 we introduce method for this problem that attains high accuracy (Section 3), demonstrating that this task is tractable.
2.2 METHOD: CONTRAST- CONSISEENT SEARCH
To make progress on the goal described above, we exploit the fact that truth has special structure: it satisfies consistency properties that few other features in language model are likely satisfy. Our method, Contrast-Consister Search (CCS), leverages this idea by finding direction in activation space that is consistent across negations. As we illustrate in Figure 1, CCS works by (1) answering each question 4 as both "Yes" (ar7) and "No" (a, ). (2) computing the representations ((x)) and (xx,) ) of each answer, (3) mapping the answer representations to probabilities p," and p; of being true, then (4) optimizing that mapping so that the probabilities are both consistent and confident.
Concretely, the input to CCS is a set of Yes-No questions, Qi...., In. and access to pretrained model's 's representations, x()): the output of CCS is a lightweight probe on top of ((-) that can answer new questions. Here, (6()) is fixed but should contain useful information about the answers to 91...........................9 In. in the sense that if one did (hypothetically) have access to the ground-truth labels for Q1..... On one would be able to train small supervised probe on d(.) that attains high accuracy. Importantly, CCS does not modify the weights of the pretrained model and it does not use labels.
Constructing contrast pairs. An important property that truth satisfies is negation consistency: the answer to a clear-cut question cannot be both "Yes" " and "No" at the same time, as these are negations of each other. Probabilistically, for each question qi, the probability that the answer qi is " "Yes" should be one minus the probability that the answer to qi is "No". To use this property, we begin by constructing contrast pairs: for each question qi we answer qi both as "Yes", resulting in the new natural language statement 2," and as "No", resulting in the natural language statement 2;- We illustrate this in Figure (left). We will then learn to classify x and xद as true or false; if x;" is true, then the answer to q should be "Yes", and if x; is true, then the answer to q: should be "No",
In practice, we convert each task into a question-answwe task-specific zero-shot prompts to format questions and answers as strings to construct each contrast pair. The opposite labels we use construct contrast pairs can be "Yes" and "No" for a generic task, or they can be other tasks-specific labels, such as "Positive" and "Negative" in the case of sentiment classification. We describe the exact prompts we use to for each task in Appendix B.
task with two possible labels, then we use
Feature extraction and normalization. Given a contrast pair (a .2,). ). CCS first computes the representations Φ(x;²) ) and (xx;) ) using the feature extractor 6(-). Intuitively, there are two salient differences between (xx²)) and o(x;) ): (1) x, ends with "Yes" while x; ends with "No", and (2) one of æ," or x; is true while the other is false. We want to find (2) rather than (1), so we first try to remove the effect of (1) by normalizing {0(x))} and { independently. In particular, we construct normalized representations d(x) as follows:
)(x == ((xt) - ut; क(a7) = )(x7))=((;;) - -4,,
where w+, -p Rd are the means of ((<x))); and {Φ( ) This normalization ensures that (ō(x;)) } and {Ø(x,) }} no longer form two separate clusters. In practice we also normalize the scale of the features, but this isn't 't essential for the method to work; see Appendix G.1 for details.
Mapping activations to probabilities. Next, we learn a probe (e,((0) that maps a (normalized) hidden state 5(x) to number between 0 and 1 representing the probability that the statement is true. We use linear projection followed by sigmoid (-)), i.e. po))(0) = (0 T + b), but nonlinear projections can also work. For simplicity, we sometimes omit the 0, b subscript in p.
Training objective. To find features that represent the truth, leverage the consistency structure of truth. First, we use the fact that a statement and its negation should have probabilities that add up to 1. This motivates the consistency loss:
Tucomiistency (0, b; g:) = [po,o(at) (1 po. ))
3

---

However, this objective alone has degenerate solution: p(x*) problem, we encourage the model to also be confident with the following confidence loss:
p(x 0.5. To avoid this
Tevnnideence (0, b; q:) min (x ), po,
We can equivalently interpret Leonfidence as imposing second consistency property on the probabilities: the law of excluded middle (every statement must be either true or false). The final unsupervised loss is the sum of these two losses, averaged across all contrast pairs:
(ocs(0,b)
Leonsistency (0, (0,b; (;): Leanfidence (0, b; g:)
Note that both losses are necessary: Lontideace alone also has degenerate solution.
Inference. Both p(x;)) and 1 p(=;) ) should represent the probability that the answer qi is "Yes". However, because we use soft consistency constraint, these may not be exactly equal. To make a prediction on an example n; after training. we consequently take the average of these:
p(a:) = 1 ((((x)) + (1-p(z77))
(1-((7)))))p(77)) p(xT))
We then predict that the answer to qi is "Yes" based on whether p(g:) is greater than 0.5. Technically, also need to determine whether p((:)) 0.5 corresponds "Yes" or "No," as this isn't specified by Locs. For simplicity in our evaluations we take the maximum accuracy over the two possible ways of labeling the predictions of given test set. However, in Appendix A we describe how one can identify the two clusters without any supervision in principle by leveraging conjunctions.
RESULTS
3.1 EXPERIMENTAL SETUP
Here we give an overview of our experimental setup: see Appendix G for full details. We provide code at https:/ /www. github. com/ http:/oooonnnoocceee disveeiiiooooiiieooooeooe latent seatteet_kooleedeeee
Models. We test six models: encoder-decoder models (T5 (Raffel et al., 2020), UnifiedQA (Khashabi et al., 2020), TO (Sanh al., 202 autoregressive models (GPT-J (Wang & Komatsuzaki, 2021)),
and encoder-only models (RoBERTa (Liu et al., 2019), DeBERTa (He et al., 2021)).
sentiment classification (IMDB (Maas et al., 201 1) and Amazon (McAuley Leskovec, 2013)), topic classification (AG-News (Zhang et al., 2015) and DBpedia- 14 (Lehmann et al., 2015)), NLI (RTE (Wang al., 2018) and QNLI (Rajpurkar al., 2016)), story completion (COPA (Roemmele et al., 201 1) and Story-Cloze (Mostafazadeh et al., 2017)), question answering (BoolQ (Clark et al., 2019)), and common sense reasoning (PIQA (Bisk et al., 2020)).
Data. We test models on 10 datasets:
We convert each dataset to a yes-no guestion-answeringg task or a binary classification task, as described in Appendix G. We balance the labels and randomly subsample 1000 examples from each dataset (except for COPA, which has only 500 examples total), then randomly split each dataset into an unsupervised training set (60% of the data) and test set (40%). We subsample each dataset for computational efficiency reasons; because we aggregate over 9 prompts per dataset, 10 datasets, and models, 1000 datapoints per dataset actually corresponds to approximately 180k examples total.
Methods. We test four main methods: zero-shot, calibrated zero-shot, Contrast-Consisterr Search (CCS), and Logistic Regression (LR). Zero-shot works by predicting the answer with the highest log probability according to the language model, averaged across the tokens that make up that label. Calibrated zero-shot works by balancing zero-shot predictions to be 50/50 for each answer, as we describe in more detail below, similar to Zhao et al. (2021). For Logistic Regression train on the training split for each dataset using (ō(x+), ((x=)) as the covariates, then evaluate on the corresponding test split. We treat LR as a cciling since it uses labeled data.
When testing CCS, we optimize it 10 times using AdamW W (Loshchilov & Hutter, 2017) with learning rate 0.01, then take the run with the lowest unsupervised loss. Unless otherwise specified, we train CCS using all prompts for single training set, then evaluate it on the corresponding test split.
Zero-shot baselines. Zero-shot outputs sometimes suffer from miscalibration (Zhao et al., 2021), in which models are biased towards predicting specific answers. Calibrating the outputs to be uniform
4

---

Method
RoBERTa
DeBERTa
GPT-J
T5
UQA
TO"
Mean
O-shot
60.1(5.7)
68.6(8.2)
53.2(5.2)
55.4(5.7)
76.8(9.6)
87.9(4.8)
62.8(6.9)
Calibrated O--hot
64.3(6.2)
76.3(6.0)
56.0(5.2)
58.8(6.1)
80.4(7.1)
90.5(2.7)
67.2(6.1)
CCS
62.1(4.1)
78.5(3.8)
61.7(2.5)
71.5(3.0)
82.1(2.7)
77.6(3.3)
71.2(3.2)
CCS (All Data)
60.1(3.7)
77.1(4.1)
62.1(2.3)
72.7(6.0)
84.8(2.6)
84.8(3.7)
71.5(3.7)
LR (Ceiling)
79.8(2.5)
86.1(2.2)
78.0(.3)
84.6(3.1)
89.8(1.9)
90.7(2.1)
83.7(2.4)
Table 1: Accuracy of each method and model averaged across all prompts and dataset, with the average standard deviation of accuracy across different prompts shown in parentheses. For most models, CCS outperforms zero-shot accuracy and exhibits lower sensitivity prompts, even though this was not our goal. This shows that we can recover knowledge from language model activations without supervision, and can do so in a way that is competitive with strong baseline methods that use model outputs, "TO was trained on 9 out of 10 of the datasets we evaluate on, including some of the data in our test splits, so we ignore it when averaging over models.
over different answers can mitigate this problem. We use a variant of the calibration method presented in Zhao al. (2021) by balancing predictions be 50/50 across the two output labels. Specifically, if 4 and 1_ are the logits for the positive and negative label respectively, then instead of classifying an example as positive 4 > L_, we classify it as positive 4 > L- + 7 where we select the threshold y E IR so that the predictions are balanced. We find this increases accuracy by about 5% on average. Unless otherwise specified, we always report zero-shot accuracy after calibration.
Encoder-only models (e.g. RoBERTa and DeBERTa) cannot be easily used to do zero-shot classifica- tion out of the box, so to evaluate them we follow the method of Yin al. (2020): we finetune both models NLI dataset (MNLI, which we do not evaluate on) and treat the difference between the entailment and contradiction probabilities as the effective logit. This provides a strong zero-shot baseline for encoder-only models that works even for non-NLI tasks (Yin et al., 2020). This finetuning isn't 't't necessary for CCS to work on encoder-only models (see Appendix C), but we test CCS using the same MNLI-finetuned models for ease of comparison.
Hidden states. We extract the hidden states corresponding to the last token in the last layer of each model for simplicity, unless otherwise specified. For encoder-decoder models, we evaluate CCS on the last layer hidden states of both the encoder and decoder, and use whichever one generally achieves a lower unsupervised loss; for TO this the decoder hidden states, while for T5 and UnifiedQA this is the encoder hidden states. See Appendix G.2 for further implementation details, such as tokenization.
To reduce prompt sensitivity, we use between and 13 prompts for each dataset (9 on average), derived or slightly modified from Sanh et al. (2021). Unless otherwise specified, we average across all prompts when showing results. To construct contrast pairs, we let z be the zero-shot prompt using qi and the first label (e.g. "Positive" for sentiment classification datasets) and let æ, be the prompt using qi and the second label (e.g. "Negative"). We describe all prompts in Appendix I.
Prompts.
3.2 EVALUATING CCS
3.2.1 CCS OUTPERFORMS ZERO-SHOT
We evaluate CCS on all 6 models and compute the average accuracy across all datasets and prompts. TO was trained on 9 out of 10 of the datasets we evaluate on, including some of the data in our test splits, so we ignore it when averaging over models avoid unfair comparisons. We display the results in Table 1. To assess prompt sensitivity, for each model and dataset we compute the standard deviation (s.d.) of accuracy across different prompts, then average the resulting standard deviations across all datasets, which we show in parentheses in Table 1. For comparison, we also include results when training CCS on all datasets simultaneously, which we refer to as CCS (All Data).
CCS attains an accuracy of 71.2% on average, compared to 67. 67.2% for calibrated zero-shot. It outperforms zero-shot accuracy for every model, except for RoBERTa (where it does 2% worse) and TO (for which zero-shot accuracy is inflated). Training all datasets improves accuracy by only an insignificant amount on average (0.3%), but with large gains for TO in particular (77.6% + 84.8%).
These results show that CCS can exceed the performance of strong baseline methods that access the model outputs, even though this wasn't our main goal. This indicates that it indeed possible to classify examples with high accuracy using only unlabeled model representations.
5

---

3.2.2 CCS IS ROBUST TO MISLEADING PROMPTS
Recall our goal: to discover latent knowledge in language model even when the model outputs false text. In particular, language models are typically trained to imitate text whether or not it is correct, so if a model sees false text it should intuitively more likely to predict that subsequent text will also be false. Based this idea, we provide an initial proof of concept that CCS can make progress toward our goal by constructing prompls that aim to mislead the outputs of language models. Specifically, we add prefix to the beginning of our zero-shot prompts that consists of questions answered incorrectly (Figure 5). The hope is that such a prefix will decrease zero-shot accuracy because the model will imitate its context and answer subsequent questions incorrectly even if it internally "knows" better." We found that while most models are robust to this type of prefix (see Appendix B), it significantly drops calibrated zero-shot performance in UnifiedQA, decreasing accuracy from 80.4% to 70.9%.
We evaluate CCS on these examples and show the results in Figure 4 of the Appendix. We find that despite the 9.5% drop in zero-shot accuracy, CCS maintains high accuracy (82.1% + 83.8%). This provides evidence that our method can still work well even when model outputs are unreliable.
3.3 ANALYZING CCS
We have shown that CCS attains strong classification performance in standard settings and when we deliberately mislead models. Moreover, we have described our motivation as discovering latent representations of truth in language models, but in practice CCS just finds a direction representation space that attains high accuracy. This raises the question: in what sense is CCS actually finding "truth" features? We now provide a preliminary investigation of this question.
3.3.1
CCS FINDS A TASK-AGNOSTIC REPRESENTATION OF TRUTH
From the results described far, it may be possible that the classifier we find is capturing dataset- specific properties, such as which of two labels (e.g. "Yes" vs. "No") is more likely to be correct. We rule this out by showing that it generalizes across completely different tasks, including ones with different label spaces (such as from "Yes" and "No" for a generic task to "Positive" and "Negative" for sentiment classification). In particular, we train and test CCS on every pair of datasets, and show the resulting transfer for several models in Figure 2. (See Appendix for results with other models.)
We find that CCS indeed transfer wells: in the majority of datasets, the transfer accuracy of CCS is competitive with training and testing on the same dataset (the last row in Figure 2, "No transfer"). Transfer performance can even outperform the no-transfer setting, especially when we train on simpler tasks, such as sentiment classification. For example, training CCS on the Amazon sentiment dataset achieves an average transfer accuracy of 71.8%, which 0.6% higher than CCS without transfer. We speculate that this performs so well because the difference in representations between correct and incorrect answers is especially pronounced for easier tasks like Amazon.
Additionally, transfer accuracy tends to be similar for many datasets. For example. training on IMDB (row of Figure 2) has similar accuracy as training on DBPedia (row 4). This provides evidence that CCS can find functionally similar direction across many different types of training datasets. Overall, these results suggest that (1) models may have a task-agnostic representation related to what is true, and that (2) CCS may approximately find this representation even without diverse data.
3.3.2 CCS DOES NOT JUST RECOVER MODEL OUTPUTS
One possibility is that CCS can only recover knowledge already contained in a model's outputs. We have shown that CCS can outperform zero-shot accuracy (Section 3.2.1), especially when model outputs are misled (Section 3.2.2), which already provides evidence against this possibility. We now provide additional evidence against this concern.
First, if CCS were just recovering knowledge in the model outputs, using the last layer of a network (right before the outputs) should presumably outperform intermediate layers (which are more causally distant from the outputs). However, for T5 and UnifiedQA, we find that using hidden states in the
In practice we found that model behavior was qualitatively similar between using this prefix with correct and incorrect answers, in agreement with the findings of Min et al. (2022b) that using incorrect labels in demonstrations often does not significantly degrade the performance of a few-shot prompt. Consequently, the prefix may instead actually be reducing accuracy because it is oft-of-disttibutiin. See also Kim et al. (2022) for more experiments the subtleties of correct vs incorrect labels for few-shot prompts.

---

UQA
T6
DeBERTH
92
99
74
88
64
82
99
72
78
59
59
78
92
94
80
54
62
63
55
66
73
61
91
92
75
60
88
62
72
9-5
92
99
76
74
BB
12
61
75
91
95
91
99
63
64
74
58
61
71
84
87
88
97
60
72
68
59
54
70
76
75
7
76
59
55
54
61
59
69
66
68
56
82
65
62
57
66
50
56
71
68
68
56
53
59
89
85
94
35
55
58
56
60
77
54
62
68
65
70
54
57
54
62
85
85
91
99
70
60
63
54
58
71
62
62
66
72
54
64
58
52
52
58
B6
96
92
99
B8
64
89
56
54
89
95
37
91
99
85
53
58
52
58
76
92
94
88
97
61
82
68
86
BOOLD
BO0LO 0R0
40
Mary
Figure 2: Transfer Performance using CCS on UnifiedQA, TO and DeBERTa. The y-axis corresponds to the training dataset, and the x-axis corresponds to the test dataset. The final row ("No Transfer") corresponds to training and testing on the same dataset, which is the same as the diagonal. On most datasets, CCS transfers well to other datasets (relative to transfer), including to different tasks with completely different labels. In some cases transfer even outperforms the no-transfer setting, See Appendix E for results with other models.
middle of the network outperform hidden states at the end of the network when using CCS (see Figure 10). This is especially true for UnifiedQA on misleading prefixes; find that using the encoder hidden states is robust to mislcading prefixes (Section 3.2.2), but that accuracy using the decoder hidden states drops from 81.0% to 73.5%, similar amount to zero-shot accuracy. This suggests that compared to the later layers of model, intermediate layers are more robust and less correlated with the model outputs, and that CCS can take advantage of this.
Finally, if CCS were just recovering knowledge in the model outputs, we would only expect it to work in cases where model outputs are informative. However, we show in Appendix C that CCS still works with masked language models when their outputs are uninformative: when we don't 't [MASK] any input tokens, and when we prompt models so that the labels used to construct contrast pairs appear in the middle of prompt rather than at the end. These results show that CCS can sometimes recover latent knowledge in a model that is distinct from-and more useful than-wha the model outputs.
3.3.3 TRUTH IS A SALIENT FEATURE
From the results we have presented so far, it possible that the direction learned by CCS is difficult to find and requires using large amount of unsupervised data. We provide evidence against this possibility by showing that finding such a direction can both (1) often be done with only a small amount of data, and can also (2) often be done by essentially taking the top principal component of slightly modified representation space.
CCS doesn't 't require much data. We now evaluate how well CCS performs with different amounts of data. Whereas before we trained CCS using the full training set and all prompts, here we use limited data and a single prompt. Specifically, we train using only unlabeled contrast pairs, using the single prompt for each model and dataset that achieves the highest zero-shot accuracy. We still test on all prompts for each dataset. We resample points 32 times for each of 1,2, 4,..., and take the average accuracy across those 32 samples. Finally, we plot the average such accuracy across all datasets and prompts for several models in Figure
We find that while CCS benefits from more data, it can often do well with very limited data. In fact, it can sometimes even do well with only a single contrast pair, though we find high variance across individual datasets; see Appendix D for more details. This suggests that the strong performance of CCS does not primarily come from using a large amount of unsupervised data, and indicates that the direction learned by CCS may be relatively easy to find.
Contrastive Representation Clustering, We now show that directions correlated with the truth may be "salient" in different way: by showing that we can also find such directions using either (1) PCA or (2) clustering. Specifically, suppose we construct contrast pairs (xt, x, ) as before. Intuitively, these two examples are qualitatively almost identical except that one is true and the other
7

---

80
TO
DeBERTa
Accuracy
76
74
16 #Samples
128 256
512
Figure 3: Accuracy when train CCS on & samples for different values of (each time averaged across 32 trials). We use the single prompt with the highest zero-shot accuracy for each dataset and model. While CCS benefits from more examples, it can often work well with limited data.
is false, so the main difference between the representations o(x²) ) and ó(x7) should relate to truth. Consequently,
take the differences in (normalized) hidden states, {6(x²) ) Ō(x; )) and cluster them. We call this method Contrastive Representation Clustering (CRC). Clustering can be achieved by taking the top principal component (TPC) and thresholding at 0, by doing "bimodal salience search" (BSS) to find direction that looks bimodal; see Appendix G.4 for further details.
We compare CCS and these variants of Contrastive Representation Clustering in Table 2, using the same setting as in Table 1. While CCS performs best, all methods attain high accuracy and competitive with zero-shot performance. This indicates both that (1) representations of truth often lie in high-variance direction in the contrastive representation space (Ø(x;) Ø(v, ) and also that (2) true and false examples are often well-clustered in this same contrastive space. This strengthens the idea that representations of truth may be salient features inside models that are relatively easy to find. This may help explain why CCS can perform well without using any supervision, and how can do so even with only limited amount of unlabeled data.
4 RELATED WORK
Zero-Shot Prompting. Since the release of GPT-3 (Brown al., 2020), one of the main paradigms for eliciting what models know has been zero-shot prompting (Liu et al., 2022; Beltagy et al., 2022). Zero-shot exploits how language models are trained to predict diverse data from the internet, which incidentally includes tasks such question-answering, If prompted appropriately, this can be used to solve various useful tasks with reasonable performance (Brown et al., 2020). However, these models trained to imitate human- -eneratee data, which bounds the quality of their outputs.
Many methods improve upon vanilla zero-shot prompting (Liu et al., 2022; Zhao et al., 2021; I; Lu et al., 2022; Wei et al., 2022b; Min et al., 2022a). While goal is not to improve zero-shot performance, some of the ideas underlying these methods are similar to CCS. Particularly relevant are methods that also leverage unsupervised consistency properties, such as Jung et al. (2022); Zhou et al. (2022). However, these methods still bootstrap from language model outputs trained via imitation learning, which limits their applicability to our main goals.
Method
RoBERTa
DeBERTa
GPT-J
T5
UQA
TO*
Mean
Calibrated 0-shot
64.3(6.2)
76.3(6.0)
56.0(5.2)
58.8(6.1)
80.4(7.1)
90.5(2.7)
67.2(6.1)
CCS
14.1(
78.5(3.8)
61.7(2.5)
71.5(3.0)
82.1(2.7)
77.6(3.3)
71.2(3.2)
CRC (TPC)
64.6(5.8)
76.5(5.7)
59.9(3.7)
66.7(5.0)
78.3(3.3)
58.9(13.0)
69.2(4.7)
CRC (BSS)
62.6(6.8)
76.8(5.3)
60.7(3.4)
69.7(3.3)
79.3(2.5)
76.7(9.6)
69.8(4.3)
Table 2: We compare CCS to two variants of Contrastive Representation Clustering: TPC, which clusters by projecting onto the top principal component, and BSS, which clusters by finding direction that looks bimodal. We show accuracy and standard deviation of each model averaged across all prompts and datasets, in the same setting Table 1. We find that CCS generally performs the best, but that all methods are competitive with zero-shot.
8

---

To illustrate this, imagine we train reinforcement learning agents to play game such as Diplomacy (FAIR et al., 2022). in which players have incentives to lie to each other. Then those agents may learn to lie in a way that is difficult to detect, but they may still internally represent whether they are lying. Because their outputs would be deliberately misleading, standard zero-shot methods may be very unreliable. In contrast, techniques like CCS may still be able to detect whether those models lying by finding representations of truth in their activations that contradict their outputs.
Truthfulness. There has been increasing interest in making language models truthful (Evans et al., 2021; Lin et al., 2022). One aspect of truthfulness that has received substantial attention is factuality (Thorne al., 2018: Maynez al., 2020). For instance, many techniques aim to improve the factuality of models by augmenting them with retrieval methods (Nakano et al., 2021; Menick et al., 2022), which allows them to cite their sources. contrast, we focus on truthfulness more generally, which also includes procedural knowledge such as reasoning or natural language inference tasks.
An approach making language models truthful in this more general setting is to finetune them using either human demonstrations (Khashabi et al., 2020; Sanh et al., 2021; Zhong et al., 2021; Wei et al., 2022a) or reinforcement learning from human feedback (Christiano et al., 2017; Stiennon et al., 2020; Askell et al., 2021; Bai et al., 2022: Ouyang et al., 2022). These techniques have been widely successful at improving performance, but unlike our method they rely on being able to provide ground truth labels, which can be intractable in many settings.
Some work has aimed to go beyond the direct supervision humans can provide by augmenting supervision with AI systems (Christiano et al., 2018; Irving et al., 2018; Leike et al., 2018; Perez et al., 2022). This may expand the range of applications we can supervise, but many of these proposals remain theoretical, and it is unclear just how far these techniques can generalize. Christiano al. (2022) reframes this issue by posing the problem of Eliciting Latent Knowledge (ELK). Like our problem statement, ELK is about eliciting knowledge from models even in cases where humans cannot evaluate that knowledge. However, ELK frames this as worst-case theoretical problem, while we frame this as an empirical problem that we can make progress on using current models.
5 DISCUSSION
5.1 LIMITATIONS AND FUTURE WORK
Our work has number of limitations. First, CCS relies on the existence of a direction in activation space that separates true and false inputs well, in the sense that supervised probe the activations would be able to attain high accuracy (if it hypothetically had access to the ground truth labels). This requires that model is both capable of evaluating the truth of given input, and also that the model actively evaluates the truth of that input. is not clear when these conditions hold precisely.
Second, did not evaluate our method on setups involving active "lying" "deception" (Kenton et al., 2021; Evans et al., 2021) by models, as we aren't aware of existing evaluation setups for this setting. If future work develops such setup, good stress test would be apply CCS to do "lie detection" in that setting. This may require modifications or extensions to the method, such as more explicitly ensuring that it recovers the truth of an input rather than what the model says.
There are also various straightforward improvements to our method that one could explore. This in- cludes adding additional consistency constraints, improving its reliability, calibrating its probabilities, generalizing it beyond the yes-no question-an grestion-answering answering setting, generalizing it to cases where answers aren't 't clear-cut, and closing the remaining gap between CCS and the logistic regression ceiling.
5.2 CONCLUSION
As language models become more capable, they will be increasingly used as components in larger AI systems trained with reinforcement learning. As this occurs, falsehoods arising from misaligned training objectives may become more common, severe, and difficult to detect. principle, models may even develop instrumental incentives to lie: for example, if human evaluator would disapprove of bad model behavior, models may learn to lie about their behavior to achieve higher reward. so, those models would be optimizing against human evaluators, making it precarious to rely on those evaluators for assessing the truth of what models say. Because unsupervised methods for eliciting answers circumvent this issue, they may still work even in this scenario. We show that it is possible to make progress on such methods today; the empirical success of our approach suggests that unsupervised methods are both tractable and underexplored research direction.

---

ACKNOWLEDGEMENTS
We are very grateful to Jared Kaplan for helpful experiment suggestions and resources early on in the project. We thank Beth Barnes and Paul Christiano for valuable discussions regarding the longer-term impacts of this work. We are also grateful to Jessy Lin, Alex Pan, Ruiqi Zhong, Yaodong Yu, the anonymous reviewers, and several others for useful feedback on earlier versions of this paper. CB is supported by Open Philanthropy AI Fellowship.
REFERENCES
Amanda Askell, Yushi Bai, Anna Chen, Dawn Drain, Deep Ganguli, T. J. Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, 7ac Hatfield-Dodds, Danny Hernandez, John Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, and Jared Kaplan. A general language assistant as laboratory for alignment. ArXiv, abs/21 12.00661, 2021.
Yushi Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, abs/2204.05862, 2022.
Iz Beltagy, Arman Cohan, Robert Logan IV, Sewon Min, and Sameer Singh. Zero- and few-shot nip with pretrained language models. In ACL, 2022.
Emily M. Bender, Timnit Gebru, Angelina McMillan- Majoor, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7432-7439) 2020.
Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In NIPS, 2016.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, ArXiv, abs/2108.07255: 2021.
10

---

bert Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020.
Paul Christiano, Ajeya Cotra, and Mark Xu. Eliciting latent knowledge. Technical re- port, ARC, 2022. https://::///oo http://ccccoooool...../..oooo Yh- wttp:/uu.H.L..aaacccoooeexeee1
Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei.
Deep reinforcement learning from human preferences. ArXiv, abs/1706.03741, 2017.
Paul Francis Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplify- ing weak experts. ArXiv, atss/8110008555 2018.
Christopher Clark, Kenton Lee, Ming-Woi Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:19055 10044, 2019.
Owain Evans, Owen Cotton-Barrati Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. Truthful ai: Developing and governing ai that does not lic. ArXiv, abs/21 10.06644, 2021.
Meta FAIR, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, pp. eade9097, 2022.
Rory A. Fisher. The use of multiple measurements in taxonomic problems. Annals of Human Genetics, 7:179-188, 1936.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: with disentangled attention. ArXiv, abs/2006.03654, 2021.
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. ArXiv, abs/2109.139116 2021.
Geoffrey Irving, Paul Francis Christiano, and Dario Amodei. Ai safety via debate. hoss/805500899, 2018
ArXiv,
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations. ArXiv, abs/2205.11822 2022.
Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. Alignment language agents. ArXiv, abs/2103. 14659, 2021.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. Unifiedqa: Crossing format boundaries with a single system. In FINDINGS, 2020.
Junyeob Kim. Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang- Woo Lee, Sang goo Lee, Kang Min Yoo, and Taeuk Kim. Ground-truth labels matter: A deeper look into input-label demonstrations. ArXiv, abs/2205.1 12662 2022.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.66 144266980, 2014.
11

---

Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. Dbpedia-a large- scale, multilingual knowledge base extracted from wikipedia. Semantic web, ((2):167-155, 2015.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: research direction. ArXiv, 00818811.07777 2018.
Stephanie C. Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In ACL, 2022.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys (CSUR), 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1197. 16692. 2019.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. ArXiv, 171.05001, 2017.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACL, 2022.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 201
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. On faithfulness and factuality in abstractive summarization. ArXiv, 0bs/2005.00661 2020.
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimen- sions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pp. 165-172, 2013.
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Hhipbell-Gillingham,, Geoffrey Irving, and Nathan McAleese. Teaching language models to support answers with verified quotes. ArXiv, abs/2203.1 11147, 2022.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannanch Hajishirzi. Metaicl: Learning to learn in context. ArXiv, abs/21 1tts/2110.1155 159449,3, 2022a.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? ArXiv, abs/2202.128377, 2022b.
Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pp. 46-51, 2017.
Reiichiro Nakano, Jacob Hilton, S. Arun Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted aanssiinn with human feedback. ArXiv, abs/21 1tts//112.0932.2 2021.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. ArXiv, has/2203.02155. 2022.
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nathan McAleese, and Geoffrey Irving. Red teaming language models with language models. ArXiv, abs/2202.03286, 2022.
12

---

Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. ArXiv, abs/1910.10683, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05256 2016.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 201/ AAAI Spring Symposium Series, 2011 1.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric Michael Smith, Y.-Lan Boureau, and Jason Weston. Recipes for building an open-domain chatbot. In EACI., 2021.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang A. Sutawika, Zaid Alyafeai, Antoine Chaffin, Armaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey. M SAIFUL BARI, Canwen Xu, Urmish Thakker, Shanya Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Stella Rose Biderman, Leo Gao, T. Owe Bers, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. ArXiv, 1is/2110.08207, 2021.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. ArXiv, Abs/2000.01325 2020.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. ArXiv, abs/1 803.05355, 2018.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint 1804.0746, 2018
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https: / gitbb... com/kingoflols. htt////sssttassssoeeeeeee formerrer---rrrrreeeoo jax, May 2021.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. ArXiv, abs/2109.01652, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning large language models. ArXiv, abs/2201. 11903, 2022b.
Laura Weidinger, John F. J. Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zachary Kenton, Sande Minnich Brown, William T. Hawkins, Tom Stepleton, Courtney Biles. Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William S. Isaac, Sean Legassick, Geoffrey Irving, and lason Gabriel. Ethical and social risks of harm from language models. ArXiv, abs/21 12.04359, 2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's 's transformers: State-of-the-art natural language processing. ArXiv, abs/1990.00771 2019.
Wenpeng Yin, Nazneen Rajani, Dragomir Radev, Richard Socher, and Caiming Xiong. Universal natural language processing with limited annotations: Try few-shot textual entailment as a start. In EMNLP, 2020.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28:649-657, 2015.
Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. ArXiv, abs/2102.09660,, 2021.
13

---

Ruiqi Zhong, Kristy Lee, Zheng Zhang. and Dan Klein. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. In EMNLP, 2021.
Chunting Zhou, Junxian He, Xuezhe Ma, Taylor Berg-Kirkpatrick and Graham Neubig. Prompt consistency for zero-shot task generalization. ArXiv, 02s/2205.000433 2022.
14

---

UQA (All Datasets and Prompts)
100
CCS
0-shot
90
y=x
Prefixed Accuracy
80
70
60
50
50
60
70
80
Regular Accuracy
Figure 4: Zero-shot accuracy drop caused by the misleading prefix shown in Figure 5 for UnifiedQA. Each point corresponds one dataset and one prompt. On average, zero-shot accuracy drops 9.5%. while CCS performs 1.7% better. This suggests that the latent knowledge inside a model that CCS recovers can be robust even when model outputs become less reliable.
A IDENTIFYING CLUSTERS
CCS predicts yes-no answers to a set of unlabeled questions .1...., In- However, does not identify which prediction label corresponds true and which corresponds false. We now describe how one can do so in principle, as long as a model is also able to take conjunctions.
Suppose we
method and find, for example, that and x' end up in different clusters. Then know that these two statements have opposite truth values. As result, if we take the conjunction of these two statements, x x' then should be false, and if we take the disjunction, it should be true. This allows us to identify which cluster is true and which is false in completely unsupervised way.
x', then
B MISLEADING PREFIX DETAILS
B. MISLEADING PREFIX
In Section 3.2, mentioned that model outputs can be biased by using the misleading few-shot prefix shown in Figure We show the effect the misleading prefix for all models Table 3, and illustrate the drop in accuracy for each dataset and prompt in Figure 4.
B.1.1 EVALUATING THE EFFECT OF THE MISLEADING PREFIX
The hope is that the misleading prefix causes models to imitate the sorts of incorrect answers in the prefix. However, we found that results are qualitatively similar with correct answers, making the actual interpretation of the effect of this prefix ambiguous. Nevertheless, to better understand what this prefix is doing, we visualize the top 100 tokens for both this misleading prefix and no prefix on average across all examples for UnifiedQA on the IMDB sentiment dataset, and show the results in Figure 6. Without this prefix, the top two tokens are the actual labels ('positive' and 'negative') In contrast, with the prefix, these tokens have lower probabilities relative to other tokens, with the highest probability tokens instead being irrelevant to the true answer. For example, since the prefix includes several answers with numbers, many of the top tokens also numbers. This provides some evidence that the prefix is actually causing the model to output false text by imitating its context in meaningful sense.
15

---

What the human life expectancy in the United States? t0 years.
Who was president the United States in 1955? Abraham Lincoln.
Which party did he belong to? The Social Democratic Party Germany.
What squere
42
work??
Fye beams are emitted by the eye and neftect back into the eye.
Where ware the 1992 Olympics held? 1992 Olympics were held on the moen.
How many squigs are bonk?
Figure The added prefix used mislead models in a way that egregiously incorrect or confusing.
in
Positivelluss
GOALELN
they
elliee
415 Where
309
Isan 55: 50
nega
VV with
e
regetively
71
20 hunans 10060
1n
Human
CO
there
5
they
what
3
it
that bad
yes
yes &
posl
uichac
Ve there
postasetig seat
2 6
422
21 33n5
positive onede zero
to
thehowhth whuaan
that 2
this Neg not the
40
22 nega What
Seaetiiigg
man
Normal
Confusing Prefix
Figure 6: The top 100 tokens with highest probability averaged across examples in IMDB for UQA. Without the prefix (left), the actual labels ("positive" and "negative") and their synonyms (e.g. "Positive" and "Neg") have high probability. With the prefix, the model's output becomes mostly irrelevant sentiment; the actual labels the top 100. but no other synonyms are.
Method
Prefix
RoBERTa
DeBERTa
GPT-J
T5
UQA
TO"
Mean
Calibrated
Regular
64.3(6.2)
76.3(6.0)
56.0(5.2)
58.8(6.1)
80.4(7.1)
90.5(2.7)
67.2(6.1)
0-shot
Prefix
65.6(5.3)
75.5(6.1)
59.2(4.6)
56.0(4.0)
70.9(7.8)
88.2(4.2)
65.4(5.6)
Regular
(4.1(
78.5(3.8)
61.7(2.5)
71.5(3.))
82.1(2.7)
77.6(3.3)
71.2(3.)
CCS
Prefix
62.2(3.6)
75.4(5.2)
61.2(1.8)
73.2(2.6)
83.8(2.4)
75.0(2.7)
71.2(3.1)
Regular
64.6(5.8)
76.5(5.7)
59.9(3.7)
66.7(5.0)
78.3(3.3)
58.9(13.0)
69.2(4.7)
TPC
Prefix
65.0(5.6)
77.0(6.0)
60.1(2.9)
68.1(5.0)
76.0(3.6)
56.5(13.6)
69.2(4.6)
Regular
62.6(6.8)
76.8(5.3)
60.7(344
69.7(3.3)
79.3(2.5)
76.7(9.6)
69.3(4.3
BSS
Prefix
59.5(5.4)
75.5(6.6)
59.8(2.8)
65.2(2.8)
83.1(2.3)
72.2(9.0)
68.6(4.0)
Regular 79.8(2.5)
86.1(2.2)
78.0(2.3)
84.6(3.1)
89.8(1.9)
90.7(2.1)
83.7(2.4)
LR
Prefix
79.4(2.7)
86.3(2.6)
79.0(2.9)
86.3(2.8)
90.1(2.2)
90.5(2.3)
84.2(2.))
Table 3: Accuracy of each method and model averaged across all prompts and datasets, with the same setting as Table "Regular" means no prefix is added, while "Prefix" corresponds to text in Figure 5. Notice that while the 0-shot accuracy decreases due to the prefix, all of our methods are more resistant the influence.
16

---

C MASKED LANGUAGE MODELING RESULTS
We now provide an initial demonstration that CCS can work well even when a model's outputs are not very useful.
First, we evaluate our method on a model trained exclusively with the masked language modeling (MLM) objective: DeBERTa-v2 (He et al., 2021) without NLI finetuming (unlike in Section 3. where we finetuned DeBERTa on an NLI task to use it in the zero-shot setting). The outputs of DeBERTa unlikely to be meaningful if we provide raw input without using any [MASK] tokens If CCS only works when the model outputs are useful, then it should perform poorly in this setting.
We also consider one other way that model outputs be uninformative. Given a sequence of tokens as input, DeBERTa has a different set of output logits for each input token. For a given contrast example question and candidate answer), the most informative output should intuitively be the logits that predict the answer. For instance, if the contrast example xt is "Is 2+2-4? Yes", then the most informative output should be the logits corresponding to candidate answer "Yes". Consequently, if we instead format each example so that the label (e.g. "Yes" or "No") appears in the middle of the prompt, then the output corresponding to the final token should not be very informative. If CCS only works well when the model outputs are useful, then it should also perform poorly in this setting (as long as CCS uses the last-token hidden states as usual).
To show that CCS can work even when model outputs aren't informative, based on the above discussion we now do an initial test of CCS when we simultaneously (1) use DeBERTa-v2 (MLM- pretrained only), and (2) format inputs that the label is in the middle of the prompt rather than at the end. In particular, we apply CCS on the Amazon dataset, where we randomly sample 1000 new (unlabeled) points and last-token hidden states for CCS. As usual, we use the default Huggingface tokenizer and we do not [MASK] any input tokens. We use the following custom prompt to test (2):
60/40 train-test split before. As usual continue to the
The Following movie review expresses a [label] sentiment An [text]
Even though unclear why the model outputs should be useful with this setup, we find that CCS can indeed still perform well, attaining an accuracy of 93.7%. For comparison, this is nearly identical to the approximately 94% accuracy of CCS when evaluated on the Amazon dataset using NLI-finetuned DeBERTa and the original prompts (see Figure 9).
We argued above that the model outputs should not be very useful in this setting. We now verify that this is the case. To do so, we evaluate modified version of zero-shot prompting adapted to work for masked language models when we use no [MASK] tokens in the input. Specifically, on an input x or x we (1) take the logit vector corresponding to the last input token, (2) select the corresponding logit for "positive" and the corresponding logit for "negative" (the two candidate labels), (3) take the difference between these, resulting in 1+ teeg for x= and 1po. !neg for x. respectively, then (4) take the difference between these, (1piox )itgg) (lpos Iegg) ). form the effective logit. Intuitively, this final expression should be large if the example is positive, and small if the example is negative. As before, we calibrate this zero-shot method so that predictions are balanced.
We evaluate this method and find that it recovers non-trivial knowledge in the model outputs, but that it still performs substantially worse than CCS: calibrated zero-shot accuracy in this setting is 71.6%. compared to almost 94% for CCS. This suggests that model outputs in this setting indeed aren't "ee't very useful, especially relative to CCS.
Overall, our results in this section show that CCS can still work in at least some settings where model outputs don't 't seem to be very useful. This provides additional evidence that out approach does not simply recover knowledge represented in the model outputs.
'The MLM objective first selects 15% of tokens to mask. Of the masked tokens, 80% are replaced with [MASKJ, 10% are randomly replaced with different token, and 10% are left unchanged (He et al., 2021). As result, for [MAK] tokens, the most likely output token should usually be the original input token itself, which is uninformative.
17

---

D COMPLETE SAMPLE COMPLEXITY RESULTS
In this section we show additional results on how the number of examples affects CCS. We show the performance (averaged across all datasets) for all models in Figure 7. We next show a more fine-grained results. Specifically, we select #amplees 1, 8, 64 and show dataset-level results in Figure for each model.
Accuracy (%)
UQA
70
GPT:
RoBEATh
DaBERT
60
16
32
128
256
#Samples
Figure 7: This figure shares the same setting with Figure 3, but it contains all six models we consider.
18

---

U06 insurptes 1
USA esamakss
ugA #Sammiks 64
300
3Samplss +1
#sampess
aSargles
Dek Seta asaniles 11
b-tthth PSampls #
D-B0Ak 25arrles de
Figure 8: CCS performance when we only use 1, 8, 64 examples, for all models and datasets. X-axis represents the dataset and y-axis is the accuracy in percentage. For each dataset, select the prompt with highest 0-shot performance, and then randomly select 1, 8, 64 data points from this prompt. We then perform CCS and test all prompts of this dataset, where each value in the barplot corresponds to one prompt.
19

---

E
COMPLETE TRANSFER RESULTS
Complete transfer results for CCS, TPC and BSS in all models are shown in Figure
CCS
53
.47
86
50
54
Gay Dir
36
52
50
69
00p442
180
-
5
92
92
94
95
Sipece
230
S7
50 40
25
35
20 36
52
92 93
39
90
32
25
90
90 39
32
86
90
90
52 49
35
32
79
32
55
51
40
96
51
90
96
55
55
90 52 52
Amaac
Oppac
76
50
59
30 05
45
80
51
39
48
33
15
50
92
09
57 95
54
60
Figure 9: Transfer Accuracy for all models using CCS, BSS, TPC and LR.
20

---

F COMPLETE INTERMEDIATE REPRESENTATIONS RESULTS
In this section we the performance CCS and LR across all hidden layers all six models. Specifically, for each model, we generate the hidden states for each dataset every 2 layers (for both encoders and decoders). Then, for each set of hidden states, we perform CCS and LR on each dataset separately, and average the performance across all datasets. We show CCS results in Figure 10 and LR results in Figure 11.
methhod OCS
DeBERTh
80
T5
TO
60
encoder
Layer
Figure 10: CCS performance when using the hidden states across different layers, using all six models consider in the paper.
method
RoBERTa
90
GPT:J
UQA
Accuracy (8))
25
85
encoder
decoder
Figure 11: Linear regression performance when using the hidden across different layers, using all six models we consider in the paper. This is the ceiling of all possible methods, and is supervised.
21

---

G CCS AND CRC IMPLEMENTATION DETAILS
In this section, we provide further implementation details for CCS and CRC.
G.1 NORMALIZATION
As described in Section 2.2, normalize the features x(x{) 6(x²)( -ut, where j² is the mean of {o(=))}² (and similarly for o(x;)) )). In practice, we also normalize the scale of the features by also dividing by the average norm of {Ø(x² ))= times v (and similarly for ~(x; )). However, this is less essential than mean normalization and there are likely many reasonable choices for scale normalization.
G.2 CONTRAST PAIRS
We use the Huggingface library (Wolf et al., 2019) for all of our experiments. We use the standard tokenizer for each model, and always take the hidden state corresponding to the last token in given layer indexed by idx. We show Huggingface-style pseudocode for extracting hidden states from in Algorithm 1.
For encoder-only and decoder-only models, we provide the full input, x+ or x which includes both the question and the proposed answer, the model; see Appendix I.1 for more formatting and tokenization details. For encoder-decoder models, our input format depends on whether we are taking the encoder hidden states or the decoder hidden states. When we take the decoder hidden states, we input the question to the encoder, and input the candidate answer to the decoder. In contrast, when we take the encoder hidden states of an encoder-decoder model, we provide the full input (including the proposed answer) to the encoder, and ignore the decoder (simply passing the empty string to the decoder and ignoring its results). This is necessary to ensure that the inputs to the encoder are not identical across contrast pairs.
Algorithm 1 Pseudocode for Getting Contrast Features
Input: Contrast Pairs Set S. model name mdl, layer index idx TuunformmersAAutoTookiiii
m transormeer-AuuoMod(((mddd
C+ C = 0.
for (xt, in S do token token = t.encode(x*)) t.encode(x")
o+ m(token** output hiddenstates = True)] True))" "hidden "wthhhdddnsstttsssllllaa statts"][idx][[]
Φ m(token outputt hiideentddesssssess ("iiddn soaiddde_stttes""]l [[ixx[[[]]
C append(
[ppennd(o)) "apppe)dd
end for Output: [et,c-]
G.3 CCS
Given contrast features from Algorithm 1, CCS works by learning parameters and by minimizing Locs(0, b) defined in Section 2.2.
In practice, we implement the bias by appending an additional dimension of is features to the input, increasing it and 0 to have dimension
We randomly initialize 0 to have unit norm. We then optimize the CCS loss = 10 times and select the run with lowest unsupervised loss. In practice, we train each time for E 1000 epochs with learning rate 7 0.01 (which found good for consistently achieving low unsupervised loss) in each run. Because we are only learning a linear probe top of the features, training be fast.
22

---

G.4 CRC: TOP PRINCIPAL COMPONENT
Intuitively, a direction that clusters examples well should have high variance. Motivated by this, a simple method is to cluster according to the Top Principal Component (TPC). We do this by first constructing contrast features from the normalized contrast pair activations, C = [ő(x)) ) ) then projecting these examples on their top principal component using PCA. We then treat examples that are less than 0 as one cluster and examples that are greater than as the other cluster. We find that despite its simplicity this method can often also find truth- like features in model activations. This method is similar in spirit to Bolukbasi al. (2016), except that our method doesn't 't require any labels; we leverage the fact that truth is consistent to construct contrast pairs in purely unsupervised way.
G.5 CRC: BIMODAL SALIENCE SEARCH
Bimodal Salience Search (BSS) is another variation of CRC. One drawback of TPC is that variance not an intrinsic quantity to a network's behavior, since different directions in representation space can be scaled without changing the behavior of the network as long as subsequent layers are rescaled accordingly. This motivates using a method that clusters examples regardless of the scale of different directions.
If examples are well-clustered, then the intra-cluster variance should be low while the inter-cluster variance (or total variance) should be high. Specifically, if center a of datapoints that O delineates the cluster boundary, then the points on either side of 0 should have low variance compared to the points overall. This suggests minimizing the following loss:
L( var ci a < 0) + var(O c; TT 0}
(1)
where c := Ø(2;) o(z;)) and where we use var(z;} as shorthand to denote the variance of a set {= C R. Because this objective is written in terms of ratio, it is invariant to the overall scale of the direction, fixing that drawback of TPC. This loss is similar to that of Linear Discriminant Analysis (L.DA) (Fisher, 1936), but unlike LDA is completely unsupervised.
In practice we use an SGD-based optimizer to find a local optimum of L(0). After computing the contrast features, we repeat the following process T times. We first initialize a random direction 0 with unit norm. Then for E epochs, we calculate the loss according to Equation (1), and update the direction by via projected gradient descent, each time projecting back onto the unit sphere. Finally, select the direction 8 that has the lowest loss among the T directions we found, and predict based on this direction. When we optimize the direction with multiple datasets, i.c. S .... Sn. use the same algorithm, but average the loss across all datasets. In practice and use Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0.1.
T F. 20,
H STATISTICAL SIGNIFICANCE
Our main accuracy results for CCS and other methods (e.g. in Table and elsewhere) are computed by evaluating the method on 40% of the 1000 (or 500 in the case of COPA) examples sampled for each dataset, then averaging the resulting accuracy across prompts per dataset (on average), 10 different datasets, and up to 5 models. This corresponds to about 180k samples in total; we performed subsampling in this way for computational efficiency reasons, as this is already substantial. For results where we average across datasets (i.e. most results in this paper), we average across 3800 IID samples (the original examples sampled from each dataset), then usually also averaged across very large number of correlated samples: 9 prompts and (often) 5 different models for a given sample.
We can compute an upper bound on the standard error of accuracies computed in this way by ignoring the averaging across different prompts and models: this gives us simple but coarse upper bound of the standard error of 2//300 2 0.8%. Applying a Wald test, the Wald statistic is W
pi0 1o- > [90
se(a)" (0.008)7
where (for example) μo is zero-shot accuracy and f is CCS accuracy. W is then x² ddsitribbteed with one degree of freedom.
We find that our main claims comparing accuracies are statistically significant at a 0.00001 level. This includes the claims that CCS outperforms zero-shot, with accuracies of 71.2% vs 67.2%, respectively;
23

---

that CCS is robust to misleading prompts while zero-shot isn't in the setting test, with accuracies of 83.8% vs 70.9%, respectively; that CCS on MMM--petrraiie DeBERTa substantially outperforms DeBERTa zero-shot, with accuracies of 93.7% 71.6%, respectively; and observations are not necessarily statistically significant, such as that CCS (All Data) outperforms CCS average (Section 3.2.1), but none of these important for main claims, and more powerful but complicated statistical tests would result in smaller p-values.
Some minor
24

---

I DATASET SETUP
In this section, we describe the setup of our data and prompts in detail. We introduce all datasets we use and the way we convert them into binary classification task. Our prompts derive from (Sanh et al., 2021)*.
Contrast Pair Example. To illustrate how we construct contrast pairs, suppose we have movie review "[text] = I loved this movie." and the sentiment of this review belongs to "[label0] = positive" or "[label1] = negative". We first format it into a binary question-ane -nnseerng or classification question: "[text] Is the sentiment of this example [label0] or [label1]] " using an existing zero-shot prompt. We then concatenate the question and candidate labels create the contrast pairs:
+ [prefix] Q: Is the sentiment of "[text]" [label0] or [label1]? A: [label0]
T [prefix] Q: Is the sentiment "[text]" [label0] or [label1]? [labell]
For instance, in this example, these would be:
x Q: the sentiment of "I loved this movie." positive or negative? A: positive
x Q: the sentiment of "I loved this movie." positive or negative? A: negative
1.1 TOKENIZATION
For all prompts consider in this paper, concatenate labels to the end of questions. The precise we do this depends on the model type. For encoder-decoder models such as T5, we replace line breaks (\n) with spaces, and we add space between the question and proposed answer if the last token of questions is not already a space. For autoregressive models, we add line break between the question and the proposed label if the last token of the question is neither a space nor line break. For DeBERTa, we add [SEP] split-token between the question and answer, and for RoBERTa, we add </>><>> between the question and answer.
Finally, we add the prefix (either prefix in Figure 5, or the empty string for the regular zero-shot setting) to the beginning of the prompt, then compute the hidden representations for this pair obtain the contrast features C.
1.2 DATASETS
We use ten datasets in our experiments. For each dataset, we select 1000 data points (except COPA, which has only 500 data points, so we only use those 500). When possible, validation split, but in other cases this is not enough data so we also use examples from the train split. For the most part this doesn't make a difference for our purposes because our methods are purely unsupervised.
just the test /
is important to note that TO is an exception among the models, because but was trained on out of the 10 datasets we consider (all except RTE). This may account for its anomalous performance. UnifiedQA was trained on BoolQ, which one of the datasets we test on, but was not trained on any other evaluation set we consider, which we don't "tn't expect has an important qualitative effect.
We now enumerate the prompts we use for each dataset.
1.2.1 AG NEWS
AG_News topic classification dataset from (Zhang et al., 2015). Given news paragraph, the label corresponds to the topic of this news, including "World politics", "Sports", "Business" and "Science and technology". For each example, we randomly select another label (which incorrect), and randomly assign the correct label and this incorrect label to "label0" and "labell". With this, we construct the classification task using the following 8 prompts:
[prefix]Consider the following example: [text]] bell]. Between choice and choice 2, the topic of this example is [label]
Choice [labe10]. Choice [la-
httpps:// //gi wttp..coo/.bgccciiieceeeco wwweeewwwwsssohoooeo workshop/prropptaours
25

---

[prefix] the following example: [text] Choice 1: [label0]. Choice 2: [la- bell].Between choice and choice 2, what is the topic of this example? [label]
[prefix]Consides the following example: " [text] Which is the topic of this example, choice I: [label0], or choice 2: [labelI]? [label]
[prefix][tex]] Choice 1: [label0]. Choice 2: [labelI] Jeetween choice 1 and choice 2, the topic of this example is [label]
[prefix][tex]] Choice I: [label0]. Choice 2: [label]]. choice 1 and choice 2, what is the topic of this example? [label]
[petfix][text] Which is the topic of this example, choice 1: [label0], or choice 2: [labeli]? [label]
[texfix[[tex]] What label best describes this news article, choice 1: [label0], choice 2: [labeli]? [label]
[prefix [text] Which section of a newspaper would this article likely appear in, choice [label0], or choice 2: [labell]? [label]
Here the last "[label]" is "choice 1 for x+ and "choice 2" for x- Notice that we only specify "[label]" for the prompts that are constructed manually. For prompts in (Sanh al., 2021), we leave their labels unchanged. (For example, we use "choice 1" or "choice 2" but they can use "Animal" or "Plant")
1.2.2 AMAZON.POLARITY
Amazon _polziipp is sentiment classification task from (McAuley & Leskovec, 2013). The content is the review of goods in Amazon, and the label can be "negative" or "positive". We use 1l different prompts in this dataset. We first take all prompts from (Sanh al., 2021) (Page 164, prompts in total), and the two of our own as follows:
the following example: [content] "* Between [label0] and [labell], the sentiment of this example is [label]
[prefix]Consider the following example: [content] " Between [label0] and [label1], which is the sentiment of this example? [label]
Here "[label]" is "negative" for x and "positive" for TT
1.2.3 BOOLQ
BOOLQ is a QA task where each example consists a yes/no question from (Clark et al., 2019). We directly use the prompts from (Sanh et al., 2021) (Page 146)
1.2.4 CОРA
COPA is a causal reasoning task determine either the cause or the effect of given premise (Roemmele et al., 2011). 1). Here the label is a short sentence. We use 10 prompts, where are from (Sanh et al., 2021) (Page 177), and we add one more prompt:
(oonsider the following premise: ... [premise] "* Choice [choice]] Choice [choice2] Q: Which one is more likely to be the [question], choice I or choice 2? [label]
1.2.5 DBPEDIA.14
DBpedia_ 14 is topic classification dataset constructed by picking 14 non-overlapping classes from DBpedia 2014 (Lehmann et al., 2015). We manually create 8 prompts. For each example, we randomly select the incorrect label from the remaining 13 classes, and randomly assign the correct label and this incorrect label to "[label0]" and "[labell]".
[prefix]Consider the following example: ". [content] "* Choice 1: [label0]. Choice [label Between choice I and choice 2, the topic of this example is [label]
26

---

[prefix]Consider the following example: [content] Choice 1: [label0]. Choice 2: [labell].Betwenn choice 1 and choice 2, what the topic of this example? [label]
[prefix]Conside the following example: [content] Which is the topic of this example, choice 1: [label0], or choice 2: [label[]? [label]
[prefix][content] Choice I: [label0]. Choice 2: [aaeellJBetteee choice and choice topic of this example is [label]
[prefix [[content] Choice 1: [label0]. Choice 2: [label1J.Betwace choice and choice 2, what is the topic of this example? [label]
6 [prefix[[content] Which is the topic of this example, choice 1: [label0], or choice 2: [label1]? [label]
[poeti]][conten] What category does the paragraph belong to, choice 1: [label0], or choice 2: [[abel1]? [label]
[conten]] What label best describes this paragraph, choice [label0], or choice [label1]? [label]
Here "[label]" "choice 1" for x and "choice 2" for x
1,2.6 IMDB
IMDB is a sentiment dataset from (Maas et al., 2011). Given a movie review, the label is either "[]abel0] negative" or "[labell] positive. We 13 prompts, where 11 from (Sanh al., 2021) (Page 168), and the rest two are as follows:
[prefix]Consider the following example: " [text] " Between [label0] and [labelI], the sentiment of this example is [label]
[prefix]Consider the following example: " [text] " Between [label0] and [label1], which is the sentiment of this example? [label]
Here "[label]" is "negative" for x and "positive" for .T.
1.2.7 PIQA
The PIQA dataset measures the physical commonsense reasoning ability of models. We use 11 prompts for PIQA, all of which are from (Sanh et al., 2021) (Page 160). The label is complete sentence that can be the solution of the question.
I.2.8 QNLI
QNLI from (Rajpurkar al., 2016) a question-anse answering dataset consisting of question partirrpp pairs, where one of the sentences in the paragraph (drawn from Wikipedia) contains the answer to the corresponding question (written by an annotator). We use prompts from (Sanh al., 2021). The label is either "yes" or "no" depending whether the information in the paragraph is enough to answer the paragraph.
I.2.9 RTE
RTE a textual entailment dataset (Wang et al., 2018). The label corresponds to whether the text entails the hypotheses. We use 11 prompts, where 10 from (Sanh al., 2021 (Page 48), and where we manually add one more prompt:
[prefix[[premise] Question: Does this imply that "[hypothesis]". yes or no? [label]
Here "[label]" is "yes" for t and "no" for oT
27

---

1.2.10 STORY .CLOEE
Story Cloze is a story completion task from (Mostafazadeh al., 2017). Given short story, the task is to determine which of two endings is more likely to continue that story. We use 9 prompts, where from (Sanh al., 2021), and the remaining are as follows:
Donsiier the following story: [input sentence 1] [input sentence 2] [in- put_sentence__) [input senput_sentence_4]] "" Choice I: [sentence_quiz) Choice 2: [sentence_quiz2] Which is the more plausible ending of this story, choice or choice 2? [label]
[prefix]Consider the following story:
[input.sentence_I] [input_sentence_]] 22] [in- put sentence 3] [input sentence 4] "" Choice 1: [sentence quizl] Choice 2: [sentence quiz2] Which is the more plausible ending of this story? [label]
1] seapputseeteene_2] fimput_sentence (input_sentence Choice I: cenntence_quiz/ Choice 2: [sentence.quiz/] Which is the more plausible ending of this story, choice or choice 2? [label]
Here "[label]" "choice I for x and "choice 2" for x
28


Reviews:
[{"review_id": "bq0oGA6O8h", "reviewer": "Reviewer_eJhP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce an unsupervised probe for knowledge in large language models to tackle a perceived challenge in verifying truthfulness of predictions/generated text. The authors' proposed method involves classifying/ranking model activations between binary labels/framings of a yes-no question. The authors demonstrate that their CCS probe is resilient to prompt-based attacks.", "summary_of_the_review": "This paper tackles an important challenge with verifying truth and knowledge in language models, but I have some questions about the basic premise of how it shows what it claims (a better way to probe for knowledge).", "strength_and_weaknesses": "Strengths:\n- The paper targets an important issue with language models: verifying truthfulness of outputs and whether knowledge is learned within the model parameters.\n- There is an interesting result in 3.2.2 for how CCS is robust to adversarial prompts. I would have liked to see more discussion on why this particular design of \"misleading prefix\" was chosen, however.\n- CCS is demonstrated to be usable with multiple models across a wide range of tasks.\n- Good discussion of limitations and possible future challenges.\n\nWeaknesses:\nI'm having a bit of a challenge reconciling the challenge of verifying truthfulness with the focus on results of how CCS out-performs zero shot approaches on accuracy - as mentioned in the paper, it *does* indicate that there is a disconnect between inferences made via one modality (ranking of activations) vs. direct model outputs, but it is difficult to see how that shows that one method is \"better\" than the other for probing knowledge.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OncAEbHwGh", "reviewer": "Reviewer_r2Hq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to understand language models' internal knowledge about truth values of the text that it outputs.\nGiven a set of yes-no questions, they train a truth classifier in an unsupervised way by first extracting the hidden states of the language model when it gives a positive or negative answer, respectively. They train the classifier to be consistent, i.e., the probabilities of the two outcomes should sum to one, and confident (one label should receive high probability).\nThe model is tested on zero-shot binary classification on a large set of tasks. It performs reasonably well, outperforming standard LM-based zero-shot classification on average by a substantial margin. They further discuss and test several alternative hypotheses for their model's success, aiming to show that their model indeed captures internal information that is qualitatively different from the information in the model's textual output.", "summary_of_the_review": "I applaud the technical and empirical novelty of the paper. I think it will be of significant interest to many in the community. The main claims are fairly well supported. As noted in the rebuttal, some aspects of the methodology are confusing and could maybe be executed and/or explained/justified better.", "strength_and_weaknesses": "Strengths:\n* very strong motivation\n* simple and effective idea\n* very valuable empirical results to the community\n* discusses relevant alternative hypotheses\n\nWeaknesses:\n* **update post rebuttal**: my biggest concerns have been addressed\n* statistical validity is unclear due to small test sets\n    * why are the datasets subsampled to include only 1000 examples, of which 400 are the test set? \n    * these are unneccessarily small test sets, causing greater uncertainty and therefore confidence in the results\n    * can you please justify why you believe that differences are unlikely to be due to randomness? Note that stds in Table 1 are relatively large.\n* evidence for disproving alternative hypotheses is weak in several cases:\n    * Section 3.2.2: CCS Is Robust To Misleading Prompts\n        * goal: \"discover latent knowledge in an LM even when its **training objective** causes the model to output false text\"\n        * method: change **test time** prompts and observe difference\n        * method is not well suited for the goal, because you don't touch the training objective of the model at all.\n        * result is found for only one out of 5 models. Combined with the statistical uncertainty above, I am not convinced that this is sufficient evidence that this is a valid, general finding.\n    * Section 3.3.2: CCS Can Find Intermediate Representations of Truth\n        * motivation: output text might encode truth, and CCS may merely correlate with output text\n        * method: show that CCS applied to intermediate layers also works well.\n        * implicit assumption: lower layers don't correlate with the produced output text (as much?). This needs evidence of some kind, e.g. a reference.\n        * results: lower layers perform worse, but according to the paper, they still perform well enough. It isn't clear what constitutes well enough.\n        * (their) conclusion: lower layers do well despite not correlating with the output text, so they have to encode truth qualitatively different from the output text.\n        * Lower performance at lower layers may be entirely explained by decreasing (but not vanishing) correlation with the output text. Since there is no quantification of the latter, we cannot make the above conclusion.\n\nAlternative method to prove that internal truth values are present independent of the input:\n* if truth values are encoded independent of textual output, you should be able to extract them by:\n    1. replacing positive and negative labels with completely uninformative, random text. Since you aim to normalize out the effect of \"Yes\" and \"No\", the appended text shouldn't matter either way, or\n    2. don't alter x_i, but train the model with two different classification heads.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "N7mfjQpnOn3", "reviewer": "Reviewer_YocT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Contrast-Consistent Search (CCS), that tries to identify latent knowledge in language models in an unsupervised manner. The process is to convert each question into binary-answer pairs, then train the LM in an unsupervised way via optimizing a consistency loss (probabilities of opposite answers should sum to 1, and a confidence loss to avoid degeneration.\n\nThe proposed method outperforms (calibrated) zero-shot accuracy by 4% on average over 10 question-answering tasks. The authors did some additional analysis like CCS's robustness to misleading prompts and the transferability of CCS.", "summary_of_the_review": "Despite its limited setting (tasks need to be converted to questions with binary answers, and the need of a large number of unsupervised samples), this paper's contribution is quite novel with relatively strong empirical results, thus I recommend weak accept.", "strength_and_weaknesses": "Strengths:\n- To the best of my knowledge, exploring the internal consistency of language models in an unsupervised way is a novel contribution. The authors proposed an interesting method via optimizing the LM with a consistency and a confidence loss to figure out the more likely choice.\n- The empirical results are significant across 4 out of 6 models, and across 10 tasks.\n\nWeakness:\n- The proposed approach applies only to questions that can be converted with binary answers (i.e., the answer choices are fixed and are from a very limited set, otherwise the inference cost would increase substantially). \n1) This largely constrains the applicability of this method to many tasks, e.g., math tasks that have numbers as answers, which would require generating an infinite number of binary-answer questions.\n2) for some tasks the conversion might be non-trivial and requires additional human processing.\n\n- Although the proposed method doesn't use any labels, it still uses input examples and requires a fairly large number of training examples to perform well (Figure 5) due to the optimization procedure. This could pose a challenge compared to true zero-shot learning which doesn't require any examples at all. For any new task, how to obtain a large number of training examples (even without labels)?\n\n- Table 1 presents average performance. Is there a more detailed breakdown of performance gains across different tasks? Are there categories of tasks that benefit from the method more vs less?\n\n- Relatedly, as the authors have stated, CCS relies on the assumption that true/false inputs can be separated reasonably well in the activation space. Are there analysis regarding what are tasks that satisfy this property and what tasks are less easily separated?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BHRdOoutSPE", "reviewer": "Reviewer_32x3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel approach to address a fundamental problem in supervised machine learning: misalignments exist between a training objective and the truth. Instead of telling the model \"what is true\" by human annotations, they proposed to learn the truth by the model itself in a purely unsupervised way. Specifically, they introduced Contrast-Consistent Search (CCS), a method that learns a linear projection of the hidden states that satisfies logical consistency (consistent across negations). Given a yes-no question, they first construct two instances with the binary answers concatenated to the question. They then train a classifier to predict the binary labels using the hidden states of a pretrained language model. To train the classifier, they add several constraints to insure logical consistency. For example, a statement and its negation should have probabilities that add up to 1. They conducted experiments using 6 different models and the empirical results on 10 question answering datasets showed they can improve the zero-shot performance by 4% compared to the baselines.", "summary_of_the_review": "In general, the proposed approach is novel and the empirical results seem strong. However, at the current stage, several claims are not well supported by the experiments and it is not super clear what the takeaways are. At the current stage, I lean toward rejection but I think improvements can be made to make it a very strong publication.\n", "strength_and_weaknesses": "Strengths: \n- The proposed approach is novel to my knowledge.\n- Strong empirical results across many datasets and models.\n- The problem this paper trying to tackle is fundamental \n\nWeaknesses:\n- My major concern about this paper is that the claims made are not well supported by the experiments.\n\t- In section 3.2.2, \"Recall our motivating goal: to discover latent knowledge in a language model even when its training objective causes the model to output false text.\" I don't see how this claim is related to the experiments in this section. The experiment is conducted in a zero-shot setting in which no training objectives are used. What they really do is use a set of contrastive prompts to mislead the model and the experimental results show that **CCS can improve the robustness of the model**. That's the takeaway I can conclude from the experiment.\n\t- In section 3.3.1, \"CCS FINDS A TASK-AGNOSTIC REPRESENTATION OF TRUTH\". I am not super convinced by this claim, as all of the models (without fine-tuning using the CCS objective) used for this experiment can already make zero-shot generalizations on those datasets to some extent. It is unclear to me what is added by the CCS objective. Does it just make the model more robust against domain shifts? Or it captures something else? Moreover, do you have a more formal definition of TRUTH here?\n\t- In section 3.3.2 \"CCS CAN FIND INTERMEDIATE REPRESENTATIONS OF TRUTH\". I don't quite get the logic behind this experiment. How can we conclude by looking at the results that lower layers perform worse? Can you elaborate on this a bit more?\n- The baseline seems a bit weak to me. How about you just fine-tune the models used in the paper using the original training objective on those datasets used for testing?\n- The \"purely unsupervised\" setting seems to be a bit overclaiming. In the experiments, they still need a non-trivial number of training examples to make the proposed approach work. It would make the paper much stronger if they could use examples from a more \"wild\" setting, e.g., sentences from a random corpus, to train CCS and show it could still work.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster