Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?
Authors: Kevin Liu
Stephen Casper
Dylan Hadfield-Menell
Jacob Andreas
Venue: EMNLP 2023 Main (2023)
Keywords: language models
truthfulness
question answering
interpretability

Abstract: Neural language models (LMs) can be used to evaluate the truth of factual statements in two ways: they can be either queried for statement probabilities, or probed for internal representations of truthfulness. Past work has found that these two procedures sometimes disagree, and that probes tend to be more accurate than LM outputs. This has led some researchers to conclude that LMs "lie" or otherwise encode non-cooperative communicative intents. Is this an accurate description of today's LMs, or can query--probe disagreement arise in other ways? We identify three different classes of disagreement, which we term confabulation, deception, and heterogeneity. In many cases, the superiority of probes is simply attributable to better calibration on uncertain answers rather than a greater fraction of correct, high-confidence answers. In some cases, queries and probes perform better on different subsets of inputs, and accuracy can further be improved by ensembling the two.

Cognitive Dissonance:
Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?
Kevin Liu"
Stephen Casper"
Dylan Hadfield-Menell
Jacob Andreas
MIT CSAIL
[kevliu, scasper, dhm, jda) t. edu
Abstract
Neural language models (LMs) can used to evaluate the truth of factual statements in two ways: they can be either queried for statement probabilities, or probed for internal represen- tations of truthfulness. Past work has found that these two procedures sometimes disagree, and that probes tend to be more accurate than LM outputs. This has led some researchers to conclude that LMs "lie" or otherwise encode non-cooperative communicative intents. Is this an accurate description of today's 's LMs, or can query-probe probe disagreement arise in other ways? We identify three different classes of disagree- ment, which we term confabulation, deception, and heterogeneity. many cases, the superi- ority of probes is simply attributable to better calibration on uncertain answers rather than a greater fraction of correct, high-confidence answers. In some cases, queries and probes perform hetter on different subsets of inputs, and accuracy can further be improved by en- sembling the two.'
Language
0.70
"Dunppaa""
Bittre predts
Sgrieeal
0.02
e Win pellon officer?
Mudii latins
Hilero
piproee a out luut....
Probe
Cariike
Prite Fane
connect 0.
2: Stooe pulle offert yo
piquery is ocenect)
905
0%
CREAK
Figure 1: Varieties disagreement between language model outputs and internal states. We evaluate two approaches for answering questions or verifying state- ments: querying models directly for answers (A), or training binary classifier probe their internal states (B). Probes and queries sometimes disagree. We pro- pose taxonomy different query-probe disagreement types (C); across several models and datasets, find that disagreements mostly occur in situations where either probes queries are uncertain (D).
1 Introduction
Text generated by neural language models (LMs) often contains factual errors (Martindale et al., 2019). These errors limit LMs' ability to gener- ate trustworthy content and serve as knowledge sources in downstream applications (Ji et al., 2023). Surprisingly, even when LM outputs are factu- ally incorrect, it is sometimes possible to assess the truth of statements by probing models' hidden states, the example shown in Fig. IA, a language model assigns high probability to the incorrect an- swer yes when prompted to answer the question Is Sting a police officer? However, linear knowl- edge probe trained on the LM's hidden represen- tations (Fig, IB) successfully classifics as the more likely answer. Knowledge probes of this kind
have consistently been found to be slightly more reliable at question answering and fact verification than direct queries to LMs (Burns et al., 2022), suggesting a mismatch between LMs' (internal) representations of factuality and their (external) expressions of statement probability.
How should we understand this behavior? One possible interpretation, suggested by several pre- vious studies, that it is analogous "deception" in human language users (Kadavath et al., 2022: Azaria and Mitchell, 2023; Zou et al., 2023). this framing, if L.M representations encode truthful-
"Equal contribution.
"ooee at gi thub. cow/lingo-miniieeeoo t/1m- ttr.funnttuuuulnees...

---

ness more reliably than outputs, then mismatches between LM queries and probes must result from situations which LMs "know" that statement untrue and nonetheless assign it high probability.
2 Preliminaries
Extracting answers from LMs We study autore- gressive LMs that compute a next-token distribu- tion PLM f<ci) by first mapping the input T<i to some hidden representation (I then us- ing this representation to predict z;. There are two standard procedures for answering questions using such an LM:
But knowledge is not a binary phenomenon in LMs humans. While probes are better than queries on average, they may not be better on every example. Moreover, both LMs and probes are prob- abilistic models: LM predictions can encode uncer- tainty over possible answers, and this uncertainty can be decoded from their internal representations (Mielke et al., 2022). There are thus a variety of mechanisms that might underlic query-probe dis- agreement, ranging from differences in calibration to differences in behavior in different input subsets.
1. Querying: Provide the question q as input to the LM, and rank answers (e.g. yes/no) according to (L q) (Petroni et al., 2019).
2. Probing: Extract the hidden representa- tion h(1g, a]) from the LM, then use ques- tion/answer pairs labeled for correctness (or unsupervised clustering methods) to train a binary classifier that maps from hLM to a dis- tribution Pprobe h) (Burns et al., 2022).
In this paper, we seek to understand mismatches between internal and external expressions of truth- fulness by understanding the distribution over pre- dictions, taking into account uncertainty in an- swers produced by both queries and probes. In doing so, we hope to provide a formal ground- ing of several terms that are widely (but vaguely and inconsistently) used describe factual errors in LMs. LMs and probing techniques are rapidly evolving, so we do not aim to provide a defini- tive answer to the question of why LMs assign false statements high probability. However, a widely studied probe class, two LMs, and three question- answering datasets, we identify three qual- itatively different reasons that probes may outper- form queries, which we call confabulation (queries produce high-confidence answers when probes are low-confidence), heterogeneity (probes and queries improve performance on different data subsets), and (in a small number of instances) what past work would characterize as deception (Fig. IC-D, in which queries and probes disagree confidently on answer probabilities). Most mismatches occur when probes queries uncertain about answers. By combining probe and query predictions, it is sometimes possible obtain better accuracy than either alone.
Causes of disagreement Past work has shown that this probing procedure is effective; recent work has shown that it often produces different, and slightly better, answers than direct querying (Burns et al., 2022). Why might this mismatch occur? In this paper, we define three possible cases:
Model Confabulation (Edwards, 2023): Dis- agreements occurring when probe confidence is low, and the completions from queries are incorrect (mid-left of Figure 1). In these cases probes may be slightly more accurate if their prediction confidence is better calibrated to the probability of correctness. In LMs exhibit- ing confabulation, large fraction of disagree- ments will occur on inputs with large probe en- tropy prrobeccorrect (correct (q, a)) and large query confidence Pquery (a q) for any a.
2. "Deception" (Azaria and Mitchell, 2023). which we define as the set of disagreements in which the probe is confidently correct and the query completion is confidently incorrect (upper-left of Figure 1). In these cases, a large fraction of disagreements will occur on ques- tions to which queries and probes both assign high confidence, but to different outputs.
Our results paint a nuanced picture of the repre- sentation of factuality in LMs: even when probes outperform LMs, they do not explain all of LMs' ability to make factual predictions. Today, many mismatches between internal and external represen- tations of factuality appear attributable to different prediction pathways, rather than an explicit rep- resentation of a latent intent to generate outputs "known" to be incorrect.
We use this terminology for consistency with past work. and do not intend any claims about the presence of specific communicative intentions in LMs (q.v. Abererombic et al., 2023; Shanahan, 2022). In humans, deception involves models of other agents' mental states (Mahon, 2008) of kind that are

---

3. Heterogeneity: Disagreements resulting from probes and queries exhibiting differential ac- curacy on specific input subsets (upper-mid of Figure 1). In these cases, probes may outper- form queries the subset of inputs on which probes are more effective is larger than the (disjoint) subset on which queries are more effective.
BoolQ
SciQ
CREAK
GPT-2
GPT-
GPT-2
GPT-
GPT-2
GPTJ
Query
61.8
61.8
76.1
84.1
51.1
50.4
Probe
62.9
62.5
78.8
88.6
63.5
71.0
Ensemble
62.7
79.7
90.5
71.3
Table 1: Accuracies of different evaluation approaches. As in past work, we find that prohes are consistently accurate than queries. Surprisingly, by ensembling probes and queries together. it is possible in 4 of 6 cases to obtain further improvements.
These three categories (along with cases of query-probe agreement, and probe confabulation and error) are visualized in Fig. IC. Behaviors may occur simultancously in single model: are ulti- mately interested in understanding what fraction of predictions corresponds to each of these categories.
BoolQ
SeIQ
CREAK
Probe
544
Kommany
Datasets We evaluate predictions on three datasets: BoolQ, a dataset of general-topic yes-no questions derived from search queries (Clark et al., 2019); SciQ, a dataset of crowdsourced multiple- choice science exam questions, and CREAK, dataset of crowdsourced (true and false) factual as- sertions (Onoe al., 2021). We evaluate model behavior on all three datasets via a binary ques- tion answering task. SciQ contains multiple wrong answers for each question; we retain the correct answer and select a single distractor.
Query
Confidence
Figure 2: Calibration of GPT-J queries and probes. Each point represents group of predictions: the horizontal axis shows the query's average confidence Ey,ap(a a(a g). the vertical axis shows the query's empirical accuracy E a is correct). and point radius shows the number of predictions in the group. Probes are substantially better calibrated than LM queries.
Models As the base LM (and implementation of Pquery), ), we use the GPT2-XL (Radford al., 2019) and GPT-J LMs (Wang and Komatsuzaki, 2021). We query LMs by evaluating the probability they assign to correct answers. In BoolQ and CREAK we normaaale their output distributions over the strings (true, false); in SciQ we use provided cor- rect and incorrect a
regression objective) to classify (question, answer) pairs as correct incorrect using the training split of each dataset described above. The input to this linear model is the LM's final hidden state in the final layer (we did not find substantial differences in probe accuracy using different layers or repre- sentation pooling techniques). As in past work, we obtain distribution over answers by normalizing over Pprobe (correct (corrett q, a). Our main results train ordinary lincar model; additional results with sparse probing objective (as in e.g. Voita and Titov, 2020) are in Appendix A.
Probes While the space of probe designs is large, many recent studies have used linear probes in LMs (Hernandez and Andreas, 2021; Ravfogel al., 2022; Burns al., 2022; Marks and Tegmark, 2023).* 4 We train a linear model (using a logistic
not exhibited by the LMs study (Sap al., 2022). What call "deception" is necessary but insufficient for an LM to "believe" one thing but choose to another.
"Just as different behavior may exhibited by different models, it may he induced by different prompts or query formats (I et al., 2022). We experimented with different query formatting strategics but found striking changes results. However, future work may more systematically study the distribution query query-probe disagreements induced by different prompts.
The success of probes over queries can largely be explained by better calibration
By evaluating accuracy of probes and queries on held-out data, we replicate the finding that probes are more accurate than queries (Table 1). some cases the difference is small (less than 1% for GPT- J on BoolQ); in other cases, it is substantial (more
"The probing paradigm hus limitations. A successful probe does not indicate that the LM necessarily uses the feature being probed for, and an unsuccessful probe does not indicate that the LM does not use the feature being probed for (Ravichander ct al., 2020: Elazar et al., 2021; Belinkov, 2022).

---

both conrect
GPIZ-XL
probe correct
query
both incorrect
/query is correct
GPT-
probe is correct %
BoolQ
SeiQ
CREAK
LEGEND
Figure 3: Distribution query and probe predictions. "Deception" -like results (orange) do not feature particularly prominently compared other outcomes. Note that heatmaps (left) use the same shade of gray for "both correct" and "both incorrect", even though they are distinguished in histograms (right) to enable direct comparison.
than 20% on CREAK).
substantially between datasets. In BoolQ and SciQ. the query correct and the probe is incorrect nearly as often as the opposite case; on both datasets, ex- amples of highly confident but contradictory pre- dictions by probes and queries are rare.
Another striking difference between probes and queries is visualized in Fig. 2, which plots the cal- ibration of predictions from GPT-J (e.g. when model predicts is it right 70% of the time?). Probes well cali- brated, while queries are reasonably well calibrated on SciQ but poorly calibrated on other datasets. This aligns with the finding by Mielke et al. (2022) that it is possible to obtain calibrated prediction of model errors (without assigning probabilities to specific answers) using similar probe.
answer with 70% probability,
Fig. ID shows the fraction of examples in each of the nine categories depicted in Fig. IC. Only in CREAK do observe significant number of instances of "deception") however, also observe many instances of probe errors. all other cases (even SciQ, where probes are substantially more accurate than queries), query- mismatches predominantly represent other types of disagree- ment.
important to emphasize that the main goal this work is understand representational cor- relates of truthfulness, not to build more accurate models. Indeed, the superior calibration and accu- racy of probes over queries might not seem surpris- ing given that probes are trained many-shot while queries use the model zero-shot. To contextual- ize these results, also finetuned GPT2-XL. true question/answer pairs and found it performed better overall than probing the pretrained model. Probe predictions exhibited 1. 1.7% better accuracy than fine-tuned model queries on BoolQ. However, querying the fine-tuned model had 6.9% and 1.9% better accuracy on SciQ and CREAK.
5 Queries and probes are complementary
Another observation in Fig. that almost all datasets exhibit a substantial portion of heterogene- ity: there are large subsets of the data in which probes have low confidence, but queries assign high probability to the correct answer (and vere-versa). We exploit this fact by constructing semble model of the form:
Pensemble (a 4) = X Pprobe g)
(1)
+ (1- A) Pquery (a
4 Most disagreements result from confabulation and heterogeneity
We select À using 500 validation examples from each dataset, and evaluate accuracy the test split. Results are shown at the bottom of Table 1; in 4/6 cases, this ensemble model is able improve over the probe. While improvements on BoolQ and CREAK are small, on SciQ they are substantial nearly as large as the improvement the probe
Next, we study the joint distribution of query and probe predictions. Results are visualized in Fig. GPT2-XL and GPT-J exhibit a similar pattern of errors on each dataset, but that distribution varies

---

over the base LM. These results underscore the fact that, even when probes significantly outper- form queries, query errors are not the source of all mismatches, and heterogeneity in probing and querying pathways can be exploited.
errors. However, better understanding of model- internal representations of factuality and truthful- ness might enable system developers to steer mod- els toward undesirable or harmful behaviors.
Finally, while we have presented techniques for slightly improving the accuracy of models on ques- tion answering tasks, models continue to make a significant number of errors, and are not suitable for deployment in applications where factuality and reliability are required.
6 Conclusion
We studied mismatches between language models' factual assertions and probes trained on their inter- nal states, and showed that these mismatches reflect a diverse set of situations including confabulation, heterogeneity, and a small number of instances of deception. In current models, disagreements between internal and external representations of truthfulness appear predominantly attributable to different prediction pathways, rather than latent intent to produce incorrect output. A variety of other model interventions are known to decrease truthfulness, including carefully designed prompts (Lin et al., 2022), and future models may exhibit more complex relationships between internal repre- sentations and generated text (especially for open- ended generation). Even in these cases, we expect the taxonomy in Fig. 1 to remain useful: not all mismatches between model and probe behavior in- volve deception, and not all model behaviors are (currently) reducible to easily decodable properties their internal states.
Acknowledgments
This work was supported by grant from the Open- Philanthropy foundation.
References
Gavin Abererombie, Amanda Cercas Curry, Tanvi Dinkar, and Zeerak Talat. 2023. Mirages: On an- thropomorphism in dialogue systems, arXiv preprint arXiv:2305.09800.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for align- ment. arXiv preprint 1rXiv:2112.000
Amos Azaria and Tom Mitchell. 2023. The internal state of LLM knows when it's lying. arXiv preprint arXiv:2304.15 13734.
7 Limitations
Yonatan Belinkov, 2022. Probing classifiers: Promises. shortcomings, and advances. Computational Linguis- rics, 48(1):207-
As seen in Fig. 1, there is significant heterogeneity in the distribution of disagreement types across datasets. These specific findings may thus not predict the distribution of disagreements in future datasets. As noted in Section 2, our experiments use only a single prompt template for each experi- ment; we believe it is likely (especially in CREAK) that better prompts exist that would substantially alter the distribution of disagreements- our goal in this work has been to establish taxonomy of er- rors for future models. Finally, we have presented only results on linear probes. The success of ensem- bling methods means that some information must be encoded non-linearly in model representations.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Stein- hardt. 2022. Discovering latent knowledge in lan- guage models without supervision. In The Eleventh International Conference on Learning Representa- tions.
Christopher Clark, Kenton Lee, Ming-Wer Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the Conference of the North American Chapter of the Association for Comptuational I Linguistics.
Benj Edwards. 2023. Why ChatGPT and Bing Chat are so good at making things up. Ars Technica.
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic probing: Behavioral expla- nation with amnesic counterfactuals, Transactions of the Association for Computational Linguistics, 9: 160 175.
8 Ethical Considerations
Our work is motivated by ethical concerns raised in past work (e.g., Askell al., 2021; Evans et al., 2021) that LMs might (perhaps unintentionally) mislead users. The techniques presented here might be used to improve model truthfulness or detect
O wwain Evans, Owen Cotton-Barran Lukas Finnve- den, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. 2021. Truthful Al:

---

Developing and governing Al that does lie. arXiv preprint urXhv22100006744
Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. 2020. Probing the probing paradigm: Does probing accuracy entail task relevance? In Proceed- ings of the Annual Meeting the European Associa- tion for Compulationl Linguistics.
Evan Hemandez and Jacob Andreas. 2021. The low- dimensional linear geometry of contextualized word representations. Proceedings of the Conference on Natural Language Learning.
Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. 2022. Neural theory-of-mind? On the limits of social intelligence in large Ims. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.
Ziwei Ji. Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su. Yan Xu, Etsuko Ishii, Ye Jin Bang. Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 35(12):1-38
Murray Shanaban. 2022. Talking about large language models. arXiv preprint vrXiv:2212.03552
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.
Elena Voita and Ivan Titov. 2020. information-theorete probing with minimum description length. In Pro- ceedings of the Conference Empirical Methods in Natural Language Processing.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: A 6 Billion Parameter Autoregressive Lan- guage Model. https://githen http://gggguuu..cc////oooooooo// com/ mesh- trrtssrrrrrr-jjjff
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the Annual Meeting of the Association for Computational Linguisties.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin. Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023. Representation engineering: A top- down approach to Al transparency. arXiv preprint arXiv:2310011055.
James Edwin Mahon. 2008. The definition of lying and deception.
Samuel Marks and Max Tegmark. 2023. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.0682
Marianna Martindale, Marine Carpuat, Kevin Duh, and Paul McNamee. 2019. Identifying fluently inade- quate output in neural and statistical machine transla- tion. In Proceedings of Machine Translation Summit XVII: Research Track.
Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y- Lan Boureau. 2022. Reducing conversational agents' overconfidence through linguistic calibration. Trans- actions of the Association for Computational Linguis- tics, 10:857-88 872.
Yasumasa Onoe, Michael IQ Zhang, Funsol Choi, and Greg Durrett. 2021. CREAK: dataset for common- sense reasoning over entity knowledge. In NeurIPS Datasets And Benchmarks.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models knowl- edge bases? In Proceedings the Conference on Empirical Methods in Natural Language Processing.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188):
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. 2022. Linear adversarial concept erasure. Proceedings of the International Confer- ence on Machine Learning.

---

A Sparse Probing Results
Ascumayy 61%
Accuticy: Spanily:
Accuracy: 62% Spuniity
Accunacy: Spunsty:
Accurccy:
Accuracy: 58% Spanilyy
Accumicy Spanily:
Accuracy:
Spansl:: 53%
Spanity:
Spaniiy,
7ype
bype
4pe
apa
9yp
by*
(a) BoolQ
(b) SciQ
Accurccy: 1056
Accuracy: 67%
Axcurccy 60%
Accuracy: 50%
Spanily: 55%
Spanity: 90%
Spanity: 99%5
Spaniit:: 100%
both
probe
query
both incorrect
(c) CREAK
Figure 4: Sparse probing results. these experiments, we train the same probes as in the main paper, but with a varying l penalty applied to the probing objective encourage discovery of sparse solutions. Here we report the level of sparsity, probe accuracy, and distribution of disagreement types vary the strength of the regularizer within {0, 0.01, 0.03, 0.1). Except at extremely high sparsity values, both accuracies and error distributions remain similar those reported the main of experiments.


Reviews:
[{"review_id": "1mUPTI6hfC", "reviewer": "Reviewer_aR1Y", "summary": "", "questions": "Please check the above section, thanks!", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We appreciate your time and feedback! \n\n### (1) Re: working on presentation in general\n\nThank you for pointing this out including specifics. We think that we can make improvements. In addition to addressing the specific items from the reviews, we will (a) comb through the paper ourselves with fresh eyes to reduce jargon, clarify terms, and more carefully explain experiments and (b) get additional help with clarifying the writing from another lab member who was not involved in the drafting of the paper. \n\n### (2) Re: lines 124-126 – what does it mean for a query to be correct?\n\nThank you for asking. When we refer to queries as incorrect, we mean that the model’s completion of the query is incorrect as depicted in Figure 1a. For example, if we prompt the model with “Q: Is the sky blue? A: ” and the model places a higher probability on the next token being “no” than “yes”, we consider this an incorrect query of the model. \n\nWe will edit how we discuss this so that we refer to the completions from the queries to be correct/incorrect instead of the queries.\n\n### (3) Re: line 129 – does confabulation refer to the mid-left orange region in fig 1c?\n\nYes. Our mention here of confabulation is meant to refer to the light orange region in the middle-left of Figure 1c. We will mention this. We will do the same for deception and heterogeneity. Thank you for the suggestion. \n\n### (4) Re: line 140 – why can the model only be confident if the probe is?\n\nIn general, this will not necessarily be the case, but we intend to use this sentence to define the set of model predictions that we label “deception”. For predictions in this set, the model is confidently wrong but a probe can be trained which is still confidently right. We do not mean to make a causal claim here, and apologize if the wording was unclear. We will remove this sentence because we think the following one on line 142 more clearly and concisely conveys the idea. \n\n### (5) Re: Confidence v. Accuracy?\n\nThank you for pointing these out. \n\n**Confidence** – the probability placed on the correct output. **Accuracy** – the percent of examples for which the confidence in the *correct* output was >0.5. We will clarify this on line 82. \nWe use the term **question** in the colloquial sense: a natural language string whose answer the LM must produce.\n\nWe use the term **query** to refer to a specific procedure for obtaining answers to questions using an LM. When querying, the question is provided as input, and the LM’s next-token prediction is used to generate an answer. (Contrast this with **probing**, in which a (question, answer) pair is provided to the LM as input, and the LM’s internal representations are inspected to determine whether the answer is correct.) This distinction is discussed around line 109; we’ll add extra text to further clarify the question/query distinction.\n\nWe use **model** to refer to the network from which answers are obtained (e.g. GPT-2-xl). Occasionally this is used attributively (“model predictions” -> “predictions produced by a model”, “model behavior” -> “behavior exhibited by a model).\n\nIn the following locations, we used the word “model” when we should have used the word “query” (in the sense above):\n- Fig 1 caption\n- Fig 2, bottom row label\n- Line 213\n\nWe will make edits to clarify these usages, say “language model” rather than “model” where appropriate, and correct the above errors. Thank you for helping us clarify!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- As summarized before, the problem itself is interesting and a genuine problem that has potential to contribute to the fundamental understanding of LMs;\n- The taxonomy of error types (Figure 1. C) sounds plausible; \n- Substantial empirical results for a short paper; \n- The synergy effect between probing and query is interesting.", "reasons_to_reject": "The paper needs much work on its presentation. I find many parts very confusing to understand, especially starting Line 124. \n\nLine 124~126: \n> Confabulation (Edwards, 2023): Disagreements occurring when probe confidence is low, and queries are incorrect.\n\nWhat does it mean the queries are incorrect? It means the queries fed to the models are incorrect, or the models' prediction is incorrect?\n\nLine 129:\n> In models exhibiting confabulation\n\nWhat does it mean? Does it mean the orange block in Figure 1.C? If so, please link them. \n\nLine 140:\n> probes outperform queries because models can produce high-confidence answers only when probes are also confident.\n\nI totally lost my understanding here. Why probes must be confident to produce high-condifence answers?\n\nI strongly urge the author teams to better present here to avoid confusions. \n\nThere are many terms that are confusing through the whole article: \n- Confidence VS Accuracy? \n- Questions VS Queries?\n- Model VS Queries? (is the model a verb or a noun is even not clear). \n\nPlease try to address the confusion here. \n\nI give a low soundness score now (=2), I will increase it if the confusions are addressed.", "questions_to_authors": "Please check the above section, thanks!", "paper_topic_and_main_contributions": "This paper studies the behavior difference between directly querying the model and the result of model's probe. The major contributions are (1) highlighting the problem of the behavior difference between two paradigm of model evaluation; (2) creating a taxonomy of querying VS probing bahavior divergences; (3) find ensemble is helpful.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Not to my knowledge.", "Typos_Grammar_Style_and_Presentation_Improvements": "Presentation style is fine, but there are many confusions as discussed in previous section."}, "extra_text": {}}, {"review_id": "9kjb5dlmEb", "reviewer": "Reviewer_M8PW", "summary": "", "questions": "1. Looking at figure 1C, and reasoning about what deceptions are, shouldn’t the description of “Deception” say “queries are INcorrect”? (line 137)\n2. What was supposed to be represented in Figure 3, top right? That subfigure appears empty.\n3. Could you comment on the extent to which linear probes can accurately indicate what information is captured in the latent space? It seems somewhat unfair to contrast prompting to linear probing (without further controls/baselines) and argue that a high p(probe is correct) but a low p(query is correct) equals \"deception\".", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We appreciate your time and feedback! \n\n### (1) Re: one type of probing, one type of prompt, one task\n\nThank you for asking about these. \n\nWe have tried experiments with MLP probes, but have not found substantially different results. The probe performance has been comparable to linear probes in our experiments, and we have not noticed any qualitative changes regarding the main findings. We also note that the prior works on truthfulness in LLMs that largely inspired this work (e.g. Burns et al.) also use only linear probes. We will mention this in the discussion. \n\nWe have tried multiple prompting strategies, and both queries and probes are sensitive to these, but fluctuations have only been a few percentage points, and we have not found any qualitative changes regarding our main findings. We will mention this on line 170.\n\nWe focus on truthfulness, but emphasize that we work with three different datasets – one involving multiple choice questions, one involving boolean questions, and one involving boolean statements. \n\nWhile it is always possible to try more models and more datasets, we believe we are at the limit of what can be communicated clearly within the ACL short paper format (aR1Y agrees).\n\n### (2) Re: Probing cannot determine when the model is “deceiving” the user.\n\nOur use of “deception” is only based on the definition we provide – when the model is confidently wrong but the probe is confidently right. We do not intend any anthropomorphic claims, and we write this in footnote 1. We also agree that it is always possible for the probe to fit features that correspond to something merely correlated with truth. We will update the paragraph on line 135 to mention that observing what we call “deception” is necessary but not sufficient for the model to be ‘believing’ one thing but actively choosing to say the other. \n\n### (3) Re: “Probing can overestimate the information captured by hidden representations”\n\nWe are not perfectly sure what overestimating means in this context, but we agree that probes can fit features even if the model does not use them. We note this in footnote 3 and by citing Ravichander et al. (2020). We think we can better discuss this limitation in the discussion alongside the discussion we add involving only using linear probes. \n\n### (4) Re: Belinkov (2022)\n\nThank you! We will discuss this alongside Ravichander et al. (2020) and Elazar et al. (2021).\n\n### (5) Re: “the probing is assumed to signal the truthfulness of the system”\n\nThis is not an assumption we want or need to make. Note that we also report instances of probe errors and probe confabulation in Figure 1 and Figure 3. However, we do consistently observe that the probe does better than the queries.\n\n### (6) Azaria and Mitchell (2023) do not mention deception\n\nWe refer to them on line 135 because the title of the paper describes language models as “lying”. Our use of “deception” also stems from literature also including [Burns et al. (2022)](https://arxiv.org/abs/2212.03827) who claim to “discover _latent_ knowledge”, [Li et al. (2023)](https://arxiv.org/abs/2306.03341) who speculate that their findings may be due to RLHF-induced deception, and [Christiano et al. (2021)](https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit) who propose probing as a solution to active deception from AI systems. \n\nThe region in Figure 1c that we refer to as “deception” is intended to be a type of observation that is necessary if there is indeed something happening like what these papers speculate may be. However, we do not intend to claim it is sufficient. We think this is important to be clear about, and we will clarify this in the paragraph on line 153. Thanks!\n\nOne of the major motivations of our work is to reply to past works’ characterizations of what probing finds to show that the picture is more nuanced than the idea that language models sometimes “lie” or otherwise encode non-cooperative communicative intents.\n\n### (6) Re: Question 1 – line 137 typo. \n\nYes. Thank you, we will fix this. \n\n### (7) Re: Question 2 – Figure 3 top right.\n\nThe top right is a key and not a missing plot. We tried labeling each plot individually, but found this to be cumbersome. We will label this as a “key”, and are open to suggestions for improving the plot in your final comments. Thanks!\n\n### (8) Re: Question 3 – How accurately can linear probes indicate what information is captured in the latent space?\n\nWe agree and discuss in footnote 3 that a successful probe does not imply that the information being probed for is used by the model; an unsuccessful probe does not imply that the model does not represent that information. We agree this makes probing challenging in general, and as mentioned above, we will stress this again when we modify the discussion section. \n\nHowever, independent of any characterization of what causes a probe’s successes or failures, the fact that they typically perform better than queries is of practical interest (e.g. for anomaly detection). For this reason, we think that our experiments are a useful approach. \n\nFinally, see our discussion above about the relationship between our definition of “deception” and more specific ones that have been used in prior works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents an appealing categorisation system that can be employed to compare and contrast predictions that were made internally and externally. Future work could benefit from adopting this terminology in the debate about the faithfulness of LLMs' predictions. \n- The paper presents a solid set of results in support of its main claim: that disagreements between probes and queries are not usually deceptions (for these datasets and models). This is an interesting piece of evidence that can contribute to the community's discussion about the reliability of LLMs.\n- In addition, the paper reproduces results from prior work, and presents a simple method to get more reliable predictions from a model.", "reasons_to_reject": "- [Major] There is very limited experimentation with the setup and hyperparameters: only one type of probing, one single prompt and one type of task is used. Particularly in the case of probing, this does not suffice to determine whether the system is really \"deceiving\" the user, when taking into account the literature on how probing can overestimate the information captured by hidden representations, which the authors do not touch upon. (The only concern that is mentioned is that these probes may not be able to capture non-linear patterns, see Limitations.) Belinkov provides a useful overview of issues of linear probing, and points out some solutions in \"Probing Classifiers: Promises, Shortcomings, and Advances\". Why is this relevant? The authors of this paper compare zero-shot generalisation via prompting (as the external signal) to the probed predictions (as the internal signal), where the probing is assumed to signal the truthfulness of the system. A stark deviation between the former and the latter is even referred to as \"deception\", which seems inaccurate when not even considering that the probes might be overestimating what the LLMs actually capture.\n- [Minor] On the one hand, the article claims that it does not want to anthropomorphize LLMs (see footnote 1). On the other hand, the terminology used is not entirely in line with this statement. Azaria and Mitchell do not mention \"deception\", this appears to be a choice by the authors. In addition, referring to the disagreement between probes and queries as \"cognitive dissonance\" (in the title) is somewhat confusing.", "questions_to_authors": "1. Looking at figure 1C, and reasoning about what deceptions are, shouldn’t the description of “Deception” say “queries are INcorrect”? (line 137)\n2. What was supposed to be represented in Figure 3, top right? That subfigure appears empty.\n3. Could you comment on the extent to which linear probes can accurately indicate what information is captured in the latent space? It seems somewhat unfair to contrast prompting to linear probing (without further controls/baselines) and argue that a high p(probe is correct) but a low p(query is correct) equals \"deception\".", "paper_topic_and_main_contributions": "The paper discusses the mismatch that exists between models’ external predictions and the predictions that can be obtained via probing internal representations of the same models. The two answers do not always agree, and the goal of this paper is to understand these mismatches by looking at predictions as distributions rather than categorical answers. Three reasons for mismatches are identified (confabulation, heterogeneity and deception). Finally, it is shown that by combining both internal and external predictions, we can obtain more accurate predictions overall. With its story, the paper combats the view that LMs try to “deceive” by performing predictions that are not in line with their internal representations.\n\nThe three causes for disagreements between probes and queries discussed are:\n1. Confabulation: low probe confidence and incorrect queries\n2. Deception: high probe and query confidence, and queries are correct (is this correct? see my question below)\n3. Heterogeneity: disagreements between probes, where there are specific input subsets on which probing or querying is better. Probes can be better overall if their subset happens to be larger.\nAll three are the result of looking at the disagreement as a graded phenomenon (see Figure 1C, which is very insightful/helpful!)\n\nTo understand the frequency of each error type, three binary QA datasets are used to query GPT2-XL and GPT-J. The probes trained are linear probes. The findings are as follows:\n- Reflecting prior work, the probes are indeed more accurate.\n- Reflecting prior work, the probes’ confidence better aligns with their accuracy compared to the models' external predictions (i.e. their calibration is better).\n- Then there is a novel finding: The most frequent type of disagreement is due to heterogeneity, followed by model confabulations. Deceptions are quite rare, they are only observed for one dataset.\n- Finally, an ensembling method in which predictions of the probes and queries are combined, leads to improvements on 4/6 experimental setups.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "To elaborate on probing and its consequences in the paper, the following articles can be a good start:\n- Belinkov, Y. (2022). Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1), 207-219.\n- Hewitt, J., & Liang, P. (2019). Designing and Interpreting Probes with Control Tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics.\n- Voita, E., & Titov, I. (2020, November). Information-Theoretic Probing with Minimum Description Length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 183-196).", "Typos_Grammar_Style_and_Presentation_Improvements": "- line 124: Edwards 2023 is only a blog, correct, not a (peer-reviewed) research article? Perhaps it's more appropriate to comment on that in a footnote, indicating that the term confabulation was coined there.\n- You are missing information on reproducibility, such as the architectures used for your experiments, the GPU usage and running times, along with hyperparameter setups used for the various experiments."}}, {"review_id": "3qo6YLn4lx", "reviewer": "Reviewer_WxBn", "summary": "", "questions": "- What is the $h_{LM}$ in L116?\n- Why are there missing values for the ensemble row in Table 1?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We appreciate your time and feedback! \n\n### (1) Re: “They identify three reasons that probes may outperform queries. I think these are different categorizations of mismatched cases, which can not be treated as the reasons.”\n\nWe agree with your perspective here that they are only observations – not reasons. We did not intend to communicate otherwise. We will modify lines 79-81, lines 124-153, and footnote 1 to make it unambiguously clear that we intend to describe observations and not speculate causally about reasons why the probe and model think differently in different cases. \n\n### (2) Re: “we are not sure whether the mismatching patterns discovered in experiments are general or specific to the language models, the probing classifiers, or the prompt template.”\n\nThank you for mentioning this. We generally agree, and we think that more experiments would be better. However, we stress that the central claims that we make in the paper are all based on what results happen across both models and all 3 datasets. We also note the 4-page limit and the general scope of EMNLP papers allow for narrow work compared to other venues. \n\nWe have tried experiments with MLP probes, but have not found substantially different results. The probe performance has been comparable to linear probes in our experiments, and we have not noticed any qualitative changes regarding the main findings. We also note that the prior works on truthfulness in LLMs that largely inspired this work (e.g. Burns et al) also use only linear probes. We will mention this in the discussion. \n\nWe have tried multiple prompting strategies, and both queries and probes are sensitive to these, but fluctuations have only been a few percentage points, and we have not found any qualitative changes regarding our main findings. We will mention this on line 170.\n\n### (3) Re: What is h_{LM}?\n\nWe mention on line 105 that it stands for a hidden latent representation inside of an LM. \n\n### (4) Re: Grammar and style\n\nThanks! We will make both of these changes. We appreciate that you read closely and suggested them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work provides a taxonomy of the mismatched cases for the language model's factual assertions and probing results, which can serve as an analysis baseline for future research.", "reasons_to_reject": "- This work overstates its contribution\n- The patterns found in this work are not conclusive.", "questions_to_authors": "- What is the $h_{LM}$ in L116?\n- Why are there missing values for the ensemble row in Table 1?", "paper_topic_and_main_contributions": "This paper focuses on analyzing the reasons behind the mismatch between the language model's factual assertions and the probing results from the internal representations. The authors first divide the mismatch into different categories based on the predicted probability of these two methods. Next, experiments are conducted using three different datasets to show what fraction of predictions correspond to these categories.\n\nAlthough it is nice to have a taxonomy for the mismatches, I think this paper is overstating its contributions. For example, in L79-81,  the authors state that they identify three reasons that probes may outperform queries.  I think these are different categorizations of mismatched cases,  which can not be treated as the reasons. They are indeed observations.\nAnother big limitation of this work is, also suggested by the authors, we are not sure whether the mismatching patterns discovered in experiments are general or specific to the language models, the probing classifiers, or the prompt template. Without more experiments, we can not conclude anything.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Need to state this is a binary classification problem earlier. I did not realize this is a binary classification problem until the description of the datasets.\n- In Figure 3, the color for \"both correct\" and \"both incorrect\" are not distinguishable enough. It would be better to have completely different colors."}}]

Author rebuttal:

Meta-review: This paper studies the difference between a models behaviour and query representation. They highlight the difference between these two paradims and create a taxonomy of clases of disagreement between the two methods. 

Both aRIY and M8PW praise the work, stating that the problem is interesting, the taxonomy is valuable and there are substantial empirical results, especially for a short paper. M8PW does have some qualms with the presentation, for which they made ample suggestions. The author response makes me (as well as the reviewer) confident that the authors will improve this for the camera ready version of the paper. M8PW furhermore points out that there is little hyperparameter search on the probe side, I believe the authors succesfully rebutted this in their response. The third reviewer, WxBN, is less enthusiastic about the work, but provides no arguments for their main stated weaknesses. Given the fact that the other reviews are more elaborate and interactive with the authors, I don't think the lower scores of WxBN should stand in the way of publication at EMNLP.
Decision: Accept-Main