LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations
Authors: Hadas Orgad
Michael Toker
Zorik Gekhman
Roi Reichart
Idan Szpektor
Hadas Kotek
Yonatan Belinkov
Venue: ICLR 2025 Poster (2025)
Keywords: hallucinations
truthfulness
interpretability
probing
LLM
Primary area: interpretability and explainable AI
TL;DR: We discover that LLMs encode much more useful information on truthfulness than previously recognized, though this information is more complex and multifaceted than expected.

Abstract: Large language models (LLMs) often produce errors, including factual inaccuracies, biases, and reasoning failures, collectively referred to as "hallucinations". Recent studies have demonstrated that LLMs' internal states encode information regarding the truthfulness of their outputs, and that this information can be utilized to detect errors. In this work, we show that the internal representations of LLMs encode much more information about truthfulness than previously recognized. We first discover that the truthfulness information is concentrated in specific tokens, and leveraging this property significantly enhances error detection performance. Yet, we show that such error detectors fail to generalize across datasets, implying that---contrary to prior claims---truthfulness encoding is not universal but rather multifaceted. Next, we show that internal representations can also be used for predicting the types of errors the model is likely to make, facilitating the development of tailored mitigation strategies. Lastly, we reveal a discrepancy between LLMs' internal encoding and external behavior: they may encode the correct answer, yet consistently generate an incorrect one. Taken together, these insights deepen our understanding of LLM errors from the model's internal perspective, which can guide future research on enhancing error analysis and mitigation.

Published as a conference paper at ICLR 2025
LLMs KNOW MORE THAN THEY SHOW: ON THE IN- TRINSIC REPRESENTATION OF LLM HALLUCINATIONS
Hadas Orgad"* 1* Michael Toker
Zorik Gekhman
Roi Reichart
Idan Szpektor
Hadas Kotek
Yonatan Belinkov!
Technion Google Research
Apple
ABSTRACT
Large language models (LLMs) often produce errors, including factual inaccura- cies, biases, and reasoning failures, collectively referred to as "hallucinations" Recent studies have demonstrated that LLMMs' internal states encode information regarding the truthfulness of their outputs, and that this information can be utilized detect errors. In this work, we show that the internal representations of LLMs encode much more information about truthfulness than previously recognized. We first discover that the truthfulness information is concentrated in specific tokens, and leveraging this property significantly enhances error detection performance. Yet, we show that such error detectors fail to generalize across datasets, implying that-contrary to prior truimu-tuttfulness. encoding is not universal but rather multifaceted. Next, we show that internal representations can also be used for pre- dicting the types of errors the model is likely to make, facilitating the development of tailored mitigation strategies. Lastly, we reveal a discrepancy between L.LMs' internal encoding and external behavior: they may encode the correct answer, yet consistently generate an incorrect one. Taken together, these insights deepen our understanding of LLM errors from the model's internal perspective, which can guide future research on enhancing error analysis and mitigation]
INTRODUCTION
The ever-growing popularity of large language models (LLM) across many domains has brought a significant limitation to center stage: their tendency to "hallucinate" which is often used to de- scribe the generation of inaccurate information. But what are hallucinations, and what causes them? A considerable body of research has sought define, taxonomize, and understand hallucinations through extrinsic, behavioral analysis, primarily examining how users perceive such errors (Bang al., [023] Ji et al.] [2033 Huang et al. [023a Rawte et al. (023). However, this approach does not adequately address how these errors are encoded within the LLMs. Alternatively, another line of work has explored the internal representations of LLMs, suggesting that LLMs encode signals of truthfulness (Kadavath et al., 2022 Li et al.) 2024, Chen et al. 2024 inter alia). However, these analyses were typically restricted detecting errors determining whether generated output contains inaccuracies-winnn without delving deeper into how such signals are represented and could be leveraged to understand or mitigate hallucinations.
In this work, we reveal that the internal representations of LLMs encode much more information about truthfulness than previously recognized. Through a series of experiments, we train classifiers on these internal representations to predict various features related to the truthfulness of generated outputs. Our findings reveal the patterns and types of information encoded in model representations, linking this intrinsic data to extrinsic LLM behavior. This enhances (while understanding the limitations of error detection), and may guide the development of more nuanced strategies based
ability to detect errors
types and mitigation methods that make use of the model's internal knowledge. Our experiments are designed to be general, covering a broad array of L.LM limitations. While the term "hallucinations" widely used, lacks a universally accepted definition (Venkit et al. 2024). Our framework adopts a broad interpretation, considering hallucinations to encompass
"Corresponding author; Work partially done during internship Apple.
OOr code is available in httph: http://ttw.....c.oe/oooooooooooa
Know

---

Published as a conference paper at ICLR 2025
all errors produced by LLM, including factual inaceuracies, biases, common- sense reasoning failures, and other woaldworld errors. This approach enables us to draw general conclusions about model errors from broad perspective.
Our first step is identifying where truthfulness signals are encoded in LLMs. Previous studies have suggested methods for detecting errors in LLM outputs using intermediate representations, logits, or probabilities, implying that LLMs may encode signals of truthfulness (Kadavath et al.] [022 Li al. 2024; Chen al. 2024). Focusing on long-form generations, which reflect real-world usage of LLMs, our analysis uncovers a key oversight: the choice of token used to extract these signals (S We find that truthfulness information is concentrated in the exact answer tokens e.g., "Hartford" in "The capital of Connecticut is Hartford, an iconic city..... . Recognizing this nu- ance significantly improves error detection strategies across the board, revealing that truthfulness encoding is stronger than previously observed.
From this point forward, we concentrate on our most effective strategy: a classifier trained on in- termediate LLM representations within the exact answer tokens, referred to 'probing classifiers' (Belinkov 202I). This approach helps us explore what these representations reveal about LLMs. Our demonstration that trained probing classifier can predict errors suggests that LLMs encode information related to their own truthfulness. However, we find that probing classifiers do not gen- eralize across different tasks (Section 4). Generalization occurs only within tasks requiring similar skills (e.g.. factual retrieval), indicating the truthfulness information is "skill-specifie" and varies across different tasks. For tasks involving different skills, e.g., sentiment analysis, these classifiers are no better-or worse-than logit-based uncertainty predictors, challenging the idea of a "universal truthfulness" encoding proposed in previous work (Marks & Tegmark 2023 Slobodkin al. 2023). Instead, our results indicate that LLMs encode multiple, distinct notions of truth. Thus, deploying trainable error detectors in practical applications should be undertaken with caution.
We next find evidence that LLMs encode not only error detection signals but also more nuanced information about error types. Delving deeper into errors within a single task, we taxonomize its errors based on responses across repeated samples (Section 5). For example, the same error being consistently generated different from an error that is generated occasionally among many other distinct errors. Using different set of probing classifiers, we find that error types are predictable from the LLM representations, drawing a connection between the models's internal representations and its external behavior. This classification offers a more nuanced understanding of errors, enabling developers to predict error patterns and implement more targeted mitigation strategies.
Finally, we find that the truthfulness signals encoded in LLMs can also differentiate between correct and incorrect answers for the same question (Section 6 Results highlight a significant misalign- ment between LLM''s internal representations and its external behavior in some cases. The model's internal encoding may identify the correct answer-yet it frequently generates an incorrect re- sponse. This discrepancy reveals that the LLM's external behavior may misrepresent its abilities, potentially pointing to new strategies for reducing errors by utilizing its existing strengths. Overall, our model- framework provides a deeper understanding of LLM errors, suggesting potential directions for improvements in error analysis and mitigation.
BACKGROUND
Defining and characterizing LLM errors. The term "hallucinations" is widely used across var- ious subfields such as conversational AI (Liu al. 2022), abstractive summarization (Zhang al.) 2019). and machine translation (Wang & Sennrich 2020), each interpreting the term differently. Yet, no consensus exists on defining hallucinations: Venkit al. (2024) identified 31 distinct frame- works for conceptualizing hallucinations, revealing the diversity of perspectives. Research efforts aim to define and taxonomize hallucinations, distinguishing them from other error types (Liu et al., 2022 Ji et al. 2023 Huang et al., 2023a Rawte et al., 2023). On the other hand, recent scholarly conversations introduce terms like "confabulations" (Millidge 2023) and "fabrications" (McGowan ct al. 2023), attributing a possible "intention" to LLMs, although the notions of LLM "intention" and other human- traits are still debated (Salles et al. 2020 Serapio-Gane Gariio et al.) 2023 Harnad 2024). These categorizations, however, adopt human-centric view by focusing on the subjec- tive interpretations of LLM hallucinations, which does not necessarily reflect how these errors are encoded within the models themselves. This gap limits our ability address the root causes of
2

---

Published as a conference paper at ICLR 2025
hallucinations, to reason about their nature. For example, it is unclear whether conclusions about hallucinations defined in one framework can be applied to another framework. Liang et al. (2024) defined hallucinations inconsistencies with the training data. While this approach engage with the possible root causes of hallucinations, our study focuses on insights from the model itself, with- requiring training data access. Instead, we adopt broad interpretation of hallucinations. Here, we define hallucinations as any type of error generated by an LLM, including factual inaccuracies, biases, failures in common-senee reasoning, and others.
Another line of research suggests that LLMs either encode information about their own errors (Ka- davath et al., 2022] Azaria & Mitchell, 2023) or exhibit discrepancies between their outputs and internal representations (Liu et al.) 2023 Gottesman & Geval (2024), indicating the presence of un- derlying mechanisms not reflected in their final outputs. Moreover, Yona et al. (2024) found that current LLMs fail to effectively convey their uncertainty through their generated outputs. Hence, we propose shifting the focus from human-centric interpretations of hallucinations to a model-centric perspective, examining the model's intermediate activations.
Error detection in LLMs. Error detection is a longstanding task in NLP, crucial for maintaining high standards in various practical applications and for constructing more reliable systems that en- sure user trust (Bommasani et al.) 2021). Over the years, many studies have proposed task-specific solutions (see Section A.I). However, the recent shift towards general- LLMs necessitates a holistic approach capable of addressing any error ty rather than focusing specific ones, making it suitable for the diverse errors generated by these models.
A line of work has addressed this challenge by loveraging external knowledge sources (Lewis al. 2020 Gao et al., 2023) or an external L.LM judge (Lin et al.) 2021 Rawte et al. 2023) to identify erroneous outputs. On the other hand, our work focuses on detection methods that rely solely on the computations of the A.L.-Specffien
output logits, probabilities after softmax, and hidden states.
Error detection in LLMs is also closely linked to uncertainty estimation, where low certainty signals potential inaccuracies and possible errors. Popular methods to derive calibrated confidence include
inspecting the model logit output values (Varshney
al.. 2023 Taubenfeld et al. 2025). agree-
ment across multiple sampled answers (Kuhn et al. [202]] Manakul et al. [203]] Tian et al. 2023a),
verbalized probability (Tian et al. 2023b). and direct prompting (Kadavath et al. 2022).
Another line of work trains probing classifiers to discover and utilize truthfulness features. This ap- proach has shown some success by probing the final token of an answer either generated (Kadavath
et al.
2022,
[Snyder et al., 2023 Yuksekgonul et al.] 2023 Zou et al. [203]] Yin et al.] [024]] Chen
et al.
2024 Simhi et al.] 2024; Gekhman et al. 2025) or not (Li et al. 2024] Marks & Tegmark
2023
Burns et al. |022] Azaria & Mitchell 2023 Rateike et al., 2023). Others probe the final token
of the prompt before the response is generated (Slobodkin et al.) 2023 Snyder et al., 2023; Simhi jet al. 2024| Gottesman & Geval 2024). Many previous studies simplify the analysis by generating answers in a few-shot setting or limiting generation to a single token. In contrast, we simulate real- world usage of LLMs by allowing unrestricted answer generation. By probing exact answer tokens, we achieve significant improvements in error detection.
3 BETTER ERROR DETECTION
This section presents experiments on detecting LLM errors through their own computations, focusing on token selection's 's impact and introducing a method that outperforms other approaches.
3.1 TASK DEFINITION
Given an LLM M, an input prompt and the I.MM-generated response ÿ. the task is to predict whether ý is correct wrong. We assume that there is access to the L.MM's internal states (i.e., white-box setting), but no access to any external resources (c.g., search engine or additional LLMs).
We use dataset D {(g,, M)) consisting of N question-label pairs, where {g: represents series of questions (e.g.. "What is the capital of Connecticut?"" and { the corresponding ground-truth answers (" ("Hartford"). For each question qt, we prompt the model M to generate a response yi, resulting in the set of predicted answers (ū. J ("The capital of Connecticut is Hart-
3

---

Published as a conference paper at ICLR 2025
ford...). Next, to build our error-detection dataset, evaluate the correctness of each generated response y; by comparing it to the ground-truthh label y4. This comparison yields a correctness label 2, E {0 (1 correct, 0 wrong). The comparison be done either via automatic heuristics with the assistance of an instruct-LLMM Our error detection dataset is: {(g: Dr. ) Note that this dataset is defined based the analyzed L.M and its generated answers. Any instances where the LLM refuses to answer are excluded, as these can easily be classified as incorrect.
3.2 EXPERIMENTAL SETUP
Datasets and models. We perform all experiments on four L.MSs: Mistral-7h (Jiang et al, 2023). Mistral-7b-instruot
(denoted Mistral-7b-inntrueet) Llama3-8b (Touvron et al. 2023) 3) and Llama3-Sb-instruct. We consider 10 different datasets spanning various domains and tasks: Triv- iaQA (Joshi et al., 2017). HotpotQA with/without context (Yang et al.) 2018), Natural Questions Kwiatkowski et al. 2019), Winobias (Zhao et al. 2018), Winogrande (Sakaguchi et al., (021), MNLI SANL(Williee (Williams et al. 2018). Math (Sun al.) 2024), IMDB review sentiment analysis (Maas) et al. 2011), and a dataset of movie roles (movies) that we curate. We allow unrestricted response generation to mimic real-world LLM usage, with answers decoded greedily. For more details on the datasets and the prompts used to generate answers, refer to Appendix A.3
Performance metric. We measure the area under the ROC curve to evaluate error detectors, pro- viding single metric that reflects their ability to distinguish between positive and negative cases across many thresholds, balancing sensitivity (true positive rate) and specificity (false positive rate).
Error detection methods. We compare methods from both uncertainty and hallucinations literature.
Aggregated probabilities / logits: Previous studies (Guerreiro et al. 2023 Kadavath et al., 2022 Varshney et al. 2023 Huang al.) 2023b) aggregate output token probabilities or logits to score LLM confidence for error detection. We implement several methods from the literature, calculating the minimum, maximum, or mean of these values. The main paper reports results for the most common approach, Logits-mean, and the best-performing one, Logits-min, with additional baselines in Appendix B
P(True):
Kadavath Kadavvthh et al.) (2022) showed that LLMs are relatively calibrated when asked to evaluate the correctness of their generation via prompting. We implement this evaluation
using the same prompt.
Probing: Probing classifiers involve training small classifier on model's intermediate activations to predict features of processed text (Belinkov) (021). Recent studies show their effectiveness for error detection in generated text (Kadavath et al., 2022 inter alia). An intermediate activation is a vector hi, from a specific LLM layer / and (either read or generated) token f. Thus, each LLM generation produces multiple such activations, Fol- lowing prior work, we use linear probing classifier for error detection alia) on static tokens: the last generated token (hy,-1). the one before it (hz,-2), and the final prompt token (hr,)). The layer is selected per token based on validation set performance.
[024]] inter
For further details on the implementation of each method, refer to Appendix A.4
the token selection for error detection, typically focusing on the last generated token taking mean. However, since LLMs typically generate long-form responses, this practice may miss crucial details (Brunner et al., 2020 Other approaches use the last token of the prompt (Slobodkin et al., 2023. inter alia). but this is inherently inaccurate due to LLMs' unidirectional nature, failing to account for the generated response and missing cases where different sampled answers from the same model vary in cor- rectness. We investigate a previously unexamined token location: the exact answer tokens, which represent the most meaningful parts of the generated response. We define exact answer tokens as those whose modification alters the answer's correctness, disregarding subsequent generated con- tent Figure illustrates the different token locations. the following experiments, we implement
Exact Answer Tokens. Existing methods often overlook a critical nuance:
For most datasets, we use heuristics to predict correctness, except for one case. See Appendix A.2
In practice, we do not use this definition for extracting the exact answer, but rather instruct model in few-shot setting. Still, the definition is useful manually verify that automatic extractions work as expected.
4

---

Published as a conference paper at ICLR 2025
Prompt
last. token

 
 What is the capital the U.S. state of Connecticut?
www.attaanswe..cee
@ast_exact_eet crrtttannsweeraa Loken
The capital city of the U.S. state of Connecticut is Hartffrd. It's one of the oldest cities in the United States and was founded in 1635. Hartford is located in the central part of the state and is home to several cultural institutions, universities, and businesses, 

Mistral
-2 -1
Figure 1: Example for the input and LLM output from the TriviaQA dataset, and the names of the tokens that can be probed.
(a) TriviaQA
(b) Winobias
(c) Math
Figure 2: AUC values of probe error detector layers and tokens, disstrl-7bbinsstrce Gen-
eration proceeds from left to right, with detection performance peaking at the exact answer tokens.
each detection method with "exact answer" version, demonstrating that it often improves performance, especially in probing. Implementation details for detecting the exact answer token are given Appendix A.2
3.3 RESULTS
Patterns of truthfulness encoding. We first focus on probing classifiers to gain insights into the internal representations of L.MMs. Specifically, we analyze the effects of layer and token selection on the error detection performance of these probing classifiers. By systematically probing all model layers, starting from the last question token to the final generated token, we observe consistent truthfulness encoding patterns. Figure shows AUC metrics of probes across Mistral-7b-Instruu layers and tokens. Middle to later layers often yield the most effective probing results (see Appendix B for more datasets and models), aligning with previous studies truthfulness encoding (Burns
et al. 2022 CH-Wang et al. 2023) and transformer representations (nostalgebraist 2020 Meng
al. 2022 Geva al. 2023). Regarding tokens, strong truthfulness signal appears immediately after the prompt, suggesting that this representation encodes information on the model's general ability answer the question correctly. This signal weakens as text generation progresses but peaks again at the exact answer tokens. Towards the end of the generation process, signal strength rises again, though remains weaker than the exact answer tokens. These patterns are consistent across nearly all datasets and models (see Appendix B}. suggesting a general mechanism by which LLMS encode and process truthfulness during text generation.
5

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of error detection techniques using AUC metric, across different models and datasets. The best- rest-performing method is bolded. Using exact answer tokens is useful for many cases, especially probing.
Mistral-7b-Instrucl
Llama 3- 3-8b-Instruct
TriviaQA
Winobias
Math
TriviaQA
Winobias
Math
Logits-mean
0.60 008
0.56 =0.017
0.55
0.66 0000
0.60 =0.028
0.75 20.018
Logits-mean-exae
0,68 +0.007
0.54 =0.012
0.51 40
0.71 +0.000
0.55
#0.019
0.80
Logits-min
0.63 =0 o0e
0.59 20.012
0.51
20.017
0.74 =0 007
0.61
20.024
0.75 20.016
Logits-min-exxs
0.75 +00 006
0.53 =0.013
0.71 +0.009
0.79 +00 .006
0.61
#0.019
0.89 +0.018
P(True)
0.66 2:0 o0e
0.45 20.021
0.48
2o o22
0.73 2: oos
0.59
20.020
0.62 20.011
P(True)-exact
0.74 +00.008 ooa
0.40
=0.021
0.60
+
0.73 +:0 0.006
0.63
#0.014
0.59 #0.018
Probe @ token
Last generated [-1]
0.71 =0 006
0.82 20.005
0.74
20 008
0.81 =: 005
0.86 20.007
0.82 200016
Before last generated [-2]
0.73 +0.00 004
0.85
=0.004
0.74
+0.007
0.75
+0 .006
0.88 #0.005
0.79 +0.020
End of question
0.76 00e
0.82
20.011
0.72
2:
0.77
007
0.80 20.018
0.72 2:0
Exact
0.85 200000
0.92 10.003
0.92 0.00s
0.83 2000002
0.93 20.004
0.95 200007
Error Detection Results. Next, we evaluate various error detection methods by comparing their performance with and without the use of exact answer tokens. Table compares the AUC across three representative datasets (additional datasets and models in Appendix B showing consistent pat- terns). Here we present results for the last exact answer token, which outperformed both the first exact answer token and the one preceding it, while the token following the last performed similarly. Incorporating the exact answer token improves the different error detection methods in almost all datasets. Notably, our probing technique (bottom line) consistently outperforms all other baselines across the board. While we did not compare all existing eлror detection methods, the primary con- clusion is that information about truthfulness highly localized in specific generated tokens, and that focusing on exact answer tokens leads to significant improvements in error detection.
4 GENERALIZATION BETWEEN TASKS
The effectiveness of a probing classifier in detecting errors suggests that LLMs encode information about the truthfulness of their outputs. This supports using probing classifiers for error detection in production, but their generalizability across tasks remains unclear. While some studies argue for a universal mechanism of truthfulness encoding in LLMs (Marks & Tegmark, 2023 Slobod- kin et al., 2023), results on probe generalization across datasets are mixed (Kadavath et al.) 2022 Marks & Tegmark 2023 CH-Wang et al.) 2023, Slobodkin et al.) 2023] Levinstein & Herrmann 2024)-observing a decline in performance, yet it remains significantly above random chance. Un- derstanding this is essential for real-world applications, where the error detector may encounter examples that significantly differ from those it trained on. Therefore, we explore whether probe trained on one dataset can detect errors in others.
Our generalization experiments conducted between all of the ten datasets discussed in Section covering a broader range of reaslistic task settings than previous work. This breadth of experiments has not been previously explored, and is crucial considering the mixed findings in previous work. We select the optimal token and layer combination for each dataset, train all probes using this com- bination on other datasets, and then test them on the original dataset. We evaluate generalization performance using the absolute AUC score, defined as max(auc, 1 - auc), to also account for cases where the learned signal in one dataset is reversed in another.
Results. Figure 3a shows the generalization results for Distral-7b-instruse with similar patterns observed for other LLMs in Appendix C In this context, values above 0.5 indicate successful gener- alization. At first glance, the results appear consistent with previous research: most heatmap values exceed 0.5, implying some degree of generalization across tasks. This observation supports the exis- tence of a universal mechanism for decoding truthfulness, since the same linear directions captured by the probe-encode encode truthfulness information across many datasets. However, upon closer inspec- tion, it turns out that most of this performance can be achieved by logit-based truthfulness detection,

---

Published as a conference paper at ICLR 2025
Trviig4
5.56
0.72
0.70
0.55
0.59
0.70
TIvbON
0:01
4.05 0.01
0.01
4.01
0.03
4.19
0.30
.0.00
002
4002400A
0.80
0.85
0.78
961
0.58
0.53
0:7
0.67
0.62
0.65
0.8
HATDORON
0.01
062
0.03
Movics
0.03 66.00
0.08 0.05
0.08
0.30
0.82
02
Train dataset
Winobiia
D.54 0059
0.2
0:2
0.
055
0.11
0.82
Train dataset
Winettia
4038
0.22
033
Winogrande-
0.00
0.00
0.5
841
0.64
0.66
0.71
0.61
0.51
0.55
Wengrande
15
4.17
-0.17
007
A3
0:24
0.21
6.19
0.3
000
0.54
0.21
9.07
0.39
0.43
00
MDB-000 0.54
0.66
962
0.62
0.60
3:7
0.60
0.51
0.58
2.56 0.5 I.80 0:7 0.51
0.52
0.33
Noth 0.16
4.22
0.02
4.10
0.02
0.4
0.22
0:10
rotporte_
0.04
40, 10,wC- -000 0.05 0.c0 0.74
0.67
0.50
0.2
0.53
0.07
WCC
-0.07
60.11
0.57
0.04 ace
0.03
.0.4
0.0
Noner
Math:
Hodeed WC
N80 WC
Hovies
ICB
Wes
wesso WC
Test dataset
Test dataset
(a) Raw AUC values. Values above 0.5 indicate some generalization.
(b) Performance (AUC) difference of the probe and the logit-based method. Values above indicate gen- eralization beyond the logit-based method.
Figure 3: Generalization between datasets, Mistral-7b 7i-iial--7-inttut.... After subtracting the logit-based method's performance, we observe that most datasets show limited or no meaningful generalization.
which only observes the output logits. Figure presents the same heatmap after subtracting results from our strongest logit-based baseline Mlogitmmin-xxact This adjusted heatmap reveals the probe's s generalization rarely exceeds what can be achieved by examining logits alone. This suggests that the observed generalization is not due to universal internal encoding of truthfulness. Instead, it likely arises from information already available through external features, such logits. Past evidence for generalization may therefore have been overstated.
Nonetheless, we do observe some successful generalization in tasks requiring similar skills, such as parametric factual retrieval (TriviaQA, HotpotQA, Movies) and common-sense reasoning (Wino- bias, Wingrande, NLI). This suggests that, although the overall pattern of truthfulness signals across tokens appeared consistent across tasks (as observed in Section 3.3). LLMs have many "skill- specific" truthfulness mechanisms rather than universal ones. However, some patterns remain unex- plained, such as the asymmetric generalization from TriviaQA to Math tasks. Overall, our findings indicate that models have a multifaceted representation of truthfulness. The internal mechanisms responsible for solving distinct problem are implemented different mechanisms (e.g.. circuits) within models (Elhage et al. 202T; Olah et al. 2023). Similarly, LLMS do not encode truthfulness through a single unified mechanism but rather through multiple mechanisms, each corresponding to different notions of truth. Further investigation is required to disentangle these mechanisms.
INVESTIGATING ERROR TYPES
Having established the limitations of error detection, we now shift to error analysis. Previously, we explored types of LLM limitations across different tasks, noting both commonalities and distinctions in their error representations. In this section, we focus on the types of errors LLMs make in specific task QiiiaQQAwhccl represents factual errors, commonly studied issue in LLMs (Kadavath
et al.
2022
Snyder et al. 2023 Li et al. 2024 Chen et al. 2024 Simhi et al. 2024).
5.1 TAXONOMY OF ERRORS
Intuitively, not all mistakes are identical. In one case, an L.LM may consistently generate an incorrect answer, considering it correct, while in another case, it could issue best guess. To analyze erros from the LLM's perspective, we sample K = 30 responses at temperature setting of T' for each example in the dataset and then analyze the resulting distribution of answers.
'We chose K = 30 the overall correctness seemed to plateau around this point; see Appendix D We found that lower temperatures generally produced less truthful answers across repeated trials.

---

Published as a conference paper at ICLR 2025
Beroo HER
Who
X
Cabby Lagan
Ose Batt 18
knownits
Wiee Amettoun other stareT
honder
The Bint
X
Cerinl Americ
undegrounddieers
109
FTe... Parte.
FSE HOTTIN
(a) The LLM mostly answers cor- rectly. but sometimes hallucinates.
(b) The LLM mostly answers in- correctly, but seems to have some knowledge on the correct answer.
(c) The LLM generates many dif- ferent answers, one them is the correct one which is generated a small fraction of the resamples.
Figure 4: Different error types in free-form generation, exposed when resampled many times.
Figure 4 illusttrats three representative error types. In one (Figure 4a the model usually gives the correct answer but occasionally make an error, implying correct information is present but sampling may lead to mistakes. In another (Figure 4b) the model often responds incorrectly, though it is capa- ble of providing the right answer, indicating some retained knowledge despite consistently making the same error. In a third type (Figure 4g the model generates wide array of mostly incorrect answers, reflecting low confidence in any generated answer.
More generally. we categorize the errors by logging three specific features for each example: (a) the number of different answers generated; (b) the frequency of the correct answer, and (c) the frequency of the most common incorrect answer. These features reveal the following error patterns:
(A) Refuses to answer: The model responds that it cannot answer the question in at least half the cases.
(B) Consistently correct: Answers correctly in at least half of the cases. This category is divided into: (B1) always correct; and (B2) mostly correct with occasional errors.
(C) Consistently incorrect: Consistently generates the same incorrect response in least half of the cases. Similarly to type B, we subdivide this type into (C1) correct answer is never produced; and (C2) correct answer appears at least once.
(D) Two competing: Generates both correct and incorrect responses at similar rates- difference in rates is 5 or less, and each response is generated at least times.
(E) Many answers: Generates over 10 distinet answers. Like types C and D, Subtypes include (EI) correct answer is never generated; and (E2) correct answer is generated at least once.
This taxonomy covers 96% of the errors in TriviaQA for examples of each type of error, see Appendix D.3] Although some overlap exists between types, our goal is to identify general patterns and explore their connection the models's 's internal repre- sentations. For a discussion on the design choices of this taxonomy, refer to Appendix [ This taxonomy classifies LLM errors based on an extrinsic, behavior-based analysis. Similarly, previous work analyzed repeated samples to assess an LLM's knowledge of the correct answer (Simhi et al.) 2024 Geekhman et al.) (024). Our approach is distinct because it also examines the nature of errors that the LLM makes. Furthermore, as we discuss next, we analyze the connection between these behavioral patterns and the model's internal encoding.
For more qualitative
5.2 PREDICTING ERROR TYPES
Our taxonomy offers an external, behavioral analysis of LLMs, which we complement by intrinsic evaluation. We explore whether LLMs encode information on potential error types within their intermediate activations, offering deeper insight into the underlying mechanisms. To investigate this, we train a probe in a one-to-many setting, where a single probe identifies a specific error type from all others. We use representations extracted from the answers produced via greedy decoding.
Table preseens the results. Our findings show that error types can be predicted from the interme- diate representations of the greedy decoding generations, suggesting that they may capture not only
8

---

Published as a conference paper at ICLR 2025
Table 2: AUC scores for error type classification (TriviaQA). Error types are predictable from the inner model representations, indicating the encoding of fine-grained information on errors.
Error type
Mistral-7b
Mistral-Instr-76
Llama3-8b
Llama3-Inss Instrr-B
(A) Refuses to answer
0.8611 002
0.85
0.87 2:0002
0.88
(B) Consistently correct
0.881 0o1
0.82 oos
0.86 oo1
0.81 10 0000
(C) Consistently incorrect
0.59 0.59+ 002
0.67 +o,
0,59 .022
0.64 coa
(D) Two competing
0.632a..00 no2
1.688s..00 DoG
0,61 #0.001
0.65 004
(E) Many answers
0.90 001
0.84 0os
0.89
0.89
output correctness but also fine-grained information about potential errors. While detection perfor- mance varies between types, the predictability of each type is valuable on its own, as it opens the possibility of tailoring targeted interventions for specific error types. Additionally, although perfor- mance on error types C and D is lower, it remains well above random, providing meaningful insights. These results suggest that internal representations encode more than just binary correctness, reveal- ing a nuanced taxonomy of error types and offering deeper insights into how these models process and encode knowledge.
6 DETECTING THE CORRECT ANSWER
After identifying that models encode diverse truthfulness-related information, we examine how this internal truthfulness aligns with their external behavior during response generation. To this end, we use our probe F trained on error detection, to select an answer from a pool of 30 generated responses to the same question. We then measure the model's s accuracy based the selected answers. A case where this accuracy does not significantly differ from traditional decoding methods (such as greedy decoding), suggests that the LLM's internal representation of truthfulness is consistent with its external behavior. In simpler terms, that the model is generating answers that it also internally considers correct. Conversely, a case where using the probe alters performance either way, would suggest a misalignment between the LLM's s internal representations and its actual behavior.
Experimental Setup The experiments were conducted on TriviaQA, Winobias, and Math. We resample each model answer the same strategy described in Section 5. The final chosen answer is the one with the highest correctness probability, assessed by the probe. We compare to three baselines: (1) greedy decoding, (2) random selection from the = 30 answer candidates; and (3) majority vote wherein the most frequently generated answer is chosen.
Results The results for Mistral- 7eraal----inn 7b-instruct are summarized in Figure with additional results for other LLMs and datasets as well as qualitative examples provided in Appendix E We only present results on error types that appear 30 times or more in our test dataset. Overall, using the probe to select answers enhances the L.MSs accuracy across all examined tasks. However, the extent of improvement varies by error type. For instance, in the TriviaQA dataset, there is minimal gain in the "mostly correct" category (B2). In contrast, substantial gains-ranging from 30 to 40 points in some cases are observed in the "mostly incorrect" (C2), "two competing answers" (D), and "many answers" (E1) categories. Interestingly, and perhaps surprisingly, the probe is most effective in cases where the LLM lacks any (external) preference for the correct answer during generation. The fact that the probe can effectively identify the correct answer in these scenarios, points significant disconnect between the LLM's internal encoding and its external behavior. These results suggest that even when the model encodes information of which answer correct, it can still generate an incorrect answer in practice.
While using the probe to select the answer proves effective, it is not proposed here as an error mitigation strategy but rather a diagnostic tool. However, these findings indicate that further research in this area could leverage the existing knowledge within LLMs to significantly reduce emror.. We recommend exploring this direction in future investigations.
We choose the hest-performing prohe for each task, which is trained on the last exact answer token.

---

Published as a conference paper at ICLR 2025
greedy
majonity
probing
100
100
190
75
56
50
45
23
11
All
Reluses to answrer
Consistently correct Willy
Cons stently correct (Mosi)
Consistently nnorrect (All)
Consistently incorrect (Most)
Twa competing
Many enswers (Non correct) (Corrat appaars)
Many answers
(a) TriviaQA
64
70
20
20
0 AlI
e
Consistently
Consistently correct (Most]
Consistently incormect (AII)
Consistently incorrect (Mosk)
correct
IAIO
(b) Math
Figure 5: Different answer choice strategies, Mistral-7B-Instruee A notable improvement in ac- curacy by using the error-detection probe is observed for error types where the LLM shows no preference for the correct answer across repeated generations.
DISCUSSION AND CONCLUSIONS
In this study, we analyzed LLM errors through their internal representations. Our approach depends on access internal representations, restricting its to open-source models. We focus on QA tasks with clear gold labels, which are key for benchmarking truthfulness detection and valued by the community. To ensure robustness, we tested 10 datasets across model architectures. Open-ended tasks are left for future research, with our work laying the groundwork for broader applications. For instance, we found that truthfulness-reehe responses, enabling practical improvements in error detection for production models. This insight could extend to tasks like summarization, by probing the most meaningful entities in an answer.
information localized in specific tokens within long-
Truthfulness features showed poor generalization across tasks and datasets, highlighting the need caution when applying trained error detectors varied settings. Some unexplained patterns suggest hidden links between unrelated tasks that warrant further research. Improving generalization could involve exploring the effects of layer-token combinations and training on diverse datasets, demonstrated by Bürger et al. 2024). Deciphering task-specific truthfulness features and their overlaps across tasks might also enhance classifier design. Still, task-specific probes could be highly valuable in critical fields like medicine and law, where reliability matters. These probes can detect errors, predict error types, and guide response selection from resampled outputs, offering significant practical benefits. Guidelines for applying these probes are provided Appendix
Finally, we identified significant discrepancy between the model's external behavior and internal states, where repeatedly outputs incorrect responses despite internally encoding the correct an- swer. It is possible that mechanisms favoring likelihood override those promoting truthfulness, as LLMS are trained to predicting likely tokens, which does not necessarily align with factual aceu- racy. Our findings imply that these models already encode valuable information that could possibly be harnessed to reduce errors. Work by [Chuang et al. (2024) shows promising results this area, while a subsequent work by Gekhman al. (2025) focused exclusively on this "hidden knowledge phenomenon, formally defining it and studying its extent. In conclusion, our findings suggest that LLMs' internal representations provide useful insights into their errors, highlights the complex link between the internal processes models and their external outputs, and hopefully paves the way for further improvements in error detection and mitigation.
10

---

Published as a conference paper at ICLR 2025
8 REPRODUCIBILITY STATEMENT
To ensure reproducibility of our work, we provide detailed instructions and necessary code. The source code, including scripts for generating model answers, probing, resampling, and error type analysis, is available in the supplementary material, where we also provide command examples and specific seeds used for experiment reproducibility. This repository includes documentation on how to set up the environment, download and preprocess datasets, and execute the experiments outlined in Sections 3-6 of the paper. Additionally, all datasets, models, and results generation steps are described in the Appendix A
ACKNOWLEDGMENTS
This research was supported by the Israel Science Foundation (grant No. 448/20), Azrieli Foun- dation Early Career Faculty Fellowship, an AI Alignment grant from Open Philanthropy. and Google gift. HO is supported by the Apple AIML PhD fellowship. This research was funded by the European Union (ERC, Control-LM, 101 165402). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
REFERENCES
Alexandre Allauzen. Error detection in confusion network. In 8th Annual Conference of the In- ternational Speech Communication Association, INTERSPEECH 2007, Antwerp, Belgium, Au- gust 27-31, 2007, pp. 1749 1752. ISCA, 2007. doi: 1021437/INTERSPPEHH22000 https: http:///...rrg11.11111113/////// 2 2007-490
URL
Amos Azaria and Tom Mitchell. The internal state of an Ilm knows when it's lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 967-976, 2023.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of chatgpt reasoning. hallucination, and interactivity. arXiv preprint S.Ki::2302.402... 2023.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances, 2021. URL https:
7/arxiv. wtwp:iiw.gg/abbboooeeo. 12422
Samuel J. Bell, Helen Yannakoudakis, and Marek Rei. Context is key: Grammatical error detection Helen Yannakoudakis, Ekaterina Kochmar, Claudia Leacock, Nitin Madnani, lldikó Pilán, and Torsten Zesch (eds.). Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, BEA @ @CL 2019, Florence, Italy, August 2, 2019, pp. 103-115, Association for Computational Linguisties, 2019. doi: 10. 10.18653/V1/W194441
with contextual word representations.
4410
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint
2021.
Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wat- tenhofer. On identifiability in transformers. In 8th International Conference on Learning Repre- sentations (ICLR vvrtaa), International Conference on Learning Representations, 2020.
Lennart Bürger, Fred Hamprecht, and Boaz Nadler. Truth is universal: Robust detection of lies in 2024.
llms. arXiv preprint
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. arXiv preprint arXiv:2212.03822 2022.
Andrew Caines, Christian Bentz, Kate M. Knill, Marek Rei, and Paula Buttery. Grammatical error detection in transcriptions of spoken english. In Donia Scott, Núria Bel, and Chengqing Zong (eds.), Proceedings the 28th International Conference Computational Linguistics, COLING
11

---

Published as a conference paper at ICLR 2025
2020, Barcelona, Spain (Online). December 8-13, 2020, pp. International Commit- tee on Computational Linguistics, 2020. doi: W10.5/////00000000IG0-0AA.M 195. URL
Sky CH- Wa-ga Benjamin Van Durme, Jason Eisner, and Chris Kedzie. Do androids know they 're only dreaming of electric sheep?, 2023.
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: LLMs' internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representatinss, 2024, URLhttps: http://///ppeeeeeeeeeeeee net few.ruuuui...eoouuuu forum?
id 2d12nz20d
Wei Chen, Sankaranarayanan Ananthakrishnan, Rohit Kumar, Rohit Prasad, and Prem Natarajan. ASR error detection conversational spoken language translation system. In IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing, ICASSP 2013, Vancouver, BC, Canada, May 26-31, 2013, pp. 7418 7422. IEEE, 2013. doi:
(D009/CCASSS2221.666999999 URL https: / /ti. org 10. 1109 ICASSP 2013 10001..663500 6639104
Yong Cheng and Mofan Duan. Chinese grammatical error detection based on BERT model. In Erhong YANG, Endong XUN, Baolin ZHANG, and Gaoqi RAO (eds.), Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications, pp. 108- 113, Suzhou, China, December 2020. Association for Computational Linguistics. https:
www.0o..pppeea.... 1t.5
Yung-Sung Chuang. Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference Learning Representations, 2024. Utt.ttps::/ /ttps://pppeevieeeee
net htte//orumm?i0 idd 2h6NyL07na
Nelson Elhage. Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
Rahhal Errattahi, Asmaa El Hannani, and Hassan Ouahmane. Automatic speech recognition er- rors detection and correction: A review. Mourad Abbas and Ahmed Abdelali (eds.), /st ternational Conference on Natural Language and Speech Processing, ICNLSP 2015, Algiers, Algeria, October 18-19 2015, volume 128 of Procedia Computer Science, pp. 32-37. Elsevier, 2015. doi: 10.101 wTt.:///......... 2018. 03. 005.
prroo.. procs.
Dan Flickinger, Michael Wayne Goodman, and Woodley Packard. Uw-stanford system description for AESW 2016 shared task on grammatical error detection. In Joel R. Tetreault, Jill Burstein, Claudia Leacock, and Helen Yannakoudakis (eds.), Proceedings of the 1 11th Workshop on Innova- five Use of NI.P for Building Educational Applications, BEA @ NAACL-HLLT 2016, June 16, 2016, San Diego. California, USA. pp. 105-111. The Association for Computer Linguistics, 2016. doi: 16-511. htths:t:: //oi. 1t5t::///bbl.oogg00111555///// wwwp://vvlwll-61 0511
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. Rarr: Researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 16477-16508, 2023.
Zorik Gekhman, Roee Aharoni, Genady Beryozkin, Markus Freitag, and Wolfgang Macherey. KoBE: Knowledge-based machine translation evaluation. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 120003207, Online, November 2020. Association for Computational Linguisties. www.tholoooy.oog/2222..... org/2020.1
doi: 10.18653/v1/2220 100.8665//122000.mddmeemmmmmmmm findings-emnlp.87. URL Findings emnlp.2pp.. 287
Zorik Gekhman, Dina Zverinski, Jonathan Mallinson, and Genady Beryozkin. RED-ACE: Robust error detection for ASR using confidence embeddings. In Yoav Goldberg, Zomitsa Kozareva,
12

---

Published as a conference paper at ICLR 2025
and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natu- ral Language Processing, pp. 2800-2808, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. doi: 10. 1t0.6/3////2222.mmlmmmm. 180. URL
https: antho logy oog..oy..ooooo......... org/ 2022. emn www.emnnlmmme ma n.. 18D]
Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. TrueTeacher: Learning factual consistency evaluation with large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natu- ral Language Processing. Pp. 2053-2070, Singapore, December 2023. Association for Computa- tional Linguisties. doi: 10.18653/v htt55533/12222.m/mm/m main. 127. URL org/2023 emn wwappmmain....mm 127
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning lims on new knowledge encourage ballucinations?? 2024.
Zorik Gekhman, Eyal Ben David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpector, Jonathan Herzig, and Roi Reichart. Inside-out: Hidden factual knowledge llms. arXiv preprint arXiv:2503.15299, 2025.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in rego-segressive language models. arXiv preprint arXiv:2304. 14767, 2023.
Daniela Gottesman and Mor Geva. Estimating knowledge in large language models without gen- erating single token. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), Miami, Florida, 2024. Association for Computational Lin- guisties.
Nuno M Guerreiro, Elena Voita, and André FT Martins. Looking for a needle in haystack: A comprehensive study of hallucinations neural machine translation. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1059- 1075, 2023.
Stevan Harnad, Language writ large: Llms, chatgpt, grounding, meaning and understanding. arXiv preprint arXiv:2402.0222 2024.
Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. q²: Evaluating factual consistency in srowledge-grouddee dialogues via question generation and question answering. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 7856-7870, Online and Punta Cana, Dominican Republic, November Association for Computational Linguistics. doi: 10. 18.13//3/v 1/t021/11111/// wwtp222....mmmmmmee/mmm moo--mmain.9999 URL wrg/
emnlp- meto-mminn.. 619
Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom. Idan Szpektor, Avinatan Hassidim, and Yossi Matias. TRUE: Re-evaluating fac- tual consistency evaluation. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pp. 3905-3920, Seat- tle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/1/20022. naacl-main.287 URL htt.tttss//aaaaattooooooooo htwphhhoooooooogg/222222 naacc main..n.......... 287
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint 2:X052231..05222 2023a.
Yuheng Huang, Jiayang Song, Zhijie Wang, Huaming Chen, and Lei Ma. Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv preprint arXiv:2307. 10236, 2023b.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
13

---

Published as a conference paper at ICLR 2025
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anee Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
//aar/ww..rg/aab//3111... 0682
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprchension. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, 2017.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Batfield-Dodde Nova DasSarma, Eli Tran-Johnson, et al. Language mod- els (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022.
Sudhanshu Kasewa, Pontus Stenetorp, and Sebastian Riedel. Wronging a right: Generating better errors improve grammatical error detection. In Ellen Riloff, David Chiang. Julia Hocken- maier, and Jun ichi Tsujüi (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 -Nvember 4, 2018, pp. 4977- 4983. Association for Computational Linguistics, 2018. URL org/D18-15417
Hadas Kotek, Rikker Dockum, and David Sun. Gender bias and stereotypes in large language models. In Proceedings of the ACM collective intelligence conference, pp. 12-24, 2023.
Wojeiech Kryseinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9332-922 9346, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1 (0/665///1/2220 http1/.emnlpppppppp/ camnlp-main7755555 m5in.700. URL hatpps:/accaanthoooog..oo hot.hhooooooooooooo/ooooooo 2020. emn wmw.pmmalaammmmmmmmmmmmmmmmmmmmm inn.nn..... 750
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Confer- ence on Learning Representations, 2023. URL htttp:://pperreriieet net/ forum? did=
VD- Adv=eee
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Al- berti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wer Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. SummaC: Re-visiting NLI-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163-177. 2022. doi: 10.1162/tac URL https:
7/ //aclanthology org. 2022. tacl-1.
Benjamin A Levinstein and Daniel A Herrmann. Still no lie detector for language models: Probing empirical and conceptual roadblocks. Philosophical Studies, pp. 1-27, 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- -aumente- genera- tion for knowledge-intenn intensiveeeeeeeeeee nlp tasks. Advances Neural Information Processing Systems, 33: 9459-9474, 2020.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. inference-time intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 36, 2024.
Wei Li and Houfeng Wang. Detection-correctie structure via general language model for grammat- ical emor correction. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of
14

---

Published as a conference paper at ICLR 2025
the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Pa- pers). ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 1748 1763. Association for Com- putational Linguisties, 2024.
org/2024
Xun Liang, Shichao Song, Zifan Zheng, Hanyu Wang, Qingchen Yu, Xunkai Li, Rong-Hua Li, Yi Wang, Zhonghao Wang, Feiyu Xiong, et al. Internal consistency and self-feedback in large language models: A survey, arXiv preprint arXiv:2407.14507, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv: 2021.
Kevin Liu, Stephen Casper, Dylan Hadfield-Menll. and Jacob Andreas. Cognitive dissonance: Why do language model outputs disagree with internal representations of truthfulness? In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Em-
pirical Methods in Natural Language Processing, pp. 4791 4791-4797, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18663311/023..mmmm
URL
org/ 2023.
Tianyu Liu, Yizhe Zhang. Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. A token-level reference-free hallucination detection benchmark for free-form text generation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers). pp. 6723-6737, Dublin, Ireland, May 2022. Association for Computational Linguis- tics. doi: 10.1 10.18653/v1/20222 awl.22..cl-ooo..444 URL. http:sss/aaaaaanhhlllgoe orgp0022..0//000...../
ac - 1an--ong... 464
Chi-kiu Lo. YiSi a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Ondřcj Bojar, Rajen Chatterjee, Christian Fed- ermann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor (eds.). Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). pp. 507-513, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/11/W19955303 URL DEtps://aclanthol/
org/ 011///1993358
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http:
wwww.acllwen org/ httpg//ahhhlooyyyll11o 1015
Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box halluci- nation detection for generative large language models. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 9004-9017, 2023.
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023.
Alessia McGowan, Yunlai Gui, Matthew Dobbs, Sophia Shuster, Matthew Cotter, Alexandria Sell- oni, Marianne Goodman, Agrima Srivastava, Guillermo A Cecchi, and Cheryl M Corcoran. Chat- gpt and bard exhibit spontaneous citation fabrication during psychiatry literature search. Psychi- atry Research, 326: 15334, 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associ- ations in GPT. Advances in Neural Information Processing Systems, 36, 2022. 2rX2v:2202.05262.
Beren Millidge. LLMs confabulate not hallucinate. Beren's 's Blog. March 2023. URL https:
www. wew.ne wtw.ereennn//223-3-- -at.:aa---o22222---------------- www.eaaaaaaaaoooooooooooeeaa hal org.://0........................
uc luciitte//e
Ritika Mishra and Navjot Kaur. A survey of spelling error detection and correction techniques. International Journal of Computer Trends and Technology, 4(3):372-374, 2013.
15

---

Published as a conference paper at ICLR 2025
nostalgebraist.
Interpreting gpt: The logit lens. LessWrong blog post, http://wwwwwwwoo.../// cctp://poooa/eccBBBaaaaaaaaotra.
2020.
URL
gnterpretinggg-ttttteee -ooggtt-t---iit-eee-- Accessed: 2024-11-18.
Chris Olah, Nelson Elhage. Neel Nanda, Catherine Schubert, Daniel Filan, et al. To-
wards monosemanticity: Decomposing language models with dictionary learning. Trans- former Circuits Thread, 2023. URL atta:s//eanstforrrrroaaaaatsssse
-essemmanni---etureess hitp//iddeee.iimml
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-leam:: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Thomas Pellegrini and Isabel Trancoso. Error detection in broadcast news ASR using markov chains. In Zygmunt Vetulani (ed.), Human Language Technology. Challenges for Computer Science and Linguistics- - 44th Language and Technology Conference, LTC 2009, Poznan, Poland, November 6-
8. 2009, Revised Selected Papers. volume 6562 of Lecture Notes in Computer Science, pp. 59-69. Springer, 2009, doi: 10. 000779988---4222099559
6. URL https: org
378-3-642-20095-22
Amy Pu, Hyung Won Chung, Ankur Parikh, Sebastian Gehrmann, and Thibault Sellam. Learn- ing compact metrics for MT. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pp. 762- Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 0036633/v12021.emmlg emnlp-mmin.58. main.58. URL
httpss://cclanthoo..os
mmmp://mm2m2..m.mmmmmmmmmmm. in. i........ 58
Gaoqi Rao, Erhong Yang, and Baolin Zhang. Overview of NLPTEA-2020 shared task for Chinese grammatical eтror diagnosis. In Erhong YANG, Endong XUN, Baolin ZHANG, and Gaoqi RAO (eds.). Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educa- fional Applications, pp. 25 35, Suzhou, China, December 2020. Association for Computational Linguisties. URL https: htt///aaaaannnoooo.aaaaaaaoooala wrg//o20.000.....ooooo wet.nnnppteee nlptea-1.
Miriam Rateike, Celia Cintas, John Wamburu, Tanya Akumu, and Skyler Speakman. Weakly super- vised detection of hallucinations in Ilm activations. arXiv preprint arXiv:2312.02799. 2023.
Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, SM Tonmoy, Aman Chadha, Amit P Sheth, and Amitava Das. The troubling emergence of hallucination in large language models-an extensive definition, quantification, and prescriptive remediations. arXiv preprint arXiv:2310.04988. 2023.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 2685- 2702, Online, November 2020. Association for Computational Linguistics. doi: 10. 18653/v1/ 2020.emnlp-main.211
pearttsss//caaaatoooooooooo org/ wrwp//20..o000.. emnlp mainn.. 213
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Philipp Koehn, Loic Barrault, Ondřej Bo- jar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewiez, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Koemi, André Martins, Makoto Mor- ishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa. Matteo Negri, Aurélie Névéol, Mar- iana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Sev- enth Conference on Machine Translation (WMT), pp. 578-585, Abu Dhabi, United Arab Emi- rates (Hybrid), December 2022a. Association for Computational Linguistics. URL https:
/acclanthoogy org/2022 wnt 5m.-
Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie,
16

---

Published as a conference paper at ICLR 2025
and André F. T. Martins. CometKiwi: IST-unbabel 2022 submission for the quality estima- tion shared task. In Philipp Koehn, Loic Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatter- jee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Koemi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 634-66- 634-645, Abu Dhabi, United Arab Emirates (Hybrid), December 2022b. Association for Computational Linguisties.
http:/aaaaaattoooooooooooooo/e/e wrg../rgg/222 wmt-1.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Arleen Salles, Kathinka Evers, and Michele Farisco. Anthropomorphism in ai. AJOB neuroscience, 11(2):88-95, 2020.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. QuestEval: Summarization asks for fact-based evaluation. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6594 6604, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 18.18653/v1 12066/////2211mmmmmmaaaa.... 2021. wwa.mppmain...... 529
URL htt.ps://cclannthol..co
org/
Thibault Sellam, Dipanjan Das. and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892 Online, July 2020. Association for Computational Linguisties. doi: 10. 704. URL httpsttt:///at www.cclllnnooooooyyoooooooooooo. www.logg.orggg/0/ www.:orgggg222... hctp//2220..cc11l0. .main.
Greg Serapio-García, Mustafa Safdari, Clément Crepy. Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. Personality traits in large language mod- 2023.
els. arXiv preprint
Adi Simhi, Jonathan Herzig, Idan Szpektor, and Yonatan Belinkov. Constructing benchmarks and interventions for combating hallucinations in llms, 2024.
Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. The curious case of ballucinatory (un)answerability: Finding truths in the hidden states of over-confident large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3607-3625, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v cemol..emnlpp-ammm. main. 220. URL http:::://aaaaaaaaaaaa htt.:htt.ps///ee//attoooooooorro
org http:r/g//222..ee0mmmeemmeme..ee htt.p://......aaooooooooooooo///
220
Ben Snyder, Marius Moisescu, and Muhammad Bilal Zafar. On early detection of hallucinations in factual question answering, 2023. URL BtL/https: /tt///aaxii wtaaaaivvvv.o.... org/ 22122 2.2..44 14183
Yuhong Sun, Zhangyue Yin, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Hui Zhao Benchmarking hallucination in large language models based on unanswerable math word problem. CoRR, 2024.
Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. Confidence improves self-consistency in 1lms. arXiv preprint arXiv:2502.06233, 2025.
Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine-tuning language models for factuality. arXiv preprint arXiv:2311.0840) 2023a.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint arXiv:2305. 144.: 2023b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023.
17

---

Published as a conference paper at ICLR 2025
Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. A stitch in time saves nine: Detecting and mitigating hallucinations of Ilms by validating low-confidence generation, 2023.
Pranav Narayanan Venkit, Tatiana Chakravorti, Vipul Gupta, Heidi Biggs, Mukund Srinath, Koustava Goswami, Sarah Rajtmajer, and Shomir Wilson. " confidently nonsensical?": A critical survey on the perspectives and challenges of hallucinations' in nlp. arXiv preprint arXiv:2404.07461, 2024.
Chaojun Wang and Rico Sennrich. On exposure bias, hallucination and domain shift in neural machine translation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Anmual Meeting of the Association for Computational Linguisties, pp- 3544-3552, Online, July 2020. Association for Computational Linguistics. doi: 10. 18653vvll 2020.acl- main. 326. URL htt.hhttt://accc http:///caaattttcca ogy owtooogg//220 acl- 326
Quanbin Wang and Ying Tan. Grammatical error detection with self attention by pairwise training. In 2020 International Joint Conference on Neural Networks, IJCNN 2020, Glasgow, United King- dom, July 19-24, 2020, pp. 1-7. IEEE, 2020. doi: 10.11091ICNNN4605 20w.:00..222220227111 15. URL
https:://o //dii. orgg 10. 109 JCNN4 B 8605 225.5.2.00 2020. 9206715
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1112-1122, Association for Computational Linguisties, 2018. URL http:/ web. org/ antho logy B
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380, 2380, 2018.
Fan Yin, Jayanth Srinivasa, and Kai-Wei Chang. Characterizing truthfulness in large language model generations with local intrinsic dimension. In Proceedings of the 41st International Conference on Machine Learning, Vienna, Austria, 2024.
Gal Yona, Roee Aharoni, and Mor Geva. Can large language models faithfully express their intrinsic uncertainty in words?, 2024. URL httpsttps://// /ttp:://xxivvvii http://wwww..og/ab//////aoe
16908
Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, and Besmira Nushi. Attention satisfies: A constraintssatisacttin lens on factual errors of language models. In The Twelfth International Conference Learning Repre- sentations, 2023.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu, ERNIE: Enhanced language representation with informative entities. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1441-1451, Florence, Italy, July 2019. Association for Computational Linguis- /ttp:p/se//aceeatoooooooooo/////
tics. doi: 10.18653/v 1/119-1199
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804 804.06876. 2018.
Lina Zhou, Yongmei Shi, Jinjuan Feng. and Andrew Sears. Data mining for detecting errors in dictation speech recognition. IEEE Trans. Speech Audio Process., (3(5-1))681-66) 2005, doi: URL httl/httss///oi.... http:///ooorrgg////// /10 1109/111. 2005888777
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan. Xuwang Yin, Mantas Mazeika, Ann- Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael Byun, Zifan Wang. Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down ap- proach transparency, 2023. URL htt.h/p:://axxi.orrgaa/////////e 01405
18

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
A.1 TASK SPECIFIC ERROR DETECTION
In this work, specifically address errors produced by modern large language models (LLMs). Given the diverse range of tasks these models are applied to, our focus on general error detection across all categories, rather than isolating specific types. Prior to the emergence of LLMs, much
research targeted error detection for specific tasks, with common examples including grammatical errors (Kasewa et al. 2018 Bell et al. [2019] Cheng & Duan 2020; Wang & Tan 2020 Flickinger
et al. 2016), spelling mistakes (Mishra & Kaur 2013), machine translation inaccuracies (Lo 2019]
Pu et al. [2021] Sellam al.) 22200] Gekhman al.
2020 Rei al.) [220 2022a b). speech recog-
nition faults (Caines et al. [020 Rao et al. 2020
Li& Wang 2024 Zhou et al. [2005; Allauzen
2007 Gekhman al. [022] Errattahi al.
2015 Pellegrini & Trancoso 2009 Chen al.
2013).
and factual consistency failures (Honovich et al.) 2022] Laban et al. 2022 Honovich et al. 2021
Gekhman et al. 2023 Scialom al. 2021
Kryscinski et al. 2020).
A.2 PROBING: IMPLEMENTATION DETAILS
We examine the intermediate representations of the exact answer tokens generated by a large lan- guage model (LL.) during the answer generation process. The intermediate representation selected for this analysis is derived from the output of the final multi-layer perceptron (MLP). This choice based on preliminary experiments comparing the MLP output, the residual stream, and the attention heads, which showed no significant differences. We leave the in-depth analysis for future work.
For the probing classifier, dregosa et al., 2011). We used the default hyperparameters, which include a norm penalty of L2 and an LBFGS solver. We initially experimented with other hyper-parameter and did not find singnificant difference. For each random seed, the dataset was split to training and validation in 80-20 ratio, and the test dataset was bootstrap sampled.
employ logistic regression model from the scikit-learn library (Pe-
Obtaining correctness label for the prohing dataset. An answer is generally considered cor- rect if it includes the correct answer label and appears before any alternative incorrect labels. We manually analyzed the results of this heuristic to confirm that it is accurate in almost all cases. How- ever, one exception is the Natural Questions with Context (NQ_WC) dataset, where identified false negatives and thus deployed a more precise validation using an instruct LLM, as demonstrated below:
Evaluate the following answers to questions. For each question you would be given an LLM answer and the correct answer. You would have to determine if the LLM answer is correct not. If the LLM answer correct, write "1' and it not correct, write '0'. For example:
Question: [Questi on 1]
Ground Truth: [Gold label 1]
LIM Answer: [LLM long answer 1]
Correctness: C
Question: [Question 2]
Ground Truth: [Gold label 2]
LLM Answer: (LLM long answer 2]
Correctness: 1
Question: [Question]
Ground Truth: [Label]
LIM Answer: [LLM long answer]
Correctness:
19

---

Published as a conference paper at ICLR 2025
Table 3: Success rate of extracting exact answer from long model answer. Each model is used to extract answers from its own output.
Mistral-7b
Mistral-Instructee
Llama3-8b
Llama--Instruct-Bb
0.99
0.96
0.99
0.95
Detecting and using exact answer tokens. Exact answers are identified from a lengthy generated answer using an external algorithm, which processes the question and the LLM's response, A(qis 9 to extract the exact answer. After extraction, we identify the exact answer tokens via simple search process, focusing four key tokens: the before the first exact answer token, the first and last exact answer tokens, and the one after the last.
For the implementation of A that detects the exact locations of answer tokens, we use a combination of heuristic methods and an instruction-tuner LLM. Specifically, when the set of possible answers finite, we rely on heuristics. For more open-ended scenarios, such as factual questions, we auto- matically locate the answer if it matches the gold label. Otherwise, we prompt an instruction- LLM, specifically Mistral-7b-Instrue (Jiang et al. 2023), to identify and extract the exact answer substring using the following prompt:
Extract from the following long the short answer, only the relevant tokens. If the long answer not answer the question, output NO ANSWER.
Q: [Question 1]
A: [LLM long answer 1]
Exact answer: [Short exact answer 1]
Q: [Question 2]
A: [LLM long
that does not the question]
Exact answer: NO ANSWER
Q: [Question]
A: [LLM long answer]
Exact answer:
To extract a valid exact answer from a long response, we prompt the instruct LLM up to five times. This process involves verifying that the exact answer is substring of the long answer unless the instruct LLM indicates that there is no answer. To avoid bias in our probing task, we only retain questions for which valid exact answer was successfully extracted. This ensures there is unfair correlation between invalid answers and incorrect answers the experiments.
(a) While it is possible to instruct LLM extract every answer regardless of its correctness, we chose the aforementioned strategy to improve the efficiency of our experiments; (b) This is just one possible implementation. For each LLM, one could use the same LLM to extract its own exact answer token, as demonstrated in a proof-of-concept over 1000 samples of TriviaQA in Table 3 Alternatively, may be more effective to train smaller system specifically designed for detecting exact answer tokens, which would be more suitable for real-world scenarios. We choose keep the extraction process as abstract as possible, as our primary focus not on the specific implementation, but on analyzing the potential gains from probing these locations.
We note the following:
Additionally, if the exact answer token is not among the first generated tokens, we examine the token immediately preceding ("before exact answer token")) If the exact answer token is not the last one, we also examine the following token. When the exact answer spans multiple tokens, the first and last exact answer tokens are probed separately.
A.3 DATASETS
We outline here all ten datasets that investigate in work. In our analysis, aimed at covering a wide range of tasks, skills required to solve the tasks, diversity of datasets and as a result also different L.M limitations such as factual inaccuracies (often referred to as "hallucinations"). biases,
20

---

Published as a conference paper at ICLR 2025
arithmetic mistakes, and more. For each dataset, explain how it covers something different from all the previous datasets. For all datasets, we present the LLM with non or a short instruct, a context (if exists for the task), and let it generate a free text. We follow this paradigm
it better mimics real-world usage of LLMs by humans, as opposed to using few-shot to force a short answer that is generated on the first token (Yuksekgonul et al., [023]] Chen et al., 2024 Simhi al.] 2024). One exception to this is a the sentiment analysis (IMDB) for which we apply 1-shot for the LLM to use the allowed labels, as it did not follow the instruction alone and we could not identify if the answer is correct or not even with manual analysis. Additionally, we implemented different prompting strategy to the instruct and non-instruct LLMs. To see the exact formats we used to prompt each dataset and LLM, refer to our code implementation at htttps://githun com/ sechnion-cs-- co- n p/ L.Ms
For each dataset we used a split of 10K training samples and 10K test samples, unless the dataset is too small, in which case we mention the size.
TriviaQA (Joshi et al. 2017): a collection of trivia question-anse answerr pairs. The questions are presented to the LLM without any context, allowing it to generate responses based solely on its internal, parametric knowledge. The dataset includes various acceptable vari- ations of the correct answer, which are used to automatically evaluate the accuracy of the generated res.
HotpotQA (Yang et al. 2018): dataset designed for diverse multi-hop question answer- ing. Each entry includes Wikipedia documents that help answering the questions. We use two different settings: (1) without context, where questions are asked directly, which cov- ers slightly different skills from TriviaQA as it requires reasoning in addition to factual knowledge; and (2) with context (HotpotQA_War
where the additional context is pro- vided, emphasizing the ability to adhere to and utilize contextual information to solve the task.
Movies: to further investigate generalization, we focused on a case of classic "halluci- nations", involving factual knowledge, within a non-diverse dataset. This approach al- lowed us to test whether generalization to other types of errors is influenced by the type of error (factual versus others) or by the dataset's diversity. For this purpose, we cre- ated the movies dataset consisting of prompts in the form: "Who acted as [figure name] in the movie [movie name]?" The figures, movies, and correct answers were sourced from "The Movies Dataset" in Kaggle: https:: /www//wwww... wwwgle.gge.... com/ dataaattsssss.ee hounakbanik/the-eae soneeemoviess dataset which is based on the MovieLens website.
Winogrande (Sakauuchi et al. 202I): we use this dataset to explore errors in common- sense reasoning. It consists of W Winograd-style coreference challenges, where each example presents sentence containing two entities and pronoun. The objective is to determine which entity the pronoun refers to, relying on common-sense reasoning. For example, in the sentence: "The trophy doesn't fit into the suitcase because it's too large," the pronoun "it" refers to the trophy, not the suitcase.
Winobias (Zhao et al., 2018): this benchmark focuses on coreference resolution in the context of gender bias, revealing different type of limitation in LLMs. Each example consists of two professions: one stereotypically male and one stereotypically female, along with gendered pronoun. The task requires the LLM to determine which profession the pronoun refers to. The sentences are unambiguous, with one correct answer. In some cases, the correct answer aligns with the stereotype, while in others, it is anti-stereotypical. For example, in the sentence "The developer argued with the designer because she did not like the design," "she" refers to the developer, which is anti-stereotypical case since "de- veloper" is considered a stereotypically male profession. Research has shown that LLMs often perform poorly on anti-stereotypical sentences (Zhao et al. 2018) and tend to base their decisions on stereotypes rather than on common-sense reasoning or linguistic rules (Kotek et al. 2023). Each split contains around 1500 samples.
NLI (Natural Language Inference): NLI involves determining whether a given "hypoth- esis" is true (entailment), false (contradiction). or undetermined (neutral) based on a pro- vided "premise." For this purpose, we use the MNLI dataset (Williams al. 2018). NLI tasks address a distinct aspect of common sennon--ennse reasoning and are generally considered complex. This complexity allows us investigate whether model's generalization ability
21

---

Published as a conference paper at ICLR 2025
is related the difficulty of the task it was trained on, other factors, such the limited diversity of labels (NLI has only three valid labels) or the type of task.
Math Sun et al. 2024): this dataset includes both unanswerable and answerable math problems. In our study, we focus exclusively on the answerable problems, as our aim is to assess the correctness of LLM'' outputs, which requires a known correct answer (gold standard). This task introduces additional, previously unexplored skill of arithmetic rea- soning. The train-test split consists of approximately 2,000 and 650 samples, respectively.
IMDB (Maas et al. 2011 cation.
contains movie reviews used for the task of sentiment classifi-
Natural Questions With Context (Kwiatkowski et al., 2019): the Natural Questions (NQ) dataset is designed evaluate and train automatic question-answering systems. consists of real, anonymized queries submitted by users to Google, with answers extracted from Wikipedia, as well as the relevant Wikipedia pages which can be given in context. We included this dataset to introduce an additional challenge that requires adherence to context, complementing the HotpotQA with context dataset.
A.4 BASELINES: IMPLEMENTATION DETAILS
Aggregated probabilities / logits. Inspired by prior work (Kadavath et al. 2022 Guerreiro al., 2023), we compute an aggregated score using the log-probabilities or raw probabilities of the gen- erated text tokens y 1/.,..., . I/N produced by the generative large language model (LLM). For in- stance, the following formulation is used to compute the Logits- mean baseline on the entire gener-
ated answer:
N
1
PwulQ. V/, Vi.....Bt-1)
N
We also explore aggregation strategies that focus solely on the exact answer tokens (PE-Exact). Following Varshney et (2023), we also experiment with aggregating the minimum and maximum values (PE-[Min--oe Max [xact)), alongside the mean aggregation described in Equation [
P(True): We follow Kadavath et al. (202) and prompt the LLM to judge whether its answer is correct. Our prompt followed the following template, from Kadavath et al. (2022):
Question: [Question]
Proposed Answer: [LLM long answer]
Is the proposed answer:
(A) True
(B) False
The proposed answer is:
B FULL ERROR DETECTION RESULTS
Figure presents the AUC values traind probe across layers and token for showing similar pattern across all datasets. We also observe similar patterns across other models. See our repo ht ttttpp::
/ttp:://tttt /ithhb.. wow/technion..ccccccccccccc cs neee-nlp/in halp/LLMsKnon for the figures.
Tables and 7 present the full error detection results across all baselines and datasets, which are consistent with the main paper results.
22

---

Published as a conference paper at ICLR 2025
Ion
(a) HotpotQA
HotpotQA with context
Movies
17
07
(d) Winogrande
(c) NLI
(f) IMDB
Figure 6: AUC values of probe error detector across layers and tokens, Mistral-7b-instie The detection performance spikes at the exact answer tokens.
23

---

Published as a conference paper at ICLR 2025
Table 4: Comparison of error detection performance (AUC) on Mistral-7B.
Mistral-7B
TriviaQA
Winobias
Math
Movies
IMDB
Logits-mean
0.67 10.004
0.49 20.010
0.41 20.015
0.67 =0.007
0.88 =0.044
Logits-mi lggis-mmean-exct
0.67 +0.004
0.50 +000010
0.56 +0,026
0.68 =00.00
0.57 =0.080
Logits-min
0.80 20.00
0.45 20.014
0.48 20.021
0.73 20.006
0.78 20.056
Nogits-min-exxx
0.80 +0.005
0.53 #-00114
0.78 +0.032
0.72 #:0.005
0.57 0.00
Logits-max
0.53 +.0.000
0.49 +000010
0.42 +0.023
0.54 =-0.005
0.83 000076
Lxgiss-mxx-exacl
0.54 2:0
0.50 20.010
0.40 20.024
0.58 200007
0.57
Probas-mean
0.76 003
0.53 #000188
0.66 +0.016
0.72 =00.007
0,87 0.004
Probas-mean-exacl
0.78 2:0. 002
0.55 20.014
0.62 20.018
0.74 = 007
0.83 000057
Probas-min
0.82
0.52
0,82
0.73
0,86
Probas-min-exact
0.85 20.003
0.58 20.011
0.84 20.015
0.74 20.006
0.83 =0.057
Probas-max
0.53 2:0 008
0.50 20.016
0.43 20.025
0.55 20.008
0.80 =0.074
Probas-max-exan
0.55
0.51
0.39
0.59
0.83
P(True)
0.57 20.007
0.53 20.018
0.56 20 02.0027
0.51 20.003
0.65 20.004
PeTrue)-exact
0.56 +00006
0.55 +0.026
0.57 400088
0.52 +0.003
0.65 #0.003
Probe @ token
Last generated [-1]
0.83 +0.002
0.65 +0.000
0.82 +0.023
0.79 =0.002
0.85 00000
Before last generated [-2]
0.82 10.00
0.84 20.012
0.83 10.010
0.78
20.003
0.95 20.004
End of question
0.74 +0.005
0.78 +0.0122
0.83 40.016
0.77
#000002
0.81
+00009
Exact answer last.
0.84 10.00s 00s
0.89
20.007
0.96 =0.008
0.78
=0.003
0.95
1:0004
Exact answer last+1
0.84 20.004
0.84 10.012
0.95 10.010
1:0.000
0.85 20.007
HotpotQA
HotpotQA-Wo
Winogrande
NLI
NQ-WC
Logits-mean
0.63 10.000 005
0.52 009
0.49 10 10.004
0.51
100000
0.69 10.006
Logits-mean-exan
0.57 +0.008
0.52 200007
0.50 +0.003
0.93 #:0.004
0.72 =0.005
Logits-min
0.72
0.59
0.50 1:
0.53 100.00
0.65
Logits-min-exact
0.72 #0.007
0.65 #0.0004
0.51 +o
0.49 =0.0006
0.70 =0.005
Logits-max
0.54 1:0. 00r
0.49 1: 110.110
0.48 1:0 005
0.48 ::0 005
0.59 00.02
Logits-max-exact
0.48
0.44
0.50
0.48 -000005
0.58
Probas-mean
0.65 1:000 004
0.55 1: 008
0.51 1:0 007
0.49 :: 003
0.63 00.008
Probas-mcan-cxact
0.62 1:0. 008
0.56 007
0.51 1:0 005
0.02 1:0 001
0.66 10.007
Probas-min
0.73
0.58
0.52
0.53 0..004
0.63
Probas-min-exact
0.78 1:00 008
0.66 1: 004
0.52 1: 008
0.49 ::0 00s
0.69 :0.006
Probas-max
0.54 +0.008
0.49 +0.007
0.50 +0.005
0.47 =0.004
0.52 =0.004
Probas-max-exact
0.48 1:000
0.44 00s
0.50 10 004
0.48 10.003
0.53 20.012
p(True)
0.55 007
0.54 +
0.51 +0.005
0.51 =0.003
0.52 00000
P(True)-exact
0.61 +0.005
0.54 +0.006
0.61 +0.006
0.51 #0.006
0.53 =0.014
Probe @ token
Last generated [-1]
0.78 +0.006
0.67 004
0.51 +0.007
0.77 =0.004
0.78 =0.003
Before last generated [-2]
0.79 10.
0.69
0.66
0.81
0.75
End of question
0.72 +0.007
0.56 400.003
0.51 +0.007
0.88
=0.004
0.70 =0.005
Exact answer last
0.80 1.
0.74 10.0007
0.69 00.006
0.84 10.004
0.81 10.009
Exact answer last+1
0.81 +0.00s
0.72 #o 400.005
0.59 +0.005
0.75 =0.006
0.84 +0.007
24

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of error detection performance (AUC) on Mistral-7B-Instruet..
Mistral-7B- Besstrl--B-Instruce
TriviaQA
Winobias
Math
Movies
IMDB
Logits-mean
0.60 1:000
0.56 20.017
0.55 4:
0.63 =0.005
0.57 =0.006
Logits- agits-mean-exact
0.68 +0.007
0.54 +0.0122
0.51 +0,005
0.70 =000004
0.87 =0.007
Logits-min
0.63 2:000
0.59 20.012
0.51 20.017
0.66 20.008
0.52 =0.007
Nogits-min-exxn
0.75 + n08
0.53 #-00113
0.71 +0.009
0.74 #00.006
0.87 = 000007
Logits-max
0.54 +-0.00 n0s
0.53 +000112
0.54 +000399
0.54 =00.004
0.47 = =0.004
Lxgiss-mxx-exall
0.55 2:0.
0.54 20.011
0.32 20.015
0.61 20.006
0.87
Probas-mean
0.60 +00.007
0.58 #-0.018
0,56 +0
0.61 #-0.002
0.54 = 00.008
Probas-mean-exacl
0.71 2:00 003
0.57 015
0.71 20.014
0.74 =0.006
0.84 000007
Probas-min
0.59
0.58
0,50
0.60
0.51
Probas-min-exact
0.74 20.004
0.57 #0.016
0.75 20.011
0.73 20.006
0.84 00.007
Probas-max
0.50 2:0. 008
0.41 20.010
0.53 20.000
0.51 20.005
0.48 =0.004
Probas-max-exan
0,51
0.54 +0.010
0.45 +0.015
0,60
0.84
P(True)
0.68 20000 00s
0.45 10.00
0.48 20.020
0.62 20.005
0.62 000009
PeTrue)-exact
0.74 #000003
0.40 +000221
0.60 40.025
0.69 #00000
0.60 =0.009
Probe @ token
Last generated [-1]
0.71 +0.006
0.82 +0.000
0.74 +0.00s
0.72 =00.00
0.92
=0.010
Before last generated [-2]
0.73 10.00
0.85 004
0.74 1:
0.72 20.006
0.94 20.006
End of question
0.76 +0.008
0.82
+000111
0.72 40.007
0.74 +000008
0.96
=0.006
Exact answer last.
0.85 1:.00
0.92 20.005
0.92 =0.008
0.81 10.003
0.97 10.005
Exact answer last+1
0.86 20.006
0.88 1: 008
0.90 10.010
0.82 1:0.003
0.96 20.006
HotpotQA
HotpotQA-We
Winogrande
NLI
NQ-WC
Logits-mean
0.61 10.00 002
0.55 009
0.59 10 000
0.64 1:0 1.0.008
0.71 10.008
Logits-mean-exan
0.66 +0.003
0.55 #0.004 004
0.49 +0.004
0.57 =0.004
0.69 =0.009
Logits-min
0.61 1:0
0.53 10.013
0.61 10.000 00.003
0.62 10.002
0.67
Logits-min-exan mxieee
0.77 #0.004
0.67 #0.018
0.48 +0.004
0.54 =0.006
0.69 =0.006
Logits-max
0.53 1:0. 00e
0.51 10.011
0.52 1: 008
0.59 008
0.63 20.011
Logits-max-exact
0.51
0.41
0.49
0.64 00.008
0.63
Probas-mean
0.63 1:000 003
0.56 010
0.58 1: 005
0.62 1: 00s
0.68 00.010
Probas-mcan-cxact
0.72 1:0. 008
0.66 11.010
0.46 1: 004
0.57 1: 003
0.65 10.008
Probas-min
0.58
0.52
0.59
0.58
0.65
Fribas-min-xxact
0.76 1:000 004
0.68 11.010
0.46 1:.000 10.005
0.57 1: 003
0.66 20.008
Probas-max
0.50 +0.005
0.53 +0.003
0.48 +0.007
0.52 00.007
0.51 =0.005
Probas-max-exact
0.46 1:000
0.46 110010
0.48 10
0.53 10.004
0.52 20.018
p(True)
0.54 00.006
0.54 +000004
0.53 +0.003
0.58 =0.003
0.57 00000
P(True)-exact
0.60 +0.008
0.48 +0.000
0.57 +0.011
0.65 =0.004
0.57 000009
Probe @ token
Last generated [-1]
0.72 +0.005
0.64 #0.000 005
0.74 +0.005
0.85 =0.004
0.82 00.006
Before last generated [-2]
0.73
0.64
0.76 1..
0.87
0.84
End of question
0.80 +0.003
0.63 #0.000
0.71 +0.007
0.79
=0.004
0.85 =0.010
Exact answer last
0.85 1.
0.75 008
0.84 00.005
0.93 1.0.003
0.86 10.003
Exact answer last+1
0.85 #0.002
0.76 #0.004
0.80 +0.004
0.92 =0.004
0.87 +0.006
25

---

Published as a conference paper at ICLR 2025
Table 6: Comparison of error detection performance (AUC) on Llama-8b.
Llama-8b
TriviaQA
Winobias
Math
Movies
IMDB
Logits-mean
0.58 +0.006
0.44 #0.015
0.43 #0.026
0.64 +0.008
0.77 +0 00.007
Logits-mcan-cxact
0.63 10.007
0.50 10.015
0.50 028
0.64 10.008
0.77 1.0 007
Logits-min
0.75 +0.0077
0.50
0.45 042
0.73 = +0.005
0.73 +0
Logits-ma Logits-min-exact
0.76 #0.000
0.53 #0.009
0. +00022
0.73 +0.006
0.77 +: .07
Logits-max
0.48
0.48
0,42 027
0.53
0.72 1.0
Dogis--ma-exxxe
0.52 #:0 007
0.49 #0.014
0.35 + 022
0.53 +: .016
0.77 4:0 007
Probas-mean
0.64 1:0 006
0.41 1: 008
0.61 1:0, 029
0.71 1:0.007
0.70 1:0 008
Probas-mean-exact
0.72
0.50
0.54 D26
0.72 #:
0.88 4:0
Probas-min
0.79 #000008
0.43 #0.004
0.75 +
0.74 +: t00000
0.68 +00006
Probas-min-exact
0.82
0.53 10.014
0.78 022
0.74 1:
0.88 1:
Probas-max
0.49 +:00006
0.50 +0.008
0.46 +0.082
0.53 +: 00007
0.60 +00009
Probas-max-exact
0.53 1: 008
0.50 10.018
0.36 1:0 032
0.54 10.007
0.88 1:0 0.0.008
p(True)
0.62
0.48
0.53
0.61
0.51
P(True)-exacl
0.67 =: 002
0.53 20.017
0.63 1:0 028
0.58 10.000 00.
0.52 1: 00.
Probe token
Last generated [-1]
0.77 #0.006
0.59 #0.024
0.83 #0.013
0.82 4: 100.005
0.94 4:0 1000002
Before last generated [-2]
0.76 +0.012
0.58 +0.021
0.82 +0.082
0.79 +00.004
0.96 +00.002
End of question
0.73 005
0.77 20.012
0.80 1:0 027
0.78
1: 000
0.68
1:0 009
Exact answer last
0.82
0.91
0.96 =0.010
0.80
0.97
Exact answer last+1
0.82 +000006
0.86 +000008
0.95 +0.007
0.82 +0,006
0.95 +00008
HolpolQA
HolpoiQA-We WC
Winogrande
NLI
NQ-WC
Logits-mean
0.65 00.
0.62 20.006
0.48 1:0003
0.47 10.002
0.53 1:0 10.010
Logits-mean-exan
0.55 +0.008
0.54 +0.006
0.49 +00004
0.48 +:0.002
0.58 +:0.009
Logits-min
0.57 =: 20.004
0.49 20.003
0.48 1:0. 003
0.48
1: 007
0.58 1:0 009
Logits-min-exen
0.69 +
0.68 +0.0006
0.49 +to.
0.48 #000007
0.61 +0.010
Logits-max
0.61
0.60 10
0.48 000
0.52
0.51 1:0
Logils-max-exact
0.47 =: 003
0.46 20.005
0.49 1:0 004
1: 0000002
0.54 1: 006
Probas-mean
0.67 +-0.002
0.62 +000006
0.49 +
0.48 +00.004
0.57 +-0.008
Probas-mean-exact
0.62 00s
0.56 005
0.51 1:0 002
0.46 1 006
0.64 1: 007
Probas-min
0.62 +00.006
0.51 +000002
0.49 ++0.003
0.50 +000100
0.62 +00.005
Probas-min-exact
0.76 005
0.67
0.51 1 10.00 002
0.50
0.69 1:0
Probas-max
0.61 #-0.004
0.58 +000004
0.48 40.000
0.48 + 60.003
0.51 4-0.012
Probas-max-exes
0.49 40.003
0.44 +000004
0.51 4.0.003
0.47 +:0.002
0.56 +-0.005
p(True)
0.52 007
0.45 20.005
0.54 100 004
0.54 =
0.56 1: 006
p(True)-exact
0.58 000000
0.50 4-0.007
0.64 4.0.004
0.62 4.0.00
0.61 4:0.002
Probe token
Last generated [-1]
0.76 20.007
0.57 #0.006
0.59 40.006
0.89 20.002
0.66 ::0 2000000
Before last generated [-2]
0.74 +0.007
0.58 +00.005
0.59 4.0.005
0.94 +.0.002
0.63 4-0.008
End of question
0.71
#0.006
0.53
20.004
0.448 10. 008
0.91
20.001
0.66
1:
Exact answer last
0.81 006
0.77
50.004
0.65 =0.004
0.94 10.
0.75
Exact answer last+1
0.82 4.0.004
0.79 4.0.001
0.57 4.0.004
0.90 0.00002
0.75 4-0.007
26

---

Published as a conference paper at ICLR 2025
Table 7: Comparison of error detection performance (AUC) on Llama- (lama-8b-ia Sb-Instruct.
Llama-8b-Insten Ih--ruttuct
TriviaQA
Winobias
Math
Movies
IMDB
Logits-mean
0.66 #0.005
0.60 #0.026
0.75 #0.018
0.75 20.005
0.59 #0.017
Logits-mcan-cxact
0.71 10.006
0.55 10.019
0.80 10.021
0.72 10.004
0.88 10.012
Logits-min
0.74 +0.0077
0.61
0.75 +0.016
0.71 =
0.55
Logits-min-exact
0.79 #000006
0.61 #0.019
0.89 +0.018
0.77 #0.006
0.88 +0.012
Logits-max
0.54
0.55 10.013
0.73 027
0.67 1:0
0.51 1:0
Dogis--ma-exxxe
0.58 +:0 100.006
0.54 #0.018
0.64 40.014
0.61 +: 001
0.88 +:0 210.012
Probas-mean
0.67 006
0.63 10..22
0,66 1:0 053
0.73 1:0 006
0.73 1.0.015
Probas-mean-exact
0.75
0.61
0.83
0.74 + 0000
0.74 +:0
Probas-min
0.67 #:0 0019
0.65 +0.019
0.64 +00
0.65 + +0.004
0.57 +0.016
Probas-min-exact
0.79
0.62 10.014
0.86 024
0.74 10.005
0.74 1:0
Probas-max
0.54 40.008
0.49 #0.020
0.57 +0.022
0.64 +0.000
0.49 +:0 t00008
Probas-max-exact
0.56 007
0.55 10.016
0.57 10.018
0.61 10.003
0.74 10.021
p(True)
0.73
0.59
0.62 40.017
0.66
0.60
P(True)-exacl
0.73 =: 005
0.63 20.014
0.59 10.018
0.63 1: 006
0.76 1: 004
Probe token
Last generated [-1]
0.81 +0.006
0.86 +0.000
0.82 40.016
0.78 +: 100000
0.81 +0.004
Before last generated [-2]
0.75 +0.0005
0.88 +0.000
0.79 +0.020
0.82 +:0 +00.006
0.83 +00.006
End of question
0.77 +0 20.007
0.80 20.016
0.72 1:0 023
0.76
1: 000
0.87 1:0 006
Exact answer last
0.83
0.93
0.95
0.85
0.96
Exact answer last+1
0.83 +000006
0.90 000.005
0.94 +0.028
0.86 +0,004
0.95 +0.004
HolpolQA
HolpoiQA-We WC
Winogrande
NLI
NQ-WC
Logits-mean
0.65 002
0.56 10.004
0.58 1:0000
0.59 10.009
0.65 1:0 006
Logits-mean-exan
0.66 +0.000
0.57 +0.005
0.48 +00000
0.49 #:0010
0.67 +00.000
Logits-min
0.67 006
0.55 20.007
0.60 1:0. 008
0.53 1: 009
0.68
1:0 004
Logits-min-exen
0.76 +100010
0.65 +0.011
0.48 +t0.004
0.50 +00000
0.68 +0.004
Logits-max
0.59
0.56 20.005
0.46 000
0.55 1:
0.56 1:0 .066
Logils-max-exact
0.52 20.006
0.48 20.002
0.48 1:0 003
0.49 1: 009
0.63 1: 008
Probas-mean
0.61 +-0. 002
0.56 +0.0100
0.57 ++0.007
0.58 +:0 0.0077
0.65 +-0.007
Probas-mean-exact
0.68 008
0.65 006
0.51 1:0 008
0.57
= .09
0.67 1: 008
Probas-min
0.60 400.004
0.51 +000007
0.59 ++0.D0T
0.55 t00000
0.64 +-0.008
Probas-min-exact
0.74 20.007
0.67 20.007
0.5l 10.000
0.59
0.66 1:0
Probas-max
0.56 40.000
0.53 #00.005
0.46 +00000
0.51 4000004
0.55 4-0.004
Probas-max-exes
0.49 +-0.007
0.47 4:0.002
0.51 4.0.005
0.50 +
0.62 4:0.006
p(True)
0.55 005
0.55 +0.008
0..77 10. 002
0.54
= .06
0.71 =: .08
p(True)-exact
0.55 00.000
0.50 4:0.005
0.50 400.00
0.50 1.0.00
0.67 4-0.007
Probe token
Last generated [-1]
0.77 #0.005
0.68
20.006 00.
0.69 4:0.000 006
0.78 20.005
0.77 4:0 0000009
Before last generated [-2]
0.76 +.0000
0.69
400.005
0.67 4.0.008
0.79 +.0.004
0.75 +-0.007
End of question
0.78 +0.004
0.60
20.003
0.65 1:0 004
0.74
20.002
0.75
1:
Exact answer last
0.83 20.005
0.76 20.003
0.78 =0.007
0.91 20.00
0.78 20.000 40.000
Exact answer last+1
0.83 4-0.000
0.76 4-0.006
0.70 4:0.000
0.90 +:0.004
0.78 4-0.007
27

---

Published as a conference paper at ICLR 2025
Trviig4
5.54
6.64
0.73
0.50
0.54
0.00
0.0 <0.00
0.02
003
4.02
062
0.00
10.11
0.25
K0C2400A
0.77
0.89
0.72
053
0.53
0.52
0:60
0.59
0.63
0.69
9.01
0.3
0.8
Movids
0.12 6.35
0.00
0.02
003
6.00
9.23
0.09
033
Train dataset
Winobiia
037 0.8
0.5
0.29
0.5
040
0.38
Train dataset
4.57
0.30
0.00
Wenogrande--
0.52 0.53 0.55
0:55
0.64
0.53
0.52
40-1
082
4.01
01
0.22
6.29
0.33
0.0
0.00
0.53
0.24
9.05
0.32
0.00
MDB-0.0 0.58
0.57
963
0.54
0.52
0:5
0.70
0.55
0.50
0.50 0.64 936 0:7
0.00
Noth
0.22
4.69
60:77
064
0.2
0.02
0.17
0.10
4.30
0.00
-0.1
rotporte_
0.30
0.00
40, 10,WC-- -0.02
0.67
0.54 0.00 0.52
0.50
940
051
3.50
0.04
WC
0.00
10.38
0.03
0.10
27.00.0
0.0
Nones
Hath:
Hocgsto WC
N80 WC
1400ICA
Movies
NCB
We
p0c000 WC
Test dataset
Test dataset
(a) Raw AUC values. Values above 0.5 indicate some
(b) Performance (AUC) difference of the probe and the logit-based method. Values above indicate gen- eralization beyond the logit-based method.
generalization.
Figure 7: Generalization between datasets, Mistral-7b.
TeviKq4 9.00 0.01 ouee
0.0 0.04
00.13
0.07
(0.38
0.03
3.76
0.82
0.70
0:5
055
051
3:59
0.75
0.63
0.55
HktiplQK
0.000 0.32
6.63 841
0.44
00.0
0.00
0.8
Moviss 0.70
0.58
0.22
D60
0.50
Movies 06 4.31
0.08 006
0.36
Train dataset
Winobils
Train dataset
Wei Wirii.koo--
0.09 0011 0:57 0.04 0.04
0.03
0.01
0.31 0.05
0.2
Winegraniie--
2.51
035
0.00
545
0.65
0.62
0:58
334
0.50
0.5
WWnograndd
033
0:2
837
932
NRJ 4.17 9035
00e 0.04
0.08
9.39
00.7
0.97
A3
8400
0.00
0.53
0.62
8:00
0.52
8.12 6:2
0.05
000
Nath
0.62
0.53
0.57 0.1 0,51 0.51
3:74
0.96
034
0.56
Wath-0.15 0.15 6.37 4.36 0.0 0.00
0.03
0.1
0.14 0.05
Hemper
Hespato0 JNC. 4.00
0.00
.WC.
0.00
0.58
ned
b:88 0:5
034
Noate
Math
NCC
Iice
Nath-
wn0000 WCC
NO.
NO
Test dataset
Test dataset
Raw AUC values. Values above 0.5 indicate generalization.
(b) Performance (AUC) difference of the probe and the logit- method. Values above indicate gen- eralization beyond the logit-based method.
Figure 8: Generalization between datasets, Llama-3-8b.
C FULL GENERALIZATION RESULTS
present the generalization results for the remaining models. While these results exhibit similar lege-level patterns to those found in the main paper on Mistral--b-instruct, notable differences suggest that these models may possess different mechanisms for encoding truthfulness.
Figures 7
g and
D TAXONOMY OF ERRORS
Figure 10 presents, for each amount of resamples, the amount percentage of answers for which at least one generated answer was correct. The experiment was done on Mistral-7b-instruu with the TriviaQA dataset. For many answers that the greedy decoding fails correctly provide answer, the LLM still able to generate the correct answer in at least one resample. The plot plateues around 30 resamples.
28

---

Published as a conference paper at ICLR 2025
TrveGk
3.04
0.74
A.1
0:74
0.5
0.59
3:50
0.01
0.59
0.70
TINDON
4.00 -0.07
000
4.05
+0.09
002
HOC2000K
0.88
0.22
033
HATDOROK
0.37
9.04
Noviec
0.66
0.04
0.03
0.7
0:53
0.52
0.63
Movita
0.00 0.87
0.04
0.56
000
0.15
0.37
0.35
0.05
Train dataset
2.57
0.59
841
0:13
052
033
3.70
033
0.22
0.38
Train cictacet
0.0
Winegranse
Wenggrndde
66.00
4.10
011
0.07
40
01
0.55 0.03
0:1
0:3
0:57
0.52
0.30
0.27
0.02
0.00
0.35 0.10
MDB-0.88
0.55
0.00
0:65
0.70
0:7
0.55
0:0
0:5
0.63
I400
0.78
4.17
048
0.20
0.06
Meth
4.89
4.02
0.01
0.31
0.05
4.00 0.10
HOGp0rJA,WW IWC 0229
0.72
925
0395
NO IMC
1073
677
8.00
0:70
0.00
0.50
NO IVC
-0.00
4.05
10.30
0:10
0.25 0.04
0.00
0.30
00.33
0.20
7W000
Moves
Math
PaGGOOR WC
NO WC
Noves
RCB
HO000O WC
ND
Test dataset
Test dataset
(a) Raw AUC values. Values above 0.5 indicate some
(b) Performance (AUC) difference of the probe and the logit-based method. Values above O indicate gen- eralization beyond the logit-based method.
generalization.
Figure 9: Generalization between datasets, Llama- 11-8-biiittrctt
0.85
0.800
Curcetmnc
0.775
0.753
0.100
Figure 10: The percentage of answers for which at least one generated answer was correct. The first step greedy decoding.
29

---

Published as a conference paper at ICLR 2025
Table 8: AUC scores for error type classification (Winobias).
Error type
Mistral-7b Mistral-Instr-7b
Llama3-8b
Ilamm3-Instre
(A) Refuses answer
(B) Consistently correct
0. 0.83 oos
0.88
0.842
0.89
(C) Consistently incorrect
0. 0.8322.000 oos
0.88 go. 002
0.79 70.0.001
0.90 003
(D) Two competing
0.68 004
0.58
0.74
0.88
(E) Many answers
Table 9: AUC scores for enror type classification (Math). Error types are predictable from the inner model representations, indicating the encoding of fine-grained information on errors.
Error type
Mistral-7b
Mistral-Instr-P7
Llama3-8b
Llama3 Instr-8h
(A) Refuses to answer
(B) Consistently correct
0.85 0.852o.om
0.84
0.83 c20
0.87 10
(C) Consistently incorrect
0.85+
0. +o.0s
0.69+a.0s
0.91 +.1.0..om 0.000
(D) Two competing
0. 0.76...028
0,57
0.79+a
(E) Many answers
0.741..000
2.791
0.69
0.90 o0e
D.1 ERROR TAXONOMY DESIGN CHOICES
The eтror taxonomy proposed in this paper is intentionally non-orthogonal, some emrors may simultaneously belong to multiple categories. For instance, an error might fall under both "con- sistently incorrect" (e.g., the same incorrect answer appears at least 15 times) and "many different answers" (e.g., the remaining answers show over 10 distinet variants).
Our taxonomy is designed to capture such nuanced cases, restricting classification to single cate- gory would hinder the generalizability insights. Instead, we aim learn general properties across different error types, providing LLM providers with actionable insights into questions exhibiting overlapping error patterns.
To support this non-orthogonal framework, probes function one-to- many classifiers, enabling precise error analysis and tailored solutions.
D.2 RESULTS ON ADDITIONAL DATASETS
Table g presents the results of error type classification on the Winobias dataset and Table 9 on the Math dataset.
D.3 QUALITATIVE EXAMPLES
Tables T0 and present qualitative examples of the error types in the TriviaQA and Math datasets.
E DETECTING THE CORRECT ANSWER FULL RESULTS
In Table 12 we present some qualitative samples from Distral-7b-instrure for the phenomenon observe at error type (C2) Consistently incorrect but generates the correct answer least one time. The samples in the table represent cases where the probe chose the correct answer. Table 13 com- pares different decoding mechanisms, including the choice via probe, on non-instruct models, and Table 14 compares the instruct models. For all datasets and models, observe similar conclu- sions to those in the main paper: significant improvement is observed for error types where the LLM shows preference to the correct answer.
30

---

Published as a conference paper at ICLR 2025
Table 10: Examples of error types in TriviaQA, Mistral-7B-Instrrutt. Correct answer bold.
Type of error
Question
Answers
What clothing-pan metaphorically classifies workers/jobs according to white or blue?
Consistently
"collar": 30
correct
Which town southeast Wales became UNESCO World Heritage Site in 2000?
Consistently incorrect
"Blaenavon", 1,
"Caerleon": 29
"Frederick Forsyth": 1,
"Jerry Patterson": 1,
"Edward Lee": 1,
"Barry Lancet":
"Jeremy Holiday": 1,
"Barry Lincoff": 1,
"Jim Marss" 1,
"John Marrs": 1.
"Anthony Lacy": 1.
"Daniel Kraus": 1,
"Ron Bass": 1,
Many different answers
Published in 2013 who wrote the novel 'The Kill List"?
"David Martiniello": 2,
"Eric Lustbader": 1,
"Barbie Latza Nadeau": 1,
"James Swallow": 1,
"Mark Sullivan": 1,
"Alex Binotto":
"David Baldacci": 1,
"Bill Cosores": 1,
"Frederie J. Brown": 1.
"Ron Capps and Tate Foley": 1,
"Barbie Wilde": 1,
"NO ANSWER": 3
What is the only letter of the alphabet which does not appear in any of the names of the 50 American states?
"The letter q": 15,
Two competing answers
"The letter X": 15
F PRACTICAL GUIDANCE ON INTEGRATING INSIGHTS FROM THIS PAPER INTO MODEL DEVELOPMENT WORKFLOWS
The findings of this study reveal critical insights into the internal mechanisms of Large Language Models (LLMs) and their implications for truthfulness and error handling. To effectively incorporate these insights into model development, consider the following strategies:
Error Detection. Focus on representations of exact answer tokens to train the error detection probe. These tokens encode significant truthfulness signals and improve the reliability error detection mechanisms. The trained probe should be integrated as part of the pipeline for specific task, e.g., math calculations. The probe provides a confidence score which can used to warn the user for unreliable outputs, to perform intervention fix the answer.
Error-Specific Interventions. The taxonomy of errors outlined in this study can be utilized to classify and analyze the types of errors that an LLMs may produce. Identifying these error types is useful for customizing strategies for error mitigation. The probes for detecting error types can be de- ployed as part of the LLM pipeline and create interventions based on their predictions. For example, Retrieval Augmented Generation (RAG) (Lewis al. 2020) can help for "consistently incorrect"
31

---

Published as a conference paper at ICLR 2025
Table 11: Examples of error types in Math, Mistral-7B-Instruet, Correct answer is in bold.
Type of error
Question
Answers
If John travels 15 miles on bike ride, and Jill travels 5 miles less, how many miles does Jim travel if he travels only 20% far Jill?
Consistently
"2": 30
correct
Joy has 30 pencils, and Colleen has 50 pencils. If they bought the pencils S4 each at the store. how much more money did Colleen pay than Joy for her pencils?
Consistently incorrect
"80$": : 1, "16S": 29 (correct)
"95": 14,
"91": 1.
"87": 1,
"15": 2,
"96": 1,
"Six":
"202 "
If the first skyscraper built 100 years ago, how many years in the future will it be years before its 200th anniversary of being built?
"2035":
Many different answers
"195": 1,
"49": 1.
"101": 1,
"199": 1,
"3 years before the
200th anniversary": 1,
"203 years after it
built": 1,
"196": 1,
"2043": 1
David did 27 more push-ups but 7 less crunches than Zachary in gym class today. If Zachary did 5 push- ups and crunches. How many more crunches than push-ups did Zachary do?
Two competing answers
"12":5, "1": 5
errors, as well as resampling and choosing the answer ranked highest by the error detection probe, weight-update, if possible, more consistent solution. For "consistently correct" error types, an intervention on the LLM's internal representations can increase the confidence in generating a correct answer (Simhi et al. 2024).
Cross-Tase Generalization. Universal generalization of probing classifiers across unrelated tasks should be approached with caution. The results in this work show that probes are mainly useful for task-specific error detection.
32

---

Published as a conference paper at ICLR 2025
Table 12: Examples of questions where Mistral-7b-Instruee consistently provided incorrect answers but occasionally generated the correct one. these instances, the probe successfully identified the right answer. For each question, the model was sampled 30 times.
Wrong Answer
Correct Answer
Question
Count
Count
Which town in southeast
Wales became UNESCO
Caerleon
29
Blaenavon
World Heritage Site in 2000?
From her first US film musical "Down Argentina Way" (1940), who became famous for extravagant hats, jewellery and dresses?
Betty Grable
Carmen Miranda
27
Men Against the Sea and Pitcairn's 's Island were two sequels to what famous novel?
Robinson Crusoe
Mutiny on the Bounty
18
2
Which is the only property on traditional UK Monopoly board which south of the River Thames?
Coventry Street
Old Kent Road
17
3
Which French Canadian became Prime Minister of Canada in 1968?
Pierre Elliott
Jean Chrétien
21
4
Trudeau
Table 13: Various answer choice strategies, non-instruct models.
Mistral-Th
Math
Winshtas
Ezroe
Gooody
Randon
Manciiy
Probing
Gieady
Rundoro
Maioiiy
Pooking
Gresdy
Randoon
Majerit
Probiux
0.50 sxoms
0:6b pooes
soow
sears
(81)
(82) Mist
0.96
0.86
1,00
0.91
0.00
0U84 mam
1.00
0.90
0.08 scace
0.73
0L00
0.91
(C) Conesically inconcet
(C7)
0:50
0.00 a.mm
0.00 anow
soow
(D) Two
0,59
(E) Mar anieen
(E contect
0.00 wause
0.00 aooes
0.00 souus
0,00 axcoa
0.00 wa.a0
0U00 waaso
0.00 aaoeo
D.00
(F2) Cormen aprscam
soowe
0130 ssem
Llamo-tb
TAvisDA
Maih
Winshias
type
Sumping
Mopeity
ProNing
Deraty
Sempling
Mayerty
Prcking
Grrdyy
Sumpling
Majertty
Proking
All
D,50
0.58
9.68 suun
8.65 auoo
0.20
D.47 anim
0.02 sxms
1.70
0.73
B.G
B,G
0.54
(A) Refracs unoser
0.66 ssuse
0L00 sxeca
0.00 sooe
0.66 soies
0.00
0.00 cwass
0.00 saneo
0.20
(Bi Conennienlly onrrec
Most
73
(C) Consiszzally inconee
(C13 All
8,00 wxcas
0.00 auees
0.00 soum
0,00 eocas
Mood
8:06 aaris
0:00 gnow
peacs
compeiing
(ET) Non connet
0,00
0.00
0.00 soues
0,00 wuoe
0.00 aa.oo
0.00 waae
0.00 aaeo
D00
(F2) Corret apriccec
0.16 po/oen
0.5% gxves
0.55 soeen
0.54 mooe
0.14 13332ass
[.17 2aas
0.65 sour
33

---

Published as a conference paper at ICLR 2025
Table 14: Various answer choice strategies, instruct models.
Hletral-Th-lnetrut
TriviaQA
Math
Winsbias
Ecroc type
Cooods
Randens
Muncciy
Probieg:
Cieody
Rusdon
Maioriy
Pooking
Greedy
Rasdion
Majotiy
Probiux
axom
axarz
Refises
0.00
(B1)
1.00
1.00
1.00
1.00
1.00 sxaeo
1.00
1.00
1.00 0.00 017
(B2) Mont
0.58 esuur
0.83 suee
0.50 soues
0.50 poice
0.89 as
0.84 ma
1.00 saamo
0000 eaarr
091 sea
0.87 28.
0.80
(C' Concismntly incomeet
Most
(D) Two competing
0.52
0.40
0.50 eours
0.78
(E) Marr aniverss
(11) zomaat
8:01
apppees
Liums- Ib-Instucct
TrivoQA
Winshiae
Fernr
Grody
Sampling
Migiity
Sempling
Mijotty
Gredy
Sampling
Majedty
Proting
All
0,50 swow
0.67 sioes
0.71 gooes
8.75 poow
0.90 so
8.91 goera
0.91 pooe
0.75 sears
0.24 ae
(A] uncoer
swass
snow
pouse
Concissenly
(81)
(B2) Must
0.96
0.90
1,60
0,52
0.94
0.92
1.00
0U0
0.94
0.88
0.53
(C) Conssically inconcet
(1T))
0:00 pa
0.00 Vom
0.00 gn
0.00
0.00
(D)
(E) Mar anisers
(RT) Non zomaa
0.00 saiies
0.00 sivees
0:00 snow
0.00 soow
(E2) Correc appeare
0.25 Soae
govorr
snow
34


Reviews:
[{"review_id": "JpPatFRyWw", "reviewer": "Reviewer_kD9V", "summary": "This paper examines the internal representations of large language models (LLMs) to understand how they encode truthfulness and produce hallucinations—errors such as factual inaccuracies and reasoning failures. Through an approach involving probing classifiers, the authors identify truthfulness signals in specific tokens, particularly in exact answer tokens. Experiments across multiple LLMs reveal that truthfulness is task-specific, with signals varying by the nature of the task. Additionally, the study develops a taxonomy of error types, finding that LLMs may encode the correct answer internally while producing an incorrect one externally, suggesting areas where error mitigation could be applied.", "questions": "1. Have the authors tested if the taxonomy of error types is consistent across various domains or tasks, and if not, would this be feasible in future work? \n\n2. Could the authors explain why certain model layers are more effective for probing in error detection? A discussion on this might enhance understanding of how internal representations encode truthfulness.\n\n3. Have the authors considered using a context window around answer tokens to improve detection accuracy, especially for tasks that involve longer responses or require more context? \n\n4. Given the diversity in model architectures tested, are there certain architectures where the probing method performs noticeably better or worse?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- By probing specific tokens associated with “correct” answers, the authors highlight underexplored dimensions of LLM truthfulness encoding. This approach captures nuanced error signals that extend beyond surface-level accuracy. \n\n- The paper demonstrates robustness by testing the approach on a variety of LLM architectures (e.g., Mistral-7b, Llama3-8b). This extensive model comparison strengthens the study’s findings by showing consistency in truthfulness encoding patterns across different architectures. \n\n- The detailed taxonomy of error types, including categories like “two competing answers” and “consistent incorrectness,” could help model developers diagnose and prioritize error types during model fine-tuning. \n\n- The study’s finding that truthfulness encoding is not universal but rather task-specific introduces an important consideration for cross-task applications.", "weaknesses": "- While probing classifiers show strong results, the interpretability of their outputs remains unclear. For example, understanding how specific probing layer choices affect error predictions or how developers might interpret these signals is crucial for practical adoption but is not fully addressed. Including examples of how different layers and token choices impact classifier predictions would clarify the interpretability of this approach. A case study showing how interpretability of classifier outputs could inform model improvements would also be beneficial. \n\n- The error taxonomy could be more robust if supported by quantitative analyses on how consistently each error type appears across varying prompts or domains. Currently, it is unclear if the identified error types remain stable under different experimental conditions or across datasets. Conducting further experiments to determine if error types remain consistent under prompt variation or across domains (e.g., factual vs. commonsense tasks) could enhance the reliability of the taxonomy. These results would provide stronger support for the taxonomy as a practical tool. \n\n- The framework’s token-level focus may overlook broader context, potentially missing how errors manifest within larger segments of generated text. For tasks like long-form QA or summarization, this might limit the approach’s utility by not capturing interactions between tokens. Testing whether incorporating context windows around answer tokens, or analyzing sentences rather than isolated tokens, would improve accuracy could make the approach more adaptable to contextually rich tasks. Including examples where broader context improves error prediction would also strengthen this aspect. \n\n- Practical guidance on integrating these insights into model development workflows is limited. Developers may find it challenging to apply these techniques without clear implementation guidelines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "moFv3wnbBz", "reviewer": "Reviewer_636k", "summary": "This paper proposes that the internal representations of LLMs' responses are predictive of error patterns. Unlike previous approaches focusing on representations of last token or averaged representations, this paper focuses on representations of exact answer tokens.", "questions": "- This approach could be only applicable to QA tasks. For open-ended questions, can it be applied? It would be better to provide performance on these kinds of tasks as well.\n- The results in table 1 show that the AUC of last generated almost match that of exact answer tokens with TriviaQA. Does this imply that actually the representations of last token and answer exact token are almost the same, especially for more general hallucinations rather than specific factual ones such as math problems? Besides, the result of Mistral is worse than that of LLaMA 3.1. Is it because the answer of Mistral involves less explanations to the question than that of LLaMA 3.1? The results in table 2 seems also demonstrate that answer exact tokens are barely better than last tokens when it comes to more generic tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This paper interprets halluciantions from the perspective of token representations, particularly the exact answer tokens, which provides a relatively new perspective.\n- This work demonstrates that the internal representations of LLMs' responses are skill specific rather than universal.\n- The paper is well-structured and the experiments are comprehensive to demonstrate the validity of the findings.", "weaknesses": "This paper has generated some interesting findings. However, the method is hardly to be taken as general.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U3aosWQX97", "reviewer": "Reviewer_sDnt", "summary": "This paper presents a novel approach to investigating hallucinations in LLMs by using probing classifiers to analyze their internal representations. The approach involves analyzing internal model representations to determine how LLMs handle factual versus hallucinatory information. The proposed approach is evaluated through multiple experiments on various LLMs and datasets, offering new insights into understanding and mitigating hallucinations in these models.", "questions": "- It would be better to clarify the specific model used for the probing classifier and provide more details on the training setup (e.g., hyperparameters, optimizer, detailed dataset split information).\n- From the experimental results presented in Section 4, it seems that the generalization ability of the probing classifier across different tasks is relatively limited. For instance, many AUC scores are in the range of 0.5 to 0.6, which suggests that the classifier's performance is only slightly better than random guessing. Could you elaborate on possible reasons for this limited generalization, and suggest any potential ways to enhance it?\n- Have you considered conducting generalization tests across different LLMs, in addition to testing across different datasets? For instance, how would a probing classifier trained on output generated by Mistral-7b perform when tested on Llama3-8b? This approach could provide better insight into whether the probing classifier remains effective when applied to activation states across different LLMs.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-structured, and the motivation is clearly communicated.\n- The experiments effectively demonstrate how probing classifiers reveal the internal mechanisms of LLMs.\n- The results highlight significant discrepancies between internal representations and generated outputs, offering a deeper understanding of how hallucinations manifest and potential ways to mitigate them.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f9bGyI2iVp", "reviewer": "Reviewer_ksBC", "summary": "This paper conducts extensive experiments and verifies through probing that LLMs possess an implicit understanding of truthfulness within their hidden representations. However, due to certain calibration issues, even though the model \"knows\" the truth, it can still produce incorrect outputs. Additionally, further experiments show that probing is not a general-purpose tool for detecting hallucinations; it is task-specific.", "questions": "- **On p.3, Section 3.1**: Which model, $M$, did you use to generate responses when constructing the triplet dataset?\n- **On p.5**: What is the accuracy of Mistral-7b-Instruct when extracting the exact answer token? Or did you assume that all extracted tokens are correct without further validation?\n- **On p.5, fig. 2**: The statement “Middle to later layers typically yield the most effective probing results” seems inaccurate. The figure does not show a clear pattern supporting this claim.\n- **On p.8**: Why were the error patterns not designed to be orthogonal? Could you provide a definitive reason for this decision?\n- **On p.10, fig.5**: What do the different colored bars represent in the figure? Please clarify the meaning of the different colors.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Originality: The understanding of hallucination is accurate, and the exploration of the model’s self-knowledge capabilities provides deeper insights. Perspective of shifting focus from human-centric to model-centric is pinpoint and insightful.\n- Quality: The experiments are thorough and well-executed.\n- Clarity: The paper is structured clearly, making it easy to follow.\n- Significance: This work is valuable for future research, especially for studies focusing on how models internally grasp factuality.", "weaknesses": "- **On the definition of hallucination**: The paper's definition of hallucination remains somewhat vague. As discussed in [this paper](https://arxiv.org/abs/2407.14507), hallucination can be understood as *inconsistency* between a model’s expression and world knowledge it has learned from its training corpus.\n\n- **Missing experiments**: On p.6, the paper states that “the primary conclusion is that information about truthfulness is highly localized in specific generated tokens.” Given this claim, I believe an additional experiment is necessary: measuring the accuracy of the model when prompted to produce exact answers under restricted conditions. This would provide more direct evidence supporting the localization of truthfulness in generated tokens.\n\n- **Lack of clear direction for next steps**: The paper does not explicitly outline what future steps can be taken. If we now accept that \"the model latently and task-specifically knows what truthfulness is,\" how can we leverage this finding to help a general-purpose LLM handle unpredictable tasks? Or, how can this work help researches avoid futile efforts? The practical applications of these insights are not clearly addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)