LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations

## Abstract

Abstract Large language models (LLMs) often produce errors, including factual inaccuracies, biases, and reasoning failures, collectively referred to as “hallucinations”.
Recent studies have demonstrated that LLMs’ internal states encode information regarding the truthfulness of their outputs, and that this information can be utilized to detect errors.
In this work, we show that the internal representations of LLMs encode much more information about truthfulness than previously recognized.
We first discover that the truthfulness information is concentrated in specific tokens, and leveraging this property significantly enhances error detection performance.
Yet, we show that such error detectors fail to generalize across datasets, implying that—contrary to prior claims—truthfulness encoding is not universal but rather multifaceted.
Next, we show that internal representations can also be used for predicting the types of errors the model is likely to make, facilitating the development of tailored mitigation strategies.
Lastly, we reveal a discrepancy between LLMs’ internal encoding and external behavior: they may encode the correct answer, yet consistently generate an incorrect one.
Taken together, these insights deepen our understanding of LLM errors from the model’s internal perspective, which can guide future research on enhancing error analysis and mitigation. 1 1 1 Our code is available in https://github.com/technion-cs-nlp/LLMsKnow .

## 1 Introduction

The ever-growing popularity of large language models (LLM) across many domains has brought a significant limitation to center stage: their tendency to “hallucinate” – which is often used to describe the generation of inaccurate information.
But what are hallucinations, and what causes them?
A considerable body of research has sought to define, taxonomize, and understand hallucinations through extrinsic, behavioral analysis, primarily examining how users perceive such errors (Bang et al., ; Ji et al., ; Huang et al., ; Rawte et al., ).
However, this approach does not adequately address how these errors are encoded within the LLMs.
Alternatively, another line of work has explored the internal representations of LLMs, suggesting that LLMs encode signals of truthfulness (Kadavath et al., ; Li et al., ; Chen et al., , inter alia).
However, these analyses were typically restricted to detecting errors—determining whether a generated output contains inaccuracies—without delving deeper into how such signals are represented and could be leveraged to understand or mitigate hallucinations.

In this work, we reveal that the internal representations of LLMs encode much more information about truthfulness than previously recognized.
Through a series of experiments, we train classifiers on these internal representations to predict various features related to the truthfulness of generated outputs.
Our findings reveal the patterns and types of information encoded in model representations, linking this intrinsic data to extrinsic LLM behavior.
This enhances our ability to detect errors (while understanding the limitations of error detection), and may guide the development of more nuanced strategies based on error types and mitigation methods that make use of the model’s internal knowledge.
Our experiments are designed to be general, covering a broad array of LLM limitations.
While the term “hallucinations” is widely used, it lacks a universally accepted definition (Venkit et al., ).
Our framework adopts a broad interpretation, considering hallucinations to encompass all errors produced by an LLM, including factual inaccuracies, biases, common-sense reasoning failures, and other real-world errors.
This approach enables us to draw general conclusions about model errors from a broad perspective.

Our first step is identifying where truthfulness signals are encoded in LLMs.
Previous studies have suggested methods for detecting errors in LLM outputs using intermediate representations, logits, or probabilities, implying that LLMs may encode signals of truthfulness (Kadavath et al., ; Li et al., ; Chen et al., ).
Focusing on long-form generations, which reflect real-world usage of LLMs, our analysis uncovers a key oversight: the choice of token used to extract these signals (Section 3).
We find that truthfulness information is concentrated in the exact answer tokens – e.g., “Hartford” in “The capital of Connecticut is Hartford, an iconic city…”.
Recognizing this nuance significantly improves error detection strategies across the board, revealing that truthfulness encoding is stronger than previously observed.

From this point forward, we concentrate on our most effective strategy: a classifier trained on intermediate LLM representations within the exact answer tokens, referred to as ‘probing classifiers’ (Belinkov, ). This approach helps us explore what these representations reveal about LLMs.
Our demonstration that a trained probing classifier can predict errors suggests that LLMs encode information related to their own truthfulness.
However, we find that probing classifiers do not generalize across different tasks (Section 4).
Generalization occurs only within tasks requiring similar skills (e.g., factual retrieval), indicating “skill-specific” truthfulness features.
For tasks involving different skills, e.g., sentiment analysis, these classifiers are no better–or worse–than logit-based uncertainty predictors, challenging the idea of a “universal truthfulness” encoding proposed in previous work (Marks & Tegmark, ; Slobodkin et al., ).
Instead, our results indicate that LLMs encode multiple, distinct notions of truth.
Thus, deploying trainable error detectors in practical applications should be undertaken with caution.

After establishing the limitations of error detection and examining how error encoding differs across tasks, we delve deeper into errors within a single task, taxonomizing its errors based on the model’s responses across repeated samples (Section 5).
For example, the same error being consistently generated is different from an error that is generated occasionally among many other distinct errors.
Using a different set of probing classifiers trained on the exact answer tokens, we find that error types are predictable from the LLM representations, drawing a connection between the models’s internal representations and its external behavior.
This raises an interesting possibility that LLMs encode information about the types of errors they may produce.
This classification offers a more nuanced understanding of errors, enabling developers to predict error patterns and implement more targeted mitigation strategies.

Finally, we investigate the alignment between the LLM’s internal representations and its external behavior (Section 6).
Our findings highlight a significant disagreement in some cases, where the model’s internal encoding contradicts its behavior. The model’s internal encoding may identify the correct answer–yet it frequently generates an incorrect response.
This discrepancy reveals that the LLM’s external behavior may misrepresent its abilities, potentially pointing to new strategies for reducing errors by utilizing its existing strengths.
Overall, our model-centric framework provides a deeper understanding of LLM errors, suggesting potential directions for improvements in error analysis and mitigation.

## 2 Background

#### Defining and characterizing LLM errors.

The term “hallucinations” is widely used across various subfields such as conversational AI (Liu et al., ), abstractive summarization (Zhang et al., ), and machine translation (Wang & Sennrich, ), each interpreting the term differently.
Yet, no consensus exists on defining hallucinations: Venkit et al. () identified 31 distinct frameworks for conceptualizing hallucinations, revealing the diversity of perspectives.
Research efforts aim to define and taxonomize hallucinations, distinguishing them from other error types (Liu et al., ; Ji et al., ; Huang et al., ; Rawte et al., ).
On the other hand, recent scholarly conversations introduce terms like “confabulations” (Millidge, ) and “fabrications” (McGowan et al., ), attributing a possible “intention” to LLMs, although the notions of LLM “intention” and other human-like traits are still debated (Salles et al., ; Serapio-García et al., ; Harnad, ).
These categorizations, however, adopt a human-centric view by focusing on the subjective interpretations of LLM hallucinations, which does not necessarily reflect how these errors are encoded within the models themselves.
This gap limits our ability to address the root causes of hallucinations, or to reason about their nature.
For example, it is unclear whether conclusions about hallucinations defined in one framework can be applied to another framework.
Instead, we adopt a broad interpretation of hallucinations. Here, we define hallucinations as any type of error generated by an LLM, including factual inaccuracies, biases, failures in common-sense reasoning, and others.

Another line of research suggests that LLMs either encode information about their own errors (Kadavath et al., ; Azaria & Mitchell, ) or exhibit discrepancies between their outputs and internal representations (Liu et al., ; Gottesman & Geva, ), indicating the presence of underlying mechanisms not reflected in their final outputs.
Moreover, Yona et al. () found that current LLMs fail to effectively convey their uncertainty through their generated outputs.
Hence, we propose shifting the focus from human-centric interpretations of hallucinations to a model-centric perspective, examining the model’s intermediate activations.

#### Error detection in LLMs.

Error detection is a longstanding task in NLP, crucial for maintaining high standards in various practical applications and for constructing more reliable systems that ensure user trust (Bommasani et al., ).
Over the years, many studies have proposed task-specific solutions (see Section A.1).
However, the recent shift towards general-purpose LLMs necessitates a holistic approach capable of addressing any error type, rather than focusing on specific ones, making it suitable for the diverse errors generated by these models.

A line of work has addressed this challenge by leveraging external knowledge sources (Lewis et al., ; Gao et al., ) or an external LLM judge (Lin et al., ; Rawte et al., ) to identify erroneous outputs.
On the other hand, our work focuses on detection methods that rely solely on the computations of the LLM—specifically, output logits, probabilities after softmax, and hidden states.

Error detection in LLMs is also closely linked to uncertainty estimation, where low certainty signals potential inaccuracies and possible errors.
Popular methods to derive calibrated confidence from LLMs include inspecting the model logit output values (Varshney et al., ), examining agreement across multiple sampled answers (Kuhn et al., ; Manakul et al., ; Tian et al., ) eliciting verbalized probability (Tian et al., ), and direct prompting (Kadavath et al., ).

Another line of work trains probing classifiers to discover and utilize truthfulness features.
This approach has shown some success by probing the final token of an answer–either generated (Kadavath et al., ; Snyder et al., ; Yuksekgonul et al., ; Zou et al., ; Yin et al., ; Chen et al., ; Simhi et al., ) or not (Li et al., ; Marks & Tegmark, ; Burns et al., ; Azaria & Mitchell, ; Rateike et al., ).
Others probe the final token of the prompt before the response is generated (Slobodkin et al., ; Snyder et al., ; Simhi et al., ; Gottesman & Geva, ).
Many previous studies simplify the analysis by generating answers in a few-shot setting or limiting generation to a single token.
In contrast, we simulate real-world usage of LLMs by allowing unrestricted answer generation.
By probing exact answer tokens, we achieve significant improvements in error detection.

## 3 Better Error Detection

This section presents our experiments on detecting LLM errors through their own computations, focusing on token selection’s impact and introducing a method that outperforms other approaches.

### 3.1 Task Definition

Given an LLM $M$, an input prompt $p$ and the LLM-generated response $\hat{y}$, the task is to predict whether $\hat{y}$ is correct or wrong.
We assume that there is access to the LLM’s internal states (i.e., white-box setting), but no access to any external resources (e.g., search engine or additional LLMs).

We use a dataset $D=\{(q_{i},y_{i})\}_{i=1}^{N}$, consisting of $N$ question-label pairs, where $\{q_{i}\}_{i=1}^{N}$ represents a series of questions (e.g., “What is the capital of Connecticut?”) and $\{y_{i}\}_{i=1}^{N}$ the corresponding ground-truth answers (“Hartford”).
For each question $q_{i}$, we prompt the model $M$ to generate a response $y_{i}$, resulting in the set of predicted answers $\{\hat{y}_{i}\}_{i=1}^{N}$ (“The capital of Connecticut is Hartford…”).
Next, to build our error-detection dataset, we evaluate the correctness of each generated response $\hat{y}_{i}$ by comparing it to the ground-truth label $y_{i}$.
This comparison yields a correctness label $z_{i}\in\{0,1\}$ ($1$ correct, 0 0 wrong).
The comparison can be done either via automatic heuristics or with the assistance of an instruct-LLM.(^2^22For most datasets, we use heuristics to predict correctness, except for one case. See Appendix A.2.)
Our error detection dataset is: $\{(q_{i},\hat{y}_{i},z_{i})\}_{i=1}^{N}$.
Any instances where the LLM refuses to answer are excluded, as these can easily be classified as incorrect.

### 3.2 Experimental Setup

#### Datasets and models.

We perform all experiments on four LLMs: Mistral-7b (Jiang et al., ), Mistral-7b-instruct-v0.2 (denoted Mistral-7b-instruct), Llama3-8b (Touvron et al., ), and Llama3-8b-instruct.
We consider 10 different datasets spanning various domains and tasks: TriviaQA (Joshi et al., ), HotpotQA with/without context (Yang et al., ), Natural Questions (Kwiatkowski et al., ), Winobias (Zhao et al., ), Winogrande (Sakaguchi et al., ), MNLI (Williams et al., ), Math (Sun et al., ), IMDB review sentiment analysis (Maas et al., ), and a dataset of movie roles (movies) that we curate.
We allow unrestricted response generation to mimic real-world LLM usage, with answers decoded greedily.
For more details on the datasets and the prompts used to generate answers, refer to Appendix A.3.

#### Performance metric.

We measure the area under the ROC curve to evaluate error detectors, providing a single metric that reflects their ability to distinguish between positive and negative cases across many thresholds, balancing sensitivity (true positive rate) and specificity (false positive rate).

Error detection methods.
We compare methods from both uncertainty and hallucinations literature.

- •
Aggregated probabilities / logits: Previous studies (Guerreiro et al., ; Kadavath et al., ; Varshney et al., ; Huang et al., ) aggregate output token probabilities or logits to score LLM confidence for error detection.
We implement several methods from the literature, calculating the minimum, maximum, or mean of these values.
The main paper reports results for the most common approach, Logits-mean, and the best-performing one, Logits-min, with additional baselines in Appendix B.
- •
P(True): Kadavath et al. () showed that LLMs are relatively calibrated when asked to evaluate the correctness of their generation via prompting. We implement this evaluation using the same prompt.
- •
Probing:
Probing classifiers involve training a small classifier on a model’s intermediate activations to predict features of processed text (Belinkov, ).
Recent studies show their effectiveness for error detection in generated text (Kadavath et al., , inter alia).
An intermediate activation is a vector $h_{l,t}$ from a specific LLM layer $l$ and (either read or generated) token $t$.
Thus, each LLM generation produces multiple such activations.
Following prior work, we use a linear probing classifier for error detection (Li et al., , inter alia) on static tokens: the last generated token ($h_{l,-1}$), the one before it ($h_{l,-2}$), and the final prompt token ($h_{l,k}$).
The layer $l$ is selected per token based on validation set performance.

For further details on the implementation of each method, refer to Appendix A.4.

Figure: Figure 1: Example for the input and LLM output from the TriviaQA dataset, and the names of the tokens that can be probed.
Refer to caption: x1.png

### 3.3 Exact Answer Tokens

Existing methods often overlook a critical nuance: the token selection for error detection, typically focusing on the last generated token or taking a mean.
However, since LLMs typically generate long-form responses, this practice may miss crucial details (Brunner et al., ).
Other approaches use the last token of the prompt (Slobodkin et al., , inter alia), but this is inherently inaccurate due to LLMs’ unidirectional nature, failing to account for the generated response and missing cases where different sampled answers from the same model vary in correctness.
We investigate a previously unexamined token location: the *exact answer tokens*, which represent the most meaningful parts of the generated response.
We define exact answer tokens as those whose modification alters the answer’s correctness, disregarding subsequent generated content.(^3^33In practice, we do not use this definition for extracting the exact answer, but rather an instruct model in a few-shot setting. Still, the definition is useful to manually verify that automatic extractions work as expected.)
Figure 1 illustrates the different token locations.
In the following experiments, we implement each error detection method with an “exact answer” version, demonstrating that it often improves performance, especially in probing.

These exact answer is identified from a lengthy generated answer using an external algorithm, which processes the question and the LLM’s response, $A(q_{i},\hat{y_{i}})$, to extract the exact answer.
In our implementation, we use Mistral-7b-Instruct in a few-shot learning setup as $A$.
However, we demonstrate that all the LLMs we evaluate are capable of extracting exact answers from their own outputs, as explained in AppendixA.2.
After extracting the exact answer, the exact answer tokens are identified through a simple search process.
We focus on four specific tokens: the one immediately preceding the first exact answer token, the first exact answer token itself, the last exact answer token, and the one immediately following it.

### 3.4 Results

Figure: (a) TriviaQA
Refer to caption: extracted/5959149/figures/probing_heatmaps/mistral-7b-instruct/triviaqa_auc.png

#### Patterns of truthfulness encoding.

We first focus on probing classifiers to gain insights into the internal representations of LLMs.
Specifically, we extensively analyze the effects of layer and token selection on activation extraction for these classifiers.
This is done by systematically probing all layers of the model, starting with the last question token and continuing through to the final generated token.
Figure 2 shows the AUC metrics of trained probes across various layers and tokens of Mistral-7b-Instruct.
While some datasets seem easier for error prediction, all exhibit consistent truthfulness encoding patterns.
Middle to later layers typically yield the most effective probing results, aligning with previous studies (Burns et al., ; CH-Wang et al., ).
Regarding tokens, a strong truthfulness signal appears immediately after the prompt, suggesting that this representation encodes information on the model’s general ability to answer the question correctly.
This signal weakens as text generation progresses but peaks again at the exact answer tokens.
Towards the end of the generation process, signal strength rises again, indicating that this representation encodes features from the entire generation, though it remains weaker than at the exact answer tokens.
These patterns are consistent across nearly all datasets and models (see Appendix B), suggesting a general mechanism by which LLMs encode and process truthfulness during text generation.

#### Error Detection Results.

**Table 1: Comparison of error detection techniques using AUC metric, across different models and datasets. The best-performing method is bolded. Using exact answer tokens is useful for many cases, especially probing.**
|  | Mistral-7b-Instruct | Llama 3-8b-Instruct |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | TriviaQA | Winobias | Math | TriviaQA | Winobias | Math |
| Logits-mean | $0.60$ $\pm 0.009$ | $0.56$ $\pm 0.017$ | $0.55$ $\pm 0.029$ | $0.66$ $\pm 0.005$ | $0.60$ $\pm 0.026$ | $0.75$ $\pm 0.018$ |
| Logits-mean-exact | $0.68$ $\pm 0.007$ | $0.54$ $\pm 0.012$ | $0.51$ $\pm 0.005$ | $0.71$ $\pm 0.006$ | $0.55$ $\pm 0.019$ | $0.80$ $\pm 0.021$ |
| Logits-min | $0.63$ $\pm 0.008$ | $0.59$ $\pm 0.012$ | $0.51$ $\pm 0.017$ | $0.74$ $\pm 0.007$ | $0.61$ $\pm 0.024$ | $0.75$ $\pm 0.016$ |
| Logits-min-exact | $0.75$ $\pm 0.006$ | $0.53$ $\pm 0.013$ | $0.71$ $\pm 0.009$ | $0.79$ $\pm 0.006$ | $0.61$ $\pm 0.019$ | $0.89$ $\pm 0.018$ |
| p(True) | $0.66$ $\pm 0.006$ | $0.45$ $\pm 0.021$ | $0.48$ $\pm 0.022$ | $0.73$ $\pm 0.008$ | $0.59$ $\pm 0.020$ | $0.62$ $\pm 0.017$ |
| p(True)-exact | $0.74$ $\pm 0.003$ | $0.40$ $\pm 0.021$ | $0.60$ $\pm 0.025$ | $0.73$ $\pm 0.005$ | $0.63$ $\pm 0.014$ | $0.59$ $\pm 0.018$ |
| Probe @ token |  |  |  |  |  |  |
| Last generated [-1] | $0.71$ $\pm 0.006$ | $0.82$ $\pm 0.004$ | $0.74$ $\pm 0.008$ | $0.81$ $\pm 0.005$ | $0.86$ $\pm 0.007$ | $0.82$ $\pm 0.016$ |
| Before last generated [-2] | $0.73$ $\pm 0.004$ | $0.85$ $\pm 0.004$ | $0.74$ $\pm 0.007$ | $0.75$ $\pm 0.005$ | $0.88$ $\pm 0.005$ | $0.79$ $\pm 0.020$ |
| End of question | $0.76$ $\pm 0.008$ | $0.82$ $\pm 0.011$ | $0.72$ $\pm 0.007$ | $0.77$ $\pm 0.007$ | $0.80$ $\pm 0.018$ | $0.72$ $\pm 0.023$ |
| Exact | 0.85 $\pm 0.004$ | 0.92 $\pm 0.005$ | 0.92 $\pm 0.008$ | 0.83 $\pm 0.002$ | 0.93 $\pm 0.004$ | 0.95 $\pm 0.027$ |

Next, we evaluate various error detection methods by comparing their performance with and without the use of exact answer tokens.
Table 1 compares the AUC across three representative datasets (additional datasets and models in Appendix B, showing consistent patterns).
Here we present results for the last exact answer token, which outperformed both the first exact answer token and the one preceding it, while the token following the last performed similarly.
Incorporating the exact answer token improves the different error detection methods in almost all datasets.
Notably, our probing technique (bottom line) consistently outperforms all other baselines across the board.
While we did not compare all existing error detection methods, the primary conclusion is that information about truthfulness is highly localized in specific generated tokens, and that focusing on exact answer tokens leads to significant improvements in error detection.

## 4 Generalization Between Tasks

The effectiveness of a probing classifier in detecting errors suggests that LLMs encode information about the truthfulness of their outputs.
This supports using probing classifiers for error detection in production, but their generalizability across tasks remains unclear.
While some studies argue for a universal mechanism of truthfulness encoding in LLMs (Marks & Tegmark, ; Slobodkin et al., ), results on probe generalization across datasets are mixed (Kadavath et al., ; Marks & Tegmark, ; CH-Wang et al., ; Slobodkin et al., ; Levinstein & Herrmann, ).
Understanding this is essential for real-world applications, where the error detector may encounter examples that significantly differ from those it was trained on.
Therefore, we explore whether a probe trained on one dataset can detect errors in others.

Our generalization experiments are conducted between all of the datasets discussed in Section 3, covering a broader range of non-synthetic settings than previous work.
We select the optimal token and layer combination for each dataset, train all probes using this combination on other datasets, and then test them on the original dataset.
We evaluate generalization performance using the absolute AUC score, defined as $\max(\text{auc},1-\text{auc})$, to also account for cases where the learned signal in one dataset is reversed in another.

#### Results.

Figure: (a) Raw AUC values. Values above $0.5$ indicate some generalization.
Refer to caption: extracted/5959149/figures/generalization/mistral_instruct.png

Figure 3(a) shows the generalization results for Mistral-7b-instruct, with similar patterns observed for other LLMs in Appendix C.
In this context, values above $0.5$ indicate successful generalization.
At first glance, the results appear consistent with previous research:
most heatmap values exceed $0.5$, implying some degree of generalization across tasks.
This observation supports the existence of a universal mechanism for decoding truthfulness, since the same linear directions—captured by the probe—encode truthfulness information across many datasets.
However, upon closer inspection, it turns out that most of this performance can be achieved by logit-based truthfulness detection, which only observes the output logits.
Figure 3(b) presents the same heatmap after subtracting results from our strongest logit-based baseline (Logit-min-exact).
This adjusted heatmap reveals the probe’s generalization rarely exceeds what can be achieved by examining logits alone.
This implies that the apparent generalization does not stem from a universal internal encoding of truthfulness but rather reflects information already accessible through external features like logits.

Nonetheless, we do observe some successful generalization in tasks requiring similar skills, such as parametric factual retrieval (TriviaQA, HotpotQA, Movies) and common-sense reasoning (Winobias, Wingrande, NLI).
This suggests that, although the overall pattern of truthfulness signals across tokens appeared consistent across tasks (as observed in Section 3.4), LLMs have many “skill-specific” truthfulness mechanisms rather than universal ones.
However, some patterns remain unexplained, such as the asymmetric generalization from TriviaQA to Math tasks.
Overall, our findings indicate that models have a multifaceted representation of truthfulness. They do not encode truthfulness through a single unified mechanism but rather through multiple mechanisms, each corresponding to different notions of truth.
Further investigation is required to disentangle these mechanisms.

## 5 Investigating Error Types

Having established the limitations of error detection, we now shift to error analysis.
Previously, we explored types of LLM limitations across different tasks, noting both commonalities and distinctions in their error representations.
In this section, we focus on the types of errors LLMs make in a specific task—TriviaQA—which represents factual errors, a commonly studied issue in LLMs (Kadavath et al., ; Snyder et al., ; Li et al., ; Chen et al., ; Simhi et al., ).

### 5.1 Taxonomy of Errors

Intuitively, not all mistakes are identical.
In one case, an LLM may consistently generate an incorrect answer, considering it correct, while in another case, it could issue a best guess.
To analyze errors from the LLM’s perspective, we sample $K=30$ responses at a temperature setting of $T=1$(^4^44We chose $K=30$ as the overall correctness seemed to plateau around this point; see Appendix D. We found that lower temperatures generally produced less truthful answers across repeated trials.) for each example in the dataset and then analyze the resulting distribution of answers.

Figure: (a) The LLM mostly answers correctly, but sometimes hallucinates.
Refer to caption: x2.png

Figure 4 illustrates three representative error types.
In one (Figure 4(a)), the model usually gives the correct answer but occasionally make an error, implying correct information is present but sampling may lead to mistakes.
In another (Figure 4(b), the model often responds incorrectly, though it is capable of providing the right answer, indicating some retained knowledge despite consistently making the same error.
In a third type (Figure 4(c)), the model generates a wide array of mostly incorrect answers, reflecting low confidence in any generated answer.

More generally, we categorize the errors by logging three specific features for each example:
(a) the number of different answers generated;
(b) the frequency of the correct answer;
and (c) the frequency of the most common incorrect answer.
These features reveal the following error patterns:

- •
(A) Refuses to answer: The model responds that it cannot answer the question in at least half the cases.
- •
(B) Consistently correct:
Answers correctly in at least half of the cases. This category is divided into: (B1) always correct; and (B2) mostly correct with occasional errors.
- •
(C) Consistently incorrect: Consistently generates the same incorrect response in at least half of the cases. Similarly to type B, we subdivide this type into (C1) correct answer is never produced; and (C2) correct answer appears at least once.
- •
(D) Two competing: Generates both correct and incorrect responses at similar rates–difference in rates is 5 or less, and each response is generated at least 5 times.
- •
(E) Many answers:
Generates over 10 distinct answers. Like types C and D, Subtypes include (E1) correct answer is never generated; and (E2) correct answer is generated at least once.

This taxonomy covers 96% of the errors in TriviaQA for Mistral-7b-instruct.
Although some overlap exists between types, our goal is to identify general patterns and explore their connection to the models’s internal representations.
This taxonomy classifies LLM errors based on an extrinsic, behavior-based analysis.
Similarly, previous work analyzed repeated samples to assess an LLM’s knowledge of the correct answer (Simhi et al., ; Gekhman et al., ).
Our approach is distinct because it also examines the nature of errors that the LLM makes.
Furthermore, as we discuss next, we analyze the connection between these behavioral patterns and the model’s internal encoding.

### 5.2 Predicting Error Types

Our taxonomy offers an external, behavioral analysis of LLMs, which we complement by an intrinsic evaluation.
We explore whether LLMs encode information on potential error types within their intermediate activations, offering a deeper insight into the underlying mechanisms.
To investigate this, we train a probe in a one-to-many setting, where a single probe identifies a specific error type from all others. We use representations extracted from the answers produced via greedy decoding.

Table 2 presents the test set results for all models.
Our findings demonstrate that the error type can be predicted from the intermediate representations of the greedy decoding generations, suggesting that they may encode not just output correctness but also features that are correlative with fine-grained information about potential errors.
This predictability opens up possibilities for targeted interventions on specific error types.

**Table 2: AUC scores for error type classification. Error types are predictable from the inner model representations, indicating the encoding of fine-grained information on errors.**
| Error type | Mistral-7b | Mistral-Instr-7b | Llama3-8b | Llama3-Instr-8b |
| --- | --- | --- | --- | --- |
| (A) Refuses to answer | $0.86\scriptscriptstyle{\pm 0.002}$ | $0.85\scriptscriptstyle{\pm 0.011}$ | $0.87\scriptscriptstyle{\pm 0.002}$ | $0.88\scriptscriptstyle{\pm 0.014}$ |
| (B) Consistently correct | $0.88\scriptscriptstyle{\pm 0.001}$ | $0.82\scriptscriptstyle{\pm 0.008}$ | $0.86\scriptscriptstyle{\pm 0.001}$ | $0.81\scriptscriptstyle{\pm 0.002}$ |
| (C) Consistently incorrect | $0.59\scriptscriptstyle{\pm 0.002}$ | $0.67\scriptscriptstyle{\pm 0.002}$ | $0.59\scriptscriptstyle{\pm 0.002}$ | $0.64\scriptscriptstyle{\pm 0.003}$ |
| (D) Two competing | $0.63\scriptscriptstyle{\pm 0.002}$ | $0.68\scriptscriptstyle{\pm 0.006}$ | $0.61\scriptscriptstyle{\pm 0.001}$ | $0.65\scriptscriptstyle{\pm 0.004}$ |
| (E) Many answers | $0.90\scriptscriptstyle{\pm 0.001}$ | $0.84\scriptscriptstyle{\pm 0.003}$ | $0.89\scriptscriptstyle{\pm 0.001}$ | $0.89\scriptscriptstyle{\pm 0.001}$ |

## 6 Detecting the Correct Answer

After identifying that models encode diverse truthfulness-related information, we examine how this internal truthfulness aligns with their external behavior during response generation.
To this end, we use our probe,(^5^55We choose the best-performing probe for each task, which is trained on the last exact answer token.) trained on error detection, to select an answer from a pool of 30 generated responses to the same question.
We then measure the model’s accuracy based on the selected answers.
A case where this accuracy does not significantly differ from traditional decoding methods (such as greedy decoding), suggests that the LLM’s internal representation of truthfulness is consistent with its external behavior.
In simpler terms, that the model is generating answers that it also internally considers as correct.
Conversely, a case where using the probe alters performance either way, would suggest a misalignment between the LLM’s internal representations and its actual behavior.

#### Experimental Setup

The experiments were conducted on TriviaQA, Winobias, and Math.
We resample each model answer in the same strategy described in Section 5.1.
The final chosen answer is the one with the highest correctness probability, as assessed by the probe.
We compare to three baselines:
(1) greedy decoding, (2) random selection from the $K=30$ answer candidates; and (3) majority vote wherein the most frequently generated answer is chosen.

#### Results

The results for Mistral-7b-instruct are summarized in Figure 5, with additional results for other LLMs and datasets as well as qualitative examples provided in Appendix E.
We only present results on error types that appear 30 times or more in our test dataset.
Overall, using the probe to select answers enhances the LLMs accuracy across all examined tasks.
However, the extent of improvement varies by error type.
For instance, in the TriviaQA dataset, there is minimal gain in the “mostly correct” category (B2).
In contrast, substantial gains—ranging from 30 to 40 points in some cases—are observed in the “mostly incorrect” (C2), “two competing answers” (D), and “many answers” (E1) categories.
Interestingly, and perhaps surprisingly, the probe is most effective in cases where the LLM lacks any (external) preference for the correct answer during generation.
The fact that the probe can effectively identify the correct answer in these scenarios, points at a significant disconnect between the LLM’s internal encoding and its external behavior. These results suggest that even when the model encodes information of which answer is correct, it can still generate an incorrect answer in practice.

While using the probe to select the answer proves effective, it is not proposed here as an error mitigation strategy but rather as a diagnostic tool.
However, these findings indicate that further research in this area could leverage the existing knowledge within LLMs to significantly reduce errors. We recommend exploring this direction in future investigations.

Figure: (a) TriviaQA
Refer to caption: extracted/5959149/figures/choose_answer/probe_choose_answer_triviaqa_mistral_instruct.png

## 7 Discussion and Conclusions

In this study, we analyzed the errors of LLMs by examining their internal representations.
We discover that information related to truthfulness is localized within the exact answer tokens, a critical insight that supports various analyses throughout the paper, as using the tokens with the strongest signals leads to more reliable conclusions.
From a practical perspective, this finding significantly enhances error detection methods applicable to production-level LLMs.
However, our approach requires access to internal representations, limiting its applicability mainly to open-source models.

Our findings also suggest that truthfulness features generalize poorly across tasks and datasets, performing better in tasks requiring similar skills.
This indicates that truthfulness may depend on “skill-specific” features, like knowledge retrieval and contextual consistency.
Some generalization patterns remain unexplained, suggesting that additional features link seemingly unrelated tasks, warranting further research.
Since the best layer and token combination for error detection varies by dataset, analyzing their effect on generalization may uncover insights into these mechanisms.
Moreover, Our results highlight the caution needed when applying a trained error detector across different settings.

We also found that the intermediate representations of LLMs can be used to predict the types of errors they might make.
As Simhi et al. () noted, different error types may require distinct mitigation strategies, such as retrieval-augmented generation (Lewis et al., ) in some cases or fine-tuning in others.
Identifying these error types is useful for customizing these strategies.

Lastly, we identified a significant discrepancy between the model’s external behavior and internal states, where a model generates an incorrect response repeatedly even though its internal representations encode the correct answer.
It is possible that mechanisms favoring likelihood override those promoting truthfulness, as LLMs are trained to predicting likely tokens, which does not necessarily align with factual accuracy.
Our findings imply that these models already encode valuable information that could possibly be harnessed to reduce errors.
Work by Chuang et al. () shows improvements in truthfulness by leveraging knowledge encoded in earlier transformer layers, pointing to a promising area for further investigation.

In conclusion, our findings suggest that LLMs’ internal representations provide useful insights into their errors, highlights the complex link between the internal processes of models and their external outputs, and hopefully paves the way for further improvements in error detection and mitigation.

## 8 Reproducibility Statement

To ensure reproducibility of our work, we provide detailed instructions and necessary code. The source code, including scripts for generating model answers, probing, resampling, and error type analysis, is available in the supplementary material, where we also provide command examples and specific seeds used for experiment reproducibility.
This repository includes documentation on how to set up the environment, download and preprocess datasets, and execute the experiments outlined in Sections 3–6 of the paper. Additionally, all datasets, models, and results generation steps are described in the Appendix A.

## Appendix A Implementation Details

### A.1 Task Specific Error Detection

In this work, we specifically address errors produced by modern large language models (LLMs). Given the diverse range of tasks these models are applied to, our focus is on general error detection across all categories, rather than isolating specific types. Prior to the emergence of LLMs, much research targeted error detection for specific tasks, with common examples including grammatical errors (Kasewa et al., ; Bell et al., ; Cheng & Duan, ; Wang & Tan, ; Flickinger et al., ), spelling mistakes (Mishra & Kaur, ), machine translation inaccuracies (Lo, ; Pu et al., ; Sellam et al., ; Gekhman et al., ; Rei et al., ; ; ), speech recognition faults (Caines et al., ; Rao et al., ; Li & Wang, ; Zhou et al., ; Allauzen, ; Gekhman et al., ; Errattahi et al., ; Pellegrini & Trancoso, ; Chen et al., ), and factual consistency failures (Honovich et al., ; Laban et al., ; Honovich et al., ; Gekhman et al., ; Scialom et al., ; Kryscinski et al., ).

### A.2 Probing: Implementation Details

We examine the intermediate representations of the exact answer tokens generated by a large language model (LLM) during the answer generation process. The intermediate representation selected for this analysis is derived from the output of the final multi-layer perceptron (MLP). This choice is based on preliminary experiments comparing the MLP output, the residual stream, and the attention heads, which showed no significant differences.
We leave the in-depth analysis for future work.
For the probing classifier, we employ a logistic regression model from the scikit-learn library (Pedregosa et al., ).

#### Obtaining correctness label for the probing dataset.

An answer is generally considered correct if it includes the correct answer label and appears before any alternative incorrect labels. We manually analyzed the results of this heuristic to confirm that it is accurate in almost all cases.
However, one exception is the Natural Questions with Context (NQ_WC) dataset, where we identified false negatives and thus deployed a more precise validation using an instruct LLM, as demonstrated below:

[backgroundcolor=blue!5, skipabove=0.5]
Evaluate the following answers to questions. For each question you would be given an LLM answer and the correct answer.
You would have to determine if the LLM answer is correct or not. If the LLM answer is correct, write ’1’ and if it is not correct, write ’0’.
For example:

Question: [Question 1]

Ground Truth: [Gold label 1]

LLM Answer: [LLM long answer 1]

Correctness: 0

Question: [Question 2]

Ground Truth: [Gold label 2]

LLM Answer: [LLM long answer 2]

Correctness: 1

Question: [Question]

Ground Truth: [Label]

LLM Answer: [LLM long answer]

Correctness:

#### Detecting and using exact answer tokens.

To detect the exact locations of answer tokens, we use a combination of heuristic methods and an instruction-tuned LLM.
Specifically, when the set of possible answers is finite, we rely on heuristics. For more open-ended scenarios, such as factual questions, we automatically locate the answer if it matches the gold label. Otherwise, we prompt an instruction-tuned LLM, specifically Mistral-7b-Instruct (Jiang et al., ), to identify and extract the exact answer substring using the following prompt:

[backgroundcolor=blue!5, skipabove=0.5]
Extract from the following long answer the short answer, only the relevant tokens. If the long answer does not answer the question, output NO ANSWER.

Q: [Question 1]

A: [LLM long answer 1]

Exact answer: [Short exact answer 1]

Q: [Question 2]

A: [LLM long answer that does not answer the question]

Exact answer: NO ANSWER

Q: [Question]

A: [LLM long answer]
Exact answer:

To extract a valid exact answer from a long response, we prompt the instruct LLM up to five times.
This process involves verifying that the exact answer is a substring of the long answer unless the instruct LLM indicates that there is no answer.
To avoid bias in our probing task, we only retain questions for which a valid exact answer was successfully extracted.
This ensures there is no unfair correlation between invalid answers and incorrect answers in the experiments.

We note the following:
(a) While it is possible to use an instruct LLM to extract every answer regardless of its correctness, we chose the aforementioned strategy to improve the efficiency of our experiments;
(b) This is just one possible implementation. For each LLM, one could use the same LLM to extract its own exact answer token, as demonstrated in a proof-of-concept over 1000 samples of TriviaQA in Table 3.
Alternatively, it may be more efficient to train a smaller system specifically designed for detecting exact answer tokens, which would be more suitable for real-world scenarios.
We choose to keep the extraction process as abstract as possible, as our primary focus is not on the specific implementation, but on analyzing the potential gains from probing these locations.

Additionally, if the exact answer token is not among the first generated tokens, we examine the token immediately preceding it (“before exact answer token”).
If the exact answer token is not the last one, we also examine the following token.
When the exact answer spans multiple tokens, the first and last exact answer tokens are probed separately.

**Table 3: Success rate of extracting exact answer from a long model answer. Each model is used to extract answers from its own output.**
| Mistral-7b | Mistral-Instruct-7b | Llama3-8b | Llama3-Instruct-8b |
| --- | --- | --- | --- |
| 0.99 | 0.96 | 0.99 | 0.95 |

### A.3 Datasets

We outline here all ten datasets that we investigate in our work.
In our analysis, we aimed at covering a wide range of tasks, skills required to solve the tasks, diversity of datasets and as a result also different LLM limitations such as factual inaccuracies (often referred to as “hallucinations”), biases, arithmetic mistakes, and more.
For each dataset, we explain how it covers something different from all the previous datasets.
For all datasets, we present the LLM with non or a short instruct, a context (if exists for the task), and let it generate a free text.
We follow this paradigm as it better mimics real-world usage of LLMs by humans, as opposed to using few-shot to force a short answer that is generated on the first token (Yuksekgonul et al., ; Chen et al., ; Simhi et al., ).
One exception to this is a the sentiment analysis (IMDB) for which we apply 1-shot for the LLM to use the allowed labels, as it did not follow the instruction alone and we could not identify if the answer is correct or not even with manual analysis. Additionally, we implemented a different prompting strategy to the instruct and non-instruct LLMs.
To see the exact formats we used to prompt each dataset and LLM, refer to our code implementation at [https://github.com/technion-cs-nlp/LLMsKnow](https://github.com/technion-cs-nlp/LLMsKnow).

For each dataset we used a split of 10K training samples and 10K test samples, unless the dataset is too small, in which case we mention the size.

- •
TriviaQA (Joshi et al., ): a collection of trivia question-answer pairs. The questions are presented to the LLM without any context, allowing it to generate responses based solely on its internal, parametric knowledge. The dataset includes various acceptable variations of the correct answer, which are used to automatically evaluate the accuracy of the generated res.
- •
HotpotQA (Yang et al., ): a dataset designed for diverse multi-hop question answering. Each entry includes Wikipedia documents that help answering the questions. We use two different settings:
(1) without context, where questions are asked directly, which covers slightly different skills from TriviaQA as it requires reasoning in addition to factual knowledge; and
(2) with context (HotpotQA_WC), where the additional context is provided, emphasizing the ability to adhere to and utilize contextual information to solve the task.
- •
Movies:
to further investigate generalization, we focused on a case of classic “hallucinations”, involving factual knowledge, within a non-diverse dataset. This approach allowed us to test whether generalization to other types of errors is influenced by the type of error (factual versus others) or by the dataset’s diversity.
For this purpose, we created the movies dataset consisting of prompts in the form: “Who acted as [figure name] in the movie [movie name]?”
The figures, movies, and correct answers were sourced from “The Movies Dataset” in Kaggle: [https://www.kaggle.com/datasets/rounakbanik/the-movies-dataset](https://www.kaggle.com/datasets/rounakbanik/the-movies-dataset), which is based on the MovieLens website.
- •
Winogrande (Sakaguchi et al., ):
we use this dataset to explore errors in common-sense reasoning.
It consists of Winograd-style coreference challenges, where each example presents a sentence containing two entities and a pronoun.
The objective is to determine which entity the pronoun refers to, relying on common-sense reasoning. For example, in the sentence: “The trophy doesn’t fit into the suitcase because it’s too large,” the pronoun “it” refers to the trophy, not the suitcase.
- •
Winobias (Zhao et al., ): this benchmark focuses on coreference resolution in the context of gender bias, revealing a different type of limitation in LLMs. Each example consists of two professions: one stereotypically male and one stereotypically female, along with a gendered pronoun. The task requires the LLM to determine which profession the pronoun refers to. The sentences are unambiguous, with one correct answer. In some cases, the correct answer aligns with the stereotype, while in others, it is anti-stereotypical. For example, in the sentence “The developer argued with the designer because she did not like the design,” “she” refers to the developer, which is an anti-stereotypical case since “developer” is considered a stereotypically male profession. Research has shown that LLMs often perform poorly on anti-stereotypical sentences (Zhao et al., ) and tend to base their decisions on stereotypes rather than on common-sense reasoning or linguistic rules (Kotek et al., ).
Each split contains around 1500 samples.
- •
NLI (Natural Language Inference): NLI involves determining whether a given “hypothesis” is true (entailment), false (contradiction), or undetermined (neutral) based on a provided “premise.” For this purpose, we use the MNLI dataset (Williams et al., ). NLI tasks address a distinct aspect of common-sense reasoning and are generally considered complex. This complexity allows us to investigate whether a model’s generalization ability is related to the difficulty of the task it was trained on, or to other factors, such as the limited diversity of labels (NLI has only three valid labels) or the type of task.
- •
Math (Sun et al., ): this dataset includes both unanswerable and answerable math problems. In our study, we focus exclusively on the answerable problems, as our aim is to assess the correctness of the LLM’s outputs, which requires a known correct answer (gold standard). This task introduces an additional, previously unexplored skill of arithmetic reasoning.
The train-test split consists of approximately 2,000 and 650 samples, respectively.
- •
IMDB (Maas et al., ): contains movie reviews used for the task of sentiment classification.
- •
Natural Questions With Context (Kwiatkowski et al., ): the Natural Questions (NQ) dataset is designed to evaluate and train automatic question-answering systems.
It consists of real, anonymized queries submitted by users to Google, with answers extracted from Wikipedia, as well as the relevant Wikipedia pages which can be given in context.
We included this dataset to introduce an additional challenge that requires adherence to context, complementing the HotpotQA with context dataset.

### A.4 Baselines: Implementation Details

#### Aggregated probabilities / logits.

Inspired by prior work (Kadavath et al., ; Guerreiro et al., ), we compute an aggregated score using the log-probabilities or raw probabilities of the generated text tokens $y_{1},y_{2},\ldots,y_{N}$ produced by the generative large language model (LLM).
For instance, the following formulation is used to compute the Logits-mean baseline on the entire generated answer:

$$ $\centering\frac{1}{N}\sum_{i=1}^{N}\mathbb{P}(y_{i}|Q,y_{1},...,y_{i-1})\@add@centering$ (1) $$

We also explore aggregation strategies that focus solely on the exact answer tokens (PE-Exact).
Following Varshney et al. (), we also experiment with aggregating the minimum and maximum values (PE-[Min—Max]-[Exact]), alongside the mean aggregation described in Equation 1.

#### P( True ):

We follow Kadavath et al. () and prompt the LLM to judge whether its answer is correct. Our prompt followed the following template, from Kadavath et al. ():

[backgroundcolor=blue!5, skipabove=0.5]

Question: [Question]

Proposed Answer: [LLM long answer]

Is the proposed answer:

(A) True

(B) False

The proposed answer is:

## Appendix B Full Error Detection Results

Figure 6 presents the AUC values of a traind probe across layers and token for Mistral-7b-instruct, showing a similar pattern across all datasets.
We also observe similar patterns across other models.
See our repo [https://github.com/technion-cs-nlp/LLMsKnow](https://github.com/technion-cs-nlp/LLMsKnow)for the figures.

Figure: (a) HotpotQA
Refer to caption: extracted/5959149/figures/probing_heatmaps/mistral-7b-instruct/hotpotqa_auc.png

Tables 4, 5, 6, and 7 present the full error detection results across all baselines and datasets, which are consistent with the main paper results.

**Table 4: Comparison of error detection performance (AUC) on Mistral-7B.**
|  | Mistral-7B |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | TriviaQA | Winobias | Math | Movies | IMDB |
| Logits-mean | $0.67$ $\pm 0.004$ | $0.49$ $\pm 0.010$ | $0.41$ $\pm 0.015$ | $0.67$ $\pm 0.007$ | $0.88$ $\pm 0.064$ |
| Logits-mean-exact | $0.67$ $\pm 0.004$ | $0.50$ $\pm 0.010$ | $0.56$ $\pm 0.026$ | $0.68$ $\pm 0.008$ | $0.57$ $\pm 0.080$ |
| Logits-min | $0.80$ $\pm 0.003$ | $0.45$ $\pm 0.014$ | $0.48$ $\pm 0.021$ | $0.73$ $\pm 0.006$ | $0.78$ $\pm 0.056$ |
| Logits-min-exact | $0.80$ $\pm 0.005$ | $0.53$ $\pm 0.014$ | $0.78$ $\pm 0.032$ | $0.72$ $\pm 0.005$ | $0.57$ $\pm 0.080$ |
| Logits-max | $0.53$ $\pm 0.008$ | $0.49$ $\pm 0.010$ | $0.42$ $\pm 0.023$ | $0.54$ $\pm 0.005$ | $0.83$ $\pm 0.076$ |
| Logits-max-exact | $0.54$ $\pm 0.009$ | $0.50$ $\pm 0.010$ | $0.40$ $\pm 0.024$ | $0.58$ $\pm 0.007$ | $0.57$ $\pm 0.080$ |
| Probas-mean | $0.76$ $\pm 0.003$ | $0.53$ $\pm 0.018$ | $0.66$ $\pm 0.016$ | $0.72$ $\pm 0.007$ | $0.87$ $\pm 0.041$ |
| Probas-mean-exact | $0.78$ $\pm 0.002$ | $0.55$ $\pm 0.014$ | $0.62$ $\pm 0.016$ | $0.74$ $\pm 0.007$ | $0.83$ $\pm 0.057$ |
| Probas-min | $0.82$ $\pm 0.003$ | $0.52$ $\pm 0.013$ | $0.82$ $\pm 0.020$ | $0.73$ $\pm 0.006$ | $0.86$ $\pm 0.032$ |
| Probas-min-exact | 0.85 $\pm 0.003$ | $0.58$ $\pm 0.011$ | $0.84$ $\pm 0.015$ | $0.74$ $\pm 0.006$ | $0.83$ $\pm 0.057$ |
| Probas-max | $0.53$ $\pm 0.008$ | $0.50$ $\pm 0.016$ | $0.43$ $\pm 0.025$ | $0.55$ $\pm 0.008$ | $0.80$ $\pm 0.074$ |
| Probas-max-exact | $0.55$ $\pm 0.009$ | $0.51$ $\pm 0.013$ | $0.39$ $\pm 0.019$ | $0.59$ $\pm 0.009$ | $0.83$ $\pm 0.057$ |
| p(True) | $0.57$ $\pm 0.007$ | $0.53$ $\pm 0.019$ | $0.56$ $\pm 0.027$ | $0.51$ $\pm 0.003$ | $0.65$ $\pm 0.004$ |
| p(True)-exact | $0.56$ $\pm 0.006$ | $0.55$ $\pm 0.026$ | $0.57$ $\pm 0.036$ | $0.52$ $\pm 0.003$ | $0.65$ $\pm 0.003$ |
| Probe @ token |  |  |  |  |  |
| Last generated [-1] | $0.83$ $\pm 0.002$ | $0.65$ $\pm 0.008$ | $0.82$ $\pm 0.023$ | $0.79$ $\pm 0.002$ | $0.85$ $\pm 0.007$ |
| Before last generated [-2] | $0.82$ $\pm 0.003$ | $0.84$ $\pm 0.012$ | $0.83$ $\pm 0.019$ | $0.78$ $\pm 0.003$ | $0.95$ $\pm 0.004$ |
| End of question | $0.74$ $\pm 0.005$ | $0.78$ $\pm 0.012$ | $0.83$ $\pm 0.016$ | $0.77$ $\pm 0.002$ | $0.81$ $\pm 0.009$ |
| Exact answer last | $0.84$ $\pm 0.005$ | 0.89 $\pm 0.007$ | 0.96 $\pm 0.008$ | $0.78$ $\pm 0.003$ | 0.95 $\pm 0.004$ |
| Exact answer last+1 | $0.84$ $\pm 0.004$ | $0.84$ $\pm 0.012$ | $0.95$ $\pm 0.010$ | 0.80 $\pm 0.002$ | $0.85$ $\pm 0.007$ |
|  | HotpotQA | HotpotQA-WC | Winogrande | NLI | NQ-WC |
| Logits-mean | $0.63$ $\pm 0.005$ | $0.52$ $\pm 0.009$ | $0.49$ $\pm 0.004$ | $0.51$ $\pm 0.004$ | $0.69$ $\pm 0.006$ |
| Logits-mean-exact | $0.57$ $\pm 0.008$ | $0.52$ $\pm 0.007$ | $0.50$ $\pm 0.003$ | 0.93 $\pm 0.004$ | $0.72$ $\pm 0.005$ |
| Logits-min | $0.72$ $\pm 0.008$ | $0.59$ $\pm 0.006$ | $0.50$ $\pm 0.007$ | $0.53$ $\pm 0.005$ | $0.65$ $\pm 0.009$ |
| Logits-min-exact | $0.72$ $\pm 0.007$ | $0.65$ $\pm 0.004$ | $0.51$ $\pm 0.007$ | $0.49$ $\pm 0.006$ | $0.70$ $\pm 0.005$ |
| Logits-max | $0.54$ $\pm 0.007$ | $0.49$ $\pm 0.010$ | $0.48$ $\pm 0.005$ | $0.48$ $\pm 0.005$ | $0.59$ $\pm 0.012$ |
| Logits-max-exact | $0.48$ $\pm 0.010$ | $0.44$ $\pm 0.007$ | $0.50$ $\pm 0.003$ | $0.48$ $\pm 0.005$ | $0.58$ $\pm 0.009$ |
| Probas-mean | $0.65$ $\pm 0.004$ | $0.55$ $\pm 0.006$ | $0.51$ $\pm 0.007$ | $0.49$ $\pm 0.003$ | $0.63$ $\pm 0.008$ |
| Probas-mean-exact | $0.62$ $\pm 0.006$ | $0.56$ $\pm 0.007$ | $0.51$ $\pm 0.005$ | $0.02$ $\pm 0.001$ | $0.66$ $\pm 0.007$ |
| Probas-min | $0.73$ $\pm 0.005$ | $0.58$ $\pm 0.007$ | $0.52$ $\pm 0.009$ | $0.53$ $\pm 0.004$ | $0.63$ $\pm 0.011$ |
| Probas-min-exact | $0.78$ $\pm 0.005$ | $0.66$ $\pm 0.004$ | $0.52$ $\pm 0.008$ | $0.49$ $\pm 0.005$ | $0.69$ $\pm 0.006$ |
| Probas-max | $0.54$ $\pm 0.008$ | $0.49$ $\pm 0.007$ | $0.50$ $\pm 0.005$ | $0.47$ $\pm 0.004$ | $0.52$ $\pm 0.004$ |
| Probas-max-exact | $0.48$ $\pm 0.010$ | $0.44$ $\pm 0.005$ | $0.50$ $\pm 0.004$ | $0.48$ $\pm 0.003$ | $0.53$ $\pm 0.012$ |
| p(True) | $0.55$ $\pm 0.007$ | $0.54$ $\pm 0.006$ | $0.51$ $\pm 0.005$ | $0.51$ $\pm 0.003$ | $0.52$ $\pm 0.008$ |
| p(True)-exact | $0.61$ $\pm 0.005$ | $0.54$ $\pm 0.006$ | $0.61$ $\pm 0.006$ | $0.51$ $\pm 0.006$ | $0.53$ $\pm 0.014$ |
| Probe @ token |  |  |  |  |  |
| Last generated [-1] | $0.78$ $\pm 0.006$ | $0.67$ $\pm 0.004$ | $0.51$ $\pm 0.007$ | $0.77$ $\pm 0.004$ | $0.78$ $\pm 0.003$ |
| Before last generated [-2] | $0.79$ $\pm 0.007$ | $0.69$ $\pm 0.007$ | $0.66$ $\pm 0.004$ | $0.81$ $\pm 0.002$ | $0.75$ $\pm 0.006$ |
| End of question | $0.72$ $\pm 0.007$ | $0.56$ $\pm 0.003$ | $0.51$ $\pm 0.007$ | $0.88$ $\pm 0.004$ | $0.70$ $\pm 0.005$ |
| Exact answer last | $0.80$ $\pm 0.008$ | 0.74 $\pm 0.007$ | 0.69 $\pm 0.006$ | $0.84$ $\pm 0.004$ | $0.81$ $\pm 0.009$ |
| Exact answer last+1 | 0.81 $\pm 0.008$ | $0.72$ $\pm 0.005$ | $0.59$ $\pm 0.005$ | $0.75$ $\pm 0.006$ | 0.84 $\pm 0.007$ |

**Table 5: Comparison of error detection performance (AUC) on Mistral-7B-Instruct.**
|  | Mistral-7B-Instruct |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | TriviaQA | Winobias | Math | Movies | IMDB |
| Logits-mean | $0.60$ $\pm 0.009$ | $0.56$ $\pm 0.017$ | $0.55$ $\pm 0.029$ | $0.63$ $\pm 0.005$ | $0.57$ $\pm 0.006$ |
| Logits-mean-exact | $0.68$ $\pm 0.007$ | $0.54$ $\pm 0.012$ | $0.51$ $\pm 0.005$ | $0.70$ $\pm 0.004$ | $0.87$ $\pm 0.007$ |
| Logits-min | $0.63$ $\pm 0.008$ | $0.59$ $\pm 0.012$ | $0.51$ $\pm 0.017$ | $0.66$ $\pm 0.008$ | $0.52$ $\pm 0.007$ |
| Logits-min-exact | $0.75$ $\pm 0.006$ | $0.53$ $\pm 0.013$ | $0.71$ $\pm 0.009$ | $0.74$ $\pm 0.005$ | $0.87$ $\pm 0.007$ |
| Logits-max | $0.54$ $\pm 0.005$ | $0.53$ $\pm 0.012$ | $0.54$ $\pm 0.039$ | $0.54$ $\pm 0.004$ | $0.47$ $\pm 0.004$ |
| Logits-max-exact | $0.55$ $\pm 0.004$ | $0.54$ $\pm 0.011$ | $0.32$ $\pm 0.015$ | $0.61$ $\pm 0.006$ | $0.87$ $\pm 0.007$ |
| Probas-mean | $0.60$ $\pm 0.007$ | $0.58$ $\pm 0.018$ | $0.56$ $\pm 0.028$ | $0.61$ $\pm 0.002$ | $0.54$ $\pm 0.008$ |
| Probas-mean-exact | $0.71$ $\pm 0.003$ | $0.57$ $\pm 0.015$ | $0.71$ $\pm 0.014$ | $0.74$ $\pm 0.006$ | $0.84$ $\pm 0.007$ |
| Probas-min | $0.59$ $\pm 0.008$ | $0.58$ $\pm 0.014$ | $0.50$ $\pm 0.025$ | $0.60$ $\pm 0.008$ | $0.51$ $\pm 0.010$ |
| Probas-min-exact | $0.74$ $\pm 0.004$ | $0.57$ $\pm 0.016$ | $0.75$ $\pm 0.011$ | $0.73$ $\pm 0.006$ | $0.84$ $\pm 0.007$ |
| Probas-max | $0.50$ $\pm 0.006$ | $0.41$ $\pm 0.010$ | $0.53$ $\pm 0.009$ | $0.51$ $\pm 0.005$ | $0.48$ $\pm 0.004$ |
| Probas-max-exact | $0.51$ $\pm 0.007$ | $0.54$ $\pm 0.010$ | $0.45$ $\pm 0.015$ | $0.60$ $\pm 0.003$ | $0.84$ $\pm 0.007$ |
| p(True) | $0.68$ $\pm 0.005$ | $0.45$ $\pm 0.021$ | $0.48$ $\pm 0.026$ | $0.62$ $\pm 0.005$ | $0.62$ $\pm 0.009$ |
| p(True)-exact | $0.74$ $\pm 0.003$ | $0.40$ $\pm 0.021$ | $0.60$ $\pm 0.025$ | $0.69$ $\pm 0.008$ | $0.60$ $\pm 0.009$ |
| Probe @ token |  |  |  |  |  |
| Last generated [-1] | $0.71$ $\pm 0.006$ | $0.82$ $\pm 0.004$ | $0.74$ $\pm 0.008$ | $0.72$ $\pm 0.005$ | $0.92$ $\pm 0.010$ |
| Before last generated [-2] | $0.73$ $\pm 0.004$ | $0.85$ $\pm 0.004$ | $0.74$ $\pm 0.007$ | $0.72$ $\pm 0.006$ | $0.94$ $\pm 0.006$ |
| End of question | $0.76$ $\pm 0.008$ | $0.82$ $\pm 0.011$ | $0.72$ $\pm 0.007$ | $0.74$ $\pm 0.003$ | $0.96$ $\pm 0.006$ |
| Exact answer last | $0.85$ $\pm 0.004$ | 0.92 $\pm 0.005$ | 0.92 $\pm 0.008$ | $0.81$ $\pm 0.003$ | 0.97 $\pm 0.005$ |
| Exact answer last+1 | 0.86 $\pm 0.006$ | $0.88$ $\pm 0.006$ | $0.90$ $\pm 0.010$ | 0.82 $\pm 0.003$ | $0.96$ $\pm 0.006$ |
|  | HotpotQA | HotpotQA-WC | Winogrande | NLI | NQ-WC |
| Logits-mean | $0.61$ $\pm 0.002$ | $0.55$ $\pm 0.009$ | $0.59$ $\pm 0.004$ | $0.64$ $\pm 0.006$ | $0.71$ $\pm 0.008$ |
| Logits-mean-exact | $0.66$ $\pm 0.009$ | $0.55$ $\pm 0.004$ | $0.49$ $\pm 0.004$ | $0.57$ $\pm 0.004$ | $0.69$ $\pm 0.009$ |
| Logits-min | $0.61$ $\pm 0.003$ | $0.53$ $\pm 0.013$ | $0.61$ $\pm 0.003$ | $0.62$ $\pm 0.002$ | $0.67$ $\pm 0.008$ |
| Logits-min-exact | $0.77$ $\pm 0.004$ | $0.67$ $\pm 0.013$ | $0.48$ $\pm 0.004$ | $0.54$ $\pm 0.005$ | $0.69$ $\pm 0.006$ |
| Logits-max | $0.53$ $\pm 0.008$ | $0.51$ $\pm 0.011$ | $0.52$ $\pm 0.006$ | $0.59$ $\pm 0.008$ | $0.63$ $\pm 0.011$ |
| Logits-max-exact | $0.51$ $\pm 0.011$ | $0.41$ $\pm 0.010$ | $0.49$ $\pm 0.007$ | $0.64$ $\pm 0.003$ | $0.63$ $\pm 0.013$ |
| Probas-mean | $0.63$ $\pm 0.003$ | $0.56$ $\pm 0.010$ | $0.58$ $\pm 0.005$ | $0.62$ $\pm 0.005$ | $0.68$ $\pm 0.010$ |
| Probas-mean-exact | $0.72$ $\pm 0.006$ | $0.66$ $\pm 0.010$ | $0.46$ $\pm 0.004$ | $0.57$ $\pm 0.003$ | $0.65$ $\pm 0.008$ |
| Probas-min | $0.58$ $\pm 0.003$ | $0.52$ $\pm 0.008$ | $0.59$ $\pm 0.002$ | $0.58$ $\pm 0.008$ | $0.65$ $\pm 0.014$ |
| Probas-min-exact | $0.76$ $\pm 0.004$ | $0.68$ $\pm 0.010$ | $0.46$ $\pm 0.005$ | $0.57$ $\pm 0.003$ | $0.66$ $\pm 0.008$ |
| Probas-max | $0.50$ $\pm 0.005$ | $0.53$ $\pm 0.003$ | $0.48$ $\pm 0.007$ | $0.52$ $\pm 0.007$ | $0.51$ $\pm 0.005$ |
| Probas-max-exact | $0.46$ $\pm 0.010$ | $0.46$ $\pm 0.010$ | $0.48$ $\pm 0.004$ | $0.53$ $\pm 0.004$ | $0.52$ $\pm 0.018$ |
| p(True) | $0.54$ $\pm 0.006$ | $0.54$ $\pm 0.004$ | $0.53$ $\pm 0.003$ | $0.58$ $\pm 0.003$ | $0.57$ $\pm 0.006$ |
| p(True)-exact | $0.60$ $\pm 0.008$ | $0.48$ $\pm 0.005$ | $0.57$ $\pm 0.011$ | $0.65$ $\pm 0.004$ | $0.57$ $\pm 0.009$ |
| Probe @ token |  |  |  |  |  |
| Last generated [-1] | $0.72$ $\pm 0.005$ | $0.64$ $\pm 0.005$ | $0.74$ $\pm 0.005$ | $0.85$ $\pm 0.004$ | $0.82$ $\pm 0.006$ |
| Before last generated [-2] | $0.73$ $\pm 0.006$ | $0.64$ $\pm 0.004$ | $0.76$ $\pm 0.004$ | $0.87$ $\pm 0.002$ | $0.84$ $\pm 0.009$ |
| End of question | $0.80$ $\pm 0.003$ | $0.63$ $\pm 0.003$ | $0.71$ $\pm 0.007$ | $0.79$ $\pm 0.004$ | $0.85$ $\pm 0.010$ |
| Exact answer last | $0.85$ $\pm 0.003$ | $0.75$ $\pm 0.006$ | 0.84 $\pm 0.005$ | 0.93 $\pm 0.003$ | $0.86$ $\pm 0.003$ |
| Exact answer last+1 | 0.85 $\pm 0.002$ | 0.76 $\pm 0.004$ | $0.80$ $\pm 0.004$ | $0.92$ $\pm 0.004$ | 0.87 $\pm 0.006$ |

**Table 6: Comparison of error detection performance (AUC) on Llama-8b.**
|  | Llama-8b |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | TriviaQA | Winobias | Math | Movies | IMDB |
| Logits-mean | $0.58$ $\pm 0.006$ | $0.44$ $\pm 0.015$ | $0.43$ $\pm 0.026$ | $0.64$ $\pm 0.008$ | $0.77$ $\pm 0.007$ |
| Logits-mean-exact | $0.63$ $\pm 0.007$ | $0.50$ $\pm 0.015$ | $0.50$ $\pm 0.028$ | $0.64$ $\pm 0.008$ | $0.77$ $\pm 0.007$ |
| Logits-min | $0.75$ $\pm 0.007$ | $0.50$ $\pm 0.022$ | $0.45$ $\pm 0.042$ | $0.73$ $\pm 0.005$ | $0.73$ $\pm 0.007$ |
| Logits-min-exact | $0.76$ $\pm 0.003$ | $0.53$ $\pm 0.009$ | $0.75$ $\pm 0.022$ | $0.73$ $\pm 0.005$ | $0.77$ $\pm 0.007$ |
| Logits-max | $0.48$ $\pm 0.006$ | $0.48$ $\pm 0.009$ | $0.42$ $\pm 0.027$ | $0.53$ $\pm 0.005$ | $0.72$ $\pm 0.007$ |
| Logits-max-exact | $0.52$ $\pm 0.007$ | $0.49$ $\pm 0.014$ | $0.35$ $\pm 0.026$ | $0.53$ $\pm 0.005$ | $0.77$ $\pm 0.007$ |
| Probas-mean | $0.64$ $\pm 0.006$ | $0.41$ $\pm 0.008$ | $0.61$ $\pm 0.029$ | $0.71$ $\pm 0.007$ | $0.70$ $\pm 0.008$ |
| Probas-mean-exact | $0.72$ $\pm 0.005$ | $0.50$ $\pm 0.018$ | $0.54$ $\pm 0.026$ | $0.72$ $\pm 0.006$ | $0.88$ $\pm 0.003$ |
| Probas-min | $0.79$ $\pm 0.008$ | $0.43$ $\pm 0.004$ | $0.75$ $\pm 0.044$ | $0.74$ $\pm 0.005$ | $0.68$ $\pm 0.005$ |
| Probas-min-exact | $0.82$ $\pm 0.003$ | $0.53$ $\pm 0.014$ | $0.78$ $\pm 0.022$ | $0.74$ $\pm 0.005$ | $0.88$ $\pm 0.003$ |
| Probas-max | $0.49$ $\pm 0.006$ | $0.50$ $\pm 0.009$ | $0.46$ $\pm 0.032$ | $0.53$ $\pm 0.007$ | $0.60$ $\pm 0.009$ |
| Probas-max-exact | $0.53$ $\pm 0.008$ | $0.50$ $\pm 0.018$ | $0.36$ $\pm 0.032$ | $0.54$ $\pm 0.007$ | $0.88$ $\pm 0.003$ |
| p(True) | $0.62$ $\pm 0.005$ | $0.48$ $\pm 0.011$ | $0.53$ $\pm 0.027$ | $0.61$ $\pm 0.005$ | $0.51$ $\pm 0.010$ |
| p(True)-exact | $0.67$ $\pm 0.002$ | $0.53$ $\pm 0.017$ | $0.63$ $\pm 0.028$ | $0.58$ $\pm 0.005$ | $0.52$ $\pm 0.008$ |
| Probe @ token |  |  |  |  |  |
| Last generated [-1] | $0.77$ $\pm 0.005$ | $0.59$ $\pm 0.024$ | $0.83$ $\pm 0.013$ | $0.82$ $\pm 0.005$ | $0.94$ $\pm 0.002$ |
| Before last generated [-2] | $0.76$ $\pm 0.012$ | $0.58$ $\pm 0.021$ | $0.82$ $\pm 0.032$ | $0.79$ $\pm 0.004$ | $0.96$ $\pm 0.002$ |
| End of question | $0.73$ $\pm 0.005$ | $0.77$ $\pm 0.012$ | $0.80$ $\pm 0.027$ | $0.78$ $\pm 0.005$ | $0.68$ $\pm 0.009$ |
| Exact answer last | 0.82 $\pm 0.006$ | 0.91 $\pm 0.007$ | 0.96 $\pm 0.010$ | $0.80$ $\pm 0.005$ | 0.97 $\pm 0.001$ |
| Exact answer last+1 | $0.82$ $\pm 0.006$ | $0.86$ $\pm 0.008$ | $0.95$ $\pm 0.007$ | 0.82 $\pm 0.006$ | $0.95$ $\pm 0.003$ |
|  | HotpotQA | HotpotQA-WC | Winogrande | NLI | NQ-WC |
| Logits-mean | $0.65$ $\pm 0.004$ | $0.62$ $\pm 0.006$ | $0.48$ $\pm 0.003$ | $0.47$ $\pm 0.002$ | $0.53$ $\pm 0.010$ |
| Logits-mean-exact | $0.55$ $\pm 0.003$ | $0.54$ $\pm 0.006$ | $0.49$ $\pm 0.004$ | $0.48$ $\pm 0.002$ | $0.58$ $\pm 0.009$ |
| Logits-min | $0.57$ $\pm 0.004$ | $0.49$ $\pm 0.003$ | $0.48$ $\pm 0.003$ | $0.48$ $\pm 0.007$ | $0.58$ $\pm 0.009$ |
| Logits-min-exact | $0.69$ $\pm 0.002$ | $0.68$ $\pm 0.006$ | $0.49$ $\pm 0.003$ | $0.48$ $\pm 0.007$ | $0.61$ $\pm 0.010$ |
| Logits-max | $0.61$ $\pm 0.005$ | $0.60$ $\pm 0.004$ | $0.48$ $\pm 0.003$ | $0.52$ $\pm 0.003$ | $0.51$ $\pm 0.008$ |
| Logits-max-exact | $0.47$ $\pm 0.003$ | $0.46$ $\pm 0.005$ | $0.49$ $\pm 0.004$ | $0.51$ $\pm 0.002$ | $0.54$ $\pm 0.005$ |
| Probas-mean | $0.67$ $\pm 0.002$ | $0.62$ $\pm 0.006$ | $0.49$ $\pm 0.002$ | $0.48$ $\pm 0.004$ | $0.57$ $\pm 0.003$ |
| Probas-mean-exact | $0.62$ $\pm 0.005$ | $0.56$ $\pm 0.005$ | $0.51$ $\pm 0.002$ | $0.46$ $\pm 0.006$ | $0.64$ $\pm 0.007$ |
| Probas-min | $0.62$ $\pm 0.006$ | $0.51$ $\pm 0.002$ | $0.49$ $\pm 0.003$ | $0.50$ $\pm 0.010$ | $0.62$ $\pm 0.005$ |
| Probas-min-exact | $0.76$ $\pm 0.005$ | $0.67$ $\pm 0.004$ | $0.51$ $\pm 0.002$ | $0.50$ $\pm 0.010$ | $0.69$ $\pm 0.008$ |
| Probas-max | $0.61$ $\pm 0.004$ | $0.58$ $\pm 0.004$ | $0.48$ $\pm 0.002$ | $0.48$ $\pm 0.003$ | $0.51$ $\pm 0.012$ |
| Probas-max-exact | $0.49$ $\pm 0.003$ | $0.44$ $\pm 0.004$ | $0.51$ $\pm 0.003$ | $0.47$ $\pm 0.002$ | $0.56$ $\pm 0.005$ |
| p(True) | $0.52$ $\pm 0.007$ | $0.45$ $\pm 0.005$ | $0.54$ $\pm 0.004$ | $0.54$ $\pm 0.007$ | $0.56$ $\pm 0.006$ |
| p(True)-exact | $0.58$ $\pm 0.005$ | $0.50$ $\pm 0.007$ | $0.64$ $\pm 0.004$ | $0.62$ $\pm 0.005$ | $0.61$ $\pm 0.002$ |
| Probe @ token |  |  |  |  |  |
| Last generated [-1] | $0.76$ $\pm 0.007$ | $0.57$ $\pm 0.006$ | $0.59$ $\pm 0.006$ | $0.89$ $\pm 0.002$ | $0.66$ $\pm 0.010$ |
| Before last generated [-2] | $0.74$ $\pm 0.007$ | $0.58$ $\pm 0.005$ | $0.59$ $\pm 0.005$ | $0.94$ $\pm 0.002$ | $0.63$ $\pm 0.008$ |
| End of question | $0.71$ $\pm 0.006$ | $0.53$ $\pm 0.004$ | $0.48$ $\pm 0.003$ | $0.91$ $\pm 0.001$ | $0.66$ $\pm 0.004$ |
| Exact answer last | $0.81$ $\pm 0.006$ | $0.77$ $\pm 0.004$ | 0.65 $\pm 0.004$ | 0.94 $\pm 0.002$ | 0.75 $\pm 0.008$ |
| Exact answer last+1 | 0.82 $\pm 0.004$ | 0.79 $\pm 0.001$ | $0.57$ $\pm 0.004$ | $0.90$ $\pm 0.002$ | $0.75$ $\pm 0.007$ |

**Table 7: Comparison of error detection performance (AUC) on Llama-8b-Instruct.**
|  | Llama-8b-Instruct |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | TriviaQA | Winobias | Math | Movies | IMDB |
| Logits-mean | $0.66$ $\pm 0.005$ | $0.60$ $\pm 0.026$ | $0.75$ $\pm 0.018$ | $0.75$ $\pm 0.005$ | $0.59$ $\pm 0.017$ |
| Logits-mean-exact | $0.71$ $\pm 0.006$ | $0.55$ $\pm 0.019$ | $0.80$ $\pm 0.021$ | $0.72$ $\pm 0.004$ | $0.88$ $\pm 0.012$ |
| Logits-min | $0.74$ $\pm 0.007$ | $0.61$ $\pm 0.024$ | $0.75$ $\pm 0.016$ | $0.71$ $\pm 0.005$ | $0.55$ $\pm 0.016$ |
| Logits-min-exact | $0.79$ $\pm 0.006$ | $0.61$ $\pm 0.019$ | $0.89$ $\pm 0.018$ | $0.77$ $\pm 0.006$ | $0.88$ $\pm 0.012$ |
| Logits-max | $0.54$ $\pm 0.007$ | $0.55$ $\pm 0.013$ | $0.73$ $\pm 0.027$ | $0.67$ $\pm 0.003$ | $0.51$ $\pm 0.009$ |
| Logits-max-exact | $0.58$ $\pm 0.005$ | $0.54$ $\pm 0.019$ | $0.64$ $\pm 0.014$ | $0.61$ $\pm 0.003$ | $0.88$ $\pm 0.012$ |
| Probas-mean | $0.67$ $\pm 0.006$ | $0.63$ $\pm 0.024$ | $0.66$ $\pm 0.033$ | $0.73$ $\pm 0.006$ | $0.73$ $\pm 0.015$ |
| Probas-mean-exact | $0.75$ $\pm 0.009$ | $0.61$ $\pm 0.014$ | $0.83$ $\pm 0.022$ | $0.74$ $\pm 0.005$ | $0.74$ $\pm 0.021$ |
| Probas-min | $0.67$ $\pm 0.009$ | $0.65$ $\pm 0.019$ | $0.64$ $\pm 0.036$ | $0.65$ $\pm 0.004$ | $0.57$ $\pm 0.016$ |
| Probas-min-exact | $0.79$ $\pm 0.008$ | $0.62$ $\pm 0.014$ | $0.86$ $\pm 0.024$ | $0.74$ $\pm 0.005$ | $0.74$ $\pm 0.021$ |
| Probas-max | $0.54$ $\pm 0.003$ | $0.49$ $\pm 0.020$ | $0.57$ $\pm 0.022$ | $0.64$ $\pm 0.006$ | $0.49$ $\pm 0.008$ |
| Probas-max-exact | $0.56$ $\pm 0.007$ | $0.55$ $\pm 0.016$ | $0.57$ $\pm 0.018$ | $0.61$ $\pm 0.003$ | $0.74$ $\pm 0.021$ |
| p(True) | $0.73$ $\pm 0.008$ | $0.59$ $\pm 0.020$ | $0.62$ $\pm 0.017$ | $0.66$ $\pm 0.004$ | $0.60$ $\pm 0.006$ |
| p(True)-exact | $0.73$ $\pm 0.005$ | $0.63$ $\pm 0.014$ | $0.59$ $\pm 0.018$ | $0.63$ $\pm 0.006$ | $0.76$ $\pm 0.004$ |
| Probe @ token |  |  |  |  |  |
| Last generated [-1] | $0.81$ $\pm 0.005$ | $0.86$ $\pm 0.007$ | $0.82$ $\pm 0.016$ | $0.78$ $\pm 0.004$ | $0.81$ $\pm 0.014$ |
| Before last generated [-2] | $0.75$ $\pm 0.005$ | $0.88$ $\pm 0.005$ | $0.79$ $\pm 0.020$ | $0.82$ $\pm 0.005$ | $0.83$ $\pm 0.006$ |
| End of question | $0.77$ $\pm 0.007$ | $0.80$ $\pm 0.018$ | $0.72$ $\pm 0.023$ | $0.76$ $\pm 0.005$ | $0.87$ $\pm 0.006$ |
| Exact answer last | 0.83 $\pm 0.002$ | 0.93 $\pm 0.004$ | 0.95 $\pm 0.027$ | $0.85$ $\pm 0.005$ | 0.96 $\pm 0.003$ |
| Exact answer last+1 | $0.83$ $\pm 0.006$ | $0.90$ $\pm 0.005$ | $0.94$ $\pm 0.023$ | 0.86 $\pm 0.004$ | $0.95$ $\pm 0.004$ |
|  | HotpotQA | HotpotQA-WC | Winogrande | NLI | NQ-WC |
| Logits-mean | $0.65$ $\pm 0.002$ | $0.56$ $\pm 0.004$ | $0.58$ $\pm 0.007$ | $0.59$ $\pm 0.009$ | $0.65$ $\pm 0.006$ |
| Logits-mean-exact | $0.66$ $\pm 0.008$ | $0.57$ $\pm 0.005$ | $0.48$ $\pm 0.003$ | $0.49$ $\pm 0.010$ | $0.67$ $\pm 0.005$ |
| Logits-min | $0.67$ $\pm 0.008$ | $0.55$ $\pm 0.007$ | $0.60$ $\pm 0.008$ | $0.53$ $\pm 0.009$ | $0.68$ $\pm 0.004$ |
| Logits-min-exact | $0.76$ $\pm 0.010$ | $0.65$ $\pm 0.010$ | $0.48$ $\pm 0.004$ | $0.50$ $\pm 0.009$ | $0.68$ $\pm 0.004$ |
| Logits-max | $0.59$ $\pm 0.005$ | $0.56$ $\pm 0.005$ | $0.46$ $\pm 0.004$ | $0.55$ $\pm 0.013$ | $0.56$ $\pm 0.006$ |
| Logits-max-exact | $0.52$ $\pm 0.006$ | $0.48$ $\pm 0.002$ | $0.48$ $\pm 0.003$ | $0.49$ $\pm 0.009$ | $0.63$ $\pm 0.008$ |
| Probas-mean | $0.61$ $\pm 0.002$ | $0.56$ $\pm 0.010$ | $0.57$ $\pm 0.007$ | $0.58$ $\pm 0.007$ | $0.65$ $\pm 0.007$ |
| Probas-mean-exact | $0.68$ $\pm 0.008$ | $0.65$ $\pm 0.006$ | $0.51$ $\pm 0.006$ | $0.57$ $\pm 0.009$ | $0.67$ $\pm 0.003$ |
| Probas-min | $0.60$ $\pm 0.004$ | $0.51$ $\pm 0.007$ | $0.59$ $\pm 0.007$ | $0.55$ $\pm 0.005$ | $0.64$ $\pm 0.008$ |
| Probas-min-exact | $0.74$ $\pm 0.007$ | $0.67$ $\pm 0.007$ | $0.51$ $\pm 0.006$ | $0.59$ $\pm 0.008$ | $0.66$ $\pm 0.004$ |
| Probas-max | $0.56$ $\pm 0.005$ | $0.53$ $\pm 0.005$ | $0.46$ $\pm 0.003$ | $0.51$ $\pm 0.004$ | $0.55$ $\pm 0.004$ |
| Probas-max-exact | $0.49$ $\pm 0.007$ | $0.47$ $\pm 0.002$ | $0.51$ $\pm 0.005$ | $0.50$ $\pm 0.009$ | $0.62$ $\pm 0.006$ |
| p(True) | $0.55$ $\pm 0.005$ | $0.55$ $\pm 0.008$ | $0.47$ $\pm 0.002$ | $0.54$ $\pm 0.006$ | $0.71$ $\pm 0.003$ |
| p(True)-exact | $0.55$ $\pm 0.004$ | $0.50$ $\pm 0.005$ | $0.50$ $\pm 0.008$ | $0.50$ $\pm 0.003$ | $0.67$ $\pm 0.007$ |
| Probe @ token |  |  |  |  |  |
| Last generated [-1] | $0.77$ $\pm 0.005$ | $0.68$ $\pm 0.006$ | $0.69$ $\pm 0.006$ | $0.78$ $\pm 0.005$ | $0.77$ $\pm 0.009$ |
| Before last generated [-2] | $0.76$ $\pm 0.002$ | $0.69$ $\pm 0.005$ | $0.67$ $\pm 0.008$ | $0.79$ $\pm 0.004$ | $0.75$ $\pm 0.007$ |
| End of question | $0.78$ $\pm 0.004$ | $0.60$ $\pm 0.003$ | $0.65$ $\pm 0.004$ | $0.74$ $\pm 0.002$ | $0.75$ $\pm 0.011$ |
| Exact answer last | 0.83 $\pm 0.005$ | 0.76 $\pm 0.003$ | 0.78 $\pm 0.007$ | 0.91 $\pm 0.005$ | 0.78 $\pm 0.006$ |
| Exact answer last+1 | $0.83$ $\pm 0.002$ | $0.76$ $\pm 0.006$ | $0.70$ $\pm 0.006$ | $0.90$ $\pm 0.004$ | $0.78$ $\pm 0.007$ |

## Appendix C Full Generalization Results

Figures 7, 8 and 9 present the generalization results for the remaining models.
While these results exhibit similar high-level patterns to those found in the main paper on Mistral-7b-instruct, notable differences suggest that these models may possess different mechanisms for encoding truthfulness.

Figure: (a) Raw AUC values. Values above $0.5$ indicate some generalization.
Refer to caption: extracted/5959149/figures/generalization/mistral.png

Figure: (a) Raw AUC values. Values above $0.5$ indicate some generalization.
Refer to caption: extracted/5959149/figures/generalization/llama.png

Figure: (a) Raw AUC values. Values above $0.5$ indicate some generalization.
Refer to caption: extracted/5959149/figures/generalization/llama_instruct.png

## Appendix D Taxonomy of Errors

Figure: Figure 10: The percentage of answers for which at least one generated answer was correct. The first step is greedy decoding.
Refer to caption: extracted/5959149/figures/correctness_across_resamples.png

Figure 10 presents, for each amount of resamples, the amount percentage of answers for which at least one generated answer was correct.
The experiment was done on Mistral-7b-instruct with the TriviaQA dataset.
For many answers that the greedy decoding fails to correctly provide an answer, the LLM is still able to generate the correct answer in at least one resample.
The plot plateues around 30 resamples.

## Appendix E Detecting the Correct Answer Full Results

In Table 8 we present some qualitative samples from Mistral-7b-instruct, for the phenomenon we observe at error type (C2) Consistently incorrect but generates the correct answer at least one time.
The samples in the table represent cases where the probe chose the correct answer.
Table 9 compares different decoding mechanisms, including the choice via probe, on non-instruct models, and Table 10 compares on the instruct models.
For all datasets and models, we observe similar conclusions to those in the main paper: significant improvement is observed for error types where the LLM shows no preference to the correct answer.

**Table 8: Examples of questions where Mistral-7b-Instruct consistently provided incorrect answers but occasionally generated the correct one. In these instances, the probe successfully identified the right answer. For each question, the model was samples 30 times.**
| Question | Wrong<br>Answer | Count | Correct<br>Answer | Count |
| --- | --- | --- | --- | --- |
| Which town in southeast<br>Wales became a UNESCO<br>World Heritage Site in 2000? | Caerleon | 29 | Blaenavon | 1 |
| From her first US film musical<br>”Down Argentina Way” (1940),<br>who became famous for<br>extravagant hats, jewellery and<br>dresses? | Betty<br>Grable | 27 | Carmen<br>Miranda | 1 |
| Men Against the Sea and<br>Pitcairn’s Island were two<br>sequels to what famous novel? | Robinson<br>Crusoe | 18 | Mutiny on<br>the Bounty | 2 |
| Which is the only property on a<br>traditional UK Monopoly board<br>which s south of the River<br>Thames? | Coventry<br>Street | 17 | Old Kent<br>Road | 3 |
| Which French Canadian became<br>Prime Minister of<br>Canada in 1968? | Jean<br>Chrétien | 21 | Pierre<br>Elliott<br>Trudeau | 4 |

**Table 9: Various answer choice strategies, non-instruct models.**
|  | Mistral-7b |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | TriviaQA | Math | Winobias |  |  |  |  |  |  |  |  |  |
| Error type | Greedy | Random | Majority | Probing | Greedy | Random | Majority | Probing | Greedy | Random | Majority | Probing |
| All | $0.63$ $\pm 0.003$ | $0.54$ $\pm 0.004$ | $0.65$ $\pm 0.002$ | $0.62$ $\pm 0.003$ | $0.25$ $\pm 0.018$ | $0.36$ $\pm 0.022$ | $0.49$ $\pm 0.019$ | $0.60$ $\pm 0.017$ | $0.69$ $\pm 0.016$ | $0.58$ $\pm 0.009$ | $0.62$ $\pm 0.009$ | $0.83$ $\pm 0.006$ |
| (A) Refuses to answer | $0.08$ $\pm 0.015$ | $0.04$ $\pm 0.009$ | $0.00$ $\pm 0.000$ | $0.13$ $\pm 0.007$ | $0.01$ $\pm 0.009$ | $0.04$ $\pm 0.019$ | $0.00$ $\pm 0.000$ | $0.22$ $\pm 0.033$ | - | - | - | - |
| (B1) All | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | - | - | - | - | - | - | - | - |
| (B2) Most | $0.98$ $\pm 0.001$ | $0.84$ $\pm 0.009$ | $1.00$ $\pm 0.000$ | $0.91$ $\pm 0.002$ | $0.96$ $\pm 0.024$ | $0.84$ $\pm 0.031$ | $1.00$ $\pm 0.000$ | $0.86$ $\pm 0.041$ | $0.96$ $\pm 0.004$ | $0.73$ $\pm 0.009$ | $0.95$ $\pm 0.003$ | $0.91$ $\pm 0.009$ |
| (C) Consistently incorrect |  |  |  |  |  |  |  |  |  |  |  |  |
| (C1) All | $0.00$ $\pm 0.003$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | - | - | - | - | - | - | - | - |
| (C2) Most | $0.03$ $\pm 0.014$ | $0.20$ $\pm 0.008$ | $0.00$ $\pm 0.000$ | $0.27$ $\pm 0.036$ | - | - | - | - | $0.19$ $\pm 0.010$ | $0.30$ $\pm 0.026$ | $0.00$ $\pm 0.000$ | $0.70$ $\pm 0.007$ |
| (D) Two competing | $0.48$ $\pm 0.006$ | $0.36$ $\pm 0.008$ | $0.52$ $\pm 0.015$ | $0.54$ $\pm 0.016$ | - | - | - | - | $0.73$ $\pm 0.018$ | $0.54$ $\pm 0.022$ | $0.47$ $\pm 0.030$ | $0.85$ $\pm 0.019$ |
| (E) Many answers |  |  |  |  |  |  |  |  |  |  |  |  |
| (E1) Non correct | $0.01$ $\pm 0.004$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.01$ $\pm 0.010$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | - | - | - | - |
| (E2) Correct appears | $0.38$ $\pm 0.009$ | $0.21$ $\pm 0.006$ | $0.42$ $\pm 0.015$ | $0.38$ $\pm 0.009$ | $0.09$ $\pm 0.010$ | $0.17$ $\pm 0.034$ | $0.36$ $\pm 0.020$ | $0.62$ $\pm 0.035$ | - | - | - | - |
|  | Llama-8b |  |  |  |  |  |  |  |  |  |  |  |
|  | TriviaQA | Math | Winobias |  |  |  |  |  |  |  |  |  |
| Error type | Greedy | Sampling | Majority | Probing | Greedy | Sampling | Majority | Probing | Greedy | Sampling | Majority | Probing |
| All | $0.66$ $\pm 0.002$ | $0.58$ $\pm 0.003$ | 0.68 $\pm 0.003$ | 0.68 $\pm 0.002$ | $0.30$ $\pm 0.023$ | $0.47$ $\pm 0.022$ | $0.62$ $\pm 0.014$ | $0.70$ $\pm 0.021$ | $0.73$ $\pm 0.011$ | $0.61$ $\pm 0.005$ | $0.66$ $\pm 0.016$ | 0.84 $\pm 0.006$ |
| (A) Refuses to answer | $0.08$ $\pm 0.005$ | $0.07$ $\pm 0.011$ | $0.00$ $\pm 0.000$ | 0.16 $\pm 0.011$ | $0.00$ $\pm 0.007$ | $0.04$ $\pm 0.015$ | $0.00$ $\pm 0.000$ | $0.25$ $\pm 0.025$ | - | - | - | - |
| (B) Consistently correct |  |  |  |  |  |  |  |  |  |  |  |  |
| (B1) All | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | - | - | - | - | - | - | - | - |
| (B2) Most | $0.98$ $\pm 0.001$ | $0.87$ $\pm 0.002$ | 1.00 $\pm 0.000$ | $0.95$ $\pm 0.002$ | $0.77$ $\pm 0.024$ | $0.88$ $\pm 0.025$ | $1.00$ $\pm 0.000$ | $0.97$ $\pm 0.014$ | $0.98$ $\pm 0.005$ | $0.75$ $\pm 0.004$ | 1.00 $\pm 0.000$ | $0.94$ $\pm 0.003$ |
| (C) Consistently incorrect |  |  |  |  |  |  |  |  |  |  |  |  |
| (C1) All | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | - | - | - | - | - | - | - | - |
| (C2) Most | $0.06$ $\pm 0.013$ | $0.18$ $\pm 0.009$ | $0.00$ $\pm 0.000$ | 0.35 $\pm 0.043$ | - | - | - | - | $0.25$ $\pm 0.026$ | $0.29$ $\pm 0.023$ | $0.00$ $\pm 0.000$ | 0.65 $\pm 0.022$ |
| (D) Two competing | $0.44$ $\pm 0.029$ | $0.42$ $\pm 0.035$ | $0.53$ $\pm 0.020$ | 0.66 $\pm 0.030$ | - | - | - | - | $0.73$ $\pm 0.025$ | $0.47$ $\pm 0.019$ | $0.41$ $\pm 0.037$ | 0.86 $\pm 0.014$ |
| (E) Many answers |  |  |  |  |  |  |  |  |  |  |  |  |
| (E1) Non correct | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | - | - | - | - |
| (E2) Correct appears | $0.46$ $\pm 0.009$ | $0.34$ $\pm 0.009$ | $0.53$ $\pm 0.007$ | 0.54 $\pm 0.005$ | $0.14$ $\pm 0.015$ | $0.17$ $\pm 0.025$ | $0.44$ $\pm 0.047$ | $0.65$ $\pm 0.031$ | - | - | - | - |

**Table 10: Various answer choice strategies, instruct models.**
|  | Mistral-7b-Instruct |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | TriviaQA | Math | Winobias |  |  |  |  |  |  |  |  |  |
| Error type | Greedy | Random | Majority | Probing | Greedy | Random | Majority | Probing | Greedy | Random | Majority | Probing |
| All | $0.63$ $\pm 0.003$ | $0.64$ $\pm 0.002$ | $0.67$ $\pm 0.004$ | $0.71$ $\pm 0.003$ | $0.55$ $\pm 0.021$ | $0.52$ $\pm 0.019$ | $0.57$ $\pm 0.025$ | $0.70$ $\pm 0.014$ | $0.77$ $\pm 0.012$ | $0.77$ $\pm 0.008$ | $0.77$ $\pm 0.010$ | $0.79$ $\pm 0.008$ |
| (A) Refuses to answer | $0.06$ $\pm 0.005$ | $0.06$ $\pm 0.011$ | $0.00$ $\pm 0.000$ | $0.28$ $\pm 0.009$ | - | - | - | - | - | - | - | - |
| (B1) All | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ |
| (B2) Most | $0.88$ $\pm 0.007$ | $0.83$ $\pm 0.009$ | $0.99$ $\pm 0.002$ | $0.89$ $\pm 0.010$ | $0.87$ $\pm 0.013$ | $0.84$ $\pm 0.024$ | $1.00$ $\pm 0.000$ | $0.96$ $\pm 0.007$ | $0.91$ $\pm 0.031$ | $0.87$ $\pm 0.029$ | $0.96$ $\pm 0.017$ | $0.89$ $\pm 0.032$ |
| (C) Consistently incorrect |  |  |  |  |  |  |  |  |  |  |  |  |
| (C1) All | $0.00$ $\pm 0.003$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.05$ $\pm 0.020$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ |
| (C2) Most | $0.11$ $\pm 0.009$ | $0.15$ $\pm 0.012$ | $0.00$ $\pm 0.000$ | $0.53$ $\pm 0.005$ | $0.10$ $\pm 0.040$ | $0.20$ $\pm 0.050$ | $0.00$ $\pm 0.000$ | $0.82$ $\pm 0.037$ | $0.18$ $\pm 0.057$ | $0.20$ $\pm 0.039$ | $0.00$ $\pm 0.000$ | $0.54$ $\pm 0.067$ |
| (D) Two competing | $0.32$ $\pm 0.010$ | $0.45$ $\pm 0.023$ | $0.50$ $\pm 0.024$ | $0.78$ $\pm 0.017$ | - | - | - | - | - | - | - | - |
| (E) Many answers |  |  |  |  |  |  |  |  |  |  |  |  |
| (E1) Non correct | $0.01$ $\pm 0.003$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | - | - | - | - | - | - | - | - |
| (E2) Correct appears | $0.23$ $\pm 0.020$ | $0.19$ $\pm 0.022$ | $0.38$ $\pm 0.009$ | $0.56$ $\pm 0.025$ | - | - | - | - | - | - | - | - |
|  | Llama-8b-Instruct |  |  |  |  |  |  |  |  |  |  |  |
|  | TriviaQA | Math | Winobias |  |  |  |  |  |  |  |  |  |
| Error type | Greedy | Sampling | Majority | Probing | Greedy | Sampling | Majority | Probing | Greedy | Sampling | Majority | Probing |
| All | $0.69$ $\pm 0.003$ | $0.67$ $\pm 0.001$ | $0.71$ $\pm 0.002$ | 0.73 $\pm 0.004$ | $0.89$ $\pm 0.010$ | $0.87$ $\pm 0.012$ | 0.91 $\pm 0.013$ | 0.91 $\pm 0.010$ | $0.75$ $\pm 0.009$ | $0.74$ $\pm 0.009$ | $0.76$ $\pm 0.012$ | 0.83 $\pm 0.009$ |
| (A) Refuses to answer | $0.06$ $\pm 0.011$ | $0.05$ $\pm 0.011$ | $0.00$ $\pm 0.000$ | 0.27 $\pm 0.025$ | - | - | - | - | - | - | - | - |
| (B) Consistently correct |  |  |  |  |  |  |  |  |  |  |  |  |
| (B1) All | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ | $1.00$ $\pm 0.000$ |
| (B2) Most | $0.93$ $\pm 0.002$ | $0.86$ $\pm 0.009$ | 1.00 $\pm 0.001$ | $0.92$ $\pm 0.004$ | $0.94$ $\pm 0.014$ | $0.92$ $\pm 0.014$ | 1.00 $\pm 0.000$ | $0.95$ $\pm 0.013$ | $0.94$ $\pm 0.006$ | $0.88$ $\pm 0.010$ | $1.00$ $\pm 0.000$ | $0.93$ $\pm 0.011$ |
| (C) Consistently incorrect |  |  |  |  |  |  |  |  |  |  |  |  |
| (C1) All | $0.00$ $\pm 0.001$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | - | - | - | - | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ |
| (C2) Most | $0.12$ $\pm 0.018$ | $0.22$ $\pm 0.010$ | $0.00$ $\pm 0.000$ | 0.43 $\pm 0.010$ | - | - | - | - | $0.11$ $\pm 0.018$ | $0.15$ $\pm 0.025$ | $0.00$ $\pm 0.000$ | 0.67 $\pm 0.016$ |
| (D) Two competing | $0.43$ $\pm 0.017$ | $0.42$ $\pm 0.014$ | $0.46$ $\pm 0.016$ | 0.60 $\pm 0.010$ | - | - | - | - | $0.39$ $\pm 0.068$ | $0.39$ $\pm 0.047$ | $0.38$ $\pm 0.042$ | 0.83 $\pm 0.050$ |
| (E) Many answers |  |  |  |  |  |  |  |  |  |  |  |  |
| (E1) Non correct | $0.00$ $\pm 0.002$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | $0.00$ $\pm 0.000$ | - | - | - | - | - | - | - | - |
| (E2) Correct appears | $0.28$ $\pm 0.006$ | $0.28$ $\pm 0.008$ | $0.40$ $\pm 0.009$ | 0.52 $\pm 0.009$ | - | - | - | - | - | - | - | - |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Have the authors tested if the taxonomy of error types is consistent across various domains or tasks, and if not, would this be feasible in future work? 

2. Could the authors explain why certain model layers are more effective for probing in error detection? A discussion on this might enhance understanding of how internal representations encode truthfulness.

3. Have the authors considered using a context window around answer tokens to improve detection accuracy, especially for tasks that involve longer responses or require more context? 

4. Given the diversity in model architectures tested, are there certain architectures where the probing method performs noticeably better or worse?
soundness: 2
strengths: - By probing specific tokens associated with “correct” answers, the authors highlight underexplored dimensions of LLM truthfulness encoding. This approach captures nuanced error signals that extend beyond surface-level accuracy. 

- The paper demonstrates robustness by testing the approach on a variety of LLM architectures (e.g., Mistral-7b, Llama3-8b). This extensive model comparison strengthens the study’s findings by showing consistency in truthfulness encoding patterns across different architectures. 

- The detailed taxonomy of error types, including categories like “two competing answers” and “consistent incorrectness,” could help model developers diagnose and prioritize error types during model fine-tuning. 

- The study’s finding that truthfulness encoding is not universal but rather task-specific introduces an important consideration for cross-task applications.
summary: This paper examines the internal representations of large language models (LLMs) to understand how they encode truthfulness and produce hallucinations—errors such as factual inaccuracies and reasoning failures. Through an approach involving probing classifiers, the authors identify truthfulness signals in specific tokens, particularly in exact answer tokens. Experiments across multiple LLMs reveal that truthfulness is task-specific, with signals varying by the nature of the task. Additionally, the study develops a taxonomy of error types, finding that LLMs may encode the correct answer internally while producing an incorrect one externally, suggesting areas where error mitigation could be applied.
weaknesses: - While probing classifiers show strong results, the interpretability of their outputs remains unclear. For example, understanding how specific probing layer choices affect error predictions or how developers might interpret these signals is crucial for practical adoption but is not fully addressed. Including examples of how different layers and token choices impact classifier predictions would clarify the interpretability of this approach. A case study showing how interpretability of classifier outputs could inform model improvements would also be beneficial. 

- The error taxonomy could be more robust if supported by quantitative analyses on how consistently each error type appears across varying prompts or domains. Currently, it is unclear if the identified error types remain stable under different experimental conditions or across datasets. Conducting further experiments to determine if error types remain consistent under prompt variation or across domains (e.g., factual vs. commonsense tasks) could enhance the reliability of the taxonomy. These results would provide stronger support for the taxonomy as a practical tool. 

- The framework’s token-level focus may overlook broader context, potentially missing how errors manifest within larger segments of generated text. For tasks like long-form QA or summarization, this might limit the approach’s utility by not capturing interactions between tokens. Testing whether incorporating context windows around answer tokens, or analyzing sentences rather than isolated tokens, would improve accuracy could make the approach more adaptable to contextually rich tasks. Including examples where broader context improves error prediction would also strengthen this aspect. 

- Practical guidance on integrating these insights into model development workflows is limited. Developers may find it challenging to apply these techniques without clear implementation guidelines.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: - This approach could be only applicable to QA tasks. For open-ended questions, can it be applied? It would be better to provide performance on these kinds of tasks as well.
- The results in table 1 show that the AUC of last generated almost match that of exact answer tokens with TriviaQA. Does this imply that actually the representations of last token and answer exact token are almost the same, especially for more general hallucinations rather than specific factual ones such as math problems? Besides, the result of Mistral is worse than that of LLaMA 3.1. Is it because the answer of Mistral involves less explanations to the question than that of LLaMA 3.1? The results in table 2 seems also demonstrate that answer exact tokens are barely better than last tokens when it comes to more generic tasks?
soundness: 4
strengths: - This paper interprets halluciantions from the perspective of token representations, particularly the exact answer tokens, which provides a relatively new perspective.
- This work demonstrates that the internal representations of LLMs' responses are skill specific rather than universal.
- The paper is well-structured and the experiments are comprehensive to demonstrate the validity of the findings.
summary: This paper proposes that the internal representations of LLMs' responses are predictive of error patterns. Unlike previous approaches focusing on representations of last token or averaged representations, this paper focuses on representations of exact answer tokens.
weaknesses: This paper has generated some interesting findings. However, the method is hardly to be taken as general.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - It would be better to clarify the specific model used for the probing classifier and provide more details on the training setup (e.g., hyperparameters, optimizer, detailed dataset split information).
- From the experimental results presented in Section 4, it seems that the generalization ability of the probing classifier across different tasks is relatively limited. For instance, many AUC scores are in the range of 0.5 to 0.6, which suggests that the classifier's performance is only slightly better than random guessing. Could you elaborate on possible reasons for this limited generalization, and suggest any potential ways to enhance it?
- Have you considered conducting generalization tests across different LLMs, in addition to testing across different datasets? For instance, how would a probing classifier trained on output generated by Mistral-7b perform when tested on Llama3-8b? This approach could provide better insight into whether the probing classifier remains effective when applied to activation states across different LLMs.
soundness: 3
strengths: - The paper is well-structured, and the motivation is clearly communicated.
- The experiments effectively demonstrate how probing classifiers reveal the internal mechanisms of LLMs.
- The results highlight significant discrepancies between internal representations and generated outputs, offering a deeper understanding of how hallucinations manifest and potential ways to mitigate them.
summary: This paper presents a novel approach to investigating hallucinations in LLMs by using probing classifiers to analyze their internal representations. The approach involves analyzing internal model representations to determine how LLMs handle factual versus hallucinatory information. The proposed approach is evaluated through multiple experiments on various LLMs and datasets, offering new insights into understanding and mitigating hallucinations in these models.
weaknesses: See questions.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: - **On p.3, Section 3.1**: Which model, $M$, did you use to generate responses when constructing the triplet dataset?
- **On p.5**: What is the accuracy of Mistral-7b-Instruct when extracting the exact answer token? Or did you assume that all extracted tokens are correct without further validation?
- **On p.5, fig. 2**: The statement “Middle to later layers typically yield the most effective probing results” seems inaccurate. The figure does not show a clear pattern supporting this claim.
- **On p.8**: Why were the error patterns not designed to be orthogonal? Could you provide a definitive reason for this decision?
- **On p.10, fig.5**: What do the different colored bars represent in the figure? Please clarify the meaning of the different colors.
soundness: 4
strengths: - Originality: The understanding of hallucination is accurate, and the exploration of the model’s self-knowledge capabilities provides deeper insights. Perspective of shifting focus from human-centric to model-centric is pinpoint and insightful.
- Quality: The experiments are thorough and well-executed.
- Clarity: The paper is structured clearly, making it easy to follow.
- Significance: This work is valuable for future research, especially for studies focusing on how models internally grasp factuality.
summary: This paper conducts extensive experiments and verifies through probing that LLMs possess an implicit understanding of truthfulness within their hidden representations. However, due to certain calibration issues, even though the model "knows" the truth, it can still produce incorrect outputs. Additionally, further experiments show that probing is not a general-purpose tool for detecting hallucinations; it is task-specific.
weaknesses: - **On the definition of hallucination**: The paper's definition of hallucination remains somewhat vague. As discussed in [this paper](https://arxiv.org/abs/2407.14507), hallucination can be understood as *inconsistency* between a model’s expression and world knowledge it has learned from its training corpus.

- **Missing experiments**: On p.6, the paper states that “the primary conclusion is that information about truthfulness is highly localized in specific generated tokens.” Given this claim, I believe an additional experiment is necessary: measuring the accuracy of the model when prompted to produce exact answers under restricted conditions. This would provide more direct evidence supporting the localization of truthfulness in generated tokens.

- **Lack of clear direction for next steps**: The paper does not explicitly outline what future steps can be taken. If we now accept that "the model latently and task-specifically knows what truthfulness is," how can we leverage this finding to help a general-purpose LLM handle unpredictable tasks? Or, how can this work help researches avoid futile efforts? The practical applications of these insights are not clearly addressed.

Meta-review: This paper proposes a new approach to investigate hallucinations in LLMs by using probing classifiers to analyze their internal representations. Reviewers found that the paper provides some interesting findings, e.g., the internal representations of LLMs' responses are skill specific rather than universal. Also, reviewers mostly agreed that the paper is well structured and the experimental results are extensive and convincing. Overall, this paper is well written, and it contributes to the understanding of LLM's internal representations.

Decision: Accept (Poster)