Are the Hidden States Hiding Something? Testing the Limits of Factuality-Encoding Capabilities in LLMs
Authors: Giovanni Servedio, Alessandro De Bellis, Dario Di Palma, Vito Walter Anelli, Tommaso Di Noia
Accepted at: acl25

Are the Hidden States Hiding Something? Testing the Limits of Factuality-Encoding Capabilities in LLMs

Giovanni Servedio1,2†    Alessandro De Bellis1†    Dario Di Palma1
Vito Walter Anelli1    Tommaso Di Noia1
1Politecnico di Bari, Italy    2Sapienza University of Rome, Italy
name.surname@poliba.it

Abstract
Factual hallucinations are a major challenge for Large Language Models (LLMs). They undermine reliability and user trust by generating inaccurate or fabricated content. Recent studies suggest that when generating false statements, the internal states of LLMs encode information about truthfulness. However, these studies often rely on synthetic datasets that lack realism, which limits generalization when evaluating the factual accuracy of text generated by the model itself. In this paper, we challenge the findings of previous work by investigating truthfulness encoding capabilities, leading to the generation of a more realistic and challenging dataset. Specifically, we extend previous work by introducing: (1) a strategy for sampling plausible true-false factoid sentences from tabular data and (2) a procedure for generating realistic, LLM-dependent true-false datasets from Question Answering collections. Our analysis of two open-source LLMs reveals that while the findings from previous studies are partially validated, generalization to LLM-generated datasets remains challenging. This study provides a foundation for future research on factuality in LLMs and offers practical guidelines for more effective evaluation. Code is provided at our GitHub Repository.

1 Introduction
In the last few years, Large Language Models (LLMs) have shown outstanding abilities in natural language processing tasks and beyond (Biancofiore et al., 2025; Di Palma, 2023). Nevertheless, factual hallucinations (Zhang et al., 2023) represent a significant obstacle, limiting their reliability and hindering their safe deployment in real-world applications (Di Palma et al., 2025) such as healthcare (Pham and Vo, 2024), education (Upadhyay et al., 2023), legal advice (Dahl et al., 2024), and language understanding (De Bellis et al., 2024; Anelli et al., 2022). Hallucinations occur when an LLM generates content that is syntactically coherent but factually inaccurate, decreasing trust in AI systems (Huang et al., 2024). Recent research suggests that LLMs may encode internal representations of factuality in their hidden states, indicating an awareness of whether a generated statement is true or false (Chen et al., 2024). These efforts led to the development of approaches to evaluate the factual accuracy of the LLM outputs given their internal representations (factuality "self-evaluation"). Self-evaluation can be used to identify gaps in the knowledge of an LLM, improving truthfulness and transparency through abstention mechanisms (Feng et al., 2024), fact verification (Wadden et al., 2020), and self-correction (Ji et al., 2023). Azaria and Mitchell (2023) suggest that LLMs have “some internal notion as to whether a sentence is true or false, as this information is required for generating (or predicting) following tokens.” Based on this assumption, they propose a neural classifier to discern factual from non-factual statements based on hidden layer activations. However, the datasets used to evaluate the probe present limitations since they contain trivially incorrect statements (e.g., “The zebra uses flying for locomotion”) that easily fail to align with the generative patterns of LLMs. Additionally, the false statements are generated using random substitutions of the true terms with little regard for the plausibility of negative samples. This misalignment not only weakens the generalizability of results but also raises concerns about the applicability of these models to real-world scenarios where false statements may be subtle or nuanced. This study addresses these gaps by generating more plausible datasets (see Figure 1) to explore LLM factuality encoding and evaluating refined models. The primary contributions are:

1. We reproduce the methodology of Azaria and Mitchell (2023) to ensure transparency.
2. We propose two strategies to generate realistic datasets and discuss how well the original and newly developed models generalize.

Specifically, we extend their work with two novel dataset creation strategies and design a strategy that better fits the factuality self-evaluation task, introducing:
• A perplexity-based negative sampling strategy that enhances the original generation mechanism and leverages the LLM token distribution.
• A novel strategy to sample realistic LLM-generated facts, leveraging Question Answering datasets to elicit responses from the LLM.

Through this analysis, we lay the groundwork for more robust factuality assessments and offer practical guidelines for enhancing the reliability of LLMs in diverse applications.

2 Reproduction of Prior Work: Settings

In their study, Azaria and Mitchell (2023) investigate whether LLMs internally represent the factuality of sentences. This section summarizes the dataset generation approach they employed and the specific probing architecture used in their study.

2.1 Dataset Generation Strategy

To explore whether LLMs internally represent the factuality of statements, the authors constructed a ‘True-False’ dataset of facts labeled as either True or False, covering six disjoint topics: Cities, Inventions, Chemical Elements, Animals, Companies, and Scientific Facts. To generate the dataset, for the first five topics, i.e. with the exception of Scientific Facts, the authors selected true statements from reliable sources (see Table 7 in the appendix) and produced false statements, replacing part of a true statement (e.g. “Hydrogen has an atomic number of 1”) with randomly sampled incorrect information (“Hydrogen has an atomic number of 34”). Meanwhile, for Scientific Facts, they employed ChatGPT (13 Feb 2023) as a generator of true and false sentences, and two human annotators manually verified their correctness. The authors publicly release the dataset, which we refer to as the “True-False dataset”. Furthermore, the authors constructed a second dataset using the OPT-6.7b model, which we refer to as the ‘OPT-Generated Dataset’. To create this dataset, the model was prompted with a true statement absent from the True-False dataset and then used to generate a subsequent sentence. The responses were manually fact-checked and annotated by three independent human judges. Non-factual responses were filtered out, resulting in a final set of 245 statements.

2.2 Internal States Analysis via SAPLMA

To investigate whether LLMs internally represent the factuality of statements, Azaria and Mitchell (2023) developed a probe (Statement Accuracy Prediction based on Language Model Activations) that predicts the factual accuracy of a statement by analyzing the hidden layer activations of an LLM. SAPLMA is a feedforward neural network designed to classify statements as true or false. It consists of three hidden layers (256, 128, 64) and a sigmoid output activation. The model is trained using the Adam optimizer for five epochs without hyperparameter tuning. The authors studied two LLMs, namely OPT-6.7b (Zhang et al., 2022) and Llama 2-7b (Touvron et al., 2023), both consisting of 32 layers. To identify which layers best capture factuality, they trained five SAPLMA models, forwarding each statement in the True-False Dataset as input to the LLMs and extracting the corresponding activation values from the 32nd (last layer), 28th, 24th, 20th, and 16th layers. These activations serve as input for training the classifiers. To ensure generalizability, i.e., making SAPLMA independent of specific topics, the authors adopted a cross-validation strategy using a leave-one-topic-out approach to train the classifier on five topics and test the probe on the held-out topic.

2.3 Reproducibility Settings

In this section, we provide details on the datasets and experimental settings for reproducing the work of Azaria and Mitchell (2023). Our goal is to answer the Research Question (RQ): Can we reproduce the results reported by Azaria and Mitchell (2023)? Although the code is not publicly accessible, the authors made it available upon request.

Dataset generation. The code provided by Azaria and Mitchell (2023) contains all the necessary material to recreate the entire dataset generation process. However, their template-matching code is influenced by randomness in the generation of false statements, and a random seed is not set. Due to this non-deterministic behavior, recreating their dataset using the original code was unfeasible. However, the authors released their dataset, allowing us to reproduce their exact dataset settings.

SAPLMA reproducibility. To reproduce the results of the original study, we trained 20 SAPLMA
<table>
  <tr>
    <th rowspan="2">Layer</th>
    <th colspan="2">Cities</th>
    <th colspan="2">Inventions</th>
    <th colspan="2">Elements</th>
    <th colspan="2">Animals</th>
    <th colspan="2">Companies</th>
    <th colspan="2">Facts</th>
    <th colspan="2">Average</th>
  </tr>
  <tr>
    <th>Llama2</th><th>OPT-6.7b</th>
    <th>Llama2</th><th>OPT-6.7b</th>
    <th>Llama2</th><th>OPT-6.7b</th>
    <th>Llama2</th><th>OPT-6.7b</th>
    <th>Llama2</th><th>OPT-6.7b</th>
    <th>Llama2</th><th>OPT-6.7b</th>
    <th>Llama2</th><th>OPT-6.7b</th>
  </tr>
  <tr>
    <td>Last</td>
    <td>0.7574</td><td>0.7796</td>
    <td>0.6735</td><td>0.5966</td>
    <td>0.6814</td><td>0.5760</td>
    <td>0.7338</td><td>0.6022</td>
    <td>0.6736</td><td>0.6925</td>
    <td>0.7444</td><td>0.6498</td>
    <td>0.7107</td><td>0.6449</td>
  </tr>
  <tr>
    <td></td>
    <td>0.7939</td><td>0.7836</td>
    <td>0.7470</td><td>0.5603</td>
    <td>0.7057</td><td>0.5656</td>
    <td>0.7133</td><td>0.5984</td>
    <td>0.6463</td><td>0.6900</td>
    <td>0.7894</td><td>0.6640</td>
    <td>0.7326</td><td>0.6437</td>
  </tr>
  <tr>
    <td>28</td>
    <td>0.8146</td><td>0.7732</td>
    <td>0.7207</td><td>0.5761</td>
    <td>0.6767</td><td>0.5907</td>
    <td>0.7249</td><td>0.5777</td>
    <td>0.6694</td><td>0.7247</td>
    <td>0.7662</td><td>0.6618</td>
    <td>0.7321</td><td>0.6507</td>
  </tr>
  <tr>
    <td></td>
    <td>0.8261</td><td>0.8014</td>
    <td>0.7221</td><td>0.5938</td>
    <td>0.6746</td><td>0.5931</td>
    <td>0.7046</td><td>0.5945</td>
    <td>0.6860</td><td>0.7252</td>
    <td>0.7976</td><td>0.6639</td>
    <td>0.7351</td><td>0.6620</td>
  </tr>
  <tr>
    <td>24</td>
    <td>0.8722</td><td>0.7963</td>
    <td>0.7816</td><td>0.6712</td>
    <td>0.6849</td><td>0.6211</td>
    <td>0.7394</td><td>0.5800</td>
    <td>0.7094</td><td>0.7758</td>
    <td>0.7858</td><td>0.6868</td>
    <td>0.7622</td><td>0.6886</td>
  </tr>
  <tr>
    <td></td>
    <td>0.8619</td><td>0.8043</td>
    <td>0.7377</td><td>0.6604</td>
    <td>0.6789</td><td>0.6172</td>
    <td>0.7415</td><td>0.6095</td>
    <td>0.7049</td><td>0.7844</td>
    <td>0.7910</td><td>0.6804</td>
    <td>0.7586</td><td>0.6927</td>
  </tr>
  <tr>
    <td>20</td>
    <td>0.8820</td><td>0.8125</td>
    <td>0.8459</td><td>0.7268</td>
    <td>0.6950</td><td>0.6197</td>
    <td>0.7758</td><td>0.6058</td>
    <td>0.8319</td><td>0.8122</td>
    <td>0.8053</td><td>0.6819</td>
    <td>0.8060</td><td>0.7098</td>
  </tr>
  <tr>
    <td></td>
    <td>0.8672</td><td>0.8118</td>
    <td>0.8584</td><td>0.7222</td>
    <td>0.6761</td><td>0.6218</td>
    <td>0.7736</td><td>0.6208</td>
    <td>0.8254</td><td>0.8160</td>
    <td>0.8065</td><td>0.6734</td>
    <td>0.8012</td><td>0.7110</td>
  </tr>
  <tr>
    <td>16</td>
    <td>0.9223</td><td>0.7435</td>
    <td>0.8938</td><td>0.6400</td>
    <td>0.6939</td><td>0.5645</td>
    <td>0.7774</td><td>0.5800</td>
    <td>0.8658</td><td>0.7570</td>
    <td>0.8254</td><td>0.6237</td>
    <td>0.8298</td><td>0.6515</td>
  </tr>
  <tr>
    <td></td>
    <td>0.9174</td><td>0.7554</td>
    <td>0.8847</td><td>0.6403</td>
    <td>0.7005</td><td>0.5732</td>
    <td>0.7883</td><td>0.5693</td>
    <td>0.8672</td><td>0.7760</td>
    <td>0.8104</td><td>0.6340</td>
    <td>0.8281</td><td>0.6580</td>
  </tr>
</table>

Table 1: Replicated SAPLMA performance on the True-False dataset across the selected layers. The results labeled as ‘Orig.’ are taken from the original work, while those labeled as ‘Repr.’ are the replicated results from this study.

probes for each of the following layers: the 32nd, 28th, 24th, 20th, and 16th, over 5 epochs, resulting in a total of 100 probes. We employed Llama 2-7b and OPT-6.7b, both using half-precision (16-bit float) parameters, with a default temperature of 0.8 for Llama 2-7b and 1.0 for OPT-6.7b. The hardware used for the experiments was an Intel(R) Core(TM) i7-5820K paired with an NVIDIA RTX 3090 graphics card. The authors do not specify from which token they extract the associated hidden state. However, code inspection led to the identification of the last token as the target state.

<table>
  <tr>
    <th>Model</th>
    <th>Cities</th>
    <th>Inventions</th>
    <th>Elements</th>
    <th>Animals</th>
    <th>Companies</th>
    <th>Facts</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>BERT</td>
    <td>0.5557</td><td>0.5557</td><td>0.5645</td><td>0.5228</td><td>0.5553</td><td>0.5302</td><td>0.5434</td>
  </tr>
  <tr>
    <td></td>
    <td>0.5257</td><td>0.5611</td><td>0.5435</td><td>0.5603</td><td>0.5302</td><td>0.5361</td><td>0.5428</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>0.5410</td><td>0.4799</td><td>0.5685</td><td>0.5650</td><td>0.5538</td><td>0.5164</td><td>0.5374</td>
  </tr>
  <tr>
    <td></td>
    <td>0.5416</td><td>0.4800</td><td>0.5683</td><td>0.5652</td><td>0.5539</td><td>0.5155</td><td>0.5368</td>
  </tr>
  <tr>
    <td>5-shot</td>
    <td>0.5416</td><td>0.4799</td><td>0.5676</td><td>0.5643</td><td>0.5540</td><td>0.5148</td><td>0.5370</td>
  </tr>
  <tr>
    <td></td>
    <td>0.5416</td><td>0.4800</td><td>0.5676</td><td>0.5643</td><td>0.5540</td><td>0.5082</td><td>0.5359</td>
  </tr>
  <tr>
    <td>It-is-true</td>
    <td>0.5240</td><td>0.5068</td><td>0.5688</td><td>0.4851</td><td>0.6883</td><td>0.5840</td><td>0.5589</td>
  </tr>
  <tr>
    <td></td>
    <td>0.5233</td><td>0.5046</td><td>0.5688</td><td>0.4831</td><td>0.6875</td><td>0.5858</td><td>0.5588</td>
  </tr>
</table>

Table 2: Replicated baselines performance on the True-False dataset. The results labeled as ‘Orig.’ are taken from the original work, while those labeled as ‘Repr.’ are the replicated results from this study.

3 Experimental Reproducibility Results

To answer RQ0, we report the results for the reproduction of the experiments in Tables 1, 2 and 3. The results labeled with ‘Orig.’ are retrieved from the original work, while the ones achieved in the reproducibility study are labeled with ‘Repr.’.

3.1 Reproduction of SAPLMA Results on the True-False Dataset

Table 1 reports SAPLMA’s performance on the True-False dataset across different layers of Llama 2-7b and OPT-6.7b for six categories: Cities, Inventions, Elements, Animals, Companies, and Facts. Results indicate that Llama 2-7b consistently outperforms OPT-6.7b across all layers and categories.

<table>
  <tr>
    <th>Layer</th>
    <th>Accuracy</th>
    <th>AUC</th>
    <th>Accuracy with Optimal Threshold</th>
    <th>Average Threshold</th>
  </tr>
  <tr>
    <td>Last-layer</td>
    <td>0.6187</td><td>0.7587</td><td>0.7052</td><td>0.8687</td>
  </tr>
  <tr>
    <td></td>
    <td>0.6406</td><td>0.7720</td><td>0.7264</td><td>0.8910</td>
  </tr>
  <tr>
    <td>28th-layer</td>
    <td>0.6362</td><td>0.7614</td><td>0.7134</td><td>0.8838</td>
  </tr>
  <tr>
    <td></td>
    <td>0.6410</td><td>0.7686</td><td>0.7203</td><td>0.8876</td>
  </tr>
  <tr>
    <td>24th-layer</td>
    <td>0.6134</td><td>0.7435</td><td>0.6988</td><td>0.8801</td>
  </tr>
  <tr>
    <td></td>
    <td>0.6206</td><td>0.7496</td><td>0.6973</td><td>0.8500</td>
  </tr>
  <tr>
    <td>20th-layer</td>
    <td>0.6029</td><td>0.7182</td><td>0.6587</td><td>0.9063</td>
  </tr>
  <tr>
    <td></td>
    <td>0.5965</td><td>0.7183</td><td>0.6669</td><td>0.8868</td>
  </tr>
  <tr>
    <td>Middle-layer</td>
    <td>0.5566</td><td>0.6610</td><td>0.6500</td><td>0.8123</td>
  </tr>
  <tr>
    <td></td>
    <td>0.5379</td><td>0.6760</td><td>0.6468</td><td>0.7948</td>
  </tr>
  <tr>
    <td>BERT</td>
    <td>0.5115</td><td>0.5989</td><td>0.5705</td><td>0.5403</td>
  </tr>
  <tr>
    <td></td>
    <td>0.5522</td><td>0.6092</td><td>0.5869</td><td>0.7939</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>0.5041</td><td>0.4845</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>0.5041</td><td>0.4845</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>5-shot</td>
    <td>0.5125</td><td>0.4822</td><td>-</td><td>-</td>
  </tr>
</table>

Table 3: Reproduced SAPLMA performance on the OPT-Generated Dataset (Section 2.1). The results labeled as ‘Orig.’ are taken from the original work, while those labeled as ‘Repr.’ are the reproduced results.

Middle layers (16, 20, 24) achieve the highest performance, while accuracy declines toward the final layer. The reproduced results closely align with the original findings, with minor deviations observed across specific categories and layers. Moreover, although OPT-6.7b shows greater variability, the ranking of layers remains consistent with the original work. Both experiments confirm that factuality information is more effectively encoded in the middle layers (16–24) than in the final layer. Additionally, we reproduce the baseline used by the authors to compare SAPLMA. Specifically, their baseline includes a trained SAPLMA on BERT activations, a few-shot approach where the LLM is prompted with a sentence and asked to label it as ‘True’ or ‘False,’ and an ‘It-is-true’ test. In this test, the LLMs were asked: Is it true that X? and Is it false that X?, where X is a dataset sample. A response was considered correct if the model assigned a higher probability to the ‘True’ token. Table 2 summarizes the baseline performance. BERT achieves the highest average performance among
non-prompted methods, demonstrating its effectiveness in factual classification, and the results indicate high reproducibility. The It-is-true baseline yields the highest average performance, particularly in the ‘Companies’ topic. The results demonstrate consistent reproducibility.

In general, the reproduction on the True-False Dataset is considered successful based on the following observations: (i) the overall performance trends remain consistent, with Llama 2-7b outperforming OPT-6.7b; (ii) observed deviations are minor and do not indicate fundamental inconsistencies; (iii) the relative ranking of layers remains unchanged, reinforcing previous findings; and (iv) baseline methods retain their rankings, confirming the validity of the original results.

3.2 Reproduction of the Results on the OPT-Generated Dataset

Table 3 presents the results of the reproduced experiments on the OPT-Generated Dataset. Performance is evaluated using Accuracy, AUC, Accuracy with an Optimal Threshold (selected by estimating it from a held-out validation set), and the Average Optimal Threshold. The final layers, specifically the 28th and final layers, outperform the middle and lower layers in terms of Accuracy and AUC. The reproduced results closely align with the original findings, exhibiting only minor variations. Accuracy with an optimal threshold consistently exceeds raw accuracy, suggesting that tuning the decision boundary improves performance.

Regarding baselines, BERT exhibits lower accuracy compared to the LLMs’ last layers, with slight improvements over the original results. Notably, the 3- and 5-shot prompting results were identical between the original and reproduced experiments.

RQ0: Can we reproduce the results reported by Azaria and Mitchell (2023)?

This reproducibility study demonstrates a high degree of alignment with the original results, confirming the validity of previous findings. It shows that the ranking and trends remain unchanged, reinforcing the robustness of the results.

4 Novel Dataset Generation Strategies for Factuality Self-Evaluation

This section introduces two novel dataset generation strategies to investigate LLM factuality self-evaluation. To contextualize our approach, we first examine the limitations of the True-False dataset by Azaria and Mitchell (2023). While it provides a structured framework for evaluating LLMs, its construction imposes constraints that may limit generalizability of the findings. Specifically, true and false statements are derived from tabular data using predefined templates. We argue that this approach suffers from several limitations:

• Adherence to predefined templates: The use of fixed templates limits the linguistic expressiveness of the dataset, potentially constraining the probe classifier’s ability to generalize beyond rigidly structured statements (e.g., <company> operates in the industry of <industry>).

• Distribution misalignment: The statements are constructed from tabular data rather than generated by the LLM itself. As a result, the dataset may not align with the LLM’s generative distribution. For instance, a niche true fact in the dataset –but unknown by the model– could have high perplexity for the LLM, undermining the study’s core premise: evaluating an LLM’s intrinsic ability to “judge” its false claims.

• Lack of consideration for the LLM’s knowledge state: LLMs exhibit uneven factual knowledge based on their pretraining data, with strengths in some domains and gaps in others. A model can assess a statement factuality only if it has prior exposure to it. The dataset does not account for these inconsistencies: it evaluates whether an LLM can detect factual errors without considering whether the model actually possesses knowledge of the fact.

• Differences in cardinality: Some properties in the dataset have fewer admissible values, making certain facts easier to evaluate. For example, a statement like "<element> appears in its standard state as _" has fewer possible values (i.e.,{Solid, Liquid, Gas}) compared to statements like "<city> is a city in _" that involve a broader set. This imbalance in complexity may bias the evaluation process.

These limitations impact the interpretability of results when evaluating an LLM’s internal representations of factuality. To ensure a more realistic assessment of an LLM’s self-evaluation of factuality, we propose two strategies to address these issues. The first strategy samples statements from tabular data to better align with the LLM’s generative predictions. The second strategy involves sampling LLM-generated facts as answers to questions from a well-known Question Answering dataset.
4.1 Perplexity-based Dataset Construction

This section presents a novel dataset generation strategy to address limitations in the Azaria and Mitchell (2023) True-False dataset, particularly distribution misalignment and implausible negative samples. To improve negative sampling based on random property-object substitutions, we introduce a perplexity-guided probabilistic sampling method, which re-weights false statements based on perplexity for better alignment with LLM output distributions. Since perplexity depends on the model, the same LLM under evaluation must generate false statements to ensure consistency, resulting in a model-dependent dataset tailored to the characteristics and biases of the LLM being studied.

True statements are initially constructed, as in Azaria and Mitchell (2023), by directly inserting correct entity-property pairs into pre-defined sentence templates. Regarding the false statements generation, we proceed as follows: (i) for each true statement, all unique alternative property values are gathered from the entire dataset; (ii) a candidate sentence is created for each alternative property by inserting it into the corresponding template; (iii) the target LLM (i.e., OPT-6.7b or Llama 2-7b) computes the perplexity of each candidate sentence. This perplexity score serves as a plausibility metric, with lower perplexity indicating a more plausible (yet incorrect) statement.

Given a true statement, we define \( C \) as the set of potential candidate sentences, which includes the true statement. Furthermore, \( C' \subset C \) is defined as the subset of candidate false sentences (i.e. excluding the true statement). Candidates \( c \in C \) are ranked based on their perplexity scores, with lower scores indicating higher plausibility.

Since perplexity can be interpreted as a measure of plausibility, we operate under the assumption that an LLM possesses factual knowledge about a fact if the fact is assigned a "sufficiently low perplexity". Conversely, a high perplexity score for the true statement suggests that the LLM lacks knowledge of the fact. Given the limitations discussed earlier, we aim to evaluate the LLM ability to discern between true and false statements when it possesses the relevant knowledge. Therefore, we exclude instances where the LLM exhibits limited knowledge about the true statement: if the true statement does not rank among the lowest \( k \) perplexity candidates, the generation of that instance is discontinued, and the next true fact is considered.

In practice, we define \( k \) as \( k = \alpha |C| \), where \( \alpha \) is a hyperparameter (\( 0 < \alpha < 1 \)). This accounts for the varying cardinality of the property ranges in the dataset, ensuring that the threshold for "sufficiently low perplexity" is adjusted based on the number of possible values for a given property.

In addition, we want to simulate a real hallucination scenario where the LLM is uncertain between the true fact and plausible alternatives: given the perplexity score function \( PP(\cdot) \), all false candidates \( c \) with a perplexity score \( PP(c) < (1 + \beta)PP(true) \), where \( 0 < \beta < 1 \) is a hyperparameter, are considered, resulting in a reduced set of candidates \( C^* \). A min-max normalization is applied to their perplexity,

\[
\text{NormPP}(c) = \frac{\text{PP}(c) - \min_{c \in C} \text{PP}(c)}{\max_{c \in C} \text{PP}(c) - \min_{c \in C} \text{PP}(c)},
\]

The normalized perplexities are transformed using a plausibility score function \( s(\cdot) \), i.e., lower perplexity scores result in higher plausibility scores. The scores are then normalized to ensure that they sum to 1 and are treated as a probability distribution over the candidates:

\[
s(c) = e^{-\text{NormPP}(c)}, \quad P(c_i) = \frac{s(c_i)}{\sum_{c_j \in C^*} s(c_j)}.
\]

The normalization guarantees that \( P(c_i) \) values are suitable for sampling. Finally, a mixture of top-k and nucleus sampling (Holtzman et al., 2020) is employed to sample the candidate for insertion into the template. Specifically, we apply top-k and nucleus sampling sequentially: we select the top-k highest-scoring candidates and then refine this set by choosing the smallest subset whose cumulative probability reaches a predefined threshold. This process generates a coherent yet factually incorrect statement that is more realistic and closely aligned with the LLM internal token prediction patterns. In Section 5.1 we detail the selected values for the hyperparameters \( \alpha, \beta, k, \) and \( p \).

4.2 LLM-Generated Dataset Construction

The strategy in Section 4.1 constructs a balanced true-false dataset from tabular data but has inherent limitations. While enabling direct comparison with Azaria and Mitchell (2023), it restricts generative models to rigid templates, limiting expressiveness. Additionally, its reliance on fixed candidate sets can lead to easily classifiable false statements
when the set is small. Finally, template-based sampling reduces diversity, likely due to token bias: we found LMs consistently assigning lower perplexity to certain property-object pairs, regardless of the subject. To more realistically assess an LLM factuality self-evaluation, we propose generating both true and false statements directly from the model, overcoming the limitations of template-based approaches. This involves (a) a method to elicit diverse factoid statements from the LLM, and (b) a strategy to annotate statement veracity, addressing biases and inconsistencies in the process. Consider a Question Answering dataset composed of \( N \) questions, \( D_{QA} = \{ (q_i, a_i) \}_{i=1}^N \), where each question \( q_i \) has a corresponding ground-truth answer \( a_i \). Given a LLM \( \mathcal{M} \), we prompt it with each question \( K \) times, yielding a set of generated answers. This results in an extended dataset \( D_{QA}^{\mathcal{M}} = \{ (q_i, a_i, \{ a_{i,k}^{\mathcal{M}} \}_{k=1}^K ) \}_{i=1}^N \). Following LLM-as-judge (Gu et al., 2024; Calderon et al., 2025) practices, the LLM-generated answers in \( D_{QA}^{\mathcal{M}} \) can be annotated using an oracle LLM, which we assume is able to evaluate the veracity of each answer \( a_{i,k}^{\mathcal{M}} \) given the ground-truth answer \( a_i \) and the question \( q_i \). This operation results in an annotated dataset

\[
\hat{D}_{QA}^{\mathcal{M}} = \{ (q_i, a_i, \{ a_{i,k}^{\mathcal{M}}, v_{i,k}^{\mathcal{M}} \}_{k=1}^K ) \}_{i=1}^N,
\]

where \( v_{i,k} \) is a veracity label assigned by the oracle, indicating whether the generated answer \( a_{i,k} \) is correct (\( v_{i,k} = 1 \)) or incorrect (\( v_{i,k} = 0 \)).

Increasing K enhances the reliability of responses by offering a more accurate evaluation of the LLM knowledge state regarding a question. This evaluation can lead to three possible outcomes: (i) a high proportion of correct answers suggests the LLM fully understands the facts; (ii) a high proportion of incorrect answers indicates the LLM lacks or has partial knowledge, preventing correct responses; (iii) a mix of correct and incorrect answers implies knowledge with a tendency toward hallucination. This study assumes that an LLM can only encode factuality regarding a generated fact if it has some knowledge about it. Therefore, we focus exclusively on the third scenario. This scenario also naturally produces a balanced dataset, including true and false variations of the same fact. We define the correct answer ratio \( p_i^{\mathcal{M}} = \frac{1}{K} \sum_{k=1}^K v_{i,k}^{\mathcal{M}} \). We consider questions whose \( p_i \) is around 0.5 with a tolerance hyperparameter \( \tau \), that is \( |p_i^{\mathcal{M}} - 0.5| < \tau \). The dataset is obtained by selecting the answers and their veracity labels satisfying the condition:

\[
D_{Facts}^{\mathcal{M}} = \{ (a_{i,k}^{\mathcal{M}}, v_{i,k}^{\mathcal{M}}) : |p_i^{\mathcal{M}} - 0.5| < \tau \}_{i=1}^N.
\]

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="2">Llama 2-7b</th>
    <th colspan="2">OPT-6.7b</th>
  </tr>
  <tr>
    <th>Sentences</th>
    <th>(%) True</th>
    <th>Sentences</th>
    <th>(%) True</th>
  </tr>
  <tr>
    <td>Cities</td>
    <td>674</td>
    <td>50</td>
    <td>756</td>
    <td>50</td>
  </tr>
  <tr>
    <td>Inventions</td>
    <td>336</td>
    <td>50</td>
    <td>202</td>
    <td>50</td>
  </tr>
  <tr>
    <td>Elements</td>
    <td>118</td>
    <td>50</td>
    <td>220</td>
    <td>50</td>
  </tr>
  <tr>
    <td>Animals</td>
    <td>116</td>
    <td>50</td>
    <td>114</td>
    <td>50</td>
  </tr>
  <tr>
    <td>Companies</td>
    <td>326</td>
    <td>50</td>
    <td>310</td>
    <td>50</td>
  </tr>
</table>

Table 4: Novel generation of our True-False dataset, following the approach described in Section 4.1, including number of sentences and percentage of true samples.

and their veracity labels satisfying the condition:

\[
D_{Facts}^{\mathcal{M}} = \{ (a_{i,k}^{\mathcal{M}}, v_{i,k}^{\mathcal{M}}) : |p_i^{\mathcal{M}} - 0.5| < \tau \}_{i=1}^N.
\]

5 Experiments and Discussion

This section describes the experimental setup used to extend the prior investigation, presents the results, and discusses their implications. We address the following Research Questions:

RQ1 Are probing classifiers trained as in Azaria and Mitchell (2023) capable of generalizing to True-False sentences with similar perplexity?

RQ2 Can the same probes generalize to facts generated by LLMs?

5.1 Dataset Generation Setup

To generate the datasets for training the probe and alleviate the limitations of Azaria and Mitchell (2023), we employ the perplexity-based sampling procedure described in Section 4.1. Specifically, we set the needed hyperparameters as follows: \( \alpha = 0.1, \beta = 0.1, k = 10 \) and \( p = 0.9 \). We excluded the Scientific Facts topic from our perplexity-based sampling procedure, as its original version was generated by ChatGPT and not from tabular data (additional details in Appendix B). Table 4 summarizes the statistics for the two refined True-False datasets, while Figure 1 shows the differences in average perplexity between the original and the proposed datasets.

We construct an LLM-generated dataset following the procedure described in Section 4.2. Our experiments use TriviaQA (Joshi et al., 2017), a dataset of question-answer pairs collected from 14 trivia and quiz-league websites. Given the limited computational resources, we limit our focus to the validation split of TriviaQA Wikipedia questions. We construct a True-False dataset from two additional QA datasets, SQuAD 2.0 (Rajpurkar et al., 2018) and TruthfulQA (Lin et al., 2022), employing the same approach, which we describe
Figure 1: Comparison of average perplexity scores for Llama 2-7b and OPT-6.7b for the Original dataset by Azaria and Mitchell (2023) and our refined version. Lower perplexity indicates that the sentences are more likely to have been generated by the model. PP(T/F) denotes average perplexity of true/false sentences.

in Appendix E. For each LLM in our analysis, we set \( K = 10 \) and \( \tau = 0.1 \), and we discard the answers composed of less than 5 tokens to improve self-consistency. For annotation, we use GPT-4o mini with a 3-shot learning strategy, where three examples are provided. More details are included in Appendix C.1, and Table 8 reports the dataset statistics.

5.2 Impact of Perplexity-Based Sampling on SAPLMA Accuracy

We assess the impact of our perplexity-based sampling strategy by training the SAPLMA probe classifier separately on the original True-False dataset from Azaria and Mitchell (2023) and our refined version. As shown in Table 5, classifiers tested on our refined dataset achieve the highest accuracy in deeper layers. For Llama 2-7b, Layer 16 shows the most consistent results, achieving the highest average accuracy across all tested configurations when trained on our dataset. Although accuracy fluctuates across different topics, overall performance remains largely consistent between the training configurations. Comparing the results with the original results in Table 1, it emerges that classifiers trained on the original dataset achieve higher accuracy when tested on the same dataset. This is particularly evident in the ‘Animals’ topic. However, it is important to consider that the perplexity values in our refined dataset are significantly lower than those in the original dataset for both true and false statements (see 1). Additionally, our refined dataset features lower and more closely aligned perplexity scores between true and false statements, with false statements often yielding even lower perplexity values than true ones. This finding implies the validity of Azaria and Mitchell hypothesis, which states that factuality information is encoded in LLM hidden states, although it might not be immediately evident from the model predicted probabilities.

RQ1: Are probing classifiers trained as in Azaria and Mitchell (2023) capable of generalizing to True-False sentences with similar perplexity?

Surprisingly, the results between the probes trained on our refined dataset and the probes trained on the original dataset are mostly comparable. This proves that the probes can generalize even when the train-test datasets have different perplexity.

5.3 Generalization of SAPLMA on LLM-generated Sentences

We extend our experiments with the SAPLMA classifier to a new setting, where both training and evaluation are conducted on sentences generated by an LLM and sourced from TriviaQA, leveraging the procedure described in Section 4.2. Table 6 reports the performance of SAPLMA on a set of factual statements extracted from TriviaQA. The results indicate that the currently used probes are inadequate for factuality self-assessment in real-world scenarios, as the observed accuracy does not reach a noteworthy threshold. Furthermore, following the suggestion of Azaria and Mitchell, we optimize the classification threshold; however, this adjustment yields no significant improvement in performance. For completeness, the interested reader may find the result obtained on the same dataset when training on the original True-False dataset of Azaria and Mitchell in Appendix D. In addition, we report results on SQuAD 2.0 and TruthfulQA in Appendix E, which align closely with the results observed on TriviaQA, further supporting the consistency of our findings.

RQ2: Can the same probes generalize to facts generated by LLMs?

In summary, this experiment partially contradicts the findings of Azaria and Mitchell (2023): the trained probes are not capable of providing good generalization to an LLM-generated dataset, even when the accuracy threshold is tuned. The motivation could stem from the nature of the dataset, as TriviaQA contains open-domain questions that result in more nuanced facts than the ones in the original True-False dataset (Section 2) or the OPT-generated dataset (Section 3.2).

We believe that further research is needed to enhance the effectiveness of factuality self-assessment
<table>
  <tr>
    <th rowspan="2">Layer</th>
    <th rowspan="2">Training Data</th>
    <th colspan="4">Cities</th>
    <th colspan="4">Inventions</th>
    <th colspan="4">Animals</th>
    <th colspan="4">Companies</th>
    <th colspan="4">Average</th>
  </tr>
  <tr>
    <th>Llama 2-7b</th>
    <th>OPT-6.7b</th>
    <th>Llama 2-7b</th>
    <th>OPT-6.7b</th>
    <th>Llama 2-7b</th>
    <th>OPT-6.7b</th>
    <th>Llama 2-7b</th>
    <th>OPT-6.7b</th>
    <th>Llama 2-7b</th>
    <th>OPT-6.7b</th>
    <th>Llama 2-7b</th>
    <th>OPT-6.7b</th>
    <th>Llama 2-7b</th>
    <th>OPT-6.7b</th>
    <th>Llama 2-7b</th>
    <th>OPT-6.7b</th>
  </tr>
  <tr>
    <td>last</td>
    <td>Orig.</td>
    <td>0.6882</td>
    <td>0.5724</td>
    <td>0.6409</td>
    <td>0.5094</td>
    <td>0.6314</td>
    <td>0.5482</td>
    <td>0.5685</td>
    <td>0.5259</td>
    <td>0.6290</td>
    <td>0.6984</td>
    <td>0.6316</td>
    <td>0.5709</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Novel</td>
    <td>0.6365</td>
    <td>0.6143</td>
    <td>0.6101</td>
    <td>0.5144</td>
    <td>0.5623</td>
    <td>0.5293</td>
    <td>0.5461</td>
    <td>0.4846</td>
    <td>0.7414</td>
    <td>0.7211</td>
    <td>0.6311</td>
    <td>0.5720</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>28</td>
    <td>Orig.</td>
    <td>0.7056</td>
    <td>0.5870</td>
    <td>0.7001</td>
    <td>0.5178</td>
    <td>0.6161</td>
    <td>0.5832</td>
    <td>0.6013</td>
    <td>0.5566</td>
    <td>0.7079</td>
    <td>0.7234</td>
    <td>0.6662</td>
    <td>0.5936</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Novel</td>
    <td>0.5091</td>
    <td>0.6057</td>
    <td>0.6591</td>
    <td>0.5473</td>
    <td>0.5665</td>
    <td>0.5655</td>
    <td>0.6052</td>
    <td>0.4662</td>
    <td>0.7061</td>
    <td>0.7065</td>
    <td>0.6092</td>
    <td>0.5811</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>24</td>
    <td>Orig.</td>
    <td>0.8286</td>
    <td>0.7026</td>
    <td>0.7250</td>
    <td>0.6022</td>
    <td>0.6432</td>
    <td>0.5918</td>
    <td>0.6310</td>
    <td>0.5439</td>
    <td>0.7121</td>
    <td>0.7366</td>
    <td>0.7080</td>
    <td>0.6354</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Novel</td>
    <td>0.6025</td>
    <td>0.6710</td>
    <td>0.6609</td>
    <td>0.6248</td>
    <td>0.5763</td>
    <td>0.5993</td>
    <td>0.5836</td>
    <td>0.4868</td>
    <td>0.7868</td>
    <td>0.7366</td>
    <td>0.6420</td>
    <td>0.6225</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>20</td>
    <td>Orig.</td>
    <td>0.8272</td>
    <td>0.7313</td>
    <td>0.7741</td>
    <td>0.6230</td>
    <td>0.6492</td>
    <td>0.6255</td>
    <td>0.5832</td>
    <td>0.5075</td>
    <td>0.7583</td>
    <td>0.7502</td>
    <td>0.7184</td>
    <td>0.6475</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Novel</td>
    <td>0.7382</td>
    <td>0.7528</td>
    <td>0.6973</td>
    <td>0.6131</td>
    <td>0.6051</td>
    <td>0.5986</td>
    <td>0.6190</td>
    <td>0.4807</td>
    <td>0.8270</td>
    <td>0.7566</td>
    <td>0.6973</td>
    <td>0.6404</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>16</td>
    <td>Orig.</td>
    <td>0.8941</td>
    <td>0.6433</td>
    <td>0.7888</td>
    <td>0.5698</td>
    <td>0.6801</td>
    <td>0.5930</td>
    <td>0.5836</td>
    <td>0.3816</td>
    <td>0.7768</td>
    <td>0.7426</td>
    <td>0.7447</td>
    <td>0.5860</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Novel</td>
    <td>0.9301</td>
    <td>0.7505</td>
    <td>0.7961</td>
    <td>0.5644</td>
    <td>0.6623</td>
    <td>0.5568</td>
    <td>0.6319</td>
    <td>0.5307</td>
    <td>0.8265</td>
    <td>0.7231</td>
    <td>0.7694</td>
    <td>0.6151</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 5: Accuracy values obtained training SAPLMA on the original True-False dataset and on our refined version, then tested on our refined version. Results are shown for the Llama 2-7b and OPT-6.7b models. ‘Orig.’ denotes the ‘original True-False dataset as training data, while ‘Novel’ denotes our version of the True-False dataset as training data. In bold we denote the best combination of layer/training dataset for each combination of model/topic.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="4">Threshold = 0.5</th>
    <th colspan="4">Optimal Threshold</th>
  </tr>
  <tr>
    <th>last</th>
    <th>28</th>
    <th>24</th>
    <th>20</th>
    <th>16</th>
    <th>last</th>
    <th>28</th>
    <th>24</th>
    <th>20</th>
    <th>16</th>
  </tr>
  <tr>
    <td>billturnbull</td>
    <td>Llama</td>
    <td>0.561</td>
    <td>0.576</td>
    <td>0.618</td>
    <td>0.621</td>
    <td>0.628</td>
    <td>0.581</td>
    <td>0.618</td>
    <td>0.634</td>
    <td>0.639</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.547</td>
    <td>0.558</td>
    <td>0.591</td>
    <td>0.537</td>
    <td>0.537</td>
    <td>0.558</td>
    <td>0.547</td>
    <td>0.539</td>
    <td>0.499</td>
  </tr>
  <tr>
    <td>derby*</td>
    <td>Llama</td>
    <td>0.568</td>
    <td>0.581</td>
    <td>0.575</td>
    <td>0.517</td>
    <td>0.566</td>
    <td>0.555</td>
    <td>0.571</td>
    <td>0.597</td>
    <td>0.597</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.533</td>
    <td>0.564</td>
    <td>0.584</td>
    <td>0.572</td>
    <td>0.562</td>
    <td>0.566</td>
    <td>0.583</td>
    <td>0.586</td>
    <td>0.540</td>
  </tr>
  <tr>
    <td>quizfree</td>
    <td>Llama</td>
    <td>0.564</td>
    <td>0.547</td>
    <td>0.523</td>
    <td>0.561</td>
    <td>0.589</td>
    <td>0.548</td>
    <td>0.521</td>
    <td>0.547</td>
    <td>0.573</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.559</td>
    <td>0.559</td>
    <td>0.575</td>
    <td>0.581</td>
    <td>0.544</td>
    <td>0.530</td>
    <td>0.533</td>
    <td>0.546</td>
    <td>0.541</td>
  </tr>
  <tr>
    <td>quizguy</td>
    <td>Llama</td>
    <td>0.578</td>
    <td>0.585</td>
    <td>0.588</td>
    <td>0.607</td>
    <td>0.635</td>
    <td>0.576</td>
    <td>0.581</td>
    <td>0.593</td>
    <td>0.637</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.579</td>
    <td>0.583</td>
    <td>0.589</td>
    <td>0.590</td>
    <td>0.569</td>
    <td>0.555</td>
    <td>0.565</td>
    <td>0.559</td>
    <td>0.548</td>
  </tr>
  <tr>
    <td>triviabug</td>
    <td>Llama</td>
    <td>0.494</td>
    <td>0.518</td>
    <td>0.521</td>
    <td>0.525</td>
    <td>0.538</td>
    <td>0.508</td>
    <td>0.530</td>
    <td>0.542</td>
    <td>0.545</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.620</td>
    <td>0.620</td>
    <td>0.596</td>
    <td>0.526</td>
    <td>0.635</td>
    <td>0.605</td>
    <td>0.598</td>
    <td>0.553</td>
    <td>0.553</td>
  </tr>
  <tr>
    <td>businessballs</td>
    <td>Llama</td>
    <td>0.566</td>
    <td>0.558</td>
    <td>0.565</td>
    <td>0.574</td>
    <td>0.582</td>
    <td>0.558</td>
    <td>0.551</td>
    <td>0.564</td>
    <td>0.575</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.559</td>
    <td>0.558</td>
    <td>0.570</td>
    <td>0.570</td>
    <td>0.545</td>
    <td>0.547</td>
    <td>0.573</td>
    <td>0.562</td>
    <td>0.551</td>
  </tr>
  <tr>
    <td>jetpunk</td>
    <td>Llama</td>
    <td>0.587</td>
    <td>0.627</td>
    <td>0.620</td>
    <td>0.644</td>
    <td>0.654</td>
    <td>0.548</td>
    <td>0.580</td>
    <td>0.534</td>
    <td>0.600</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.606</td>
    <td>0.612</td>
    <td>0.614</td>
    <td>0.596</td>
    <td>0.621</td>
    <td>0.618</td>
    <td>0.618</td>
    <td>0.601</td>
    <td>0.619</td>
  </tr>
  <tr>
    <td>odquiz</td>
    <td>Llama</td>
    <td>0.531</td>
    <td>0.536</td>
    <td>0.546</td>
    <td>0.572</td>
    <td>0.573</td>
    <td>0.537</td>
    <td>0.535</td>
    <td>0.550</td>
    <td>0.567</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.576</td>
    <td>0.576</td>
    <td>0.576</td>
    <td>0.576</td>
    <td>0.576</td>
    <td>0.576</td>
    <td>0.576</td>
    <td>0.576</td>
    <td>0.576</td>
  </tr>
  <tr>
    <td>quiz-zone</td>
    <td>Llama</td>
    <td>0.356</td>
    <td>0.357</td>
    <td>0.358</td>
    <td>0.361</td>
    <td>0.411</td>
    <td>0.349</td>
    <td>0.355</td>
    <td>0.369</td>
    <td>0.415</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.59</td>
    <td>0.570</td>
    <td>0.582</td>
    <td>0.552</td>
    <td>0.537</td>
    <td>0.530</td>
    <td>0.538</td>
    <td>0.508</td>
    <td>0.508</td>
  </tr>
  <tr>
    <td>quizballs</td>
    <td>Llama</td>
    <td>0.603</td>
    <td>0.575</td>
    <td>0.572</td>
    <td>0.578</td>
    <td>0.571</td>
    <td>0.602</td>
    <td>0.561</td>
    <td>0.567</td>
    <td>0.557</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.558</td>
    <td>0.565</td>
    <td>0.574</td>
    <td>0.571</td>
    <td>0.540</td>
    <td>0.549</td>
    <td>0.564</td>
    <td>0.539</td>
    <td>0.539</td>
  </tr>
  <tr>
    <td>quizwise</td>
    <td>Llama</td>
    <td>0.560</td>
    <td>0.565</td>
    <td>0.579</td>
    <td>0.609</td>
    <td>0.618</td>
    <td>0.561</td>
    <td>0.551</td>
    <td>0.574</td>
    <td>0.619</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.560</td>
    <td>0.563</td>
    <td>0.565</td>
    <td>0.570</td>
    <td>0.555</td>
    <td>0.556</td>
    <td>0.573</td>
    <td>0.573</td>
    <td>0.573</td>
  </tr>
  <tr>
    <td>sfquiz</td>
    <td>Llama</td>
    <td>0.568</td>
    <td>0.554</td>
    <td>0.554</td>
    <td>0.559</td>
    <td>0.575</td>
    <td>0.564</td>
    <td>0.559</td>
    <td>0.569</td>
    <td>0.583</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.584</td>
    <td>0.591</td>
    <td>0.590</td>
    <td>0.547</td>
    <td>0.570</td>
    <td>0.572</td>
    <td>0.573</td>
    <td>0.573</td>
    <td>0.573</td>
  </tr>
  <tr>
    <td>triviacountry</td>
    <td>Llama</td>
    <td>0.536</td>
    <td>0.554</td>
    <td>0.559</td>
    <td>0.566</td>
    <td>0.560</td>
    <td>0.530</td>
    <td>0.532</td>
    <td>0.549</td>
    <td>0.549</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.536</td>
    <td>0.551</td>
    <td>0.550</td>
    <td>0.587</td>
    <td>0.544</td>
    <td>0.476</td>
    <td>0.511</td>
    <td>0.472</td>
    <td>0.472</td>
  </tr>
  <tr>
    <td>wrexham**</td>
    <td>Llama</td>
    <td>0.570</td>
    <td>0.563</td>
    <td>0.569</td>
    <td>0.565</td>
    <td>0.567</td>
    <td>0.548</td>
    <td>0.541</td>
    <td>0.546</td>
    <td>0.566</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.548</td>
    <td>0.573</td>
    <td>0.578</td>
    <td>0.584</td>
    <td>0.553</td>
    <td>0.552</td>
    <td>0.579</td>
    <td>0.571</td>
    <td>0.571</td>
  </tr>
  <tr>
    <td>Average</td>
    <td>Llama</td>
    <td>0.562</td>
    <td>0.564</td>
    <td>0.568</td>
    <td>0.579</td>
    <td>0.584</td>
    <td>0.552</td>
    <td>0.550</td>
    <td>0.558</td>
    <td>0.567</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.567</td>
    <td>0.575</td>
    <td>0.583</td>
    <td>0.585</td>
    <td>0.553</td>
    <td>0.560</td>
    <td>0.562</td>
    <td>0.567</td>
    <td>0.587</td>
  </tr>
</table>

Table 6: Performance of SAPLMA on a fact dataset generated from TriviaQA. The original topic-wise leave-one-out strategy is adopted. Results are shown for the Llama 2-7b and OPT-6.7b models.

techniques, particularly in settings involving LLM-generated content. Promising research directions may be leveraging datasets that are closely aligned with the distribution of LLM-generated text and exploring alternative techniques such as uncertainty-aware classification.

6 Related Work

Probing techniques have become central for the layer-wise interpretation of deep learning models (Alain, 2016). This approach was then extended to Large Language Models (LLMs) to assess LLMs ability to encode syntax and semantics (Conneau et al., 2018; Tenney et al., 2019). Specifically, among the different applications, a technique that emerges is self-evaluation, defined as a model’s ability to assess the accuracy of its own outputs (Kadavath et al., 2022). Among the various works, notable research by Kadavath et al. (2022) explored estimating a well-calibrated \( P(True) \) directly from output probabilities to reflect answer accuracy. Azaria and Mitchell (2023) demonstrate that LLMs are capable of detecting false claims in synthetic true-false datasets. Building upon this initial study, several works (Marks and Tegmark, 2023; Bürger et al., 2024; Levinstein and Herrmann, 2024) investigate the generalization capabilities of probes in detecting hallucinations in LLMs. While these studies offer valuable insights into how factual knowledge is encoded in LLMs, they stop short of examining whether LLMs can assess the factuality of their own generations. Instead, they primarily evaluate models on artificially constructed or perturbed datasets, rather than on claims produced by the LLMs themselves. Orgad et al. (2024) explore error detection and hallucination mitigation in Question Answering by probing the internal representations of question-answer pairs. Their study shows that while LLMs encode factuality signals, these signals do not generalize well across task-specific datasets, suggesting that factuality encoding is task-dependent rather than universal. Gekhman et al. (2025) show that LLMs can internally encode knowledge, detectable by probing the hidden states, yet still fail to express it in their generated output. Chen et al. (2024) introduce the INSIDE framework, which detects hallu-
cinations using the EigenScore metric to measure self-consistency across multiple LLM outputs for a single input. Similarly, Zhang et al. (2024) leverage a probe model, PINOSE, trained via offline consistency-checking, to perform online hallucination detection. Although INSIDE and PINOSE primarily focus on self-consistency in internal representations across multiple generated responses, they do not interpret hidden states or look for explicit factuality encoding.

7 Conclusion and Future Work

In this paper, we investigate the factuality-encoding capabilities of LLMs. Our work replicates the methodology of Azaria and Mitchell (2023) to ensure reproducibility and extend their approach with two novel dataset construction strategies: perplexity-based negative sampling and fact generation based on QA datasets. We applied these strategies to analyze two open-source LLMs, and found that although the findings from previous studies are partially validated even on more challenging synthetic datasets, transferring these findings to LLM-generated datasets proves difficult. This study paves the way for more reliable LLM evaluations and offers practical guidelines for improving model transparency and trustworthiness in real-world applications.

Limitations

There are a few aspects of our study that could be explored further. We made every effort to check the datasets for inconsistencies, but a more thorough manual verification by human annotators would be beneficial for ensuring their robustness and minimizing potential biases. Additionally, our analysis is based on a limited set of models. While these models provide valuable insights, it is possible that larger or more complex models could demonstrate enhanced performance, particularly in the context of self-evaluation. Future work could expand on this, incorporating a wider range of models to investigate whether scalability can improve results.

Lastly, we highlight important considerations regarding the use of perplexity as a proxy for plausibility in relation to model knowledge. Recent findings suggest that LLMs often assign similar perplexity scores to both seen and unseen sentences (Duan et al., 2024), which may limit the reliability of perplexity in distinguishing between plausible and implausible content. This challenge intersects with broader issues of memorization and data provenance. While our work focuses on extending and refining prior approaches to factuality self-evaluation, we recognize that a rigorous assessment of perplexity effectiveness in this context is a critical avenue for future investigation. We therefore encourage further research into more robust and theoretically grounded measures of plausibility for LLM-generated content.

Ethical Statement

A major concern when working with LLMs is their tendency to generate factually inaccurate information. When training probe classifiers to assess factual accuracy, biases and beliefs from the LLM may transfer to the probe, potentially reinforcing cultural, demographic, or ideological biases in factuality self-evaluation. With careful design, probing techniques can be adapted not only to minimize bias but also to actively mitigate its consequences.

Acknowledgments

This study was supported by the MOST – Sustainable Mobility National Research Center funded by the European Union Next-GenerationEU (Italian National Recovery and Resilience Plan (NRRP) – M4C2, Investment 1.4 – D.D. 1033 17/06/2022, CN00000023 - CUP: D93C22000410001). We acknowledge ISCRA for awarding this project access to the LEONARDO supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CINECA (Italy). Patti Territoriali WPI, OVS: Fashion Retail Reloaded, Lutech Digitale 4.0, Natuzzi S.p.A. del Contratto di Sviluppo Industriale ai sensi dell’art. 9 del Decreto del Ministro dello Sviluppo Economico del 09.12.2014. EPANSA (FAIR) - Enhancing Personal Assistants with Neuro-Symbolic AI and Knowledge Graphs, funded by the European Union Next-GenerationEU (NRRP – M4C2, Investment 1.3, D.R. No. 123 of 16/01/2024, PE00000013, CUP: H97G22000210007). XA14AMELIA - Implementation of Explainable Artificial Intelligence Technologies for Advancing Interoperability and Analysis in AMELIA, funded by the European Union Next-GenerationEU (NRRP – M4C2, Investment 1.3, D.D. 341 del 15/03/2022, CUP: J33C22002910001). Furthermore, this work has been carried out while Giovanni Servedio was enrolled in the Italian National Doctorate on Artificial Intelligence run by Sapienza University of Rome in collaboration with Politecnico di Bari.
References

Guillaume Alain. 2016. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644.

Vito Walter Anelli, Giovanni Maria Biancofiore, Alessandro De Bellis, Tommaso Di Noia, and Eugenio Di Sciascio. 2022. Interpretability of BERT latent space through knowledge graphs. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022, pages 3806–3810. ACM.

Amos Azaria and Tom M. Mitchell. 2023. The internal state of an LLM knows when it’s lying. In EMNLP (Findings), pages 967–976. Association for Computational Linguistics.

Giovanni Maria Biancofiore, Dario Di Palma, Claudio Pomo, Fedelucio Narducci, and Tommaso Di Noia. 2025. Conversational user interfaces and agents. In Human-Centered AI: An Illustrated Scientific Quest, pages 399–438. Springer.

Lennart Bürger, Fred A. Hamprecht, and Boaz Nadler. 2024. Truth is universal: Robust detection of lies in llms. In NeurIPS.

Nitay Calderon, Roi Reichart, and Rotem Dror. 2025. The alternative annotator test for llm-as-a-judge: How to statistically justify replacing human annotators with llms. CoRR, abs/2501.10970.

Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024. Inside: Llms’ internal states retain the power of hallucination detection. arXiv preprint arXiv:2402.03744.

Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. arXiv preprint arXiv:1805.01070.

Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E. Ho. 2024. Large legal fictions: Profiling legal hallucinations in large language models. CoRR, abs/2401.01301.

Alessandro De Bellis, Vito Walter Anelli, Tommaso Di Noia, and Eugenio Di Sciascio. 2024. PRONTO: prompt-based detection of semantic containment patterns in MLMs. In ISWC (2), volume 15232 of Lecture Notes in Computer Science, pages 227–246. Springer.

Dario Di Palma. 2023. Retrieval-augmented recommender system: Enhancing recommender systems with large language models. In RecSys, pages 1369–1373. ACM.

Dario Di Palma, Felice Antonio Merra, Maurizio Sfilio, Vito Walter Anelli, Fedelucio Narducci, and Tommaso Di Noia. 2025. Do llms memorize recommendation datasets? a preliminary study on movielens-lm. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2025, Padua, Italy July 13-18, 2025. ACM.

Michael Duan, Anshuman Suri, Nilooofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. 2024. Do membership inference attacks work on large language models? CoRR, abs/2402.07841.

Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. 2024. Don’t hallucinate, abstain: Identifying LLM knowledge gaps via multi-LLM collaboration. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14664–14690, Bangkok, Thailand. Association for Computational Linguistics.

Zorik Gekhman, Eyal Ben-David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpektor, Jonathan Herzig, and Roi Reichart. 2025. Inside-out: Hidden factual knowledge in llms. CoRR, abs/2503.15299.

Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhao Wang, and Jian Guo. 2024. A survey on llm-as-a-judge. CoRR, abs/2411.15594.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2024. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst.

Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023. Towards mitigating LLM hallucination via self reflection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1827–1843, Singapore. Association for Computational Linguistics.

Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL (1), pages 1601–1611. Association for Computational Linguistics.

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario
Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know. CoRR, abs/2207.05221.

Benjamin A Levinstein and Daniel A Herrmann. 2024. Still no lie detector for language models: Probing empirical and conceptual roadblocks. Philosophical Studies, pages 1–27.

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In ACL (1), pages 3214–3252. Association for Computational Linguistics.

Samuel Marks and Max Tegmark. 2023. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. CoRR, abs/2310.06824.

Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Bisk. 2024. Lms know more than they show: On the intrinsic representation of LLM hallucinations. CoRR, abs/2410.02707.

Duy Khoa Pham and Bao Quoc Vo. 2024. Towards reliable medical question answering: Techniques and challenges in mitigating hallucinations in language models. CoRR, abs/2408.13808.

Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In ACL (2), pages 784–789. Association for Computational Linguistics.

Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. arXiv preprint arXiv:1905.06316.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, and Yasmine Babaei et al. 2023. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288.

Shriyash Upadhyay, Etan Ginsberg, and Chris Callison-Burch. 2023. Improving mathematics tutoring with A code scratchpad. In BEA@ACL, pages 20–28. Association for Computational Linguistics.

David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylem, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online. Association for Computational Linguistics.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuhui Chen, Christopher DeWan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.

Xiaokang Zhang, Zijun Yao, Jing Zhang, Kaifeng Yun, Jifan Yu, Juanzi Li, and Jie Tang. 2024. Transferable and efficient non-factual content detection via probe training with offline consistency checking. arXiv preprint arXiv:2404.06742.

Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. Siren’s song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219.
Appendix

A  Dataset Statistics

A.1  True-False Dataset (Azaria and Mitchell, 2023) Statistics

Table 7 provides a summary of the number of sentences and the distribution of true and false statements across each topic in the original True-False dataset (Azaria and Mitchell, 2023) on which we base our replication experiments.

<table>
  <tr>
    <th>Dataset</th>
    <th>Sentences</th>
    <th>Source</th>
    <th>(%) True</th>
  </tr>
  <tr>
    <td>Cities</td>
    <td>1458</td>
    <td>SimpleMaps dataset</td>
    <td>50</td>
  </tr>
  <tr>
    <td>Inventions</td>
    <td>876</td>
    <td>Wikipedia’s list of inventors</td>
    <td>53</td>
  </tr>
  <tr>
    <td>Chemical Elements</td>
    <td>930</td>
    <td>PubChem’s periodic table</td>
    <td>50</td>
  </tr>
  <tr>
    <td>Animals</td>
    <td>1008</td>
    <td>National Geographic Kids</td>
    <td>50</td>
  </tr>
  <tr>
    <td>Companies</td>
    <td>1200</td>
    <td>Forbes Global 2000 List 2022: The Top 200</td>
    <td>50</td>
  </tr>
  <tr>
    <td>Scientific Facts</td>
    <td>612</td>
    <td>ChatGPT and human annotators</td>
    <td>50</td>
  </tr>
</table>

Table 7: True-False dataset categories, original sources, and label splits from (Azaria and Mitchell, 2023).

A.2  LLM-Generated Trivia Facts Dataset

Table 8 illustrates the number of annotated facts extracted from TriviaQA (Joshi et al., 2017) employing the procedure described in section 4.2. Statistics are presented for each of the 14 question sources. Table 9 reports examples of generated facts for the Llama 2-7b and OPT-6.7b models.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="2">Llama 2-7b</th>
    <th colspan="2">OPT-6.7b</th>
  </tr>
  <tr>
    <th>Sentences</th>
    <th>(%) True</th>
    <th>Sentences</th>
    <th>(%) True</th>
  </tr>
  <tr>
    <td>triviacountry</td>
    <td>118</td>
    <td>49.15%</td>
    <td>118</td>
    <td>49.15%</td>
  </tr>
  <tr>
    <td>wwwodquiz</td>
    <td>700</td>
    <td>49.71%</td>
    <td>525</td>
    <td>49.71%</td>
  </tr>
  <tr>
    <td>trivibug</td>
    <td>97</td>
    <td>50.52%</td>
    <td>88</td>
    <td>51.14%</td>
  </tr>
  <tr>
    <td>derby*</td>
    <td>342</td>
    <td>50.00%</td>
    <td>325</td>
    <td>50.00%</td>
  </tr>
  <tr>
    <td>quiz-zone</td>
    <td>187</td>
    <td>48.13%</td>
    <td>239</td>
    <td>50.63%</td>
  </tr>
  <tr>
    <td>businessballs</td>
    <td>433</td>
    <td>50.84%</td>
    <td>443</td>
    <td>49.43%</td>
  </tr>
  <tr>
    <td>wrexham**</td>
    <td>216</td>
    <td>50.00%</td>
    <td>276</td>
    <td>50.72%</td>
  </tr>
  <tr>
    <td>sfquiz</td>
    <td>1054</td>
    <td>49.80%</td>
    <td>1085</td>
    <td>50.39%</td>
  </tr>
  <tr>
    <td>quizwise</td>
    <td>565</td>
    <td>49.91%</td>
    <td>704</td>
    <td>50.42%</td>
  </tr>
  <tr>
    <td>billturnbull</td>
    <td>216</td>
    <td>49.54%</td>
    <td>105</td>
    <td>50.48%</td>
  </tr>
  <tr>
    <td>jetpunk</td>
    <td>139</td>
    <td>51.80%</td>
    <td>364</td>
    <td>50.27%</td>
  </tr>
  <tr>
    <td>quizballs</td>
    <td>500</td>
    <td>49.60%</td>
    <td>488</td>
    <td>50.82%</td>
  </tr>
  <tr>
    <td>quizguy</td>
    <td>240</td>
    <td>51.67%</td>
    <td>341</td>
    <td>50.74%</td>
  </tr>
  <tr>
    <td>quiz4free</td>
    <td>171</td>
    <td>49.12%</td>
    <td>141</td>
    <td>48.94%</td>
  </tr>
</table>

Table 8: Summary of the dataset obtained by extracting factoid sentences from TriviaQA (Joshi et al., 2017) Wikipedia validation split, following the procedure described in Section 4.2.

B  Perplexity-Based Refinement of the True-False Dataset (Azaria and Mitchell, 2023)

We base our dataset generation strategy described in Section 4 on the same tabular data employed by Azaria and Mitchell, which was made available upon request. The properties used for sampling are analogous to the ones employed by the original authors in their dataset generation. However, for the Cities topic, we restrict the analysis to facts like "<city> is city in <country>." Differently from the original authors, we avoid generating facts such as "<city> is the name of a city/country" having only two possible values, potentially resulting in too many easy-to-classify samples. Table 10 reports examples of false sentences generated following the perplexity-based sampling procedure described in Section 4.1 and the Llama 2-7b model.

<table>
  <tr>
    <th>Model</th>
    <th>Label</th>
    <th>Sentence</th>
  </tr>
  <tr>
    <td>Llama 2-7b</td>
    <td>1</td>
    <td>Arthur was married to Guinevere.</td>
  </tr>
  <tr>
    <td></td>
    <td>1</td>
    <td>Arthur's most famous wife was Guinevere.</td>
  </tr>
  <tr>
    <td></td>
    <td>1</td>
    <td>Guinevere was married to Arthur.</td>
  </tr>
  <tr>
    <td></td>
    <td>0</td>
    <td>Guinevere's husband was King Arthur.</td>
  </tr>
  <tr>
    <td></td>
    <td>0</td>
    <td>Lancelot was married to Guinevere.</td>
  </tr>
  <tr>
    <td></td>
    <td>0</td>
    <td>Sir Lancelot was married to Queen Guinevere.</td>
  </tr>
  <tr>
    <td></td>
    <td>0</td>
    <td>Sir Lancelot was married to Queen Guinevere.</td>
  </tr>
  <tr>
    <td>OPT-6.7b</td>
    <td>0</td>
    <td>Anakin Skywalker is Darth Vader's son.</td>
  </tr>
  <tr>
    <td></td>
    <td>0</td>
    <td>Darth Vader's son is Luke Skywalker.</td>
  </tr>
  <tr>
    <td></td>
    <td>1</td>
    <td>Darth Vader's son is Luke Skywalker, a member of the Rebel Alliance.</td>
  </tr>
  <tr>
    <td></td>
    <td>1</td>
    <td>Darth Vader's son is Luke.</td>
  </tr>
  <tr>
    <td></td>
    <td>0</td>
    <td>Darth Vader's son is known as Darth Vader.</td>
  </tr>
  <tr>
    <td></td>
    <td>1</td>
    <td>Luke's father is Darth Vader.</td>
  </tr>
</table>

Table 9: Examples of true and false facts generated by Llama 2-7b and OPT-6.7b based on the questions in TriviaQA (Joshi et al., 2017), obtained following the procedure detailed in Section 4.2.

<table>
  <tr>
    <th>True Sentence</th>
    <th>Original Negative</th>
    <th>Generated Negative (Llama 2-7b)</th>
  </tr>
  <tr>
    <td>The crocodile has a habitat of freshwater.</td>
    <td>The crocodile has a habitat of various freshwater.</td>
    <td>The crocodile has a habitat of grassland/sea.</td>
  </tr>
  <tr>
    <td>The zebra has distinctive black and white stripes, which may help deter flies and provide camouflage.</td>
    <td>The zebra is a fast swimmer and can maintain high speeds for extended periods of time.</td>
    <td>The zebra is the fastest land animal, reaching speeds up to 60-70 mph.</td>
  </tr>
  <tr>
    <td>Tantalum has the symbol Ta.</td>
    <td>Tantalum has the symbol Co.</td>
    <td>Tantalum has the symbol Tin.</td>
  </tr>
</table>

Table 10: Examples of false sentences generated with the Llama 2-7b model, using the perplexity-based generation strategy described in Section 4.1.

C  LLM-Generated Dataset Extraction Details

C.1  Factoid Answer Generation

Below is an example prompt used in the text generation pipeline for TriviaQA (Joshi et al., 2017). The 10 examples are sampled from the train Wikipedia split. The answers for the examples are manually crafted by looking at the available ground truth.

Question: Where in England was Dame Judi Dench born?
Answer: The English actress Dame Judi Dench was born in York, England.

Question: From which country did Angola achieve independence in 1975?
Answer: Angola achieved independence from Portugal in 1975.

Question: Which city does David Soul come from?
Answer: David Soul hails from Chicago, Illinois.

Question: Who won Super Bowl XX?
Answer: The Chicago Bears won Super Bowl XX.

Question: Which was the first European country to abolish capital punishment?
Answer: Norway was the first European country to abolish capital punishment.

Question: In which country did the widespread use of ISDN begin in 1988?
Answer: The widespread use of ISDN began in Japan in 1988.

Question: What is Bruce Willis' real first name?
Answer: Bruce Willis' real first name is Walter.

Question: Which William wrote the novel Lord of the Flies?
Answer: The William who wrote Lord of the Flies was William Golding.

Question: How is Joan Molinsky better known?
Answer: Joan Molinsky is better known as Joan Rivers.

Question: In which branch of the arts is Patricia Neary famous?
Answer: Patricia Neary is famous in the field of ballet.

To generate responses, the model is provided with a continuation prompt such as:

Question: Ford Prefect came from a star in which constellation?
Answer:

The model then generates a response, adhering to the pattern set by the previous examples. Responses are filtered based on length and uniqueness to ensure meaningful and high-quality outputs.

The generation process uses the following hyper-parameters for all the employed models (i.e. Llama 2-7b and OPT-6.7b):

• max_new_tokens: 128

• top_k: 50

• top_p: 0.95

• temperature: 1.0

• num_return_sequences: 10

C.2 Factoid Answer Annotation

Below is an example prompt to GPT-4o mini (2024-08-01), the LLM that we use as the oracle for the procedure described in Section 4.2. In this setup, we employ a 3-shot learning strategy, where the model is provided with three examples to establish a clear pattern of judgment. These examples demonstrate the task of evaluating answers against a given ground truth. The model is then tasked with using this context to evaluate a new answer to a question, based on the same criteria. In our experiments, we employ a temperature setting of 1.

SYSTEM:

You are a judge and your role is to judge whether the provided answer is correct for the given question, based on the provided ground truth. Answer with a 1 if the answer is correct and 0 if the answer is incorrect.
Here are a few examples:

USER:

Question: Who was the next British Prime Minister after Arthur Balfour?
Answer: Arthur Balfour was followed by David Lloyd George.
Ground truth: "{\"aliases\": ['Sir Henry Campbell-Bannerman', 'Campbell-Bannerman', 'Campbell Bannerman', 'Sir Henry Campbell Bannerman', 'Henry Campbell-Bannerman'], 'normalized_aliases': ['henry campbell bannerman', 'sir henry campbell bannerman', 'campbell bannerman'], 'matched_wiki_entity_name': '', 'normalized_matched_wiki_entity_name': '', 'normalized_value': 'campbell bannerman', 'type': 'WikipediaEntity', 'value': 'Campbell -Bannerman'}"
Evaluation:

ASSISTANT:
0

USER:

Question: Who had a 70s No 1 hit with Kiss You All Over?
Answer: The band Exile had a 70s No 1 hit with Kiss You All Over.
Ground truth: "{\"aliases\": ['Internal exile', 'Exiles', 'Transported for life', 'Exile (politics and government)', 'Voluntary exile', 'Sent into exile', 'Exile and Banishment', 'Self-exile', 'Forced
exile', 'Exile', 'Exile in Greek tragedy', 'Banish', 'Banishment'], 'normalized_aliases': ['exiles', 'voluntary exile', 'forced exile', 'banish', 'self exile', 'exile in politics and government', 'exile in greek tragedy', 'sent into exile', 'banishment', 'transported for life', 'exile', 'internal exile', 'exile and banishment'], 'matched_wiki_entity_name': '', 'normalized_matched_wiki_entity_name': '', 'normalized_value': 'exile', 'type': 'WikipediaEntity', 'value': 'Exile'})

Evaluation:

ASSISTANT:

1

USER:

Question: Which common mineral is used to make casts, moulds, blackboard chalk and plaster of Paris?
Answer: The common mineral used to make casts, moulds, blackboard chalk and plaster of Paris is calcium carbonate.
Ground truth: "{'aliases': ['CaSO4.2H2O', 'Gypsum', 'Calcium sulfate dihydrate', 'CaSO4*2H2O', 'Gipsum'], 'normalized_aliases': ['calcium sulfate dihydrate', 'caso4 2h2o', 'gipsum', 'caso4.2h2o', 'gypsum'], 'matched_wiki_entity_name': '', 'normalized_matched_wiki_entity_name': '', 'normalized_value': 'gypsum', 'type': 'WikipediaEntity', 'value': 'Gypsum'}"

Evaluation:

ASSISTANT:

0

D Additional Experiments on the LLM-Generated Dataset

Table 11 reports the performances of the SAPLMA classifier trained on the original True-False dataset by Azaria and Mitchell (2023) and tested on the dataset generated from TriviaQA. Similarly to Table 6, SAPLMA does not generalize well over LLM-generated facts. Moreover, tuning an optimal threshold did not provide solid enhancements.

E Experiments on Additional LLM-Generated Datasets

We extended our LLM-generated dataset construction approach (Section 4.2) to TruthfulQA (Lin et al., 2022) and SQuAD 2.0 (Rajpurkar et al., 2018) to further support our findings.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="6">Threshold = 0.5</th>
    <th colspan="6">Optimal Threshold</th>
  </tr>
  <tr>
    <th>last</th>
    <th>28</th>
    <th>24</th>
    <th>20</th>
    <th>16</th>
    <th>last</th>
    <th>28</th>
    <th>24</th>
    <th>20</th>
    <th>16</th>
  </tr>
  <tr>
    <td>billturnbull</td>
    <td>Llama</td><td>579</td><td>560</td><td>560</td><td>593</td><td>648</td><td>605</td><td>563</td><td>632</td><td>631</td><td>691</td>
    <td>OPT</td><td>543</td><td>533</td><td>533</td><td>486</td><td>476</td><td>527</td><td>500</td><td>486</td><td>500</td><td>500</td>
  </tr>
  <tr>
    <td>derby*</td>
    <td>Llama</td><td>556</td><td>576</td><td>350</td><td>544</td><td>602</td><td>542</td><td>529</td><td>529</td><td>357</td><td>375</td>
    <td>OPT</td><td>526</td><td>548</td><td>554</td><td>563</td><td>535</td><td>535</td><td>583</td><td>579</td><td>579</td><td>561</td>
  </tr>
  <tr>
    <td>quiz4free</td>
    <td>Llama</td><td>602</td><td>573</td><td>544</td><td>538</td><td>608</td><td>575</td><td>600</td><td>525</td><td>350</td><td>642</td>
    <td>OPT</td><td>525</td><td>489</td><td>511</td><td>525</td><td>504</td><td>545</td><td>535</td><td>556</td><td>576</td><td>495</td>
  </tr>
  <tr>
    <td>quizguy</td>
    <td>Llama</td><td>606</td><td>571</td><td>546</td><td>567</td><td>571</td><td>601</td><td>595</td><td>586</td><td>577</td><td>565</td>
    <td>OPT</td><td>557</td><td>537</td><td>587</td><td>569</td><td>551</td><td>586</td><td>594</td><td>552</td><td>546</td><td>552</td>
  </tr>
  <tr>
    <td>triviabug</td>
    <td>Llama</td><td>612</td><td>619</td><td>526</td><td>598</td><td>577</td><td>485</td><td>474</td><td>474</td><td>344</td><td>344</td>
    <td>OPT</td><td>552</td><td>602</td><td>576</td><td>614</td><td>500</td><td>629</td><td>597</td><td>629</td><td>497</td><td>352</td>
  </tr>
  <tr>
    <td>businessballs</td>
    <td>Llama</td><td>580</td><td>577</td><td>539</td><td>575</td><td>587</td><td>586</td><td>605</td><td>546</td><td>583</td><td>392</td>
    <td>OPT</td><td>564</td><td>578</td><td>555</td><td>581</td><td>521</td><td>585</td><td>579</td><td>585</td><td>547</td><td>437</td>
  </tr>
  <tr>
    <td>jetpunk</td>
    <td>Llama</td><td>612</td><td>600</td><td>583</td><td>519</td><td>600</td><td>582</td><td>561</td><td>520</td><td>592</td><td>675</td>
    <td>OPT</td><td>569</td><td>571</td><td>592</td><td>604</td><td>566</td><td>561</td><td>631</td><td>631</td><td>631</td><td>635</td>
  </tr>
  <tr>
    <td>odquiz</td>
    <td>Llama</td><td>546</td><td>516</td><td>537</td><td>559</td><td>564</td><td>537</td><td>522</td><td>547</td><td>563</td><td>571</td>
    <td>OPT</td><td>511</td><td>541</td><td>591</td><td>579</td><td>552</td><td>563</td><td>535</td><td>592</td><td>571</td><td>565</td>
  </tr>
  <tr>
    <td>quiz-zone</td>
    <td>Llama</td><td>578</td><td>519</td><td>588</td><td>562</td><td>578</td><td>617</td><td>534</td><td>534</td><td>534</td><td>357</td>
    <td>OPT</td><td>544</td><td>577</td><td>602</td><td>611</td><td>586</td><td>542</td><td>518</td><td>554</td><td>601</td><td>613</td>
  </tr>
  <tr>
    <td>quizballs</td>
    <td>Llama</td><td>610</td><td>568</td><td>592</td><td>582</td><td>582</td><td>617</td><td>563</td><td>587</td><td>586</td><td>586</td>
    <td>OPT</td><td>535</td><td>578</td><td>559</td><td>594</td><td>549</td><td>532</td><td>564</td><td>519</td><td>576</td><td>576</td>
  </tr>
  <tr>
    <td>quizwise</td>
    <td>Llama</td><td>572</td><td>588</td><td>570</td><td>572</td><td>586</td><td>581</td><td>598</td><td>611</td><td>616</td><td>616</td>
    <td>OPT</td><td>540</td><td>550</td><td>574</td><td>568</td><td>551</td><td>580</td><td>550</td><td>554</td><td>582</td><td>576</td>
  </tr>
  <tr>
    <td>sfquiz</td>
    <td>Llama</td><td>530</td><td>533</td><td>528</td><td>538</td><td>560</td><td>545</td><td>581</td><td>584</td><td>570</td><td>575</td>
    <td>OPT</td><td>531</td><td>546</td><td>568</td><td>590</td><td>545</td><td>595</td><td>597</td><td>590</td><td>611</td><td>553</td>
  </tr>
  <tr>
    <td>triviacountry</td>
    <td>Llama</td><td>602</td><td>636</td><td>602</td><td>602</td><td>619</td><td>639</td><td>602</td><td>639</td><td>599</td><td>578</td>
    <td>OPT</td><td>525</td><td>542</td><td>585</td><td>551</td><td>525</td><td>494</td><td>602</td><td>506</td><td>578</td><td>446</td>
  </tr>
  <tr>
    <td>wrexham**</td>
    <td>Llama</td><td>532</td><td>519</td><td>528</td><td>574</td><td>583</td><td>474</td><td>526</td><td>533</td><td>566</td><td>579</td>
    <td>OPT</td><td>500</td><td>533</td><td>594</td><td>583</td><td>525</td><td>572</td><td>552</td><td>562</td><td>608</td><td>567</td>
  </tr>
  <tr>
    <td>Average</td>
    <td>Llama</td><td>566</td><td>569</td><td>558</td><td>573</td><td>593</td><td>569</td><td>560</td><td>560</td><td>572</td><td>596</td>
    <td>OPT</td><td>538</td><td>553</td><td>577</td><td>573</td><td>535</td><td>567</td><td>563</td><td>563</td><td>581</td><td>551</td>
  </tr>
</table>

* derby is adopted as abbreviation of derbyshirepubquizleague
**: wrexham is adopted as abbreviation of wrexhamquizleague

Table 11: Accuracy values obtained training SAPLMA on the original True-False dataset and testing on our facts dataset generated from TriviaQA. The original topic-wise leave-one-out strategy is adopted. Results are shown for the Llama 2-7b and OPT-6.7b models.

<table>
  <tr>
    <th>Model</th>
    <th>Total Samples</th>
    <th>True Samples (%)</th>
  </tr>
  <tr>
    <td>Llama 2-7b</td>
    <td>842</td>
    <td>49.76%</td>
  </tr>
  <tr>
    <td>OPT-6.7b</td>
    <td>331</td>
    <td>49.85%</td>
  </tr>
</table>

Table 12: Summary of the dataset obtained by extracting factoid sentences from TruthfulQA (Lin et al., 2022), following the procedure described in Section 4.2.

Following the same procedure detailed in Section 4.2, we extract true and false facts from TruthfulQA. Table 12 reports the statistics of the LLM-generated datasets constructed with Llama 2-7b and OPT-6.7b. Table 13 presents the performance of the SAPLMA classifier when both trained and tested on the TruthfulQA dataset. Additionally, Table 14 shows the results obtained by SAPLMA on TruthfulQA after being trained on the original True-False dataset. Since the dataset lacks a topic-based split, we adopt a 50-50 holdout strategy for evaluation. The results on the TruthfulQA dataset
<table>
  <tr>
    <th colspan="2">OPT-6.7b</th>
    <th colspan="2">Llama 2-7b</th>
  </tr>
  <tr>
    <th>Random Split 1</th>
    <th>Random Split 2</th>
    <th>Random Split 1</th>
    <th>Random Split 2</th>
  </tr>
  <tr>
    <td>last</td>
    <td>0.487</td>
    <td>0.488</td>
    <td>0.491</td>
    <td>0.504</td>
  </tr>
  <tr>
    <td>28</td>
    <td>0.477</td>
    <td>0.495</td>
    <td>0.498</td>
    <td>0.506</td>
  </tr>
  <tr>
    <td>24</td>
    <td>0.478</td>
    <td>0.513</td>
    <td>0.507</td>
    <td>0.507</td>
  </tr>
  <tr>
    <td>20</td>
    <td>0.512</td>
    <td>0.535</td>
    <td>0.518</td>
    <td>0.509</td>
  </tr>
  <tr>
    <td>16</td>
    <td>0.493</td>
    <td>0.492</td>
    <td>0.520</td>
    <td>0.514</td>
  </tr>
</table>

Table 13: Performance of SAPLMA on a fact dataset generated from TruthfulQA. A 50-50 holdout strategy is adopted. Results are shown for the Llama 2-7b and OPT-6.7b models.

<table>
  <tr>
    <th colspan="2">OPT-6.7b</th>
    <th colspan="2">Llama 2-7b</th>
  </tr>
  <tr>
    <th>Random Split 1</th>
    <th>Random Split 2</th>
    <th>Random Split 1</th>
    <th>Random Split 2</th>
  </tr>
  <tr>
    <td>last</td>
    <td>0.473</td>
    <td>0.560</td>
    <td>0.518</td>
    <td>0.539</td>
  </tr>
  <tr>
    <td>28</td>
    <td>0.515</td>
    <td>0.566</td>
    <td>0.520</td>
    <td>0.514</td>
  </tr>
  <tr>
    <td>24</td>
    <td>0.503</td>
    <td>0.476</td>
    <td>0.518</td>
    <td>0.501</td>
  </tr>
  <tr>
    <td>20</td>
    <td>0.455</td>
    <td>0.542</td>
    <td>0.499</td>
    <td>0.525</td>
  </tr>
  <tr>
    <td>16</td>
    <td>0.479</td>
    <td>0.500</td>
    <td>0.485</td>
    <td>0.530</td>
  </tr>
</table>

Table 14: Accuracy values obtained training SAPLMA on the original True-False dataset and testing on our facts dataset generated from TruthfulQA. A 50-50 holdout strategy is adopted. Results are shown for the Llama 2-7b and OPT-6.7b models.

support our findings: the probe classifier fails to generalize effectively to LLM-generated datasets.

E.2 LLM-Generated Facts from SQuAD 2.0

For the SQuAD 2.0 dataset, we limit our dataset construction procedure (Section 4.2) to the validation split and the five most common topics, as the large number of questions makes applying our procedure to the full dataset computationally impractical due to time constraints. Table 16 presents the performance of the SAPLMA classifier when trained and tested on the SQuAD 2.0 dataset. Table 17, instead, reports the results obtained on SQuAD 2.0 after training on the original True-False dataset from Azaria and Mitchell (2023).

The results on the SQuAD 2.0 dataset further support our conclusion: existing probes fall short in accurately assessing factuality in real-world settings. Across all cases, accuracy remains below a meaningful threshold, highlighting the limited reliability of these methods for self-assessment in scenarios involving LLM-generated sentences.

<table>
  <tr>
    <th>Topic</th>
    <th>Model</th>
    <th>Sentences</th>
    <th>(%) True</th>
  </tr>
  <tr>
    <td>Economic_inequality</td>
    <td>Llama 2-7b</td>
    <td>224</td>
    <td>50.45%</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT-6.7b</td>
    <td>135</td>
    <td>50.37%</td>
  </tr>
  <tr>
    <td>Immune_system</td>
    <td>Llama 2-7b</td>
    <td>167</td>
    <td>49.70%</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT-6.7b</td>
    <td>72</td>
    <td>50.00%</td>
  </tr>
  <tr>
    <td>Rhine</td>
    <td>Llama 2-7b</td>
    <td>120</td>
    <td>50.83%</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT-6.7b</td>
    <td>59</td>
    <td>47.46%</td>
  </tr>
  <tr>
    <td>Warsaw</td>
    <td>Llama 2-7b</td>
    <td>140</td>
    <td>51.43%</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT-6.7b</td>
    <td>102</td>
    <td>50.00%</td>
  </tr>
  <tr>
    <td>Yuan Dynasty</td>
    <td>Llama 2-7b</td>
    <td>109</td>
    <td>49.54%</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT-6.7b</td>
    <td>53</td>
    <td>52.83%</td>
  </tr>
</table>

Table 15: Summary of the dataset obtained by extracting factoid sentences from SQuAD 2.0 (Rajpurkar et al., 2018), following the procedure described in Section 4.2.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="5">last</th>
  </tr>
  <tr>
    <th>28</th>
    <th>24</th>
    <th>20</th>
    <th>16</th>
  </tr>
  <tr>
    <td>Economic_inequality</td>
    <td>Llama</td>
    <td>0.525</td>
    <td>0.510</td>
    <td>0.515</td>
    <td>0.492</td>
    <td>0.502</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.517</td>
    <td>0.516</td>
    <td>0.499</td>
    <td>0.496</td>
    <td>0.464</td>
  </tr>
  <tr>
    <td>Immune_system</td>
    <td>Llama</td>
    <td>0.486</td>
    <td>0.490</td>
    <td>0.503</td>
    <td>0.502</td>
    <td>0.547</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.436</td>
    <td>0.439</td>
    <td>0.401</td>
    <td>0.438</td>
    <td>0.435</td>
  </tr>
  <tr>
    <td>Rhine</td>
    <td>Llama</td>
    <td>0.525</td>
    <td>0.520</td>
    <td>0.515</td>
    <td>0.524</td>
    <td>0.526</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.484</td>
    <td>0.485</td>
    <td>0.522</td>
    <td>0.547</td>
    <td>0.521</td>
  </tr>
  <tr>
    <td>Warsaw</td>
    <td>Llama</td>
    <td>0.558</td>
    <td>0.542</td>
    <td>0.545</td>
    <td>0.546</td>
    <td>0.555</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.521</td>
    <td>0.531</td>
    <td>0.545</td>
    <td>0.544</td>
    <td>0.509</td>
  </tr>
  <tr>
    <td>Yuan_dynasty</td>
    <td>Llama</td>
    <td>0.509</td>
    <td>0.522</td>
    <td>0.514</td>
    <td>0.529</td>
    <td>0.544</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.544</td>
    <td>0.560</td>
    <td>0.512</td>
    <td>0.542</td>
    <td>0.512</td>
  </tr>
</table>

Table 16: Performance of SAPLMA on a fact dataset generated from SQuAD 2.0, restricted to the validation set and the top-5 most popular topics. The original topic-wise leave-one-out strategy is adopted. Results are shown for the Llama 2-7b and OPT-6.7b models.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="5">last</th>
  </tr>
  <tr>
    <th>28</th>
    <th>24</th>
    <th>20</th>
    <th>16</th>
  </tr>
  <tr>
    <td>Economic_inequality</td>
    <td>Llama</td>
    <td>0.558</td>
    <td>0.531</td>
    <td>0.589</td>
    <td>0.531</td>
    <td>0.545</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.600</td>
    <td>0.600</td>
    <td>0.548</td>
    <td>0.533</td>
    <td>0.533</td>
  </tr>
  <tr>
    <td>Immune_system</td>
    <td>Llama</td>
    <td>0.569</td>
    <td>0.593</td>
    <td>0.593</td>
    <td>0.575</td>
    <td>0.599</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.611</td>
    <td>0.611</td>
    <td>0.611</td>
    <td>0.708</td>
    <td>0.542</td>
  </tr>
  <tr>
    <td>Rhine</td>
    <td>Llama</td>
    <td>0.467</td>
    <td>0.492</td>
    <td>0.475</td>
    <td>0.442</td>
    <td>0.525</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.525</td>
    <td>0.559</td>
    <td>0.593</td>
    <td>0.593</td>
    <td>0.576</td>
  </tr>
  <tr>
    <td>Warsaw</td>
    <td>Llama</td>
    <td>0.614</td>
    <td>0.579</td>
    <td>0.607</td>
    <td>0.600</td>
    <td>0.593</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.48</td>
    <td>0.500</td>
    <td>0.539</td>
    <td>0.598</td>
    <td>0.549</td>
  </tr>
  <tr>
    <td>Yuan_dynasty</td>
    <td>Llama</td>
    <td>0.523</td>
    <td>0.523</td>
    <td>0.541</td>
    <td>0.505</td>
    <td>0.550</td>
  </tr>
  <tr>
    <td></td>
    <td>OPT</td>
    <td>0.453</td>
    <td>0.547</td>
    <td>0.491</td>
    <td>0.528</td>
    <td>0.547</td>
  </tr>
</table>

Table 17: Accuracy values obtained training SAPLMA on the original True-False dataset and testing on the facts dataset generated from SQuAD 2.0. Results are shown for the Llama 2-7b and OPT-6.7b models.

F Examples of SAPLMA Predictions

The tables in this appendix provide illustrative examples that complement the quantitative results discussed in the main text. Specifically, Table 18 presents sentence-level predictions on facts generated from the TriviaQA dataset using Llama 2-7b. In this case, the probe classifier is trained adopting a topic-wise leave-one-out strategy, as in Table 6. Table 19 shows additional predictions obtained using the probe classifier, which is trained on all but one topic and tested on our refined version of the held-out topic (as in the Orig. setting of Table 5).
<table>
  <tr>
    <th>Sentence</th>
    <th>Predicted</th>
    <th>GT</th>
    <th>Confidence</th>
  </tr>
  <tr>
    <td>Sir John Suckling was the first poet to be buried at Poet’s Corner in London’s Westminster Abbey.</td>
    <td>False</td>
    <td>False</td>
    <td>0.373</td>
  </tr>
  <tr>
    <td>The first poet to be buried at Poet’s Corner in London’s Westminster Abbey was Geoffrey Chaucer.</td>
    <td><span style="color:green">True</span></td>
    <td>True</td>
    <td>0.551</td>
  </tr>
  <tr>
    <td>The poet who was the first to be buried in Westminster Abbey was Geoffrey Chaucer.</td>
    <td>False</td>
    <td>True</td>
    <td>0.456</td>
  </tr>
  <tr>
    <td>Roquefort cheese is made from sheep’s milk.</td>
    <td><span style="color:green">True</span></td>
    <td>True</td>
    <td>0.794</td>
  </tr>
  <tr>
    <td>Roquefort cheese is made from cow’s milk.</td>
    <td><span style="color:green">True</span></td>
    <td>False</td>
    <td>0.851</td>
  </tr>
  <tr>
    <td>The milk of sheep is used to make ‘Roquefort Cheese’.</td>
    <td>False</td>
    <td>True</td>
    <td>0.488</td>
  </tr>
  <tr>
    <td>South Korea held its first ‘Grand Prix’ motor race in 1999.</td>
    <td>False</td>
    <td>False</td>
    <td>0.291</td>
  </tr>
  <tr>
    <td>South Korea held its first ‘Grand Prix’ motor race in 2010.</td>
    <td><span style="color:green">False</span></td>
    <td>True</td>
    <td>0.305</td>
  </tr>
  <tr>
    <td>South Korea hosted its first ‘Grand Prix’ motor race in 1966.</td>
    <td>False</td>
    <td>False</td>
    <td>0.182</td>
  </tr>
</table>

Table 18: Examples of sentence classification on facts generated from the TriviaQA dataset using Llama 2-7b. The original topic-wise leave-one-out strategy is adopted. Correct predictions are highlighted in green, while incorrect ones are shown in red.

<table>
  <tr>
    <th>Sentence</th>
    <th>Predicted</th>
    <th>GT</th>
    <th>Confidence</th>
  </tr>
  <tr>
    <td>Tin has the symbol Sn.</td>
    <td><span style="color:green">True</span></td>
    <td>True</td>
    <td>0.990</td>
  </tr>
  <tr>
    <td>Mercury is a liquid at room temperature and used in thermometers and some electrical switches.</td>
    <td>False</td>
    <td>True</td>
    <td>0.113</td>
  </tr>
  <tr>
    <td>Fluorine is in the Alkaline earth metal group.</td>
    <td>False</td>
    <td>False</td>
    <td>0.012</td>
  </tr>
  <tr>
    <td>Tellurium has the atomic number of 82.</td>
    <td><span style="color:green">True</span></td>
    <td>False</td>
    <td>0.998</td>
  </tr>
</table>

Table 19: Examples of predictions using SAPLMA and Llama 2-7b. In these examples, the probe classifier is trained on the original version of the True-False dataset (Azaria and Mitchell, 2023) (excluding the element topic) and tested on our refined version of the element topic. Correct predictions are highlighted in green, while incorrect ones are shown in red.

The code is provided in our GitHub Repository.

Reviewer paper summaries: This paper investigates whether or not LLMs encode in their inner states information regarding the truthfulness of a sentence. Probes are used against the inner representations of models, using a (pre-existing) synthetic dataset and two new datasets that should test the ability of the proposed analysis to generalize to more complex and realisticc scenarios.

The paper studies the extent that hidden states of an LLM encode information about truthfulness, by introducing a new setup to test an existing methodology. Concretely, the author(s) argue that previous work addressed this research question in a simplified setup, that did not take into account plausible false facts. To improve this, they design a more complex method to construct a dataset that allows for more insightful findings. The results  1) on the hand, confirm previous findings about hidden states encoding truthfulness information, 2) on the other hand, show that this does not generalize to less templatized data.
Summary of strengths: - The objective of the paper is challenging and interesting: the results suggesting that LLM hinner states encode truthfulness may not be fully generalizable and the paper clearly explains the limitations of a dataset previously used to detect this.
- The underlying assumption that there is a difference between the knowledge an LLM may have versus what is true outside the LLM (in the paper, distributional misalignment) is an important point to make.
- The results on the dataset generated by LLM are interesting and clarify a problem with this type of analysis, namely how the nature of the data can invalidate some results.

- The paper addresses an important topic for the NLP community, naming LLM's factual accuracy and hallucination detection.
- The paper identifies limitation in previous work and address this to test analogous questions in a more challenging and insightful setup (less obvious false statements and less templatized data). 
- The paper is overall well-written and clear.
Summary of weaknesses: - Data generated by another LLM may help to obtain more variable and realistic data, but must be evaluated by humans: GPT-4 may still suffer from hallucinations and, at least on a subset of data, the quality of the generation must be verified to make the dataset truly useful.
- The use of perplexity as a proxy for plausibility with respect to model knowledge is questionable: LLMs, for a variety of reasons, seem to obtain similar loss (and thus perplexity) for sentences that are observed during training as for sentences that are not in training [1]. This aspect, at least in theory, should be investigated. More information in the comments.
- Previous works have also discussed some limitations of the Azaria and Mitchell dataset and their effect on the probes. These works should be discussed, in relation to the proposed dataset and probes: [2] show that the original probes do not generalise to small perturbations, such as negations; [3] discuss and curate a set of new true/false statements and support the thesis of a linear direction for truthfulness; and [4] further discuss why in some cases the probes fail to generalise.

[1] Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., ... & Hajishirzi, H. [Do Membership Inference Attacks Work on Large Language Models?.](https://openreview.net/forum?id=av0D19pSkU#discussion) In First Conference on Language Modeling.

[2] Levinstein, Benjamin A., and Daniel A. Herrmann.["Still no lie detector for language models: Probing empirical and conceptual roadblocks."](https://link.springer.com/article/10.1007/s11098-023-02094-3) Philosophical Studies (2024): 1-27.

[3] Marks, S., & Tegmark, M. (2023). [The geometry of truth: Emergent linear structure in large language model representations of true/false datasets.](https://arxiv.org/abs/2310.06824) arXiv preprint arXiv:2310.06824.

[4] Bürger, L., Hamprecht, F. A., & Nadler, B. (2024). [Truth is universal: Robust detection of lies in llms. Advances in Neural Information Processing Systems](https://proceedings.neurips.cc/paper_files/paper/2024/hash/f9f54762cbb4fe4dbffdd4f792c31221-Abstract-Conference.html), 37, 138393-138431.

While I have very much appreciated the topic and ideas in this paper, I find that the paper does not focus enough on the most important parts of its contributions (exemplifying the new dataset and discussing the results), while focusing excessively on the reproducibility of previous work. More concretely:
- While it is very important that the author(s) verify the reproducibility of previous work, I believe that the level of detail at which they report this effort (Section 2 and 3; ~2 pages) is excessive for being reported in the main body of the paper, as it does not contribute novel insights. In my opinion, most of this content can go to the Appendix. This allows to get back some space that can then be used to expand other sections (see below). 
- Section 4 should include examples of the data generated (in particular highlighting differences with samples with the datasets from previous work). 
- The results (5.2 and 5.3) deserve to be discussed more at length in the paper, with potentially more analyses or visualizations allowing to better understand them. What I'm specifically missing are 1) examples of correct and incorrect predictions, 2) deeper insights trying to understand more specifically e.g., variations across question classes or layers, or the effect of non-templatized data.
Comments, suggestions and typos: Regarding the use of perplexity as a metric, the models under consideration do not provide training material, so perplexity *might be a good indicator* of plausibility, but there is no way to test it rigorously. There are LLMs that are completely open - open train data and open parameters - and although inspection of the training data is difficult due to the scale of such a dataset, it is important to at least state that this should be the investigation to be performed. The research directions that should be considered and mentioned, if not adopted, during this evaluation phase are those related to the extraction of Training Data Extractions, Membership Inference Attacks and Data Contamination (some survey for reference [5, 6, 7]).

[5] Rigaki, Maria, and Sebastian Garcia. ["A survey of privacy attacks in machine learning."](https://dl.acm.org/doi/full/10.1145/3624010) ACM Computing Surveys 56.4 (2023): 1-34

[6] Miranda, Michele, et al. ["Preserving privacy in large language models: A survey on current threats and solutions."](https://openreview.net/forum?id=Ss9MTTN7OL), TLMR.

[7] Xu, Cheng, et al. ["Benchmark data contamination of large language models: A survey."](https://arxiv.org/abs/2406.04244) arXiv preprint arXiv:2406.04244 (2024).

Given what noted in the Weaknesses section, I recommend the author(s) to revise the content and structure of the paper, and zoom in on the results, providing more in-depth discussion. Adding examples from the data in the text and referring to them in the discussion would be very helpful to highlight the contributions and help the reader appreciate them.
Overall assessment scores: 4

3
Metareview score: 3.5
Metareview: The authors study whether the activations of an LLM encode information about the factuality/truthfulness of its output generations. They rerun experiments from previous studies that explored a similar question. They claim that previous studies used an oversimplified approach with unnatural prompts and instead propose a new experiment that better controls for plausible false facts. They confirm that latent activations do indeed store information about truthfulness but this result does not generalize when the data is less templated.
Summary of reasons to publish: The paper identifies a central limitation in earlier work and proposes a new method to address the limitation and replicate the study in a better-controlled setting. Studying the question of how language models reason about truthfulness is important, as it may help us to better understand hallucination, and possibly how to mitigate it. The paper is well-written and easy to understand.