Transferable and Efficient Non-Factual Content Detection via Probe Training with Offline Consistency Checking (2024)

Abstract: Detecting non-factual content is a longstanding goal to increase the trustworthiness of large language models (LLMs) generations. Current factuality probes, trained using humanannotated labels, exhibit limited transferability to out-of-distribution content, while online selfconsistency checking imposes extensive computation burden due to the necessity of generating multiple outputs. This paper proposes PINOSE, which trains a probing model on offline self-consistency checking results, thereby circumventing the need for human-annotated data and achieving transferability across diverse data distributions. As the consistency check process is offline, PINOSE reduces the computational burden of generating multiple responses by online consistency verification. Additionally, it examines various aspects of internal states prior to response decoding, contributing to more effective detection of factual inaccuracies. Experiment results on both factuality detection and question answering benchmarks show that PINOSE achieves surpassing results than existing factuality detection methods. Our code and datasets are publicly available on this github repository.

heading: 1 Introduction
text: Large language models (LLMs), after pre-training on massive corpora (Brown et al., 2020; Touvron et al., 2023a; Jiang et al., 2023), show a surprising ability to generate knowledgeable content (Sun et al., 2023; Yu et al., 2022). Although this ability facilitates a wide range of applications, such as question answering (QA) (Abdallah et al., 2023; Liu et al., 2022; Li et al., 2022) and information retrieval (Mao et al., 2021; Ma et al., 2023), the propensity of LLMs to occasionally produce nonfactual knowledge (Lin et al., 2022; Wang et al.,
†Equal Contribution. ‡Work was done when interned at Zhipu AI. *Corresponding author.
2023a) potentially hinders the practical utilization of generated content. Thus, it is necessary to detect whether LLMs generate non-factual content.
Previous studies offer evidence that the internal representation vectors in LLMs determine whether they produce factual answers to the input question (Azaria and Mitchell, 2023a; Kadavath et al., 2022; Zou et al., 2023). Specifically, the factual behavior entailed is extracted from the feed-forward layer activations of tokens before the generated content using linear probes (Alain and Bengio, 2017; Belinkov, 2022). However, their construction relies on the labor-intensive process of annotating natural language questions, as well as labeling LLMs’ outputs with factuality annotations, a factor that limits their applicability to questions and responses with unseen distributions.
To avoid the annotation process, the most recent studies detect non-factual content via online selfconsistency checking (Wang et al., 2022). They assume that if LLMs give contradictory responses to the same prompt, the model is more likely to hallucinate to give that answer (Elazar et al., 2021). In this way, detecting non-factual content is reduced to the mutual-entailment analysis among multiple generations, which is usually realized as natural language inference (NLI) models (Kuhn et al., 2023; Manakul et al., 2023a) or heuristic comparison of the hidden representation similarity (Anonymous, 2023). However, self-consistency checking introduces extensive computation overhead to sample multiple responses. In addition, due to the lack of training process, these methods are less robust than previous factuality probes.
Giving these limitations of existing methods, we propose PINOSE, a method to predict non-factual responses from LLMs. The main idea of PINOSE is to construct a probing model that learns from offline self-consistency checking. It aims to present two core advantages over existing methods:
Transferability. Comparing with existing prob-
12348
ing methods, PINOSE eliminates human annotation for training data. This is achieved with bootstrapped natural language questions and generated pseudo factuality labels through an offline consistency checking mechanism. Moreover, as PINOSE does not rely on specific training data, it transfers effortlessly to any different data distributions.
Efficiency and Effectiveness. Comparing with online consistency checking, PINOSE avoids the computational burden associated with multiple generations during inference, thus enhancing time efficiency. Additionally, by analyzing the continuous internal representations of LLMs rather than discrete tokens in the response, PINOSE gains access to a broader spectrum of information, enhancing its prediction effectiveness.
We conduct comprehensive experiments on established factuality detection benchmarks and variations of QA datasets. Our results reveal several key findings: (1) PINOSE outperforms supervised probing-based baselines by 7.7-14.6 AUC across QA datasets, despite being trained without annotated labels. (2) Moreover, our PINOSE achieves significant performance improvements (3-7 AUC) compared to unsupervised consistency checking baselines, while also demonstrating superior time efficiency. (3) Additionally, the dataset generated via offline self-consistency checking shows promise for transferring to probe various LLMs.

heading: 2 Preliminaries
text: This study concentrates on identifying non-factual content using decoder-only LLMs. It begins by formally defining the task and then elaborates on decoder-only LLMs and the construction of probes for these models. Additionally, it discusses the distinctions between online and offline selfconsistency checking.
Task Definition. Formally, given a question q = ⟨q1, q2, . . . , qn⟩ composed of n tokens, and its corresponding response r = ⟨r1, r2, . . . , rm⟩ consisting of m tokens, non-factual content detection aims to assign a binary judgment f ∈ {True, False}, determining the factual correctness of r. For example, given the question “What is the capital city of China?”, “Beijing” serves as the response with True judgement, while “Shanghai” is classified as False. Without losing generality, we allow q to be null (i.e., q = ∅), in which scenario, the task evaluates the factuality of the standalone assertion.
To detect whether LLMs generate non-factual
content, in our setting, the response r is usually sampled from an LLM. In this case, we expect the model to assess the factuality of content generated by itself without the need of another LLM.
Decoder-only LLMs. Decoder-only LLMs comprise a stack of Transformer decoder layers (Vaswani et al., 2017). After embedding tokens into the hidden representation H(0), each layer manipulates the hidden representation of the previous layer as follows1:
H(l) = FFN ( Attn ( H(l−1) )) ,
where Attn(·) is the attention mechanism. FFN(·) is the feed-forward network composed of two consecutive affine transformations and activation functions. Thus, the intermediate representations of decoder-only LLMs are usually extracted from the output of FFN(·) operation. In the following of this paper, we denote the hidden representation of the ith token extracted from layer l as H(l)[i].
Language Model Probing. Probing method extracts implicit information from the intermediate representation. It is usually implemented as a simple classification model that maps from the hidden representation of certain token into discrete classes.
The probe in PINOSE is a two-layer feedforward network with binary classification outputs:
Probe ( H(l)[i] ) = σ2 ( W2σ1 ( W1H (l)[i] + b1 ) + b2 ) ,
(1)
where σ1 is the Sigmoid function, σ2 is any nonlinear function, and W,b are trainable parameters. Probe ( H(l)[i] ) is the probability for True.
Consistency Checking. Consistency checking requires LLMs to generate multiple responses towards the same question, and utilize these semantic consistency to judge whether the generations are correct. Previous methods for non-factual content detection are online self-consistency checking, where LLMs need to generate extensive responses to answer a single question to obtain factuality labels. Our method falls into the offline consistency checking category, where consistency checking is solely used to generate labels for training the probe. During online checking, LLMs only need to produce a single response and obtain the factuality label from the probe.
1We omit residual connection (He et al., 2016) and layer normalization (Ba et al., 2016) for simplicity.

heading: 3 Methodology
text: The construction of PINOSE involves three main stages: (1) In the data preparation stage, we bootstrap natural language questions and generate multiple responses, which together serve as model inputs; (2) In the offline consistency checking stage, we employ a peer review mechanism to generate pseudo factuality label for each response; (3) In the probe construction stage, these pseudo factuality labels are used to train a language model probe. Figure 1 illustrates the overall process.

heading: 3.1 Stage 1: Data Preparation
text: In alignment with the requirements of the nonfactual content detection task, the supporting data consists of three elements: natural language questions q, their corresponding responses r, and factuality labels f . This stage concentrates on generating large-scale data containing the initial two elements (i.e., q and r).
Question Bootstrapping. PINOSE leverages natural language questions to prompt LLMs to generate responses for consistency checking. However, natural language questions are not always available across all domains. Furthermore, both the diversity and quantity of questions can significantly impact the quality of the prepared data. Therefore, we aim to enable LLMs to bootstrap questions with minimal human involvement.
Fortunately, as Honovich et al. (2023) point out, high-performing language models show significant capacity in question generation. Inspired by these findings, we manually annotate a set of seed questions and employ them as demonstrations for LLMs to generate a large volume of questions via incontext learning (Brown et al., 2020, ICL). To enhance diversity in generation, we broaden the scope of seed questions by incorporating the generated ones and sample diverse combinations from the seed questions for subsequent generation. Detailed prompt for question generation is provided in Figure 4 of Appendix A.5.
Diverse Response Generation. We use previously generated questions as input for LLMs to generate multiple responses for subsequent consistency checking. We design two strategies to encourage the diversity of multiple responses to the same input question. (1) From the perspective of decoding, we adjust the decoding strategy by applying a vanilla sampling method with a relatively high sampling temperature (t = 1). (2) From the
perspective of model input, we instruct LLMs to answer a question using a variety of prompts (as shown in Figure 5 in Appendix A.5).
The outcome of this stage is a dataset containing questions paired with multiple responses, designated as {(q, {ri})}, where the number of responses k = |{ri}| serves as a hyperparameter that determines the quantity of responses per question for the subsequent consistency check.

heading: 3.2 Stage 2: Offline Consistency Checking
text: We engage LLMs in the offline consistency check process via a peer review mechanism. First, we gather reviews by asking LLMs to determine for each response whether it is consistent with other responses. Then, we enrich reviews by sampling multiple consistency judgements by varying model inputs. Finally, we integrate reviews to form the pseudo-factuality label for each response and filter out low-quality responses.
Review Gathering. Formally, consistency reviewing involves asking an LLM to evaluate whether the response ri to the question q is semantically consistent with other responses (i.e., rj , j ̸= i). If ri has equivalent meaning with other responses, it is considered factual. To ensure unambiguous judgment, we require the LLM to make pairwise comparisons with other k − 1 responses. For each comparison, it must output one of three labels: “Consistent”, “Neutral”, or “Non-Consistent”. To achieve this, we specify the output format with in-context demonstrations and prompt instructions (as shown in Figure 6 in Appendix A.5).
Review Enrichment. To enhance the diversity of reviews, we introduce variability in the input provided to the LLM during consistency assessments. Recognizing the significant impact of demonstrations on LLM judgments in ICL (Wang et al., 2023b), we utilize a range of diverse demonstration combinations for ICL to elicit varied reviews from the LLM for each pairwise comparison. Diverse demonstrations facilitate the collection of multiple reviews, each potentially providing a unique perspective. In total, we gather N round of reviews for each pairwise comparison, where N is a hyperparameter.
Integration and Filtering. We integrate N reviews for each pairwise comparison, and subsequently integrate k − 1 pairwise comparisons for each response through the same majority voting mechanism. Here is how the voting works: we first
consider Neutral consistency judgement as an abstention for voting. Then, to guarantee the quality of the final dataset, we exclude controversial judgements where no single label (Consistent, Neutral, Non-Consistent) receives over 50% of the votes. This step ensures that only the most widely agreed-upon judgements are retained for analysis. Finally, we assign the factuality label True (False) to responses that are predominantly considered consistent (non-consistent) with others.
This stage outputs the dataset with full elements for consistency checking, i.e., {(q, {(ri, fi)})}.

heading: 3.3 Stage 3: Probe Construction
text: PINOSE predicts the factuality of responses via a probing model, as defined in Equation 1. To be more specific, PINOSE integrates the response with the question, both formatted according to the template outlined in Figure 7 in Appendix A.5, into the LLM for detection. Subsequently, the hidden representation of the last token in the response at the middle layer of the LLM is employed as the input for the probing model. We train the probing model to maximize the probability of the factuality label while freezing all the parameters of the LLM. Formally, the construction process of the probe optimizes the following cross-entropy loss:
loss = − ∑
{q,rj ,fj} log Probe
( H(l)[i] ) 1 ( fj = True ) ,
where 1(·) is the indicator function, i is the index of the last input token, and l represents half the layer number of the LLM to be detected.

heading: 3.4 Discussion
text: We discuss the rationality of PINOSE and the involvement of LLMs for implementing PINOSE.
The reason of why PINOSE successfully detect non-factual responses comes from the model and data perspective. (1) Model property. LLMs are well-calibrated after massive pre-training (Kadavath et al., 2022; Zhu et al., 2023). This indicates that for non-factual responses, LLMs tend to assign less probability, while preserve relatively high probability for factual generations. This calibration property guarantees the feasibility of distilling factuality detection dataset from offline consistency checking. It also suggests that the internal states of LLMs tracks whether they are producing factual contents, which PINOSE tries to uncover with probing model. (2) Data quality. Offline consistency checking gathers diverse instances of inconsistency between responses from LLMs, potentially enhancing the quality of training data for the probing model. Consequently, it enables the model to address a broader range of inconsistency scenarios compared to online consistency checking. Moreover, as the data collection process is fully automated, the dataset can be significantly larger than existing training data for factuality probes. The feasibility of this principle is also widely verified in distant supervision (Quirk and Poon, 2017).
To implement PINOSE, LLMs are multiply invoked during the construction process, including data preparation, peer reviewing in consistency checking, and finally non-factual detection. For a coherent implementation, we employ the same
LLM for detecting the factuality of responses as the one used for generation and checking consistency. This implementation strategy aligns with our setting, where no third-party LLM is available, and it also enhances the transferability of our method.

heading: 4 Experiment
text: We conduct experiments to examine the performance of PINOSE by comparing it to baseline methods for factuality detection. Additionally, we assess its transferability and efficiency.

heading: 4.1 Experiment Setup

heading: 4.1.1 Datasets
text: The datasets include both factuality detection benchmark and variations of QA datasets. We introduce the purpose to incorporate each dataset and their data specifications. Detailed statistics are shown in Table 1.
Benchmark. We follow previous research to use True-False benchmark (Azaria and Mitchell, 2023b). True-False provides statements generated by LLMs along with corresponding factuality labels examined by humans. It does not include questions for each statement (i.e., q = ∅). True-False comes with both training dataset and test dataset.
Variation of QA. Given that probing statements, such as those in True-False dataset, is less practical compared to examining responses to questions from LLMs, for practical evaluation, we establish test sets based on existing QA datasets: Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and WebQ (Berant et al., 2013). In particular, we sample 1, 000 questions from each dataset and employ Llama2-7B (Touvron et al., 2023b) to generate responses accordingly. The factuality label for each response is annotated by human annotators through comparing with the original ground-truth of each question.

heading: 4.1.2 Baselines
text: We compare PINOSE against probing-based and consistency-checking-based methods. Additionally, to ensure a comprehensive comparison, we implement heuristic confidence-based methods as baselines.
• Probing Based: SAPLMA (Azaria and Mitchell, 2023a) utilizes a feed-forward neural network for factuality detection, trained on the True-False training data. RepE (Zou et al., 2023) conduct principal component analysis (PCA) on the internal representations of True-False training data. It selects a factual direction vector using the factuality labels. During testing, RepE compute the dot product between the internal representation of the given response and the factual direction vector. • Consistency Checking Based: We compare against SelfCheckGPT (Manakul et al., 2023b), which performs factuality detection via online self-consistency checking. We implement two variants. SelfCheckGPT-NLI (SCGPTNLI) uses a BERT-based NLI model (Williams et al., 2018) for consistency checking, while SelfCheckGPT-Prompt (SCGPT-PT) elicits LLM itself to evaluate the consistency between two responses via prompt. The prompt used for SCGPT-PT is shown in Figure 8 in the Appendix. • Confidence Based: We also utilize model confidence as an indicator for factuality detection. Perplexity-AVE (PPL-AVE) and PerplexityMax (PPL-MAX) (Kadavath et al., 2022; Azaria and Mitchell, 2023a; Zou et al., 2023) quantify the average and maximum token-level probabilities of statements within each test set generated by the evaluated LLM. It-is-True (Azaria and Mitchell, 2023a) compares the probabilities between sentences “It is true that q||r.” and “It is false that q||r.”, where || denotes concatenation.
It is worth noting that probing based baselines rely on training data. We thus implement them using the training dataset from True-False. Besides, as SCGPT-PT and SCGPT-NLI needs input questions to generate multiple responses, it is infeasible to test on True-False, where we mark their results as “N/A” in Table 2.

heading: 4.1.3 Evaluation Metrics
text: We follow conventions (Azaria and Mitchell, 2023b) in factuality detection, employing the area
under the receiver operating characteristic curve (AUC) and accuracy (ACC) as evaluation metrics.

heading: 4.1.4 Implementation Details
text: To implement PINOSE, we uniformly use Llama27B for data preparation, consistency checking, and factuality detection. For hyperparameters, we set the number of sampled responses N to 9 and the round of peer review k to 7. For fair comparison, we also allow SelfCheckGPT to generate N = 9 responses for consistency checking. The training dataset for PINOSE consists of 20, 000 constructed triplets {(q, ri, fi)}. The threshold for calculating accuracy is determined by selecting the value that yields the highest accuracy among 100 validation instances partitioned from the test sets.

heading: 4.2 Main Results
text: Table 2 presents the AUC and ACC scores for all the compared methods across four test sets. Meanwhile, Table 3 provides insights into the average detection time required by SCGPT-NLI and our PINOSE for each instance. In general, PINOSE outperforms probing-based methods across all QA variation dataset, despite being trained without annotated labels. Additionally, PINOSE exhibits superior performance compared to consistency checking methods and is also more efficient. Some detailed findings include:
Leveraging factuality labels substantially improves factuality detection accuracy. This is evidenced by PINOSE ’s superior performance trained on factuality labels, compared to confidence-based methods such as PPL and It-is-True.
Limitations of annotated labels on model transferability. Despite being trained using annotated labels, probing-based methods like RepE and SAPLMA lag behind a large margin compared to PINOSE on the three QA variation test sets.
This disparity arises because the two baselines are trained on True-False’s training data, which consists of statements rather than question and responses. This difference in input distribution significantly limits the transferability of these models to out-of-distribution datasets. In contrast, PINOSE is trained on a diverse range of questions, leading to superior performance across the QA datasets. The slight lag observed in True-False from SAPLMA is also attributed to the model’s training on questions. However, probing the factuality of responses to questions is more practical than evaluating statements given by True-False. Therefore, training on datasets guided questions is reasonable.
Self-consistency correlates well with factuality. Despite SCGPT lacking supervision from factuality labels, it surpasses supervised probingbased baselines, suggesting a strong correlation between its self-consistency principle and factuality. Furthermore, PINOSE, also adhering to the selfconsistency principle, outperforms SCGPT. This is due to PINOSE being exposed to numerous instances with diverse inconsistencies between responses, unlike SCGPT, which focuses solely on responses related to the given question. Moreover, PINOSE evaluates the consistency of internal representations rather than discrete output responses like SCGPT, allowing it access to a wider range of information, thereby enhancing its predictive accuracy.
PINOSE’s detection time is significantly shorter than SCGPT, as shown in Table 3. This is because PINOSE relies on offline consistency checking, incorporating consistency characteristics into internal representations during training. As a result, its online inference depends solely on internal representations, eliminating the need for multiple online inferences like those performed by SCGPT.

heading: 4.3 Cross-model Evaluation
text: To implement PINOSE, LLMs are invoked multiple times during the construction process, including data preparation, peer reviewing in consistency
checking, and finally non-factual detection. By default, we employ the same LLM for all stages, leveraging an LLM’s calibration property. Additionally, we explore whether training data generated by one LLM can effectively train a probe to detect the factuality of content generated by other LLMs. To study this, we use Llama2-7B for data preparation but vary the detection target to Llama2-13B and Mistral-7B. We further switch the LLM for consistency checking to Llama2-13B and Mistral-7B, consistent with the model to be detected. It’s important to note that for a given LLM to be detected, the probe needs to be aligned with it, specifically in terms of probe’s input, which consists of the internal representation of the response along with the question, that must be generated by the LLM. The crucial findings, as presented in Table 4, include:
More powerful LLMs brings better detection performance. Comparing group 2 and 3 with group 1, where the training data remains consistent (created by Llama2-7B), probes built based on more powerful LLMs demonstrate higher performance, attributed to the enhanced representational capacity of these models.
Generated data facilitates probing across various LLMs. Switching the LLM for consistency checking to match the LLM being detected results in comparable performances between groups 4 and 2, as well as between groups 5 and 3, respectively. This indicates that we can generate the training data, comprising (question, response, factuality label) triplets, once, regardless of the LLMs being probed, and utilize them uniformly to train probes for any LLM.

heading: 4.4 Ablation Studies on Data Preparation
text: We examine the impact of question distribution in the data preparation stage with two variants:
• PINOSE with self questions: Assumes a sce-
nario where the training dataset consists of questions from the same distribution as the test questions. We utilize questions from the training data that belong to the same dataset as the test set. • PINOSE with external questions: Considers a scenario where questions from the same distribution as the test questions are unavailable. We utilize questions from the training data that belong to a different dataset from the test set.
We evaluate these variants across three QA test sets, maintaining a consistent number of training questions (1, 000 per set) for the first variant. For the second variant, we collect 5, 000 training questions from the remaining two datasets for each target test set. Additionally, we vary the number of generated questions within the range of [1K, 2K, 3K, 4K, 5K, 10K] to assess its impact. We maintain fairness in our evaluation approach by applying an identical labeling methodology, specifically offline consistency checking, to each dataset.
Figure 2(a)(b)(c) presents the performance of these variants on the three QA test sets. We find:
Training on questions of the same distribution as the test set yields significantly better results than a different distribution. Despite having more external questions (5, 000 vs 1, 000), the second variant still lags behind the first.
Training on generated questions could enhance transferability of the probe. Across the three test sets, training with fewer than 5, 000 generated questions (approximately 3, 000+, 2, 000+, 1, 000+ questions on NQ, TriviaQA, and WebQA, respectively) can achieve performance comparable to using 5, 000 external questions. Additionally, training with approximately 6, 000, 10, 000, and 3, 000 generated questions on these three datasets respectively could outperform training using 1, 000 self-questions. These results indicate that generated questions offer greater diversity, facilitating the transferability of the probe across different test sets, despite the diverse distributions observed among the three test sets.

heading: 4.5 Ablation Studies on Consistency Checking
text: We investigate the impact of two hyperparameters, k (the number of responses) and N (the number of review rounds per response), in the consistency checking stage.
Figure 2(d) displays the detection performance on NQ with varying values of k from 1 to 9 with interval 2 and different values of N (1, 3, 5, 7).
It’s worth noting that N = 1 corresponds to the review strategy in SCGPT. Remarkably, the best performance is achieved with N = 7, indicating that multiple inferences from an LLM, each guided by different demonstrations acting as instructions, contribute to more robust and confident review outcomes akin to opinions from multiple reviewers. The figure also illustrates that the performance exhibits a smooth increase as more responses are used, also suggesting that multiple responses could result in more confident consistency checking.

heading: 4.6 Ablation Studies on Probe Construction
text: We investigate feature selection at the probe construction stage, exploring the use of internal representations from the last (32nd), middle(16th), and first layers of Llama2-7B. Additionally, we experiment with averaging representations of all tokens within a layer or using only the last token. The default configuration includes the middle-layer representation and the last token in a layer. The results, as depicted in Table 5, indicate that the middlelayer representation and the last token are optimal choices within our setting.

heading: 5 Related work
text: Factuality detection for LLM generated content mainly falls into two categories: consistency-based and probing-based.
Consistency-based methods detect non-factual generations by comparing model generated content with other information. Among these methods, the most widely adopted assumption is that, LLMs usually fail to give consistent responses to the same prompt when generating multiple times (Elazar et al., 2021; Mündler et al., 2023; Pacchiardi et al., 2023), thus motivating a series method to detect non-factual content (Manakul et al., 2023b; Cohen et al., 2023; Azaria and Mitchell, 2023a) or reduce non-factual generations (Dhuliawala et al., 2023; Kuhn et al., 2023). Another thread of works evaluate model self-consistency via model’s confidence to the generated content (Kadavath et al., 2022; Azaria and Mitchell, 2023a; Zou et al., 2023). Except for self-consistency checking, there are also attempts that use consistency between model generated content and external information as factuality indicator (Wang et al., 2023c; Gao et al., 2023; Chern et al., 2023).
Probing-based methods possesses the belief that the hidden representation entails certain property of generated content and can be extracted via a light weight model (Alain and Bengio, 2017; Gurnee and Tegmark, 2023). Probing whether LLMs are producing factual content is proved to be feasible (Kadavath et al., 2022), thus motivating researchers to develop more accurate probes (Azaria and Mitchell, 2023a; Zou et al., 2023; Chen et al., 2023). Comparing with these works, which rely on annotated training data, PINOSE provides a method that dis-
till consistency patterns from LLMs into a probe.

heading: 6 Conclusion
text: This paper presents PINOSE, a probing method for non-factual content detection that learns from offline consistency checking. PINOSE achieves good transferability among different distributed datasets as its does not rely on manually annotated data. It also avoids the computational burden for online consistency checking. In the future, PINOSE potentially paves way to build more faithful LLMs.

heading: Limitations
text: The limitations of this work are as follows: (1) Data Preparation: PINOSE employs an offline consistency checking method to automatically generate factuality labels for training a probe model. Although the probe model efficiently infers the factuality label of a response from an LLM in a single pass, the offline data preparation stage requires a large amount of data. This involves multiple inferences of the LLM for generating questions, responses, and reviews, resulting in high offline construction costs. Fortunately, as online usage increases, the amortized cost of offline construction decreases. (2) Open-sourced LLMs: PINOSE is limited to detecting factuality errors in opensourced LLMs because it requires the internal representation of the input response along with the question as features input to the probing model for detection. (3) Factuality Error: PINOSE is constrained to detecting the factuality errosr in responses to questions or statements. Other aspects of errors, such as logical error, require further investigation.

heading: Ethical Considerations
text: We discuss the ethical considerations and broader impact of this work in this section: (1) Intellectual Property: The datasets employed in this study, comprising True-False, NQ, TriviaQA, and WebQ, are widely accessible and established resources designed to facilitate extensive research in artificial intelligence and natural language processing (NLP). We are confident that these resources have been adequately de-identified and anonymized. (2) Data Annotation: We recruit 10 annotators from commercial data production companies to label factual accuracy for three QA test sets, seed questions for question generation, and consistent/inconsistent response pairs for consistency checking. Annotators
are compensated fairly based on agreed-upon working hours and rates. Prior to annotation, annotators are briefed on the data’s processing and usage, which is formalized in the data production contract. (3) Intended Use: The proposed PINOSE is utilized for detecting non-factual content generated by LLMs. (4) Misuse Risks: There is a risk that PINOSE could be exploited for adversarial learning, potentially enabling LLMs to generate more implicit non-factual content that is more challenging to detect. (5) Potential Risk Control: The trained PINOSE is made publicly available to the open-source community, which may help mitigate the risks associated with its potential misuse for adversarial learning.

heading: Acknowledgments
text: This work is supported by the National Key Research & Develop Plan (2023YFF0725100), the National Natural Science Foundation of China (62322214), Zhipu Project Fund, and a grant from the Institute for Guo Qiang, Tsinghua University (2019GQB0003).

heading: A Appendix
text: A.1 Perplexity-based Baseline . . . . . 12 A.2 Cross-model Evaluation on TriviaQA and WebQ . . . . . . . . . . 12 A.3 Evaluation of Different Layers . . 13 A.4 Training Dataset Samples . . . . . 13 A.5 Employed Prompts . . . . . . . . 13

heading: A.1 Perplexity-based Baseline
text: The perplexity-based baseline is formulated as:
PPL(AVE) = − 1 J
∑
j
log pij , (2)
PPL(MAX) = max j (− log pij), (3)
where i denotes the i-th statement, and j denotes the j-th token in the i-th statement. J is the number of tokens in the i-th statement. pij denotes the probability of the j-th token in the i-th statement generated by the LLM. PPL(AVE) measures the average likelihood of all tokens, while PPL(MAX) measures the likelihood of the least likely token in the statement.

heading: A.2 Cross-model Evaluation on TriviaQA and WebQ
text: We present the cross-model evaluation results on TriviaQA in Table 6 and on WebQ in Table 7 to explore whether training data generated by one LLM
can effectively train a probe to detect the factuality of other LLMs. The settings are the same as those presented for NQ in Table 4. In addition to the default setting where we employ the same LLM for all stages of data preparation, consistency checking, and factuality detection, we also investigate two other settings. The first setting involves using Llama2-7B for data preparation but varying the detection target to Llama2-13B and Mistral-7B. The second setting involves further switching the LLM for consistency checking to Llama2-13B and Mistral-7B, consistent with the LLM to be detected.
Tables 6 and 7 present observations: for the first setting, when the detection target is changed to more powerful LLMs, the detection performance increases, indicating that more powerful LLMs can improve detection performance. Additionally, for the second setting, when changing the LLM for
consistency checking to the same more powerful LLMs, the detection performance remains almost unchanged. This suggests that we can generate training data for questions, responses, and reviews once, regardless of the LLMs being probed, and uniformly employ them to train probes for any LLMs.

heading: A.3 Evaluation of Different Layers
text: We vary the internal representations obtained per layer from the 1st to the last layer (32nd) of Llama27B, use them to construct probes respectively, and show the evaluated AUC on the three QA test sets in Figure 3. The results demonstrate that the detection performance of PINOSE generally increases and then decreases with the increase in the number of layers. Typically, the best performance is achieved at the middle layer.

heading: A.4 Training Dataset Samples
text: As shown in Table 8, the extensive internal knowledge of Large Language Models (LLMs) enables them to generate datasets encompassing a broad array of topics. This diversity enhances the effectiveness of training downstream detectors.

heading: A.5 Employed Prompts
text: We introduce the prompts used at different steps of our proposed PINOSE, and also other baseline methods that need prompts.
Prompt for Question Generation in PINOSE is shown in Figure 4, where the seed questions are randomly sampled from an initial set of questions annotated by humans, which is then expanded by the newly generated questions.

heading: Prompt for Response Generation in PINOSE is
text: shown in Figure 5, where five instructions for generating responses are presented. We randomly select an instruction from this set each time to encourage diverse response generation.

heading: Prompt for Consistency Checking in PINOSE
text: is depicted in Figure 6. It instructs LLMs to determine whether two responses are “Consistent”, “Neutral”, or “Non-Consistent” given a question. For each judgment, three demonstrations are randomly selected from a set of 16 consistency judgment pairs. These diverse demonstrations facilitate the collection of multiple judgments (reviews) for each comparison between two responses, potentially offering unique perspectives.

heading: Prompt for Probe Construction in PINOSE is
text: depicted in Figure 7, where a question and the answer generated by the LLM under detection are organized as input for the LLM to obtain their internal representation. This representation then serves as input for the probe to predict its factual label.
Prompt for SelfCheckGPT-Prompt is displayed in Figure 8, identical to the one presented in (Manakul et al., 2023b). SelfCheckGPT-Prompt facilitates consistency checking by presenting a sentence and its context to LLM, enabling it to judge whether the context adequately supports the sentence.

heading: Prompt for Question Generation in PINOSE.

heading: Prompt for Consistency Checking in PINOSE.

heading: Prompt for SelfCheckGPT with Prompt.

heading: Future Work
text: In the future, PINOSE potentially paves way to build more faithful LLMs