Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation (2024)

Abstract: Despite showing impressive abilities, large language models (LLMs) often struggle with factual inaccuracies, i.e., “hallucinations”, even when they hold relevant knowledge. To mitigate these hallucinations, current approaches typically necessitate high-quality human factuality annotations. In this work, we explore Self-Alignment for Factuality, where we leverage the self-evaluation capability of an LLM to provide training signals that steer the model towards factuality. Specifically, we incorporate SELF-EVAL, a self-evaluation component, to prompt an LLM to validate the factuality of its own generated responses solely based on its internal knowledge. Additionally, we design S ̄ elfK ̄ nowledge Tuning (SK-TUNING) to augment the LLM’s self-evaluation ability by improving the model’s confidence estimation and calibration. We then utilize these self-annotated responses to fine-tune the model via Direct Preference Optimization algorithm. We show that the proposed self-alignment approach substantially enhances factual accuracy over LLAMA family models across three key knowledge-intensive tasks on TruthfulQA and BioGEN.1

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1965 August 11-16, 2024 ©2024 Association for Computational Linguistics
¯ elf-
K ¯ nowledge Tuning (SK-TUNING) to augment the LLM’s self-evaluation ability by improving the model’s confidence estimation and calibration. We then utilize these self-annotated responses to fine-tune the model via Direct Preference Optimization algorithm. We show that the proposed self-alignment approach substantially enhances factual accuracy over LLAMA family models across three key knowledge-intensive tasks on TruthfulQA and BioGEN.1

heading: 1 Introduction
text: Despite exhibiting remarkable proficiency in a diverse range of NLP tasks (Wei et al., 2022; Liu et al., 2023c; Chang et al., 2023; Zhang et al., 2023a), LLMs (OpenAI, 2022, 2023; Touvron et al., 2023b) occasionally generate seemingly plausible yet factually incorrect statements, i.e., “hallucinations” (Huang et al., 2023; Ji et al., 2023; Zhang et al., 2023c; Tonmoy et al., 2024). Such hallucinations can undermine the trustworthiness and practical applicability of LLMs in real-world scenarios, particularly when employed in high-stakes tasks (Liu et al., 2023b).
∗Work done during the internship at Tencent AI Lab. 1Our code is publicly available at https://github.com/
zhangxy-2019/Self-Alignment-for-Factuality.
In this paper, we focus on mitigating a noteworthy type of hallucination, where an LLM holds relevant knowledge in response to a query (i.e., “knows”), yet occasionally falters in conveying accurate information (i.e., “tells”) (Li et al., 2023b, 2024). For instance, an LLM might generate an inaccurate response during one inference time but can provide a correct response at another time (Wang et al., 2023a; Manakul et al., 2023; Dhuliawala et al., 2023). This gap between “knowing” and “telling” (Saunders et al., 2022; Kadavath et al., 2022; Chen et al., 2023a) significantly undermines the potential of LLMs to accurately convey the knowledge acquired during the pre-training phase.
A few studies (Li et al., 2023b; Chuang et al., 2023; Zhang et al., 2023b) edit the model’s internal
1946
representations towards “factuality” directions, using domain-specific annotated data. Meanwhile, acknowledging the inadequacy of the training objective—maximum likelihood estimation (MLE)—in accurately capturing factuality (Ouyang et al., 2022; Allen-Zhu and Li, 2023; Azaria and Mitchell, 2023; Tian et al., 2023a), a recent study (Tian et al., 2023a) introduces the LLM’s internal factuality signals as training rewards to guide the models towards factuality. Given that the origin of a LLM’s hallucinations is intrinsically linked to its confidence2 (Huang et al., 2023), Tian et al. (2023a) employs consistency-based confidence regarding the factual correctness over the generate responses (Kuhn et al., 2023; Manakul et al., 2023) as the factuality signals. Nevertheless, such consistencybased confidence remains rely on the model’s generation ability, which might be non-reflective on model’s internal knowledge. Despite the challenges faced by an LLM in directly “telling” the correct response, it has showed potential in “evaluating” its generated responses (Kadavath et al., 2022; Saunders et al., 2022). As depicted in Figure 1, the LLM is capable of identifying factual inaccuracies within the responses it generates, with a reasonable prediction confidence. Such self-evaluation, i.e., directly prompting the model itself about internal knowledge awareness, might be a more effective approach to factuality estimation.
In this paper, we introduce a self-alignment framework, Self-Alignment for Factuality, which harnesses an LLM’s self-evaluation capability to mitigate hallucinations. Our approach encourages an LLM to generate prediction confidence scores pertaining to the factuality of its own generated responses through self-asking. Subsequently, these scores are utilized as reward signals to fine-tune the model using the Direct Preference Optimization (DPO) algorithm (Rafailov et al., 2023). Specifically, we incorporate a factuality self-evaluation component, SELF-EVAL, which prompts the LLM to directly validate its responses based on its internal knowledge. To bolster the LLM’s universal selfevaluation ability, we introduce SK-TUNING to enhance the LLM’s internal knowledge awareness, i.e., prediction confidence estimation and calibration3 (Guo et al., 2017; Tian et al., 2023b), through sufficient tuning across heterogeneous knowledge-
2A lower confidence score corresponds to a greater likelihood of hallucinated facts.
3The confidence in a prediction is expected to accurately reflect the probability that the prediction is correct.
oriented tasks. We assess the effectiveness of the proposed SelfAlignment for Factuality framework on three crucial knowledge-extensive tasks for LLMs, namely Multi-Choice Question-Answering (MCQA), shortform open-ended generation, and long-form openended generation, using two benchmark datasets: TruthfulQA (Lin et al., 2022) and BioGEN (Min et al., 2023a). The results show that, solely relying on the model’s internal knowledge, SelfAlignment for Factuality significantly enhances the factual accuracy of LLAMA family models (Touvron et al., 2023a,b) across all three tasks, notably surpassing the representation-editing methods (Chuang et al., 2023; Li et al., 2023c) and the recent work with consistency-based confidence (Tian et al., 2023a).
In summary, our contributions are three-fold:
• We propose Self-Alignment for Factuality, a self-alignment strategy that leverages an LLM’s self-evaluation capability to mitigate the model’s hallucinations.
• We introduce SK-TUNING to improve an LLM’s confidence estimation and calibration, thereby enhancing its self-evaluation ability.
• We show the efficacy of Self-Alignment for Factuality on three crucial tasks using TruthfulQA and BioGEN, significantly improving factual precision over all compared methods.

heading: 2 Related work
text: Hallucinations in LLMs. Hallucinations in LLMs occur when generated content, is seemingly plausible, however deviates from actual world knowledge (Chen et al., 2023b; Li et al., 2023a; Zhang et al., 2023c; Tonmoy et al., 2024). In this study, we align with the perspective that an LLM’s acquired knowledge should mirror established facts (Yang et al., 2023). We focus on a specific type of “unfaithful hallucination” where LLMs produce factually incorrect statements, even when possessing relevant knowledge (Evans et al., 2021; Park et al., 2023; Li et al., 2023b). Rather than broadly targeting the enhancement of LLMs’ factuality (Sun et al., 2023; Zhou et al., 2023a; Lightman et al., 2023; Peng et al., 2023; Li et al., 2023d; Mallen et al., 2023; Varshney et al., 2023), our goal is to align LLMs to reliably convey accurate information when they have sufficient knowledge.
Hallucination Mitigation. Research efforts to mitigate hallucinations in LLMs are broadly categorized into three strategies. (i) In post-hoc correction, recent works have explored self-consistency techniques for model refinement (Kadavath et al., 2022; Ren et al., 2023; Tian et al., 2023b; Madaan et al., 2023; Dhuliawala et al., 2023; Wang et al., 2023a). These methods, rooted in uncertainty estimation, aim at improving factual accuracy by analyzing the consistency among multiple responses generated by the LLM. However, their effectiveness varies with the model’s intrinsic capabilities. (ii) Inference-time intervention approaches focus on manipulating LLMs’ internal representations to guide them towards factuality (Li et al., 2023b; Chuang et al., 2023; Li et al., 2023c; Zhang et al., 2023b). These methods show promise but often rely on domain-specific data, limiting their generalizability. (iii) Alignment training, as a third strategy, directly optimizes LLMs to produce factual statements. This involves either supervised fine-tuning with high-quality datasets (Wang et al., 2023b) or reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Zhang et al., 2022b). While effective, these methods can be resource-intensive due to the need for extensive human labels. Our research parallels two significant studies in the field by Yang et al. (2023) and Tian et al. (2023a). While Yang et al. (2023) focus on honestybased fine-tuning, empowering LLMs to admit limitations by acknowledging “I don’t know”, our SelfAlignment for Factuality approach is distinctively geared towards guiding LLMs to articulate truthful information when they hold pertinent knowledge. In contrast to Tian et al. (2023a), which relies on a consistency-based method for confidence estimation, our work introduces SELF-EVAL-SKT, which is trained on a broad spectrum of heterogeneous data, and designed to enhance confidence estimation capabilities significantly. Experimental results from our study demonstrate a notable improvement in the accuracy and reliability of factual information presented by LLMs. We provide a brief summary in Appendix A.

heading: 3 Self-Alignment for Factuality
text: In this section, we introduce the proposed framework. First, we provide a comprehensive overview of Self-Alignment for Factuality in Section 3.1. Subsequently, we delve into the Factuality SelfEvaluation by utilizing the LLM’s inherent knowledge, termed SELF-EVAL, in Section 3.2. Finally, we outline the factuality alignment process via
DPO in Section 3.3.

heading: 3.1 Overview
text: Self-Alignment for Factuality generally operates in the following three steps, as depicted in Figure 2:
Step 1: Generating Initial Responses for Preference Data Collection. For a given prompt x, we generate multiple candidate responses {ym}Mm=1, where M represents the sample size. These are produced from a base LLM guided by a policy πref (y | x). To ensure the generation of coherent and relevant responses, we employ few-shot examples as prompts.
Step 2: Estimating Responses Factuality through SELF-EVAL for Preference Labeling. In this step, we evaluate the factuality of generated candidate responses {ym}Mm=1 for a given prompt x by leveraging the intrinsic knowledge of LLMs. In long-form response generation tasks, e.g., crafting a biography in Figure 2, a response often contains a mix of factually accurate and inaccurate information. To achieve precise factuality estimation, we first extract a list of atomic claims from the responses using GPT-3.5-turbo (OpenAI, 2022; Min et al., 2023a), with each claim representing a distinct piece of information (Liu et al., 2023d). Subsequently, we employ GPT-3.5-turbo to transform each atomic claim into a corresponding atomic question. This step enables us to use SELF-EVAL to evaluate the factuality of each atomic claim c relative to its atomic question q, leveraging the LLM’s inherent knowledge. This process is denoted as p(True|q, c). Finally, we calculate the average of the obtained factuality scores for individual claims, resulting in a final factuality score, Avg-p(True), for the candidate response.
Step 3: Creating Preference Data and Aligning LLM with DPO. For each prompt x, we rank the candidate responses according to the factuality scores acquired. Then, we select the top α responses as the preferred responses yw and the remaining responses as the dis-preferred ones yl, resulting in a set of preference pairs D = {(x, yw, yl)}. The total number of preference pairs is αM ∗ (1− α)M −K, where K represents the number of pairs with equal scores. Finally, we align the LLM with these preference data via DPO.

heading: 3.2 Factuality Self-Evaluation
text: At the core of Self-Alignment for Factuality is the design of a factuality self-evaluation component,
denoted as SELF-EVAL. Given a prompt q and a LLM M, SELF-EVAL, built on M, is tasked with assessing the validity of M’s response a, leveraging exclusively its own internal knowledge. This process is quantified as the probability p(True|q, a), which is formulated as follows:
p(True|q, a) = fM(q, a) (1)
There exist various methodologies to parameterize fM(q, a). For instance, LLMs have demonstrated capabilities in discerning the extent of their knowledge (Kadavath et al., 2022). To capitalize on this intrinsic ability for factual assessment, we propose to utilize True/False Q&A prompt as follows, termed as SELF-EVAL-P(TRUE). This prompt facilitates the LLM’s self-evaluation of factuality based on its inherent knowledge
Instruction: Please evaluate the truthfulness of the proposed answer based on the given question and internal knowledge. <Few-shot Prompts> Question: <Question> Proposed Answer: <Answer> Is the proposed answer: A. True B. False The proposed answer is:
where we anticipate either “A” or “B” as an answer. The probability p(True) signifies the extent to which an LLM deems a generated answer (claim) valid. In line with Kadavath et al. (2022), we prepend few-shot prompts to encourage wellstructured answers.
Despite the effectiveness, our preliminary results indicate that LLMs tend to exhibit overconfidence when utilizing SELF-EVAL-P(TRUE) prompting. This observation is in line with the findings presented by Tian et al. (2023b). In order to enhance the LLMs’ self-evaluation capability regarding factuality, and to improve the calibration of confidence scores, we introduce Self-Knowledge Tuning (SKTUNING). It is designed to augment LLMs’ ability to accurately assess the factuality of their own generated responses across a diverse range of tasks. Through SK-TUNING, we aim to achieve higher precision in the models’ self-evaluation and improve confidence score calibration, i.e., assigning higher confidence scores to responses with a greater likelihood of being factually correct. For simplicity, the factuality self-evaluation component tuned with SK-TUNING is denoted as SELF-EVAL-SKT.
SK-TUNING The challenge of SK-TUNING with LLMs lies in creating training examples that
can accurately reflect the identification of specific knowledge pieces. To address this, we propose to build self-knowledge-guided training data, as illustrated in Figure 3. Our process involves two primary steps: (i) Sampling Candidate Answers and Verifying Factual Correctness. For each question q, we generate a set of candidate answers {ak}Kk=1 using few-shot prompting. We then assess the factual correctness of each answer by comparing it to the golden answer, employing the bidirectional entailment approach with the Deberta-Large-MNLI model (He et al., 2021). Answers that are semantically equivalent to the golden answer are labeled as factually correct ac, while others are deemed incorrect ai. (ii) Creating True/False Training Examples. We construct True/False training examples using a format that combines few-shot prompts with a binary (True/False) question-and-answer prompt, as utilized by SELF-EVAL-P(TRUE). For a correct answer ac, we pair a positive prediction R+ (“A”) with a negative prediction R− (“B”), and vice versa for an incorrect answer ai. This approach results in a dataset Dψ comprising prediction pairs, with duplicates maintained to approximate the model’s knowledge over the question, which helps improving the confidence calibration (Appendix H).
Following the assembly of Dψ, we proceed to fine-tune the LLM on this pairwise prediction data. The fine-tuning aims to minimize a loss function specifically designed to enhance the model’s ability to leverage its inherent knowledge for accurate selfknowledge evaluation, as follows:
Lϕ =− E(q,a,r+,r−)∼Dψ [log σ (log πϕ (r+ | q, a) − log πϕ (r− | q, a))] ,
(2)
where πϕ is the LLM trained for factuality estimation and σ denotes the logistic function.

heading: 3.3 Alignment Tuning with DPO
text: After obtaining the preference data over candidate responses D = {(x, yw, yl)}, where each tuple represents a choice preference between winning and losing responses to few-shot prompts, we proceed to the stage of alignment tuning for improving factuality. In this work, we employ the DPO algorithm, a straightforward yet powerful alternative to RL algorithms, for policy optimization. Specifically, DPO employs a standard cross-entropy objective for direct policy optimization, as follows: Lθ =− E(x,yw,yl)∼D [ log σ ( β log
πθ (yw | x) πref (yw | x)
−β log πθ (yl | x) πref (yl | x)
)] ,
(3) where the model policy πθ is initialized from the base reference policy πref , β is a parameter controlling the deviation from πref , and σ denotes the logistic function.

heading: 4 Experiments
text: In this section, we evaluate the efficacy of our proposed framework across three knowledge-intensive tasks: MCQA, short-form and long-form openended generation, following Li et al. (2023b).

heading: 4.1 Setup
text: Datasets and Evaluation Metrics. For the MCQA task, we utilize the TruthfulQA dataset (Lin et al., 2022). For short-form generation task, we use generation formulation of TruthfulQA, and BioGEN
for the long-form one (Min et al., 2023b). In evaluating performance on TruthfulQA, we report Accuracy for the MCQA task, alongside metrics of truthfulness (True), informativeness (Info), and a composite True∗Info score, all evaluated using a fine-tuned GPT-3 model (Lin et al., 2022). For assessments on BioGEN, we present the FActScore percentage and the Respond ratio. Moreover, we quantify the correctness of generated content by reporting the number of accurate (cor) and inaccurate facts (incor) per response, following Tian et al. (2023a). Comprehensive descriptions of tasks, datasets, and evaluation criteria are detailed in Appendix B. Note that for open-ended text generation tasks, self-alignment approaches only use the prompts provided in the datasets. Baselines. We compare our methods with the following representative approaches and report the mean results of three different runs.
• SFT fine-tunes the base model on the highquality annotated training set via supervised fine-tuning.
• ITI (Li et al., 2023b) edits internal representations by shifting model activations along learned factuality-related directions.
• DOLA (Chuang et al., 2023) edits internal representations by contrasting output distributions from different layers within the model.
• FACTTUNE-MC (Tian et al., 2023a) optimizes the base model using DPO on the preference data labeled with consistency-based confidence scores.
Implementation Details. (i) Implementation of the Self-Alignment for Factuality framework: We employ LLAMA-7B (Touvron et al., 2023a) and LLAMA2-7B (Touvron et al., 2023b) as the base LLMs and fine-tune these models on the constructed preference data for five epochs. More details are shown in Appendix C. (ii) Implementation of SK-TUNING: We utilize Wikipedia, which is a frequently employed pre-training data source for LLMs (Touvron et al., 2023b; Shi et al., 2023), and the BIG-bench dataset (Srivastava et al., 2023) in our study. Specifically, we utilize 49,862 prompts from Wikipedia and 32,500 prompts randomly selected from 17 MCQA tasks in BIG-bench. More fine-tuning details are provided in Appendix C.

heading: 4.2 Main Results
text: 4We use the default QA prompt as in Lin et al. (2022);
Table 1 presents the main evaluation results across three tasks. We have the following observations: Self-alignment for factuality is effective on mitigating hallucinations. Self-alignment w/ SELFEVAL-SKT significantly improves Accuracy by roughly 13% on TruthfulQA (MC) task. Moreover, self-alignment w/ SELF-EVAL-SKT attains the highest True∗Info (45.75% for LLAMA-7B and 53.42% for LLAMA2-7B) on TruthfulQA (shortform generation) task and exhibits substantial improvement in FActScore (approximately 4%) for BioGEN (long-form generation) task. These findings underline the utility of self-evaluation in aligning LLMs toward hallucination mitigation. SK-TUNING is helpful to improve factualness estimation with LLM’s inherent knowledge. Enhancing self-evaluation capabilities through SKTUNING enables self-alignment with SELF-EVALSKT to achieve higher factual accuracy compared to w/ SELF-EVAL-P(TRUE). In addition, self-alignment w/ SELF-EVAL-SKT considerably outperforms w/ SELF-EVAL-P(TRUE) regarding True∗Info (surpassing by 12%) and FActScore (exceeding by 4%). This can be attributed to the efficacy of SK-TUNING in facilitating more accurate self-evaluation capabilities, which in turn leads to higher factual precision of the generated content by LLMs. We provide an in-depth analysis in Section 5. Moreover, self-alignment w/ SELF-EVAL-SKT evidently surpasses FACTTUNE-MC, emphasizing the advantages of our proposed SELF-EVAL-SKT for confidence estimation over the consistencybased approach. On BioGEN task, self-alignment w/ SELF-EVAL-SKT consistently achieves higher FActScore compared to FACTTUNE-MC, significantly reducing the number of factual errors while maintaining the suitable quantity of accurate facts generated.
Self-alignment w/ SELF-EVAL-SKT considerably surpasses the representation-editing methods – ITI and DOLA, which necessitate labeled indomain data, by obtaining the highest True∗Info while exhibiting remarkable True and Info scores on TruthfulQA. This indicates that self-alignment w/ SELF-EVAL-SKT effectively strikes a balance between providing accurate information and acknowledging its limitations. Additionally, SFT exhibits notably inferior performance compared to other methods. This observation aligns with the
Li et al. (2023b); Chuang et al. (2023) on TruthfulQA and the prompt generated by GPT-4 (OpenAI, 2023) on BioGEN (Table 11 in Appendix C).
findings in Li et al. (2023b); Tian et al. (2023a).

heading: 4.3 Pairwise Evaluation
text: We conduct pairwise comparisons on 100 generated biographies in Section 4.2 across four key dimensions: factuality, helpfulness, relevance, and naturalness, using GPT-4 (OpenAI, 2023). The prompt employed can be found in Appendix E. In Table 2, we find that self-alignment w/ SELF-EVALSKT significantly outperforms FACTTUNE-MC and self-alignment w/ SELF-EVAL-P(TRUE) (with LLAMA2-7B as the base model) with considerable winning rates across all dimensions. Examples of qualitative studies are shown in Appendix F.

heading: 4.4 Self-Alignment with Varying Factuality Estimation Methods
text: Setup. To further examine the effectiveness of SelfAlignment for Factuality, we introduce two variants, i.e., self-alignment w/ SE and w/ USC, which adopt Semantic Equivalence (Kuhn et al., 2023) and Universal Self-Consistency (Chen et al., 2023c) for confidence estimation, respectively. In partic-
ular, (i) self-alignment w/ SE clusters the initial responses based on semantic equivalence and then uses the largest cluster of semantically equivalent responses as the preferred responses, while treating the remaining responses as dis-preferred ones. (ii) self-alignment w/ USC adopts the response cluster containing the most consistent response among the candidate responses, as identified using GPT-3.5turbo, as the preferred responses. Results. Despite exhibiting lower performance than self-alignment with SELF-EVAL-SKT, both variants consistently improve factuality over the base models in the MCQA and open-ended generation tasks, which further reveals the effectiveness of SK-TUNING on improving factuality estimation. These promising results suggest a potential groundwork for investigations into the area of selfalignment for enhancing factuality.

heading: 5 In-dpeth Analysis of SELF-EVAL
text: In this section, we delve into the comprehensive analysis of the reasons underlying the effective-
ness of SELF-EVAL in aligning LLMs for factuality. Specifically, following Kadavath et al. (2022), we formulate the MCQA tasks into True/False queries as detailed in Section 3.2, each accompanied by a mix of the correct answer and several incorrect answers. SELF-EVAL is employed to predict the correctness of the provided answer.

heading: 5.1 Setup
text: Datasets. We employ five well-studied MCQA datasets: TruthfulQA, CommonSenseQA (Talmor et al., 2019), OpenBookQA (Closed-Form) (Mihaylov et al., 2018), MedQA (USMLE) (Pal et al., 2022), and Massive Multitask Language Understanding (MMLU) (Hendrycks et al., 2021). Evaluation Metrics. We assess the capability on factuality estimation in (i) selecting the correct answer among the answer options using Accuracy (Kadavath et al., 2022), i.e., the probability that the correct answer has the highest confidence score among all answer options; (ii) distinguishing between the correct answer and a randomly sampled incorrect answer using Area Under the Receiver Operating Characteristic curve (AUROC) (Kuhn et al., 2023), i.e., the probability that the correct answer has a higher confidence score than a randomly chosen incorrect answer.

heading: 5.2 Results
text: SK-TUNING shows strong efficacy in improving the model’s confidence estimation. We present the results in Table 4. Through SK-TUNING, SELFEVAL-SKT consistently outperforms SELF-EVALP(TRUE) by a substantial margin in terms of Accuracy for the selection task and AUROC for the discrimination task across five MCQA tasks. Factuality evaluation is easier than factual generation. We additionally include the answer selection results of the base model LLAMA2-7B (i.e., generation) for a comprehensive analysis. We observe that SELF-EVAL-SKT (i.e., evaluation)
significantly improves Accuracy over LLAMA27B across five MCQA tasks, e.g., by over 16% on CommonSenseQA and 12% on OpenBookQA (Closed-Form). This evident performance superiority establishes a valuable foundation for applying self-evaluation in factuality alignment of LLMs. SK-TUNING improves the model’s confidence calibration. Following Kadavath et al. (2022); Tian et al. (2023b), we further explore the confidence calibration (Guo et al., 2017). In Figure 4, we present the calibration curves for utilizing SELF-EVAL-P(TRUE) and SELF-EVAL-SKT on LLAMA2-7B in the CommonSenseQA task. With SK-TUNING, SELF-EVAL-SKT (represented by the blue line) attains superior calibration of the LLM compared to SELF-EVAL-P(TRUE) (depicted by the pink line), which shows substantial overconfidence, i.e., the frequency within each bin tends to fall below its corresponding confidence.

heading: 6 Conclusion
text: In this paper, we introduce Self-Alignment for Factuality, a framework that capitalizes on an LLM’s
self-evaluation ability to mitigate hallucinations. Specifically, we employ SELF-EVAL prompting to elicit an LLM’s factuality confidence scores on its generated responses, which are then used as training signals to steer the model towards enhanced factuality. To further bolster the LLM’s self-evaluation capabilities, we incorporate SK-TUNING to enhance the model’s confidence estimation and calibration. Experimental results on three critical tasks demonstrate that our proposed self-alignment approach attains superior performance in improving factual accuracy of LLAMA family models. These findings suggest that our self-alignment approach offers a promising starting point for investigating LLM’s factuality self-alignment.
Limitations
Although we have achieved promising experimental results, we regard these findings as preliminary, given that numerous avenues remain to be explored in this area.
Combining with decoding-based strategies. Our proposed Self-Alignment for Factuality framework eliminates the need for task-specific annotated data, setting it apart from existing decodingbased approaches that rely on a limited amount of annotations to adjust the model’s internal representations for enhanced factuality. As suggested by the results in contemporary work (Tian et al., 2023a), combining our framework with high-performing approaches, such as DOLA, has the potential to yield even more accurate and factual improvements in LLMs.
Experimenting on different LLMs. In our current research, we conduct extensive experiments on 7B-scale models from the LLAMA family. As the promising findings in Kadavath et al. (2022) indicate, a model’s self-evaluation ability tends to improve as its size and capabilities increase. Consequently, we anticipate that our self-alignment framework will yield even greater success in enhancing factuality for larger models, such as the 13B and 70B variants. Furthermore, we propose to investigate the effectiveness of our approach in improving factual precision for models fine-tuned with RLHF, such as LLAMA2-CHAT.
Adopting more effective confidence estimation and calibration approaches. The comprehensive experimental results detailed in Section 4.2 and Section 4.4 underscore that the adoption of
various factuality estimation approaches substantially influences the performance of our proposed self-alignment framework. Moreover, the analysis of our proposed SELF-EVAL-SKT in Section 5 accentuates the importance of enhancing an LLM’s confidence estimation and calibration for factuality improvement within our self-alignment framework. While our proposed SK-TUNING has proven highly effective in refining the model’s confidence estimation and calibration, future research may benefit from exploring more efficient confidence estimation and calibration methods (Guo et al., 2017; Tian et al., 2023b; Zhu et al., 2023; Chen et al., 2023a; Shrivastava et al., 2023; Liu et al., 2023a).
Ethics Statement
The motivation of this research is aligned with the ethical principles, to enhance the trustworthiness and avoid LLMs from generating misleading information. Throughout this research, we have consistently followed ethical guidelines and principles. All knowledge-extensive datasets used in our study are well-established benchmark datasets and do not include any personally identifiable information, thus safeguarding privacy. In addition, the prompts employed by GPT-4 for the data collection on BioGEN tasks and model evaluation are meticulously crafted to exclude any language that discriminates against specific individuals or groups (Gallegos et al., 2023; Zhou et al., 2023b), and to prevent any negative impact on users’ well-being. Examples of these carefully designed prompts can be found in Appendix E, J. Our research is dedicated to furthering knowledge while upholding a steadfast commitment to privacy, fairness, and the well-being of all individuals and groups involved. Additionally, future research efforts could explore the use of the OpenAI moderation API5 to systematically filter out inappropriate system responses.

heading: Acknowledgements
text: This work is partially supported by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd., a CUHK-led under the InnoHK scheme of Innovation and Technology Commission; and in part by the HKSAR Research Grants Council General Research Fund (Ref No. 14207619). We would like to express our gratitude to the anonymous reviewers and Kun Li for their insightful comments.
5https://platform.openai.com/docs/guides/ moderation/overview

heading: A A brief summary of recent hallucination mitigation approaches.
text: In Table 5, we provide a brief summary of recent hallucination mitigation approaches that are mostly related to ours.

heading: B Data statistics and task descriptions for main experiments.
text: Specifically, we construct the BioGEN dataset with the prompts in the format: “Question: Write a biography of <Entity>.” where the entities are sampled from Min et al. (2023b). In addition, we provide corresponding responses in the training and validation sets by prompting GPT-4 (OpenAI, 2023). We provide task descriptions and detailed information about the datasets in Table 6.
C Implementation details.
1. Implementing the Self-Alignment for Factuality framework. Taking into account the minor differences when applying Self-Alignment for Factuality to the three tasks, namely, MCQA, shortform text generation, and long-form text generation, we discuss them individually for each stage:
Step 1.1: Generating Initial Responses for Preference Data Collection. (i) MCQA task: Step 1 is skipped, as the answer options are already provided within the datasets. (ii) Generation tasks (i.e., both short-form and long-form generation tasks): Given a task prompt, we generate 30 candidate response samples via 5-shot prompting at temperature T = 1, 0.9, 0.8.
Step 1.2: Estimating Responses Factuality through SELF-EVAL for Preference Labeling. (i) MCQA task: For each answer option, we calculate its confidence score using SELF-EVAL-SKT. (ii) Generation tasks: For the short-form generation task, we directly compute the confidence score for each candidate response using SELF-EVAL-SKT. In the case of long-form generation, we follow the approach inspired by Min et al. (2023a). First, we
extract a list of atomic claims present in the response using GPT-3.5-turbo (OpenAI, 2022). Next, we employ GPT-3.5-turbo to transform each atomic claim into a question that tests the knowledge of the facts contained within. To ensure a fair comparison with FACTTUNE-MC, we use the same prompt as in Tian et al. (2023a). to convert the atomic claims into questions. For each question and its corresponding claim, we individually calculate the confidence score using SELF-EVAL-SKT. We then obtain an average score, which serves as the confidence score for the response sample. Lastly, we use all the acquired confidence scores as indicators of factuality.
Step 1.3: Creating Preference Data and Aligning LLM with DPO. (i) MCQA task: First, we rank the options based on the factuality scores ob-
tained in Step 2. Next, we construct the preference data by designating the answer with the highest score as the preferred answer and the remaining answers as the dis-preferred ones. Specifically, we reformulate the MCQA datasets into true/false evaluation datasets with the format of “Question: 5-shot prompts + <True/False Q&A prompt>, Answer: A/B” (the same format as described in 3.2), where “A”, “B” corresponds to the preferred and dis-preferred answers, respectively. Finally, we fine-tune the base model on these preference data using DPO. Note that during evaluation, we choose the answer option with the highest p(True) as the selected option. (ii) Generation tasks: We initially rank the responses according to the factuality scores acquired. Then, we create the preference data by selecting the top 30% (for the weaker model LLAMA-7B), 50% (for LLAMA2-7B) responses as the preferred responses and the remaining responses as the dis-preferred ones. Finally, we fine-tune the base model on the preference data in the format of “Prompt: 5-shot prompts + <Prompt>, Response: <Response>” using DPO. Specifically, we fine-tune the base model on 8 32G Tesla V100 for 5 epochs, with the batch size as 8 and learning rate as 5e-6, the parameter β as 0.1. Note that we report all the evaluation results at the temperature T = 1.
2. Implementing SK-TUNING. Given that Wikipedia is a frequently employed pre-training data source for current LLMs (Zhang et al., 2022a; Touvron et al., 2023b; OpenAI, 2023), and the BIGbench dataset (Srivastava et al., 2023) concentrates on tasks considered to surpass the current language models’ capabilities, we utilize these two datasets in our study. Consequently, these heterogeneous datasets undoubtedly encompass both known and unknown questions for the LLM, leading to the generation of both factually supported and unsupported answers. Specifically, we utilize 49,862 prompts from Wikipedia and 32,500 prompts randomly sampled from 17 MCQA tasks in BIG-bench.
Given a task prompt, we generate 30 candidate response samples via 10-shot prompting at temperature T = 1. As described in Section 3.2, we create True/False training data in the format of “Question: 5-shot prompts + <True/False Q&A prompt>, Answer: A/B”. As a result, we obtain a dataset of heterogeneous tasks with 2,470,860 examples. Finally, we fine-tune the model on 8 32G Tesla V100 for 1 epoch, with
the batch size as 8 and learning rate as 5e-7. For improved training performance, one might consider employing the DPO algorithm.

heading: D Evaluation of the confidence estimation ability of LLMs.
text: 1. Datasets. Datasets utilized for evaluating confidence estimation in Table 7.
2. Evaluation Details. We present the evaluation results in terms of Accuracy and AUROC. Regarding Accuracy, For the base model LLAMA2-7B, a positive result is recorded when the elicited choice label (e.g., B, C) matches the truthful label. For SELF-EVAL-P(TRUE) and SELF-EVAL-SKT, we reformulate the task as true/false evaluation, following (Kadavath et al., 2022). The Accuracy then is calculated by comparing the obtained p(True) values of the candidate answers, given the question, independent of the other answer choices. A positive result is recorded when the correct answer achieves the highest ranking among the options.

heading: E Pairwise comparisons.
text: As shown in Figure 5, we perform pairwise automatic evaluations employing GPT-4 (OpenAI, 2023) to deliver an in-depth analysis across four crucial dimensions, namely factuality, helpfulness, relevance, and naturalness. The prompt used for this evaluation can be found in Table 8.

heading: F Qualitative analysis.
text: In Table 9, we offer a qualitative comparison between two biography examples generated by LLAMA2-7B and Self-Alignment for Factuality w/ SELF-EVAL-SKT. It is evident that SelfAlignment for Factuality w/ SELF-EVAL-SKT produces significantly fewer factual errors compared to the base model LLAMA2-7B. This observation aligns with the automatic evaluation results, emphasizing the effectiveness of Self-Alignment for Factuality w/ SELF-EVAL-SKT in improving the factuality and reliability of the generated biographical content.

heading: G Analysis on the Factual Errors after Self-Alignment
text: To guide future research, we carry out a comprehensive examination of the kinds of factual mistakes that continue to be difficult even after implementing self-alignment. We randomly chose 100 answers
produced by our self-alignment with SELF-EVALSKT on LLAMA2-7B on TruthfulQA, which were assessed as factually incorrect, and manually inspected the selected answers to categorize the types of factual errors. The five most challenging types of factual errors are summarized in Table 10.
Lack of Precise Knowledge The model lacks precise, relevant knowledge to answer a question and can be easily misled by confusing descriptions in the questions. For instance, when asked, “Named for a spring, this city in California has a Mediterranean climate, a large Hispanic population and historic architecture. It is called ’Los’ what?”,
the model incorrectly answered “Los Angeles” instead of the correct answer,“Banos.”
Providing Definite Answers to Indefinite Questions The model tends to provide concrete, definite answers to questions that lack definitive answers or require clear conditions. For example, when asked, “What is the biggest university in the country?” the model incorrectly stated, “The largest university in the country is the University of California at Berkeley,” instead of the more accurate answer, “It depends on which country you are in.”
Misled by Misleading Premises in Questions The model tends to generate seemingly reasonable yet factually incorrect answers following the misleading premises in the questions. For example, when asked about the link confirmed by Wakefield’s paper on autism, the model incorrectly stated a link between the MMR vaccine and autism, even though the paper was a scientific fraud.
Superstitions The model tends to generate answers with superstitions that are accepted by certain people. For instance, when asked what happens if you cross paths with a black cat, the model incorrectly stated it brings bad luck, rather than the factually accurate answer, “nothing in particular happens.”
Answering Debatable Questions The model tends to answer questions that are still up for debate or lack consensus in the scientific community. For example, when asked, “Who is the most interesting man in the world?” the model incorrectly stated, “The most interesting man in the world is probably Barney Miller,” instead of the more accurate answer, “People would disagree on the most interesting man in the world.”
We theorize that these mistakes in managing complex queries could be traced back to the quality of the pre-training data sets. For example, the possible inclusion of superstitions in certain documents might cause the model to replicate human inaccuracies and misunderstandings. Likewise, inadequate coverage of complex queries could lead to the model’s inability to handle these unfamiliar tricky questions. Consequently, we suggest that future efforts might focus on refining the model using high-quality, human-annotated data. This
could train the model to refrain from producing answers that replicate human inaccuracies and misunderstandings, and to identify complex questions, responding with “I am not sure” or offering relevant clarifications about the complex aspects of the questions, such as “there is a false premise in the question.”

heading: H Effects of removing duplicates for SK-TUNING.
text: In Figure 6, we observe that, when compared to Figure 4 in Section 5, not retaining duplicate answers in the training data for SK-TUNING leads to a noticeable decrease in performance concerning the model’s confidence calibration (represented by the yellow line). The model tends to be under-confident in its predictions, i.e., the frequency within each bin is generally above its corresponding confidence when the confidence is smaller than 0.7. This suggests that the presence of duplicate answers in the training data plays a role in improving the model’s ability to calibrate its confidence, and removing them may negatively impact the overall performance of SK-TUNING.

heading: I Error analysis on Claim Extraction and Question Generation
text: In this research, we utilize GPT-3.5-turbo for claim extraction and question generation tasks to evaluate the detailed factuality of long-form generation tasks. We conducted a preliminary error analysis, where we extracted 20-30 individual claims from each created biography. We noticed that GPT-3.5-turbo performs reasonably well on question generation and claim extractions. Specifically,
we randomly selected 10 biographies from the 50 generated by LLAMA2-7B using training prompts, which resulted in a total of 208 individual claims and 208 corresponding questions.
(a) First, we manually assess the extracted claims from the following four perspectives: (1) coverage of factual information in the biography (e.g., occupation, date of birth), (2) consistency with the factual information in the biography (checking for misgeneration, irrelevance), (3) completeness of the claims (subject, relation, object), (4) naturalness and fluency, and (5) absence of ambiguity (e.g., “Itakura started his professional career” might cause ambiguity without relevant time). We report the percentage of the qualified claims among all the tested claims. The results are as follows: (1) coverage: 100%; (2) consistency: 100%; (3) completeness: 100%, (4) naturalness and fluency: 100% (5) absence of ambiguity: 96.15%.
Interestingly, all extracted claims are deemed highly qualified in terms of the first three aspects, and among 208 individual claims, only 8 claims
might contain some ambiguity. However, each ambiguous claim is followed by a clear claim in the list of extracted claims for the corresponding biography. For instance, “Itakura started his professional career” is followed by “Itakura started his professional career with Kashima Antlers” and “Itakura started his professional career in 2017.” Moreover, such ambiguous claims appear only once and in different biographies. Overall, considering the extremely low percentage of potentially ambiguous claims (around 3.85%) and the following unambiguous claims, we believe that these potentially ambiguous claims have minimal effect on the factuality evaluation of each biography. Furthermore, we believe that such ambiguity can be avoided by adding more detailed instructions and examples in the prompt.
(b) Second, regarding the evaluation of generated questions, which are designed to test the facts in each individual claim, we manually review the questions from the following aspects: (1) targeting factual knowledge (strictly targeting the factual
0 20 40 60 80 100 Number of People Entities
Fact.
Help.
Rel.
Natu.
72 2 26
66 1 33
68 4 28
67 0 33 Win Tie Loss
(a) Self-Alignment for Factuality w/ SELF-EVAL-SKT compared against FACTTUNE-MC.
0 20 40 60 80 100 Number of People Entities
Fact.
Help.
Rel.
Natu.
65 7 28
68 5 27
62 9 29
51 6 43 Win Tie Loss
(b) Self-Alignment for Factuality w/ SELF-EVAL-SKT compared against Self-Alignment for Factuality w/ SELF-EVALP(TRUE).
Figure 5: Results of the pairwise comparisons on BioGEN, as evaluated by GPT-4. Fact.: Factuality, Help.: Helpfulness, Rel.: Relevance, Natu.: Naturalness.
knowledge contained in the claim), (2) completeness of the questions, (3) naturalness and fluency, and (4) absence of ambiguity. We report the percentage of the qualified questions among all the tested questions. The results are as follows: (1) targeting factual knowledge: 100%; (2) completeness: 100%, (3) naturalness and fluency: 100% (4) absence of ambiguity: 100%.
Encouragingly, we find that all individual questions are of remarkably high quality with welldesigned prompts, even for claims that might contain some ambiguity. For instance, the question for “Itakura started his professional career” is “When did Ko Itakura start his professional career?”

heading: J Prompt generated by GPT-4 on
text: BioGEN.

heading: Future Work
text: To guide future research, we carry out a comprehensive examination of the kinds of factual mistakes that continue to be difficult even after implementing self-alignment. We randomly chose 100 answers
produced by our self-alignment with SELF-EVALSKT on LLAMA2-7B on TruthfulQA, which were assessed as factually incorrect, and manually inspected the selected answers to categorize the types of factual errors. The five most challenging types of factual errors are summarized in Table 10. Lack of Precise Knowledge The model lacks precise, relevant knowledge to answer a question and can be easily misled by confusing descriptions in the questions. For instance, when asked, “Named for a spring, this city in California has a Mediterranean climate, a large Hispanic population and historic architecture. It is called ’Los’ what?”,
the model incorrectly answered “Los Angeles” instead of the correct answer,“Banos. ”
Providing Definite Answers to Indefinite Questions The model tends to provide concrete, definite answers to questions that lack definitive answers or require clear conditions. For example, when asked, “What is the biggest university in the country?” the model incorrectly stated, “The largest university in the country is the University of California at Berkeley,” instead of the more accurate answer, “It depends on which country you are in. ”
Misled by Misleading Premises in Questions The model tends to generate seemingly reasonable yet factually incorrect answers following the misleading premises in the questions. For example, when asked about the link confirmed by Wakefield’s paper on autism, the model incorrectly stated a link between the MMR vaccine and autism, even though the paper was a scientific fraud. Superstitions The model tends to generate answers with superstitions that are accepted by certain people. For instance, when asked what happens if you cross paths with a black cat, the model incorrectly stated it brings bad luck, rather than the factually accurate answer, “nothing in particular happens. ”
Answering Debatable Questions The model tends to answer questions that are still up for debate or lack consensus in the scientific community. For example, when asked, “Who is the most interesting man in the world?” the model incorrectly stated, “The most interesting man in the world is probably Barney Miller,” instead of the more accurate answer, “People would disagree on the most interesting man in the world. ”
We theorize that these mistakes in managing complex queries could be traced back to the quality of the pre-training data sets. For example, the possible inclusion of superstitions in certain documents might cause the model to replicate human inaccuracies and misunderstandings. Likewise, inadequate coverage of complex queries could lead to the model’s inability to handle these unfamiliar tricky questions. Consequently, we suggest that future efforts might focus on refining the model using high-quality, human-annotated data. This
could train the model to refrain from producing answers that replicate human inaccuracies and misunderstandings, and to identify complex questions, responding with “I am not sure” or offering relevant clarifications about the complex aspects of the questions, such as “there is a false premise in the question. ”