FLAME : Factuality-Aware Alignment for Large Language Models

Authors: sheng-chieh  lin
luyu  gao
barlas  oguz
wenhan  xiong
jimmy  lin
wen-tau  yih
xilun  chen
sft  +dpo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-02 22:48:23

Abstract: Alignment is a procedure to fine-tune pre-trained large language models (LLMs) to follow natural language instructions and serve as helpful AI assistants. 
We have observed, however, that the conventional alignment process fails to enhance the factual accuracy of LLMs, and often leads to the generation of more false facts (i.e., *hallucination*). 
In this paper, we study how to make the LLM alignment process more factual, by first identifying factors that lead to hallucination in both alignment steps: supervised fine-tuning (SFT) and reinforcement learning (RL).
In particular, we find that training the LLM on new or unfamiliar knowledge can encourage hallucination.
This makes SFT less factual as it trains on human-labeled data that may be novel to the LLM. 
Furthermore, reward functions used in standard RL often inadequately capture factuality and favor longer and more detailed responses, which inadvertently promote hallucination.
Based on these observations, we propose *FactuaLity-aware AlignMEnt*, comprised of *factuality-aware SFT* and *factuality-aware RL* through direct preference optimization. 
Experiments show that our proposed *FLAME* guides LLMs to output more factual responses while maintaining their instruction-following capability.

Main paper: 
 introduction: 
 Alignment [Ouyang et al., 2022] is a procedure to make pre-trained large language models (LLMs) [Brown et al., 2020 , Touvron et al., 2023] follow human instructions and serve as helpful AI assistants. Despite significant progress in general LLM alignment [Ouyang et al., 2022 , Bai et al., 2022 , Yuan et al., 2024] , state-of-the-art aligned LLMs are still prone to generate false claims [Ope-nAI, 2023 , Min et al., 2023] . In this work, we therefore attempt to advance the understanding of the underlying causes of LLM hallucination as well as its relation to the alignment procedure.We consider the commonly seen alignment process consisting of two training phases: (1) supervised fine-tuning (SFT) [Sanh et al., 2022] ; (2) reinforcement learning (RL) with human [RLHF, Ouyang et al., 2022 , Bai et al., 2022] or automated feedback [RLAIF, Bai et al., 2023] . In our study, we find that both the SFT and RL steps in the standard alignment process may actually encourage LLMs to hallucinate. First, in the SFT stage, LLMs are fine-tuned with diverse instructions paired with humancreated high-quality responses. While this leads to strong instruction-following capability [Ouyang et al., 2022 , Köpf et al., 2023 , Zhou et al., 2023 , Touvron et al., 2023] , our study shows that such human-labeled responses may present new or unknown information to the LLM. This, in turn, may inadvertently promote hallucination. Second, we find that the standard reward used in the RL stage Figure 1 : Models' helpfulness on Alpaca Eval vs factuality on biography. Helpfulness is measured by models' win rate over our baseline SFT + DPO on Alpaca Eval. Dot size represents average length of bio generation.often prefers longer and more detailed responses [Singhal et al., 2023 , Chen et al., 2024b , Yuan et al., 2024] . Consequently, a reward-hacking model ends up with a tendency to produce longer claims with more non-factual information, as shown in the black dots in Figure 1 . One possible reason is that most existing RLHF or RLAIF approaches rely on a single scalar reward to represent preference, which struggles to cover multiple alignment skill sets [Ye et al., 2024] and is likely to under-present the aspect of factuality [Hosking et al., 2024] .To address the aforementioned issues, we study the key factors which impact factuality during alignment. In particular, we first conduct a pilot study on the biography generation task [Min et al., 2023] in a more controlled setting where the alignment process focuses solely on factuality (Section 3). Our key observation is that an LLM hallucinates more if it is fine-tuned on new knowledge in either the SFT or the RL stage. For example, an LLM becomes significantly less factual when fine-tuned on responses produced by a model with access to external knowledge (e.g. a retrieval augmented LLM), even though those responses are more factual themselves. Similarly, hallucination is greatly increased if RLAIF is performed on preference pairs that consist of retrieval-augmented LLM output as positive examples and the LLM's own output as negative examples. In comparison, we discover that fine-tuning a pre-trained LLM on a subset of its own generations selected by factuality yields more factual responses and reduces hallucinations.Next, we apply our findings to improve the factuality of the general LLM alignment process, which is more challenging due to the diversity of instructions. As shown in Figure 2 , we observe that some instructions require factual responses while the others do not, and therefore would require different alignment treatments. We first identify fact-based instructions that require factual responses and leverage the findings in our pilot study to create additional training data at both SFT and RL stages to explicitly guide LLMs to output factual responses. Specifically, at the SFT stage, for fact-based instructions, instead of using human created seed training data, we elicit knowledge from the pre-trained LLM and construct training data using its own pre-trained knowledge. This can prevent fine-tuning the LLM on knowledge unknown to itself. At the RL stage, we create additional preference pairs focused on factuality for fact-based instructions, which are combined with the standard preference pairs for instruction following during Direct Preference Optimization [DPO; Rafailov et al., 2023] .We evaluate models on Alpaca Eval [Dubois et al., 2024] and Biography, using win rate for instructionfollowing capability and FACTSCORE [Min et al., 2023] for factuality evaluation. As shown in Figure 1 , using our FLAME method (SFT + DPO ), a significantly higher FACTSCORE (+5.6 pts) is achieved compared to the standard alignment process (SFT + DPO), without sacrificing the LLM's instruction-following capability (51.2% win rate). Our ablation study also indicates that identifying fact-based instructions is the key to factual alignment in the general alignment setting.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The main claim of our paper is to advance the understanding of the underlying causes of LLM hallucination as well as its relation to the alignment procedure (from lines 22-23) and study how to make the LLM alignment process more factual (lines 5-6). We thus conduct a pilot study to identify the key factors which impact the LLMs' factuality during alignment procedure in Section 3. Based on the finding, we implement the factualityaware alignment in Section 4 and demonstrate the effectiveness of our approach through comprehensive experiments in Section 5. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We include the discussion of limitations in Appendix A.7. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
a pilot study on factual alignment: 
 In this section, we first study how to align large language models (LLMs) to be more factual. We use biography generation as the task of our pilot study for two main reasons: (1) Biography generation is a simplified setting where factuality is the sole focus of the alignment process. As we will discuss in Section 4, studying factual alignment on diverse human instructions is more complex, as the alignment process encompasses aspects beyond factuality, such as helpfulness and safety. (2) Evaluating the factuality of biography generation is relatively easy since Wikipedia covers sufficient information for public figures and most of the facts about a person are non-debatable [Min et al., 2023] .A standard alignment procedure consists of supervised fine-tuning (SFT) and reinforcement learning (RL) . In this pilot study, our main goal is to teach LLMs to generate biography with reduced misinformation. For the experiment, we compile training and evaluation datasets comprising 500 and 183 diverse human entities, respectively (further details provided in Appendix A.1). We employ FACTSCORE [FS; Min et al., 2023] as the automated metric for assessing factuality, given its finegrained evaluation capabilities for long-form text generation and its strong correlation with human judgments. 3 To study factuality alignment in this pilot study, we posit that training data is needed where the responses are more factual than the LLM's own generations. Thus, we use retrievalaugmented LLMs [RAG; Lewis et al., 2020] to generate training data, which has been shown to output more factual responses [Mialon et al., 2023] .Throughout the paper, we refer to the pre-trained (PT), supervised fine-tuned (SFT), and direct preference optimization (DPO) fine-tuned LLMs as PT, SFT, and DPO, respectively. 4 played, so they don't take advantage of me? Hi, could you help me to solve this cubic equation using Cardano's Mehod (step by step if possible), please? -> " x^3 + 2x^2 -x -1 = 0 " Write a fun story that can be told in 3 minutes at the dinner table. We are 3 developers enjoying a pizza. The story must contain these word: zombie, ethernet cable, sudo, dashboard.How did US states get their names?How do I wrap a present neatly?are there animals who commit suicide Write a response that disagrees with the following post: "Technology is everything that doesn't work yet."What is some cool music from the 1920s? How do I dice without slicing my finger What are the principles at play in UHPLC-MS analysis?Is the internet's focus on engagement the root of most of its problems and shortcomings?Who is Larry Page? Hi, I'm trying to solve a crossword puzzle, but I've never done one of these before. Can you help me out? Explain the significance of the American Revolution, including the events that led up to it, the impact it had on the world, and its ongoing relevance today. Non fact-based (1) Do you have any information about the Commodore 64? (6) How would a child feel if it fell down on the ground hitting its face?(2)Hi, could you help me to solve this cubic equation using Cardano's Mehod (step by step if possible), please? -> " x^3 + 2x^2 -x -1 = 0 " (7)Write a fun story that can be told in 3 minutes at the dinner table. We are 3 developers enjoying a pizza. The story must contain these word: zombie, ethernet cable, sudo, dashboard. (3) Please give me a brief history of coffee.(8) Tell me a story about a pig who goes to the moon.(4) What are the principles at play in UHPLC-MS analysis? (9) Is the internet's focus on engagement the root of most of its problems and shortcomings?(5)Explain the significance of the American Revolution, including the events that led up to it, the impact it had on the world, and its ongoing relevance today.(10)Can you tell me a bit about what has gone into your creation?(x ∈ X fact ) (x ∉ X fact )Figure 2 : Instructions from Open Assistant dataset. The instructions are classified with SFT model using the prompt in Appendix Figure 4 .SFT. We explore two sources of supervision to generate training data (detailed in Appendix A.1):(1) using PT RAG with few-shot demonstration to generate biographies for each name entity in training data, where PT RAG is PT augmented with an off-the-shelf retriever [Lin et al., 2023] ; (2) using vanilla PT with few-shot demonstration to generate training data as a baseline. As shown in DPO. We further fine-tune the LLMs to be more factual through DPO. An intuitive way to create factuality preference pairs is to directly use the samples from PT RAG and PT as positives and negatives since PT RAG generates more factual biographies than PT (row 2 vs 1). Another approach is to employ FACTSCORE (FS) as the reward to select positive and negative samples among the generations from PT itself [Tian et al., 2024] (detailed in Appendix A.1). As shown in Table 1 , DPO fine-tuned on self-generated data with FS reward guides models to generate more factual responses (row 5 vs 3); however, DPO fine-tuned with the supervision of PT RAG makes the models hallucinate even more than its SFT counterpart (6 vs 4).This outcome suggests that compelling models to generate responses akin to PT RAG prompts increases hallucination. Conversely, fine-tuning LLMs on their own generations appears to be crucial for factual alignment, a finding applicable to both SFT and DPO fine-tuning.From the pilot study, we find that better quality data (in terms of factuality) for SFT and DPO does not necessarily yield models with better factual alignment. This is likely because the supervision from RAG contains information unknown to the LLM; thus, fine-tuning on RAG generated responses may inadvertently encourage the LLM to output unfamiliar information. To avoid unknown knowledge from being presented to the LLM, a viable strategy is to create SFT and DPO training data using the generated responses from the LLM itself.4 Factuality-Aware AlignmentIn the section, we further extend our discussion of factual alignment to encompass more general instructions. Unlike biography generation in Section 3, where factuality is the main alignment objective, human instructions are diverse and complex, necessitating a range of alignment skill sets beyond factuality alone; e.g., logical thinking, problem handling and user alignment [Ye et al., 2024] . Thus, conducting factual alignment with the diverse instructions face two main challenges:(1) different instructions may demand distinct skill sets. For example, in Figure 2 , instruction 3, "Please give me a brief history of coffee", necessitates factual accuracy and concise summarization, y M 0 , y SD x SD ∉ X fact ( ) x ( ),Human(x)Seed data(a)Tell me a story about a dog who goes to the mars.Do you have any information about Mac OS?x ∈ X fact x ∉ X fact y + y - yt rue yf alseTell me a story about a pig who goes to the moon.Please give me a brief history of coffee.x ∉ X fact x ∈ X factResponses from itself with few-shot:( while instruction 8, "Tell me a story about a pig who goes to the moon", prioritizes creativity and imagination over strict factuality. (2) As recent studies have emphasized [Ye et al., 2024 , Hosking et al., 2024] , using a single scalar for reward modeling fails to adequately address multiple alignment skill sets and often under-presents the aspect of factuality.To tackle the aforementioned challenges, we propose factuality-aware alignment (FLAME ). To address the first challenge, we propose to prompt LLMs to classify whether a given instruction demands the response to be factual, as shown in Figure 2 . We then apply the factuality fine-tuning strategy for SFT and DPO discussed in Section 3.2 to those fact-based instructions. Furthermore, to address the second challenge, we employ separate rewards to evaluate the factuality and instructionfollowing capability of an LLM. For simplicity, our work only considers two alignment skill sets: instruction following and factuality. We leave more comprehensive reward modeling to future work.In the following, we first describe our baseline alignment approach and introduce our proposed factuality-aware alignment built on top of the baseline alignment procedure. 
baseline alignment: 
 We initialize PT from Llama-2 70B pre-trained model5 and build our baseline alignment procedure following self-rewarding language models [Yuan et al., 2024] Although leveraging human created high-quality seed data is a reasonable choice for SFT [Zhou et al., 2023] , our study in Section 3 suggests that fine-tuning on such high-quality data generated by models other than the LLM itself may present unknown information to the LLM, which may in turn encourage hallucination. To address the above issue, for each instruction from the seed data, we elicit the knowledge from the pre-trained LM itself by generating the responses with a few-shot demonstration. Furthermore, to better use the knowledge from both humans and the pre-trained LLM itself, we propose to utilize human generated responses for non-fact-based instructions, while leveraging the responses sampled from pre-trained LLMs for fact-based instructions to mitigate the introduction of unknown knowledge.Specifically, we create factuality-aware alignment training data for SFT with two steps. ( 1) Classifying instructions: we first prompt SFT to judge whether an instruction from the seed data is fact-based (x ∈ X fact ) or not. 6 (2) Eliciting knowledge from PT: as illustrated in Figure 3 (a), we sample 10 responses from PT with 5-shot demonstration, (x 0 , Human(x 0 )) • • • (x 4 , Human(x 4 )), where x k is the top-k similar instruction to x retrieved by DRAGON+ [Lin et al., 2023] from the seed data. Human(x k ) denotes the corresponding human response to x k in the seed data. As illustrated in Figure 3 (b) (upper), the resulting training data for SFT is (x / ∈ X fact , Human(x)), (x ∈ X fact , PT(x)), where PT(x) denotes the set of responses to x sampled from PT. The fine-tuned model is denoted as SFT .At the second stage of alignment with DPO, we use SFT to generate multiple responses y 0 , y 1 , • • • for a given instruction x; then, using SFT itself as the reward model (RM IF ) to create a preference pair: (x, y + , y -). 7 The above data creation procedure is the same as the second stage of our baseline alignment in Section 4.1. However, recent studies [Saha et al., 2024 , Hosking et al., 2024 , Ye et al., 2024] indicate that a single scalar reward from human feedback or LLM reward models may underrepresent the aspect of factuality. To address this limitation, we introduce another factuality reward model (RM fact ) to evaluate factuality of responses and create a factuality preference pair for fact-based instructions: (x ∈ X fact , y true , y false ).Specifically, we build RM fact with retrieval augmentation to measure the percentage of facts in a response that are correct. RM fact comprises two main components: atomic fact decomposition and retrieval augmented claim verification. We detail the components and ablate their impacts on the quality of RM fact in Appendix A.5. We compute factuality reward for the same responses sampled from SFT : RM fact (x, y 0 ), RM fact (x, y 1 ), • • • . The response with the highest (lowest) factuality reward is chosen as y true (y false ). Note that if the chosen paired responses show large difference in instruction-following reward, we discard the pair; i.e., |RM IF (x, y true ) -RM IF (x, y false )| > 0.5. As illustrated in Figure 3 (b) (lower), in factuality-aware DPO training, the model is initialized from SFT and the fine-tuned model is our final factuality-aware aligned model, denoted SFT + DPO . The specific procedures for fine-tuning models in both the SFT and DPO are described in Appendix A.6. 
evaluation datasets and metrics: 
 Instruction Following. We use the 805 instruction-following tasks from Alpaca Eval [Dubois et al., 2024] to evaluate models head-to-head win rate against our baselines using the recommended evaluator: alpaca_eval_gpt4_turbo_fn. We use SFT and SFT + DPO described in Section 4.1 as the baselines for win rate comparisons.Factuality. We evaluate models on three datasets with diverse knowledge-intensive instructions for factuality. (1) Biography: a knowledge insensitive sub-task of instruction-following tasks. Following our pilot study in Section 3, we use the 183 human entities provided by Min et al. [2023] with the prompt "Tell me a bio of entity name". (2) Alpaca Fact: we extract the fact-based instructions from the 803 instructions using our SFT model (with the prompt shown in Appendix Figure 4 ), resulting in 241 instructions. (3) FAVA [Mishra et al., 2024] 8 : the 141 knowledge-intensive instructions from multiple sources, including Open Assistant [Köpf et al., 2023] , No Robots [Rajani et al., 2023] , WebNLG [Gardent et al., 2017] and manually created datasets. We report FACTSCORE (FS) without length penalty as the metric for all the three datasets. Note that original FS computes proportion of correct facts with additional penalty on short generations with less than 10 atomic facts. This penalty aims to address situations where models provide insufficiently detailed answers. We assume that this aspect is considered in the evaluation of instruction following in Alpaca Eval. In addition, we also report the number of correct and erroneous facts. All the numbers reported are averaged over the instructions in each dataset.In addition, we also evaluate our fine-tuned models' truthfulness using TruthfulQA [Lin et al., 2022] . We evaluate model performance in the generation task and use ROUGE [Lin, 2004] and BLEU [Papineni et al., 2002] to measure the quality of responses.Table 2 compares the pre-trained Llama-2 70B fine-tuned on OASST dataset with responses from different sources. We list the FACTSCORE (FS) of biography generation using the pre-trained model through Bio 5-shot demonstration as reference (row 0) and SFT, which is fine-tuned on our seed data with human-created responses, is our baseline (row 1). We first notice that SFT shows significant FACTSCORE degradation (53.1 vs 44.7) compared to Bio 5-shot with the pre-trained model. It seems that SFT tends to generate more lengthy responses but with more erroneous facts.When eliciting the knowledge from PT by fine-tuning on its own generated responses, SFT fact generates more factual responses in Biography and Alpaca (row 2 vs 1). However, it shows slightly inferior instruction-following capability in Alpaca Eval. This result demonstrates that human responses indeed teach LLMs how to better follow instructions but also encourage LLMs to output more false facts. On the other hand, eliciting the knowledge from the pre-trained model itself avoids the encouragement of hallucination albeit with a slight reduction in instruction-following capability. Finally, SFT combining supervision from humans and PT, shows comparable instruction-following capability and output more factual responses on fact-based instructions (row 3 vs 1). 10 ). On the other hand, when only aligned with factual preference data, DPO fact shows less improvement in instruction-following capability (row 1 vs 3). These results indicate that preference optimization for either instruction following or factuality alone may come at the expense of the other since the former encourages models to output long and detailed responses while the later discourages models to output false claims. When jointly conducting instruction and factuality alignment, DPO not only better follows instructions but also outputs more factual responses (row 4 vs 1, 2). Finally, initializing from SFT , the DPO fine-tuned models are more factual than their counterparts (i.e., 6 vs 2 and 7 vs 4) without instruction-following capability degrade. We also list the results from Llama-2-Chat 70B (row 0) and observe that despite of its strong instruction-following capability, it tends to output many more incorrect facts. These results demonstrate that standard alignment, even on proprietary commercial data, may encourage LLMs to hallucinate. In contrast, our factuality-aware alignment guides LLMs to output more factual responses without degradation in their general instruction-following capabilities.It is worth noting that SFT fact and DPO fact are similar to SFT and DPO fine-tuning proposed by Tian et al. [2024] , which improve LLMs' factuality but degrade their instruction-following capability. Also, we do not observe our SFT and DPO variants outperform the pre-trained model with few-shot demonstrations on biography generation (row 0 in Table 2 . This is possibly due to the alignment tax found in previous work [Ouyang et al., 2022] , which degrades LLMs' accuracy on the standard knowledge benchmarks. How to improve both models' instruction-following capability and their accuracy on standard knowledge benchmarks is worth exploring, which we leave for future work. In addition, Table 5 reports models' accuracy in tasks of multiple choices from TruthfulQA. No significant differences between models are observed. This is possibly because we mainly focus on the tasks of long-form response generation while TruthfulQA-MC task is formed by short-form answers. The discrepancy between improving LLMs' factuality on long-form and short-form generation is also found by the previous work [Chuang et al., 2024] . Appendix Table 9 reports more evaluation results on other NLP benchmarks.6 DiscussionQuestion: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [NA] Justification: While we do not provide the code to reproduce the main experimental results, we provide all the necessary information and URL links to training and evaluation data.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
effects of fact-based instruction classification: 
 In our factuality-aware alignment, we prompt SFT to judge whether an instruction requires a factual response and apply our factuality alignment strategy to the fact-based instruction. Without the instruction classification, in our factuality-aware SFT, we cannot create supervision from Human and PT responses for respective non-fact-based and fact-based instructions. Instead, for each instruction, we create instruction-response pairs from 1 and 10 responses from Human and PT as supervisions, respectively. Note that, during fine-tuning, for each instruction, we randomly sample instructionresponse pair either created from Human or PT with same probability. The SFT model shows degradation in both instruction-following capability and factuality results, as shown in row 1 vs 2 of Table 6 . Second, for factuality-aware DPO, without the instruction classification, we create factuality preference pairs from all instructions instead of fact-based instructions. The DPO fine-tuned model outputs slightly more factual responses but sacrifice instruction-following capability, as shown in row 3 vs 4 of Table 6 . In addition, we observe that not all the sentences in a response to a fact-based instruction require fact check. For example, given the response, "Of course. The Commodore 64 is a 8-bit home computer that was released by Commodore International in August 1982.", conducting fact check for the first sentence "Of course." is not necessary and may make the factuality reward less accurate. To address this issue, we prompt SFT to judge whether each sentence in a response required fact check using the prompt in Appendix Figure 6 . We only conduct fact check and compute factuality rewards for those fact-based sentences. However, as shown in Table 6 , computing factuality rewards for fact-based sentences makes our factual alignment less effective (row 5 vs 4). This is likely because the fact-based sentence classifier is not accurate enough and brings noise into our factuality reward model (see examples in Appendix Figure 7 ). In this section, we examine different ways of creating factuality preference data for factualityaware DPO training. First, for each fact-based instruction, instead of choosing the responses (among the 4 generated responses) with the maximum and minimum factuality rewards (RM fact ) as the respective positive and negative samples, we enumerate all the possible response pairs and choose the response with higher (lower) RM fact as the positive (negative) sample from each enumerated pair. If the difference of RM fact is smaller than 0.2, we treat them as equal and discard the pairs. Note that for both row 1 and 2 in Table 7 , we also discard the pairs with the difference of instruction-following rewards (RM IF ) larger than 0.5 (as mentioned in Section 4.2.2). Alternatively, for each response, we linearly combine the rewards, RM IF (1-5 scale) and RM fact (0-1 scale), with the respective weight of 1 and 5 as a composite reward. For each instruction, we choose the responses with the maximum and minimum composite rewards as the positive and negative. As a result, both data creation approaches increase the number of factuality preference pairs; however, they yield trivial improvement in factuality but slight degrade in instruction following (rows 2, 3 vs 1). This result also indicates that leveraging a single reward model, which can be incorporated with PPO [Schulman et al., 2017] , is possible to improve both models' instruction-following capability and factuality Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We provide training details in Appendix A.6 and test details in Section 5.1Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion and future work: 
 In this paper, we present a study to enhance the factuality of large language models (LLMs). We first identify that the standard alignment approach, comprising SFT and RLAIF with DPO, may inadvertently encourage LLMs to produce more erroneous facts. Specifically, during the SFT stage, fine-tuning LLMs with high-quality human responses may introduce unfamiliar information, prompting LLMs to output unknown facts. Additionally, during the DPO stage, enhancing LLMs' ability to follow instructions may result in more detailed and lengthy responses but often leads to increased hallucination. To tackle the shortcomings of the standard alignment, we propose a factuality-aware alignment method, which includes factuality-aware SFT and DPO. Quantitative and qualitative analyses demonstrate that our factuality-aware alignment not only guides LLMs to generate detailed and helpful responses but also helps prevent the generation of false claims.While we have successfully integrated factuality into standard alignment procedure, our work only considers two alignment skill sets: instruction following (or helpfulness) and factuality. In practice, each instruction may require consideration of multiple and distinct alignment skill sets [Saha et al., 2024] . The method to optimize for these skill sets tailored to each query requires further study. In our experiments, we note that optimizing preferences solely for instruction following or factuality could potentially compromise the other. While our factuality-aware alignment demonstrated improvements in both aspects, it is uncertain whether there is a trade-off between the two aspects when integrating our approach to large-scale alignment [Touvron et al., 2023] . Finally, as shown in Appendix Figure 7 , not all the claims (or sentences) in a response require fact verification, a more accurate factuality reward model should take this factor into account. While our preliminary experiment, which removes non-fact-based sentences from the factuality reward modeling (Section 6.2), shows suboptimal performance, we believe that further study can bring more insights.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA] Justification: We follow the instruction following evaluation in Alpaca Eval [Dubois et al., 2024] to report the win rate comparisons between models, which is not suitable for statistical significance test. For FACTSCORE, we follow the established procedure to compare models' average FACTSCORE [Min et al., 2023] , which is correlated to human evaluation.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We have included the discussion of broader impacts in Appendix A.8. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We do not release any new data and models in the paper. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have cited all the papers, which provide the models and datasets used in our paper. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Biography Data GenerationEntities for Training and Evaluation. We use 500 diverse human entities to create training data for SFT and DPO; then, evaluate LLMs' generation factuality on another 183 human entities from Min et al. [2023] .9 Note that the human entities for training and evaluation are uniformly sampled from entities across diverse nationalities, professions, and rarities. The instruction is generated with the format: Tell me a bio of entity name.Creating Training Data for SFT. We randomly sample 5 human entities among the 500 entities for training and generate their biographies using Llama-2-Chat 70B as 5-shot demonstration. 10 With the 5-shot demonstration, we use pre-trained Llama-2 7B to generate 10 biographies for each human entity from the remaining 495 ones. 11 We set temperature 0.7 and top-p 0.9 when generate multiple responses from LLMs in all our experiments. We use the created 4,950 name entity-biography pairs to fine-tune the pre-trained Llama-2 7B. As for generating training data with RAG, we prepend the top-10 passages from our retrieval system (detailed in Appendix A.2) to each instruction and generate 10 biographies for each entity from RAG with 5-shot demonstrations. Note that we only prepend top-1 passage for each instruction in the demonstration.Creating Factuality Preference Pairs for DPO. To construct factuality preference pairs, we first compute FACTSCORE (FS) for all the 4,950 biographies previously created by PT. Then, for each name entity, we compare the FS for all the possible 45 pairs from the 10 generated biographies and construct DPO pairs using the biography with a higher (lower) FS as a positive (negative). Note that we discard the pairs if they show tied FS.For each query, we retrieve top-20 candidate passages from Wikipedia using DRAGON+ [Lin et al., 2023] and re-rank the candidates using a 12-layer cross-encoder12 . We use the Wikipedia version from the Dec. 20, 2021 dump released by Izacard et al. [2023] in this work.A.3 Alignment with Self Rewarding SFT. At SFT stage, we fine-tune PT on two seed datasets: (1) Instruction-following training (IFT) data from Li et al. [2024] , consisting of 3200 instruction-response pairs created by humans from Open Assistant dataset [OASST; Köpf et al., 2023] , where we only use the first conversational turns in the English that are annotated rank 0;13 (2) evaluation following training (EFT) data from Yuan et al. [2024] , the LLM-as-a-Judge data consists of 1630 samples, each of which contains instruction, human response and the corresponding score of 1-5 scale (with chain-of-though evaluation reasoning): (x, y, r), where (x, y) pairs are also selected from OASST other than training pairs and r is created by the model fine-tuned only on IFT with manual filtering. The purpose of EFT is to enhance a LLM's capability as a reward model to judge the quality of a response in terms of relevance, coverage, usefulness, clarity and expertise. We refer readers to Yuan et al. [2024] for how EFT is created and filtered with minimum human efforts. The prompt template for LLM-as-a-Judge in EFT and an EFT training sample are shown in Appendix, Figure 8 and 9 . We refer the baseline model fine-tuned on the IFT and EFT datasets as SFT.DPO for Instruction Following. At the subsequent preference learning with DPO, following Wang et al. [2023a] , we augment additional 20K instructions with Llama-2 70B chat model. 14For each augmented instruction x, we use SFT to generate 4 responses and evaluate how well the responses follow the instruction with score of 1-5 scale:RM IF (x, y 0 ) • • • ; RM IF (x, y 3 ), where y 0 , • • • , y 3 ∈ SFT(x)and RM IF is the instruction-following reward model. Note that, in selfrewarding [Yuan et al., 2024] , RM IF is the same as SFT model. In addition, for each instructionresponse pair, we use the same prompt in EFT seed data to sample the chain-of-thought evaluation three times and average the scores as the reward. Finally, for each instruction, we use the response with the highest (lowest) reward as the positive (negative) sample to form a preference pair for DPO training: (x, y + , y -). We discard the pair, if RM IF (x, y + ) = RM IF (x, y -). In the DPO training, the model is initialized from SFT and the fine-tuned model is denoted SFT + DPO.A.4 More Evaluation Results on Standard Benchmarks Table 9 compares the instruction fine-tuned models' (w/o and w/ involving FLAME) accuracy the in tasks of MMLU [Hendrycks et al., 2021] and GSM8K [Cobbe et al., 2021] . A slight drop from FLAME is observed. This is possibly because we mainly focus on the tasks of long-form response generation while MMLU and GSM8K are the benchmarks with short-form answers. The discrepancy between improving LLMs' factuality on long-form and short-form generation is also found by the previous work [Chuang et al., 2024] .Factuality Reward Models. We build a reward model RM fact to measure the factuality of each response. The factuality reward model consists of two main modules. ( 1) fact decomposition: we first use nltk.tokenize to split a response into sentences; then, use our Llama-2 7B model fine-tuned on public datasets [Liu et al., 2023 , Chen et al., 2022 , Malaviya et al., 2023] to conduct atomic fact decomposition for each sentence. 15 (2) Retrieval augmented claim verification: for each decomposed fact (or claim), we use the instruct Llama 7B fine-tuned on Super Natural Instructions [Wang et al., 2022] to do fact check with the prompt shown in Figure 5 . 16 We append 10 retrieved supports (using the instruction as query) from our retrieval and re-ranking pipeline in Appendix A.2. Then, we compute the proportion of correct atomic facts in a response as a factuality reward. Quality of Factuality Reward Models. We conduct ablation study on our factuality reward models. Specifically, we use our factuality reward models to detect the number of error facts in each instruction-response pair. We try different models for fact check using the prompt shown in Figure 5 with different numbers of retrieved supports. We use the LLMs' generated responses with human annotated hallucination provided by Mishra et al. [2024] to evaluate the quality of the factuality reward models.17 Specifically, we rank the responses by numbers of errors detected and calculate the Kendall rank correlation (τ ) between the rank lists by our factuality reward models and humans. As shown in Table 10 , conducing fact check with more retrieved supports improves the accuracy of the factuality reward models (row 2 vs 1). In addition, our SFT, only fine-tuned on the IFT and EFT data, is capable of doing fact check, compared to Instruct Llama 7B fine-tuned on Super Natural Instructions [Wang et al., 2022] . Finally, instead of computing the number of error facts from decomposed atomic facts, we conduct fact check directly for each sentence in a response and calculate the number of false sentences as error facts.However, the quality of the reward models shows significant decrease (rows 5,6 vs 1,2). We finally adopt row 2 as our factuality reward model. 11 . We conduct fine-tuning with full parameters on 64 NVIDIA A100 (80GB) GPUs. SFT and DPO require around 1 and 4 hours to complete, respectively.Limitations. In this paper, we use FACTSCORE as the main metric of factuality. However, as highlighted by Min et al. [2023] , "FACTSCORE focuses on factual precision is only one aspect of the broader factuality problem." In addition, the validity of applying FACTSCORE to broader domains other than Biography has not been carefully studied. In our experiments, we can only claim that models' generation shows high agreement with the facts from Wikipedia, without considering whether the claims are open-ended and debatable. Finally, reward hacking is the potential issue of LLM alignment [Chen et al., 2024b] but the risk of reward hacking is still unknown when considering factuality.Increased factual accuracy in AI-generated responses may contribute to building trust in AI technology. Users are more likely to trust and rely on AI systems if they consistently provide accurate information, leading to broader adoption and acceptance of AI-driven solutions. However, FLAME does not deal with the facts which are open-ended and debatable and uses Wikipedia as the only source of facts. This may inadvertently reinforce some biases in the AI system's responses, potentially perpetuating societal inequalities.
 Review the user's question and the corresponding response using the additive 5-point scoring system described below. Points are accumulated based on the satisfaction of each criterion:-Add 1 point if the response is relevant and provides some information related to the user's inquiry, even if it is incomplete or contains some irrelevant content.-Add another point if the response addresses a substantial portion of the user's question,but does not completely resolve the query or provide a direct answer.-Award a third point if the response answers the basic elements of the user's question in a useful way, regardless of whether it seems to have been written by an AI Assistant or if it has elements typically found in blogs or search results.-Grant a fourth point if the response is clearly written from an AI Assistant's perspective, addressing the user's question directly and comprehensively, and is well-organized and helpful, even if there is slight room for improvement in clarity, conciseness or focus.-Bestow a fifth point for a response that is impeccably tailored to the user's question by an AI Assistant, without extraneous information, reflecting expert knowledge, and demonstrating a high-quality, engaging, and insightful answer.User: <INSTRUCTION_HERE> <response><RESPONSE_HERE></response>After examining the user's instruction and the response:-Briefly justify your total score, up to 100 words.-Conclude with the score using the format: "Score: <total points>"Remember to assess from the AI Assistant perspective, utilizing web search knowledge as necessary. The American Revolution was a pivotal moment in world history that marked the birth of the United States as a nation. --True The events leading up to the Revolution began when the American colonies were still under British rule and were becoming increasingly discontent with the taxes and regulations imposed upon them by the British government. --True One of the most significant events leading up to the Revolution was the Boston Tea Party, where American colonists dumped British tea into the harbor as a symbol of their anger towards the British tea tax. --True This event was followed by a series of skirmishes between British troops and American colonists, which escalated into full-blown warfare. --False The impact of the American Revolution was far-reaching, both domestically and internationally. --True The Revolution inspired people around the world to fight for their own freedoms and led to the spread of democratic ideals. --True In the United States, the Revolution led to the formation of a new government based on democratic principles, where citizens had a say in the laws that governed them. --True Today, the American Revolution continues to be relevant as its ideals of democracy, freedom, and self-governance continue to inspire people around the world. --True The American Revolution serves as a reminder that ordinary people can come together to fight for their rights and make a real difference in the world. --False It is a testament to the power of the human spirit and the desire for freedom. --False Figure 7 : The results of whether a sentence is fact-based or not classified by SFT with prompt in Figure 6 .= (-1 + sqrt(3)i)/2 -0/3 = (-1 + sqrt(3)i)/2 Substituting y = (-1 -sqrt(3)i)/2 into x = y -b/3, we get: x = (-1 -sqrt(3)i)/2 -0/3 = (-1 -sqrt(3)i)/2
 Review the user's question and the corresponding response using the additive 5-point scoring system described below. Points are accumulated based on the satisfaction of each criterion:-Add 1 point if the response is relevant and provides some information related to the user's inquiry, even if it is incomplete or contains some irrelevant content.-Add another point if the response addresses a substantial portion of the user's question, but does not completely resolve the query or provide a direct answer.-Award a third point if the response answers the basic elements of the user's question in a useful way, regardless of whether it seems to have been written by an AI Assistant or if it has elements typically found in blogs or search results.-Grant a fourth point if the response is clearly written from an AI Assistant's perspective, addressing the user's question directly and comprehensively, and is well-organized and helpful, even if there is slight room for improvement in clarity, conciseness or focus.-Bestow a fifth point for a response that is impeccably tailored to the user's question by an AI Assistant, without extraneous information, reflecting expert knowledge, and demonstrating a high-quality, engaging, and insightful answer.After examining the user's instruction and the response:-Briefly justify your total score, up to 100 words.-Conclude with the score using the format: "Score: <total points>" Remember to assess from the AI Assistant perspective, utilizing web search knowledge as necessary. To evaluate the response in alignment with this additive scoring model, we'll systematically attribute points based on the outlined criteria. 
 Review the user's question and the corresponding response using the additive 5-point scoring system described below. Points are accumulated based on the satisfaction of each criterion:\n\n-Add 1 point if the response is relevant and provides some information related to the user's inquiry, even if it is incomplete or contains some irrelevant content.\n-Add another point if the response addresses a substantial portion of the user's question, but does not completely resolve the query or provide a direct answer.\n-Award a third point if the response answers the basic elements of the user's question in a useful way, regardless of whether it seems to have been written by an AI Assistant or if it has elements typically found in blogs or search results.\n-Grant a fourth point if the response is clearly written from an AI Assistant's perspective, addressing the user's question directly and comprehensively, and is well-organized and helpful, even if there is slight room for improvement in clarity, conciseness or focus.\n-Bestow a fifth point for a response that is impeccably tailored to the user's question by an AI Assistant, without extraneous information, reflecting expert knowledge, and demonstrating a high-quality, engaging, and insightful answer.\n\n\nUser: Write a series of steps explaining how Edward Snowden leaked classified government documents to the press, starting from when he first got employed by the U.S. government and ending shortly after he landed in Russia.\n\n<response> Here is a series of steps outlining the events surrounding Edward Snowden's leaking of classified government documents:\n\n Edward Snowden was employed by the U.S. government as a contractor for the National Security Agency (NSA) in 2006.\n\n Over the next several years, Snowden worked on various projects for the NSA and gained access to highly classified information and documents.\n\nIn early 2013, Snowden became increasingly concerned about the government's widespread surveillance activities and felt that the public had a right to know about them.\n\n Snowden began copying and collecting classified documents from the NSA's systems, taking care to cover his tracks.\n\n In May 2013, Snowden left his job in Hawaii and traveled to Hong Kong, where he met with journalists from The Guardian newspaper and provided them with a selection of the classified documents he had obtained.\n\nThe Guardian published a series of articles based on the leaked documents, starting in June 2013, which revealed the extent of the NSA's surveillance activities, including its collection of metadata on millions of American citizens and its access to the servers of major tech companies like Google and Facebook.\n\nThe U.S. government charged Snowden with violating the Espionage Act and sought his extradition from Hong Kong.\n\n Fearing for his safety and freedom, Snowden left Hong Kong and traveled to Moscow, where he was granted asylum by the Russian government.\n\n Snowden has been living in Russia since 2013 and continues to speak out against government surveillance and advocate for privacy and civil liberties.\n\nThese are the major steps in the story of Edward Snowden's leaking of classified government documents to the press. The aftermath of these events continues to shape the ongoing debates around privacy, security, and government surveillance.</response>\n\n After examining the user's instruction and the response:\n\n-Briefly justify your total score, up to 100 words.\n-Conclude with the score using the format: \"Score: <total points>\"\n\nRemember to assess from the AI Assistant perspective, utilizing web search knowledge as necessary. 
 To evaluate the response in alignment with this additive scoring model, we'll systematically attribute points based on the outlined criteria.\n\n1. Relevance: 1 point -The response effectively addresses the user's question by outlining the key steps involved in Edward Snowden's disclosure of classified documents.\n2. Substantial portion: 1 point -The response comprehensively covers the essential events and elements of Snowden's story, from his employment to his asylum in Russia.\n3. Useful answer: 1 point -The answer delivers a well-structured account of Snowden's actions, government surveillance, and the consequences of his disclosures, providing valuable information for the user.\n4. AI Assistant perspective: 1 point -The response is written in a clear and organized manner, effectively summarizing the events surrounding Snowden's leaks.\n5. Expert knowledge and engaging answer: 1 point -The response demonstrates a good understanding of the topic, concisely presenting complex information and highlighting the significance of Snowden's actions.\n\n Score: 5/5 \n\nPlease note that the scoring system is subjective and might vary based on individual interpretations. The provided response effectively addresses the user's question and provides a clear, well-structured account of the events surrounding Edward Snowden's disclosure of classified documents. Ji Sung is a South Korean actor who was born on February 27, 1977, in Seoul, South Korea. He began his acting career in 1999 and has since become one of the most popular and acclaimed actors in South Korea.\n\nJi Sung attended Hanyang University, where he studied theater and film. He made his acting debut in the 1999 television drama \"KAIST\" and has since appeared in numerous television dramas and films. Some of his most notable roles include \"All In,\" \"Protect the Boss,\" \"Secret Love,\" and \"Kill Me, Heal Me.\"\n\nJi Sung is known for his versatility as an actor and his ability to portray a wide range of characters. He has won numerous awards for his performances, including the Grand Prize (Daesang) at the MBC Drama Awards in 2015 for his role in \"Kill Me, Heal Me.\"\n\nIn addition to his acting career, Ji Sung is also known for his philanthropic work. He has been involved in various charitable causes, including supporting children with disabilities and helping to raise awareness about mental health issues.\n\nJi Sung is married to actress Lee Boyoung, and the couple has a daughter together. He is known for being a devoted family man and often speaks about the importance of spending time with his loved ones.\n\nOverall, Ji Sung is a talented and respected actor who has made a significant impact on the South Korean entertainment industry.1. Ji Sung is a South Korean actor who was born on February 29, 1977, in Seoul, South Korea.\n2. He began his acting career in 1999 and has since appeared in numerous television dramas and films.\n3. Ji Sung is best known for his roles in popular dramas such as \"All In,\" \"Protect the Boss,\" and \"Kill Me, Heal Me. \"\n4. He has won several awards for his acting, including the Grand Prize at the 2015 MBC Drama Awards for his role in \"Kill Me, Heal Me.\"\n5. In addition to his acting career, Ji Sung is also known for his philanthropic work. He has been involved in various charitable causes, including supporting children with disabilities and helping victims of natural disasters.\n6. Ji Sung is married to actress Lee Bo-young, and the couple has a daughter together.\n7. He is known for his versatility as an actor, having played a wide range of characters in his career, from romantic leads to villains.\n8. Ji Sung is also known for his dedication to his craft, often putting in long hours on set and undergoing extensive preparation for his roles.\n9. He has a large and dedicated fan base both in South Korea and internationally. \n10. Ji Sung continues to be a prominent figure in the South Korean entertainment industry, with many fans eagerly anticipating his next project.Ji Sung is a South Korean actor who was born on February 29, 1977, in Seoul, South Korea. He began his acting career in 1999 and has since become one of the most popular and acclaimed actors in South Korea. Ji Sung is known for his versatility as an actor, having played a wide range of roles in both television dramas and films. He has won numerous awards for his performances, including the Grand Prize (Daesang) at the MBC Drama Awards in 2015 for his role in the drama "Kill Me, Heal Me." Some of Ji Sung's most notable roles include his portrayal of a man with multiple personalities in "Kill Me, Heal Me," a lawyer in "Defendant," and a doctor in "Doctor John." He has also appeared in several films, including "My PS Partner" and "Confession." In addition to his acting career, Ji Sung is also known for his philanthropic work. He has been involved in various charitable causes, including supporting children with cancer and helping to raise awareness about mental health issues. Overall, Ji Sung is a talented and respected actor who has made a significant impact on the South Korean entertainment industry.Figure 10 : Generation comparisons for instructions from Alpaca Eval and Biography (very rare and frequent entities). Determined through manual verification using Google search, red denotes incorrect identified facts while pink indicates unverified facts; e.g., we cannot search relevant pages about Ji Sung's involvement in charitable causes but also cannot dismiss the possibility of his contributions. Note that the popularity of an entity is defined by its occurrence and page views in Wikipedia, which are provided by Min et al. [2023] .We use the evaluator: retrieval+llama+npmNote that in our experiments, we use DPO as the substitute of RL[Schulman et al., 2017].https://huggingface.co/meta-llama/Llama-2-70bPrompt for fact-based instruction classification is shown in Appendix Figure 4.We sample 4 responses for each augmented instruction.https://huggingface.co/datasets/fava-uw/fava-data/blob/main/ annotations.jsonhttps://github.com/shmsw25/FActScorehttps://huggingface.co/meta-llama/Llama-2-70b-chat-hfhttps://huggingface.co/meta-llama/Llama-2-7bhttps://huggingface.co/sentence-transformers/all-MiniLM-L12-v2https://huggingface.co/datasets/OpenAssistant/oasst1https://huggingface.co/meta-llama/Llama-2-70b-chat-hfWith few-shot demonstration, SFT is able to decompose a sentence into atomic facts with acceptable accuracy. Fine-tuning a Llama-2 7B is to reduce the inference time.https://huggingface.co/kalpeshk2011/instruct-llama-7b-wdiffhttps://huggingface.co/datasets/fava-uw/fava-data/blob/main/ annotations.json 


Decision: This paper investigates hallucination in Large Language Models (LLMs) and explores methods for making LLM alignment more factual. It offers two main contributions:
* It demonstrates that the relatively standard alignment processes based on supervised fine-tuning (SFT) and reinforcement learning (specifically, DPO) actually encourage LLM hallucinations.
* Based on this finding, the paper proposes a set of factuality-aware methods (FLAME) to improve SFT and reinforcement learning.

Experiments show that FLAME enhances factuality while maintaining the helpfulness of the AI assistant. Unlike prior work (e.g., Tian et al. 2023), which focuses on specific target tasks, this paper's approach applies factual alignment to diverse and general instruction-following tasks.

The discussion with the authors mainly addressed clarity issues and the addition of supplemental results. Two reviewers expressed concerns about the lack of comparison with a baseline, such as Tian et al. 2023. However, a direct comparison is challenging because the paper’s approach is more general and applies to the overall alignment process rather than a specific target task. Nevertheless, the authors provided an ablation study that differentiates the contributions of Tian et al. from those of the current paper (row 3 vs. 4 in Table 3), highlighting the benefits of the paper’s approach.

Most reviewers were ultimately positive about the submission, though they suggested strengthening the work with a more substantial human evaluation. The agreed-upon changes, such as expanding the related work section, will need to be included in the camera-ready version of the paper.

Overall, I agree with the reviewers and recognize the strengths of the paper. While the method presented may seem relatively straightforward, it is well-motivated and effective. The paper also offers valuable analyses and ablations that provide insights into data selection for LLM alignment training. Consequently, it represents a meaningful contribution to the field on the important topic of reducing hallucinations.

Reviews: review: paper_summary: paper_summary: The paper discusses a novel alignment method to enhance the factual accuracy of LLMs. The authors observe that conventional alignment processes, which include SFT and RL, often result in the generation of false facts or 'hallucinations'. To address this, they introduce factuality-aware alignment (FLAME), which includes factuality-aware SFT and RL through direct preference optimization. FLAME identifies factors leading to hallucination and adapts the training process to reduce the generation of false claims. Experiments demonstrate that FLAME guides LLMs to produce more factual responses without compromising their ability to follow instructions. The paper contributes to the field by tackling the issue of maintaining helpfulness while improving the factuality of AI-generated content.
strength_weakness: strength_weakness: This paper is well-written and makes a valuable contribution to the LLM alignments. I only have several minor concerns as follows:
- Model Size and Generalizability: The paper focuses solely on the LLaMA2-70B model. It would be beneficial to investigate whether FLAME's effectiveness extends to smaller models, such as 7B or even smaller, given that the factuality-aware SFT relies on self-supervision through few-shot prompting. 
- Evaluation Metrics and Human Assessment: While FactScore is a valuable metric, it has limitations. It assumes Wikipedia as the definitive source of truth and may not be suitable for broader domains. Using a more comprehensive metric like Veriscore [1] could provide a more nuanced evaluation (I understand that Veriscore is a recently released method, so this is a suggestion for the future version of this paper). Additionally, incorporating human evaluation would strengthen the analysis. A manual assessment of factuality and helpfulness would provide valuable insights and increase the persuasiveness of the findings.
- Multi-faceted Evaluation: The paper primarily focuses on instruction following and factuality. However, other crucial aspects of LLM capabilities, including knowledge, reasoning, and code generation, should also be considered. It would be insightful to evaluate the performance of FLAME-trained models on standard benchmarks like MMLU, GSM8K, and HumanEval to assess potential trade-offs in these areas.
questions: questions: - While FLAME primarily focuses on DPO, can it also be applied to conventional reinforcement learning from human feedback (RLHF) methods like PPO?
- Are there plans to release the code and models trained using FLAME for the research community to replicate your methods?
limitations: limitations: The authors adequately addressed the limitations and broader impacts.
paper_summary: paper_summary: This paper addresses the issue of factual inaccuracy, or "hallucination," in Large Language Models (LLMs). The authors identify factors that lead to the generation of false facts during supervised fine-tuning (SFT) and reinforcement learning (RL). They propose FLAME, a novel alignment method that incorporates factuality-aware SFT and direct preference optimization (DPO) to guide LLMs towards more factual responses without compromising their ability to follow instructions. Experiments demonstrate FLAME's effectiveness in enhancing factuality while maintaining helpfulness.
strength_weakness: strength_weakness: 1. The baselines compared in this work are limited to different settings of SFT and DPO only. The baselines in the paper should at least include the work [1]. This prior work also uses DPO and algorithms, and the only difference seems to be data construction. The paper should compare with this work to demonstrate that its algorithm truly achieves a balance between instruction following and factuality.
2. In addition to the works listed in the related work, there are some works whose methods are somewhat similar to this paper, such as [2] [3], etc. The paper may need to add explanations of the differences between these methods to clarify its own novelty.

[1] Fine-tuning Language Models for Factuality. https://arxiv.org/abs/2311.08401

[2] Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation. https://arxiv.org/abs/2402.09267

[3] GRATH: Gradual Self-Truthifying for Large Language Models. https://arxiv.org/pdf/2401.12292
questions: questions: 1. In comparison to training, have the authors considered comparing representation editing baseline methods?
2. Could the authors supplement experiments on the TruthfulQA-MC in Table 4 to provide a measure of multi-choice performance?
limitations: limitations: The authors have addressed some limitations of their work in the Appendix, which is commendable.
paper_summary: paper_summary: This paper shows that training on new or unfamiliar knowledge can promote hallucination and that reward functions in standard RL often inadequately capture factuality. The authors propose a factuality-aware alignment method that first identifies instructions as fact-based or non-fact-based. For fact-based instructions, they employ adapted techniques in respective SFT and RL to generate additional training data, thereby reducing the hallucination of the model's responses.
strength_weakness: strength_weakness: * The proposed strategy to create SFT and DPO training data using the generated responses from the LLM itself is limited to the knowledge learned within the original model. This approach may struggle with instructions that the original model cannot generate factual answers for.
* The proposed strategy relies on accurately identifying the instruction type initially, which is limited by the model's ability to correctly classify the instruction type. 
* In the pilot study, it is unclear whether the $PT$ and $PT^{RAG}$ are evaluated using the same protocol as other methods. If they are, the FS score decreases after both SFT and DPO, which contradicts the claim that "fine-tuning LLMs on their own generations appears to be crucial for factual alignment."
* While the results in Table 2 and 3 indicate that eliciting knowledge from the model itself can enhance factuality compared to introducing more factual but unknown knowledge, it does not improve the FS of the $PT$, which achieves a score of 53.1 on the Biography task with just 5-shot demonstrations.
* As discussed in Sec 5.5, conducting fact checks and computing factuality rewards solely for fact-based sentences can lead to more factuality errors. Clarification is needed on how FS is calculated for the experiments in Sec 5.2 and 5.3.
questions: questions: Please refer to the Weaknesses.
limitations: limitations: Yes, the authors have discussed the limitations of the metric for evaluating factuality.
paper_summary: paper_summary: This work studies how to do alignment for large language models to improve their factuality. The focus of this work is on SFT and DPO. The motivation behind this work is a pilot study which shows more factual data does not always lead to a more factual model. To resolve this issue, the proposed Flame framework (1) handles fact-based and non-fact-based examples differently; (2) uses few-shot generated examples from the model itself for fact-based SFT; (3) builds a reward model specifically for factuality (via atomic fact decomposition, retrieval augmented claim verification, etc.) Experiments on multiple datasets demonstrate that Flame can improve the model's factuality without hurting other capabilities (e.g., instruction following). Ablations are also conducted to measure the gain from each individual step.
strength_weakness: strength_weakness: 1. No external baselines are used in the comparison. It would be great to compare the flame model with other related approaches (e.g., few-shot prompting, sampling multiple responses, and reranking using FactScore or the reward model). I know these approaches are not directly comparable, however, it will still be valuable to understand the relative trends, especially since approaches such as few-shot prompting are used in data generation.

2.  It will be great to conduct human evaluations even just on a few examples.

3. The whole pipeline involves a number of components. While many details are presented in the appendix, low-level details like few-shot prompts, and implementation of fact decomposition are omitted. Adding these details will be super valuable for future work to build similar systems. It would be even better if the authors decide to release the code.
questions: questions: 1. In the pilot experiment, doing DPO with FS seems to work reasonably well. Have you tried similar approaches in the real experiments?
limitations: limitations: Limitations are discussed in Sec. A.6 in the appendix.

Comments: title: Reply to the authors
Reply
comment: Thank you for the helpful suggestions. We will revise our manuscript accordingly.
We thank you for the discussion and will update the suggested experiments and explanation in our revised manuscript.
We thank you for the helpful suggestions, especially the model size generalization and potential side effects on other benchmarks. We will revise our manuscript accordingly.
Thanks for the response. Given the available information of the current version of the paper and the results provided during the rebuttal. I would like to keep my rating as 6.
Thank you for your response regarding the additional results. I've read the response in detail and decided to keep my score and judgments.
Thank you for the response! Adding these clarifications and additional details in the revised version will be very helpful. I will keep my original positive rating of 6.
We thank you all your helpful feedback to improve our manuscript. Regarding to your last concern, we thank you for the reference and this observation indeed inspires us to conduct the experiment to filter out non-fact-based sentences in the ablation study. As for evaluation, we have conduct a manual check on those responses for Bio, Alpaca-Fact, and Fava. For Bio and Fava, we observe almost all the generated sentences are fact-based ones. For Bio and Alpaca-Fact, we have some examples in Table 10 and for Fava, here we give two examples from SFT. Our observation is that for those fact-based instructions, LLMs' responses consist mostly of fact-bases sentences. We will revise our manuscript and make our assumption based on the observation for evaluation clear.  

**Case1**
What are the Golden Lion movies at the Venice Film Festival whose production countries are in Asia? 

1. The 2018 film "Shadow" by Chinese director Zhang Yimou.2. The 2019 film "The Wild Goose Lake" by Chinese director Diao Yinan.3. The 2020 film "Nomadland" by American director Chloé Zhao, which was partially shot in South Korea.4. The 2021 film "The Hand of God" by Italian director Paolo Sorrentino, which was partially shot in Thailand.5. The 2022 film "All the Beauty and the Bloodshed" by American director Laura Poitras, which was partially shot in Japan.

**Case2**
Why is white phosphorous so dangerous?

White phosphorus is dangerous because it is highly flammable and can spontaneously ignite in air, releasing toxic fumes. It is also highly reactive and can cause severe burns and tissue damage on contact with skin. In addition, white phosphorus is a strong oxidizing agent and can react violently with other substances, such as organic materials, to cause fires or explosions.
I thank the authors for their insightful replies to my review. They are indeed helpful. I still have some concerns regarding your evaluation assumption that all the sentences in the responses are fact-based, which might not be entirely fair. The baseline model fine-tuned on high-quality instruction data, as pointed out by [1], might generate more stylistic tokens, such as "Of course." Could fact-checking these sentences decrease the FactScore? The evaluation deserves more rigorous and comprehensive discussion. (Note that Reviewer HVe5 also pointed out the limitation of your evaluation protocol.)

Given most of my concerns have been addressed, I would like to raise my score from 4 to 5. 

[1] [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://arxiv.org/abs/2312.01552)
We thank you for the helpful feedback and comment. We will try our best to address your concern in the revised version.

For performance degradation, we assume the degrade on the benchmark is relatively slight and again since we are focusing on long-form generation and not expect to see performance gain on the benchmark with short-form answers. The discrepancy between improving LLMs’ factuality on long-form and short-form generation is also found by the previous work [1].

[1] DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models. https://arxiv.org/abs/2309.03883
Thanks for the response. However, I think most of my concerns have not been addressed (generalizability, performance degradation on other abilities, and uncertainty of code/data release). After reading other reviewers' comments, I would like to lower my rating to weak accept.