FLAME : Factuality-Aware Alignment for Large Language Models
Authors: Sheng-Chieh Lin
Luyu Gao
Barlas Oguz
Wenhan Xiong
Jimmy Lin
Wen-tau Yih
Xilun Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: large language models
factuality
alignment
Primary area: natural_language_processing
TL;DR: We find that the standard alignment process encourages hallucination, and propose factuality-aware alignment while maintaining the LLM's general instruction-following capability.

Abstract: Alignment is a procedure to fine-tune pre-trained large language models (LLMs) to follow natural language instructions and serve as helpful AI assistants. 
We have observed, however, that the conventional alignment process fails to enhance the factual accuracy of LLMs, and often leads to the generation of more false facts (i.e., *hallucination*). 
In this paper, we study how to make the LLM alignment process more factual, by first identifying factors that lead to hallucination in both alignment steps: supervised fine-tuning (SFT) and reinforcement learning (RL).
In particular, we find that training the LLM on new or unfamiliar knowledge can encourage hallucination.
This makes SFT less factual as it trains on human-labeled data that may be novel to the LLM. 
Furthermore, reward functions used in standard RL often inadequately capture factuality and favor longer and more detailed responses, which inadvertently promote hallucination.
Based on these observations, we propose *FactuaLity-aware AlignMEnt*, comprised of *factuality-aware SFT* and *factuality-aware RL* through direct preference optimization. 
Experiments show that our proposed *FLAME* guides LLMs to output more factual responses while maintaining their instruction-following capability.

FLAME : Factuality-Aware Alignment for Large Language Models
Sheng-Chieh Lin', Luyu Gav?, Barlas Oguz², Wenhan Xiong", Jimmy Lin", Wen-tau Yih Xilun Chen
University of Waterloo', Carnegie Mellon University². Meta A1
s269lin@uwaterlo.. ca, xilunêmeta.com
Abstract
Alignment is procedure to fine-tune pre-trained large language models (LLMs) to follow natural language instructions and serve as helpful Al assistants. We have observed, however, that the conventional alignment process fails to enhance the factual accuracy of L.Mss, and often leads to the generation of more false facts (i.e., hallucination). In this paper, study how make the LLM align- ment process more factual, by first identifying factors that lead to hallucination in both alignment steps: supervised fine-tuning (SFT) and reinforcement learning (RL). In particular, we find that training the LLM on new or unfamiliar knowledge can encourage hallucination. This makes SFT less factual as trains on human- labeled data that may be novel to the LLM. Furthermore, reward functions used standard RL often inadequately capture factuality and favor longer and more detailed responses, which inadvertently promote hallucination. Based on these observations, we propose Factuality-aware AlignMEnt (FLAME") comprised of factuality-awen SFT and factuality-avare RL. through direct preference optimiza- tion. Experiments show that our proposed FLAME" guides LLMs to output more factual responses while maintaining their inslmactionnfollooinng capability.
1 Introduction
Alignment [Ouyang et al. 2022] is a procedure to make pre-trained large language models (LLMs) [Brown et al., 2020 Touvron et al. 2023) follow human instructions and serve as helpful AT assistants. Despite significant progress in general L.M alignment [Ouyang et al. 2022 Bai et al. 2022, Yuan et al. 2024), 1 state-of-the-art aligned L.LMs are still prone to generate false claims [Ope- nAl 2023 Min et al. [202]. In this work, therefore attempt to advance the understanding of the underlying causes of LLM hallucination as well as its relation to the alignment procedure.
We consider the commonly seen alignment process consisting of two training phases: (1) supervised fine-tuning (SFT) [Sanh et al., 2022]: : (2) reinforcement learning (RL) with human [RLHF, Ouyang et al. 2022 Bai et al. 2022 or automated feedback [RLAIF, Bai et al.) [023). In our study, we find that both the SFT and RL steps the standard alignment process may actually encourage LLMs to hallucinate. First, in the SFT stage, LLMs are fine-tuned with diverse instructions paired with human- created high-quality responses. While this leads to strong instruction-followw followinn capability [Ouyang et al, 2022, Köpf et al. 2023 Zhou et al.) 2023 Touvron et al.) 2023), our study shows that such haman-labeled responses may present new or unknown information to the LLM. This, in turn, may inadvertently promote hallucination. Second, we find that the standard reward used in the RL stage
This work done during Sheng- Chieh's internship at Meta.
Xilun and Sheng- Chich contributed equally to this work.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

55
Factuality (FACTSCORE)
50
SET
length: 1.0k
+DPO*
length: 1.4k
longth: 1.2k
+DPO
45
length: 1:4k
SFT
+PP
length: 1.5k
40
length: 1.6k
35
RI.F
30
20
30
40
50
60
70
Helpfulness
Figure I: Models' helpfulness on Alpaca Eval factuality on biography. Helpfulness is measured by models' win rate over our baseline SFT DPO on Alpaca Eval. Dot size represents average length of bio generation.
often prefers longer and more detailed responses (Singhal et al., 2023 Chen al. 2024b Yuan et al. 2024). Consequently, a reward-hacking model ends up with tendency to produce longer claims with more non-factual information, as shown in the black dots in Figure [ One possible reason that single scalar reward to represent preference,
most existing RLHF or RLAIF approaches rely which struggles to cover multiple alignment skill sets Ye et al. 2024 and likely to under-present
the aspect of factuality (Hosking al. 2024].
To address the aforementioned issues, we study the key factors which impact factuality during alignment. In particular, we first conduct a pilot study on the biography generation task [Min et al. 2023] in more controlled setting where the alignment process focuses solely on factuality (Section B) Our key observation is that an LLM hallucinates more if is fine-tuned on new knowledge in either the SFT or the RI. stage. For example, an LLM becomes significantly less factual when fine-tuned on responses produced by model with access to external knowledge (e.g. retrieval augmented LLM), even though those responses are more factual themselves. Similarly, hallucination is greatly increased if RLAIF is performed preference pairs that consist of retrieval-augmented LLM output as positive examples and the LLM's own output as negative examples. In comparison, we discover that fine-tuning pre- trained LLM subset of its generations selected by factuality yields more factual responses and reduces hallucinations.
Next, we apply our findings to improve the factuality of the general LLM alignment process, which more challenging due to the diversity of instructions. As shown in Figure 2 we observe that some instructions require factual responses while the others do not, and therefore would require different alignment treatments. We first identify fact-based instructions that require factual responses and leverage the findings in our pilot study create additional training data at both SFT and RL stages to explicitly guide LLMS to output factual responses. Specifically, at the SFT stage, for fact-based instructions, instead of using human created seed training data, we elicit knowledge from the pre-trained LLM and construct training data using its own pre- pre-trained knowledge. This can prevent fine-tuning the LLM knowledge unknown itself. At the RL stage, create additional preference pairs focused on factuality for fact-based instructions, which are combined with the standard preference pairs for instruction following during Direct Preference Optimization [DPO; Rafailov et al. 2023]
We evaluate models Alpaca Eval Dubois et al. 2024 and Biography, using win rate for instruction- following capability and FACTSCORE [Min et al. 2023) for factuality evaluation. As shown in Figure I using our FLAME" method (SFT* + DPO*) ), a significantly higher FACTSCORE (+5.6 pts) is achieved compared to the standard alignment process (SFT + DPO), without sacrificing the LLM'' folllloiooooooowiinng capability (51 51.2% win rate). Our ablation study also indicates that identifying fact-based instructions is the key to factual alignment in the general alignment setting.

---

2 Related Work
Alignment. Since pre-trained LLMS cannot accurately follow human instructions, bunch of work has been proposed to improve LLM alignment through SFT and RL. Some propose to improve SFT
through data curation [hoo et al. 2023 Chen et al. [024a]. diverse instruction augmentation WWagg ct al.
2023a Li et al. 2024] while others focus on RL with human feedback [Ouyang et al.. 2022 Bai al.
2022 AT Teedback [Bi al., 2023 Sun al.
2024 Yuan et al. 2024] The main goal of these cntruuctionfooloiinn following capability (or helpfulness), which may guide LLMs to
aligmment approaches is instruction
output detailed and lengthy responses (Singhal al. 2023
but inevitably encourage hallucination.
Factuality, Prior work has highlighted the issue of hallucination in LLMs [Gao et al., 2022
Kandpal al. 2023 Mallen al. [023] To address the issue, important research lines are factuality
evaluation [Min et al.
2023
Wang et al.
2023b Chern et al.) 2023] and improvement. Some training-
free approaches to improve LLMs' factuality include external knowledge augmentation [Gao et al.
2022
Kandpal et al.
2023
Cheng el al.
2023 Jiang et al. 2023 and specialized decoding
Li et al.
2023
Chuang al.,
2024).
Recent studies apply RL to improve LLMs' factuality. For example, Tian et al. [024] propose to
construct factuality preference pairs for direct preference optimization [DPO; Ratailov et al. 2023].
which is closely related to our work. However, they focus solely on enhancing L.LMs' factuality through DPO but overlook its potential impact the models' cntrucctionfollowinng capability, as demonstrated our experiments. In contrast, our work provides a comprehensive examination
of improving LLMs' factuality and instruction-follooing ability through fine-tuning approaches encompassing both SFT and DPO. Concurrent to our work, Kang et al. [2024) find that LLMs tend to hallucinate when facing unfamiliar queries. They consider improving LLMs' factuality as teaching LLMS to output abstaining or less detailed responses on such unfamiliar queries, a similar behavior observed from our LLMs fine-tuned with FLAME (see case studies in Section 6.5). It is worth mentioning that both prior studies focus on a simplified scenario as our pilot study in Section 3 fine-tuning LLMs to improve factuality on single task (e.g., eie-ttunigg and evaluating on biography generation). In contrast, we consider the general alignment task, where L.MSs are given diverse and complex instructions.
3 A Pilot Study on Factual Alignment
In this section, we first study how to align large language models (LLMs) to be more factual. We use biography generation as the task of our pilot study for two main reasons: (1) Biography generation is a simplified setting where factuality is the sole focus of the aligmment process. As we will discuss in Section E studying factual alignment on diverse human instructions is more complex, as the alignment process encompasses aspects beyond factuality, such as helpfulness and safety. (2) Evaluating the factuality of biography generation relatively easy since Wikipedia covers sufficient information for public figures and most of the facts about a person are non-debatable [Min et al. 2023
3.1 Alignment for Biography Generation
A standard alignment procedure consists of supervised fine-tuning (SFT) and reinforcement learning (RL). In this pilot study, our main goal is to teach LLMs to generate biography with reduced misinformation. For the experiment, and 183 diverse human entities, respectively (further details provided in Appendix [I.]. We employ FACTSCORE [FS: Min al. 20233 as the automated metric for assessing factuality, given its fine- grained eva vvaluation capabilities for long-form text generation and its strong correlation with human judgments To study factuality alignment in this pilot study, we posit that training data is needed where the responses are more factual than the LLM's 's own generations. Thus, we use retrieval- augmented LLMs [RAG: [ewis Lewis et al. 2020] to generate training data, which has been shown to output more factual responses Mialon et al. 2023
compile training and evaluation datasets comprising 500
Throughout the paper, we refer to the pre-trained (PT), supervised fine-tuned (SFT), and direct preference optimization (DPO) fine-tuned LLMs as PT, SFT, and DPO, respectively
*e use the evaluator: retri i evva 1 +1 1ama +mpm
'Note that in our experiments, we DPO the substitute of RL (Schulman al. 2017
3

---

Fact-bused (t )
Non fact-hased (x E X
( Do you have any information about the Commodore 64?
(6) How would child feel if fell down on the ground hitting its face??
Hi, could you help mhe solve this cubic equation using Cardano's
Write a fun story that can told 3 minutes the diunee table.
(3)
Mehod (step by step possible), please? >> xn3 2x"2 1=
(7) are 3 developers enjoying pizza The story must contain these word: zombie, ethernet cable. sudo, duchhoard.
e Please give a brief history of coffee.
(8) Tell me story about who to the moon.
What are the principles at play in UHPLC-MS analysis?
Is the internet's focus on engagement the rool of most of it problems
(4)
(9)
and shortcomings? ?
Explain the signifleance of the American Revolution, including the
Can you tell me a bit about whitt has gone into your creation?
(5)
events that led up to the impact it had on the world, and its ongoing relevance today.
(10)
Instructions from Open Assistant dataset. The instructions are classified with SFT model using the prompt in Appendix Figure 4
Figure 2:
SFT. We explore two sources of supervision to generate training data (detailed in Appendix A.I]p (1) using PT with few-shot demonstration to generate biographies for each name entity in training
data, where Рт is PT augmented with an off-the-shelf retriever [Lin et al. 2023): (2) using vanilla PT with few-shot demonstration to generate training data as a baseline. As shown in Table [
PT is indeed much more factual than PT. However, surprising discovery in the pilot study is that fine fie-tuning on such more factual cdtructiio-biogrape pairs generated by PT results in less factual SFT model (row 4 vs 3).
DPO. We further fine-tune the LLMs to be more factual through DPO. An intuitive way to create factuality preference pairs to directly use the samples from PT and PT as positives and negatives since PTRAG generates more fac- tual biographies than PT (row 2 vs 1). Another approach is to employ FACTSCORE (FS) as the reward to select positive and negative samples among the generations from PT itself (Tian et al., 2024) (detailed in Appendix A.I. As shown in Table DPO fine-tuned on self-generated data
Table I: Pilot study on bio generation. Pos. de- notes the positives for SFT or DPO. Neg. denotes the negatives for DPO. FS denotes FACTSCORE
Llama-2 7B
stc. of supervision
Bio
Pos.
Nog.
FS
# Con: / Err.
(1) PT
39.1
14.4/
(2) P'G
55.4
18.6/ 15.9
(3) SFT
PT
37.9
13.47218
(4)
PT
35.7
13.5/23.7
(5) DPO
PT
PT
41.6
15,4720.
with FS reward guides models to generate more factual responses (row 5 vs 3); however, DPO fine-tuned with the supervision of PTRAG makes
(6)
PTRAG
PT
23.5
/3.9
FACTSCORE is used to select positives and negatives.
the models hallucinate even more than its SFT counterpart (6 vs 4).
This outcome suggests that compelling models to generate responses akin to PT prompts increases hallucination. Conversely, mun-tuning LLMs on their own generations appears to be crucial for factual alignment, finding applicable to both SFT and DPO fine-tuning.
3.2 Strategies for Factual Alignment
From the pilot study, we find that better quality data (in terms of factuality) for SFT and DPO does not necessarily yield models with better factual alignment. This is likely because the supervision from RAG contains information unknown to the LLM; thus, fine-tuning on RAG generated responses may inadvertently encourage the LLM to output unfamiliar information. To avoid unknown knowledge from being presented to the L.M,, viable strategy is to create SFT and DPO training data using the generated responses from the LLM itself.
4 Factuality-Aware Alignment
In the section, we further extend our discussion of factual alignment to encompass more general instructions. Unlike biography generation in Section 3 where factuality is the main aligmment objective, human instructions are diverse and complex, necessitating range of alignment skill sets beyond factuality alone; e.g., logical thinking, problem handling and user alignment (Ye et al.. 2024]. Thus, conducting factual alignment with the diverse instructions face two main challenges: (1) different instructions may demand distinct skill sets. For example, in Figure 2 instruction 3, "Please give me brief history of coffee", necessitates factual accuracy and concise summarization,

---

Instruction
Human response
Factuality-Aware SFT
Source of supersisonn
(Gnnerin outline history Turkey
sdl
Here brief
Nistory
E X
Please give me brter Hissory
Resporsess with few-shot PTOA)
Lx. J.Human(x))
TuRkyyrrin
SFT
bried history Duke
Can miveme
Duke basketbal was established 1905 and one of
Tell me story about who gams
Haman resportiss
(bikkalball?
Seed
PT(x)
Factuality-Awar
Source of Fugerrsiion
k-shot demo
X Please give me bnef history cofine.
Coffee belloved
oocuualoonol
retrieve top k instructions
PT
18th century: Coffee
Coffee crigi nated
E
X
about
you any
DPO
Ehiopia the 9th
X
centuryin2. ....
who the mars
nucleus sampling
(a)
(b)
Figure 3: Illustrations of (a) response generation using a pre-trained LLM (PT) with few-shot demonstration; (b) factuality aware alignment.
while instruction 8, "Tell me a story about pig who goes to the moon", prioritizes creativity and imagination over strict factuality. (2) As recent studies have emphasized (Ye al. 2024 Hosking
et al. 2024), using a single scalar for reward modeling fails to adequately address multiple alignment skill sets and often under-presents the aspect of factuality.
To tackle the aforementioned challenges, we propose fuctuality-avare alignment (FLAME*). To address the first challenge, we propose to prompt LLMs to classify whether given instruction We then apply the factuality fine-tuning
demands the response to be factual, as shown in Figure 2 strategy for SFT and DPO discussed in Section 3.22iionn322te to those fact-based instructions. Furthermore, to address the second challenge, we employ separate rewards to evaluate the factuality and instruction- work only considers two alignment skill sets:
following capability of an L.M.. For simplicity, instruction following and factuality. We leave more comprehensive reward modeling to future work.
In the following, we first describe our baseline alignment approach and introduce our proposed factuality-aware alignment built on top of the baseline alignment procedure.
4.1 Baseline Alignment
We initialize PT from Llama-2 70B pre-trained model and build our baseline alignment procedure following celf-rewarding language models (Yuan al. 2024 due to its simplicity and independence of other strong LLMs (e.g., GPT4) or human evaluators as a reward model. The alignment comprises two steps: (1) building SFT model fine-tuned on high-quality seed data consisting of 3,200 instructions and each instruction is paired with the best response created by humans from Open Assistant dataset (OASST:KKöpt et al. [023]: (2) further fine-tuning SFT through DPO on instruction- following preference data (I, /+: y-) constructed by itself (SFT) as the reward model, RM where y+ and y- are the positive and negative responses for a given prompt ær respectively. The resulting fine-tuned model is denoted as SFT DPO. Note that, following Yuan et al. (2024), we use additional augmented 20K instructions to create the preference training data for DPO fine-tuning. Further details are provided in Appendix A.3
4.2
Our Approach
4.2.1
Factuality-Aware SFT (SFT*)
Although leveraging human created high-quality seed data is a reasonable choice for SFT [Zoou al. 2023]. our study in Section 3 suggests that fine-tuning on such high-quality data generated by models other than the LLM itself may present unknown information to the LLM, which may in turn encourage hallucination. To address the above issue, for each instruction from the seed data, we elicit the knowledge from the pre-trained LM itself by generating the responses with few-shot demonstration. Furthermore, to better use the knowledge from both humans and the pre-trained LLM itself, we propose to utilize human generated responses for non-fact-based instructions, while leveraging the responses sampled from pre-trained LLMs for fact-based instructions to mitigate the introduction of unknown knowledge.
https:://uggiiggeeee wtt.://e....mmmtaateee
70p
5

---

Specifically, we create factuality-aww aware alignment training data for SFT with two steps. (1) Classifying instructions: we first prompt SFT judge whether an instruction from the seed data is fact-based (x
X or not 6 (2) Elliciiing knowledge from PT:: as illustrated in Figure 3 a),
Ba), we sample 10 responses (Ta. (r)ma( where 2y is the top-k
from PT with 5-shot demonstration, (ro, Human(zo))
similar instruction to retrieved by DRAGON, [Lin et al. 2023) from the seed data. Human (2*)) denotes the corresponding human response to 27k in the seed data. As illustrated in Figure 3(b) (upper), the resulting training data for SFT is (z & X X Human(z)), (x X PT(()),, where PT(x) denotes the set of responses sampled from PT. The fine-tuned model is denoted as SFT*
4.2.2 Factuality-Awark DPO (DPO*)
At the second stage of alignment with DPO, we use SFT to generate multiple responses yo, y1. for a given instruction z; then, using SFT* itself as the reward model (RM") to create a preference pair: (I, , The above data creation procedure is the same as the second stage of our baseline
alignment in Section 4.1 However, recent studies (Saha et al., 2024 Hosking et al. 2024 Ye et al. 2024] indicate that single scalar reward from human feedback or LLM reward models may underrepresent the aspect of factuality. To address this limitation, we introduce another factuality reward model (RM to evaluate factuality of responses and create a factuality preference pair for fact-based instructions: (ar E X fact Yrues Vtale).
Specifically, we build RM with retrieval augmentation to measure the percentage of facts in response that are correct. RM comprises two main components: atomic fact decomposition and retrieval augmented claim verification. We detail the components and ablate their impacts on the quality of RM in Appendix A.5 We compute factuality reward for the same responses sampled from SFT*: RM faet (z, yo). R.M (I, 111....... The response with the highest (lowest) factuality reward is chosen as Youe (ylaae)). Note that if the chosen paired responses show large difference in instruction-follooiin reward, we discard the pair; i.c., RRM Ylmee) RM (z., Maaae) ) 0.5. As illustrated in Figure 3 (lower), factuality- aware DPO training, the model is initialized from SFT* and the fine-tuned model is our final factuality- aware aligned model, denoted SFT + +PO*. The specific procedures for fine-tuning models in both the SFT and DPO are described in Appendix A.6
Experiments
5.1 Evaluation Datasets and Metrics
Instruction Following, We use the 805 cestruction-foloviiii tasks from Alpaca Eval [Dubois et al. 2024 to evaluate models head-to-head win rate against baselines using the recommended evalua- tor: a as the baselines for win rate comparisons.
w_tpppt__tuur___p______ fn. We use SFT and SFT + DPO described in Section 4.1
Factuality, We evaluate models on three datasets with diverse knowledge-intense intensive instructions for factuality, (1) Biography: knowledge insensitive sub-task of caatucttoonfollowiing tasks. Following our pilot study in Section 3 we use the 183 human entities provided by Min et al. [2023] with the prompt "Tell me bio of entity name". (2) Alpaca Fact: we extract the fact-based instructions from the 803 instructions using our SFT model (with the prompt shown in Appendix Figure 4 re-
sulting in 241 instructions. (3) FAVA [Mishra et al. 2024 the 141 mrowledge-intensinn instructions
from multiple sources, including Open Assistant Köpf et al. 2023]. No Robots [Rajani et al.) 2023
WebNLG (Gardent al. 2017 7 and manually created datasets. We report FACTSCORE (FS) without length penalty as the metric for all the three datasets. Note that original FS computes proportion of correct facts with additional penalty aims to address situations where models provide insufficiently detailed answers. We assume that this aspect considered in the evaluation of instruction following in Alpaca Eval. In addition, also
short generations with less than 10 atomie facts. This penalty
'Prompt for fact-based instruction classification is shown in Appendix Figure We sample responses for each augmented instruction. 'https: htap://:///gggnngacc.o annotations.c json

---

Table 2: Experimental results of supervised fine-tuning Open Assistant dataset. PT denotes pre- pre-trained Llama2 70B with 5-shot demonstration. SFT denotes the variant which only optimizes factuality. FS denotes FACTSCORE.
1lama-2 70B
src. of supervision
Alpaca Eval
Bio
Alpaca Fact
FAVA
Human
PT
win rate over (1)
FS
# Corr. EEr.
F$
Com. / Bim.
FS
Cor,. / Err.
(0) PT
53.1
13.3713.5
SF)
50.0
44.7
26.17
16.7/29.0
21.2/25X
(2) SFT6
48.5
19.6/ 20.6
42.0
17.5/2844
18.3 24.2
(3) SFT*
6'
51.2
49.5
19.5
41.4
18.3/27.7
54.2
19.3/22.4
"FTT" uses supervision from Human und PT for non-fact-bused and fact-bused instructions, respectively.
Table 3: Experiments of direct preference optimization (DPO). IF. and Fact. denote instruction following (I, (I, --) and factuality (z E X Mumes Vtatee preference data, where X denotes the set of fact-based instructions. DPO denotes the variant which only optimizes factuality. The preference data statistics is listed in Appendix, Table TT.
SEC. of supervision
Alpaca Eval
Alpaca Fact
FAVA
Llama-2 70B
Fact.
win rate over (2)
Corr. Err.
Corr. Erz.
Corz. Etr.
(0) Chat
Proprictary
66.2
33.2
23.4743.6
39.3
22.37 36.4
47.5
28.07313
SFT
27.1
44.7
21.1 / 26.8
38.6
16.7 / 29.0
54.4
21.27
(2) DPO
x
50,0
42.3
24.6/35.0
41.6
22.9 34.6
52.9
28.1 26.88
(3) DP
x
/
40.8
47.1
19.8/23.9
48.2
17.5 19.0
57.9
20.0/ 15.9
(4) DPO*
v
/
51.7
44.9
23.7 23.7/30.3
45.0
23.1 28.7
56.4
27.1 /23.3
(5) SFT*
29.1
49.5
19.9
41.4
18.3
54.2
19.3/22.4
(6) DPO
30.4
46.3
24.07 28.7
43.9
21.67 28.8
55.0
25.472200
(7) DPO
51.2
47.9
25.9 28.5
48.7
24.1
/25.5
58.9
29.0/22.2
report the number of correct and erroneous facts. All the numbers reported are averaged over the instructions each dataset.
In addition, we also evaluate our fine-tuned models' truthfulness using TruthfulQA [Lin et al.
2022
We evaluate model performance in the generation task and use ROUGE Lin 2004 and
BLEU Pappineni et al. [2022] to measure the quality of responses.
5.2 Comparisons of SFT
Table compares the pre-trained Llama-2 70B fine-tuned OASST dataset with responses from different sources. We list the FACTSCORE (FS) of biography generation using the pre-trained model with
through Bio 5-shot demonstration as reference (row 0) and SFT, which is fine-tuned our seed data
responses, is our baseline (row 1). We first notice that SFT shows significant FACtScoRE degradation (53.1 44.7) compared Bio 5-shot with the pre-trained model. It seems that SFT tends to generate more lengthy responses but with more erroncous facts.
When eliciting the knowledge from PT by fine-tuning on its own generated responses, SFT gener- ates more factual responses in Biography and Alpaca (row 2 vs 1). However, shows slightly inferior instruction- followiinnn capability in Alpaca Eval. This result demonstrates that human responses indeed teach LLMs how to better follow instructions but also encourage LLMs to output more false facts. On the other hand, eliciting the knowledge from the pre-trained model itself avoids the encouragement of hallucination albeit with a slight reduction in instruction following capability. Finally, SFT* intructtion-oolloinng combining supervision from humans and PT, shows comparable instruction-follwviii capability and output more factual responses on bast- instructions (row 3 vs 1).
5.3 Comparisons of DPO
Table 3 compares different DPO training recipes. First, we conduct DPO fine-tuning on our SFT baseline, SFT. When further aligning the model to follow instructions, DPO sees a significant improvement in instruction--fllloiing capability (row 2 vs 1) with win rate 72.9 over SFT:; however, the instruction aligned model tends to output lengthy responses with more factual errors (see examples in Appendix Figure 10). On the other hand, when only aligned with factual preference data, DPOffct shows less improvement in cnstructionfoollwiing capability (row I vs 3). These results indicate that
7

---

preference optimization for either instruction following or factuality alone may come at the expense of the other since the former encourages models to output long and detailed responses while the later discourages models to output false claims. When jointly conducting instruction and factuality alignment, DPO not only better follows instructions but also outputs more factual responses (row 4 vs 1, 2). Finally, initializing from SFT*, the DPO fine-tuned models are more factual than their counterparts (i.e.. 6 vs 2 and vs 4) without cntrructionfollowinng capability degrade. We also list the results from Llama-2-Char 70B (row 0) and observe that despite of its strong cnstruction-followwie capability, it tends to output many more incorrect facts. These results demonstrate that standard alignment, even on proprietary commercial data, may encourage LLMs to hallucinate. In contrast, our factuality-awarr alignment guides LLMMs to output more factual responses without degradation in their general instruction-followinm capabilities.
It is worth noting that SFT and DPO are similar to SFT and DPO fine-tuning proposed by Tian et al. [2044), which improve LLMs' factuality but degrade their unstucttionfollowiii capability. Also, we do not observe our SFT and DPO variants outperform the pre-trained model with few-shot demonstrations on biography generation (row in Table This is possibly due to the alignment tax found in previous work [Ouyang et al. 2022]. which degrades LLMs' accuracy on the standard knowledge benchmarks. How to improve both models' instruction-followinn capability and their accuracy on standard knowledge benchmarks is worth exploring, which we leave for future work.
5.4 Results on TruthfulQA
Table compares models performance on Truth- fulQA. Generally, we observe that our factuality- aware alignment training guides LLMs to output more truthful responses. For example, factuality- aware SFT improves LLMs' truthfulness (row 5 vs I). In addition, DPO fine-tuning on the factual- ity preference data guides LLMs to output more truthful responses (rows 3,4 vs and 7 vs 6). Note that observe that SFT and DPO models show a reverse trend in BLUE and ROUGE. This is likely because SFT models tend to generate shorter responses than the DPO ones do.
Table 4: Results on TruthfulQA.
SIC. of supervision
TruthfulQA
Llama-2 70B
IF
Fact.
BLUE
ROUGE
(0) Chat
Proprietary data
0.21
1.16
(I)SFT
0.20
(2) + DPO
x
0.03
0.54
(3)+ DtO
X
/
0.30
1.12
(4)+ DPO*
V
v
0.15
0.80
SFT
0.39
0.51
(6) DPO
V
x
0.07
0.91
(7)+ DPO
V
0.20
0.96
In addition, Table reports models' accuracy in tasks of multiple choices from TruthfulQA. No significant differences between models
Table 5: Results on TruthfulQA multiple choices.
SIC. of supervision
TruthfulQA-MC
Llama 70B
are observed. This is possibly because we
IF
Fact.
MC1
MC2
MC3
mainly focus the tasks of form re- sponse generation while TruthfulQA-MC task is formed by short- form answers. The
(0) Chat
Proprictury data
32.2
50.2
25.4
(F)
30.8
45.7
23.9
(2) + DPQ
x
30.5
46.0
23.4
(3) + DPO
X
/
31.8
46.8
24.3
discrepancy between improving LLMs' factu- ality long- form and short-form generation is also found by the previous work [Chuang let al. 2024]. Appendix Table 9 reports more evaluation results on other NL.P benchmarks.
(4) + DPO
/
/
30.8
46.0
23.6
SFT
29.9
44.8
22.5
(6) + DPO
x
31.5
47.0
24.0
+PDPP*
/
30.5
45.4
23.1
Diseussion
6.1 Effects of Fact-Based Instruction Classification
In our factuality-aware alignment, we prompt SFT to judge whether an instruction requires factual response and apply our factuality alignment strategy the fact-based instruction. Without the instruction classification, in our factuality-awan aware SFT, we cannot create supervision from Human and PT responses for respective non-fact-based and fact-based instructions. Instead, for each instruction, we create instruction-respons pairs from and 10 responses from Human and PT as supervisions, respectively. Note that, during fine-tuning, for each instruction, we randomly sample instruction- response pair either created from Human PT with same probability. The SFT model shows
8

---

degradation in both cnaruuctionfolloviing capability and factuality results, as shown in row 1 vs 2 of Table Second, for factuality-awar DPO, without the instruction classification, we create factuality preference pairs from all instructions instead of fact-based instructions. The DPO fine-tuned model outputs slightly more factual responses but sacrifice istruuctionfoolooiing capability, as shown row 3 vs 4 of 6
6.2 Effects of Fact-Based Sentence Classification
In addition, we observe that not all the sentences in a response to fact-based instruction require fact check. For example, given the response, "Of course. The Commodore 64 is a 8-bit home com- puter that was released by Commodore Interna- tional in August 1982.", conducting fact check for the first sentence "Of course." is not necessary and may make the factuality reward less accurate. To address this issue, prompt SFT to judge whether each sentence in a response required fact check using the prompt in Appendix Figure 6 We only conduct fact check and compute factu-
Table 6: Effects of fact-based classification.
Classifier Alpaca Eval
Bio
Inst. Sent.
win
FS
Cor.. / Err.
47.6"
48.4
20.5721.4
SFT
51.2*
49.5
19.9/ 19.5
X
x
46.84
46.8
25.7//5.3
(4) SFT + DPO
x
51.7A
45.0
/3.7/30.33
51.34
42.9
25.5/368 /6.8
comparing with SFT buseline, SFT.
comparing with DPO baseline, SFT + DPO.
ality rewards for those fact-based sentences. However, as shown Table 6 computing factuality rewards for fact-based sentences makes our factual alignment less effective (row 5 vs 4). This is likely because the fact-based sentence classifier not accurate enough and brings noise into our factuality reward model (see examples in Appendix Figure 7).
6.3 Ablations Factuality Preference Data Creation
In this section, we examine different ways of creating factuality preference data for factuality- aware DPO training. First, for each fact-based instruction, instead of choosing the responses (among the 4 generated responses) with the max- imum and minimum factuality rewards (RMffac) as the respective positive and negative samples, we enumerate all the possible response pairs and choose the response with higher (lower) R.M fact as the positive (negative) sample from each enu- merated pair. If the difference of R.M is smaller than 0.2, we treat them as equal and discard the pairs. Note that for both row 1 and 2 in Table [ we also discard the pairs with the difference of instruction forlloonnnoooowingggooo rewards (RM") larger than 0.5 (as mentioned in Section 4.2.2). Alternatively, for each response, we linearly combine the rewards, RM (1-5 scale) and RM Taat (0-1 scale), with the respective weight of 1 and 5 as composite reward. For each instruction, we choose the responses with the maximum and minimum composite rewards as the positive and negative. As result, both data creation approaches increase the number of factuality preference pairs; however, they yield trivial improvement in factuality but slight degrade in instruction following (rows 2, 3 vs 1). This result also indicates that leveraging single reward model, which can be incorporated with PPO (Schulman) et al. 2017]. possible to improve both models' cestruction-folloviii capability and factuality
Table 7: Ablation factuality preference data.
Factuality preference data
Alpaca Eval
Bio
Reward model
Pos.,Neg.
# pairs
win rate
FS
RM
max, min
3,315
51.7
44.9
RM
enum.
5,126
50.7
45.0
RM" + 5-RMPa
max, min
6.340
50.1
45.1
2 comparing with DPO baseline, SPT DPO.
6.4 Impacts of DPO on Generation Length
Table lists the average length of models' re- sponses for each dataset. We observe that DPO fine-tuned models tend to output lengthy re-
Table 8: Effects of DPO on response length.
Alpaca Eval
Bio
Alpaca Fact
FAVA
sponses than SFT except for (ao on Biog- raphy. This trend indicates that our instruction- following reward model RM guides LLMs to output more detailed and lengthy responses. In addition, we observe that although DPO* out-
(1) SET
897
1221
969
912
(2) + DPO
1470
1494
1586
1540
(3) DR
1160
1166
1192
1104
(4)+
1474
1395
1528
1422

---

puts responses with similar length as DPO on Alpaca Eval, DPO generates slightly shorter responses for the fact-based instructions in the other three datasets. This results show that our factuality-awan DPO training mainly impacts models' responses for fact-based instructions. The impact is mainly to reduce the false claims, evidenced by the numbers of erroneous facts in rows and 4 of Table
6.5 Case Studies
Figure TO (in Appendix) showcases the generations of different models, SFT, SFT DPO and SFT* DPO Alpaca Eval and Biography. Given the instruction, "What are the names of some famous actors that started their careers on Broadway?", SFT only lists some names of Broadway actors while DPO fine-tuned models generate detailed information for each listed Broadway actor. As for biography generations, we observe that given the instruction to generate a biography for a rare name entity, Marianne McAndrew, SFT + DPO generates a detailed response but with many wrong facts while SFT and SFT* + DPO* give relatively short responses. For the frequent entity, Sung. all the models generate detailed and mostly correct responses. This qualitative analysis shows that SFT* + DPO tends to generate detailed responses for most instructions, but for those instructions required tailed knowledge (e.g., rare entity) likely unknown to LLMs [Mallen al., 2023] it reduces erroneous facts by giving less detailed responses, which is also observed by Kang el al. 2024
7 Conclusion and Future Work
In this paper, we present study to enhance the factuality of large language models (LLMs). We first identify that the standard alignment approach, comprising SFT and RLAIF with DPO, may inadvertently encourage LLMs produce more erroneous facts. Specifically, during the SFT stage, fine-tuning LLMs with high-quality human responses may introduce unfamiliar information, prompting I.IMS to output unknown facts. Additionally, during the DPO stage, enhancing L.LMs' ability to follow instructions may result in more detailed and lengthy responses but often leads to increased hallucination. To tackle the shortcomings of the standard alignment, we propose a factuality -aware alignment method, which includes factuality SFT and DPO, Quantitative and qualitative analyses demonstrate that our factuality-awan alignment not only guides LLMs to generate detailed and helpful responses but also helps prevent the generation of false claims.
While we have successfully integrated factuality into standard alignment procedure, our work only considers two alignment skill sets: instruction following (or helpfulness) and factuality. In practice. each instruction may require consideration of multiple and distinct alignment skill sets (Saha et al.) 2024]. The method to optimize for these skill sets tailored to each query requires further study. In our experiments, we note that optimizing preferences solely for instruction following or factuality could potentially compromise the other. While our factuality- in both aspects, it is uncertain whether there is trade-off between the two aspects when integrating our approach to large-scale alignment [Touvron al. 2023 Finally, as shown in Appendix Figure 7 not all the claims (or sentences) in a response require fact verification, a more accurate factuality reward model should take this factor into account. While our preliminary experiment, which removes non-fact- base- sentences from the factuality reward modeling (Section 6.2), shows suboptimal performance, we believe that further study can bring more insights.
alignment demonstrated improvements
Acknowledgements
We thank Bhargavi Paranjape for sharing fine- tuned Llama-2 7B for atomic fact decomposition and Jing Xu, Weizhe Yuan and Jason Weston for their helpful suggestions.
References
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Ólsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan.
10

---

Training a helpful and harmless assistant with reinforcement learning from human feedback. 2xXiv:2204.05862, 2022.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional Al: Harmlessness from Al feedback. 0rXiv:2212.08073. 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proc. NeurIPS, pages 1877-1901, 2020.
Jifan Chen, Aniruddh Sriram, Eunsol Choi, and Greg Durrett. Generating literal and implied subquestions fact-check complex claims. In Proc. EMNLP, pages 3495-3516, 2022.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. AlpaGasus: Training a better Alpaca model with fewer data, In Proc. ICLR, 2024a.
Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: Disentangled reward mitigates hacking in RLHF. arXiv:2402.07319, 2024b.
Silei Cheng. Zhe Gan, Zhengyuan Yang. Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. Prompting GPT-3 be reliable. In Proc. ICLR, 2023.
I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig. Pengfei Liu, et al. FacTool: Factuality detection in generative Al-a tool augmented framework for multi-task and multi-domain scenarios. arXiw:2307.13528, 2023.
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. DoLa: Decoding by contrasting layers improves factuality in large language models. In Proc. ICLR, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaFarm: A simulation framework for methods that learn from human feedback. arXiv:2305. 14387, 2024.
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, N. Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. RARR: Researching and revising what language models say, using language models. In Proc. ACL, 2022.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura
The WebNLG
challenge: Generating text from RDF data. In Proc. INLG, pages 124-133, 2017.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proc. ICLR, 2021.
Tom Hosking, Phil Blunsom, and Max Bartolo. Human feedback not gold standard. Proc. ICLR, 2024.

---

Gautier Izacard, Patrick Lewis. Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, pages 1-43, 2023.
Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-YY Yu, Yiming Yang. Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proc. EMNLP, pages 7969-7992, 2023.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In Proc ICML, 2023.
Katie Kang, Eric Wallace, Claire Tomlin, Aviral Kumar, and Sergey Levine. Unfamiliar finetuning examples control how language models hallucinate. arXiv:2403.05612 2024.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley. Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. OpenAssistant Conversations democratizing large language model alignment. arXiv:2304. 07332 2023.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sehastian Riedel, and Douwe Kiela. Retrieval--uggeene generation for nowledge-intensiing NLP tasks. Proc. NewrIPS, pages 1459-9474, 2020.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Proc. NeurIPS, 2023.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. eeff-aligmment with instruction backtranslation. In Proc. ICLR, 2024.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74 81, July 2004.
Sheng-Chich Chich Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. How to train your DRAGON: Diverse augmentation towards generalizable dense retrieval. In Proc. Findings of EMNLP, pages 6385-6400, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proc. ACI., pages 3214 42522, 2022.
Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation. In Proc. ACI, pages 4140-4170, 2023.
Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. Ex- pertQA: Expert-curated questions and attributed answers. arXiv:2309.07853 2023.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das. Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proc. ACL. pages 9802-9888888 9822, 2023.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Roziere, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: a survey. Transactions on Machine Learning Research, 2023.
Sewon Min, Kalpesh Krishna. Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proc. EMNLP, pages 12076-12101 12100, 2023.
Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. Fine-grained hallucination detection and editing for language models. arXiv:2401.06855. 2024.
12

---

OpenAl. GPT-4 technical report. 20X1:2333.08774,200 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. Proc. NeurlPS, pages 27730-27744, 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, BLEU: method for automatic evaluation of machine translation. In Proc. ACL, pages 311-318, 2002.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Proc. NeurIPS, pages 53728-5323 -3741, 2023.
Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. No robots. Hugging Face repository, 2023.
Swamnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. Branch- Solve-Merge improves large language model evaluation and generation. In Proc. NAACL, pages 8352-8370, 2024.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In Proc. ICLR, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv: /707 707477 2017.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in RLHF arXiv:2310.037/6 2023.
Zhiqing Sun, Yikang Shen, Hongxin Zhang. Qinhong Zhou, Zhenfang Chen, David Daniel Cox, Yiming Yang. and Chuang Gan. SALMON: Self-alignment with principleffollowing reward models. In Proc. ICLR, 2024.
Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine-tuning language models for factuality. In Proc. ICLR, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288, 2023.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri,
13

---

Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro. Tanay Dixit, and Xudong Shen. Super=Vaturallnstrrciionss Generalization via declarative instructions on 1600+ NLP tasks. In Proc. EMNLP, pages 5085-5109, 2022.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self- Instruct::: Aligning language models with self seff--geeertedd instructions. In Proc. ACL, pages 13484-13508, 2023a.
Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Ruba- shevskii, Jiahui Geng, Osama Mohammed Afzal, Lianginigg Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, and Preslav Nakov. Factcheck-GPT: End-to-end fine-grained document-level fact-checking and correction of LLM output. arXiv:231 1,09000, 2023b.
Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thome, Juho Kim, and Minjoon Seo. FLASK: Fine-grained language model evaluation based on alignment skill sets. In Proc. ICLR, 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-Rewarding language models. In Proc. ICML, pages 57905-57923, 2024.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. Proc. NeurlPS, pages 55006-55021, 2023.
14

---

A Appendix
A.1 Biography Data Generation
Entities for Training and Evaluation. We use 500 diverse human entities to create training data for SFT and DPO; then, evaluate LLMs' generation factuality on another 183 human entities from Min et al. [2033] 9 Note that the human entities for training and evaluation are uniformly sampled from entities across diverse nationalities, professions, and rarities. The instruction generated with the format: Tell me bio of entity name.
Creating Training Data for SFT. We randomly sample human entities among the 500 entities for training and generate their biographies using -lama-2-Chhat 70B as 5-shot demonstration T0 With the 5-shot demonstration, we use pre-trained Llama-2 7B to generate 10 biographies for each human entity from the remaining 495 ones We set temperature 0.7 and top-p 0.9 when generate multiple responses from LLMs in all our experiments. We use the created 4,950 name entity-biography pairs to fine-tune the pre-trained Llama-2 7B. As for generating training data with RAG, prepend the top-10 passages from our retrieval system (detailed in Appendix A.2) to each instruction and generate 10 biographies for each entity from RAG with 5-shot demonstrations. Note that only prepend top-1 passage for each instruction in the demonstration.
Creating Factuality Preference Pairs for DPO. To construct factuality preference pairs, we first compute FACTSCORE (FS) for all the 4,950 biographies previously created by PT. Then, for each name entity, we compare the FS for all the possible 45 pairs from the 10 generated biographies and construct DPO pairs using the biography with higher (lower) FS as positive (negative). Note that we discard the pairs if they show tied FS.
A.2 Retrieval Models
For each query, we retrieve top-20 candidate passages from Wikipedia using DRAGON+ (Lin et al., 2023) and re-rank the candidates using a 12-layer cross-encoder We use the Wikipedia version from the Dec. 20, 2021 dump released by Izacard et al. [2023 in this work.
A.3 Alignment with Self Rewarding
SFT. At SFT stage, we fine-tune PT on two seed datasets: (1) Instruction-following training (IFT) data from Li et al. [2024), consisting of 3200 eestuccion-responee pairs created by humans from Open Assistant dataset [OASST; Köpf et al. [2023, where we only use the first conversational turns in the English that are annotated rank O (2) evaluation following training (EFT) data from Yuan
et al. 2024], the J.LM-as-a-Judee data consists of 1630 samples, each of which contains instruction,
human response and the corresponding score of 1-5 scale (with chain-of-thouud evaluation reasoning):
(z. r)r where (z, y) pairs are also selected from OASST other than training pairs and r is created by the model fine-tuned only on IFT with manual filtering. The purpose of EFT to enhance LLM's capability as a reward model to judge the quality of a response in terms of relevance, coverage, usefulness, clarity and expertise. We refer readers to Yuan et al. [2024] for how EFT is created and filtered with minimum human efforts. The prompt template for LLM-as-a-Jude in EFT and an EFT training sample are shown in Appendix, Figure and We refer the baseline model fine-tuned on
the IFT and EFT datasets as SFT.
DPO for Instruction Following. At the subsequent preference learning with DPO, following Wang et al. [2023a), we augment additional 20K instructions with Llama-2 70B chat model [4] For each augmented instruction x, we use SFT to generate responses and evaluate how well the responses follow the instruction with score of 1-5 scale: RM (2. ;RM (z, 13), where
15

---

yo.... E SFT(z) and RM is the instruction foolloiigg reward model. Note that, in self- rewarding Yuan al. 2024], RM" is the same as SFT model. In addition, for each instruction- response pair, we use the same prompt in EFT seed data to sample the chain-of-thought evaluation three times and average the scores as the reward. Finally, for each instruction, we use the response with the highest (lowest) reward as the positive (negative) sample to form a preference pair for DPO training: (æ, V-) We discard the pair, if RM (æ, 1/+) = RM (z, y-). In the DPO training, the model is initialized from SFT and the fine-tuned model is denoted SFT + DPO.
A.4 More Evaluation Results on Standard Benchmarks
Table Sableae the instruction fine-tuned mod- els' (w/o and w/ involving FLAME) accuracy the in tasks of MMLU (Hendrycks et al. 2021] and
Table 9: Results on MMLU and GSMSK.
Llama-2 70B
STC. of supervision
MMLU
GSM8K
GSM8K [Cobbe et al. 2021]. A slight drop from FLAME is observed. This is possibly because we mainly focus on the tasks of long-form response
(I) SF++ DPO (2) S DPO
69.3
59.3
V
69.1
58.2
generation while MMLU and GSM8K are the benchmarks with short-form answers. The discrepancy between improving LLMs' factuality on long-form and short-form generation is also found by the previous work [Chuang et al. 2024
A.5 Factuality Reward Modeling
Factuality Reward Models. We build a reward model RM to measure the factuality of each response. The factuality reward model consists of two main modules. (1) fact decomposition: we first use nltk. token i ze split response into sentences; then, use our Llama-2 7B model fine-tuned
on public datasets (Liu et al., 2023 Chen et al. 2022 Malaviya et al. 2023 to conduct atomic fact decomposition for each sentence (2) Retrieval augmented claim verfication: for each decomposed
fact (or claim), we use the instruct Llama 7B fine-tuned on Super Natural Instructions (Wang et al.
10 We append 10 retrieved supports (using the instruction as query) from our retrieval and re-ranking pipeline in Appendix A.2 Then, we compute the proportion of correct atomic facts in a response as a factuality reward.
2022] to fact check with the prompt shown in Figure 5
Quality of Factuality Reward Models. We conduct ablation study on our factuality reward models. Specifically, we use our factuality reward models to detect the number of error facts in each instroction-response pair. We try different models for fact check using the prompt shown in Figure 5 with different numbers of retrieved supports. We use the LLMs' generated responses
Table 10: A comparison of factuality reward mod- els. denotes the correlation between human annotation.
fact check model
# sup.
fact unit
5
0.32
Instruct Llama
10
atom.
0.34
with human annotated hallucination provided by
(3) SFT (Llama-2 70B)
10
5
atom.
0.31
0.28
Mishra et al. [2024] to evaluate the quality of
5
0.20
the factuality reward models 17 Specifically, we
(6) Instruct Llama
10
sent.
0.25
rank the responses by numbers of errors detected and calculate the Kendall rank correlation (r)
between the rank lists by our factuality reward models and humans. As shown in Table 10 conducing fact check with more retrieved supports improves the accuracy of the factuality reward models (row vs 1). In addition, our SFT, only fine-tuned on the IFT and EFT data, is capable of doing fact check, compared to Instruct Llama 7B fine-tuned on Super Natural Instructions [Wang et al., [002]] Finally, instead of computing the number of error facts from decomposed atomic facts, we conduct fact check directly for each sentence in a response and calculate the number of false sentences as error facts. However, the quality of the reward models shows significant decrease (rows 5,6 vs 1,2). We finally adopt row 2 as our factuality reward model.
With few-shot demonstration, SFT is able to decompose a sentence into atomic facts with acceptable accuracy. Fine-tuming a 2 7B is to reduce the inference time.
main/
annotations.jso
16

---

Table 11: Training data statistics for different variants. IF. and Fact. denote instruction following (z, y++ .-) and factuality (xr E X Vune Mtase) preference data, where X denotes the set of fact-based instructions.
Seed IFT (# of Inst.)
Preference (# of pairs)
model variant
I X
E X
IF.
Fact.
SFT DPO
18,454
SFT DPQ
2,187
1,013
3,315
SFT + DPO
18,454
3,315
SFT* + DPO
18,603
2,187
1,013
SFT* DPO
18,603
4,211
A.6 Training Details
We fine-tune our models for 500 steps with batch size of 32 and 64 on respective SFT and DPO stages. The learning rate and maximum sequence length is set le 6 (which decays to le 7) and 2048, respectively. At SFT stage, we mix the IFT and EFT while at DPO stage, we set 3 0.1 and uniformly sample between self rewarding (z,, V+ V-) and factuality reward (x, Youe Viase) preference data. Note that SFT (SFT*)= + DPO meaning that we use SFT (SFT") to create preference data, serve cestruction-followinn reward model RM and as the initialization of DPO. The data used to fine-tune different variants are listed in Table T1 We conduct fine-tuning with full parameters on 64 NVIDIA A100 (80GB) GPUs. SFT and DPO require around and 4 hours complete, respectively.
A.7 Limitations
Limitations. In this paper, we use FACTSCORE as the main metric of factuality. However, as highlighted by Min al. [2023]. "FACTSCORE focuses factual precision is only one aspect of the broader factuality problem." " addition, the validity of applying FACTSCORE to broader domains other than Biography has not been carefully studied. In experiments, we can only claim that models' generation shows high agreement with the facts from Wikipedia, without considering whether the claims are open-ended and debatable. Finally, reward hacking is the potential issue of LLM alignment [Chen et al. 2024b) but the risk of reward hacking is still unknown when considering factuality.
A.8 Broader Impacts
Increased factual accuracy in Al-generated responses may contribute to building trust Al technology. Users more likely to trust and rely on AI systems if they consistently provide accurate information, leading to broader adoption and acceptance of Al-driven solutions. However, FLAME does not deal with the facts which open-ended and debatable and uses Wikipedia the only source of facts. This may inadvertently reinforce some biases in the Al system's 's responses, potentially perpetuating societal inequalities.
17

---

(1867) Hore question fron "<008377015" the above question, de you seed the ractual knowledge from RiKipdiia dive using
Tornat: "Anavera You Bot, [/IXac)
Figure 4: Prompt to check whether an instruction is fact-based.
(3N9T] Ansver the queation about "sinstractionn" based en the aiven contest. Vaha schatzieved Suppos tay Nahaleoat:
tadexi pa Jacc es starencce teee az Palse7lnbusposs 1/1X92)
Figure 5: Prompt for fact check.
(I8BT] Does: the given sencence concain. factual knowledge from Xikipedia? Questions with Yea or No. (/2892)
"cQUERT>"> Bentenoes
"<CLATM>" Answer
Figure 6: Prompt to check whether claim is fact- fact-based.
Eeyiato the ugstieaerr of Aperiios Ri incattice incteding
To lase information aboal Co
Maked ttip by map if papiiil plase 2'3 20-2-20
cas help you
oit opaatias Ser3 T- 0 sakig The Nveatior Reoelaic was a aooud meneed world liaay Sur
The Earenbes aie heee
velonee
linted
Tran
Connodes bacmit and in August 1oe2 Thiz tall uich ool. That
[toe the vape Sapp the the sqases in the fermees you 315 The oreas loding in the Ronduatie buga when Accatcan cileto
Lave 6 m do
Eir saus and rapiidiin iepoad open fice by the teikis givennace.
Trrck was widd bhe oppinitem Ire.
home grod wrinyy no bith
Dnp rent sipeiicnt morn leading the Rouduase
epeetto dors t-tryts wi
carpel
Commadder 136 tiie
-U sley Safve
bator syete
Then seeining coceuaity Couaaaees lara wound
TE eoor oan tilkood secas d'aaa
lepest Aeceiia Revolaaio bless Biiik
who
the
of andio
sprtia Stg Fealthe
Take
Sevalaiing lat -y 577 gen
ruclling bay diceniially
and heato dndueet de revice falle
onaam comttes
The dos-iiaann inppodd poinic amont dhe ward to fight for the ove
oftwee or prograsein beggagei place led Devir aik Falle
ii - i cppttao Seliiiinng 1. wonh las k. an the Undell Siam the Beuch fur lil to forutias of we "poric
oo sprt speclay and santi Thow twns comples oancel oquntoc 2012
pein glln,
Taday, Amcaian Revouaion outtacc sdoraat or
and
people
The Assricn Revelak seres nunlaler tua orlicar
exme tagetir right hor ahe right and mike not extarnee in the
lloaai korkmon power the come oprr and the come for trookes
Figure 7: The results of whether a sentence is fact-based or not classified by SFT with prompt in
Figure 6
(RET) savius the usesta queation. and Ube cocceapand.ng pespenav using the catilive a poins acozing vystimm desceibed below, Pointa accurcisted buand on the autistaction of each critasion:
response containa aue ixcelevent content.
point
provides intocmation
usezta inquary, wven incumptete
And another point it the resgonse addresses a aubstantial porsion of the usexta guestion, but does not completaly resolve the quary or provioe a disse. ansse.
haard thind point if the rergonte answers the basto elenents of nsente grestiin in weeful way, regoriller of whether St seena have been weitten at Al Asolatant or is it has atenenta uppioiiyy tound in bLoge or searon resuito.
Graet fourts point ie the response to clearly settten from an At Aestetane's perepeative, addieessing the yeeete guarttion strectly corprehenstve's y, and vell-orgenized and helpfu), even there io o'ighe conciseneas
inprowenent clartry,
Beatow ti poter for resgance that is sepeccably extlered to the userts guertton by an XT Acctetant, without surraneeus Infomation, se veflicning expert knowledge, and denonrcracing a 21 high-gialiny, engrytng, and Snotghtto", srover
Teors enaeranctoon STARS
crappasss GESPONEE c3stsc//eapoonee
Aftez exandzing the saacts inateuction and the reaponns:
Bristly juility your total acoss, up to :00 vocts.
Sonclude with the acoce using the Eocnatz "Scoce: <tosal peintas"
assistast parspentive, otilising search knovlooge neceapiiy. evallate zeaponse
aligument with this add acosing mudel, we'll systanatioalll astribeie pointa based on ouslines criteria. [/1802]
Figure& 8: Prompt to evaluate models' instructionnfollowiing capability from Yuan et al. 2024
18

---

[OwT] sovine the noorts quostion and the cormppaddngg Moree ared bated the on artefaction of each criterion Sann rdd polnt if the responto is relerat and provideo seeo inforestion related to the s00ale tngirry, ooes
adfleive Sepoint
openow deseribed below, Polnte
Incorplato or containe cre Inveloennt content dew arother point the rerponte aidrascer
competeely
responss aravara Ansistant or Lt is hes alenutea typicelly found in blogs or maree pesucta. in Geatt tourt point it the cuapozze is clescly seitben fcoe un kastatant's pazspection, antreraing use''s qustior directly and crppemmminiiy and will prpariced and seiptei, even shere question
basic eleaanta
cascty quation
repactions whathez aser
veltten by
At Aasiatant, without and insightzul ensver. talzlatteess Reite sectes et scaps esp latnun soe Shvard fnowee cealee classitien gorerteant cocoennt to the peass, oatiining
inborattinn rutlersing
inosieege, and sesnstrating higs quality,
cheecraponneo eaployed vata adoaed
muccoundin sdwsed incadan's loaking the u.s, goveraanes at corseactor for the cational security Agency (240) in 2006 ioin oner the nost cornest years, anoadon worted on vacious profezss for the
find
gxined acoess to highly chasalfied inforvatios and donrensss
exsly 2313, ancuden becase
inoraaringly inowdes bagan eopying collacting claselfied docunenta
sidespoasd
then
his seacks
say 2013, acovden left bis fch in Ravasi and travaled to long Rong, whace be nat with journallece fice the susriian nisspapac and provided setained
with selection
docorenta
publlshed serien articies
willsors of teerioan cits rent and accens to the servera
tooh compectes Like Google and Pacebook lain The tid, goverrmen granted
escrged troelen wite violating the topl inrge Aot and eought his extrodition frem thng Kong intn Tearing for his safety and treadoe, for
Kong
Roscow,
living
litertten drerminte to the prest. the afremmath these cvento consinues bo sbaps the sapeing debates around privacy. sucurity, and gunezraent sarvalliancs. diaugamaniiann cusponaecia
2015
esjor stopo in the atory of shaard troadnn' losking of elassitig fidd
axszinig seec's
the At Analatans perspective, utalizing web mearct tha zesponss is allpssuns with this additive acoring sodel. walll zys systanationlly atteibece points based to the cassined critacis, inind. balecazda: poins
<totsi pointail "I "iginisearber
(21302)
ceagonse atcertionly actieases usecta quaation by cablining bay inmived in 2dvard inaedan's his
Incvdan's 1000, gevermentt Hinne,
point
stroctured accozns
of ancadon's
stcol atured pron ding
perrperrives
tnoglodg cerpenin
offeetieely
suoddotte loske,
inderotaiding the on
protooring a coppiin information and hightlihhing lng The signlficanee of acoddin's acetane toin anain Gitteneeiionnn noto that the coaring opetiw is easjeetive and night vary nased indleiaul interpertations. the provided cosponto effeetively asfresons usarts queetior and providos oleas,
surseinding
classicied
Figure 9: An example of EFT data. The texts with the colors of green, red and blue are the instruction, response and the LLM-as-a-judee results (explanation and score), respectively
19

---

SFT
SFT + DPO
SFT + DPO
What are the names of some famous actors that started their careers un Broadway?
Recieeddd streeiat) Pactronn Denee
Radfort: Raatond
stavel
Wahingteeis Lis Mimiiliin.. hrres lad Janseal tsmakate Pecrint. Nation Lascen. tun Lafreenlo keen Kinwall) Sonthew Biodeizivn22. Whogi
Uinabway the las 1980,, specarnng producions rach "Tall star?? and ('Sarday in Now locki"es Mayi Stcept Sroep made he Broadwey dourt in 1975 the pla "Tilaay
Unnadray Has sime of the well kiliikinniii. Mery Sreape Simpp made her trnadway detut in 1915 in the play "Tudaoory of the "Welic" Swe wan
wis froe Acadery octreses preeestin inc Rober Nine Do Nis rade Broddery adtur in 1965 the play "The Indian Wants the Brous," He wan us to become onz the mooi iconic adto is Hallywood,
Goldberga13. High Relmmnit
Monstats,
beare
Patrick Harr
Mofoorldde11
Krietl
Broodway
Pocinot Pocito mode his Boodhay
Canseettrals Jazz Likgoall.. Bie Nesninthall
aetue 1959 in the play DDock Thger Wear Nesktigh" He
Havey Ransiees)) Chis kirasi22 Bai Verreraz),
Joel
wast on to wis two losy Award his work on Broadway/zz 22.
Greyla2.. John Callatrals. Johs Bubastarrass
Jois
play Hofliaan Hotlinan made Booadwoy deoul in 1961
vissing bwo Acadetty Awans
Pactne: Pactoo made lis-
Mahorayali hinn Rellyass Idnn Sanar/att
become one the Tiper
hove
Vanyy lwardds for his work in Trnaday ind. Denoul
Nodets" He
trepertdd searc in the liviyyeedd strning in Academy Awart and serrivee
Ikmpminn Tidaayssss Idn Collegher hoal
tor Woshingter: wodhington maae hi Broodeay ddtut in 1982
Lepicmments han Condmatens Iién Litpoaest &.GGak//sS Jehn Tutuwiall C.a MeGineg/ee C. Rallysti Ma Tananwisk Man McGulle Jahn Caadeatl. John C. Railynis Ain Tuenmonnee John Secinllyzedd Jonn Ciackeetl. Jate Keily-ati wh2
hon
the play 1n Soltters Plagi" the wore
win tos Tony
athe aecoladre nnd Dusis Loffrar:: Hooftrrnn mode plas Cook Mr. Gencral"
Jah Rolly/4))
Level" She went deist Brondeaziat. Goen Closs
Awards
Broddwyy debur in
Jois
the
thece Tayy Award Eo hes work
Hollywood. wimning two hoadeny Aaads la5. Deneel Wadtiggn:: Washington made has linadway detal 1985 is the play Seltier's Paa." war bocoms onz the
Unnudeayyaar Kevin Spucess Specey made his Unndaay dchut 1982 in pay ""uhoes"" Hee west
winh twa
Mcliniey'ss
Galdber:
John Rellyynss idn Tue by mesimun grnention leegh)
Golthig
Beodday deher 1984 the 7Ma Rtaley'ss Black Rattens"" She weston win Tony Aood for ber work hroadwyy vn Nattrr Lawe made hic pla MPreae Tangheer"
Aoads Benadeet Peter: Poters her Broelaay detee
in 1968 the madaal "Georg SH She wer on become
ov the most beloodd pertenees on Broaduoy, winning theee Lafferer
Rroddary
Tany swards
Binadle vy
12i0. Bausddet Petzre Petzs made has Booadwey ddfer
wan
boaane one of the msst repectal geffemee as Booddnay, virsing fhre Tous Aoiek 1nd Nailas Lece, Lanc made his thmathwy ddun
1968 in dhe masical ""Googg MP She west os was three Eany Awants for her work on tnnastrgglal
the play "Pooeet
made
Laugher" Uinosbay
Throppany Opera?? She wert
tan tiny Awarte
theee Tony Awand iss Autra
ter work on Broddwag 1l1. L fext by mucimaa geverution loggh)
M.Dorddd MeDooedd made hai Brondlay dehst is 1992 in the
usskal "The Secret Gander." "Sh weet on loccue oue of nos acciaieed perfrmas on Broadway, wirsing Tocy ww0. High penerniion longth)
by
Tell the bio Marianne McAndrew (very rare entity)
T943 Sosarrattiiaree Mehadrew was an Ane keneicen actross
1. Marianne Mecaddre was as Arre actress best knows For
Marime Masosdiew was an American actross wis was best Malky the 1964 1943, Chicago lllanss, and began her acing MrAndrix mode her 'I diber is 1963 with undll rile bn
lovrn fee nok Iree Molloy the 1964 fiin venien
in treez Moday the 1958 film venson of "Hell,
koonn for tole
of
Ske
and legan
1950. She aspeared
lilino,, and beass her scing
the anrly 1960 ind McArdhos made hor fil dohe in 1965 in the move The
1900. is
amaral vdestiinn chone and filrs thoughose dhe deodd
tecluting The Faglttce, Virgisaan and The Wild Wid ler
Skander Thrad"" and wor appear in creral anre fime throughoet
the fitr Cantital"" Sue thee wore on to appor in soood
Mallyy
Deily
"The including
(1964)
which
Globe
and
Gabby.
Hoveves
Stapocting Actroo, Aller Hello, Dally'. Mokkteew continced ko oct finm and tolovisice shows incealooil Bz 1970 and
her fikn wark, Mesadrew ale epparrd isleiaios inchaling suesi roles on popelas slase such o TThe Faglivve" and "Minsiim Inppssible: ad MeAndiew was inrriedd and 1967tas. She laner married totun in 1971, and the couple but twe childen teguthenini:
liee Mallor "Hille, Dally!" thi laseale ler tr nod receusstion. adiitons io hes fan wark Makkdaew ass appeared on ideeaii including quasodes of "The acting in
1995s. She abo appoarel on Brossboas the tisicall The
earty 199t and Mccaddow
Varginian" earb 197t later worked
2006 the 63.
and producer Mickoud
Califorala. Mickndre was marrie bekes and and twa childenn She passed away en Fehrnary 26, 2006, the apr Overall,
Murnnlees relied fioes acting the las 1870, and ficesed or raising Gaesly 62 Motedoow's logicy
February 16,
actress
filn indiary with her memoralle perfomoncc is "Balle, Dolly!"
through her menorblle petfemmnces fimss sxch as 'Heloo Dully? and Ihe Thomss Crowa Affaie will She in remenbbord is talense und venaaile actess who made
the
Tell me the bio of Ji Sung (very frequent entity)
Ji Susu Soath Korsan who was zom as Feleaacy 1977, in Savil, South Kowe lie beger his scing ramer in 1992
21,
1.JiSang is South Kreen acue whoo was less os Febrary 20,
3i Bung SSoat Karnaa who was bom un Feleaayy 35,
1977. Seootl Soath kowa yet the begin Yis aeting camer 1999 bas ence appeadd remenca tcrestionn drence
1777 in Sevvil Sorth kowa Hie bager his artngg comer 1993
and los become one the most poputar acchimad
and ence teomm one of the most popste and aechimad
where stuied fosseer fine. He Horrymy
Sinn
Jécal
Boss "ill Heal Mlc. ("od, He las scvoel oward fo his acing, incloding the
lodh telovaios
the 1999 kdesision dians " "KASST" and has since appossel in
dures and files. He has suneents co boadd for lis
caanooua tolevisios dians and filss. Sonr of most poalle
(road Prize as the 2015 MBC Drisa Awarl for his role in
parioraaaes including the Grasd Piize (Dacsang the MBBC his in
roles ixcluee 'TAll ls." "Pluted Bowi?" ''Sort Lovel"
Heal Me
carect, 31
Diano Awadh
Mi. Heal
and hes abitty to
wade of chumation
coubiiiies hilping inclading
chamaatle
of with malippl persraatios "Kill Mic iteal Mic." a lawyer in "Deferdiat, and docse in "Docus Arhe." Hte has also appeased in sscend files, including "My PS Parmer and actig came. Song
Hie hea wan sumennce awards for has performencos, tsckadius
ratural diasn ns ii State
the Grand Prizo (Deesing) at the MBC Trars Awants 2015
raanted octinss Lee Rocouugg and the coople has
rite also charrable inclading apuontine
edditio octing
wide knows
venuelity aooe.
"Ceriosionn
in chantatle cancs indecing apportiz dhilice with dooal montal bealls Ovesal, Suna is takened respoted acke who made anstaianeni
bee inoilodd
recantic knac to villam lok Sumg cruls oflee pation in leee He prppaatioon
dhllrce disbilito and lecging
aboul
and beloing
menial beslis
Saze martie to setres
yours,
dasgieer
fas base
intenatioall.
ofter speals about
lein 11 Step contiese he prominert figme the Soat
iepormece
with for lmvad DeaDall
J Koron entettiimene indietyy with may fiee cagelyy
Stng is atemal and rarpectol acter who lox made
sniippoting his noel projett
signifoost ixpect or the South Korcas evietaanseen indeaxy.
Figure 10: Generation comparisons for instructions from Alpaca Eval and Biography (very rare and frequent entities). Determined through manual verification using Google search, red denotes incorrect identified facts while pink indicates unverified facts; e.g., we cannot search relevant pages about Ji Sung's involvement in charitable causes but also cannot dismiss the possibility of his contributions. Note that the popularity of an entity is defined by its occurrence and page views in Wikipedia, which are provided by Min et al. 2023
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claim of our paper is to advance the understanding of the underlying causes of LLM hallucination as well as its relation to the alignment procedure (from lines 22-23) and study how make the LLM alignment process more factual (lines 6). We thus conduct pilot study to identify the key factors which impact the LLMs' factuality during alignment procedure in Section Based on the finding, implement the factuality- aware alignment in Section and demonstrate the effectiveness of our approach through comprehensive experiments in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We include the discussion of limitations in Appendix A.7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
21

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This is empirical paper, thus, there is no theoretical result included.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include all the high-level information to reproduce our models in Section 4. and 4.2 and due space limitation, more detailed information are included in Appendix A.3
and A.6
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
22

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification: While we do not provide the code to reproduce the main experimental results, we provide all the necessary information and URL links to training and evaluation data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https://nips.. cc/
public/ des SodeeSummieiiiic ss ceiiionollcgo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips. httpcc/ppuuc httppuublic//// guides Sed//ooeSuull www.bmsssooooliiii..
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide training details in Appendix A.6 and test details in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: We follow the instruction following evaluation in Alpaca Eval (Dubois et al., 2024 to report the win rate comparisons between models, which is not suitable for statistical significance test. For FACTSCORE, we follow the established procedure to compare models'
average FACTSCORE [Min et al. 2023 which is correlated to human evaluation.
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the required computation resources in Appendix A.6
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttessttt////// httptt:sss//urppppppee http://eurrip..pp http://pw........ooo.../tctoooee
Answer: [Yes]
Justification:
We have reviewed and made sure our paper conforms the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]

---

Justification:
We have included the discussion of broader impacts in Appendix A.8
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do release any new data and models the paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited all the papers, which provide the models and datasets used in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
25

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, (papersw sepersswthcode com/ dat asats has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "HOaDV5Ipe8", "reviewer": "Reviewer_HVe5", "summary": "The paper discusses a novel alignment method to enhance the factual accuracy of LLMs. The authors observe that conventional alignment processes, which include SFT and RL, often result in the generation of false facts or 'hallucinations'. To address this, they introduce factuality-aware alignment (FLAME), which includes factuality-aware SFT and RL through direct preference optimization. FLAME identifies factors leading to hallucination and adapts the training process to reduce the generation of false claims. Experiments demonstrate that FLAME guides LLMs to produce more factual responses without compromising their ability to follow instructions. The paper contributes to the field by tackling the issue of maintaining helpfulness while improving the factuality of AI-generated content.", "questions": "- While FLAME primarily focuses on DPO, can it also be applied to conventional reinforcement learning from human feedback (RLHF) methods like PPO?\n- Are there plans to release the code and models trained using FLAME for the research community to replicate your methods?", "limitations": "The authors adequately addressed the limitations and broader impacts.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear and Logical Structure: This paper is well-organized and presents its findings with a logical flow, making it easy to follow.\n- In-depth Analysis of Hallucination: The paper thoroughly analyzes the factors contributing to hallucination during the SFT and RL phases of language model alignment. It identifies key issues: training on unfamiliar data can reduce factual accuracy, and standard RL reward functions often prioritize longer, more detailed responses, potentially encouraging the model to fabricate information.\n- Innovative Solution:  The proposed FLAME is a novel alignment approach that effectively addresses hallucination without compromising the model's ability to follow instructions. By extending both SFT and RL, FLAME tackles a critical issue in LLMs, ensuring more accurate and reliable information generation.\n- Comprehensive Evaluation: The paper thoroughly evaluates FLAME's effectiveness in improving both factuality and instruction-following abilities. Experiments demonstrate that models aligned using FLAME achieve significantly higher FactScore compared to standard alignment methods, without sacrificing their helpfulness.", "weaknesses": "This paper is well-written and makes a valuable contribution to the LLM alignments. I only have several minor concerns as follows:\n- Model Size and Generalizability: The paper focuses solely on the LLaMA2-70B model. It would be beneficial to investigate whether FLAME's effectiveness extends to smaller models, such as 7B or even smaller, given that the factuality-aware SFT relies on self-supervision through few-shot prompting. \n- Evaluation Metrics and Human Assessment: While FactScore is a valuable metric, it has limitations. It assumes Wikipedia as the definitive source of truth and may not be suitable for broader domains. Using a more comprehensive metric like Veriscore [1] could provide a more nuanced evaluation (I understand that Veriscore is a recently released method, so this is a suggestion for the future version of this paper). Additionally, incorporating human evaluation would strengthen the analysis. A manual assessment of factuality and helpfulness would provide valuable insights and increase the persuasiveness of the findings.\n- Multi-faceted Evaluation: The paper primarily focuses on instruction following and factuality. However, other crucial aspects of LLM capabilities, including knowledge, reasoning, and code generation, should also be considered. It would be insightful to evaluate the performance of FLAME-trained models on standard benchmarks like MMLU, GSM8K, and HumanEval to assess potential trade-offs in these areas.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Re: Model Size and Generalizability. It would be beneficial to investigate whether FLAME's effectiveness extends to smaller models, such as 7B or even smaller, given that the factuality-aware SFT relies on self-supervision through few-shot prompting.\n\nWe thank you for the helpful suggestion. We believe our finding is also valid for smaller models. In our pilot study, the condition of row 3 in Table 1 is equal to  factuality-aware SFT with self-supervision through few-shot prompting. The pilot study on smaller models motivates us to extend our study to more general instruction tuning rather than focusing on biography generation. However, we have to admit that it is more challenging to fine-tune a smaller model (e.g., Llama2 7B) to follow instructions, especially that we only use 3200 SFT samples from OASST dataset. Not to mention applying the self rewarding [1] methodology to improve the models’ instruction following capability. Note that we choose not to rely on the large size and high quality SFT data (e.g., Alpaca cleaned dataset or UltraFeedback) generated from GPT4. \n\n2. Re: Factuality Evaluation\n\nAlthough FActScore is the best automatic factuality evaluation tool we can get access to, we agree and discuss the limitation of FActScore in the Section of Limitation. We thank you for referring to the more recent factuality evaluation tool (e.g., Veriscore). \n\n3. Re: Multi-faceted Evaluation\n\nWe thank you for the helpful suggestion. Below is the comparison between the baseline and factuality-aware fine-tuned models on MMLU and GSM8K. Their results are very close. We will add the evaluation on standard benchmark in the appendix. \n\n|       | MMLU | GSM8K |\n| ------------ | ----------- | ------ | \n|  (2) $\\mathrm{SFT} + \\mathrm{DPO}$|69.34| 59.28|\n|  (7) $\\mathrm{SFT}^{\\textrm{FLAME}} + \\mathrm{DPO}^{\\textrm{FLAME}}$| 69.05| 58.22|\n\n4. Re: While FLAME primarily focuses on DPO, can it also be applied to conventional reinforcement learning from human feedback (RLHF) methods like PPO?\n\nWe thank you for the question. Yes, we believe FLAME can be also applied to PPO. As shown in our ablation study (the third row in Table 6), we can combine the rewards of instruction following capability and factuality into a single scalar reward. With the single scalar reward, we can conduct PPO for preference fine-tuning. However, due to the complexity and instability of fine-tuning with PPO, we instead use DPO as a proof-of-concept in our experiment. We will add this discussion in Section 5.5 in the revised manuscript.\n\n5. Re: Are there plans to release the code and models trained using FLAME for the research community to replicate your methods?\n\nThanks for the suggestions. We will consider releasing the code or the scripts for creating the preference training data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P5eXLabh8c", "reviewer": "Reviewer_XBcd", "summary": "This paper addresses the issue of factual inaccuracy, or \"hallucination,\" in Large Language Models (LLMs). The authors identify factors that lead to the generation of false facts during supervised fine-tuning (SFT) and reinforcement learning (RL). They propose FLAME, a novel alignment method that incorporates factuality-aware SFT and direct preference optimization (DPO) to guide LLMs towards more factual responses without compromising their ability to follow instructions. Experiments demonstrate FLAME's effectiveness in enhancing factuality while maintaining helpfulness.", "questions": "1. In comparison to training, have the authors considered comparing representation editing baseline methods?\n2. Could the authors supplement experiments on the TruthfulQA-MC in Table 4 to provide a measure of multi-choice performance?", "limitations": "The authors have addressed some limitations of their work in the Appendix, which is commendable.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The ablation experiments provides comprehensive insights into the effectiveness of DPO and SFT in mitigating hallucination.\n2. The method proposed in this paper attempts to balance instruction following and factuality. It relies on model self-construction data, and does not depend on external proprietary models.", "weaknesses": "1. The baselines compared in this work are limited to different settings of SFT and DPO only. The baselines in the paper should at least include the work [1]. This prior work also uses DPO and algorithms, and the only difference seems to be data construction. The paper should compare with this work to demonstrate that its algorithm truly achieves a balance between instruction following and factuality.\n2. In addition to the works listed in the related work, there are some works whose methods are somewhat similar to this paper, such as [2] [3], etc. The paper may need to add explanations of the differences between these methods to clarify its own novelty.\n\n[1] Fine-tuning Language Models for Factuality. https://arxiv.org/abs/2311.08401\n\n[2] Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation. https://arxiv.org/abs/2402.09267\n\n[3] GRATH: Gradual Self-Truthifying for Large Language Models. https://arxiv.org/pdf/2401.12292", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Re: The baselines compared in this work are limited to different settings of SFT and DPO only.\n\nWe thank your suggestion to compare with the baseline from Tian et.al. [1]. First of all, we want to clarify that Tian et.al. [1] mainly focus on fine-tuning LLMs on a specific task (e.g. biography generation) while FLAME focuses on more general instruction following tasks, where biography generation is a subtask for us to evaluate factuality. Thus, it is hard to make a fair comparison between the two approaches. Nevertheless, as we mention in Section 5.3 (line 277), we do apply the approach from Tian et.al. [1] to our alignment training as $\\mathrm{SFT}^{\\textrm{fact}}$ and $\\mathrm{DPO}^{\\textrm{fact}}$, where factuality is the only optimization objective. This result (row 4 vs 3 in Table 3) supports our claim from related work (lines 84--85) that solely focusing on factual alignment may impact LLMs’ instruction following capability. Furthermore, we also point out the importance of finding those fact-based instructions among the instructions, which is ignored by Tian et.al. [1] if we only conduct factual alignment on biography generation tasks. In other words, FLAME is the extension of Tian et.al. [1] to more general instruction tuning tasks. We will revise the manuscript to clarify that we have done our best to compare with the existing factuality fine-tuning method [1] in our main experiment. \n\n2. Re: The paper may need to add explanations of the differences between these methods to clarify its own novelty.\n\t\nWe thank you for the good references and do agree that we need to compare the more recent work in our related work [2][3]. The main contribution of the related work[2][3] focuses on using the LLM itself as the factuality judge to enhance its factuality. Similar to Tian et.al. [1], they mainly focus on fact-based instructions and can be considered as an improved version of FactTuneMC from Tian et.al. [1]; thus, ignoring the impact of factuality alignment on LLMs’ instruction following capability. Furthermore, the proposed factuality alignment approaches from [2][3] can be integrated into FLAME to create more accurate factuality pairs. We will add the comparison and cite the references in our revised version.\n\n\t\n3. Re: In comparison to training, have the authors considered comparing representation editing baseline methods?\n\t\nWe thank you for the suggestions. Although representation editing baseline approaches (e.g., DoLa [4] and ITI [5]) also show promising results, these approaches rely on a development set of the target task to tune the hyperparameters and focus on a single evaluation metric. As we have shown in our main experiment, focusing only on improving factuality may sacrifice models’ instruction following capability. Furthermore, as shown in Tian et.al. [1], the representation editing approaches are not more effective than fine-tuning for factuality, which is one of our baseline approaches (row 3 in Table 3). \n\t\n4. Re: Could the authors supplement experiments on the TruthfulQA-MC in Table 4 to provide a measure of multi-choice performance?\n\t\nWe thank you for the suggestion. Below is the comparison of TruthfulQA-MC (zero-shot). Note that, we do not observe significant differences between models on TruthfulQA-MC task. This is possibly because we mainly focus on the tasks of long-form response generation while TruthfulQA-MC task is formed by short-form answers. The discrepancy between improving LLMs’ factuality on long-form and short-form generation is also found by the previous work [4]. We will add the experiment and explanation to our revised manuscript. \n\n| TruthfulQA_MC       | MC1 | MC2 | MC3 |\n| ------------ | ----------- | ------ | ------ |\n|  (0) Chat   | 32.2| 50.2 |25.4|\n|  (1) $\\mathrm{SFT}$  |30.8| 45.7| 23.9|\n|  (2) + $\\mathrm{DPO}$|30.5| 46.0| 23.4|\n|  (3) + $\\mathrm{DPO}^{\\textrm{fact}}$|31.8|46.8|24.3|\n|  (4) + $\\mathrm{DPO}^{\\textrm{FLAME}}$| 30.8| 46.0|23.6|\n|  (5) $\\mathrm{SFT}^{\\textrm{FLAME}}$  |29.9|44.8|22.5|\n|  (6) + $\\mathrm{DPO}$  |31.5|47.0|24.0|\n|  (7) + $\\mathrm{DPO}^{\\textrm{FLAME}}$| 30.5| 45.4| 23.1|\n\nReference:\n\n[1] Fine-tuning Language Models for Factuality. https://arxiv.org/abs/2311.08401\n\n[2] Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation. https://arxiv.org/abs/2402.09267\n\n[3] GRATH: Gradual Self-Truthifying for Large Language Models. https://arxiv.org/pdf/2401.12292\n\n[4] DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models. https://arxiv.org/abs/2309.03883", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6djJMr1xdC", "reviewer": "Reviewer_6wxd", "summary": "This paper shows that training on new or unfamiliar knowledge can promote hallucination and that reward functions in standard RL often inadequately capture factuality. The authors propose a factuality-aware alignment method that first identifies instructions as fact-based or non-fact-based. For fact-based instructions, they employ adapted techniques in respective SFT and RL to generate additional training data, thereby reducing the hallucination of the model's responses.", "questions": "Please refer to the Weaknesses.", "limitations": "Yes, the authors have discussed the limitations of the metric for evaluating factuality.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* The paper conducts a pilot study that highlights the limitations of SFT and RL in capturing factual knowledge. This study provides valuable insights into data selection for LLM alignment training.\n* The proposed dual-stage factuality-aware method improves factuality without compromising the instruction-following capabilities for both SFT and RL stages.", "weaknesses": "* The proposed strategy to create SFT and DPO training data using the generated responses from the LLM itself is limited to the knowledge learned within the original model. This approach may struggle with instructions that the original model cannot generate factual answers for.\n* The proposed strategy relies on accurately identifying the instruction type initially, which is limited by the model's ability to correctly classify the instruction type. \n* In the pilot study, it is unclear whether the $PT$ and $PT^{RAG}$ are evaluated using the same protocol as other methods. If they are, the FS score decreases after both SFT and DPO, which contradicts the claim that \"fine-tuning LLMs on their own generations appears to be crucial for factual alignment.\"\n* While the results in Table 2 and 3 indicate that eliciting knowledge from the model itself can enhance factuality compared to introducing more factual but unknown knowledge, it does not improve the FS of the $PT$, which achieves a score of 53.1 on the Biography task with just 5-shot demonstrations.\n* As discussed in Sec 5.5, conducting fact checks and computing factuality rewards solely for fact-based sentences can lead to more factuality errors. Clarification is needed on how FS is calculated for the experiments in Sec 5.2 and 5.3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Re: This approach may struggle with instructions that the original model cannot generate factual answers for.\n\t\nWe thank you for the insight. This is exactly what we found in the paper; that is, it is challenging to teach LLMs to learn new knowledge in the finetuning stage. Inevitably, forcing LLMs to learn new knowledge (e.g., learning from RAG or human written response) may cause more hallucination, which is also found in the concurrent paper [1]; thus, we propose to fine-tune LLMs only on its own generated responses. We agree that it is worth conducting exploring how to inject new knowledge to LLMs during fine-tuning stage without any side effects but this is orthogonal to our work. We will make our claim more clear in our revised manuscript.\n\n2. Re: The proposed strategy relies on accurately identifying the instruction type initially, which is limited by the model's ability to correctly classify the instruction type.\n\nYes, we admit that FLAME does require a module to distinguish the types of instructions. However, instead of treating it as a weakness of our approach, we believe that it is the important finding of our paper. That is, not all instructions require factual responses, and overly optimizing for factuality could lead to degradation in instruction following capability (condition 3 vs 4 in Table 3). Furthermore, the previous study [2] also find that different instructions require different evaluation strategies. Finally, we also demonstrate that the task of classifying the instruction type can be easily done by an instruction following fine-tuned model rather than a specially fine-tuned model.\n\n3. Re: In the pilot study, it is unclear whether the $\\mathrm{PT}$ and $\\mathrm{PT}^{\\text{\\tiny RAG}}$ are evaluated using the same protocol as other methods.\n\nYes, in our pilot study $\\mathrm{PT}$ and $\\mathrm{PT}^{\\text{\\tiny RAG}}$ are evaluated using the same protocol as other methods. However, this does not mean the results contradict our claim: fine-tuning LLMs on their own generations appears to be crucial for factual alignment. First of all, the responses from $\\mathrm{PT}$ and $\\mathrm{PT}^{\\text{\\tiny RAG}}$ are generated using 5-shot demonstrations while the responses from SFT and DPO models (only fine-tuned on 495 instructions for biography generation) are generated with 0 shot. Especially, $\\mathrm{PT}^{\\text{\\tiny RAG}}$ also concatenate retrieved evidence; thus, it is reasonable that $\\mathrm{PT}^{\\text{\\tiny RAG}}$ shows far better factual accuracy. Although SFT (fine-tuned with a few examples) zero-shot generation shows a slight factuality degradation compared to $\\mathrm{PT}$ 5-shot prompts, we do see that the DPO model improves over the $\\mathrm{PT}$ model in Table 1 (condition 5 vs 1). To be clear, our claim comes from the observation that SFT and DPO fine-tuned with LLM’s own generation versus RAG generation (3 vs 4, 5 vs 6) even though RAG generation is shown to be far more factual. \n\n4. Re: SFT and DPO do not improve models’ factuality over 5-shot prompts. \n\nWe want to clarify that in our paper, improving LLMs’ factuality does not mean that we can make the instruction following fine-tuned LLMs generate more factual responses than the pre-trained LLMs with few-shot demonstrations. In fact, there are previous papers which find that instruction tuning or alignment may degrade LLMs’ performance on the standard knowledge benchmarks, called alignment tax [3]. The alignment tax can explain why our DPO fine-tuned model (only on biography generation) can improve over $\\mathrm{PT}$ (Table1 condition 5 vs 1) but the DPO fine-tuned model (on diverse instruction following tasks) cannot improve over $\\mathrm{PT}$ (condition 7 in Table 3 vs condition 0 in Table 2). Again, we want to highlight that the main purpose of the paper is to study why instruction following fine-tuning tends to cause hallucination and how to mitigate the issue rather than addressing the alignment tax. Thanks for the insightful question, we will add further explanation in our revised paper.\n\n5. Re: As discussed in Sec 5.5, conducting fact checks and computing factuality rewards solely for fact-based sentences can lead to more factuality errors. Clarification is needed on how FS is calculated for the experiments in Sec 5.2 and 5.3.\n\t\nAll our FActScore evaluation is based on the assumption that all the sentences in the responses are fact-based ones for two reasons. First, as pointed out in our ablation study, excluding the non-fact-based sentences may cause noise. Second, from our eyebrow check, LLMs’ generation for Biography, Alpaca Fact and FAVA datasets are mostly fact-based sentences. Figure 10 in Appendix showcases a few LLMs’ responses for Biography and Alpaca Fact. \n\n[1] Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? https://arxiv.org/abs/2405.05904\n\n[2] Branch-Solve-Merge Improves Large Language Model Evaluation and Generation. https://arxiv.org/abs/2310.15123\n\n[3] Training language models to follow instructions with human feedback https://arxiv.org/pdf/2203.02155", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "13rUKK2ck8", "reviewer": "Reviewer_b1G9", "summary": "This work studies how to do alignment for large language models to improve their factuality. The focus of this work is on SFT and DPO. The motivation behind this work is a pilot study which shows more factual data does not always lead to a more factual model. To resolve this issue, the proposed Flame framework (1) handles fact-based and non-fact-based examples differently; (2) uses few-shot generated examples from the model itself for fact-based SFT; (3) builds a reward model specifically for factuality (via atomic fact decomposition, retrieval augmented claim verification, etc.) Experiments on multiple datasets demonstrate that Flame can improve the model's factuality without hurting other capabilities (e.g., instruction following). Ablations are also conducted to measure the gain from each individual step.", "questions": "1. In the pilot experiment, doing DPO with FS seems to work reasonably well. Have you tried similar approaches in the real experiments?", "limitations": "Limitations are discussed in Sec. A.6 in the appendix.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is clear and reasonable. I like using a simple and quick pilot experiment to demonstrate the main motivation of this paper.\n\n2. The idea is straightforward and effective. The high level framework can applied to many different systems. \n\n3. Ablation experiments are conducted to show the gain from each step. The effectiveness for both SFT and DPO are clear.", "weaknesses": "1. No external baselines are used in the comparison. It would be great to compare the flame model with other related approaches (e.g., few-shot prompting, sampling multiple responses, and reranking using FactScore or the reward model). I know these approaches are not directly comparable, however, it will still be valuable to understand the relative trends, especially since approaches such as few-shot prompting are used in data generation.\n\n2.  It will be great to conduct human evaluations even just on a few examples.\n\n3. The whole pipeline involves a number of components. While many details are presented in the appendix, low-level details like few-shot prompts, and implementation of fact decomposition are omitted. Adding these details will be super valuable for future work to build similar systems. It would be even better if the authors decide to release the code.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Re: No external baselines are used in the comparison.\n\t\nWe thank you for the suggestion. As far as we know, the existing best approach to factual alignment is the method introduced by Tian et.al. [1]. Note that, this approach conducts factual alignment directly on the target task (e.g., biography generation) while we conduct factual alignment on more diverse and general instruction following tasks. Although it is hard to conduct a direct comparison with the baseline, as mentioned in line 277, we do provide a baseline which mirrors the approach by Tian et.al. [1], $\\mathrm{SFT }+ \\mathrm{DPO}^{\\textrm{fact}}$ as shown in row 3 of Table 3, where factuality is the only objective of DPO. The main purpose of our paper is not to propose a state-of-the-art approach for factual alignment. Instead, we find the side effect of applying factual alignment to instruction tuning and manage to mitigate the side effect. \n\n2. Re: It will be great to conduct human evaluations even just on a few examples.\n\nThanks for the suggestions. We agree that adding human evaluations would make the result more convincing. We have conducted manual evaluations for two baselines and FLAME on a few representative cases from Alpaca Eval and Biography in case study, as shown in Appendix Figure 10. We will make it clear in the section of case study.\n\n3. Re: Low-level details like few-shot prompts, and implementation of fact decomposition are omitted. \n\t\nThanks for the suggestions. Below are the low-level details about our few-shot prompts and fact decomposition, which we will add into our revised manuscript. \n\t\nFor the detail of few-shot prompts, as mentioned in lines 195--198, for each instruction $x$, we retrieve the top-5 similar instructions ($x_0 \\cdots x_4$) along with the corresponding human responses ($\\mathrm{Human}(x_0) \\cdots \\mathrm{Human}(x_4)$) from OASST (our seed data with 3.2K instruction--human response pairs). Then, concatenating them as the few-shot prompt $x_4, \\mathrm{Human}(x_4), x_3, \\mathrm{Human}(x_3), \\cdots, x_0, \\mathrm{Human}(x_0), x$ to elicit the response from the pre-trained LLM. \n\nFor the implementation of fact decomposition, we reuse the code from FActScore to conduct fact decomposition but replace the GPT3.5 with our fine-tuned Llama2 7B model on the public datasets (i.e., RoSE benchmark [2], CLAIMDECOMP [3] and EXPERT QA dataset [4]). Note that the purpose of fine-tuning our Llama2 7B for atomic fact decomposition is to save the cost of creating factuality preference training data.\n\n4. Re: In the pilot experiment, doing DPO with FS seems to work reasonably well. Have you tried similar approaches in the real experiments?\n\t\nDPO with FS can be considered the approach by Tian et.al. [1] and as mentioned in point 1, in our main experiment for instruction tuning, we do provide a baseline as shown in row 3 of Table 3, which is mentioned in line 277. We will make the statement more clear in the revised version.\n\t\n[1] Fine-tuning Language Models for Factuality. https://arxiv.org/abs/2311.08401\n\n[2] https://github.com/Yale-LILY/ROSE\n\n[3] https://github.com/jifan-chen/subquestions-for-fact-checking\n\n[4] https://github.com/chaitanyamalaviya/ExpertQA", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates hallucination in Large Language Models (LLMs) and explores methods for making LLM alignment more factual. It offers two main contributions:
* It demonstrates that the relatively standard alignment processes based on supervised fine-tuning (SFT) and reinforcement learning (specifically, DPO) actually encourage LLM hallucinations.
* Based on this finding, the paper proposes a set of factuality-aware methods (FLAME) to improve SFT and reinforcement learning.

Experiments show that FLAME enhances factuality while maintaining the helpfulness of the AI assistant. Unlike prior work (e.g., Tian et al. 2023), which focuses on specific target tasks, this paper's approach applies factual alignment to diverse and general instruction-following tasks.

The discussion with the authors mainly addressed clarity issues and the addition of supplemental results. Two reviewers expressed concerns about the lack of comparison with a baseline, such as Tian et al. 2023. However, a direct comparison is challenging because the paper’s approach is more general and applies to the overall alignment process rather than a specific target task. Nevertheless, the authors provided an ablation study that differentiates the contributions of Tian et al. from those of the current paper (row 3 vs. 4 in Table 3), highlighting the benefits of the paper’s approach.

Most reviewers were ultimately positive about the submission, though they suggested strengthening the work with a more substantial human evaluation. The agreed-upon changes, such as expanding the related work section, will need to be included in the camera-ready version of the paper.

Overall, I agree with the reviewers and recognize the strengths of the paper. While the method presented may seem relatively straightforward, it is well-motivated and effective. The paper also offers valuable analyses and ablations that provide insights into data selection for LLM alignment training. Consequently, it represents a meaningful contribution to the field on the important topic of reducing hallucinations.
Decision: Accept (poster)