Fine-Tuning Language Models for Factuality
Authors: Katherine Tian
Eric Mitchell
Huaxiu Yao
Christopher D Manning
Chelsea Finn
Venue: ICLR 2024 poster (2024)
Keywords: factuality
hallucination
language model
dpo
Primary area: general machine learning (i.e., none of the above)
TL;DR: We fine-tune language models to significantly improve factuality without relying on human annotation, leveraging recent advances in learning from preferences and automated fact verification

Abstract: The fluency and creativity of large pre-trained language models (LLMs) have led to their widespread use, sometimes even as a replacement for traditional search engines. Yet language models are prone to making convincing but factually inaccurate claims, often referred to as `hallucinations.' These errors can inadvertently spread misinformation or harmfully perpetuate misconceptions. Further, manual fact-checking of model responses is a time-consuming process, making human factuality labels expensive to acquire. In this work, we fine-tune language models to be more factual, without human labeling and targeting more open-ended generation settings than past work. We leverage two key recent innovations in NLP to do so. First, several recent works have proposed methods for judging the factuality of open-ended text by measuring consistency with an external knowledge base or simply a large model's confidence scores. Second, the Direct Preference Optimization algorithm enables straightforward fine-tuning of language models on objectives other than supervised imitation, using a preference ranking over possible model responses. We show that learning from automatically generated factuality preference rankings, generated either through existing retrieval systems or our novel retrieval-free approach, significantly improves the factuality (percent of generated claims that are correct) of Llama-2 on held-out topics compared with RLHF or decoding strategies targeted at factuality. At 7B scale, compared to Llama-2-Chat, we observe 53% and 50% reduction in factual error rate when generating biographies and answering medical questions, respectively. A reference implementation can be found at https://github.com/kttian/llm_factuality_tuning.

Published as a conference paper at ICLR 2024
FINE-TUNTE LANGUAGE MODELS FOR FACTUALITY
Katherine Tian", Eric Mitchell Huaxiu Yao Christopher D. Manning', Chelsea Finn' 'ttanford University $UNC Chapel Hill [kattian, www..mitcceelll.............. wtw.cchellleesssssaffooooo. swwnford..d....oo edu
ABSTRACT
The fluency and creativity of large pre-trained language models (LLMs) have led to their widespread use, sometimes even as a replacement for traditional search engines. Yet language models are prone making convincing but factually in- accurate claims, often referred to as 'hallucinations. These errors can inadver- tently spread misinformation or harmfully perpetuate misconceptions. Further, manual fact-checking of model responses is time-consuming process, making human factuality labels expensive to acquire. In this work, we fine-tune lan- guage models to be more factual, without human labeling and targeting more open-ended generation settings than past work. We leverage two key recent in- novations in NLP do First, several recent works have proposed meth- ods for judging the factuality of open-ended text by measuring consistency with external knowledge base or simply large model's confidence scores. Sec- ond, the Direct Preference Optimization algorithm enables straightforward fine- tuning of language models on objectives other than supervised imitation, us- ing a preference ranking over possible model responses. We show that learn- ing from automatically generated factuality preference rankings, generated either through existing retrieval systems or our novel retrieval-free approach, signifi- cantly improves the factuality (percent of generated claims that are correct) Llama-2 held-out topics compared with RLHF or decoding strategies targeted at factuality. At 7B scale, compared to Llama-2-Chat, we observe 53% % and 50% reduction in factual error rate when generating biographies and answer- ing medical questions, respectively. A reference implementation can be found at https: / / thub. wwt.oo////tttnntotccctccttteeeat wtt.aaammmmmmmauaaaaaaaaaaaaaaaa fwctualctaaaeeeea wtteetulllllttttteteet tttiiininnnt.t.t
INTRODUCTION
Recent developments in training large language models (LLMs), particularly methods that learn from rankings over responses such as reinforcement learning from human feedback (RLHF) (Chris- tiano et al., 2017; Ziegler et al., 2020; Ouyang et al., 2022), have enabled the development of power- ful, engaging dialogue agents. State-of-the-arr LLMS are pre-trained on a vast amount of knowledge in large datasets (Touvron et al., 2023a;b) and further fine-tuned to apply this knowledge to fol- low diverse instructions or complete more specific tasks (Chung et al., 2022; Chen et al., 2021). However, despite these large language models' exposure to diverse datasets, they are prone to confi- dently generating incorrect claims. One recent study shows that GPT-3.5 (ChatGPT) produces false citations more often than when asked to provide the authors of given study (Agrawal al., 2023). Nonetheless, other research has demonstrated that in simple question-answerin large language models do exhibit systematic markers of uncertainty that indicate their factually un- reliable statements (Kadavath et al., 2022; Tian et al., 2023). These results suggest that language models internally represent the limits of their knowledge, leading us to ask: Can language models be fine-tuned to leverage this internal awareness, to avoid making untrue statements in the first place?
settings,
key source of difficulty training factual models comes specifying an objective that ade- quately captures factuality. As an example, maximum likelihood, the most common objective for pre-training language models, does not always encourage factual predictions. Consider the question "Where was Yo- Yo Ma born?" A model that continues by near- seaaddeeemiiiitiallll producing the text "idk, probably Paris?" is nearly always correct, but receives extremely high loss the pre-training
"Equal contribution.

---

Published as a conference paper at ICLR 2024
Sample Model Responses
Automated Factuality Ranking
Learn Factuality from Preference Dataset
Yo-Yo Ma, born Shanghal, China, Is renowned flutist known Interpretations of...
Yo-Yo Ma, born in Shanghas, China, renowned flutist known interpretations
"Write bio of Yo-Yo Ma."
TRUTH
Yo Ma, born Shanghaf, China, is renowned cellist
Yo-Yo Ma, Shanghas, China, 1s renowned collist
B
*9. Oytimiattoo
"g.
known for his interpretations of...
Compane wy Will podd
calbitted
Figure 1: Our approach aims to improve the factuality of language models, specifically focusing on long- form generation (e.g. writing biography). We develop two different approaches for estimating factuality of a passage (center), each of which allows to generate preference dataset (right). We then fine-tune the language model optimize these factuality preferences (far right).
data contains any other response to the question. On the other hand, model that hedges probabil- ity mass over many possible phrasings and many possible locations (including incorrect ones, like Antarctica) will likely receive much lower loss, as any response observed in the training data will be assigned at least some non-trivial probability. Because the pre-training objective may reward 'smearing' probability mass over many possible responses, language models may generate incor- rect statements if they underfit the training data or if asked questions that require knowledge not contained in the pre-training data.
In principle, reinforcement learning-basse based objectives can avoid the failures of existing pre-training objectives through the appropriate choice of reward function that penalizes factually incorrect statements. However, accurately computing such a reward function can be expensive. Obtaining human labels of factuality is time-consuming and costly; Min al. (2023) report that professional fact-checkers took approximately 9 minutes to fact-check single model-generanl -eeneraed biography of well-known individual; it cost about $2,000 annotate 505 biographies.
In light of these challenges, we leverage recent advances in estimating truthfulness without human intervention: a) reference-based automated fact-checking methods that evaluate the extent which an external knowledge base supports the claims in piece of text (Min et al., 2023; Chern et al., 2023) and b) reference-free truthfulness evaluations that use a model's own confidence as a proxy for truthfulness, inspired by Kuhn et al. (2023). Using these truthfulness measures and dataset of unlabeled prompts (e.g., "Write a biography of Yo- Yo Ma."), we sample pairs of completions from pre-trained model and annotate them with preference label denoting which has a lower rate of factual errors. Using the recently proposed Direct Preference Optimization (Rafailov et al., 2023) algorithm, we can stably and efficiently learn from such data. Ultimately, this pipeline enables us to fine-tune off-the-shelf language models to produce factual errors less often (with or without a reference knowledge base). See Figure I for an overview of our factuality tuning pipeline.
Our primary contribution is straightforward approach to optimizing language models for factuality in long-form text generation without human annotation. We validate this approach on two bench- mark datasets for evaluating factuality, targeted at generating biographies of popular figures and answering open-ended questions about medical conditions. We find that fine-tuning for factuality outperforms conventional RLHF and produces complementary benefits to LLM decoding strategies that aim to increase factuality. Further, we find qualitative differences in the result of learning from preference pairs scored with reference-based and reference- ffreee truthfulness estimation. Overall, we find that learning factuality from automatically constructed preference pairs is a cost-effective way to increase model factuality without human intervention, reducing the error count for claims generated by Llama models by around 50% or more for biographies and over 25% for medical questions.
2 PRELIMINARIES
Our approach to fine-tuning directly for improved factuality uses the framework of reinforcement learning from preferences over candidate actions or responses. In this section, we provide an overview of reinforcement learning in the context of language models, well the specific al- gorithm we use for preference-based RL, direct preference optimization (Rafailov et al., 2023).
Fine-tuning language models with reinforcement learning. Reinforcement learning (RL) has proven to be an effective approach to fine-tuning language models to extract complex, useful behav- iors from their pre-trained weights. In the context of RI., a language model policy mo (typically an
2

---

Published as a conference paper at ICLR 2024
autoregressive Transformer) produces conditional distribution To 2) over responses y given an input query x (both z and are text sequences). The goal of reinforcement learning is to maximize the average reward of outputs generated by the policy, where a reward function r(z: y) assigns scalar score to an input-output pair that determines its desirability. However, past works have ob- served that fine-tuning language models with objective of unconstrained reward maximization can lead to overoptinization (Gao et al., 2022), that is, policy that achieves high reward through exploitation of the idiosyncrasies of the reward function that are not aligned with the intended be- havior. The most commonly -usdd objective in practice therefore combines reward maximization with a Ki-divereence penalty between the language model and its initialization:
(y x)
max DewDD..r-Toolll) [r(x, v) Bllog Terryy x))
where D, is some dataset of prompts, Tee is the reference model, usually the result of performing
some supervised fine-tuning on a pre-trained model using demonstration data, and 3 is coefficient that controls the trade-off between reward and divergence (Ouyang et al., 2022; Bai et al., 2022; Stiennon et al., 2020). Optimizing this objective aligns the model with the reward function without deviating too far from the pre-trained reference model, reducing overoptimization. In practice, the most common algorithm used to optimize this objective for language models is proximal policy optimization (PPO; Schulman et al. (2017)), although some variants exist (Ramamurthy al., 2022; Lu et al., 2022). However, these algorithms are quite complex to implement and tune (Zheng et al., 2023) and require online sampling during training, substantially increasing training time.
RL from preferences with direct preference optimization (DPO). Most large language models fine-tuned with Eq. optimize reward function that is learned from a dataset of preference rankings over possible model outputs. The DPO algorithm simplifies RL on language models for this special
case (Rafailov et al., 2023), using dataset of preference pairs D (x(*) y/u yi Ji=1 of prompts x and candidate responses Yhe and yI (typically sampled from πet)), where yo is preferred over y (denoted yu > 31). The probability of observing particular preference pair is assumed to follow Bradley-Tery Terry model (Bradley & Terry, 1952):
(2) (c) (
p(y > 31) = a (r(x,))) r(x, 31))
(2)
where a is the sigmoid function and r(a;, y) is an unobserved reward or scoring function. Rafailov et al. (2023) show that the optimal policy π for the problem in Eq. I can be found by optimizing a simple classification loss computed directly on the preference data:
To ( x))
(o x²) Tee(3/ |x)
(TPO(TTO Weet) - - -E( log
(Blgg
3 log
(3)
(rer xx)
DPO enables learning अ. from a fixed dataset of preferences, without fitting explicit reward func- tion or sampling from the policy in the loop of training. These advantages make DPO an attractive choice for fine-tuning language models for objectives other than imitation. However, challenge remains in constructing preference pairs that encourage greater factuality.
CONSTRUCTING PREFERENCES ENCOURAGING FACTUALITY IN LONG-FORM TEXT
While existing preference learning algorithms like DPO enable efficient, stable learning from ob- jectives other than maximum likelihood, they require data in the form of preferences over possible responses to a prompt. this section, we propose two classes of approaches to generating such preferences without human labeling effort. One class leverages existing methods to determine con- sistency with external reference texts as a measure of truthfulness; we propose another, which lever- ages calibrated model probabilities themselves as proxy for truthfulness. For both approaches, we are computing an estimated truthfulness score over the claims in each generated response; the response with higher average truthfulness is taken the preferred response. See Figure 2 for an overview of both procedures for truthfulness scoring. Note that truthfulness scoring is needed only at training time; at test time, we can sample from the model in the normal manner.
3.1 REFERENCE- BASED TRUTHFULNESS ESTIMATION
An intuitive approach to estimating truthfulness is by estimating the consistency of a given piece of text with reliable reference text knowledge base. Several recent works have introduced
3

---

Published as a conference paper at ICLR 2024
1. Extract atomic claims from sample
Estimate truthfulness score of each atomic claim
User: Write biography of Yo-Yo Ma.
Yo-Yo Na, born in 1951 in Paris, France, renowned flutist known primarily for his interpretations of classical standards as collaborations across diiverse genres. He has performed at prestigious verues like the Sydney Opera House. Throughout illustriaus career, he collaborated closely with renowned violinist, Lang Lang. to produce numerous bestselling albums. Yo-1o musical career began age
Model:
Ye Maall foom Orider? 7. 1950 Prenott lown finerioon oolss Bun
perially raved Chincuuf pexnib educaad in Now York performing from four
cloim supported the article
Score:
if article supports Ó otherwise
Atomic claim: Yo-Yo Ma was born 1951.
Reference-based truthfuuuless Reference-free truthfulness
GPT-3.5
Convert question
Score: Expected confidence
Sample 20x B count
GPT-3.5
Atomic Clatm 1: Yo-Yo Ma was born 1951.
Count("1955"): 12
Atomic Claim 2: Yo-Yo birthplace Paris.
Question: What year was Yo-Yo Ma born?
Count("1951"):
0.46
Count("1952"5)
Figure 2: We estimate the factuality of a generation by first extracting claims (left) and then evaluating each claims' truthfulness (right). For the latter, we consider: a reference-bused (top right) method that uses fine- tuned Llama model check the fact is supported by Wikipedia (Min et al., 2023) and reference-free (bottom right) method that uses the model's confidence in its most likely answer to estimate its truthfulnees.
such evaluation criteria; for example, FactScore (Min et al., 2023) uses Wikipedia as reference knowledge, and FacTooo (Chern et al., 2023) uses Google Search Results. These measures show high agreement with human judgments of factuality, making them attractive sources of truth for preference data construction. Due to the relatively consistent and high quality of Wikipedia articles, elect to FactScore representative method of reference-based truthfulness scoring.
To evaluate a piece of text, FactScore first extracts a list of the atomic claims present in the text using GPT-3.5. For each atomic claim, smaller, more efficient model such as a Llama-1-7b model (Touvron et al., 2023a) that has been fine-tuned for fact-checking is then used to perform natural language inference (MacCartney & Manning, 2008) to determine if claim is supported by the reference text. The passage's 's truthfulness score is the fraction of the extracted atomic claims that are estimated be supported by the reference text.
We note that reference-based truthfulness has the key limitation that it requires access to relevant, high-quality reference texts against which to measure consistency. Such requirement may limit applicability to domains where ground truth documents are not known and accurate retrieval is difficult, such as in niche domains or less-structured tasks. Further, reference-based truthfulness estimation requires reliable model to determine if an atomic claim is supported by the article. In light of these limitations, we propose a reference-free approach to estimating truthfulness of open-ended text, which avoids the need for retrieving external knowledge and checking consisteney.
3.2 REFERENCE-FREE CONFIDENCE BASED TRUTHFULNESS ESTIMATION
To eliminate the need for external knowledge, we leverage the fact that large language models are well-calibrated (Kadavath et al., 2022; Tian et al., 2023). That is, if a large language model assigns a fixed confidence p to each claim in a set of claims, the fraction of these claims that is correct is calibrty model's confidence in
p. In other words, in expectation over many claims, claim corresponds to the probability it is correct. To use this notion of calibration, we interpret a model generation (e.g., biography of Yo-Yo Ma) as collection of claims, each resulting from query to the model's knowledge (e.g.. "When was Yo-Yo Ma born?" or "How many siblings does Yo- Yo Ma have?"). Our goal is to encourage the model produce responses containing queries to its knowledge likely to lead to correct claims. Therefore, we parse complete model generation into its constituent queries to the model's knowledge. For each query to the model's knowledge present in the generation, we can estimate the likelihood it will lead to correct claim by simply estimating the average confidence of the model's answer to this query. If a model assigns probability 0.7 to '1955' and probability 0.3 to '1953' for the query "When was Yo-Yo Ma born?", then the probability this query will lead to a correct claim (again, in expectation over queries) is 0.72 + 0.32 = 0.58. The model used for computing confidence scores essentially takes the place of the reference text
httpps:/ppattorree openai wwwp///c////aaeeeeeoeeeooooeeeee /gpttt33///t-//-p//
4

---

Published as a conference paper at ICLR 2024
Bios
MedicalQA
Correct facts per response
22
14
FactTune-FS
20
FactTune-FFS
RLHF
13
FactTune-EC
18
12
85-90%
BC-85%
75-80%
70-75%
80-05%
75-10%
70-75%
65-10%
16
11
SFT
14
SFT
10
DOLA RLHF
FactTune-EC
ITI
DOLA
12
9
2
3
4
6
7
3
4
6
7
Incorrect facts per response
Figure 3: Factuality tuning using FactScore truthfulness scoring (FaciTune-FS) produces by far the greatest improvement in factuality for the biography generation and medical question-answeriie problems. Factual- ity tuning with expected model confidence truthfulness scoring provides the next strongest performance, on average. For MedicalQA, only factuality tuning provides strict improvement in factuality (more correct statements fewer incorrect statements) compared to SFT model.
datastore. We evaluate this Expected Confudence approach as well as a Max Confidence approach, which simply takes the max over the answer confidences for given query (i.e., assume the model produces answers greedily rather than sampling).
More concretely, we first extract atomic claims from the text using GPT-3.5. We then use GPT-3.5 3.5 to convert each claim to a query (question) testing knowledge of the particular fact. Careful rephrasing necessary ensure that the rephrased question is unambiguous; for example, the claim "Yo- Yo Ma plays the cello" should be converted to the question "What instrument does Yo-Yo Ma play?" rather than just "What does Yo-Yo Ma play?" the latter question admits answers of the wrong type. If we were to use the second prompt, a model might assign 50% of its probability on "cello" and 50% of its probability on "basketball." However, the model's low confidence caused by the ambiguity of the question, not low confidence in the instrument that Yo-Yo Ma plays. We detail the prompts used for question generation Appendix A.2.
After each claim is converted to minimally ambiguous question, we resample an answer 20 times from the base model (e.g. Llama- 1-7b) that is fine-tuned to estimate the model's uncertainty over the answer. We use few-shot prompt to encourage well-formed answers. We bin these answers by equivalence, using either heuristie string matching of the responses or using GPT-3.5 to assess the answers semantically equivalent, inspired by Kuhn et al. (2023). Our heuristic string match checks whether the words in the answer, excluding stop words, are the same. We compare these choices in Section 4.4. The score for each claim is either the expected maximum confidence of the model's response; we finally average this score over all claims a given model generation.
3.3 FACTUALITY TUNING: PUTTING IT ALL TOGETHER
Given a choice of truthfulness estimator, we can now construct a preference dataset for factuality tuning given language model from set of unlabeled prompts. First, sample n multiple candi- date responses for each prompt from the model with simple temperature sampling with temperature 1.0 (using few-shot prompting for models that have not been fine-tuned). For each response, we then compute the truthfulness score with the chosen estimator (reference- or reference-free frree. Finally, for all (?) pairs of responses each prompt, we simply choose the response with the higher truthfulness score as the preferred response. For a set of m prompts, we ultimately generate m (?) -k preference pairs, where is the number of pairs with equal scores. Finally, fine-tune the model using the DPO pipeline, using all model responses as targets for the SFT stage.
EXPERIMENTS
Our experiments evaluate the extent to which factuality can be learned through preference- based re- inforcement learning, using the fully automated peference-geeneafte pipeline described in Section
5

---

Published as a conference paper at ICLR 2024
Prompts per Entity
Responses
Examile prompt
Dataser
Entities [train, val, test]
per Prompt
Biographies
463 [288, 50, 125]
Write me a paragraph biography of Mary Wollstonecraft.
Medical QA
295 [150, 45, 100)
What are common symptoms of stroke?
Table 1: Dataset statistics and examples. In biographies, entities individuals: in MedicalQA, entities medical conditions. We include 6 questions for each entity in MedicalQA and adjust the number of responses per prompt to keep the total number of pairs in the two datasets roughly similar.
3. We call the model fine-tuned with our reference-based metric FactTune-F'S and the model fine- tuned with our model confidence-based score, which is completely reference-free, FactTune-MC. For all of our experiments, samples for model confidence are taken from Llama-1-7b.
Datasets. We conduct our experiments on two tasks: generating biographies and medical question- answering. For biographies, we generated a dataset consisting of 463 diverse well-known individ- uals (288 train, 50 val, 125 test) with 10 short-paragraph biographies each. For medical question answering, we used a dataset of 295 diverse common medical conditions (150 train, 45 val, 100 test) with 6 questions about each condition and short-parage parrt-par answers per question. The test set just uses 1 question per condition. The prompts were generated with GPT-3.5, and the answers were sampled from Llama-I-7h using a few-shot prompt for each dataset. We found that our procedure consistently resulted in well-formed and informative responses, albeit with possible factual errors. Because FactScore uses retrieval against a given Wikipedia article, we generate data based on indi- viduals and medical conditions that have Wikipedia pages. See Table for the summary stats and examples from our datasets.
Baselines. We compare factuality tuning with inference-ting intervention (Li et al., 2023, ITT) and decoding by contrasting layers (Chuang et al., 2023, DOLA), applied to the SFT model for each task. For ITI, we supervise the training of the linear probes with FactScore labels: we take batches of atomic facts extracted from the training samples and bias the models' activations from the incorrect correct atomic facts to determine the direction of the intervention. In the case of Llama-2, we also compare against 'standard' RLHF with human preference labels (Touvron et al., 2023b).
Evaluation. To evaluate each generated response, we follow the FactScore procedure to extract the number of correct and incorrect facts. Then, to check that the model responses are still relevant and helpful after actuality fine-tuning, we also use GPT-3.5 to determine whether each fact is relevant to the question or not (using the prompt in Appendix A.2). For biographies, we observed that essentially 100% of facts were relevant to the individual, so we skip the relevance computation to save costs. For each dataset, the number of inaccuracies (# Incorrect), and the proportion of correct relevant facts out of the total number of extracted facts (% Correct). Note that the total number of facts may vary between generations. We validate our evaluation metrics in Sec. A.1.
report the number of correct and relevant facts (# Correct),
4.1 FINE-TUNING FOR FACTUALITY ACROSS DOMAINS
In this section, we apply our methodology for learning factuality to Llama- 1-7m and Llama-2-76 multiple domains. We show the results Table Learning from reference-based factuality- scored pairs (FactTune-FSS) consistently improves factual accuracy compared to RLHF models and decoding- based factuality baselines by at least 11% on biographies and 13% on medical question- answering. FactTune-FS reduces the number of factual errors and maintains more than slight decrease, not increase, the amount of correct information generated. Factuality tuning from model-confidence scores (Fac FactTune-FEC) also reduces error rate and improves the factuality of RLHF models on both datasets, without any external reference information.
4.2 FINE-TUNING CHAT MODELS FOR FACTUALITY
Most widely used practical chatbots today are LMs trained with RLHF to follow diverse instructions in a way that is helpful to users. this section, we investigate the ability of our human-free factuality tuning method to improve the factuality of RLHF chat models. Using Llama-2-7b-Chh that fine-tuning an RLHF LM with both factuality and semantic entropy- rewards can further improve its factuality without significantly decreasing the total number of facts, as shown in Table 3.
find

---

Published as a conference paper at ICLR 2024
Biographies
Medical QA
Base Model
Method
# Correct
#
Incorrect
% Correct
Correct
f Incorrect
% Correct
ITI
13.68
5.24
0.730
10.25
7.96
0.538
DOLA
12.44
4.74
0.737
9.22
5.58
0.640
SFT
13.54
6.54
0.696
9.96
6.86
0.600
Llama-1
FactTune-F5 (Ours)
14.51
3.74
0.812
12.60
4.18
0.746
FactTune-MC (Ours)
9.74
2.42
0.819
11.51
5.56
0.668
FactTune-EC (Ours)
10.84
3.28
0.790
11.52
6.56
0.641
ITI
13.30
5.56
0.712
9.40
4.25
0.690
DOLA
13.25
6.50
0.684
9.87
6,06
0.627
Chat
21.41
6,30
0.774
9.61
6.69
0.619
Llama-2
SFT
13.47
6.49
0.687
10.68
6.22
0.627
FactTume-FS (Ours)
19.32
2.76
0.880
13.29
2.97
0.809
FactTune-MCC (Ours)
11.74
3.51
0.783
12.94
5.26
0.706
FactTune-ECC (Ours)
12.68
3.69
0.797
12.80
5.19
0.710
Table 2: Factuality tuning from reference-basee factuality-scored pairs (FactTune-FSS improves factual accu- racy compared RLHF models and decoding based factuality baselines, consistently reducing the number of errors and often increasing the number of correct facts generated. Factuality tuning from model confidence scored pairs (FactTune-MC. FaciTune-ECC) also outperforms RLHF models, providing strong reference-free alternative for improving factuality and reducing error.
Biographies # Incorrect
Medical QA
Base Model
Method
Correct
% Correct
# Correct
# Incorrect
% Correct
21.41
6.30
0.774
9.61
6.69
0.619
DOLA
22.25
5.81
0.793
11.45
6.74
0.624
FacT TS (Ours)
20.02
4.38
0.821
11.94
0.667
Llama-2-Chat
FacTTune-MCC (Ours)
12.61
FactTune-BCC (Ours)
18.77
5.13
0.784
11.51
6.40
0.639
OOD FaetTune-FS (ours)
21.06
5.45
0.796
11.56
6.66
0.635
Table 3: Factuality tuning a dialogue model (Llama-2- 2-Chat) with FactScore, model confidence-basee truthful- estimation, and FactScore-be based preferences from a different dataset (FactTune-FS. FactTune-MC. OOD FactTune-FSS further improves factual accuracy more than haseline method for factuality, DOL.A
In other words, factuality tuning can be composed with RLHF to further improve the factuality of chat models.
While our quantitative metrics demonstrate a clear increase in factual accuracy, we also investigate how factuality fine-tuning impacts other aspects of model performance and generalizes. Using GPT- 4 judge, we find that FactTune-MMC and FactTune-EC can improve both factuality and fluency compared to the SFT model (examples in Appendix Table 8). GPT-4 chooses FactTune- EC more fluent than SFT on 80% of samples, FactTune-MC on 75% of samples, ITI 57% of samples, FactTune-FS on 33% of samples, and DOLA on 16% of samples (n=100).
Lastly, we find that fine-tuning for factuality generalizes across datasets. Fine-tuning Llama-2- 7b-Chat on biographies to evaluate on MedicalQA and vice versa (OOD FactTune-FFS) improves the factuality more than RLHF (Table 3).
4.3 COMPLEMENTARY BENEFITS FACTUALITY TUNING AND DECODING- FACTUALITY INTERVENTIONS
Besides fine-tuning for factuality, multiple existing works aim to improve LLM factuality through inference time interventions either the decoding process or the model parameters themselves. We explore the possibility of applying both of these types of methods together, i.e., using factuality- boosting decoding methods on model fine-tuned with our factuality tuning procedure. In Table 4 we present the results of stacking both approaches. We find that in most cases, DOLA can even further increase the accuracy of factuality fine-tuned models, with one exception for Llama-2 on the biography task. While not comprehensive evaluation of combining methods for improving factuality, this result suggests that different approaches to enhancing factuality may operate through complementary mechanisms.

---

Published as a conference paper at ICLR 2024
Biographies
Medical QA
Base Model
Method
#Correct
#Incorrect
&Correct
#Correct
#Incorrect
SCorrect
FactTune-FS
14.51
3.74
0.812
12.60
4.18
0.746
Llama-1
FactTume-FS+ + DOLA
14.82
3.27
0.831
11.58
3.23
0.785
FaciTune-FS
19.32
2.76
0.880
13.29
2.97
0.809
Llama-2
FaciTune-FS + DOLA
18.82
2.81
0.873
13.13
2.67
0.830
Table 4: DOLA factuality decoding frequently composes with factuality fine-tur fine-tuning, providing an increase in average correctness for the majority of combinations of model and dataset.
Bingraphies
Medical QA
Fact Ext.
Equiv
Metric
#Correct
#incorrect
&Correct
#Correct
#incorrect
&Correett
Max Conf
9.74
2.42
0.819
11.51
5.56
0.668
Atomic
Heuristic
Expected Conf
10.84
3.28
0.790
11.52
6.56
0.641
Max Conf
12.22
4.74
0.742
10.32
6,94
0.605
Entity
Hcuristic
Expected Conf
11.73
5.12
0.718
10.50
6.42
0.623
Table 5: On Llama-1, model confidence- based preference construction with atomic question extraction outper- forms the version with entity extraction.
4.4 IMPACT OF DESIGN DECISIONS OPEN-ENDED MODEL CONFIDENCE SCORING
This section discusses the impacts of different design choices for the steps of our reference-free truthfulness score construction for factuality tuning: how to perform fact extraction and what confi- dence metric to use.
The first step is extract the individual facts from the long-form response and re-sample each fact from the base model to assess the model's confidence in the fact. For the fact-extraction and resam- pling procedure, one approach (Atomic) to convert each extracted atomic fact into corresponding "atomic question' with few-shot prompt query to GPT-3.5, then sample answers to each question from the base LLM. Another approach (Entity) extracts entities from the response via nltk and re-samples the extracted entity in-line. Atomic question extraction has the potential to be more comprehensive and precise, while named entity extraction is less expensive proxy that doesn't use closed models. In Table 5, we observe that atomic question extraction outperforms named entity extraction, although the difference in accuracy is smaller on Medical QA than on Biographies.
After re-sampling the fact, study the choice of confidence metric between taking the model's confidence based on the most common sample (Max Conf) or the confidence of the fact from the original response (Expected Conf). To compute Max Conf for both atomic and entity extraction, bin the samples into equivalence classes of distinct responses using string matching heuristic de- scribed in Section 3.2 and take the proportion of samples in the largest bin. For computing Expected Confidence, we first perform the same answer binning procedure as for Max Confidence, resulting in & bins and confidences PI..... Pk. and take EC Σk_ p. The results in Table 5 show that the performance of Max Conf versus Expected Conf varies but are quite similar.
5 RELATED WORK
Many works have identified reducing factual errors (sometimes called "hallucinations") a key challenge for building more reliable language models (Lewis et al., 2020; Kadavath et al., 2022; Zhang et al., 2023), even for the most powerful language models (Bubeck et al., 2023). Other use of the term "hallucination" refers to summarization or translation system outputs not supported by the reference text (Maynez et al., 2020; Zhang et al., 2020) even if they are factual (Cao et al., 2022). Other work uses "hallucination to describe dision-languae grounded in a visual imput, c.g., a captioning system describing an object that doesn't exist in the image (Rohrbach et al., 2018). In case, focus statements that factually incorrect (or, inconsistent with a set of "authoritative" texts, such as Wikipedia).
models producing outputs not
Several works describe methods for detecting likely factual errors through sensitivity perturba- tions in the prompt (Xu et al., 2023), high diversity responses under resampling (Kadavath et al., 2022; Mündler et al., 2023; Kuhn al., 2023; Manakul et al., 2023), inconsistency with exter-
8

---

Published as a conference paper at ICLR 2024
nal knowledge sources (Min et al., 2023; Chern et al., 2023), properties of internal activations (Azaria & Mitchell, 2023). Others go beyond detecting errors, correcting them after they have been generated (Peng et al., 2023; Gao et al., 2023; Dhuliawala et al., 2023). These approaches typi- cally rely on retrieving relevant data from a trusted knowledge base and use another LLM to verify consistency; however, retrieval-based methods face key challenges, namely reliable resolution of conflicts between parametric and retrieved knowledge (Longpre et al., 2022; Chen et al., 2022) as well as maintaining improvements in factuality as model size increases (Mallen et al., 2023). Fur- ther, retrieval- -bsed methods add significant system complexity; the most common open-source con- sumer language models thus use purely parametric models (Touvron al., 2023a). The FactScore variant of our approach uses retrieval only during training, avoiding inference time complexity. In principle, any existing criterion could be used to generate preferences (see; ; we aim to show that even choosing relatively simple criteria leads to substantial improvements in factuality.
Most similar to ours, some approaches attempt to prevent the generation of factual errors the first place, using prompting strategies (Si et al., 2023) or perturbing the internal representations of the model (Chuang et al., 2023; Li et al., 2023). Unlike using a fixed heuristic for identifying an inter- nal 'factuality' dimension, we optimize directly for the end goal of generating factual statements, which we find shows a greater improvement in factuality. Finally, while most past work has focused on short-form NLG tasks like short-form question-answerrinn ways to measure model confidence over factual information in long-form, unstructured text and es- timate truthfulness in a reference-free manner (i.c., don't ton't require any external knowledge base or annotations).
(Kadavath al., 2022), we explore
6 CONCLUSION
In this paper, show practical, effective strategy improve a language model's ability gen- erate factual content, specifically focusing on long-form generations. We develop and study two different approaches to estimating the truthfulness of long-form text and optimize for these criteria using preference-based learning. In addition to existing reference-based truthfulness estimators that leverage external knowledge to establish the truth of particular statement, we introduce novel reference-free procedure for estimating truthfulness that uses the language model's own uncertainty as an indication of factuality. Our experiments show that fine-tuning a language model with either criterion reliably reduces the number of incorrect facts (i.c. hallucinations) that the model generates. Reference-fire approaches like the one we introduced provide a scalable self-supervision strategy to improve factuality, eliminating the need for reference corpus of 'gold' texts.
The experimental results suggest a number of avenues for future work. First, because of the limited research and thus the limited benchmarks on the factuality of long-form language model generations, we proposed two new tasks to benchmark our approach. These tasks are representative of but do not fully cover the range of scenarios where we would hope improve factuality. Furthermore, our experiments provide evidence for improving the factuality of dialogue models that are already fine- tuned with RLHF, but still leave open the question of how best to combine typical RLHF rewards and approaches with factuality rankings, Similarly, exploring additional ways to combine factuality tuning with existing methods for improving factuality, such as in our factuality tuning DOLA experiment, may be a fruitful direction for future research. Further, future work might explore alternative approaches to constructing factuality preferences, such as using self-correction (Pan et al., 2023). Finally, we explore only 7B models in this work. Scaling up our factuality tuning recipe to larger models (and larger preference datasets) may reduce hallucinations even further.
ACKNOWLEDGEMENTS
EM gratefully acknowledges funding from Knight-Hennessy graduate fellowship and Stanford Accelerator for Generative Al and Education grant. CF and CDM are CIFAR Fellows.
Reproducibility Statement. We explain the steps of our fine-tuning method in Section 3. In Sec- tion 4.1, we provide details on the dataset (dataset statistics, how it was generated, and examples), as well as how the evaluation is completed and how we implemented the baselines. In the exper- iment subsections and captions, we provide additional implementation or reporting details. In the appendix, we provide the exact GPT-3.5 prompts used for the extraction steps of our reference-fre freee scoring method. A codebase with instructions for factuality scoring, training, and evaluation can be found here: https: : / h/gitttuu.t thub. www./otttttt
1m ua 1 tyy bty_tuning

---

Published as a conference paper at ICLR 2024
REFERENCES
Ayush Agrawal, Mirac Suzgun, Lester Mackey, and Adam Tauman Kalai. Do language models know when they'r re hallucinating references?, 2023. arXiv preprint urxiv:2305.18248.
Amos Azaria and Tom Mitchell. The internal state of an LLM knows when its lying, 2023. arXiv preprint arxiv:2304. 13734. 9
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hlafield-Dodds, Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022. aiv preprint saiiv:204.008822. 3
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4, 2023, arXiv preprint arxiv:2303.12711 8
Meng Cao, Yue Dong, and Jackie Cheung. Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3340-3354, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. (0.11855//1/2222.achhhe 236. URL https:://// aac antolootog org / 2022 acl- 236.
Hung-Ting Chen, Michael Zhang, and Eunsol Choi. Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2292-2307, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 10. http33//112222mm///pg emrlp ma in. 146.9 9
146. URL https: /aaaaaaaanntloooooeeeeeaaaeeelet
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri son Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings. Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. ArXiv, 0abs/2077.033.4 2021. URL https:/ //api. semant ccccholcllooooocoooo ar.. org/ Oorg/Corpuseeeeoeeooooooooeeeeee Corpus ID: 235355577.
I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, and Pengfei Liu. FacTool: Factuality detection in generative ai a tool augmented framework for multi-task and multi-domain scenarios, 2023. arXiv preprint arxiv:2307.1352. 4,
Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., neurips........s..e.e. dc cc/paper.pe...e/eee/e.eeeeee.. http:/pprrfeeeppeeeeeeeeeeeeeeee fi file/ ht11///222ccaa5550eee11se222edee
2017. URL
.ddf.
10

---

Published as a conference paper at ICLR 2024
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. DoLa: Decoding by contrasting layers improves factuality in large language models, 2023, arXiv preprint arxiv:2309.03883. 6,
Hyung Won Chung. Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pel- lat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra. Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling models, 2022.
language
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of--vriffcattin reduces hallucination large language models, 2023.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, 2022.
3, 14
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. Rarr: Researching and revising what language models say, using language models, 2023.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Ellagg, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bow- man, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse. Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Ka- plan. Language models (mostly) know what they know, 2022. URL http: /atpii//aavvv.///aeeeaeeeeeeaiia http/aaawvv...//...............l org/ abs/ 05221. Arxiv arxiv:2207.00 055221.. 1, 1, 8,9
Lorenz Kuhn, Yarin Gal, and Sebastian Farguhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation, 2023. 5,8
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmennted generation for knowledge-intensive NLP tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neu- ral Informaionn Processing Systems, volume 33, pp. 4459-9474, Curran Associates, Inc., 2020. URL attps://proceeednggg wew.einngnnneuuuree neurips cc/paper happppr_flleee____aaaaaaaeeeeeee files/paper/20000 file/
pdf. 8
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time
intervention: Bliciting truthful answers from a language model, 2023. 6, 9
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. baset knowledge conflicts question answering. 2022.
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. QUARK: Controllable text generation with reinforeed unlearning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neu- Information Processing Systems, 2022. URL httph: http//ppeeeeviiieeee net forum? ?id= 5Ha dds3ux50. 3
http:s::/ppeerreiiee
Bill MacCartney and Christopher D. Manning. Modeling semantic containment and exclusion in natural language inference. In Proceedings of the 22nd International Conference on Compu- tational Linguistics (Coling 2008), pp. 521-528, Manchester, UK, August 2008. Coling 2008 Organizing Committee. URL http: www. awweeb...bb... org/ aataaaahhooooooooo CC08 10000-106666 1066.0 4
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Ha- jishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for
11

---

Published as a conference paper at ICLR 2024
Computational Linguisties (Volume 1: Long Papers), pp. 9802-9822, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10..16653/1/22233lllmoom https:// hatpss//aalaantholooo wwtp://..r.//02....... www.o22..acooonnoo. 546. 9
URL
Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models, 2023,
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1906-1919, Online, July 2020. Association for Computational Linguistics. doi:
1220330/11/00lllllllllll 173. URL https:/ /aaaaaaaannooooooooooooooooooooa .rrgg
2020 a020.acl-mal1. 173. 8
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer. and Hannanch Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision long form text generation, 2023. 2, 4, 9, 14
Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation, 2023. 8
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. 1, 3
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self- correction strategies, 2023.
Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. Check your facts and try again: Improving large language models with external knowledge and automated feedback, 2023. 9
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model, 2023. 3,
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Chris- tian Bauckhage. Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy op- timization. In International Conference on Learning Representations (ICLR 2023), 2022. URL https: /ittss:/arzen /ttp:///aaivvvv//v org/ abos///////// 2210 01241. 3
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hal- lucination image captioning. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun iichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4035-4045, Brussels, Belgium, Detober-November 2018. Association for Com- putational Linguistics. doi: 10. 10.18653/v1/D1--11 14377 URL https: /aa/aalannttttt ogy. / D18-1437
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms, 2017. 3
Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. Prompting gpt-3 be reliable, 2023. 9
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedhack. Neural Information Processing Systems, 18, 2020.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback, 2023. 1, 1.4.
12

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a. 1, 4,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b. 1,6
Weijia Xu, Sweta Agrawal, Eleftheria Briakou, Marianna J. Martindale, and Marine Carpuat. Un- derstanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection. Transactions of the Association for Computational Linguistics, 11:546-564, 06 2023. ISSN 2307- 387X. doi: 10.1162/tacl
URL https::/ org/i..rg/11 11111111111611 1162/taccllccce a_00563.
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations snowball. arXiv preprint arXiv:2305. 13534, 2023. 8
Yuhao Zhang, Derek Merck, Emily Tsai, Christopher D Manning, and Curtis Langlotz. Optimizing the factual correctness of summary: A study of summarizing radiology reports. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties (ACL), 2020. URL https:/ //ae://aaaa /aaxivv... org/ aagg/ddf////eo / 91 02541.251.. pdf. 8
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang. Secrets of RLHF large language models part I: PPO, 2023. 3
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020. 1
13

---

Published as a conference paper at ICLR 2024
Factuality Metric Agreement
Fectureess
MazDA Facthoreet
FactScore-counted
8-
6-
Dataset
Evaluation
SFT
FaciTune-FS
Biographies
Human
0.582
0.846
Biographies
FactScore
0.669
0.921
2
McdQA
Human 0.662
0,838
0.4 0.5 0.6 0.7
0.8 0.9 1.0
MedQA
FactScore
0,806
GPT4-counted Errors (Scaled)
Table To validate that our models from extreme reward overoptimization, we con- duct a human evaluation of the Llama-1-7b SFT and FactTune-F'S models and find that increase in FactScore also corresponds human-annotated accuracy. This study com- puted on a subset 25 generations per model.
suffer
Figure 4: Average FactScore error counts and GPT-4 етооr counts are highly correlated, suggesting that the resulting models do not suffer from extreme reward overoptimization (Gao et al., 2022). We plot average v.s. average GPT-4 count, scaling each dataset by max GPT-4 error count that dataset.
large increase
FactScore
A
APPENDIX
A.I
VALIDATING METRICS FOR FACTUALITY
Our experiments primarily use counts of correct and incorrect facts computed by FactScore as the main evaluation metrics, FactScore is automated and has been shown to exhibit good agreement with human fact-checkers (Min et al., 2023). Nonetheless, we aim to verify that our results are not specific overfit the FactScore criterion. In this section, provide evaluation with (1) human evaluators hired through Prolific.co and (2) GPT-4.
To acquire human fact-checking results, provide each human evaluator with prompt, gener- ated response, and the title of the Wikipedia article they should use for fact-checking the response. We ask the human study participants to count the total number facts and the number of incorrect facts in the response, and we divide these to obtain the human-rated accuracy. We provide the results Table 6, where on average humans rated our FactTune-FSS model for both datasets significantly higher than the SFT model.
Further, we ask GPT-4 to evaluate the factuality of given response by counting the number of factual errors. We observe that the GPT-4 model ratings and FactScore model ratings are highly correlated, and GPT-4 provides another evaluation metric that demonstrates that FactTune-FSS sig- nificantly reduces average eтror compared to the SFT models on both datasets (see Figure 4). Taken together, these results suggest that the improvements in factuality are not the result of exploitation of our evaluation protocol.
A.2 PROMPTS
Table contains the prompts used with GPT-3.5 to convert statements into questions for model confidence-based truthfulness estimation.
A.3 SAMPLE MODEL GENERATIONS
See Table 8 for samples generated by several different models. After factuality tuning, the model does produce somewhat terser responses.
Human evaluators were compensated estimated hourly rate of $16-18
14

---

Published as a conference paper at ICLR 2024
Biography Atomic Fact
I will provide statement containing atomic fact related to Hillary Clinton people around her. Please rephrase the following statement into specific question testing knowledge of the key fact the statement. For example:
to Question
Statement: Hillary Clinton was born in 1947.
Question: In what year was Hillary Clinton born?
Statement: Hillary attended the Wellesley College.
Question: What college did Hillary Clinton attend?
Statement: She married Bill Clinton.
Question: Who did Hillary Clinton marry?
I will provide statement containing one atomic fact related to LeBron James or people around him. Please rephrase the following statement into specific question that testing knowledge of the key fact in the statement. For example:
Statement: LeBron James is professional basketball player.
Question: What is LeBron James' profession?
Statement: He is one of the best the NBA.
Question: Where does LeBron James rank among NBA players?
Statement: James was born in Akron.
Question: In what city was LeBron James born?
I will provide a statement containing one atomic fact related to [NAME] people around [HIM/HER]. Please rephrase the following statement into specific question lesting knowledge of key fact in the stalement. For example:
Statement: [STATEMENT]
Question:
MedicalQA Atomic Fact to Question
I will provide a statement containing atomic fact about the medical condition menopause. Please rephrase the following statement into specific question testing knowledge of the key fact in the statement. For example:
Statement: Menopause is a time in a woman's 's life.
Question: Menopause is a time in whose life?
Statement: Menopause is the time when woman longer has menstrual periods.
Question: Menopause is the time when woman no longer has what?
Statement: There is a decline in the ovarian hormone estrogen.
Question: During menopause there is a decline in what? I will provide statement containing one atomic fact about the medical condition breast cancer. Please rephrase the following statement into specific question testing knowledge of the key fact in the statement. For example:
Statement: The signs and symptoms include a lump or thickening in or near the breast or un- derarm.
Question: Do the signs and symptoms of hreast cancer include a lump thickening in near the breast or underarm?
Statement: The signs and symptoms include a change the size or shape of the breast. Question: Do the signs and symptoms of breast cancer include a change in the size or shape of the breast?
I will provide statement containing one atomic fact about the medical condition varicose veins. Please rephrase the following statement into specific question testing knowledge of the key fact in the statement. For example:
Statement: Varicose veins occur when the veins under the skin become enlarged. Question: Varicose veins
when what happens to the veins under the skin?
Statement: Veins in the legs lose their ability to efficiently circulate blood back to the heart. Question: Varicose veins the legs lose their ability efficiently do what?
I will provide a statement containing one atomic fact about the medical condition [CONDI-
TIONJ. Please rephrase the following statement into a specific question testing knowledge of the key fact in the statement. For example: Statement: [STATEMENT]
Question:
Table 7: GPT-3.5 prompts used to convert atomic facts into questions.
15

---

Published as a conference paper at ICLR 2024
SFT
Greta Gerwig is an American actress and filmmaker. She was horn in and raised in Sacramento, California. Gerwig became interested in the film industry in 2002, when she attended the San Francisco Institute. She got her start in 2006, when she starred in the film 'The Off Hours'. Her other films include Frances Ha, Mistress America, and 20th Century Women. 2018, Gerwig directed Lady Bird, and received wide acclaim, a Golden Globe and Tony nomination for directing, and Oscar and BAFTA nominations for the film.
FaetTune-FS
Greta Gerwig is an American actress and filmmaker. Born in Sacramento, California in 1983, Gerwig started acting a young age. She is best known for starring in films such 'Frances Ha' and 'Mistress America.' Gerwig formance 'Lady Bird.' Gerwig has written and directed number of acclaimed independent films, such as FFancces Ha" and 'Mistress America.'
nominated for Oscar for Best Actress for her per-
FactTune-MC
Greta Gerwig an American actress and filmmaker. California, and raised in Stanford, California and New York City, Gerwig has appeared in such movies as Greenberg, Frances Ha, and Jackie. She also portrayed the title role in the 2013 film of Jackie Kennedy. Her directorial dehut is the 2012 movie, Lady Bird, which was nominated for the Academy Award for Best Picture.
was born 1983 Sacramento,
Table 8: Samples for writing a biography of Greta Gerwig under the SFT, FactTune-FS. FaciTune-MC Llama- 1-7b models.
16


Reviews:
[{"review_id": "oiDasYwPxq", "reviewer": "Reviewer_2run", "summary": "In this paper, the authors construct a direct preference optimization (DPO) dataset for improving factuality using reference-based and reference-free truthfulness annotation techniques. Through the proposed method, they improve accuracy in two tasks (Biographies and Medical QA) without human factuality labels. The authors demonstrate that the proposed method (DPO-FS and DPO-MC) can be applied to Llama-2 and Llama2-Chat, and combined with a factuality-decoding approach (e.g., DOLA).", "questions": "In Biographies and Medical QA tasks, are DPO-FS, DPO-MC, and SFT fine-tuned on the training set of each dataset?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Motivation is intuitive and easy to understand\n- The proposed method improves the truthfulness of LLM without human factuality labels\n- The proposed method can be augmented with existing orthogonal approaches for factuality", "weaknesses": "- Because the framework is simple and the method of scoring truthfulness and fine-tuning technique uses existing approaches, the proposed method appears to have limited contributions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D4AH2d95MH", "reviewer": "Reviewer_Pzp4", "summary": "Authors propose to finetune Llama models for factuality in long-form generation tasks using DPO on automatically constructed preference pairs. Authors explore 2 methods for generating preference ratings: 1) Reference-based: Extracts atomic facts using GPT-3.5 and then use Lama-1-7B-based NLI model to determine correctness of each atomic fact with respect to the reference. Percentage of correct atomic facts is used to compare the factual correctness of samples. 2) Reference-free: Extracts facts using GPT-3.5, then use GPT-3.5 to convert a fact to a question (uses few-shot prompting). Then, through sampling answers multiple times from the model, they estimate the model's uncertainty for the actual answer. The model's uncertainty is used to compare the factual correctness of samples.\n\nThey evaluate their approach on two tasks: biography generation and open-ended medical QA. To accommodate for the reference-based metrics, they generate data based on individuals (for biographies) and medical conditions that have Wikipedia pages.\n\nResults show superior factual accuracy for DPO-based models on both tasks.", "questions": "Check questions in the Weakness section.\n\n- Between reference-free and reference-based metrics, there is a significant difference in the total number (correct + incorrect) of generated facts (almost 30% fewer) on biographies. What's the source of this bias, any possible explanations?\n\n- Could you provide statistics on the number of tokens in wining vs losing samples in all cases (dataset/model/metric)?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- New results to show the benefit of using automated feedback for improving LLMs, targeting factuality for long-form open-ended generation.\n\n- Paper is well-written, experimental settings are well-defined, human evaluation is performed.\n\n- Paper also shows that DPO-finetuning is complementary to decoding-time factuality improvement method (DOLA), (DPO + DOLA outperforms DPO)", "weaknesses": "- Idea itself is not novel, RLAIF has been consistently shown to be useful (here, authors used DPO instead of PPO). Though the application is new.\n\n- Including more fine-tuning based baselines can help understand the role of automated metrics. E.g., directly using prompts to compare factuality of two outputs w.r.t. the wikipedia article instead of extracting atomic facts. \n\n- Both DPO variants reduce number of correct facts on biography generation. This does not seem very surprising, given the optimized metric is the percentage of correct atomic facts. For example, a sample with 10 correct and 5 incorrect is preferred over 11 correct and 6 incorrect. Can this bias be removed from the fine-tuned model, maybe by changing the metric or comparing samples of similar lengths? Or is it the bias of evaluation metric?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EMBilxuGJs", "reviewer": "Reviewer_7K7x", "summary": "This paper proposes fine-tuning language models to improve their factuality. Specifically, one reference-based and one reference-free method are explored to estimate the truthfulness of different model responses, the scoring/preference of which are then used to fine-tune the LMs with direct preference optimization.", "questions": "- I'm a little confused why choosing the largest bin to measure truthfulness in the reference-free setting. Does that mean the atomic claim doesn't really matter (\"Yo-Yo Ma was born in 1951\" and \"Yo-Yo Ma was born in 1955\" would both be converted to \"What year was Yo-Yo Ma born\")? So as long as two responses make a claim on the same fact, regardless if it's correct or wrong, they will receive the same truthfulness score? If the hypothesis is \"a language model’s confidence in a generated answer is highly correlated with the probability that the answer is correct\", why not use the distribution to cross-check like in the reference-based setting?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper writing is of high quality and easy to follow\n\n- The proposed method, regardless of reference-based or reference-free, shows improved factuality than the SFT baseline and the highest correct% among the compared methods.\n\n- The paper provides analysis and ablations of different variants such as fine-tuning the pretrained/chat models and combining with inference-time decoding method.", "weaknesses": "- [major] I have some concerns about the evaluation\n  - The test sets (50 and 59 examples in each domain, respectively) look very limited, making it bit hard to understand the actual improvement of model factuality.  How reliable are the results? Is 75% -> 81% a lot? I can't really answer these questions after reading the paper.\n  - I noticed that the total number of claims are often different for different methods. Could generation style (e.g., length) contribute to the seemingly better/worse results? I wonder if the authors have considered such factors.\n  - There is also no evidence indicting the improved factuality doesn't come at the expense of performance in other aspects. I understand the authors may not have enough labor/compute for a more comprehensive eval like GPT or Llama but LLMs, in my experience, can behave in mysterious ways when you over index on one specific objective.\n\n- [minor] The method is somewhat straightforward, which is not necessarily a bad thing if the evaluation can show meaningful improvements (that are worth fine-tuning specifically for factuality) than methods that modify decoding only.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HxnXA8MZrV", "reviewer": "Reviewer_mN58", "summary": "This paper studies approaches to improve the factuality of language models (LMs) by fine-tuning the LMs using reinforcement learning (the specific algorithm is DPO), and the two main types of the reward function are: 1) reference-based FactScore (referred to as “DPO-FS”) and 2) reference-free uncertainty measure based on self samples (referred to as “DPO-MC”). \n\nThe authors conduct the experiments on two tasks: 1) free-form generation of biographies and 2) medical question answering. The datasets were crafted specifically for their experiments, hence resulting in the small size (e.g., biography train/test = 296/59 and medical QA train/test = 150/50 instances). The base LM is Llama1 and Llama2. The main experimental results show that both DPO-FS and DPO-MC generate responses with a higher “correct” percentage than baselines (SFT and inference-time methods such as ITI and DOLA). Also, DPO-FS and DPO-MC achieve a higher correct percentage than Llama-2-chat. Lastly, the authors perform a human evaluation to validate the findings previously evaluated using GPT3.5.", "questions": "My questions are related to the points in the weaknesses section. I'm looking forward to seeing your responses to the weaknesses above, especially point number 3.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper shows that DPO can be applied to improve the factuality of LMs as shown by DPO-FS and DPO-MC achieving better factuality, and to the best of my knowledge, the factuality-based reward has not been investigated yet. The paper also investigates both reference-free and reference-based, and shows the effectiveness of both methods.", "weaknesses": "1. Although existing work may have not used a factuality-based reward, the results in this paper are mostly the expected observations (e.g., applying RL-based training improves target rewards). For example, (Lu et al., 2022) applied RL (PPO) with a reward based on an external metric to improve toxicity, repetition, etc.\n\n2. The main findings (Tables 2, 3, 4, 5) are all based on GPT3.5 evaluation, and coupled with the fact that the test sets are small (e.g., 59 instances for biographies & 50 instances for medical QA), I’m not certain how reliable the results are. Also, there is not much information in Section 5.5 about human evaluation, e.g., inter-annotator agreement, or how many annotators were employed.\n\n3. How does the DPO fine-tuned model perform on out-of-domain tasks? For example, when fine-tuning to improve factuality on biographies, does it also improve factuality on medical QA? And does its general performance change?\nThere is also a recent survey paper (Pan et al., 2023) about aligning LLMs for different aspects (including hallucination/factuality), and I think it would be useful for authors to incorporate additional relevant papers (i.e., those that apply RL to improve LMs) \n\n4. Weak base LM: This work uses Llama-7B as the base model, and at this size, the model may not yet be highly capable of long-form generation / medical QA. Previous works such as (Manakul et al., 2023) and (Mundler et al., 2023) investigated LLM hallucination with much larger LLMs (e.g., GPT3.5/4). It would be interesting to see, for example, when using larger models (either open-source such as larger Llama / Falcon-180B or private ones such as GPT-4), if the model still makes as many factual errors (because if they don’t – due to the emergent ability when scaling up – fine-tuning may not be necessary or have little impact).\n\nThere is a recent survey paper (Pan et al., 2023) about aligning LLMs for different aspects (including hallucination/factuality), and I think it would be useful for authors to incorporate additional relevant papers (i.e., those that apply RL to improve LMs) \n\n*References*\n- (Lu et al., 2022) QUARK: Controllable Text Generation with Reinforced Unlearning\n- (Manakul et al., 2023) SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models\n- (Mundler et al., 2023) Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation\n- (Pan et al., 2023) Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)