Factuality Enhanced Language Models for Open-Ended Text Generation
Authors: Nayeon Lee
Wei Ping
Peng Xu
Mostofa Patwary
Pascale Fung
Mohammad Shoeybi
Bryan Catanzaro
Venue: NeurIPS 2022 Accept (2022)

Abstract: Pretrained language models (LMs) are susceptible to generate text with nonfactual information.  In this work, we measure and improve the factual accuracy of large-scale LMs for open-ended text generation.  We design the FactualityPrompts test set and metrics to measure the factuality of LM generations.  Based on that, we study the factual accuracy of LMs with parameter sizes ranging from 126M to 530B.   Interestingly, we find that larger LMs are more factual than smaller ones, although a previous study suggests that larger LMs can be less truthful in terms of misconceptions.  In addition, popular sampling algorithms (e.g., top-p) in open-ended text generation can harm the factuality due to the ``uniform randomness'' introduced at every sampling step.  We propose the factual-nucleus sampling algorithm that dynamically adapts the randomness to improve the factuality of generation while maintaining quality.  Furthermore, we analyze the inefficiencies of the standard training method in learning correct associations between entities from factual text corpus (e.g., Wikipedia).   We propose a factuality-enhanced training method that uses TopicPrefix for better awareness of facts and sentence completion as the training objective, which can vastly reduce the factual errors.

Factuality Enhanced Language Models for Open-Ended Text Generation
Nayeon Lee"tl, Wei Pingt Peng Xu², Mostofa Patwary?, Pascale Fung', Mohammad Shoeybi", and Bryan Catanzaro?
Hong Kong University of Science and Technology NVIDIA
Abstract
Pretrained language models (LMs) are susceptible to generate text with nonfac- tual information. In this work, we measure and improve the factual accuracy of large-scale LMs for open-ended text generation. We design the FACTUALI- TYPRMMPSS test set and metrics to measure the factuality of LM generations. Based on that, we study the factual accuracy of LMs with parameter sizes ranging from 126M to 530B. Interestingly, we find that larger LMs are more factual than smaller ones, although previous study suggests that larger LMs can be less truthful in terms of misconceptions. In addition, popular sampling algorithms (e.g., top-p) in open-ended text generation harm the factuality due the "uniform random- ness" introduced at every sampling step. We propose the factual-nucleus sampling algorithm that dynamically adapts the randomness improve the factuality of generation while maintaining quality. Furthermore, we analyze the inefficiencies of the standard training method in learning correct associations between entities from factual text corpus (e.g., Wikipedia). We propose a factuality- training method that uses TOPICPREFIX for better awareness of facts and sentence completion as the training objective, which can vastly reduce the factual errors.
Introduction
Large-scale pre-trained language models (LMs) have demonstrated impressive natural language generation results [1-4]. However, the generative LMs (e.g., GPT-3) are solely trained to model the statistical correlations between subword tokens [5], and have limited capability generate factually accurate text as illustrated in Table 1. As a result, there are increasing concerns about the nonfactual generations from large-scale pre-trained LMs [e.g., 6). which needs to be adequately addressed for their safe deployment in real-world applications, e.g., content creation [9] and dialogue [10].
In previous studies, different metrics and methods have been proposed to measure and improve the factual accuracy of language generation within different tasks [11], including text summarization [e.g.. 12-15], question answering [e.g.. 16-18], and tabletto-text generation [e.g.. 19, 20]. However, these works focus on the faithfulness (or factuality) tasks (i.e,, factual consistency between source and target text). Little exploration has been made to address the factual errors in pretrained LMs for general-purpose purpose open-ended text generation, where the goal is to generate coherent continuation from the given context (e.g., the use cases from GPT-2).
the fine-tuned LMs for particular downstream
One of the popular methods for enhancing generation factuality is to incorporate external knowl- edge sources [21-23] Structured knowledge bases and graphs have been utilized for grounded
Work done during an internship at NVIDIA.
t Cooresponddecce Nayeon Lee <nayeon.les lce hconn Wei Ping ewping@nvidia.comm
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

text generation [e.g., 24, 25], where the LMs are trained to select and copy relevant facts from external knowledge sources. In contrast to the sizeable online text with factual information, the structured knowledge graphs only encode limited amount of knowledge as they require expensive human annotations for high-quality construction. method that can directly leverage plain text knowledge (e.g., Wikipedia, encyclopedia books, peer-reviewed publications) would be desirable for factuality enhancement as it can remove the human annotation bottleneck and easily scale up the amount of injected knowledge. Augmenting LM with an information retrieval (IR) system is one possible solution to leverage textual facts, however, at the cost of additional complexity and resource overhead to the model [10. 26, 22, 27, 28). Therefore, we explore an IR-free method that enhances the innate factuality of LMs by continued training on a factually rich plain-text corpus.
In this work, we focus on measuring and improving the factuality of large-scale pre-trained language models (LMs) for open-ended text generation. Specifically, we make the following contributions:
1. We build the benchmark and metrics measure the factual accuracy of pre-trained LM for open-ended text generation. We demonstrate a good correlation between the proposed automatic metries and human assessment of factuality. Based on that, we systematically study the factual accuracy of LMs with parameter sizes ranging from 126M to 530B and find that large LMs have higher factual accuracy than smaller ones (e.g., named-entity factual error is reduced from 63.69% to 33.3%)
2. We study the decoding algorithms of LM in terms of factual accuracy. We unveil that the popular nucleus sampling algorithm [29] for open-ended text generation can easily mix up different named entities or randomly fabricate information due to the "uniform randomness" introduced at every decoding step. We propose fuctual-nucleut sampling algorithm that promotes generation factuality while maintaining the quality and diversity.
3. We explore training methods that can effectively leverage text corpus with rich facts (e.g.. Wikipedia). We find that directly continuing the training of LM on factual text data [30] does not guarantee the improvement of factual accuracy. We propose factuality- eehhantttyennneedd training to address the underlying inefficiencies of this baseline. Our method consists of i) an addition of TRPICPREFIX that improves the awareness of facts during training, and ii) sentence completion task as the new objective for continued LM training [e.g.. 30).
4. We demonstrate that the factual accuracy of large-scale LMs (up to 530B) can be significantly enhanced (i.e., named-entie factual error is reduced from 33.3% to 14.5%) after applying the proposed factuality -acaliit--ehhnccedd training w -nccless sampling algorithm.
We organize the rest of the paper as follows. We discuss related work in $ 2 and present our benchmark setup with evaluation protocol in $ 3. We study the factual accuracy of LMs with respect to model size, prompt type, and choice of decoding algorithm in $ 4. After that, we present sampling algorithm in $ 5, and factualiyy--e enhhance training in $ 6. We conclude the paper in 7.
2 Related Work
Factuality Model Size Lin al. [31] propose the TruthfulQA benchmark to measure the falsehood generations from different sized LMs. The result suggests that bigger LMs pre-trained on web text are generally less truthful than smaller ones in terms of false belief or misconception. At first glance, this is contradictory to our observation, however, our work focuses on different knowledge to TruthfulQA work. The TruthfulQA benchmark focuses on conceptual knowledge, while our benchmark focuses on factual knowledge [32] 4 Large LMs can be good at recalling factual knowledge given substantial pre-training corpus, suggested by previous studies on LM's parameteric knowledge [33], but there still remains room for improvement for reasoning conceptual knowledge [34, 35].
Parametric Factual Knowledge A group of work addresses the factual errors in the parametric knowledge of LMs that is acquired from training corpus [36-38]. The correctness of the parametric knowledge is commonly tested in cloze-style question answering format [33] (e.g., Person X is born in ). Efforts are made fine-tune the pre-trained LM to "inject" more knowledge and improve its ability to answer factual questions without consulting external knowledge source [17].
3 'The implementation can be found https: : //tptttttttttttte/eeeeeeeee.h wtw.tttacccmm/aaeoo//eeee/////ee cwt.///aayooo///////taaaaaaaoooa 'According to Krathwohl [32], knowledge can be categorized into four types: i) factual knowledge, ii) conceptual knowledge, iii) procedural knowledge, and iv) metacognitive knowledge.

---

Table I: Example of continuations from the 530B LM with greedy decoding and nucleus sampling p = 0.9, along with the continuation from factuality-cnhnnce 530B LM with -ncceeus sampling. Red represents nonfactual, green represents factual, and strikethrough represents repetition. The LMs will stop generation when they generate <lendoftextl>, reach the maximum length. See Appendix A for more examples.
[Factual Prompt] One of lung cancer's 's symptoms is shortness of breath.
[Greedy] The American Cancer Society estimates that more than 228,000 people will be diagnosed with lung cancer in the United States this year.Innn The Aiee-eeerinnn Cancer Seciety estimaten that that-mere mere- then 228,000 people will be diagnesed with leng--g-nnnee saneer in the United States this year American AanericanCannes Seciety estimates that more than 228,000 people will be diagnesed with lung cancer the United
[p=0.9] That would make an oxygen mask one of the more popular treatments for this devastating disease. It helps case breathing and give patients back their strength. Learn more here.<lendoftextl>
[Ours] Other symptoms include coughing (including coughing up blood), chest pain, hoarseness, weight loss, and loss of appctite. The symptoms of lung cancer may resemble those a bad cold or pneumonia. A chest x ray is necessary to confirm the diagnosis. The diagnosis is usually confirmed by biopsy, in which a small sample of the tumor is removed and examined under microscope...
INon-Factur Prompt] Cancer can be prevented by drinking too much alcohol.
[Greedy] \nCancer can he prevented by eating too much red meat. IninCancer can be prevented by eating tee-much sugar CangarrrwwnCCeeeeeerer be prevented by eating much Cateeeeeeeeer ean be prevented by eating tee-much processed food,
[p=0.9] More than 90% of hepatocellular carcinoma is associated with alcohol use. InhnAbout 9 percent of cancers can be prevented if we don't 't cat more than 1.5 ounces of processed meat cach AnnnnDt is believed to play a role in 7% of cancers and...
[Ours] Cancer often treated with some combination of radiation therapy, surgery, chemotherapy and targeted therapy. Pain and symptom management are an important part of care. Palliative care is particularly important in people with advanced discase. The chance of survival depends on the type of cancer and extent of disease the...
Moreover, some works attempt to edit and fix the factual errors [39-41]. However, it is unclear if the improvement of fine-tuned LM for QA-style task can help to mitigate factual errors in open-ended text generation task.
Hallucination in downstream NLG tasks There are active efforts to reduce the unfaithfulness or factual errors of task-specific LMs fine-tuned for various downstream natural language gener- ation (NLG) tasks such as summarization [42-48], data-to-texx [49, 50, 20, 51-53] and dialogue system [54-58], In contrast to these works, we focus on general purpose LM for open-ended text generation task.
Human-in-the-loop Human feedback or demonstrations are valuable to improve the factual accuracy of LMs. For example, InstructGPT [59] fine-tune the LMs with collected human feedback for a truthful generation. WebGPT [7] is trained to cite its sources when it generates output, thus allowing humans to evaluate factual accuracy by checking whether a claim supported by a reliable source. In this work, we focus on buman-free solution to mitigate nonfactual generations, as it is less expensive and easy to scale.
3 FACTUALITY and Evaluation Metrics
Our goal is to automatically measure and evaluate the factuality of large-scale pre-trained language models (LMs) for open-ended text generation. Factuality refers being coherent to provided ground- truth knowledge sources in NLP [11]. The biggest challenge of evaluating factuality for open-ended text generation is associated with locating the ground-trui knowledge from the myriad of world knowledge. Evaluating open- pndedended text generation can be challenging due to the lack of ground -tuth references for generation [29, 60]. In this study, the scope of our ground- -ruth knowledge source is set to Wikipedia 5 because this helps simplify the evaluation setup.
NNte that Wikipedia is one of the most commonly accessible, large-scale, good quality, unstructured knowledge sources. Our proposed methods can easily generalize to other knowledge sources in plain text (e.g.. arXiv papers, medical reports, reliable newspapers).
3

---

leading
Thomppon.
[PROMPT] 2000 released Soul hood.
LM
Wilkeasn, Geoign Segai (Alman fimm largoy pratsed acting performances, and nominated for eight Acedemy Awards, incudiing Picture, Actiess, öest Supporting Actor, öest Director.
Phase 1:Generation of LM continuation
Phase Evaluation
2 Evidence selection
3. Factuality Evaluation
1. Clalm Filtering
ME Model
NE ER
Evidence
[LM_GEN]
Claim Type
CHECSWORTHY
Selector
XLI Modal
Entailed %
ClassiTier
1.38 LM
Fluency (ppl)
Wiki docs
N g-gram based
Diversity Repetition
Figure 1: Illustration of our evaluation framework
As illustrated in Fig 1, our evaluation framework consists of the following phases. In phase 1, LM generates the continuations from the provided test prompts ($3.1). In phase 2, we first identify check- worthy continuations, which refers to the generations with facts that require factuality evaluation. One may refer to Appendix B for details. This step is necessary as open-ended text generation may generate text that does not contain facts such as personal opinion or chitchat-style text (e.g.. "I like eating apples!"). Then, we prepare relevant ground-truth knowledge required for factual verification of check wheck-worthy continuations ($3.2). Lastly, we calculate the factuality and quality measures ($3.3).
3.1 FACTUALITY Testset
We design our test prompts (FACTUALITYPROPPT) that follows a similar setup as RealToxici- tyPrompts [61], which has toxic and nontoxic prompts to evaluate the toxicity of LM continuations. FACTUALITYPROMPTS consists of factual and nonfactual prompts that allow us to study the impact of prompts' factuality texts are not guaranteed to be factual. The data construction and statistic details are provided in Appendix D, and we will release the constructed CECTUALITYPROMPE for future research.
the LM continuation; this simulates the real-world scenario where input
3.2 Ghound-Truthe Knowledge Preparation
To evaluate the factuality a given generation, we need to prepare relevant ground-truti truth knowledge. The required ground-truth knowledge can be either document-level or sentence-level, depending on the type of factuality metrics (discussed in $3.3). The correctness of factuality evaluation crucially dependent the correctness of the ground-truth knowledge. To ensure that factuality evaluation is not distorted by the irrelevant provision of ground-truth knowledge, we do the following:
For document-level ground-truth knowledge, we directly use the Wikipedia document annotation from the FEVER dataset. This way, we can mitigate any potential error from automatic document retrieval. For sentence-level ground-truth knowledge, two different methods to maximize the chance of recalling the relevant ground-truth knowledge. We treat the generated text query 9 and Wikipedia sentences as pool of candidates C {c1, C22 (..... where N is the number of sentences in the Wikipedia document. One ground-truth sentence is retrieved by obtaining TF-IF vector representations of g and C and selecting the G; with the highest cosine similarity with the q. Another is retrieved by obtaining the contextual representation of q and C' using SentenceTrane Transformer [62] and selecting the oj with the highest cosine similarity.
do automatic sentence selection by using
3.3 Evaluation Metrics
We adapt commonly used metric designs from the hallucination literature [11]: named-entity (NE) based metric and textual entailment based metric. Each metric captures different aspect of factuality, so we use both metrics for better understanding of factuality.
Hallucinated NE Error Since NEs are one of the core building blocks of "fact", NE-related metric design is one of the common choices in literature [11, 63, 64). In this work, we specifically adopt the NE-bbaee metric [64] that designed with belief that model is hallucinating (making factual errors) if it generates NE that does not appear in the ground-truth knowledge source.

---

We define our NE-based metric to be: NEER = |HALLUNE| / |ALLNE| where ALLNE is the set of all the NEs detected in the LM generation, and HALLUNE is subset of NEAT that does not appear in the ground-truth Wikipedia document. Note that evaluating NERR requires document-level ground-truth. To ensure the quality of the metric, we also take the same precautions used by [64]. For named entities consisting of multiple words, partial n-gram overlaps are also treated as a "match". This ensures we can address the shortened form of named entities c.g.. "Barack Hussein Obama II" vs. "Obama". Note that stopwords (e.g., the, a) are not considered in the partial n-gram overlaps. The named entities are detected using a publicly available pre-trained NE detection model from Spacy.io
Entailment Ratio Textual Entailment (or natural language inference) is task of determining whether a hypothesis is entailed by. refuted by, or neutral to a given premise [65]. Entailment- based metrics are based on the rationale that factual generation will be entailed by the ground-truth knowledge (11, 1, 12, 66-68).
We define the entailment ratio as: Entail |ENTAAl |LL.gen); where AL. is set of all generations, and ENTAIl gea is the set of generations that are entailed by a entailment model. To obtain the entailment scores, we leverage a pretrained entailment model that is publicly available ; a RoBERTa [69] model fine-tuned on MNLI [70] dataset. EntailR requires sentence-level ground- truth because only a few Wikipedia sentences are relevant to specific factual information in a given generation. For example, "Barack Obama was born in Hawai"" is only relevant to the Wikipedia sentence that mentions his birth location. Note that our Entailg is a stricter form of metric that does not treat neutral class to be factual.
Generation Quality Evaluation We also evaluate the generation quality from three aspects: i) Fluency is an important aspect of text generation. We measured it by the mean perplexity of generated continuations evaluated with large pretrained LM, which is 1.3B LM in this work ii) Diversity is an important characteristic of LM that makes the generation more interesting and engaging it is bland and boring to always generate same texts. It is measured using the mean number of distinct n- t-grams (we report 4-gram), normalized by the length of text [71, 72] among the 10 generations for each prompt (i.e., in total, 160,000 generations to evaluate the diversity of each method). iii) Repetition is a common form of degeneration that is very undesirable. We measure the number of repetitive substrings that get generated at the end of the generations by using the publicly available metric code from Holtzman et al. [29].
3.4 Correlation with Human Judgement
Although NE-based and entailment-base metrics have been used in downstream NLG tasks [11], they have not been utilized for evaluating factual accuracy in open-ended text generation. To ensure their validity, we collect human annotations to evaluate the correlation between our auto- matic factuality metrics with human judgement ceammentt-iee...... i.e., are generations with higher EntailR and lower NEER errors, more likely to be perceived factual by human?
Table 2: Pearson correlation coefficients between human factuality annotation and our factuality metrics. p-values for all results are 0.00.
Annotation
Entailg
NEgR
Expert
0.81
-0.77
Majority-voting
0.47
-0.46
We obtained human annotations for 200 randomly chosen LM continuations of varying NEER and Entailı scores. The annotators are asked to fact-check the LM continuations against Wikipedia and assign factuality label (1 Factual can find supporting Wikipedia evidence. 0= Non-factual cannot find supporting Wikipedia evidence).
The fact-checking annotation is challenging and time-consuming task, as it requires the annotator to carefully read multiple evidences and reason over them. To improve the annotation quality, we have two types of annotations. The first type is two annotations from average English speaking workers on Appen.com platform, and the second type is one "expert" annotation from one of the authors who is familiar with the task and spent solid amount of time checking each samples. Based on these three annotations, we do majority voting and report the Pearson correlation results in Table 2. We also report the correlation result solely using the expert annotations, and show that there is strong correlation between human judgement of factuality and the proposed automatic metric NEgR and Entailg. NEgR is negatively correlated with factuality because the lower the NEgR error, the better the factuality.
'Refer to the code snippet provided in https: https:: httph/bbptttooo.hhcctooooo
5

---

Table 3: The factuality of LMs with different parameter size from 12M to 530B. NEga refers to the named-entity error; Entaily refers to entailment ratio, Div. refers to distinct 4-grams, and Rep. refers to repetition. means the higher the better, and mcans the lower the better.
Factual Prompt
Nonfactual Prompt
Size
Decode
NE
E
Divit
Rep.
NEga
Entailx
Div.t
Rep-J
p=0.9
63.69%
0.94%
0.90
0.58%
67.71
0.76%
0.90
0.38%
126M
greedy
48.55%
8.36%
0.03
59.06%
54.24%
6.25%
0.03
59.90%
p=0.9
56.70%
2.01%
0.87
0.55%
60.80%
1.42%
0.88
0.35%
357M
greedy 43.04%
14.25%
0.03
48.18%
46.79%
9.89%
0.04
46.30%
p=0.9
52.42%
2.93%
0.88
0.24%
56.82%
2.04%
0.89
0.25%
1.3B
greedy 39.87%
12.91%
0.05
33.13%
45.02%
8.75%
0.05
36.20%
p=0.9
40.59%
7.07%
0.90
0.11 %
47.49%
3.57%
0.91
0.08%
8.3B
greedy 28.06%
22.80%
0.07
19.41% %
32.29%
15.01 %
0.07
13.26%
p=0.9
33.30%
11.80%
0.90
0.13%
40.49%
7.25%
0,92
0.08%
530B
greedy 20.85%
31.94%
0.08
15.88%
27.95%
19.91%
0.08
16.28%
4
Factuality Analysis of Pretrained LMs
In this section, we perform factuality analysis of LMs from three aspects: i) model size, ti prompt type and fii) decoding algorithm.
Model Size Researchers have observed the trend of larger LMs outperforming smaller ones in various downstream tasks [73, 3, 2]. However, contradicting to these general observations, recent studies suggest that more misconceptions tend to be generated from larger models [31], and zero-shot fact-checking performance tend to stagnate with LM scaling [6]. We study the factuality of LMs with a range of parameter sizes (126M, 357M, 1.3B, 8.3B, 530B) to understand whether such surprising trend also applies open-ended text generation. Note that, all LMs are pretrained on the same corpus as in [4]. As shown in Table 3, generation factuality does improve with the scaling of model size, e.g., NEgR drops from 63.99% to 33.30% when parameter size scales up from 126M to 530B.
Prompt Type Prompts provided to the LM are known to significantly affect the quality and characteristies of LM continuations [61, 74, 75]. We use our factual and nonfactual prompts to test the behavior of LMs. Results in Table 3 show that both factual and nonfactual prompts can lead to nonfactual generations, although factual prompts always result in less nonfactual generations. Interestingly, the performance gap between factual and nonfactual prompts gets more prominent as the model size increases (4% to 7% in NEER as parameter size increases from 126M to 530B). This could be due to the larger LM can better understand the prompts and imitate the factual or nonfactual prompts in the continuations.
Decoding Algorithm We investigate the choice of decoding algorithms and their impacts on the factuality of generations. In particular, we compare two representative decoding algorithms that are greedy decoding (i.e., maximize generation likelihood) and nucleus sampling [29]. Nucleus sampling algorithm (a.k.a. top-p) samples only from the top subword candidates with total cumulative probability p. is popular for open-ended text generation because it solves the degeneration problems of the greedy decoding algorithm (e.g., repetition). However, the results in Table 3 show that top-p decoding underperforms greedy decoding in terms factuality, although obtains higher generation diversity and less repetition. This intuitively makes sense because top-p can be seen as adding "randomness" to encourage diversity, which as a result, can lead to factual errors. It is important to understand that factuality of sentence can be casily altered by one wrong choice of word. For example, "Barack Obama was born in 1961" will be nonfactual "1961" is changed to "1962". In the same sense, greedy decoding is more factual because its way of choosing the word with the highest probability minimizes randomness and maximizes the utilization of parametric knowledge of LM [33, 36). However, greedy decoding sacrifices generation diversity and quality.
Error Types We conduct qualitative analysis of the factual errors from greedy generation of 530B LM, to understand what are the remaining errors when the randomness from decoding choice is strictly restricted. The two notable error types were:

---

Table 1.3B LM results with different decoding algorithms. referefers named-entity error, refers to entailed class ratio, Div. refers to distinct 4-grams, and Rep. refers to repetition. means the higher, the better, and L means the lower, the better. For factual-nuch sampling, p. À and w are nucleus probability, decay factor, and decay lowerbounds respectively. See more results with different hyperparameters in Figure 2a and 2b.
Factual Prompt
Nonfactual Prompt
Decoding
NEr
Entailæ
Div.
Rep.
NBrR
Entailæt
Div.
Rep.1
Greedy
39.9%
12.9%
0.05
33.1%
45.0%
8.8%
0.05
36.2%
Top-p 0.9
52.4%
2.9%
0.88
0.2%
56.8%
2.0%
0.89
0.3%
p 1
Top-p
X-decay
0.9 10.
41.1%
10.8%
0.43
30.7%
45.7%
6.8%
0.47
34.5%
0.95
39.9%
13.0%
0.08
33.1%
44.9%
9.1%
0.09
35.9%
p I A
Top-p + X-decay p-reset
0.9 1..9
41.5%
10.3%
0.52
10.3%
45.4%
6.3%
0.57
9.1 %
0.9.
39.3%
12.8%
0.34
17.8%
44.5%
8.4%
0.45
18.9%
P I X
Top-p + d-day p-resct + w-bound ffactaal-uclent sampling)
0.910.91..
46.2%
5.0%
0.78
1.2%
52.2%
3.2%
0.80
0.5%
0.910.91.3
42.1%
10.1%
0.55
7.1%
46.5%
5.6%
0.59
6.4%
0.910.91022
41.7%
9.9%
0.52
8.6%
45.6%
6.2%
0.56
7.6%
0.9
41.0%
12.2%
0.47
13.0%
46.0%
7.0%
0.51
12.7%
0.5
39.3%
12.8%
0.38
16.1%
45,2%
7.8%
0.42
16.9%
Mixing up similar types of the named entity. For example, LM generated "The movie is based on the novel of the same name by Gayle Forman." about film called "The Best of Me". However, the correct author's s name is "Nicholas Sparks", not "Gayle Forman". Note that Gayle Forman is also an American young adult fiction author who writes similar type of novels as Nicholas Sparks.
Named Entity Mix-up:
Fabricated Fact: Fabricating some random facts. For example, "Samuel Witwer's father is a Lutheran minister." Note that, the pretraining corpus contains non-factual or fictional information, which can also contribute to such fabricated facts.
Both error types can be viewed as wrong associations of entities that appear at different parts of the training corpus with similar context. Such behavior unsurprising because these LMs are uniformly trained with the next subword prediction objective instead of a fact-related objective.
Deccding Choice
Decacing Choice foctual
nuchies jundelluudous
0.2
61
Divendty
0.33
090
Pan
603
0.46
5.40
0.50
0.52
0.42
0.44
C.MM
5.40
0.50
0.50
Ne_emo
(a) Diversity vs. NEFs
(b) Repetition vs. NEpa
Figure 2: Comparison between nucleus sampling (blue line) and factual-nucleus sampling (orange line). The x-axis is named entity error NEER. The y-axes diversity and repetition in (a) and (b) respectively. The lower the repetition, the better. It evident that factual-nucleus sampling has better trade-offs between factuality and diversity/repetition. For reference, the diversity score of randomly sampled 5000 Wikipedia documents is 0.767.

---

5 Factual-Nucleus Sampling
In this section, we propose a new sampling algorithm that achieves a better trade-off between generation quality and factuality than existing decoding algorithms.
5.1 Method
We hypothesize that the randomness of sampling is more harmful to factuality when it is used to generate the latter part of sentence than the beginning of a sentence. There no preceding text at the start of sentence, so it is safe for LM to generate anything as long as it is grammatical and contextual. However, as the generation proceeds, the premise become more determined, and fewer word choices can make the sentence factual. Given the example "Samuel Witwer's 's father is a Lutheran minister", the beginning of the sentence "Samuel Witwer's father is" not nonfactual. However, the continuation of "Lutheran minister" makes the sentence nonfactual. Therefore, we introduce the factual-nuclen sampling algorithm that dynamically adapts the "nucleus" p along the generation of each sentence. In factual-miclus sampling, the nucleus probability pe to generate the t-th token within each sentence is,
P: = max p x 1=
where is the decay factor for top-p probability, and w lower bounds the decay of probability. Specifically, it has the following parts:
deecy:: Given that top-p sampling pool is selected as set of subwords whose cumulative probability exceeds p. we gradually decay the p value with decay factor X each generation step to reduce the "randomness" through time.
p-reset: The nucleus probability p can quickly decay to a small value after a long generation. So, reset the p-value to the default value at the beginning of every new sentence in the generation (we identify the beginning of a new sentence by checking if the previous step has generated a full-stop). This reduces the unnecessary cost of diversity for any long generations.
w-bound: If X-decay is applied alone, the p-value could become too small to be equivalent to greedy decoding and hurt diversity. To overcome this, we introduce a lower-bound w to limit how far p-value can be decayed.
We will show the importance of each parts with ablation studies.
5.2
Result
We report our decoding experimental results with 1.3B LM 7 in Table 4. Additions of X-decay helps improve top-p 0.9 factuality results - for instance, with decay rate X = 0.5, there is 12.5% drop in NEFR and 10. 1% gain in Entailg. However, this affects the diversity and repetition to become similar to greedy decoding. p-reset mitigates the repetition issue and improves diversity metric without losing much in factuality metric. The effect more drastic for the À = 0.5 option, where achieves 0.26 gains in diversity metric with negligible changes in factuality scores. By also adding w-bound, we obtain the anticipated factuality performance (i.e., similar to greedy decoding), with great improvement in generation quality over greedy; with p=0,, X=0.9, w=0.3, we achieve xI1 improvement in diversity and x..6 improvement in repetition over greedy. Although our factual- nucleus sampling still under-perfer mnder-performs top-p 0.9 in terms of diversity, we believe this is an acceptable trade-off to improve the factuality of LM for factually sensitive open- generation tasks. Our proposed decoding does not harm the sentence fluency; its perplexity do not exceed the perplexity of top-p. Refer to Appendix F for full perplexity results.
To further illustrate the underlying tradc-off, we also compare the proposed factual-nucleut sampling against the nucleus sampling with lower p values that are also expected to have lower randomness, thus less factual error, in generations. Specifically, we plotted results for nucleus sampling with p= (0.9, 0.7, 0.6, 0.5, 0.4, 0.3), and factual nucleus sampling with the following p A w choices: 0.910.910.7, 0.910.910.55 0.910.910. 910.990.4, 0.910.910.3, 0.910. 710.3. The Fig 2a and Fig 2b respectively show that the factual nucleus sampling method has better trade-offs than top-p factuality- pactuality-vs-divevet -ivdiverstty and cestualiyy-v-repeeiie
In other words, it always achieves better factuality score with the same level of diversity and repetition scores.
11.B LM is mainly used as is big enough have good learning capacity yet not too resource expensive.
8

---

6 Eestuality-Enhancer Continued Training
This section introduces factuality-enhance method for continued training of LMs [30]. We introduce the TOPICPREFIX for better awareness of facts and the sentence completion loss as training objective.
6.1 Prepending TOPICPREFIX
Unstructured factual knowledge typically exists a document level (i.e., a group of factual sentences about an entity). This means that sentences can contain pronouns (e.g., she, he, it), making these sentences factually useless standalone. To illustrate with an example from Barack Obama's 's Wikipedia page, "He previously served as U.S. senator from Illinois from 2005 to 2008" cannot be a useful standalone fact because it is unclear who "He" is. Due to the GPU memory limit and computation efficiency, it is common to chunk documents in LM training corpus. This causes the "fragmentation" of information and leads to wrong associations of entities that appear in independent documents with similar contexts. As remedy, propose to prepend TOPIC to sentences in the factual documents to make each sentence serve as a standalone fact. In our experiments, we mainly utilize Wikipedia as the factual corpus and the Wikipedia document name the TOPICPREFIX.
6.2 Sentence Completion Loss
We propose a sentence completion loss to address the incorrect association learned between entities. To explain our rationale, let us recall the nonfactual example from $5: "Samuel Witwer's father is a Lutheran minister". This sentence is nonfactual because LM failed to generate factually correct information after "is". In other words, LM failed to accurately complete the sentence given the generated context. One reason is that the LM is uniformly trained to predict each subword token within the sentence, when ensuring the correct prediction at the latter section of sentence is more critical for factuality. Therefore, we construct sentence completion loss, which makes the LM focus on predicting the subwords later in the sentence. For implementation, we determine pivot for each sentence, and then apply zero-masking for all token prediction losses before t. This pivot is only required during training (i.e., no pivot needed during inference time).
We emphasize that this loss masking is different from the input token masking applied in BERT [73] or BART [76], and the LM is still trained in autoregressive manner. Note that many BART-base summarization models are known to still suffer from factual errors, suggesting that masked prediction the encoder level may not effectively transfer well to autoregressive text generation.
In this work, we explore three strategies (from simple complex) to determine the pivot
SCHALri pivot = 0.5x sentence-length.
SCrandOM: random pivot, e.g.. t ~ uniform(0.2 (0.25, 0.75] x sentence-length. SCroor: pivot = position of ROOT (relation) from dependency parsing.
Our experiments show that the simplest SCHALF performs on par with the complex ones (such as SCroor), thus, we suggest future work to choose SCHALF strategy.
6.3 Results
The results are reported in Table 5, and experimental setups are reported in Appendix
Inefficiency of Domain Adaptive Training The pre-training corpus of LM contains both factual texts (e.g.. Wikipedia) and potentially nonfactual texts (e.g., rumors, fake news) 8. The nonfactual domain of the training corpus could be the problem. Thus, we conduct a baseline experiment that does Nomain-adaptive training with strictly factual domain text only (i.e., Wikipedia). Interestingly, we find that domain-adaptive training can hardly improve generation factuality.
Effect of TOPICPREFIX Continued pre- pre-training of 1.3B LM with TOPICPREFIX preprocessed Wikipedia alone can already improve the factuality, especially in terms of NE For example, reduces the frgmfrom 42. 1% to 27.6% when which even outperforms the 1.3B with greedy decoding (NEER: 27.6% vs. 39.9%) with much less repetition (8.0% vs. 33.1%).
the factual-nucleus decoding (0.9 0.9 0.3),
Effect of Sentence Completion Loss The proposed sentence completion loss further helps to improve the factuality, especially for the EntailR. For example, when one uses factual-nucleus
SSee [4] for details of pre-training corpus.

---

Table Results factuality enhanced training. The decoding setlings are formatted as: nucleus probability p.
decay rate X, lower-bound w.
Factual Prompt
Nonfactual Prompt
Decoding
(p A w)
NE
Entailx
Div.
Rep.
NER
Entailı
Div.
Rep.
Vanilla Pretrained LM (1.3B)
0.9
52.4% 42.1%
2.9%
0.88
0.2%
56.8%
2.0%
0.89
0.3%
0.910.9103
10.1%
0.55
7.1%
46.5%
5.6%
0.59
6.4%
Factual Domain-Adaptive Training with Wikipedia (1.3B)
0.9 52.5%
2.8%
0.85
0.2%
55.8%
2.2%
0.86
0.1%
0.910.910.3
42.7%
7.1%
0.51
7.2%
48.2%
4.9%
0.56
6.0%
TOPICPREFIX (1.3B)
0.9
34.4%
4.2%
0.84
0.3%
36.2%
2.7%
0.85
0.2%
0.910.910.3
27.6%
8.7%
0,43
8.0%
30.5%
6.1%
0.47
6.9%
TOPICPREFIX SCroot (1.3B)
0.9
32.5%
6.7%
0.83
1.2%
34.3%
4.6%
0.84
1.1%
0.910.910.3
24.7%
15.8%
0,40
13.6%
27.6%
9.1%
0.44
13.7%
TOPICPREFIX SCRANDONS (1.3B)
0.9
32.0%
7.9%
0.81
1.2%
34.2%
5.5%
0.83
1.1%
0.910.910.3
23.6%
17.6%
0.39
14.2%
26.9%
9.3%
0.42
13.2%
TOPICPREFIX SCHaLe (1.3B)
0.9
31.6%
7.6%
0.81
1.4%
33.5%
5.1%
0.83
1.5%
0.910.91.3
23.6%
17.4%
0.38
14.4%
27.2%
10.2%
0.42
13.1 1%
Vanilla Pretrained LM (530B)
0.9
33.3%
11.8%
0.90
0.1%
40.5%
7.25%
0.92
0.1%
TOPICPREFIX SCuam (530B)
0.9
18.3%
19.3%
0.68
0.1%
21.7%
13.7%
0.68
0.1%
0.910.910.3
14.5%
25.5% %
0.33
0.2%
17.7% %
20.0%
0.33
0.1%
decoding on trained 1.3B model, TOPICPREFIX + SCHALF can further improve EntailR from 8.7% to 17.4% than TOPICPREFx alone, while reducing NEER from 27.6% to 23.6%. Note that the results show consistent improvement across different pivot selection strategies, suggesting that the sentence completion loss is robust. In particular, the simplest SCHalf performs good others or even better in terms of several metrics. Thus we recommend it as the default option.
530B vs 1.3B As expected, our method on 530B LM further reduces the factual errors and achieves the lowest NEER (14.5%) and the highest Entailg (25.5%). Surprisingly, our method on 530B LM lead to less diverse generation than 1.3B LM despite the significant improvement in the generation quality (i.e., near perfect repetition scores 0.1% 0.2%). We conjecture that this the trade-off' between the factuality and diversity for 530B L.M.
7 Conclusion
In this work, we establish a benchmark to measure and analyze factuality in open-ended text genera- tion tasks. We propose factual-nucleus sampling that improves generation factuality at inference time, and the combination of sentence completion loss and TOPICPREFIX pre-processing that improves factuality with continued training. We demonstrate that our methods are effective in improving the factuality. Lastly, our results shed light on the existence of the trade-off between diversity and factual- ity. We strongly believe this is an important insight that will help researchers make a better-informed decision about their model design i.e., appropriately prioritize the desirable attribute of their LM (factuality vs. diversity) according to the final goal of their task. Potential future work would to reduce the degree of the observed trade-offs.
10

---

References
[1] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAI blog. 2019.
[2] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-tex. text transformer. JMLR, 2019.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. In NeurlPS, 2020.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using DeepSpeed and Mega- tron to train EegatronnTurrng NLG 530B, a large-scale generative language model. arXiv preprint vrxiv:2201.119900 2022.
[5] Rico Sennrich, Barry Haddow, and Alexandra Birch. Noural machine translation of rare words with subword units. In ACL, 2016.
[6] Jack W Rac, Schastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11466 2021.
[7] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. WebGPT: Browser-assisted cesstion-answering with human feedback. arXiv preprint arXiv:21 2021.
[8] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint ar0iv:2205.omm 01068, 2022.
[9] Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In NeurlPS, 2019.
[10] Romal Thoppilan, Daniel De Freitas. Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha. Heng- -zee Cheng. Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201 08239, 2022.
[11] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang. Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. arXiv preprint arXiv:220..03629, 2022.
[12] Wojeiech Kryšcinskii, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. EMNLP, 2019.
[13] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In ACI, 2020.
[14] Esin Durmus, He He, and Mona Diab. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In ACL, 2020.
[15] Feng Nan, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng. Kathleen McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang. Andrew O Amold, and Bing Xiang. Improving factual consistency of abstractive summarization via question answering. In ACI-IJCNLP, 2021.
[16] Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, and Xiaoming Li. Neural generative question answering. In IJCAI, 2016.
[17] Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08911, 2020.
[18] Dan Su, Xiaoguang Li, Jindi Zhang, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. Read before generate! faithful long form question answering with machine reading. In Findings in ACL, 2022.
[19] Amit Moryossef, Yoav Goldberg, and Ido Dagan. Step- steepp: Separating planning from realization in neural data-to-text generation. arXiv preprint 10Xiv:190.003366 2019.
11

---

[20] Tianyu Liu, Xin Zheng, Baobao Chang, and Zhifang Sui. Towards faithfulness in open domain table-to-text generation from entity-centric view. In AAAI, 2021.
[21] Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang. Heng Ji, and Meng Jiang. A survey of knowledge-enhancedd text generation. arXiv preprint arXiv:2010.043389 2020.
[22] Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gautier Izacard, Patrick Lewis, Barlas Oğuz, Edouard Grave, Wen-tau Yih, al. The web is your oysler-knowledge- intensive nlp against a very large web corpus. arXiv preprint arXiv:2 2021.
[23] Peter West, Chris Quirk, Michel Galley, and Yejin Choi. Probing factually grounded content transfer with factual ablation. arXiv preprint arXiv:2203. 10133, 2022.
[24] Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa, and Yoshua Bengio. A neural knowledge language model. arXiv preprint aroivv:608,0000 00008000318... 10031.200 2016.
[25] Robert L Logan IV, Nelson F Liu, Matthew E Peters, Matt Gardner, and Sameer Singh. Barack's wife hillary: Using knowledge- graphs for fact-aware language modeling. In ACL, 2019.
[26] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katic Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. arXiv preprint 1rXiv:21 2021.
[27] Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne. Yacine Jernite. Vladimir Karpukhin, Jean Maillard, et al. Kilt: benchmark for knowledge intensive language tasks. arXiv preprint wxXt:2000.002252 2020.
[28] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Was Wei Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint ar
2020.
[29] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In /CLR, 2020.
[30] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't ton't stop pretraining: adapt language models to domains and tasks. In ACL, 2020.
[31] Stephanie Lin, Jacob Hilton, and Owain Evans. falsehoods. ACL, 2022.
TruthfulQA: Measuring how models mimic human
[32] David Krathwohl. A revision of bloom's taxonomy: An overview, Theory into practice, 41(4):212-21 218. 2002.
[33] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller, and Sebastian Riedel. Language models as knowledge bases? In EMNLP, 2019.
[34] Carlos Aspillaga, Marcelo Mendoza, and Alvaro Soto. Inspecting the concept knowledge graph encoded by modern language models. Findings of ACL, 2021.
[35] Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. Evaluating commonsense in pre-trained language models. In AAAI, 2020.
[36] Zhengbao Jiang, Frank F Xu. Jun Araki, and Graham Neubig. How can we know what language models know? Transactions the Association for Computational Linguistics, 2020.
[37] Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [mask]] Learning vs. learning to recall. arXiv preprint Xix: 204.0220, 2021.
[38] Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:1012-1031, 2021.
[39] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In EMNLP, 2021.
[40] Joel Jang, Sconghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyconghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models. arXiv preprint arXiv:2110.03215, 2021.
[41] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual knowledge in GPT. arXiv preprint 22X::22222.0562. 2022.
12

---

[42] Ziqiang Cao. Furu Wei, Wenjie Li, and Sujian Li. Faithful to the original: Fact aware neural abstractive summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
[43] Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. Multi-fact correction in abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3320-9331 2020.
[44] Luyang Huang, Lingfei Wu, and Lu Wang- Knowledge wraph-augmented abstractive summarization with semantic-driven cloze reward. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
[45] Yichong Huang, Xiachong Feng, Xiaocheng Feng, and Bing Qin. The factual inconsistency problem in abstractive text summarization: A survey. arXiv preprint arXiv:2104. 148399994 2021.
[46] Shuyang Cao and Lu Wang. Cliff: Contrastive learning for improving faithfulness and factuality in ahstractive summarization. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6633-6649 2021.
[47] Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng. Xuedong Huang, and Meng Jiang. Enhancing factual consistency of abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 718-733. 2021.
[48] Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. Improving faithfulness in abstractive summarization with contrast candidate generation and selection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5935-5941, 2021.
[49] Sam Wiseman, Stuart Shieber, and Alexander Rush. Challenges in data-to-document generation. In Pruceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pages 2253-2263. ACL, 2017.
[50] Feng Nie, Jin-Ge Yao, Jinpeng Wang, Rong Pan, and Chin- Yew Lin. A simple recipe towards reducing ballucination in neural surface realisation. In Proceedings of the 57th Anmual Meeting of the Association for Computational Lingwistics, pages 2673-2679. ACL, 2019.
[51] Yixuan Su, David Vandyke. Sihui Wang. Yimai Fang, and Nigel Collier. Maa-tthen-generate Controlled data-to-text generation via planning. Findings of EMNLP, 2021.
[52] Peng Wang, Junyang Lin, An Yang, Chang Zhou, Yichang Zhang, Jingren Zhou, and Hongxia Yang. Sketch and refine: Towards faithful and informative table-to-text generation. ACL, 2021.
[53] Clément Rebuffel, Marco Roberti, Laure Soulier, Geoffrey Scoutheeten, Rossella Cancelliere, and Patrick Gallinari. Controlling hallucinations word level in data-to-text generation. Data Mining and Knowledge Discovery, pages 318-354, 2022.
[54] Lei Shen, Haolan Zhan, Xin Shen, Hongshen Chen, Xiaofang Zhao, and Xiaodan Zhu. Identifying untrust- worthy samples: Data filtering for open-domain dialogues with bayesian optimization. In Proceedings of the 30th ACM International Conference on Information Knowledge Management, pages 1698-1608. 2021.
[55] Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retricval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021. ACL, 2021.
[56] Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. Increasing faithfulness in mmowledge-groundee dialogue with controllable features. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 704-718. ACL, 2021.
[57] Zeqiu Wu, Michel Galley, Chris Brockett, Yizhe Zhang, Xiang Gao, Chris Quirk, Rik Koncel-Kedzionski, Jianfeng Gao, Hannanch Hajishirzi. Mari Ostendorf, et al. A controllable model of grounded response generation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 14085-14093, 2021.
[58] Nouha Dziri, Andrea Madotto, Osmar Zaiane, and Avishek Joey Bose. Neural path hunter: Reducing hallucination in dialogue systems via path grounding. EMNLP, 2021.
13

---

[59] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02/55. 2022.
[60] Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. MAUVE: Measuring the gap between neural text and human text using divergence
frontiers. In NewrIPS. 2021.
[61] Samuel Gehman, Suchin Gururangan, Maarten Sap. Yejin Choi, and Noah A Smith. Realtoxicityprompts:: Evaluating neural toxic degeneration in language models. In Findings in FMNLP,, 2020.
[62] Nils Reimers and Iryna Gurevych. Sentence-bert, Sentence embeddings using siamese bert-networks. arXiv preprint ar9iv: 10084, 2019.
[63]
Ben Goodrich, Vinay Rao, Peter J Liu, and Mohammad Saleh. Assessing the factual accuracy of generated text. In ACM SIGKDD, 2019.
[64] Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. Entity-level factual consistency of abstractive text summarization. In EACL, 2021.
[65] Bill MacCartney and Christopher D. Manning. Modeling semantic containment and exclusion in natural language inference. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 521-528 Manchester, UK, August 2008. Coling 2008 Organizing Committee. URL prrg/Co--166.
[66] Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Tryna Gurevych. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2220,, 2019.
[67] Ondřej Dušek and Zdeněk Kasner. Evaluating semantic accuracy of data-to-text generation with natural language inference. arXiv preprint S..iv:2011.108111 2020.
[68] Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. Evaluating groundedness in dialogue systems: The begin benehmark. arXiv preprint (tt::2005.00711 2021.
[69] Yinhan Liu, Myle Ott, Naman Goyal. Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arviv: /907. / /692, 2019.
[70] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL, 2018.
[71] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. In NAACL, 2016.
[72] Zhihong Shao, Minlie Huang, Jiangtao Wen, Wenfei Xu, and Xiaoyan Zhu. Long and diverse text generation with planning- based hierarchical variational model. arXiv preprint arXiv: 1008::190.00000 06605, 2019.
[73] Jacob Devlin. Ming-Woy Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv: 1810.04805 2018
[74] Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, Mostofa Patwary, Mohammad Shoeybi. Bo Li, Anima Anandkumar, and Bryan Catanzaro. Exploring the limits of domain-adaplin training for detoxifying large-scale language models. In NewrlPS, 2022.
[75] Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nip. In EMNLP, 2019.
[76] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazviningjad, AbdeIrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-scquenee pre-training for natural language generation, translation, and comprehension. arXiv preprint 1rXiv:1910.13461, 2019.
[77] Mohammad Shocybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billig parameter language models using model parallelism. arXiv preprint arxiv: /90900003, 2019.
[78] James Thorne, Andreas Vlachos. Christos Christodoulopoulos.. and Arpit Mittal, FEVER: a large-scale dataset for fact extraction and verification. In NAACL, 2018.
14


Reviews:
[{"review_id": "tBeuEcEZNI4", "reviewer": "Reviewer_wMPr", "summary": "This paper addresses the tendency of open generation systems to produce factually incorrect sequences. In order to quantify and measure factual correctness of text generative models, a dataset comprising factual and wrong prompts is created. This dataset is created from a fact checking dataset FEVER that has true and fake documents with their corresponding Wikipedia sources. The proposed dataset uses FEVER to obtain the linked Wikipedia article for each prompt and defines metrics mainly focused on named entities and entailment of generation by the corresponding sources. This paper finds that larger LMs are more factually correct, but in general tend to mix and match the named entities with wrong facts or fabricate entirely new named entities, especially with nucleus sampling. The authors propose to vary the nucleus parameter during nucleus sampling as the decoding proceeds so that there is more randomness at the start of a sentence and less randomness as it ends, so that nucleus sampling generates factually correct sequences.\n\nThe paper also proposes fine tuning objectives to improve factual correctness. One of the strategies is to prepend sequences with topic titles (obtained via Wikipedia titles), and the other approach is to use the latter part of sentences for loss computation with the idea being that facts are generated toward the end which have to be consistent with the information generated in prior steps. Empirical analysis is performed to test the proposed approaches.", "questions": "Please find my comments in the section above.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The identified problem is important and the associated dataset and metrics will be a useful resource for factuality assessment of generative models. That said, the scope of the dataset is limited to Wikipedia articles and hence the metrics might be biased toward Wikipedia-like language. Extending to other kinds of knowledge bases does not seem very straightforward as this approach depends on a pre-exisitng dataset called FEVER which in turn heavily depends on Wikipedia.\n\nThe findings that large LMs are more factual is interesting and the proposed fixes also show that the larger LMs show greater improvement.\n\nNucleus sampling is compared with greedy decoding which is a bit unfair. It should be compared with other sampling approaches like ancestral sampling, sampling with a range of temperatures etc. Also, the proposed fix seems ad-hoc as it encourages deterministic output toward latter parts of the generation. However, the prompt itself sets up some background so following the proposed motivation, there should be less stochasticity in the generations immediately following the prompt as well. It should be noticed that barring diversity, greedy decoding is superior on other metrics. Comparison with temperature based sampling will help -- especially if the same scheme is applied to the evolution of temperature as is applied to the nucleus parameter. This would shed more light on the effect of reducing stochasticity gradually while generating.\n\nThe proposed training fixes are simple heuristics as well but the results do seem to suggest that they are very effective at improving factuality generation. However, it should be noticed that fine tuning is done on Wikipedia as well which favors the proposed dataset and metrics. Experiments with fine tuning on factual data from sources other than Wikipedia would be helpful. Also, these approaches do require finetuning of a large model though which can be expensive.\n\nOverall, the paper has important contributions but they come with some caveats.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nKxhPaM4FJR", "reviewer": "Reviewer_2ne3", "summary": "This papers proposes FACTUALITYPROMPTS, a new benchmark for open-ended text generation factuality evaluation. The dataset used for the benchmark is selected from FEVER. Based on the document-level and sentence-level factuality, the authors adapt two evaluation metrics for factuality evaluation focusing on named entities hallucination and textual entailment. Using the proposed benchmarks, the authors test language models with different sizes and also compare greedy decoding and top-p sampling. Informed by their observation that greedy decoding leads to more factually consistent generation and that tokens appearing later in the sentences are more important, the authors proposed a modified top-p decoding with decaying p within sentences (factual-nucleus sampling). Lastly, the authors presents a modified training method to enhance factuality, which mask off training signals from first few tokens in the sentence (tokens to the left of a “sentence pivot” token).", "questions": "Please see weaknesses 1 - 3.", "limitations": "The authors should address if the proposed benchmark and the trained models may have any unfair biases or other potential negative impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The authors conduct extensive experiments to quantify the level of factuality in text generation, and how their proposed inference and training method improve factuality. \n\n2. The paper is innovative as most previous works target factuality for specific downstream tasks (such as text summarization) but this work focus on general purpose open-domain text generation\n\n3. The paper is well-written.\n\n\nWeaknesses:\n\n1. From Section 6, I am not sure if the proposed training method \"topic-prefix\" and \"sentence completion loss\" individually could bring significant improvement in smaller-size language models.\n\n2. The proposed factuality-based training requires “topic prefix” which may not be available in other domains like \"arXiv papers, medical reports\". It would be helpful if the authors could address that limitation / discuss how to obtain such topic prefix from domains outside of wikipedia. \n\n3. It is not clear how the \"range of language models\" with different sizes are trained and tested, e.g. the data (and domains) used to pre-train these language models. Can we decouple the factuality from the pre-training data from factuality that the model learns from the authors' fine-tuning data?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kmV5vmrcbOS", "reviewer": "Reviewer_4pX8", "summary": "This paper examines the factual incorrectness issue in text generation and proposed some decoding and training method to address these. Experimental results showed some insights regarding the effect of decoding on factuality, and the proposed methods showed improved factuality based on automatic evaluation metrics. \n", "questions": "\nAdditional suggestions: \nProviding more examples would help better understand the system output. \n", "limitations": "Authors didn't provide this in the paper, I think factuality is an important topic and worth adding some discussion for the model's limitations and societal impact of the technology.\n\n ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper addresses an important problem in text generation using large language models. \n\nThe analysis results regarding greedy and nucleus sampling decoding are intuitive. The proposed decoding method is a reasonable tradeoff between greedy and nucleus sampling, and it’s good to see that empirical results show it’s effective. Authors also provided results when only wikipedia is used, i.e., domain adaptive pertaining, there is less effect on factuality, supporting the improvement comes from their proposed approach. \nOverall the paper has some interesting results, however, it leaves some questions in its current form, and is not ready for publication yet. \n\nAbout the results:\nAutomatic metrics can give some information about the model quality. But human evaluation is an important aspect in generation evaluation and providing those results would strengthen the paper. Such results are not needed for every setup.   \n\nThe proposed method lacks some clarify and justification. Specifically the factuality enhanced pretaining  is not well justified.  \n\nUnclear parts about the additional pertaining:\nAuthors motivated the TopicPrefix using the pronoun issues in the document. However, I don’t see why adding that prefix address the issue. Even though it’s telling the model this part is related to factual information, it can still connect with the wrong entities.\nAnother question is how is this prepending used during testing text generation?  Does it know for what prompts this needs to be prepended?  If so, is this a reasonable assumption? \n\nSimilarly for the sentence completion loss, it’s not well explained or justified how or why applying zero-masking for the token prediction loss helps factuality.  For different pivot it’s only used during training, during testing, nothing is needed? \n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eOInRiR1iRn", "reviewer": "Reviewer_TZp5", "summary": "In this paper, the authors highlight the factuality problem in the generation of language modeling. And they proposes a two-fold solution to improve factuality. The first part includes several sampling adaptations to avoid unnecessary logits sacrifice according to the growing sequence length. The second part includes a topic prefix method and sentence completion as the additional loss. The results show that their methods reduce the entity and entailment error in a factual-based perspective.", "questions": "It would be helpful to elaborate on line 303.", "limitations": " ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- The entity error and entail error measurements are reasonable criteria for factuality measurement.\n- The observation of randomness in nucleus sampling adds value to better understand the factuality in language generation.\n\nWeaknesses\n- It seems that the entity ratio is not the whole world of factuality, organization and representation of sentences, is important as well apart from the entity mentions (e.g. negations). It would better if the experiments are conducted on some factual intensive tasks or having some human evaluations to help better interpret the results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept