SlowGuess/NAACL_2021_OCR huggingface dataset:
Recent pre-trained abstractive summarization systems have started to achieve credible performance, but a major barrier to their use in practice is their propensity to output summaries that are not faithful to the input and that contain factual errors. While a number of annotated datasets and statistical models for assessing factuality have been explored, there is no clear picture of what errors are most important to target or where current techniques are succeeding and failing. We explore both synthetic and human-labeled data sources for training models to identify factual errors in summarization, and study factuality at the word-, dependency-, and sentence-level. Our observations are threefold. First, exhibited factual errors differ significantly across datasets, and commonly-used training sets of simple synthetic errors do not reflect errors made on abstractive datasets like XSUM. Second, human-labeled data with fine-grained annotations provides a more effective training signal than sentence-level annotations or synthetic data. Finally, we show that our best factuality detection model enables training of more factual XSUM summarization models by allowing us to identify non-factual tokens in the training data.<sup>1</sup>
Hallucination of unsupported or incorrect facts is a known shortcoming of current text generation and summarization models (Cao et al., 2018; Falke et al., 2019). This has been established for both abstractive summarization models (Maynez et al., 2020) and extractive summarization models (Kryscinski et al., 2020; Falke et al., 2019). Past work has explored using off-the-shelf frameworks such as entailment models (Falke et al., 2019) or QA systems (Durmus et al., 2020; Wang et al.,
2020) to detect and sometimes correct errors in generated summaries. Another line of recent work has used synthetically generated data to specifically train models on the factuality detection task (Kryscinski et al., 2020; Zhao et al., 2020; Goyal and Durrett, 2020a). However, these efforts have focused on different datasets, summarization systems, and error types, often shedding little light on what errors state-of-the-art systems are actually making and how to fix them.
In this paper, we aim to answer two main questions. First, while synthetic data generation approaches are specifically designed for factuality evaluation, do these align with actual errors made by generation models? We find the answer is no: techniques using surface-level data corruption (Kryscinski et al., 2020; Zhao et al., 2020; Cao et al., 2020) or paraphrasing (Goyal and Durrett, 2020a) target inherently different error distributions than those seen in actual model generations, and factuality models trained on these datasets perform poorly in practice. Furthermore, we show that different summarization domains, CNN/Daily Mail (Hermann et al., 2015; Nallapati et al., 2016) and XSum (Narayan et al., 2018) (which differ in the style of summaries and degree of abstraction), exhibit substantially different error distributions in generated summaries, and the same dataset creation approach cannot be used across the board.
Second, we investigate the best approach for modeling and learning factuality, particularly for highly abstractive summarization settings (Narayan et al., 2018). Specifically, we compare the utility of fine-grained human annotations (such as error highlighting at the word- or span-level) with sentence-level factuality annotations. We use a prior factuality detection model capable of leveraging such fine-grained annotations (Goyal and Durrett, 2020a) and show that these allow us to more reliably detect errors as well as localize those errors within generated texts. In fact, fine-grained
$^{1}$ Code and data available at https://github.com/tagoyal/factuality-datasets
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1449-1462. June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: Examples from the synthetic and human annotated factuality datasets. The entity-centric and generation-centric approaches produce bad summaries from processes which can label their errors. All models can be adapted to give word-level, dependency-level, or sentence-level highlights, except for Gen-C.
human annotations are almost essential for any of our techniques to work well with high-performing summarizers in the challenging XSUM setting.
Finally, we demonstrate a practical application for such error localization capabilities beyond interpretibility. Given noisy training data for summarization, we employ a modified training objective that leverages information about error spans in gold summaries, derived from factuality models, to train the summarizer. Our results show that models trained using this approach are inherently more factual than standard training objectives when dealing with error-prone gold datasets.
2 Training Datasets to Compare
We first seek to answer how well synthetic training data can help address factuality errors observed in real summarization datasets. Figure 1 shows a summary of the approaches we consider, which we describe in detail in Section 2.1 and 2.2.
The summarization models we analyse are trained on two English-language domains: (1) XSUM, an "extreme" summarization dataset from British Broadcasting Corporation (BBC) articles, where the first sentence of the article is treated as a summary of the article. These summaries are highly abstractive in nature: summarization models trained on this dataset have to learn to model long-range dependencies and may still be unable to recover all information in the gold summary. (2) CNN/DAILYMAIL, a multi-sentence abstrac
tive summary dataset. The level of abstraction in this dataset is considerably lower and reference summaries exhibit high overlap with source articles (Zhang et al., 2018).
For both of these domains, we compare the distribution of factuality errors from synthetic training data with the distribution of observed factuality errors from models trained on that data. In Section 4, we further dive into factuality models' performance in these settings.
2.1 Entity-centric Synthetic Data (Ent-C)
A recent thread of work has focused on leveraging synthetic data transformations for evaluating factuality (Kryscinski et al., 2020), imposing decoding-time constraints (Zhao et al., 2020), or post-correction of summaries (Cao et al., 2020). Each of these approaches assumes that corruption strategies will yield useful non-factual summaries, while gold summaries are treated as factual. Figure 1 illustrates this process: these approaches apply transformations to either the source article (shown) or a reference summary to obtain a corrupted summary (Ohio instead of Norfolk).
We call this set of approaches entity-centric because the transformations largely focus on perturbing entities and noun phrases and addressing these types of hallucinations. The approach from Kryscinski et al. (2020) has the broadest set of transformations out of this line of prior work, so we follow them to generate training examples representative of this class of techniques. The data
Gold: Apple has been accused of misleading customers over its new iPad 3.0 version.
Entity Swap	Apple has been accused of ... over its new iPhone 3.0 version.
Number Swap	Apple has been accused of ... over its new iPhone 10 version.
Pronoun Swap	Apple has been accused of ... over her new iPhone version.
Negation	Apple has not been accused of ... over its new iPhone version.
Noise Injection	Apple has has been accused of ... over its new iPhone version.
Paraphrase	Customers have accused Apple of misinformation over its new iPad 3.0 version.
Figure 2: Set of transformations/data corruption techniques from Kryscinski et al. (2020) used to generate training data for the entity-centric approach.
corruptions or transformations included are entity and number swapping, pronoun swapping, sentence negation, and arbitrary noise injection. Additionally, backtranslation is used to paraphrase summaries and further augment the dataset. Figure 2 illustrates the complete set of transformations applied to the reference summary to construct the synthetic dataset.
For CNN/DM, we use a dataset of 50k labeled pairs that is a subset of the data distributed by Kryscinski et al. (2020); this subset is sufficient to reproduce the performance of their factuality classifier. We generate a similarly-sized dataset for XSUM. Note that although the data creation procedure produces sentence-level annotations, since data corruptions are introduced in a rule-based manner, we can highlights spans within the summaries where the error was actually introduced to get span-level factuality annotations as well. Figure 1 illustrates these spans in red. The figure also demonstrates how to obtain dependency-level factuality judgements from error span highlights; what these mean and how these are derived is explained in Section 2.2.
2.2 Generation-centric Synthetic Data (Gen-C)
Goyal and Durrett (2020a) introduce a different method for obtaining factuality annotations that more closely align with errors made by generation models. The core assumption of that generation-centric approach (see Figure 1) is that gener-
ated paraphrases at the bottom of a paraphrasing model's beam (the 10th-best paraphrase) are more likely to contain factual errors than 1-best generations, and new information in these generations can be labeled non-factual. Moreover, these generations align with realistic errors made by generation models, unlike purely synthetic entity swaps. In addition to sentence-level annotations, this approach also extracts factuality labels corresponding to each dependency arc of the generated summary. According to the definition given in Goyal and Durrett (2020a), an arc is factual (or entailed) if the semantic relationship described by that particular dependency arc is entailed by the source article. Figure 1 shows a non-factual created  $\rightarrow$  necklace collapsed dependency arc.
To adapt this data creation approach for our current experimental setting, we generated paraphrases of gold summaries using the paraphrase generation model of Goyal and Durrett (2020b). We use the 10th-best generated summaries to generate both sentence-level and dependency-level annotations automatically. See Figure 1 for an example of this process. We generate 40k training examples for both CNN/DM and XSUM domains.
2.3 Types of supervision
The two techniques, Ent-C and Gen-C, naturally generate annotations at different levels. We take steps to unify these formats to enable apples-to-apples comparison of them.
For Ent-C as well as human-labeled data (discussed later), we have access to span highlights within the summary that are non-factual with respect to the source article. From these, we can derive dependency-level annotations in the following way: for each arc in the summary, if either the head word or the child word is highlighted as non-factual, the dependency arc is annotated as non-factual. Otherwise, the arc is factual. This process is demonstrated in Figure 1.
Table 1 gives a summary of the type of annotations available for the 3 types of training datasets. Mapping Gen-C dependency-level annotations to word-level classification decisions is less well-defined, so we do not attempt to do this. Our focus in this work will be on training sentence-level and dependency-level classification models, which is possible on all our sources of data.
Source Article US technology firm Apple has offered to refund Australian customers who felt misled about the 4G capabilities of the new iPad. The country's consumer watchdog has taken Apple to court for false advertising because the tablet computer does not work on Australia's 4G network. Apple's lawyers said they were willing to publish a clarification. [...] At a preliminary hearing, Apple lawyer Paul Anastassiou said Apple had never claimed the device would work fully on the current 4G network operated by Telstra. Apple says the new iPad works on what is globally accepted to be a 4G network. The matter will go to a full trial on 2 May.
Figure 3: Taxonomy of error types considered in our manual annotation. On the right are example summaries with highlighted spans corresponding to the error types; the first summary is an actual BART generated summary while others are manually constructed representative examples.
Dataset Source	Sent-level	Word-level	Dep-level
Ent-C	✓	✓	✓d
Gen-C	✓	✓
HUMAN-XSUM	✓	✓	✓d
Table 1: Summary of the annotations available for each training dataset.  $\checkmark$  indicates that annotations at that granularity can be directly obtained from the data creation process.  $\checkmark^d$  indicates that annotations can be derived.
3 Analysis of Error Types
Past work using synthetic training data implicitly assumes that training a factuality model on such data will allow it to transfer to realistic settings. We start by qualitatively analyzing the actual errors produced by summarization models to see how these align with the synthetic data, which helps us better understand this assumption.
We identify four broad categories of errors (see Figure 3) that we will identify through manual inspection. Each of these categories is further divided into Intrinsic (errors that arise as a result of misinterpreting information from the source article) and Extrinsic (errors that hallucinate new information or facts not present in the source article), following the characterization from Maynez et al. (2020).
1. Entity-Related: errors specifically related to surface realization of named entities, quantities,
dates, etc. Hallucination of new entities is an extrinsic error; incorrectly combining distinct entities from the source article is an intrinsic error (Paul Telstra in Figure 3).
2. Event-Related: errors with incorrect claims about events in the summary, such as predicates with arguments filled by incorrect entities. Hallucinations of new events (held a press conference in Figure 3) are extrinsic; mixed-up attributes from within the source article are intrinsic (apple lawyer never claimed in Figure 3, incorrect agent).
3. Noun Phrase-Related: errors related to noun phrases other than the entity-specific errors. Examples include hallucinating new NP modifiers (extrinsic) or combining with a wrong modifier from the article (intrinsic).
4. Other Errors: errors such as ungrammatical text, repeated words, highly erroneous spans, etc. that don't fall into one of the above categories. These are not broken down by intrinsic/extrinsic.
Our taxonomy of summarization errors differs from that of Lux et al. (2020): theirs is targeted at the effects on the reader, whereas ours is more directly tied to the grammatical role of the error, which we believe is more useful to improve our
Figure 4: Fractions of examples in each dataset exhibiting different error types (note a single example may have multiple errors). The graphs show a significant mismatch between the error distributions of actual generation models and synthetic data corruptions.
data and our systems. We use the above taxonomy to annotate examples from both summarization domains. For XSUM, we use the state-of-the-art BART model (Lewis et al., 2020) to generate summaries followed by manual annotation (100 examples). For CNN/DM, annotation was done on the 50 summaries across 10 different models collected by Kryscinski et al. (2020). We additionally do this annotation for the artificially introduced errors in Ent-C and Gen-C. $^{2}$
Results Figure 4 shows the distribution of errors for these different settings. First, we see that summarization models from different domains make substantially different types of errors. Models trained on XSUM learn to hallucinate new content and consequently produce extrinsic errors:  $60\%$  of the errors made by BART models are extrinsic. One reason for this is that the XSUM data was automatically constructed and contains gold summaries that are noisy or non-factual (75% of gold summaries, according to Maynez et al. (2020)). In addition to this, the gold summaries are also highly abstractive, and XSum-trained summarization models learn to combine informa
Discussion of inter-annotator agreement is included in Appendix A.
tion from different parts of an article, leading to models making long-range dependency errors. This misinterpretation of content is largely responsible for the  $40\%$  of the errors which are intrinsic.
On the other hand, the CNN/DM summarization datasets contain human written gold summaries and are therefore generally much more reliable. The models trained on this dataset reflects that. Only  $14\%$  of the generated summaries contains errors in the CNN/DM validation set from (Kryscinski et al., 2020). Of these  $14\%$ , the bulk of the errors produced are intrinsic errors, primarily event-related caused by sentence compression or fusion, which is common in this dataset (Lebanoff et al., 2019). For example, the two Delaware boys are in critical condition at the U.S. Virgin Islands should instead be ...at the hospital after a trip to the U.S. Virgin Islands. The generation models rarely makes extrinsic hallucinations, and we observed that these are even less common in recent systems like PEGASUS (Zhang et al., 2020a). This aligns with the findings from prior work analysing summarization models (Fabbri et al., 2021).
Comparing these with synthetic error distributions, we can see that synthetic datasets do not reflect the error distributions of actual generation models. To the extent that Ent-C covers intrinsic event-related errors, these are almost exclusively from pronoun swaps. Moreover, because CNN/DM and XSUM feature such different errors, a synthetic dataset inspired by observed errors on one setting is not likely to be effective on the other. Later (in Section 5.1), we provide further evidence of this mismatch for both datasets: models trained on this synthetic data perform poorly when evaluated on actual generation errors. Also, models trained on human annotated XSUM training data do not transfer to the CNN/DM domain.
4 Factuality Models to Compare
Next, we investigate how factuality models trained on these synthetic datasets perform on real generation errors. Given a document  $D$ , a factuality model predicts whether all the information in a generated summary  $S$  is supported by the source document  $D$ .<sup>3</sup> We consider two factuality modeling formulations: (1) a Sentence-Factuality model that
3Factuality is ill-defined: whether inferences, world knowledge, implicatures, etc. are viewed as factual is not standardized and is dependent on human annotators for each dataset or task. However, existing generation models only rarely exhibit tricky cases along these dimensions.
Figure 5: The dependency arc entailment (DAE) model from (Goyal and Durrett, 2020a). A pre-trained encoder is used to obtain arc representations; these are used to predict arc-level factuality decisions.
makes a factuality judgment at the entire summary-level, and (2) an Arc-Factuality model (Goyal and Durrett, 2020a) that makes independent factuality judgments for dependency arcs of the generated summary, which are then combined to obtain a sentence-level decision. This helps in localizing factuality errors and was shown to be more effective than sentence-level models in prior work. $^4$
4.1 Sentence-Factuality Model
Prior work (Kryscinski et al., 2020) used a BERT-based sequence-pair classification model (Devlin et al., 2019) as follows: the source document  $D$  and the generated summary  $S$  are concatenated and fed into a pre-trained transformer encoder model (BERT, ELECTRA, etc.). The representation of the [CLS] token is fed into a linear and softmax layer that outputs a probability distribution over the output labels  $(y = \{\text{Factual}, \text{Non-Factual}\})$ . This model can be trained on any data with summary-level factuality labels.
4.2 Arc-Factuality model
The Dependency Arc Entailment (DAE) model (Goyal and Durrett, 2020a) evaluates factuality at the dependency arc level. Let  $d(S)$  be the dependency-parse of the generated summary  $S$ . For each arc  $a \in d(S)$ , the DAE model predicts whether the relationship described by the arc is entailed by the input document. Note that these factuality judgements are made independently for each arc in the summary, and can differ across arcs within the same summary. For instance, in the ex
ample in Figure 5, the arc arrested  $\leftarrow$  games is non-factual: in context, it is not the case that the games are being arrested. However, the arc seven  $\leftarrow$  games is supported by the input (there are seven games) and hence, entailed.
The model architecture is detailed in Figure 5. First, the document  $D$  and summary  $S$  are concatenated and fed through a pre-trained encoder  $E$ . Arc representations  $\mathbf{r}_a$  are derived for each dependency arc  $a \in d(S)$ :  $\mathbf{r}_a = [E(D; S)_{a_h}; E(D; S)_{a_c}]$ . Here,  $a_h$  and  $a_c$  correspond to the head and child words of arc  $a$  respectively. The arc representation  $\mathbf{r}_a$  is fed into a classification layer that outputs a probability distribution over the output labels ( $y_a = \{\text{Factual}, \text{Non-Factual}\}$ ). Finally, summary-level judgments are extracted from these arc-level decisions: if any dependency arc is non-factual, the generated summary is labeled as non-factual.
The DAE model is trained from arc-labeled examples of the form  $(D, S, \{y_a\}_a \in d(S))$ . These are derived from either synthetic or human-labeled data, as described in Section 2.
DAE with weak supervision (DAE-Weak) DAE training requires gold annotations at the dependency-level; however, such fine-grained annotations may not always be available. We extend the DAE framework to address this. The core idea behind our approach is that the sentence-level labels naturally impose loose constraints on the arc-level labels.
The constraints are as follows: for a factual example, all individual arcs in the summary must be factual. For a non-factual example, at least one arc must be non-factual, and this arc should be one not present in the source document. The DAE-Weak model is trained to maximize the marginal likelihood of all labelings that obey these constraints.
Let  $F$  be the set of all arcs that should be factual (contains all arcs with sent-label  $= 1$  and arcs common with the source article for sent-label  $= 0$ ). The above constraints are formulated as the following training objective:
$$
\begin{array}{l} \mathcal {L} = \log \left[ \prod_ {a \in F} P (y _ {a} = 1 \mid D, S) \right] \\ + \log \left[ 1 - \prod_ {a \in D (S) \backslash F} P (y _ {a} = 1 \mid D, S) \right] \\ \end{array}
$$
The second term in the above equation is the probability of predicting at least one non-factual arc in
<sup>4</sup>We describe models for single-sentence summaries to align with the available human-annotated test set (described later in Section 5.1). However, it is straightforward to extend these frameworks for multi-sentence summaries.
Ent-C	Gen-C
Majority Label	50	50
Kryscinski et al. (2020)	74.1	-
Sent-Factuality	72.3	64.4
DAE	76.7	72.1
DAE-Weak	75.2	71.1
a non-factual summary.5
5.1 Evaluation of Synthetic Training Datasets
CNN/DAILYMAIL First, we compare the performance of the three models (Sent-Factuality, DAE and DAE-Weak) trained on the two synthetic factuality datasets (outlined in Section 2) on the CNN/DAILYMAIL domain. We compare their performance on the human-annotated test dataset from Kryscinski et al. (2020). The test set contains human-annotated sentence-level factuality judgements for 503 (article, summary) pairs for summaries generated using 10 different generation models. We use the validation set provided by the authors to choose the best model checkpoint across all settings. Similar to the original paper, we report class-balanced accuracy values.