SlowGuess/NAACL_2021_OCR huggingface dataset:
Artidoro Pagnoni Vidhisha Balachandran Yulia Tsvetkov
{apagnoni,vbalacha,ytsvetko}@cs.cmu.edu
Modern summarization models generate highly fluent but often factually unreliable outputs. This motivated a surge of metrics attempting to measure the factuality of automatically generated summaries. Due to the lack of common benchmarks, these metrics cannot be compared. Moreover, all these methods treat factuality as a binary concept and fail to provide deeper insights on the kinds of inconsistencies made by different systems. To address these limitations, we devise a typology of factual errors and use it to collect human annotations of generated summaries from state-of-the-art summarization systems for the CNN/DM and XSum datasets. Through these annotations we identify the proportion of different categories of factual errors in various summarization models and benchmark factuality metrics, showing their correlation with human judgement as well as their specific strengths and weaknesses.
Factuality is defined as a measure of "whether eventualities are characterized as corresponding to facts, possibilities, or situations that do not hold in the world" (Sauri, 2008; Saurí and Pustejovsky, 2012). In summarization, this "world" is the article, which is taken as ground-truth, and the output summary must be faithful to the article's facts. Despite advancements in neural abstractive summarization (Narayan et al., 2018; Liu and Lapata, 2019; Lewis et al., 2020),  $\sim 30\%$  of summaries have factual inconsistencies (Cao et al., 2018). With summarization being an integral component of information consumption, this highlights a need for ensuring summarization systems are factually consistent and developing methods for evaluating them.
Common evaluation metrics for summarization based on n-gram overlap - BLEU, ROUGE, and
METEOR (Papineni et al., 2002; Lin, 2004; Lavie and Agarwal, 2007) – are insufficient to measure the factual correctness of summaries and fail to correlate with the human judgements of factuality (Falke et al., 2019; Kryscinski et al., 2019). More recent metrics proposed to improve the evaluation of summarization factuality (Kryscinski et al., 2020; Durmus et al., 2020; Wang et al., 2020; Maynez et al., 2020) cannot be compared due to the lack of common benchmarks. More critically, while these approaches differ in the way they model factuality, they all consider factuality as a binary concept, labeling summaries of any length as factual or non-factual. They do not provide any fine-grained understanding of the factual errors made by different systems that could serve as an actionable feedback on a system's limitations.
The binary factuality of a text can be difficult to determine. Falke et al. (2019) show relatively low crowd-expert agreement, indicating the presence of subjectivity in the annotation process. Moreover, not all factual errors are equally important and the number of errors can have a significant impact on the perceived factuality of a text. This suggests that non-factuality should be modeled as a multidimensional construct and not a label.
In this work, we propose a linguistically motivated typology of factual errors for fine-grained analysis of factuality in summarization systems (§2). Our typology is theoretically grounded in frame semantics (Fillmore et al., 1976; Palmer et al., 2005) and linguistic discourse theory (Brown and Yule, 1983). It provides several benefits. First, we find that decomposing the concept of factuality in (relatively) well-defined and grounded categories makes the final binary decision more objective leading to near perfect agreement between crowd and expert annotators ( $\kappa = 0.86$ ). Second, this approach provides some measure of the degree of non-factuality both in terms of the quantity and the category of factual violations that appear
$^{1}$ Code, data, and online leaderboard will be available at https://github.com/artidoro/frank
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4812-4829 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: We propose a linguistically grounded typology of factual errors. We select crowd workers to annotate summaries from two datasets according to this typology achieving near perfect agreement with experts. We collect FRANK, the resulting dataset, to benchmark factuality metrics and state-of-art summarization systems.
in the text. This typology also provides us with the means to categorize the types of errors made by summarization systems, helping us gain deeper insights than simply categorizing content as factual or hallucinated.
We define an annotation protocol of factuality based on our typology and collect a dataset of human judgements over a diverse set of model generated summaries on the CNN/DM (Hermann et al., 2015) and XSum (Narayan et al., 2018) datasets  $(\S 3)$ . Through this dataset, we aim to both assess the factuality of summarization systems and benchmark recently proposed factuality metrics. In  $\S 4$  we discuss various state-of-art models and show a detailed analysis of the factual errors they make. Finally, in  $\S 5$  we evaluate multiple summarization metrics against our benchmark and show their strengths and weaknesses in detecting specific types of factual errors. Figure 1 shows an overview of this work.
2 Typology of Factual Errors
Previous studies of factuality in summarization only distinguish factual and hallucinated content (Kryscinski et al., 2019; Maynez et al., 2020) and provide limited insights on the fine-grained types of factual errors. In the simplest case, factual errors appear within a single proposition. However, as summaries include several sentences, discourse markers describe relations across propositions. These cross-sentence links, such as causality or temporal ordering, can introduce inconsistencies with the article. Furthermore, information in the summary should be verifiable given the article. This understanding outlines different levels of linguistic structure where factual mistakes can arise in summaries: at the semantic frame level, at the discourse level, or because the content cannot be verified. Below we define a typology of
factual errors further detailing these three levels. This typology is theoretically grounded in frame semantics (Fillmore et al., 1976; Baker et al., 1998; Palmer et al., 2005) and linguistic discourse analysis (Brown and Yule, 1983). Examples for each category are shown in Table 1.
2.1 Semantic Frame Errors
A semantic frame is a schematic representation of an event, relation, or state, which consists of a predicate and a list of participants, called frame elements (Baker et al., 1998). A semantic frame has both core and non-core frame elements (FE). Core frame elements are essential to the meaning of the frame, while non-core (e.g. location, time) provide additional descriptive information. Our first three categories capture factual errors in each of these components (frame, core and non-core FE) respectively.
Predicate Error (PredE): Category PredE encompasses errors where the predicate in a summary statement is inconsistent with the source text. More generally, this represents cases where the frame from a summary statement does not align with what is expressed in the source text.
Entity Error (EntE): Category EntE captures errors where the primary arguments (like entities) of the predicate are wrong or have the wrong attributes, although the relation was expressed in the original text. More generally, these account for cases where the core frame elements in a frame are wrong. This also captures directionality errors where the elements are interchanged (similar to agent-patient swap).
Circumstance Error (CircE): In additional to the core arguments, predicates can be further specified using additional information or attributes that describe the circumstance in which the arguments
Category	Description	Example
PredE	Relation Error	The predicate in the summary statement is inconsistent with the source article.	The Ebola vaccine was rejected by the FDA in 2019.
EntE	Entity Error	The primary arguments (or their attributes) of the predicate are wrong.	The COVID-19 vaccine was approved by the FDA in 2019.
CircE	Circumstance Error	The additional information (like location or time) specifying the circumstance around a predicate is wrong.	The first vaccine for Ebola was approved by the FDA in 2014.
CorefE	Coreference Error	A pronoun/reference with wrong or non-existing antecedent.	The first vaccine for Ebola was approved in 2019. They say a vaccine for COVID-19 is unlikely to be ready this year.
LinkE	Discourse Link Error	Error in how multiple statements are linked together in the discourse (for example temporal ordering/causal link).	To produce the vaccine, scientists have to show successful human trials, then sequence the DNA of the virus.
OutE	Out of Article Error	The statement contains information not present in the source article.	China has already started clinical trials of the COVID-19 vaccine.
GramE	Grammatical Error	The grammar of the sentence is so wrong that it becomes meaningless.	The Ebola vaccine accepted have already started.
Table 1: Typology of factual errors. Original text for the examples: The first vaccine for Ebola was approved by the FDA in 2019 in the US, five years after the initial outbreak in 2014. To produce the vaccine, scientists had to sequence the DNA of Ebola, then identify possible vaccines, and finally show successful clinical trials. Scientists say a vaccine for COVID-19 is unlikely to be ready this year, although clinical trials have already started.
and predicates interact (e.g. location, time, manner, direction, modality). Category CircE captures errors where one or more such attributes (non-core frame elements within a frame) are wrong.
2.2 Discourse Errors
The communicative intent of an author is also expressed through relations that hold between parts of the text. Factual errors in summarized text can often extend beyond a single semantic frame introducing erroneous links between discourse segments. Below we outline such categories of errors which are grounded in discourse analysis and rhetorical structure theory (RST) (Brown and Yule, 1983; Mann and Thompson, 1988). RST is an elaborate system for annotating coherence relations in discourse. Some examples of such relations include: "Elaboration", "Background", "Motivation", and "Volitional Cause". Here we depart from semantic frame terminology as its rooting in a single frame does not allow us to represent such errors.
Coreference Error (CorefE): Category CorefE accounts for errors where pronouns and other types of references to previously mentioned entities either are incorrect or have no clear antecedents, making them ambiguous.
Discourse Link Error (LinkE): Category LinkE encompasses errors involving a discourse link between different statements. These include errors of incorrect temporal ordering or incorrect
discourse links (e.g. RST relations, discourse connectors) between statements.
2.3 Content Verifiability Errors
Often statements in a summary cannot be verified against the source text due to difficulty in aligning them to the source. Below we outline two categories of errors for such cases.
Out of Article Error (OutE): Since summaries of a document should only contain information that can be deduced from the original text, we include a category for such errors OutE (prior work refers to this as extrinsic hallucinations (Maynez et al., 2020)).
Grammatical Error (GramE): We use GramE to categorize statements that are not well formed. When grammatical mistakes make the meaning of a statement incomprehensible or ambiguous, it cannot be verified against the source and is thus considered trivially wrong. Minor grammatical errors are acceptable.
Finally, for completeness in our annotation exercise, we add two additional categories Others (OthE) for factually errors that do not correspond to any of the above categories and Not an Error (NE) for statements that do not contain any errors.
Beyond theoretical grounding, we empirically verify our typology through large scale human annota
tions of five abstractive summarization models on the CNN/DM dataset and four on the XSum dataset. Through our dataset, we aim to have a broad coverage of different types of errors made by neural summarization systems, with human judgements on their fine-grained factuality errors.
Annotation Data For the annotation, we include model summaries from CNN/DM and XSum datasets as they present different characteristics. CNN/DM summaries are longer, with three sentences on average, while XSum has only single sentence summaries. Having longer summaries is crucial to identify discourse level errors. On the other hand, XSum summaries are more abstractive and include more factual errors on average (Maynez et al., 2020). For a diverse set of model summaries, we collect publicly available model outputs from different summarization models with differing factuality capabilities. For the CNN/DM dataset, we use model outputs from a LSTM Seq-to-Seq model (S2S) (Rush et al., 2015), a Pointer-Generator Network (PGN) model (See et al., 2017), a Bottom-Up Summarization (BUS) model (Gehrmann et al., 2018), a Bert based Extractive-Abstractive model (BertSum) (Liu and Lapata, 2019) and a jointly pretrained transformer based encoder-decoder model BART (Lewis et al., 2020). For the XSum dataset, we collect model outputs from a Topic-Aware CNN Model (Narayan et al., 2018), a Pointer-Generator Network (PGN) model, a randomly initialized (TransS2S) (Vaswani et al., 2017) and one initialized with Bert-Base (BertS2S) (Devlin et al., 2019). Details of the models used are provided in §A.1.
Annotation Collection Using the above model generated summaries, we collect human annotations from three independent annotators for 250 articles from each dataset (with a total of 1250 model outputs on CNN/DM and 1000 on XSum). We annotate each sentence of a summary to break the judgement of factuality into smaller units. We present sentences in the context of the entire summary to identify discourse errors spanning multiple sentences. Annotations are a two step process: for each sentence in the summary, the annotator first selects whether the sentence is factual, and if marked not factual, identifies the category of each
error based on our typology.  ${}^{3}$  A sentence can be annotated with more than one category of errors to account for multiple errors within a sentence. We conduct the annotation task on the Amazon Mechanical Turk (MTurk) platform. To achieve high quality crowd-sourced annotations, we build an intuitive interface  ${}^{4}$  which combines:
1. Clear Instructions: We explain the annotation scheme without assuming linguistic knowledge and give several examples for each category.
2. Training and Evaluation: We setup training tutorials for first time users to train and provide feedback on the task. We also setup a qualification test which tests their understanding of our annotation scheme and require annotators to obtain  $>85\%$  score to qualify. Further, we continuously evaluate annotators during the task against artificially generated factual errors to ensure continued high quality.
3. Fair Pay and Bonus: All workers are paid  $50\%$  more than the average American minimum wage. We offer bonuses for scores of  $60\%$  or above on the continuous evaluation, and for completing sets of 10 annotations.
Further details on our interface are added in  $\S A.6$
Inter-Annotator Agreement: We report inter-annotator agreement in terms of Fleiss Kappa  $\kappa$  (Fleiss, 1971). Following Durmus et al. (2020), we report the percentage  $p$  of annotators that agree with the majority class. Each datapoint in our dataset corresponds to a sentence in a summary. We compute agreement on all 4942 annotated sentences. On the annotation of whether a sentence is factual or not we obtain  $\kappa = 0.58$ , with  $p = 91\%$  of annotators agreeing with the majority class. As a comparison, Durmus et al. (2020) reports  $p = 76.7\%$  average agreement. When all three annotators agree that a sentence is not factual, we obtain  $\kappa = 0.39$  with  $p = 73.9\%$  of annotators agreeing with the majority class on the eight category annotation (seven categories of errors and "other") which indicate a moderate agreement.
Agreement with Domain Expert: We measure agreement between the majority class of the three
<sup>3</sup>We experimented with Likert scale evaluation of full summaries in a pilot study. Such an annotation would not provide precise information about where in the summary an error appears and also resulted in lower agreement. Hence, we opted for sentence level judgements.
4We make the interface available for future human annotations that follow our typology
As we use publicly available model outputs, the summaries across different datasets are from different models owing to their availability.
Figure 2: Proportion of summaries with factual errors based on collected annotations, with breakdown of the categories of errors within. Full specification of categories of errors in Table 1.
annotators and one expert annotator on 201 datapoints (10 summaries from CNN/DM and 10 summaries from XSum). We find a Kohen Kappa of  $\kappa = 0.86$  indicating nearly perfect agreement. Previous work found agreement of  $\kappa = 0.65$  between three crowd annotators and expert annotations of factuality (Falke et al., 2019). Even with more than nine workers, they report agreement with expert annotations of at most  $\kappa = 0.74$ . This improvement validates the robustness of our annotation interface and protocol which achieves higher agreement with fewer workers.
4 Summarization Model Analysis
We evaluate the performance of different summarization models in terms of factuality. Figure 2 visualizes the percentage of summaries with factual errors for each category model and dataset, with a breakdown of proportion of different error types within each. A summary is considered incorrect if it contains at least one sentence with a factual error. A sentence contains a factual error if the majority of annotators indicate the presence of an error (here we do not consider annotations where all three annotators disagree on the category).
How factual are generated summaries across different datasets? From our annotations, we observe that  $60\%$  of the summaries that were annotated contain at least one factual error. From Figure 2, we see that the XSum dataset has more factually incorrect model summaries  $(92\%)$  than CNN/DM  $(43\%)$ . It poses more significant challenges in terms of factuality as all models produce  $>80\%$  summaries with factual errors, with the best model (BertS2S) producing  $83\%$  wrong summaries. On the CNN/DM dataset, while state-of-the-art pretrained models like BERTSum and BART have better factuality numbers, the percentage of factually incorrect summaries is still high  $(23\%)$  for
BERTSum and  $27\%$  for BART). The proportion of errors across different categories vary widely between the two datasets. For the CNN/DM dataset, the most frequent classes of errors are Entity Error (EntE) and Coreference Error (CorefE). For the XSum dataset they are Out of Article Error (OutE) and Entity Error (EntE). Note that there are no discourse errors (CorefE, LinkE) in the XSum dataset because the data only contains single sentence summaries. Additionally, we observe that OthE makes up a very small percentage  $(\sim 1\%)$  of errors overall showing that our typology is complete with most errors being mapped to one of our existing categories.
How factual are generated summaries across different models? From Figure 2, we observe that LSTM based models like S2S and BUS generate many incorrect summaries. Interestingly, PGN on CNN/DM has fewer summaries with factual errors (26%) compared to S2S (74%) and BUS (62%) potentially due to the extractive nature of CNN/DM and the copy based objective in PGN. PGN has been previously shown to produce highly extractive summaries on CNN/DM copying large portions of text (often entire sentences) (Gehrmann et al., 2018; Balachandran et al., 2021). On the more abstractive dataset XSum, PGN produces  $>96\%$  factually incorrect summaries. We also observe that large-scale pretrained models improve factuality on both datasets, as also noted by Durmus et al. (2020), with more significant gains on CNN/DM. On CNN/DM, BERTSum and BART display half the error rate of BUS. In contrast, on XSum, BertS2S improves over non-pretrained models by  $\sim 10\%$  only, showing that XSum poses a significant challenge for factuality even in pretrained models.
Different models also exhibit different distributions in the error categories. LSTM based mod
els have higher proportion of Grammatical Errors (GramE) while transformer and CNN based models have a lower proportion. For pretrained transformer models, we observe that the improved error-rate on the CNN/DM dataset can be attributed to improvements at the frame level (PredE, EntE, CircE) while the discourse level errors still remain a challenge. Errors CorefE, LinkE account for a higher proportion of errors in BERTSum and BART compared to the other models.
5 Factuality Metric Evaluation
We propose the FRANK dataset resulting from the human annotation study as a common benchmark to assess different factuality metrics. We provide an evaluation protocol of factuality metrics, which controls for dataset biases, and a fine grained analysis of the strengths of each metric.
5.1 Benchmark
The FRANK benchmark provides a diverse dataset for evaluating various metrics on their ability to capture factual errors. Notably, our benchmark has factual error diversity, as it covers all types of errors described in the typology in §2, and data diversity as it combines 2250 summaries from different systems and datasets. Our annotations go beyond binary labels of factuality on a summary by providing fine-grained category annotations for every sentence. This allows us to determine how well each metric can capture each type of error. Furthermore, through averaging of sentence level judgements, we can also obtain a factuality scores (0 to 1 range) for a summary. To measure the degree that automated metrics capture a certain characteristic, we compute their correlation with human judgements and report Pearson correlation and Spearman rank correlation along with their p-values.
We evaluate different classes of metrics against the FRANK benchmark. We select four general summarization metrics. ROUGE, BLEU, and Meteor are n-gram based metrics and computed with respect to the reference summary. BERTScore (Zhang et al., 2020) computes BERT (Devlin et al., 2019) contextual embeddings on summary and source article and measures distances between matched embeddings. We select five metrics focused on factuality. As Goodrich et al. (2019), we use a simple OpenIE (Banko et al., 2007) baseline. This involves extracting OpenIE triples and matching them through sentence embeddings (Reimers
Figure 3: Correlation between metrics and human judgement on subsets of data. The  $x$  and  $y$  axis represent the human judgement the metric scores respectively. The red line is a linear regression fitted on full data. Each dotted line is a linear regression fitted on a model-dataset subset. Each colored point has coordinates equal to average factuality judgement, and metric score for its corresponding partition.
and Gurevych, 2019). FactCC (Kryscinski et al., 2020) and DAE (Goyal and Durrett, 2020) are entailment based metrics. FactCC operates with sentences as claims, while DAE uses dependency level entailment. FEQA (Durmus et al., 2020) and QAGS (Wang et al., 2020) are two question answering and generation metrics (QGA). More details on the differences between these metrics is in §A.2.
5.2 Controlling for Dataset Biases