SlowGuess/ACL_2020_OCR huggingface dataset:
Esin Durmus*
ed459@cornell.edu
hehe@cs.nyu.edu
Mona Diab
The George Washington
mtdiab@gwu.edu
Neural abstractive summarization models are prone to generate content inconsistent with the source document, i.e. unfaithful. Existing automatic metrics do not capture such mistakes effectively. We tackle the problem of evaluating faithfulness of a generated summary given its source document. We first collected human annotations of faithfulness for outputs from numerous models on two datasets. We find that current models exhibit a trade-off between abstractiveness and faithfulness: outputs with less word overlap with the source document are more likely to be unfaithful. Next, we propose an automatic question answering (QA) based metric for faithfulness, FEQA, $^{1}$  which leverages recent advances in reading comprehension. Given question-answer pairs generated from the summary, a QA model extracts answers from the document; non-matched answers indicate unfaithful information in the summary. Among metrics based on word overlap, embedding similarity, and learned language understanding models, our QA-based metric has significantly higher correlation with human faithfulness scores, especially on highly abstractive summaries.
Abstractive summarization models must aggregate salient content from the source document(s) and remain faithful, i.e. being factually consistent with information in the source documents. Neural abstractive models are effective at identifying salient content and producing fluent summaries (See et al., 2017; Chen and Bansal, 2018; Gehrmann et al., 2018). However, the generated summary may not always contain faithful information, which is vital for real-world applications.
Source. The world's oldest person has died a few weeks after celebrating her 117th birthday. Born on March 5, 1898, the great-grandmother had lived through two world wars, the invention of the television and the first successful powered aeroplane flight by the wright brothers...
Output sentence. The world's oldest person has died on March 5, 1898.
Table 1: An example of unfaithful output (highlighted in red); generated by Gehrmann et al. (2018).
Table 1 shows an example of unfaithful generation. Recent studies have shown that around  $30\%$  of generated summaries contain unfaithful information (Cao et al., 2018; Falke et al., 2019a; Krysciński et al., 2019), especially when the sentence combines content from multiple source sentences (Lebanoff et al., 2019).
In this paper, we address the problem of evaluating faithfulness of generated summaries given their source documents. Our key insight is that current models are limited by a trade-off between abstractiveness and faithfulness (Section 2). On a wide range of systems and two datasets with varying levels of abstractiveness (CNN/DM and XSum), we show that the number of unfaithful sentences (annotated by humans) increases as the summary becomes more abstractive (i.e. less overlap with the source document). Next, we investigate a diverse set of existing automatic evaluation metrics such as ROUGE, BERTScore (Zhang et al., 2019a), and learned entailment models. We find that their correlations with human scores of faithfulness drop significantly on highly abstractive summaries, where deeper text understanding beyond surface similarity is needed.
Recently, question answering (QA) based automatic metrics have been proposed for evaluating
*Most of the work is done while the authors were at Amazon Web Services AI.
Faithfulness Evaluation with Question Answering.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055-5070 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
content selection in summarization (Eyal et al., 2019; Scialom et al., 2019; Chen et al., 2018). Specifically, cloze-style QA is used to evaluate whether important information in the source is recovered from the summary. Inspired by prior work, we use automatically generated QA pairs to represent information in the summary and validate it against the source. Concretely, we generate a set of "groundtruth" QA pairs from the summary, using a learned model that converts a declarative sentence and an answer span to a question (Section 3). Then, off-the-shelf reading comprehension models are evaluated on this set by extracting answer spans from the source documents. High accuracy means that the summary and the source document tend to produce the same answers, thus they are factually consistent with respect to the questions. Compared to prior approaches using cloze tests, our question generation approach enables evaluation with a broader range of QA models and answer types (e.g. extractive and generative), thus maximally taking advantage of progress in QA.
Among automatic metrics based on  $n$ -gram overlap, word embeddings, and language understanding models (relation extraction and entailment), FEQA has significantly higher correlation with human scores of faithfulness and is the only metric that correlates with human scores on highly abstractive summaries from XSum.
2 The Abstractiveness-Faithfulness Tradeoff
While extractive summarizers are largely faithful (since they copy sentences from the source document), current abstractive models struggle to produce faithful summaries without copying. Similar to Lebanoff et al. (2019), we observe that factual errors occur more frequently as models generate more abstractive summary sentences, i.e. less overlap with the source document. In this section, we analyze generated summaries along two dimensions: abstractiveness and faithfulness. Specifically, we aim to answer the following questions: (1) How to quantify abstractiveness of a summary? (2) Is abstractiveness encouraged more by the data or the model? (3) How does being abstractive affect faithfulness?
2.1 Characterizing Abstractiveness of a Summary
Abstractive summarization involves rephrasing important content into brief statements, ranging from minor editing of a source sentence to condensing multiple sentences in new words. Given a source document and a summary, we want to measure the level of abstractiveness of the summary.
Prior work measures abstractiveness by overlapped text spans between the summary and the document (Grusky et al., 2018; Zhang et al., 2018), or indirectly by the effectiveness of extractive baselines such as LEAD-3 (Nallapati et al., 2016a). While metrics such as extractive fragment coverage and density (Grusky et al., 2018) provide a continuous measure of the level of abstractiveness, we define a more fine-grained categorization of abstractiveness by analyzing how each sentence in the summary is formed.
A more abstractive summary sentence aggregates content over a larger chunk of source text; consequently it must copy fewer words to maintain brevity. Therefore, we define the following abstractiveness types based on the amount of copying, e.g. copying a source sentence, one or more partial fragments from the source sentence, and individual words.
1. Sentence extraction: the summary sentence is exactly the same as one of the source sentences.
2. Span extraction: the summary sentence is a substring of one of the source sentences, e.g. "the plane was coming back from the NCAA final" is a span extracted from "the plane was coming back from the NCAA final, according to spokesman John Twork".
3. Word extraction: the summary sentence is formed by a subset of the tokens in a source sentence, e.g. "Capybara Joejoe has almost 60,000 followers" is a result of deleting words in "Capybara Joejoe who lives in Las Vegas has almost 60,000 followers on Instagram".
4. Perfect fusion $_k$ : the summary sentence is constructed by piecing together the substrings from  $k$  ( $k > 1$ ) source sentences in their original order, e.g. "Capybara Joejoe has almost 60,000 followers" is a perfect fusion of the sentences "Capybara Joejoe lives
To quantify the amount of abstractiveness of a set of summaries, we label each sentence with the first qualified type in the order above if it fits to one of these categories.
We then define the score of each type as the percentage of sentences labeled by that category. The types are ordered by increasing levels of abstractiveness. For example, a summary with higher fusion scores and lower extraction scores is considered more abstractive. In addition, we compute the percentage of novel  $n$ -grams that do not appear in the source document as another metric for abstractiveness.
2.2 Is abstractiveness from the model or the data?
Equipped with the metrics for abstractiveness above, we want to further understand how abstractive the generated summaries are, and whether the amount of abstractiveness is a result of the training data or the model. Therefore, we compute abstractiveness scores for both the reference summaries and summaries generated from a diverse set of models on two datasets.
Datasets. We use the CNN/DailyMail (Hermann et al., 2015; Nallapati et al., 2016b) (CNN/DM) and the XSum (Narayan et al., 2018) datasets, which are both used for single-document news summarization tasks. CNN/DM consists of articles from the CNN and Daily Mail websites, where the summaries comprise highlights in bullet points. XSum consists of BBC articles, where the summaries comprise a single-sentence summary that is written as the opening introductory sentence for the article. XSum was released in particular to promote research on highly abstractive summarization systems. Appendix A provides statistics on CNN/DM and XSum datasets: they contain around 288k and 204k training examples, respectively; CNN/DM includes longer documents and summaries on average.
Models. Most neural abstractive summarization models are based on sequence-to-sequence models. They differ in how summarization-specific operations such as copying/extraction are instantiated. We consider 5 prominent models and sum
Systems	Extractor	Encoder	Decoder
PGC	-	LSTM	LSTM+copy
FASTRL	sentences	LSTM	LSTM+copy
BOTTOMUP	words	LSTM	LSTM+copy
TCONV	-	CNN+topic	CNN
BERTSUM	-	BERT-based	Transformer
Table 2: Comparison of summarization systems in terms of model architecture.
marize their characteristics in Table 2. Details of each model can be found in Appendix B. PGC (See et al., 2017) uses the copy mechanism during decoding to allow extraction. FASTRL (Chen and Bansal, 2018) and BOTTOMUP (Gehrmann et al., 2018) decouple extraction and abstractive generation by learning to select sentences and words respectively in the first step; this model has been shown to generate more abstractive summaries compared to PGC. TCONV (Narayan et al., 2018) is initially designed for XSum, thus it does not include any explicit copying/extraction components and focuses on long text representation using convolutional neural networks. BERTSUM (Liu and Lapata, 2019) consists of a BERT-based encoder and a 6-layer Transformer decoder. It incorporates extraction implicitly by first fine-tuning the encoder on the extractive summarization task.
Results. Our goal is to understand the level of abstractiveness of summaries generated by different models, and the influence on abstractiveness from the training data. Therefore, we analyzed summaries generated by the above models on CNN/DM and XSum. We computed the metrics described in Section 2.1 for both the generated summaries and the reference summaries on the test sets. The results are shown in Table 3.
First, CNN/DM is more extractive than XSum. Extraction scores of the reference summaries in CNN/DM shows that almost half of the sentences are formed by deleting words in one of the source sentences. This shows that sentence compression (Knight and Marcu, 2002) is the main technique used for this dataset. In contrast, none of the summary sentences in XSum are formed by copying from a single source sentence. They are generated mostly by paraphrasing the input content, indicated by the large fraction of novel  $n$ -grams.
in Las Vegas." and "He has almost 60,000 followers on Instagram."
2We use state-of-the-art models proposed for each dataset at the time of writing.
3We use the BERTSUMEXTABS variation.
Dataset	Model	Extraction	Perfect fusion	Novel n-grams
Sentence	Span	Word	k=2	k≥2	n=1	n=2	n=3
CNN/DM	Ref	1.39	2.14	9.27	12.92	14.87	12.40	51.03	71.22
PGC	35.45	34.18	15.45	10.90	1.61	0.62	3.33	7.42
FASTRL	8.94	40.06	39.64	4.22	0.84	0.82	10.89	20.74
BOTTOMUP	7.65	17.98	36.75	21.86	6.77	0.86	11.44	22.40
BERTSUM	-	13.73	53.40	16.18	4.39	5.23	14.55	23.09
XSum	Ref	-	-	-	0.87	0.77	39.20	84.98	96.05
PGC	-	-	-	0.41	3.47	30.08	74.27	91.27
TCONV	-	-	-	0.35	2.31	34.07	80.62	95.12
BERTSUM	-	-	-	0.33	3.15	28.93	75.85	91.41
Table 3: Abstractiveness measures of the models on CNN/DM and XSum datasets. The numbers for Extraction and Perfect fusion indicate % of sentences generated with these strategies. Numbers for novel  $n$ -grams indicate % of  $n$ -grams that are present in the output sentence but is not present in the source.
Second, training data has a larger influence on the abstractiveness of model outputs. Similar to Zhang et al. (2018), we find that models trained on CNN/DM are near-extractive. However, the same models trained on XSum are significantly more abstractive. In fact, none of the models produced any sentence that copies words/phrases from a single source sentence, which is consistent with characteristics of the reference summaries in XSum. The content is more often rephrased in novel words/phrases. However, on both datasets, current models struggle to achieve the same level of abstractiveness as the reference summaries, indicating that additional inductive bias is needed to condense multiple sentences by rephrasing.
Third, different models have different ways of doing extraction. When trained on CNN/DM, PGC generates the majority of sentences by copying complete source sentences, whereas FASTRL, BOTTOMUP and BERTSUM do simple compression by deletion more often. In addition, BOTTOMUP does more fusion compared to PGC, FASTRL and BERTSUM.
2.3 Annotating Summary Faithfulness<sup>4</sup>
To understand faithfulness of current systems and its relation to abstractiveness, we crowd-sourced human annotations on the output of each model-dataset pair described in Section 2.2. Since a near-extractive sentence is very likely to be grammatical and faithful, we focus on more abstractive cases by excluding output sentences that are either an exact copy or a substring of one of the source sentences.
A key challenge to reliable human annotation is that the inter-annotator agreement on faithfulness is relatively low (Lebanoff et al., 2019). Our pi
lot study shows that workers often do not agree on incoherent sentences, e.g. whether "Chelsea beat Chelsea  $5 - 3$  in the Premier League on Saturday." is faithful or not. To standardize the annotation process, we design hierarchical questions to distinguish among failed generation that render a sentence meaningless, low-level grammatical errors that hardly affect semantic understanding, and faithfulness errors that convey incorrect (yet meaningful) information.
Figure 1 shows the decision tree of our human annotation steps. We first evaluate the grammaticality of generated sentences (independent from the source document). We show annotators a summary sentence and ask them to choose whether the given sentence is meaningful or nonsensical to determine if the given sentence is structurally and semantically sound. If the annotator can make sense of the sentence, we then ask whether it is grammatical or has minor grammaticality problems which a person can easily correct.
Next, for sentences labeled as meaningful in the first step, we ask workers whether they are faithful to the provided source document. In case the worker labels a sentence as unfaithful, we conduct a simple error analysis by asking them to indicate if the sentence contains information that is absent from or conflicting with the source document, which corresponds to hallucination and contradiction errors, respectively. More details about the annotation schema and guidelines are included in the Appendix C. Next, we describe our human evaluation results.
2.3.1 Human Annotation Results
For each dataset-model pair described in Section 2.2, we randomly sampled 1000 sentence-source pairs eliminating output sentences that are either an exact copy or substring of a source sen
<sup>4</sup>We make our data and code available for reproducibility at: https://github.com/esdurmus/summary-faithfulness.
Figure 1: The decision diagram of our human annotation process. Decision nodes are rectangular and outcome nodes are circular. We show the annotation path of two summary sentences, S1 (green arrows) and S2 (red arrows). S2 is annotated as nonsensical thus is not considered for faithfulness. S1 is annotated as unfaithful due to hallucinated content.
Dataset	Model	Grammaticality	Faithfulness
Score	Agreement	Abstractiveness	Score	Agreement	Abstractiveness
CNN/DM	PGC	93.34	94.04	10.05	70.05	77.28	13.35
FASTRL	83.06	88.05	44.46	68.27	77.45	49.74
BOTTOMUP	85.83	89.19	29.62	64.17	76.04	42.36
BERTSUM	97.53	97.65	29.44	95.03	95.14	39.16
XSum	PGC	65.85	81.03	91.10	40.33	71.63	97.06
TCONV	70.85	85.03	94.94	38.96	69.90	98.81
BERTSUM	90.44	91.80	91.50	60.54	70.00	97.60
Table 4: Grammaticality and faithfulness results of human annotations. Score is computed by taking the percentage of annotators that selected "meaningful" and "faithful" for grammaticality and faithfulness annotation tasks, respectively, and then averaging these values across all the examples for the given annotation task. Agreement is computed by taking the percentage of the workers that annotate the majority class for the given example. Abstractiveness is measured by the percentage of novel trigrams in a given sentence.
tence. We collected grammaticality annotations for these sentences from 5 annotators. We consider a sentence meaningful if at least 4 out of 5 annotators label it as meaningful in the first stage. We sampled 200 meaningful sentences randomly to collect annotations for faithfulness. Table 4 shows the results of the grammaticality and faithfulness human evaluations.
Grammaticality. Overall, outputs from all models are scored high on grammaticality with high inter-annotator agreement. However, on more abstractive summaries (i.e. when trained on XSum), the grammaticality scores drop significantly. One exception is BERTSUM, which maintains good performance on XSum and achieves the highest grammaticality score on both datasets.
Faithfulness. Near-extractive summaries generated from models trained on CNN/DM have significantly higher faithfulness scores than highly
abstractive summaries from models trained on XSum. We find that PGC and TCONV has faithfulness errors in more than half of the sentences they generate when trained on XSum. Although BERTSUM generates fewer unfaithful sentences, it still suffers from performance drop on XSum. Interestingly, human agreement on faithfulness is also lower for abstractive summaries from XSum. This suggests that faithfulness errors are harder to catch for humans as well in more abstractive settings. We further observe conflicting information is more common among models trained on CNN/DM while hallucination is more common among models trained on XSum. Table 5 shows examples of meaningful but unfaithful sentences.
3 FEQA: Faithfulness Evaluation with Question Answering
Our analysis above shows that the number of unfaithful sentences increases significantly as more abstractive summaries are generated. Thus the key challenge to faithfulness evaluation is to verify highly abstractive sentences against the source document, where surface similarity match-
${}^{5}$  Majority of the sentences  $\left( { > {70}\% }\right)$  identified as "meaningful" are annotated as "perfectly grammatical" for each model-dataset pair.
Source	Output Sentence	Domain	Category
...However, Winger Ross Wallace (knee) and right-back Steven Reid (calf) could return for the Barclays pre-mier league contest...	Dean Marney and Steven Reid could return for the Bar-clays Premier League match.	CNN/DM	IC
...Odom also played for the US in the 2004 Athens Olympics, winning the bronze medal. His condition is un-known but well-wishers tweeted their support following the news...	NBA basketball player Odom has been found dead in a helicopter crash in the US state of Nevada.	XSum	H
Table 5: Examples of meaningful but unfaithful sentences. Category corresponds to the faithfulness error type for the output sentence. IC: Incorrect Concatenation, H: Hallucination. More examples are provided in Table 11.
Figure 2: Overview of FEQA. Given a summary sentence and its corresponding source document, we first mask important text spans (e.g. noun phrases, entities) in the summary. Then, we consider each span as the "gold" answer and generate its corresponding question using a learned model. Lastly, a QA model finds answers to these questions in the documents; its performance (e.g. F1 score) against the "gold" answers from the summary is taken as the faithfulness score.
ing would fail. If we have a good semantic representation of the sentence abstracting away its surface form (e.g. a list of facts about who did what to whom), we can simply compare the sentence representation to the document representation (e.g. check whether the fact list from the summary is a subset of the list from the document). Ideally, the representation should be domain-general and interpretable for easy error analysis.
Motivated by the fast progress in reading comprehension (Chen, 2018; Gao et al., 2018) we propose to use QA pairs as a generic meaning representation of sentences for faithfulness evaluation. Given a summary sentence, we produce a list of questions asking about key information in the sentence and their corresponding answers. To verify this information against the source, we use a QA model to predict answers from the document. The questions and the QA model thus extract comparable information from two pieces of text. More matched answers from the document implies a more faithful summary since the information addressing these questions are consistent between the summary and the source document.
Figure 2 shows the workflow of FEQA.
Question generation. Prior work (Eyal et al., 2019; Scialom et al., 2019) uses cloze tests as questions by masking entities. To go beyond cloze-style QA and leverage more recent extractive (Rajpurkar et al., 2016) or even generative (Alec et al., 2019) QA models, we generate natural language questions from the summary sentence automatically. Specifically, we mask important text spans in a sentence, including noun phrases extracted by a constituency parser (Kitaev and Klein, 2018) and named entities extracted by the Stanford CoreNLP NER model (Finkel et al., 2005; Manning et al., 2014). We consider each span as the gold answer and generate its corresponding question by fine-tuning a pretrained BART language model (Lewis et al., 2019). To train the question generator, we adapt the QA2D dataset Demszky et al. (2018). The input is a declarative sentence with masked answers and the output is a question. A training example might look like:
Input: Sally was born in  $<\mathrm{m}> 1958 </\mathrm{m}>$
Output: When was Sally born?
Since the transformation from declarative sen
tences to questions is almost rule-based without much paraphrasing, we expect the model to generalize to various domains.
Answer verification. Given the QA pairs generated from a summary sentence, we run off-the-shelf QA models to get answers to these questions from the source document. We then measure the average F1 score against the "gold" answers from the summary, which is our faithfulness score for the given sentence. This step does not have any constraint on the QA model. We experiment with the pretrained BERT-base model (Devlin et al., 2019) fine-tuned on SQuAD-1.1 (Rajpurkar et al., 2016) and SQuAD-2.0 (Rajpurkar et al., 2018). Note that in the case of SQuAD-2.0, the model may be able to hypothesize that a question is unanswerable. This case is equivalent to getting an answer incorrect (i.e. unfaithful).