SlowGuess/NAACL_2022_OCR huggingface dataset:
Hwanhee Lee $^{1*}$ , Kang Min Yoo $^{2}$ , Joonsuk Park $^{2,3}$ , Hwaran Lee $^{2\dagger}$  and Kyomin Jung $^{1\dagger}$ \
 $^{1}$ Seoul National University,  $^{2}$ NAVER AI Lab,  $^{3}$ University of Richmond
{wanted1007,kjung}@snu.ac.kr
{kangmin.yoo, hwaran.lee}@navercorp.com
Despite the recent advances in abstractive summarization systems, it is still difficult to determine whether a generated summary is factual consistent with the source text. To this end, the latest approach is to train a factual consistency classifier on factually consistent and inconsistent summaries. Luckily, the former is readily available as reference summaries in existing summarization datasets. However, generating the latter remains a challenge, as they need to be factually inconsistent, yet closely relevant to the source text to be effective. In this paper, we propose to generate factually inconsistent summaries using source texts and reference summaries with key information masked. Experiments on seven benchmark datasets demonstrate that factual consistency classifiers trained on summaries generated using our method generally outperform existing models and show a competitive correlation with human judgments. We also analyze the characteristics of the summaries generated using our method. We will release the pre-trained model and the code at https://github.com/hwanheelee1993/MFMA.
As textual content available on- and offline explodes, automated text summarization is becoming increasingly crucial (El-Kassas et al., 2020); with the advances in neural text generation methods, abstractive summarization systems that generate paraphrases are quickly replacing extractive ones that simply select essential sentences from the source text (Nallapati et al., 2017). While abstractive summaries can be more coherent and informative (given the same length) than their extractive counterparts, they frequently contain information inconsistent with the source text. This is a critical
Article: Guus Hiddink, the Russia and Chelsea coach, has had much to smile about in his 22-year managerial career. ..., Enjoying success around the world - at different levels with different players in different cultures - has made Guus Hiddink one of the most admired bosses around. ..., Hiddink's resume includes stints in other high-pressure jobs such as Fenerbahce, Valencia and Real Madrid. ..., But the straight-speaking Dutchman is loyal to the project he has in charge of the Russian national side and insists he will leave Chelsea at the end of the season regardless.
Reference Summary: Born in 1946, Hiddink has become one of the best managers in the world. Dutchman has enjoyed huge success at club and international level. He's currently coach of Russia and is in charge of Chelsea until end of the season.
Mask-and-fill Summary Without Article:
Born in 1946, Dutchman has become one of the most respected politicians in the world. Dutchman is enjoyed success at the Olympics and World Cup. He's currently the President of Russia and is in charge of the country until the end of the season.
Mask-and-fill Summary With Masked Article: Born in 1946, Hiddink has become one of the most admired managers in the world. Dutchman has enjoyed successful spells at Chelsea and Real Madrid. He's currently manager of Russia and is in charge of the country until the end of the season.
Figure 1: An example of generated negative summary using masked article. Spans that are highlighted are masked when generating the negative summary. Note that red spans are factually inconsistent with the given article and blue spans are factually consistent.
issue, as it directly affects the reliability of the generated summaries. (Cao et al., 2018; Zhao et al., 2020; Maynez et al., 2020).
Unfortunately, existing approaches to identify such factual inconsistency without constructing new resources have not been satisfactory. Directly measured similarity between the summary and its source text—using popular n-gram similarity metrics such as ROUGE (Lin, 2004) and BLEU (Papineni et al., 2002)—exhibits low correlation with
*Work done during an internship at Naver AI Lab.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1019 - 1030
human judgments for factual consistency. Also, leveraging related tasks—such as natural language inference (NLI) (Bowman et al., 2015) and fact verification (Thorne et al., 2018)—is not ideal. This is because these tasks aim to identify relations between two sentences, whereas factual consistency checking involves a multi-sentence summary and an even longer source text (Bora-Kathariya and Haribhakta, 2018; Falke et al., 2019).
A remaining solution is to train a factual consistency classifier with a dataset specifically constructed for this purpose. Note that positive summaries are readily available. That is, the reference summaries from existing text summarization datasets can be assumed to be factually consistent with the respective source texts. Thus, the main challenge is in generating effective negative summaries, i.e., summaries that are factually inconsistent with the source text. Recent works generate negative summaries by simply replacing keywords in the reference summaries or sentences extracted from the source texts (Kryscinski et al., 2020; Yin et al., 2021). This, however, results in negative summaries that significantly diverge from the source texts and positive summaries, which is not ideal for training factual consistency classifiers. For instance, Figure 1 shows that coach in the reference summary is changed to President of Russia, which is an inconsistency that is too obvious.
In this paper, we propose a novel method, Masked-and-Fill with Masked Article (MFMA), where parts of the source text and reference summary are masked and later inferred to generate a plausible but factually inconsistent summary. Experiments on seven benchmark datasets demonstrate that factual consistency classifiers trained on negative summaries generated with our method mostly outperform existing models and show a competitive correlation with human judgment. We also analyze the characteristics of the negative summaries generated. Our main contributions are as follows:
- We propose a novel negative summary generation method for training factual consistency classifiers for abstractive summaries.
- We show the efficacy of our method on seven benchmark datasets using classification performance and correlation with human judgment.
- We analyze the characteristics, such as affinity and diversity, of the negative summaries generated using our method.
2.1 Factual Inconsistency in Summarization Systems
Previous works (Maynez et al., 2020; Zhao et al., 2020; Cao et al., 2018) have studied the factual inconsistency in abstractive summarization systems. Especially, (Cao et al., 2018) demonstrates that  $30\%$  of the model generated summaries have at least one factual error, and this obstacle the practical usage. (Maynez et al., 2020) specifies these factual errors in the abstractive summarization system into two types: intrinsic errors and extrinsic errors. Intrinsic errors occur using the contents present in the source article like "Switzerland" and "England" in the negative summary example in Figure 2. On the other hand, extrinsic errors are the errors generated by ignoring the source article when generating summaries. "in the second half" in Figure 2, which is not included in the source article, is an example of extrinsic errors.
In this work, we propose a system for detecting these various factual errors that are necessary for developing a summarization system. We propose a unified method for intentionally modeling both types of errors to build a dataset for training this system.
2.2 Measuring Factual Consistency
As a better way to evaluate the factual consistency, recent works such as QAGS (Wang et al., 2020) and QuestEval (Scialom et al., 2021) adopt question generation and question answering frameworks to evaluate the factual consistency. Both methods firstly generate questions using entities or noun phrases in the candidate summary and then compare the answers of these questions between the source and the summary. Although these methods do not require any reference summaries, they have a higher correlation with human judgments than previous metrics in consistency checking. Also, the generated questions and their answers are easily interpretable. But due to their complicated structure, the computational complexity of these methods is relatively heavy and the errors in each component can be cascaded.
Following the idea that all of the contents in the summaries should be entailed by source document, models from the related tasks such as Natural Language Inference(NLI) (Bowman et al., 2015; Williams et al., 2018; Falke et al., 2019) are also used to verify the factual consistency of the sum
Figure 2: Overall flow of our proposed negative summary generation method Mask-and-Fill-with-Masked Article.
maries. These approaches are simpler and more intuitive than QA-based metrics. But the data pairs in these datasets are usually composed of single sentences, and this makes it difficult to be directly used for factual consistency checking in summarization where the task requires multi-sentence level reasoning. For this reason, two recent studies FactCC (Kryscinski et al., 2020) and DocNLI (Yin et al., 2021) have studied ways to make synthetic datasets for training factual consistency checking model. Both works create synthetic negative summaries using the pre-defined rules such as entity substitution or mask-and-fill. In this paper, we propose a more general negative summary generation method additionally using the masked source.
CoCo (Xie et al., 2021) compares the likelihood of the generated summaries using the original source and the masked source to estimate the counterfactual samples. Different from CoCo, our work directly augments the negative summaries and train the classifier using them.
For a given article  $A$  and a summary  $S$ , we aim to develop a factual consistency checking system that can evaluate whether  $S$  is factual consistent with  $A$ . In other words, the system is required to discriminate a factual consistent summary  $S_{C}$  with the factual inconsistent summary  $S_{I}$  that consists of at least one factual error. We consider this problem as a classification task between  $S_{C}$  and  $S_{I}$ . However, large-scale human-annotated training datasets for this task have not been constructed yet, especially for the inconsistent summaries  $S_{I}$ .
In this paper, we focus on effective augmentation
methods of the inconsistent summaries. In order for that, there are two crucial conditions: 1) guarantee of inconsistency; the generated summaries should be indeed inconsistent with the source article, 2) relevance to the source article; the generated summaries should include contents related to the article. These two factors are in trade-off relations, which means that when the generated summaries are strongly inconsistent they might not be related to the article and vice versa. Therefore appropriate negative summary augmentation is required to improve the factual consistency classifier.
To generate confusing and hard negative summaries, we propose a summary generation using a masked article and a masked reference summary where some salient information is hidden. By doing so, we let the summarizer model infer hidden information through the masked article to generate plausible negative summaries. Note that, previous works such as FactCC and DocNLI generate negative summaries  $S_{I}$  by changing positive summaries  $S_{C}$  through entity replacements or mask-and-fill methods without referring to the source article. We observe that previous methods can easily guarantee negativeness, but they often generate summaries that are very irrelevant to the source article or unnatural as shown in Figure 1.
3.1 Mask-and-Fill with Masked Article
To model inconsistent summaries but related to the article, we propose a method, Mask-and-Fill with Masked Article (MFMA), which generates negative summaries with masked articles and masked reference summaries, as shown in Figure 2.
Specifically, we assumed noun phrases and entities in the articles are salient information, and mask
them with the ratio of  $\gamma_{A}$ , resulting in masked article  $\overline{A}_{\gamma_A}$ . Similarly, we also mask the salient spans in the positive summary, i.e., reference summary, with the ratio of  $\gamma_{S}$  to form a masked summary  $\overline{S}_{\gamma_S}$ . Then, we concatenate  $\overline{A}_{\gamma_A}$  and  $\overline{S}_{\gamma_S}$  by prepending prefix token for each input text (i.e., "Summary:  $\overline{S}_{\gamma_S}$ , Article:  $\overline{A}_{\gamma_A}$ ) as shown in Figure 2. Next, we train a summarizer based on an encoder-decoder model, BART (Lewis et al., 2020), to reconstruct the original summary  $S$  with the following loss:
$$
\mathcal {L} = \sum_ {t} - \log P (S _ {t} | S _ {<   t}, [ \overline {{S}} _ {\gamma_ {S}}; \overline {{A}} _ {\gamma_ {A}} ]). \qquad (1)
$$
After training, we generate negative summaries of unseen and masked article-summary pairs through inference. Obviously, if the mask ratio is high enough, the model is hard to correctly fill the masked contents from the erased article and reference summary. However, we assume the trained reconstruction model is able to fill the masks with plausible contents by inferring the related contents with the masked article.
3.2 Masked Summarization
As a variant of MFMA, we also study another negative summary generation model, Masked SuMmarization(MSM). The model aims to generate summaries using masked articles  $\overline{A}_{\gamma_A}$  but without masked reference summaries as follows:
$$
\mathcal {L} = \sum_ {t} - \log P (S _ {t} | S _ {<   t}, \bar {A} _ {\gamma_ {A}}). \tag {2}
$$
The MSM model is trained to generate the entire summaries without the information guidance of masked reference summaries, so MSM has merits in generating more diverse summaries than MFMA.
3.3 Training Factual Consistency Checking Model
Finally, for the factual consistency checking model, we train a binary classifier of consistent summaries and inconsistent generated summaries. The pair of summary and the corresponding article are concatenated and then fed into the classification model as an input. We fine-tuned the pre-trained ELECTRA (Clark et al., 2019) by adding a classifier head with binary cross-entropy loss.
Negative Summary Generation We randomly split the training set of CNN/DM dataset (Nallapati et al., 2016) in half and use half for training negative summarizer and the other half for generating negative summary after training. We use spaCy for finding entities and noun phrases in both summaries and articles. We train bart-base<sup>1</sup> for five epochs to train MFMA, and use bart-base model without fine-tuning for MF. We use t5-small (Raffel et al., 2020)<sup>2</sup> for MSM, which shows better results than bart-base for this task. We attach the further details in Appendix.
Training Classifier We train google/electra-base-discriminator<sup>3</sup> for five epochs with learning rate 2e-5, batch size of 96 using adam optimizer (Kingma and Ba, 2015) with the dataset we generate using MF, MFMA and MSM. For DocNLI and FactCC, we get the original training dataset that each author release, and we train a model with the same setting as our method except for the training datasets for a fair comparison. We choose model using the balanced accuracy on validation set of FactCC (Kryscinski et al., 2020) which consists of 1k human annotated summaries.
For evaluating the performance of factual consistency checking system, it is necessary to compare the human judgments of the consistency for the summary with the system. And these human judgments exist in two forms, binary level (consistent, inconsistent) or numerical levels such as likert scale. In general, in the case of binary level data, performance is measured through accuracy with human judgments. For the case of numerical levels, correlation with human judgments is measured. In addition to using the results for the existing benchmark dataset in this way, we also report the accuracy by casting these numerical level datasets to the binary level dataset since we develop classifier based system. We report the results on the following datasets.
FC-Test (Kryscinski et al., 2020) release a human-annotated factual consistency for the model generated summaries for CNN/DM Dataset in
<sup>1</sup>https://huggingface.co/facebook/bart-base
2https://huggingface.co/t5-small
<sup>3</sup>https://huggingface.co/google/electra-based-discriminator
Table 1: Macro F1-score(F1) and class-balanced accuracy(BA) of the human annotated factual consistency for the benchmark datasets based on CNN/DM.
Dataset	FactCC-Test	SummEval	QAGS-CNN/DM	FRANK-CNN/DM	Average
Metric	F1	BA	F1	BA	F1	BA	F1	BA	F1	BA
Baselines
FactCC	71.0	71.3	65.1	68.2	69.3	69.6	64.1	63.9	67.4	68.2
DocNLI	67.2	71.0	71.5	71.3	62.4	66.2	66.0	66.0	66.8	68.6
MNLI	55.0	56.0	51.7	51.7	48.6	53.4	50.4	53.3	51.4	53.6
FEVER	57.9	56.2	52.6	53.6	39.4	53.3	49.8	55.6	49.9	54.7
MF	59.9	64.1	68.2	67.5	47.6	56.9	62.4	62.7	59.5	62.8
Ours
MFMA	79.7	84.5	71.3	69.6	70.5	72.3	69.5	69.2	72.8	73.9
MSM	70.6	72.7	66.8	68.2	67.6	68.7	69.6	69.3	68.6	69.7
Table 2: Macro F1-score(F1) and class-balanced accuracy(BA) of the human annotated factual consistency for the benchmark datasets based on XSum.
Dataset	XSumHall	QAGS-XSum	FRANK-XSum	Average
Metric	F1	BA	F1	BA	F1	BA	F1	BA
Baselines
FactCC	52.1	61.8	63.6	63.7	50.7	58.0	55.5	61.2
DocNLI	55.1	56.4	65.3	66.0	60.3	63.4	60.2	61.9
MNLI	33.3	52.1	45.2	51.1	28.8	50.6	35.8	51.3
FEVER	53.1	55.5	62.2	63.7	54.9	63.5	56.7	60.9
MF	53.6	53.3	54.6	54.9	55.7	55.3	54.6	54.5
Ours
MFMA	55.5	56.0	66.6	67.0	59.6	59.6	60.6	60.9
MSM	52.6	53.9	50.8	55.5	50.8	51.3	51.4	53.6
binary-level to test the performance of FactCC. There are 513 instances in this dataset.
XSumHall (Maynez et al., 2020) study the types of hallucination in the generated summaries and collect the annotation on the errors in the 2K model generated summary for BBC XSum dataset (Narayan et al., 2018). We use the datasets as binary level benchmark for XSum dataset as in (Kryscinski et al., 2020).
SummEval (Fabbri et al., 2021) collect the likert scale human judgments for the 1600 summaries generated from sixteen abstactive summarizer on CNN/DM testset. This dataset provides human judgments scores in terms of "coherence", "consistency", "fluency", and "relevance" by three expert annotators in likert scale. We only use "consistency" score of three annotators, for evaluating our proposed metric. For casting this score to binary level, we let the cases where at least one annotators give less than 5 points for "consistency" as inconsistent, otherwise consistent.
QAGS-CNN/DM & XSum (Wang et al., 2020) release a human judgments for factual consistency
on the model generated summaries for 235 summaries on CNN/DM testset and 239 summaries on XSum testset. Each summary is annotated by three annotators. We also cast the dataset to binary level by assigning inconsistent if at least one annotators give inconsistent label, otherwise consistent.
FRANK-CNN/DM & XSum (Pagnoni et al., 2021) releases a benchmark dataset FRANK for summarization factual metrics which consists of 2246 summaries on the model generated summaries for 1250 summaries in CNN/DM and 996 summaries XSum. Three annotators evaluated factual consistency of the generated summaries in this dataset. We also convert this dataset to binary level as same as QAGS-CNN/DM and QAGS-XSum.
4.3 Baseline Metrics
We compare our methods with the following metrics. For all of the baseline metrics, we manually compute the score using the official repository which each author provided or reproducing the model for a fair comparison.
Table 3: Summary level Pearson Correlation(r) and Spearman's Correlation(ρ) between various automatic metrics and human judgments of factual consistency for the model generated summaries. Note that we use the confidence of consistency label for entailment based metrics.
Dataset	SummEval	QAGS-CNN/DM	QAGS-XSum	FRANK-CNN/DM	FRANK-XSum
Metric	r	ρ	r	ρ	r	ρ	r	ρ	r	ρ
Baselines
ROUGE-L	0.16	0.14	0.29	0.24	0.13	0.13	0.16	0.13	0.16	0.13
BLEU-4	0.11	0.12	0.18	0.23	0.03	0.03	0.16	0.17	0.11	0.14
METEOR	0.18	0.16	0.26	0.25	0.11	0.12	0.29	0.28	0.18	0.16
BERTScore	0.16	0.14	0.37	0.36	0.11	0.13	0.33	0.30	0.19	0.17
QuestEval	0.35	0.30	0.42	0.36	0.20	0.20	0.46	0.41	0.19	0.18
CoCo	0.42	0.36	0.67	0.57	0.20	0.18	0.50	0.45	0.14	0.12
FactCC	0.38	0.36	0.45	0.48	0.30	0.30	0.32	0.36	0.09	0.08
DocNLI	0.51	0.41	0.60	0.59	0.36	0.35	0.49	0.49	0.25	0.21
MNLI	0.11	0.13	0.19	0.22	0.08	0.10	0.15	0.16	0.02	0.03
FEVER	0.33	0.32	0.40	0.34	0.38	0.41	0.38	0.43	0.20	0.19
MF	0.44	0.35	0.43	0.30	0.10	0.10	0.40	0.39	0.10	0.13
Ours
MFMA	0.52	0.38	0.62	0.65	0.37	0.38	0.52	0.45	0.16	0.17
MSM	0.43	0.36	0.50	0.48	0.20	0.22	0.51	0.48	0.05	0.09
Entailment Based Metrics We adopt the model trained on MNLI (Bowman et al., 2015) and FEVER (Thorne et al., 2018) for factual consistency checking as in (Kryscinski et al., 2020). FactCC (Kryscinski et al., 2020) and DocNLI (Yin et al., 2021) are also entailment based models trained on synthetic dataset as in our work.
QA-Based Metrics QuestEval (Scialom et al., 2021) uses the question generation and answering framework for evaluating the factual consistency of the summaries. QuestEval generates the question both the generated summaries and the source article, and then compare the answers of them with both summaries and the article to compute the factuality score of the summary.
N-gram Similarity Metrics BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and METEOR (Banerjee and Lavie, 2005) are widely used for evaluating the summaries. Among them, ROUGE-L, which uses F-measure based on the longest common subsequence between a candidate summary an the reference is the most widely used.
Other Metrics BERTScore (Zhang et al., 2020) utilizes cosine similarity of BERT (Devlin et al., 2019) embeddings between the reference and the generated summary. CoCo (Xie et al., 2021) computes the difference of likelihood of the summarizer between the summary with the original source and the summary with the masked source.
Classification Accuracy Due to the imbalance in each dataset, we report the macro-F1 and class balanced accuracy in Table 1 and Table 2. We observe that macro-F1 score of our proposed methods MFMA outperforms baseline entailment metrics in five of seven benchmark datasets. MFMA shows better performances than other methods in especially for CNN/DM benchmarks, and shows similar performance to other baseline in XSum datasets. We explain that this is because we only use training set of CNN/DM to construct training set. On the other hand, DocNLI additionally uses the human annotated datasets from related tasks such as ANLI (Nie et al., 2020) and SQuAD (Rajpurkar et al., 2016) except for synthetic negative summaries. Another proposed method MSM also shows competitive performance for CNN/DM benchmarks, but relatively lower performance in XSum based benchmark datasets. We explain the performance gap between MSM and MFMA is due to the properties that directly generates summaries, resulting in many noisy samples that are relatively easy to be distinguished.
Correlation with Human Judgments To compare with general metrics that are not classification level, we also report the correlation with human judgments for five datasets in Table 3. We demonstrate that our proposed method has higher pearson correlation coefficient with human judgments in three of five benchmark datasets and competitive
with the best results results in the spearman correlation coefficient. Especially, entailment based methods, which are relatively easy to compute, including our proposed methods show better results than QA-based QuestEval or likelihood based CoCo. Also, reference based methods such as ROUGE-L show very lower performance than other methods that do not require any references.