SlowGuess/NAACL_2022_OCR huggingface dataset:
Figure 3: Validation Performance among Masked Ratio for Mask-and-Fill with Masked Article. We experiment with each of the five combinations of article mask ratio and summary mask ratio, and then plot the interpolated results.
Figure 4: Generated negative summaries among various masking ratio in CNN/DM dataset. For MFMA and MF, we fix the summary masking  $\gamma_{S} = 0.6$ :
Performance among Masked Ratio We analyze the effects of the mask ratio for both source article and summary in our proposed method MFMA and present results using the validation set in Figure 3. Through this experiment, we investigate the tradeoff in adjusting both the article masking ratio and summary masking ratio for generating negative summaries. As shown in Figure 3, we find that too high masking ratio decreases performance by sacrificing affinity. On the other hand, if the masking ratio is insufficient, the generated negative sample is often not really negative. In other words, too lower masking ratio leads to generate positive samples that are almost same as the original summary, and this degrades the performance of factual consistency checking model. Also, we can infer that there is an optimal masking ratio combination where the performance of factual consistency checking model is maximized.
Generated Samples among Masking Ratio We visualize the generated negative summaries through our proposed method MFMA and MSM using CNN/DM in Figure 4. We also visualize the example through MF, which simply fills in the mask
Article: Tropical Storm Andrea formed in the Gulf of Mexico on Wednesday, marking the first storm of the 2013 Atlantic hurricane season. The storm was about 270 miles west-southwest of Tampa, Florida, and moving north at  $6\mathrm{mph}$ , said the National Hurricane Center. It had maximum sustained winds of  $40\mathrm{mph}$ . Thursday afternoon and move over southeastern Georgia and eastern South Carolina Thursday night and Friday. Little change in strength is expected over the next 48 hours.
Reference Summary: Storm moving north at faster rate (6 mph) A storm warning is issued for parts of the Florida coast. Little change in strength is forecast for the next 48 hours. Andrea the first storm of the 2013 Atlantic hurricane season.
MF: Storm moving north at  $6\mathrm{mph}$  (11 kph) A storm warning is issued for parts of the Carolinas. Little change in strength is forecast for the Carolinas. Andrea the first storm of the season.
$\mathbf{MFMA}\gamma_{A} = 0.4$  : Andea moving north at faster rate (6 mph) A storm warning is issued for parts of the Florida coast. Little change in strength is forecast for the next 48 hours. Andea the first storm of the 2013 Atlantic hurricane season.
$\mathbf{MFMA}\gamma_{A} = 0.6$  : Storm moving north at faster rate (6 mph) Tropical storm watch is issued for west coast of Florida. Little change in strength is forecast for next 48 hours. Storm the first storm of the 2013 Atlantic hurricane season.
$\mathbf{MFMA}\gamma_{A} = 0.8$  : Andreae moving north at  $12\mathrm{mph}$  (6 mph) A tropical storm warning is issued for parts of the East Coast. A significant increase in strength is forecast for Thursday. Andreae the first storm of the Atlantic hurricane season.
$\mathbf{MFMA}\gamma_{A} = 1.0$  : Storm moving north at  $6\mathrm{mph}$ $(10\mathrm{kph})$  A tropical storm watch is issued for the west coast of Florida. A hurricane watch in effect is forecast for the East Coast. The center of the storm forms.
MSM  $\gamma_{A} = 0.4$ : The storm is about 270 miles west-southwest of Fort Myers, Florida. NEW: The storm is in effect for the west coast of Florida. The storm is the first of the 2013 Atlantic hurricane season.
without the article. We observe that if the article masking ratio  $\gamma_{A}$  is too low, the generated summaries become almost similar to the original summary since there are enough information to fill the mask. However, if the  $\gamma_{A}$  is too high, the generated examples are too far from the article, resulting in too negative summary similar to filling the mask without article.
Table 4: Balanced accuracy of the human annotated factual consistency among masking unit. NP/Ent denotes noun phrases and entities.
Dataset	Avg-CNN/DM	Avg-XSum
NP/Ent	73.9	60.9
Token	58.6	53.9
Sentence	53.5	53.4
Performance among Masking Unit We basically perform masking operation in the noun phrases and entities units for both summary and article. In order to see the effect of the masking
Figure 5: Validation Set Performance among BERTScore between the original reference summaries and the negative summaries we generate using the various combinations of article and summary masking ratios.
Figure 6: Validation Set Performance among diversity among various combinations of article masking ratio and summary masking ratio. Diversity is computed as negative of the pairwise BERTScore between four negative samples generated by each masking ratio.
Figure 7: Case study on entailment based models. First example comes from and FactCC-Test and second example comes from XSumHall.
unit, we also conduct an experiment on word level masking and sentence level masking, and present the classification level results in Table 4. We observe that noun phrases level masking shows the best results following the work (Goyal and Durrett, 2021) where many errors in summarization system are related to noun phrases and entities.
Distance from Original Reference Summary Using the results on various combinations of article masking ratio and summary masking ratio for MFMA as presented in Figure 3, we also investigate the relation between the average distance from the reference summary on each mask ratio combination and the performance. We compute BERTScore between original reference summary and the negative summary generated using the reference summary to get the distance. Interestingly, as shown in Figure 5, we observe the distribution in
Article: Nkaisery told reporters the university will be able to confirm Saturday if everyone has been accounted for. Thursday's attack by al-Shabaab militants killed 147 people, three security officers and two university security personnel. The attack left 104 people injured, including 19 who are in critical condition, Nkaisery said,...
Candidate Summary: 147 people, including 142 students, are in critical condition.
Ground Truth: INCONSISTENT
MFMA: INCONSISTENT
MSM: INCONSISTENT
DocNLI: INCONSISTENT
FactCC: CONSISTENT
Article: Media playback is not supported on this device. United remain 15 points clear at the top of the table with eight games left after a 1-0 win at Sunderland. "We are not concerned with what we have left behind us, we are only focusing on what is in front of us," said Ferguson. "...
Candidate Summary: Manchester United manager Sir Alex Ferguson says he is not concerned about his side's unbeaten start to the season as they attempt to win the Premier League title.
Ground Truth: CONSISTENT
which performance is maximized within the appropriate distance around 0.8 as the two-dimensional distribution with an  $R^2$  of 0.74. This result shows how far the synthetic negative summaries must be from the reference summaries to help training the factual consistency checking model.
Diversity among Masked Ratio Our proposed method can generate various samples depending on the location of the mask for the same summary- article pair with the fixed mask ratio. Hence, we analyze the diversity of the generated negative summaries among the combinations of mask ratio for MFMA and present the result using validation set in Figure 6. We define the diversity of each mask ratio combination as the negation of pairwise similarity score for each sample following (Tevet and Berant, 2021). We sample four negative summaries using the given article for each method and then compute the pairwise similarity scores for all of the combinations. We also use BERTScore as a similarity measure. Similar to the distance, we observe that diversity has also similar to a two-dimensional form with an  $R^2$  of 0.7, in which the accuracy is maximized at an appropriate point.
Case Study To understand the pros and cons of our proposed factual consistency checking system, we conduct a case study and illustrate the repre
sentative success and failure cases in Figure 7. We observe that our system is good at judging the facts themselves in the summary like the first example, but still not perfect in examples that require high-level reasoning like the second example. We expect the system can be improved by adopting MFMA and MSM to the datasets that have more abstractive summaries which require more reasoning to check the factual consistency.
In this paper, we proposed an effective generation method of factually inconsistent summaries, called MFMA. In this method, some proportion of the source text and corresponding reference summaries is hidden, then a summarization model generates plausible but factually inconsistent summaries by inferring the masked contents. Experiments on seven benchmark datasets demonstrate that factual consistency classifiers trained using our method generally outperform existing models and show a competitive correlation with human judgment.
Our approach creates a synthetic dataset using a public dataset to train a factual consistency checking model. Therefore, in the process of generating such samples, ethically problematic datasets can be generated due to the bias of the pre-trained models, similar to other text generation tasks. For this reason, once the training process is completed, we remove the generated sample. And, we will not release the synthetic dataset itself, and will release only the trained factual consistency checking model.
K. Jung is with ASRI, Seoul National University, Korea. This research was supported by SNU-NAVER Hyperscale AI Center.
Satanjeev Banerjee and Alon Lavie. 2005. ME-TEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
Rajeshree Bora-Kathariya and Yashodhara Haribhakta. 2018. Natural language inference as an evaluation measure for abstractive summarization. In 2018 4th International Conference for Convergence in Technology (I2CT), pages 1-4. IEEE.
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2019. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Wafaa S El-Kassas, Cherif R Salama, Ahmed A Rafea, and Hoda K Mohamed. 2020. Automatic text summarization: A comprehensive survey. Expert Systems with Applications, page 113679.
Alexander R Fabbri, Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391-409.
Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214-2220.
Tanya Goyal and Greg Durrett. 2021. Annotating and modeling fine-grained factuality in summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1449-1462.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR (Poster).
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Thirty-First AAAI Conference on Artificial Intelligence.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885-4901.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with frank: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4812-4829.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. QuestEval: Summarization asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594-6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Guy Tevet and Jonathan Berant. 2021. Evaluating the evaluation of diversity in natural language generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 326-346, Online. Association for Computational Linguistics.
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018. The fact extraction and VERIFICATION (FEVER) shared task. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 1-9, Brussels, Belgium. Association for Computational Linguistics.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008-5020, Online. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122.
Yuexiang Xie, Fei Sun, Yang Deng, Yaliang Li, and Bolin Ding. 2021. Factual consistency evaluation
for text summarization via counterfactual estimation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 100-110.
Wenpeng Yin, Dragomir Radev, and Caiming Xiong. 2021. DocNLI: A large-scale dataset for document-level natural language inference. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 4913-4922, Online. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with BERT*. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Zheng Zhao, Shay B. Cohen, and Bonnie Webber. 2020. Reducing quantity hallucinations in abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2237-2249, Online. Association for Computational Linguistics.
A.1 Reproducibility Checklist
Source Code We attach the source in the submission and we will release the pre-trained factual consistency checking model.
Computing Infrastructure We use Intel(R) Xeon(R) Silver 4210R CPU (2.40 GHz) with NVIDIA RTX A5000 24GB for the experiments. The software environments are Python 3.8.8 and PyTorch 1.10.1.
Dataset Statistics We use the training of CNN/DM dataset that consists of 287113 examples. We divide it in half randomly and use one for MSM or MFMA training and the other for generating negative summaries. Then, we merge the generated article-negative summaries pairs and the article-positive summaries we used for training MFMA and MSM to construct the training set for factual consistency checking model.
Average runtime for each approach For training MFMA and MSM, it takes 10 hours to train the whole model. And it takes 3 hours to generate whole negative summaries that is to be used for training factual consistency checking. For training factual consistency checking model, it takes 7 hours using a single GPU.
Hyperparameters We train five epochs for MFMA and MSM using bart-base for MFMA and t5-small for MSM respectively. We train the model with batch size of 48, max input sequence size of 1024, and max target sequence size of 140. We conduct experiment with various article masking  $\gamma_{A}$  ratio-summary masking ratio  $\gamma_{S}$  combinations, at 0.2 intervals from (0.2, 0.2) to (1.0, 1.0). For the case of training classifier, we train google/electra-base-discriminator for five epochs with learning rate 2e-5 and batch size of 96. We choose the best parameters using the validation set provided by the (Kryscinski et al., 2020). The best mask ratio combination is  $\gamma_{A} = 0.6$  and  $\gamma_{S} = 0.8$ .
Number of Model Parameters The number of parameters for negative summary generation model is 139M for MFMA, is 0.6M (t5-small) and the factual consistency classifier is 109M.
A.2 Computing Baseline Metrics
Even with the same dataset, the results may be different due to some factors such as type of
tokenizer or case, so we calculate baseline ourselves as follows. For n-gram similarity metrics BLEU-4, ROUGE-L and METEOR, we compute the scores using the package language evaluation which is based on COCOeval. For BERTScore, QuestEval and CoCO, we use the official repository with the default setting. For MNLI, we use roberta-large-mnli and use tals/albert-base-vitaminc-fever for FEVER.
A.3 Significance Test
We adopt standard way to test the significance of the correlation coefficient for all of the reported related correlation coefficients in Table 3. We compute the p-value for each coefficient with a t-test that uses a null hypothesis, which is an absence of association.
<sup>4</sup>https://github.com/bckim92/language-evaluation
$^{5}$ https://github.com/tylin/coco-caption
$^{6}$ https://github.com/Tiiger/bert_score
<sup>7</sup>https://github.com/ThomasScialom/QuestEval
<sup>8</sup>https://github.com/xieyxclack/factual_coco
<sup>9</sup>https://huggingface.co/roberta-large-mnli
10https://huggingface.co/tals/albert-base-vitaminc-fever