SlowGuess/NAACL_2021_OCR huggingface dataset:
Model	100×Fact Score	ROUGE-1	ROUGE-2	ROUGE-L
CNN/DailyMail
BOTTOMUP	83.9	41.22	18.68	38.34
Corrected by FC	85.3* (↑1.4%)	40.95	18.37	37.86
UNILM	87.2	43.33	20.21	40.51
Corrected by FC	87.0 (↓0.2%)	42.75	20.07	39.83
SEQ2SEQ	85.0	41.03	18.04	37.93
FASUM	87.8*	40.53	17.84	37.40
Corrected by FC	88.1 (↑0.3%)	40.38	17.67	37.23
XSum
BOTTOMUP	78.0	26.91	7.66	20.01
Corrected by FC	78.9* (↑0.9%)	28.21	8.00	20.69
TCONVS2S	79.8	31.89	11.54	25.75
Corrected by FC	82.9* (↑3.1%)	32.44	11.83	26.02
UNILM	83.2	42.14	19.53	34.13
Corrected by FC	83.4 (↑0.2%)	42.18	19.53	34.15
SEQ2SEQ	80.6	31.44	10.91	24.69
FASUM	81.5	30.28	10.03	23.76
Corrected by FC	81.7 (↑0.2%)	30.20	9.97	23.68
Table 3: Factual consistency score and ROUGE scores on CNN/DailyMail and XSum test set. *p-value< 0.05.
Model	RMR1↑	RMR2↑	NLI↓
UNILM	60.0	39.6	10.2
FC correction	61.4	40.7	10.0
SEQ2SEQ	53.8	32.2	11.2
FASUM	65.0	46.0	9.3
FC correction	67.0	47.4	8.3
Table 4: Average relation matching rate (RMR, Eq. 6) and NLI contradictory ratio between article and summary in CNN/DailyMail test set. The arrow indicates whether larger or smaller value means better result.
(RMR) to measure the ratio of correct hits:
$$
\mathrm {R M R} _ {1} = 1 0 0 \times \frac {C}{C + W} \tag {5}
$$
$$
\mathrm {R M R} _ {2} = 1 0 0 \times \frac {C}{C + W + M} \tag {6}
$$
Note that this metric is different from the ratio of overlapping tuples proposed in Goodrich et al. (2019), where the ratio is computed between the ground-truth and the candidate summary. Since even the ground-truth summary may not cover all the salient information in the article, we choose to compare the knowledge tuples in the candidate summary directly against those in the article. An additional advantage of our metric is that it does not require ground-truth summaries to be available.
Table 4 displays the result of this metric in
CNN/DailyMail's testset. As shown, FASUM achieves the highest precision of correct hits under both measures. And there is a considerable boost from the knowledge graph (FASUM vs SEQ2SEQ):  $11.2\%$  in  $\mathrm{RMR}_1$  and  $13.8\%$  in  $\mathrm{RMR}_2$ . And the correction from the FC model can further improve the metric for both FASUM and UNILM.
We also compute factual consistency via natural language inference models following Maynez et al. (2020). We use the BERT-Large model finetuned on MNLI dataset (Williams et al., 2018) provided by fairseq (Ott et al., 2019). The model predicts the relationship between the article and summary to be one of the following: entailment, neutral and contradiction. We report the ratio of contradiction as predicted by the model in Table 4. As shown, FASUM achieves the lowest ratio and FC helps further reducing conflicting facts in generated summaries.
4.6.3 Human Evaluation
We conduct human evaluation on the factual consistency and informativeness of summaries. We randomly sample 100 articles from the test set of CNN/DailyMail. Then, each article and summary pair is labelled by 3 people from Amazon Mechanical Turk (AMT) to evaluate the factual consistency and informativeness. Each labeller gives a score in each category between 1 and 3 (3 being perfect). The kappa-ratio between reviewer scores is 0.32 for
Model	Factual Score	Informativeness
BOTTOMUP	2.32	2.23
UNILM	2.65	2.45
SEQ2SEQ	2.59	2.30
FASUM	2.74*	2.42
Table 5: Human evaluation results of summaries for 100 randomly sampled articles in CNN/DailyMail test set. *p-value< 0.05.
BOTTOMUP is better	FC is better	Same
15.0%	42.3%	42.7%
UNILM is better	FC is better	Same
20.4%	31.2%	48.4%
Table 6: Human evaluation results for side-by-side comparison of factual consistency on 100 randomly sampled articles from CNN/DailyMail test set where FC makes modifications.
factual consistency and 0.28 for informativeness.
Here, factual consistency indicates whether the summary's content is faithful with respect to the article; informativeness indicates how well the summary covers the salient information in the article.
As shown in Table 5, our model FASum achieves the highest factual consistency score, higher than UNILM and considerably outperforming BOTTOMUP. We conduct a statistical test and find that compared with UNILM, our model's score is statistically significant with p-value smaller than 0.05 under paired t-test. In terms of informativeness, our model is comparable with UNILM and outperforms BOTTOMUP. Finally, without the knowledge graph component, the SEQ2SEQ model generates summaries with both less factual consistency and informativeness.
To assess the effectiveness of the correction model FC, we conduct a human evaluation of side-by-side summaries. In CNN/DailyMail, we randomly sample 100 articles where the summaries generated by BOTTOMUp are modified by FC. 3 labelers are asked whether the original or the corrected version is factually more correct. We collect all the feedbacks and compute the ratio of judgements for each case. To reduce bias, we randomly shuffle the two versions of summaries. We conduct similar evaluation on UNILM.
As shown in Table 6, the corrected summaries are significantly more likely to be judged as more factually correct for both baseline models. For example,  $42.3\%$  of the judgements think the corrected
summaries are factually more correct,  $42.7\%$  think the corrected version neither improves nor worsens the factual consistency, while only  $15.0\%$  think that the corrected version becomes worse than the original BOTTOMUP summary. Therefore, FC can help boost the factual consistency of summaries from given systems.
Finally, to evaluate the quality of the relation matching rate (RMR), we compute the correlation coefficient  $\gamma$  between the factual score given by human labelers and the RMR value. The result shows that  $\gamma = 0.43$ , indicating observable relationship between RMR and human evaluation results.
In this paper, we extract factual information from the article to be represented by a knowledge graph. We then integrate this factual knowledge into the process of producing summaries. The resulting model FASUM enhances the ability to preserve facts during summarization, demonstrated by both automatic and human evaluation. We also present a correction model, FC, to rectify factual errors in candidate summaries. Furthermore, we propose an easy-to-compute model-free metric, relation matching rate, to measure factual consistency based on the overlapping ratio of relational tuples.
For future work, we plan to integrate knowledge graphs into pre-training for more accurate and factually consistent summarization. Moreover, we will combine the internally extracted knowledge graph with an external knowledge graph (e.g. ConceptNet) to enhance the commonsense capability of summarization models.
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D Manning. 2015. Leveraging linguistic structure for open domain information extraction. In ACL 2015, pages 344-354.
Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual error correction for abstractive summarization models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6251-6258, Online. Association for Computational Linguistics.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Thirty-Second AAAI Conference on Artificial Intelligence.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197.
Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020. Multi-fact correction in abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9320-9331, Online. Association for Computational Linguistics.
Esin Durmus, He He, and Mona Diab. 2020. Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization. arXiv preprint arXiv:2005.03754.
Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214-2220.
Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018. Bottom-up abstractive summarization. arXiv preprint arXiv:1808.10792.
Ben Goodrich, Vinay Rao, Peter J Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 166-175.
Belize Gunel, Chenguang Zhu, Michael Zeng, and Xuedong Huang. 2019. Mind the facts: Knowledgeboosted coherent abstractive text summarization. In Proceedings of The Workshop on Knowledge Representation and Reasoning Meets Machine Learning in NIPS 2019.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, pages 1693-1701.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Wojciech Krysciński, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019a. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 540-551, Hong Kong, China. Association for Computational Linguistics.
Wojciech Krysciński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019b. Evaluating the factual consistency of abstractive text summarization. arXiv preprint arXiv:1910.12840.
Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. arXiv preprint arXiv:1808.06226.
Friedrich Wilhelm Levi. 1942. Finite geometrical systems.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2018. Ensure the correctness of the summary: Incorporate entailment knowledge into abstractive sentence summarization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1430-1441.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out..
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Kazuki Matsumaru, Takase Sho, and Okazaki Naoaki. 2020. Improving truthfulness of headline generation. arXiv preprint arXiv:2005.00882.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304.
Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-First AAAI Conference on Artificial Intelligence.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, pages 5998-6008.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008-5020, Online. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtopicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019a. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
Yuhao Zhang, Derek Merck, Emily Bao Tsai, Christopher D Manning, and Curtis P Langlotz. 2019b. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. arXiv preprint arXiv:1911.02541.
Chenguang Zhu, Ziyi Yang, Robert Gmyr, Michael Zeng, and Xuedong Huang. 2019. Make lead bias in your favor: A simple and effective method for news summarization. arXiv preprint arXiv:1912.11602.
Model	Incorrect
Random	50.0%
BERT (Falke et al., 2019)	35.9%
ESIM (Falke et al., 2019)	32.4%
FactCC (Krysciński et al., 2019b)	30.0%
FactCC (our version)	26.8%
Table 7: Percentage of incorrectly ordered sentence pairs using different consistency prediction models in CNN/DailyMail, using data from Falke et al. (2019).
For hyperparameter search, we tried 4 layers with 4 heads, 6 layers with 6 heads and 10 layers with 10 heads.
There're 108.3M parameters in the FASum model and it takes 2 hours (CNN/DailyMail) / 0.5 hours (XSum) for 4 v100 GPUs to train 1 epoch. The batch size is set to 48 for both datasets.
On validation datasets, FASUM achieves ROUGE-1  $41.08\%$ , ROUGE-2  $18.35\%$  and ROUGE-L  $37.95\%$  on CNN/DailyMail, and it achieves ROUGE-1  $30.28\%$ , ROUGE-2  $10.09\%$  and ROUGE-L  $23.85\%$  on XSum.
B Factual Consistency Evaluator
To automatically evaluate the factual consistency of a summary, we leverage the FactCC model (Kryscinski et al., 2019b), which maps the consistency evaluation as a binary classification problem, namely finding a function  $f:(A,C)\longrightarrow [0,1]$  where  $A$  is an article and  $C$  is a summary sentence defined as a claim.  $f(A,C)$  represents the probability that  $C$  is factually correct with respect to the article  $A$ . If a summary  $S$  is composed of multiple sentences  $C_1,\ldots ,C_k$ , we define the factual score of  $S$  as:  $f(A,S) = \frac{1}{k}\sum_{i = 1}^{k}f(A,C_i)$ .
To generate training data, we adopt backtranslation as a paraphrasing tool. The ground-truth summary is translated into an intermediate language, including French, German, Chinese, Spanish and Russian, and then translated back to English. Together with the original summaries, these claims are used as positive training examples. We then apply entity swap, negation and pronoun swap to generate negative examples (KrySciński et al., 2019b).
Following Krysciński et al. (2019b), we finetune the  $\mathrm{BERT}_{\mathrm{BASE}}$  model using the same hyperparameters to finetune FactCC. We concatenate the article and the generated claim together with special tokens [CLS] and [SEP]. The final embedding of
[CLS] is used to compute the probability that the claim is entailed by the article content.
As shown in Table 7, on CNN/Daily Mail, our reproduced model achieves better accuracy than that in Krysciński et al. (2019b) on the human-labelled sentence-pair-ordering data (Falke et al., 2019). Thus, we use this evaluator for all the factual consistency assessment tasks in the following.
Table 8, 9 and 10 show examples of CNN/DailyMail articles and summaries generated by our model and several baseline systems. The factual errors in the summary are marked in red, the correct facts in the summaries of FASUM are marked in green and the corresponding facts are marked in bold in the article.
As shown, while baseline systems like BOTOMUP and UNILM achieve high ROUGE scores, they are susceptible to factual errors. For instance, in Article 5, both BOTOMUP and SEQ2SEQ wrongly state that Rickie Fowler accused Alexis. In fact, Alexis, Rickie's girlfriend, was accused by an online hater. In Article 1, UNILM mistakenly summarizes that Arsenal lost 4-1 where in fact Arsenal won 4-1 against Liverpool.
In comparison, our proposed fact-aware summarizer FASUM could faithfully summarize the salient information in the article. And it can re-organize the phrasing instead of merely copying content from the article.
Table 11 and Table 12 show examples of CNN/DailyMail articles, summaries generated by BOTTOMUP/UNILM and the corrected version by FC. As shown, our correction model can select the wrong entities and replace them with correct ones. For instance, in Article 1, BOTTOMUP's summary states that Rual Castro, who appears elsewhere in the article, is the President of Venezuela, while FC correctly replaces it with Nocolas Maduro. In Article 4, UNILM wrongly attributes the statement to Scott's lawyer (probably because "Scott" appears closer to the statement in the article), while it was actually said by Slager's lawyer. This error is corrected by FC.
<sup>3</sup>We use the same setting and train another evaluator for XSum dataset.
Table 10: Continuation of Table 8.
Table 11: Example articles, summaries and corrections made by FC in CNN/DailMail. The factual errors in the original summary are marked in red, the corrected part by FC are marked in green and the corresponding facts are marked in bold in the article.
Table 12: Continuation of Table 11