SlowGuess/NAACL_2022_OCR huggingface dataset:
Competing Methods. We compare our model with some competing methods, including extractive and abstractive summarization models. EXT-ORACLE (Narayan et al., 2018) and MATCHSUM (Zhong et al., 2020) are extractive models. EXT-ORACLE selects a single best sentence of the document by referring to the target. MATCHSUM reranks the candidate summaries produced by BertExt (Liu and Lapata, 2019) and achieves state-of-the-art extractive results on various summarization datasets. For abstractive summarization models, FASum (Zhu et al., 2021) and GSum (Dou et al., 2021) are models designed for faithful summarization. FASum extracts the relation triplets and uses a knowledge graph to synthesize information. Then the graph information is fed into the Transformer architecture. GSum is a general framework for guided neural summarization, which investigates four types of guidance signals and achieves state-of-the-art performance on various popular datasets.
5https://qihub.com/Huffon/factsumm
$^{6}$ https://nlp.stanford.edu/software/openie.html
7https://github.com/ThomasScialom/ QuestEval
We use the prediction files of the competing models provided in EXPLAINABOARD  $^{8}$  other than running the models. It is noted that the difference between the performance of EXPLAINABOARD and the results in the original paper is below 1 point in terms of ROUGE.
5.2 Main Results and Analysis
Main results. Table 2 presents the detailed results on the test set of the datasets including traditional metrics and factual consistency metrics. Compared with the results published in EXPLAINABOARD, our reimplement of BART is inferior by about 0.6 points in terms of ROUGE-L. It is noted that the performance of BART on XSUM dataset has been discussed in fairseq repository<sup>9</sup>. The results on base models implicate that our implementation is fair for our study.
We apply GISG on XSUM and CNNDM with BART as the backbone. As seen, GISG achieves higher performance for lexical overlay on both datasets compared to BART. It achieves 0.8 and 0.5 points improvement in terms of ROUGE-L on XSUM and CNNDM datasets, which is a considerable improvement over strong baselines for summarization. It is noted that GSum in Table 2 use the key sentences as the guidance. Although there is the version that GSum uses keywords as the guidance in (Dou et al., 2021), which is more relevant to our work. EXPLAINABOARD does not provide the output files and we report the results using key sentences as the guidance.
For the factual consistency metrics, GISG outperforms BART on all factual consistency metrics which indicates that jointly training keyphrases induction and summary generation benefit the faithful consistency. Compared with a strong factual guidance baseline FASum and GSum, our approach consistently outperforms FASum and GSum.
Compared with extractive summarization baseline MATCHSUM and EXT-ORACLE, the abstractive summarization models have a large margin in terms of factual consistency, even if these models achieve much higher performance on the lexical overlap. It indicates that extractive summarization models can get better factual consistency at the cost of being relevant and fluent.
Figure 3: Distribution of the number of keyphrases.
Figure 4: Fine tuning on the hyper-parameter  $\lambda$ .
For the results between XSUM and CNNDM, all factual consistency metrics on XSUM are much lower than CNNDM. This is consistent with the conclusion that summaries in XSUM are much more abstractive. It is more difficult for the model to generate consistent results on XSUM.
Due to the extreme abstractive nature of XSUM dataset, it is ideal to evaluate the models' ability to capture the facts of the document. In the rest of this section, we present in-depth analyses to better understand our model with XSUM as the testbed.
Distribution of the number of keyphrases. We assume the maximum number of the keyphrases is  $B$  in Section 4.3 and the model selects fewer keyphrases by selecting one candidate repeatedly. In this section, We investigate the distribution of the number of keyphrases for the test set and the model prediction in Table 3.
As seen, most of the reference summaries have about 4 keyphrases while most of the reference summaries have less than eight keyphrases. Thus  $B$  is set to eight according to the ground truth distribution.
Moreover, the number of keyphrases in the model prediction is larger than the ground truth. We argue that it is because the model tends to take advantage of all the latent variables and selects redundancy and similar candidates.
Fine tuning on hyper-parameter  $\lambda$ . In Eqn. (7),
http://explainaboard.nlpedia.ai/leaderboard/task-summ/index.phphttps://github.com/pytorch/fairseq/issues/1971
Figure 5: Case study.
we use  $\lambda$  to keep a trade-off between  $\mathcal{L}_{\mathrm{CVAE}}$  and  $\mathcal{L}_{\mathrm{w}}$ . We analyze the effect of  $\lambda$  in Figure 4.
As seen, ROUGE-L is boosted with the increment of  $\lambda$  until 0.2, showing that predicting the words of the summary by the keyphrases contributes to the performance.
Subsequently, a larger value of  $\lambda$  reduces the ROUGE-L and the performance is even lower than without  $\mathcal{L}_{\mathrm{w}}$ . We argue that it is because  $f_{\mathrm{key}}$  is constrained to predict the words of the summary. A larger value of  $\lambda$  would disturb the word prediction item  $y_{t}$ , which would hurt the performance. Therefore, we set the hyperparameter  $\lambda$  to 0.2 to control the effect.
Ablation on the keyphrases prediction network. We first predict the keyphrases and generate the summaries conditioned on the source document and the keyphrases. We investigate the influence of the keyphrases prediction network and replace the module with a random selection of  $B$  keyphrases. The results are shown in Table 3.
The results show that both ROUGE, BERTScore and factual consistency metrics have a descend without the keyphrases prediction module, which indicates the effectiveness of the guidance prediction module.
Ablation on the number of keyphrases in testing. To investigate the effectiveness of the keyphrases prediction network, we make ablation of the keyphrases in testing in Figure 6, where we increase the number of the attending keyphrases gradually.
As shown in the figure, the performance increases as more keyphrases are used to
Models	Prediction	Rand
R-L	36.82	36.25
BERTScore	92.11	91.07
Close	2.14	1.96
Open	2.18	2.07
QAGS	14.31	13.94
QuesEval	37.09	37.02
Table 3: Ablation on the keyphrases prediction.
Figure 6: Ablation on the number of keyphrases in testing.
generate the summaries. Without attending to any keyphrases of the module prediction, the performance drops about 1 point in terms of ROUGE-L. It indicates that the keyphrase prediction filters the information and helps the decoder to generate a more accurate summarization.
Case Study To further demonstrate the effectiveness of our method, we give a case study in Figure 5. We compare the summary generated based on our approach and baseline which is based
on BART. As shown in Figure 5, the baseline model generated hallucination, "They are not allowed to spend any time together", which is inconsistent with the source document, "They visit at Charleston County Jail". Our model first predicts the keyphrases from the source document and generates the summary conditioned on the source document. As shown in the figure, our result is more faithful, which confirms the effectiveness of our approach.
In this paper, we propose to learn guidance induction and summary generation jointly via conditional variational autoencoders. We use phrases as the information granularity of our guidance and we induce the keyphrases of the source document. These summaries are generated conditioned on the source document and the keyphrases, ensuring the important information is consistent with the source document. The experiments show that our approach can generate more faithful summaries than the existing state-of-the-art approaches, according to multiple factual consistency metrics.
We are grateful to the anonymous reviewers and meta reviews for their insightful comments and suggestions. This work is funded by the National Natural Science Foundation of China (Grant No. U1908216).
Mehdi Allahyari, Seyed Amin Pouriyeh, Mehdi Assefi, Saeid Safaei, Elizabeth D. Trippe, Juan B. Gutierrez, and Krys J. Kochut. 2017. Text summarization techniques: A brief survey. CoRR, abs/1707.02268.
Rahul Aralikatte, Shashi Narayan, Joshua Maynez, Sascha Rothe, and Ryan Thomas Mcdonald. 2021. Focus attention: Promoting faithfulness and diversity in summarization. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of The Conference on Computational Natural Language Learning, pages 10-21.
Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual error correction for
abstractive summarization models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 6251-6258.
Shuyang Cao and Lu Wang. 2021. CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 6633-6649.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).
Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. 2021a. Improving faithfulness in abstractive summarization with contrast candidate generation and selection. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pages 5935-5941.
Wenqing Chen, Jidong Tian, Yitian Li, Hao He, and Yaohui Jin. 2021b. De-confounded variational encoder-decoder for logical table-to-text generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, pages 5532-5542.
Hal Daumé III and Daniel Marcu. 2006. Bayesian query-focused summarization. In Proceedings of the International Conference on Computational Linguistics and Annual Meeting of the Association for Computational Linguistics, pages 305-312. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171-4186.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems.
Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020. Multifact correction in abstractive text summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 9320-9331.
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2021. Gsum: A general framework for guided neural abstractive summarization. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pages 4830-4842.
Esin Durmus, He He, and Mona Diab. 2020. Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 5055-5070.
Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. 2021. GO FIGURE: A meta evaluation of factuality in summarization. In Findings of the Association for Computational Linguistics, pages 478-487.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, volume 28, page 1693-1701.
Yichong Huang, Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021. The factual inconsistency problem in abstractive text summarization: A survey.
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations.
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In International Conference on Learning Representations.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 9332-9346.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2018. Ensure the correctness of the summary: Incorporate entailment knowledge into abstractive sentence summarization. In Proceedings of the International Conference on Computational Linguistics, pages 1430-1441.
Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, page 71-78.
Wei Liu, Huanqin Wu, Wenjing Mu, Zhen Li, Tao Chen, and Dan Nie. 2021. Co2sum: contrastive learning for factual-consistent abstractive summarization.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing, pages 3730-3740.
James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi. 2019. Don't blame the elbo! a linear vae perspective on posterior collapse. In Proceedings of the International Conference on Neural Information Processing Systems.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1906-1919.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404-411, Barcelona, Spain. Association for Computational Linguistics.
Ines Montani, Matthew Honnibal, Matthew Honnibal, Sofie Van Landeghem, Adriane Boyd, Henning Peters, Maxim Samsonov, Jim Geovedi, Jim Regan, György Orosz, Paul O'Leary McCann, Søren Lind Kristiansen, Duygu Altinok, Roman, Leander Fiedler, Grégy Howard, Wannaphong Phatthiyaphaibun, Explosion Bot, Sam Bozek, Mark Amery, Yohei Tamura, Björn Böing, Pradeep Kumar Tippa, Leif Uwe Vogelsang, Ramanan Balakrishnan, Vadim Mazaev, GregDubbin, jeannefukumaru, Jens Dahl Møllerhøj, and Avadh Patel. spaCy: v3.0.0rc: Transformer-based pipelines, new training system, project templates, custom models, improved component API, type hints & lots more [online]. 2020.
Feng Nan, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Kathleen McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew O. Arnold, and Bing Xiang. 2021. Improving factual consistency of abstractive summarization via question answering. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, pages 6881-6894.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1797-1807.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pages 4812-4829.
Clement Rebuffel, Thomas Scialom, Laure Soulier, Benjamin Piwowarski, Sylvain Lamprier, Jacopo Staiano, Geoffrey Scoutheeten, and Patrick Gallinari. 2021. Data-QuestEval: A referenceless metric for data-to-text semantic evaluation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 8029-8036.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the International Conference on Machine Learning, volume 32, pages 1278-1286.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. 2015. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems, volume 28.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 5008-5020.
Zijun Wu, Atharva Naik, Zi Xuan Zhang, and Lili Mou. 2021. Weakly supervised explainable phrasal reasoning with neural fuzzy logic. arXiv:2109.08927.
Shusheng Xu, Xingxing Zhang, Yi Wu, and Furu Wei. 2022. Sequence level contrastive learning for text summarization. In Proceedings of the Association for the Advance of Artificial Intelligence.
Wang Xu, Kehai Chen, and Tiejun Zhao. 2021a. Discriminative reasoning for document-level relation extraction. In *Findings of the Annual Meeting of the Association for Computational Linguistics*, pages 1653-1663.
Wang Xu, Kehai Chen, and Tiejun Zhao. 2021b. Document-level relation extraction with reconstruction. In Proceedings of the Association for the Advance of Artificial Intelligence, 16, pages 14167-14175.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020a. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the International Conference on Machine Learning, pages 11328-11339.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b. *Bertscore: Evaluating text generation with bert*. In International Conference on Learning Representations.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders.
In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 654-664.
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization as text matching. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 6197-6208.
Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 2021. Detecting hallucinated content in conditional neural sequence generation. In Findings of the Association for Computational Linguistics, pages 1393-1404.
Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2021. Enhancing factual consistency of abstractive summarization. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pages 718-733.
Qile Zhu, Wei Bi, Xiaojiang Liu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. 2020. A batch normalized inference network keeps the kl vanishing away. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 2636-2649.