SlowGuess/ACL_2020_OCR huggingface dataset:
We first select OpenIE triples from the salient context and filter out those that have any overlapping content word with the correct answer. For argument pair questions, we create one candidate answer by swapping the subject and the object (e.g. candidate B as in Fig. 3) and two candidates by replacing the subject or the object with another argument of the same role extracted from the salient context (e.g. candidates C and D). If not enough answers are created, we further consider randomly selecting sentences from the input. For predicate questions, we use predicates in other triples from the context as candidate answers. Among all candidates, we select the three that are able to construct the most fluent questions using perplexity predicted by BERT (Devlin et al., 2019).
In case reference summaries do not yield OpenIE triples, we create additional entity pair questions. We remove two co-occurring entities from the summary and create three candidate answers in the same way as described above.
QA Model. We fine-tune RoBERTa (Liu et al., 2019) to build our QA model. We use the salient context described above as the context for training. We then concatenate the context, the question, and each of the four candidate answers, and pass the final [CLS] representation through a fully-connected layer, from which the answer is predicted.
6 Experimental Setups
Datasets. We experiment with two popular summarization datasets with summaries containing multiple sentences: the New York Times annotated corpus (NYT) (Sandhaus, 2008) and the CNN/Daily Mail dataset (CNN/DM) (Hermann et al., 2015). We follow the preprocessing steps and experimental setups from prior work (Paulus et al., 2018; See et al., 2017) for both datasets. For NYT, the training, validation, and test sets contain 588,909, 32,716, and 32,703 samples. For CNN/DM, the numbers are 287,188, 13,367, and 11,490.
To train our cloze QA model for NYT, we construct 1, 414, 336 question-answer pairs from human-written summaries in the training set based on the method described in § 5. On CNN/DM, we collect 1, 361, 175 question-answer samples from the training set. For both datasets, we set aside 20,000 samples as a validation set and 20,000 samples as a test set. Our QA model achieves an accuracy of  $97\%$  on NYT and  $95\%$  on CNN.
Training Details and Parameters. We use the base version of RoBERTa model to extract token features for all experiments. We truncate input articles to 1024 (NYT) and 512 (CNN/DM) BPEs. We employ LSTM models with 256-dimensional hidden states for the document encoder (128 each direction) and the decoder. For the residual connection of the graph encoder, we use 4 heads, each with a dimension of 72. For DocGraph training and inference, we prune isolated graphs with fewer than three nodes to increase robustness and reduce redundancy. We set  $\gamma_{1} = 0$ ,  $\gamma_{2} = 0.75$  on NYT and  $\gamma_{1} = 0.33$ ,  $\gamma_{2} = 0.33$  on CNN/DM after tuning on the validation set. For both datasets, we set  $\gamma_{close} = 0.05$ . More details about parameters and graph statistics are in the Appendices.
Baselines and Comparisons. For both datasets,
System	ROUGE-1	ROUGE-2	ROUGE-L
LEAD-3	32.59	16.49	29.17
POINTGEN+COV	41.06	25.71	37.28
DEEPREINFORCE	47.03	30.72	43.10
BOTTOMUP	47.38	31.23	41.81
DCA	48.08	31.19	42.33
SENeca	47.94	31.77	44.34
BART	53.25	36.61	48.78
Our Models
NoGRAPH	47.15	32.02	43.65
+Rrouge	49.17	33.19	46.44
ASGARD-DOC	49.51	33.82	45.72
+Rrouge	50.18	33.91	46.84
+Rrouge + Rcloze	50.59	33.98	48.24
ASGARD-SEG	49.54	33.84	45.75
+Rrouge	50.47	33.95	47.43
+Rrouge + Rcloze	51.29	34.97	48.26
we include an extractive baseline LEAD-3. We further add the following abstractive models for comparison: (1) a pointer-generator model with coverage (See et al., 2017) (POINTGEN+COV); (2) a deep reinforcement learning-based model (Paulus et al., 2018) (DEEPREINFORCE); (3) a bottom-up model (Gehrmann et al., 2018) (BOTTOMUP); (4) a deep communicating agents-based summarization model (Celikyilmaz et al., 2018) (DCA). We also report results by fine-tuning BART model (Lewis et al., 2019). In Lewis et al. (2019), fine-tuning is only performed on CNN/Daily Mail. We apply the same method for NYT.
For NYT, we add results by SENECA model (Sharma et al., 2019) from our prior work, which previously achieved the best ROUGE-2.
On CNN/Daily Mail, we include comparisons of a two-stage fine-tuned model (first on an extractor, then on an abstractor) with BERT (Liu and Lapata, 2019) (BERTSUMEXTABS), and a unified pretrained language model for generation (Dong et al., 2019) (UNILM).
In addition to ASGARD-DOC and ASGARD-SEG, which are trained with an ML objective, we report results trained with ROUGE as the reward  $(R_{\text{rouge}})$ , and with an additional cloze reward  $(R_{\text{cloze}})$ . Lastly, we consider a variant NOGRAPH by ablating the graph encoder.
Table 1: Automatic evaluation with ROUGE on New York Times. Best results are in **boldface**. Best of our models are in **italics**. ASGARD-SEG+R<sub>rouge</sub>+R<sub>close</sub> yields significantly higher scores than our other models with approximate randomization test ( $p < 0.0005$ ).
System	ROUGE-1	ROUGE-2	ROUGE-L
LEAD-3	40.23	17.52	36.34
POINTGEN+COV	39.53	17.28	36.38
DEEPREINFORCE	41.16	15.75	39.08
BOTTOMUP	41.22	18.68	38.34
DCA	41.69	19.47	37.92
BERTSUMEXTABS	42.13	19.60	39.18
UNILM	43.33	20.21	40.51
BART	44.16	21.28	40.90
Our Models
NoGRAPH	39.55	17.89	36.75
+Rrouge	41.37	17.63	37.99
ASGARD-DOC	40.38	18.40	37.51
+Rrouge	43.10	17.58	39.41
+Rrouge + Rcloze	43.93	20.37	40.48
ASGARD-SEG	40.09	18.30	37.30
+Rrouge	42.94	17.93	39.36
+Rrouge + Rcloze	43.81	20.22	40.37
Table 2: Automatic evaluation with ROUGE on CNN/Daily Mail. Best results of our model variants are in italics. Both ASGARD-SEG  $+R_{\text{rouge}} + R_{\text{close}}$  and ASGARD-DOC  $+R_{\text{rouge}} + R_{\text{close}}$  obtain significantly better scores than other model variants  $(p < 0.0005)$ .
7.1 Automatic Evaluation
Results on NYT. As displayed in Table 1, our ASGARD-SEG model trained with ROUGE and cloze rewards achieves better ROUGE scores (Lin and Hovy, 2003) than all other comparisons except the fine-tuned BART. However, our ASGARD-SEG's ROUGE-L score is comparable to BART. This indicates the effectiveness of our graphaugmented summarization framework.
Moreover, both our ASGARD-DOC and ASGARD-SEG models yield significantly higher ROUGE scores than the variant without the graph encoder (NOGRAPH). This demonstrates the benefit of using structured representation to encode entity interactions. Furthermore, both ASGARD-DOC and ASGARD-SEG with cloze reward  $(R_{\text{cloze}})$  obtain significantly higher scores compared to the models trained with ROUGE reward only. This signifies that our multi-choice cloze reward can guide better semantic interpretation of content, leading to the generation of more informative summaries. We also find that ASGARD-SEG outperforms ASGARD-DOC, indicating that ASGARD-SEG better captures topic drift through multiple paragraphs.
Results on CNN/DM. We observe similar trends on the CNN/DM articles as shown in Table 2. No
Figure 4: Evaluation with QA model prediction probability and accuracy on our multiple choice cloze test, with higher numbers indicating better summaries.
ticeably, ASGARD-DOC trained with the combined ROUGE and cloze reward produces better ROUGE scores than BERTSUMEXTABS and UNILM, which are carefully fine-tuned from large pretrained language models, and the numbers are also comparable to the fine-tuned BART.
Evaluation with Cloze Test. We further evaluate model-generated summaries with our proposed cloze test. Here, we report two scores in Fig. 4: the average probability of the correct answers output by our QA model, and its prediction accuracy. We first calculate one score per summary, then take the average over all summaries. We can see that our models with graph encoders perform better than the variant without it.
7.2 Human Evaluation
We further conduct human evaluation to analyze the informativeness and fluency of the generated summaries, as well as to investigate the unfaithful errors made by different models. We sample 100 articles from the NYT test set and hire three native or fluent speakers of English to rate summaries generated by our two systems, NOGRAPH  $+R_{\text{rouge}}$  and ASGARD-SEG  $+R_{\text{rouge}} + R_{\text{close}}$ , along with outputs by BART and human-written summaries (presented in random order). After reading the articles, each judge scores summaries on a Likert scale from 1 (worst) to 5 (best) on informativeness—whether the summary covers important information from the input, and fluency—whether the summary is grammatically correct.
We consider three types of unfaithful errors: (i) hallucination error—creating content not present in the input, (ii) out-of-context error—generating facts without including required context or within
System	Inf.↑	Flu.↑	Hal.↓	Out.↓	Del./Sub.↓
HUMAN	4.47	4.65	21%	10%	10%
NOGRAPH + Rrouge	3.94	3.65	9%*	26%	22%
ASGARD-SEG
+Rrouge + Rcloeze	4.12†	3.77†	23%	14%†	9%*
BART	4.44*	4.66*	16%	15%	12%
Table 3: Human evaluation on informativeness (Inf.) and fluency (Flu.) (1-to-5), and percentages of unfaithful errors of hallucination (Hal.), out-of-context (Out.) and deletion or substitution (Del./Sub.).  $\star$ : significantly different from all other models.  $\dagger$ : ASGARD-SEG is significantly better than NoGRAPH ( $p < 0.05$ ). Interrater agreement with Krippendorf's  $\alpha$  for all columns: 0.61, 0.70, 0.57, 0.50 and 0.43.
Family Court in Burlington County, NJ, rules that lesbian couple can list both their names as parents on birth certificate of newborn; state attorney general's office drops opposition to move; court ruling negates couple's having to go through adoption proceedings to establish full parental rights for both.
NoGraph  $+R_{\text{rouge}}$
Lesbian couple in South Jersey wins court approval to have both of their names listed as parents on birth certificate of their newborn. it will no longer oppose such applications
ASGARD-doc+Rrouge  $^+$  Rcloze:
Lesbian couple in South Jersey, won court approval to have both of their names listed as parents on birth certificate of their newborn. attorney general's office says it will no longer oppose such applications
ASGARD-seg+Rrouge  $^+$  Rcloze:
Lesbian couple in South Jersey wins court approval to have both of their names listed as parents on birth certificate of newborn and attorney general's office will no longer oppose such applications. decision stems from Oct 0 ruling by New Jersey Supreme Court holding that same-sex couples are entitled to same legal rights and protections as heterosexual couples
Figure 5: Sample summaries for an NYT article. Summaries by our models with the graph encoder are more informative than the variant without it.
incorrect context, and (iii) deletion or substitution error—mistakenly deleting or substituting subjects, objects, or clauses. We ask the annotators to label each type as 1 for existence of errors, and 0 otherwise. Detailed guidelines are in the Appendices.
From Table 3, we can see that our ASGARD-SEG model obtains better scores in informativeness and fluency, compared to the variant without the graph encoder. This indicates the effectiveness of leveraging knowledge graph representation. Sample output summaries by our models can be found in Fig. 5. Meanwhile, fine-tuned BART model produces outputs with similar informativeness and fluency of human-constructed summaries, suggest
ing a future direction of building our model on top of a large-pretrained encoder-decoder model.
For unfaithful errors, we report the percentage of errors calculated by majority voting (i.e., more than one annotator vote as incorrect). First, we find that our ASGARD-SEG model has a comparable error pattern as human summaries. Specifically, for out-of-context and deletion or substitution errors, our graph-enhanced model produces significantly fewer mistakes in these categories, compared to the model without graph information. This implies that knowledge graph-enhanced models can improve summary faithfulness.
Interestingly, human-written summaries are also discerned to contain a nontrivial amount of hallucination errors. After inspection, we find that human tends to leverage world knowledge to include content that is not covered by the articles. For instance, for an article discussing events in "Boston", the human writer may describe them as happening in "Massachusetts" in the summary.
7.3 Analyzing Automatic Metrics and Summary Errors
We further plot the distributions of automatic evaluation scores regarding the three types of unfaithful errors based on majority voting in Fig. 6. First, summaries with out-of-context and deletion or substitution errors receive lower cloze and ROUGE scores overall.
Nevertheless, with regard to hallucination errors, we do not see such pattern; there is even a slightly reversed relation with both cloze scores and ROUGE scores, wherein summaries with more hallucination errors tend to score higher. This echos our previous observation that human summaries can be hallucinatory too, where world knowledge is used for writing the summaries.[2]
Furthermore, we find a weak correlation between the three variants of ROUGE scores and three types of errors, e.g., the minimum and the maximum values of Pearson's  $r$  are -0.19 and 0.14. This suggests that new metrics should be designed to better gauge summary quality. We plan to study this direction in future work.
Figure 6: Distribution of automatic summarization metrics with three types of unfaithful errors. "True" indicates summaries with such type of error.
We presented a novel knowledge graph-augmented abstractive summarization framework, along with a novel multiple choice cloze reward for reinforcement learning. Our models capture both local characteristics and global interactions of entities from the input, thus generating summaries of higher quality. In tandem with the graph representation, our cloze reward further improves summary content. Human evaluation further confirms that our graph-augmented models trained with the cloze reward produce more informative summaries and significantly reduces unfaithful errors.
This research is supported in part by National Science Foundation through Grant IIS-1813341, and by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract # FA8650-17-C-9116. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. We thank the anonymous reviewers for their suggestions.
2During human evaluation, we do not ask human judges to distinguish the source of hallucination errors, i.e. from world knowledge or out of fabrication, since this requires significant domain knowledge.
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. 2015. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 344-354, Beijing, China. Association for Computational Linguistics.
Kristjan Arumae and Fei Liu. 2019. Guiding extractive summarization with question-answering rewards. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2566-2577, Minneapolis, Minnesota. Association for Computational Linguistics.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations (ICLR).
Regina Barzilay and Lillian Lee. 2004. Catching the drift: Probabilistic content models, with applications to generation and summarization. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 113-120.
Daniel Beck, Gholamreza Haffari, and Trevor Cohn. 2018. Graph-to-sequence learning using gated graph neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 273-283, Melbourne, Australia. Association for Computational Linguistics.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI).
Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1662-1675, New Orleans, Louisiana. Association for Computational Linguistics.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686.
Marco Damonte and Shay B Cohen. 2019. Structural neural encoders for amr-to-text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3649-3658.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 13063-13075. Curran Associates, Inc.
Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research, 22:457-479.
Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3938-3948, Minneapolis, Minnesota. Association for Computational Linguistics.
Angela Fan, Claire Gardent, Chloé Braud, and Antoine Bordes. 2019. Using local knowledge graph construction to scale Seq2Seq models to multi-document inputs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4177-4187, Hong Kong, China. Association for Computational Linguistics.
Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Structured neural summarization. In International Conference on Learning Representations.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium. Association for Computational Linguistics.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman,
and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693-1701.
Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018. A unified model for extractive and abstractive summarization using inconsistency loss. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 132-141, Melbourne, Australia. Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR).
Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. 2019. Text Generation from Knowledge Graphs with Graph Transformers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2284-2293, Minneapolis, Minnesota. Association for Computational Linguistics.
Wojciech Krysciński, Romain Paulus, Caiming Xiong, and Richard Socher. 2018. Improving abstraction in text summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1808-1817.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Chin-Yew Lin and Eduard Hovy. 2003. Automatic Evaluation of Summaries Using N-gram Cooccurrence Statistics. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume I, pages 71-78.
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In International Conference on Learning Representations.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3721-3731, Hong Kong, China. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.