SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 4: Human evaluation on Factualness, Succinctness, Informativeness. All model variants of S-BART received significantly higher ratings than BART (student t-test,  $p < 0.05$ ).
ROUGE scores (Lin and Och, 2004)  $^{3}$ , and reported ROUGE-1, ROUGE-2, and ROUGE-L in Table 2. We found that, compared to simple sequence-to-sequence models (Pointer Generator and Transformer), incorporating extra information such as commonsense knowledge from ConceptNet  $(D\text{-} HGN)$  increased the ROUGE metrics. When equipped with pre-trained models and simple conversation structures such as topics and conversation stages, Multi-View Seq2Seq boosted ROUGE scores. Incorporating discourse relation graphs or action graphs helped the performances of summarization, suggesting the effectiveness of explicitly modeling relations between utterances and the associations between speakers and actions within utterances. Combining two different structured graphs produced better ROUGE scores compared to previous state-of-the-art methods and our base models,
3We followed fairseq and used https://github. com/pltrdy/rouge to calculate ROUGE scores. Note that different tools may result in different ROUGE scores.
2The version on 10/7 in https://huggingface.co/ transformers/model_doc/bart.html
with an increase of  $2.0\%$  on ROUGE-1,  $4.3\%$  on ROUGE-2, and  $1.2\%$  on ROUGE-L compared to our base model, BART. This indicates that, our structure-aware models with discourse and action graphs could help abstractive conversation summarization, and these two graphs complemented each other in generating better summaries.
Human Evaluation We conducted human evaluation to qualitatively evaluate the generated summaries. Specifically, we asked annotators from Amazon Mechanical Turk to score a set of randomly sampled 100 generated summaries from ground-truth, BART and our structured models, using a Likert scale from 1 (worst) to 5 (best) in terms of factualness (e.g., associates actions with the right actors), succinctness (e.g., does not contain redundant information), and informativeness (e.g., covers the most important content) (Feng et al., 2020; Huang et al., 2020a). To increase annotation quality, we required turkers to have a  $98\%$  approval rate and at least 10,000 approved tasks for their previous work. Each message was rated by three workers. The scores for each summary were averaged. The Intra-Class Correlation was 0.543, showing moderate agreement (Koo and Li, 2016).
As shown in Table 4,  $S$ -BART that utilized structured information from discourse relation graphs and action graphs generated significantly better summaries with respect to factualness, succinctness, and informativeness. This might because that the incorporation of structured information such as discourse relations helped  $S$ -BART to recognize the salient parts in conversations, and thus improve the succinctness and informativeness over BART. Modeling the connections between speakers and actions greatly helped generate more factual summaries than the baselines, e.g., with an increase of 0.27 from BART to  $S$ -BART w. Action.
4.5 Results on Out-Of-Domain Corpus
To investigate the generalizability of our structure-aware models, we then tested the  $S$ -BART model trained on SAMSum corpus directly on the debate summarization domain (ADSC Corpus (Misra et al., 2015)) in a zero-shot setting. Besides the differences in topics, utterances in debate conversations were generally longer and include more action triples (37.20 vs 6.81 as shown in Table 1) and fewer participants. The distribution of discourse relation types also differed a lot across different
Graph Types	R-1	R-2	R-L
S-BART w. Discourse Graph	45.89	22.50	44.83
S-BART w. Random Graph	45.28	21.80	44.30
Table 5: ROUGE-1, ROUGE-2 and ROUGE-L scores of S-BART with either the constructed discourse relation graphs or random graphs. Results are averaged over three random runs.
Combination Strategy	R-1	R-2	R-L
Parallel	46.07	22.60	45.00
Sequential (discourse, action)	45.40	22.14	44.67
Sequential (action, discourse)	45.62	22.41	44.62
Table 6: ROUGE-1, ROUGE-2 and ROUGE-L scores of S-BART models using different ways to combine discourse relation graphs and action graphs. Results are averaged over three random runs.
domains $^4$  (e.g., more Contrast in debates  $(19.5\%)$  than in daily conversations  $(1.0\%)$ ).
As shown in Table 3, our single graph models S-BART w. Discourse and S-BART w. Action boosted ROUGE scores compared to BART, suggesting that utilizing structures can also increase the generalizability of conversation summarization methods. However, contrary to in-domain results in Table 2, action graphs led to much more gains than discourse graphs. This indicated that when domain shifts, action triples were most robust in terms of zero-shot setups; differences in discourse relation distributions could limit such generalization. Consistent with in-domain scenarios, our S-BART w. Discourse&Action achieved better results, with an increase of  $66.2\%$  on ROUGE-1,  $373.4\%$  on ROUGE-2, and  $82.2\%$  on ROUGE-L over BART.
This part conducted ablation studies to show the effectiveness of structured graphs in our  $S$ -BART.
The Quality of Discourse Relation Graphs We showed how the quality of discourse relation graphs affected the performances of conversation summarization in Table 5. Specifically, we compared the ROUGE scores of S-BART using our constructed discourse relation graphs (S-BART w. Discourse Graph) and S-BART using randomly generated discourse relation graphs S-BART w. Random Graph where both connections between nodes and relation types were randomized. The number of edges in two graphs was kept the same. We found that S-BART with our discourse graphs outperformed
4The detailed distributions were shown in the Appendix.
Figure 3: Averaged  $\alpha$  over decoder layers in the trained S-BART models using different graphs
models with random graphs, indicating the effectiveness of the constructed discourse relation graphs and the importance of their qualities.
Different Ways to Combine Graphs We experimented with different ways to combine discourse relation graphs and action graphs in our S-BART w. Discourse & Action, and presented the results in Table 6. Here, parallel strategy performed cross attentions on different graphs separately and then combined the attended results with feed-forward networks as discussed in Section 3.3; sequential strategy performed cross attentions on two graphs in a specific order (from discourse relation graphs to actions graphs, or vice versa). We found that the parallel strategy showed better performances and the sequential ones did not introduce gains compared to S-BART with single graphs. This demonstrates that discourse relation graphs and action graphs were both important and provided different signals for abstractive conversation summarization.
Visualizing ReZero Weights We further tested our structure-aware BART with two ReZero settings: (i) initializing  $\alpha$  from 0, (ii) initializing  $\alpha$  from 1, and found initializing  $\alpha$  from 1 would bring in more performance gains (see Appendix). We then visualized the average  $\alpha$  over different decoder layers after training in Figure 3, and observed that (i) when  $\alpha$  was initialized with 1, the final  $\alpha$  was much larger than the setting where  $\alpha$  was initialized with 0, which might because randomly initialized modules barely received supervisions at early stages and therefore contributes less to BART. (ii) Compared to discourse graphs, action graphs received higher  $\alpha$  weights after training in both initializing settings, suggesting that the information from structured action graphs might be harder for the end-to-end BART models to capture. (iii) Utilizing both graphs spontaneously led to higher
Conversations	# Num	# Dis.	# Act.
Test Set	819	8.63	6.81
Similar	373	8.31	6.36
Increase	208	9.13	7.40
Challenging	160	9.58	7.85
Table 7: The total number of examples, average number of Discourse edges and Action triples in different set of conversations in the SAMSUM test set.
ReZero weights, further validating the effectiveness of combining discourse relation graphs and action graphs and their complementary properties.
4.7 Error Analyses
To inspect when our summarization models could help the conversations summarization, we visualized the average number of discourse edges and the average number of action triples in three sets of conversations in Table 7: (i) Similar: examples where  $S$ -BART generated similar ROUGE scores (the differences were less than 0.1) compared to BART; (ii) Increase: examples where  $S$ -BART resulted in higher ROUGE scores (the differences were larger than 1.0) compared to BART; (iii) Challenging: examples where both  $S$ -BART and BART showed low ROUGE scores (ROUGE-1 < 20.0, ROUGE-2 < 10.0, ROUGE-L < 10.0).
When the structures in conversations were simpler (fewer discourse edges and fewer action triples than the average),  $BART$  showed similar performance as  $S-BART$ . As the structures of conversations become more complex with more discourse relations and more action mentions,  $S-BART$  outperformed  $BART$  as it explicitly incorporated these structured graphs. However, both  $BART$  and  $S-BART$  struggled when there were much more interactions beyond certain thresholds, calling for better mechanisms to model structures in conversations for generating better summaries.
In this work, we introduced a structure-aware sequence-to-sequence model for abstractive conversation summarization by incorporating discourse relations between utterances, and the connections between speakers and actions within utterances. Experiments and ablation studies on SAMSum corpus showed the effectiveness of these structured graphs in aiding the task of conversation summarization via both quantitative and qualitative eval
uation metrics. Results in zero-shot settings on ADCS Corpus further demonstrated the generalizability of our structure-aware models. In the future, we plan to extend our current conversation summarization models for various application domains such as emails, debates, and podcasts, and in conversations that might involve longer utterances and more participants in an unsynchronized way.
We would like to thank the anonymous reviewers for their helpful comments, and the members of Georgia Tech SALT group for their feedback. This work is supported in part by grants from Google, Amazon and Salesforce.
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. 2015. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 344-354, Beijing, China. Association for Computational Linguistics.
Nicholas Asher, Julie Hunter, Mathieu Morey, Benamara Farah, and Stergos Afantenos. 2016. Discourse structure and dialogue acts in multiparty dialogue: the stac corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 2721-2727.
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W Cottrell, and Julian McAuley. 2020. Rezero is all you need: Fast convergence at large depth. arXiv preprint arXiv:2003.04887.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, pages 1171-1179.
Trung Bui, Matthew Frampton, John Dowding, and Stanley Peters. 2009. Extracting decisions from multi-party dialogue using directed graphical models and semantic similarity. In Proceedings of the SIGDIAL 2009 Conference, pages 235-243, London, UK. Association for Computational Linguistics.
Ziqiang Cao, Furu Wei, W. Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In AAAI.
Jiaao Chen, Jianshu Chen, and Zhou Yu. 2019. Incorporating structured commonsense knowledge in
story completion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6244-6251.
Jiaao Chen and Diyi Yang. 2020. Multi-view sequence-to-sequence models with conversational structure for abstractive dialogue summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4106-4118, Online. Association for Computational Linguistics.
Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020. Multi-fact correction in abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9320-9331, Online. Association for Computational Linguistics.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383-1392.
Angela Fan, Claire Gardent, Chloe Braud, and Antoine Bordes. 2019. Using local knowledge graph construction to scale seq2seq models to multi-document inputs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4177-4187.
Xiachong Feng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2020. Incorporating commonsense knowledge into abstractive dialogue summarization via heterogeneous graph networks. arXiv preprint arXiv:2010.10044.
Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. 2018. Structured neural summarization. In International Conference on Learning Representations.
Michel Galley. 2006. A skip-chain conditional random field for ranking meeting utterances by importance. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 364-372.
Shen Gao, Xiuying Chen, Zhaochun Ren, Dongyan Zhao, and Rui Yan. 2020. From standard summarization to new tasks and beyond: Summarization with manifold information. arXiv preprint arXiv:2005.04684.
James Paul Gee. 2014. An introduction to discourse analysis: Theory and method. Routledge.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop
on New Frontiers in Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguistics.
Chih-Wen Goo and Yun-Nung Chen. 2018. Abstractive dialogue summarization with sentence-gated modeling optimized by dialogue acts. 2018 IEEE Spoken Language Technology Workshop (SLT).
Vishal Gupta and Gurpreet Singh Lehal. 2010. A survey of text summarization extractive techniques. Journal of emerging technologies in web intelligence, 2(3):258-268.
Luyang Huang, Lingfei Wu, and Lu Wang. 2020a. Knowledge graph-augmented abstractive summarization with semantic-driven cloze reward. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5094-5107, Online. Association for Computational Linguistics.
Shanshan Huang, Kenny Q. Zhu, Qianzi Liao, Libin Shen, and Yinggong Zhao. 2020b. Enhanced story representation by conceptnet for predicting story endings. In Proceedings of the 29th ACM International Conference on Information Knowledge Management, CIKM '20, page 3277-3280, New York, NY, USA. Association for Computing Machinery.
Paul A Kirschner, Simon J Buckingham-Shum, and Chad S Carr. 2012. Visualizing argumentation: Software tools for collaborative and educational sensemaking. Springer Science & Business Media.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Open-source toolkit for neural machine translation.
Terry K Koo and Mae Y Li. 2016. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chiropractic medicine, 15(2):155-163.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Manling Li, Lingyu Zhang, Heng Ji, and Richard J. Radke. 2019. Keep meeting summaries on topic: Abstractive multi-modal meeting summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2190-2196, Florence, Italy. Association for Computational Linguistics.
Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram
statistics. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, page 605. Association for Computational Linguistics.
Chunyi Liu, Peng Wang, Jiang Xu, Zang Li, and Jieping Ye. 2019a. Automatic dialogue summary generation for customer service. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD19, page 1957-1965, New York, NY, USA. Association for Computing Machinery.
H. Liu and P. Singh. 2004. Conceptnet — a practical commonsense reasoning tool-kit. BT Technology Journal, 22(4):211-226.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3721-3731.
Zhengyuan Liu, Angela Ng, Sheldon Lee, Ai Ti Aw, and Nancy F. Chen. 2019b. Topic-aware pointer-generator networks for summarizing spoken conversations. 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU).
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55-60, Baltimore, Maryland. Association for Computational Linguistics.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, Online. Association for Computational Linguistics.
K. McKeown, Lokesh Shrestha, and Owen Rambow. 2007. Using question-answer pairs in extractive summarization of email conversations. In CICling.
Amita Misra, Pranav Anand, Jean E. Fox Tree, and Marilyn Walker. 2015. Using summarization to discover argument facets in online ideological dialog. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 430-440, Denver, Colorado. Association for Computational Linguistics.
Beverly Moser. 2001. An introduction to discourse analysis. theory and method.
Gabriel Murray, S. Renals, and J. Carletta. 2005. Extractive summarization of meeting recordings. In INTERSPEECH.
Gabriel Murray, Steve Renals, Jean Carletta, and Johanna Moore. 2006. Incorporating speaker and discourse features into speech summarization. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 367-374, New York City, USA. Association for Computational Linguistics.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1747-1759.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations.
Kochen Qin, Lu Wang, and Joseph Kim. 2017. Joint modeling of content and discourse relations in dialogues. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 974-984.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379-389, Lisbon, Portugal. Association for Computational Linguistics.
Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. 1978. A simplest systematics for the organization of turn taking for conversation. In Studies in the organization of conversational interaction, pages 7-55. Elsevier.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Guokan Shang, Wensi Ding, Zekun Zhang, Antoine Tixier, Polykarpos Meladianos, Michalis Vazirgiannis, and Jean-Pierre Lorre. 2018. Unsupervised abstractive meeting summarization with multisentence compression and budgeted submodular maximization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 664-674, Melbourne, Australia. Association for Computational Linguistics.
Zhouxing Shi and Minlie Huang. 2019. A deep sequential model for discourse parsing on multi-party dialogues. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7007-7014.
Kaiqiang Song, Chen Li, Xiaoyang Wang, Dong Yu, and Fei Liu. 2020. The ucf podcast summarization system at trec 2020. arXiv preprint arXiv:2011.04132.
Matthew Stone, Una Stojnic, and Ernest Lepore. 2013. Situated utterances and discourse relations. In Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013)-Short Papers, pages 390-396.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations.
Lu Wang and Claire Cardie. 2013. Domain-independent abstract generation for focused meeting summarization. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1395-1405.
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Discourse-aware neural extractive text summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5021-5031, Online. Association for Computational Linguistics.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. 2019. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. arXiv preprint arXiv:1912.08777.
Zhou Zhao, Haojie Pan, Changjie Fan, Yan Liu, Linlin Li, Min Yang, and Deng Cai. 2019. Abstractive meeting summarization via hierarchical adaptive segmental network learning. In The World Wide Web Conference, WWW '19, page 3455-3461, New York, NY, USA. Association for Computing Machinery.
Chenguang Zhu, William Hinthorn, Ruochen Xu, Qing kai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2020a. Boosting factual correctness of abstractive summarization with knowledge graph. ArXiv, abs/2003.08612.
Chenguang Zhu, Ruochen Xu, Michael Zeng, and Xuedong Huang. 2020b. A hierarchical network for abstractive meeting summarization with cross-domain pretraining. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computational Linguistics.
A Discourse Relation Distributions
We pre-trained a deep sequential model (Shi and Huang, 2019) on STAC Corpus (1,062 dialogues)