ExplainMeetSum: A Dataset for Explainable Meeting Summarization Aligned with Human Intent
Year: 2023

Abstract:
To enhance the explainability of meeting summarization, we construct a new dataset called “ExplainMeetSum,” an augmented version of QMSum, by newly annotating evidence sentences that faithfully “explain” a summary. Using ExplainMeetSum, we propose a novel multiple extractor guided summarization, namely Multi-DYLE, which extensively generalizes DYLE to enable using a supervised extractor based on human-aligned extractive oracles. We further present an explainabilityaware task, named “Explainable Evidence Extraction” (E3), which aims to automatically detect all evidence sentences that support a given summary. Experimental results on the QMSum dataset show that the proposed Multi-DYLE outperforms DYLE with gains of up to 3.13 in the ROUGE-1 score. We further present the initial results on the E3 task, under the settings using separate and joint evaluation metrics.1

heading: Future Work
text: In future work, we would like to invent a joint learning framework for meeting summarization and E3, extensively employing human-supervised explainable signals from ExplainMeetSum, towards better explainable meeting summarization. Furthermore, developing a novel joint evaluation metric for meeting summarization and E3 to overcome the limitations of the ROUGE-driven scores would be worthwhile. Limitations
This paper presents ExplainMeetSum to enhance the explainability of meeting summarization and provides Multi-DYLE as a generalized version of DYLE by employing multiple extractors. One limitation of our work is the restricted exploration of using ExplainMeetSum for meeting summarization. Although we propose the use of multiple extractors, it can go beyond DYLE’s extractivegenerator framework, thereby extending and generalizing other extract-then-generate methods, such as Dou et al. (2021). In addition, we currently use single and dual extractors (i. e. , M = 1 or M = 2) for Multi-DYLE. However, other advanced settings using more extractors (M > 2) were not examined in this experiment. Another limitation is the current joint evaluation metrics, such as precision, recall, F1 scores, and Joint ROUGE scores, adopted as initial trials of evaluating of the “summarize-then-explain” joint setting. In particular, Joint ROUGE scores inherit the limitations of the original ROUGE scores. An important remaining issue is to design a more stable and agreeable joint evaluation metric that can be used as a standard evaluation metric for the joint setup of the summarize-then-explain task. Furthermore, our current applications using ExplainMeetSum are limited to meeting summarization and E3. However, ExplainMeetSum can be used as a suitable benchmark dataset to compare various interpretable and explainable models on summarization results. Given the emerging importance of interpretable and explainable models, arguably valuable work is to extensively examine the usefulness of ExplainMeetSum in more interpretable-related tasks by exploring and evaluating interpretable models, such as Ribeiro et al. (2016); Lundberg and Lee (2017); Sundararajan et al. (2017); Sanyal and Ren (2021); Saha et al. (2022). Some parts of the annotations in ExplainMeetSum were not fully utilized in this work. We also annotated the evidence sentences for General(long) types of queries in AMI and ICSI, however, our work on meeting summarization used only QMSum as a benchmark dataset. Thus, it would be valuable to obtain additional results using ExplainMeetSum for meeting summarization in AMI and ICSI to examine whether the use of ExplainMeetSum leads to improvements in other types of datasets. Acknowledgement
This work was supported by Institute of Information & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (2022-0-00989,Development of Artificial Intelligence Technology for Multi-speaker Dialog Modeling) and (2019-0-00004,Development of semi-supervised learning language intelligence technology and Korean tutoring service for foreigners)
We would like to thank all anonymous reviewers for their valuable comments and suggestions. References Mousumi Akter, Naman Bansal, and Shubhra Kanti
Karmaker. 2022. Revisiting automatic evaluation of extractive summarization task: Can we do better than ROUGE? In Findings of the Association for Computational Linguistics: ACL 2022, pages 1547– 1560, Dublin, Ireland. Association for Computational Linguistics. Ahsaas Bajaj, Pavitra Dangati, Kalpesh Krishna, Pradhiksha Ashok Kumar, Rheeya Uppaal, Bradford Windsor, Eliot Brenner, Dominic Dotterrer, Rajarshi Das, and Andrew McCallum. 2021. Long document summarization in a low resource setting using pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 71–80, Online. Association for Computational Linguistics. Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska, Iain McCowan, Wilfried Post, Dennis Reidsma, and Pierre Wellner. 2006. The ami meeting corpus: A pre-announcement. In Machine Learning for Multimodal Interaction, pages 28–39, Berlin, Heidelberg. Springer Berlin Heidelberg. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675– 686, Melbourne, Australia. Association for Computational Linguistics. Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2021. GSum: A general framework for guided neural abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, pages 4830–4842, Online. Association for Computational Linguistics. Xiachong Feng, Xiaocheng Feng, Bing Qin, Xinwei Geng, and Ting Liu. 2020. Dialogue discourse-aware graph convolutional networks for abstractive meeting summarization. volume abs/2012. 03502. Quentin Grail, Julien Perez, and Eric Gaussier. 2021. Globalizing BERT-based transformer architectures for long document summarization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1792–1810, Online. Association for Computational Linguistics. A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolcke, and C. Wooters. 2003. The icsi meeting corpus. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP ’03). , volume 1, pages I–I. Logan Lebanoff, Kaiqiang Song, Franck Dernoncourt, Doo Soon Kim, Seokhwan Kim, Walter Chang, and Fei Liu. 2019. Scoring sentence singletons and pairs for abstractive summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2175–2189, Florence, Italy. Association for Computational Linguistics. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledgeintensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc. Annie Louis and Ani Nenkova. 2009. Automatically evaluating content selection in summarization without human models. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 306–314, Singapore. Association for Computational Linguistics. Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. Bing Ma, Cao Liu, Jingyu Wang, Shujie Hu, Fan Yang, Xunliang Cai, Guanglu Wan, Jiansong Chen, and Jianxin Liao. 2021. Distant supervision based machine reading comprehension for extractive summarization in customer service. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, page 1895–1899, New York, NY, USA. Association for Computing Machinery. Ziming Mao, Chen Henry Wu, Ansong Ni, Yusen Zhang, Rui Zhang, Tao Yu, Budhaditya Deb, Chenguang Zhu, Ahmed Awadallah, and Dragomir Radev. 2022. DYLE: Dynamic latent extraction for abstractive long-input summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1687–1698, Dublin, Ireland. Association for Computational Linguistics. Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145–152, Boston, Massachusetts, USA. Association for Computational Linguistics. MengNan Qi, Hao Liu, YuZhuo Fu, and Ting Liu. 2021. Improving abstractive dialogue summarization with hierarchical pretraining and topic segment. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1121–1130, Punta Cana, Dominican Republic. Association for Computational Linguistics. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should i trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1135–1144, New York, NY, USA. Association for Computing Machinery. Swarnadeep Saha, Shiyue Zhang, Peter Hase, and Mohit Bansal. 2022. Summarization programs: Interpretable abstractive summarization with neural modular trees. Soumya Sanyal and Xiang Ren. 2021. Discretized integrated gradients for explaining language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10285–10299, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Natalie Schluter. 2017. The limits of automatic summarisation according to ROUGE. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 41–45, Valencia, Spain. Association for Computational Linguistics. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 3319–3328. JMLR. org. Jesse Vig, Alexander Fabbri, Wojciech Kryscinski, Chien-Sheng Wu, and Wenhao Liu. 2022. Exploring neural models for query-focused summarization. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1455–1468, Seattle, United States. Association for Computational Linguistics. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. Wenhan Xiong, Anchit Gupta, Shubham Toshniwal, Yashar Mehdad, and Wen-tau Yih. 2022. Adapting pretrained text-to-text models for long text sequences. Jiacheng Xu and Greg Durrett. 2019. Neural extractive text summarization with syntactic compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3292–3303, Hong Kong, China. Association for Computational Linguistics. Haoyu Zhang, Jingjing Cai, Jianjun Xu, and Ji Wang. 2019. Pretraining-based natural language generation for text summarization. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 789–797, Hong Kong, China. Association for Computational Linguistics. Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Awadallah, Dragomir Radev, and Rui Zhang. 2022. Summn: A multi-stage summarization framework for long input dialogues and documents. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1592–1604, Dublin, Ireland. Association for Computational Linguistics. Yusen Zhang, Ansong Ni, Tao Yu, Rui Zhang, Chenguang Zhu, Budhaditya Deb, Asli Celikyilmaz, Ahmed Hassan Awadallah, and Dragomir Radev. 2021. An exploratory study on long dialogue summarization: What works and what’s next. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4426–4433, Punta Cana, Dominican Republic. Association for Computational Linguistics. Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905–5921, Online. Association for Computational Linguistics. Chenguang Zhu, Ruochen Xu, Michael Zeng, and Xuedong Huang. 2020. A hierarchical network for abstractive meeting summarization with crossdomain pretraining. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 194–203, Online. Association for Computational Linguistics. A Dataset Construction
A. 1 Annotation Process
ExplainMeetSum was built on top of the QMSum dataset. Here, we choose QMSum because its dataset is one of the most widely used datasets and its further annotated dataset, ExplainMeetSum, is also invaluable, which is likely to be of wide research interest. For annotation, we recruited four annotators and a coordinator based on their proficiency in English and prior experience with English dataset annotations. To create ExplainMeetSum, they were required to select and annotate evidence sentences from transcripts that are considered as aligned to “explain“ well each gold sentence summary in the approximately 2,000 queries distributed among 232 transcripts; once selecting an evidence sentence, an annotator labeled its main type, CES and PES, as well as matching characteristics (i. e. , subtypes of CES and PES). Despite QMSum being relatively limited in its size, the annotation work is time-consuming because the annotation tasks are non-trivial and involve extensive revisions, based on multiple feedbacks and comments provided by the coordinator. Considering the large amount of annotation work, we set aside more than six months for this task based on the two-level feedback pipeline on which the annotators interacted with both a coordinator and an expert. The coordinator thoroughly checked the annotation process and interacted with the annotators, while the expert acted as a metareviewer who periodically inspected random samples in-depth, provided feedback, and updated the guidelines. The total cost of the annotator’s work was $40,000 which equates to an estimation of $20 per query. A. 2 Examples in ExplainMeetSum
Table 7 shows examples in ExplainMeetSum collected across various queries in QMSum. Being categorized with main types and matching characteristics (i. e. , subtypes), CESs and PESs aligned for each of the gold summary sentences indicated by ‘[G-Sent]’ are presented in the column ‘Evidence Sentence. ’
Matching characteristics are classified into six labels: CES has exact, semantic, and supportive subtypes, whereas PES has illustrative, introductory, and connective subtypes, defined as follows:
• CES/exact: A subtype of CES whose keywords are “exactly” matched with the contents in the target sentence of the summary. • CES/semantic: A subtype of CES whose keywords “semantically” match the contents in the target sentence of the summary. • CES/supportive: A subtype of CES that includes information or examples that directly support or entail abstractive expressions in the target sentence of the summary. • PES/illustrative: A subtype of PES that includes relevant information or examples, such as enumerating further detailed information in addition to CES, on which a reasoning step is required to match the target sentence of the summary. • PES/introductory: A subtype of PES, which includes relevant information that initiates the subject of the extracted evidences; it usually appears before CES. • PES/connective: A subtype of PES that includes relevant information used to build the connectivity between the extracted evidences; it is usually based on conjunctions that connect two subsequent CESs