SlowGuess/NAACL_2021_OCR huggingface dataset:
in this paper. We use fairseq library<sup>5</sup> to implement BART model. For PGNet and BART, we truncate the input text to 2,048 tokens, and remove the turns whose lengths are less than 5. All results reported in this paper are averages of three runs.
5.2 Effectiveness of Locator
First, we need to verify the effectiveness of the Locator to ensure that it can extract spans related to the query. Instead of the accuracy of capturing relevant text spans, we focus on the extent of overlap between the selected text spans and the gold relevant text spans. It is because whether the summarization process is built on similar contexts with references or not is essential for Summarizer. Therefore, we use ROUGE-L recall to evaluate the performance of different models under the setting of extracting the same number of turns.
We introduce two additional baselines: Random and Similarity. The former refers to randomly extracting a fixed number of turns from the meeting content, while the latter denotes that we obtain turn embedding and query embedding through a feature-based BERT, and then extract the most similar turns by cosine similarity. As shown in Table 2, because there are usually a large number of repeated conversations in the meetings, Random can get a good ROUGE-L recall score, which can be used as a baseline to measure the performance of the model. Similarity performs badly, even worse than Random, which may be due to the great difference in style between the BERT pre-trained corpus and meeting transcripts. Pointer Network is only slightly better than Random. We think this is because in the text of with an average of more than 500 turns, only three <start, end> pairs are given as supervision signals, which is not very informative and therefore is not conducive to model learning.
Table 2: ROUGE-L Recall score between the predicted spans and the gold spans.  $1/6$  means that the turns extracted by the model account for  $1/6$  of the original text.
Models	R-1	R-2	R-L
Random	12.03	1.32	11.76
Ext. Oracle	42.84	16.86	39.20
TextRank	16.27	2.69	15.41
PGNet	28.74	5.98	25.13
BART	29.20	6.37	25.49
PGNet*	31.37	8.47	27.08
BART*	31.74	8.53	28.21
HMNet*	32.29	8.67	28.17
PGNet†	31.52	8.69	27.63
BART†	32.18	8.48	28.56
HMNet†	36.06	11.36	31.27
Table 3: Experimental results on QMSum dataset. We use standard ROUGE F-1 score to evaluate different models. The models with * denotes they use the spans extracted by our Locator as the input and † indicates this Summarizer uses gold spans as the input.
On the contrary, our hierarchical ranking-based Locator always greatly exceeds the random score, which demonstrates that it can indeed extract more relevant spans in the meeting. Even if 1/6 of the original text is extracted, it can reach a 72.51 ROUGE-L recall score, which significantly reduces the burden of subsequent Summarizer processing long text while ensuring the amount of information.
5.3 Experimental Results on QMSum
For comparison, we introduce two basic baselines: Random and Extractive Oracle. We randomly sample 10 turns of the original meeting for each query as an answer and this is the Random baseline in Table 3. Besides, we implement the Extractive Oracle, which is a greedy algorithm for extracting the highest-scoring sentences, usually regarded as the the upper bound of the extractive method (Nallapati et al., 2017). An unsupervised method, TextRank is also included in our experiment. We treat each turn as a node and add a query node to fully connect all nodes. Finally, the 10 turns with the highest scores are selected as the summary.
Table 3 shows that the performance of three typical neural network models is significantly better than Random and TextRank. When equipped with our Locator, both PGNet and BART have brought evident performance improvements (PGNet: 28.74 -> 31.37 R-1, BART: 29.20 -> 31.74 R-1). Compared to PGNet*, the advantage of BART* lies in the ROUGE-L score (1.13 improvement), which indicates that it can generate more fluent sentences.
$^{5}$ https://github.com/pytorch/fairseq/tree/master/examples/bart
Datasets	Product	Academic	Committee	All
R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L
Pro.	35.43	10.99	31.37	22.59	3.41	19.82	24.48	3.84	21.94	30.02	7.58	26.62
Aca.	27.19	4.86	24.09	26.69	4.32	22.58	27.84	4.29	25.10	27.22	4.59	24.02
Com.	25.56	3.48	22.17	23.91	2.99	20.23	32.52	6.98	27.71	27.07	4.28	23.21
All	34.93	10.78	31.21	26.47	5.05	23.01	31.16	6.47	27.52	32.18	8.48	28.56
The current state-of-the-art meeting summarization model HMNet achieves the best performance, which may be attributed to its cross-domain pretraining process making HMNet more familiar with the style of meeting transcripts.
In addition, we also use the gold text spans as the input of different models to measure the performance loss caused by Locator. Surprisingly, for models (PGNet and BART) that need to truncate the input text, although Locator is an approximate solution, the models equipped with it can achieve comparable results with the models based on gold span inputs. Therefore, in this case, our two-stage pipeline is a simple but effective method in the meeting domain. However, for some models (HM-Net) that use a hierarchical structure to process long text, inputting gold text spans can still bring huge performance improvements.
5.4 Experiments on Different Domains
In addition, we also conduct multi-domain and cross-domain experiments. First, we perform indomain and out-domain tests in the three domains of QMSum dataset. In Table 4, we can conclude that there are obvious differences between these three domains. For instance, the models trained on the Academic and Committee domains perform poorly when tested directly on the Product domain, with only the ROUGE-L scores of 24.09 and 22.17 respectively. However, the model trained on the single domain of Product can achieve a ROUGE-L score of 31.37, which illustrates although these domains are all in the form of meeting transcript, they still have visible domain bias.
On the other hand, when we train all the domains together, we can obtain a robust summarization model. Compared with models trained on a single domain, models trained on QMSum can always achieve comparable results. In the Academic domain, the model with multi-domain train
Table 4: Multi-domain and cross-domain summarization experiments. Each row represents the training set, and each column represents the test set. The gray cells denote the best result on the dataset in this column. We use BART† for these experiments and use standard ROUGE F-1 score to evaluate the model performance.
Opin.	Inter.	Con./Dec.	Reason	Overall
Num.	22	40	19	7	12
Diff. 1	2.2	1.6	1.7	2.4	2.0
Diff. 2	1.9	2.1	2.3	2.2	1.6
R-L	27.0	30.1	26.1	24.9	30.9
Table 5: The number, human evaluation and model performance of different types of queries. Diff. 1 represents the difficulty of locating relevant information and Diff. 2 represents the difficulty of organizing content.
ing can even get higher ROUGE-2 (5.05 vs 4.32) and ROUGE-L (23.01 vs 22.58) scores. These results show that the multi-domain setting in meeting summarization task is apparently necessary and meaningful. Meeting transcripts cover various fields, making the transfer of models particularly difficult. Therefore, we need to introduce multidomain training to make the model more robust, so it can be applied to more practical scenarios.
In this section, we conduct comprehensive analysis of query types and errors in the model output.
6.1 Analysis of Query Types
We manually divide the query in QMSum into five aspects: personal opinion, multi-person interaction, conclusion or decision, reason, and overall content. For example, "Summarize the whole meeting." requires a summary of the overall content and "Why did A disagree with B?" requires a summary of some reasons. The questions we are concerned about are: what is the distribution of different types of queries in QMSum? Are there differences in the difficulty of different types of queries?
To figure out the above issues, we randomly sample 100 queries from the test set, count the number of each type, and score the difficulty of each query. Table 5 illustrates that answering  $40\%$  of queries re
requires summarizing the interaction of multiple people, and the queries that focus on personal opinions and different aspects of conclusions or decisions account for almost  $20\%$  each. Besides, queries about a specific reason are less frequent in the meetings.
We also perform a human evaluation of the difficulty of various query types. For each query, the relevant text spans and query-summary pair are shown to annotators. Annotators are asked to score the difficulty of this query in two dimensions: 1) the difficulty of locating relevant information in the original text; 2) the difficulty of organizing content to form a summary. For each dimension, they can choose an integer between 1 and 3 as the score, where 1 means easy and 3 means difficult.
As we can see from Table 5, query about reasons is the most difficult to locate key information in related paragraphs, and this type of query is also challenging to organize and summarize reasonably. Queries about multi-person interaction and overall content are relatively easy under human evaluation scores. The relevant paragraphs of the former contain multi-person conversations, which are usually redundant, so the effective information is easier to find; the latter only needs to organize the statements in the chronological order of the meeting to write a summary, so it has the lowest Diff. 2 score. The model performance also confirms this point, BART can get more than 30 R-L score on these two types of queries, but performs poorly on the rest. Therefore, the remaining three types of queries in QMSum are still very challenging even for powerful pre-trained models, and further research is urgently needed to change this situation.
Although ROUGE score can measure the degree of overlap between the generated summary and the gold summary, it cannot reflect the factual consistency between them or the relevance between the predicted summary and the query. Therefore, in order to better understand the model performance and the difficulty of the proposed task, we sample 100 generated summaries for error analysis. Specifically, we ask 10 graduate students to do error analysis on the sampled summaries. Each summary is viewed by two people. They discuss and agree on whether the sample is consistent with the original facts and whether it is related to the query.
According to Cao et al. (2018), nearly  $30\%$  of summaries generated by strong neural models con
tain factual errors. This problem is even more serious on QMSum: we find inconsistent facts in  $74\%$  of the samples, which may be because the existing models are not good at generating multi-granularity summaries. Although BART can achieve state-of-the-art performance in the single-document summarization task, it does not seem to be able to truly understand the different aspects of the meeting, thus create factual errors. What's worse,  $31\%$  summaries are completely unrelated to the given query. This not only encourages us to design more powerful models or introduce more prior knowledge to overcome this challenge, but also shows better metrics are needed to evaluate model performance in generating multi-granularity summaries.
We propose a new benchmark, QMSum, for query-based meeting summarization task. We build a locate-then-summarize pipeline as a baseline and further investigate variants of our model with different Locators and Summarizers, adopt different training settings including cross-domain and multidomain experiments to evaluate generalizability, and analyze the task difficulty with respect to query types. The new task and benchmark leave several open research directions to explore: 1) how to process the long meeting discourses; 2) how to make a meeting summarization model generalize well; 3) how to generate summaries consistent with both meeting transcripts and queries. 4) how to reduce the annotation cost for meeting summarization.
The Language, Information, and Learning lab at Yale LILY) would like to acknowledge the research grant from Microsoft Research. We would also like to thank the annotators for their hard work and anonymous reviewers for their valuable comments.
Ethics Consideration
We propose a novel query-based meeting summarization task, accompanying with a high-quality dataset QMSum. Since the paper involves a new dataset and NLP application, this section is further divided into the following two parts.
7.1 New Dataset
Intellectual Property and Privacy Rights Collecting user data touches on the intellectual prop
erty and privacy rights of the original authors: both of the collected meeting transcripts and recruited annotators. We ensure that the dataset construction process is consistent with the intellectual property and privacy rights of the original authors of the meetings. All the meeting transcripts we collected are public and open to use according to the regulation  $^{6789}$ . The annotation process is consistent with the intellectual property and privacy rights of the recruited annotators as well.
Compensation for Annotators We estimated the time for annotating one meeting is around 1 - 2 hours. Therefore, we paid annotators around  $14 for each product and academic meeting and$ 28 for each committee meeting. To further encourage annotators to work on annotations, we proposed bonus mechanism: the bonus of each of the 5th to 8th meetings would be $4; the bonus of each of the 9th to 12th meetings would be $5, and so on. Some of the authors also did annotations and they were paid as well.
Steps Taken to Avoid Potential Problems The most possible problems which may exist in the dataset is bias problem and the inconsistency among queries, annotated summaries and original meeting contents. With regard to bias problem, we find that product meeting dataset rarely contains any explicit gender information, but annotators still tended to use 'he' as pronoun. To avoid the gender bias caused by the usage of pronouns, we required annotators to replace pronouns with speaker information like 'Project Manager', 'Marketing' to avoid the problem. Also, when designing queries based on query schema list, we found that annotators usually used the same query schema, which might lead to bias towards a certain type of query. Therefore, we asked the annotators to use different schemas as much as possible. For the inconsistency problem, each annotation step was strictly under supervision by 'experts' which are good at annotation and could be responsible for reviewing.
7.2 NLP Applications
Intended Use The query-based meeting summarization application is aiming at summarizing meetings according to queries from users. We could foresee that the trained model could be applied in companies to further improve the efficiency of workers, and help the staff comprehensively understand the meeting contents. The annotated QM-Sum dataset could be used as a benchmark for researchers to study how to improve the performance of summarization on such long texts and how to make models more generalizable on the meetings of different domains.
Failure Mode The current baseline models still tend to generate ungrammatical and factually inconsistent summaries. If a trained baseline model was directly applied in companies, the misinformation would negatively affect comprehension and further decision making. Further efforts are needed to generate high-quality summaries which are fluent and faithful to the meeting transcripts and queries.
Bias Training and test data are often biased in ways that limit system accuracy on domains with small data size or new domains, potentially causing distribution mismatch issues. In the data collection process, we control for the gender bias caused by pronouns such as 'he' and 'she' as much as possible. Also, we attempt to control the bias towards a certain type of query schema by requiring annotators to use diverse schemas as much as possible. However, we admit that there might be other types of bias, such as political bias in committee meetings. Thus, the summarization models trained on the dataset might be biased as well. and We will include warnings in our dataset.
Misuse Potential We emphasize that the application should be used with careful consideration, since the generated summaries are not reliable enough. It is necessary for researchers to develop better models to improve the quality of summaries. Besides, if the model is trained on internal meeting data, with the consideration of intellectual property and privacy rights, the trained model should be used under strict supervision.
Collecting Data from Users Future projects have to be aware of the fact that some meeting transcripts are intended for internal use only. Thus, researchers should be aware of the privacy issues about meeting data before training the model.
http://groups.inf.ed.ac.uk/ami/  
corpus/licenses.shtml  
http://groups.inf.ed.ac.uk/ami/icsi/  
license.shtml  
https://senedd.wales/en/help/  
our-information/Pages/Open-data.aspx  
https://www.ourcommons.ca/en/  
important-notices
Chenxin An, Ming Zhong, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2021. Enhancing scientific papers summarization with citation graph.
Tal Baumel, Matan Eyal, and Michael Elhadad. 2018. Query focused abstractive summarization: Incorporating query relevance, multi-document coverage, and summary length constraints into seq2seq models. arXiv preprint arXiv:1801.07704.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 4784-4791. AAAI Press.
Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, et al. 2005. The am meeting corpus: A pre-announcement. In International workshop on machine learning for multimodal interaction, pages 28-39. Springer.
Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 1662-1675.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686, Melbourne, Australia. Association for Computational Linguistics.
Yiran Chen, Pengfei Liu, Ming Zhong, Zi-Yi Dou, Dan- qing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. An empirical study of cross-dataset evaluation for neural summarization systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020, pages 3679- 3691. Association for Computational Linguistics.
Yun-Nung Chen and Florian Metze. 2012. Integrating intra-speaker topic modeling and temporal-based inter-speaker topic modeling in random walk for improved multi-party meeting summarization. In Thirteenth Annual Conference of the International Speech Communication Association.
Sangwoo Cho, Logan Lebanoff, Hassan Foroosh, and Fei Liu. 2019. Improving the similarity measure of determinantal point processes for extractive multi-document summarization. In Proceedings of the
57th Annual Meeting of the Association for Computational Linguistics, pages 1027-1038.
Sumit Chopra, Michael Auli, and Alexander M Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93-98.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 615-621.
H. Dang. 2005. Overview of duc 2005.
Hoa Trang Dang. 2006. DUC 2005: Evaluation of question-focused summarization systems. In Proceedings of the Workshop on Task-Focused Summarization and Question Answering, pages 48-55, Sydney, Australia. Association for Computational Linguistics.
Hal Daumé III and Daniel Marcu. 2006. Bayesian query-focused summarization. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 305-312, Sydney, Australia. Association for Computational Linguistics.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 708-719, New Orleans, Louisiana. Association for Computational Linguistics.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693-1701.
Tatsuya Ishigaki, Hen-Hsen Huang, Hiroya Takamura, Hsin-Hsi Chen, and Manabu Okumura. 2020. Neural query-biased abstractive summarization using copying mechanism. In European Conference on Information Retrieval, pages 174-181. Springer.
Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke, et al. 2003. The icsi meeting corpus. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP'03), volume 1, pages I-I. IEEE.