Venue: NAACL 2021

# QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization

Ming Zhong\({}^{\dagger}\)  Da Yin\({}^{\clubsuit}\)  Tao Yu\({}^{\ddagger}\) Ahmad Zaidi\({}^{\sharp}\)

**Mutethia Mutuma\({}^{\ddagger}\) Rahul Jha\({}^{\lx@paragraphsign}\) Ahmed Hassan Awadallah\({}^{\lx@paragraphsign}\) Asli Celikyilmaz\({}^{\lx@paragraphsign}\) Yang Liu\({}^{\lx@sectionsign}\) Xipeng Qiu\({}^{\dagger}\) Dragomir Radev\({}^{\ddagger}\) Fudan University \({}^{\clubsuit}\)University of California, Los Angeles \({}^{\ddagger}\)Yale University \({}^{\lx@paragraphsign}\)Microsoft Research \({}^{\lx@paragraphsign}\)Microsoft Cognitive Services Research mzhong18@fudan.edu.cn da.yin@cs.ucla.edu {tao.yu, dragomir.radev}@yale.edu

###### Abstract

Meetings are a key component of human collaboration. As increasing numbers of meetings are recorded and transcribed, meeting summaries have become essential to remind those who may or may not have attended the meetings about the key decisions made and the tasks to be completed. However, it is hard to create a single short summary that covers all the content of a long meeting involving multiple people and topics. In order to satisfy the needs of different types of users, we define a new query-based multi-domain meeting summarization task, where models have to select and summarize relevant spans of meetings in response to a query, and we introduce QMSum, a new benchmark for this task. QMSum consists of 1,808 query-summary pairs over 232 meetings in multiple domains. Besides, we investigate a _locate-then-summarize_ method and evaluate a set of strong summarization baselines on the task. Experimental results and manual analysis reveal that QMSum presents significant challenges in long meeting summarization for future research. Dataset is available at [https://github.com/Yale-LILY/QMSum](https://github.com/Yale-LILY/QMSum).

+
Footnote †: These two authors contributed equally. The order of authorship decided by the flip of a coin.


standing out, while others may be more interested in what other attendees thought about different elements of the design. It is challenging to compress or compose a short summary that contains all the salient information. Alternatively, summarization systems should adopt a more flexible and interactive approach that allows people to express their interests and caters to their diverse intents when generating summaries [15, 20, 34].

With comprehensive consideration of the multi-granularity meeting contents, we propose a new task, query-based meeting summarization. To enable research in this area, we also create a high-quality multi-domain summarization dataset. In this task, as shown in Figure 1, given a query and a meeting transcript, a model is required to generate the corresponding summary. The query-based approach is a flexible setup that enables the system to satisfy different intents and different levels of granularity. Besides the annotated queries and corresponding gold summaries at different levels of granularity, our new dataset contains a rich set of annotations that include the main topics of each meeting and the ranges of relevant text spans for the annotated topics and each query. We adopt a hierarchical annotation structure that could not only assist people to find information faster, but also strengthen the models' summarization capacity.

In this paper, we employ a two-stage meeting summarization approach: _locate-then-summarize_. Specifically, given a query, a model called _Locator_ is used to locate the relevant utterances in the meeting transcripts, and then these extracted spans are used as an input to another model called _Summarizer_ to generate a query-based summary. We present and evaluate several strong baselines based on state-of-the-art summarization models on QMSum. Our results and analysis from different perspectives reveal that the existing models struggle in solving this task, highlighting the challenges the models face when generating query-based meeting summaries. We are releasing our dataset and baselines to support additional research in query-focused meeting summarization.

Overall, our contributions are listed as follows: 1) We propose a new task, query-based multi-domain meeting summarization, and build a new benchmark QMSum with a hierarchical annotation structure. 2) We design a _locate-then-summarize_ model and conduct comprehensive experiments on its strong variants and different training settings. 3) By human evaluation, we further pose the challenges of the new task, including the impact of different query types and factuality errors.

## 2 Related Work

### Text Summarization

Most prior work in text summarization [68, 79, 80, 13, 16, 14, 15, 17, 18, 19, 20, 21, 22] investigate how to generate better summaries on news article data, such as CNN/DailyMail [16], Newsroom [13], etc. Scientific paper summarization is another important branch [14, 15, 16]. Our paper mainly focuses on meeting summarization, a more challenging task compared to news summarization. With the burst of demand for meeting summarization, this task attracts more and more interests from academia [15, 16, 17, 18] and becomes an emerging branch of text summarization area.

### Query-based Summarization

Query-based summarization aims to generate a brief summary according to a source document and a given query. There are works studying this task [16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 111, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 222, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 311, 320, 321, 333, 341, 342, 343, 344, 345, 346, 347, 348, 359, 360, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 422, 443, 445, 446, 447, 448, 450, 409, 421, 432, 446, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 51, 52, 53, 54, 55, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 111, 109, 110, 111, 113, 114, 115, 116, 117, 118, 119, 120, 121, 123, 124, 125, 126, 127, 128, 129, 130, 141, 150, 151, 161, 172, 173, 174, 175, 176, 177, 178, 179, 180, 190, 191, 193, 194, 195, 196, 197, 198, 199, 200, 203, 204, 205, 206, 207, 208, 209, 210, 209, 211, 223, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 251, 252, 254, 255, 256, 257, 258, 259, 261, 270, 282, 293, 294, 295, 296, 297, 298, 299, 300, 310, 320, 321, 323, 324, 326, 327, 328, 329, 333, 343, 344, 345, 346, 347, 348, 359, 360, 371, 383, 390, 392, 301, 393, 394, 395, 396, 397, 398, 399, 400, 411, 423, 445, 447, 450, 461, 462, 473, 478, 489, 490, 410, 424, 493, 494, 495, 496, 497, 498, 499, 500, 51, 52, 53, 54, 55, 56, 57, 5 

and Cardie (2013) attempt to make use of decisions, action items and progress to generate the whole meeting summaries. Oya et al. (2014) leverages the relationship between summaries and the meeting transcripts to extract templates and generate summaries with the guidance of the templates. Shang et al. (2018) utilize multi-sentence compression techniques to generate summaries under an unsupervised setting. Li et al. (2019) attempt to incorporate multi-modal information to facilitate the meeting summarization. Zhu et al. (2020) propose a model which builds a hierarchical structure on word-level and turn-level information and uses news summary data to alleviate the inadequacy of meeting data.

Unlike previous works, instead of merely generating summaries for the complete meeting, we propose a novel task where we focus on summarizing multi-granularity contents which cater to different people's need for the entire meetings, and help people comprehensively understand meetings.

## 3 Data Construction

In this section, we show how we collected meeting data from three different domains: academic meetings, product meetings, and committee meetings. In addition, we show how we annotated the three types of meeting data while ensuring annotation quality for query-based meeting summarization.

### Data Collection

We introduce the three types of meetings that we used to annotate query-summary pairs.

Product MeetingsAMI1Carletta et al. (2005) is a dataset of meetings about product design in an industrial setting. It consists of 137 meetings about how to design a new remote control, from kick-off to completion over the course of a day. It contains meeting transcripts and their corresponding meeting summaries.

Footnote 1: [http://groups.inf.ed.ac.uk/ami/download/](http://groups.inf.ed.ac.uk/ami/download/)

Academic MeetingsICSI2Janin et al. (2003) dataset is an academic meeting dataset composed of 59 weekly group meetings at International Computer Science Institute (ICSI) in Berkeley, and their summaries. Different from AMI, the contents of ICSI meetings are specific to the discussions about research among students.

Committee MeetingsParliamentary committee meeting is another important domain of meetings. These meetings focus on the formal discussions on a wide range of issues (e.g., the reform of the education system, public health, etc.) Also, committee meetings are publicly available, which enables us to access large quantities of meetings. We include 25 committee meetings of the Welsh Parliament3 and 11 from the Parliament of Canada4 in our dataset.

Footnote 2: [http://groups.inf.ed.ac.uk/ami/icsi/index.shtml](http://groups.inf.ed.ac.uk/ami/icsi/index.shtml)

### Annotation Pipeline

After collecting meeting transcripts, we recruited annotators and required them to annotate by following annotation instruction. As illustrated in Figure 2, the annotation process is composed by three stages: topic segmentation, query generation, and query-based summarization.

Topic SegmentationMeeting transcripts are usually long and contain discussions about multiple topics. To assist further annotations, we asked annotators to write down the main topics discussed in the meetings, and their relevant text spans, which makes the meeting structure clear. As shown in Figure 2, "scope of the project and team building" is one of the annotated main topics, and its relevant text spans of the topic are (_Turn_ 25 - 50, _Turn_ 73 - 89). More details are listed in Appendix A.2.1.

Query GenerationTowards the query-based task, we further asked annotators to design queries by themselves. To cater to the need for multi-granularity contents, we categorized two types of queries: queries related to general information (e.g., the contents of whole meetings, etc.) are called _general queries_; queries focusing on relatively detailed information (e.g., the discussion about certain topics, etc.) are called _specific queries_.

To alleviate the influence of extremely hard queries and focus on the evaluation of query-based summarization capacity, rather than designing queries in an unconstrained way, we asked annotators to generate queries according to the schema. Details of the query schema list are shown in Appendix A.1. The list consists of important facets people might be interested in, including overall contents of discussions, speakers' opinions, the reasons why a speaker proposed an idea, etc., which cover the most common queries over meetings involving multiple people discussing several topics.

 

To query multi-granularity meeting contents, we further divided the query schema list into general and specific ones, and asked annotators to design queries towards general and specific meeting contents, respectively. In terms of _general query generation_, the annotators were asked to design 1 - 2 general queries according to the general schema list. For _specific query generation_, annotators were asked to first select 2 - 4 main topics and their relevant text spans, and then design around 3 specific queries based on the specific schema list for each main topic. To ensure the task to be _summarization_ instead of _question answering_, we asked annotators to design queries of which the relevant text spans are more than 10 turns or 200 words. Therefore, our proposed task would differ from _question answering_ tasks where models merely need to extract phrases or generate answers based on short text spans, and focus on how to _summarize_ based on large stretches of texts. Additional details are in Appendix A.2.2.

Query-based SummarizationAccording to the designed queries and meeting transcripts, annotators were asked to do faithful summarization. Being accorded with the meeting transcripts and queries is the most important criterion. We also required annotators to write informative summarization. For example, they could add more details about the reasons why the group/committee made such decisions, and which important ideas the group/committee members proposed, etc. Besides, the annotated summaries should be abstractive, fluent and concise. We set word limits for the answers of general queries (50 - 150 words) and specific queries (20 - 100 words) to keep conciseness. More details are shown in Appendix A.2.3.

In the end, we organize all the meeting data after accomplishing the three annotation stages. Detailed annotations of one product meeting and one committee meeting are shown in Appendix A.4. Each meeting transcript is accompanied with annotated main topics, queries, their corresponding summaries, and relevant text span information.

### Additional Details of Annotation Process

This section describes how we recruited annotators and how we review the annotations in detail.

Annotator RecruitmentTo guarantee annotation quality given the complexity of the task, instead of employing tasks on Amazon Mechanical Turker, we anonymously recruited undergraduate students who are fluent in English. The annotation team consists of 2 native speakers and 10 non-native speakers majoring in English literature.

Annotation ReviewTo help the annotators fully comprehend the instruction, annotators were

Figure 2: Overall annotation pipeline. It is divided into three stages: Stage 1 is to annotate main topics and their relevant text spans; Stage 2 is to generate queries based on query schema lists; Stage 3 is to annotate the summaries according to the queries. The pipeline was implemented upon the collected meetings of multiple domains.

 

trained in a _pre-annotation_ process. Annotations were reviewed across all stages in our data collection process by expert of this annotation task. More details of review standards could be found in Appendix A.3.

### Dataset Statistics and Comparison

Statistics of the final QMSum dataset is shown in Table 1. There are several advantages of QMSum dataset, compared with the previous datasets.

Number of Meetings and SummariesQMSum includes 232 meetings, which is the largest meeting summarization dataset to our best knowledge. For each query, there is a manual annotation of corresponding text span in the original meeting, so there are a total of 1,808 question-summary pairs in QMSum. Following the previous work, we randomly select about 15% of the meetings as the validation set, and another 15% as the test set.

BrieftyThe average length of summaries in QMSum 69.6 is much shorter than that of previous AMI and ICSI datasets. It is because our dataset also focuses on specific contents of the meetings, and the length of their corresponding summaries would not be long. It leaves a challenge about how to precisely capture the related information and compress it into a brief summary.

Multi-domain SettingPrevious datasets are specified to one domain. However, the model trained on the summarization data of a single domain usually has poor generalization ability Wang et al. (2019); Zhong et al. (2019b); Chen et al. (2020). Therefore, QMSum contains meetings across multiple domains: Product, Academic and Committee meetings. We expect that our dataset could provide a venue to evaluate the model's generalization ability on meetings of different domains and help create more robust models.

## 4 Method

In this section, we first define the task of query-based meeting summarization, then describe our two-stage _locate-then-summarize_ solution in detail.

### Problem Formulation

Existing meeting summarization methods define the task as a sequence-to-sequence problem. Specifically, each meeting transcript \(X=(x_{1},x_{2},\cdots,x_{n})\) consists of \(n\) turns, and each turn \(x_{i}\) represents the utterance \(u_{i}\) and its speaker \(s_{i}\), that is, \(x_{i}=(u_{i},s_{i})\). Additionally, each utterance contains \(l_{i}\) words \(u_{i}=(w_{1},\cdots,w_{l_{i}})\). The object is to generate a target summary \(Y=(y_{1},y_{2},\cdots,y_{m})\) by modeling the conditional distribution \(p(y_{1},y_{2},\cdots,y_{m}|(u_{1},s_{1}),\cdots,(u_{n},s_{n}))\).

However, meetings are usually long conversations involving multiple topics and including important decisions on many different matters, so it is necessary and practical to use queries to summarize a certain part of the meeting. Formally, we introduce a query \(Q=(w_{1},\cdots,w_{|Q|})\) for meeting summarization task, the objective is to generate a summary \(Y\) by modeling \(p(y_{1},y_{2},\cdots,y_{m}|Q,(u_{1},s_{1}),\cdots,(u_{n},s_{n}))\).

### Locator

In our two-stage pipeline, the first step requires a model to locate the relevant text spans in the meeting according to the queries, and we call this model a Locator. The reason why we need a Locator here is, most existing abstractive models cannot process long texts such as meeting transcripts. So we need to extract shorter, query-related paragraphs as input to the following Summarizer.

We mainly utilize two methods to instantiate our Locator: Pointer Network Vinyals et al. (2015) and a hierarchical ranking-based model. Pointer

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
* & 97 / 20 / 20 \\ ICSI & 59 & 819.0 & 13317.3 & 488.5 & 6.3 &
- & 41 / 9 / 9 \\ \hline Product & 137 & 535.6 & 6007.7 & 70.5 & 4.0 & 7.2 & 690 / 145 / 151 \\ Academic & 59 & 819.0 & 13317.3 & 53.7 & 6.3 & 6.3 & 259 / 54 / 56 \\ Committee & 36 & 207.7 & 13761.9 & 80.5 & 34.1 & 12.6 & 308 / 73 / 72 \\ All & 232 & 556.8 & 9069.8 & 69.6 & 9.2 & 7.8 & 1,257 / 272 / 279 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of meeting summarization datasets. The top half of the table is the existing meeting datasets, and the bottom half is the statistics of QMSum. Because a meeting may have multiple queries, #Pairs here means how many query-summary pairs are contained in the train / valid / test set.
 

Network has achieved widespread success in extractive QA tasks (Wang and Jiang, 2017). For each question, it will point to the <start, end> pair in the source document, and the span is the predicted answer. Specific to our task, Pointer Network will point to the start turn and the end turn for each query. It is worth noting that one query can correspond to multiple spans in our dataset, so we always extract three spans as the corresponding text for each query when we use Pointer Network as Locator in the experiments.

In addition, we design a hierarchical ranking-based model structure as the Locator. As shown in Figure 3, we first input the tokens in each turn to a feature-based BERT to obtain the word embedding, where feature-based means we fix the parameters of BERT, so it is actually an embedding layer. Next, CNN (Kim, 2014) is applied as a turn-level encoder to capture the local features such as bigram, trigram and so on in each turn. Here we do not use Transformer because previous work (Kedzie et al., 2018) shows that this component does not matter too much for the final performance. We combine different features to represent the utterance \({\bf u}_{i}\) in each turn, and concatenate the speaker embedding \({\bf s}_{i}\) as the turn-level representation: \({\bf x}_{\bf i}=[{\bf u}_{\bf i};{\bf s}_{\bf i}]\), where \([;]\) denotes concatenation and \({\bf s}_{i}\) is a vector randomly initialized to represent the speaking style of meeting participants.

Then these turn representations will be contextualized by a document-level Transformer (Vaswani et al., 2017) encoder. Next, we introduce query embedding \({\bf q}\) which is obtained by a CNN (shared parameters with CNN in turn-level encoder) and use MLP to score each turn. We use binary cross-entropy loss to train our Locator. Finally, turns with the highest scores are selected as the relevant text spans of each query and will be inputted to the subsequent Summarizer.

### Summarizer

Given the relevant paragraphs, our goal in the second stage is to summarize the selected text spans based on the query. We instantiate our Summarizer with the current powerful abstractive models to explore whether the query-based meeting summarization task on our dataset is challenging. To be more specific, we choose the following three models:

**Pointer-Generator Network**(See et al., 2017) is a popular sequence-to-sequence model with copy mechanism and coverage loss, and it acts as a baseline system in many generation tasks. The input to Pointer-Generator Network (PGNet) is: "<<S>> Query 
 Relevant Text Spans 
".

**BART**(Lewis et al., 2020) is a denoising pre-trained model for language generation, translation and comprehension. It has achieved new state-of-the-art results on many generation tasks, including summarization and abstractive question answering. The input to BART is the same as PGNet.

**HMNet**(Zhu et al., 2020) is the state-of-the-art meeting summarization model. It contains a hierarchical structure to process long meeting transcripts and a role vector to depict the difference among speakers. Besides, a cross-domain pretraining process is also included in this strong model. We add a turn representing the query at the beginning of the meeting as the input of HMNet.

## 5 Experiments

In this section, we introduce the implementation details, effectiveness of Locator, experimental results and multi-domain experiments on QMSum.

### Implementation Details

For our ranking-based Locator, the dimension of speaking embedding is 128 and the dimension of turn and query embedding is 512. Notably, we find that removing Transformers in Locator has little impact on performance, so the Locator without Transformer is used in all the experiments. To reduce the burden of the abstractive models, we utilize Locator to extract 1/6 of the original text and input them to Summarizer. The hyperparameters used by PGNet and HMNet are consistent with the original paper. Due to the limitation of computing resources, we use the base version of pre-trained models (including feature-based BERT and BART)

Figure 3: Hierarchical ranking-based locator structure.

 

in this paper. We use fairseq library5 to implement BART model. For PGNet and BART, we truncate the input text to 2,048 tokens, and remove the turns whose lengths are less than 5. All results reported in this paper are averages of three runs.

Footnote 5: [https://github.com/pytorch/fairseq/tree/master/examples/bart](https://github.com/pytorch/fairseq/tree/master/examples/bart)

### Effectiveness of Locator

First, we need to verify the effectiveness of the Locator to ensure that it can extract spans related to the query. Instead of the accuracy of capturing relevant text spans, we focus on the extent of overlap between the selected text spans and the gold relevant text spans. It is because whether the summarization process is built on similar contexts with references or not is essential for Summarizer. Therefore, we use ROUGE-L recall to evaluate the performance of different models under the setting of extracting the same number of turns.

We introduce two additional baselines: Random and Similarity. The former refers to randomly extracting a fixed number of turns from the meeting content, while the latter denotes that we obtain turn embedding and query embedding through a feature-based BERT, and then extract the most similar turns by cosine similarity. As shown in Table 2, because there are usually a large number of repeated conversations in the meetings, Random can get a good ROUGE-L recall score, which can be used as a baseline to measure the performance of the model. Similarity performs badly, even worse than Random, which may be due to the great difference in style between the BERT pre-trained corpus and meeting transcripts. Pointer Network is only slightly better than Random. We think this is because in the text of with an average of more than 500 turns, only three <start, end> pairs are given as supervision signals, which is not very informative and therefore is not conducive to model learning.

On the contrary, our hierarchical ranking-based Locator always greatly exceeds the random score, which demonstrates that it can indeed extract more relevant spans in the meeting. Even if 1/6 of the original text is extracted, it can reach a 72.51 ROUGE-L recall score, which significantly reduces the burden of subsequent Summarizer processing long text while ensuring the amount of information.

### Experimental Results on QMSum

For comparison, we introduce two basic baselines: Random and Extractive Oracle. We randomly sample 10 turns of the original meeting for each query as an answer and this is the Random baseline in Table 3. Besides, we implement the Extractive Oracle, which is a greedy algorithm for extracting the highest-scoring sentences, usually regarded as the the upper bound of the extractive method Nallapati et al. (2017). An unsupervised method, TextRank is also included in our experiment. We treat each turn as a node and add a query node to fully connect all nodes. Finally, the 10 turns with the highest scores are selected as the summary.

Table 3 shows that the performance of three typical neural network models is significantly better than Random and TextRank. When equipped with our Locator, both PGNet and BART have brought evident performance improvements (PGNet: 28.74 ~> 31.37 R-1, BART: 29.20 ~> 31.74 R-1). Compared to PGNet\({}^{*}\), the advantage of BART\({}^{*}\) lies in the ROUGE-L score (1.13 improvement), which indicates that it can generate more fluent sentences.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{4}{c}{**Extracted Length**} \\
**Models** & \(\nicefrac{{1}}{{6}}\) & \(\nicefrac{{1}}{{5}}\) & \(\nicefrac{{1}}{{4}}\) & \(\nicefrac{{1}}{{3}}\) \\ \hline Random & 58.86 & 63.20 & 67.56 & 73.81 \\ Similarity & 55.97 & 59.24 & 63.45 & 70.12 \\ Pointer & 61.27 & 65.84 & 70.13 & 75.96 \\ Our Locator & **72.51** & **75.23** & **79.08** & **84.04** \\ \hline \hline \end{tabular}
\end{table}
Table 2: ROUGE-L Recall score between the predicted spans and the gold spans. \(\nicefrac{{1}}{{6}}\) means that the turns extracted by the model account for 1/6 of the original text.

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Models** & **R-1** & **R-2** & **R-L** \\ \hline Random & 12.03 & 1.32 & 11.76 \\ Ext. Oracle & 42.84 & 16.86 & 39.20 \\ \hline TextRank & 16.27 & 2.69 & 15.41 \\ PGNet & 28.74 & 5.98 & 25.13 \\ BART & 29.20 & 6.37 & 25.49 \\ PGNet\({}^{*}\) & 31.37 & 8.47 & 27.08 \\ BART\({}^{*}\) & 31.74 & 8.53 & **28.21** \\ HMNet\({}^{*}\) & **32.29** & **8.67** & 28.17 \\ \hline PGNet\({}^{\dagger}\) & 31.52 & 8.69 & 27.63 \\ BART\({}^{\dagger}\) & 32.18 & 8.48 & 28.56 \\ HMNet\({}^{\dagger}\) & **36.06** & **11.36** & **31.27** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Experimental results on QMSum dataset. We use standard ROUGE F-1 score to evaluate different models. The models with \({}^{*}\) denotes they use the spans extracted by our Locator as the input and \({}^{\dagger}\) indicates this Summarizer uses gold spans as the input.

 

The current state-of-the-art meeting summarization model HMNet achieves the best performance, which may be attributed to its cross-domain pretraining process making HMNet more familiar with the style of meeting transcripts.

In addition, we also use the gold text spans as the input of different models to measure the performance loss caused by Locator. Surprisingly, for models (PGNet and BART) that need to truncate the input text, although Locator is an approximate solution, the models equipped with it can achieve comparable results with the models based on gold span inputs. Therefore, in this case, our two-stage pipeline is a simple but effective method in the meeting domain. However, for some models (HMNet) that use a hierarchical structure to process long text, inputting gold text spans can still bring huge performance improvements.

### Experiments on Different Domains

In addition, we also conduct multi-domain and cross-domain experiments. First, we perform in-domain and out-domain tests in the three domains of QMSum dataset. In Table 4, we can conclude that there are obvious differences between these three domains. For instance, the models trained on the Academic and Committee domains perform poorly when tested directly on the Product domain, with only the ROUGE-L scores of 24.09 and 22.17 respectively. However, the model trained on the single domain of Product can achieve a ROUGE-L score of 31.37, which illustrates although these domains are all in the form of meeting transcript, they still have visible domain bias.

On the other hand, when we train all the domains together, we can obtain a robust summarization model. Compared with models trained on a single domain, models trained on QMSum can always achieve comparable results. In the Academic domain, the model with multi-domain training can even get higher ROUGE-2 (5.05 vs 4.32) and ROUGE-L (23.01 vs 22.58) scores. These results show that the multi-domain setting in meeting summarization task is apparently necessary and meaningful. Meeting transcripts cover various fields, making the transfer of models particularly difficult. Therefore, we need to introduce multi-domain training to make the model more robust, so it can be applied to more practical scenarios.

## 6 Analysis

In this section, we conduct comprehensive analysis of query types and errors in the model output.

### Analysis of Query Types

We manually divide the query in QMSum into five aspects: personal opinion, multi-person interaction, conclusion or decision, reason, and overall content. For example, "Summarize the whole meeting." requires a summary of the overall content and "Why did A disagree with B?" requires a summary of some reasons. The questions we are concerned about are: _what is the distribution of different types of queries in QMSum? Are there differences in the difficulty of different types of queries?_

To figure out the above issues, we randomly sample 100 queries from the test set, count the number of each type, and score the difficulty of each query. Table 5 illustrates that answering 40% of queries re

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**Product**} & \multicolumn{3}{c}{**Academic**} & \multicolumn{3}{c}{**Committee**} & \multicolumn{3}{c}{**All**} \\
**Datasets** & R-1 & R-2 & R-L & R-1 & R-2 & R-L & R-1 & R-2 & R-L & R-1 & R-2 & R-L \\ \hline
**Pro.** & 35.43 & 10.99 & 31.37 & 22.59 & 3.41 & 19.82 & 24.48 & 3.84 & 21.94 & 30.02 & 7.58 & 26.62 \\
**Aca.** & 27.19 & 4.86 & 24.09 & 26.69 & 4.32 & 22.58 & 27.84 & 4.29 & 25.10 & 27.22 & 4.59 & 24.02 \\
**Conv.** & 25.56 & 3.48 & 22.17 & 23.91 & 2.99 & 20.23 & 32.52 & 6.98 & 27.71 & 27.07 & 4.28 & 23.21 \\
**All** & 34.93 & 10.78 & 31.21 & 26.47 & 5.05 & 23.01 & 31.16 & 6.47 & 27.52 & 32.18 & 8.48 & 28.56 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Multi-domain and cross-domain summarization experiments. Each row represents the training set, and each column represents the test set. The gray cells denote the best result on the dataset in this column.We use BART\({}^{\dagger}\) for these experiments and use standard ROUGE F-1 score to evaluate the model performance.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & **Opin.** & **Inter.** & **Con./Dec.** & **Reason** & **Overall** \\ \hline Num. & 22 & 40 & 19 & 7 & 12 \\ Diff. 1 & 2.2 & 1.6 & 1.7 & 2.4 & 2.0 \\ Diff. 2 & 1.9 & 2.1 & 2.3 & 2.2 & 1.6 \\ R-L & 27.0 & 30.1 & 26.1 & 24.9 & 30.9 \\ \hline \hline \end{tabular}
\end{table}
Table 5: The number, human evaluation and model performance of different types of queries. Diff. 1 represents the difficulty of locating relevant information and Diff. 2 represents the difficulty of organizing content.

 

quires summarizing the interaction of multiple people, and the queries that focus on personal opinions and different aspects of conclusions or decisions account for almost 20% each. Besides, queries about a specific reason are less frequent in the meetings.

We also perform a human evaluation of the difficulty of various query types. For each query, the relevant text spans and query-summary pair are shown to annotators. Annotators are asked to score the difficulty of this query in two dimensions: 1) the difficulty of locating relevant information in the original text; 2) the difficulty of organizing content to form a summary. For each dimension, they can choose an integer between 1 and 3 as the score, where 1 means easy and 3 means difficult.

As we can see from Table 5, query about reasons is the most difficult to locate key information in related paragraphs, and this type of query is also challenging to organize and summarize reasonably. Queries about multi-person interaction and overall content are relatively easy under human evaluation scores. The relevant paragraphs of the former contain multi-person conversations, which are usually redundant, so the effective information is easier to find; the latter only needs to organize the statements in the chronological order of the meeting to write a summary, so it has the lowest Diff. 2 score. The model performance also confirms this point, BART can get more than 30 R-L score on these two types of queries, but performs poorly on the rest. Therefore, the remaining three types of queries in QMSum are still very challenging even for powerful pre-trained models, and further research is urgently needed to change this situation.

### Error Analysis

Although ROUGE score can measure the degree of overlap between the generated summary and the gold summary, it cannot reflect the factual consistency between them or the relevance between the predicted summary and the query. Therefore, in order to better understand the model performance and the difficulty of the proposed task, we sample 100 generated summaries for error analysis. Specifically, we ask 10 graduate students to do error analysis on the sampled summaries. Each summary is viewed by two people. They discuss and agree on whether the sample is consistent with the original facts and whether it is related to the query.

According to Cao et al. (2018), nearly 30% of summaries generated by strong neural models contain factual errors. This problem is even more serious on QMSum: we find inconsistent facts in 74% of the samples, which may be because the existing models are not good at generating multi-granularity summaries. Although BART can achieve state-of-the-art performance in the single-document summarization task, it does not seem to be able to truly understand the different aspects of the meeting, thus create factual errors. What's worse, 31% summaries are completely unrelated to the given query. This not only encourages us to design more powerful models or introduce more prior knowledge to overcome this challenge, but also shows better metrics are needed to evaluate model performance in generating multi-granularity summaries.

## 7 Conclusion

We propose a new benchmark, QMSum, for query-based meeting summarization task. We build a _locate-then-summarize_ pipeline as a baseline and further investigate variants of our model with different Locators and Summarizers, adopt different training settings including cross-domain and multi-domain experiments to evaluate generalizability, and analyze the task difficulty with respect to query types. The new task and benchmark leave several open research directions to explore: 1) how to process the long meeting discourses; 2) how to make a meeting summarization model generalize well; 3) how to generate summaries consistent with both meeting transcripts and queries. 4) how to reduce the annotation cost for meeting summarization.

## Acknowledgements

The Language, Information, and Learning lab at Yale LILY) would like to acknowledge the research grant from Microsoft Research. We would also like to thank the annotators for their hard work and anonymous reviewers for their valuable comments.

## Ethics Consideration

We propose a novel query-based meeting summarization task, accompanying with a high-quality dataset QMSum. Since the paper involves a new dataset and NLP application, this section is further divided into the following two parts.

### New Dataset

Intellectual Property and Privacy RightsCollecting user data touches on the intellectual prop 

erty and privacy rights of the original authors: both of the collected meeting transcripts and recruited annotators. We ensure that the dataset construction process is consistent with the intellectual property and privacy rights of the original authors of the meetings. All the meeting transcripts we collected are public and open to use according to the regulation 6789. The annotation process is consistent with the intellectual property and privacy rights of the recruited annotators as well.

Footnote 5: [http://groups.inf.ed.ac.uk/ami/corpus/license.shtml](http://groups.inf.ed.ac.uk/ami/corpus/license.shtml)

Compensation for AnnotatorsWe estimated the time for annotating one meeting is around 1 - 2 hours. Therefore, we paid annotators around SS14 for each product and academic meeting and SS28 for each committee meeting. To further encourage annotators to work on annotations, we proposed bonus mechanism: the bonus of each of the 5th to 8th meetings would be $4; the bonus of each of the 9th to 12th meetings would be SS5, and so on. Some of the authors also did annotations and they were paid as well.

Steps Taken to Avoid Potential ProblemsThe most possible problems which may exist in the dataset is bias problem and the inconsistency among queries, annotated summaries and original meeting contents. With regard to bias problem, we find that product meeting dataset rarely contains any explicit gender information, but annotators still tended to use 'he' as pronoun. To avoid the gender bias caused by the usage of pronouns, we required annotators to replace pronouns with speaker information like 'Project Manager', 'Marketing' to avoid the problem. Also, when designing queries based on query schema list, we found that annotators usually used the same query schema, which might lead to bias towards a certain type of query. Therefore, we asked the annotators to use different schemas as much as possible. For the inconsistency problem, each annotation step was strictly under supervision by 'experts' which are good at annotation and could be responsible for reviewing.

### NLP Applications

Intended UseThe query-based meeting summarization application is aiming at summarizing meetings according to queries from users. We could foresee that the trained model could be applied in companies to further improve the efficiency of workers, and help the staff comprehensively understand the meeting contents. The annotated QMSum dataset could be used as a benchmark for researchers to study how to improve the performance of summarization on such long texts and how to make models more generalizable on the meetings of different domains.

Failure ModeThe current baseline models still tend to generate ungrammatical and factually inconsistent summaries. If a trained baseline model was directly applied in companies, the misinformation would negatively affect comprehension and further decision making. Further efforts are needed to generate high-quality summaries which are fluent and faithful to the meeting transcripts and queries.

BiasTraining and test data are often biased in ways that limit system accuracy on domains with small data size or new domains, potentially causing distribution mismatch issues. In the data collection process, we control for the gender bias caused by pronouns such as 'he' and 'she' as much as possible. Also, we attempt to control the bias towards a certain type of query schema by requiring annotators to use diverse schemas as much as possible. However, we admit that there might be other types of bias, such as political bias in committee meetings. Thus, the summarization models trained on the dataset might be biased as well. and We will include warnings in our dataset.

Misuse PotentialWe emphasize that the application should be used with careful consideration, since the generated summaries are not reliable enough. It is necessary for researchers to develop better models to improve the quality of summaries. Besides, if the model is trained on internal meeting data, with the consideration of intellectual property and privacy rights, the trained model should be used under strict supervision.

Collecting Data from UsersFuture projects have to be aware of the fact that some meeting transcripts are intended for internal use only. Thus, researchers should be aware of the privacy issues about meeting data before training the model 



## References

* An et al. (2021) Chenxin An, Ming Zhong, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2021. Enhancing scientific papers summarization with citation graph.
* Baumel et al. (2018) Tal Baumel, Matan Eyal, and Michael Elhadad. 2018. Query focused abstractive summarization: Incorporating query relevance, multi-document coverage, and summary length constraints into seq2seq models. _arXiv preprint arXiv:1801.07704_.
* Cao et al. (2018) Ziqiang Cao, Furu Wei, Wenjie Li, and Sujan Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_, pages 4784-4791. AAAI Press.
* Carletta et al. (2005) Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, et al. 2005. The ami meeting corpus: A pre-announcement. In _International workshop on machine learning for multimodal interaction_, pages 28-39. Springer.
* Celikyilmaz et al. (2018) Asli Celikyilmaz, Antoine Bossellut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, volume 1, pages 1662-1675.
* Chen and Bansal (2018) Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 675-686, Melbourne, Australia. Association for Computational Linguistics.
* Chen et al. (2020) Yiran Chen, Pengfei Liu, Ming Zhong, Zi-Yi Dou, Danguing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. An empirical study of cross-dataset evaluation for neural summarization systems. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020_, pages 3679-3691. Association for Computational Linguistics.
* Chen and Metze (2012) Yun-Nung Chen and Florian Metze. 2012. Integrating intra-speaker topic modeling and temporal-based inter-speaker topic modeling in random walk for improved multi-party meeting summarization. In _Thirteenth Annual Conference of the International Speech Communication Association_.
* Cho et al. (2019) Sangwoo Cho, Logan Lebanoff, Hassan Foroosh, and Fei Liu. 2019. Improving the similarity measure of determinantal point processes for extractive multi-document summarization. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1027-1038.
* Chopra et al. (2016) Sumit Chopra, Michael Auli, and Alexander M Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 93-98.
* Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 615-621.
* Dang (2005) H. Dang. 2005. Overview of duc 2005.
* Dang (2006) Hoa Trang Dang. 2006. DUC 2005: Evaluation of question-focused summarization systems. In _Proceedings of the Workshop on Task-Focused Summarization and Question Answering_, pages 48-55, Sydney, Australia. Association for Computational Linguistics.
* Daume III and Marcu (2006) Hal Daume III and Daniel Marcu. 2006. Bayesian query-focused summarization. In _Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics_, pages 305-312, Sydney, Australia. Association for Computational Linguistics.
* Grusky et al. (2018) Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 708-719, New Orleans, Louisiana. Association for Computational Linguistics.
* Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In _Advances in neural information processing systems_, pages 1693-1701.
* Ishigaki et al. (2020) Tatsuya Ishigaki, Hen-Hsen Huang, Hiroya Takamura, Hsin-Hsi Chen, and Manabu Okumura. 2020. Neural query-biased abstractive summarization using copying mechanism. In _European Conference on Information Retrieval_, pages 174-181. Springer.
* Janin et al. (2003) Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke, et al. 2003. The lcsci meeting corpus. In _2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP'03)._, volume 1, pages I-I. IEEE

Ruipeng Jia, Yanan Cao, Hengzhu Tang, Fang Fang, Cong Cao, and Shi Wang. 2020. Neural extractive summarization with hierarchical attentive heterogeneous graph network. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3622-3631.
* Kedzie et al. (2018) Chris Kedzie, Kathleen McKeown, and Hal Daume III. 2018. Content selection in deep learning models of summarization. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1818-1828.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1746-1751.
* Koay et al. (2020) Jia Jin Koay, Alexander Roustai, Xiaojin Dai, Dillon Burns, Alec Kerrigan, and Fei Liu. 2020. How domain terminology affects meeting summarization performance. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 5689-5695, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Kulkarni et al. (2020) Sayali Kulkarni, Sheide Chammas, Wan Zhu, Fei Sha, and Eugene Ie. 2020. Aquamuse: Automatically generating datasets for query-based multi-document summarization. _arXiv preprint arXiv:2010.12694_.
* Lakkar et al. (2020) Md Tahmid Rahman Laskar, Enamol Hoque, and Jimmy Huang. 2020. Query focused abstractive summarization via incorporating query relevance and transfer learning with transformer models. In _Canadian Conference on Artificial Intelligence_, pages 342-348. Springer.
* Lebanoff et al. (2019) Logan Lebanoff, Kaiqiang Song, Franck Dernoncourt, Doo Soon Kim, Seokhwan Kim, Walter Chang, and Fei Liu. 2019. Scoring sentence singletons and pairs for abstractive summarization. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2175-2189.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 7871-7880. Association for Computational Linguistics.
* Li et al. (2019) Manling Li, Lingyu Zhang, Heng Ji, and Richard J. Radke. 2019. Keep meeting summaries on topic: Abstractive multi-modal meeting summarization. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2190-2196, Florence, Italy. Association for Computational Linguistics.
* Litvak and Vanetik (2017) Marina Litvak and Natalia Vanetik. 2017. Query-based summarization using MDL principle. In _Proceedings of the MultiLine 2017 Workshop on Summarization and Summary Evaluation Across Source Types and Genres_, pages 22-31, Valencia, Spain. Association for Computational Linguistics.
* Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3721-3731.
* Mehdad et al. (2013) Yashar Mehdad, Giuseppe Carenini, Frank Tompa, and Raymond T. Ng. 2013. Abstractive meeting summarization with entailment and fusion. In _Proceedings of the 14th European Workshop on Natural Language Generation_, pages 136-146, Sofia, Bulgaria. Association for Computational Linguistics.
* Droz et al. (2018) Joseph E. Droz, Joseph A. Allen, Dana C. Verhoeven, and Marissa L. Shuffler. 2018. Do we really need another meeting? the science of workplace meetings. _Current Directions in Psychological Science_, 27(6):484-491.
* Nallapati et al. (2017) Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In _Thirty-First AAAI Conference on Artificial Intelligence_.
* Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Ca glar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. _CoNLL 2016_, page 280.
* Nema et al. (2017) Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven attention model for query-based abstractive summarization. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1063-1072, Vancouver, Canada. Association for Computational Linguistics.
* Otterbacher et al. (2009) Jahna Otterbacher, Gunes Erkan, and Dragomir Radev. 2009. Biased lexrank: Passage retrieval using random walks with question-based priors. _Information Processing & Management_, 45(1):42-54.
* Oya et al. (2014) Tatsuro Oya, Yashar Mehdad, Giuseppe Carenini, and Raymond Ng. 2014. A template-based abstractive meeting summarization: Leveraging summary and source text relationships. In _Proceedings of the 8th International Natural Language Generation Conference (INLG)_, pages 45-53, Philadelphia, Pennsylvania, U.S.A. Association for Computational Linguistics.
* Rush et al. (2015) Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 379-389 

* See et al. (2017) Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 1073-1083.
* Shang et al. (2018) Guokan Shang, Wensi Ding, Zekun Zhang, Antoine Tixier, Polykarpos Meladianos, Michalis Vazirgiannis, and Jean-Pierre Lorre. 2018. Unsupervised abstractive meeting summarization with multi-sentence compression and budgeted submodular maximization. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 664-674, Melbourne, Australia. Association for Computational Linguistics.
* Spataro (2020) Jared Spataro. 2020. The future of work--the good, the challenging the unknown.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, pages 5998-6008.
* Vinyals et al. (2015) Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In _Advances in neural information processing systems_, pages 2692-2700.
* Wang et al. (2020) Danqing Wang, Pengfei Liu, Yining Zheng, Xipeng Qiu, and Xuanying Huang. 2020. Heterogeneous graph neural networks for extractive document summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 6209-6219. Association for Computational Linguistics.
* Wang et al. (2019) Danqing Wang, Pengfei Liu, Ming Zhong, Jie Fu, Xipeng Qiu, and Xuanjing Huang. 2019. Exploring domain shift in extractive text summarization. _arXiv preprint arXiv:1908.11664_.
* Wang and Cardie (2013) Lu Wang and Claire Cardie. 2013. Domain-independent abstract generation for focused meeting summarization. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1395-1405, Sofia, Bulgaria. Association for Computational Linguistics.
* Wang et al. (2016) Lu Wang, Hema Raghavan, Vittorio Castelli, Radu Florian, and Claire Cardie. 2016. A sentence compression based framework to query-focused multi-document summarization. _arXiv preprint arXiv:1606.07548_.
* Wang and Jiang (2017) Shuohang Wang and Jing Jiang. 2017. Machine comprehension using match-lstm and answer pointer. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
* Xu and Durrett (2019) Jiacheng Xu and Greg Durrett. 2019. Neural extractive text summarization with syntactic compression. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_, Hong Kong, China. Association for Computational Linguistics.
* Xu et al. (2019) Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Discourse-aware neural extractive model for text summarization. _arXiv preprint arXiv:1910.14142_.
* Yasunaga et al. (2019) Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander R Fabbri, Irene Li, Dan Friedman, and Dragomir R Radev. 2019. Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 7386-7393.
* Zhong et al. (2020) Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xiepeng Qiu, and Xuanjing Huang. 2020. Extractive summarization as text matching. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 6197-6208. Association for Computational Linguistics.
* Zhong et al. (2019a) Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2019a. Searching for effective neural extractive summarization: What works and what's next. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1049-1058.
* Zhong et al. (2019b) Ming Zhong, Danqing Wang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2019b. A closer look at data bias in neural extractive summarization models. _EMNLP-IJCNLP 2019_, page 80.
* Zhu et al. (2020) Chenguang Zhu, Ruochen Xu, Michael Zeng, and Xuedong Huang. 2020. A hierarchical network for abstractive meeting summarization with cross-domain pretraining. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 194-203, Online. Association for Computational Linguistics.
* Zhu et al. (2019) Haichao Zhu, Li Dong, Furu Wei, Bing Qin, and Ting Liu. 2019. Transforming wikipedia into augmented data for query-focused summarization. _CoRR_, abs/1911.03324.

 

Appendix

### Query Schema List

As mentioned in Section 3.2, we make query schema list to help annotators design queries. Towards general and specific meeting contents, we further divide the query schema list into _general query schema list_ and _specific query schema list_. The detailed list is shown in Table 6.

### Other Details of Annotation Instruction

We show other details except those in Section 3.

#### a.2.1 Topic Segmentation

Topics.We require annotators to use noun phrases to represent the main topics. As we hope to select the most important topics, the number of annotated main topics should not be too much, and 3 to 8 is proper range.

Distribution of Relevant Text Spans in Meeting Transcripts.Relevant text spans of a main topic may be scattered in different parts of meetings, e.g., the main topic 'scope of the project and team building' in the leftmost part of Figure 2. So annotators are asked to label _all_ the relevant spans, and these spans are allowed to be not contiguous.

Whether Chatting Belongs to an Independent Main Topic or not.Main topics should be objectively cover most of the meeting contents. In meetings, group members might take much time chatting. Though this is not close to the main theme of the meetings, it should also be counted as a main topic if the chat took lots of time.

#### a.2.2 Query Generation

Diverse Query Types.For the specific queries, we encourage the annotators to choose different schemas to compose queries, since we intend to diversify the query types and reduce the bias towards certain query types.

Independent Query-answer Pairs.Each query-answer pair should be independent, that is, not dependent on previous queries and answers. For example, for the query 'How did they reach agreement afterwards?', it seems that the query is dependent on the previous annotations, and it is ambiguous to know when they reached agreement and what the agreement referred to if we treat the query independently. Annotators should specify the information of when they reached an agreement and what the agreement is to make the query clear. For example, annotators could rewrite the query as 'How did the group agree on the button design when discussing the design of remote control?'.

#### a.2.3 Query-based Summarization

Informative Writing.When answering queries like 'What did A think of X?', annotators were required to not only summarize what A said, but also briefly mention the context which is relevant to what A said. This is designed to rich the contents of summaries and further challenge the model's capability of summarizing relevant contexts.

Relevant text spans' annotation.Since there might be multiple relevant text spans, we asked annotators to annotate all of them.

The Usage of Tense.Since all the meetings happened, we ask annotators to use past tense.

How to Denote Speakers.If the gender information is unclear, we would ask annotators not to use 'he/she' to denote speakers. We also asked them not to abbreviations (e.g., PM) to denote speakers, and use the full name like 'Project Manager' instead.

Abbreviations.In the raw meeting transcripts, some of abbreviations are along with character '\({}^{-}\)'. If the annotators encountered abbreviations, e.g., 'L_C_D_', 'A_A_A_', etc., they would be required to rewrite them like LCD or AAA.

### Annotation Review Standards

We launched a _'pre-annotation'_ stage in which annotators were asked to try annotating one meeting and the _experts_ who are good at our annotation task would review it and instruct the annotators by providing detailed feedback. Requirements include 1) faithfulness, 2) informativeness, 3) lengths of the relevant text spans of designed queries, 4) typo errors, etc. They could continue annotating only if they passed _'pre-annotation'_ stage. After _'pre-annotation'_, _experts_ would keep carefully reviewing all the annotations according to the requirements. We write down the standards for the three annotation stages individually in annotation instruction. Details of annotation review standards could be referred to Appendix A.2.

### Examples of QMSum Dataset

We show two examples of our proposed QMSum dataset. One belongs to product meeting (Table 7), and the other one is about committee meeting (Table 8) 



**General Query Schema List**

A: **Speaker**

1. Summarize the whole meeting. / What did the group/committee discuss in the meeting? (Mandatory)
2. What did A say in the meeting? / Summarize what A said.
3. What was the conclusion / decision of the meeting?
4. What was the purpose of the meeting?
5. How did the group/committee split the work?

**Specific Query Schema List**

A, B: **Speakers, X: Annotated Main Topics, Y: Subtopics regarding to X**

1. Summarize the discussion about X. / What did the group/committee discuss X? (Mandatory)
2. Why did the group/committee decide to do sth. when discussing X?
3. What did A think of Y when discussing X? / Summarize A's opinions towards Y.
4. What did A think of X? / Summarize A's opinions towards X. / What did A propose in the discussion about X?
5. What was the advantage / disadvantage of sth. with regard to X?
6. Why did A think regarding to X?
7. Why did A agree / disagree with certain ideas? / Provide the reasons why A held certain opinions towards X.
8. Why did A think of Y when discussing X?
9. What was the decision / conclusion of the discussion about X? / Summarize the decision of the discussion about X.
10. Why did A agree / disagree with B when discussing X?
11. What did A recommend to do when discussing X and why?
12. What did A learn about topic X?
13. What did A and B discuss X?



**Product Meeting (IS1000a)****Color: Speakers, Main Topics, Subtopics****_Turn 0**_: User Interface Designer: Okay.
* .........
* .......
* __Turn 243: Project Manager: Well, this uh this tool seemed to work.
* ......
* __Turn 257: Project Manager: More interesting for our company of course, uh profit aim, about fifty million Euro. So we have to sell uhu quite a lot of this u things. ......
* _Turn 258: User Interface Designer: Ah yeah, the sale man, four million._Turn 259: User Interface Designer: Maybe some uh Asian countries. Um also important for you all is um the production cost must be maximal uh twelve uht twelve Euro and fifty cents.
* ......
* _Turn 275: Project Manager: So uhl well I think when we are working on the international market , uh in principle it has enough customers._Turn 276: Industrial Designer: Yeah._Turn 277: Project Manager: Uhs owhen we have a good product we uh we could uh meet this this aim, I think. So, that about finance. And uh now just let have some discussion about what is a good remote control and uh well keep in mind this this first point, it has to be original, it has to be trendy, it has to be user friendly. ......
* ......
* _Turn 400: Project Manager: Keep it in mind._

**Annotated Main Topics******

Scope of the project and team building (_Turn 41 - 245_)****Cost constraints and financial targets of the new remote control project (_Turn 246 - 277_)****Remote control style and use cases (_Turn 277 - 295_)****Prioritizing remote control features (_Turn 343 - 390_)****Queries and Annotated Summaries****_Query 1_: Summarize the whole meeting._Answer:_Project Manager introduced a new remote control project for television sets, and the team got acquainted with each other and technical devices. The remote control would be priced at 25 Euros and a production cost of 12.5 Euros. ......
* ......
* _Query 2_: What did the group discuss about prioritizing remote control features?_Answer:_User Interface Designer and Industrial Designer expressed a desire to integrate cutting-edge features into the remote. Marketing pointed out that most of the market would buy it for standard use, like changing channels and adjusting volume ......
* 390___Summary 4_: Why did Marketing disagree with Industrial Designer when discussing prioritizing remote control features?_Answer:_Marketing believed that fancy features like IP would not be used by most people. The overwhelming majority of users would want convenient channel browsing and volume adjustment features ......
* _Relevant Text Spans: Turn 358__Summary 7_: What did Project Manager think of the cost constraints and financial targets of the new remote control project?_Answer:_Project Manager introduced the financial information: 25 Euro selling price and 12.5 Euro production cost. Project Manager then went on to elaborate that the target market would primarily consist of Europe and North America. ......





**Committee Meeting (Education 4)****Color:**Speakers, Main Topics, Subtopies__

_Turn 0_: Lynne Neagle AM: Okay, good morning, everyone. Welcome to the Children, Young People and Education Committee this morning. I've received apologies for absence from ......

_Turn 31_: David Hopkins: Yes, sure. The delegation levels are already very high in most authority areas, and we've got agreements in place with the Government to make sure that more money, or as much money as possible .....

_Turn 32_: Sian Gwenllian AM: Okay. But just the pressures coming in with the new Act et cetera could mean more expulsions.

_Turn 33_: David Hopkins: It shouldn't, but it could. It's difficult to know how headteachers and governing bodies will react. .......

_Turn 44_: Sharon Davies: As Nick said, it does get more difficult at key stage 4, and it's working, then, with. It comes back to that team-around-the-family approach ......

_Remark 47_: David Hopkins: I don't think I'm allowed to say at this point.

_Remark 228_: Lynne Neagle AM: Item 4, then, is papers to note. Just one paper today, which is the Welsh Government's response to the committee's report on the scrutiny of the Welsh Government's draft budget 2020-1. ......

**Annotated Main Topics**

An increase of exclusions from school and current measures against it (_Turn 1 - 19_, _Turn 158 - 172_)

The funding issues (_Turn 20 - 38_, _Turn 177 - 179_)

The networking within the PRU and the transition arrangements (_Turn 39 - 56_)

....

Schools' awareness of early trauma ACEs (_Turn 180 - 188_)

**Queries and Annotated Summaries**

_Query 1_: Summarize the whole meeting.

_Answer_: The meeting was mainly about the reasons behind and the measurements against the increasing exclusions from school.

The increase brought more pressure to EOTAS in the aspects of finance, transition, curriculum arrangement and the recruitment of professional staff. Although much time and finance had been devoted to the PRU ......

_Query 4_: What was considered by David Hopkins as the factor that affected exclusions?

_Answer_: David Hopkins did not think that the delegation levels were not high enough in most authority areas. Instead, he thought they had got agreements with the government to make sure that enough money was devolved to school. The true decisive factor w-as the narrow measure at the end of Stage 4 that drove the headquarters to exclude students or put them into another school.

_Relevant Text Spans_: _Turn 31 - 33_

....

_Query 6_: What was the major challenge of the transition of the excluded students?

_Answer_: The students coming to the end of their statutory education were facing the biggest challenge, for it would be far more difficult for them to go back into the mainstream education process when they turned 15 or 16, not to mention the transition into further education, such as colleges.

_Relevant Text Spans_: _Turn 44 - 49_

....

\begin{table}
\begin{tabular}{p{341.4pt}} \hline
**Committee Meeting (Education 4)****Color:**Speakers, Main Topics, Subtopies__