SlowGuess/NAACL_2021_OCR huggingface dataset:
Ming Zhong*† Da Yin*♣ Tao Yu† Ahmad Zaidi‡
Mutethia Mutuma† Rahul Jha¶ Ahmed Hassan Awadallah¶
Asli Celikyilmaz $^{\dagger}$  Yang Liu $^{\S}$  Xipeng Qiu† Dragomir Radev‡
†Fudan University ‡University of California, Los Angeles ‡Yale University
mzhong18@fudan.edu.cn da.yin@cs.ucla.edu
{tao.yu,dragomir.radev}@yale.edu
Meetings are a key component of human collaboration. As increasing numbers of meetings are recorded and transcribed, meeting summaries have become essential to remind those who may or may not have attended the meetings about the key decisions made and the tasks to be completed. However, it is hard to create a single short summary that covers all the content of a long meeting involving multiple people and topics. In order to satisfy the needs of different types of users, we define a new query-based multi-domain meeting summarization task, where models have to select and summarize relevant spans of meetings in response to a query, and we introduce QMSum, a new benchmark for this task. QMSum consists of 1,808 query-summary pairs over 232 meetings in multiple domains. Besides, we investigate a locate-then-summarize method and evaluate a set of strong summarization baselines on the task. Experimental results and manual analysis reveal that QMSum presents significant challenges in long meeting summarization for future research. Dataset is available at https://github.com/Yale-LILY/QMSum.
Meetings remain the go-to tool for collaboration, with 11 million meetings taking place each day in the USA and employees spending six hours a week, on average, in meetings (Mroz et al., 2018). The emerging landscape of remote work is making meetings even more important and simultaneously taking a toll on our productivity and wellbeing (Spataro, 2020). The proliferation of meetings makes it hard to stay on top of this sheer volume of information and increases the need for automated methods for accessing key information exchanged during them. Meeting summarization (Wang and Cardie, 2013; Shang et al., 2018;
Figure 1: Examples of query-based meeting summarization task. Users are interested in different facets of the meeting. In this task, a model is required to summarize the contents that users are interested in and query.
Li et al., 2019; Zhu et al., 2020) is a task where summarization models are leveraged to generate summaries of entire meetings based on meeting transcripts. The resulting summaries distill the core contents of a meeting that helps people efficiently catch up to meetings.
Most existing work and datasets on meeting summarization (Janin et al., 2003; Carletta et al., 2005) pose the problem as a single document summarization task where a single summary is generated for the whole meeting. Unlike news articles where people may be satisfied with a high-level summary, they are more likely to seek more detailed information when it comes to meeting summaries such as topics (Li et al., 2019), opinions, actions, and decisions (Wang and Cardie, 2013). This poses the question of whether a single paragraph is enough to summarize the content of an entire meeting?
Figure 1 shows an example of a meeting about "remote control design". The discussions in the meeting are multi-faceted and hence different users might be interested in different facets. For example, someone may be interested in learning about the new trends that may lead to the new product
* These two authors contributed equally. The order of authorship decided by the flip of a coin.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905-5921 June 6-11, 2021. ©2021 Association for Computational Linguistics
standing out, while others may be more interested in what other attendees thought about different elements of the design. It is challenging to compress or compose a short summary that contains all the salient information. Alternatively, summarization systems should adopt a more flexible and interactive approach that allows people to express their interests and caters to their diverse intents when generating summaries (Dang, 2005, 2006; Litvak and Vanetik, 2017; Baumel et al., 2018).
With comprehensive consideration of the multi-granularity meeting contents, we propose a new task, query-based meeting summarization. To enable research in this area, we also create a high-quality multi-domain summarization dataset. In this task, as shown in Figure 1, given a query and a meeting transcript, a model is required to generate the corresponding summary. The query-based approach is a flexible setup that enables the system to satisfy different intents and different levels of granularity. Besides the annotated queries and corresponding gold summaries at different levels of granularity, our new dataset contains a rich set of annotations that include the main topics of each meeting and the ranges of relevant text spans for the annotated topics and each query. We adopt a hierarchical annotation structure that could not only assist people to find information faster, but also strengthen the models' summarization capacity.
In this paper, we employ a two-stage meeting summarization approach: locate-then-summarize. Specifically, given a query, a model called Loca-tor is used to locate the relevant utterances in the meeting transcripts, and then these extracted spans are used as an input to another model called Sum-marizer to generate a query-based summary. We present and evaluate several strong baselines based on state-of-the-art summarization models on QM-Sum. Our results and analysis from different perspectives reveal that the existing models struggle in solving this task, highlighting the challenges the models face when generating query-based meeting summaries. We are releasing our dataset and baselines to support additional research in query-focused meeting summarization.
Overall, our contributions are listed as follows: 1) We propose a new task, query-based multidomain meeting summarization, and build a new benchmark QMSum with a hierarchical annotation structure. 2) We design a locate-then-summarize model and conduct comprehensive experiments on
its strong variants and different training settings. 3) By human evaluation, we further pose the challenges of the new task, including the impact of different query types and factuality errors.
2.1 Text Summarization
Most prior work in text summarization (Rush et al., 2015; Chopra et al., 2016; Nallapati et al., 2016; See et al., 2017; Celikyilmaz et al., 2018; Chen and Bansal, 2018; Zhong et al., 2019a; Xu and Durrett, 2019; Liu and Lapata, 2019; Lebanonff et al., 2019; Cho et al., 2019; Zhong et al., 2020; Wang et al., 2020; Xu et al., 2019; Jia et al., 2020) investigate how to generate better summaries on news article data, such as CNN/DailyMail (Hermann et al., 2015), Newsroom (Grusky et al., 2018), etc. Scientific paper summarization is another important branch (Cohan et al., 2018; Yasunaga et al., 2019; An et al., 2021). Our paper mainly focuses on meeting summarization, a more challenging task compared to news summarization. With the burst of demand for meeting summarization, this task attracts more and more interests from academia (Wang and Cardie, 2013; Oya et al., 2014; Shang et al., 2018; Zhu et al., 2020) and becomes an emerging branch of text summarization area.
2.2 Query-based Summarization
Query-based summarization aims to generate a brief summary according to a source document and a given query. There are works studying this task (Daumé III and Marcu, 2006; Otterbacher et al., 2009; Wang et al., 2016; Litvak and Vanetik, 2017; Nema et al., 2017; Baumel et al., 2018; Ishigaki et al., 2020; Kulkarni et al., 2020; Laskar et al., 2020). However, the models focus on news (Dang, 2005, 2006), debate (Nema et al., 2017), and Wikipedia (Zhu et al., 2019). Meeting is also a genre of discourses where query-based summarization could be applied, but to our best knowledge, there are no works studying this direction.
2.3 Meeting Summarization
Meeting summarization has attracted a lot of interest recently (Chen and Metze, 2012; Wang and Cardie, 2013; Mehdad et al., 2013; Oya et al., 2014; Shang et al., 2018; Li et al., 2019; Zhu et al., 2020; Koay et al., 2020). Specifically, Mehdad et al. (2013) leverage entailment graphs and ranking strategy to generate meeting summaries. Wang
and Cardie (2013) attempt to make use of decisions, action items and progress to generate the whole meeting summaries. Oya et al. (2014) leverages the relationship between summaries and the meeting transcripts to extract templates and generate summaries with the guidance of the templates. Shang et al. (2018) utilize multi-sentence compression techniques to generate summaries under an unsupervised setting. Li et al. (2019) attempt to incorporate multi-modal information to facilitate the meeting summarization. Zhu et al. (2020) propose a model which builds a hierarchical structure on word-level and turn-level information and uses news summary data to alleviate the inadequacy of meeting data.
Unlike previous works, instead of merely generating summaries for the complete meeting, we propose a novel task where we focus on summarizing multi-granularity contents which cater to different people's need for the entire meetings, and help people comprehensively understand meetings.
In this section, we show how we collected meeting data from three different domains: academic meetings, product meetings, and committee meetings. In addition, we show how we annotated the three types of meeting data while ensuring annotation quality for query-based meeting summarization.
We introduce the three types of meetings that we used to annotate query-summary pairs.
Product Meetings AMI $^{1}$  (Carletta et al., 2005) is a dataset of meetings about product design in an industrial setting. It consists of 137 meetings about how to design a new remote control, from kick-off to completion over the course of a day. It contains meeting transcripts and their corresponding meeting summaries.
Academic Meetings ICSI $^2$  (Janin et al., 2003) dataset is an academic meeting dataset composed of 59 weekly group meetings at International Computer Science Institute (ICSI) in Berkeley, and their summaries. Different from AMI, the contents of ICSI meetings are specific to the discussions about research among students.
Committee Meetings Parliamentary committee meeting is another important domain of meetings. These meetings focus on the formal discussions on a wide range of issues (e.g., the reform of the education system, public health, etc.) Also, committee meetings are publicly available, which enables us to access large quantities of meetings. We include 25 committee meetings of the Welsh Parliament<sup>3</sup> and 11 from the Parliament of Canada<sup>4</sup> in our dataset.
3.2 Annotation Pipeline
After collecting meeting transcripts, we recruited annotators and required them to annotate by following annotation instruction. As illustrated in Figure 2, the annotation process is composed by three stages: topic segmentation, query generation, and query-based summarization.
Topic Segmentation Meeting transcripts are usually long and contain discussions about multiple topics. To assist further annotations, we asked annotators to write down the main topics discussed in the meetings, and their relevant text spans, which makes the meeting structure clear. As shown in Figure 2, "scope of the project and team building" is one of the annotated main topics, and its relevant text spans of the topic are (Turn 25 - 50, Turn 73 - 89). More details are listed in Appendix A.2.1.
Query Generation Towards the query-based task, we further asked annotators to design queries by themselves. To cater to the need for multi-granularity contents, we categorized two types of queries: queries related to general information (e.g., the contents of whole meetings, etc.) are called general queries; queries focusing on relatively detailed information (e.g., the discussion about certain topics, etc.) are called specific queries.
To alleviate the influence of extremely hard queries and focus on the evaluation of query-based summarization capacity, rather than designing queries in an unconstrained way, we asked annotators to generate queries according to the schema. Details of the query schema list are shown in Appendix A.1. The list consists of important facets people might be interested in, including overall contents of discussions, speakers' opinions, the reasons why a speaker proposed an idea, etc., which cover the most common queries over meetings involving multiple people discussing several topics.
1http://groups.inf.ed.ac.uk/ami/ download/ 2http://groups.inf.ed.ac.uk/ami/icsi/ index.shtml
3https://record.assembly.wales/ 4https://www.ourcommons.ca/Committees/ en/Home
Figure 2: Overall annotation pipeline. It is divided into three stages: Stage 1 is to annotate main topics and their relevant text spans; Stage 2 is to generate queries based on query schema lists; Stage 3 is to annotate the summaries according to the queries. The pipeline was implemented upon the collected meetings of multiple domains.
To query multi-granularity meeting contents, we further divided the query schema list into general and specific ones, and asked annotators to design queries towards general and specific meeting contents, respectively. In terms of general query generation, the annotators were asked to design 1 - 2 general queries according to the general schema list. For specific query generation, annotators were asked to first select 2 - 4 main topics and their relevant text spans, and then design around 3 specific queries based on the specific schema list for each main topic. To ensure the task to be summarization instead of question answering, we asked annotators to design queries of which the relevant text spans are more than 10 turns or 200 words. Therefore, our proposed task would differ from question answering tasks where models merely need to extract phrases or generate answers based on short text spans, and focus on how to summarize based on large stretches of texts. Additional details are in Appendix A.2.2.
Query-based Summarization According to the designed queries and meeting transcripts, annotators were asked to do faithful summarization. Being accorded with the meeting transcripts and queries is the most important criterion. We also required annotators to write informative summarization. For example, they could add more de
tails about the reasons why the group/committee made such decisions, and which important ideas the group/committee members proposed, etc. Besides, the annotated summaries should be abstractive, fluent and concise. We set word limits for the answers of general queries (50 - 150 words) and specific queries (20 - 100 words) to keep conciseness. More details are shown in Appendix A.2.3.
In the end, we organize all the meeting data after accomplishing the three annotation stages. Detailed annotations of one product meeting and one committee meeting are shown in Appendix A.4. Each meeting transcript is accompanied with annotated main topics, queries, their corresponding summaries, and relevant text span information.
3.3 Additional Details of Annotation Process
This section describes how we recruited annotators and how we review the annotations in detail.
Annotator Recruitment To guarantee annotation quality given the complexity of the task, instead of employing tasks on Amazon Mechanical Turker, we anonymously recruited undergraduate students who are fluent in English. The annotation team consists of 2 native speakers and 10 nonnative speakers majoring in English literature.
Annotation Review To help the annotators fully comprehend the instruction, annotators were
Datasets	# Meetings	# Turns	# Len. of Meet.	# Len. of Sum.	# Speakers	# Queries	# Pairs
AMI	137	535.6	6007.7	296.6	4.0	-	97 / 20 / 20
ICSI	59	819.0	13317.3	488.5	6.3	-	41 / 9 / 9
Product	137	535.6	6007.7	70.5	4.0	7.2	690 / 145 / 151
Academic	59	819.0	13317.3	53.7	6.3	6.3	259 / 54 / 56
Committee	36	207.7	13761.9	80.5	34.1	12.6	308 / 73 / 72
All	232	556.8	9069.8	69.6	9.2	7.8	1,257 / 272 / 279
Table 1: Statistics of meeting summarization datasets. The top half of the table is the existing meeting datasets, and the bottom half is the statistics of QMSum. Because a meeting may have multiple queries, #Pairs here means how many query-summary pairs are contained in the train / valid / test set.
trained in a pre-annotation process. Annotations were reviewed across all stages in our data collection process by expert of this annotation task. More details of review standards could be found in Appendix A.3.
3.4 Dataset Statistics and Comparison
Statistics of the final QMSum dataset is shown in Table 1. There are several advantages of QMSum dataset, compared with the previous datasets.
Number of Meetings and Summaries QMSum includes 232 meetings, which is the largest meeting summarization dataset to our best knowledge. For each query, there is a manual annotation of corresponding text span in the original meeting, so there are a total of 1,808 question-summary pairs in QMSum. Following the previous work, we randomly select about  $15\%$  of the meetings as the validation set, and another  $15\%$  as the test set.
Briefly The average length of summaries in QM-Sum 69.6 is much shorter than that of previous AMI and ICSI datasets. It is because our dataset also focuses on specific contents of the meetings, and the length of their corresponding summaries would not be long. It leaves a challenge about how to precisely capture the related information and compress it into a brief summary.
Multi-domain Setting Previous datasets are specified to one domain. However, the model trained on the summarization data of a single domain usually has poor generalization ability (Wang et al., 2019; Zhong et al., 2019b; Chen et al., 2020). Therefore, QMSum contains meetings across multiple domains: Product, Academic and Committee meetings. We expect that our dataset could provide a venue to evaluate the model's generalization ability on meetings of different domains and help create more robust models.
In this section, we first define the task of query-based meeting summarization, then describe our two-stage locate-then-summarize solution in detail.
Existing meeting summarization methods define the task as a sequence-to-sequence problem. Specifically, each meeting transcript  $X = (x_{1}, x_{2}, \dots, x_{n})$  consists of  $n$  turns, and each turn  $x_{i}$  represents the utterance  $u_{i}$  and its speaker  $s_{i}$ , that is,  $x_{i} = (u_{i}, s_{i})$ . Additionally, each utterance contains  $l_{i}$  words  $u_{i} = (w_{1}, \dots, w_{l_{i}})$ . The object is to generate a target summary  $Y = (y_{1}, y_{2}, \dots, y_{m})$  by modeling the conditional distribution  $p(y_{1}, y_{2}, \dots, y_{m}|(u_{1}, s_{1}), \dots, (u_{n}, s_{n}))$ .
However, meetings are usually long conversations involving multiple topics and including important decisions on many different matters, so it is necessary and practical to use queries to summarize a certain part of the meeting. Formally, we introduce a query  $Q = (w_{1}, \dots, w_{|Q|})$  for meeting summarization task, the objective is to generate a summary  $Y$  by modeling  $p(y_{1}, y_{2}, \dots, y_{m} | Q, (u_{1}, s_{1}), \dots, (u_{n}, s_{n}))$ .
4.2 Locator
In our two-stage pipeline, the first step requires a model to locate the relevant text spans in the meeting according to the queries, and we call this model a Locator. The reason why we need a Locator here is, most existing abstractive models cannot process long texts such as meeting transcripts. So we need to extract shorter, query-related paragraphs as input to the following Summarizer.
We mainly utilize two methods to instantiate our Locator: Pointer Network (Vinyals et al., 2015) and a hierarchical ranking-based model. Pointer
Figure 3: Hierarchical ranking-based locator structure.
Network has achieved widespread success in extractive QA tasks (Wang and Jiang, 2017). For each question, it will point to the <start, end> pair in the source document, and the span is the predicted answer. Specific to our task, Pointer Network will point to the start turn and the end turn for each query. It is worth noting that one query can correspond to multiple spans in our dataset, so we always extract three spans as the corresponding text for each query when we use Pointer Network as Locator in the experiments.
In addition, we design a hierarchical ranking-based model structure as the Locator. As shown in Figure 3, we first input the tokens in each turn to a feature-based BERT to obtain the word embedding, where feature-based means we fix the parameters of BERT, so it is actually an embedding layer. Next, CNN (Kim, 2014) is applied as a turn-level encoder to capture the local features such as bigram, trigram and so on in each turn. Here we do not use Transformer because previous work (Kedzie et al., 2018) shows that this component does not matter too much for the final performance. We combine different features to represent the utterance  $\mathbf{u}_i$  in each turn, and concatenate the speaker embedding  $\mathbf{s}_i$  as the turn-level representation:  $\mathbf{x_i} = [\mathbf{u_i};\mathbf{s_i}]$ , where  $[\cdot ]$  denotes concatenation and  $\mathbf{s}_i$  is a vector randomly initialized to represent the speaking style of meeting participants.
Then these turn representations will be contextualized by a document-level Transformer (Vaswani et al., 2017) encoder. Next, we introduce query embedding  $\mathbf{q}$  which is obtained by a CNN (shared parameters with CNN in turn-level encoder) and use MLP to score each turn. We use binary cross-entropy loss to train our Locator. Finally, turns with the highest scores are selected as the relevant text spans of each query and will be inputted to the
subsequent Summarizer.
4.3 Summarizer
Given the relevant paragraphs, our goal in the second stage is to summarize the selected text spans based on the query. We instantiate our Summarizer with the current powerful abstractive models to explore whether the query-based meeting summarization task on our dataset is challenging. To be more specific, we choose the following three models:
Pointer-Generator Network (See et al., 2017) is a popular sequence-to-sequence model with copy mechanism and coverage loss, and it acts as a baseline system in many generation tasks. The input to Pointer-Generator Network (PGNet) is: “
 Query 
 Relevant Text Spans 
”.
BART (Lewis et al., 2020) is a denoising pretrained model for language generation, translation and comprehension. It has achieved new state-of-the-art results on many generation tasks, including summarization and abstractive question answering. The input to BART is the same as PGNet.
HMNet (Zhu et al., 2020) is the state-of-the-art meeting summarization model. It contains a hierarchical structure to process long meeting transcripts and a role vector to depict the difference among speakers. Besides, a cross-domain pretraining process is also included in this strong model. We add a turn representing the query at the beginning of the meeting as the input of HMNet.
In this section, we introduce the implementation details, effectiveness of Locator, experimental results and multi-domain experiments on QMSum.
For our ranking-based Locator, the dimension of speaking embedding is 128 and the dimension of turn and query embedding is 512. Notably, we find that removing Transformers in Locator has little impact on performance, so the Locator without Transformer is used in all the experiments. To reduce the burden of the abstractive models, we utilize Locator to extract 1/6 of the original text and input them to Summarizer. The hyperparameters used by PGNet and HMNet are consistent with the original paper. Due to the limitation of computing resources, we use the base version of pre-trained models (including feature-based BERT and BART)
Models	Extracted Length
1/6	1/5	1/4	1/3
Random	58.86	63.20	67.56	73.81
Similarity	55.97	59.24	63.45	70.12
Pointer	61.27	65.84	70.13	75.96
Our Locator	72.51	75.23	79.08	84.04