SlowGuess/NAACL_2021_OCR huggingface dataset:
Ruipeng Jia, Yanan Cao, Hengzhu Tang, Fang Fang, Cong Cao, and Shi Wang. 2020. Neural extractive summarization with hierarchical attentive heterogeneous graph network. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3622-3631.
Chris Kedzie, Kathleen McKeown, and Hal Daume III. 2018. Content selection in deep learning models of summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1818-1828.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751.
Jia Jin Koay, Alexander Roustai, Xiaojin Dai, Dillon Burns, Alec Kerrigan, and Fei Liu. 2020. How domain terminology affects meeting summarization performance. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5689-5695, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Sayali Kulkarni, Sheide Chammas, Wan Zhu, Fei Sha, and Eugene Ie. 2020. Aquamuse: Automatically generating datasets for query-based multi-document summarization. arXiv preprint arXiv:2010.12694.
Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Huang. 2020. Query focused abstractive summarization via incorporating query relevance and transfer learning with transformer models. In Canadian Conference on Artificial Intelligence, pages 342-348. Springer.
Logan Lebanoff, Kaiqiang Song, Franck Dernoncourt, Doo Soon Kim, Seokhwan Kim, Walter Chang, and Fei Liu. 2019. Scoring sentence singletons and pairs for abstractive summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2175-2189.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871-7880. Association for Computational Linguistics.
Manling Li, Lingyu Zhang, Heng Ji, and Richard J. Radke. 2019. Keep meeting summaries on topic: Abstractive multi-modal meeting summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2190-2196, Florence, Italy. Association for Computational Linguistics.
Marina Litvak and Natalia Vanetik. 2017. Query-based summarization using MDL principle. In Proceedings of the MultiLing 2017 Workshop on Summarization and Summary Evaluation Across Source Types and Genres, pages 22-31, Valencia, Spain. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3721-3731.
Yashar Mehdad, Giuseppe Carenini, Frank Tompa, and Raymond T. Ng. 2013. Abstractive meeting summarization with entailment and fusion. In Proceedings of the 14th European Workshop on Natural Language Generation, pages 136-146, Sofia, Bulgaria. Association for Computational Linguistics.
Joseph E. Mroz, Joseph A. Allen, Dana C. Verhoeven, and Marissa L. Shuffler. 2018. Do we really need another meeting? the science of workplace meetings. Current Directions in Psychological Science, 27(6):484-491.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarrunner: A recurrent neural network based sequence model for extractive summarization of documents. In Thirty-First AAAI Conference on Artificial Intelligence.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Ca glar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. CoNLL 2016, page 280.
Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven attention model for query-based abstractive summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1063-1072, Vancouver, Canada. Association for Computational Linguistics.
Jahna Otterbacher, Gunes Erkan, and Dragomir R Radev. 2009. Biased lexrank: Passage retrieval using random walks with question-based priors. Information Processing & Management, 45(1):42-54.
Tatsuro Oya, Yashar Mehdad, Giuseppe Carenini, and Raymond Ng. 2014. A template-based abstractive meeting summarization: Leveraging summary and source text relationships. In Proceedings of the 8th International Natural Language Generation Conference (INLG), pages 45-53, Philadelphia, Pennsylvania, U.S.A. Association for Computational Linguistics.
Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379-389.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1073-1083.
Guokan Shang, Wensi Ding, Zekun Zhang, Antoine Tixier, Polykarpos Meladianos, Michalis Vazirgiannis, and Jean-Pierre Lorre. 2018. Unsupervised abstractive meeting summarization with multisentence compression and budgeted submodular maximization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 664-674, Melbourne, Australia. Association for Computational Linguistics.
Jared Spataro. 2020. The future of work—the good, the challenging the unknown.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in neural information processing systems, pages 2692-2700.
Danqing Wang, Pengfei Liu, Yining Zheng, Xipeng Qiu, and Xuanjing Huang. 2020. Heterogeneous graph neural networks for extractive document summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 6209-6219. Association for Computational Linguistics.
Danqing Wang, Pengfei Liu, Ming Zhong, Jie Fu, Xipeng Qiu, and Xuanjing Huang. 2019. Exploring domain shift in extractive text summarization. arXiv preprint arXiv:1908.11664.
Lu Wang and Claire Cardie. 2013. Domain-independent abstract generation for focused meeting summarization. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1395-1405, Sofia, Bulgaria. Association for Computational Linguistics.
Lu Wang, Hema Raghavan, Vittorio Castelli, Radu Florian, and Claire Cardie. 2016. A sentence compression based framework to query-focused multi-document summarization. arXiv preprint arXiv:1606.07548.
Shuohang Wang and Jing Jiang. 2017. Machine comprehension using match-lstm and answer pointer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Jiacheng Xu and Greg Durrett. 2019. Neural extractive text summarization with syntactic compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Hong Kong, China. Association for Computational Linguistics.
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Discourse-aware neural extractive model for text summarization. arXiv preprint arXiv:1910.14142.
Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander R Fabbri, Irene Li, Dan Friedman, and Dragomir R Radev. 2019. Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7386-7393.
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization as text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 6197-6208. Association for Computational Linguistics.
Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, and Xuan-Jing Huang. 2019a. Searching for effective neural extractive summarization: What works and what's next. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1049-1058.
Ming Zhong, Danqing Wang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2019b. A closer look at data bias in neural extractive summarization models. EMNLP-IJCNLP 2019, page 80.
Chenguang Zhu, Ruochen Xu, Michael Zeng, and Xuedong Huang. 2020. A hierarchical network for abstractive meeting summarization with cross-domain pretraining. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 194-203, Online. Association for Computational Linguistics.
Haichao Zhu, Li Dong, Furu Wei, Bing Qin, and Ting Liu. 2019. Transforming wikipedia into augmented data for query-focused summarization. CoRR, abs/1911.03324.
A.1 Query Schema List
As mentioned in Section 3.2, we make query schema list to help annotators design queries. Towards general and specific meeting contents, we further divide the query schema list into general query schema list and specific query schema list. The detailed list is shown in Table 6.
A.2 Other Details of Annotation Instruction
We show other details except those in Section 3.
A.2.1 Topic Segmentation
Topics. We require annotators to use noun phrases to represent the main topics. As we hope to select the most important topics, the number of annotated main topics should not be too much, and 3 to 8 is proper range.
Distribution of Relevant Text Spans in Meeting Transcripts. Relevant text spans of a main topic may be scattered in different parts of meetings, e.g., the main topic 'scope of the project and team building' in the leftmost part of Figure 2. So annotators are asked to label all the relevant spans, and these spans are allowed to be not contiguous.
Whether Chatting Belongs to an Independent Main Topic or not. Main topics should be objectively cover most of the meeting contents. In meetings, group members might take much time chatting. Though this is not close to the main theme of the meetings, it should also be counted as a main topic if the chat took lots of time.
A.2.2 Query Generation
Diverse Query Types. For the specific queries, we encourage the annotators to choose different schemas to compose queries, since we intend to diversify the query types and reduce the bias towards certain query types.
Independent Query-answer Pairs. Each query-answer pair should be independent, that is, not dependent on previous queries and answers. For example, for the query 'How did they reach agreement afterwards?', it seems that the query is dependent on the previous annotations, and it is ambiguous to know when they reached agreement and what the agreement referred to if we treat the query independently. Annotators should specify the information of when they reached an agreement and what the agreement is to make the query clear.
For example, annotators could rewrite the query as 'How did the group agree on the button design when discussing the design of remote control?'
A.2.3 Query-based Summarization
Informative Writing. When answering queries like 'What did A think of X?', annotators were required to not only summarize what A said, but also briefly mention the context which is relevant to what A said. This is designed to rich the contents of summaries and further challenge the model's capability of summarizing relevant contexts.
Relevant text spans' annotation. Since there might be multiple relevant text spans, we asked annotators to annotate all of them.
The Usage of Tense. Since all the meetings happened, we ask annotators to use past tense.
How to Denote Speakers. If the gender information is unclear, we would ask annotators not to use 'he/she' to denote speakers. We also asked them not to abbreviations (e.g., PM) to denote speakers, and use the full name like 'Project Manager' instead.
Abbreviations. In the raw meeting transcripts, some of abbreviations are along with character  $\underline{\cdot}_{-}^{\cdot}$  If the annotators encountered abbreviations, e.g., L_C_D_,A_A_A_',etc., they would be required to rewrite them like LCD or AAA.
A.3 Annotation Review Standards
We launched a 'pre-annotation' stage in which annotators were asked to try annotating one meeting and the experts who are good at our annotation task would review it and instruct the annotators by providing detailed feedback. Requirements include 1) faithfulness, 2) informativeness, 3) lengths of the relevant text spans of designed queries, 4) typo errors, etc. They could continue annotating only if they passed 'pre-annotation' stage. After 'pre-annotation', experts would keep carefully reviewing all the annotations according to the requirements. We write down the standards for the three annotation stages individually in annotation instruction. Details of annotation review standards could be referred to Appendix A.2.
A.4 Examples of QMSum Dataset
We show two examples of our proposed QMSum dataset. One belongs to product meeting (Table 7), and the other one is about committee meeting (Table 8).
General Query Schema List
A:Speaker
1. Summarize the whole meeting. / What did the group/committee discuss in the meeting? (Mandatory)
2. What did A say in the meeting? / Summarize what A said.
3. What was the conclusion / decision of the meeting?
4. What was the purpose of the meeting?
5. How did the group/committee split the work?
Specific Query Schema List
A, B: Speakers, X: Annotated Main Topics, Y: Subtopics regarding to X
1. Summarize the discussion about X. / What did the group/committee discuss X? (Mandatory)
2. Why did the group/committee decide to do sth. when discussing X?
3. What did A think of Y when discussing X? / Summarize A's opinions towards Y.
4. What did A think of X? / Summarize A's opinions towards X. / What did A propose in the discussion about X?
5. What was the advantage / disadvantage of sth. with regard to X?
6. Why did A think regarding to X?
7. Why did A agree / disagree with certain ideas? / Provide the reasons why A held certain opinions towards X.
8. Why did A think of Y when discussing X?
9. What was the decision / conclusion of the discussion about X? / Summarize the decision of the discussion about X.
10. Why did A agree / disagree with B when discussing X?
11. What did A recommend to do when discussing X and why?
12. What did A learn about topic X?
13. What did A and B discuss X?
Table 6: General and specific query schema lists. A, B denote speaker names. X indicates one of the annotated main topics, and Y means the subtopics regarding to X.
Product Meeting (IS1000a)
Color: Speakers, Main Topics, Subtopics
Turn 0: User Interface Designer: Okay.
Turn 243: Project Manager: Well, this uh this tool seemed to work.
Turn 257: Project Manager: More interesting for our company of course, uh profit aim, about fifty million Euro. So we have to sell uh quite a lot of this uh things. ....
Turn 258: User Interface Designer: Ah yeah, the sale man, four million.
Turn 259: User Interface Designer: Maybe some uh Asian countries. Um also important for you all is um the production cost must be maximal uh twelve uh twelve Euro and fifty cents.
Turn 275: Project Manager: So uh well I think when we are working on the international market, uh in principle it has enough customers.
Turn 276: Industrial Designer: Yeah.
Turn 277: Project Manager: Uh so when we have a good product we uh we could uh meet this this aim, I think. So, that about finance. And uh now just let have some discussion about what is a good remote control and uh well keep in mind this this first point, it has to be original, it has to be trendy, it has to be user friendly. ....
Turn 400: Project Manager: Keep it in mind.
Annotated Main Topics
Scope of the project and team building (Turn 41 - 245)
Cost constraints and financial targets of the new remote control project (Turn 246 - 277)
Remote control style and use cases (Turn 277 - 295)
Prioritizing remote control features (Turn 343 - 390)
Queries and Annotated Summaries
Query 1: Summarize the whole meeting.
Answer: Project Manager introduced a new remote control project for television sets, and the team got acquainted with each other and technical devices. The remote control would be priced at 25 Euros and a production cost of 12.5 Euros. ....
Query 2: What did the group discuss about prioritizing remote control features?
Answer: User Interface Designer and Industrial Designer expressed a desire to integrate cutting-edge features into the remote. Marketing pointed out that most of the market would buy it for standard use, like changing channels and adjusting volume ....
Relevant Text Spans: Turn 343 - 390
Query 4: Why did Marketing disagree with Industrial Designer when discussing prioritizing remote control features?
Answer: Marketing believed that fancy features like IP would not be used by most people. The overwhelming majority of users would want convenient channel browsing and volume adjustment features ....
Relevant Text Spans: Turn 358
Query 7: What did Project Manager think of the cost constraints and financial targets of the new remote control project?
Answer: Project Manager introduced the financial information: 25 Euro selling price and 12.5 Euro production cost. Project Manager then went on to elaborate that the target market would primarily consist of Europe and North America. ....
Relevant Text Spans: Turn 248 - 277
Table 7: A product meeting annon example in QMSum dataset.
Committee Meeting (Education 4)
Turn 0: Lynne Neagle AM: Okay, good morning, everyone. Welcome to the Children, Young People and Education Committee this morning. I've received apologies for absence from ....
Turn 31: David Hopkins: Yes, sure. The delegation levels are already very high in most authority areas, and we've got agreements in place with the Government to make sure that more money, or as much money as possible ....
Turn 32: Sian Gwenllian AM: Okay. But just the pressures coming in with the new Act et cetera could mean more expulsions.
Turn 33: David Hopkins: It shouldn't, but it could. It's difficult to know how headteachers and governing bodies will react. ....
Turn 44: Sharon Davies: As Nick said, it does get more difficult at key stage 4, and it's working, then, with. It comes back to that team-around-the-family approach ....
Turn 47: David Hopkins: I don't think I'm allowed to say at this point.
Turn 228: Lynne Neagle AM: Item 4, then, is papers to note. Just one paper today, which is the Welsh Government's response to the committee's report on the scrutiny of the Welsh Government's draft budget 2020-1. ....
An increase of exclusions from school and current measures against it (Turn 1 - 19, Turn 158 - 172)
The funding issues (Turn 20 - 38, Turn 177 - 179)
The networking within the PRU and the transition arrangements (Turn 39 - 56)
Schools' awareness of early trauma ACEs (Turn 180 - 188)
Answer: The meeting was mainly about the reasons behind and the measurements against the increasing exclusions from school. The increase brought more pressure to EOTAS in the aspects of finance, transition, curriculum arrangement and the recruitment of professional staff. Although much time and finance had been devoted to the PRU ....
Query 4: What was considered by David Hopkins as the factor that affected exclusions?
Answer: David Hopkins did not think that the delegation levels were not high enough in most authority areas. Instead, he thought they had got agreements with the government to make sure that enough money was devolved to school. The true decisive factor was as the narrow measure at the end of Stage 4 that drove the headmasters to exclude students or put them into another school.
Relevant Text Spans: Turn 31 - 33
Query 6: What was the major challenge of the transition of the excluded students?