SlowGuess/NAACL_2022_OCR huggingface dataset:
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.11692.
Zhengyuan Liu and Nancy F. Chen. 2021. Dynamic sliding window for meeting summarization.
Potsawee Manakul and Mark J. F. Gales. 2021. Sparsity and sentence structure in encoder-decoder attention of summarization systems.
Ziming Mao, Chen Henry Wu, Ansong Ni, Yusen Zhang, Rui Zhang, Tao Yu, Budhaditya Deb, Chenguang Zhu, Ahmed H. Awadallah, and Dragomir Radev. 2021. Dyle: Dynamic latent extraction for abstractive long-input summarization.
Ramesh Nallapati, Bing Xiang, and Bowen Zhou. 2016. Sequence-to-sequence rnns for text summarization. ArXiv preprint, abs/1602.06023.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Laura Perez-Beltrachini, Yang Liu, and Mirella Lapata. 2019. Generating summaries with topic templates and structured convolutional decoders. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5107-5116, Florence, Italy. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Eva Sharma, Chen Li, and Lu Wang. 2019. BIGPATENT: A large-scale dataset for abstractive and coherent summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2204-2213, Florence, Italy. Association for Computational Linguistics.
Dan Su, Yan Xu, Tiezheng Yu, Farhad Bin Siddique, Elham Barezi, and Pascale Fung. 2020. CAiRE-COVID: A question answering and query-focused
multi-document summarization system for COVID-19 scholarly information management. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online. Association for Computational Linguistics.
Dan Su, Tiezheng Yu, and Pascale Fung. 2021. Improve query focused abstractive summarization by incorporating answer relevance. In *Findings of the Association for Computational Linguistics: ACLIJCNLP* 2021, pages 3124-3131, Online. Association for Computational Linguistics.
Xiaojun Wan, Jianwu Yang, and Jianguo Xiao. 2007. Manifold-ranking based topic-focused multi-document summarization. In *IJCAI*, volume 7, pages 2903-2908.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv preprint, abs/1910.03771.
Yujia Xie, Tianyi Zhou, Yi Mao, and Weizhu Chen. 2020. Conditional self-attention for query-based summarization.
Yumo Xu and Mirella Lapata. 2020. Coarse-to-fine query focused multi-document summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3632-3645, Online. Association for Computational Linguistics.
Yumo Xu and Mirella Lapata. 2021a. Generating query focused summaries from query-free resources. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6096-6109, Online. Association for Computational Linguistics.
Yumo Xu and Mirella Lapata. 2021b. Text summarization with latent queries. ArXiv preprint, abs/2106.00104.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. PEGASUS: pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 11328-11339. PMLR.
Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed H. Awadallah, Dragomir Radev, and Rui Zhang. 2021a.  $\mathrm{Summ}^{\mathrm{N}}$ : A multi-stage summarization framework for long input dialogues and documents.
Yusen Zhang, Ansong Ni, Tao Yu, Rui Zhang, Chenguang Zhu, Budhaditya Deb, Asli Celikyilmaz, Ahmed Hassan Awadallah, and Dragomir Radev. 2021b. An exploratory study on long dialogue summarization: What works and what's next.
Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. SEAL: segment-wise extractive-abstractive long-form text summarization. ArXiv preprint, abs/2006.10213.
Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021a. Dialoglm: Pre-trained model for long dialogue understanding and summarization.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021b. QMSum: A new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905-5921, Online. Association for Computational Linguistics.
Locator Model Parameters For MARGE experiments, we apply the original fine-tuned BERT-base checkpoint from Xu and Lapata (2021a), while for DPR, we fine-tune a BERT-base model for both query and passage encoders following Karpukhin et al. (2020).
We report results for RELREG fine-tuned from an Electra-large checkpoint (Clark et al., 2020). For a fair comparison with other metrics, we also fine-tuned RELREG from a BERT-base checkpoint. This version still outperformed DPR by about a point in R-1, R-2, and R-L, demonstrating the advantage of this locator approach beyond the chosen base model.
We apply RELREGTT fine-tuned from a distilled RoBERTa base (Liu et al., 2019) checkpoint initially fine-tuned for the task of entailment. This approach of continuing fine-tuning from an entailment checkpoint is suggested by the sentence transformers library (Reimers and Gurevych, 2019). We also experimented with fine-tuning the REL-REGTT model from BERT-base and Electra-large checkpoints, but these locators did not perform better in initial experiments.
Model	Lexical Overlap b/w
Top-1	Top-5	Top-15	All	Top-5	Top-5	Top-5	Top-5	Top-5	Top-15	All	All	All	All	All	Precision	Recall
R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	x̂	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	X̂	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L	x̂	R-1	R-2	R-L
GOLD SPANS	15.00	3.80	11.10	60	20.89	6.05	15.04	218	19.62	5.99	14.28	386	16.09	5.60	12.47	660	11.01	2.75	9.90	60	7.30	1.58	6.24	218	4.73	1.10	4.07	386	3.53	0.93	3.05	660	0.75	1.00
Table 8: Performance of extractor models on the validation set. The left and middle sections present the lexical overlap between utterances retrieved by extractor models and the reference summaries and summary queries, accordingly. Lexical overlap is evaluated by means of ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L) metrics. Segments of the section focus on the lexical overlap between the highest ranked 1 (Top-1), 5 (Top-5), 15 (Top-15) utterances, and all utterances truncated to a 1024 token limit (All). The table also includes the average word counts of all extracted utterances, denoted as  $\bar{x}$ . The right section shows the span overlap between the utterance spans retrieved by the extractor models and those collected from human annotators by the authors of QMSum. The performance is evaluated by means of Precision and Recall scores and uses the highest ranked utterances truncated to the limit of 1024 tokens.
Summarization Model Parameters In all experiments described in this work, the LED model was initialized from the allenai/led-large-16384 checkpoint. Two model hyperparameters, maximal input size and attention window size, were chosen through a hyperparameter search with candidate models selected based on their performance on the validation set. Best hyperparameters were found to be: 16384 maximum input size, and 1024 attention window size. LED models were trained for 10 epochs, with a batch size 1, gradient accumulation set to 4 steps, and learning rate set to 0.000005. The SEGENC model was initialized from the facebook/bart-large checkpoint. The model hyperparameters, maximal input size and attention window size, were chosen through a hyperparameter search with candidate models selected based on their performance on the validation set, with results reported in the paper. Best hyperparameters were found to be: 16384 maximum input size, and 512 attention window size. The SEGENC models were trained for 10 epochs, with a batch size of 1 and learning rate set to 0.000005.
QMSum Details QMSum contains 1,808 query-summary pairs in total, with a train/validation/test split of 1257/272/281. It is made available through an MIT license<sup>5</sup>, which aligns with our use for research purposes. Non-identifying names are used in place of real names.
AQuaMuse Details We experiment the V3, abstractive version of AQuaMuse, consisting of 7725 query-summary pairs, with a train/validation/test
split of 5566/596/734. The original AQuaMuse paper reported results on V2 of the dataset, which contains a slightly different input document set due to variations in the semantic overlap threshold used to retrieve documents. Some input documents could not be retrieved due to differences in the Common crawl index used; we use the cleaned, reproduced version of the C4 dataset (Raffel et al., 2020) from the Common crawl made available by AI $^{6}$ . We kept examples for which all input documents were found, which resulted in a dataset of 6896 examples. The natural language questions it contains are made available through an Apache 2.0 license $^{7}$ , which aligns with our use for research purposes. This dataset uses publicly available entities from Wikipedia.
B Human Annotation Interface
The instructions shown to the annotators during human studies are presented in Figure 1
$^{6}$ https://github.com/allenai/allennlp/discussions/5056
$^{7}$ https://github.com/  
google-research-datasets/  
natural-questions/blob/master/License
https://github.com/Yale-LILY/QMSum/blob/main/License
In this task you will evaluate the quality of summaries of a conversation.
The summaries were written to answer a question about the conversation.
1. Carefully read the Question and the Conversation, be aware of the information they contain.
2. Read the proposed summaries A-C (3 in total).
3. Rate each summary on a scale from 1 (worst) to 5 (best) by its fluency, relevance, completeness, factuality.
This rating measures the grammatical quality of the Summary text, is it well-written and grammatically correct.
Check the quality of individual sentences.
Fluency can be rated without considering the Conversation or the Question.
Relevance:
The rating measures how relevant the Summary is to the Question ignoring the Conversation.
Check whether the content of the Summary is on topic with respect to the Question.
Relevance must be rated considering the Question.
Completeness:
The rating measures how completely/comprehensively the Summary answers the Question.
Check whether all necessary information from the Conversation is present in the Summary.
Completeness must be rated considering the Question and Conversation.