SlowGuess/NAACL_2022_OCR huggingface dataset:
synthetic queries from an external dataset using masking. We find that the single-encoder REL-REG outperforms the best dual-encoder model; the cross-attention term in the single-encoder REL-REG model allows it to better attend to the query when determining relevance. Intuitively, the ordering of results corresponds to the span overlap recall with the gold spans; the ability of the extractor to select produce high-recall rankings directly affects abstractor performance. We see that increasing the input segment length used in training and inference for RELREG improves at 256 tokens but decreases at 512 tokens, suggesting that a balance is found between including additional context for ranking versus enabling a greater number of shorter segments that may capture more diverse content from the source.
4.2 End-to-End Approaches
We explore hyperparameter choices for two end-to-end architectures described in §3.2: the Longformer Encoder-Decoder (LED) and Segment Encoder (SEGENC). For both models, we consider different choices for input size (4096, 8192, or 16384
Model	Input	Attn	R-1	R-2	R-L
BART	1024	1024	32.42	9.62	28.37
LED	4096	256	31.55	8.89	27.62
512	32.25	9.27	28.29
1024	32.16	9.05	28.27
8192	256	31.79	8.97	27.75
512	32.76	9.38	28.65
1024	32.85	9.26	28.73
16384	256	31.94	9.16	27.73
512	32.88	9.82	28.90
1024	32.98	9.60	29.08
SEGENC	4096	256	35.35	10.37	30.91
512	35.25	10.36	30.85
1024	34.36	9.85	30.13
8192	256	36.51	11.36	31.87
512	36.68	11.71	32.08
1024	35.48	10.97	31.21
16384	256	37.21	12.14	32.67
512	37.47	12.47	32.95
1024	36.30	11.71	32.01
SEGENC-D	16384	512	36.68	11.97	32.35
Table 3: Performance of end-to-end models on the QM-Sum validation set, across varying input and attention window sizes (in number of tokens). SEGENC-D is a variant of SEGENC in which the segments are disjoint rather than overlapping; this ablation was evaluated on the best-performing SEGENC hyperparameters.
tokens) and attention window size $^4$  (256, 512, or 1024 tokens). For SEGENC, we also consider two different segmentation strategies: overlapping segments (50% overlap) and disjoint segments. Validation set results for both models and a baseline BART model are reported in Table 3.
We notice that both the LED and SEGENC benefit from increasing the input size and perform best with the input limit set to 16,384 tokens. The optimal attention window for LED is 1024, while SEGENC performs best with an attention window
4For SEGENC, attention window size is equivalent to segment size.
Model	R-1	R-2	R-L
No Transfer	32.42	9.62	28.37
AnswerSumm	34.36	9.64	30.22
AQuaMuse	34.57	9.78	30.42
WikiHowQA	33.08	9.03	28.48
CNNDM	33.87	9.36	28.48
WikiSum	34.73	9.80	30.54
of 512 tokens. For SEGENC, using overlapping segments improves performance compared to using disjoint segments, suggesting that the additional context provided by the former approach is helpful for locating relevant content. The SEGENC model achieves the highest performance out of the end-to-end architectures with ROUGE scores of 37.47  $R-1$ , 12.47  $R-2$ , and 32.95  $R-L$  on the validation set.
The results also highlight that while the LED model matches or slightly outperforms the BART baseline for higher maximum input and window sizes, it performs substantially worse than SEGENC. This observation is consistent with prior findings on the QMSum dataset (Zhang et al., 2021b). One possible explanation for the lower performance of LED relative to SEGENC is that LED must adapt its parameters for a global attention mechanism that is absent from the backbone BART encoder model, whereas SEGENC relies solely on local self-attention that is aligned with the backbone model. This may be particularly relevant to QMSum given its relatively small size.
Practitioners may wish to consider the computational cost and efficiency of various hyperparameter settings. Computational complexity increases with both input length and attention window size (since attention grows quadratically in attention-window size). Complexity is also greater with the overlapping segment strategy compared to the disjoint segment strategy for the SEGENC model, due to the greater number of resulting segments that are passed through the encoder and decoder modules.
4.3 Task-Specific Transfer
Having determined the best-performing models, we examine whether performance can be further improved by fine-tuning a model that has already been fine-tuned for a different summarization task. We conduct this study using the end-to-end BART on 1024 tokens, as this model is the backbone, al
Table 4: QMSum validation-set performance of the end-to-end BART models first fine-tuned on related summarization tasks and then further fine-tuned on QMSum data. The model indicates the task first fine-tuned on, and input is truncated to 1024 tokens.
Model	R-1	R-2	R-L
Baselines
DYLE	34.42	9.71	30.10
SUMMN	34.03	9.28	29.48
BART	31.87	9.08	27.50
BART-W	32.68	8.97	28.74
BART-W (Gold)	39.54	15.65	35.17
Two-stage
DPR	32.28	9.73	28.34
RELREGTT	33.02	10.17	28.90
MARGE	31.99	8.97	27.93
RELREG	34.91	11.91	30.73
RELREG-W	36.45	12.81	32.28
End-to-end
LED	34.18	10.32	29.95
SEGENC	37.05	13.03	32.62
SEGENC-W	37.80	13.43	33.38
Table 5: QMSum test-set performance of two-stage and end-to-end models that performed best on the validation set (Tables 2 and 3), including versions fine-tuned from the WikiSum-finetuned checkpoint (denoted by -W). Results reported in prior work are italicized. Also included is an extractive-oracle model that takes the gold spans (§3.3) as input.
beit in varying ways, of both our two-step and end-to-end models. We test the transferring capabilities of models trained on the news summarization task from CNN/DailyMail (Nallapati et al., 2016), which performed best among non query-focused datasets in Zhang et al. (2021b). We also explore transferring from the previously-mentioned query- and topic-focused summarization tasks: AnswerSumm, AQuaMuSe, WikiHowQA, and WikiSum. We compare to fine-tuning from the original BART checkpoint, with results shown in Table 4.
We find that transferring from any of the tasks improves over no transfer in R-1 and R-L. Transferring from any of the constrained, query-focused tasks outperforms transferring from unconstrained news summarization. Furthermore, transferring from WikiSum outperforms transfer from other datasets, which aligns with other work that shows the generalizability of Wikipedia as a source of data for task transfer (Fabbri et al., 2021a).
4.4 Final Results
We now measure the test set performance of the best-performing architectures from §4.1 and §4.2 in combination with the optimal transfer-learning approach from §4.3. Results are presented in Table 5 along with baseline models.
We find that RELREG and SEGENC outperform existing state-of-the-art models by a substantial margin, and that initializing the model from the
Model	Flu.	Rel.	Comp.	Fact.
BART	4.08	3.68	3.22	3.31
RELREG-W	3.87	3.81	3.67	3.70
SEGENC-W	3.93	3.87	3.81	3.63
Table 6: Human evaluation of two best-performing models from Section 4, along with a baseline BART model. Summaries were evaluated across four dimensions: fluency (Flu.), relevance (Rel.), completeness (Comp.), and factuality (Fact.).
Wikisum-fine-tuned checkpoint further improves performance, with the best model exceeding current state-of-the-art performance by a difference of 3.38 R-1, 3.72 R-2, and 3.28 R-L. Comparing the best models from each category, we find that the end-to-end approach outperforms the two-stage. Within the two-stage dual-encoder models, RELREGTT outperforms DPR on the test set despite the slightly worse performance on the validation set. We attribute this variation to the small size of the validation set, and our other findings remain consistent across validation and test sets. The single-encoder RELREG outperforms the best dual-encoder model, with RELREG-W improving upon the current state-of-the-art performance by a difference of 2.03 R-1, 3.10 R-2, and 2.18 R-L.
In this section we conduct further analysis of the best performing models from Section 4. First, we offer additional insights into the performance of those models on the QMSum dataset through a human-based study. Next, we discuss the generalization abilities of those models by running experiments on the AQuaMuSe dataset.
To gain a better understanding of the performance of the models on the QMSum dataset, human judges were hired and asked to assess the quality of generated summaries. Summaries were evaluated across four dimensions: 1) fluency, measuring their grammatical quality, 2) relevance, assessing their relevance to the input query, 3) completeness, evaluating their comprehensiveness considering the input conversation and query, and 4) factuality, measuring their factual consistency with respect to the conversation. Scores were assigned on a Likert scale from 1 to 5 (best), where each example was evaluated by 3 judges with the final score averaged. Results are presented in Table 6.
Model	R-1	R-2	R-L
Hi-MAP	30.34	14.82	26.86
BART	48.74	33.96	46.02
RELREG-W	54.06	38.51	51.07
SEGENC-W	63.62	51.27	61.37
Table 7: AQuaMuSe test-set performance of two best-performing models from §4, along with a baseline BART model and previously reported results (in italics) for Hi-MAP (Fabbri et al., 2019a) from Kulkarni et al. (2020). Note that the version of the dataset used for previous results would have been slightly different due to variations in document selection parameters and Common Crawl indices (see Appendix).
We find that the RELREG-W and SEGENC-W models achieved comparable performance across all of the evaluated dimensions, with summaries generated by SEGENC-W rated as slightly more complete. The BART baseline was rated highest in the fluency dimension, however, it was substantially outperformed by both of the introduced models on completeness and factuality. One possible explanation for the slightly lower fluency scores for the RELREG-W and SEGENC-W models is that they are better able to retrieve content from the source, which itself may have low fluency due to its conversational nature. The results also highlight a gap between the performance of existing models and perfect scores, which shows that there is potential for improvement in future work.
5.2 Dataset Generalization
To test that the automated evaluation results generalize beyond the QMSum dataset, we trained and evaluated the best-performing models on AQuaMuSe, another high-quality dataset for QFS that includes long documents (§2.1, §3.3). Test-set performance for the best-performing two-stage and end-to-end models, along with a baseline BART model, are shown in Table 7. Results are consistent with those for the QMSum dataset (Table 5), with the best end-to-end model (SEGENC-W) outperforming the best two-stage model (RELREG-W), and both outperforming the baseline (BART) model.
In this work, we conducted an exploratory study of neural models for query-focused summarization. We studied two categories of models: two-stage and end-to-end, and presented two architectures, RELREG and SEGENC, both of which improve ROUGE performance over prior state of the art
by a substantial margin. We also explored task-specific transfer learning, which further improved model performance. Besides model performance, we discussed issues of computational efficiency that practitioners may factor into their modeling choices. Finally, we conducted a human study suggesting that the summaries produced by the best-performing models are more factually correct and complete than a baseline model by a substantial margin. We hope that the analysis and modeling contributions of this paper will be a resource for future research on query-focused summarization.
Dataset Biases QMSum and AQuaMuSe contain meeting transcripts and documents in English and thus mainly represent the culture of the English-speaking populace. Political or gender biases may also exist in the dataset, and models trained on these datasets may propagate these biases. Additionally, the pretrained BART model carries biases from the data it was pretrained on. We did not stress test these models for biases and request that the users be aware of these potential issues in applying the models presented.
Crowdsourcing Protocols Workers were compensated  $1 per example, calibrated to equal a$ 12/hour payrate. We use the following qualifications to recruit MTurk workers with good track records: HIT approval rate greater than or equal to 97%, number of HITs approved greater than or equal to 10000, and located in one of the following English native-speaking countries: Australia, Canada, New Zealand, United Kingdom, United States.
Misuse Potential and Failure Mode When properly used, the summarization models described in this paper can be time-saving. However, the current model outputs may be factually inconsistent with the input documents, and in such a case could contribute to misinformation on the internet. This issue is present among all current abstractive summarization models and is an area of active research.
Environmental Cost The experiments described in the paper primarily make use of A100 GPUs. We typically used a single GPU per experiment, and the experiments may take up to a day when repeating across random seeds. The largest backbone model used, BART-Large, has 400 million parameters. While our work required extensive experiments,
future work and applications can draw upon our insights and need not repeat these comparisons.
We thank Divyansh Agarwal for his insights on existing model architectures and assistance with training DPR. We thank Semih Yavuz for providing the initial implementation of the Fusion-in-Decoder model.
Ahsaas Bajaj, Pavitra Dangati, Kalpesh Krishna, Pradhiksha Ashok Kumar, Rheeya Uppaal, Bradford Windsor, Eliot Brenner, Dominic Dotterrer, Rajarshi Das, and Andrew McCallum. 2021. Long document summarization in a low resource setting using pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 71-80, Online. Association for Computational Linguistics.
Tal Baumel, Matan Eyal, and Michael Elhadad. 2018. Query focused abstractive summarization: Incorporating query relevance, multi-document coverage, and summary length constraints into seq2seq models. ArXiv preprint, abs/1801.07704.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer.
Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2021. Summscreen: A dataset for abstractive screenplay summarization. ArXiv preprint, abs/2104.07091.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686, Melbourne, Australia. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pretraining text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 615-621, New Orleans, Louisiana. Association for Computational Linguistics.
Peng Cui and Le Hu. 2021. Sliding selector network with dynamic memory for extractive summarization of long documents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5881-5891, Online. Association for Computational Linguistics.
Hoa Trang Dang. 2005. Overview of duc 2005. In Proceedings of the document understanding conference, volume 2005, pages 1-12.
Yang Deng, Wai Lam, Yuexiang Xie, Daoyuan Chen, Yaliang Li, Min Yang, and Ying Shen. 2020. Joint learning of answer selection and answer summary generation in community question answering. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7651-7658. AAAI Press.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Elozino Egonmwan, Vittorio Castelli, and Md Arafat Sultan. 2019. *Cross-task knowledge transfer for query-based text summarization*. In *Proceedings of the 2nd Workshop on Machine Reading for Question Answering*, pages 72-77, Hong Kong, China. Association for Computational Linguistics.
Alexander Fabbri, Simeng Han, Haoyuan Li, Haoran Li, Marjan Ghazvininejad, Shafiq Joty, Dragomir Radev, and Yashar Mehdad. 2021a. Improving zero and few-shot abstractive summarization with intermediate fine-tuning and data augmentation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 704-717, Online. Association for Computational Linguistics.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019a. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy. Association for Computational Linguistics.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019b. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy. Association for Computational Linguistics.
Alexander R. Fabbri, Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir R. Radev. 2021b. Summeval: Re-evaluating summarization evaluation. Trans. Assoc. Comput. Linguistics, 9:391-409.
Alexander R. Fabbri, Xiaojian Wu, Srini Iyer, Haoran Li, and Mona Diab. 2021c. Answersumm: A manually-curated dataset and pipeline for answer summarization.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419-1436, Online. Association for Computational Linguistics.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874-880, Online. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 540-551, Hong Kong, China. Association for Computational Linguistics.
Wojciech Krysciński, Nazneen Fatema Rajani, Dipyansh Agarwal, Caiming Xiong, and Dragomir R. Radev. 2021. Booksum: A collection of datasets for long-form narrative summarization. ArXiv preprint, abs/2105.08209.
Sayali Kulkarni, Sheide Chammas, Wan Zhu, Fei Sha, and Eugene Ie. 2020. Aquamuse: Automatically generating datasets for query-based multi-document summarization.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. 2020. Pytorch distributed: Experiences on accelerating data parallel training. Proc. VLDB Endow., 13(12):3005-3018.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Marina Litvak and Natalia Vanetik. 2017. Query-based summarization using MDL principle. In Proceedings of the MultiLing 2017 Workshop on Summarization and Summary Evaluation Across Source Types and Genres, pages 22-31, Valencia, Spain. Association for Computational Linguistics.
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018a. Generating wikipedia by summarizing long sequences. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018b. Generating wikipedia by summarizing long sequences. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Yang Liu and Mirella Lapata. 2019. Hierarchical transformers for multi-document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5070-5081, Florence, Italy. Association for Computational Linguistics.
Ye Liu, Jian-Guo Zhang, Yao Wan, Congying Xia, Lifang He, and Philip S. Yu. 2021. Hetformer: Heterogeneous transformer with sparse attention for long-text extractive summarization.