SlowGuess/NAACL_2022_OCR huggingface dataset:
Jesse Vig* Alexander R. Fabbri* Wojciech Krysciński* Chien-Sheng Wu Wenhao Liu
{jvig, afabbri, kryscinski, wu.jason, wenhao.liu} @salesforce.com
Query-focused summarization (QFS) aims to produce summaries that answer particular questions of interest, enabling greater user control and personalization. While recently released datasets, such as QMSum or AQuaMuSe, facilitate research efforts in QFS, the field lacks a comprehensive study of the broad space of applicable modeling methods. In this paper we conduct a systematic exploration of neural approaches to QFS, considering two general classes of methods: two-stage extractive-abstractive solutions and end-to-end models. Within those categories, we investigate existing models and explore strategies for transfer learning. We also present two modeling extensions that achieve state-of-the-art performance on the QMSum dataset, up to a margin of 3.38 ROUGE-1, 3.72 ROUGE-2, and 3.28 ROUGE-L when combined with transfer learning strategies. Results from human evaluation suggest that the best models produce more comprehensive and factually-consistent summaries compared to a baseline model. Code and checkpoints are made publicly available: https://github.com/salesforce/query-focused-sum.
Text summarization aims at transforming long documents into short snippets that contain only the most important information from the source document. The field has seen substantial progress driven by the availability of large-scale models pre-trained on vast amounts of data (Devlin et al., 2019; Lewis et al., 2020), the development of summarization-specific pre-training strategies (Zhang et al., 2020; Zhao et al., 2020), and computationally efficient neural architectures (Zaheer et al., 2020).
The majority of recent research efforts in text summarization assume an unconstrained setting
in which models are given only a source document as input and are expected to generate a general summary covering the salient aspects from the source. The performance of such models has been evaluated on benchmark datasets spanning various domains: news articles (Nallapati et al., 2016; Narayan et al., 2018; Fabbri et al., 2019a), legal documents (Sharma et al., 2019), scientific writing (Cohan et al., 2018), or creative writing (Kryscinski et al., 2021; Chen et al., 2021). However, it has been shown that summarization in an unconstrained setting is an ill-defined task where multiple generated summaries are equally relevant (Kryscinski et al., 2019). This in turn hinders the ability to evaluate and understand the models' content selection capacity. In addition, such generic summarization models lack control mechanisms that would allow end users to customize summaries to their particular needs and expectations.
Query-focused summarization (QFS) is a subtask within text summarization that focuses on generating summaries where the summary content is tailored to a user-specified query that is passed alongside the source document as input to the model. Each source document can be associated with multiple unique queries inquiring about different information from that document. In this setting, end users are enabled to explicitly specify their preferences for the summary, and the relevance of the output summary may be evaluated more precisely with respect to the input query. Research on this task has been accelerated by the recently introduced high-quality datasets, such as QMSum (Zhong et al., 2021b) and AQuaMuSe (Kulkarni et al., 2020).
In this work we conduct a systematic, exploratory study of different approaches to query-focused text summarization, considering both two-step and end-to-end neural methods. We present two models, RELREG and SEGENC, which achieve state-of-the-art ROUGE scores on the QMSum dataset, up to a margin of 3.38 R-1, 3.72 R-2, and
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1455 - 1468 July 10-15, 2022 ©2022 Association for Computational Linguistics
3.28 R-L when combined with transfer learning methods. The RELREG model uses a two-step approach to solving the problem, where the first step extracts content relevant to the given query and the next step synthesizes the extracted fragments into a coherent summary. The SEGENC method follows an end-to-end framework in which individual document segments are separately encoded to avoid the computational bottleneck of long input documents, and the decoder jointly attends to all encoded segments when producing the summary. Through quantitative studies, we compare our models with other baselines and discuss the trade-offs of the end-to-end methods and pipelined approaches. We also perform human evaluation to understand the qualitative differences between the models. Together with this manuscript, we share the code base and model checkpoints to enable future research in this area.
2.1 Query-Focused Summarization
Query-focused summarization aims to generate a summary of a given text conditioned upon a query. Initial work in this area centered around unsupervised extractive approaches (Wan et al., 2007; Litvak and Vanetik, 2017) due to the limited availability of task-specific training data (Dang, 2005). More recent work has taken advantage of the relationship between query-focused summarization and the more data-rich task of question answering for extractive summarization (Egonmwan et al., 2019), reranking documents within a retrieval pipeline (Su et al., 2020), and abstractive summarization (Su et al., 2021; Baumel et al., 2018; Xie et al., 2020). Xu and Lapata (2020) introduce a pipeline consisting of a relevance estimator filter followed by query-focused evidence and centrality estimators, while other work converts generic summarization dataset to query-focused training data (Xu and Lapata, 2021a) or performs latent query modeling (Xu and Lapata, 2021b).
Recently, several query-focused summarization datasets have been introduced, which can be further divided into short-document datasets, whose source document length does not exceed the input limits of standard pretrained models, and long-document datasets. Within short-document, query-focused summarization, AnswerSumm (Fabbri et al., 2021c) is composed of summaries of answers to queries from StackExchange forums, while Wik
iHowQA (Deng et al., 2020) proposes the task of answer selection followed by the summarization of individual response articles to queries from the how-to site WikiHow. Within long-document summarization, WikiSum (Liu et al., 2018a) consists of Wikipedia article titles as queries, the first paragraph of the article as the summary, and documents referenced by the article as the input. AQuaMuSe (Kulkarni et al., 2020) is a query-focused multi-document summarization dataset with user-written queries and human-verified long-answer summaries from the Natural Questions dataset (Kwiatkowski et al., 2019), and QMSum (Zhong et al., 2021b) is a manually-curated dataset for query-focused dialog summarization. QMSum and AQuaMuSe are of particular interest to our study due to the combined challenges of query-focused and long-document summarization and the presence of high-quality, curated query-summary pairs.
Recent work on QMSum has introduced task-specific denoising objectives for meeting summarization (Zhong et al., 2021a), generated final fine-grained summaries based on multiple coarse-grained steps (Zhang et al., 2021a), and treated the extractive text of an extractive-abstractive model as a latent variable (Mao et al., 2021). Zhang et al. (2021b) analyze the challenges of long dialogue summarization such as the input length, the role of queries, and domain adaptation. Our work builds on QA-motivated methods and presents two approaches yet to be applied in query-focused summarization that each achieve state-of-the-art results, including a two-step model and an end-to-end model.
2.2 Long Document Summarization
Long document summarization addresses the setting where source document length exceeds the input limits of standard pre-trained models. Approaches to this task can largely be divided into two categories: two-step extractive-abstractive frameworks, which first extract a subset of the text as input to an abstractive model, and end-to-end models, which process the input within a single model. The two-step pipeline has been applied to topic-focused Wikipedia summarization (Liu et al., 2018b; Liu and Lapata, 2019; Perez-Beltrachini et al., 2019), low-resource summarization (Bajaj et al., 2021), and single-document summarization Chen and Bansal (2018). End-to-end approaches address the input-length problem using sparse
attention models. Beltagy et al. (2020) introduce the Longformer, consisting of local attention as well as global attention between select input tokens. Other approaches make use of dynamic attention mechanisms (Zhao et al., 2020; Manakul and Gales, 2021; Cui and Hu, 2021), sliding window strategies (Liu and Chen, 2021), and other mechanisms to introduce sparsity into the model (Huang et al., 2021; Liu et al., 2021). Izacard and Grave (2021) concatenate the outputs of multiple encoders as input to a generator component for the task of open domain question answering. In our work we build on these models for query-focused summarization and perform extensive hyperparameter ablations, achieving state-of-the-art results over other two-step and end-to-end models.
We present existing methods and propose modeling extensions to address the challenges of query-focused summarization.
3.1 Two-Step Approaches
Two-step approaches consist of an extractor model, which extracts parts of the source document relevant to the input query, and an abstractor model, which synthesizes the extracted segments into a final summary. We consider score-and-rank extractor models, which first score each source passage for relevance to the query and then rank the passages in descending order of relevance, with the concatenated and truncated results passed to the abstractor. In this work we present two types of scoring models: single-encoder models and dual-encoder models, which we describe below. All two-step approaches share the same abstractor, a BART-large model.
3.1.1 Single-Encoder Models
Single encoder models concatenate a query and source passage as input to the scoring function that produces the similarity score. Those models benefit from full cross-attention between query and passage, resulting in richer data representations.
MARGE (Xu and Lapata, 2021a) is a single-encoder, Masked ROUGE extractor that aims to improve upon low-resource query-focused summarization by synthesizing query-focused data from more resource rich, generic summarization datasets. This model is trained to predict the relevance of each passage in the source document with respect
to a query, where the proxy for relevance is the ROUGE overlap between the passage and the reference summary. For training on generic summarization datasets, MARGE uses pseudo-queries that are created by masking content words in the reference summaries.
When performing inference using real queries, certain query words (e.g., wh-words) are masked to better align the queries to the pseudo-queries from the training process. Following Xu and Lapata (2021a), we apply MARGE trained for masked relevance prediction on Multi-News (Fabbri et al., 2019b) without training on our target dataset.
RELREG Motivated by the retrieval component of MARGE, we propose the RELREG (RELevance REGression) model, which trains a relevance prediction model directly on QFS data using the original, non-masked query. Like MARGE, this model is trained to predict the ROUGE overlap between a source passage and the reference summary, using only the passage and query as input. A single-encoder model jointly encodes the delimiter-separated query and passage, and the final layer of the model outputs the predicted relevance value.
3.1.2 Dual-Encoder Models
Dual-encoder models separately encode a query and source passage before calculating the cosine similarity between the embeddings to compute the relevance score. This class of models offers computational benefits, as passage embeddings may be precomputed and stored for a given input, while the single-encoder model must be run over all passages should a new query be introduced.
DPR (Karpukhin et al., 2020) is a dual-encoder model that separately encodes queries and passages into an embedding space optimized for calculating semantic similarity between the two, showing improved results over traditional vector-space models. We fine-tune a DPR extractor model directly on the target dataset. As opposed to other locators that optimize with respect to the continuous ROUGE overlap, DPR uses the ROUGE score between the passage and reference summary to identify binary positive and negative passages and optimizes the negative log likelihood of the positive passages.
RELREGTT (RELevance REGression Two Tower) is a more computationally-efficient version of RELREG that uses a dual-encoder architecture to predict ROUGE-based relevance scores. This
model is implemented with a backbone architecture of Sentence-BERT (Reimers and Gurevych, 2019), using a shared-parameter encoder for each of the query and passage and a special token appended to each input that identifies it is as either query or passage, following the suggested best practices of Reimers and Gurevych (2019). The final output for the model is based on the inner product of the pooled embeddings for the query and passage.
3.2 End-to-End Approaches
Two-step pipelines depend on the strength of the retrieval component, and may still fail to capture all relevant content despite an ideal retriever, due to length limitations of the generation component. This motivates our experiments on end-to-end models that can incorporate longer input texts.
BART (Lewis et al., 2020) As a baseline end-to-end model, we consider BART, an encoder-decoder Transformer model pre-trained using a denoising objective. BART is composed of a bidirectional encoder module and an autoregressive decoder model that attends to the encoder's final layer outputs. Due to the quadratic memory complexity of the encoder's full self-attention mechanism, the model input size is limited to 1024 tokens. In our experiments, we prepare the input to BART by concatenating the query, a delimiter token, and the source document, and then truncating the combined text to the model's input size.
LED To circumvent the input size limitations of the BART model, we include the Longformer Encoder-Decoder (Beltagy et al., 2020) (LED) in our study LED replaces the quadratic self-attention mechanism of traditional Transformers with a memory-efficient version that combines local attention with sparse global attention. The architecture allowed us to run experiments with input sizes up to 16384 tokens. Based on insights from the original work on tuning the model to the QA task, we configure the global attention mechanism to span the entire query.
SEGENC We also consider a simpler form of sparse attention in the encoder based solely on windowed local attention, combining elements of LED with Fusion-in-Decoder (FiD) (Izacard and Grave, 2021), a model for open-domain question answering. In our Segment Encoder (SEGENC) model, the source document is split into fixed
length overlapping<sup>1</sup> segments, each of which is separately appended to the query and encoded using a standard Transformer model. Similar to FiD, these encodings are then concatenated into a single embedding sequence and passed to a decoder model that generates the summary. Since there is no cross-attention between the encoded segments, the attention mechanism scales linearly in the number of segments and hence the length of the source document. Nonetheless, the decoder can attend to all encoded segments jointly, enabling the encoder-decoder architecture to operate in an end-to-end fashion. This model is motivated by two hypotheses: 1) query-relevant sections within a source document are often small enough to be processed by standard Transformer models (e.g. 1024 tokens), and 2) each query-relevant section may be understood independently of other sections, removing the need for cross-attention between the segments.
3.3 Data
We analyze our methods on two high-quality query-focused, long-document datasets.
QMSum (Zhong et al., 2021b) is a query-focused dialogue summarization dataset consisting of 1,808 query-summary pairs over 232 meetings from product design, academic, and political committee meetings, all conducted in English. QMSum also includes additional annotations such as topic segmentations and highlighted text spans associated with reference summaries. We leverage the provided span annotations to run oracle experiments. We focus our analysis on QMSum due to the availability of prior work as points of comparison.
AQuaMuSe (Kulkarni et al., 2020) is a query-focused multi-document summarization dataset consisting of 5,519 query-long answer summary pairs from the Natural Questions question-answering dataset (Kwiatkowski et al., 2019) and associated input documents from the Common Crawl<sup>2</sup>. Input documents for the original dataset were selected based on embedding similarity with respect to the summary, and hyperparameters can be chosen to control the level of semantic overlap between the input document set and the summary. Data replication details are found in the Appendix. We use AQuaMuSe to examine the generalizability of our QMSum results.
<sup>1</sup>We use segments that are  $50\%$  overlapping, though other configurations may be considered.
3.4 Experiment Setup
Implementation Models were implemented using the PyTorch (Li et al., 2020) and Huggingface (Wolf et al., 2019) libraries. Model weights were initialized from pre-trained checkpoints available through the Huggingface Model Hub<sup>3</sup>. All BART models were based on the facebook/bart-large checkpoint, the LED-model was based on the allenai/led-large-16384 checkpoint, which itself is based on BART-large.
Training & Inference Models were trained for 10 epochs with final checkpoints selected based on the average of ROUGE- $\{1,2,L\}$  ( $R-1,R-2,R-L$ ) scores achieved on the validation set. Gradient checkpointing (Chen et al., 2016) was used for the LED and SEGENC models to reduce the memory footprint. Model outputs were decoded using beam search with 4 beams. To ensure high consistency of results, all experiments in §4 were repeated 5 times with results averaged across runs.
Evaluation Models were automatically evaluated using the ROUGE- $\{1,2,L\}$  metrics (Lin, 2004) included in the SummEval toolkit (Fabbri et al., 2021b). Models were also manually evaluated by hired human annotators. Annotators were hired through the Amazon Mechanical Turk platform. Workers were selected from English speaking countries and offered an hourly rate of approximately 12 USD. The study was conducted on 50 model generated examples chosen at random from the test set of QMSum.
4 Model Exploration
In this section, we first analyze the effects of model-specific architectural and hyperparameter choices on the performance of two-stage (§4.1) and end-to-end models (§4.2). Next, we study the task-specific knowledge transfer capabilities of different pretraining strategies in §4.3. Lastly, we conduct a final evaluation and comparison of all discussed models in §4.4. All experiments and analyses presented in this section were conducted on QMSum.
4.1 Two-Stage Approaches
For two-stage models, we first focus on evaluating the extractor component and comparing performance to baseline heuristics. We quantify extractor
performance using two metrics: 1) lexical overlap between the extracted utterances and reference summaries, computed using R-1, R-2, and R-L metrics, 2) span overlap between the extracted and golden spans included with QMSum represented by Precision and Recall scores, with results shown in Table 1. In both cases, we first order utterances of the conversation according to the scores assigned by the extractor models, then concatenate the utterances and finally truncate the result to 1024 tokens (excluding the space reserved for the query) to mimic the input length limits of downstream abstractor models; we present those numbers as the All columns in the table. For the lexical overlap, we also show the scores for the best 1 (Top-1), 5 (Top-5), and 15 (Top-15) utterances.
The results show that the best-performing extractor model is RELREG closely followed by REL-REGTT in the Top-1 evaluation and DPR in the Top-5, Top-15, and All cases. We note that both the RELREG and RELREGTT models tend to select longer utterances than the other extractors; the regression-based training mirrors the ROUGE overlap score which favors longer, more informative utterances. However, despite their strong performance in extracting top-matching utterances, the results also expose a considerable gap between model-based approaches and human annotations when considering the entirety of extracted spans. This shows a promising topic for future work in this matter. We also notice that despite the simplicity of the LEAD heuristic, which extracts the first  $k$  utterances in their original order, it remains competitive with the data-driven extractor models when we consider the All case. An extended version of this study, which includes the lexical overlap between extracted spans and input queries is presented in Table 8 in the Appendix.
Next, we analyze how the performance of the extractor components carries over to the final summarization task. For the best-performing model, we additionally test the effect of varying the input segment size used during training and inference between 256 and 512 tokens. Validation-set results for all models are reported in Table 2.
We find that DPR slightly outperforms REL-REGTT for dual-encoder models. Among single-encoder models, RELREG outperforms MARGE by over a full R-1 point, which may explained by RELREG using more direct supervision based on an in-domain query, rather than creating
<sup>3</sup>https://huggingface.co/models
Model	Lexical Overlap b/w Extractors and References	Span Overlap b/w Extractors and Golden Spans
Top-1	Top-5	Top-15	All	All
R-1	R-2	R-L	x	R-1	R-2	R-L	x	R-1	R-2	R-L	x	R-1	R-2	R-L	x	Precision Recall
GOLD SPANS	15.00	3.80	11.10	60	20.89	6.05	15.04	218	19.62	5.99	14.28	386	16.09	5.60	12.47	660	0.75	1.00
LEAD	8.17	0.98	6.30	82	12.84	1.69	9.17	309	13.13	1.81	9.21	463	8.77	1.79	6.77	978	0.09	0.20
DPR	11.31	1.99	8.72	34	17.46	2.86	12.21	156	15.38	2.74	10.64	394	9.75	2.23	7.42	932	0.22	0.27
RELREGTT	23.67	3.34	15.66	82	16.13	3.35	11.18	413	9.65	2.58	7.31	930	9.16	2.52	6.99	994	0.07	0.24
MARGE	7.13	0.72	5.81	20	13.76	1.39	10.22	92	14.85	1.74	11.09	269	9.21	1.52	7.16	896	0.15	0.21
RELREG	24.57	4.33	16.57	88	17.52	4.11	12.21	418	10.56	3.04	8.06	884	9.62	2.87	7.47	989	0.11	0.28
Table 1: Performance of extractor models on the QMSum validation set. The left section presents the lexical overlap between the utterances retrieved by extractor models and the reference summaries, evaluated by means of ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L) metrics. Segments of the section focus on the lexical overlap between the highest ranked 1 (Top-1), 5 (Top-5), 15 (Top-15) utterances, and all utterances truncated to a 1024 token limit (All). The table also includes the average word counts of all extracted utterances, denoted as  $\bar{x}$ . The right section shows the span overlap between the utterance spans retrieved by the extractor models and those collected from human annotators by the authors of QMSum. The performance is evaluated by means of Precision and Recall scores and uses the highest ranked utterances truncated to the limit of 1024 tokens.
Model	R-1	R-2	R-L
DPR	32.79	9.82	28.91
RELREGTT	32.65	9.00	28.57
MARGE	31.90	9.10	28.17
RELREG	33.43	9.77	29.40
RELREG (256)	34.67	11.53	30.66
RELREG (512)	32.22	10.29	29.49
Table 2: Performance of two-step models on the QM-Sum validation set, divided into dual-encoder and single-encoder extractors. Input segment lengths are indicated in parentheses, and otherwise the model operates on utterance-level input.