Conclusions	In this paper, we propose to automatically construct a large-scale query-focused summarization dataset WikiRef using Wikipedia articles and the corresponding references. The statements, supporting citations and article title along with section titles of the statements are used as summaries, documents and queries respectively. The WikiRef dataset serves as a means of data augmentation on DUC benchmarks. It also is shown to be a eligible query-focused summarization benchmark. Moreover, we develop a BERT-based extractive query-focused summarization model to extract summaries from the documents. The model makes use of the query-sentence relationships and sentence-sentence relationships jointly to score sentences. The results on DUC benchmarks show that our model with data augmentation outperforms the state-of-the-art. As for future work, we would like to model relationships among documents for multi-document summarization.

Query-Focused Summarization Model	Figure FIGREF2 gives an overview of our BERT-based extractive query-focused summmarization model. For each sentence, we use BERT to encode its query relevance, document context and salient meanings into a vector representation. Then the vector representations are fed into a simple output layer to predict the label or estimate the score of each sentence.

On the other hand, the section titles give a hint about which aspect of the document is the summary's focus. Therefore, we use the article title and the section titles of the statement to form the query. Given that Wikipedia is the largest online encyclopedia, we can automatically construct massive query-focused summarization examples. Most systems on the DUC benchmark are extractive summarization models. These systems are usually decomposed into two subtasks, i.e., sentence scoring and sentence selection. Sentence scoring aims to measure query relevance and sentence salience for each sentence, which mainly adopts feature-based methods , , . Sentence selection is used to generate the final summary with the minimal redundancy by selecting highest ranking sentences one by one. In this paper, we develop a BERT-based model for query-focused extractive summarization. The model takes the concatenation of the query and the document as input. The query-sentence and sentence-sentence relationships are jointly modeled by the self-attention mechanism . The model is fine-tuned to utilize the general language representations of BERT .

Transforming Wikipedia into Augmented Data for Query-Focused Summarization	The manual construction of a query-focused summarization corpus is costly and timeconsuming. The limited size of existing datasets renders training data-driven summarization models challenging. In this paper, we use Wikipedia to automatically collect a large query-focused summarization dataset (named as WIKIREF) of more than 280,000 examples, which can serve as a means of data augmentation. Moreover, we develop a query-focused summarization model based on BERT to extract summaries from the documents. Experimental results on three DUC benchmarks show that the model pre-trained on WIKIREF has already achieved reasonable performance. After fine-tuning on the specific datasets, the model with data augmentation outperforms the state of the art on the benchmarks.

Related Work	A wide range of unsupervised approaches have been proposed for extractive summarization. Surface features, such as n-gram overlapping, term frequency, document frequency, sentence positions , sentence length , and TF-IDF cosine similarity . Maximum Marginal Relevance (MMR)  greedily selects sentences and considered the trade-off between saliency and redundancy.  ilp treat sentence selection as an optimization problem and solve it using Integer Linear Programming (ILP).  lin2010multi propose using submodular functions to maximize an objective function that considers the trade-off between coverage and redundancy terms. Graph-based models make use of various inter-sentence and query-sentence relationships are also widely applied in the extractive summarization area. LexRank  scores sentences in a graph of sentence similarities.  qfsgraph apply manifold ranking to make use of the sentence-to-sentence and sentence-to-document relationships and the sentence-to-query relationships. We also model the above mentioned relationships, except for the cross-document relationships, like a graph at token level, which are aggregated into distributed representations of sentences. Supervised methods with machine learning techniques , ,  are also used to better estimate sentence importance.

In recent years, few deep neural networks based approaches have been used for extractive document summarization.  cao-attsum propose an attention-base model which jointly handles sentence salience ranking and query relevance ranking. It automatically generates distributed representations for sentences as well as the document. To leverage contextual relations for sentence modeling,  Ren-crsum propose CRSum that learns sentence representations and context representations jointly with a two-level attention mechanism. The small data size is the main obstacle of developing neural models for query-focused summarization.

Experiments ::: Implementation Details	We use the uncased version of BERT-base for fine-tuning. The max sequence length is set to 512. We use Adam optimizer  with learning rate of 3e-5, $\beta _1$ = 0.9, $\beta _2$ = 0.999, L2 weight decay of 0.01, and linear decay of the learning rate. We split long documents into multiple windows with a stride of 100. Therefore, a sentence can appear in more than one windows. To avoid making predictions on an incomplete sentence or with suboptimal context, we score a sentence only when it is completely included and its context is maximally covered. The training epoch and batch size are selected from {3, 4}, and {24, 32}, respectively.

Experiments ::: Evaluation Metrics	For summary evaluation, we use Rouge  as our automatic evaluation metric. Rouge is the official metrics of the DUC benchmarks and widely used for summarization evaluation. Rouge-N measures the summary quality by counting overlapping N-grams with respect to the reference summary. Whereas Rouge-L measures the longest common subsequence. To compare with previous work on DUC datasets, we report the Rouge-1 and Rouge-2 recall computed with official parameters that limits the length to 250 words. On the WikiRef dataset, we report Rouge-1, Rouge-2 and Rouge-L scores.

Query-Focused Summarization Model ::: BERT Encoding Layer	In this layer, we use BERT , a deep Transformer  consisting of stacked self-attention layers, as encoder to aggregate query, intra-sentence and inter-sentence information into sentence representations. Given the packed input embeddings $\mathbf {H}^0=\left[\mathbf {x}_1,...,\mathbf {x}_{|x|}\right]$, we apply an $L$-layer Transformer to encode the input: Hl=Transformerl(Hl-1) where $l\in \left[1,L\right]$. At last, we use the hidden vector $\mathbf {h}_i^L$ of the $i$-th [CLS] token as the contextualized representation of the subsequent sentence.

Experiments ::: Experiments on DUC Datasets ::: Baselines ::: CrSum	 is the contextual relation-based neural summarization system that improves sentence scoring by utilizing contextual relations among sentences.

Experiments ::: Experiments on DUC Datasets ::: Ablation Study	To further verify the effectiveness of our data augmentation method, we first pre-train models on the WikiRef dataset and then we vary the number of golden examples for fine-tuning. Here we take the DUC 2007 dataset as test set and use DUC 2005 and 2006 as training set. In Figure FIGREF33, we present Rouge-2 scores of fine-tuning BERT on DUC datasets for comparison. Either using DUC 2005 alone or DUC 2006 alone yields inferior performance than using both. Our proposed data augmentation method can obtain competitive results using only no more than 30 golden examples and outperform BERT fine-tuning thereafter.

Experiments ::: Experiments on DUC Datasets ::: Results	Table TABREF22 shows the Rouge scores of comparison methods and our proposed method. Fine-tuning BERT on DUC datasets alone outperforms previous best performing summarization systems on DUC 2005 and 2006 and obtains comparable results on DUC 2007. Our data augmentation method further advances the model to a new state of the art on all DUC benchmarks. We also notice that models pre-trained on the augmentation data achieve reasonable performance without further fine-tuning model parameters. It implies the WikiRef dataset reveals useful knowledge shared by the DUC datatset. We pre-train models on augmentation data under both sentence classification and sentence regression supervision. The experimental results show that both supervision types yield similar performance.

Experiments ::: Experiments on WikiRef ::: Results	The results are shown in Table TABREF16. Our proposed model with classification output layer achieves 18.81 Rouge-2 score on the WikiRef test set. On average, the output summary consists of 1.8 sentences. Lead is a strong unsupervised baseline that achieves comparable results with the supervised neural baseline Transformer. Even though WikiRef is a large-scale dataset, training models with parameters initialized from BERT still significantly outperforms Transformer. The model trained using sentence regression performs worse than the one supervised by sentence classification. It is in accordance with oracle labels and scores. We observe a performance drop when generating summaries without queries (see “-Query”). It proves that the summaries in WikiRef are indeed query-focused.

Experiments ::: Experiments on DUC Datasets ::: Settings	We follow standard practice to alternately train our model on two years of data and test on the third. The oracle scores used in model training are Rouge-2 recall of sentences. In this paper, we score a sentence by only considering the query and the its document. Then we rank sentences according to the estimated scores across documents within a cluster. For each cluster, we fetch the top-ranked sentences iteratively into the output summary with redundancy constraint met. A sentence is redundant if more than half of its bigrams appear in the current output summary. The WikiRef dataset is used as augmentation data for DUC datasets in two steps. We first fine-tune BERT on the WikiRef dataset. Subsequently, we use the DUC datasets to further fine-tune parameters of the best pre-trained model.

Experiments ::: Experiments on WikiRef ::: Baselines ::: Transformer	uses the same structure as the BERT with randomly initialized parameters.

Query-Focused Summarization Model ::: Input Representation	The query $\mathcal {Q}$ and document $\mathcal {D}$ are flattened and packed as a token sequence as input. Following the standard practice of BERT, the input representation of each token is constructed by summing the corresponding token, segmentation and position embeddings. Token embeddings project the one-hot input tokens into dense vector representations. Two segment embeddings $\mathbf {E}_Q$ and $\mathbf {E}_D$ are used to indicate query and document tokens respectively. Position embeddings indicate the absolute position of each token in the input sequence. To embody the hierarchical structure of the query in a sequence, we insert a [L#] token before the #-th query token sequence. For each sentence, we insert a [CLS] token at the beginning and a [SEP] token at the end to draw a clear sentence boundary.

Then we append the sentence one by one to form the summary if it is not redundant and scores higher than a threshold. We skip the redundant sentences that contain overlapping trigrams with respect to the current output summary as in  ft-bert-extractive. The threshold is searched on the development set to obtain the highest Rouge-2 F1 score.

Experiments ::: Experiments on DUC Datasets ::: Baselines	We compare our method with several previous query-focused summarization models, of which the AttSum is the state-of-the-art model:

Experimental results on three DUC benchmarks show that the model achieves competitive performance by fine-tuning and outperforms previous state-of-the-art summarization models with data augmentation. Meanwhile, the results demonstrate that we can use WikiRef as a large-scale dataset to advance query-focused summarization research.

Experiments ::: Experiments on WikiRef ::: Baselines	We apply the proposed model and the following baselines:

Experiments	In this section, we present experimental results of the proposed model on the DUC 2005, 2006, 2007 datasets with and without data augmentation. We also carry out benchmark tests on WikiRef as a standard query-focused summarization dataset.

Query-Focused Summarization Model ::: Training Objective	The training objective of sentence classification is to minimize the binary cross-entropy loss: where $y_i\in \lbrace 0,1\rbrace $ is the oracle label of the i-th sentence. The training objective of sentence regression is to minimize the mean square error between the estimated score and the oracle score: L=1nin (r(siQ,D) - f(siS*))2 where $\mathcal {S}^*$ is the oracle summary and $\textrm {f}(s_i\vert \mathcal {S}^*)$ is the oracle score of the i-th sentence.

Experiments ::: Experiments on WikiRef ::: Settings	We first train our extractive summarization model on the WikiRef dataset through sentence classification. And we need the ground-truth binary labels of sentences to be extracted. However, we can not find the sentences that exactly match the reference summary for most examples. To solve this problem, we use a greedy algorithm similar to  zhou-etal-2018-neural-document to find an oracle summary with document sentences that maximizes the Rouge-2 F1 score with respect to the reference summary. Given a document of $n$ sentences, we greedily enumerate the combination of sentences. For documents that contain numerous sentences, searching for an global optimal combination of sentences is computationally expensive. Meanwhile it is unnecessary since the reference summaries contain no more than four sentences. So we stop searching when no combination with $i$ sentences scores higher than the best the combination with $i$-1 sentences. We also train an extractive summarization model through sentence regression. For each sentence, the oracle score for training is the Rouge-2 F1 score. During inference, we rank sentences according to their predicted scores.

Introduction	Query-focused summarization aims to create a brief, well-organized and informative summary for a document with specifications described in the query. Various unsupervised methods , , , , ,  and supervised methods , , , ,  have been proposed for the purpose. The task is first introduced in DUC 2005 , with human annotated data released until 2007. The DUC benchmark datasets are of high quality. But the limited size renders training query-focused summarization models challenging, especially for the data-driven methods. Meanwhile, manually constructing a large-scale query-focused summarization dataset is quite costly and time-consuming. In order to advance query-focused summarization with limited data, we improve the summarization model with data augmentation. Specifically, we transform Wikipedia into a large-scale query-focused summarization dataset (named as WikiRef). To automatically construct query-focused summarization examples using Wikipedia, the statements' citations in Wikipedia articles as pivots to align the queries and documents. Figure FIGREF1 shows an example that is constructed by the proposed method. We first take the highlighted statement as the summary. Its supporting citation is expected to provide an adequate context to derive the statement, thus can serve as the source document.

Experiments ::: Experiments on DUC Datasets ::: Human Evaluation	To better understand the improvement brought by augmentation data, we conduct a human evaluation of the output summaries before and after data augmentation. We sample 30 output summaries of the DUC 2006 dataset for analysis. And we find that the model augmented by the WikiRef dataset produces more query-related summaries on 23 examples. Meanwhile,the extracted sentences are usually less redundant. We attribute these benefits to the improved coverage and query-focused extraction brought by the large-scale augmentation data.

Query-Focused Summarization Model ::: Output Layer	The output layer is used to score sentences for extractive query-focused summarization. Given $\mathbf {h}_i^L\in \mathbb {R}^d$ is the vector representation for the i-th sentence. When the extracive summarization is carried out through sentence classification , the output layer is a linear layer followed by a sigmoid function: P(siQ,D)=sigmoid(WchiL+bc) where $\mathbf {W}_c$ and $\mathbf {b}_c$ are trainable parameters. The output is the probability of including the i-th sentence in the summary. In the setting of sentence regression, a linear layer without activation function is used to estimate the score of a sentence: r(siQ,D)=WrhiL+br where $\mathbf {W}_r$ and $\mathbf {b}_r$ are trainable parameters.

Experiments ::: Discussion	The improvement introduced by using the WikiRef dataset as augmentation data is traceable. At first, the document in the DUC datasets are news articles and we crawl newspaper webpages as one source of the WikiRef documents. Secondly, queries in the WikiRef dataset are hierarchical that specify the aspects it focuses on gradually. This is similar to the DUC datasets that queries are composed of several narratives to specify the desired information. The key difference is that queries in the WikiRef dataset are composed of key words, while the ones in the DUC datasets are mostly natural language. At last, we construct the WikiRef dataset to be a large-scale query-focused summarization dataset that contains more than 280,000 examples. In comparison, the DUC datasets contain only 145 clusters with around 10,000 documents. Therefore, query relevance and sentence context can be better modeled using data-driven neural methods with WikiRef. And it provides a better starting point for fine-tuning on the DUC datasets.

WikiRef: Transforming Wikipedia into Augmented Data ::: Data Creation ::: Data Curation	To make sure the statement is a plausible summary of the cited document, we process and filter the examples through a set of fine-grained rules. The text is tokenized and lemmatized using Spacy. First, we calculate the unigram recall of the document, where only the non-stop words are considered. We throw out the example whose score is lower than the threshold. Here we set the threshold to 0.5 empirically, which means at least more than half of the summary tokens should be in the document. Next, we filter the examples with multiple length and sentence number constraints. To set reasonable thresholds, we use the statistics of the examples whose documents contain no more than 1,000 tokens. The 5th and the 95th percentiles are used as low and high thresholds of each constraint. Finally, in order to ensure generating the summary with the given document is feasible, we filter the examples through extractive oracle score. The extractive oracle is obtained through a greedy search over document sentence combinations with maximum 5 sentences.

We factorize the probability of predicting $\hat{\mathcal {S}}$ as the output summary $P(\hat{\mathcal {S}}\vert \mathcal {Q},\mathcal {D})$ of document $\mathcal {D}$ given query $\mathcal {Q}$ as: P(SQ,D)=siS P(siQ,D) ) In sentence regression, extractive summarization is achieved via sentence scoring and sentence selection. The former scores $\textrm {r}(s_i\vert \mathcal {Q},\mathcal {D})$ a sentence $s_i$ by considering its relevance to the query $\mathcal {Q}$ and its salience to the document $\mathcal {D}$. The latter generates a summary by ranking sentences under certain constraints, e.g., the number of sentences and the length of the summary.

WikiRef: Transforming Wikipedia into Augmented Data ::: Human Evaluation	We also conduct human evaluation on 60 WikiRef samples to examine the quality of the automatically constructed data. We partition the examples into four bins according to the oracle score and then sample 15 examples from each bin. Each example is scored in two criteria: (1) “Query Relatedness” examines to what extent the summary is a good response to the query and (2) “Doc Salience” examines to what extent the summary conveys salient document content given the query. Table TABREF15 shows the evaluation result. We can see that most of the time the summaries are good responses to the queries across bins. Since we take section titles as query and the statement under the section as summary, the high evaluation score can be attributed to Wikipedia pages of high quality. When the oracle scores are getting higher, the summaries continue to better convey the salient document content specified by the query. On the other hand, we notice that sometimes the summaries only contain a proportion of salient document content. It is reasonable since reference articles may present several aspects related to topic. But we can see that it is mitigated when the oracle scores are high on the WikiRef dataset.

WikiRef: Transforming Wikipedia into Augmented Data ::: Data Statistics	Table TABREF11 show statistics of the WikiRef dataset. The development set and the test set contains 12,000 examples each. The statistics across splits are evenly distributed and no bias observed. The numerous Wikipedia articles cover a wide range of topics. The average depth of the query is 2.5 with article titles are considered. Since the query are keywords in WikiRef, it is relatively shorter than the natural language queries with an average length of 6.7 tokens. Most summaries are composed of one or two sentences. And the document contains 18.8 sentences on average.

Experiments ::: Experiments on DUC Datasets ::: Baselines ::: AttSum	 is a neural attention summarization system that tackles query relevance ranking and sentence salience ranking jointly.

Experiments ::: Experiments on DUC Datasets ::: Baselines ::: Svr	 is a supervised baseline that extracts both query-dependent and query-independent features and then using Support Vector Regression to learn the weights of features.

Experiments ::: Experiments on DUC Datasets ::: Baselines ::: Query-Sim	is an unsupervised method that ranks sentences according to its TF-IDF cosine similarity to the query.

Experiments ::: Experiments on DUC Datasets ::: Baselines ::: Lead	is a simple baseline that selects leading sentences to form a summary.

Experiments ::: Experiments on DUC Datasets	DUC 2005-2007 are query-focused multi-document summarization benchmarks. The documents are from the news domain and grouped into clusters according to their topics. And the summary is required to be no longer than 250 tokens. Table TABREF29 shows statistics of the DUC datasets. Each document cluster has several reference summaries generated by humans and a query that specifies the focused aspects and desired information. We show an example query from the DUC 2006 dataset below: EgyptAir Flight 990? What caused the crash of EgyptAir Flight 990? Include evidence, theories and speculation. The first narrative is usually a title and followed by several natural language questions or narratives.

We show an example in Figure FIGREF8 to illustrate the raw data collection process. The associated query, summary and the document are highlighted in colors in the diagram. At last, we have collected more than 2,000,000 English examples in total after the raw data collection step.

Experiments ::: Experiments on WikiRef ::: Baselines ::: Lead	is a straightforward summarization baseline that selects the leading sentences. We take the first two sentences for that the groundtruth summary contains 1.4 sentences on average.

Experiments ::: Experiments on WikiRef ::: Baselines ::: All	outputs all sentences of the document as summary.

Here we adopt Rouge-2 recall as scoring metric and only the examples with an oracle score higher than 0.2 are kept. After running through the above rules, we have the WikiRef dataset with 280,724 examples. We randomly split the data into training, development and test sets and ensure no document overlapping across splits.

WikiRef: Transforming Wikipedia into Augmented Data	We automatically construct a query-focused summarization dataset (named as WikiRef) using Wikipedia and corresponding reference web pages. In the following sections, we will first elaborate the creation process. Then we will analyze the queries, documents and summaries quantitatively and qualitatively.

WikiRef: Transforming Wikipedia into Augmented Data ::: Data Creation	We follow two steps to collect and process the data: (1) we crawl English Wikipedia and the references of the Wikipedia articles and parse the HTML sources into plain text; (2) we preprocess the plain text and filter the examples through a set of fine-grained rules.

WikiRef: Transforming Wikipedia into Augmented Data ::: Data Creation ::: Raw Data Collection	To maintain the highest standards possible, most statements in Wikipedia are attributed to reliable, published sources that can be accessed through hyperlinks. In the first step, we parse the English Wikipedia database dump into plain text and save statements with citations. If a statement is attributed multiple citations, only the first citation is used. We also limit the sources of the citations to four types, namely web pages, newspaper articles, press and press release. A statement may contain more than one sentence. The statement can be seen as a summary of the supporting citations from a certain aspect. Therefore, we can take the body of the citation as the document and treat the statement as the summary. Meanwhile, the section titles of a statement could be used as a natural coarse-grained query to specify the focused aspects. Then we can form a complete query-focused summarization example by referring to the statement, attributed citation and section titles along with the article title as summary, document and query respectively. It is worth noticing that the queries in WikiRef dataset are thus keywords, instead of natural language as in other query-focused summarization datasets.

Problem Formulation	Given a query $\mathcal {Q}=(q_1, q_2,...,q_m)$ of $m$ token sequences and a document $\mathcal {D}=(s_1, s_2, ..., s_n)$ containing $n$ sentences, extractive query-focused summarization aims to extract a salient subset of $ \mathcal {D}$ that is related to the query as the output summary $\mathcal {\hat{S}}=\left\lbrace \hat{s_i}\vert \hat{s_i} \in \mathcal {D}\right\rbrace $. In general, the extrative summarization task can be tackled by assigning each sentence a label to indicate the inclusion in the summary or estimating scores for ranking sentences, namely sentence classification or sentence regression. In sentence classification, the probability of putting sentence $s_i$ in the output summary is $P\left(s_i\vert \mathcal {Q},\mathcal {D}\right)$.

Question: How does their BERT-based model work?
Answer: The model takes the concatenation of the query and the document as input. The query-sentence and sentence-sentence relationships are jointly modeled by the self-attention mechanism . The model is fine-tuned to utilize the general language representations of BERT .