Transforming Wikipedia into Augmented Data for Query-Focused Summarization

Abstract: The manual construction of a query-focused summarization corpus is costly and timeconsuming. The limited size of existing datasets renders training data-driven summarization models challenging. In this paper, we use Wikipedia to automatically collect a large query-focused summarization dataset (named as WIKIREF) of more than 280,000 examples, which can serve as a means of data augmentation. Moreover, we develop a query-focused summarization model based on BERT to extract summaries from the documents. Experimental results on three DUC benchmarks show that the model pre-trained on WIKIREF has already achieved reasonable performance. After fine-tuning on the specific datasets, the model with data augmentation outperforms the state of the art on the benchmarks.

Query-focused summarization aims to create a brief, well-organized and informative summary for a document with specifications described in the query. Various unsupervised methods BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5 and supervised methods BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10 have been proposed for the purpose. The task is first introduced in DUC 2005 BIBREF11, with human annotated data released until 2007. The DUC benchmark datasets are of high quality. But the limited size renders training query-focused summarization models challenging, especially for the data-driven methods. Meanwhile, manually constructing a large-scale query-focused summarization dataset is quite costly and time-consuming.
In order to advance query-focused summarization with limited data, we improve the summarization model with data augmentation. Specifically, we transform Wikipedia into a large-scale query-focused summarization dataset (named as WikiRef). To automatically construct query-focused summarization examples using Wikipedia, the statements' citations in Wikipedia articles as pivots to align the queries and documents. Figure FIGREF1 shows an example that is constructed by the proposed method. We first take the highlighted statement as the summary. Its supporting citation is expected to provide an adequate context to derive the statement, thus can serve as the source document. On the other hand, the section titles give a hint about which aspect of the document is the summary's focus. Therefore, we use the article title and the section titles of the statement to form the query. Given that Wikipedia is the largest online encyclopedia, we can automatically construct massive query-focused summarization examples.
Most systems on the DUC benchmark are extractive summarization models. These systems are usually decomposed into two subtasks, i.e., sentence scoring and sentence selection. Sentence scoring aims to measure query relevance and sentence salience for each sentence, which mainly adopts feature-based methods BIBREF0, BIBREF7, BIBREF3. Sentence selection is used to generate the final summary with the minimal redundancy by selecting highest ranking sentences one by one.
In this paper, we develop a BERT-based model for query-focused extractive summarization. The model takes the concatenation of the query and the document as input. The query-sentence and sentence-sentence relationships are jointly modeled by the self-attention mechanism BIBREF12. The model is fine-tuned to utilize the general language representations of BERT BIBREF13.
Experimental results on three DUC benchmarks show that the model achieves competitive performance by fine-tuning and outperforms previous state-of-the-art summarization models with data augmentation. Meanwhile, the results demonstrate that we can use WikiRef as a large-scale dataset to advance query-focused summarization research.
A wide range of unsupervised approaches have been proposed for extractive summarization. Surface features, such as n-gram overlapping, term frequency, document frequency, sentence positions BIBREF10, sentence length BIBREF9, and TF-IDF cosine similarity BIBREF3. Maximum Marginal Relevance (MMR) BIBREF0 greedily selects sentences and considered the trade-off between saliency and redundancy. BIBREF2 ilp treat sentence selection as an optimization problem and solve it using Integer Linear Programming (ILP). BIBREF14 lin2010multi propose using submodular functions to maximize an objective function that considers the trade-off between coverage and redundancy terms.
Graph-based models make use of various inter-sentence and query-sentence relationships are also widely applied in the extractive summarization area. LexRank BIBREF1 scores sentences in a graph of sentence similarities. BIBREF3 qfsgraph apply manifold ranking to make use of the sentence-to-sentence and sentence-to-document relationships and the sentence-to-query relationships. We also model the above mentioned relationships, except for the cross-document relationships, like a graph at token level, which are aggregated into distributed representations of sentences.
Supervised methods with machine learning techniques BIBREF6, BIBREF7, BIBREF8 are also used to better estimate sentence importance. In recent years, few deep neural networks based approaches have been used for extractive document summarization. BIBREF9 cao-attsum propose an attention-base model which jointly handles sentence salience ranking and query relevance ranking. It automatically generates distributed representations for sentences as well as the document. To leverage contextual relations for sentence modeling, BIBREF10 Ren-crsum propose CRSum that learns sentence representations and context representations jointly with a two-level attention mechanism. The small data size is the main obstacle of developing neural models for query-focused summarization.
Given a query $\mathcal {Q}=(q_1, q_2,...,q_m)$ of $m$ token sequences and a document $\mathcal {D}=(s_1, s_2, ..., s_n)$ containing $n$ sentences, extractive query-focused summarization aims to extract a salient subset of $ \mathcal {D}$ that is related to the query as the output summary $\mathcal {\hat{S}}=\left\lbrace \hat{s_i}\vert \hat{s_i} \in \mathcal {D}\right\rbrace $. In general, the extrative summarization task can be tackled by assigning each sentence a label to indicate the inclusion in the summary or estimating scores for ranking sentences, namely sentence classification or sentence regression.
In sentence classification, the probability of putting sentence $s_i$ in the output summary is $P\left(s_i\vert \mathcal {Q},\mathcal {D}\right)$. We factorize the probability of predicting $\hat{\mathcal {S}}$ as the output summary $P(\hat{\mathcal {S}}\vert \mathcal {Q},\mathcal {D})$ of document $\mathcal {D}$ given query $\mathcal {Q}$ as: P(SQ,D)=siS P(siQ,D) )
In sentence regression, extractive summarization is achieved via sentence scoring and sentence selection. The former scores $\textrm {r}(s_i\vert \mathcal {Q},\mathcal {D})$ a sentence $s_i$ by considering its relevance to the query $\mathcal {Q}$ and its salience to the document $\mathcal {D}$. The latter generates a summary by ranking sentences under certain constraints, e.g., the number of sentences and the length of the summary.
Figure FIGREF2 gives an overview of our BERT-based extractive query-focused summmarization model. For each sentence, we use BERT to encode its query relevance, document context and salient meanings into a vector representation. Then the vector representations are fed into a simple output layer to predict the label or estimate the score of each sentence.
The query $\mathcal {Q}$ and document $\mathcal {D}$ are flattened and packed as a token sequence as input. Following the standard practice of BERT, the input representation of each token is constructed by summing the corresponding token, segmentation and position embeddings. Token embeddings project the one-hot input tokens into dense vector representations. Two segment embeddings $\mathbf {E}_Q$ and $\mathbf {E}_D$ are used to indicate query and document tokens respectively. Position embeddings indicate the absolute position of each token in the input sequence. To embody the hierarchical structure of the query in a sequence, we insert a [L#] token before the #-th query token sequence. For each sentence, we insert a [CLS] token at the beginning and a [SEP] token at the end to draw a clear sentence boundary.
In this layer, we use BERT BIBREF13, a deep Transformer BIBREF12 consisting of stacked self-attention layers, as encoder to aggregate query, intra-sentence and inter-sentence information into sentence representations. Given the packed input embeddings $\mathbf {H}^0=\left[\mathbf {x}_1,...,\mathbf {x}_{|x|}\right]$, we apply an $L$-layer Transformer to encode the input: Hl=Transformerl(Hl-1) where $l\in \left[1,L\right]$. At last, we use the hidden vector $\mathbf {h}_i^L$ of the $i$-th [CLS] token as the contextualized representation of the subsequent sentence.
The output layer is used to score sentences for extractive query-focused summarization. Given $\mathbf {h}_i^L\in \mathbb {R}^d$ is the vector representation for the i-th sentence. When the extracive summarization is carried out through sentence classification , the output layer is a linear layer followed by a sigmoid function: P(siQ,D)=sigmoid(WchiL+bc) where $\mathbf {W}_c$ and $\mathbf {b}_c$ are trainable parameters. The output is the probability of including the i-th sentence in the summary.
In the setting of sentence regression, a linear layer without activation function is used to estimate the score of a sentence: r(siQ,D)=WrhiL+br where $\mathbf {W}_r$ and $\mathbf {b}_r$ are trainable parameters.
The training objective of sentence classification is to minimize the binary cross-entropy loss:
where $y_i\in \lbrace 0,1\rbrace $ is the oracle label of the i-th sentence.
The training objective of sentence regression is to minimize the mean square error between the estimated score and the oracle score: L=1nin (r(siQ,D) - f(siS*))2 where $\mathcal {S}^*$ is the oracle summary and $\textrm {f}(s_i\vert \mathcal {S}^*)$ is the oracle score of the i-th sentence.
We automatically construct a query-focused summarization dataset (named as WikiRef) using Wikipedia and corresponding reference web pages. In the following sections, we will first elaborate the creation process. Then we will analyze the queries, documents and summaries quantitatively and qualitatively.
We follow two steps to collect and process the data: (1) we crawl English Wikipedia and the references of the Wikipedia articles and parse the HTML sources into plain text; (2) we preprocess the plain text and filter the examples through a set of fine-grained rules.
To maintain the highest standards possible, most statements in Wikipedia are attributed to reliable, published sources that can be accessed through hyperlinks. In the first step, we parse the English Wikipedia database dump into plain text and save statements with citations. If a statement is attributed multiple citations, only the first citation is used. We also limit the sources of the citations to four types, namely web pages, newspaper articles, press and press release. A statement may contain more than one sentence.
The statement can be seen as a summary of the supporting citations from a certain aspect. Therefore, we can take the body of the citation as the document and treat the statement as the summary. Meanwhile, the section titles of a statement could be used as a natural coarse-grained query to specify the focused aspects. Then we can form a complete query-focused summarization example by referring to the statement, attributed citation and section titles along with the article title as summary, document and query respectively. It is worth noticing that the queries in WikiRef dataset are thus keywords, instead of natural language as in other query-focused summarization datasets.
We show an example in Figure FIGREF8 to illustrate the raw data collection process. The associated query, summary and the document are highlighted in colors in the diagram. At last, we have collected more than 2,000,000 English examples in total after the raw data collection step.
To make sure the statement is a plausible summary of the cited document, we process and filter the examples through a set of fine-grained rules. The text is tokenized and lemmatized using Spacy. First, we calculate the unigram recall of the document, where only the non-stop words are considered. We throw out the example whose score is lower than the threshold. Here we set the threshold to 0.5 empirically, which means at least more than half of the summary tokens should be in the document. Next, we filter the examples with multiple length and sentence number constraints. To set reasonable thresholds, we use the statistics of the examples whose documents contain no more than 1,000 tokens. The 5th and the 95th percentiles are used as low and high thresholds of each constraint. Finally, in order to ensure generating the summary with the given document is feasible, we filter the examples through extractive oracle score. The extractive oracle is obtained through a greedy search over document sentence combinations with maximum 5 sentences. Here we adopt Rouge-2 recall as scoring metric and only the examples with an oracle score higher than 0.2 are kept. After running through the above rules, we have the WikiRef dataset with 280,724 examples. We randomly split the data into training, development and test sets and ensure no document overlapping across splits.
Table TABREF11 show statistics of the WikiRef dataset. The development set and the test set contains 12,000 examples each. The statistics across splits are evenly distributed and no bias observed. The numerous Wikipedia articles cover a wide range of topics. The average depth of the query is 2.5 with article titles are considered. Since the query are keywords in WikiRef, it is relatively shorter than the natural language queries with an average length of 6.7 tokens. Most summaries are composed of one or two sentences. And the document contains 18.8 sentences on average.
We also conduct human evaluation on 60 WikiRef samples to examine the quality of the automatically constructed data. We partition the examples into four bins according to the oracle score and then sample 15 examples from each bin. Each example is scored in two criteria: (1) “Query Relatedness” examines to what extent the summary is a good response to the query and (2) “Doc Salience” examines to what extent the summary conveys salient document content given the query.
Table TABREF15 shows the evaluation result. We can see that most of the time the summaries are good responses to the queries across bins. Since we take section titles as query and the statement under the section as summary, the high evaluation score can be attributed to Wikipedia pages of high quality. When the oracle scores are getting higher, the summaries continue to better convey the salient document content specified by the query. On the other hand, we notice that sometimes the summaries only contain a proportion of salient document content. It is reasonable since reference articles may present several aspects related to topic. But we can see that it is mitigated when the oracle scores are high on the WikiRef dataset.
In this section, we present experimental results of the proposed model on the DUC 2005, 2006, 2007 datasets with and without data augmentation. We also carry out benchmark tests on WikiRef as a standard query-focused summarization dataset.
We use the uncased version of BERT-base for fine-tuning. The max sequence length is set to 512. We use Adam optimizer BIBREF15 with learning rate of 3e-5, $\beta _1$ = 0.9, $\beta _2$ = 0.999, L2 weight decay of 0.01, and linear decay of the learning rate. We split long documents into multiple windows with a stride of 100. Therefore, a sentence can appear in more than one windows. To avoid making predictions on an incomplete sentence or with suboptimal context, we score a sentence only when it is completely included and its context is maximally covered. The training epoch and batch size are selected from {3, 4}, and {24, 32}, respectively.
For summary evaluation, we use Rouge BIBREF16 as our automatic evaluation metric. Rouge is the official metrics of the DUC benchmarks and widely used for summarization evaluation. Rouge-N measures the summary quality by counting overlapping N-grams with respect to the reference summary. Whereas Rouge-L measures the longest common subsequence. To compare with previous work on DUC datasets, we report the Rouge-1 and Rouge-2 recall computed with official parameters that limits the length to 250 words. On the WikiRef dataset, we report Rouge-1, Rouge-2 and Rouge-L scores.
We first train our extractive summarization model on the WikiRef dataset through sentence classification. And we need the ground-truth binary labels of sentences to be extracted. However, we can not find the sentences that exactly match the reference summary for most examples. To solve this problem, we use a greedy algorithm similar to BIBREF17 zhou-etal-2018-neural-document to find an oracle summary with document sentences that maximizes the Rouge-2 F1 score with respect to the reference summary. Given a document of $n$ sentences, we greedily enumerate the combination of sentences. For documents that contain numerous sentences, searching for an global optimal combination of sentences is computationally expensive. Meanwhile it is unnecessary since the reference summaries contain no more than four sentences. So we stop searching when no combination with $i$ sentences scores higher than the best the combination with $i$-1 sentences.
We also train an extractive summarization model through sentence regression. For each sentence, the oracle score for training is the Rouge-2 F1 score.
During inference, we rank sentences according to their predicted scores. Then we append the sentence one by one to form the summary if it is not redundant and scores higher than a threshold. We skip the redundant sentences that contain overlapping trigrams with respect to the current output summary as in BIBREF18 ft-bert-extractive. The threshold is searched on the development set to obtain the highest Rouge-2 F1 score.
We apply the proposed model and the following baselines:
outputs all sentences of the document as summary.
is a straightforward summarization baseline that selects the leading sentences. We take the first two sentences for that the groundtruth summary contains 1.4 sentences on average.
uses the same structure as the BERT with randomly initialized parameters.
The results are shown in Table TABREF16. Our proposed model with classification output layer achieves 18.81 Rouge-2 score on the WikiRef test set. On average, the output summary consists of 1.8 sentences. Lead is a strong unsupervised baseline that achieves comparable results with the supervised neural baseline Transformer. Even though WikiRef is a large-scale dataset, training models with parameters initialized from BERT still significantly outperforms Transformer. The model trained using sentence regression performs worse than the one supervised by sentence classification. It is in accordance with oracle labels and scores. We observe a performance drop when generating summaries without queries (see “-Query”). It proves that the summaries in WikiRef are indeed query-focused.
DUC 2005-2007 are query-focused multi-document summarization benchmarks. The documents are from the news domain and grouped into clusters according to their topics. And the summary is required to be no longer than 250 tokens. Table TABREF29 shows statistics of the DUC datasets. Each document cluster has several reference summaries generated by humans and a query that specifies the focused aspects and desired information. We show an example query from the DUC 2006 dataset below:
EgyptAir Flight 990?
What caused the crash of EgyptAir Flight 990?
Include evidence, theories and speculation.
The first narrative is usually a title and followed by several natural language questions or narratives.
We follow standard practice to alternately train our model on two years of data and test on the third. The oracle scores used in model training are Rouge-2 recall of sentences. In this paper, we score a sentence by only considering the query and the its document. Then we rank sentences according to the estimated scores across documents within a cluster. For each cluster, we fetch the top-ranked sentences iteratively into the output summary with redundancy constraint met. A sentence is redundant if more than half of its bigrams appear in the current output summary.
The WikiRef dataset is used as augmentation data for DUC datasets in two steps. We first fine-tune BERT on the WikiRef dataset. Subsequently, we use the DUC datasets to further fine-tune parameters of the best pre-trained model.
We compare our method with several previous query-focused summarization models, of which the AttSum is the state-of-the-art model:
is a simple baseline that selects leading sentences to form a summary.
is an unsupervised method that ranks sentences according to its TF-IDF cosine similarity to the query.
BIBREF7 is a supervised baseline that extracts both query-dependent and query-independent features and then using Support Vector Regression to learn the weights of features.
BIBREF9 is a neural attention summarization system that tackles query relevance ranking and sentence salience ranking jointly.
BIBREF10 is the contextual relation-based neural summarization system that improves sentence scoring by utilizing contextual relations among sentences.
Table TABREF22 shows the Rouge scores of comparison methods and our proposed method. Fine-tuning BERT on DUC datasets alone outperforms previous best performing summarization systems on DUC 2005 and 2006 and obtains comparable results on DUC 2007. Our data augmentation method further advances the model to a new state of the art on all DUC benchmarks. We also notice that models pre-trained on the augmentation data achieve reasonable performance without further fine-tuning model parameters. It implies the WikiRef dataset reveals useful knowledge shared by the DUC datatset. We pre-train models on augmentation data under both sentence classification and sentence regression supervision. The experimental results show that both supervision types yield similar performance.
To better understand the improvement brought by augmentation data, we conduct a human evaluation of the output summaries before and after data augmentation. We sample 30 output summaries of the DUC 2006 dataset for analysis. And we find that the model augmented by the WikiRef dataset produces more query-related summaries on 23 examples. Meanwhile,the extracted sentences are usually less redundant. We attribute these benefits to the improved coverage and query-focused extraction brought by the large-scale augmentation data.
To further verify the effectiveness of our data augmentation method, we first pre-train models on the WikiRef dataset and then we vary the number of golden examples for fine-tuning. Here we take the DUC 2007 dataset as test set and use DUC 2005 and 2006 as training set. In Figure FIGREF33, we present Rouge-2 scores of fine-tuning BERT on DUC datasets for comparison. Either using DUC 2005 alone or DUC 2006 alone yields inferior performance than using both. Our proposed data augmentation method can obtain competitive results using only no more than 30 golden examples and outperform BERT fine-tuning thereafter.
The improvement introduced by using the WikiRef dataset as augmentation data is traceable. At first, the document in the DUC datasets are news articles and we crawl newspaper webpages as one source of the WikiRef documents. Secondly, queries in the WikiRef dataset are hierarchical that specify the aspects it focuses on gradually. This is similar to the DUC datasets that queries are composed of several narratives to specify the desired information. The key difference is that queries in the WikiRef dataset are composed of key words, while the ones in the DUC datasets are mostly natural language. At last, we construct the WikiRef dataset to be a large-scale query-focused summarization dataset that contains more than 280,000 examples. In comparison, the DUC datasets contain only 145 clusters with around 10,000 documents. Therefore, query relevance and sentence context can be better modeled using data-driven neural methods with WikiRef. And it provides a better starting point for fine-tuning on the DUC datasets.
In this paper, we propose to automatically construct a large-scale query-focused summarization dataset WikiRef using Wikipedia articles and the corresponding references. The statements, supporting citations and article title along with section titles of the statements are used as summaries, documents and queries respectively. The WikiRef dataset serves as a means of data augmentation on DUC benchmarks. It also is shown to be a eligible query-focused summarization benchmark. Moreover, we develop a BERT-based extractive query-focused summarization model to extract summaries from the documents. The model makes use of the query-sentence relationships and sentence-sentence relationships jointly to score sentences. The results on DUC benchmarks show that our model with data augmentation outperforms the state-of-the-art. As for future work, we would like to model relationships among documents for multi-document summarization.

Figure and table captions:
Figure 1: An example of the automatic query-focused summarization example construction. Given a statement in Wikipedia article “Marina Beach”, we take the body text of citation as the document, use the article title along with section titles to form a query (i.e., “Marina Beach, Incidents”), and the statement is the summary.
Figure 2: The overview of the proposed BERT-based extractive summarization model. We use special tokens (e.g., [L1], [L2]) to indicate hierarchial structure in queries. We surround each sentence with a [CLS] token before and a [SEP] token after. The input representations of each token are composed of three embeddings. The hidden vectors of [CLS] tokens from the last layer are used to represent and score sentences.
Figure 3: Illustration of WIKIREF examples creation using Wikipedia and reference pages.
Table 1: Statistics of the WIKIREF dataset.
Table 2: Quality rating results of human evaluation on the WIKIREF dataset. “Query Relatedness”: 2 for summary completely related to the query, 1 for summary partially related to the query, 0 otherwise. “Doc Salience”: 2 for summary conveys all salient document content, 1 for summary conveys partial salient document content, 0 otherwise.
Table 3: ROUGE scores of baselines and the proposed model on WIKIREF dataset. “Class” and “Reg” represent classification and regression, which indicate the supervision type used for training. “- Query” indicates removing queries from the input.
Table 4: ROUGE scores on the DUC 2005, 2006 and 2007 datasets. “*” indicates results taken from Ren et al. (2017). “†” indicates results taken from Cao et al. (2016). “DA” is short for data augmentation using WIKIREF dataset. “DA Pre-trained” denotes applying the model pre-trained on augmentation data to directly extract summaries for DUC datasets. “Class” and “Reg” represent classification super and regression, which indicate the supervision type used on augmentation data.
Table 5: Statistics of DUC datasets.
Figure 4: ROUGE-2 score on the DUC 2007 evaluation set with various training data. The horizontal lines indicate training on DUC 2005, on DUC 2006, and on both. The green line indicates training on only a proportion of DUC 2005 and DUC 2006 examples with the WIKIREF data augmentation. The x-axis indicates the number of used training examples, along with data augmentation.

Questions about this paper:
How does their BERT-based model work?
How do they use Wikipedia to automatically collect a query-focused summarization dataset?

Answers:
It takes the query and document as input and encodes the query relevance, document context and salient meaning to be passed to the output layer to make the prediction.
They use the article and section titles to build a query and use the body text of citation as the summary.