Neural Summarization by Extracting Sentences and Words

Abstract: Traditional approaches to extractive summarization rely heavily on human-engineered features. In this work we propose a data-driven approach based on neural networks and continuous sentence features. We develop a general framework for single-document summarization composed of a hierarchical document encoder and an attention-based extractor. This architecture allows us to develop different classes of summarization models which can extract sentences or words. We train our models on large scale corpora containing hundreds of thousands of document-summary pairs. Experimental results on two summarization datasets demonstrate that our models obtain results comparable to the state of the art without any access to linguistic annotation.

The need to access and digest large amounts of textual data has provided strong impetus to develop automatic summarization systems aiming to create shorter versions of one or more documents, whilst preserving their information content. Much effort in automatic summarization has been devoted to sentence extraction, where a summary is created by identifying and subsequently concatenating the most salient text units in a document.
Most extractive methods to date identify sentences based on human-engineered features. These include surface features such as sentence position and length BIBREF0 , the words in the title, the presence of proper nouns, content features such as word frequency BIBREF1 , and event features such as action nouns BIBREF2 . Sentences are typically assigned a score indicating the strength of presence of these features. Several methods have been used in order to select the summary sentences ranging from binary classifiers BIBREF3 , to hidden Markov models BIBREF4 , graph-based algorithms BIBREF5 , BIBREF6 , and integer linear programming BIBREF7 .
In this work we propose a data-driven approach to summarization based on neural networks and continuous sentence features. There has been a surge of interest recently in repurposing sequence transduction neural network architectures for NLP tasks such as machine translation BIBREF8 , question answering BIBREF9 , and sentence compression BIBREF10 . Central to these approaches is an encoder-decoder architecture modeled by recurrent neural networks. The encoder reads the source sequence into a list of continuous-space representations from which the decoder generates the target sequence. An attention mechanism BIBREF11 is often used to locate the region of focus during decoding.
We develop a general framework for single-document summarization which can be used to extract sentences or words. Our model includes a neural network-based hierarchical document reader or encoder and an attention-based content extractor. The role of the reader is to derive the meaning representation of a document based on its sentences and their constituent words. Our models adopt a variant of neural attention to extract sentences or words. Contrary to previous work where attention is an intermediate step used to blend hidden units of an encoder to a vector propagating additional information to the decoder, our model applies attention directly to select sentences or words of the input document as the output summary. Similar neural attention architectures have been previously used for geometry reasoning BIBREF12 , under the name Pointer Networks.
One stumbling block to applying neural network models to extractive summarization is the lack of training data, i.e., documents with sentences (and words) labeled as summary-worthy. Inspired by previous work on summarization BIBREF7 , BIBREF13 and reading comprehension BIBREF9 we retrieve hundreds of thousands of news articles and corresponding highlights from the DailyMail website. Highlights usually appear as bullet points giving a brief overview of the information contained in the article (see Figure 1 for an example). Using a number of transformation and scoring algorithms, we are able to match highlights to document content and construct two large scale training datasets, one for sentence extraction and the other for word extraction. Previous approaches have used small scale training data in the range of a few hundred examples.
Our work touches on several strands of research within summarization and neural sequence modeling. The idea of creating a summary by extracting words from the source document was pioneered in bankoetal00 who view summarization as a problem analogous to statistical machine translation and generate headlines using statistical models for selecting and ordering the summary words. Our word-based model is similar in spirit, however, it operates over continuous representations, produces multi-sentence output, and jointly selects summary words and organizes them into sentences. A few recent studies BIBREF14 , BIBREF15 perform sentence extraction based on pre-trained sentence embeddings following an unsupervised optimization paradigm. Our work also uses continuous representations to express the meaning of sentences and documents, but importantly employs neural networks more directly to perform the actual summarization task.
rush2015neural propose a neural attention model for abstractive sentence compression which is trained on pairs of headlines and first sentences in an article. In contrast, our model summarizes documents rather than individual sentences, producing multi-sentential discourse. A major architectural difference is that our decoder selects output symbols from the document of interest rather than the entire vocabulary. This effectively helps us sidestep the difficulty of searching for the next output symbol under a large vocabulary, with low-frequency words and named entities whose representations can be challenging to learn. Gu:ea:16 and gulcehre2016pointing propose a similar “copy” mechanism in sentence compression and other tasks; their model can accommodate both generation and extraction by selecting which sub-sequences in the input sequence to copy in the output.
We evaluate our models both automatically (in terms of Rouge) and by humans on two datasets: the benchmark DUC 2002 document summarization corpus and our own DailyMail news highlights corpus. Experimental results show that our summarizers achieve performance comparable to state-of-the-art systems employing hand-engineered features and sophisticated linguistic constraints.
In this section we formally define the summarization tasks considered in this paper. Given a document $D$ consisting of a sequence of sentences $\lbrace s_1, \cdots , s_m\rbrace $ and a word set $\lbrace w_1, \cdots , w_n\rbrace $ , we are interested in obtaining summaries at two levels of granularity, namely sentences and words.
Sentence extraction aims to create a summary from $D$ by selecting a subset of $j$ sentences (where $j<m$ ). We do this by scoring each sentence within $D$ and predicting a label $y_L \in {\lbrace 0,1\rbrace }$ indicating whether the sentence should be included in the summary. As we apply supervised training, the objective is to maximize the likelihood of all sentence labels $\mathbf {y}_L=(y_L^1, \cdots , y_L^m)$ given the input document $D$ and model parameters $\theta $ : 
$$\log p(\mathbf {y}_L |D; \theta ) = \sum \limits _{i=1}^{m} \log p(y_L^i |D; \theta )$$   (Eq. 5) 
Although extractive methods yield naturally grammatical summaries and require relatively little linguistic analysis, the selected sentences make for long summaries containing much redundant information. For this reason, we also develop a model based on word extraction which seeks to find a subset of words in $D$ and their optimal ordering so as to form a summary $\mathbf {y}_s = (w^{\prime }_1, \cdots , w^{\prime }_k), w^{\prime }_i \in D$ . Compared to sentence extraction which is a sequence labeling problem, this task occupies the middle ground between full abstractive summarization which can exhibit a wide range of rewrite operations and extractive summarization which exhibits none. We formulate word extraction as a language generation task with an output vocabulary restricted to the original document. In our supervised setting, the training goal is to maximize the likelihood of the generated sentences, which can be further decomposed by enforcing conditional dependencies among their constituent words: 
$$\hspace*{-5.69046pt}\log p(\mathbf {y}_s |D;
\theta )\hspace*{-2.84544pt}=\hspace*{-2.84544pt}\sum \limits _{i=1}^{k}\hspace*{-2.84544pt}\log p(w^{\prime }_i | D, w^{\prime }_1,\hspace*{-2.84544pt}\cdots \hspace*{-2.84544pt}, w^{\prime }_{i-1}; \theta )$$   (Eq. 7) 
In the following section, we discuss the data elicitation methods which allow us to train neural networks based on the above defined objectives.
Data-driven neural summarization models require a large training corpus of documents with labels indicating which sentences (or words) should be in the summary. Until now such corpora have been limited to hundreds of examples (e.g., the DUC 2002 single document summarization corpus) and thus used mostly for testing BIBREF7 . To overcome the paucity of annotated data for training, we adopt a methodology similar to hermann2015teaching and create two large-scale datasets, one for sentence extraction and another one for word extraction.
In a nutshell, we retrieved hundreds of thousands of news articles and their corresponding highlights from DailyMail (see Figure 1 for an example). The highlights (created by news editors) are genuinely abstractive summaries and therefore not readily suited to supervised training. To create the training data for sentence extraction, we reverse approximated the gold standard label of each document sentence given the summary based on their semantic correspondence BIBREF7 . Specifically, we designed a rule-based system that determines whether a document sentence matches a highlight and should be labeled with 1 (must be in the summary), and 0 otherwise. The rules take into account the position of the sentence in the document, the unigram and bigram overlap between document sentences and highlights, the number of entities appearing in the highlight and in the document sentence. We adjusted the weights of the rules on 9,000 documents with manual sentence labels created by woodsend2010automatic. The method obtained an accuracy of 85% when evaluated on a held-out set of 216 documents coming from the same dataset and was subsequently used to label 200K documents. Approximately 30% of the sentences in each document were deemed summary-worthy.
For the creation of the word extraction dataset, we examine the lexical overlap between the highlights and the news article. In cases where all highlight words (after stemming) come from the original document, the document-highlight pair constitutes a valid training example and is added to the word extraction dataset. For out-of-vocabulary (OOV) words, we try to find a semantically equivalent replacement present in the news article. Specifically, we check if a neighbor, represented by pre-trained embeddings, is in the original document and therefore constitutes a valid substitution. If we cannot find any substitutes, we discard the document-highlight pair. Following this procedure, we obtained a word extraction dataset containing 170K articles, again from the DailyMail.
The key components of our summarization model include a neural network-based hierarchical document reader and an attention-based hierarchical content extractor. The hierarchical nature of our model reflects the intuition that documents are generated compositionally from words, sentences, paragraphs, or even larger units. We therefore employ a representation framework which reflects the same architecture, with global information being discovered and local information being preserved. Such a representation yields minimum information loss and is flexible allowing us to apply neural attention for selecting salient sentences and words within a larger context. In the following, we first describe the document reader, and then present the details of our sentence and word extractors.
The role of the reader is to derive the meaning representation of the document from its constituent sentences, each of which is treated as a sequence of words. We first obtain representation vectors at the sentence level using a single-layer convolutional neural network (CNN) with a max-over-time pooling operation BIBREF16 , BIBREF17 , BIBREF18 . Next, we build representations for documents using a standard recurrent neural network (RNN) that recursively composes sentences. The CNN operates at the word level, leading to the acquisition of sentence-level representations that are then used as inputs to the RNN that acquires document-level representations, in a hierarchical fashion. We describe these two sub-components of the text reader below.
We opted for a convolutional neural network model for representing sentences for two reasons. Firstly, single-layer CNNs can be trained effectively (without any long-term dependencies in the model) and secondly, they have been successfully used for sentence-level classification tasks such as sentiment analysis BIBREF19 . Let $d$ denote the dimension of word embeddings, and $s$ a document sentence consisting of a sequence of $n$ words $(w_1, \cdots , w_n)$ which can be represented by a dense column matrix $\mathbf {W} \in \mathbb {R}^{n \times d}$ . We apply a temporal narrow convolution between $\mathbf {W}$ and a kernel $\mathbf {K} \in \mathbb {R}^{c \times d}$ of width $c$ as follows: 
$$\mathbf {f}^{i}_{j} = \tanh (\mathbf {W}_{j : j+c-1} \otimes \mathbf {K} + b)$$   (Eq. 12) 
where $\otimes $ equates to the Hadamard Product followed by a sum over all elements. $\mathbf {f}^i_j $ denotes the $j$ -th element of the $i$ -th feature map $\mathbf {f}^i$ and $b$ is the bias. We perform max pooling over time to obtain a single feature (the $i$ th feature) representing the sentence under the kernel $\mathbf {K}$ with width $c$ : 
$$\mathbf {s}_{i, \mathbf {K}}= \max _j \mathbf {f}_j^i$$   (Eq. 13) 
In practice, we use multiple feature maps to compute a list of features that match the dimensionality of a sentence under each kernel width. In addition, we apply multiple kernels with different widths to obtain a set of different sentence vectors. Finally, we sum these sentence vectors to obtain the final sentence representation. The CNN model is schematically illustrated in Figure 2 (bottom). In the example, the sentence embeddings have six dimensions, so six feature maps are used under each kernel width. The blue feature maps have width two and the red feature maps have width three. The sentence embeddings obtained under each kernel width are summed to get the final sentence representation (denoted by green).
At the document level, a recurrent neural network composes a sequence of sentence vectors into a document vector. Note that this is a somewhat simplistic attempt at capturing document organization at the level of sentence to sentence transitions. One might view the hidden states of the recurrent neural network as a list of partial representations with each focusing mostly on the corresponding input sentence given the previous context. These representations altogether constitute the document representation, which captures local and global sentential information with minimum compression.
The RNN we used has a Long Short-Term Memory (LSTM) activation unit for ameliorating the vanishing gradient problem when training long sequences BIBREF20 . Given a document $d=(s_1,
\cdots , s_m)$ , the hidden state at time step $t$ , denoted by $\mathbf {h_t}$ , is updated as: 
$$\begin{bmatrix}
\mathbf {i}_t\\ \mathbf {f}_t\\ \mathbf {o}_t\\ \mathbf {\hat{c}}_t
\end{bmatrix} =
\begin{bmatrix} \sigma \\ \sigma \\ \sigma \\ \tanh \end{bmatrix} \mathbf {W}\cdot \begin{bmatrix} \mathbf {h}_{t-1}\\ \mathbf {s}_t
\end{bmatrix}$$   (Eq. 15) 
$$ \mathbf {c}_t = \mathbf {f}_t \odot \mathbf {c}_{t-1} +
\mathbf {i}_t \odot \mathbf {\hat{c}}_t$$   (Eq. 16) 
where $\mathbf {W}$ is a learnable weight matrix. Next, we discuss a special attention mechanism for extracting sentences and words given the recurrent document encoder just described, starting from the sentence extractor.
In the standard neural sequence-to-sequence modeling paradigm BIBREF11 , an attention mechanism is used as an intermediate step to decide which input region to focus on in order to generate the next output. In contrast, our sentence extractor applies attention to directly extract salient sentences after reading them.
The extractor is another recurrent neural network that labels sentences sequentially, taking into account not only whether they are individually relevant but also mutually redundant. The complete architecture for the document encoder and the sentence extractor is shown in Figure 2 . As can be seen, the next labeling decision is made with both the encoded document and the previously labeled sentences in mind. Given encoder hidden states $(h_1, \cdots , h_m)$ and extractor hidden states $(\bar{h}_1, \cdots , \bar{h}_m)$ at time step $t$ , the decoder attends the $t$ -th sentence by relating its current decoding state to the corresponding encoding state: 
$$\bar{\mathbf {h}}_{t} = \text{LSTM} ( p_{t-1} \mathbf {s}_{t-1}, \mathbf {\bar{h}}_{t-1})$$   (Eq. 20) 
$$p(y_L(t)=1 | D ) = \sigma (\text{MLP} (\mathbf {\bar{h}}_t : \mathbf {h}_t) )$$   (Eq. 21) 
where MLP is a multi-layer neural network with as input the concatenation of $\mathbf {\bar{h}}_t$ and $\mathbf {h}_t$ . $p_{t-1}$ represents the degree to which the extractor believes the previous sentence should be extracted and memorized ( $p_{t-1}$ =1 if the system is certain; 0 otherwise).
In practice, there is a discrepancy between training and testing such a model. During training we know the true label $p_{t-1}$ of the previous sentence, whereas at test time $p_{t-1}$ is unknown and has to be predicted by the model. The discrepancy can lead to quickly accumulating prediction errors, especially when mistakes are made early in the sequence labeling process. To mitigate this, we adopt a curriculum learning strategy BIBREF21 : at the beginning of training when $p_{t-1}$ cannot be predicted accurately, we set it to the true label of the previous sentence; as training goes on, we gradually shift its value to the predicted label $p(y_L(t-1)=1
| d )$ .
Compared to sentence extraction which is a purely sequence labeling task, word extraction is closer to a generation task where relevant content must be selected and then rendered fluently and grammatically. A small extension to the structure of the sequential labeling model makes it suitable for generation: instead of predicting a label for the next sentence at each time step, the model directly outputs the next word in the summary. The model uses a hierarchical attention architecture: at time step $t$ , the decoder softly attends each document sentence and subsequently attends each word in the document and computes the probability of the next word to be included in the summary $p(w^{\prime }_t = w_i|
d, w^{\prime }_1, \cdots , w^{\prime }_{t-1})$ with a softmax classifier: 
$$\bar{\mathbf {h}}_{t} = \text{LSTM} ( \mathbf {w^{\prime }}_{t-1},
\mathbf {\bar{h}}_{t-1})\footnote {We empirically found that feeding
the previous sentence-level attention vector as additional
input to the LSTM would lead to small performance improvements.
This is not shown in the equation.}$$   (Eq. 25) 
$$a_j^t = \mathbf {z}^\mathtt {T} \tanh (\mathbf {W}_e \mathbf {\bar{h}}_t + \mathbf {W}_r \mathbf {h}_j), h_j \in D$$   (Eq. 26) 
In the above equations, $\mathbf {w}_i$ corresponds to the vector of the $i$ -th word in the input document, whereas $\mathbf {z}$ , $\mathbf {W}_e$ , $\mathbf {W}_r$ , $\mathbf {v}$ , $\mathbf {W}_{e^{\prime }}$ , and $\mathbf {W}_{r^{\prime }}$ are model weights. The model architecture is shown in Figure 3 .
The word extractor can be viewed as a conditional language model with a vocabulary constraint. In practice, it is not powerful enough to enforce grammaticality due to the lexical diversity and sparsity of the document highlights. A possible enhancement would be to pair the extractor with a neural language model, which can be pre-trained on a large amount of unlabeled documents and then jointly tuned with the extractor during decoding BIBREF23 . A simpler alternative which we adopt is to use $n$ -gram features collected from the document to rerank candidate summaries obtained via beam decoding. We incorporate the features in a log-linear reranker whose feature weights are optimized with minimum error rate training BIBREF24 .
In this section we present our experimental setup for assessing the performance of our summarization models. We discuss the datasets used for training and evaluation, give implementation details, briefly introduce comparison models, and explain how system output was evaluated.
Table 1 (upper half) summarizes our results on the DUC 2002 test dataset using Rouge. nn-se represents our neural sentence extraction model, nn-we our word extraction model, and nn-abs the neural abstractive baseline. The table also includes results for the lead baseline, the logistic regression classifier (lreg), and three previously published systems (ilp, tgraph, and urank).
The nn-se outperforms the lead and lreg baselines with a significant margin, while performing slightly better than the ilp model. This is an encouraging result since our model has only access to embedding features obtained from raw text. In comparison, lreg uses a set of manually selected features, while the ilp system takes advantage of syntactic information and extracts summaries subject to well-engineered linguistic constraints, which are not available to our models. Overall, our sentence extraction model achieves performance comparable to the state of the art without sophisticated constraint optimization (ilp, tgraph) or sentence ranking mechanisms (urank). We visualize the sentence weights of the nn-se model in the top half of Figure 4 . As can be seen, the model is able to locate text portions which contribute most to the overall meaning of the document.
Rouge scores for the word extraction model are less promising. This is somewhat expected given that Rouge is $n$ -gram based and not very well suited to measuring summaries which contain a significant amount of paraphrasing and may deviate from the reference even though they express similar meaning. However, a meaningful comparison can be carried out between nn-we and nn-abs which are similar in spirit. We observe that nn-we consistently outperforms the purely abstractive model. As nn-we generates summaries by picking words from the original document, decoding is easier for this model compared to nn-abs which deals with an open vocabulary. The extraction-based generation approach is more robust for proper nouns and rare words, which pose a serious problem to open vocabulary models. An example of the generated summaries for nn-we is shown at the lower half of Figure 4 .
Table 1 (lower half) shows system results on the 500 DailyMail news articles (test set). In general, we observe similar trends to DUC 2002, with nn-se performing the best in terms of all rouge metrics. Note that scores here are generally lower compared to DUC 2002. This is due to the fact that the gold standard summaries (aka highlights) tend to be more laconic and as a result involve a substantial amount of paraphrasing. More experimental results on this dataset are provided in the appendix.
The results of our human evaluation study are shown in Table 2 . Specifically, we show, proportionally, how often our participants ranked each system 1st, 2nd, and so on. Perhaps unsurprisingly, the human-written descriptions were considered best and ranked 1st 27% of the time, however closely followed by our nn-se model which was ranked 1st 22% of the time. The ilp system was mostly ranked in 2nd place (38% of the time). The rest of the systems occupied lower ranks. We further converted the ranks to ratings on a scale of 1 to 6 (assigning ratings 6 $\dots $ 1 to rank placements 1 $\dots $ 6). This allowed us to perform Analysis of Variance (ANOVA) which revealed a reliable effect of system type. Specifically, post-hoc Tukey tests showed that nn-se and ilp are significantly ( $p < 0.01$ ) better than lead, nn-we, and nn-abs but do not differ significantly from each other or the human goldstandard.
In this work we presented a data-driven summarization framework based on an encoder-extractor architecture. We developed two classes of models based on sentence and word extraction. Our models can be trained on large scale datasets and learn informativeness features based on continuous representations without recourse to linguistic annotations. Two important ideas behind our work are the creation of hierarchical neural structures that reflect the nature of the summarization task and generation by extraction. The later effectively enables us to sidestep the difficulties of generating under a large vocabulary, essentially covering the entire dataset, with many low-frequency words and named entities.
Directions for future work are many and varied. One way to improve the word-based model would be to take structural information into account during generation, e.g., by combining it with a tree-based algorithm BIBREF31 . It would also be interesting to apply the neural models presented here in a phrase-based setting similar to lebret2015phrase. A third direction would be to adopt an information theoretic perspective and devise a purely unsupervised approach that selects summary sentences and words so as to minimize information loss, a task possibly achievable with the dataset created in this work.
We would like to thank three anonymous reviewers and members of the ILCC at the School of Informatics for their valuable feedback. The support of the European Research Council under award number 681760 “Translating Multiple Modalities into Text” is gratefully acknowledged.
In addition to the DUC 2002 and 500 DailyMail samples, we additionally report results on the entire DailyMail test set (Table 3 ). Since there is no established evaluation standard for this task, we experimented with three different ROUGE limits: 75 bytes, 275 bytes and full length.

Figure and table captions:
Figure 3: Neural attention mechanism for word extraction.
Figure 2: A recurrent convolutional document reader with a neural sentence extractor.
Table 2: Rankings (shown as proportions) and mean ranks given to systems by human participants (lower is better).
Table 1: ROUGE evaluation (%) on DUC-2002 and DailyMail corpora.
Figure 4: Visualization of the summaries for a DailyMail article. The top half shows the relative attention weights given by the sentence extraction model. Darkness indicates sentence importance. The lower half shows the summary generated by the word extraction.

Questions about this paper:
What dataset do they use?

Answers:
DailyMail news articles