Venue: NAACL 2016

# Abstractive Sentence Summarization

with Attentive Recurrent Neural Networks

 Sumit Chopra

Facebook AI Research

spchopra@fb.com &Michael Auli

Facebook AI Research

michaelauli@fb.com &Alexander M. Rush

Harvard SEAS

srush@seas.harvard.edu

###### Abstract

Abstractive Sentence Summarization generates a shorter version of a given sentence while attempting to preserve its meaning. We introduce a conditional recurrent neural network (RNN) which generates a summary of an input sentence. The conditioning is provided by a novel convolutional attention-based encoder which ensures that the decoder focuses on the appropriate input words at each step of generation. Our model relies only on learned features and is easy to train in an end-to-end fashion on large data sets. Our experiments show that the model significantly outperforms the recently proposed state-of-the-art method on the Gigaword corpus while performing competitively on the DUC-2004 shared task.

## 1 Introduction

Generating a condensed version of a passage while preserving its meaning is known as text summarization. Tackling this task is an important step towards natural language understanding. Summarization systems can be broadly classified into two categories. _Extractive models_ generate summaries by cropping important segments from the original text and putting them together to form a coherent summary. _Abstractive models_ generate summaries from scratch without being constrained to reuse phrases from the original text.

In this paper we propose a novel recurrent neural network for the problem of _abstractive sentence summarization_. Inspired by the recently proposed architectures for machine translation Bahdanau et al. (2014), our model consists of a conditional recurrent neural network, which acts as a decoder to generate the summary of an input sentence, much like a standard recurrent language model. In addition, at every time step the decoder also takes a conditioning input which is the output of an encoder module. Depending on the current state of the RNN, the encoder computes scores over the words in the input sentence. These scores can be interpreted as a soft alignment over the input text, informing the decoder which part of the input sentence it should focus on to generate the next word. Both the decoder and encoder are jointly trained on a data set consisting of sentence-summary pairs. Our model can be seen as an extension of the recently proposed model for the same problem by Rush et al. (2015). While they use a feed-forward neural language model for generation, we use a recurrent neural network. Furthermore, our encoder is more sophisticated, in that it explicitly encodes the position information of the input words. Lastly, our encoder uses a convolutional network to encode input words. These extensions result in improved performance.

The main contribution of this paper is a novel convolutional attention-based conditional recurrent neural network model for the problem of abstractive sentence summarization. Empirically we show that our model beats the state-of-the-art systems of Rush et al. (2015) on multiple data sets. Particularly notable is the fact that even with a simple generation module, which does not use any extractive feature tuning, our model manages to significantly outperform their ABS+ system on the Gigaword data set and is comparable on the DUC-2004 task.



Previous Work

While there is a large body of work for generating extractive summaries of sentences (Jing, 2000; Knight and Marcu, 2002; McDonald, 2006; Clarke and Lapata, 2008; Filippova and Altun, 2013; Filippova et al., 2015), there has been much less research on abstractive summarization. A count-based noisy-channel machine translation model was proposed for the problem in Banko et al. (2000). The task of abstractive sentence summarization was later formalized around the DUC-2003 and DUC-2004 competitions (Over et al., 2007), where the Topinary system (Zajic et al., 2004) was the state-of-the-art. More recently Cohn and Lapata (2008) and later Woodsend et al. (2010) proposed systems which made heavy use of the syntactic features of the sentence-summary pairs. Later, along the lines of Banko et al. (2000), Moses was used directly as a method for text simplification by Wubben et al. (2012). Other works which have recently been proposed for the problem of sentence summarization include (Galanis and Androutsopoulos, 2010; Napoles et al., 2011; Cohn and Lapata, 2013). Very recently Rush et al. (2015) proposed a neural attention model for this problem using a new data set for training and showing state-of-the-art performance on the DUC tasks. Our model can be seen as an extension of their model.

## 3 Attentive Recurrent Architecture

Let \({\bf x}\) denote the input sentence consisting of a sequence of \(M\) words \({\bf x}=[x_{1},\ldots,x_{M}]\), where each word \(x_{i}\) is part of vocabulary \({\cal V}\), of size \(|{\cal V}|=V\). Our task is to generate a target sequence \({\bf y}=[y_{1},\ldots,y_{N}]\), of \(N\) words, where \(N<M\), such that the meaning of \({\bf x}\) is preserved: \({\bf y}=\operatorname*{argmax}_{y}P(y|{\bf x})\), where y is a random variable denoting a sequence of \(N\) words.

Typically the conditional probability is modeled by a parametric function with parameters \(\theta\): \(P(y|{\bf x})=P(y|{\bf x};\theta)\). Training involves finding the \(\theta\) which maximizes the conditional probability of sentence-summary pairs in the training corpus. If the model is trained to generate the next word of the summary, given the previous words, then the above conditional can be factorized into a product of individual conditional probabilities:

\[P({\bf y}|{\bf x};\theta)=\prod_{t=1}^{N}p(y_{t}|\{y_{1},\ldots,y_{t-1}\},{ \bf x};\theta). \tag{1}\]

In this work we model this conditional probability using an RNN Encoder-Decoder architecture, inspired by Cho et al. (2014) and subsequently extended in Bahdanau et al. (2014). We call our model RAS (Recurrent Attentive Summarizer).

### Recurrent Decoder

The above conditional is modeled using an RNN:

\[P(y_{t}|\{y_{1},\ldots,y_{t-1}\},{\bf x};\theta)=P_{t}=g_{\theta_{1}}(h_{t},c_ {t}),\]

where \(h_{t}\) is the hidden state of the RNN:

\[h_{t}=g_{\theta_{1}}(y_{t-1},h_{t-1},c_{t}).\]

Here \(c_{t}\) is the output of the encoder module (detailed in SS3.2). It can be seen as a context vector which is computed as a function of the current state \(h_{t-1}\) and the input sequence \({\bf x}\).

Our Elman RNN takes the following form (Elman, 1990):

\[h_{t} = \sigma(W_{1}y_{t-1}+W_{2}h_{t-1}+W_{3}c_{t})\] \[P_{t} = \rho(W_{4}h_{t}+W_{5}c_{t}),\]

where \(\sigma\) is the sigmoid function and \(\rho\) is the softmax, defined as: \(\rho(o_{t})=e^{o_{t}}/\sum_{j}e^{o_{j}}\) and \(W_{i}\) (\(i=1,\ldots,5\)) are matrices of learnable parameters of sizes \(W_{\{1,2,3\}}\in{\mathcal{R}}^{d\times d}\) and \(W_{\{4,5\}}\in{\mathcal{R}}^{d\times V}\).

The LSTM decoder is defined as (Hochreiter and Schmidhuber, 1997):

\[i_{t} = \sigma(W_{1}y_{t-1}+W_{2}h_{t-1}+W_{3}c_{t})\] \[i^{\prime}_{t} = \tanh(W_{4}y_{t-1}+W_{5}h_{t-1}+W_{6}c_{t})\] \[f_{t} = \sigma(W_{7}y_{t-1}+W_{8}h_{t-1}+W_{9}c_{t})\] \[o_{t} = \sigma(W_{10}y_{t-1}+W_{11}h_{t-1}+W_{12}c_{t})\] \[m_{t} = m_{t-1}\odot f_{t}+i_{t}\odot i^{\prime}_{t}\] \[h_{t} = m_{t}\odot o_{t}\] \[P_{t} = \rho(W_{13}h_{t}+W_{14}c_{t}).\]

Operator \(\odot\) refers to component-wise multiplication, and \(W_{i}\) (\(i=1,\ldots,14\)) are matrices of learnable parameters of sizes \(W_{\{1,...,12\}}\in{\mathcal{R}}^{d\times d}\), and \(W_{\{13,14\}}\in{\mathcal{R}}^{d\times V}\).

 



### Attentive Encoder

We now give the details of the encoder which computes the context vector \(c_{t}\) for every time step \(t\) of the decoder above. With a slight overload of notation, for an input sentence \(\mathbf{x}\) we denote by \(x_{i}\) the \(d\) dimensional learnable embedding of the \(i\)-th word (\(x_{i}\in\mathcal{R}^{d}\)). In addition the position \(i\) of the word \(x_{i}\) is also associated with a learnable embedding \(l_{i}\) of size \(d\) (\(l_{i}\in\mathcal{R}^{d}\)). Then the full embedding for \(i\)-th word in \(\mathbf{x}\) is given by \(a_{i}=x_{i}+l_{i}\). Let us denote by \(B^{k}\in\mathcal{R}^{q\times d}\) a learnable weight matrix which is used to convolve over the full embeddings of consecutive words. Let there be \(d\) such matrices (\(k\in\{1,\dots,d\}\)). The output of convolution is given by:

\[z_{ik}=\sum_{h=-q/2}^{q/2}a_{i+h}\cdot b_{q/2+h}^{k}, \tag{2}\]

where \(b_{j}^{k}\) is the \(j\)-th column of the matrix \(B^{k}\). Thus the \(d\) dimensional aggregate embedding vector \(z_{i}\) is defined as \(z_{i}=[z_{i1},\ldots,z_{id}]\). Note that each word \(x_{i}\) in the input sequence is associated with one aggregate embedding vector \(z_{i}\). The vectors \(z_{i}\) can be seen as a representation of the word which captures the position in which it occurs in the sentence and also the context in which it appears in the sentence. In our experiments the width \(q\) of the convolution matrix \(B^{k}\) was set to \(5\). To account for words at the boundaries of \(\mathbf{x}\) we first pad the sequence on both sides with dummy words before computing the aggregate vectors \(z_{i}\)'s.

Given these aggregate vectors of words, we compute the context vector \(c_{t}\) (the encoder output) as:

\[c_{t}=\sum_{j=1}^{M}\alpha_{j,t-1}x_{j}, \tag{3}\]

where the weights \(\alpha_{j,t-1}\) are computed as

\[\alpha_{j,t-1}=\frac{\exp(z_{j}\cdot h_{t-1})}{\sum_{i=1}^{M}\exp(z_{i}\cdot h _{t-1})}. \tag{4}\]

### Training and Generation

Given a training corpus \(\mathcal{S}=\{(\mathbf{x}^{i},\mathbf{y}^{i})\}_{i=1}^{S}\) of \(S\) sentence-summary pairs, the above model can be trained end-to-end using stochastic gradient descent by minimizing the negative conditional log likelihood of the training data with respect to \(\theta\):

\[\mathcal{L}=-\sum_{i=1}^{S}\sum_{t=1}^{N}\log P(y_{t}^{i}|\{y_{1}^{i},\ldots, y_{t-1}^{i}\},\mathbf{x}^{i};\theta), \tag{5}\]

where the parameters \(\theta\) constitute the parameters of the decoder and the encoder.

Once the parametric model is trained we generate a summary for a new sentence \(\mathbf{x}\) through a word-based beam search such that \(P(y|\mathbf{x})\) is maximized, \(\operatorname*{argmax}P(y_{t}|\{y_{1},\ldots,y_{t-1}\},\mathbf{x})\). The search is parameterized by the number of paths \(k\) that are pursued at each time step.

## 4 Experimental Setup

### Datasets and Evaluation

Our models are trained on the annotated version of the Gigaword corpus (Graff et al., 2003; Napoles et al., 2012) and we use only the annotations for tokenization and sentence separation while discarding other annotations such as tags and parses. We pair the first sentence of each article with its headline to form sentence-summary pairs. The data is pre-processed in the same way as Rush et al. (2015) and we use the same splits for training, validation, and testing. For Gigaword we report results on the same randomly held-out test set of \(2000\) sentence-summary pairs as (Rush et al., 2015).1 We also evaluate our models on the DUC-2004 evaluation data set comprising 500 pairs (Over et al., 2007). Our evaluation is based on three variants of ROUGE (Lin, 2004), namely, ROUGE-1 (unigrams), ROUGE-2 (bigrams), and ROUGE-L (longest-common substring).

Footnote 1: We remove pairs with empty titles resulting in slightly different accuracy compared to Rush et al. (2015) for their systems.

### Architectural Choices

We implemented our models in the Torch library ([http://torch.ch/](http://torch.ch/))2. To optimize our loss (Equation 5) we used stochastic gradient descent with mini-batches of size \(32\). During training we measure the perplexity of the summaries in the validation set and adjust our hyper-parameters, such as the learning rate, based on this number.



For the decoder we experimented with both the Elman RNN and the Long-Short Term Memory (LSTM) architecture (as discussed in SS 3.1). We chose hyper-parameters based on a grid search and picked the one which gave the best perplexity on the validation set. In particular we searched over the number of hidden units \(H\) of the recurrent layer, the learning rate \(\eta\), the learning rate annealing schedule \(\gamma\) (the factor by which to decrease \(\eta\) if the validation perplexity increases), and the gradient clipping threshold \(\kappa\). Our final Elman architecture (RAS-Elman) uses a single layer with \(H=512\), \(\eta=0.5\), \(\gamma=2\), and \(\kappa=10\). The LSTM model (RAS-LSTM) also has a single layer with \(H=512\), \(\eta=0.1\), \(\gamma=2\), and \(\kappa=10\).

## 5 Results

On the Gigaword corpus we evaluate our models in terms of perplexity on a held-out set. We then pick the model with best perplexity on the held-out set and use it to compute the F1-score of ROUGE-1, ROUGE-2, and ROUGE-L on the test sets, all of which we report. For the DUC corpus however, inline with the standard, we report the recall-only ROUGE. As baseline we use the state-of-the-art attention-based system (ABS) of Rush et al. (2015) which relies on a feed-forward network decoder. Additionally, we compare to an enhanced version of their system (ABS+), which relies on a range of separate extractive summarization features that are added as log-linear features in a secondary learning step with minimum error rate training [6].

Table 1 shows that both our RAS-Elman and RAS-LSTM models achieve lower perplexity than ABS as well as other models reported in Rush et al. (2015). The RAS-LSTM performs slightly worse than RAS-Elman, most likely due to over-fitting. We attribute this to the relatively simple nature of this task which can be framed as English-to-English translation with few long-term dependencies. The ROUGE results (Table 2) show that our models comfortably outperform both ABS and ABS+ by a wide margin on all metrics. This is even the case when we rely only on very fast greedy search (\(k=1\)), while as ABS uses a much wider beam of size \(k=50\); the stronger ABS+ system also uses additional extractive features which our model does not. These features cause ABS+ to copy 92% of words from the input into the summary, whereas our model copies only 74% of the words leading to more abstractive summaries. On DUC-2004 we report recall ROUGE as is customary on this dataset. The results (Table 3) show that our models are better than ABS+. However the improvements are smaller than for Gi

\begin{table}
\begin{tabular}{l r} \hline Model & Perplexity \\ \hline Bag-of-Words & 43.6 \\ Convolutional (TDNN) & 35.9 \\ Attention-based (ABS) & 27.1 \\ RAS-Elman & **18.9** \\ RAS-LSTM & 20.3 \\ \hline \end{tabular}
\end{table}
Table 1: Perplexity on the Gigaword validation set. Bag-of-words, Convolutional (TDNN) and ABS are the different encoders of Rush et. al., 2015.

\begin{table}
\begin{tabular}{l c c c} \hline  & RG-1 & RG-2 & RG-L \\ \hline ABS & 29.55 & 11.32 & 26.42 \\ ABS+ & 29.76 & 11.88 & 26.96 \\ RAS-Elman (\(k=1\)) & 33.10 & 14.45 & 30.25 \\ RAS-Elman (\(k=10\)) & **33.78** & **15.97** & **31.15** \\ RAS-LSTM (\(k=1\)) & 31.71 & 13.63 & 29.31 \\ RAS-LSTM (\(k=10\)) & 32.55 & 14.70 & 30.03 \\ \hline Luong-NMT & 33.10 & 14.45 & 30.71 \\ \hline \end{tabular}
\end{table}
Table 2: F1 ROUGE scores on the Gigaword test set. ABS and ABS+ are the systems of Rush et al. 2015. \(k\) refers to the size of the beam for generation; \(k=1\) implies greedy generation. RG refers to ROUGE. Rush et al. (2015) previously reported ROUGE recall, while as we use the more balanced F-measure.

\begin{table}
\begin{tabular}{l l c c} \hline  & RG-1 & RG-2 & RG-L \\ \hline ABS & 26.55 & 7.06 & 22.05 \\ ABS+ & 28.18 & 8.49 & 23.81 \\ RAS-Elman (\(k=1\)) & 29.13 & 7.62 & 23.92 \\ RAS-Elman (\(k=10\)) & **28.97** & **8.26** & **24.06** \\ RAS-LSTM (\(k=1\)) & 26.90 & 6.57 & 22.12 \\ RAS-LSTM (\(k=10\)) & 27.41 & 7.69 & 23.06 \\ \hline Luong-NMT & 28.55 & 8.79 & 24.43 \\ \hline \end{tabular}
\end{table}
Table 3: ROUGE results (recall-only) on the DUC-2004 test sets. ABS and ABS+ are the systems of Rush et al. 2015. \(k\) refers to the size of the beam for generation; \(k=1\) implies greedy generation. RG refers to ROUGE.

 

gaword which is likely due to two reasons: First, tokenization of DUC-2004 differs slightly from our training corpus. Second, headlines in Gigaword are much shorter than in DUC-2004.

For the sake of completeness we also compare our models to the recently proposed standard Neural Machine Translation (NMT) systems. In particular, we compare to a smaller re-implementation of the attentive stacked LSTM encoder-decoder of Luong et al. (2015). Our implementation uses two-layer LSTMs for the encoder-decoder with \(500\) hidden units in each layer. Tables 2 and 3 report ROUGE scores on the two data sets. From the tables we observe that the proposed RAS-Elman model is able to match the performance of the NMT model of Luong at al. (2015). This is noteworthy because RAS-Elman is significantly simpler than the NMT model at multiple levels. First, the encoder used by RAS-Elman is extremely light-weight (attention over the convolutional representation of the input words), compared to Luong's (a 2 hidden layer LSTM). Second, the decoder used by RAS-Elman is a single layer standard (Elman) RNN as opposed to a multi-layer LSTM. In an independent work, Nallapati et. al (2016) also trained a collection of standard NMT models and report numbers in the same ballpark as RAS-Elman on both datasets.

In order to better understand which component of the proposed architecture is responsible for the improvements, we trained the recurrent model with Rush et. al., (2015)'s ABS encoder on a subset of the Gigaword dataset. The ABS encoder, which does not have the position features, achieves a final validation perplexity of \(38\) compared to \(29\) for the proposed encoder, which uses position features as well as context information. This clearly shows the benefits of using the position feature in the proposed encoder.

Finally in Figure 1 we highlight anecdotal examples of summaries produced by the RAS-Elman system on the Gigaword dataset. The first two examples highlight typical improvements in the RAS model over ABS+. Generally the model produces more fluent summaries and is better able to capture the main actors of the input. For instance in Sentence 1, RAS-Elman correctly distinguishes the actions of "pepe" from "ferreira", and in Sentence 2 it identifies the correct role of the "think tank". The final two examples highlight typical mistakes of the models. In Sentence 3 both models take literally the figurative use of the idiom "a silence that spoke volumes," and produce fluent but nonsensical summaries. In Sentence 4 the RAS model mistakes the content of a relative clause for the main verb, leading to a summary with the opposite meaning of the input. These difficult cases are somewhat rare in the Gigaword, but they highlight future challenges for obtaining human-level sentence summary.

## 6 Conclusion

We extend the state-of-the-art model for abstractive sentence summarization Rush et al. (2015) to a recurrent neural network architecture. Our model is a simplified version of the encoder-decoder framework for machine translation Bahdanau et al. (2014). The model is trained on the Gigaword corpus to generate headlines based on the first line of each news article. We comfortably outperform the previous state-of-the-art on both Gigaword data and the DUC-2004 challenge even though our model does not rely on additional extractive features.

Figure 1: Example sentence summaries produced on Gigaword. **I** is the input, **G** is the true headline, **A** is Abs+, and **R** is RAS-Elman.

 



## References

* [Bahdanau et al.2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _CoRR_, abs/1409.0473.
* Banko et al. [2000] Michele Banko, Vibhu O Mittal, and Michael J Witbrock. 2000. Headline generation based on statistical translation. In _Proceedings of the 38th Annual Meeting on Association for Computational Linguistics_, pages 318-325. Association for Computational Linguistics.
* Cho et al. [2014] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In _Proceedings of EMNLP 2014_, pages 1724-1734.
* Clarke and Lapata [2008] James Clarke and Mirella Lapata. 2008. Global inference for sentence compression: An integer linear programming approach. _Journal of Artificial Intelligence Research_, pages 399-429.
* Cohn and Lapata [2008] Trevor Cohn and Mirella Lapata. 2008. Sentence compression beyond word deletion. In _Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1_, pages 137-144. Association for Computational Linguistics.
* Cohn and Lapata [2013] Trevor Cohn and Mirella Lapata. 2013. An abstractive approach to sentence compression. _ACM Transactions on Intelligent Systems and Technology (TIST'13)_, 4,3(41).
* Elman [1990] Jeffrey L. Elman. 1990. Finding structure in time. _Cognitive Science_, 14(2):179-211.
* Filippova and Altun [2013] Katja Filippova and Yasemin Altun. 2013. Overcoming the lack of parallel data in sentence compression. In _EMNLP_, pages 1481-1491.
* Filippova et al. [2015] Katja Filippova, Enrique Alfonseca, Carlos A Colmenares, Lukasz Kaiser, and Oriol Vinyals. 2015. Sentence compression by deletion with lstms. In _EMNLP_.
* Galanis and Androutsopoulos [2010] Dimitrios Galanis and Ion Androutsopoulos. 2010. An extractive supervised two-stage method for sentence compression. In _Proceedings of NAACL-HLT 2010_.
* Graff et al. [2003] David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003. English gigaword. _Linguistic Data Consortium, Philadelphia_.
* Hochreiter and Schmidhuber [1997] S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. _Neural Computation_, 9(8):1735-1780.
* Jing [2000] H Jing. 2000. Sentence reduction for automatic text summarization. In _ANLP-00_, pages 703-711.
* Knight and Marcu [2002] Kevin Knight and Daniel Marcu. 2002. Summarization beyond sentence extraction: A probabilistic approach to sentence compression. _Artificial Intelligence_, 139(1):91-107.
* Lin [2004] Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _Text Summarization Branches Out: Proceedings of the ACL-04 Workshop_, pages 74-81.
* Luong et al. [2015] Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1412-1421, Lisbon, Portugal, September. Association for Computational Linguistics.
* McDonald [2006] R McDonald. 2006. Discriminative sentence compression with soft syntactic evidence. In _EACL-06_, pages 297-304.
* Nallapati et al. [2016] Ramesh Nallapati, Bing Xiang, and Zhou Bowen. 2016. Sequence-to-sequence rnns for text summarization. In _[http://arxiv.org/abs/1602.06023_](http://arxiv.org/abs/1602.06023_).
* Napoles et al. [2011] Courtney Napoles, Chris Callison-Burch, Juri Ganitkevitch, and Benjamin Van Durme. 2011. Paraphratic sentence compression with a character-based metric: Tightening without deletion. In _Proceedings of the Workshop on Monolingual Text-To-Text Generation (MTTG'11)_.
* Napoles et al. [2012] Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In _Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction_, pages 95-100. Association for Computational Linguistics.
* Och [2003] Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In _Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1_, pages 160-167. Association for Computational Linguistics.
* Over et al. [2007] Paul Over, Hoa Dang, and Donna Harman. 2007. Duc in context. _Information Processing & Management_, 43(6):1506-1520.
* Rush et al. [2015] Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In _EMNLP_.
* Woodsend et al. [2010] Kristian Woodsd, Yansong Feng, and Mirella Lapata. 2010. Generation with quasi-synchronous grammar. In _Proceedings of the 2010 conference on empirical methods in natural language processing_, pages 513-523. Association for Computational Linguistics.
* Wubben et al. [2012]Sander Wubben, Antal Van Den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1_, pages 1015-1024. Association for Computational Linguistics.
* Zajic et al. [2004] David Zajic, Bonnie Dorr, and Richard Schwartz. 2004. Bbn/umd at duc-2004: Topjary. In _Proceedings of the HLT-NAACL 2004 Document Understanding Workshop, Boston_, pages 112-119