Venue: NAACL 2018

# A Discourse-Aware Attention Model for

Abstractive Summarization of Long Documents

 Arman Cohan\({}^{\dagger}\)  Franck Dernoncourt\({}^{\star}\)  Doo Soon Kim\({}^{\star}\)  Trung Bui\({}^{\star}\)

Seokhwan Kim\({}^{\star}\)   Walter Chang\({}^{\star}\)    Nazli Goharian\({}^{\dagger}\)

\({}^{\dagger}\)Department of Computer Science, Georgetown University, Washington, DC

{arman,nazli}@ir.cs.georgetown.edu

\({}^{\star}\)Adobe Research, San Jose, CA

{dernonco,dkim,bui,seokim,wachang}@adobe.com

###### Abstract

Neural abstractive summarization models have led to promising results in summarizing relatively short documents. We propose the first model for abstractive summarization of single, longer-form documents (e.g., research papers). Our approach consists of a new hierarchical encoder that models the discourse structure of a document, and an attentive discourse-aware decoder to generate the summary. Empirical results on two large-scale datasets of scientific papers show that our model significantly outperforms state-of-the-art models.

## 1 Introduction

Existing large-scale summarization datasets consist of relatively short documents. For example, articles in the CNN/Daily Mail dataset Hermann et al. (2015) are on average about 600 words long. Similarly, existing neural summarization models have focused on summarizing sentences and short documents. In this work, we propose a model for effective abstractive summarization of longer documents. Scientific papers are an example of documents that are significantly longer than news articles (see Table 1). They also follow a standard discourse structure describing the problem, methodology, experiments/results, and finally conclusions Suppe (1998).

Most summarization works in the literature focus on extractive summarization. Examples of prominent approaches include frequency-based methods Vanderweronde et al. (2007), graph-based methods Erkan and Radev (2004), topic modeling Steinberger and Jezek (2004), and neural models Nallapati et al. (2017). Abstractive summarization is an alternative approach where the generated summary may contain novel words and phrases and is more similar to how humans summarize documents Jing (2002). Recently, neural methods have led to encouraging results in abstractive summarization Nallapati et al. (2016); See et al. (2017); Paulus et al. (2017); Li et al. (2017). These approaches employ a general framework of sequence-to-sequence (seq2seq) models Sutskever et al. (2014) where the document is fed to an encoder network and another (recurrent) network learns to decode the summary. While promising, these methods focus on summarizing news articles which are relatively short. Many other document types, however, are longer and structured. Seq2seq models tend to struggle with longer sequences because at each decoding step, the decoder needs to learn to construct a context vector capturing relevant information from all the tokens in the source sequence Shao et al. (2017).

Our main contribution is an abstractive model for summarizing scientific papers which are an example of long-form structured document types. Our model includes a hierarchical encoder, capturing the discourse structure of the document and a discourse-aware decoder that generates the summary. Our decoder attends to different discourse sections and allows the model to more accurately represent important information from the source resulting in a better context vector. We also introduce two large-scale datasets of long and structured scientific papers obtained from arXiv and PubMed to support both training and evaluating models on the task of long document summarization. Evaluation results show that our method outperforms state-of-the-art summarization models1.

Footnote 1: Data/code: [https://github.com/acohan/long-summarization](https://github.com/acohan/long-summarization)

## 2 Background

In the seq2seq framework for abstractive summarization, an input document x is encoded using a Recurrent Neural Network (RNN) with \(\mathbf{h}_{i}^{(e)}\) being the hidden state of the encoder at timestep \(i\). The last step of the encoder is fed as input to another RNN which decodes the output one token

at a time. Given an input document along with the corresponding ground-truth summary \({\bf y}\), the model is trained to output a summary \(\hat{\mathbf{y}}\) that is close to \({\bf y}\). The output at timestep \(t\) is predicted using the decoder input \({\bf x^{\prime}_{t}}\), decoder hidden state \({\bf h}_{t}^{(d)}\), and some information about the input sequence. This framework is the general seq2seq framework employed in many generation tasks including machine translation Sutskever et al. (2014); Bahdanau et al. (2014) and summarization Nallapati et al. (2016); Chopra et al. (2016).

**Attentive decoding** The attention mechanism maps the decoder state and the encoder states to an output vector, which is a weighted sum of the encoder states and is called context vector Bahdanau et al. (2014). Incorporating this context vector at each decoding timestep (attentive decoding) is proven effective in seq2seq models. Formally, the context vector \(c_{t}\) is defined as: \({\bf c}_{t}=\sum_{i=1}^{N}\alpha_{i}^{(t)}{\bf h}_{i}^{(e)}\) where \(\alpha_{i}^{(t)}\) are the attention weights calculated as follows:

\[\small\begin{split}&\alpha_{i}^{(t)}\!=\!\underset{i}{\rm softmax }({\rm score}({\bf h}_{i}^{(e)},{\bf h}_{i-1}^{(d)}))\\ \end{split} \tag{1}\]

where \(\underset{i}{\rm softmax}\) means that the denominator's sum in the softmax function is over \(i\). The \({\rm score}\) function can be defined in bilinear, additive, or multiplicative ways Luong et al. (2015). We use the additive scoring function:

\[{\rm score}({\bf h}_{i}^{(e)},{\bf h}_{i-1}^{(d)})={\bf v}_{a}^{\top}\tanh \big{(}\operatorname*{linear}({\bf h}_{i}^{(e)},{\bf h}_{i-1}^{(d)})\big{)} \tag{2}\]

where \({\bf v}_{a}\) is a weight vector and linear is a linear mapping function. I.e.,

\[\operatorname*{linear}({\bf x}_{1},{\bf x}_{2})={\bf w}_{1}{\bf x}_{1}+{\bf w }_{2}{\bf x}_{2}+{\bf b} \tag{3}\]

where \({\bf W}_{1}\) and \({\bf W}_{2}\) are weight matrices and \({\bf b}\) is the bias vector.

## 3 Model

We now describe our discourse-aware summarization model (shown in Figure 1).

**Encoder** Our encoder extends the RNN encoder to a hierarchical RNN that captures the document discourse structure. We first encode each discourse section and then encode the document. Formally, we encode the document as a vector \({\bf d}\) according to the following:

\[{\bf d}={\rm RNN}_{doc}\left(\{{\bf h}_{1}^{(s)},...,{\bf h}_{N}^{(s)}\}\right)\]

\({\rm RNN}(.)\) denotes a function which is a recurrent neural network whose output is the final state of the network encoding the entire sequence. \(N\) is the number of sections in the document and \({\bf h}_{j}^{(s)}\) is representation of section \(j\) in the document consisting of a sequence of tokens.

\[{\bf h}_{j}^{(s)}={\rm RNN}_{sec}\left({\bf x}_{(j,1)},...{\bf x}_{(j,M)}\right)\]

where \({\bf x}_{(j,i)}\) are dense embeddings corresponding to the tokens \(w_{(j,i)}\) and \(M\) is the maximum section length. The parameters of \({\rm RNN}_{sec}\) are shared for all the discourse sections. We use a single layer bidirectional LSTM (following the LSTM formulation of Graves et al. (2013)) for both \({\rm RNN}_{doc}\) and \({\rm RNN}_{sec}\); further extension to multilayer LSTMs is straightforward. We combine the forward and backward LSTM states to a single state using a simple feed-forward network:

\[{\bf h}={\rm relu}({\bf W}([\overrightarrow{\bf h},\overleftarrow{\bf h}]+{ \bf b})\]

where \([,]\) shows the concatenation operation. Throughout, when we mention the RNN (LSTM) state, we are referring to this combined state of both forward and backward RNNs (LSTMs).

**Discourse-aware decoder** When humans summarize a long structured document, depending on the domain and the nature of the document, they write about important points from different discourse sections of the document. For example, scientific paper abstracts typically include the description of the problem, discussion of the methods, and finally results and conclusions Suppe (1998). Motivated by this observation, we propose a discourse-aware attention method. Intuitively, at each decoding timestep, in addition to the words

Figure 1: Overview of our model. The word-level RNN is shown in blue and section-level RNN is shown in green. The decoder also consists of an RNN (orange) and a “predict” network for generating the summary. At each decoding time step \(t\) (here \(t\)=3 is shown), the decoder forms a context vector \(c_{t}\) which encodes the relevant source context (\(c_{0}\) is initialized as a zero vector). Then the section and word attention weights are respectively computed using the green “section attention” and the blue “word attention” blocks. The context vector is used as another input to the decoder RNN and as an input to the ”predict” network which outputs the next word using a joint pointer-generator network.

 

in the document, we also attend to the relevant discourse section (the "section attention" block in Figure 1). Then we use the discourse-related information to modify the word-level attention function. Specifically, the context vector representing the source document is:

\[{\bf c}_{t}=\sum\nolimits^{N}_{j=1}\sum\nolimits^{M}_{i=1}\alpha^{(t)}_{(j,i)} \mathbf{h}^{(e)}_{(j,i)} \tag{4}\]

where \(\mathbf{h}^{(e)}_{(j,i)}\) shows the encoder state of word \(i\) in discourse section \(j\) and \(\alpha^{(t)}_{(j,i)}\) shows the corresponding attention weight to that encoder state. The scalar weights \(\alpha^{(t)}_{(j,i)}\) are obtained according to:

\[\alpha^{(t)}_{(j,i)}=\operatorname*{softmax}_{(i,j)}\left(\beta^{(t)}_{j} \operatorname{score}(\mathbf{h}^{(e)}_{(j,i)},\mathbf{h}^{(d)}_{t-1})\right) \tag{5}\]

The \(\operatorname{score}\) function is the additive attention function (Equation 2) and the weights \(\beta^{(t)}_{j}\) are updated according to:

\[\beta^{(t)}_{j}=\operatorname*{softmax}_{j}(\operatorname{score}(\mathbf{h}^ {(s)}_{j},\mathbf{h}^{(d)}_{t-1})) \tag{6}\]

At each timestep \(t\), the decoder state \(\mathbf{h}^{(d)}_{t}\) and the context vector \({\bf c}_{t}\) are used to estimate the probability distribution of next word \(y_{t}\):

\[p(y_{t}|y_{1:t-1})=\operatorname*{softmax}\big{(}\mathbf{V}^{\top} \operatorname{linear}\big{(}\mathbf{h}^{(d)}_{t},{\bf c}_{t}\big{)}\big{)} \tag{7}\]

where \(\mathbf{V}\) is a vocabulary weight matrix and \(\operatorname*{softmax}\) is over the entire vocabulary.

**Copying from source** There has been a surge of recent works in sequence learning tasks to address the problem of _unkown_ token prediction by allowing the model to occasionally copy words directly from source instead of generating a new token Gu et al. (2016); See et al. (2017); Paulus et al. (2017); Wiseman et al. (2017). Following these works, we add an additional binary variable \(z_{t}\) to the decoder, indicating generating a word from vocabulary (\(z_{t}\)=\(0\)) or copying a word from the source (\(z_{t}\)=\(1\)). The probability is learnt during training according to the following equation:

\[p(z_{t}\text{$=$}1|y_{1:t-1})=\sigma(\operatorname{linear}(\mathbf{h}^{(d)} _{t},{\bf c}_{t},{\bf x}^{\prime}_{t})) \tag{8}\]

Then the next word \(y_{t}\) is generated according to:

\[p(y_{t}|y_{1:t-1})=\sum_{z}p(y_{t},z_{t}\text{$=$}z|y_{1:t-1});z=\{0,1\}\]

The joint probability is decomposed as:

\[p(y_{t},z_{t}\text{$=$}z)=\begin{cases}p_{c}(y_{t}|y_{1:t-1})~{}p(z_{t}\text{ $=$}z|y_{1:t-1}),&z\text{$=$}1\\ p_{g}(y_{t}|y_{1:t-1})~{}p(z_{t}\text{$=$}z|y_{1:t-1}),&z\text{$=$}0\end{cases}\]

\(p_{g}\) is the probability of generating a word from the vocabulary and is defined according to Equation 7. \(p_{c}\) is the probability of copying a word from the source vector \({\bf x}\) and is defined as the sum of the word's attention weights. Specifically, the probability of copying a word \(x_{\ell}\) is defined as:

\[p_{c}(y_{t}=x_{\ell}|y_{1:t-1})=\sum\nolimits_{(j,i):x_{(j,i)}=x_{\ell}} \alpha^{(t)}_{(j,i)} \tag{9}\]

**Decoder coverage** In long sequences, the neural generation models tend to repeat phrases where the softmax layer predicts the same phrase multiple times over multiple timesteps. To address this issue, following See et al. (2017), we track attention coverage to avoid repeatedly attending to the same steps. This is done with a coverage vector \(\mathbf{cov}^{(t)}\), the sum of attention weight vectors at previous timesteps: \(\operatorname*{cov}^{(t)}_{(j,i)}=\sum\nolimits^{t-1}_{k=0}\alpha^{(k)}_{( j,i)}\)

The coverage implicitly includes information about the attended document discourse sections. We incorporate the decoder coverage as an additional input to the attention function:

\[\alpha^{(t)}_{(j,i)}=\operatorname*{softmax}_{(i,j)}\Big{(}\beta^{(t)}_{j} \operatorname{score}(\mathbf{h}^{(e)}_{(j,i)},\operatorname*{cov}^{(t)}_{(j ,i)},\mathbf{h}^{(d)}_{t-1})\Big{)}\]

## 4 Related work

Neural abstractive summarization models have been studied in the past Rush et al. (2015); Chopra et al. (2016); Nallapati et al. (2016) and later extended by source copying Miao and Blunsom (2016); See et al. (2017), reinforcement learning Paulus et al. (2017), and sentence salience information Li et al. (2017). One model variant of Nallapati et al. (2016) is related to our model in using sentence-level information in attention. However, our model is different as it contains a hierarchical encoder, uses discourse sections in the decoding step, and has a coverage mechanism. Similarly, Ling and Rush (2017) proposed a coarse-to-fine attention model that uses hard attention to find the text chunks of importance and then only attend to words in that chunk. In contrast, we consider all the discourse sections using soft attention. The closest model to ours is that of See et al. (2017) and Paulus et al. (2017) who used a joint pointer-generator network for summarization. However, our model extends theirs by _(i)_ a hierarchical encoder for modeling long documents and _(ii)_ a discourse-aware decoder that captures the information flow from all discourse sections of the document. Finally, in a recent work, Liu et al. (2018) proposed a model based on the transformer network Vaswani et al. (2017) for abstractive generation of Wikipedia articles. However, their focus 

is on multi-document summarization.

Our datasets are obtained from scientific papers. Scientific document summarization has been recently received extended attention [20, 17, 34]. In contrast to ours, existing approaches are extractive and rely on external information such as citations, which may not be available for all papers.

## 5 Data

Seq2seq models typically have a large number of parameters and thus they require large training data with ground truth summaries. Researchers have constructed such training data from news articles (e.g., CNN, Daily Mail and New York Times articles), where the abstracts or highlights of news articles are considered as ground truth summaries [26, 8]. However, news articles are relatively short and not suitable for the task of long-from document summarization. Following these works, we take scientific papers as an example of long documents with discourse information, where their abstracts can be used as ground-truth summaries. We introduce two datasets collected from scientific repositories, arXiv.org and PubMed.com.

The choice of scientific papers for our dataset is motivated by the fact that scientific papers are examples of long documents that follow a standard discourse structure and they already come with ground truth summaries, making it possible to train supervised neural models. We follow existing work in constructing large-scale summarization datasets that take news article abstracts as ground truth.

We remove the documents that are excessively long (e.g., theses) or too short (e.g., tutorial announcements), or do not have an abstract or discourse structure. We use the level-1 section headings as the discourse information. For arXiv, we use the LaTeX files and convert them to plain text using Pandoc ([https://pandoc.org](https://pandoc.org)) to preserve the discourse section information. We remove figures and tables using regular expressions to only reserve the textual information. We also normalize math formulas and citation markers with special tokens. We analyze the document section names and identify the most common concluding sections names (e.g. _conclusion_, _concluding remarks_, _summary_, etc). We only keep the sections up to the conclusion section of the document and we remove sections after the conclusion.

The statistics of our datasets are shown in Table 1. In our datasets, both document and summary lengths are significantly larger than the existing large-scale summarization datasets. We retain about 3% (5%) of PubMed (ArXiv) as validation data and about another 3% (5%) for test; the rest is used for training.

## 6 Experiments

SetupSimilar to the majority of published research in the summarization literature [9, 26, 27], evaluation was done using the Rouge automatic summarization evaluation metric [10] with full-length F-1 Rouge scores. We lowercase all tokens and perform sentence and word tokenization using spaCy [14].

Implementation detailsWe use Tensorflow 1.4 for implementing our models. We use the hyperparameters suggested by See et al. [27]. In particular, we use two bidirectional LSTMs with cell size of 256 and embedding dimensions of 128. Embeddings are trained from scratch and we did not find any gain using pre-trained embeddings. The vocabulary size is constrained to 50,000; using larger vocabulary size did not result in any improvement. We use mini-batches of size 16 and we limit the document length to 2000 and section length to 500 tokens, and number of sections to 4. We use batch-padding and dynamic unrolling to handle variable sequence lengths in LSTMs. Training was done using Adagrad optimizer with learning rate 0.15 and an initial accumulator value of 0.1. The maximum decoder size was 210 tokens which is in line with average abstract length in our datasets. We first train the model without coverage and added it at the last two epochs to help the model converge faster. We train the models on NVIDIA Titan X Pascal GPUs. Training is performed for about 10 epochs and each training step takes about 3.2 seconds. We used beam search at

\begin{table}
\begin{tabular}{l r r r} \hline \hline Datasets & \# docs & avg. doc. & avg. summary length (words) \\ \hline CNN & 92K & 656 & 43 \\ Daily Mail & 219K & 693 & 52 \\ NY Times & 655K & 530 & 38 \\ PubMed (this work) & 133K & 3016 & 203 \\ arXiv (this work) & 215K & 4938 & 220 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of our arXiv and PubMed datasets compared with existing large-scale summarization corpora, CNN and Daily Mail [26] and NY Times [8].

 

decoding time with beam size of 4. We train the abstractive baselines for about 250K iterations as suggested by their authors.

**Comparison** We compare our method with several well-known extractive baselines as well as state-of-the-art abstractive models using their open-sourced implementations, when available; we follow the same training setup described in the corresponding papers. The compared methods are: _LexRank_[1], _SumBasic_[37], _LSA_[69], _Attn-Seq2Seq_[80, 10], _Pntr-Gen-Seq2Seq_[60]. The first three are extractive models and last two are abstractive. _Pntr-Gen-Seq2Seq_ extends _Attn-Seq2Seq_ by using a joint pointer network during decoding. For _Pntr-Gen-Seq2Seq_ we use their reported hyperparameters to ensure that the result differences are not due to hyperparameter tuning.

**Results** Our main results are shown in Tables 2 and 3. Our model significantly outperforms the state-of-the-art abstractive methods, showing its effectiveness on both datasets. We observe that in our Rouge-1 score is respectively about 4 and 3 points higher than the abstractive model _Ptr-Gen-Seq2Seq_ for the arXiv and PubMed datasets, providing a significant improvement. Our method also outperforms most of the extractive methods except for _LexRank_ in one of the Rouge scores. We note that since extractive methods copy salient sentences from the document, it is usually easier for them to achieve higher Rouge scores.

Figure 2 illustrates the effectiveness of our model extensions in capturing various discourse information from the papers. It can be observed that the state-of-the-art _Pntr-Gen-Seq2Seq_ model generates a summary that mostly focuses on introducing the problem, whereas our model generates a summary that includes more information about the methodology and impacts of the target paper. This indicates that the context vector in our model compared with _Pntr-Gen-Seq2Seq_ is better able to capture important information from the source by attending to various discourse sections.

## 7 Conclusions and future work

This work was the first attempt at addressing neural abstractive summarization of single, long documents. We presented a neural sequence-to-sequence model that is able to effectively summarize long and structured documents such as scientific papers. While our results are encouraging, there is still much room for improvement for this challenging task; our new datasets can help the community to further explore this problem.

We note that following the convention in the summarization research, our quantitative evaluation is performed by Rouge automatic metric. While Rouge is an effective evaluation framework, nuances in the coherence or coverage of the summaries are not captured with it. It is non-trivial to evaluate such qualities especially for long document summarization; future work can design expert human evaluations to explore these nuances.

## Acknowledgements

We thank the three anonymous reviewers for their comments and suggestions.

\begin{table}
\begin{tabular}{l l r r r r} \hline \multicolumn{2}{c}{Summarizer} & RG-1 & RG-2 & RG-3 & RG-L \\ \hline \multirow{4}{*}{\(\text{\textsuperscript{word}}^{\dagger}\)} & SumBasic & 29.47 & 6.95 & 2.36 & 26.30 \\  & LexRank & 33.85 & 10.73 & **4.54** & 28.99 \\  & LSA & 29.91 & 7.42 & 3.12 & 25.67 \\ \cline{2-6}  & Attn-Seq2Seq & 29.30 & 6.00 & 1.77 & 25.56 \\  & Pntr-Gen-Seq2Seq & 32.06 & 9.04 & 2.15 & 25.16 \\  & This work & \({}^{\dagger\ddagger}\)**35.80** & \({}^{\dagger\ddagger}\)**11.05** & \({}^{\dagger\ddagger}\)3.62 & \({}^{\dagger\ddagger}\)**31.80** \\ \hline \end{tabular}
\end{table}
Table 2: Results on the arXiv dataset, RG: Rouge. For our method \({}^{\dagger}\) (’) shows statistically significant improvement with \(p\)<0.05 over other abstractive methods (all other methods).

\begin{table}
\begin{tabular}{l l c c c} \hline \multicolumn{2}{c}{Summarizer} & RG-1 & RG-2 & RG-3 & RG-L \\ \hline \multirow{4}{*}{\(\text{\textsuperscript{word}}^{\dagger}\)} & SumBasic & 37.15 & 11.36 & 5.42 & 33.43 \\  & LexRank & **39.19** & 13.89 & 7.27 & 34.59 \\  & LSA & 33.89 & 9.93 & 5.04 & 29.70 \\ \cline{2-5}  & Attn-Seq2Seq & 31.55 & 8.52 & 7.05 & 27.38 \\  & Pntr-Gen-Seq2Seq & 35.86 & 10.22 & 7.60 & 29.69 \\  & This work & \({}^{\dagger\ddagger}\)**38.93** & \({}^{\dagger\ddagger}\)**15.37** & \({}^{\dagger\ddagger}\)**9.97** & \({}^{\dagger\ddagger}\)**35.21** \\ \hline \end{tabular}
\end{table}
Table 3: Results on PubMed dataset, RG:Rouge. For our method, ‘t’ shows statistically significant improvement with \(p\)<0.05 over abstractive methods (all other methods).

Figure 2: Example of a generated summary





## References

* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _arXiv preprint arXiv:1409.0473_ .
* Chopra et al. (2016) Sumit Chopra, Michael Auli, Alexander M Rush, and SEAS Harvard. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In _HLT-NAACL_. pages 93-98.
* Cohan and Goharian (2015) Arman Cohan and Nazli Goharian. 2015. Scientific article summarization using citation-context and article's discourse structure. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Lisbon, Portugal, pages 390-400. [http://aclweb.org/anthology/D15-1045](http://aclweb.org/anthology/D15-1045).
* Cohan and Goharian (2017a) Arman Cohan and Nazil Goharian. 2017a. Contextualizing citations for scientific summarization using word embeddings and domain knowledge. _arXiv preprint arXiv:1705.08063_ .
* Cohan and Goharian (2017b) Arman Cohan and Nazli Goharian. 2017b. Scientific document summarization via citation contextualization and scientific discourse. _International Journal on Digital Libraries_ pages 1-17. [https://doi.org/10.1007/s00799-017-0216-8](https://doi.org/10.1007/s00799-017-0216-8).
* Erkan and Radev (2004) Gunes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. _Journal of Artificial Intelligence Research_ 22:457-479.
* Graves et al. (2013) Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In _Acoustics, speech and signal processing (icassp), 2013 ieee international conference on_. IEEE, pages 6645-6649.
* Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, pages 1631-1640. [http://www.aclweb.org/anthology/P16-1154](http://www.aclweb.org/anthology/P16-1154).
* Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In _Advances in Neural Information Processing Systems_. pages 1693-1701.
* Honnibal and Johnson (2015) Matthew Honnibal and Mark Johnson. 2015. An improved non-monotonic transition system for dependency parsing. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Lisbon, Portugal, pages 1373-1378. [https://aclweb.org/anthology/D/D15/D15-1162](https://aclweb.org/anthology/D/D15/D15-1162).
* Jing (2002) Hongyan Jing. 2002. Using hidden markov modeling to decompose human-written summaries. _Computational linguistics_ 28(4):527-543.
* Li et al. (2017) Piji Li, Wai Lam, Lidong Bing, Weiwei Guo, and Hang Li. 2017. Cascaded attention based unsupervised information distillation for compressive summarization. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. pages 2071-2080.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _Text summarization branches out: Proceedings of the ACL-04 workshop_. Barcelona, Spain, volume 8.
* Ling and Rush (2017) Jeffrey Ling and Alexander Rush. 2017. Coarse-to-fine attention models for document summarization. In _Proceedings of the Workshop on New Frontiers in Summarization_. Association for Computational Linguistics, Copenhagen, Denmark, pages 33-42. [http://www.aclweb.org/anthology/W17-4505](http://www.aclweb.org/anthology/W17-4505).
* Liu et al. (2018) Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In _International Conference on Learning Representations_. [https://openreview.net/forum?id=Hyg0VvbWC-](https://openreview.net/forum?id=Hyg0VvbWC-).
* Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. _arXiv preprint arXiv:1508.04025_ .
* Miao and Blunsom (2016) Yishu Miao and Phil Blunsom. 2016. Language as a latent variable: Discrete generative models for sentence compression. _arXiv preprint arXiv:1609.07317_ .
* Nallapati et al. (2017) Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarnner: A recurrent neural network based sequence model for extractive summarization of documents. _AAAI_ 1:1.
* Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. _arXiv preprint arXiv:1602.06023_ .
* Paulus et al. (2017) Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. _arXiv preprint arXiv:1705.04304_ .
* Qazvinian et al. (2013) Vahed Qazvinian, Dragomir R Radev, Saif M Mohammad, Bonnie Dorr, David Zajic, Michael Whidby, and Taesun Moon. 2013. Generating extractive summaries of scientific paradigms. _Journal of Artificial Intelligence Research_ 46:165-201.
* Rush et al. (2015) Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. _arXiv preprint arXiv:1509.00685_ .



* See et al. (2017) Abigail See, Christopher Manning, and Peter Liu. 2017. Get to the point: Summarization with pointer-generator networks. In _Association for Computational Linguistics_. [https://arxiv.org/abs/1704.04368](https://arxiv.org/abs/1704.04368).
* Shao et al. (2017) Yuanlong Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. 2017. Generating high-quality and informative conversation responses with sequence-to-sequence models. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. pages 2210-2219.
* Steinberger and Jezek (2004) Josef Steinberger and Karel Jezek. 2004. Using latent semantic analysis in text summarization and summary evaluation. In _Proc. ISIM'04_. pages 93-100.
* Suppe (1998) Frederick Suppe. 1998. The structure of a scientific paper. _Philosophy of Science_ 65(3):381-405.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in neural information processing systems_. pages 3104-3112.
* Vanderwende et al. (2007) Lucy Vanderwende, Hisami Suzuki, Chris Brockett, and Ani Nenkova. 2007. Beyond sumbasic: Task-focused summarization with sentence simplification and lexical expansion. _Information Processing & Management_ 43(6):1606-1618.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_, Curran Associates, Inc., pages 6000-6010. [http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf](http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf).
* Wiseman et al. (2017) Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. Challenges in data-to-document generation. _arXiv preprint arXiv:1707.08052_ .