Venue: NAACL 2019

SEQ\({}^{3}\): Differentiable Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression

Christos Baziotis\({}^{1,2}\), Ion Androutsopoulos\({}^{2}\), Ioannis Konstas\({}^{3}\), Alexandros Potamianos\({}^{1}\)

\({}^{1}\) School of ECE, National Technical University of Athens, Athens, Greece

\({}^{2}\) Department of Informatics, Athens University of Economics and Business, Athens, Greece

\({}^{3}\) Interaction Lab, School of Math. and Comp. Sciences, Heriot-Watt University, Edinburgh, UK

cbaziotis@mail.ntua.gr, ion@aueb.gr

i.konstas@hw.ac.uk, potam@central.ntua.gr

###### Abstract

Neural sequence-to-sequence models are currently the dominant approach in several natural language processing tasks, but require large parallel corpora. We present a sequence-to-sequence-to-sequence autoencoder (seq\({}^{3}\)), consisting of two chained encoder-decoder pairs, with words used as a sequence of discrete latent variables. We apply the proposed model to unsupervised abstractive sentence compression, where the first and last sequences are the input and reconstructed sentences, respectively, while the middle sequence is the compressed sentence. Constraining the length of the latent word sequences forces the model to distill important information from the input. A pretrained language model, acting as a prior over the latent sequences, encourages the compressed sentences to be human-readable. Continuous relaxations enable us to sample from categorical distributions, allowing gradient-based optimization, unlike alternatives that rely on reinforcement learning. The proposed model does not require parallel text-summary pairs, achieving promising results in unsupervised sentence compression on benchmark datasets.

## 1 Introduction

Neural sequence-to-sequence models (seq2seq) perform impressively well in several natural language processing tasks, such as machine translation Sutskever et al. (2014); Bahdanau et al. (2015) or syntactic constituency parsing Vinyals et al. (2015). However, they require massive parallel training datasets Koehn and Knowles (2017). Consequently there has been extensive work on utilizing non-parallel corpora to boost the performance of seq2seq models Sennrich et al. (2016); Gulcehre et al. (2015), mostly in neural machine translation where models that require absolutely no parallel corpora have also been proposed Artetxe et al. (2018); Lample et al. (2018).

Unsupervised (or semi-supervised) seq2seq models have also been proposed for summarization tasks with no (or small) parallel text-summary sets, including unsupervised sentence compression. Current models, however, barely reach lead-_N_ baselines Fevry and Phang (2018); Wang and Lee (2018), and/or are non-differentiable Wang and Lee (2018); Miao and Blunsom (2016), thus relying on reinforcement learning, which is unstable and inefficient. By contrast, we propose a sequence-to-sequence-to-sequence autoencoder, dubbed seq\({}^{3}\), that can be trained end-to-end via gradient-based optimization. seq\({}^{3}\) employs differentiable approximations for sampling from categorical distributions Maddison et al. (2017); Jang et al. (2017), which have been shown to outperform reinforcement learning Havrylov and Titov (2017). Therefore it is a generic framework which can be easily extended to other tasks, e.g., machine translation and semantic parsing via task-specific losses. In this work, as a first step, we apply seq\({}^{3}\) to unsupervised abstractive sentence compression. seq\({}^{3}\) (SS2) comprises two attentional encoder-decoder Bahdanau et al. (2015) pairs (Fig. 1): a compressor \(C\) and a reconstructor \(R\). \(C\) (SS2.1) receives an input text \(\mathbf{x}=\langle x_{1},\dots,x_{N}\rangle\) of \(N\) words, and generates a summary **y**\(=\langle y_{1},\dots,y_{M}\rangle\) of \(M\) words (_M_<_N_), **y** being a latent variable. \(R\) and \(C\) communicate only through the discrete words of the summary **y** (SS2.2). \(R\) (SS2.3) produces a sequence \(\hat{\mathbf{x}}=\langle\hat{x}_{1},\dots,\hat{x}_{N}\rangle\) of \(N\) words from **y**, try

Figure 1: Overview of the proposed seq\({}^{3}\) autoencoder.

 

ing to minimize a reconstruction loss \(L_{R}=(\mathbf{x},\hat{\mathbf{x}})\) (SS2.5). A pretrained language model acts as a prior on **y**, introducing an additional loss \(L_{P}(\textbf{x},\textbf{y})\) that encourages seq3 to produce human-readable summaries. A third loss \(L_{T}(\textbf{x},\textbf{y})\) rewards summaries **y** with similar topic-indicating words as **x**. Experiments (SS3) on the Gigaword sentence compression dataset Rush et al. (2015) and the duc-2003 and duc-2004 shared tasks Over et al. (2007) produce promising results.

Our contributions are: (1) a fully differentiable sequence-to-sequence-to-sequence (seq3) autoencoder that can be trained without parallel data via gradient optimization; (2) an application of seq3 to unsupervised abstractive sentence compression, with additional task-specific loss functions; (3) state of the art performance in unsupervised abstractive sentence compression. This work is a step towards exploring the potential of seq3 in other tasks, such as machine translation.

## 2 Proposed Model

### Compressor

The bottom left part of Fig. 2 illustrates the internals of the compressor \(C\). An embedding layer projects the source sequence **x** to the word embeddings \(\textbf{e}^{s}=\langle e_{1}^{s},\ldots,e_{N}^{s}\rangle\), which are then encoded by a bidirectional rnn, producing \(\textbf{h}^{s}=\langle h_{1}^{s},\ldots,h_{N}^{s}\rangle\). Each \(h_{t}^{s}\) is the concatenation of the corresponding left-to-right and right-to-left states (outputs in lstms) of the bi-rnn.

To generate the summary **y**, we employ the attentional rnn decoder of Luong et al. (2015), with their global attention and input feeding. Concretely, at each timestep (\(t\in\{1,\dots,M\}\)) we compute a probability distribution \(a_{i}\) over all the states \(h_{1}^{s},\ldots,h_{N}^{s}\) of the source encoder conditioned on the current state \(h_{t}^{c}\) of the compressor's decoder to produce a context vector \(c_{t}\).

\[a_{i}=\text{softmax}(h_{i}^{s\intercal}W_{a}\,h_{t}^{c}),~{}~{}c_{t}=\sum_{i=1 }^{N}a_{i}\,h_{i}^{s}\]

The matrix \(W_{a}\) is learned. We obtain a probability distribution for \(y_{t}\) over the vocabulary \(V\) by combining \(c_{t}\) and the current state \(h_{t}^{c}\) of the decoder.

\[o_{t}^{c} =\text{tanh}(W_{o}\,[c_{t};h_{t}^{c}]+b_{o}) \tag{1}\] \[u_{t}^{c} =W_{v}\,o_{t}^{c}+b_{v}\] (2) \[p(y_{t}|y_{<t},\textbf{x}) =\text{softmax}(u_{t}^{c}) \tag{3}\]

\(W_{o},b_{o},W_{v},b_{v}\) are learned. \(c_{t}\) is also used when updating the state \(h_{t}^{c}\) of the decoder, along with the embedding \(e_{t}^{c}\) of \(y_{t}\) and a countdown argument \(M-t\) (scaled by a learnable \(w_{d}\)) indicating the number of the remaining words of the summary Fevry and Phang (2018); Kikuchi et al. (2016).

\[h_{t+1}^{c}=\overrightarrow{\textsc{RNN}}_{c}(h_{t}^{c},e_{t}^{c},c_{t},w_{ d}\,(M-t)) \tag{4}\]

For each input \(\textbf{x}=\langle x_{1},\ldots,x_{N}\rangle\), we obtain a target length \(M\) for the summary \(\textbf{y}=\langle y_{1},\ldots,y_{M}\rangle\) by sampling (and rounding) from a uniform distribution \(U(\alpha N,\beta N)\); \(\alpha,\beta\) are hyper-parameters (\(\alpha<\beta<1\)); we set \(M\) = 5, if the sampled \(M\) is smaller. Sampling \(M\), instead of using a static compression ratio, allows us to train a model capable of producing summaries with varying (e.g., user-specified) compression ratios. Controlling the output length in encoder-decoder architectures has been explored in machine translation Kikuchi et al. (2016) and summarization Fan et al. (2018).

### Differentiable Word Sampling

To generate the summary, we need to sample its words \(y_{t}\) from the categorical distributions \(p(y_{t}|y_{<t},\textbf{x})\), which is a non-differentiable process.

Figure 2: More detailed illustration of seq3. The compressor (\(C\)) produces a summary from the input text, and the reconstructor (\(R\)) tries to reproduce the input from the summary. \(R\) and \(C\) comprise an attentional encoder-decoder each, and communicate only through the (discrete) words of the summary. The lm prior incentivizes \(C\) to produce human-readable summaries, while topic loss rewards summaries with similar topical-indicating words as the input text.

 



**Soft-Argmax** Instead of sampling \(y_{t}\), a simple workaround during training is to pass as input to the next timestep of \(C\)'s decoder and to the corresponding timestep of \(R\)'s encoder a weighted sum of all the vocabulary's (\(V\)) word embeddings, using a peaked softmax function Goyal et al. (2017):

\[\overline{e}^{c}_{t}=\sum_{i}^{|V|}\ e(w_{i})\,\text{softmax}(u^{c}_{t}/\tau) \tag{5}\]

where \(u^{c}_{t}\) is the unnormalized score in Eq. 2 (i.e., the logit) of each word \(w_{i}\) and \(\tau\in(0,\infty)\) is the temperature. As \(\tau\to 0\) most of the probability mass in Eq. 5 goes to the most probable word, hence the operation approaches the \(\arg\max\).

**Gumbel-Softmax** We still want to be able to perform sampling, though, as it has the benefit of adding stochasticity and facilitating exploration of the parameter space. Hence, we use the Gumbel-Softmax (gs) reparametrization trick Maddison et al. (2017); Jang et al. (2017) as a low variance approximation of sampling from categorical distributions. Sampling a specific word \(y_{t}\) from the softmax (Eq. 3) is equivalent to adding (element-wise) to the logits an independent noise sample \(\xi\) from the Gumbel distribution1 and taking the \(\arg\max\):

Footnote 1: \(\xi_{i}=-\log(-\log(x_{i})),~{}~{}x_{i}\sim U(0,1)\)

\[y_{t}\sim\text{softmax}(u^{c}_{t})\leftrightarrow y_{t}=\arg\max(u^{c}_{t}+\xi) \tag{6}\]

Therefore, using the gs trick, Eq. 5 becomes:

\[\tilde{e}^{c}_{t}=\sum_{i}^{|V|}\ e(w_{i})\,\text{softmax}((u^{c}_{t}+\xi)/\tau) \tag{7}\]

**Straight-Through** Both relaxations lead to mixtures of embeddings, which do not correspond to actual words. Even though this enables the compressor to communicate with the reconstructor using continuous values, thus fully utilizing the available embedding space, ultimately our aim is to constrain them to communicate using _only_ natural language. In addition, an unwanted discrepancy is created between training (continuous embeddings) and test time (discrete embeddings). We alleviate these problems with the Straight-Through estimator (st) Bengio et al. (2013). Specifically, in the forward pass of training we discretize \(\tilde{e}^{c}_{t}\) by using the \(\arg\max\) (Eq. 6), whereas in the backward pass we compute the gradients using the gs (Eq. 7). This is a biased estimator due to the mismatch between the forward and backward passes, but works well in practice. st gs reportedly outperforms scheduled sampling Goyal et al. (2017) and converges faster than reinforcement learning Havrylov and Titov (2017).

### Reconstructor

The reconstructor (upper right of Fig. 2) works like the compressor, but its encoder operates on the embeddings \(e^{c}_{1},\dots,e^{c}_{M}\) of the words \(y_{1},\dots,y_{M}\) of the summary (exact embeddings of the sampled words \(y_{t}\) in the forward pass, approximate differentiable embeddings in the backward pass).

### Decoder Initialization

We initialize the hidden state of each decoder using a transformation of the concatenation \([\overrightarrow{h^{s}_{N}};\overleftarrow{h^{s}_{1}}]\) of the last hidden states (from the two directions) of its bidirectional encoder and a length vector, following Mallinson et al. (2018). The length vector for the decoder of the compressor \(C\) consists of the target summary length \(M\), scaled by a learnable parameter \(w_{v}\), and the compression ratio \(\frac{M}{N}\).

\[h^{c}_{0}=\text{tanh}(W_{c}\,[\overrightarrow{h^{s}_{N}};\overleftarrow{ h^{s}_{1}};w_{v}M;\frac{M}{N}])\]

\(W_{c}\) is a trainable hidden layer. The decoder of the reconstructor \(R\) is initialized similarly.

### Loss Functions

**Reconstruction Loss**\(L_{R}(\mathbf{x},\hat{\mathbf{x}})\) is the (negative) log-likelihood assigned by the (decoder of) \(R\) to the input (correctly reconstructed) words \(\mathbf{x}=\langle x_{1},\dots,x_{N}\rangle\), where \(p_{R}\) is the distribution of \(R\).

\[L_{R}(\mathbf{x},\hat{\mathbf{x}})=-\sum_{i=1}^{N}\log p_{R}(\hat{x}_{i}=x_{ i})\]

We do not expect \(L_{R}(\mathbf{x},\hat{\mathbf{x}})\) to decrease to zero, as there is information loss through the compression. However, we expect it to drive the compressor to produce such sentences that will increase the likelihood of the target words in the reconstruction.

**LM Prior Loss** To ensure that the summaries \(\mathbf{y}\) are readable, we pretrain an rnn language model (see Appendix) on the _source_ texts of the full training set. We compute the Kullback-Leibler divergence \(D_{\text{KL}}\) between the probability distributions of the (decoder of) the compressor (\(p(y_{t}|y_{<t},\mathbf{x})\), Eq. 3) and the language model (\(p_{\text{LM}}(y_{t}|y_{<t},\mathbf{x})\)). Similar priors have been used in sentence compression Miao and Blunsom (2016) and agent communication Havrylov and Titov (2017).



We also use the following task-specific losses.

**Topic Loss Words with high tf-idf scores are indicative of the topic of a text Ramos et al. (2003); Erkan and Radev (2004). To encourage the compressor to preserve in the summary y the topic-indicating words of the input x, we compute the tf-idf-weighted average \(v^{x}\) of the word embeddings of x and the average \(v^{y}\) of the word embeddings of y and use their cosine distance as an additional loss \(L_{T}=1-\cos(v^{x},v^{y})\).**

\[v^{x}=\sum_{i=1}^{N}\frac{\textsc{IDF}(x_{i})\,e_{i}^{s}}{\sum_{t=1}^{N} \textsc{IDF}(x_{t})}~{}~{}~{}v^{y}=\frac{1}{M}\sum_{i=1}^{M}e_{i}^{c}\]

**(Using Tf-idf in \(v^{y}\) did not help.) All idf scores are computed on the training set.**

**Length Penalty A fourth loss \(L_{L}\) (not shown in Fig. 1) helps the (decoder of the) compressor to predict the end-of-sequence (eos) token at the target summary length \(M\). \(L_{L}\) is the cross-entropy between the distributions \(p(y_{t}|y_{<t},\textbf{x})\) (Eq. 3) of the compressor at \(t=\textit{M}+1\) and onward, with the one-hot distribution of the eos token.**

### Modeling Details

**Parameter Sharing We tie the weights of layers encoding similar information, to reduce the number of trainable parameters. First, we use a shared embedding layer for the encoders and decoders, initialized with 100-dimensional GloVe embeddings Pennington et al. (2014). Additionally, we tie the shared embedding layer with the output layers of both decoders Press and Wolf (2017); Inan et al. (2017). Finally, we tie the encoders of the compressor and reconstructor (see Appendix). OOVs Out-of-vocabulary words are handled as in Fevry and Phang (2018) (see Appendix).**

## 3 Experiments

**Datasets We train seq\({}^{3}\) on the Gigaword sentence compression dataset Rush et al. (2015).2 It consists of pairs, each containing the first sentence of a news article (**x**) and the article's headline (**y**), a total of 3.8M/189k/1951 train/dev/test pairs. We also test (without retraining) seq\({}^{3}\) on duc-2003 and duc-2004 shared tasks Over et al. (2007), containing 624/500 news articles each, paired with 4 reference summaries capped at 75 bytes. Methods compared We evaluated seq\({}^{3}\) and an ablated version of seq\({}^{3}\). We only used the article sentences (sources) of the training pairs from Gigaword to train seq\({}^{3}\); our model is never exposed to target headlines (summaries) during training or evaluation, i.e., it is completely unsupervised. Our code is publicly available.3

Footnote 2: github.com/harvardnlp/sent-summary

We compare seq\({}^{3}\) to other unsupervised sentence compression models. We note that the extractive model of Miao and Blunsom (2016) relies on a pre-trained attention model using at least 500K parallel sentences, which is crucial to mitigate the inefficiency of sampling-based variational inference and reinforce. Therefore it is not comparable, as it is semi-supervised. The results of the extractive model of Fevry and Phang (2018) are also not comparable, as they were obtained on a different, not publicly available test set. We note, however, that they report that their system performs worse than the Lead-8 baseline in rouge-2 and rouge-l on Gigaword. The only directly comparable unsupervised model is the abstractive 'Pretrained Generator' of Wang and Lee (2018). The version of 'Adversarial Reinforce' that Wang and Lee (2018) consider unsupervised is actually weakly supervised, since its discriminator was exposed to the summaries of the same sources the rest of the model was trained on. As baselines, we use lead-8 for Gigaword, which simply selects the first 8 words of the source, and prefix for duc, which includes the first 75 bytes of the source article. We also compare to supervised abstractive sentence compression methods (Tables 1-3). Following previous work, we report the average F1 of rouge-1, rouge-2, rouge-l Lin (2004). We implemented seq\({}^{3}\) with lstms (see Appendix) and durning inference we perform greedy-sampling. Results Table 1 reports the Gigaword results. seq\({}^{3}\) outperforms the unsupervised Pretrained Generator across all metrics by a large margin. It also surpasses lead-8. If we remove the lm prior, performance drops, esp. in rouge-2 and rouge-l. This makes sense, since the pretrained lm rewards correct word order. We also tried removing the topic loss, but the model failed to converge and results were extremely poor (Table 1). Topic loss acts as a bootstrap mechanism, biasing the compressor to generate words that maintain the topic of the input text. This greatly reduces variance due to sampling in early stages of training, alleviating the need to pretrain individual 

components, unlike works that rely on reinforcement learning (Miao and Blunsom, 2016; Wang and Lee, 2018). Overall, both losses work in synergy, with the topic loss driving _what_ and the lm prior loss driving _how_ words should be included in the summary. seq\({}^{3}\) behaves similarly on duc-2003 and duc-2004 (Tables 2-3), although it was trained on Gigaword. In duc-2003, however, it does not surpass the prefix baseline.

Finally, Fig. 3 illustrates three randomly sampled outputs of seq\({}^{3}\) on Gigaword. In the first one, seq\({}^{3}\) copies several words esp. from the beginning of the input (hence the high rouge-l) exhibiting extractive capabilities, though still being adequately abstractive (bold words denote paraphrases). In the second one, seq\({}^{3}\) showcases its true abstractive power by paraphrasing and compressing multi-word expressions to single content words more heavily, still without losing the overall meaning. In the last example, seq\({}^{3}\) progressively becomes ungrammatical though interestingly retaining some content words from the input.

## 4 Limitations and Future Work

The model tends to copy the first words of the input sentence in the compressed text (Fig. 3). We hypothesize that since the reconstructor is autoregressive, i.e., each word is conditioned on the previous one, errors occurring early in the generated sequence have cascading effects. This inevitably encourages the compressor to select the first words of the input. A possible workaround might be to modify seq\({}^{3}\) so that the first encoder-decoder pair would turn the inputs to longer sequences, and the second encoder-decoder would compress them trying to reconstruct the original inputs. In future work, we plan to explore the potential of seq\({}^{3}\) in other tasks, such as unsupervised machine translation (Lample et al., 2018a; Artetxe et al., 2018) and caption generation (Xu et al., 2015).

## Acknowledgments

We would like to thank Ryan McDonald for helpful discussions and feedback. This work has been partially supported by computational time granted from the Greek Research & Technology Network (GR-NET) in the National HPC facility - ARIS. We thank NVIDIA for donating a TitanX GPU.

\begin{table}
\begin{tabular}{p{79.7pt} p{34.1pt} p{34.1pt} p{34.1pt} p{34.1pt}} \hline
**Type** & **Supervision** & **Methods** & & \\ \hline  & & ABS (Rush et al., 2015) & 29.55 & 11.32 & 26.42 \\ Supervised & 3.8M & SESAS (Zhou et al., 2017) & 36.15 & 17.54 & 33.63 \\  & & words-lvt5K-I sent (Nallapati et al., 2016) & **36.4** & **17.7** & **33.71** \\ \hline Weakly supervised & (3.8M) & Adversarial REINFORCE (Wang and Lee, 2018) & 28.11 & 9.97 & 25.41 \\ \multirow{6}{*}{Unsupervised} & \multirow{6}{*}{0} & Lead-8 (Baseline) & 21.86 & 7.66 & 20.45 \\  & & Pretrained Generator (Wang and Lee, 2018) & 21.26 & 5.60 & 18.89 \\  & & seg\({}^{3}\) (Full) & **25.39** & **8.21** & **22.68** \\  & & seq\({}^{3}\) w/o lm prior loss & 24.48 & 6.68 & 21.79 \\  & & seq\({}^{3}\) w/o topic loss & 3.89 & 0.1 & 3.75 \\ \hline \end{tabular}
\end{table}
Table 1: Average results on the (English) Gigaword dataset for abstractive sentence compression methods.

Figure 3: Good/bad example summaries on Gigaword.

 



## References

* Artetxe et al. (2018) Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In _Proceedings of the Annual Meeting of International Conference on Learning Representations_.
* Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _Proceedings of the International Conference on Learning Representations_, San Diego, California.
* Bengio et al. (2013) Yoshua Bengio, Nicholas Leonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. _arXiv preprint arXiv:1308.3432_.
* Bowman et al. (2016) Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In _Proceedings of the SIGNLL Conference on Computational Natural Language Learning_, pages 10-21, Berlin, Germany. Association for Computational Linguistics.
* Chopra et al. (2016) Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In _Proceedings of the Conference of the NAACL-HLT_, pages 93-98, San Diego, California.
* Erkan and Radev (2004) Gunes Erkan and Dragomir R. Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. _Journal of Artificial Intelligence Research_, 22:457-479.
* Fan et al. (2018) Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In _Proceedings of the Workshop on Neural Machine Translation and Generation_, pages 45-54, Melbourne, Australia.
* Fevry and Phang (2018) Thibault Fevry and Jason Phang. 2018. Unsupervised sentence compression using denoising auto-encoders. In _Proceedings of the Conference on Computational Natural Language Learning_, pages 413-422, Brussels, Belgium.
* Goyal et al. (2017) Kartik Goyal, Chris Dyer, and Taylor Berg-Kirkpatrick. 2017. Differentiable scheduled sampling for credit assignment. In _Proceedings of the Annual Meeting of the ACL_, pages 366-371, Vancouver, Canada.
* Gu et al. (2018) Jiatao Gu, Daniel Jiwoong Im, and Victor O. K. Li. 2018. Neural machine translation with gumbelbergly decoding. In _Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, February 2-7, 2018_, pages 5125-5132.
* Gulcehre et al. (2017) Caglar Gulcehre, Sarath Chandar, and Yoshua Bengio. 2017. Memory augmented neural networks with wormhole connections. _arXiv preprint arXiv:1701.08718_.
* Gulcehre et al. (2015) Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. _arXiv preprint arXiv:1503.03535_.
* Havrylov and Titov (2017) Serhii Havrylov and Ivan Titov. 2017. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Proceedings of the Advances in Neural Information Processing Systems_, pages 2149-2159.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_, 9(8):1735-1780.
* Inan et al. (2017) Hakan Inan, Khashayar Khosravi, and Richard Socher. 2017. Tying word vectors and word classifiers: A loss framework for language modeling. _CoRR_, abs/1611.01462.
* Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparametrization with gumble-softmax. In _Proceedings of the International Conference on Learning Representations_, Toulon, France.
* Kikuchi et al. (2016) Yuta Kikuchi, Graham Neubig, Royhehei Sasano, Hiroya Takamura, and Manabu Okumura. 2016. Controlling output length in neural encoder-decoders. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 1328-1338, Austin, Texas.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _Proceedings of the International Conference on Learning Representations_.
* Kingma and Welling (2014) Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In _Proceedings of the International Conference on Learning Representations_, Banff, AB, Canada.
* Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. _arXiv preprint arXiv:1706.03872_.
* Lample et al. (2018a) Guillaume Lample, Alexis Conneau, Ludovic Denoyer, Herve Jegou, et al. 2018a. Word translation without parallel data. In _Proceedings of the International Conference on Learning Representations_, Vancouver, Canada.
* Lample et al. (2018b) Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ronzato. 2018b. Unsupervised machine translation using monolingual corpora only. In _Proceedings of the International Conference on Learning Representations_, Vancouver, Canada

* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. _Text Summarization Branches Out_.
* Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 1412-1421, Lisbon, Portugal.
* Maddison et al. (2017) Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The concrete distribution: A continuous relaxation of discrete random variables. In _Proceedings of the International Conference on Learning Representations_, Toulon, France.
* Mallinson et al. (2018) Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2018. Sentence compression for arbitrary languages via multilingual pivoting. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 2453-2464, Brussels, Belgium.
* Miao and Blunsom (2016) Yishu Miao and Phil Blunsom. 2016. Language as a Latent Variable: Discrete Generative Models for Sentence Compression. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 319-328, Austin, Texas.
* Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In _Proceedings of the Conference on Computational Natural Language Learning_, pages 280-290, Berlin, Germany.
* Over et al. (2007) Paul Over, Hoa Dang, and Donna Harman. 2007. Duc in context. _Information Processing and Management_, 43(6):1506-1520.
* Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 1532-1543, Doha, Qatar.
* Press and Wolf (2017) Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In _Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics_, pages 157-163, Valencia, Spain. Association for Computational Linguistics.
* Ramos et al. (2003) Juan Ramos et al. 2003. Using tf-idf to determine word relevance in document queries. In _Proceedings of the 1st International Conference on Machine Learning_, volume 242, pages 133-142.
* Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 379-389, Lisbon, Portugal.
* See et al. (2017)bigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In _Proceedings of the Annual Meeting of the ACL_, pages 1073-1083, Vancouver, Canada.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In _Proceedings of the Annual Meeting of the ACL_, pages 86-96, Berlin, Germany.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Proceedings of the Advances in Neural Information Processing Systems_, pages 3104-3112.
* Vinyals et al. (2015) Oriol Vinyals, L ukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In _Proceedings of the Advances in Neural Information Processing Systems_, pages 2773-2781.
* Wang and Lee (2018) Yasushian Wang and Hung-yi Lee. 2018. Learning to encode text as human-readable summaries using generative adversarial networks. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 4187-4195, Brussels, Belgium.
* Woodsend et al. (2010) Kristian Woodsend, Yansong Feng, and Mirella Lapata. 2010. Title generation with quasi-synchronous grammar. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 513-523, Cambridge, MA.
* Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In _Proceedings of the International Conference on Machine Learning_, volume 37, pages 2048-2057, Lille, France. PMLR.
* Zajic et al. (2007) David Zajic, Bonnie J. Dorr, Jimmy Lin, and Richard Schwartz. 2007. Multi-candidate reduction: Sentence compression as a tool for document summarization tasks. _Information Processing Management Special Issue on Summarization_, 43(6):1549-1570.
* Zhou et al. (2017) Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. 2017. Selective encoding for abstractive sentence summarization. In _Proceedings of the Annual Meeting of the ACL_, pages 1095-1104, Vancouver, Canada.

 

Appendix

### Temperature for Gumbel-Softmax

Even though the value of the temperature \(\tau\) does not affect the forward pass, it greatly affects the gradient computation and therefore the learning process. Jang et al. (2017) propose to anneal \(\tau\) during training towards zero. Gulcehre et al. (2017) propose to learn \(\tau\) as a function of the compressor's decoder state \(h_{t}^{c}\), in order to reduce hyper-parameter tuning:

\[\tau(h_{t}^{c})=\frac{1}{\log(1+\exp(w_{\tau}^{\intercal}h_{t}^{c}))+1} \tag{8}\]

where \(w_{\tau}\) is a trainable parameter and \(\tau(h_{t}^{c})\in(0,1)\). Havrylov and Titov (2017) add \(\tau_{0}\) as a hyper-parameter which controls the upper bound of the temperature.

\[\tau(h_{t}^{c})=\frac{1}{\log(1+\exp(w_{\tau}^{\intercal}h_{t}^{c}))+\tau_{0}} \tag{9}\]

In our experiments, we had convergence problems with the learned temperature technique. We found that the compressor preferred values close to the upper bound, which led to unstable training, forcing us to set \(\tau_{0}>1\) to stabilize the training process. Our findings align with the behavior reported by Gu et al. (2018). Consequently, we follow their choice and fix \(\tau=0.5\), which worked well in practice.

### Out of Vocabulary (OOV) Words

The vocabulary of our experiments comprises the 15k most frequent words of Gigaword's training input texts (without looking at their summaries). To handle oovs, we adopt the approach of Fevry and Phang (2018), which can be thought of as a simpler form of copying compared to pointer networks (See et al., 2017). We use a small set (10 in our experiments) of special oov tokens \(\textsc{oov}_{1}\), \(\textsc{oov}_{2}\), ..., \(\textsc{oov}_{10}\), whose embeddings are updated during learning. Given an input text \(\textbf{x}=\langle x_{1},\dots,x_{N}\rangle\), we replace (before feeding **x** to seq3) each unknown word \(x_{i}\) with the first unused (for the particular **x**) oov token, taking care to use the same oov token for all the occurrences of the same unknown word in **x**. For example, if 'John' and 'Rome' are not in the vocabulary, then "John arrived in Rome yesterday. While in Rome, John had fun." becomes "oov\({}_{1}\) arrived in oov\({}_{2}\) yesterday. While in oov\({}_{2}\), oov\({}_{1}\) had fun." If a new unknown word \(x_{i}\) is encountered in **x** and all the available oov tokens have been used, \(x_{i}\) is replaced by 'unk', whose embedding is also updated during learning. The oov tokens (and 'unk') are included in the vocabulary, and seq3 learns to predict them as summary words, in effect copying the corresponding unknown words of **x**. At test time, we replace the oov tokens with the corresponding unknown words.

Footnote 3: [https://github.com/google-transformers/gumbel-softmax](https://github.com/google-transformers/gumbel-softmax)

### Reconstruction Word Drop

Our model is an instance of Variational AutoEncoders (vae) (Kingma and Welling, 2014). A common problem in vaes is that the reconstructor tends to disregard the latent variable. We weaken the reconstructor \(R\), in order to force it to fully utilize the latent sequence **y** to generate \(\hat{\mathbf{x}}\). To this end, we employ word dropout as in Bowman et al. (2016) and randomly drop a percentage of the input words, thus forcing \(R\) to rely solely on **y** to make good reconstructions.

### Implementation and Hyper-parameters

We implemented seq3 in PyTorch (Paszke et al., 2017). All the rnns are lstms(Hochreiter and Schmidhuber, 1997). We use a shared encoder for the compressor and the reconstructor, consisting of a two-layer bidirectional lstm with size \(300\) per direction. We use separate decoders for the compressor and the reconstructor; each decoder is a two-layer unidirectional lstm with size \(300\). The (shared) embedding layer of the compressor and the reconstructor is initialized with \(100\)-dimensional GloVe embeddings (Pennington et al., 2014) and is tied with the output (projec

Figure 4: Plot of Eq. 9, with different values for the upper bound \(\tau_{0}\).

 

tion) layers of the decoders and jointly finetuned during training. We apply layer normalization (Ba et al., 2016) to the context vectors (Eq. 1) of the compressor and the reconstructor. We apply word dropout (SSA.3) to the reconstructor with \(p=0.5\).

During training, the summary length \(M\) is sampled from \(U(0.4\,N,0.6\,N)\); during testing, \(M=0.5\,N\). The four losses are summed, \(\lambda\)s being scalar hyper-parameters.

\[L=\lambda_{R}\,L_{R}+\lambda_{P}\,L_{P}+\lambda_{T}\,L_{T}+\lambda_{L}L_{L}\]

We set \(\lambda_{R}=\lambda_{T}=1\), \(\lambda_{L}=\lambda_{P}=0.1\). We use the Adam (Kingma and Ba, 2015) optimizer, with batch size 128 and the default learning rate \(0.001\). The network is trained for 5 epochs.

**LM Prior** The pretrained language model is a two-layer lstm of size \(1024\) per layer. It uses its own embedding layer of size \(256\), which is randomly initialized and updated when training the language model. We apply dropout with \(p=0.2\) to the embedding layer and dropout with \(p=0.5\) to the lstm layers. We use Adam (Kingma and Ba, 2015) with batch size 128 and the network is trained for 30 epochs. The learning rate is set initially to \(0.001\) and is multiplied with \(\gamma=0.5\) every 10 epochs.

**Evaluation** Following Chopra et al. (2016), we filter out pairs with empty headlines from the test set. We employ the pyrouge package with "_-m -n 2 -w 1.2_" to compute rouge scores. We use the provided tokenizations of the Gigaword and duc-2003, duc-2004 datasets. All hyper-parameters were tuned on the development set.