Venue: NAACL 2019

# Evaluating Text GANs as Language Models

Guy Tevet\({}^{*,1,2}\) Gavriel Habib\({}^{*,1,2}\) Vered Shwartz\({}^{3}\) Jonathan Berant\({}^{1,4}\)

###### Abstract

Generative Adversarial Networks (GANs) are a promising approach for text generation that, unlike traditional language models (LM), does not suffer from the problem of "exposure bias". However, A major hurdle for understanding the potential of GANs for text generation is the lack of a clear evaluation metric. In this work, we propose to approximate the distribution of text generated by a GAN, which permits evaluating them with traditional probability-based LM metrics. We apply our approximation procedure on several GAN-based models and show that they currently perform substantially worse than state-of-the-art LMs. Our evaluation procedure promotes better understanding of the relation between GANs and LMs, and can accelerate progress in GAN-based text generation.

+
Footnote †: The authors contributed equally

\({}^{1}\)School of Computer Science, Tel-Aviv University

\({}^{2}\)Department of Electrical Engineering, Tel-Aviv University

\({}^{3}\)Computer Science Department, Bar-Ilan University

\({}^{4}\)Allen Institute for Artificial Intelligence

{guytevet@mail,gavrielhabib@mail,joberant@cs}.tau.ac.il

vered1986@gmail.com

## 1 Introduction

Neural networks have revolutionized the field of text generation, in machine translation Sutskever et al. (2014); Neubig (2017); Luong et al. (2015); Chen et al. (2018), summarization See et al. (2017), image captioning You et al. (2016) and many other applications Goldberg (2017).

Traditionally, text generation models are trained by going over a gold sequence of symbols (characters or words) from left-to-right, and maximizing the probability of the next symbol given the history, namely, a language modeling (LM) objective. A commonly discussed drawback of such LM-based text generation is _exposure bias_Ranzato et al. (2015): during training, the model predicts the next token conditioned on the ground truth history, while at test time prediction is based on predicted tokens, causing a train-test mismatch. Models trained in this manner often struggle to overcome previous prediction errors.

Generative Adversarial Networks Goodfellow et al. (2014) offer a solution for _exposure bias_. Originally introduced for images, GANs leverage a discriminator, which is trained to discriminate between real images and generated images via an adversarial loss. In such a framework, the generator is not directly exposed to the ground truth data, but instead learns to imitate it using global feedback from the discriminator. This has led to several attempts to use GANs for text generation, with a generator using either a recurrent neural network (RNN) Yu et al. (2017); Guo et al. (2017); Press et al. (2017); Rajeswar et al. (2017), or a Convolutional Neural Network (CNN) Gulrajani et al. (2017); Rajeswar et al. (2017).

However, evaluating GANs is more difficult than evaluating LMs. While in language modeling, evaluation is based on the log-probability of a model on held-out text, this cannot be straightforwardly extended to GAN-based text generation, because the generator outputs discrete tokens, rather than a probability distribution. Currently, there is no single evaluation metric for GAN-based text generation, and existing metrics that are based on n-gram overlap are known to lack robustness and have low correlation with semantic coherence Semeniuta et al. (2018).

In this paper, we propose a method for evaluating GANs with standard probability-based evaluation metrics. We show that the expected prediction of a GAN generator can be viewed as a LM, and suggest a simple Monte-Carlo method for approximating it. The approximated probability distribution can then be evaluated with standard LM metrics such as perplexity or Bits Per Character (BPC).

To empirically establish our claim, we implement our evaluation on several RNN-based GANs: Press et al. (2017); Yu et al. (2017); Guo et al. (2017). We find that all models have substantially lower BPC compared to state-of-the-art LMs. By directly comparing to LMs, we put in perspective the current performance of RNN-based GANs for

text generation. Our results are also in line with recent concurrent work by Caccia et al. (2018), who reached a similar conclusion by comparing the performance of textual GANs to that of LMs using metrics suggested for GAN evaluation.

Our code is available at: [http://github.com/GuyTevet/SeqGAN-eval](http://github.com/GuyTevet/SeqGAN-eval) and [http://github.com/GuyTevet/rnn-gan-eval](http://github.com/GuyTevet/rnn-gan-eval).

## 2 Background

Following the success of GANs in image generation, several works applied the same idea to texts using convolutional neural networks Gulrajani et al. (2017); Rajeswar et al. (2017), and later using RNNs Press et al. (2017); Yu et al. (2017). RNNs enable generating variable-length sequences, conditioning each token on the tokens generated in previous time steps. We leverage this characteristic in our approximation model (SS4.1).

A main challenge in applying GANs for text is that generating discrete symbols is a non-differentiable operation. One solution is to perform a continuous relaxation of the GAN output, which leads to generators that emit a nearly discrete continuous distribution Press et al. (2017). This keeps the model differentiable and enables end-to-end training through the discriminator. Alternatively, SeqGAN Yu et al. (2017) and LeakGAN Guo et al. (2017) used policy gradient methods to overcome the differentiablity requirement. We apply our approximation to both model types.

## 3 Evaluating GANs and LMs

LM Evaluation.Text generation from LMs is commonly evaluated using probabilistic metrics. Specifically, given a test sequence of symbols \((t_{1},\dots,t_{n})\), and a LM \(q\), the average cross-entropy over the entire test set is computed: \(ACE=-\frac{1}{n}\sum_{i=1}^{n}\log_{2}q(t_{i}\mid t_{1}...t_{i-1})\). For word-based models, the standard metric is perplexity: \(PP=2^{ACE}\), while for character-based models it is \(BPC=ACE\) directly.

Intrinsic improvement in perplexity does not guarantee an improvement in an extrinsic downstream task that uses a language model. However, perplexity often correlates with extrinsic measures Jurafsky and Martin (2018), and is the de-facto metric for evaluating the quality of language models today.

GAN-based Text Generation Evaluation.By definition, a text GAN outputs a discrete sequence of symbols rather than a probability distribution. As a result, LM metrics cannot be applied to evaluate the generated text. Consequently, other metrics have been proposed:

* N-gram overlap: Yu et al. (2017); Press et al. (2017): Inspired by BLEU Papineni et al. (2002), this measures whether n-grams generated by the model appear in a held-out corpus. A major drawback is that this metric favors conservative models that always generate very common text (e.g., _"it is"_). To mitigate this, self-BLEU has been proposed Lu et al. (2018) as an additional metric, where overlap is measured between two independently sampled texts from the model.
* LM score: The probability of generated text according to a pre-trained LM. This has the same problem of favoring conservative models.
* Zhao et al. (2017) suggested an indirect score by training a LM on GAN-generated text, and evaluating it using perplexity. The drawback in this setting is the coupling of the performance of the GAN with that of the proxy LM.
* Heusel et al. (2017) used Frechet InferSent Distance (FID) to compute the distance between distributions of features extracted from real and generated samples. However, this approach relies on a problematic assumption that features are normally distributed.
* Rajeswar et al. (2017) used a context-free grammar (CFG) to generate a reference corpus, and evaluated the model by the likelihood the CFG assigns to generated samples. However, simple CFGs do not fully capture the complexity of natural language.
* To overcome the drawbacks of each individual method, Semeniuta et al. (2018) proposed a unified measure based on multiple evaluation metrics (N-grams, BLEU variations, FID, LM score variations and human evaluation). Specifically, they argue that the different measures capture different desired properties of LMs, e.g., quality vs. diversity.
* Following Semeniuta et al. (2018), and in parallel to this work, Caccia et al. (2018) proposed a temperature sweep method that trades-off quality for diversity using a single parameter. Similar to our findings, they concluded that GANs perform worse than LMs on this metric 

Overall, current evaluation methods cannot fully capture the performance of GAN-based text generation models. While reporting various scores as proposed by Semeniuta et al. (2018) is possible, it is preferable to have a single measure of progress when comparing different text generation models.

## 4 Proposed Method

We propose a method for approximating a distribution over tokens from a GAN, and then evaluate the model with standard LM metrics. We will describe our approach given an RNN-based LM, which is the most commonly-used architecture, but the approximation can be applied to other auto-regressive models Vaswani et al. (2017).

### Language Model Approximation

The inputs to an RNN at time step \(t\), are the state vector \(h_{t}\) and the current input token \(x_{t}\). The output token (one-hot) is denoted by \(o_{t}\). In RNN-based GANs, the previous output token is used at inference time as the input \(x_{t}\)Yu et al. (2017); Guo et al. (2017); Press et al. (2017); Rajeswar et al. (2017). In contrast, when evaluating with BPC or perplexity, the gold token \(x_{t}\) is given as input. Hence, LM-based evaluation neutralizes the problem of exposure bias addressed by GANs. Nevertheless, this allows us to compare the quality of text produced by GANs and LMs on an equal footing. Figure 1 illustrates the difference between inference time and during LM approximation.

```
0:\(G_{t}(\cdot)\): the generator function at time step \(t\)\((x_{0},...,x_{t})\): previous gold tokens \(x_{t+1}\): the gold next token (as ground truth) \(f(\cdot,\cdot)\): a LM evaluation metric \(N\): number of samples
1:for\(n\gets 1\) to \(N\)do
2:\(g_{t,n}\longleftarrow\) sample from \(G_{t}(x_{0}...x_{t})\)
3:\(\tilde{G}_{t,N}=\frac{1}{N}\Sigma^{m}_{n=1}q_{l,n}\)
4:return\(f(\tilde{G}_{t,N},x_{t+1})\)
```

**Algorithm 1** LM Evaluation of RNN-based GANs

We can therefore define the generator function at time step \(t\) as a function of the initial state \(h_{0}\) and the past generated tokens \((x_{0}\ldots x_{t})\), which we denote as \(o_{t}=G_{t}(h_{0},x_{0}...x_{t})\) (\(x_{0}\) is a start token). Given a past sequence \((x_{0}\ldots x_{t})\), \(G_{t}\) is a stochastic function: the stochasticity of \(G_{t}\) can be gained either by using a noise vector as the initial state \(h_{0}\)Press et al. (2017), or by sampling from the GAN's internal distribution over possible output tokens Yu et al. (2017); Guo et al. (2017). Since \(h_{0}\) is constant or a noise vector that makes \(G_{t}\) stochastic, we can omit it to get \(G_{t}(x_{0}\ldots x_{t})\). In such a setup, the expected value \(\mathbb{E}[G_{t}(x_{0}\ldots x_{t})]\) is a distribution \(q\) over the next vocabulary token \(a_{t}\):

To empirically approximate \(q\), we can sample from it \(N\) i.i.d samples, and compute an approximation \(\tilde{G}_{t,N}=\frac{1}{N}\Sigma^{N}_{n=1}g_{t,n}\), where \(g_{t,n}\) is one sample from \(G_{t}(x_{0}...x_{t})\). Then, according to the strong law of large numbers:

\[\mathbb{E}[G_{t}(x_{0}\ldots x_{t})]=\lim_{N\rightarrow\infty}\tilde{G}_{t,N} \tag{1}\]

Given this approximate LM distribution, we can evaluate a GAN using perplexity or BPC. We summarize the evaluation procedure in Algorithm 1.1

Footnote 1: Our evaluation algorithm is linear in the length of the test set and in the number of samples \(N\).

### Approximation Bound

We provide a theoretical bound for choosing a number of samples \(N\) that results in a good approximation of \(\tilde{G}_{t,N}\) to \(\mathbb{E}[G_{t}]\).

Perplexity and BPC rely on the log-probability of the ground truth token. Since the ground truth token is unknown, we conservatively define the bad event \(B\) in which there exists \(v\in V\) such that \(|\{E[G_{t}]\}_{v}-\{\tilde{G}_{t,N}\}_{v}|>\gamma\), where \(V\) is the vocabulary. We can then bound the probability of \(B\) by some \(\epsilon\). We define the following notations:

1. The probability of a token \(a_{t}\) to be \(v\) is \(p_{v}\stackrel{{\Delta}}{{=}}q(a_{t}=v|a_{0}\dots a_{t-1})=\{ \mathbb{E}[G_{t}(x_{0}\ldots x_{t})]\}_{v}\).
2. \(\chi_{v,n}\stackrel{{\Delta}}{{=}}\left\{g_{t,n}\right\}_{v}\) is a random variable representing the binary value of the \(v\)'th index of

Figure 1: Generator recurrent connections. \(\{h_{t}\}\) is the internal state sequence and \(\{o_{t}\}\) is the generator prediction sequence (one-hot). During inference, the outputs \(\{o_{t}\}\) are fed back as the input for the next time step (dashed lines). During LM approximation, the input \(\{x_{t}\}\) is a sequence of one-hot vectors from the test set.

 

\(g_{t,n}\) which is a single sample of \(G_{t}\). Note that the average of \(\chi_{v,n}\) over \(N\) samples is \(X_{v}\stackrel{{\Delta}}{{=}}\frac{1}{N}\sum_{n=1}^{N}\chi_{v,n}= \left\{\frac{1}{N}\sum_{n=1}^{N}g_{t,n}\right\}_{v}=\left\{\tilde{G}_{t,N}\right\} _{v}\).

Using the above notation, we can re-define the probability of the bad event \(B\) with respect to the individual coordinates in the vectors:

\[Pr(B) =Pr\left(|\mathbb{E}[G_{t}]-\tilde{G}_{t,N}||_{\infty}>\gamma\right) \tag{2}\] \[=Pr\left(\bigcup_{v\in V}|p_{v}-X_{v}|>\gamma\right)\overset{!}{<}\epsilon\]

We note that \(\chi_{v,n}\sim Bernoulli(p_{v})\), and given that \(\left\{\chi_{v,n}\right\}_{n=1}^{N}\) are i.i.d., we can apply the Chernoff-Hoeffding theorem Chernoff et al. (1952); Hoeffding (1963). According to the theorem, for every \(v\in V\), \(Pr(|X_{v}-p_{v}|>\gamma)<2e^{-2N\gamma^{2}}\). Taking the union bound over \(V\) implies:

\[Pr(B)=Pr\left(\bigcup_{v\in V}|X_{v}-p_{v}|>\gamma\right)<2|V|e^{-2N\gamma^{2 }}<\epsilon \tag{3}\]

Hence, we get a lower bound on \(N\):

\[N>\frac{1}{2\gamma^{2}}\ln\left(\frac{2|V|}{\epsilon}\right) \tag{4}\]

As a numerical example, choosing \(\gamma=10^{-3}\) and \(\epsilon=10^{-2}\), for a character-based LM over the text8 dataset, with \(|V|=27\), we get the bound: \(N>4.3\cdot 10^{6}\). With the same \(\gamma\) and \(\epsilon\), a typical word-based LM with vocabulary size \(|V|=50,000\) would require \(N>8.1\cdot 10^{6}\).

In practice, probability vectors of LMs tend to be sparse Kim et al. (2016). Thus, we argue that we can use a much smaller \(N\) for a good approximation \(\tilde{G}_{t,N}\). Since the sparsity of LMs is difficult to bound, as it differs between models, we suggest an empirical method for choosing \(N\).

The approximation \(\tilde{G}_{t,N}\) is a converging sequence, particularly over \(\|\cdot\|_{\infty}\) (see Equation 1). Hence, we can empirically choose an \(N\) which satisfies \(\|\tilde{G}_{t,N-\alpha}-\tilde{G}_{t,N}\|_{\infty}<\gamma^{\prime},\quad \alpha\in\mathbb{N}\). In Section 5 we empirically measure \(\|\tilde{G}_{t,N-\alpha}-\tilde{G}_{t,N}\|_{\infty}\) as a function of \(N\) to choose \(N\). We choose a global \(N\) for a model, rather than for every \(t\), by averaging over a subset of the evaluation set.

## 5 Evaluation

### Models

We focus on character-based GANs as a test-case for our method. We evaluate two RNN-based GANs with different characteristics. As opposed to the original GAN model Goodfellow et al. (2014), in which the generator is initialized with random noise, the GANs we evaluated both leverage gold standard text to initialize the generator, as detailed below.

Recurrent GAN Press et al. (2017)is a continuous RNN-based generator which minimizes the improved WGAN loss Gulrajani et al. (2017). To guide the generator, during training it is initialized with the first \(i-1\) characters from the ground truth, starting the prediction in the \(i\)th character. Stochasticity is obtained by feeding the generator with a noise vector \(z\) as a hidden state. At each time step, the input to the RNN generator is the output distribution of the previous step.

SeqGAN Yu et al. (2017)is a discrete RNN-based generator. To guide the generator, it is pre-trained as a LM on ground truth text. Stochasticity is obtained by sampling tokens from an internal distribution function over the vocabulary. To overcome differentiation problem, it is trained using a policy gradient objective Sutton et al. (2000).

We also evaluated **LeakGAN**Guo et al. (2017), another discrete RNN-based generator, but since it is similar to SeqGAN and performed worse, we omit it for brevity.

### Evaluation Settings

To compare to prior work in LM, we follow the common setup and train on the text8 dataset.2 The dataset is derived from Wikipedia, and includes 26 English characters plus spaces. We use the standard 90/5/5 split to train/validation/test. Finally, we measure performance with BPC.

Footnote 2: [http://mattmahoney.net/dc/textdata](http://mattmahoney.net/dc/textdata)

We tuned hyper-parameters on the validation set, including sequence length to generate at test time (7 for Press et al. (2017), 1000 for Yu et al. (2017)). We chose the number of samples \(N\) empirically for each model, as described in Section 4.2. We set \(\alpha\) to 10, and the boundary to \(\gamma^{\prime}=10^{-3}\) as a good trade-off between accuracy and run-time. Figure 2 plots the approximate error \(\|\tilde{G}_{t,N-\alpha}-\tilde{G}_{t,N}\|_{\infty}\) as a function of \(N\). For both models, \(N>1600\) satisfies this condition (red line in Figure 2). To be safe, we used \(N=2000\).

### Results

Table 1 shows model performance on the test set.

 

Because SeqGAN models output a distribution over tokens at every time step, we can measure the true BPC and assess the quality of our approximation. Indeed, we observe that approximate BPC is only slightly higher than the true BPC.

GAN-based models perform worse than state-of-the-art LMs by a large margin. Moreover, in SeqGAN, the pre-trained LM performs better than the fully trained model with approximate BPC scores of 1.95 and 2.06, respectively, and the BPC deteriorates as adversarial training continues.

Finally, we note that generating sequences larger than 7 characters hurts the BPC of Press et al. (2017). It is difficult to assess the quality of generation with such short sequences.

In Table 2 we present a few randomly generated samples from each model. We indeed observe that adversarial training slightly reduces the quality of generated text for SeqGAN, and find that the quality of 100-character long sequences generated from Press et al. (2017) is low.

## 6 Conclusions

We propose an evaluation procedure for text GANs that is based on approximating the GAN output distribution and using standard LM metrics. We provide a bound for the number of samples required for the approximation and empirically show in practice as few as \(2000\) samples per time-step suffice. We evaluate character-based GAN models using our procedure, and show their performance is substantially lower than state-of-the-art LM. We hope our simple evaluation method leads to progress in GAN-based text generation by shedding light on the quality of such models.

## Acknowledgements

We would like to thank Shimi Salant for his comments and suggestions. This research was partially supported by The Israel Science Foundation grant 942/16, the Blavatnik Computer Science Research Fund, and The Yandex Initiative for Machine Learning.

\begin{table}
\begin{tabular}{l l c c} \hline \hline
**Approach** & **Model** & **BPC** & **Approx. BPC** \\ \hline  & mLSTM + dynamic eval Krause et al. (2017) & 1.19 & \\  & Large mLSTM + emb + WN +VD Krause et al. (2016) & 1.27 & \\
* pre-trained LM Yu et al. (2017) & 1.85 & 1.95 \\ \hline
* full adversarial training Yu et al. (2017) & 1.99 & 2.08 \\  & Recurrent GAN without pre-training Press et al. (2017) & 3.31 \\ \hline \multicolumn{4}{c}{Uniform Distribution} & 4.75 \\ \hline \end{tabular}
\end{table}
Table 1: Test set evaluation of different character-based models on the text8 dataset. State-of-the-art results are taken from [https://github.com/sebastianruder/NLP-progress/blob/master/language_modeling.md](https://github.com/sebastianruder/NLP-progress/blob/master/language_modeling.md). The uniform distribution is equivalent to guessing the next character out of \(|V|=27\) characters.
 



## References

* Caccia et al. (2018) Massimo Caccia, Lucas Cascia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. 2018. Language gans falling short. _arXiv preprint arXiv:1811.02549_.
* Chen et al. (2018) Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, et al. 2018. The best of both worlds: Combining recent advances in neural machine translation. _arXiv preprint arXiv:1804.09849_.
* Chernoff et al. (1952) Herman Chernoff et al. 1952. A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. _The Annals of Mathematical Statistics_, 23(4):493-507.
* Chung et al. (2016) Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. 2016. Hierarchical multiscale recurrent neural networks. _arXiv preprint arXiv:1609.01704_.
* Cooijmans et al. (2016) Tim Cookjmans, Nicolas Ballas, Cesar Laurent, Caglar Gulcehre, and Aaron Courville. 2016. Recurrent batch normalization. _arXiv preprint arXiv:1603.09025_.
* Goldberg (2017) Yoav Goldberg. 2017. Neural network methods for natural language processing. _Synthesis Lectures on Human Language Technologies_, 10(1):1-309.
* Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In _Advances in neural information processing systems_, pages 2672-2680.
* Gulrajani et al. (2017) Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. 2017. Improved training of wasserstein gans. In _Advances in Neural Information Processing Systems_, pages 5767-5777.
* Guo et al. (2017) Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. 2017. Long text generation via adversarial training with leaked information. _arXiv preprint arXiv:1709.08624_.
* Heussel et al. (2017) Martin Heussel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Gunter Klambauer, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a nash equilibrium. _arXiv preprint arXiv:1706.08500_.
* Hoeffding (1963) Wassily Hoeffding. 1963. Probability inequalities for sums of bounded random variables. _Journal of the American statistical association_, 58(301):13-30.
* Jurafsky and Martin (2018) Dan Jurafsky and James H Martin. 2018. _Speech and language processing: Third Edition_, volume 3. Pearson London.
* Kim et al. (2016) Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In _AAAI_, pages 2741-2749.
* Krause et al. (2017) Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. 2017. Dynamic evaluation of neural sequence models. _arXiv preprint arXiv:1709.07432_.
* Krause et al. (2016) Ben Krause, Liang Lu, Iain Murray, and Steve Renals. 2016. Multiplicative lstm for sequence modelling. _arXiv preprint arXiv:1609.07959_.
* Lu et al. (2018) Sidi Lu, Yaoming Zhu, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Neural text generation: Past, present and beyond. _arXiv preprint arXiv:1803.07133_.
* Luong et al. (2015) Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015. Multi-task sequence to sequence learning. _arXiv preprint arXiv:1511.06114_.
* Neubig (2017) Graham Neubig. 2017. Neural machine translation and sequence-to-sequence models: A tutorial. _arXiv preprint arXiv:1703.01619_.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting on association for computational linguistics_, pages 311-318. Association for Computational Linguistics.
* Press et al. (2017) Ofir Press, Amir Bar, Ben Bogin, Jonathan Berant, and Lior Wolf. 2017. Language generation with recurrent generative adversarial networks without pretraining. In _1st Workshop on Learning to Generate Natural Language at ICML 2017_.
* Rajeswar et al. (2017) Sai Rajeswar, Sandeep Subramanian, Francis Dutil, Christopher Pal, and Aaron Courville. 2017. Adversarial generation of natural language. _arXiv preprint arXiv:1705.10929_.
* Ranzato et al. (2015) Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. _arXiv preprint arXiv:1511.06732_.
* See et al. (2017) Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks.
* Semeniuta et al. (2018) Stanislau Semeniuta, Aliaksei Severyn, and Sylvain Gelly. 2018. On accurate evaluation of gans for language generation. _arXiv preprint arXiv:1806.04936_.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in neural information processing systems_, pages 3104-3112.
* Sutton et al. (2000) Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. 2000. Policy gradient methods for reinforcement learning with function approximation. In _Advances in neural information processing systems_, pages 1057-1063

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, pages 5998-6008.
* You et al. (2016) Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016. Image captioning with semantic attention. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 4651-4659.
* Yu et al. (2017) Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. Seqgan: Sequence generative adversarial nets with policy gradient. In _AAAI_, pages 2852-2858.
* Zhao et al. (2017) Junbo Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M Rush, and Yann LeCun. 2017. Adversarially regularized autoencoders for generating discrete structures. _CoRR, abs/1706.04223_.
* Zilly et al. (2016) Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnik, and Jurgen Schmidhuber. 2016. Recurrent highway networks. _arXiv preprint arXiv:1607.03474_.