Title: CaptainGAN: Navigate Through Embedding Space For Better Text Generation

Abstract: Score-function-based text generation approaches such as REINFORCE, in general, suffer from high computational complexity and training instability problems. This is mainly due to the non-differentiable nature of the discrete space sampling and thus these methods have to treat the discriminator as a reward function and ignore the gradient information. In this paper, we propose a novel approach, CaptainGAN, which adopts the straight-through gradient estimator and introduces a "re-centered" gradient estimation technique to steer the generator toward better text tokens through the embedding space. Our method is stable to train and converges quickly without maximum likelihood pre-training. On multiple metrics of text quality and diversity, our method outperforms existing GAN-based methods on natural language generation.

1. Introduction:

Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) have led to many advances in image generation, image editing, style transfer, and representation learning (Karras et al., 2017; Brock et al., 2018; Karras et al., 2019) . Unsurprisingly, much effort has been devoted to adopting the GAN framework for unsupervised text generation (Yu et al., 2017; Che et al., 2017; Balagopalan et al., 2018; Fedus et al., 2018; Guo et al., 2018; de Masson d'Autume et al., 2019; Zhang et al., 2017; Nie et al., 2019) . However, as finding a Nash equilibrium is not as straightforward as finding a local optima, researchers have been forced to develop many ad-hoc tricks and techniques to make GAN training well-behaved. In the text generation setting, they are also faced with the additional obstacle of passing discrete tokens through a non-differentiable operation, which prohibits back-propagating the gradient signal to the generator.

To address the issue of non-differentiablity, researchers and practitioners use score function gradient estimators such as REINFORCE to train GANs for text generation, where the discriminator is cast as a reward function for the generator. However, these methods still suffer from poor sample efficiency due to the credit assignment problem. We argue that it is disadvantageous to utilize the discriminator as simply a reward function when it is known that gradient-based backpropagation is a far more efficient way to perform credit assignment.

In this paper, we propose a novel unsupervised text generation technique, called Captain-GAN, which propagates a modified gradient signal from discriminator to generator in order to improve the efficiency and accuracy of the estimator. Our contributions are as follows:

• An update procedure for the generator to incorporate gradient information from the discriminator during generator training.

• Lower memory and computational requirements than other RL-based counterparts.

• Near SOTA results without maximum likelihood pretraining.

Please see Appendix A for a detailed description of the notation used in this paper.

2. Background:

The Generative Adversarial Network (GAN) proposed in Goodfellow et al. (2014) is an innovative approach to the generative modeling problem. Rather than using the maximum likelihood estimation (MLE) directly to learn a probabilistic model, the GAN is a two-player minimax game in which the goal of one player, the generator G θ , is to generate samples x from p θ , and the goal of the other player, the discriminator D ϕ , is to learn to classify whether or not a sample was generated from real data p data or the generator.

V D = E x∼p data [log D ϕ (x)] + E x∼p θ [log(1 -D ϕ (x))]

(1)

EQUATION

where V D and V G are respectively the objective functions of the discriminator D ϕ and the generator G θ . Equation 2 is the alternative generator loss suggested by the original work as its gradient does not vanish when D ϕ (x) is small.

In the standard GAN architecture, the generator's output is directly connected as the input to the discriminator in a fully differentiable manner, which means the gradients from the discriminator's loss function can be back-propagated to the parameters of the generator. However, it requires sampling a sequence of tokens from a discrete distribution in text generation, which is essentially non-differentiable. In order to avoid the intractability of the gradients, text GANs resort to some sort of estimation.

2.1. Continuous Relaxation:

Continuous relaxation approaches such as Gumbel-Softmax (Jang et al., 2016) approximate a stochastic categorical distribution in terms of a deterministic continuous function. While this apparently allows us to remove the non-differentiable discrete sampling altogether (Kusner & Hernández-Lobato, 2016; Nie et al., 2019) , it creates several serious issues.

The continuous distribution generates the expectation of embeddings -a weighted sum with no direct correspondence to an exact word or token. This means all the discriminator has to do is spot the difference the actual word embeddings and expectation. In turn, the generator will try to compensate by producing extremely "spiky" predictions.

Furthermore, this way of generating data creates a major inconsistency in that during inference , the generator has to sample a discrete sequence from distribution whereas during training, it's only trained to generate an expectation that is feasible to discriminator.

2.2. Score-Function Gradient Estimator:

The score-function gradient estimator (Fu, 2006; Glynn, 1990) , also known as the REIN-FORCE (Williams, 1992 ) is a common solution for non-differentiable issue as mentioned above. Applying the REINFORCE algorithm, the gradient of the expectation of reward function f ϕ can be written as

EQUATION

Since it does not require f ϕ to be differentiable or even continuous as a function of x, the gradient of E x∼p θ [f ϕ (x)] can be back-propagated to the generator G θ . In the context of GANs, E x∼p θ [f ϕ (x)] can be seen as the objective function of the generator and the reward function f ϕ can be replaced with D ϕ .

Although REINFORCE is an unbiased estimator, it still has a number of disadvantages such as high variance, low sample efficiency and the credit assignment problem. Therefore, much effort is devoted to reducing the variance using special methods (Gu et al., 2016; Grathwohl et al., 2017) , or to providing more dense rewards such as in Yu et al. (2017) ; Che et al. Figure 1 : The non-differentiable operation between the generator and the discriminator.

2.3. Straight-Through Gradient Estimator:

Another approach is using a straight-through gradient estimator (Bengio et al., 2013; Jang et al., 2016) . The basic idea is to perform non-differentiable operation during the forward pass, but approximate it with a differentiable proxy during the backward pass.

Consider sampling from a categorical distribution (notation borrowed from Jang et al.

(

EQUATION

The following operation is performed during the forward pass

EQUATION

where a categorical sample x is encoded as a v-dimensional one-hot vector z (v being the given vocabulary size). During the backward pass, the corresponding straight-through estimator of function f with respect to θ is

ĝθ = ( ∂f ∂z ) ⊤ ∂m θ ∂θ (5)

derived by using the approximation ∂z ∂θ ≈ ∂m θ ∂θ , where m θ is a differentiable proxy for the non-differentiable one-hot encoding vector z (Jang et al., 2016) . Moreover, Jang et al. (2016) suggest the choice of

p θ = [p θ (x 1 ), . . . , p θ (x v )] ⊤ (6)

as proxy, where {x 1 , . . . , x v } are all possible tokens in vocabulary V . Compared to the former method, this approach updates all the possible tokens' probability, even if the tokens were not sampled.

3. Proposed Method:

The non-differentiable connection between the generator G θ and the discriminator D ϕ is our primary focus in this paper (Figure 1 ). Starting from the straight through gradient estimator, we derive a re-centered estimator which helps the generator learn to select better tokens.

3.1. Gradient Estimation in the Generator:

Consider a discriminator whose first layer is an embedding lookup layer. During forward propagation, the operation below is performed first:

e x = E.lookup(x) = Ez (7)

where E = [e 1 , . . . , e v ] and z is one-hot encoding of vocabulary. As described in Section 2.3, this is a non-differentiable categorical sampling operation with a straight-through estimator of the gradient of V G with respect to θ, formulated as follows:

EQUATION

)

where xt is the selected token at time-step t, T is the maximum time-step and V is the given vocabulary. The first term of Equation 8 can be derived from Equation 7by applying the chain rule.

3.2. Re-centered Estimator:

We turn our attention to the first term of Equation 9:

EQUATION

Denoting it as

EQUATION

)

The coefficient δ xt→x [V G ] can be considered as an affinity factor indicating how much the discriminator is 'attracted' to the token x.

Recall that the gradient ∂V G ∂e (e x) has the direction of greatest increase of the objective V G at e x. Since this direction is relative to e x, we suggest taking the inner product with the vector e x -e x which has been re-centered relative to e x (Figure 2 ).

EQUATION

This inner product approximates the increase of V G when xt is replaced by x.

Because this approximation only holds in the neighborhood of e xt , |e x -e xt | ≪ 1, we also add a kernel to dampen the affinity factor for e x that are far away. That is,

EQUATION

where

EQUATION

)

See Appendix B for more details. As a result, Equation 9becomes

EQUATION

)

In score-function based approaches, the generator is only given feedback on the tokens it samples, so it requires a large amount of trial-and-error in order to figure out how to write realistic sentences.

To use an analogy, imagine a captain at sea (embedding space) searching for treasure. Each iteration of the game, the captain chooses random locations in the space and receives feedback regarding the choices. Score-function based approaches tell us how good the location is but say nothing about how to improve it. In contrast, our method, gives the captain a navigational direction to the next best location. Even if the direction is biased, it should allow the captain to find the treasures with fewer iterations. 

* →B [V G ], δ RC * →A [V G ]

have opposite signs, and thus better at reflecting the relative suitability of A and B as judged by the discriminator.

4. Evaluation Metrics:

We evaluate our model's ability to generate realistic and diverse texts with BLEU-based metric, Fréchet Embedding Distance and language model based metric.

4.1. BLEU and Self-BLEU:

BLEU (Papineni et al., 2002) and Self-BLEU (Zhu et al., 2018 ) capture text quality and diversity respectively. BLEU, a modified form of n-gram precision, measures local consistency between a set of reference and a set of candidate texts. Self-BLEU is a version of BLEU in which both the reference and candidate texts are drawn from the same corpus. A high Self-BLEU means that members within the corpus are too highly correlated, indicating a lack of diversity.

4.2. Fréchet Embedding Distance:

Fréchet Embedding Distance (FED) (de Masson d'Autume et al., 2019) measures sentimental similarity, global consistency and diversity between reference and candidate texts. Semeniuta et al. (2018) claims that Fréchet Distance (FD) is not sensitive to the choice of embedding model. However, we notice significant discrepancies between FD scores calculated using different embedding models. The details are discussed in Appendix G.

4.3. Language Model Scoring Methods:

Following Caccia et al. (2018); Zhao et al. (2017) , we also evaluate the quality and the diversity of generated samples by Language Model Perplexity (LM) and Reverse Language Model Perplexity (RLM).

Language Model Score (LM).:

We use a language model trained on real data to estimate the negative log likelihood per word of generated sentences. Samples which the language model considers improper, such as ungrammatical or nonsense sentences, have high perplexity.

Reverse Language Model Score (RLM). If we instead train a language model on generated data while evaluating on real data, we can judge the diversity of the generated samples. Low-diversity generated data leads to an overfitted model which generalizes poorly on real data, as indicated by a high perplexity.

5. Results & Discussion:

In this section, we use two datasets to evaluate CaptainGAN: COCO Image Captions1 and EMNLP 2017 News2 . We discuss our results from different perspectives via the evaluation metrics presented in the last section. Furthermore, an ablation study is performed in order to know the contribution of each technique. More details about our experiment setup, datasets, architecture and training techniques such as temperature control are described in Appendix D. 

5.1. Quality and Diversity:

On metrics of local text consistency and diversity, CaptainGAN significantly outperforms prior GAN-based models which rely heavily on pre-training, with the exception of RelGAN.

The BLEU/Self-BLEU temperature sweeps shown in Figure 3a for CaptainGAN indicate that CaptainGAN approaches the performance of a language model. On the other hand, in Figure 3b , the language model scores show similar results for CaptainGAN and MLE model. Results for COCO are shown in Appendix E.

5.2. Temperature Sensitivity:

CaptainGAN is robust to temperature variations. This is demonstrated in Figure 4a , which depicts how the softmax temperature influences the FED score. Regardless of model, changes in softmax temperature are certain to affect the FED, however, CaptainGAN shows the least variation.

5.3. Global Consistency:

CaptainGAN improves global consistency comparing to prior works and language model on LM perplexity and FED score, respectively. As shown in Table 1 , our method significantly reduces the gap of perplexity between GAN-based method and language model, which is directly trained to minimize the perplexity. Besides, we show that CaptainGAN outperforms MLE trained model on FED score as shown in Table 1 . 

5.4. Matching N-grams of Training Data:

Text generating models are often criticized for simply memorizing texts seen during training instead of learning how to model the true data distribution. We show the longest matching ngrams between text from the training data and samples in Figure 4b . CaptainGAN generates more samples with small matching n-grams (n < 5) comparing to MLE model, implying that our model can generate novel sentences without resorting to memorizing phrases from the training data.

5.5. CaptainGan As A Language Model:

CaptainGAN's strong performance on the evaluation metrics described earlier shows that it is capable of generating realistic text. Since the generator in our case is a RNN which can produce conditional probabilities at each timestep, this raises the question of whether or not an explicit language model is being learned.

We feed real data to our generator and calculate the perplexity to measure the generator's suitability as a language model. Surprisingly, as shown in We explain this phenomenon as follows: MLE models are trained specifically for low perplexity, since they minimize KL divergence. However, CaptainGAN minimizes alternative loss (Equation 2), which is similar to minimizing reverse KL divergence (Arjovsky & Bottou, 2017) . This objective assigns an extremely low cost to mode dropping and does not force the generator to mimic all aspects of real data. This can result in poor modeling of the likelihood, but does not necessarily lead to poor sample generation. (Theis et al., 2016; Hashimoto et al., 2019) .

Following 5.4, we conclude that our model's ability to generate realistic text cannot be the result of simply plagiarizing training samples. Note: Perplexity is calculated as geometric mean over sentences.

5.6. Comparison to score function-based methods:

Our method is able to outperform score function-based approaches using small batch sizes. Score function-based approaches often rely on large batch sizes for variance reduction (de Masson d'Autume et al., 2019) , but this does not appear to be necessary for our recentered estimator. This confirms the reasoning at the end of Section 3.2, and greatly reduces our memory requirements during training.

In addition, since gradients are naturally calculated for every timestep, there is no need for Monte-Carlo estimation of rewards like in score function-based approaches. Because our generator receives a richer learning signal, we use a 5:1 ratio for discriminator-generator updating. This greatly lessens our computational requirements.

5.7. Ablation Study:

To understand the impact of each component of the CaptainGAN, we conduct an ablation study. In Table 3 , we show the influence of adding new features, namely spectral normalization, re-center estimator, pretrained embeddings and discriminator regularization decoupled weight decay (Loshchilov & Hutter, 2018) and one-sided label smoothing (Salimans et al., 2016) , by scoring FED on validation data.

We observe that:

• As shown in Zhou et al. (2019) , an unrestricted objective leads to an uninformative gradient. Without spectral normalization, the Lipschitz constant of the discriminator is unbounded making it difficult to update the generator with a gradient-based approach.

• Rather unexpectedly, pretrained embeddings do not lead to any significant improvement, which is surprising given our method's dependence on the discriminator's embedding space. The embeddings which the discriminator learns from scratch are capable of giving the generator an effective learning signal.

• The application of the re-centered estimator is responsible for a 20% improvement in FED, confirming its effectiveness.

• Equation 14 is dependent on the norm of the embeddings, which explains the effectiveness of discriminator weight regularization at improving FED. 

6. Conclusion and Future Work:

In this work we have presented CaptainGAN, an effective gradient-based method for quickly training a text generating GAN without pretraining. Starting from the straight-through estimator, we derive a re-centered gradient estimator that improves the quality/diversity of generated texts as measured by various standard metrics. In future work, we plan to investigate our estimator in more theoretical detail and further decrease its bias.

Appendix A Notation:

Symbol Meaning V = {x 1 , . . . , x v } a predefined vocabulary x a token belongs to V x a token sampled from V x a sequence of tokens belong to V x a sequence of tokens sampled from V ⊤ a transpose operation

Appendix B The Importance of Kernel:

To verify the necessity of the kernel term (Equation 14), we directly apply Equation 12to Equation 11 as follows:

EQUATION

)

where x is the selected token and x is a tokens in the vocabulary V.

Let f (x) = ⟨e x , ∂V G ∂e (e x)⟩ and b = ⟨e x, ∂V G ∂e (e x)⟩. b does not depend on x and is constant in the summation.

EQUATION

Without the kernel, the update to θ will be the same as prior work and thus carries the same drawback. Table 4 : Generator structure, d E is the embedding size, and V is the vocabulary size.

C.1 Generator:

The purpose of the first dense layer is to project the GRU cell outputs into the embedding space. The second dense layer converts the projected embedding vectors to logits over the tokens and is initialized as the transpose of the embedding weights.

At the first timestep, the input to the embedding layer is the start-of-sentence token. The vocabulary also contains an end-of-sentence token. If the generator outputs this token at any timestep, the sentence ends and its length is set to this timestep. To perform consistent convolution on sentences of different lengths, we use a special masking mechanism on all layers. For all convolution and pooling layers, we mask out input features where the receptive field is out-of-bounds with respect to the unpadded input sentence. 

C.2 Discriminator:



COCO:

-a person standing in a boat on the water .

-three sheep are grazing on the city sidewalk .

-kenya airways airplane wing , engine and cabin on the tarmac .

-light poles in the snow with yellow traffic lights mounted on them .

-we are looking at the floor between the toilet and the wall .

-someone taking a photo of a small residential bathroom .

-a cat sits on the seat of a bicycle and looks down at another cat on a snowy day .

-large set of motorcycles all lined up down a street . EMNLP 2017 News -apple clearly doesn ' t want to see the dollar value of its u . k . earnings fall by a similar amount .

-we do look at the number of hours we produce , and measure that against the religious make -up of society .

-the u . s . has struggled to find an effective ground force to take on isis in syria , where president obama has ruled out a u . s . ground combat role .

-it is a happy and kind world that we live in on this show and that is where i hope we can live in real life .

-while men and women may end up earning roughly the same amount in the same jobs , men are more likely to end up in higher -paying roles in the tech industry .

-but while they were beaten by a better side , the tie did reveal what i think has been city ' s biggest problem this season : they have lost the ability to score against good teams .

-according to facebook ' s policies , accounts can be suspended if law enforcement believe individuals are at risk of harm .

-she has to take personal responsibility for this -when she was health secretary she was told by those who know best that her decision to cut student places would have a damaging impact . 

-the bathroom bowl with focus on the vanity .

-a man racer in a car brushing a carved cake .

-a lady riding a motorcycle and a street while sitting on a pedestal track .

-a commercial man are standing in the bicycle next to the back of an orange computer .

-an airplane the bus is headed as something in the sky .

-people riding on a bike on a surfer surfing doing the ocean .

-a plane propped up in front of a vehicle .

-a man wearing a cat in front of a motorcycle . EMNLP 2017 News -a one -game event and a limited number this dropped by 38 per cent to find out what we could want , to be as much done as we finish as the games .

-more than 30 per cent of calais camps at risk were taken between children and child refugees crossing the mediterranean to fall apart .

-the uk government got on the deal when it came to other eu countries that allowed workplace checks for 26 .

-so women i didn ' t want my parents to work hard with me at the time of the task of making it presents .

-black voters feel there may not be enough choice between former president george w . bush and i have to go along with problems and support the people that are in a swing state for president .

-he sold out to its eighth list , but a stunning showing that what cost the ability to start the league failed ? -let ' s catch him down his back phone one morning john f kennedy , where he is in a us meeting group with supporters of love not being successful in the fields .

-then there ' s a couple of chances that they won competition --the result from what is of me going out there that i really will play for the foundation . -a bathroom with a mirror , a shower and a toilet .

-two people sit on a building in front of a building .

-a group of people walking around the corner of a building .

-people are riding bikes on a city street .

-there is a picture of a bathroom in a bathroom .

-an empty city street with cars parked on the side of the road .

-a man taking a picture of a white bathroom . EMNLP 2017 News -if i go to the party , it ' s a disaster , and i ' m going to keep it right .

-i think i ' ve had some great strength in my life and i want that to be here for me .

-obama pointed to reporters claiming he was willing to discuss the situation , but the u . s . have been military concern by american troops , including a nuclear deal in march .

-a 16 -year -old man was arrested after the death of a man shot and arrested him on suspicion on his behalf .

-the problem is that they are hard to find out what they want to see where they are and what they can achieve their past two years .

-now it ' s important to me and i am thinking so hard to make it the best for a long time .

-great britain is suggesting the u . s . -led coalition figures at the australian foreign investor have raised more than $ 250 , 000 in the construction of operations .

-it is a simple way to believe that if it can happen , then i ' m not the right choice . 

:

The preprocessed COCO dataset is available at https://github.com/pclucas14/ GansFallingShort/tree/master/real_data_experiments/data/coco

The preprocessed EMNLP dataset is available at https://github.com/pclucas14/ GansFallingShort/tree/master/real_data_experiments/data/news

http://www.statmt.org/wmt17/

The preprocessed COCO dataset is available at https://github.com/pclucas14/ GansFallingShort/tree/master/real_data_experiments/data/coco

The preprocessed EMNLP dataset is available at https://github.com/pclucas14/ GansFallingShort/tree/master/real_data_experiments/data/news

https://tfhub.dev/google/universal-sentence-encoder/2

https://tfhub.dev/google/universal-sentence-encoder-large/3

Figures and Tables:

(2017); Fedus et al. (2018); de Masson d'Autume et al. (2019), where Monte-Carlo roll-outs are used to obtain per-word rewards.

Figure 2: Conceptual contour graph for objective V G in embedding space. ∇ e V G is the gradient of V G at the sampled token * . B is a nearby token with a lower V G . Affinity factors for each token are calculated as the inner product between the red vectors and the blue vector. (a) The inner products of A and B with ∇ e V G are both positive even though B leads to a lower V G . (b) After applying re-centering in the inner product, the affinity factors δ RC * →B [V G ], δ RC * →A [V G ] have opposite signs, and thus better at reflecting the relative suitability of A and B as judged by the discriminator.

(a) Negative BLEU-5 versus Self-BLEU-5.(b) Language-and reverse language-model scores.

Figure 3: BLEU scores (left) and language model scores on (right) on EMNLP News. Resultsfor LeakGAN(Guo et al., 2017),MaliGAN(Che et al., 2017), RankGAN(Lin et al., 2017)   and SeqGAN(Yu et al., 2017) are fromCaccia et al. (2018).

Figure 4: (a) The influence of different temperature values on FED scores measured on EMNLP 2017 News. (b) Longest matching n-grams on EMNLP 2017 News training data.

Figure 5: Model architecture and computation flow

Figure 8: FED scores on different sampling size of training data and CaptainGAN samples.

FED & LM scores on EMNLP 2017 News

Table content: <table><tr><td>Model</td><td colspan="3">Train FED Val. FED LM Score</td></tr><tr><td>Training data</td><td>0.0050</td><td>0.0120</td><td>3.22</td></tr><tr><td>MLE</td><td>0.0100</td><td>0.0194</td><td>3.43</td></tr><tr><td>SeqGAN</td><td>0.1234</td><td>0.1422</td><td>6.09</td></tr><tr><td>MaliGAN</td><td>0.1280</td><td>0.1504</td><td>6.30</td></tr><tr><td>RankGAN</td><td>0.1418</td><td>0.1431</td><td>5.76</td></tr><tr><td>LeakGAN</td><td>0.0718</td><td>0.0691</td><td>4.90</td></tr><tr><td>RelGAN</td><td>0.0462</td><td>0.0408</td><td>3.49</td></tr><tr><td>CaptainGAN</td><td>0.0120</td><td>0.0184</td><td>4.04</td></tr></table>



Table content: <table/>

Word level perplexity on EMNLP 2017 News.

Table content: <table><tr><td>Model</td><td colspan="2">Train Perp. Val. Perp.</td></tr><tr><td>Random.</td><td>5.3k</td><td>5.3k</td></tr><tr><td>MLE</td><td>26</td><td>34</td></tr><tr><td>CaptainGAN</td><td>14.9k</td><td>16.8k</td></tr></table>

Ablation study on EMNLP 2017 News validation data

Table content: <table><tr><td>Model</td><td>Val. FED</td></tr><tr><td>straight through estimator</td><td>0.383</td></tr><tr><td>+ spectral normalization</td><td>0.047</td></tr><tr><td>+ re-center estimator</td><td>0.037</td></tr><tr><td>+ pre-trained embeddings</td><td>0.035</td></tr><tr><td>+ discriminator regularization</td><td>0.018</td></tr></table>

Discriminator, all conv & pooling layers use 'same' padding.

Table content: <table><tr><td>embedding lookup</td></tr><tr><td>conv3-512 relu</td></tr><tr><td>conv4-512 relu</td></tr><tr><td>avg pool2</td></tr><tr><td>conv3-1024 relu</td></tr><tr><td>conv4-1024 relu</td></tr><tr><td>global average pool</td></tr><tr><td>dense 1024 relu</td></tr><tr><td>dense 1</td></tr></table>

Training samples on EMNLP 2017 News and COCO dataset.

Table content: <table/>

Randomly selected MLE samples on EMNLP 2017 News and COCO with temperature = 1.0.

Table content: <table/>

Randomly selected CaptainGAN samples on EMNLP 2017 News and COCO with temperature = 1.0.

Table content: <table/>

Peer review (iclr2020):
The authors propose CaptainGAN, a method using the straight-through gradient estimator to improve training of the generator for text generation. The paper is well-written and the evaluation seems thorough, comparing to relevant baselines. Comments: Figure 3: the caption refers to Caccia et al. for results on LeakGAN, MaliGAN and seqGAN, but unless I’ve missed it, RelGAN hasn’t yet been introduced by name as a baseline? The citation is given in the opening part of the introduction, in an enumeration, but isn’t revisited later in the text - not even here where the results of the model are introduced. Given that it seems, according to the presented results, to be the most competitive of the GAN models that the authors are comparing to, maybe it’s worth adding more contextual information on RelGAN to the Background section? For their method, the authors should report an average performance over several random seeds and provide the standard deviation / confidence intervals, for the readers to be able to assess the stability of the method and the significance of the improvement reported in the results. I find Section 5.5. particularly interesting, as well as the reported perplexity in Table 2. The authors provide 3 bullet points to explain the unusually high perplexity of the generator on the training and validation data. I feel that the explanations that are given are at the moment vague and not visibly backed by data, therefore being speculative. Obviously, point 1) is hard to quantify - but point 2) could possibly be at least partially quantified - if the hypothesis is that names, places, punctuation marks etc play an important role in the reported perplexity score, then maybe the authors could test this by correlating model perplexity on sentences with whether those sentences contain these types of words?