CaptainGAN: Navigate Through Embedding Space For Better Text Generation
Authors: Chun-Hsing Lin
Alvin Chiang
Chi-Liang Liu
Chien-Fu Lin
Po-Hsien Chu
Siang-Ruei Wu
Yi-En Tsai
Chung-Yang (Ric) Huang
Venue:  (2020)
Keywords: Generative Adversarial Network
Text Generation
Straight-Through Estimator

Abstract: Score-function-based text generation approaches such as REINFORCE, in general, suffer from high computational complexity and training instability problems. This is mainly due to the non-differentiable nature of the discrete space sampling and thus these methods have to treat the discriminator as a reward function and ignore the gradient information. In this paper, we propose a novel approach, CaptainGAN, which adopts the straight-through gradient estimator and introduces a ”re-centered” gradient estimation technique to steer the generator toward better text tokens through the embedding space. Our method is stable to train and converges quickly without maximum likelihood pre-training. On multiple metrics of text quality and diversity, our method outperforms existing GAN-based methods on natural language generation.

Under review as a conference paper at ICLR 2020
CAPTAINGAN: NAVIGATE THROUGH EMBEDDING SPACE FOR BETTER TEXT GENERATION
A Anonymous authors Paper under double-blind review
ABSTRACT
Score-function-basedd text generation approaches such as REINFORCE, in general, suffer from high computational complexity and training instability problems. This is mainly due to the non-differentiabll nature of the discrete space sampling and thus these methods have to treat the discriminator as a reward function and ignore the gradient information. In this paper, we propose novel approach, CaptainGAN, which adopts the straight-through gradient estimator and introduces a "re-centered" gradient estimat ion tech- nique to steer the generator toward better text tokens through the embed- ding space. Our method is stable to train and converges quickly without maximun likelihood pre-training On multiple metrics of text quality and divensity, our method outperforms existing GAN-based methods on natural language generation.
1 INTRODUCTION
Generative Adversarial Networks (GAN) Goodfellow et al, 2014) have led to many advances image.editing,
in image generalion image editing style transfer, and representation learning (Karras et al
2017:
Brock et al., 2018; Karras et al., 2019). Unsurprisingly, much has been devoled
ccoaaoptinggtto the GAN framework for for insupervised text.gemeaatioonn (Yu et 2017 7 Che et al 2018; Guo al, 2018 de Masson d' d'Autume 2017 Nic et al., 2019). However, as finding a Nash equilibrium
2017
Balagopalan et al.
2018
Fedus et_
et al 2019: Zhang et al
not as straightforward as finding local optima, rescarchers have been forced develop many ad-hoc tricks and techniques to make GAN training well-behaved. In the text gen- eration setting, they are also faced with the additional obstacle passing discrete tokens through non-differentiable operalion, which prohibits back pack-propagaling the gradient signal to the generator.
To address the issue of non-differentiabbity, researchers and practitioners score function gradient estimators such as REINFORCE to train GANs for text generation, where the discriminator is cast suffer from poor sample efficiency due to the credit assignment problem. We argue that is disadvantageous utilize the discriminator known that gradient-based backpropagation is a far more efficient way to perform credit. assignment.
reward function for the generator. However, these methods still
simply reward function when it is
In this paper, we propose a novel unsupervised text generation technique, called Captain- GAN, which propagates modified gradient signal from discriminator to generator in order to improve the efficiency and accuracy of the estimator. Our contributions are as follows:
An updale procedure for the generator to incorporate gradient information from the discriminator during generator training.
Lower memory and computational requirements than other RL-based counterparts.
Near SOTA results without maximum likelihood pretraining.
Please Appendix for detailed description of the notation used in this paper.

---

Under review as a conference paper at ICLR 2020
2 BACKGROUND
The Generative Adversarial Network (GAN) proposed in Goodfellow et al. (2014) is innovative approach to the generative modeling problem. Rather than using the maximum likelihood estimation (MLE) directly to learn a probabilistic model, the GAN minimax game in which the goal of one player, the generator Ge, is to generate samples from pe; and the goal of the other player, the discriminator Dø, is to learn classify whether or not sample was generated from real data Pdata or the generator.
(wo-player
DD = Para (log (x) + Ex-po [log(1 - Do(x)))
(1)
V E po loog( (x))
(2)
where Vo and Va are respectively the objective functions of the discriminator Do and the generator Gg. Equation 2 the alternative generator loss suggested by the original work as its gradient does not vanish when (x is small.
In the standard GAN architecture, the generator's output is directly connected as the input to the discriminator in a fully differentiable manner, which mcans the gradients from the discriminator's loss function can be pack-propagatod to the paramcters of the generator. However, it requires sampling a sequence of tokens from a discrete distribution in text generation, which is essentially non-dillerentiable. In order to avoid the intractability of the gradients, text. GANs resort to some sort of estimation.
2.1 CONTINUOUS RELAXATION
Continuous relaxation approaches such Gumbel-Softmar (Jang et al, 2016) approximate stochastic categorical distribution in terms of a dcterministic continuous function. While apparently allows us to remove the non-differentiabl discrete sampling altogether (Kusner
Hernandez-Lobato 2016; Nie al. 2019), it creates several serious issues.
The continuous distribution generates the expectation of embeddings weighted sum with no direct correspondence to an exact word or token. This means all the discriminator has to do is spot the difference the actual word embeddings and expectation. In turn, the generator will try to compensate by producing extremely "spiky" predictions.
Furthermore, this way of generating data creates major inconsistency in that during in- ference, the generator has to sample a discrete sequence from distribution whereas during training, it's only trained to generate expectation that is feasible discriminator.
2.2 SCORE-FUNCTION GRADIENT ESTIMATOR
The score-finction gradient estimator Fr. 2006; Glynn, 1990), also known the REIN-
FORCE (Williams, 1992) is a common solution for non-differentiable issue as mentioned above. Applying the RÉINFORCE algorithm, the gradient of the expectation of reward function fa can be written as
i 00
Ex [oo( Rx (x( log po (x)).
(3)
00
Since it does not require fo be differentiable or even continuous as function of x, the gradient of Ex-pp [fx()] can be back-propagated to the generator Gg. In the context of GANs, Ex [x)( can be seen as the objective function of the generator and the reward function fa can be replaced with Dz.
Although REINFORCE is an unbiased estimator, it still has a number of disadvantages such as high variance, low sample efficiency and the credit assignment, prangmmmttppobble,n Therefore, much affort, ig dexoted to reducing the variance using special methods (Ca et al 2016 Grathwnhl etlal. 2017) or to, providing more dense rewards. such in Yu et al (2017); Che et al. [2017]; Fedus et al. (2018); de Masson d' A'utumm et al. (2019). where Monte- Carlo roll-outs used to obtain per-word rewards.
2

---

Under review as a conference paper at ICLR 2020
Gererator Gp
*ake sequence
Distriminator D
Embeddling Loo
Feature Extraction
Classifiation
Furward Prupagatioon
ferentilerentibble Forward Propagation
real sequence
Backward Propagetion
Estimated Backward Propagaticn
Figure The non-diffeerntiable operation wetween the generator and the discriminator.
2.3 STRAIGHI-THROUUG GRADIENT ESTIMATOR
Another approach is using a straight-through gradient estimator (Bengio al,, 2013; Jang et al. 2016). The basic idea is to perform non-diflerentiable operalion during the forward pass, but approzimate with a differentiable proxy during the backward pass.
Consider sampling from categorical distribution (notation borrowed from Jang et al. (2016)): : The following operation is performed during the forward pass
2 one-hot(1))
(4)
where categorical sample & is encoded v-dimensional one-hot vector (v being the given vocabulary size). During the backward pass, the corresponding thraigh esti- mator of function f with respect to is
0s 8ms Oz
go
00
8x derived by using the approximation non-differentiable one-hot encoding vector Jang et al. 2016). Moreover, Jang et al. (2016) suggest the choice of
30
~
Sme
where ma is a differentiablee ble proxy for the
Po (po(x))...... ((((1)))....o(2.)))) po(x)))
as proxy, where (.I,..., F are all possible Lokens in vocabulary V. Compared to the former method, this approach updates all the possible tokens' probability, even if the tokens were not sampled.
PROPOSED METHOD
The non-differentiable connection between the generator Go and the discriminator D& is our primary focus in this paper (Figure 1) Starting from the straight through gradient estimator, we derive a re-centered estimator which helps the generator learn to select better tokens.
3.1 GRADIENT ESTIMATION IN THE GENERATOR.
Consider a discriminator whose first layer an embedding lookup layer. During forward propagation, the operation below is performed first:
e E.lookup($) Ez
where E = je..... e,,.,,,,,]]] and z is one-hot encoding vocabulary. As described in Section 2.3 this is non-differentiable categorical sampling operation with straight-through estimator
3

---

Under review as a conference paper at ICLR 2020
of the gradient of Va with respect to 8, formulated as follows:
VVG
VVG
ppi.
90
E
(ex.))
(8)
dz.
00
de
00
t=1
T
VVG de
0p(c(x1111)
Ce:
(
(9)
00
1=1
where 8g is the selected token at time-step T is the maximum time-step and V is the given vocabulary. The first term of Equation can be derived from Equation by applying the chain rule.
3.2 RE-CENTERED ESTIMATOR
We turn our attention to the first term of Equation 1
OVa (eax))
(exy Oc
(10)
Denoting as 6z Vel, Equation becomes
T
Opo (p0(2/33a11)
Σ
82 [Va]
(11)
00
1=1 DEV
The cocfficient [ell can be considered discriminator is 'attracted' to the token x.
affindty factor indicating how much the
Recall that the gradient aVe (es) has the direction of greatest increase of the objective Va at eş- Since this direction is relative to eg, we suggest taking the inner product with the vector e ey which has been re-centered relative es (Figure 2
OVG
(e, es., de
(ez,))
(12)
This inner product approximates the increase of VG when á is replaced by æ.
Because this approximation only holds in the neighborhood of ea,: le, ea, < 1, we also add kernel to dampen the affinity factor for e, that are far away. That is,
OVG (ox,))
RC 8. + Vcl K(e,, o2, Me, Ca de
(13)
where
K(u, v)
(14)
V1 + 11 v
See Appendix B for more details. As a result, Equation becomes
T
Op( x1111)
9
RC
Σ
Σ El
RC B. +x [cl
(15)
00
In score-function based approaches, the generator is only given feedback on the tokens it samples, so requires large amount of trial-and-error in order figure out how write realistic sentences.
To use an analogy, imagine a captain at sea (embedding space) searching for treasure. Each iteration of the game, the captain chooses random locations in the space and receives feed- back regarding the choices. Scoro-function based approaches tell us how good the location is but say nothing about how to improve it. In contrast, our method, gives the captain navigational direction to the next best location. Even if the direction biased, it should allow the captain to find the treasures with fewer iterations.

---

Under review as a conference paper at ICLR 2020
VeVG
CA
C.
eg
(b)
Figure 2: Conceptual contour graph for objective Ve in embedding space. V.Va is the gradient of Ve at the sampled token *, is nearby token with a lower Va. Affinity factors for each token are calculated the inner product between the red vectors and the blue vector.
(a) The inner products of A and B with VaVa are both positive even though B leads to lower Va- (b) After applying re-centering in the inner product, the affinity factors & [Va], +++B S [Va] have opposite signs, and thus better at reflecting the relative suitability A and B as judged by the discriminator.
EVALUATION METRICS
We evaluate model's ability to generate realistic and diverse texts with BLEU-based metric, Fréchet Embedding Distance and language model based metric.
4.1 BLEU AND BLLF-BLEU
BLEU (Papineni et al. 2002] and Self-BLEU (Zhu al, 2018) capture text quality and di- versity respectively. BL,UU, a modified form of n-gram precision, measures local consistency between set of reference and a set of candidate texts. Self-BLEU is version of BLEU in which both the reference and candidate texts are drawn from the same corpus. A high Self-BLEU means that members within the corpus are too highly correlated, indicating lack of diversity.
4.2 FRÉCHET EMBEDDING DISTANCE
Fréchet Embedding Distance (FED) (de Masson d' Autume et al, 2019) measures sentimen- tal similarity, global consistency and diversity between reference and candidate texts.
Semeniuta al. (2018) claims that Fréchet Distance (FD) not sensitive to the choice of embedding model. However, we notice significant discrepancies between FD scores calcu- lated using different embedding models. The details are discussed Appendix G
4.3 LANGUAGE MODEL SCORING METHODS
Following Caccia et al, (2018); Zhao et al (2017). we also evaluate the quality and the diversity of generated samples by Language Model Perplexity (LM) and Reverse Language Model Perplexity (RLM).
Language Model Score (LM). We use a language model trained on real data to estimate the negative log likelihood per word of generated sentences. Samples which the language model considers improper, such as ungrammatical or nonsense sentences, have high per- plexity.
Reverse Language Model Score (RLM). If we instead train language model on generated data while evaluating real data, judge the diversity of the generated samples. Low-diversity generated data leads to an overfitted model which generalizes poorly real data, as indicated by high perplexity.
5

---

Under review as a conference paper at ICLR 2020
5 RESULTS & DISCUSSION
In this scction, we use two datascts to evaluate CaptainGAN: COCO Image Captions and EMNLP 2017 News B. We discuss our results from different perspectives via the evaluation metrics presented in the last section. Furthermore, to know the contribution of each technique. More details about our experiment selup, datasets, architecture and training techniques such temperature control are described in Appendix D
ablation study is performed in order
10
0.6
CaptalnGAN various
CaptainGAN various T
MLE various T
MLE al various T
0.5
log(RLM Perplexity
training data
LeakGAN
validalion
Selt Seif BLBB--
0.4
ReIGAN
RankGAN SeqGAN
5
MaliGAN
training data-
-0.4
-0.3
-0.2
-0.1
0.0
4
10
BLEU 5-uegalive
loy(LM Porplexily)
(a) Negative BLEU-5 versus Self-BLEU-5.
(b) Language- and scores.
reverse
language-modd model
Figure 3: BLEU scores (left. and language modelscoree on (right) on EMNLP News. Results for LeakGAN Guo et. al 1 MaliGAN el_ 2017), RankGAN al, 2017] and SeqGAN Yu et al., 2017) are from Caccia el. al. (2018).
5.1 QUALITY AND DIVERSITY
On metrics of local text consistency and diversity, CaptainGAN significantly outperforms prior GAN-based models which rely heavily on pre-training, with the exception of RelGAN. The BLEU/ /SEI--BLUU lemperature sweeps shown in Figure Bal for CaplainGAN indicate that CaptainGA GAN approaches the performance of language model. On the other hand, in Figure 3b. the language model scores show similar results for CaptainGAN and MLE model. Results for COCO are shown in Appendix E
5.2 TEMPERATURE SENSITIVITY
CaptainGAN is robust to temperature variations. This is demonstrated in Figure da which depicts how the softmax temperature influences the FED score. Regardless of model, changes in softmax temperature are certain to affect the FED, however, CaptainGAN shows the least variation.
5.3 GLOBAL CONSISTENCY
CaplainGAN improves global consistency comparing to prior works and language model on LM perplexity and FED score, respectively. As shown in Table [ our method significantly reduces the gap of perplexity between GAN-based method and language model, which is directly trained to minimize the perplexity. Besides, show that CaptainGAN outperforms MLE trained model on FED score as shown in Table 1.
GanfFallinghatr/trressssseeeeel data _tt._daeeeremeeteeeeeaoooo https///ggitaabb.c////cllalll
TThe preprocessed COCO dataset
available at https:
"The preprocessed EMNLP dataset is available at https: httpp://ggihubb.co//cccccass
Bannaalllnggoot//rre/a/ttttrr/ll ottpa////eeeeeeoo.oooooprsaaa/ar

---

Under review as a conference paper at ICLR 2020
Table 1: FED & LM scores on EMNLP 2017 News
Model
Train FED
Val. FED
LM Score
Training data
0.0050
0.0120
3.22
MLE
0.0100
0.0194
3.43
SeqGAN
0.1234
0.1422
6.09
MaliGAN
0.1280
0.1504
6.30
RankGAN
0.1418
0.1431
5.76
LeakGAN
0.0718
0.0691
4.90
RelGAN
0.0462
0.0408
3.49
CaptainGAN
0.0120
0.0184
4.04
Note: All the GANs, except CaplainGAN, are pretrained by MLE.
0.10
0.10
ScratchGAN at various T
MLE
0.35
Validation Dals
CaptainGAN various
Proportion of samples
0.30
CaptalnGAN
0.25
0.20
Vol
0.15
0.10
0.02
0.05
0.00
0.00
0.6
0.8
1.0
1.2
1.4
8
10
12
lumporatuve
Longusl malching n-grams in lraining sol
(a)
(b)
Figure 4: (a) The influence of different temperature values on FED scores measured
EMNLP 2017 News. (b) Longest matching n-grams on EMNLP 2017 News training data.
5.4 MATCHING N-GRAMS OF TRAINING DATA
Text generating models are often criticized for simply memorizing texts seen during training instead of learning how model the true data distribution. We show the longest matching n-- grams between text from the training data and samples in Figure 4b, CaptainGAN generates more samples with small matching n-grams (n < 5) comparing to MLE model, implying that our model can generate novel sentences without resorting to memorizing phrases from the training data.
5.5 CAPTAINGAN AS A LANGUAGE MODEL
CaptainGAN's strong performance on the evaluation metrics described carlier shows that is capable generating realistic text. Since the generator in our case a RNN which can produce conditional probabilities at each limestep, this raises the question of whether or not an explicit language model is being learned.
We feed real data to generator and calculate the perplexity to measure the generator's suitability as a language model. Surprisingly, as shown in Table 2. the perplexity of the generator hoth training and validation data is unusually high. A similar phenomenon was noted in de Masson d' d'Autume et al. (2019), though not as extreme.
We explain this phenomenon follows: MLE models trained specifically for low per- plexity, since they minimize KL divergence. However, CaptainGAN minimizes alternative loss (Equation 2), which is similar to minimizing reverse KT. divergence (Arjovsky & Bottou,

---

Under review as a conference paper at ICLR 2020
2017). This objective assigns extremely low cost to mode dropping and does not force the generator to mimic all aspects real data. This can result in poor modeling, of the likelihood, but does not necessarily lead to poor sample generation. (Theis et al 2016
Hashimoto et a 2019).
Following 5.4 conclude that model's ability to generate realistic text cannot be the result of simply plagiarizing training samples.
Table 2: Word level perplexity on EMNLP 2017 News.
Model
Train Perp.
Val. Perp.
Random. MLE
5.3k
5.3k
34 14.9k
26
CaptainGAN
16.8k
Note: Perplexity is calculated as geometric mean over sentences.
5.6 COMPARISON TO SCORE FUNCTION-BASED METHODS
Our method is able to outperform score function-based approaches using small batch sizes. Score fumetion-hased approaches often rely on large batch sizes for variance reduction (de Masson d' d'Autume et al, 2019). but this does not appear to be necessary for our re- centered estimator. This confirms the reasoning at the end of Section 3.2. and greatly reduces memory requirements during training.
In addition, since gradients are naturally calculated for every timestep, there no need for Monte-Carlo estimation of rewards like in score funetion-based approaches. Because our generator receives a richer learning signal, we usee a 5:1 ratio for discriminator-generaaoo updating. This greatly lessens computational requirements.
5.7 ABLATION STUDY
To understand the impact of each component of the CaptainGAN, conduct ablation study. In Table g we show the influence of adding new features, namely spectral normaliza- tion, re-center pekimattor_pettrainng weight decay (Loshchilov & Hutter, 2018) and one-sided label smoothing (Salimans et al 2016), by scoring FED on validation data.
embeddings and discriminator regularization decouppee
We observe that:
As shown in Zhou et al. (2019), an unrestricted objective leads to an uninformative gradient. Without spectral normalization, the Lipschitz constant of the discrimina- tor is unbounded making difficult to update the generator with a gradient-basee approach.
Rather unexpectedly, pretrained embeddings do not lead to any significant improve- ment, which is surprising given method's dependence the discriminator's embedding space. The embeddings which the discriminator learns from scratch are capable of giving the generator effective learning signal.
The application of the re-centered estimator is responsible for a 20% improvement in FED, confirming its effectiveness.
Equation 14 is dependent on the norm of the embeddings, which explains the effec- tiveness discriminator weight regularization improving FED.
8

---

Under review as a conference paper at ICLR 2020
Table 3: Ablation study on EMNLP 2017 News validation data
Model
Val. FED
straight through estimator
0.383
+ spectral normalization
0.047
re-center estimator
0.037
pre-trained embeddings
0.035
+ discriminator regularization
0.018
CONCLUSION AND FUTURE WORK
In this work we have presented CaptainGAN, an effective gradient-based method for quickly training a text generating GAN without pretraining. Starting from the straight- estimator, derive re-centered gradient estimator that improves the qualityddvversse of generated texts as measured by various standard metrics. In future work, we plan to investigate our estimator in more theoretical detail and further decrease ils bias.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifong Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Vićgas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. FeosorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.
Martín Arjovsky and Léon Bottou. Towards principled methods for training generalive adversarial networks. ArXie, abs/1701.04$65 2017.
Aparna Balagopalan, Satya Gorti, Mathieu Ravaut, and Raeid Saqur. Regan: Re |ax] bar) inforce] based sequence generation using gans. arXiv preprint 1r0i::1805. 02788, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating propagating gradients through stochastic neurons for conditional computation. CoRR, abs/ 1408,233,, 2013.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. arXiv preprint arXiv: 1607.04666,, 2016.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. Xi preprint arXiv: 11066, 2018.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. Language gans falling short. CoRR, abs/1811.02553 2018.
Danicl Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Lyn Untalan Limtiaco, Rhomni St. John, Noah Constant, Mario Ceajardo-Céspedes,, Steve Yuan, Chris Tar, Yun hsuan Sung, Brian Strope, and Ray Kurzweil. Universal sentence encoder. In In submission to: EMNLP demonstration, Brussels, Belgium, 2018. In submission.
Tong Che, Yanran Li, Ruixiang Zhang, R. Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. Maximum likell--oieiioood augmented discrete generative adversarial networks. CoRR, hts//702.07995 2017.
Boxing Chen and Colin Cherry. A systematic comparison of smoothing techniques for sentence-level bleu. In Proceedings of the Ninth Workshop on Statistical Machine Trans- latson, pp. 362 367, 2014.

---

Under review as a conference paper at ICLR 2020
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evalu- ation server. CoRR, 0ass/504.0033 2015.
Cyprien de Masson d' d'Autmme, Mihaela Rosca, Jack W. Rae, and Shakir Mohamed. Training language gans from scratch. CoRR, abs/ has/1905.0999) 09992,, 2019.
William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: better text generation via filling in the_ arXiv preprint arXXi:1801.07730 2018.
Michael C Fu. Gradient estimation. Handbooks in operations research and management science, 13:575-616, 2006.
Peter W Glymn. Likelihood ratio gradient estimation for stochastic systems. Communica- tions of the ACM, 33(10):75- 84, 1990.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Parley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahra- mani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 2672 2680. Curran Associates, Inc., 2014.
Will Grathwohl, Dami Choi, Yuhuai Wu, GeofTrey Roeder, and David K. Duvenaud. Back- propagation through the void: Optimizing control variates for black-box gradient estima- tion. CoRR, abs/ 1iss1771.,012,, 2017.
Shixiang Gu, Sergey Levine, llya Sutskever, and Andriy Muih. Muprop: Unbiased back- propagation for stochastic neural networks. In 4th International Conference on Learning Representations (ICLR), May 2016.
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text gener-
ation via adversarial training with leaked information. CoRR, fbss/170.00822. 2017.
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text gener- ation via adversarial training with Icaked information. In AAAI, 2018.
Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. CoRR, abs/1904.02792, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel- softmax. arXiv preprint arXis:1611.0114, 2016.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint Xix: 10196, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for gener- ative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pallern Recogrilion, pp. 4401-4410, 4410, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:14 142::4412.6988 2014.
Matt J. Kusner and José Miguel Hernández-Lobato Gans for sequences of discrete elements with the gumbel-softmax distribution. ArXie, abs/1611.04051. 2016.
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-ting Sun. Adversarial ranking for language generation. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Pro- cessing Systems 30. pp. 3155 3165. Curran Associates, Inc.. 2017. URL http: //paprrs. pdf.
nips.cc/paaer//6908 aedteersarialrraniiina
1lya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normal- ization for generative adversarial networks. CoRR, abs/1880.009957, 2018.
10

---

Under review as a conference paper at ICLR 2020
Weili Nie, Nina Narodytska, and Ankit Patel. RelGAN: Relational generative adversarial networks for text generation. In International Conference on Learning Representations, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-jing Zhu. Bleu: method for auto- matic evaluation of machine translation. 10 2002. doi: (0.3115/1073003.1011 1073135.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances neural information processing systems, pp. 2234 2242, 2016.
Stanislau Semeniuta, Aliaksci Severyn, and Sylvain Gelly. On accurate evaluation of gans for language gencration. arXiv preprint arXiv: 10::::806.000 04966, 2018.
L. Theis, A. van den Oord, and M. Bethge. A note on the evaluation of generative models. In International Conference on Learning Representations, 2016. arXiv:1511.0184.
Ronald Williams. Simple statistical podiient-flllowing algorithms for connectionist rein- forcement learning. Machine learning, 8(3-4):229 256, 1992.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seggan: Sequence generative adver- sarial nets with policy gradient. In AAAI, 2017.
Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. Adversarial feature matching for text generation. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 4006 4015. JMLR. org, 2017.
Junbo Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M. Rush, and Yann LeCun. Adversar- ially regularized autoencoders for generating discrete structures. CoRR, abs/ 2os/1706.04223, 2017.
Zhiming Zhou, Jiadong Liang, Yuxuan Song, Lantao Yu, Hongwei Wang, Weinan Zhang, Yong Yu, and Zhihua Zhang. Lipschitz generative adversarial nets, 2019.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. SIGIR, 2018
11

---

Under review as a conference paper at ICLR 2020
APPENDIX A NOTATION
Symbol
Meaning
=
(M).
Tol
a predefined vocabulary
token belongs to V a token sampled from V a sequence of tokens belong V a sequence of tokens sampled from V a transpose operation
X
APPENDIX B THE IMPORTANCE OF KERNEL
To verify the necessity of the kernel term (Equation
14), we directly apply Equation to
Equation 11 as follows:
OV.
(p)
Σ NEV
(r
(ex))
de
00
Va
(c+)) (p)(2)
Σ øEV
(rx
de
00
OVG
ppo(a)
Σ zEV
(e
(e+))
(16)
de
00
where & is the selected token and x a tokens in the vocabulary V.
Let f(x) = (e,, 8Ve the summation.
de (ea)) and (ea,
aVe
(e=)). does not depend on and constant in de
E(1()) + b) (p)(2)
00
xEV
8
(f(x)) + b)po(x))
0A
f(x)po(x)) + +b6 Σ po (x))
0A
sEV
zEV
f(x))o(x)) + 1
30
ÆEV
a
Σ f(x)po(x)
(17)
00
ZEV
Without the kernel, the update to 0 will be the same as prior work and thus carries the same drawback.
APPENDIX C MODEL STRUCTURE
We use a pre-trained word embedding of 300 dimensions to initialize all embedding weights. The embeddings are pre-trained using the fastTexx library Bojanowski et al, (2016) on the corresponding training data of each task.
12

---

Under review as a conference paper at ICLR 2020
Generator Gg
GRU
Linear
g
Discriminator
D p
Embedding Layer
GRU
Linear
Forward Propagation
Backward Propagation
Estimated Backward Propagation
Figure 5: Model architecture and computation flow
embedding lookup
GRU 1024
densse de dense V
Table 4: Generator structure, dg is the embedding size, and the vocabulary size.
C.1 GENERATOR
The purpose of the first dense layer is to project the GRU cell outputs into the embedding space. The second dense layer converts the projected embedding vectors to logits over the the transpose of the embedding weights.
tokens and is initialized
At the first timestep, the input to the embedding layer is the start-of-sentence token. The vocabulary also contains end-of-sentence token. If the generator outputs this token at any timestep, the sentence ends and its length is set to this timestep.
C.2 DISCRIMINATOR
embedding lookup
conv3-512 relu conv4-512 relu avg pool2
conv3-1024 relu
conv4-1024 relu
global average pool
dense 1024 relu dense
Table 5: Discriminator, all conv & pooling layers use 'same' padding.
To perform consistent convolution sentences of different lengths, mechanism on all layers. For all convolution and pooling layers, we mask out input features where the receptive field is out-of-bounds with respect to the unpadded input sentence.
special masking
13

---

Under review as a conference paper at ICLR 2020
APPENDIX D EXPERIMENTAL DETAILS
D.1 SETUP
D.1.1 DATASETS
We two datasets COCO Image Caption dataset Chen et al. (2015) and EMNLP 2017 News datased Results are reported on EMNLP 2017 News if not specified. For each datasets, 10k sentences are set aside as validation data.
COCO Image Captions. Sentences are limited in length to 24 tokens. The vocabulary size 4.6k tokens. Training data consists of 10k sentences
EMNLP 2017 News. Sentences in this dataset are much longer and more complicated, with maximum length of 50 tokens and a vocabulary of 5.7k words. Training data consists of 300k sentences,
D.1.2 ARCHITECTURE AND TECHNIQUES
Our architecture is described Appendix G Spectral normalization of the discriminator's weights Miyato et al (2018) is critical to our method as without it the gradients are too unstable and convergence becomes impossible (Section 5.7
D.1.3 TEMPERATURE CONTROL
Following Caccia et al. (2018). we adjust the softmax temperature parameter at sampling time to measure the trade-off between quality and diversity. Increasing the temperature lowers the differences between softmax probabilities, which leads to diverse but low-quality samples. On the other hand, reducing the temperature leads to high-quality yet low-diversity samples.
D.2 TRAINING DETAILS
The discriminator and generator updates are performed with a 5:1 ratio. We use Adam Kingma & Ba (2014) with learning rate -0 f 0.5, and B2 0.999. Decoupled weight decay regularization Loshchilov & Hutter (2018) is applied to all variables in the discriminator, and X 0.05 learning rale. All our models trained using NVIDIA
GTX 1080 Ti.
To evaluate samples BLEU-5, LM versus RLM, and FED, tences as reference texts for both COCO and EMNLP 2017 News. For Self-BLEU, we randomly select half of the 10k samples as reference texts and leave the remainder target texts. A smoothing function is used on BLEU-based metrics. See Appendix D.4 for more details. The maximum likelihood model architecture this paper follows the setting of de Masson d' Autume et al (2019). We could not reproduce the exact results reported in de Masson d' A'lume et al. (2019), which we believe stems from discrepancies between our Trained language model and theirs.
10k validation
D.3 ONE-SIDED LABEL SMOOTHING
One-sided label smoothing has been shown reduce the vulnerability of neural networks to adversarial examples and it is strongly recommended for GAN training Salimans et al. 2016). Therefore, the first term Equation reformulated as
Exopanea (a log (x (1 a) log(1 (x))
(18)
where 0.9 in experiments.
http://wwwstttttorrtooo
The preprocessed COCO dataset available at https: https///gittabb..cmmppluuaaaa
Gangaallaggaot//rrreaaaatterrree http://eeaadddataeeeee httpaaaadddeeeeeeeeeeeetetaaaaaa
'The preprocessed EMNLP dataset is available at
14

---

Under review as a conference paper at ICLR 2020
D.4 BLEU SMOOTHING
One of the issue with BLEU that in the case that a higher order n-gram precision of a sentence is then the BLEU score will be O, resulting in severely underestimation. This is due to the fact that BLEU calculated by the geometric mean of precision. To solve this, applied the smoothing technique follows:
n = c if m, =0
(19)
where mh is the original match count, my is the one used during B BLEU calculation. We've 've picked the smoothing factor 0.1 as proposed by Chen & Cherry (2014).
APPENDIX E COCO RESULTS
10
ReIGAN
Cap at vartous T
CaptainGAN various
MLE various
log(RL M. Perplexaty)
8
training data
Self BLE15
validation data
LeakIAAN
Sen MalIGAN
6
0.4
0.3
training data
4
0.2
0.1
2
-0.30
-0.25
-0.20
-0.15
-0.10
-0.05
0.00
2
4
6
8
10
BLEU 5-negative
log(LM Perplexity)
(a) Negative BLEU-5 versus BLf--,
(b) Language- and reverse language-model scores.
Figure 6: BLEU scores (left) and language model scores on (right) COCO. Results for LeakGAN Guo et, al. 2017), MaliGAN et al. 2017), RankGAN Lin et al, 2017) and SeqGAN Yu et al. 2017) are from Caccia et al. (2018).
See Figure
APPENDIX F SENTENCE LENGTH CONSISTENCY
The sentence length samples which are generated by CaptainGAN is highly related to the training data, shown in Figure 0 This suggests that CaptainGAN is able to fully learn the distribution of sentence length by gradient estimator without additional mechanism like positional encoding or dense reward.
APPENDIX G DETAILS ON FRÉCHET EMBEDDING DISTANCE
Caleulation of the FED score depends on the Universal Sentence Encoder Cer et al (2018) However, the version of Universal Septence Encoder used in de Masson d'Autume et (2019) is incompatible with Tensorflow/Abet et al. (2015) version 1.7 or newer. Furthermore, would be infeasible limit the experimentation environment Tensorfloy. Therefore, as a work around, we decide to report our FED score, as shown in Table C. with Universal Sentence Encoder Largel instead, which is compatible with the current version of Tensorflow (1.14 as of Jul 2019).
such ancient version of
While there are many benefits to using the FED metric, it not without drawback. A drawback that we've observed is that the FED score will be significantly underostimated
15

---

Under review as a conference paper at ICLR 2020
0.10
CaptainGAN
Training data
Normalizod Count
Validation data
0.06
0.04
0.02
0.00
40
45
50
Sentence Length
Figure 7: Normalized counts of different sentence length on EMNLP 2017 News.
Table 6: FED scores EMNLP 2017 News based on Universal Sentence Encoder Large
Model
Train FED
Val. FED
Training data
0.0059
0.0136
MLE
0.0130
0.0244
SeqGAN
0.1333
0.1414
MaliGAN
0.1392
0.1412
RankGAN
0.1554
0.1474
LeakGAN
0.0884
0.0878
ReIGAN
0.0504
0.0514
CaplainGAN
0.0164
0.0232
whenever we are using a small number of samples, as shown in Figure Therefore, we're 've ensured that sufficient samples are used in our evaluation.
APPENDIX H GENERATED SAMPLES
Training samples of COCO and EMNLP 2017 News be found in Table Randomly picked samples from MLE model and CaptainGAN are available in Table and Table
16

---

Under review as a conference paper at ICLR 2020
0.018
0.016
FED score
0.014
0.012
0.010
Number of sentences
Figure 8: FED scores on different sampling size of training data and CaplainGAN samples.
COCO
a person standing in a boat on the water
three sheep grazing the city sidewalk.
kenya airways airplane wing engine and cabin on the tarmac
light poles in the snow with yellow traffic lights mounted on them
we are looking at the floor between the toilet and the wall
somcone taking a photo of small residential bathroom
cat sits on the seat of bicycle and looks down at another cat snowy day large set of motorcycles all lined up down a street.
EMNLP 2017 News
apple clearly doesn t want to see the dollar value of its k earnings fall by similar amount
do look at the number of hours we produce and measure that against the religious make- up of socicty..
the u. has struggled to find an effective ground force take on isis in syria where president obama has ruled out a u. 8 ground combat role is a happy and kind world that we live in on this show and that where hope we can live in real life..
while men and women may end up earning roughly the same amount in the same jobs men are more likely to end up in higher paying roles in the tech industry..
but while they were beaten by a better side the tie did reveal what i think has been city s biggest problem this season they have lost the ability to score against good teams according to facebook s policies, accounts can suspended if law enforcement believe individuals are risk of harm
she has to take personal responsibility for this when she was health secretary she was told by those who know best that her decision to cut student places would have a damaging impact.
Table 7: Training samples FMNLP 2017 News and COCO dataset.
17

---

Under review as a conference paper at ICLR 2020
COCO
the bathroom bowl with locus on the vanily. a man racer in a car brushing a carved cake
lady riding motoreycle and street while sitting pedestal track
a commercial man are standing in the bicycle next to the back of an orange computer airplane the bus is headed something in the sky people riding on a bike on a surfer surfing doing the ocean..
plane propped up in front of vehicle
a man wearing a cat in front of a motorcycle.
EMNLP 2017 News
one game event and a limited number this dropped by 38 per cent to find out what we could want, to be much done as we finish as the games
more than 30 per cent of calais camps at risk were laken between children and child refugees crossing the mediterranean to fall apart.
the uk government got on the deal when came to other eu countries that allowed workplace checks for 26
so women didn t want my parents to work hard with me at the time of the task of making it presents.
black voters feel there may not be enough choice between former president george bush and i have to go along with problems and support the people that are in swing state for president.
he sold out to its eighth list but a stunning showing that what cost the ability to start the league failed ?
let s catch him down his back phone one morning john f kennedy where he in a us meeting group with supporters of love not being successful in the fields.
then there a couple of chances that they won competition the result from what of me going out there that really will play for the foundation..
Table 8: Randomly selected MLE samples on EMNLP 2017 News and COCO with temper-
ature = 1.0.
COCO
black motorcycle parked next to a river. bathroom with a mirror, a shower and toilet.
two people sit on a building in front of a building
group of people walking around the corner of building.
people are riding bikes on a city street...
there is picture of a bathroom in bathroom an empty city street with cars parked on the side of the road man taking a picture of white bathroom
EMNLP 2017 News
- if i go to the party, it disaster, and i going to keep it right - think i ve had some great strength in my life and want that to be here for me. -obama pointed to reporters claiming he was willing to discuss the situation but the
s. have been military concern by american troops including a muclear deal in march
-a 16 year old man was arrested after the death of man shot and arrested him on suspicion on his behalf.
the problem is that they are hard to find out what they want to see where they are and what they can achieve their past two years
now it s important to me and am thinking so hard to make it the best for a long time great britain is suggesting the u.
led coalition figures at the australian foreign investor have raisod more than 250. 000 in the construction of operations
it is simple way to believe that if it can happen then i m not the right choice
Table 9: Randomly selected CaptainGAN samples on EMNLP 2017 News and COCO with lemperature = 1.0.
18


Reviews:
[{"review_id": "rkl9nFTF9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes to train a GAN on discrete sequences using the straight-through Gumbel estimator introduced in Jang et al. (2016) in combination with gradient centering. The proposed approach is evaluated on COCO and EMNLP News in terms of BLEU and Self-BLEU scores, Fréchet Embedding Distance, Language Model Score, and Reverse Language Model Score.\n\nMy assessment is that the submission is below the acceptance bar, mainly due to clarity and novelty concerns. The proposed approach does have empirical backing, but I would argue that it is a very straightforward application of the straight-through Gumbel estimator to GANs, which is itself similar to existing work on applying the Gumbel-softmax estimator to GANs (Kusner & Hernández-Lobato, 2016). Detailed comments can be found below.\n\nThe submission does not feel self-contained. For instance, it borrows notation from Jang et al. (2016) without explicitly acknowledging it, and my personal experience is that reading Jang et al. (2016) beforehand makes a big difference in terms of clarity in Section 2.2.\n\nThe notation is inconsistent and confusing, and gets in the way of understanding the proposed approach. Here’s a (non-exhaustive) list of examples:\n\n- The reward function is first introduced as f_\\phi(\\mathbf{x}) above Equation 3, but all subsequent mentions of the reward function use f_\\phi(\\hat{\\mathbf{x}}).\n- The \\mathbf{m}_\\theta variable is introduced in Equation 5 and is immediately replaced with \\mathbf{p}_\\theta, which adds notational overhead without any benefit.\n- The difference between \\hat{\\mathbf{x}} and \\hat{x} is not explained in the text. From the context I understand that \\hat{x} is a categorical scalar in {1, …, V}; is this correct?\n- In Equation 6, x_1, …, x_V are used to denote the *values* that \\hat{x} can take. This clashes with the previous convention that \\mathbf{x} is a sequence sampled from p_{data} (Equation 1). Given that convention and the difference between bolded and non-bolded variables discussed above, I would have expected that x_1, …, x_V would correspond to the categorical values of elements of the \\mathbf{x} sequence. That contributes to confusion in Equation 9, where \\mathbf{e}_{x_t} and p_\\theta(x_t) are *not* time-dependent.\n- Equation 8 sums over time steps, but the first summation that appears in Equation 8 does not make use of the temporal index. There is also a symbol collision for T, which is used both as the sequence length and as the \"transpose\" symbol.\n\nAs a result, the proposed centering method and the rationale for it is still not entirely clear to me. In particular, is the gradient centering approach necessary to avoid the drawback of score function-based approaches (i.e. the generator is only given feedback on the tokens it samples), or does the non-centered, straight-through variant of the proposed approach also avoid this drawback?\n\nI’m also not convinced that the centering heuristic is a crucial component of the proposed approach when the biggest improvement observed over the straight-through baseline is obtained by adding spectral normalization. I would argue that the proposed approach is a straightforward application of the straight-through Gumbel gradient estimator to GAN training, which is similar in spirit to work by Kusner & Hernández-Lobato (2016) (not cited in the submission) -- the main difference being that the latter uses the Gumbel-softmax distribution directly and anneals the temperature parameter over the course of training. A comparison between the two would be warranted.\n\nReferences:\n\n- Kusner, M. J., & Hernández-Lobato, J. M. (2016). GANs for sequences of discrete elements with the Gumbel-softmax distribution. arXiv:1611.04051.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HkxIxHwaFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper attempts to solve the problem of non-differentiable connection between the generation and discriminator of a GAN. The authors come up with an estimator of the gradient for the generator from the gradient of the discriminator, which was disconnected previously. With this change, the model should be able to  select better tokens than random selection, which could leads to more robust training. The experiment results on both COCO Image Captions and EMNLP 2017 News datasets justify the authors' argument. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HJgZb4nntr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose CaptainGAN, a method using the straight-through gradient estimator to improve training of the generator for text generation.\n\nThe paper is well-written and the evaluation seems thorough, comparing to relevant baselines.\n\nComments:\n\nFigure 3: the caption refers to Caccia et al. for results on LeakGAN, MaliGAN and seqGAN, but unless I’ve missed it, RelGAN hasn’t yet been introduced by name as a baseline? The citation is given in the opening part of the introduction, in an enumeration, but isn’t revisited later in the text - not even here where the results of the model are introduced. Given that it seems, according to the presented results, to be the most competitive of the GAN models that the authors are comparing to, maybe it’s worth adding more contextual information on RelGAN to the Background section?\n\nFor their method, the authors should report an average performance over several random seeds and provide the standard deviation / confidence intervals, for the readers to be able to assess the stability of the method and the significance of the improvement reported in the results.\n\nI find Section 5.5. particularly interesting, as well as the reported perplexity in Table 2. The authors provide 3 bullet points to explain the unusually high perplexity of the generator on the training and validation data. I feel that the explanations that are given are at the moment vague and not visibly backed by data, therefore being speculative. Obviously, point 1) is hard to quantify - but point 2) could possibly be at least partially quantified - if the hypothesis is that names, places, punctuation marks etc play an important role in the reported perplexity score, then maybe the authors could test this by correlating model perplexity on sentences with whether those sentences contain these types of words?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to train generative adversarial nets for text generation. The paper proposes to address the challenge of discrete sequences using straight-through and gradient centering. The reviewers found that the results on COCO Image Captions and EMNLP 2017 News were interesting. However, this paper is borderline because it does not sufficiently motivate one of its key contributions: the gradient centering. The paper establishes that it provides an improvement in ablation, but more in-depth analysis would significantly improve the paper. I strongly encourage the authors to resubmit the paper once this has been addressed.
Decision: Reject