A Text GAN for Language Generation with Non-Autoregressive Generator
Authors: Fei Huang
Jian Guan
Pei Ke
Qihan Guo
Xiaoyan Zhu
Minlie Huang
Venue:  (2021)
Keywords: language generation
non-autoregressive text generation
generative adversarial networks
GANs
latent variable

Abstract: Despite the great success of Generative Adversarial Networks (GANs) in generating high-quality images, GANs for text generation still face two major challenges: first, most text GANs are unstable in training mainly due to ineffective optimization of the generator, and they heavily rely on maximum likelihood pretraining; second, most text GANs adopt autoregressive generators without latent variables, which largely limits the ability to learn latent representations for natural language text. In this paper, we propose a novel text GAN, named NAGAN, which incorporates a non-autoregressive generator with latent variables. The non-autoregressive generator can be effectively trained with gradient-based methods and free of pretraining. The latent variables facilitate representation learning for text generation applications. Experiments show that our model is competitive comparing with existing text GANs in unconditional text generation, and it outperforms existing methods on sentence manipulation in latent space and unsupervised text decipherment.

Under review as a conference paper at ICLR 2021
A TEXT GAN FOR LANGUAGE GENERATION WITH NON-AUTOREGRESSIVE GENERATOR
Anonymous authors
Paper under double-blind review
ABSTRACT
Despite the great success of Generative Adversarial Networks (GANs) in generat- ing high-quality images, GANs for text generation still face two major challenges: first, most text GANs are unstable in training mainly due to ineffective optimization of the generator; and they heavily rely on maximum likelihood pretraining: second, most text GANs adopt autoregressive generators without latent variables, which largely limits the ability to learn latent representations for natural language text. this paper, we propose novel text GAN, named NAGAN, which incorporates a non-autoregressiive generator with latent variables. The non-autoregressivee genera- tor can be effectively trained with gradient- based methods and free of pretraining. The latent variables facilitate representation learning for text generation applica- tions. Experiments show that our model is competitive compared with existing text GANs unconditional text generation, and it outperforms existing methods on sentence manipulation in latent space and unsupervised text decipherment.
INTRODUCTION
Generative Adversarial Networks (GANs) Goodfellow et al. 2014) have achieved great success in generating continuous data, such as images with high resolution and fidelity (Brock et al. 2019). Unsurprisingly, GANs are also widely studied for text generation, but the adaptation is by no means trivial. The mainstream text GANs (Yu et al. 2017b] Guo et al. (018)) apply a different framework tailored for discrete sequence data, but there are remaining several unsolved research problems.
One problem lies in ineffective optimization. Most text GANs resort to gradient-free RL (reinforce- ment lcarning) algorithms, mainly due to the nature of discrete text data. However, since RL methods abandon the gradient information, they suffer from unstable training processes (Ke et al., 2019). Though some works (Chen et al., 2018) explored the feasibility of gradient- based methods, the optimization is still ineffective. As result, most text GANs heavily rely on MLE pretraining, and some even report worse performance after GAN training (Caccia et al.) 2020).
Another problem can be attributed to the generative model. Most text GANs adopt an autoregressive generator, which defines an explicit likelihood without any latent variable. Latent variables have empowered image GANs with various applications, such unsupervised style transfer (Taigman ctt al. 2017) and image editing (Brock et al., 2017). However, most text GANs merely generate sentences from the learned distribution with autoregressive decoding, thereby hardly applicable to text style transfer or controlled text generation which may require latent representations.
We therefore challenge the conventional design of existing text GANs and argue that incorporating non-auttoeeresssvve generator can benefit from both efficient gradient- gradient-based optimization methods and the use of latent variables. Our proposed model, named Aon-Auvoregressive GAN (NAGAN), consists of a non-autoregressivve generator and a regularized discriminator. The non-autoregresssiee generator naturally translates latent variables the tokens in parallel, and the gradient-basee optimization on our feed-forward structure is significantly more effective than the same method on an autoregressive generator. The discriminator is regularized by Max Gradient Penalty (Zhou et al. 2019). which is another key to effective optimization. Our contributions are summarized as follows:
We propose -onaeutorerressvve GAN, which characterizes itself by employing non-autoregressive generator and latent variables, and efficiently training from scratch with gradient- based methods. To our knowledge, NAGAN is the first text GAN which learns latent representations from scratch.

---

Under review as a conference paper at ICLR 2021
Table 1: Differences between various GANs. Autoregressive using autoregressive generators.
Explicit: using explicit generative models. Latent: equipped with latent variables. Pretrain: pretraining required or not. The mainstream TexIGANs include Yu et (2017b); Guo et al. (2018);
Shi et al. (2018); Che et al. (017) Lin et al. (2017); [edus al. (2018).
Model
Data
Antoregressive
Explicit
Latent Optimization Gradientt-hsed Gradient-busee
Pretrain
Vanilla GAN (Goodlellow al. 2014
Image
K
SOTA ImigeGANN (Brock et 2019
Image
x
Mainstream NexIGANN []]
Text
RL RL Gradient-
ScratchGAN (de Masson Autume et 2019
Text
ReIGAN (Nie 2019
Text
FMGAN (Chen 2018
Text
Gradient-basee
NAGAN((Urr)
Text
Gradient- based
We conduct experiments on synthetic and real data, and show that NAGAN without MLE pretrain- ing is competitive in unconditional text generation compared with the pretrained text GANs.
By taking advantage of the latent variables and the non-autoregrressive generator, our model can be applied sentence manipulation in latent space and unsupervised text decipherment, where NAGAN significantly outperforms previous methods.
MOTIVATION OF NON-AUTOREGRESIVE GENERATION IN TEXT GANS
Non-autoregressive (NAR) generators have been widely used image GANs (Goodfellow et al. 2014) and achieve great success in generating high-quality images (Brock et al.) 2019). However, for generating texts, text GANs apply a different framework where autoregressive (AR) generators are used. In this section, we will discuss the differences between image and text GANs, and show why we need non-autoregressive generators in text generation.
2.1 GENERATIVE MODELS
Image and text GANs differ substantially in generative models. The image GAN is an implicit generative model, where sample generated in steps:
p(z);
x G(zz
(1)
where z a latent variable, and p(z) is the prior distribution. G is a deterministic function from the latent space to the data space, usually parameterized by NAR generator, where each pixel of x is generated simultaneously.
A text GAN is usually an explicit generative model. A discrete sequence sample [m), 222 ... ,rr. sampled by stochastic process from the distribution (ar), where
Pa(x) II
(2)
Eq (2) shows an autoregressive model, where tokens are sampled sequentially conditioned on previously generated prefixes. Most text GANs do not adopt latent variables, partially because Eq
(2) is convenient for the MLE pretraining. There are also some explorations (Chen et al. 2018) on
equipping text GANs with latent variables, but elaborate VAE-like pretraining is required.
However, directly injecting latent variables into the explicit generative model of text GANs can be problematic. For instance, when defining Pa(x) = Eg>p(z)) 11 Pa(0,/=<> 2). we may face two problems: (1) Solely optimizing Pş does not make the model learn the latent representations. Without further constraints" PI
the model can degenerate and simply ignores 2, which becomes almost vanilla language model, even if the generation distribution Pg perfectly fits the real distribution. (2) The representation in AR text GANs can be hardly applied to downstream tasks. In image GANs, x fully determined by the sampled z, so latent variable. Moreover, the mapping from the latent space to the data space is continuous and differentiable. thereby facilitating applications such as image editing (Brock et al. 2017). In text
control the generated images by manipulating the
FMGAN uses explicit generative models in pretraining and implicit ones in GAN training. For example, the reconstruction loss VAFs alleviates degeneration. More discussed in Appendix A.I.I
2

---

Under review as a conference paper at ICLR 2021
GANs, translating to is still stochastic process when the latent variable is fixed, indicating that the generated sentences may not be entirely controllable. The non-differentiable generator can also be an obstacle for downstream applications.
Based on the above reasons, we explore the feasibility of adopting an implicit generative model with NAR generator for text GANs. The NAR generator can be applied to various applications conveniently and will not degenerate into a language model (more discussed in Appendix AA.I.
2.2 OPTIMIZATIONS
In image GANs, the generator Go and the discriminator D& play minimax game: min max VDD,, Ga) = Ex (x) (log (x)x)] + E2-p(e) [log(1 - Do(GG(z))))
(3)
0 &
Since Go and D& are fully differentiable, they can be optimized alternately by gradient- based methods. In the text GANs, there is usually no latent variable, and the minimax game becomes
min max V (o) Paetua (ax) llg [a(x)) Ez--o (m) [log (1 Dz(2))).
(4)
0
The discriminator D& can be trained by gradient- based methods, but the output of Ge is discrete, so the gradient cannot be passed from De back to Go. Instead, the generator Go can be optimized by the gradient-free REINFORCE algorithm (Williams 1992):
VoE (a) [R(z)] = Eæ-Ps, Ps, (x) R(x) Vo log Ps.())),
(5)
where R(x) is a reward function measuring the fitness of generated sequence. If R(xx) = log(1
D(x)), it recovers the second term of Eg (4). Existing text GANs also devise various rewards tailored
for text data (Guo et al. 2018 Shi et al. 2018 Lin et al. 2017 Fedus et al., 2018: Xu et al.; 2018).
Gradient-free RL algorithms naturally suffer from high variance. In text GANs, the huge action spaces and the changing reward function"
the problem, so most text GANs heavily rely on the MLE pretraining. Some text GANs improve the training algorithms from the perspective of RL (Guo et al.] 2018 Shi et al. 2018), but recent empirical studies show the RL training does not always improve the performance over the pretraining (Caccia al. 2020 Semeniutal et al. 2018). ScratchGAN (de Masson d' Autume et al. 2019) devises improved RL techniques, for the first time freeing text GANs from MLE pretraining by increasing the batch size (-10x) and computation cost.
Some works (Kusner& & Hernández-Lobato/ 2016 Chen et al. 2018) explore the gradient-based methods and adopt continuous relaxations gradient estimators for discrete text data. However, these models still rely on pretraining since the optimization is ineffective as well.
Different from existing works that focus on improvements of RL or the approximation methods, we find that one reason for the ineffective optimization may come from the generator architecture. To obtain gradients, an autoregressive generator with the gradient-based optimization method has to apply the gradient estimator recurrently, because the generator reads discrete tokens inputs. In other words, the gradient flow from the last token will be approximated multiple times when it reaches the start of the sequence. On the contrary, non-autoregressive generator has feed-forward structure, and only one approximation is required at the last layer of the generator.
2.3 NON-AUTOREGGEESSI TEXT GENERATION
-on-autoregressvve text generation is first proposed for machine translation to generate the whole sequence in parallel with low latency (Gu al. 2018 Ma et al., 2019). Most non-autoregressive generators are trained with cross-entropy loss, which assumes the tokens are conditional independent of each other when the input given. As a result, the non-autoregressive generators can only be applied to the tasks where the outputs can be fully determined by the inputs (c.g., machine translation). and they are failed to capture complex distributions, known as the multimodality problem (Gu et al.. 2018). Most non- aon-autoregresive models require knowledge distillation to reduce the dataset complex- ity, otherwise they will experience a serious performance drop (Zhou et al. 2020h. We assume that a key problem behind the phenomenon is the unreasonable independence assumption, so we replace 'The reward estimated by the discriminator, and keeps changing during training even for the same ar.
3

---

Under review as a conference paper at ICLR 2021
A Generator
B Discriminator
D
Argmas
Straight Through
Onehot
Estimator
MLP
S;
St
Max Pooling
MLP
MLP
MLP
MLP
Rg
h2
Rg
Transformer
Transformer
Positional
Difforentiable
Positional
Encading
Encoding
Non- iffrentiable
2
2z
Gradient Estimator
O2
O2
12 ..... z Figure 1: Architecture of NAGAN. (A) The generator converts the latent variable Z
to sentence O = 02102... ..]]. where each o a one-hot vector. The gradient of non-differential operation is estimated by the straight through estimator. (B) The discriminator produces score D(O) for the sentence O. The gradient from the discriminator can be passed back to the generator.
the cross-entropy loss with the GANs objective. The GAN objective considers the whole sequence and punishes implausible sentences, which does not rely on the independence assumption. In this paper, we explore the feasibility generating diverse sentences with a non- non-autoregressive generator.
NON-AUTOREGREESSVVE GAN
As shown in Figure our proposed non- non-autoregressive GAN (NAGAN) consists of non- -anautrreeres-
sive generator with latent variables and a regularized discriminator. The framework is very similar to image GANs but differs from the mainstream text GANs substantially in two aspects: (I) We use implicit von-auveregressive generative models equipped with latent variables, (2) We use the gradient-based optimization method with Max Gradient Penalty stabilize the training process.
3.1 GENERATOR
Our implicit generative model is defined by sampling process. First, we sample the latent variable from the prior distribution. Instead of single latent vector, we sample sequence of continuous latent variables Z |2| z1), where L is desired length of the target sequence and each 2, is sampled from AO f) independently for S t s L. This idea is inspired by Ma et al. 2019). which provides a good way for the non-autoregressive generator to leverage the latent variables. Then, non-autoreerressive generator G converts Z sequence O [010g oL). which is the one-hot representation of the generated sentence. We use one-hot representations here for the convenience of obtaining gradients, The process can be formulated as follows:
Z P(Z);
O G(Z).
(6)
g is implemented by a Transformer network (Gu et al. h2 hz] Transformer((z))
2018 Vaswani et al. 2017):
=1]),
(7)
8, MLP(hh) RVV O: onehot(arg max, (ax,
where V is the size of the vocabulary, 8t,r the e-th dimension of st, and o, (0,
Straight-Through Estimator. The argmax operation is con-differentiable such that we cannot pass gradient from o, back to the parameters of G. Previous work (Kusner & Hermández-Lobato 2016)
has introduced the straight-through estimator (Bengio et al. 2013) to solve the problem. Given
scalar temperature if r 0, we have da, x dsofimax TT)
Note that the straight-through estimator
Te:
is only used for obtaining gradients, and the forward pass remains unchanged as Eq (8)
*T different for each sequence. See details in Section 3.3 and Appendix A.1.2
4

---

Under review as a conference paper at ICLR 2021
3.2 DISCRIMINATOR
Similar to the vanilla GANs, the discriminator is trying to distinguish the real sentences and the gen- erated sentences as a binary classification, Once receiving the generated sentence O = 00 ... or.). the discriminator produces O()) score for the sentence. We simply choose Transformer for our discriminator. Some other architectures are discussed in Section 4.2
rI. rT. (O))
Transformer
o,.)).
(9)
(210 rL)))
(10)
Max Gradient Penalty. The gradient vanishing (Arjovsky al. 2017) problem is critical factor for causing the unstable training process in image GANs. When an unregularized discriminator is per- fectly trained, the gradient becomes zero and cannot provide signals for the generator's optimization. We borrow common training technique named Max Gradient Penalty from image GANs (Zhou al. 2019). The technique restricts the Lipschitz constant of the discriminator, which is proven effective in gradient-bar based GAN's optimization. Max Gradient Penalty can be formulated as follows:
OD(O') 2
LGP max O'
(11)
80'
where O' are sampled uniformly along the straight lines between pairs of points sampled from the real data and the generated data (Gulrajani et al. 2017).
3.3 TRAINING
The discriminator and the generator will be optimized alternatively, where the discriminator tries to distinguish the real and generated sentences, and the generator is updated to achieve a higher score ())(
We utilize the GAN objective with non-saturating loss (Goodfellow et al. 2014).
Lp = -BrePaa Pun log D(onchot(z)) - Ez-r(z) log ())(
(12)
Lg = -E -Ez~r(z) log (Z)).))
(13)
We further apply Max Gradient Penalty on the objective, and the loss of the discriminator becomes f = LD + ALap, where À is hyper-parameter balancing the regularization terms. This objective can be regarded as special case of Lipschitz GANs (Zhou et al. 2019).
With the straight- and LG can be optimized alternately by gradient-based methods. The training steps are consistent with the vanilla GANs but essentially different from previous text GANs: we do not use MIE pretraining sampling techniques devised for RL.
estimator, both the generator and the discriminator are differentiable, so C'p
Predicting the Length. Non-autoregressive generators require a sentence length before generation in parallel. Existing non-autoregressive generators (Gu et al. 2018) usually leverage a classifier P(L[C) to predict conditioned on the input C (e.g., the source sentence in machine translation). In unconditional text generation, directly estimate P(L) by counting the lengths of the training samples since C not provided. During training and inference, is sampled from the same PLL).
3.4 APPLICATIONS
Manipulate Sentences in Latent Space. To show the utility of latent variables, we introduce sentence editing task: given a sentence (whose latent representation is Z.) which contains source word, the task aims to obtain new latent representation Z,, which can translated to sentence that contains target word. Some cases shown in Table 6 A previous method (Zhao al. 2018) modifies the latent variable with an offset vector, which is obtained by subtracting the mean latent variable of the source word from that of the target word. This method ignores the context around the edited word and thus leads to a low success rate. More details are deseribed in Appendix A..5
NAGAN use the gradient information to make specific modification for different contexts. If we want to modify the t-th word x, in the source sentence to a desired word x', we can apply gradient descent the latent variable maximize the -' dimension of o, in Eq One descent step can
5

---

Under review as a conference paper at ICLR 2021
be formulated as Z, : Z, + 2 where is a small weight. The process will be repeated until success or exceeding pre-specified iteration number. Note that the two methods are only applicable to the models with latent variables, where most text GANs are not applicable in this task.
Unsupervised Decipherment. Our model can apply to unsupervised decipherment (Yang al. 2018) within the framework of CycleGAN (Zhu et al. 2017). The task provides two unparallel corpora: the plaintext X and the ciphertext y. where we aim to learn mapping Fy-x to decrypt the ciphertext. In order to adapt NAGAN to this task, we introduce an encoder E' to encode the data to a shared latent space and define y == g (yy ey); ex), where y E 3/, ey and ex are labels indicating the type for the source and target text. In a similar way, Ex-y == ((F(x,, cx), ey). Following Dai et al. (2019). we utilize three losses: the cycle loss, the adversarial loss, and the reconstruction loss. We first define the three losses on X. and the losses on ) can be obtained similarly. The cycle where d is a distance function. The adversarial loss matches the real sample and the generated samples: Ladv, Eu~Px [D(x cx)] Ey--yy (D(Fy-x(y)) cx)]. The reconstruction loss helps the unsupervised encoder-decode training: F Ex-Px [d(ar, ((E(2,, cx), cx))]. The final objective sums three losses both X' and V, i.e. ming maxp Laye. + L + Oohadv + aaaddr + ,Lrree.x + [9Lren.y], where o, B are hyperparameters to balance the losses. In this task, NAGAN learns shared latent representations for two types of text, and then the alignment between two text spaces are strengthened by the cycle loss. We adapt NAGAN to this task with the non-autoregressive generator unchanged, where we further introduce an encoder and utilize cross-entropy based discriminator (de Masson d' l'Autmme et al. 2019). More details are described in Appendix A.6.
loss is defined as Laye, = Ex-Px [d(x:
EXPERIMENTS
Experiment Settings. We test NAGAN on both synthetic and real data. The synthetic data (vocab size = 500, length = 20) is generated by an oracle Hidden Markov Model with fixed parameters. We do not use LSTM as the oracle model, because it may make the evaluation biased to the LSTM generators. We extract the text from COCO image caption ( (vocab size = 4,839, max length = 32) (Chen et al. 2013) and SNLI (vocab size 42,981, max length = 40) (Bowman al., 2015) dataset as the real data, which are adopted by Guo et al. (2018): Semeniutal et al. (208)).
Dropout in NAGAN. We utilize dropout (Srivastava et al. 2014 in our generator, which introduces noises for generating diverse samples and stabilizes GAN training (Isola et al.] 2017 Sonderby et al. 2017). Note that dropout in GANs should also be applied inference, which ensures that the generated samples are from the same distribution during GAN training. However, we find that varying the dropout rate in inference is in fact balancing the quality and the diversity of generated sentences, which will be shown in Section
Evaluation Metrics. For the synthetic data, we calculate the oracle negative log- likeeliod (Oracle NLL). For the real data, adopt language model score (Caccia et al. 2020), n-gram based metrics (Shi et al. 2018), and Fréchet embedding distance (FED) (de Masson d' d'Autuee et al. 2019). In these metrics, LM score and BLEU, F measure quality, BLEU, n measures diversity, BLEU H and FED measure overall performance. The detailed definition are shown in Appendix A.4.1
4.1 RESULTS ON UNCONDITIONAL TEXT GENERATION
We first test NAGAN on the synthetic data. The chosen baselines include GRU model trained by MLE (Graves) 2013), two mainstream text GANs YYu et al.) 2017b Guo et al. 2018), and ScratchGAN (de Masson d' A'ttume et al., 2019). As shown in Figure 2 NAGAN obtains lower Oracle NLL than the baselines, particularly than the other models without pretraining (solid lines). LeakGAN without pretraining not shown because of too large scores (Oracle NI..=7.28).
On the real data, we test NAGAN against the MLE-trained GRU (Graves 2013), Transformer (Vaswani et al.] 2017), and the SOTA text GANs, The result is shown in Table 2 Our best model outperforms the pretrained baselines in LM Score and BLEUp. With respect to diversity, MLE- trained models are superior to all text GANs except ReIGAN in terms of BLEU H on SNLI dataset. The result suggests that GAN training in these pretrained baselines, particularly with RL optimization (SeqGAN, LeakGAN, IRL), harms the diversity and even the overall performance, which was also reported previously (Caccia et al. 2020
Semeniutal et al. 2018

---

Under review as a conference paper at ICLR 2021
Table Generation performance on COCO and SNLI datasets. Bold scores indicate the best perfor- mance in all models while underline scores are the best non-pretrained models.
Model
COCO
SNLI
Score
BLEU,
BLBU
BLEUHA FED
BLEU,
BLBU
BLEUNA
FED
Prefrained Models:
GRU GGrvees 2013
4.13
0.292
0.325
0.308
0.094
4.59
0.246
0.229
0.237
0.031
Trunsfurmee Furmer Vaswani el 2017
3.90
0.327
0.321
0.324
0.094
4.26
0,274
0.224
0.246
0.039
SeGGAN
0.298
0.285
0.291
0.108
0.293
LeakGAN Gloreral 2018
3.81
0.321
0.239
0.273
0.324
4.40
0.259
0.200
0.226
0.054
IRL SShi cl 2018
3.80
0.314
0.235
0.265
0.140
4.07
0,283
0.183
0.221
0.125
ReGGAN [Ne
2019
3.71
0.347
0.288
0.315
0.164
3.76
0.332
0.215
0.261
0.086
FMGAN (C lien el ale
2013
3.90
0.295
0.308
0.301
0.095
4.49
0,214
0.209
0.211
0.034
Non-preinaured MModeis:
SeçGAN w/o pretrain
9.06
0.019
0.021
0.020
0.812
6.99
0.148
0.065
0.090
0.262
ReIGAN w/o pretrain
7.45
0.040
1.092
0.009
0.003
0.004
0.584
FMGAN w/o pretrain
8.83
0.004
0.027
0.007
0.928
9.65
0.002
0.001
0.002
0.601
Scratch:GAN
4.13
0,278
0.251
0.264
0.127
4.57
0,237
0.192
0.212
0.062
NAGAN (dropout=0.255
3.69
0.315
0.257
0.283
0.108
3.96
0.279
0.205
0.236
0.039
NAGAN (dropout=0.20)
3,54
0,342
0.256
0.293
0.111
3.78
0,310
0.205
0.246
0.067
NAGAN (dropout=0.15)
3.41
0.371
0.252
0.301
0.118
0.338
0.203
0.078
NAGAN (dropout=0.10)
3,30
0.391
0.245
0.301
0.128
3.48
0.363
0.196
0.254
0.096
NAGAN (dropout=0.00)
3.21
0.415
0.221
0.289
0.191
3.48
0.376
0.149
0.213
0.265
Among the non-pretrained models, NAGAN (dropout=0:25 0.2) outperforms previous text GANs in all
metrics, where ScratchGAN (de Masson A'ttume et al., 2019) is the only baseline that can generate readable sentences. The other non- non-pretrain pretrained baselines fail to produce reasonable sentences, their optimization methods are ineffective.
We test models with different dropout rates in inference (where the training dropout rates remain unchanged as 0.25), and the result shows a smaller dropout rate can lead to higher quality. However, it also leads to less diversity (BLEUp B) and makes larger gap between the distributions of the real sentences and the generated sentences (FED). It suggests that varying the dropout rate in inference method balance the quality and diversity, where the larger noises introduced by the dropout encourage more diverse generated samples. We provide explanations in Appendix A.1.3
--- SANGAN
NAGANidrqpout-O2.
5.0
ARlSSof)
53-
LANKGAN
AR[Sumbel)
NLL Oracle
8-
MAGANES 1)
NLL Oracle
4.5
4.0
4-
3.5-
x
3.0
10
15
20
Training Steps
Sentence Length
Training curves synthetic data. The vertical dashed line denotes the end of pre- training for SegGAN and LeakGAN.
Figure 2:
Figure 3: Comparison autoregressive (AR)
generators on synthetic data of different lengths. Lower values are better.
4.2 WHY CAN NON-AUTOREGRESSIVE GAN WORK?
Comparing Autoregressive (AR) and Non-autoregressive (NAR) Generator. In Section 2 we mention that the AR generator can be an obstacle effective optimization. To verify the conjecture. we replace NAGAN's NAR generator with an AR generator, while preserving the other parts unchanged. The non-differentiable issue should be tackled before optimizing AR generator with gradient-based methods, so we adopt two methods proposed in previous text GANs. The first one is called Sefi-Embedding approximation (Chen et al. 2018). The generator predicts a word distribution at each position and obtains soft word embedding weighted by the distribution, where the soft embedding is treated as the next input. The second one called Gumbel-Soffman approximation 2016 Nie et al.) 2019). In each step, token is sampled from the
(Kusner & Hernández- word distribution with a reparameterized trick, where the straight-through estimator is also used for gradient estimation. We implement the AR generator using Transformer with the same size of
7

---

Under review as a conference paper at ICLR 2021
Table 4: Performance with different discriminator architectures on COCO dataset.
Table 5: Unsupervised decipherment. # and are reported in CCen et al. (2018) and He et al. (2020). respectively.
LM Score & BLEU 1 BLEUg BL FED 4
Without Max Gradient Penalty:
Model
Accuracy on BrownW-200
Transformer
4.62
0.111
0.051
0.070
0.734
Gomez et al.
2018
75.7
CNN
5.52
0.202
0.188
0.195
0.167
Chen et al 2018
77.2*
RNN
5.27
0.187
0.110
0.138
0.446
NAGAN
95.0
With Max Gradient Penalty:
BLEU WordSub
Shen et al 2017
50.8"
Transformer
3.69
0.315
0.257
0.283
0.108
Yang el al 2018
49.3"
CNN
4.08
0.309
0.243
0.272
0.110
78.4"
RNN
4.17
0.235
0.215
0.225
0.169
NAGAN
82,5
parameters with NAGAN. We also equip the generator with a latent variable, which is concatenated with the input at every step. Detailed implmentations are described in Appendix A..4.4
We first compare NAGAN with these two methods on synthetic data of different lengths. As shown in Figure NAGAN dominates the AR models on all lengths, and the AR models become worse rapidly as the length increases. Then we evaluate these models on SNLI dataset (max (ongth=40) where NAGAN remarkably outperforms the AR methods, as shown in Table 3 The results support our claim that the feed-forwart architecture benefits gradient- basedbaadddddee optimization and allows GAN training from scratch possible. The AR generator suffers from ineffective optimization because of the non-differentiable issue at every step's 's input, especially with long sequences.
Table Comparison to autoregressive (AR) generators on SNLI dataset.
BLEU
NAGAN AR(Soff AR(Gumbel)
3,96
0.279
0.205
0.236
6.21
0.100
0.080
0.089
0.086
Investigating Discriminator Architectures and Max Gradient Penalty (MaxGP). To investigate the effects of the discriminator architectures and MaxGP technique, we utilize Transformer, CNN. and RNN as the discriminator, and evaluate them with and without MaxGP on COCO dataset. Detailed architectures are described in Appendix A.4.4 As shown in Table 4 MaxGP boosts the performance on all architectures, where the weakest RNN discriminator can realize effective optimization in the non-pretrained setting. Although not critical, the discriminator architectures indeed affect the results, suggesting that there is room for further improvement in the discriminator design.
4.3 APPLICATION I: MANIPULATING SENTENCES IN LATENT SPACE
In this task, we set dropout rate to zero in inference, where the generated text can be fully controlled by the latent variable. We evaluate NAGAN on COCO dataset with two metrics. Success rafe denotes the proportion of edited sentences containing the desired word. For overlap, we first compute the ratio of the longest common subsequence's 's length the maximal length of the two sentences, and then average the ratios over 100 successfully edited pairs.
We compare our proposed method (called GD, gradient descent) against the offset vector method (called OV) (Zhao et al. 2018). In GD, the moving step will be repeated for several iterations. For a fair comparison, we also allow OV to try the moving step Z, == Z. + 3A for several iterations. where A is the offset vector. We also compare NAGAN against FMGAN (Chen et al. 2018). where the latter is equipped with latent variables but requires VAE pretraining. However, FMGAN cannot be applied to GD because it uses Soft-Embeddiig approximation to tackle the non-differentiable problem, which cannot provide gradient for latent variables during inference.
As presented in Table 6 NAGAN(OV) and FMGAN(OV) are comparable, but NAGAN(GD) achieves a higher success rate with better overlap scores, as it benefits from the direct gradient signals of the non-autoregressive generator. We also provide overall results over 100 random pairs of top-50 frequent words. The overall results are lower than the chosen cases because the random pairs may have different parts of speech, such as modifying red to people, making the task harder.
8

---

Under review as a conference paper at ICLR 2021
Table 6: Results for sentence manipulation (modifying a sentence containing a source word the one containing a targer word). OV = Offset Vector; GD = Gradient Descent (Ours).
FMGAN(OV) NAGIAN(OV) NAGAN(GD)
of Sentence Manipulation by NAGAN(GD)
sre/lar. words Suce. Overlap Suce. Overlap Succ. Overlap
white
black
0.59
0.32
0.72
0.39
0.99
0.75
A black cat laying outside in front of a large table.
white cal laying outside in front
large table.
white
A white cut sitting the top of large chair.
black
0.80
0.37
0.40
0.47
black cat sitting in the middle of large chair
standing
0.81
0.30
0.74
0.46
1,00
0.71
people are standing next to window in an umbrella.
sitting
A man is sitting next window an umbrella.
satling
0.47
0.33
0.75
0.45
0.98
0.64
A man sitting in Tront of window on a lake.
standing
man standing front
on
Overall
0.62
0.24
0.54
0.33
0.76
0.47
4.4 APPLICATION II: UNSUPERVISED DECIPHERMENT
We use Word Substitution dataset Yangg et al. 2018) and BrownW-200 (Gomez et al. 2018) for the task, where the text is encrypted by the substitution and the Vigenère cipher, respectively. Following previous work, WordSub is reported with BLEU-4, and BrownW-200 is reported with accuracy.
As shown in Table NAGAN significantly outperforms existing methods. Compared with AR models, the NAR generator facilitates effective optimization for the cycle loss and the adversarial loss. where AR generators are facing the same optimizing problem discussed in Section 4.2 The gradient of both losses can be back-propagdae to better alignment between two text spaces and thus a better decipher result.
through our feed-forward generator smoothly, leading
CONCLUSIONS
We present a novel text GAN, named NAGAN, which incorporates a non- aon-autoregresive generator to facilitate efficient gradient-based based training from scratch and the use of latent variables. NAGAN adopts novel formulation for adversarial text generation, which connects itself with image GANs and can potentially benefit from the success of image generation. As preliminary work of adversarial non-autoregressive text generation, NAGAN shows promising results on generating diverse sentences without conditions, which may suggest direction for solving the multi-modaling problem (Gu et al. 2018) and can be applied to general grquence-to-sqqeene generation as future work.
REFERENCES
Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN, CORR, abs/1 77111701077780 00775, 2017. http:///t//axxxi.....
abs 07777.7..77
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013. URL
htEp::/aaxxv.org/abb... http://////a///////// 3432
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp. 232-642, 2015. Utthtpps: / www. acl web. org/ wwwhaanngoooo---e-aeaeaaggg htt.a//www...ooootaaaaaoooooo/// 1075/
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, and Samy Bengio. Generating sentences from a continuous space. In Proceedings of the 20th SIGNIJ. Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11- 2016, pp. 10-21, 2016. URL https: / /www aclweb. org/ anthology / /16 10000000//
Andrew Brock, Theodore Lim, James M. Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. In 5th International Conference on Learning Representations, ICI.R 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
net/ htte//orummiee HKNKF iGex
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference Learning Representations, ICLR 2019.

---

Under review as a conference paper at ICLR 2021
New Orleans, LA, USA, May 6-9, 2019, 2019. BRLLatttss///eenerieeeeee//////t id-BIxaq109FFm
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. Language gans falling short. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, OpenReview.net, 2020. URL https //openreview.o net/ (tte://oommmidBJcaaaaaaa
Daniel Cer, Yinfei Yang. Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Quajardo-Cespedees Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. Universal sentence encoder for english. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31- November 4, 2018, pp. 169-174, 2018. URL https: /tt.st/////ww..... http:://wwwwwwwww..b....... 201-20ogy/DDB-2099//
Tong Che, Yanran Li, Ruixiang Zhang, R Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. Maximum-likelihen augmented discrete generative adversarial networks. arXiv preprint 17Xi::1702.07983. 2017.
Liqun Chen, Shuyang Dai, Chenyang Tao, Haichao Zhang, Zhe Gan, Dinghan Shen, Yizhe Zhang, Guoyin Wang. Ruiyi Zhang, and Lawrence Carin. Adversarial text generation via feature-mover's distance, Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pp. 4671-4682 2018. URL http: pwwprspppppssaaa nips..ipppp..................o.. cc/ /aper/ 7717-adver
www.reteeeraaae on - www.eooeeeeeesdaeeeeeeeeee
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp.
2016. URL http:://apprrs http://ppprr.iis........ wttp:://ssin...ccppppppp.p.ppp
Xinlei Chen, Hao Fang. Tsung-Ye Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 2015.
Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. Style transformer: Unpaired text style transfer without disentangled latent representation.
Anna Korhonen, David R. Traum, and Lluís Marquez (eds.). Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 6997- Association for Computational Linguistics, 2019. doi: 10. 11186//11p119-1600 URL www.:oo.orgg/00/ /Bt3://////..113//1/////3//////0 1601
Cyprien de Masson Autume, Shakir Mohamed, Mihacla Rosca, and Jack W. Rac. Training language gans from scratch. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Achéé- Buc, Emily B. Fox, and Roman Gameett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference December 2019, Vancouver, BC, Canada, pp. 4302 2019. URL http:httppppprrrs./// nips 00/ cccppape/// 682 -ele-ttranni weaiiiing-ll ng- lag-languagee ge gans from scratch
Neural Information Processing Systems 2019, NeurIPS 2019, 8-14
William Fedus, Ian J. Goodfellow, and Andrew M. Dai. Maskgan: Better text generation via filling in the In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL net http://ooommmmmom/oee
Aidan N. Gomez, Sicong Huang, Ivan Zhang, Bryan M. Li, Muhammad Osama, and Lukasz Kaiser. Unsupervised cipher cracking using discrete gans. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver. BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL battps://openrestt Bkea07x0-
net forum?id
10

---

Under review as a conference paper at ICLR 2021
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems. pp.
2014.
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv: 1308.0850 2013.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. Non-autoregressive neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL net wwt.:eeet/ooteeee
https:://pperev///
Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Improved training of wasserstein gans. CoRR, abs/1704.00028, 2017. URL httt:://aaaiiy
org/ hogg/abs/ie arg/abs/1704 1704. 00028
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (1AAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (FAAI-
18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 5141-5148, 2018. URL https
7 //www. www..aaai....m www.aaaaaaaaaaaaaaaaaa.aaaaaaaaa
Junxian He, Xinyi Wang. Graham Neubig, and Taylor Kirg-Kiirp.t A probabilistic formulation of unsupervised text style transfer. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net 2020. URI https
7/openreview net/ fore:/ormmmaam id HJLAOC4TPS
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2016. Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.AA
hRLhttps: doi. http://tiooooo..///////o www.111099......
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA. USA. pp. 6626 2017. URL http: htttt:::/pappss.... www.errssniiiiio... http:///as.aiis.cccpppppppppee/e 7240 www.ggaanttand-bberroraaaaa orr00ggnnnnttaaaaeee---r----ootn otg.ee/eeeec.....oo..oeeeoooeeee
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL htlth:ts:// https////ppneeeeee.eeeeee net forttmfoummt umm Bk Bk1r3j0cKX
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei (eds.), Proceedings the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMIR Workshop and Conference Proceedings, pp. 448-456. JMLR.org. 2015. URL /ttt:://proce www.rroeeeigg........... ngs .o........ ml wwwppreessv...... wttps///////offffff5e htmll
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 0167555679999 IEEE Computer Society, 2017. doi: 10. 2..119/CVPR.201..cc 2001 632
URL https: //:. wt.../0..0.../11.......
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In 5th International Conference Learning Representations. ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings, 2017. htt.hhtts:///
http::::///ppeeeevvieeee net/ fene/foormee www.mmrmmii. ? d
rkE3y85ee
11

---

Under review as a conference paper at ICLR 2021
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for gen-
erative adversarial networks. In IEEE Conference on Computer Vision and Pattern
Recognition, CVPR 2019, Long Beach, CA. USA, June 16-20, 2019, pp. 4401-4410. Computer Vision Foundation / IEEE, 2019. doi: WOO////P..009.000000 http:://peeaccesss tharccccsss.heee.....o com content, CVPR_ STFFR_2019htmm///e Style Based Generator wttprrrrrhhhhhhheheeeaee for Generative Adversariaa
URL
Networks CVPR_ 2019 paper.htm.
Pei Ke, Fei Huang. Minlie Huang, and Xiaoyan Zhu. ARAML: A stable adversarial training framework for text generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language
Processing (EMNLP-IJCNLP)), pp. 4262-42) 4272, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 11111165///DDD--II 1436. URL http:tttt/////t. /www ac aclweb
org/ antholooggo /Dwg 1436
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. htth
V/arxiv. w/t.xiv.rggaab//////o wttp:////bb///aaa/a 6980
Matt Kusner and José Miguel Hrmnnndez-Lobatte Gans for sequences of discrete elements with the gumbel-softmax distribution. arXiv preprint arXiv: 1,0000051 2016.
Chunyuan Li, Hao Liu, Changyou Chen, Yunchen Pu, Liqun Chen, Ricardo Henao, and Lawrence Carin. ALICE: towards understanding adversarial learning for joint distribution matching. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N.
Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 5495-5503, 2017. URL
nips. cc/paper.cpppppeppppo jorntttt-------------o-ooo-o- htt.:aoo---oooootttdddddrroooooo
Kevin Lin, Dianqi Li, Xiaodong He, Ming- Sun, and Zhengyou Zhang. Adversarial ranking for language generation. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 nips
December 2017, Long Beach, CA, USA, pp. 3155-3165, 2017. htL:: CC/ cc/paper/ 6908 www.aaaveaar.ar ranking for Language gereratiegggnnnnnn
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianteng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. / net/
Thomas Lucas and Jakob Verbeek. Auxiliary guided autoregressive variational autoencoders. In Michele Berlingerio, Francesco Bonchi, Thomas Gärtner, Neil Hurley, and Georgiana Ifrim (eds.), Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2018, Dublin, Ireland, September 10-14, 2018, Proceedings, Part I, volume 11051 of Lecture Notes in Computer Science, pp. 443 458 Springer. 2018. doi: 10. 0010//99788--00-009922-- 10. 1001 001199888-00000000-------00
URL wttpt:ttt//....oooo
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard H. Hovy. Flowseq: Non- autoregressive conditional sequence generation with generative flow. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joini Conference Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 4281-429) Association for Computational Linguistics, 2019. doi: 10.196533///D19-14533 1437. URL https 1.o...222/10..1853//////3 111
Lars M. Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of gans. In Is- abelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017
12

---

Under review as a conference paper at ICLR 2021
Long Beach, CA. USA, pp. 1825-1835, 2017. URL http: http::/papers..att//////// .ww.ssnnpsss.................... wwwpp.ccpp.pp......eccccppppppee cwweeerriss------o
Lars M. Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 3478 3487. PMLR, 2018. URL http:://proceedingg ml wtt.h//gw................mmm html
Vaishnavh Nagarajan and J. Zico Kolter. Gradient descent GAN optimization is locally stable. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, niss. cc/paper/
Long Beach, CA, USA. pp.
2017. URL
Hermann Ney, Ute Essen, and Reinhard Kneser. On structuring probabiliisiic dependences in stochastic language modelling. Computer Speech & Language, 8(1):1-38, 1994.
Weili Nie, Nina Narodytska, and Ankit Patel. Relgan: Relational generative adversarial networks for text generation. In 7th International Conference on Learning Representations,, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL https: bttts:://peenzevvew..ete IJedV3R5tm
Stanislau Semeniutal, Aliaksei Severyn, and Sylvain Gelly, On accurate evaluation of gans for language generation. arXiv preprint arXiv: 0996.0493, 2018.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-aligmment. In Advances in neural information processing systems, pp. 6830-6841, 2017.
Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Toward diverse text generation with inverse reinforcement learning. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden., pp. 4361-4367, 2018. doi: 10.24493/ijcai.20113 2018///0..111100 Utttttps:: /oii 2tt.://29/1..293/11111 111.:1...24499.9 /itp/////jaaa... 2018 606
Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. Latent-variable non-autoregressive neural machine translation with deterministic inference using a delta posterior. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference. IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, AAAI Press, 2020. URL piktps://aaal/ org.iioog//oo wwwp/sssssnsien.....oe// phad/
2020, pp. 18846-8853
/AA./article/vieeeeee 6413
Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, and Ferene Huszár. Amortised MAP inference for image super-resolution. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
/tnttps://openreview. http://wwwww..teeefoooooom/aaaa- 6GGlee
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1): 1929-1958, 2014. BRLhhttp://dl
acm ttt//ootttiooooooooooo/oo
Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross-domain image generation. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track P Proceedings, 2017. URL RRRLhhhts::/// http:///ppeeeiiiiee htttps:///penreeiiee
net forum? d=
Sk2 Im5
Lifu Tu, Richard Yuanzhe Pang, Sam Wiseman, and Kevin Gimpel. ENGINE: energy-based inference networks for non-auutoregressive machine translation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 2819 2826. Association for Computational Linguistics, 2020. URL https:/ www aclweb. org/ ant httoooy//2ooo0
acl sel-maing n. 2517
13

---

Under review as a conference paper at ICLR 2021
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems. pp. 5998 6008, 2017.
Ronald J. Williams. Simple statistical gradient-followingg algorithms for connectionist reinforcmeet learning. Mach. Learn. 8:229-2566 1992. doi: 10.1007/BF00999666 URL https://dbi org /00 10000000009966665
Jingjing Xu, Xu Sun, Xuancheng Ren, Junyang Lin, Bingzhen Wei, and Wei Li. DP-GAN: diversity-
promoting generative adversarial network for generating informative and diversified text. CoRR, abs/ 802.0132,,, 2018. httphtt:// /ttp:://axxiiii org/ abs/1802 01345
Kun Xu, Chongxuan Li, Huanshu Wei, Jun Zhu, and Bo Zhang. Understanding and stabilizing gans' training dynamics with control theory. CoRR, Abs//999..3188, 2019. URL http: /ttp:///axta
erg/abs/1909 13188
Zichao Yang, Zhiting Hu, Chris Dyer, Eric P. Xing, and Taylor Berg-Kirkpattick. Unsu- pervised text style transfer using language models as discriminators. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Con- ference on Neural Information Processing Systems 2018, NeurlPS 2018, 3-8 December 2018, Montréal, Canada, pp. 1299-73091 2018 htthttp:: ptattt::/paperrtt nips. wwtppp/ppp....... 7959- www-pervvseedddeeeeeeeeeeeee -eassseddeex-ete style transfer Setpnnsferuiiing--ee Laieeuguaun----gggeeeee sonngugge-mddesss models as discriiimatmorrs
Yasin Yazici, Chuan-Sheng Foo, Stefan Winkler, Kim-Hui Yap. Georgios Piliouras, and Vijay Chandrasekhar. The unusual effectiveness of averaging in GAN training. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URLhttts:://ppezeeeee wttp://wwwww...eeeeeeeeeeeeeeeee wttp:mmmmiiisss.iee _S___________F____________
Fisher Yu, Vladlen Koltun, and Thomas A. Funkhouser. Dilated residual networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI. USA, July 21-26, 2017, pp. 636-644, 2017a. doi: 10. 10.1109/CVPR.2211..55 URL https
//doi.org/// /00 L119//CP..2217.......
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Thirty-First AAAI Conference on Artificial Intelligence, 2017b.
Junbo Jake Zhao, Michaël Mathieu, and Yann LeCun. Energy-based generative adversarial networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24- 26, 2017, Conference Truck Proceedings. OpenReview.net, 2017. URL /tttp:://///ttt /pw./evveeeeeeeeeeeeeeeeeee ew net / E/rummrrmco ?dd ryh9pmcee
Junbo Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M. Rush, and Yann LeCun. Adversarially regularized autoencoders. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässen, Stockholm, Sweden, July 10-15, 2018, pp. 5897-5906, 2018.
mlrr www...press/vee 2e.00/2hhllle html
Chunting Zhou, Jiatao Gu, and Graham Neubig. Understanding knowledge distillation in non- autoregressive machine translation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30. 2020. OpenReview.net, 2020. URL https
//openreview.oo net wwtp://tteeo.tiieeteee forummirooooo ftt d-ycdFARKDDF
Zhiming Zhou, Jiadong Liang, Yuxuan Song, Lantao Yu, Hongwei Wang, Weinan Zhang, Yong Yu, and Zhihua Zhang. Lipschitz generative adversarial nets. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 7588477993. PMLR, 2019. URL http:tt:ppp/ccei...ee. mlr. www.ppress/eee. / wchoulll.h.....
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In IEEE International Conference Computer Vision, ICCV 2017. Venice Italy, October 22-29, 2017, pp. 2242-2251, IEEE Computer Society, 2017. 0tt///10.11100 1109/ TCCV. 2017.240
doi: 10.1 1199/CCV.2017.244 244. URL UtI. h
14

---

Under review as a conference paper at ICLR 2021
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08- 08-12, 2018, pp. 1097-1100, 2018. doi: 10. 145/3209978.52100008 7/doi.org/ 1105://0011444437795777
URL https:
A APPENDIX
A.1 MORE DISCUSSIONS
A.1.1 DEGENERATION OF AUTOREGRESSIVE MODELS WITH LATENT VARIABLE
An autoregressive model with latent variables can be defined follows (Bowman et al. 2016
Lucas
& Verbeek 2018):
L
(a(a)) = Expp(x) II 1=1
Po Paxs1/00 z).
(14)
It can degenerate into a language model, where the latent variable z is ignored when generating æ, This problem has been found in Variational Autoencoders
i.., mutual information 45 (2x (VAEs), known as the KL vanishing problem (Bowman et al. 2016). However, the reconstruction loss in VAEs optimizes the lower bound of Ig(x, ., z). proven by Li al. (217)) in Corollary
Ig (a:, 2) 2 Ex Eu-paana
[x (a(a]=)] + H Hpaato (x)
(15)
Pdiat
-rreoo + constant
(16)
where H means entropy, and q is the approximate posterior distribution. Therefore, optimizing Crecon avoids the degeneration, making VAEs able to learn latent representations.
However, text GANs only use an adversarial loss, which merely matches Pg and Preat. We can easily construct an ideal generator defined as (g(m: [eei 2) = Preat (I; <i))), where Pş perfectly fits Preat: but (x, z) equals O. We can conclude that the text GAN with an autoregressive generator is not guaranteed to learn latent representations.
Things are different if G is deterministic function, where the generator G is forced to use = if Pg can fit P Preat well. To explain the statement, we present another lower bound of mutual information. Let Gg be an generator with parameters 0, and p(z) be the prior distribution of z, we have:
Iga (xx 2) = Ex-p(*)
DKL (Po. (x)=))
log Preat(()) (x))
E Exsp(*)
DKL. (z) Preat (2)) + Eew (2))))
log Ps. (2:)
Ex-p(*)
DKL (Po. (m| [ Preat (x)) - DKL (Poo (x) Preal
(17)
min DKL. (Po.. (x1x")) (reat( - DKL (Poo Preat
(18)
In Eq 17 we relax the first term by fixing z and 0 to minimize the KL divergence. In the first term of Eq (18). x" is constant, and thus Ps.- (z x²) can be regarded as generator with fixed input, which ignores the latent variable. In the second term, Pg. (x) is the expectation of Ps. (a|z) over p(z), which represents the model distribution considering latent variables. Then the first term indicates the best KL divergence if G ignores the latent variable, and the second term indicates the KL divergence when Go uses the latent variable. Therefore, Ig (x, 2) is bounded by the performance gap between the generators with and without the latent variable.
If is deterministic function, Ps.- (æ) x²) must be a one-point distributiorf which is far from fitting the real data. As long as Go can be effectively optimized, the second term of Eq (18) should
Ps.. (zr/²") is equal when Go- (z") and equal 0 otherwise.
15

---

Under review as a conference paper at ICLR 2021
be smaller than the first term. It indicates Ig, (a. x) 0, and Go cannot ignore the latent variable. Moreover, when Ge is optimized to approach the real distribution, the lower bound of the mutual information becomes larger, which usually indicates better representations (Chen et al. 2016 Hjelm
et al. 2019).
A.1.2 SENTENCE LENGTH IN NON-AUTOREGRESSIV GENERATORS
Autoregressive text generators use a special token (e.g., <eos>) to determine the length, but most non-autoregressive text generators require the sentence length before the parallel generation. Strictly speaking, a non-autoregressive text generator is a conditional generator, which generates the text conditioned on the length I, and the input C, i.e., = g(C, L.). Therefore, the generation process is split into two steps: determine the sentence length, and then generate sentences according to the length. We use the non-autoregressivv generator only in the second step, where the first step can vary in different tasks.
In machine translation, con-autoregressvve text generator usually uses a length predictor to predict the target length from the input sentence (Gu et al., 2018), which is trained separately from the non-autoregressive generator. There also an alternative method called Length-paralle Decoding (Shu et al. 2020), where the generator tries all possible lengths (by enumerating a pre-specified range) and generates a set of sentences, and then select the best one using an external ranking model.
In unconditional text generation, we aim to match Pa(x)) with the real distribution. Regarding the length as another latent variable, (o(x) can be formulated as a sampling process: L ~~ p(L), 2 ~ p(z[L), x- G(Z, L). We intuitively estimate pLL) by the sentence length distribution of the training data, which helps Pg (x) approach Preat (z) well.
In the unsupervised word decipherment application, have already known the target length should be equal to the source length, so we can simply obtain the target length. However, it is theoretically possible to learn length predictor Pc([Tly) to predict the target length, where y is the ciphertext. Then the decipher model Fy--x can be formulated as a sampling process: Z = Ely, ey), I. ~ Po(L)y), [[)[)y))a x Gz,,cx, L). The classifier Pe can be trained to minimize the final objective described in Section 3.4 by reinforcement learning or other optimization methods.
A.1.3 How DROPOUT WORK IN GANS
This subsection is added during the discussion.
Dropout is a common trick in training image GANs, but it is more than simple regularizer when used in GANs' generators. We provide two explanations about why dropout can stabilize the training process and balance fluency and diversity.
First, dropout can be regarded as noises to diversify the generated samples. GANs training usually suffers from the mode collapse problem, which can be alleviated by keeping the diversity of the generated samples. We refer the reader to the instance noise trick (Sønderby et al., (017). which is also a method that keeps the diversity. The author claim that the diverse samples can stabilize the gradient flows by avoiding the situation that Pg and Pdata have disjoint supports.
When removing the dropout in NAGAN, we can observe spiky losses and unstable training process. However, it still outperforms some baselines without MLE pretraining. The results of no dropout training is demonstrated Table IT
Second, dropout can be regarded as method of introducing extra latent variables. Unlike the common networks that learn to produce the same output when dropout masks change, it is shown that random noises introduced in the intermediate layer of GANs' generator will affect the generated samples (Karras et al. 2019). We can explicitly take these random noises as latent variables, where different noises will be mapped to different samples in the GANs training. In our case, the latent variables are the dropout masks.
A linear layer with dropout can be formulated as follows:
1
y =
W & m),
(19)
p
16

---

Under review as a conference paper at ICLR 2021
where is the imput vector, y is the output vector, W is the weight matrix, m is the dropout mask, p the dropout rate, and dlement-wise multiplication. Taking m random variable, we can obtain the mean and the variance of y.
Elyi 1
1
Σ
W( & P(m)
me (0.1))
1
W Σ (z.m;P(m.))
P
mu
1
W1 - = Wæ
(20)
1-P
Varly)
EEyy") (yyy
1
Σ
W
Σ
((mm))(mm))- (Wa)²
(1 - p)
myE(o.1)
1)(22)
(21)
N is the dimension of æ, W, is the i-th column of W. When = O, the variance of y is 0. For larger p. the mean keeps unchanged but the variance becomes larger.
In the GANs training and inference, we usually keep the distribution of dropout masks, i.e., the dropout rate, unchanged (Isola et al. 2017), that the generator's distribution remains unchanged. However, if we reduce the dropout rate in inference, the distribution will be more concentrated, bringing samples with higher quality but lower diversity.
This phenomenon is not significant in autoregressive text GANs. One reason is that the implicit generative model uses deterministic generator, where the generated sample is fully determined by the latent variables (including Z and dropout masks). However, the autoregressive generator samples from word distribution at each step, which brings more diversity than a non-autoregressive generator, so the effect of dropout will be weakened.
A.2 EXPLORING THE LATENT SPACE AND GENERATION PROCESS
This section is added during the discussion.
For better understanding of the learned latent space and the non-autoregressive generation process, further conduct four experiments the COCO dataset.
A.2.1 SENTENCE INTERPOLATION
This subsection added during the discussion.
We set I. = 12 and randomly sample two sequences of latent variables Z,, Zz from N(0. T) independently. Then we obtain the linearly interpolated latent variables ZA = (1 Z) + >Zz and translate them to the text space. Cases are shown in Table 7
A.2.2 SMOOTHNESS OF SENTENCES WITH DIFFERENT LENGTHS
This subsection added during the discussion.
We randomly sample a sequence of latent variables and then concatenate a new sampled =1+1 the end of Z. We translate two sequences of latent variables text space and find that the two sentences are very similar, except the length differs. We repeat the step and generate sentences with length from 10 17. We also try to concatenate new sampled variable the front of the sequence. The generated sentences are shown in Table
A.2.3 CONNECTION BETWEEN Z, AND O;
This subsection is added during the discussion.
17

---

Under review as a conference paper at ICLR 2021
Table 7: Cases of sentence interpolation. Gray words indicate unchanged parts compared with the preceding sentence.
A
Generated Sentences
0.0
picture of a dog laying on posts in the forest. A picture of a person laying a runway in the ocean.
0.1
0.2
A picture of an person on motorcycle in the ocean.
0.3
A picture of an elephant riding surfboard in the ocean.
0.4
A picture an elephant riding a surfboard in the ocean.
0.5
A picture of the ocean while some waves in the ocean.
0.6
A picture of the ocean while some surfboards in the ocean.
0.7
A picture of the ocean that heing sailing in the water.
0.8
A picture of the occan that walking the water.
0.9
A man the ocean that is surfing in the ocean.
1.0
A man in a suit that is surfing in the ocean.
0.0
A big teddy bear laying
rock in a field.
0.1
A herd toddy bear laying on a rock in a field. herd sheep laying on a rock in field.
0.2
0.3
A herd of cattle grazing on a rock in a field.
0.4
A group of cattle standing on motorcycle in field.
0.5
A group of cattle standing on motorcycle in the field.
0.6
A group of people standing on motorcycle the park.
0.7
A group of people standing on a horse in the rain.
0.8
A group of people standing riding a horse in the rain.
0.9
A group of young man riding horse in the rain.
1.0
A group of young man riding horse in the rain.
0.0
A man in a suit riding a skateboard in the snow.
0.1
A man in suit riding some surfhoards in the snow.
0.2
A man in a suit riding some surfboards in the occan,
0.3
A picture of a person riding some surfboards in the rain.
0.4
A picture a giraffe trying to someone in the rain.
0.5
A group of cattle standing next to someone in the rain.
0.6
A group of people standing next to someone in the distance.
0.7
A group of giraffes standing next someone in the pasture.. group of giraffes standing next to someone in the pasture.
0.8
0.9
A group of giraffes standing next to someone in the pasture.
1.0
A rows of giraffes stand next to other in the pasture.
Table 8: Smoothness of sentences with different lengths. We convert one sentence to the next one by adding a new sampled latent variable to the end (or the front) of the latent variable sequence, Gray words indicate unchanged parts compared with the preceding sentence.
Length
Added to the end A pile of fresh fries, broccoli and mashed A pile of braccoli and sandwich in mashed fries.
10
11
12
A slice of bread with a white sandwich on their fries
13
A white white plate with white cream arranged on a plate.
14
A white plate with chicken sized bunches of broccoli, and french fries
15
white plate with various meat and mashed broccoli. braccoli and french fries..
white plate with meat peppers and mashed potatoes broccoli, and french fries..
17
A white plate with chicken peppers and mashed potatoes, , braccoli. and french fries nearby.
Length
Added to the front
A pile of bread on plate in pan.
11
A plate is pie on a plate in a pan.
12
A slice of bread with sliced toppings sauce on plate.
13
A white plate with sliced toppings sitting on dinner white plate. , plate of fresh fruits sitting on a plate on a white pan A pile of fruits and vegetables on a plate with broccoli and a fork.
14
15
16
A pile of broccoli and vegetables on top with sandwich and french fries nearby
17
A pile of broccoli and vegetables on top with sandwich, and plate nearby
18

---

Under review as a conference paper at ICLR 2021
As the length of Z is always equal to the length of generated sentences, it seems natural if 2; is highly related to O, However, the following experiment negates the hypothesis.
We first sample a sequence of latent variables Z (L 10), and translate it to sentence O. Then random choose a position i and replace z; by an new sampled latent variable from N I). The new sequence is denoted by Z' and translated times and count the probability j)i,j) that o changed when modifying 2;. If the hypothesis is true, high probabilities should be observed in the diagonal of the matrix P. However, P(i,j) is highly correlated with the position i regardless of j. as shown in Figure + We can conclude that there is no strong connection between 2; and 2,. and each latent variable plays similar role in generating 24-
sentence O' We repeat the trials for 1,600
This property is not desired and can make NAGAN less interpretable. However, we think that adding regularizers will be good way make the tokens more correlated to nearby latent variables, which is left for future work.
1.0
1
Position of latent variable
3
0.8
0.6
5
0.4
7
0.2
9
0.0
2
6
8
10
Position of generated token
Figure 4: Connection between tokens and latent variables at different positions. The element in the i-th row and the j-th column is the probability that the token æ, changes when modifying the latent variable z,. The last token is full stop and never changed.
A.2.4 INVESTIGATING GENERATION PROCESS
This subsection added during the discussion.
Autoregressive generators always generate tokens from left to right, but non-autoregressive generators produce the whole sequence in parallel. We investigate the generation process of NAGAN's generator and find some interesting properties: The non-autoregressive generator will gradually determine the hidden states as the data flows through the Transformer layers; tends determine some of the tokens first and then the others.
We collect the hidden states of each Transformer's layer and calculate the cosine similarities between the intermediate hidden states and the last outputs. To be specific, sim,, cos(h1,. h.n)). where h4,1 is the output of the 1-th layer the t-th position, and is the number of layers. In our experiment, L= 5 and < 1 < 10.
As shown in Figure the similarities each position are usually slowly increased from the first layer to the last layer, which means the generator gradually determine their hidden states. Some tokens at the front (t = 1), in the middle (4 < 7), and at the end (t 10) will be determined before the 3rd layer, and the other tokens will be determined in the last two layers. The tokens at the front are usually definite or indefinite articles, and the tokens at the end are full stops. The tokens in the middle usually prepositions (e.g., in, top, and next), which split the whole sentences in two halves. This strategy may make the generation of the remained tokens easier, because it is similar to a masked language model when the near context is given.
19

---

Under review as a conference paper at ICLR 2021
0.40
2
0.35
3
4
5
0.30
man
riding
skateboard
grassy field
0.25
2
0.20
4
5
0.15
white
cat
sitting
on
bop
table
0.10
2
0.05
4
lntble
girl
sitting
next
small
blanket
Figure 5: Cases of cosine similarities between intermediate hidden states and the last outputs. The i-th row indicates the hidden states of the i-th Transformer layer (1 sis 5).
A.3 IMPLEMENTATION DETAILS
Here introduce the implementation details of NAGAN. The codes available in the supplemen-
tary material.
A.3.1 GENERATOR
The generator's Transformer based on the architecture of the con-autoregressive text decoder al. 2018), which contains P_layers Transformer blocks. The attention layers attending to the source sentence are replaced by attention layers attending to the latent variable.
In the training stage, introduce Gumbel noises when obtaining s,. Introducing noises is common technique used in image GANs (Zhao et al. 2017 Sønderby et al. (007)) where noises can encourage the model generate diverse samples and avoid mode-collapse problem in the early training stage. To he specific, 8, Ml.((h,) + 9t. where 9: is sampled from the standard Gumbel distribution. We use Gumbel noises rather than the other noises (c.g., Gaussian noises) because they more suitable for categorical distribution (Jang et al. 2017). Different from the Gumbel-Softmal tricks, we do not use the noise or sample from the categorical distribution in inference.
As mentioned in Section 3.1 the truighh-throuug estimator is only used when obtaining gradient. We provide alternative description for the implementation as follows:
o onehot (arg max,((st..)..
(22)
of" (ottmax(8t/)))
(23)
O, = stop goo__rradent(o, of of') + o."
(24)
where St,v is the v-th dimension of s;. means that o, of in the forward pass, and o = of + constant when obtaining gradient.
'Our implementation is based http://gittaalcmmmsssefrrrooooon
20

---

Under review as a conference paper at ICLR 2021
A.3.2 DISCRIMINATOR
The discriminator's Transformer has the same architecture the encode in Gu al. (2018). which contains n n_layers Transformer blocks.
Max Gradient Penalty. Following Zhou et al. (2019) we calculate max sampled gradient as follows:
(D()))
CGP
max (=12, .n
(25)
80;
where n denotes the batch size. The intermediate sequences O are sampled in the following way: first, sample a sentence from the real data, whose one-hot representation Owi second, generate fake sample Os with the same length of Ori. Finally,
0 = Ar (1 (1- A,)
(26)
where A; is sampled from [0, 1] uniformly. Note that O may not be a one-hot sequence and does not represent a discrete sentence. However, not a problem because it is only used regularize the discriminator.
Score Regularizer. We also adopt a 12-regularizer on the predicted score D(O), which is proposed to stabilize GAN training (Xu et al. 2019). It can be formulated as follows:
LSR = |[D
(27)
A.3.3 TRAINING
The training procedure is presented in Algorithm
Algorithm Training Procedure of NAGAN
Require: Max Training Steps: iters Batch Size:
I: for T 1,,,...... ittrs do Training the discriminator.
2: Sample batch of real data O,:(1 SiS n), whose length is L,,
3: Sample batch of latent variables Z, Si n), where Z. R² dimxI.
4: Translate the latent variables Z: to generated sentences Oy, via the generator G, for 1 SiS
5: Calculate LGP via Eq (255)
6: Update the discriminator D via Cnp Lo + A(Cor LSR) (See Eq for LD).
Training the generator.
Sample new batch latent variables Z, similar (3).
8: Update the generator G via Eq (13).
9: end for
Parameter Averaging. Some studies (Mescheder et al. 2017 Nagarajan & Kolter 2017 Mescheder
al. 2018) analyze the training dynamics of GANs and show that the generator does not converge to a point but becomes periodic function around the Nash equilibrium point. Therefore we adopt common method used in image GAN, called parameter averaging (Yazici et al. 2019).
Suppose the generator's 's parameter at step is ,,. Then, the exponential moving average is defined as
0 = 1= + (1-7)0,
(28)
where y is often number slightly smaller than 1. is noticeable that Ú, has much smaller variance than 0r, and 0, is then used for the evaluation.
A.4 EXPERIMENT SETTINGS
A.4.1 EVALUATION METRICS
Oracle NLL. For synthetic data, calculate the negative log-likelihood (NLL) per token of the oracle model evaluated on 5,000 generated sentences. We do not use perplexity because the likelihood of implicit generative models is intractable.
21

---

Under review as a conference paper at ICLR 2021
Language Model Scores. Language model score (Ke et al. 2019 Caccia et al. 2020) is the NLL of a language model trained on the test set and evaluated on 5,000 generated sentences. As suggested by de Masson d' A'Atumne al. (2019), this score may favor the models which have the same architecture of the language model. Thus we use a 4-gram language model with Kneser-Ney smoothing (Ney et al. 1994) instead of using RNN language models.
n-gram Based Metrics. To evaluate both the quality and the diversity (Shi et al. 2018). we adopt three BLBU scores: Forward BLEU (BLEUF), Backward BLEU (BLEUB). and their harmonic mean (BLEUHa). Forward BLEU measures quality, which uses the test set as references, and evaluates 5,000 generated sentences with the BLEU score. Backward BLEU measures diversity, which swaps the roles of the test set and the generated sentences. We use the BLEU-5 score in our experiments.
Some previous work adopts self-BLEU (Zhu al., 2018 for evaluating diversity, which calculates the BLEU score of each generated sentence using the other generated sentences as references. The metric suffers from meaningless diversity, where model that randomly generates tokens can achieve perfect self-BLEU, We use Backward BLEU to measure diversity, which is symmetric with Forward BLEU and avoids the meaningless diversity problem.
Fréchet Embedding Distance (FED). FED (de Masson d' Autume et al., 2019) is text version of Fréchet Inception Distance (Heusel al., 2017), which has been widely used image GANs. measures the similarity between the representation distributions of the test data and the generated data, where we use 5,000 generated sentences as the generated data. Sentences are represented with vectors by Universal Sentence Encoder" Cer al. 2018
A.4.2 SYNTHETIC DATA
The synthetic data are generated by an HMM mode with 100 hidden states. randomly initialized HMM may have a flat distribution over all sequences, which is remarkably different from natural language text, so we fit the HMM on 5,000 real sentences from the COCO dataset before generating the synthetic data. However, the generated synthetic data are still different from the COCO dataset, where the synthetic sentences have a fixed length (length=20 in Figure 2 and a smaller vocabulary of 500 tokens. The parameters of the oracle HMM will be released. We generate 50,000 sentences as the training set. We do not use a validation set or test set, because the Oracle NLL does not need real samples for evaluation. The Oracle NLL is always averaged over 5,000 generated sentences.
The baselines with the RNN architecture are using LSTT with 128 cells or GRU with 256 cells. The MLE baseline is 256-dim GRU. The other hyper-parameter of baselines are setting according to the official implementation
o The hyper- parameters of NAGAN are shown in Table
In Table 10 we provide the best Oracle NLLs as a supplement of Figure 2 Note that one training step in Figure 2 may be different across models. For MLE, ScratchGAN, NAGAN, and the pretraining part of SeqGAN and LeakGAN, one training step means one epoch training. For the GAN training part of SeqGAN and LeakGAN, one training step contains several updating steps in the generator and the discriminator, which is consistent with the original implementations.
A.4.3 REAL DATA
2015) we follow previous work (Guo et al. 2018 Shi et al. 2018 Ke et al. 2019) and use the image captions the real sentences, where the images are discarded. The dataset is processed by Shi et al. (2018) 10 which contains 80,000 samples in the training set, 5,000
For the COCO dataset (Chen et al.
samples in the test set, and 4,838 words in the vocabulary. However, the dataset does not contain validation set, so we randomly choose 5,000 samples from the training set as our validation set.
For the SNLI dataset (Bowman et al. 2015). we follow previous work (Semeniutal et al., 2018 Zhao et al. 2018) to extract the sentences, where the connections between the paired sentences are ignored. The SNLI dataset contains 714,667 samples in the training set, 5,000 samples in both the validation and test set, and 42,981 words in the vocabulary. We trim long sentences to the first 40 tokens.
7 "The pretrained model can be downloaded at http://///htato//////////////e/d
We use the implementation of /ttp::///ttaaacccoom/mmmm/////mm hhttps://githubb... wttp:/cwooo/ccc/////oc.e/ http:///gguub..oo//uuann...////t
22

---

Under review as a conference paper at ICLR 2021
Table 9: Hlper-parametter of NAGAN on the syn-
thetic data and the real data.
Table 10: Results on the synthetic data. NA-
GAN are tagged with droprates in inference (Training droprate is always 0.25).
Hyperparameler
Value
MaxGP Weight (Synthetic/ Real)
0.17T
Softmax Temperature r
Model
Oraele NI.I
Purameter Averaging
0,9999
Pretruined Modelsy
Batch Size (Synthetic / Real)
128 64
MLE (Graves 201 7hh
Transformer Blocks (t____y__oee layers)
SegGiA To
3.84
Transformer Heads
LeakGAN al, 2015
3.45
Dim. of Transformer (tf (tf_d
128
Non- Non-pretraneed Models:
Dim, of Latent Variable (Z/ dim)
64
SegGAN wio pretrain
3,44
Dim. Word Emhedding (E (e_dm))
LeakGAN prettaiii
7.28
ScratchGAN
d'Autumo
Dmpprate in Training
11.25
NAGAN (100poit=0.225)
3.54
L2 Regularizer (D (9)
0.05 /
NAGAN (dropont=0..2
3.50
Optimization method
RAdam (Liu et 2020
NAGAN (dropout:0.15)
Leuming rate (D ()
5c-47 20-4
NAGAN (drupout-0.1)
3.23
9., B2 in RAdam
0.5, 0.9
Table 11: Generation performance on COCO and SNLI datasets with mean and variance of three runs with different random seeds.
Model
COCO
LM Score L
BLEU, f
BLEUg
BLEUHA f
FED
NAGAN (dropout=0.25)
3.69:0.05
0.315:000077
0.257=0.013
0.28310.011
0.008
NAGAN (dropout=0.20)
3,5410.05
0,342:10.008
0.256-0.011
0.293+0.010
0.101
NAGAN (dropout=0.15)
3,41+0.07
0.371++0.018
0.252-0.014
0.301+00.015
0.118++0000
NAGAN (dropout=0.10)
3.30=10.07
0.391::0.023
0.245=0.017
0.001
0.128:
NAGAN (dropout=0.00)
3.21+0.11
0.415++0.048
0.221=0.022
0.289+00003
0.191
NAGAN (no training dropoutj
3.90:20.11
0.267:10.012
0.192-0.012
0.223:0.011
0.160=1.0054
Model
SNLI
Score 1
BLBUe f
BLEUs
BLEUWA f
FED
NAGAN (dropout=0.25
3.96:10.15
0.279:000088
0.205=0.003
0.236:0.004
0.059:0.011
NAGAN (dropout=0.20)
3.78+0.15
0.310=+0.021
0.205=0.004
0.246++0.004
0.067=+0.014
NAGAN (dropout=0.15)
3.61:0011
0.338:10.019
0.203=0.006
0.25340.001
0,078:0.019
NAGAN (dropout=0.10)
3.48::0.13
0.363:100019
0.196=0.009
0.25440.003
0.096:000027
NAGAN (dropout=0.00)
3,48:10.37
0.376:000055
0.149=0.032
0.213:100000
0.265:10.129
NAGAN (no training dropout]
4.44:10.14
0.313-50.024
0.189=0.004
0.235:10.010
0.081 10.015
The hyper-parameters NAGAN are shown Table which are tuned to minimize the FED on the validation set. We report the generator architecture, numbers of parameters for the generator and the discriminator in Table 12 The Transformer bascline and NAGAN has the same number of Transformer blocks, attention heads, and size of hidden features, except two differences: the baseline is an autoregressive generator trained by MLE; NAGAN does not need word embeddings because it receives the latent variable Z as input. Unless otherwise specified, the other hyper-parameterr of baselines are set according the official implementation [T] The reported results in Table are averaged over three runs with different random seeds. As a supplement, we report the result with mean and variance in Table TT We also show some generated sentences in Table
A.4.4 IMPLEMENTATION OF ABLATION MODELS
Autoregressive (AR) Generators. In Section 4.2 we compare NAGAN against two models with AR generators. These two models differ from NAGAN only in the generator, which is implemented by an autoregressive Transformer with the same architecture except for the attention mask.
AR(Soft) implemented following Chen et al. 2018). At step AR(Soft) predicts the next word distribution P. = P(æ I<t. 2) R. Then we obtain the soft word embedding e, EP,, where E E E [_dimxxV is the word embedding matrix. After that, e is concatenated with the latent variable
IRR: https://gitubbcom//uuaannNN SeqGAN, LeakGAN; http:://gihhbbcco///eeea/a/////t ReIGAN; Ssatffaf.iuiiiininooommmmmmmmmme
FMGAN: www.raawiiiiiiiiiinaanoaaaooooon
23

---

Under review as a conference paper at ICLR 2021
Table 12: Parameter size of all models when vo- cabulary size is 4838. Note that NAGAN do not need word embeddings because it receives latent variables as imputs.
Table 13: Cases of unconditional generation on the COCO dataset.
A group of people walking in the grass covered held.
A man riding motorcycle on snowy day.
Model
Generator
@Parums &Parums
A young
riding skate board down hill
GRL
256-2im GRU
3.1M
NIA
A woman sitting on table with brown laptop
Transformer
AR Transfommer
3.0M
N/A
A group of people standing next to bunch bananas-
SoqGAN
128-dim LSTM
0.9M
Two people are sitting on the side of read.
LeakGAN
Specially Designed
410M
1.0M
A toddy bear laying
rock field...
IRL
128-dim dim LSTM
2.0M
5.4M
A group young man riding a honse the
Specially Designed
3.3M
A stop sign with sily street
EMGAN
128-dim LSTM
1.7M
38M
A plati filkdd with chicken, and a salad in a basket sitting on table
SemtchGAAN
2-layer 512-dim I STM
15.4M
2,3
NAGAN
NAK Transtormer
2.2M
2_imm and fed into the Transformer as the next input. The latent variable is sampled from the normal distribution (o0 I). Note that the model generates a sequence embeddings training, which does not represent any discrete sentence. We follow Chen et al. (2018) and use greedy decoding to generate texts in inference.
ARGGumbel) is implemented following Kusner & Hernández-Lobato (016). AR(Gumbel) predicts the next word distribution at step t. Then we obtain the next word in one-hot representation with the Gumbel-softmax tricks (Jang et al. 2017):
02 onehot (arg max, (log P(z, = u| z) + gx)) E 13,
(29)
The Gumbel-softmax trick is reparameterized trick, where o, can be regarded as the one-hot representation of random sample from the categorical distribution P( P(æz//<2: z) (Jang et al. (017)) The straight- straight-through estimator is also adopted to obtain gradients for o,. After that, O, is converted to word embedding et, which is then similarly concatenated with z and fed into the Transformer as the next input. AR(Gumbel) uses random decoding inference and thus keeps the same behavior between training and test.
Other Discriminator Architectures. In Section 4.2 nators.
we equip NAGAN with two different discrimi-
The CNN discriminator is composed of 12 dilated residual blocks Yu et al. 2017a), where each block sequentially contains batch normalization layer (loffe & Szegedy) 2015), ID dilated convolutional layer, and a residual connection (He et al. 2016). The channel size is 300, and the kernel size 3. The dilation of the first 4 convolutional layers is set to [1, 1. 3). which is repeated two times for the left 8 layers. The convolutional layer does not change the sequence length, where the feature is then fed into max-pooling layer and an MLP like Eq (10).
The RNN discriminator is a one-layer bidirectional GRU with 256 hidden cells. The encoded feature is also processed by Eq (10 to obtain the final score DOO).
A.5 APPLICATION I: MANIPULATE SENTENCES IN LATENT SPACE
In the Offset Vector method (OV) (Zhao et al. 2018), we first randomly generate 100k sentences and obtain the offset vector A = Z, Z, where Z, is the average of latent variables whose corresponding sentences containing the target word, and Zg is that for the source word. Then, we modify the latent variable by Z, : Z, + 3A until the sentence generated from Z, contains the target word or the process exceeds a pre-specified iteration number.
Our proposed method, called gradient descent (GD), edits the sentence by applying gradient descent over the latent variable. As described in Section 3.4 we maximize 0r,x', where t is the edited position and x' is the target word. In addition to the maximization term, we also add another term to keep the other parts unchanged, which can be formulated as maximizing Eipr oi where æ; is the 1-th token in the source sentence. The final update step is
Z, = Z, H + 3 Eite Oi.z,)
(30)
az.
where 3 = 0.1 in our experiments. The update step needs a pre-specified position f, where we enumerate all editing positions / (1, T.) and choose successful one with the smallest changes. We
24

---

Under review as a conference paper at ICLR 2021
also find using different optimizers rather than the simple Gradient Descent optimizer improve the performance, where we try Adan/Kingma & Ba (2013) and SGD with different learning rates.
A.6 APPLICATION II: UNSUPERVISED DECIPHERMENT
A.6.1 IMPLEMENTATION DETAILS
Unsupervised decipherment a task similar text style transfer, where we learn sequence-to- sequence model without parallel data. Following several previous models style transfer (Zhu et al., 2017 Yang et al. 2018 Dai et al. 2019), we adapt NAGAN to the task by introducing an encoder. An overview of the model is shown in Figure which can be formulated as:
Fx-y G cx), Fy-x ((E ex).
(31)
Cx and Cy are labels that specify the input type for the encoder and the desired type for the generator. Note a non-autoregressive generator, where the output length is equal to the source length. (The equal length is requirement of the task.)
For the reconstruction loss and the cycle loss, we use:
L Px- (( cx));
(32)
Layc, Egw [xd(x, Fy- (Fx-y(z)))]-
(33)
in Eq (32 is a noisy text modified from æ. where randomly drop and swap the order of some tokens (He al. 2020). d1r, (()) distance function, where we adopt the cross-entropy:
d(x, (()) = log(so)tmmx((:))1a))
(34)
where 8, is defined in Eq (8) and softmax( la,, means the 2ı-th dimension of the vector after softmax. The cross-entropy loss is commonly used in previous non- san-auttorerrssivee generators (Gu
et al. 2018).
For the adversarial loss, utilize discriminator based on language model following Yang al.) (2018), which is proven effective on style transfer tasks. The discriminator trained by cross-entropy based loss (de Masson Autume al., 2019):
L
LD.x = -Eax
log Pplaslare,,.x
+ E
(o(aa)(ak (35)
In Eq X' is the distribution from the generator, where x' is obtained following two steps: ~ Py. xt' Fx) The adversarial loss for the generator is defined as:
Ladv, Fealooee
Σ
log Po Po(zi|a's, cxc)
(36)
Tig
By swapping the roles X and V, we can obtain the other losses: Leye,y, Ludu,y, Cree.y, and LD.Y. The final losses be formulated follows:
Lg = Leye,X + Loye. + CLaads. + oLLadnyy + 3Lree,x + BBroey
(37)
LD = LD + LD.Y.
(38)
where a, 3 are hyper-parameters. G and D are optimized alternatively by gradient-based methods.
We utilize Transformer the encoder, whose input is the sum of the source sentence embedding and an embedding for the label Cx (or cy). The output of the Transformer regarded as the latent variable Z B 2_dimxx in Figure 6 The architecture is the same as the Transformer in the discriminator described in Appendix A.3.2
We adopt Transformer-based language model the discriminator, where the input is also added with the label embedding. The architecture of the Transformer the same as the encoder except for the attention mask. Note that the discriminator should pass the gradient back to the generator, where we also use one-hot representations of sentences and the trraight-througe estimator. A previous work adopt the same method, refer the interested reader to Tu al. (2020) for more details.
25

---

Under review as a conference paper at ICLR 2021
(A) Training
(B) Inference
cx
ex
Shared
Enc
Zx
Gen
Latent Space
GX
Crea
FX-M;
Z
Enc
2
Gen
y
Enc
Gen
Fy-+::
Zy
сy
g
Gen
Zz
Enc
Enc
Newdl intredazed Brooder
Gen
Non autoropresivve Cenerator
Ds
Droweentnayy baned Dsorimirutoo
Ladr. y
Dis
Figure 6: Overview of NAGAN on unsupervised decipherment. (A) Three losses of generators:
Crec, L'y Lade.y. The other three losses can be obtained by swapping the roles of X and V. Red dotted lines indicate operations bringing the non-difeerentiable problems. (B) The decipher model contains the newly introduced encoder and the non-autoregressive generator, where two types of text are encoded into shared latent space.
The non-autoregressive generator is kept unchanged, which is the core of our model. Autoregres- sive generators suffer from the non-differentiable problem when optimizing the cycle loss and the adversarial loss (Red arrows in Figure A)), where our non-autoregressive can optimized more effectively.
A.6.2 EXPERIMENT DETAILS
The Word Substitution dataset Yang et al. 2018) uses the word substitution cipher encrypt the plaintext. The dataset contains 9,445 words in vocabulary. 200,000 unpaired sentences for both A and Y in the training set, and 5,000 sentence pairs in both the validation and the test set. Following previous work (He et al. 2020), we use BLEU-4 for evaluation. We show several cases on the Word Substitution dataset in Table
The Brown W-200 dataset (Gomez et al. 2018) uses the Vigenère cipher for encryption. The dataset contains 200 words in the vocabulary and 51,606 (11,468) sentence pairs for the training (test) set. We break the connections of the paired sentences when training. Following previous work (Chen) al. 2018), we use accuracy for evaluation, which is calculated by the average proportion of words correctly deciphered. The dataset does not contain a validation set, so we choose the best model on the training set as our final model.
For hyper-parameters, we set a = 1. 3 = at the beginning of the training for the Word Subtituion dataset. Then they decay to 0 linearly in the first 100k batches. (We train the models for 200k batches.) The annealing method is proposed by He et al. 2020). Similarly, we set 0.3, 3 at the beginning for BrownW-22 dataset. On both datasets, we use the RAdam (Liu et al. (0220) optimizer with learning rate of le-3 to optimize the generator and the discriminator alternatively.
Table 14: Cases of unsupervised decipherment on the WordSub dataset. We show the deciphered text and the golden answer, where the ciphertext is omitted. Red words indicates the decipherment errors.
Generated
cost num dollars.
Golden
entry cost $ num dollars.
Generated
the wait staff and bartenders are rude.. the wait staff and bartenders are rude.
Golden
Generated
everyone wants help
Golden
everyone wants to help
Generated
enjoyed the area and searched for some drinks and sunglasses. all cleaned up enjoyed the and headed for drinks and gaming.
all cleaned
Golden
Generated
our server john was incredible cute, patient, attentive and funny our server james was incredible cute, patient, attentive and funny
Golden
26

---

Under review as a conference paper at ICLR 2021
A.7
OTHER DETAILS
For the COCO and SNLI dataset, we train our model for 100 epochs (each epoch contains 1500 batches of samples), evaluate it every epoch, and select the model with the best FED on the validation set. Each training run used approximately Intel Xeon E5-2690 v4 CPUs at 2.60GHz, and 1 Nvidia GeForee RTX 2080 Ti GPU. Although the results of 10-hour training are close to the reported performance (with gap of 0.01 in terms of BLEUHA H A), finish the 100 epochs for around 28 hours.
For the synthetic data, we train our model for 50 epochs. Each training run costs around 15 hours with the devices described above.
We also report the latency of one generator training step and the inference in Table 1. Compared with an autoregressive Transformer of the same architecture, NAGAN is slower in training (because our generator's updates need gradients from the discriminator) but faster in inference (6.66x speed up). The fast inference is brought by the parallel decoding. Compared with autoregressive text GANs of the same architecture, NAGAN also has a faster training step because autoregressive GANs need to read generated tokens as input.
For the unsupervised decipherment task, we train the model for 200k batches, evaluate it every 1.5k batches, and select the model with the best BLEU-4 on the validation set for the Word Subtitution dataset (the best accuracy the training set for the BrownW-200 dataset). Each training run cost around 37 hours with the same devices described above.
Table 15: Training and inference latency of an autoregressive Transformer and NAGAN on the COCO dataset. All results are evaluated with batch size of 32.
Model
Generator Training Step
Inference Latency
MLE-Transfomner NAGAN
145ms (1.00x)
213ms (1.00x)
210ms (0.69x)
32ms (6.66x)
27


Reviews:
[{"review_id": "vuZv-au-5jy", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces the non-autoregressive generator in the GAN-based text generation, making textGAN can be trained without pre-training and better utilize latent variables to control the style of generated text.\n\nIntroducing non-autoregressive architectures into GAN-based text generator is a natural idea, and the modelling ability of non-autoregressive generator has been verified at BERT. I think that this paper is above the average because it provides comprehensive experiments (including unconditional text generation, unsupervised decipherment and sentence manipulation), showing the significant improvement in various evaluation metrics compared to the text model without NAR and baselines.\n\nHowever, this paper should have more analysis of how non-autoregressive architectures work in the GAN-based text generation.\n1.\tHow latent variables in the non-autoregressive generator influence the content of the generated text? Can you provide some analysis or examples about it? (e.g. change the value of some latent variable continually (from 0 to 1?) and give the generated text).\n2.\tCan you give some analysis about the generation process of the non-autoregressive generator, (e.g. attention map), which makes the generator more interpretable.\n3.\tThe user study is absent.\n4.\tIs dropout necessary for the non-autoregressive generator? What if the dropout rate is 0, how the performance of generator changes?\n5.    Can you give more details about experiments, such as model parameters, training time, inference time and GPU you used?", "extra_scores": {}, "extra_text": {}}, {"review_id": "VZIslghG1gD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper creatively extends text GAN by introducing non-autoregressive generator, which is a well-known notion in translation and VAE like generation but not often applied in a GAN setting. The paper argues that a non-autoregressive generator brings more effective gradient-based optimization and also good latent representation learning capability. \n\nThe comparison between NAGAN and other text GANS reads okay, but the reviewer concerns the limited scope and significance of this paper.\n\n1, Given very strong text generation capability of MLE learning and pre-training, NAGAN makes little contribution to push the generation SOTA. Audiences of this approach are also limited. In this paper, given a very old baseline of MLE and a bunch of text GANs, the overall performance of NAGAN is still not much leading. Let alone compare it to other strong pre-trained generators.\n\n2. When claiming good latent representation learning capability, there should be a big gap between NAGAN and text VAEs in this aspect. If the author adds more control and manipulation experiments in text VAE, NAGAN will be not as shining as now.\n\n3. Non-autoregressive generator has difficulties in generalizing to long text generation and conditional generation. How does the author consider such settings, instead of simple unconditional generation in toy datasets like COCO?\n\nOverall, the reviewer thinks this is a well-written paper, but a boardline one considering its limited significance for the venue.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HLdDYmY3A", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThis paper proposed a new text GAN framework by combining non-autoregressive text generator based on transformer, straight-through gradient approximation, and various regularization techniques such as gradient penalty and dropout. The paper demonstrates the superiority of non-autoregressive generator in the context of text GANs through various experiments including unconditional text generation, latent space manipulation and unsupervised decipherment.\n\n**Pros**\n- This work narrows the gap between image GAN and text GAN by leveraging recent advances on non-autoregressive text generators and a straight-through gradient approximation. While these components are well studied in previous works, I think this work presents a neat combination of them in order to solve a well-known problem.\n\n- The paper provides rich discussions in training text GAN and comprehensive experiments and ablations to demonstrate the usefulness of an implicit text generator in different contexts.\n\n**Concerns & Questions to Answer during rebuttal**\n- The original text GAN papers were mainly motivated to address the *exposure bias* problem in maximum likelihood estimation for autoregressive generators. In other words, when we use an autoregressive generator to sequentially generate tokens one by one, there is a distribution mismatch between training and test phase. In your case, now that you already have a non-autoregressive text generator, is there any *theoretical motivation/insights* for using the adversarial training framework? We know that MLE is statistically efficient (achieving Cram´er–Rao Lower Bound) and possesses many good properties, maybe training the non-autoregressive text generator with MLE (e.g., FlowSeq [1] or variational inference) is a better choice?\n\n- The non-autoregressive (NA) text generator have been well studied recently so the novelty of this work is more on the integration of NA generator with adversarial training. Thus the main challenge here is how to solve the non-differentiability problem. The paper directly leverages a traditional workaround, the straight through estimator, which is a biased gradient approximation. Is the bias going to be an issue and is there any better strategy? I think the paper need to provide more discussions on this aspect. Overall the method section need to be polished with more details, as I feel this part is currently hard to follow.\n\n- In figure 1, $z_1, \\ldots, z_L$ are sampled independently, which are sent to a transformer and later produced the sample. Is the independence between $z_1, \\ldots, z_L$ going to be a problem? When we use transformer to do neural machine translation, the attention mechanism will capture the dependence in the input sentence ($z_1, \\ldots, z_L$ in this context) and then produce the output correspondingly. Hence will the independence in $z_1, \\ldots, z_L$ lead to a less expressive sample distribution in your text generator (although this is not an issue in image GAN)?\n\n- The paper also propose to use the Max Gradient Penalty from image GAN domain. The Max Gradient Penalty was introduced under the framework of Wasserstein GAN or Lipschitz GAN framework, which aims to constraint the function space to be Lipschitz smooth. However this work uses the vanilla GAN objective (eq 12 and eq 13), which is not the WGAN or LGAN framework. Thus the regularization may not be theoretically correct. Also why not instead use the WGAN objective which is empirically more stable and theoretically sound?\n\n- Experiments: In table 2, all the results for NAGAN use the dropout with a positive ratio. How does NAGAN perform without dropout? Also I wonder if the comparison in the table and figures are fair, since most previous methods.baselines such as MLE or SeqGAN only use a vanilla RNN/LSTM, while NAGAN has a more complicated structure with transformers, and additional regularization such as gradient penalty and dropout. Perhaps we should control at least the number of parameters to be in the same level.\n\n[1] FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow\n\nUpdate after rebuttal: \nAfter seeing the author response below, no change to my score.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes GAN-training of a non-autoregressive generator for text. To circumvent the usual problems with non-differentiability of text GANs, the authors turn to Gumbel-Softmax parameterisation and straight-through estimation. 

There are a number of aspects to this submission and they are not always clearly positioned. I will concentrate on the two aspects that seem most crucial:

1. The authors position their generator as an implicit generator, but it really isn't. If we take the continuous interpretation of the output distributions: the Gumbel-Softmax transformation does correspond to a tractable density, the Concrete density of Maddison et al, with known parameter. If we take the discrete interpretation of the output distribution: Gumbel-argmax is just an alternative to sampling from a Categorical distribution with known parameter. In either case, the generator maps the noise source to a collection of conditionally independent distributions each of which has a known parameter and analytical density/mass function. The authors do, however, train the architecture using a GAN-type objective *as if* the generator were implicit.

2. In the discussion phase the authors added that GAN training overcomes the independence assumptions made by the generator. Whereas that makes intuitive sense, it suddenly changes the emphasis of the contributions, from proposing an implicit generator (presumably powerful for it being implicit) to proposing a way to circumvent the strong independence assumptions of the generator with a mechanism other than more traditional approximate marginalisation of VAEs. In their rebuttal, the authors commented on the use of non-autoregressive VAEs in neural machine translation, and though those observations have indeed been made, they might well be specific to MT. The simplest and more satisfactory response would be to ablate the use of the GAN objective (that is, to train a non-autoregressive VAE, also note that, with the same choice of likelihood, posterior collapse is rather unlikely to happen).

Other problems raised by reviewers were addressed in the rebuttal, and I would like to thank the authors for that. For example, ablating the non-autoregressive generator and comparing to REINFORCE. I believe these improved the submission. 

Still, I cannot recommend this version for publication. I would suggest that the authors consider careful ablations of the components they see as precisely important for the results (that currently seems to be the GAN-like objective despite the model not, strictly speaking, requiring it). 





Decision: Reject