Improving Sequence Generative Adversarial Networks with Feature Statistics Alignment
Authors: Yekun Chai
Qiyue Yin
Junge Zhang
Venue:  (2021)

Abstract: Generative Adversarial Networks (GAN) are facing great challenges in synthesizing sequences of discrete elements, such as mode dropping and unstable training. The binary classifier in the discriminator may limit the capacity of learning signals and thus hinder the advance of adversarial training. To address such issues, apart from the binary classification feedback, we harness a Feature Statistics Alignment (FSA) paradigm to deliver fine-grained signals in the latent high-dimensional representation space. Specifically, FSA forces the mean statistics of the fake data distribution to approach that of real data as close as possible in a finite-dimensional feature space. Experiments on synthetic and real benchmark datasets show the superior performance in quantitative evaluation and demonstrate the effectiveness of our approach to discrete sequence generation. To the best of our knowledge, the proposed architecture is the first that employs feature alignment regularization in the Gumbel-Softmax based GAN framework for sequence generation. 

Under review as a conference paper at ICLR 2021
IMPROVING SEQUENCE GENERATIVE ADVERSARIAL NETWORKS WITH FEATURE STATISTICS ALIGNMENT
Anonymous authors Paper under double-blind review
ABSTRACT
Generative Adversarial Networks (GAN) facing great challenges in synthesiz- ing sequences of discrete elements, such as mode dropping and unstable training. The binary classifier in the discriminator may limit the capacity of learning signals and thus hinder the advance of adversarial training. To address such issues, apart from the binary classification feedback, we harness a Feature Statistics Alignment (FSA) paradigm to deliver fine-graing aeneggrined signals in the latent nigh-dimensional rep- resentation space. Specifically, FSA forces the mean statisties of the fake data dis- tribution to approach that of real data as close as possible in finite-dimensionl feature space. Experiments on synthetic and real benchmark datasets show the superior performance in quantitative evaluation and demonstrate the effectiveness our approach to discrete sequence generation. To the best of our knowledge, the proposed architecture is the first that employs feature alignment regularization in the Gumbel-Softrax based GAN framework for sequence generation.
INTRODUCTION
Unsupervised sequence generation is the cornerstone for a plethora of applications, such machine translation (Wu et al. (0166), image captioning (Anderson et al. (2018), and dialogue generation (Li al. 2017). The most common approach to autoregressive sequence modeling is maximizing the likelihood of each token in the sequence given the previous partial observation. However, using maximum likelihood estimation (MLE) for sequence modeling is inherently prone to the exposure bias problem (Bengio et al. 2015), which results from the discrepancy between the training and inference stage: the generator predicts the next token conditioned on its previously generated ones during inference but conditioned on its prefix ground-truth tokens during training. yielding accumu- lative mismatch along with the increment of generated sequence length.
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) can serve as alternative to models trained by MLE, which have achieved promising results in generating sequences of discrete elements, in particular, language sequences (Kusner & Hernández-
[016 Yu al.) 2017] Lin ltt al. 2017 Guo et al., 2001 8; Fedus et al.] [2088] et al. [2119 de Masson Autume et al. 2019 Zhou 2020 (Scialom | 2020). GANs consist of two competing networks: discriminator
that is trained to distinguish the generated samples from real data, and a generator that aims to generate high-quality samples to fool the discriminator.
Although having succeeded in avoiding exposure bias issues, GANs still suffer from some intrinsic problems, such mode dropping, reward sparsity, and training instability. To enrich the informa- tiveness of the discriminator's 's training signal, several approaches have been proposed by measuring the latent features, such as feature distribution matching (Zhang et al.] 2017 Chen al. (2018) and comparative discriminators (Lin et al. 2017 Zhou et al. 2020). Zhang et al. (2017) and Chen et al. (218)) leveraged feature matching mechanism by minimizing the kernel-based moment-matching metric, such as Maximum Mean Discrepancy and Earth-Man Mover's Distance, between encoded fea- tures. However, merely adopting feature matching in licu of the original learning signal may lack some guiding feedback at the initial stage of training.
Another approach is to compare the finite latent features with comparative discriminators like ranker and relativistic discriminator. Lin al. (2017) maintained that the binary classification in the dis- criminator network limits the learning capacity of tasks because the diversity and richness are cir- cumscribed by the degenerated distribution. RankGAN (Lin et al. 2017) replaced the binary clas-

---

Under review as a conference paper at ICLR 2021
Real
Relativistic Discriminator
Real
Real
Fake
Discriminator
Real/ Fake
Freat
Gonerator
Generator
Feature Alignment
fyuke
Standard GANs
FSA GANs
Figure 1: (a) Standard GANs using binary classifier as its discriminator; (b) GANs with Feature Statistics Alignment and relativistic discriminator that provide more instructive signals for updating the generator.
sifier with pairwise feature ranker by comparing the similarities between sample features in the latent space. SAL (Zhou al., 2020) classified the encoded features of constructed pairwise train- ing examples into three categories, i.e., better / worse / indistinguishable. Nevertheless, adopting comparative discriminator could provide the sine-grained signals for updating the generator network and may require some coarse credits for further improvements.
In this work, we propose to improve the GANs for sequence generation by jointly considering both the feature statistics matching and relativistic discriminator to serve fine-graie ine-grained and coarse learn- ing signals respectively. We leverage the Feature Statistics Aligmment (FSA) paradigm to embed the latent feature representations in finite feature space and force the distribution of generated samples to approach the real data distribution by minimizing the distance between their respec- tive feature representation centroids. Intuitively, matching the mean feature representations of fake and real samples could make the two data distributions closer. Besides, the relativistic discrim- inator
2019) is employed to measure the comparative information between generated and real sequences and empirically to show the effectiveness during the model training.
Our experimental results illustrate the effectiveness of FSA techniques and large batch size al- leviate the gradient vanishing problem and stabilize the training process in comparison with the vanilla Hambel--Sffmaa GANs. Besides, our models could generate discrete text sequences with high quality in terms of the semantic coherence and grammatical correctness of language, as eval- uated with crowdsourcing. Furthermore, we empirically demonstrate that the proposed architecture overshadows most existing models in terms of quantitative and qualitative evaluation. To the best of our knowledge, the proposed framework is the first to adopt the statistics feature alignment paradigm in the Gumbel-Soffi Softmax based GAN framework for discrete sequence generation.
ADVERSARIAL SEQUENCE GENERATION
Adversarial sequence generation has attracted broad attention for its properties to solve the exposure bias issue suffered with maximum likelihood estimation (MLE) for generating language sequences. Based on the game theory, its goal is train generator network G(2; ((G) that produces sam- ples from the data distribution Pasa (x)) ) by decoding the randomly initialized noise z (i.e., standard normal distribution) into the sequence æ Giz; e (G)). where the training signal is provided by the discriminator network (ax; ((D) that is trained to distinguish between the samples drawn from the real data distribution Puata and those produced by the generator. The minimax objective of adversarial training is formulated as:
min max Earpar log D(x; (D)) + Eznpp [log (1 (1-D Daci (G(zz; (z) @( (DD)
Despite the impressive results of GANs in the sequence generation (Yu et al. 2017 Guirajani et al.) 2017 Scialom et al.) 2020), there are still several fundamental issues in the GAN training: (a) Training instability, which arises from the intrinsic nature of minimax games in GANs; (b) Mode dropping, which is the fact that GANs only generate samples with limited patterns in the real data distribution instead of attending to diverse patterns (Chen et al. 2018); (c) Reward sparsity, which because that it is easier to train the discriminator than the generator, making it difficult to acquire the instructive feedback (Zhou et al. 2020).
2

---

Under review as a conference paper at ICLR 2021
Due to the non-differentiability of gradients caused by sampling operations between the generator and discriminator for sequence generation, the majority previous works have resorted to rein- forcement learning (RL) heuristics with Monte Carlo search collect the credits from the discrimi- nator. The usage of RI may further deteriorate the instability of model training and exacerbate the reward sparsity problem.
relaxation has proven be alternative RI. tech- niques (Kusner & Hernández-Lobald 2016) Nie et al. 2019). How to efficiently train GANs with the Gumbel-Softmax trick still remains under-explored. Therefore, utilize the Gumbel-Softmax reparameterization instead of conventional policy gradients in our framework.
METHODOL.OGY
As illustrated in Fig. standard GANs employ the real/fake binary classifier as the discriminator, which is prone to overtrained in comparison with the generator (Salimans et al.] 2016). To prevent the discriminator from overfitting and further stabilize the training process, we propose to leverage the FSA techniques and relativistic discriminator by comparing the fake and real distributions from two different aspects.
Compared with conventional GANs, the proposed framework enjoys the following advantages: (a) In the earlier training stage, the relativistic discriminator could estimate the probability that how much better the real data is in comparison with the generated samples. This could provide the rela- tively "coarse" credits to the generator. (b) When it comes to the later training stage and the quality of generated samples becomes high, FSA measures the "fine-grained" difference between latent fea- tures of fake and real data batches, precluding the discriminator from being overly confident. In other words, the FSA can be regarded as a kind regularization for adversarial training.
3.1 FEATURE STATISTICS ALIGNMENT
Intuitively, aligning the statistics of embedded feature representations increases the opportunities to capture the various modes of the data distribution. For brevity, we utilize the first-order mean statisties our framework and leave the higher-order statistics for future work.
Denoting the feature extractor as F. parameterized by w, a minibatch of real data samples as æ with the batch size of N, we propose two variants of FSA formulations, which calculate the mean squared difference and Euclidean distance between the minibatch centroids of real and fake feature representations. To reduce the parameter amount, we share the weights between the discriminator and feature extractor of FSA.
Mean Squared Alignment (MSA) We take the mean squared difference between the centroids of fake and generated distributions the feature alignment metrie:
LMSA
|| Bamppn (a))
Ezspa
Fss (G(2;
N
N
(x
FG (G)
(3)
Mean Distance Alignment (MDA) Another intuitive approach is to calculate the distance be- tween two sample centroids, which is equivalent to the square root of MSA mathematically:
LMDA = 1 [a)) Ezzop, (((
(4)
LMMA
(5)
By forcing the mean statistics of fake data to be close to the real samples, the generator could receive more informative signals in the training process. It is worth noting that the large batch size helpful to reduce the variance of small mini-batches.
3

---

Under review as a conference paper at ICLR 2021
3.2 RELATIVISTIC DISCRIMINATOR
We consider the Relativistic Discriminator (ollcoeur-Martineuu 2019) to take into account the rel- ative confidence that the given real data is more realistic than the randomly sampled fake data. In the standard GAN, defining the diseriminator as D(x) = sigmoid( H (æ)), where H(.) represents the non con-transformed layer before the final non-linearity. The objectives for the discriminator and generator in terms of the Relativistic Discriminator are defined as:
CRD -Ex & pmpp [log a (H H(G((; ()))
(6)
Lr -ee- [log a (G(( - 11 (x))),
(7)
where a represents the activation function to be relativistic (we use sigmoid function in our experi- ments), RD and RG denote the loss terms for the discriminator and generator respectively.
3.3 OVERALL TRAINING OBJECTIVES
Previous works using moment matching techniques to support the training on both the discriminator and generator (Zhang et al. 2017 Chen al. 2018). However, the generator is always more difficult [
to train than the discriminator, resulting in the training instability and reward sparsity. To relieve these issues, we only adopt the FSA techniques enhance the generator but keep the objective of the discriminator unchanged. This could pass more informative signals only to the generator, and also prevent the discriminator to be overtrained.
Therefore, the overall training objectives for the proposed framework are defined as:
LD = LRD.
(8)
LG = LRG LESA
(9)
where Lesa takes the form of LMSA and LMDA as Eq. 2 and Eq.
The goal of the discriminator is to maximize the gap between the generated and real data, whereas the generator jointly considers two different aspects simultancously: it not only competes with the discriminator by maximizing the gap in terms of the relativistic signals but takes into account the additional leaked feature information from the discriminator. The idea leaked features from the discriminator is similar to LeakGAN (Guo et al. 2018). The FSA term the RHS can also be regarded as a dynamic regularizer for the sequence generator.
3.4 TRAINING WITH DISCRETE SEQUENCE
3.4.1
GUMBEL-SOFTMAX DISTRIBUTION
Conventional GANs for generating discrete sequences are inherently unable backpropagate the gradient through samples due to the din-differentiable sampling from categorical distribution. Gumbel-SSftmax distribution (Jang et af., 2017] Maddison et al., (017)) was proposed to deal with this issue by smoothly annealed to approximate the categorical distribution.
Denoting the output probabilities I .T. ... TVV,, where V represents the output vocabulary size in the generator, the Gunbel-Max trick (Maddison et al. 2014) can be parameterized as:
y/ one hot arg maxig: + log =.)) :or==1..... VV,
(10)
where [g. = 1,.... ... [V] are i.i.d from the Gumbel(0,1) distribution, that is, 9i = - log(- - log u.) 1, with is drawn from standard uniform distribution Uniform(O.1). one hot represents the |V|- dimensional one hot encoding.
Gunbel-SSfttax approximates the non-differential arg operation using the softmax function: f exp ((log(Ti) + g:)/T)
for 1,... ,
(11)
==1 exp (log(az) + 9
where 7 denotes the softmax temperature to modulate the exploitation and exploration during train- ing. When 7 approaches too high, the approximation is nearly equiprobable, encouraging the gener- ator explore different options. In contrast, the lower could discourage the exploration and tend to exploit during training. In particular, when 7 + 0, 9, approaches the result of one hot operator as in Eq. 10 whereas ý will degenerated into uniform distribution when xc.
4

---

Under review as a conference paper at ICLR 2021
3.4.2 ARCHITECTURE AND ADVERSARIAL TRAINING
RNN Generator Due to the free-running mode of GAN's generator, it is unsuitable to adopt the transformer-base models due to the non-recurrence nature. Thus, investigate the Recurrent Neural Network (RNN) based models, such as Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber 1997). and Relational Memory Core (RMC) (Santoro et al. 2018).
CNN Discriminator & Feature Extractor We use the convolutional neural networks (CNN) ar- chitecture (Kim 2014) the discriminator and feature extractor for input sequences. We employ the multi-channel convolution using multiple filters with various window sizes to extract the distinct n-gram features, followed by max-over-time pooling operation to gather the most salient features, Le., features with the highest value for each feature map.
Adversarial Training Algorithm Alg. T illustrates the overall training process of the proposed framework. The Relativistic Discriminator and the generator could reach the Nash Equilibrium when the generator could fool the discriminator into accepting its output as being true. Since the discriminator is easy to be overtrained, we do not pretrain the discriminator but only pretrain the generator using MLE for few epochs.
Algorithm Adversarial Training with Feature Statisites Alignment
Require: generator Gø: discriminator Dş: samples of real data 8; generator training step gi discriminator training step k; the generator pretraining epochs m.
2: Pretrain Go using MLE on S for m epochs
3: repeat
4: for ! steps do
5: Sample minibatch from real data S
6: Generate minibatch of samples 2g ~ Go
7: Update Gg via (q.
8: end for
9:
for k steps do
10: Sample minibatch from real data Sample minibatch from the generated data Train the discriminator D& by Eq. end for until convergence
12:
14:
EXPERIMENTS
4.1
EXPERIMENTAL SETTING
Similar to (Lin et al. 2017 Guo et al. 2018 Nie et al. 2019 Zhou et al.) 2020). we evaluate the
proposed framework based the Texygen benchmark platform (Zhu et al. 2018 for adversarial text generation. Experiments were conducted on the synthetic and real datasets: (a) synthetic data,
which is generated by oracle single-layer LSTM as in (Yu al. 2017); (b) MS COCO Image
Caption dataset (Chen et al. 2013); (c) EMNLP WMT 201 News dataset (Guo et al. 2018 Table summarizes the statisties of benchmark datasets for evaluation.
Table 1: Summary of experimental datasets.
dataset
vocabulary size
sequence length
training
test
synthetic data
5,000
20
/ 40
10,000
10,000
MS COCO
4.657
37
10,000
10,000
EMN1.P2017 WMT News
5,255
51
27,8586
10,000
For the synthetic data experiments, we utilize a single-layer LSTM initialized by the standard nor- mal distribution as the oracle model, which is used generate 10,000 samples of length 20 and 40
5

---

Under review as a conference paper at ICLR 2021
respectively as the real samples. We use the negative log-likelihood (NLL) under the oracle data dis- tribution for evaluation, termed NoLoracee- For the real data experiments, the BLEU score PPaiineni et al. 2002 serves metric to evaluate the n-gram statistics overlapping the whole dataset.
To measure the diversity of generated samples, the NLL of the generator (denoted as NLLgea) is used by computing the NLL of reference samples in the test set by the generator. Considering that BLEU scores always focus on the local text statistics and may be insufficient for evaluating the overall quality of texts, we conducted additional human evaluation via crowdsoureing on the generated samples of all comparison models. See Appendix A for more experimental details.
We compare the proposed framework with the MLE baseline and other state-of-the-art models, involving SegGAN (Yu et al., 2017), RankGAN (Lin et al., (017)) LeakGAN (Guo et al. 2018). ReIGAN (Nie al. 2019), and Sclf-Adversarial Learning (SAL) (Zhou et al. 2020).
4.2
EXPERIMENTAL RESULTS
4.2.1
SYNTHETIC DATA
Siblttiilustrrttes the performance of different models on NLLorale- Since our experiments achieved better results using MSA instead of MDA on synthetic data, we only report the optimal results with MSA in Table Our models outperform other prevalent adversarial models in terms of the generated sample quality, demonstrating the effectiveness our proposed method.
In practice, found that the LSTM generator could outperform the RMC generator for producing the sequence with the length of 20, whereas RMC generators exceed LSTMs for long sequence generation. This may be due to the fact that LSTMs may forget the long-term dependencies with the sequence length increases, but the self-attention based relational memory cell RMC could mitigate the issues using interactive memory slots. This demonstrates that the proposed model could produce samples with high quality. As to the diversity, our method achieves competitive NLL NLLggnn score compared with baselines models (see Appendix C.T for the detail).
Table 2: The NLLonall performance of different models (r 1) the synthetic dataset with the sequence length of 20 and 40 respectively. For NLL, the lower, the better.
Length
MLE
SeqGAN
RankGAN
LeakGAN
ReIGAN
SAL
Ours (LSTM)
Ours (RMC)
Real
20
9.038
8.736
8.247 9,958
7.038 7.191
6.680
7.71
5.047
5.819 5.087
5.750 4.071
40
10.411
10.310
6.765
9.31
5,909
4.2.2
MS COCO DATASET
To further test the performance the real data, run and evaluate our model MS COCO image caption datasets. The data and preprocessing remain the same as in Texygen (Zhu et al. 2018)) Empirically, we found that RMC achieves better results in terms of the long sequences, and thus reports the results using the RMC generator if not otherwise specified.
Table 3: BLEU and NLLggen performance on MS COCO image captions with T 0.1 for the proposed models with MSA and MDA. For BLEU scores, the higher, the better.
Model
BLEU-2
BLEU-3
BLEU-4
BLEU-5
NLL
MILE
0.731
0.497
0.305
0.1889
0.718
SeqGAN
0.745
0.498
0.294
0.180
1.082
RankGAN
0.743
0.467
0.264
0.156
1.344
LeakGAN
0.746
0.528
0.355
0.230
0.679
RelGAN
0.849
0.687
0.502
0.331
0.756
SAL
0.785
0.581
0.362
0.227
0.873
Ours (MSA)
0.959
0.866
0.759
0.630
0.760
Ours (MDA)
0.938
0.863
0.731
0.582
0.717
Table 3 exhibits the final results of the BLEU and NLLgen scores different comparison models. Our models reported in Table the softmax temperature as 0.1 for the model with MSA and that with MDA. Notably, model shows the significant improvement tently overshadowing the state-of-the-an models in terms of the sample quality (indicated by BLEU scores) while maintaining the diversity (indicated by NLIggen).
previous methods, consis-

---

Under review as a conference paper at ICLR 2021
4.2.3 EMNLP2017 WMT NEWS DATASET
presetts the considerable improvements of our model (w.rt. quality) on EMNLP2017 WMT News dataset, with the temperature of for models with MSA and MDA. The maximum length in the EMNLP2017 dataset is 51, greatly challenging the generation task.
It can be observed that only RankGAN and LeakGAN slightly outrank the MLE in terms of the NL which may be due to the fact that feature ranking or leakage information could pass more internal information from the discriminator to the generator, and thereby smoothly assist the training process of the generator. By leveraging the FSA methods, our model greatly outperforms these models and yielding the long sequences with promising qualities.
Table 4: The BLEU and NLLgen performance on EMNLP2017 WMT News dataset with tempera- tures of I for the proposed models with MSA and MDA.
Model
BLEU-2
BLEU-3
BLEU-4
BLEU-S
NLL gon
MLE
0.768
0.473
0.240
0.126
2.382
SeqGAN
0.777
0.491
0.261
0.138
2.773
RankGAN
0.727
0.435
0.209
0.101
3.345
LeakGAN
0.826
0.645
0.437
0.272
2.356
ReIGAN
0.881
0.705
0.501
0.319
2.482
SAL
0.788
0.523
0.281
0.149
2.578
Ours (MSA)
0.932
0.798
0.585
0.404
3.999
Ours (MDA)
0.916
0.784
0.592
0.386
2.732
Apart from the automatic quantitative evaluation, we also conducted human evaluation on the MS COCO Image Captioning dataset. We randomly sampled 100 sentences for each model and the real data, then asked 12 different people to score them on the scale of 1-5 with anonymizing the model's identity. Please see Appendix. C.2 for more details of human evaluation.
Our models received the highest score in comparison with other models, where the model with MSA receives higher credits than that with MDA. By manually going through the generated samples, found that models with MSA tend to generate long sentences with higher quality but run into the mode dropping issue. contrast, models with MDA posses better trade-off between the diversity and quality of samples, yielding promising results in comparison with previous models.
Table 5: Mean and standard deviation results of human evaluation w.r.t different models on MS COCO Image Caption dataset. Note that "Real" indicates the real data samples.
Model
MLE
SeqGAN
RankGAN
LeakGAN
MaliGAN
Human score
3.127 0.124 3.062 0.100 3.048 0.127
3.018 0.104 3.031 0.123
Model
TexIANN
ReIGAN
Ours (MSA)
Ours (MDA)
Real
Hurman score
1.973 + 0.168 3.687 + 0.181
4.209 + 0.245
3.878 + 0.276
3.444 1 0.122
4.3 DISCUSSION
Fig. 2 (Left) illustrates the ablation of FSA terms of the BLEU-4 score (See Appendix D.2 for all the results). It can be observed that the increasing trend for models with FSA outreached that w/o feature alignment mechanism and our models achieve superior performance in the later training stage. The utilization of FSA leads to significant performance gains, as it could provide the consecutive ffined-graindd smoother learning signals to update the generator.
Fig. 3 demonstrates the relative importance of each component of our models with ablation test. The usage of FSA results in the most significant performance gain, followed by Gumbel-Softmnn and large batch size. A large batch size can boost the performance due to the variance reduction, accurate estimates of feature statistics, and stabilizing effect during the adversarial training (see Fig. 2 (Right)).
As to the Gumbel-Sofine Softmax trick, we conduct experiments on various temperature values and find that fined-tuned temperature hyperparameter could account for the second important factor to contribute to the overall performance boost. Please check Appendix for the detailed results w.r.t the Gumbel-Softmax temperature.
7

---

Under review as a conference paper at ICLR 2021
Cura we 154
9as IPSA)
Cura IMDN
Oux (VG0))OR 6an-M
Okiss SMOA
2.7
24
2.
24-
53-
03
3000
1500
500
1000
1508
2093
Pacstion
Figure 2: (Left) BLEU-4 scores of the proposed model w/ and w/o FSA mechanism. (Right) BLEU- 4 scores of the proposed models with various batch size (64 v.s. 128). The vertical dash lines indicate the end of generator pretraining. MS COCO results unless otherwise specified.
RELATED WORK
There has been large category of GANs for sequence generation, which heavily rely on the
Figure 3: Ablation Study of the proposed model.
RL paradigm. SeqGAN (Yu et al. 2017) re- gards the sequence generation
Markov de-
cision making process, estimates the rewards via Monte Carlo search, and trains the genera- tor with policy gradient. RankGAN (Lin et al.] 2017) and SAL (Zhou al., 2020) replace the binary classifier in the discriminator as compar- ative discriminators to take into account the re- lation between constructed pair samples. MaliGAN (Che et al. 2017) utilizes the information in the discriminator as an additional source of training signals on the MLE objective to reduce the variance of gradients. LeakGAN (Guo et al.) 2018) leaks the intermediate feature information via a manager to guide the generator, which is inspired by hierarchical RL. ColdGAN (Scialom et al.) 2020) integrates the advance of importance sampling, Proximal Policy Optimization (PPO) algo- rithm (Schulman et al.] (017)), and nucleus sampling (Holtzman et al.) 2019) for finetuning the pretrained T5 (Raffel et al. 2019) and BART (Lewis et 2019).
Another approach applics non-RL methods for adversarial sequence generation by either ap- proximating the categorical sampling or directly using the continuous latent representation. TextGAN (Zhang et al.) 2017) uses feature matching techniques via a kernelized discrepancy in the Reproducing Kernel Hilbert Space. FMGAN (Chen et al. 2018) proposes to match the feature distributions using a Feature-Mover's Distance. Similar to our proposed model, both of them ap- ply annealed soft-argmax for approximation. ARAML et al. (2019) utilizes Reward Augmented Maximum Likelihood by sampling from the stationary distribution to acquire rewards. However, none of them adopt the Gumbel-Max trick reparameterize the categorical sampling. Besides, they applied feature matching as the training objectives of both the discriminator and generator, whereas we only apply the feature statistics matching to modulate the generator. Gumbel-SSftmn
(GS) GAN (Kusner & Hernández-Lobato 2016) and RelGAN (Nie et al.) (019) prove the effectiveness of Gumbel-Softmax on unsupervised sequence generation. Dialoge WAR Gu et al. (2018) employs a GS GAN within the latent variable space for dialogue generation. However, improving the training of GS GANs still remains an open problem. Our model aims to promote the GS GAN with the proposed FSA and other techniques to boost the training of language GANs. We also report list of other techniques we tried but proved to be unsuccessful or unnecessary in Appendix B
6 CONCLUSION
We propose an adversarial training framework for discrete sequence generation, by leveraging the advance of Feature Statistics Alignment and Gumbel-Softmax relaxation. Our model empirically shows superior performance in terms of the quantitative and human evaluation. In the future, would be promising direction extend the proposed model to conditional text generation, such as text style transfer.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Peter Anderson, Xiaodong Hc, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. uottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 6077-6086, 2018.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Sys- tems, pp. 1171-1179, 2015.
Tong Che, Yanran Li, R. Zhang, R. Devon Hjelm, W. Li, Y. Song, and Yoshua Bengio. Maximum- likelihood augmented discrete generative adversarial networks. ArXiv, abs/1702.07983, 2017.
Liqun Chen, Shuyang Dai, Chenyang Tao, Dinghan Shen, Zhe Gan, H. Zhang, Yizhe Zhang, and
L.. Carin. Adversarial text generation via feature-mover's distance. ArXiv, abs/180) 1809 180906297,, 2018.
Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015.
Cyprien de Masson d' Autume, Mihaela Rosca, Jack W. Rae, and Shakir Mohamed. Training lan- guage gans from scratch. ArXiv, abs/1905.09922. 2019.
William Fedus, lan Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the_. arXiv preprint arXiv: 2018.
Ian Goodfellow, Jean Pouget-Abadie, M. Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Xiaodong Gu, Kyunghyun Cho, Jung-Woo Woo Ha, and Sunghun Kim. Dialogwae: Multimodal response generation with conditional wasserstein auto-encoder. arXiv preprint arXiv: 12352, 2018.
Ishaan Gulrajani, F. Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Improved training of wasserstein gans. In NIPS, 2017.
Jiaxian Guo, S. Lu, Han Cai, W. Zhang, Y. Yu, and Wang. Long text generation via adversarial training with leaked information. ArXiv, abs/ hass/709.08624 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint 1rXiv:1904.09733 090050 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. ArXiv, abs/1611.01144 2017.
Alexia Jolicoeur- Martineau. The relativistic discriminator: a key element missing from standard gan. ArXiv, abs/1807 00074007744.11 2019.
Pei Ke, Fei Huang, Minlie Huang, and Xiaoyan Zhu, Araml: A stable adversarial training framework for text generation. arXiv preprint 19Xiv:198.007111 2019.
Yoon Kim. Convolutional neural networks for sentence classification. arXiv preprint 14X8:1408.5882 2014.
Matt J. Kusner and José Miguel Hermández-Lobato. Gans for sequences of discrete elements with the gumbel-softmax distribution. ArXiv, abs/1611.0405/ 2016.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequenee pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 1910. 13461, 2019.

---

Under review as a conference paper at ICLR 2021
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1701.06547, 2017.
Kevin Lin, Dianqi Li, Xiaodong He, M. Sun, and Zhengyou Zhang. Adversarial ranking for language generation. ArXiv, abs/1705. 1001, 2017.
1001,
Chris Maddison, Daniel Tarlow, and Tom Minka. A sampling. In Advances Neural Information Processing Systems, pp. 3086-3094, 2014.
Chris J. Maddison, A. Mnih, and Y. Tch. The concrete distribution: A continuous relaxation of discrete random variables. ArXiv, abs/16l1 1/s/16111000122 2017.
Weili Nie, Nina Narodytska, and Ankit B. Patel. Relgan: Relational generative adversarial networks for text generation. In ICIRR, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Tim Salimans, Ian J. Goodfellow, W. Zaremba, Vicki Cheung, A. Radford, and Xi Chen. Improved techniques for training gans. ArXiv, abs/1666.03444 2016.
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. In Advances in neural information processing systems, pp. 7299-7310, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1700.7077777777 2017.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. Coldgans: Taming language gans with cautious sampling strategies. ArXiv, abs/2006.04643, 2020.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 08144, 2016.
Lantao Yu, W. Zhang, J. Wang, and Y. Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI, 2017.
Yizhe Zhang, Zhe Gan, K. Fan, Z. Chen, Ricardo Henao, Dinghan Shen, and L. Carin. Adversarial feature matching for text generation. In ICML, 2017.
Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, and M. Zhou. Self-adversarial learning with com- parative discrimination for text generation. ArXiv, abs/2001. 11611, 2020.
Yaoming Zhu, S. Lu, L.. Zheng, Jiaxian Guo, W. Zhang, J. Wang, and Y. Yu. Texygen: A bench- marking platform for text generation models. The 41st International ACM SIGIR Conference on Research Development in Information Retrieval, 2018
10

---

Under review as a conference paper at ICLR 2021
A EXPERIMENTAL DETAILS
A.1 TRAINING DETAILS
Generator The input embedding dimension for the generator is 32. As for the LSTM generator, we use the single-layer LSTM with the hidden dimension of 32. Then adopt a linear transformation get the logits at each time step, and iteratively feed the sampled output tokens into the generator at the next time step. As for the RMC generator, we follow the experimental settings as (Nie al., 2 setting the memory size as 256, memory slots as I, attention head number as 2. After the one-layer RMC, linear projection is applied to get the output logits at each step.
CNN Discriminator The input embedding dimension for the discriminator set to 64. We adopt the filter size of {2, 3, 4, with the number of 300 channels for each. A max-over-time pooling adopted after the convolution layer. Afterward, a highway layer that is identical to SeqGAN (Yu et al., (017) is used followed by linear transformation with the dimension of 100. Finally, apply a linear transformation to get the final logits. The feature extractor for Feature Statistics Alignment shares the identical architecture and weights with the CNN discriminator, and the leaked feature dimension is set to 100.
Optimization We use Adam optimizer with 0.9 and 0.999. The initial learning rate for the generator was set to le-2 and le-4 for pretraining and adversarial training. We set the initial learning rate as le-4 for the discriminator during the adversarial training. To prevent overfitting, we clip the gradients of parameters whose L2 norm exceeds 5.
Training Procedure We conduct experiments finetune the following experiments: the batch size of {32, 64, 128), the Gumbel-Softmax temperature 7 E 0.5, 0.1, 0.1,0.01.0.0011 The training steps of generator and discriminators are set to g = 1 and = 5, respectively. The generator is pretrained for 150 epochs before adversarial training. Finally, the optimal batch size is set to 128 for both synthetic and real datasets. is worth noting that we also test the batch size to 256, which requires too much GPU resource but do not show obvious improvement.
B NEGATIVE RESULTS
Here we list some approaches that we tried but proved unsuccessful:
Using Mogrifier LSTM as the generator, which achieves similar results as vanilla LSTMs the synthetic data.
Using a Wasserstein loss instead of current Relativistic Discriminator. Not as stable as current solutions.
Using the Transformer model as the discriminator. It achieves unsatisfied results with the current experimental settings.
Using interleaved training instead of two-stage training, i.e., adversarial training after pre- training. It is unsuccessful to train the generator for 15 iterations after one iteration using MILE.
Using top-k sampling and nucleus sampling. instead of the argmax in the Gumbel-Max trick. This does not always boost the final performance.
Using hinge loss the discriminator. This did not improve the current relativistic loss.
C EVALUATION DETAILS
C.I NLLGEN ON SYNTHETIC DATA
Table δ reportt the NLLgen metric of comparison models. It can be seen that the proposed model with RMC generator achieves comparative results in comparison with baselines in terms of both short and long texts.
11

---

Under review as a conference paper at ICLR 2021
Table 6: The NLLgen performance different models (r 1) on the synthetic dataset with the sequence length of 20 and 40 respectively. For the NLL score, the lower, the better.
Length
MLE
SeqGAN
RankGAN
SAL
Ours (LSTM) Ours (RMC)
20
5.96
6.61
7.14
6.58
7.73
5,12
40
6.55
6.98
7.05
6.97
7.59
6.89
C.2 HUMAN EVALUATION
Acceptance (i.e. if sentence is acceptable), grammaticality (i.e., if sentence is grammatically correct), and meaningfulness (i.e., if sentence makes sense) are three main standards for the text quality evaluation. Please note that any minor text formatting issues which will not negatively influ- ence the understanding and correctness of the sentences (e.g., punctuation, capitalization, spelling errors, extra spaces) can be ignored. Please also note: sentence consists of less than 10 words should get one point deducted. Table helow gives more detailed criteria.
is worth mention that the human evaluation used to measure the quality of generated sentences rather than the diversity.
Table The human evaluation scale from 1 with corresponding criteria and example sentences.
Scale
Criterion & Example
Grammatical, acceptable, meaningful. For example, "a man is carving under yellow planes Include 2 tiny grammatical errors, and the whole sentence is mostly acceptable and meaningful. For example, "two giraffe standing in front of them Include major grammatical errors, but the whole sentence is still acceptable and making sense. For example, "a kitchen with a grill roll from him
5 Excellent
4 Good
3 Fair
Include severe grammatical errors, and the whole sentence does not make sense, but some parts still acceptable. For example, "a motorcycle It hasically string of words with random order and totally ungrammatical. The entire sentence does make any sense. For example, city..
2 2- Poor
paved road the freeway
Unacceptable
C.3 HUMAN EVALUATION ANALYSIS
The model with MSA tends to generate grammatically correct sentences, and the sentences tend to be longer. For example, "A man is sitting motorcycle busy street, in city." Though it produces the samples with high quality by human evaluation, however, it does not solve the mode dropping collapse.
In contrast, models with MDA tend to generate more variety of sentences rather than repeated ones. Most sentences are grammatically correct and meaningful. They follow the SVO sentence structure with Preposition Phrase (PP) placed at the acceptable position in sentence. Even though some of the auxiliary or main action verbs are missing, the meaning of each sentence can still be understandable and making sense. There is no obvious mode dropping issues according to the generated samples of MDA.
D DETAILED RESULTS OF ABLATION STUDY
D.1 IMPACT OF FEATURE STATISTICS ALIGNMENT
See Fig. 4 for the results of the ablation study on FSA techniques.
D.2 IMPACT OF LARGE BATCH SIZE
Fig. 5 shows the training curve of our model various batch sizes. It is observed that the increase of batch size could provide the performance boost, due to the variance reduction of gradients and the stability of adversarial dynamics.
12

---

Under review as a conference paper at ICLR 2021
blouz
0.0
0.4
3.5
1000 Iteration
1500
2000
1000
3503
2000
Iteration
Oum we PSA
Ours (MSA)
Durs (MDA)
Ours win FSA
Ours (MSA)
Ours IWDN
0.0
0.5
25
0.4
0.2
1000
15.00
2000
500
2000
1503
2000
Iteration
Iteration
Figure 4: Training curves of BLEU scores on MS COCO Image Caption dataset w/ and w/ FSA mechanism.
D.3 IMPACT OF GUMBEL-SOFTMAT TEMPERATURE
Fig. reports the BLEU scores with different temperatures on MS COCO dataset. It can be seen that suitable r could greatly advance the automatic evaluation scores.
GENERATED SAMPLES ON REAL DATASET
E. GENERATED SAMPLES ON MS COCO DATASET
Table displays samples of generated samples from all baseline models and references on MS COCO dataset.
Table 9 shows the randomly sampled sentences from the proposed models generated on MS COCO Dataset.
E.2 GENERATED SAMPLES ON EMNLP2017 WMT NEWS DATASET
Table 10
presents the random sampled sentences from models generated EMNLP2017 WMT News Dataset.
13

---

Under review as a conference paper at ICLR 2021
Our (MGA)INRin SENN
Cars balch 120
(VISA) boach
Oue (MSA)
0.75
0.73
0.5
0.95
0.63
1000
1500
1500
Ourn (MI))) size
Guts (16AA) dal 128
Ourr. (USAA) hokh site
Our (MGA) bunnh wan
066
0.5
0.4
1500
203
1500
leraton
lension
Figure 5: Training curves BLEU scores on MS COCO Image Caption dataset with various batch sizes.
Table Samples of baseline models and real dataset on MS COCO Image Captioning dataset.
Samples
a single kite flies high above body water as a person stands on the edge the water. a man wearing an apron in an industrial kitchen reaching for pot
Real
watches on his bike, in lake field.
MLE
women is standing behind orange table in helmet child in the background.
some people sitting on top of luggage near a truck.
SeqGAN
a man sitting a bath tub on tops
riding motorcycle. bathroom with a sink and table
TextGAN
a man standing next to her cell phone on a street sign. a woman is holding child in the air
LeakGAN
woman is standing and another nak cake drain.
MaliGAN
a man standing in kitchen with her laptop and two tables
a colorful bike is is down next to large mirror.. man is riding bike down track.
RankGAN
a woman walking with a dog the city front a city bus
RelGAN
a man sitting on bed a room with chair on the couch.
man is sitting motorcycle motorcycle crowded street near building with bicycle in parking lot
busy street, in city
Ours (MSA)
man sitting
a person is riding a motorcycle on a city street with a woman standing on the back of it.
Ours (MDA)
a man with a woman standing next to fire hydrant wearing a backpack.
14

---

Under review as a conference paper at ICLR 2021
NDA
VDA
MOA
0.2
UN62
0.4
0.5
1.00
303
1560
1ention
lentoo
MDA
0.5
0.4
0.3
500
1300
1500
2000
503
1000
1560 2000
Figure 6: Training curves of BLEU scores MS COCO Image Caption dataset with various Gumbel-Sone Softmmax temperature values.
Table 9: Randomly sampled 10 samples trained on MS COCO dataset, with MDA (top row) and MSA (bottom row). We can observe that the phrases "a man" and "motorcycle" occur too many times the samples generated by MSA-enhanced models. Thus, models with MSA lack the diver- sity of text whereas those with MDA perform well on both the diversity and quality.
sitting bench a person riding a metorcyele on a city street with woman standing on the back it.
bicycle parked sidewalk.
cat sitting motorcycle car seat wanting to out the car window.
a woman is sitting
bench next to a bicycle.
a man is riding a motorcycle a city street with parked vehicles.
group people standing a woman is leaving a bathzoom with a toilet and a sink.
a person riding a motorcycle
intersection
with people standing it.
bench a little girl is sitting on the back of storefrent bathroom with his orange bike next to him. wall.
little gid sitting
bicycle.
person sitting on beach next bicycle parked
a man sitting a motorcycle a husy street, city
man is sitting on a motorcycle on a crowded street, near huilding with a bike nearby. bicycle. :
motorcycle a a photo of small plane sitting on a busy runway.
a man sitting
street.
parking lot. a group of peoplc sitting on motorcycle on side of road.
sitting motorcycle crowded street
sitting motorcycle crowded street, a person sitting on a motorcycle on a sidewalk next to hicycle. sitting moloreycle a man is sitting on a motorzycle on a husy street, in city.
busy sidewalk next bicycle.
15

---

Under review as a conference paper at ICLR 2021
Table 10: Randomly sampled 10 samples trained on EMNLP2011 WMT News dataset, with MDA (top row) and MSA (bottom row).
the british people would have to commit travelling curope and has been priority for the first time in decade. his priority finally hecomes a hope for the police to be informed hy the attack and is not on the scene there is more than a year before the start the day after the european first, but it was not in the last two years.
not something that about be 15. percent the fourth quarter. according to a report from the third the week. part of
" i that" why have to he part of the UK
two years he said statement from the bbe today .
i' not saying that was the first of the kind of people who were in the wrong but it is true that it to be determined.
he will be a key for the first time in a decade. and has helped to stop the spread of the decade over the past year.
so what if that" the reason that they can be within the last two years..
was one the most in the first quarter.
was the first of the nearly two months since the start first the day.
you' re a new and you have he part of the team
news, she has already been major despite conflict world. when you but i ve been instead, there is no evidence to suggest that the united kingdom.
world. when they growing
of the group for christmas.
the department of health and the enforcement and defense agencies last Thursday
you re the only in world, time when
m sure
he will be more than it sometimes it was number of the people who were in the back of the way. it is an easy for trump ; the other at the same
the control of water.
16


Reviews:
[{"review_id": "dF_eK3Sowmp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\n \nThe paper proposes an improvement to sequence generative adversarial networks (GAN) to cope with the common training issues of GANs. For the sake, the paper combines Gumbel-Softmax based GAN, relativistic discrimination  function with  the matching of mean representations of true and generated samples in a latent feature space. This feature statistics alignment allows to leak information from the discriminator to the generator as the used features are extracted from the discriminator network. Experimental evaluations on synthetic and real datasets show the improvement achieved by the proposed method over existing sequence generation networks.\n\n\n\nReasons for score: \n \nThe paper straightforwardly combines existing procedures (relativistic discriminator, Gumbel-Softmax approximation for categorical distribution, features matching) to improved upon vanilla sequence generation networks and somehow lacks novelty. Although the ablation study is interesting and shows the improvement brought by each module, examples of lengthy generated sequences illustrate that the sentences produced by the GAN are not semantically meaningful. \n \n\nPros:\n- Overall, the paper is well written. In particular, the rationale behind the proposed method is justified. Empirical evaluations support these intuitions and show how they contribute to the observed quality of the generated sequences.  \n- The paper aims at addressing a major issue in training GAN for sequence generation: how to strengthen the learning of the generator compared to the discrimination network which is easier to train? The approach promoted in the paper consists to align the mean statistics of true sequences and fake ones. Specifically, the statistics are computed over features extracted from discrimination network. The objective function of the generator is therefore composed of the usual GAN loss term and the distance between those mean representations. As such, this idea of guiding the generation network with information from the discriminator is interesting and plays a key role in the performances improvement. \n- In the same vein, the use of Gumbel-Softmax distribution (instead of the discrete distribution) and of the relativistic discrimination function (instead of the classical classification function) helps to learn a better generation model. However these ideas are not novel and were investigated separately in previous research works. \n- Experimental evaluations, including both qualitative analysis and quantitative results, are provided in the paper and in the supplementary to show the effectiveness of the proposed framework. The newly proposed GAN achieve superior performances. The comprehensive ablation study is interesting and helps to understand how each module (feature alignment, Gumbel-Softmax, batch size) contributes to the enhanced performances.\n \nCons: \n- Although the proposed method, according to the empirical results, show improved performances, it lacks novelty as features matching, relativistic discrimination or Gumbel-Softmax are not new ideas. The main contribution resides in the better quantitative results compared to existing sequence generation networks. However when one examines the generated sentences, it appears that they lack semantic meaningfulness especially for long sentences (see for instance Table 8). This shows that the proposed GAN (as well as the competitors) is not effective yet.  \n- Features distribution alignment is an interesting way to measure how close are the marginal distributions of the real and fake sequences. The paper considers the Mean Distance Alignment (MDA) and the Mean Square alignment (MSA) which are respectively the distance and the squared distance between the mean latent representations of the real and generated sequences. Several comments can be made as hereafter.\n     * MSA and MDA encode the same matching up  to a power 2. It’s unclear why they lead to different empirical results.\n     * Instead of matching only the mean statistics, the overall distributions of the latent representations can be aligned by considering metrics such as MMD or Wasserstein distance. How would the results look like in that setting?\n     * It should be clarified earlier in the paper that the used features are extracted from the discrimination network (as the weights between the discriminator and feature extractor are shared). Also the paper should make explicit from which layer of the discrimination network the features are extracted. \n- The findings of human evaluation (see Table 5) are not unequivocal. MSA and MDA achieve higher scores than the real sentences. The best model, the one with MSA, is not preferred because of a lack of diversity and quality. This raises the question of how reliable is the human evaluation score.\n \nOther comments: \n- Page 3, definition of MSA: in the sentence “mean squared difference between the centroids...”, the term mean is over-used as Eq. (2) or (3) represents only the squared distance between centroids.\n- Page 4: in “The FSA term on the RHS can also be regarded as a dynamic regularizer for the sequence generator” the notion of dynamic regularizer is unclear. In which sense FSA induces a dynamic regularization?\n- In Equation (10) the function “one_hot” should be defined. Also in (10), I think $y_i$ should read the $|V|$-dimensional vector $y$.\n- Equation (11) is to be checked carefully as the parameter $\\tau$ simplifies in numerator and denominator. \n- Algorithm 1: the update of the generator $G_\\theta$ requires a minibatch from real dataset in order to minimize $L_{RG}$ + $L_{FSA}$ as $L_{FSA}$ relies on the mean of real data latent representation.\n\nAfter rebuttal\n-  I read the response of the authors. The spotted typos are fixed in the revision. Some  questions/concerns  have been tentatively. However the novelty in the paper is still not blatant or how the use of distance such as MMD or Wasserstein to match the features is under-explored. Hence I intend to keep my rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "VHkLDQqF-h2", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\nThis paper proposes a new GAN-based text generation method that incorporates feature statistics alignment and gumbel-softmax for reparameterization to deal with mode collapse and unstable training. For feature statistics alignment, the authors design two methods such as mean square and mean distance alignments. They evaluate the proposed method on a synthetic dataset, MS COCO caption, and EMNLP2017 WMT news dataset, comparing them with RL-based and non RL-based models. With extensive experiments including ablation studies, the proposed method show promising results.\n\n[Recommendation]\nOverall, this paper is clear and well-written. So I lean to acceptance. But I have some concerns as well.\n\n[Strength]\n- Mode collapse is challenging issue in GAN training.\n- Text generation is important problem.\n\n[Weakness]\n- The authors insist the use of Gumbel-softmax in GAN tranining is under-explored. But It is not clear. There are more method using Gumbel-softmax [Gu et al. 2019] and a similar softmax with temperature annealing. It is not clear for the authors to explicitly discriminate using Gumbel-softmax and other smoothed softmax methods. \n- Some related work  were missed such as DialogWAE [Gu et al. 2019] and ARAML [Ke et al. 2019]. In particular, DialogWAE uses GAN and Gumbel-softmax for text generation even if it focuses on dialog generation.\n- For verifying mode collapse issues, how about using Self-BLUE in addition to BLUE scores as a metric to evaluate the diversity? \n- Novelty might be incremental. It seems that the novelty is from using feature statistics alignment. To emphasize the contribution of feature statistics, comparing between the latent feature visualization with and without FSA might be helpful in addition to ablation study. \n\n[Minor] \nIn p3, the given real data is --> are\n\n\n[Gu et al. 2019] DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder. ICLR 2019.\n[Ke et al. 2019] ARAML: A Stable Adversarial Training Framework for Text Generation. EMNLP 2019. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "CnsHXulT1e", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Main Claim:**\n\nIn this work, the authors propose to use the Feature Statistics Alignment paradigm to enrich the learning signal from the discriminator in a sentence generation GAN. The proposed model can generate sentences with better likelihood and BLEU on one synthetic and two real datasets.\n\n**Contributions:**\n\nThis work introduces an novel and interesting idea of Feature Statistics Alignment in training GANs. \n\nThe authors follow the convention in this domain, and evaluate the model on three datasets.\n\nThe experiment results show that the proposed model outperforms existing models. However, the authors need to clarify some details to make the results trustworthy (see weakness). \n\n\n**Strong points:**\n\nThe idea is novel and interesting. \n\nThe model and training procedure is clearly explained. Related works are cited well. \n\n\n**Weak points:**\n\nIn Table 2: \n\n- The LSTM model gets NLL lower than the real data. This is a clear evidence of overfitting. \n- In SAL (Zhou et. al, 2020), NLL_{gen} is used to evaluate the diversity of the generator. But this metric is missing here without explanation. \n\nIn Table 3:\n\n- The BLEU metric in this paper is the BLEU(F) metric in SAL (Zhou et. al, 2020). This metric evaluates the generated sentences using the test set as a reference. Thus the BLEU(F) metric cannot show the diversity of examples. \n- The BLEU(B) (Zhou et. al, 2020) metric is missing. BLEU (B) metric evaluates the test set using the generated sentences as a reference, so it can detect mode collapse of a generative model. \n\nIn section 4, although authors clearly cite previous works for experiment settings, I think it’s worthwhile to repeat the definition of each metric, and some other key points in the paper, so that readers can easily understand the notations and jargons in this section. \n\n**Recommendation:**\nReject. \n\nThere’s a major flaw in the evaluation metrics. On both synthetic and real datasets, the evaluation metrics prefers overfitted models, i.e. if the model can remember one example from the training set, and repeat that sentence, it can get a very high score. \n\nI will reconsider my recommendation if (1) I miss interpret the metrics or (2) the authors provide more evidence on the diversity of the generated sentences, for example showing the NLL_{gen} metric on the synthetic dataset, and BLEU(B) metric on real datasets. \n\n**Questions:**\n\nHow is NLL_gen computed?\n\n**After Rebuttal**\nThe author's reply partially resolved my concerns, although the diversity of models has not improved, nor has it significantly decreased. Thus I have increased my score from 3 to 4. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "2E-Y6UFOVI", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper addresses the task of improving GANs for sequence generation and proposed a method based on the relativistic discriminator. The proposed method employs a Feature Statistics Alignment (FSA) paradigm to reduce the gap between real and generated data distributions. It relies on the relativistic discriminator for \"coarse\" differences and FSA for \"fine-grained\" differences between real and generated data distributions. It is evaluated on synthetic and real datasets, and it significantly outperforms the baselines. It also outperforms baselines on human evaluation based on the acceptance, grammaticality, and meaningfulness of the generated sentences. \n\nStrengths:\n\nThe proposed approach is very effective, as demonstrated by significant performance improvements in the experiments across synthetic and real datasets. Also, it can generate better sentences compared to the baselines, as shown by human evaluation.\n\nWeakness:\n\nAlthough the proposed model is thoroughly evaluated and empirically effective, it is not very different from existing methods, except for FSA. The application of FSA in this context might be novel; however, the proposed approach seems to be a simple combination of two existing approaches. Therefore, the novelty of the model is limited.\n\nMinor comments:\n1. Correction:  Did you mean\n\nIn contrast, the lower τ could discourage exploration and tend to explore during training. -> In contrast, the lower τ could discourage exploration and tend to exploit during training. ?\n\n2. It would have been good to see a head-on comparison of the generated samples (baseline vs. proposed approach) in the paper's main text.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work introduces a method that uses the Feature Statistics Alignment paradigm to improve sequence generation with GANs. The contribution is interesting and novel (although marginally), clarity is also good.
However the reviewers raised several concerns calling for more comprehensive and thorough evaluation. Experiments show an improvement comparing to selected baselines and the revised paper addressed, at least partially, a serious evaluation concern of one reviewer.
Although the excellent revision work some important open questions still seem to remain, in particular the choose of alignment metrics and a thorough evaluation.

Decision: Reject