Improving Sequence Generative Adversarial Networks with Feature Statistics Alignment

1 INTRODUCTION . Unsupervised sequence generation is the cornerstone for a plethora of applications , such as machine translation ( Wu et al. , 2016 ) , image captioning ( Anderson et al. , 2018 ) , and dialogue generation ( Li et al. , 2017 ) . The most common approach to autoregressive sequence modeling is maximizing the likelihood of each token in the sequence given the previous partial observation . However , using maximum likelihood estimation ( MLE ) for sequence modeling is inherently prone to the exposure bias problem ( Bengio et al. , 2015 ) , which results from the discrepancy between the training and inference stage : the generator predicts the next token conditioned on its previously generated ones during inference but conditioned on its prefix ground-truth tokens during training , yielding accumulative mismatch along with the increment of generated sequence length . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) can serve as an alternative to models trained by MLE , which have achieved promising results in generating sequences of discrete elements , in particular , language sequences ( Kusner & Hernández-Lobato , 2016 ; Yu et al. , 2017 ; Lin et al. , 2017 ; Guo et al. , 2018 ; Fedus et al. , 2018 ; Nie et al. , 2019 ; de Masson d ’ Autume et al. , 2019 ; Zhou et al. , 2020 ; Scialom et al. , 2020 ) . GANs consist of two competing networks : a discriminator that is trained to distinguish the generated samples from real data , and a generator that aims to generate high-quality samples to fool the discriminator . Although having succeeded in avoiding exposure bias issues , GANs still suffer from some intrinsic problems , such as mode dropping , reward sparsity , and training instability . To enrich the informativeness of the discriminator ’ s training signal , several approaches have been proposed by measuring the latent features , such as feature distribution matching ( Zhang et al. , 2017 ; Chen et al. , 2018 ) and comparative discriminators ( Lin et al. , 2017 ; Zhou et al. , 2020 ) . Zhang et al . ( 2017 ) and Chen et al . ( 2018 ) leveraged feature matching mechanism by minimizing the kernel-based moment-matching metric , such as Maximum Mean Discrepancy and Earth-Mover ’ s Distance , between encoded features . However , merely adopting feature matching in lieu of the original learning signal may lack some guiding feedback at the initial stage of training . Another approach is to compare the finite latent features with comparative discriminators like ranker and relativistic discriminator . Lin et al . ( 2017 ) maintained that the binary classification in the discriminator network limits the learning capacity of tasks because the diversity and richness are circumscribed by the degenerated distribution . RankGAN ( Lin et al. , 2017 ) replaced the binary clas- sifier with a pairwise feature ranker by comparing the similarities between sample features in the latent space . SAL ( Zhou et al. , 2020 ) classified the encoded features of constructed pairwise training examples into three categories , i.e. , better / worse / indistinguishable . Nevertheless , adopting a comparative discriminator could provide the fine-grained signals for updating the generator network and may require some coarse credits for further improvements . In this work , we propose to improve the GANs for sequence generation by jointly considering both the feature statistics matching and relativistic discriminator to serve as fine-grained and coarse learning signals respectively . We leverage the Feature Statistics Alignment ( FSA ) paradigm to embed the latent feature representations in a finite feature space and force the distribution of generated samples to approach the real data distribution by minimizing the distance between their respective feature representation centroids . Intuitively , matching the mean feature representations of fake and real samples could make the two data distributions closer . Besides , the relativistic discriminator ( Jolicoeur-Martineau , 2019 ) is employed to measure the comparative information between generated and real sequences and empirically to show the effectiveness during the model training . Our experimental results illustrate the effectiveness of FSA techniques and large batch size to alleviate the gradient vanishing problem and stabilize the training process in comparison with the vanilla Gumbel-Softmax GANs . Besides , our models could generate discrete text sequences with high quality in terms of the semantic coherence and grammatical correctness of language , as evaluated with crowdsourcing . Furthermore , we empirically demonstrate that the proposed architecture overshadows most existing models in terms of quantitative and qualitative evaluation . To the best of our knowledge , the proposed framework is the first to adopt the statistics feature alignment paradigm in the Gumbel-Softmax based GAN framework for discrete sequence generation . 2 ADVERSARIAL SEQUENCE GENERATION . Adversarial sequence generation has attracted broad attention for its properties to solve the exposure bias issue suffered with maximum likelihood estimation ( MLE ) for generating language sequences . Based on the game theory , its goal is to train a generator network G ( z ; θ ( G ) ) that produces samples from the data distribution pdata ( x ) by decoding the randomly initialized noise z ( i.e. , standard normal distribution ) into the sequence x = G ( z ; θ ( G ) ) , where the training signal is provided by the discriminator networkD ( x ; φ ( D ) ) that is trained to distinguish between the samples drawn from the real data distribution pdata and those produced by the generator . The minimax objective of adversarial training is formulated as : min θ ( G ) max φ ( D ) Ex∼pdata [ logD ( x ; φ ( D ) ) ] + Ez∼pz [ log ( 1−Dφ ( D ) ( G ( z ; θ ( G ) ) ) ) ] . ( 1 ) Despite the impressive results of GANs in the sequence generation ( Yu et al. , 2017 ; Gulrajani et al. , 2017 ; Scialom et al. , 2020 ) , there are still several fundamental issues in the GAN training : ( a ) Training instability , which arises from the intrinsic nature of minimax games in GANs ; ( b ) Mode dropping , which is the fact that GANs only generate samples with limited patterns in the real data distribution instead of attending to diverse patterns ( Chen et al. , 2018 ) ; ( c ) Reward sparsity , which is because that it is easier to train the discriminator than the generator , making it difficult to acquire the instructive feedback ( Zhou et al. , 2020 ) . Due to the non-differentiability of gradients caused by sampling operations between the generator and discriminator for sequence generation , the majority of previous works have resorted to reinforcement learning ( RL ) heuristics with Monte Carlo search to collect the credits from the discriminator . The usage of RL may further deteriorate the instability of model training and exacerbate the reward sparsity problem . Gumbel-Softmax relaxation has proven to be an alternative to RL techniques ( Kusner & Hernández-Lobato , 2016 ; Nie et al. , 2019 ) . How to efficiently train GANs with the Gumbel-Softmax trick still remains under-explored . Therefore , we utilize the Gumbel-Softmax reparameterization instead of conventional policy gradients in our framework . 3 METHODOLOGY . As illustrated in Fig . 1 , standard GANs employ the real/fake binary classifier as the discriminator , which is prone to be overtrained in comparison with the generator ( Salimans et al. , 2016 ) . To prevent the discriminator from overfitting and further stabilize the training process , we propose to leverage the FSA techniques and relativistic discriminator by comparing the fake and real distributions from two different aspects . Compared with conventional GANs , the proposed framework enjoys the following advantages : ( a ) In the earlier training stage , the relativistic discriminator could estimate the probability that how much better the real data is in comparison with the generated samples . This could provide the relatively “ coarse ” credits to the generator . ( b ) When it comes to the later training stage and the quality of generated samples becomes high , FSA measures the “ fine-grained ” difference between latent features of fake and real data batches , precluding the discriminator from being overly confident . In other words , the FSA can be regarded as a kind of regularization for adversarial training . 3.1 FEATURE STATISTICS ALIGNMENT . Intuitively , aligning the statistics of embedded feature representations increases the opportunities to capture the various modes of the data distribution . For brevity , we utilize the first-order mean statistics in our framework and leave the higher-order statistics for future work . Denoting the feature extractor as Fω parameterized by ω , a minibatch of real data samples as x with the batch size of N , we propose two variants of FSA formulations , which calculate the mean squared difference and Euclidean distance between the minibatch centroids of real and fake feature representations . To reduce the parameter amount , we share the weights between the discriminator and feature extractor of FSA . Mean Squared Alignment ( MSA ) We take the mean squared difference between the centroids of fake and generated distributions as the feature alignment metric : LMSA = ∥∥Ex∼pdata [ Fω ( x ) ] − Ez∼pz [ Fω ( G ( z ; θ ( G ) ) ) ] ∥∥22 ( 2 ) = ∥∥ 1 N N∑ i=1 Fω ( xi ) − 1 N N∑ i=1 Fω ( G ( zi ; θ ( G ) ) ) ∥∥2 2 . ( 3 ) Mean Distance Alignment ( MDA ) Another intuitive approach is to calculate the distance between two sample centroids , which is equivalent to the square root of MSA mathematically : LMDA = ∥∥Ex∼pdata [ Fω ( x ) ] − Ez∼pz [ Fω ( G ( z ; θ ( G ) ) ) ] ∥∥2 ( 4 ) = √ LMSA . ( 5 ) By forcing the mean statistics of fake data to be close to the real samples , the generator could receive more informative signals in the training process . It is worth noting that the large batch size is helpful to reduce the variance of small mini-batches . 3.2 RELATIVISTIC DISCRIMINATOR . We consider the Relativistic Discriminator ( Jolicoeur-Martineau , 2019 ) to take into account the relative confidence that the given real data is more realistic than the randomly sampled fake data . In the standard GAN , defining the discriminator as D ( x ) = sigmoid ( H ( x ) ) , where H ( · ) represents the non-transformed layer before the final non-linearity . The objectives for the discriminator and generator in terms of the Relativistic Discriminator are defined as : LRD = −Ex∼pdata , z∼pz [ log a ( H ( x ) −H ( G ( z ; θ ( G ) ) ) ) ] , ( 6 ) LRG = −Ex∼pdata , z∼pz [ log a ( H ( G ( z ; θ ( G ) ) ) −H ( x ) ) ] , ( 7 ) where a represents the activation function to be relativistic ( we use sigmoid function in our experiments ) , RD and RG denote the loss terms for the discriminator and generator respectively . 3.3 OVERALL TRAINING OBJECTIVES . Previous works using moment matching techniques to support the training on both the discriminator and generator ( Zhang et al. , 2017 ; Chen et al. , 2018 ) . However , the generator is always more difficult to train than the discriminator , resulting in the training instability and reward sparsity . To relieve these issues , we only adopt the FSA techniques to enhance the generator but keep the objective of the discriminator unchanged . This could pass more informative signals only to the generator , and also prevent the discriminator to be overtrained . Therefore , the overall training objectives for the proposed framework are defined as : LD = LRD , ( 8 ) LG = LRG + LFSA , ( 9 ) where LFSA takes the form of LMSA and LMDA as Eq . 2 and Eq . 4 . The goal of the discriminator is to maximize the gap between the generated and real data , whereas the generator jointly considers two different aspects simultaneously : it not only competes with the discriminator by maximizing the gap in terms of the relativistic signals but takes into account the additional leaked feature information from the discriminator . The idea of leaked features from the discriminator is similar to LeakGAN ( Guo et al. , 2018 ) . The FSA term on the RHS can also be regarded as a dynamic regularizer for the sequence generator .

Summary: The paper addresses the task of improving GANs for sequence generation and proposed a method based on the relativistic discriminator. The proposed method employs a Feature Statistics Alignment (FSA) paradigm to reduce the gap between real and generated data distributions. It relies on the relativistic discriminator for "coarse" differences and FSA for "fine-grained" differences between real and generated data distributions. It is evaluated on synthetic and real datasets, and it significantly outperforms the baselines. It also outperforms baselines on human evaluation based on the acceptance, grammaticality, and meaningfulness of the generated sentences.