CoT: Cooperative Training for Generative Modeling of Discrete Data

1 INTRODUCTION . Generative modeling is essential in many scenarios , including continuous data modeling ( e.g . image generation ( Goodfellow et al. , 2014 ; Arjovsky et al. , 2017 ) , stylization ( Ulyanov et al. , 2016 ) , semisupervised classification ( Radford et al. , 2015 ) ) and sequential discrete data modeling ( e.g . neural text generation ( Bahdanau et al. , 2014 ; Yu et al. , 2017 ; Lu et al. , 2018 ) ) . For discrete data with tractable density like natural language , generative models are predominantly optimized through Maximum Likelihood Estimation ( MLE ) , inevitably introducing exposure bias ( Ranzato et al. , 2015 ) , which results in that given a finite set of observations , the optimal parameters of the model trained via MLE do not correspond to the ones maximizing the generative quality . Specifically , the model is trained on the data distribution of inputs and tested on a different distribution of inputs , namely , the learned distribution . This discrepancy implies that in the training stage , the model is never exposed to its own errors and thus in the test stage , the errors made along the way will quickly accumulate . On the other hand , for general generative modeling tasks , an effective framework , named Generative Adversarial Network ( GAN ) ( Goodfellow et al. , 2014 ) , was proposed to train an implicit density model for continuous data . GAN introduces a discriminator Dφ parametrized by φ to distinguish the generated samples from the real ones . As is proved in ( Goodfellow et al. , 2014 ) , GAN essentially optimizes an approximately estimated Jensen-Shannon divergence ( JSD ) between the currently learned distribution and the target distribution . GAN shows promising results in many unsupervised and semi-supervised learning tasks . The success of GAN results in the naissance of a new paradigm of deep generative models , i.e . adversarial networks . However , since the gradient computation requires backpropagation through the generator ’ s output , GAN can only model the distribution of continuous variables , making it non-applicable for generating discrete sequences like natural language . Researchers then proposed Sequence Generative Adversarial Network ( SeqGAN ) ( Yu et al. , 2017 ) , which uses model-free policy gradient algorithm to optimize the original GAN objective . With SeqGAN , the expected JSD between current and target discrete data distribution is minimized if the training is perfect . SeqGAN shows observable improvements in many tasks . Since then , many variants of SeqGAN have been proposed to improve its performance . Nonetheless , SeqGAN is not an ideal algorithm for this problem , and current algorithms based on it can not show stable , reliable and observable improvements that covers all scenarios , according to a previous survey ( Lu et al. , 2018 ) . The detailed reason will be discussed in detail in Section 2 . In this paper , we propose Cooperative Training ( CoT ) , a novel , low-variance , bias-free algorithm for training likelihood-based generative models on discrete data by directly optimizing a wellestimated Jensen-Shannon divergence . CoT coordinately trains a generative module G , and an auxiliary predictive module M , called mediator , for guiding G in a cooperative fashion . For theoretical soundness , we derive the proposed algorithm directly from the definition of JSD . We further empirically and theoretically demonstrate the superiority of our algorithm over many strong baselines in terms of generative performance , generalization ability and computational performance in both synthetic and real-world scenarios . 2 BACKGROUND . Notations . P denotes the target data distribution . θ denotes the parameters of the generative module G. φ denotes the parameters of the auxiliary predictive mediator module M . Any symbol with subscript g and m stands for that of the generator and mediator , respectively . s stands for a complete sample from the training dataset or a generated complete sequence , depending on the specific context . st means the t-length prefix of the original sequence , i.e . an incomplete sequence of length t. x denotes a token , and xt stands for a token that appears in the t-th place of a sequence . Thus st = [ x0 , x1 , x2 , . . . , xt−1 ] while the initial case s0 is ∅ . 2.1 MAXIMUM LIKELIHOOD ESTIMATION . Maximum likelihood estimation is equivalent to minimizing the KL divergence using the samples from the real distribution : min θ Es∼pdata [ − logGθ ( s ) ] , ( 1 ) where Gθ ( s ) is the estimated probability of s by Gθ and pdata is the underlying real distribution . Limitations of MLE . MLE is essentially equivalent to optimizing a directed Kullback–Leibler ( KL ) divergence between the target distribution P and the currently learned distribution G , denoted as KL ( P‖G ) . However , since KL divergence is asymmetric , given finite observations this target is actually not ideal . As stated in ( Arjovsky & Bottou , 2017 ) , MLE tries to minimize KL ( P‖G ) = ∑ s P ( s ) log P ( s ) G ( s ) . ( 2 ) • When P ( s ) > 0 and G ( s ) → 0 , the KL divergence grows to infinity , which means MLE assigns an extremely high cost to the “ mode dropping ” scenarios , where the generator fails to cover some parts of the data . • When G ( s ) > 0 and P ( s ) → 0 , the KL divergence shrinks to 0 , which means MLE assigns an extremely low cost to the scenarios , where the model generates some samples that do not locate on the data distribution . Likewise , optimizing KL ( G‖P ) will lead to exactly the reversed problems of the two situations . An ideal solution is to optimize a symmetrized and smoothed version of KL divergence , i.e . the Jensen-Shannon divergence ( JSD ) , which is defined as JSD ( P‖G ) = 1 2 ( KL ( P‖M ) +KL ( G‖M ) ) , ( 3 ) where M = 12 ( P + G ) . However , directly optimizing JSD is conventionally considered as an intractable problem . JSD can not be directly evaluated and optimized since the equally interpolated distribution M is usually considered to be unconstructable , as we only have access to the learned model G instead of P . 2.2 SEQUENCE GENERATIVE ADVERSARIAL NETWORK . SeqGAN incorporates two modules , i.e . the generator and discriminator , parametrized by θ and φ respectively , as in the settings of GAN . By alternatively training these two modules , SeqGAN optimizes such an adversarial target : min θ max φ Es∼pdata [ log ( Dφ ( s ) ) ] + Es∼Gθ [ log ( 1−Dφ ( s ) ) ] . ( 4 ) Algorithm 1 Cooperative Training Require : Generator Gθ ; mediator Mφ ; samples from real data distribution P ; hyper-parameter Nm . 1 : Initialize Gθ , Mφ with random weights θ , φ . 2 : repeat 3 : for Nm steps do 4 : Collect two equal-sized mini-batch of samples { sg } and { sp } from Gθ and P , respectively 5 : Mix { sg } and { sp } as { s } 6 : Update mediator Mφ with { s } via Eq . ( 9 ) 7 : end for 8 : Generate a mini-batch of sequences { s } ∼ Gθ 9 : Update generator Gθ with { s } via Eq . ( 13 ) 10 : until CoT converges The objectives of generator Gθ and discriminator Dφ in SeqGAN can be formulated as Generator : min θ −Es∼Gθ [ n∑ t=1 Qt ( st , xt ) · logGθ ( xt|st ) ] ( 5 ) Discriminator : max φ Es∼pdata [ log ( Dφ ( s ) ) ] + Es∼Gθ [ log ( 1−Dφ ( s ) ) ] , ( 6 ) where s ∼ Gθ = [ x1 , ... , xn ] denotes a complete sequence sampled from the generator and the action value Qt ( st , xt ) = Es∼Gθ ( ·|st+1 ) [ Dφ ( s ) ] is the expectation of the discriminator ’ s evaluation on the completed sequences sampled from the prefix st+1 = [ st , xt ] , which can be approximated via Monte Carlo search . Limitations of SeqGAN & its Variants . First , SeqGAN is an algorithm of high variance , which relies on pre-training via Maximum Likelihood Estimation as a variance reduction procedure . Besides , during the adversarial epochs , even if with variance reduction techniques such as Actor-Critic methods ( Sutton , 1984 ) , the fact that SeqGAN is essentially based on model-free reinforcement learning makes it a non-trivial problem for SeqGAN to converge well . As a result , SeqGAN usually gets stuck in some fake local optimals . Specifically , although the discriminator can distinguish the samples from the generator easily , it is not able to effectively guide the generator because of the vanishing gradient , as is discussed in a recent survey ( Lu et al. , 2018 ) . Although this problem can be alleviated by reshaping the reward signals based on the relative rankings of the outputs in a mini-batch ( Lin et al. , 2017 ; Guo et al. , 2017 ) , they are more technical workarounds than essential solutions . Second , SeqGAN trained via REINFORCE ( Williams , 1992 ) suffers from the “ mode collapse ” problem , which is similar to the original GAN . That is to say , the learned distribution “ collapses ” to the other side of KL divergence , i.e . KL ( G‖P ) , which leads to the loss of diversity of generated samples . In other words , SeqGAN trains the model for better generative quality at the cost of diversity . 3 COOPERATIVE TRAINING . 3.1 MOTIVATION . To be consistent with the goal that the target distribution should be well-estimated in both quality and diversity senses , an ideal algorithm for such models should be able to optimize a symmetric divergence or distance . For sequential discrete data modeling , since the data distribution is decomposed into a sequential product of finite-dimension multinomial distributions ( always based on the softmax form ) , the failures of effectively optimizing JSD when the generated and real data distributions are distant , as discussed in ( Arjovsky et al. , 2017 ) , will not appear . As such , to optimize JSD is feasible . However , to our knowledge , no previous algorithms provide a direct , low-variance optimization of JSD . In this paper , we propose Cooperative Training ( CoT ) , as shown in Algorithm 1 , to directly optimize a well-estimated unbiased JSD for training such models . 3.2 ALGORITHM DERIVATION . Each iteration of Cooperative Training mainly consists of two parts . The first part is to train a mediator Mφ , which is a density function that estimates a mixture distribution of the learned generative distribution Gθ and target latent distribution P = pdata as Mφ ' 1 2 ( P +Gθ ) . ( 7 ) Since the mediator is only used as a density prediction module during training , the directed KL divergence is now free from so-called exposure bias for optimization of Mφ . Denote 12 ( P +Gθ ) as M∗ , we have : Lemma 1 ( Mixture Density Decomposition ) ∇φJm ( φ ) =∇φKL ( M∗‖Mφ ) =∇φ E s∼M∗ [ log M∗ ( s ) Mφ ( s ) ] =∇φ ( − E s∼M∗ [ logMφ ( s ) ] ) =∇φ 1 2 ( E s∼Gθ [ − log ( Mφ ( s ) ) ] + E s∼P [ − log ( Mφ ( s ) ) ] ) ( 8 ) By Lemma 1 , for each step , we can simply mix balanced samples from training data and the generator , then train the mediator via Maximum Likelihood Estimation with the mixed samples . The objective Jm ( φ ) for the mediator M parametrized by φ therefore becomes Jm ( φ ) = 1 2 ( E s∼Gθ [ − log ( Mφ ( s ) ) ] + E s∼P [ − log ( Mφ ( s ) ) ] ) . ( 9 ) Since the objective of MLE is bias-free for predictive purposes , the estimated Mφ is also bias-free when adopted for estimating JSD . The training techniques and details will be discussed in Section 4 . After each iteration , the mediator is exploited to optimize an estimated Jensen-Shannon divergence for Gθ : ∇θJg ( θ ) =∇θ ( − ˆJSD ( Gθ‖P ) ) = ∇θ ( − 1 2 [ KL ( Gθ‖Mφ ) +KL ( P‖Mφ ) ] ) =∇θ ( −1 2 E s∼Gθ [ log Gθ ( s ) Mφ ( s ) ] − 1 2 E s∼P [ log P ( s ) Mφ ( s ) ] ) = ∇θ ( −1 2 E s∼Gθ [ log Gθ ( s ) Mφ ( s ) ] ) . ( 10 ) Note that the gradient Eq . ( 10 ) should be performed for only one step because once Gθ is updated the current mediator ’ s estimation Mφ becomes inaccurate . For any sequence or prefix of length t , we have : Lemma 2 ( Markov Backward Reduction ) ∇θ ( − 1 2 E st∼Gθ [ log Gθ ( st ) Mφ ( st ) ] ) ( 11 ) =∇θ ( −1 2 E st−1∼Gθ [ ∑ st Gθ ( st|st−1 ) log Gθ ( st|st−1 ) Mφ ( st|st−1 ) ] − 1 2 E st−1∼Gθ [ log Gθ ( st−1 ) Mφ ( st−1 ) ] ) . ( 12 ) The detailed derivations can be found in the supplementary material . Note that Lemma 2 can be applied recursively . That is to say , given any sequence st of arbitrary length t , optimizing st ’ s contribution to the expected JSD can be decomposed into optimizing the first term of Eq . ( 12 ) and solving an isomorphic problem for st−1 , which is the longest proper prefix of st . When t = 1 , since in Markov decision process the probability for initial state s0 is always 1.0 , it is trivial to prove that the final second term becomes 0 . Therefore , Eq . ( 10 ) can be reduced through recursively applying Lemma 2 . After removing the constant multipliers and denoting the predicted probability distribution over the action space , i.e . Gθ ( ·|st ) andMφ ( ·|st ) , as πg ( st ) and πm ( st ) respectively , the gradient∇θJg ( θ ) for training generator via Cooperative Training can be formulated as ∇θJg ( θ ) = ∇θ E s∼Gθ [ n−1∑ t=0 πg ( st ) > ( log πm ( st ) − log πg ( st ) ) ] . ( 13 ) For tractable density models with finite discrete action space in each step , the practical effectiveness of this gradient is well guaranteed for the following reasons . First , with a random initialization of the model , the supports of distributions Gθ and P are hardly disjoint . Second , the first term of Eq . ( 13 ) is to minimize the cross entropy between G and M∗ , which tries to enlarge the overlap of two distributions . Third , since the second term of Eq . ( 13 ) is equivalent to maximizing the entropy of G , it encourages the support of G to cover the whole action space , which avoids the case of disjoint supports between G and P . The overall objective of CoT can be formulated as finding the maximal entropy solution of max θ max φ E s∼pdata [ log ( Mφ ( s ) ) ] + E s∼Gθ [ log ( Mφ ( s ) ) ] . ( 14 ) Note the strong connections and differences between the optimization objective of CoT ( 14 ) and that of GAN ( 4 ) . Figure 1 illustrates the whole Cooperative Training process .

Summary: The paper proposes an interesting method, where the discriminator is replaced by a component that estimates the density that is the mixture of the data and the generator's distributions. In a sense, that component is only a device that allows estimating a Jensen-Shannon divergence for the generator to then be optimized against. Other GAN papers have replaced their discriminator by a similar device (e.g., WGANs, ..), but the present formulation seems novel. The numerical experiments presented on a synthetic Turing test and text generation from EMNLP's 2017 news dataset appear promising.