Learning Implicit Generative Models by Teaching Explicit Ones

1 INTRODUCTION . Deep generative models ( Kingma & Welling , 2013 ; Goodfellow et al. , 2014 ; Oord et al. , 2016 ) have the ability to capture the distributions over complicated manifolds , e.g. , natural images . Most recent state-of-the-art deep generative models ( Radford et al. , 2015 ; Arjovsky et al. , 2017 ; Karras et al. , 2017 ) are usually implicit statistical models ( Mohamed & Lakshminarayanan , 2016 ) , also called implicit probability distributions . Implicit distributions are flexible as they are specified by a sampling procedure rather than a tractable density . However , this implicit nature makes them difficult to train since maximum likelihood estimation ( MLE ) is not directly applicable . Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) address this difficulty by adopting a minimax game , where a discriminator D is introduced to distinguish whether a sample is real ( from the data distribution ) or fake ( from a generator G ) , while G tries to fool D via generating realistic samples . In practice , G and D are optimized alternatively and GANs suffer from the mode collapse problem . This is because G is optimized to generate samples which are considered as real ones with high confidence by the current D , and won ’ t be penalized for missing modes in data distribution . Although various methods ( Metz et al. , 2016 ; Zhao et al. , 2016 ; Arjovsky et al. , 2017 ) try to modify the vanilla GANs to alleviate the problem , they are still formulated in the minimax framework and do not address the intrinsic weakness of GANs . Therefore , the mode collapse problem of training implicit models is still largely open . This problem mainly arises from the objective function in GAN , i.e. , JS-divergence over data distribution p and the generator distribution pG , which is more tolerant to mode collapse compared to the KL-divergence . As illustrated in the example in Fig . 1 , local optima with mode collapse can be found by optimizing JS-divergence whereas KL ( p||pG ) achieves its optima iff p = pG . To address this issue , we propose a novel teacher-student framework to optimize the KL-divergence where we learn an implicit generator G ( also referred as a teacher ) by teaching a likelihood estimator E ( also referred as a student ) to match the data distribution . In particular , the training scheme is as follows : ( a ) The student E is trained to maximize the log-likelihood of samples generated by the teacher G. ( b ) The student E is evaluated on the real data samples in terms of log-likelihood as well , and the teacher G is trained to improve such log-likelihood based on the signal from E. According to the scheme , we refer to our framework as learning by teaching ( LBT ) . Intuitively , in LBT , E tells G what must be generated whereas in GAN , D tells G what can not be generated . Specifically , in LBT , E learns pG and G aims to adjust its distribution to maximize the log likelihood of real samples evaluated by the learned E. If pG misses some modes , then pE on these modes is low , and G will be penalized heavily . In other words , in LBT , the estimator directs the generated samples to overspread the support of data distribution . In GAN , the goal of D is to distinguish whether a sample is real or fake and the goal of G is to fool D by generating realistic samples . G will be penalized much more heavily when generating fake patterns than when missing modes . Therefore , we can say that LBT and GAN are complementary to each other , where E helps G to overspread the data distribution and D helps G to generate realistic samples . Thus , we propose the combined LBT-GAN to improve the performance . Formally , LBT is formulated as a bilevel optimization ( Colson et al. , 2007 ) problem , where an upper level optimization problem ( step ( b ) ) is dependent on the optimal solution of a lower level problem ( step ( a ) ) . The gradients of the upper problem w.r.t . the parameters of G are unknown since the optimal solution of E can not be analytically expressed by G ’ s parameters . Though the influence function ( Koh & Liang , 2017 ) provides a principle way to differentiate through the bilevel optimization problem , it is computationally expensive . Instead , we propose to use the unrolling technique ( Metz et al. , 2016 ) to efficiently obtain approximate gradients , which are closely connected to the exact ones given by the influence function . Theoretically , under non-parametric conditions , LBT ensures that both of the teacher G and the student E converge to the data distribution ( See proof in Sec . 4 ) . Besides , we provide further empirical analysis of the case where a E with limited capbility can still help G to resist to mode collapse in Appendix B . To summarize , our contributions are threefold : 1 . We propose a novel framework LBT to train an implicit generator by teaching a density estimator . LBT intrinsically avoids the mode collapse problem and is fully compatible with GANs . 2 . Theoretically , we prove that the implicit model will converge to the data distribution in both LBT and LBT-GAN . 3 . Empirically , we show the effectiveness of LBT and LBT-GAN on both synthetic and real datasets . 2 RELATED WORK . Implicit statistical models ( Mohamed & Lakshminarayanan , 2016 ) are of great interests with the emergence of GANs ( Goodfellow et al. , 2014 ) which introduces a minimax framework to train implicit generative models . Nowozin et al . ( 2016 ) generalize the original GANs via introducing a broad class of f -divergence for optimization . In comparison , LBT provides a different way to optimize the KL-divergence and achieves good results on avoiding the mode collapse problem and generating samples when combined with GAN . Arjovsky et al . ( 2017 ) propose to minimize the earth mover ’ s distance to avoid the problem of gradient vanishing in vanilla GANs . Besides , Li et al . ( 2015 ) train implicit models by matching momentum between generated samples and real samples . Mode collapse is a well-known problem in practical training of GANs . Much work has been done to analyze and alleviate the problem ( Arjovsky & Bottou , 2017 ; Arjovsky et al. , 2017 ; Mao et al. , 2017 ; Metz et al. , 2016 ; Srivastava et al. , 2017 ) . Unrolled GAN ( Metz et al. , 2016 ) propose to unroll the update of the discriminator in GANs . The unrolling helps capturing how the discriminator would react to a change in the generator . Therefore it reduces the tendency of the generator to collapse all samples into a single mode . Srivastava et al . ( 2017 ) propose VEEGAN that introduces an additional reconstructor net to map data back to the noise space . Then a discriminator on the joint space is introduced to learn the joint distribution , similar as in ALI ( Dumoulin et al. , 2016 ) . Lin et al . ( 2017 ) propose to modify the discriminator to distinguish whether multiple samples are real or generated . Though such methods can resist mode collapsing to some extent , they are still restricted to the minimax formulation , which makes the training extremely unstable . We directly compare LBT with existing methods ( Metz et al. , 2016 ; Srivastava et al. , 2017 ; Lin et al. , 2017 ) in our experiments . In LBT , we need to evaluate the influence of the training data of the estimator , i.e. , generated samples , on the likelihood of test data , i.e. , real samples , which is closely related to the influence function methods . Koh & Liang ( 2017 ) propose to use influence function to model the affect of training data on the loss of test data , which is equivalent to a quadratic approximation at the optimal point . Zhang et al . ( 2018 ) apply this method to debug the training data using a set of trusted items , where the authors presume that there is bias in the training data . Our method can be treated as another instance of the influence function , where we try to learn an implicit generative model using the influence of the generated samples on the log-likelihood of the real data evaluated by the estimator . 3 METHOD . Consider an implicit model ( or a generator ) G ( · ; θ ) parameterized by θ that maps a simple random variable z ∈ RH to a sample x in the data space RD , i.e. , x = G ( z ; θ ) . Here , z is typically drawn from a standard Gaussian distribution pZ and G is typically a feed-forward neural network . The sampling procedure defines a distribution over the data space , denoted as pG ( x ; θ ) . Our goal is to train the generator to approximate the data distribution p ( x ) . Since the generator distribution is implicit , it is infeasible to adopt maximum likelihood estimation directly to train the generator . To address the problem , we propose learning by teaching ( LBT ) , which introduces an auxiliary density estimator pE ( x ; φ ) parameterized by φ ( e.g. , a VAE ( Kingma & Welling , 2013 ) ) to fit the generator distribution pG ( x ; θ ) by maximizing the log-likelihood on generated samples . We train the generator to maximize the estimator ’ s likelihood evaluated on the training data . Formally , LBT is defined as a bilevel optimization problem ( Colson et al. , 2007 ) : max θ Ex∼p ( x ) [ log pE ( x ; φ ? ( θ ) ) ] , s.t . φ ? ( θ ) = argmax φ Ez∼pZ [ log pE ( G ( z ; θ ) ; φ ) ] , ( 1 ) where φ ? ( θ ) clarifies that the optimal φ ? depends on θ . For notational convenience , we denote fG ( φ ? ( θ ) ) =Ex∼p ( x ) [ log pE ( x ; φ ? ( θ ) ) ] , fE ( θ , φ ) =Ez∼pZ [ log pE ( G ( z ; θ ) ; φ ) ] , ( 2 ) in the sequel . In Sec . 4 , we provide theoretical analysis to show that the generator distribution can converge to the data distribution given that the generator and the estimator have enough capacity . 3.1 DIFFERENTIATE THROUGH THE BILEVEL OPTIMIZATION PROBLEM . The bilevel problem is generally challenging to solve . Here , we present a stochastic gradient ascend algorithm ( i.e. , Algorithm 1 ) by using an unrolling technique to derive the gradient . Specifically , to perform gradient ascend , we calculate the gradient of fG with respect to θ as follows : ∂fG ( φ ? ( θ ) ) ∂θ = ∂fG ( φ ? ( θ ) ) ∂φ ? ( θ ) ∂φ ? ( θ ) ∂θ = ∂fG ( φ ? ( θ ) ) ∂φ ? ( θ ) ∫ z ∂φ ? ( θ ) ∂G ( z ; θ ) ∂G ( z ; θ ) ∂θ pZdz , ( 3 ) where both ∂fG ( φ ? ( θ ) ) ∂φ ? ( θ ) and ∂G ( z ; θ ) ∂θ are easy to calculate . However , ∂φ ? ( θ ) ∂G ( z ; θ ) is intractable since φ ? ( θ ) can not be expressed as an analytic function of the generated samples G ( z ; θ ) . In the following , we rewrite G ( z ; θ ) as xz and ∂fE ( θ , φ ) ∂φ as∇φ for simplicity . On one hand , the influence function ( Koh & Liang , 2017 ) provides a way to calculate the gradient of φ ? ( θ ) w.r.t . the generated samples xz as follows : ∂φ ? ( θ ) ∂xz = −H−1φ ? ∇xz ( ∂fE ( θ , φ ) ∂φ ∣∣∣∣ φ ? ) , ( 4 ) Algorithm 1 Stochastic Gradient Ascend Training of LBT with the Unrolling Technique Input : data x , learning rate ηθ and ηφ , unrolling steps K and inner update iterations M . Initialize parameters θ0 and φ0 , and t = 1. repeat φ0t ← φt−1 for i = 1 to M do φit ← φi−1t + ηφ · ∂fE ( θ , φ ) ∂φ ∣∣∣ φi−1t end for Update φ : φt ← φMt φ0t ← φt Unrolling : φKt ← φ0t + ∑K i=1 ηφ · ∂fE ( θ , φ ) ∂φ ∣∣∣ φi−1t Update θ : θt ← θt−1 + ηθ ∂fG ( φ K t ) ∂θ Update t : t← t+ 1 until Both θ and φ converge . where Hφ ? is the Hessian of the objective fE w.r.t . φ at φ ? and is negative semi-definite ( Koh & Liang , 2017 ) . However , calculating the Hessian and its inverse is computationally expensive . On the other hand , a local optimum φ̂ ? of the density estimator parameters can be expressed as the fixed point of an iterative optimization procedure , φ0 =φ φk+1 =φk + η · ∂fE ( θ , φ ) ∂φ ∣∣∣∣ φk ( 5 ) φ̂ ? = lim k→∞ φk , where η is the learning rate1 . Since the samples used to evaluate the likelihood fE ( θ , φ ) are generated by G , each step of the optimization procedure is dependent on θ . We thus write φk ( θ , φ0 ) to clarify that φk is a function of θ and the initial value φ0 . Since ∂fE ( θ , φ ) ∂φ is differentiable w.r.t . xz for most density estimators such as VAEs , φk ( θ , φ0 ) is also differentiable w.r.t . xz . By unrolling for K steps , namely , using φK ( θ , φ0 ) to approximate φ ? ( θ ) in the objective fG ( φ ? ( θ ) ) , we optimize a surrogate objective for the generator formulated as fG ( φK ( θ , φ0 ) ) . Thus , the term ∂φ ? ( θ ) ∂xz is approximated as ∂φ ? ( θ ) ∂xz ≈ ∂φ K ( θ , φ0 ) ∂xz , which is known as the unrolling technique ( Metz et al. , 2016 ) . We now build connections between the above approximate gradients and the exact gradients given by the influence function . Assuming that the parameters of the density estimator is at its optimum φ ? , i.e. , φ0 = φ ? , we examine the case of K = 1 . The result of one step unrolling is given by : ∂φ1 ∂xz = ∂ ( φ0 + η∇φ ) ∂xz = η ∂∇φ ∂xz = η∇xz ( ∂fE ( θ , φ ) ∂φ ∣∣∣∣ φ0 ) = η∇xz ( ∂fE ( θ , φ ) ∂φ ∣∣∣∣ φ ? ) . ( 6 ) Note that the inner product of ∂φ 1 ∂xz and ∂φ ? ∂xz given by the influence function is positive because the Hessian Hφ ? is negative semi-definite ( Koh & Liang , 2017 ) . Therefore , the unrolling technique essentially gives an approximation of the influence function in Eqn . ( 4 ) under the condition that the estimator is good enough ( near to its optimaum φ ? ) . Besides , the unrolling technique is much more efficient as it does not need to inverse the Hessian matrix . Finally , the generator and the likelihood estimator can be updated using the following process , θ ← θ + ηθ ∂fG ( φ K ( θ , φ ) ) ∂θ , φ← φ+ ηφ ∂fE ( θ , φ ) ∂φ , ( 7 ) where ηθ and ηφ are the learning rates for the generator and the estimator , respectively . We perform several updates of φ per update of θ to keep the estimator good . Note that for other gradient-based optimization methods such as Adam ( Kingma & Ba , 2014 ) , the unrolling procedure is similar ( Metz et al. , 2016 ) . In our experiments , only a few steps of unrolling , e.g. , 5 steps , are sufficient for the training . The whole training algorithm is described in Algorithm 1 . 1We have omitted the learning rate decay for simplicity .

Summary: This work introduces a framework for learning implicit models that is robust to mode collapse. It consists in learning an explicit model of the implicit model through maximum likelihood while the later is used to teach the explicit model to better match the data distribution. The resulting bi-level optimization is carried out with truncated unrolled stochastic gradient descent.