EnGAN: Latent Space MCMC and Maximum Entropy Generators for Energy-based Models

1 INTRODUCTION . The early work on deep learning relied on unsupervised learning ( Hinton et al. , 2006 ; Bengio et al. , 2007 ; Larochelle et al. , 2009 ) to train energy-based models ( LeCun et al. , 2006 ) , in particular Restricted Boltzmann Machines , or RBMs . However , it turned out that training energy-based models without an analytic form for the normalization constant is very difficult , because of the challenge of estimating the gradient of the partition function , also known as the negative phase part of the log-likelihood gradient ( described in more details below , Sec . 2 ) . Several algorithms were proposed for this purpose , such as Contrastive Divergence ( Hinton , 2000 ) and Stochastic Maximum Likelihood ( Younes , 1998 ; Tieleman , 2008 ) , relying on Monte-Carlo Markov Chains ( MCMC ) to iteratively sample from the energy-based model . However , because they appear to suffer from either high bias or high variance ( due to long mixing times ) , training of RBMs and other Boltzmann machines has not remained competitive after the introduction of variational auto-encoders ( Kingma & Welling , 2014 ) and generative adversarial networks or GANs ( Goodfellow et al. , 2014 ) . In this paper , we revisit the question of training energy-based models , taking advantage of recent advances in GAN-related research , and propose a novel approach to training energy functions and sampling from them , called EnGAN . The main inspiration for the proposed solution is the earlier observation ( Bengio et al. , 2013 ) made on stacks of auto-encoders that sampling in latent space ( and then applying a decoder to map back to data space ) led to faster mixing and more efficient sampling . The authors observed that whereas the data manifold is generally very complex and curved , the corresponding distribution in latent space tends to be much simpler and flatter . This was verified visually by interpolating in latent space and projecting back to data space through the decoder , observing that the resulting samples look like data samples ( i.e. , the latent space manifold is approximately convex , with most points interpolated between examples encoded in latent space also having high probability ) . We propose a related approach , EnGAN , which also provides two energy functions , one in data space and one in latent space . A key ingredient of the proposed approach is the need to regularize the generator ( playing the role of the decoder in auto-encoders , but with no need for an encoder ) so as to increase its entropy . This is needed to make sure to produce negative examples that can kill off spurious minima of the energy function . This need was first identified by Kim & Bengio ( 2016 ) , who showed that in order for an approximate sampler to match the density associated with an energy function , a compromise must be reached between sampling low energy configurations and obtaining a high-entropy distribution . However , estimating and maximizing the entropy of a complex high-dimensional distribution is not trivial , and we take advantage for this purpose of very recently proposed GAN-based approaches for maximizing mutual information ( Belghazi et al. , 2018 ; Oord et al. , 2018 ; Hjelm et al. , 2018 ) , since the mutual information between the input and the output of the generator is equal to the entropy at the output of the generator . In this context , the main contributions of this paper are the following : • proposing EnGAN , a general architecture , sampling and training framework for energy functions , taking advantage of an estimator of mutual information between latent variables and generator output and approximating the negative phase samples with MCMC in latent space , • showing that the resulting energy function can be successfully used for anomaly detection , improving on recently published results with energy-based models , • showing that EnGAN produces sharp images - with competitive Inception and Frechet scores - and which also better cover modes than standard GANs and WGAN-GPs , while not suffering from the common blurriness issue of many maximum likelihood generative models . 2 LIKELIHOOD GRADIENT ESTIMATOR FOR ENERGY-BASED MODELS AND DIFFICULTIES WITH MCMC-BASED GRADIENT ESTIMATORS . Let x denote a sample in the data space X and Eθ : X → R an energy function corresponding to minus the logarithm of an unnormalized density density function pθ ( x ) = e−Eθ ( x ) Zθ ∝ e−Eθ ( x ) ( 1 ) where Zθ : = ∫ e−Eθ ( x ) dx is the partition function or normalizing constant of the density sample in the latent space . Let pD be the training distribution , from which the training set is drawn . Towards optimizing the parameters θ of the energy function , the maximum likelihood parameter gradient is ∂Ex∼pD [ log pθ ( x ) ] ∂θ = Ex∼pD [ ∂Eθ ( x ) ∂θ ] − Ex∼pθ ( x ) [ ∂Eθ ( x ) ∂θ ] ( 2 ) where the second term is the gradient of logZθ , and the sum of the two expectations is zero when training has converged , with expected energy gradients in the positive phase ( under the data pD ) matching those under the negative phase ( under pθ ( x ) ) . Training thus consists in matching the shape of two distributions : the positive phase distribution ( associated with the data ) and the negative phase distribution ( where the model is free-running and generating configurations by itself ) . This observation has motivated the pre-GAN idea presented by Bengio ( 2009 ) that “ model samples are negative examples ” and a classifier could be used to learn an energy function if it separated the data distribution from the model ’ s own samples . Shortly after introducing GANs , Goodfellow ( 2014 ) also made a similar connection , related to noise-contrastive estimation ( Gutmann & Hyvarinen , 2010 ) . One should also recognize the similarity between Eq . 2 and the objective function for Wasserstein GANs or WGAN ( Arjovsky et al. , 2017 ) . In the next section , we examine a way to train what appears to be a particular form of WGAN that makes the discriminator compute an energy function . The main challenge in Eq . 2 is to obtain samples from the distribution pθ associated with the energy functionEθ . Although having an energy function is convenient to obtain a score allowing to compare the relative probability of different x ’ s , it is difficult to convert an energy function into a generative function . The commonly studied approaches for this are based on Monte-Carlo Markov chains , in which one iteratively updates a candidate configuration , until these configurations converge in distribution to the desired distribution pθ . For the RBM , the most commonly used algorithms have been Contrastive Divergence ( Hinton , 2000 ) and Stochastic Maximum Likelihood ( Younes , 1998 ; Tieleman , 2008 ) , relying on the particular structure of the RBM to perform Gibbs sampling . Although these MCMC-based methods are appealing , RBMs ( and their deeper form , the deep Boltzmann machine ) have not been competitive in recent years compared to autoregressive models ( van den Oord et al. , 2016 ) , variational auto-encoders ( Kingma & Welling , 2014 ) and generative adversarial networks or GANs ( Goodfellow et al. , 2014 ) . What has been hypothesized as a reason for the poorer results obtained with energy-based models trained with an MCMC estimator for the negative phase gradient is that running a Markov chain in data space is fundamentally difficult when the distribution is concentrated ( e.g , near manifolds ) and has many modes separated by vast areas of low probability . This mixing challenge is discussed by Bengio et al . ( 2013 ) who argue that a Markov chain is very likely to produce only sequences of highly probable configurations . If two modes are far from each other and only local moves are possible ( which is typically the case with MCMCs ) , it becomes exponentially unlikely to traverse the ’ desert ’ of low probability which can separate two modes . This makes mixing between modes difficult in high-dimensional spaces with strong concentration of probability mass in some places ( e.g . corresponding to different categories ) and very low probability elsewhere . In the same papers , the authors propose a heuristic method for jumping between modes , based on performing the random walk not in data space but in the latent space of an auto-encoder . Data samples can then be obtained by mapping the latent samples to data space via the decoder . They argue that auto-encoders tend to flatten the data distribution and bring the different modes closer to each other . The EnGAN sampling method proposed here is highly similar but leads to learning both an energy function in data space and one in latent space , from which we find that better samples are obtain . The energy function can be used to perform the appropriate Metropolis-Hastings rejection . Having an efficient way to approximately sample from the energy function also opens to the door to estimating the log-likelihood gradient with respect to the energy function according to Eq . 2 , as outlined below . 3 TURNING GAN DISCRIMINATORS INTO ENERGY FUNCTIONS WITH ENTROPY MAXIMIZATION . Turning a GAN discriminator into an energy function has been studied in the past ( Kim & Bengio , 2016 ; Zhao et al. , 2016 ; Dai et al. , 2017 ) but in order to turn a GAN discriminator into an energy function , a crucial and difficult requirement is the maximization of entropy at the output of the generator . Let ’ s see why . In Eq . 2 , we can replace the difficult to sample pθ by another generative process , say pG , such as the generative distribution associated with a GAN generator : LE = Ex∼pD [ ∂Eθ ( x ) ∂θ ] − Ex∼pG ( x ) [ ∂Eθ ( x ) ∂θ ] + Ω ( 3 ) where Ω is a regularizer which we found necessary to avoid numerical problems in the scale ( temperature ) of the energy . In this paper we use a gradient norm regularizer ( Gulrajani et al. , 2017 ) Ω = ∣∣∣∣∇xEθ ( x ) ∣∣∣∣2 for this purpose . This is similar to the training objective of a WGAN as to Eq . 2 , but this interpretation allows to train the energy function only to the extent that pG is sufficiently similar to pθ . To make them match , consider optimizingG to minimize the KL divergenceKL ( pG||pθ ) , which can be rewritten in terms of minimizing the energy of the samples from the generator while maximizing the entropy at the output of the generator : KL ( pG||pθ ) = H [ pG ] − EpG [ log pθ ( x ) ] as already shown by Kim & Bengio ( 2016 ) . When taking the gradient of KL ( pG||pθ ) with respect to the parameters w of the generator , the partition function of pG disappears and we equivalently can optimize w to minimize LG = −H [ pG ] + Ez∼pzEθ ( G ( z ) ) ( 4 ) where pz is the prior distribution of the latent variable of the generator . In order to maximize the entropy at the output of the generator , we propose to exploit another GANderived framework in order to estimate and maximize mutual information between the input and output of the generator network . The entropy at the output of a deterministic function ( the generator in our case ) can be computed using an estimator of mutual information between the input and output of that function , since the conditional entropy term is 0 because the function is deterministic . With x = G ( z ) the function of interest : I ( X , Z ) = H ( X ) −H ( X|Z ) = H ( G ( Z ) ) − : 0 H ( G ( Z ) |Z ) Hence , any neural mutual information maximization method such as MINE ( Belghazi et al. , 2018 ) , noise constrastive estimation ( Oord et al. , 2018 ) and DeepINFOMAX ( Hjelm et al. , 2018 ) can be applied to estimate and maximize the entropy of the generator . All these estimators are based on training a discriminator which separates the joint distribution p ( X , Z ) from the product of the corresponding marginals p ( X ) p ( Z ) . As proposed by Brakel & Bengio ( 2017 ) in the context of using a discriminator to minimize statistical dependencies between the outputs of an encoder , the samples from the marginals can be obtained by creating negative examples pairing an X and a Z from different samples of the joint , e.g. , by independently shuffling each column of a matrix holding a minibatch with one row per example . The training objective for the discriminator can be chosen in different ways . In this paper , we used the Deep INFOMAX ( DIM ) estimator ( Hjelm et al. , 2018 ) , which is based on maximizing the Jensen-Shannon divergence between the joint and the marginal ( see Nowozin et al . for the original F-GAN formulation ) . IJSD ( X , Z ) = Ep ( X , Z ) [ −s+ ( −T ( X , Z ) ) ] − Ep ( X ) p ( Z ) [ s+ ( T ( X , Z ) ) ] ( 5 ) where s+ ( a ) = log ( 1+ea ) is the softplus function . The discriminator T used to increase entropy at the output of the generator is trained by maximizing IJSD ( X , Z ) with respect to the parameters of T . WithX = G ( Z ) the output of the generator , IJSD ( G ( Z ) , Z ) is one of the terms to be minimized the objective function for training G , with the effect of maximizing the generator ’ s output entropy H ( G ( Z ) ) . The overall training objective for G is LG = −IJSD ( G ( Z ) , Z ) + Ez∼pzEθ ( G ( z ) ) ( 6 ) where Z ∼ pz , the latent prior ( typically a N ( 0 , I ) Gaussian ) .

Summary: In this paper, the authors extend the framework proposed by Kim&Bengio 2016 and Dai et.al. 2017, which introduce an extra step to fit a generator to approximate the current model for estimating the deep energy model. Specifically, the generator is fitted by reverse KL divergence. To bypass the difficulty in handling the entropy term, the authors exploit the Deep INFOMAX formulation, which introduces one more discriminator. Finally, to obtain better samples, the authors inject the Metropolis-adjusted Langevin algorithm within the learned generator to generate samples in latent space. They demonstrate the better performances of the proposed algorithms in both synthetic and real-world datasets, and apply the learned model for anomaly detection task.