Year: 2018

Abstract:
Unsupervised learning is about capturing dependencies between variables and driven by the contrast between the probable vs improbable configurations of these variables, often either via a generative model which only samples probable ones or with an energy function (unnormalized log-density) which is low for probable ones and high for improbable ones. Here we consider learning both an energy function and an efficient approximate sampling mechanism for the corresponding distribution. Whereas the critic (or discriminator) in generative adversarial networks (GANs) learns to separate data and generator samples, introducing an entropy maximization regularizer on the generator can turn the interpretation of the critic into an energy function, which separates the training distribution from everything else, and thus can be used for tasks like anomaly or novelty detection. This paper is motivated by the older idea of sampling in latent space rather than data space because running a Monte-Carlo Markov Chain (MCMC) in latent space has been found to be easier and more efficient, and because a GAN-like generator can convert latent space samples to data space samples. For this purpose, we show how a Markov chain can be run in latent space whose samples can be mapped to data space, producing better samples. These samples are also used for the negative phase gradient required to estimate the log-likelihood gradient of the data space energy function. To maximize entropy at the output of the generator, we take advantage of recently introduced neural estimators of mutual information. We find that in addition to producing a useful scoring function for anomaly detection, the resulting approach produces sharp samples (like GANs) while covering the modes well, leading to high Inception and Frchet scores.

heading: 1 INTRODUCTION
text: The early work on deep learning relied on unsupervised learning (Hinton et al., 2006; Bengio et al., 2007; Larochelle et al., 2009) to train energy-based models (LeCun et al., 2006), in particular Restricted Boltzmann Machines, or RBMs. However, it turned out that training energy-based models without an analytic form for the normalization constant is very difficult, because of the challenge of estimating the gradient of the partition function, also known as the negative phase part of the log-likelihood gradient (described in more details below, Sec. 2). Several algorithms were proposed for this purpose, such as Contrastive Divergence (Hinton, 2000) and Stochastic Maximum Likelihood (Younes, 1998; Tieleman, 2008), relying on Monte-Carlo Markov Chains (MCMC) to iteratively sample from the energy-based model. However, because they appear to suffer from either high bias or high variance (due to long mixing times), training of RBMs and other Boltzmann machines has not remained competitive after the introduction of variational auto-encoders (Kingma & Welling, 2014) and generative adversarial networks or GANs (Goodfellow et al., 2014).
In this paper, we revisit the question of training energy-based models, taking advantage of recent advances in GAN-related research, and propose a novel approach to training energy functions and sampling from them, called EnGAN. The main inspiration for the proposed solution is the earlier observation (Bengio et al., 2013) made on stacks of auto-encoders that sampling in latent space (and then applying a decoder to map back to data space) led to faster mixing and more efficient sampling. The authors observed that whereas the data manifold is generally very complex and curved, the corresponding distribution in latent space tends to be much simpler and flatter. This was verified visually by interpolating in latent space and projecting back to data space through the decoder,
observing that the resulting samples look like data samples (i.e., the latent space manifold is approximately convex, with most points interpolated between examples encoded in latent space also having high probability). We propose a related approach, EnGAN, which also provides two energy functions, one in data space and one in latent space. A key ingredient of the proposed approach is the need to regularize the generator (playing the role of the decoder in auto-encoders, but with no need for an encoder) so as to increase its entropy. This is needed to make sure to produce negative examples that can kill off spurious minima of the energy function. This need was first identified by Kim & Bengio (2016), who showed that in order for an approximate sampler to match the density associated with an energy function, a compromise must be reached between sampling low energy configurations and obtaining a high-entropy distribution. However, estimating and maximizing the entropy of a complex high-dimensional distribution is not trivial, and we take advantage for this purpose of very recently proposed GAN-based approaches for maximizing mutual information (Belghazi et al., 2018; Oord et al., 2018; Hjelm et al., 2018), since the mutual information between the input and the output of the generator is equal to the entropy at the output of the generator.
In this context, the main contributions of this paper are the following:
• proposing EnGAN, a general architecture, sampling and training framework for energy functions, taking advantage of an estimator of mutual information between latent variables and generator output and approximating the negative phase samples with MCMC in latent space,
• showing that the resulting energy function can be successfully used for anomaly detection, improving on recently published results with energy-based models,
• showing that EnGAN produces sharp images - with competitive Inception and Frechet scores - and which also better cover modes than standard GANs and WGAN-GPs, while not suffering from the common blurriness issue of many maximum likelihood generative models.

heading: 2 LIKELIHOOD GRADIENT ESTIMATOR FOR ENERGY-BASED MODELS AND DIFFICULTIES WITH MCMC-BASED GRADIENT ESTIMATORS
text: Let x denote a sample in the data space X and Eθ : X → R an energy function corresponding to minus the logarithm of an unnormalized density density function
pθ(x) = e−Eθ(x)
Zθ ∝ e−Eθ(x) (1) where Zθ := ∫ e−Eθ(x)dx is the partition function or normalizing constant of the density sample in the latent space. Let pD be the training distribution, from which the training set is drawn. Towards optimizing the parameters θ of the energy function, the maximum likelihood parameter gradient is
∂Ex∼pD [log pθ(x)] ∂θ
= Ex∼pD [ ∂Eθ(x)
∂θ
] − Ex∼pθ(x) [ ∂Eθ(x)
∂θ
] (2)
where the second term is the gradient of logZθ, and the sum of the two expectations is zero when training has converged, with expected energy gradients in the positive phase (under the data pD) matching those under the negative phase (under pθ(x)). Training thus consists in matching the shape of two distributions: the positive phase distribution (associated with the data) and the negative phase distribution (where the model is free-running and generating configurations by itself). This observation has motivated the pre-GAN idea presented by Bengio (2009) that “model samples are negative examples” and a classifier could be used to learn an energy function if it separated the data distribution from the model’s own samples. Shortly after introducing GANs, Goodfellow (2014) also made a similar connection, related to noise-contrastive estimation (Gutmann & Hyvarinen, 2010). One should also recognize the similarity between Eq. 2 and the objective function for Wasserstein GANs or WGAN (Arjovsky et al., 2017). In the next section, we examine a way to train what appears to be a particular form of WGAN that makes the discriminator compute an energy function.
The main challenge in Eq. 2 is to obtain samples from the distribution pθ associated with the energy functionEθ. Although having an energy function is convenient to obtain a score allowing to compare the relative probability of different x’s, it is difficult to convert an energy function into a generative function. The commonly studied approaches for this are based on Monte-Carlo Markov chains, in
which one iteratively updates a candidate configuration, until these configurations converge in distribution to the desired distribution pθ. For the RBM, the most commonly used algorithms have been Contrastive Divergence (Hinton, 2000) and Stochastic Maximum Likelihood (Younes, 1998; Tieleman, 2008), relying on the particular structure of the RBM to perform Gibbs sampling. Although these MCMC-based methods are appealing, RBMs (and their deeper form, the deep Boltzmann machine) have not been competitive in recent years compared to autoregressive models (van den Oord et al., 2016), variational auto-encoders (Kingma & Welling, 2014) and generative adversarial networks or GANs (Goodfellow et al., 2014).
What has been hypothesized as a reason for the poorer results obtained with energy-based models trained with an MCMC estimator for the negative phase gradient is that running a Markov chain in data space is fundamentally difficult when the distribution is concentrated (e.g, near manifolds) and has many modes separated by vast areas of low probability. This mixing challenge is discussed by Bengio et al. (2013) who argue that a Markov chain is very likely to produce only sequences of highly probable configurations. If two modes are far from each other and only local moves are possible (which is typically the case with MCMCs), it becomes exponentially unlikely to traverse the ’desert’ of low probability which can separate two modes. This makes mixing between modes difficult in high-dimensional spaces with strong concentration of probability mass in some places (e.g. corresponding to different categories) and very low probability elsewhere. In the same papers, the authors propose a heuristic method for jumping between modes, based on performing the random walk not in data space but in the latent space of an auto-encoder. Data samples can then be obtained by mapping the latent samples to data space via the decoder. They argue that auto-encoders tend to flatten the data distribution and bring the different modes closer to each other. The EnGAN sampling method proposed here is highly similar but leads to learning both an energy function in data space and one in latent space, from which we find that better samples are obtain. The energy function can be used to perform the appropriate Metropolis-Hastings rejection. Having an efficient way to approximately sample from the energy function also opens to the door to estimating the log-likelihood gradient with respect to the energy function according to Eq. 2, as outlined below.

heading: 3 TURNING GAN DISCRIMINATORS INTO ENERGY FUNCTIONS WITH ENTROPY MAXIMIZATION
text: Turning a GAN discriminator into an energy function has been studied in the past (Kim & Bengio, 2016; Zhao et al., 2016; Dai et al., 2017) but in order to turn a GAN discriminator into an energy function, a crucial and difficult requirement is the maximization of entropy at the output of the generator. Let’s see why. In Eq. 2, we can replace the difficult to sample pθ by another generative process, say pG, such as the generative distribution associated with a GAN generator:
LE = Ex∼pD [ ∂Eθ(x)
∂θ
] − Ex∼pG(x) [ ∂Eθ(x)
∂θ
] + Ω (3)
where Ω is a regularizer which we found necessary to avoid numerical problems in the scale (temperature) of the energy. In this paper we use a gradient norm regularizer (Gulrajani et al., 2017) Ω = ∣∣∣∣∇xEθ(x)∣∣∣∣2 for this purpose. This is similar to the training objective of a WGAN as to Eq. 2, but this interpretation allows to train the energy function only to the extent that pG is sufficiently similar to pθ. To make them match, consider optimizingG to minimize the KL divergenceKL(pG||pθ), which can be rewritten in terms of minimizing the energy of the samples from the generator while maximizing the entropy at the output of the generator: KL(pG||pθ) = H[pG] − EpG [log pθ(x)] as already shown by Kim & Bengio (2016). When taking the gradient of KL(pG||pθ) with respect to the parameters w of the generator, the partition function of pG disappears and we equivalently can optimize w to minimize
LG = −H[pG] + Ez∼pzEθ(G(z)) (4)
where pz is the prior distribution of the latent variable of the generator.
In order to maximize the entropy at the output of the generator, we propose to exploit another GANderived framework in order to estimate and maximize mutual information between the input and output of the generator network. The entropy at the output of a deterministic function (the generator in our case) can be computed using an estimator of mutual information between the input and output of that function, since the conditional entropy term is 0 because the function is deterministic. With
x = G(z) the function of interest:
I(X,Z) = H(X)−H(X|Z) = H(G(Z))− : 0 H(G(Z)|Z)
Hence, any neural mutual information maximization method such as MINE (Belghazi et al., 2018), noise constrastive estimation (Oord et al., 2018) and DeepINFOMAX (Hjelm et al., 2018) can be applied to estimate and maximize the entropy of the generator. All these estimators are based on training a discriminator which separates the joint distribution p(X,Z) from the product of the corresponding marginals p(X)p(Z). As proposed by Brakel & Bengio (2017) in the context of using a discriminator to minimize statistical dependencies between the outputs of an encoder, the samples from the marginals can be obtained by creating negative examples pairing an X and a Z from different samples of the joint, e.g., by independently shuffling each column of a matrix holding a minibatch with one row per example. The training objective for the discriminator can be chosen in different ways. In this paper, we used the Deep INFOMAX (DIM) estimator (Hjelm et al., 2018), which is based on maximizing the Jensen-Shannon divergence between the joint and the marginal (see Nowozin et al. for the original F-GAN formulation).
IJSD(X,Z) = Ep(X,Z)[−s+(−T (X,Z))]− Ep(X)p(Z)[s+(T (X,Z))] (5) where s+(a) = log(1+ea) is the softplus function. The discriminator T used to increase entropy at the output of the generator is trained by maximizing IJSD(X,Z) with respect to the parameters of T . WithX = G(Z) the output of the generator, IJSD(G(Z), Z) is one of the terms to be minimized the objective function for training G, with the effect of maximizing the generator’s output entropy H(G(Z)). The overall training objective for G is
LG = −IJSD(G(Z), Z) + Ez∼pzEθ(G(z)) (6) where Z ∼ pz , the latent prior (typically a N(0, I) Gaussian).

heading: 4 PROPOSED LATENT SPACE MCMC AND MAXIMUM ENTROPY GENERATOR FOR ENERGY-BASED MODELS
text: Next, the proposed z̃t+1 is accepted or rejected using the Metropolis Hastings algorithm, by computing the acceptance ratio
r = p(x̃t+1)
p(xt) = exp
{ − Eθ(Gω(z̃t+1)) + Eθ(Gω(zt)) } and accepting (setting zt+1 = z̃t+1) with probability r.
The overall training procedure for EnGAN is detailed in Algorithm 1, with MALA referring to the above procedure for sampling by MCMC, with nmcmc steps. When nmcmc=0, we recover the base
case where z is only sampled from the prior and passed through G, and no MCMC is done to clean up the sample.
Algorithm 1 EnGAN Training Procedure Default values: Adam parameters α = 0.0001, β1 = 0.5, β2 = 0.9;λ = 0.1; nϕ = 5
Require: Score penalty coefficient λ, number of energy function updates nϕ per generator updates, number of MCMC steps nmcmc, number of training iterations T , Adam hyperparameters α, β1 and β2. Require: Energy function Eθ with parameters θ, entropy statistics network Tφ with parameters φ, generator network Gω with parameters ω, minibatch size m for t = 1, ..., T do
for 1, ..., nϕ do Sample minibatch of real data {x(1), ...,x(m)} ∼ PD. Sample minibatch of latent variables {z(1)0 , ..., z (m) 0 } ∼ Pz .
for τ = 1, ..., nmcmc do zτ+1 ← MALA(zτ , Eθ ◦G) end for x̃← Gω(znmcmc)
LE ← 1
m [ m∑ i Eθ(x (i))− m∑ i Eθ(x̃ (i)) + λ m∑ i ∣∣∣∣∇x(i)Eθ(x(i))∣∣∣∣2] θ ← Adam(LE , θ, α, β1, β2)
end for Sample minibatch of latent variables z = {z(1), ..., z(m)} ∼ Pz . x̃← Gω(z) Per-dimension shuffle of the minibatch z of latent variables, obtaining {z̃(1), ..., z̃(m)}.
LH ← 1
m m∑ i [ log σ(Tφ(x (i), z(i)))− log ( 1− σ(Tφ(x(i), z̃(i))) )] LG ← 1
m [ m∑ i Eθ(x̃ (i)) ] + LH
ω ← Adam(LG, ω, α, β1, β2) φ← Adam(LH , φ, α, β1, β2)
end for The gradient-based updates can be performed with any gradient-based learning rule. We used Adam in our experiments.

heading: 5 EXPERIMENTAL SETUP

heading: 5.1 SYNTHETIC TOY DATASETS
text: Generative models trained with maximum likelihood often suffer from the problem of spurious modes and excessive entropy of the trained distribution, where the model incorrectly assigns high probability mass to regions not present in the data manifold. Typical energy-based models such as RBMs suffer from this problem partly because of the poor approximation of the negative phase gradient, as discussed above.
To check if EnGAN suffers from spurious modes, we train the energy-based model on synthetic 2D datasets (swissroll, 25gaussians and 8gaussians) similar to Gulrajani et al. (2017) and visualize the energy function.
From the probaility density plots on Figure 1, we can see that the energy model doesn’t suffer from spurious modes and learns a sharp energy distribution.

heading: 5.2 DISCRETE MODE COLLAPSE EXPERIMENT
text: GANs have been notoriously known to have issues with mode collapse, by which certain modes of the data distribution are not at all represented by the generative model. Similar to the mode dropping issue that occurs in GANs, our generator is prone to mode dropping as well, since it is matched with the energy model’s distribution using a reverse KL penalty DKL[PG || PE ]. Although the entropy maximization term attempts to fix this issue by maximizing the entropy of the generator’s distribution, it is important to verify this effect experimentally. For this purpose, we follow the same experimental setup as Metz et al. (2016) and Srivastava et al. (2017). We train our generative model on the StackedMNIST dataset, which is a synthetic dataset created by stacking MNIST on different channels. The number of modes can be counted using a pretrained MNIST classifier, and the KL divergence can be calculated empirically between the mode count distribution produced by the generative model and true data (assumed to be uniform).
From Table 1, we can see that our model naturally covers all the modes in that data, without dropping a single mode. Apart from just representing all the modes of the data distribution, our model also better matches the data distribution as evidenced by the very low KL divergence scores as compared to the baseline WGAN-GP.
We noticed empirically that modeling 103 modes was quite trivial for benchmark methods such as WGAN-GP (Gulrajani et al., 2017). Hence, we also try evaluating our model on a new dataset with 104 modes (4 stacks). The 4-StackedMNIST was created to have similar statistics to the original 3-StackedMNIST dataset. We randomly sample and fix 128 × 104 images to train the generative model and take 26× 104 samples for evaluations.

heading: 5.3 PERCEPTUAL QUALITY OF SAMPLES
text: Generative models trained with maximum likelihood have often been found to produce more blurry samples. Our energy model is trained with maximum likelihood to match the data distribution and the generator is trained to match the energy model’s distribution with a reverse KL penalty. To evaluate if our generator exhibits blurriness issues, we train our EnGAN model on the standard benchmark 32x32 CIFAR10 dataset for image modeling. We additionally train our models on the 64x64 cropped CelebA - celebrity faces dataset to report qualitative samples from our model. Similar to recent GAN works (Miyato et al., 2018), we report both Inception Score (IS) and Frchet Inception Distance (FID) scores on the CIFAR10 dataset and compare it with a competitive WGAN-GP baseline.
From Table 2, we can see that in addition to learning an energy function, EnGAN trains generative model producing samples comparable to recent adversarial methods such as WGAN-GP (Gulrajani et al., 2017) widely known for producing samples of high perceptual quality. Additionally, we attach samples from the generator trained on the CelebA dataset and the 3-StackedMNIST dataset for qualitative inspection. As shown below in Fig. 4, the visual quality of the samples can be further improved by using the proposed MCMC sampler.

heading: 5.4 APPLICATION TO ANOMALY DETECTION
text: Apart from the usefulness of energy estimates for relative density estimation (up to the normalization constant), energy functions can also be useful to perform unsupervised anomaly detection. Unsupervised anomaly detection is a fundamental problem in machine learning, with critical applications in many areas, such as cybersecurity, complex system management, medical care, etc. Density estimation is at the core of anomaly detection since anomalies are data points residing in low probability
density areas. We test the efficacy of our energy-based density model for anomaly detection using two popular benchmark datasets: KDDCUP and MNIST.
KDDCUP We first test our generative model on the KDDCUP99 10 percent dataset from the UCI repository (Lichman et al., 2013).
Our baseline for this task is Deep Structured Energy-based Model for Anomaly Detection (DSEBM) (Zhai et al., 2016), which trains deep energy models such as Convolutional and Recurrent EBMs using denoising score matching instead of maximum likelihood, for performing anomaly detection. We also report scores on the state of the art DAGMM (Zong et al., 2018), which learns a Gaussian Mixture density model (GMM) over a low dimensional latent space produced by a deep autoencoder.
We train our model on the KDD99 data and use the score norm ||∇xEθ(x)||22 as the decision function, similar to Zhai et al. (2016).
From Table 3, we can see that our EnGAN energy function outperforms the previous SOTA energybased model (DSEBM) by a large margin (+0.1990 F1 score) and is comparable to the current SOTA model (DAGMM) specifically designed for anomaly detection.
MNIST Next we evaluate our generative model on anomaly detection of high dimensional image data. We follow the same experiment setup as (Zenati et al., 2018) and make each digit class an anomaly and treat the remaining 9 digits as normal examples. We also use the area under the precision-recall curve (AUPRC) as the metric to compare models.
From Table 4, it can be seen that our energy model outperforms VAEs for outlier detection and is comparable to the SOTA BiGAN-based anomaly detection methods for this dataset (Zenati et al., 2018) which train bidirectional GANs to learn both an encoder and decoder (generator) simultaneously. An advantage with our method is that it has theoretical justification for the usage of energy function as a decision function, whereas the BiGAN-σ model lacks justification for using a combination of the reconstruction error in output space as well as the discriminator’s cross entropy loss for the decision function.

heading: 5.5 MCMC SAMPLING
text: To show that the Metropolis Adjusted Langevin Algorithm (MALA) performed in latent space produced good samples in observed space, we attach samples from the beginning (with z sampled from a Gaussian) and end of the chain for visual inspection. From the attached samples, it can be seen that the MCMC sampler appears to perform a smooth walk on the image manifold, with the initial and final images only differing in a few latent attributes such as hairstyle, background color, face orientation, etc.

heading: 6 CONCLUSIONS
text: We proposed EnGAN, an energy-based generative model that produces energy estimates using an energy model and a generator that produces fast approximate samples. This takes advantage of novel methods to maximize the entropy at the output of the generator using a GAN-like technique. We have shown that our energy model learns good energy estimates using visualizations in toy 2D data and through performance in unsupervised anomaly detection. We have also shown that our generator produces samples of high perceptual quality by measuring Inception and Frchet scores and shown that EnGAN is robust to the respective weaknesses of GAN models (mode dropping) and maximumlikelihood energy-based models (spurious modes). We found that running an MCMC in latent space rather than in data space (by composing the generator and the data-space energy to obtain a latentspace energy) works substantially better than running the MCMC in data-space.

heading: 7 APPENDIX

heading: 7.1 MCMC IN DATA SPACE

heading: 7.2 ARCHITECTURE AND HYPERPARAMETERS
text: For all experiments we use Adam as the optimizer with α = 0.0001, β1 = 0.5, β2 = 0.9. We used nmcmc = 0 (no MCMC steps during training) for all scores reported in the paper.
Toy Data: The generator, energy-model and the statistics network are simple 3-hidden layer MLPs with dimensionality 512. The input to the statistics network is a conatenation of the inputs x and latents z. For these experiments, we use the energy norm co-efficient λ = 0.1
StackedMNIST: : In line with previous work, we adopt the same architectural choices for the generator and energy-model / discriminator as VeeGAN (Srivastava et al., 2017). The statistics network is modeled similar to the energy-model, except with the final MLP which now takes as input both the latents z and reduced feature representation of x produced by the CNN.
CIFAR10: For the CIFAR10 experiments, we adopt the same ’Standard CNN’ architecture as in SpectralNorm (Miyato et al., 2018). We adapt the architecture for the Statistics Network similar to the StackedMNIST experiments as mentioned above. For these experiments, we use the energy norm co-efficient λ = 10
Anomaly Detection: For the KDD99 dataset, we adopt the same architecture as (Zenati et al., 2018). We noticed that using nψ = 1 and λ = 105 worked best for these experiments. A large energy norm coefficient was specifically necessary since the energy model overfit to some artifacts in the data and exploded in value.
For the MNIST anomaly detection experiments, we use the same architecture as the StackedMNIST experiments.

Review summaries:
In this paper, the authors extend the framework proposed by Kim&Bengio 2016 and Dai et.al. 2017, which introduce an extra step to fit a generator to approximate the current model for estimating the deep energy model. Specifically, the generator is fitted by reverse KL divergence. To bypass the difficulty in handling the entropy term, the authors exploit the Deep INFOMAX formulation, which introduces one more discriminator. Finally, to obtain better samples, the authors inject the Metropolis-adjusted Langevin algorithm within the learned generator to generate samples in latent space. They demonstrate the better performances of the proposed algorithms in both synthetic and real-world datasets, and apply the learned model for anomaly detection task.