Generalized Energy Based Models
Authors: Michael Arbel
Liang Zhou
Arthur Gretton
Venue: ICLR 2021 Poster (2021)
Keywords: Sampling
MCMC
Generative Models
Adversarial training
Optimization
Density estimation

Abstract: We introduce the Generalized Energy Based Model (GEBM) for generative modelling. These models combine two  trained components: a base distribution (generally an implicit model), which can learn the support of data with low intrinsic dimension in a high dimensional space; and an energy function, to refine the probability mass on the learned support. 
Both the energy function and base jointly constitute the final model, unlike GANs, which retain only the base distribution (the "generator").  
GEBMs are trained by alternating between learning the energy and the base. 
We show that both training stages are well-defined: the energy is learned by maximising a generalized likelihood, and the resulting energy-based loss provides informative gradients for learning the base.
Samples from the posterior on the latent space of the trained model can be obtained via MCMC, thus finding regions in this space that produce better quality samples.
Empirically, the GEBM samples on image-generation tasks are of much better quality than those from the learned generator alone, indicating that all else being equal, the GEBM will outperform a GAN of the same complexity. When using normalizing flows as base measures, GEBMs succeed on density modelling tasks returning comparable performance to direct maximum likelihood of the same networks.

Published as a conference paper at ICLR 2021
GENERALIZED ENERGY BASED MODELS
Michael Arbel', Liang Zhou & Arthur Gretton Gatsby Computational Neuroscience Unit, University College London
ABSTRACT
We introduce the Generalized Energy Based Model (GEBM) for generative mod- elling. These models combine two trained components: a base distribution (generally implicit model), which can learn the support of data with low intrinsic dimension in a high dimensional space; and an energy function, to refine the probability mass on the learned support. Both the energy function and base jointly constitute the final model, unlike GANs, which retain only the base distribution (the "generator"). ) GEBMs are trained by alternating between learning the energy and the base. We show that both training stages are well-defined: the energy is learned by maximising generalized likelihood, and the resulting energy-based loss provides informative gradients for learning the base. Samples from the posterior on the latent space of the trained model can be obtained via MCMC, thus finding regions in this space that produce better qual- ity samples. Empirically, the GEBM samples image-geen genggeggneattion tasks are of much better quality than those from the learned generator alone, indicating that all clse being equal, the GEBM will outperform a GAN of the same complexity. When using normalizing flows as base measures, GEBMs succeed on density modelling tasks, re- turning comparable performance direct maximum likelihood of the same networks.
INTRODUCTION
Energy-based models (EBMs) have a long history in physics, statistics and machine learning (LeCun et al., 2006). 210,2 They belong to the class of explicit models, and can be described by a family of energies E which define probability distributions with density proportional to exp(-E), Those models are often known up a normalizing constant Z (E), also called the furtitionfunction. The learning task consists of finding an optimal function that best describes a given system or target distribution P. This can be achieved using maximum likelihood estimation (MLE), however the intractability of the normalizing partition function makes this learning task challenging. Thus, various methods have been proposed to address this (Hinton, 2002; Hyvärinen, 2005; Gutmann and Hyvärinen, 2012; Dai et al,, 201 9a;b). All these methods estimate EBMs that are supported over the whole space. In many applications, however, I is believed to be supported on an unknown lower dimensional manifold. This happens in particular when there are strong dependencies between variables in the data (Thiry et al., 2021), l), and suggests incorporating a low-dimensionality hypothesis the model.
function.
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are particular way to enforce low dimensional structure in a model. They rely on an implicit model, the generator, to produce samples supported low-dimensional manifold by mapping pre-defined latent noise to the sample space using a trained function. GANs have been very successful in generating high-quality samples on various tasks, especially for unsupervised image generation (Brock et al., 2018). The generator is trained adversarially against discriminator network whose goal is to distinguish samples produced by the generator from the target data. This has inspired further research to extend the training procedure to more general losses (Nowozin et al., 2016 6; Arjovsky et al., 2017: 7: Li al., 2017; Binkkwski et al., 2018; Arbel al., 2018) and to improve its stability (Miyato al., 2018; Gulrajani al., 2017: Nagarajan and Kolter, 2017; Kodali et al., 2017). While the generator of a GAN has effectively a fow-dimensional support, it remains challenging to refine the distribution of mass on that support using pre-defined latent noise. For instance, as shown by Cornish et al. (2020) for normalizing flows, when the latent distribution is unimodal and the target distribution possesses multiple disconnected low-dimensional components, the generator, as continuous map, compensates for this mismatch using steeper slopes. In practice, this implies the need for more complicated generators.
"Correspondence: mi chael n.. arbel @gma com.

---

Published as a conference paper at ICLR 2021
In the present work, we propose new class of models, called Generalized Energy Based Models (GEBMs), which can represent distributions supported on low-dimensional manifolds, while offering more flexibility in refining the mass those manifolds. GEBMs combine the strength of both implicit and explicit models in two separate components: a base distribution (often chosen to be an implicit model) which learns the low-dimensional support of the data, and an energy function that can refine the probability mass on that learned support. We propose to train the GEBM by alternating between learning the energy and the base, analogous to f-GAN training (Goodfellow et al., 2014; Nowozin et al., 2016). The energy is learned by maximizing a generalized notion of likelihood which we relate to the Donsker- Varadhan lower-bound (Donsker and Varadhan, 1975) and Fenchel duality, as in (Nguyen et al., 2010; Nowozin et al., 2016). Although the partition function is intractable in general, we propose a method to learn it an amortized fashion without introducing additional surrogate models, as done in variational inference (Kingma and Welling, 2014; Rezende et al., 2014) by Dai et al. (201 20)9a;b). The resulting maximum likelihood estimate, the KL Approximate Lower-boung Estimate ( (KALE), is then used a loss for training the base. When the class of energies is rich and smooth enough, we show that KALE leads to a meaningful criterion for measuring weak convergence of probabilities. Following recent work by Chu et al. (2020); Sanjabi et al. (2018), show that KALE possesses well defined gradients w.r.t. the parameters of the base, ensuring well-behaved training. We also provide convergence rates for the empirical estimator of KALE when the variational family is sufficiently well behaved, which may be of independent interest.
The main advantage of GEBMs becomes clear when sampling from these models: the posterior over the latents of the base distribution incorporates the learned energy. putting greater mass on regions in this latent space that lead to better quality samples. Sampling from the GEBM can thus be achieved by first sampling from the posterior distribution of the latents via MCMC in the low-d oow-dimeniional latent space, then mapping those latents to the input space using the implicit map of the base. This is in contrast standard GANs, where the latents of the base have fixed distribution. We focus on a class of samplers that exploit gradient information, and show that these samplers enjoy fast convergence properties by leveraging the recent work of Eberle et al. (2017). While there has been recent interest in using the discriminator to improve the quality of the generator during sampling (Azadi et al., 2019; Turner et al., 2019; Neklyudov et al., 2019; Grover et al., 2019; Tanaka, 2019; Wu et al., 2019b), our approach emerges naturally from the model we consider.
We begin in Section 2 by introducing the GEBM model. In Section 3, we describe the learning procedure using KALE, then derive a method for sampling from the learned model in Section 4. In Section 5 we discuss related work. Finally, experimental results are presented in Section with code available at ht https: / / g thub. com MMi chae 1 Arbel /Generalll ZedEBM.
2 GENERALIZED ENERGY-BASED MODELS
(a) Ground Truth
(b) GAN
(c) GEBM
(d) EBM
Figure 1:
Data generating distribution supported on a line and with higher density at the extremities. Models are learned using either GAN, GEBM, or EBM. More details are provided in Appendix G.3.
In this section, introduce generalized energy based models (GEBM), that combine the strengths of both energy-based models and implicit generative models, and admit the first of these as a special case. An energy-based model (EBM) is defined by set E of real valued functions called energies, where each E E & specifies probability density over the data space X' C Rdd up to normalizing constant,
Q (dxx) = exlr)=exp((Eii))) exp(-E(x)) AAddz, A-log (fexo(-E(a))d)))
While EBMs have been shown recently to be powerful models for representing complex high dimen- sional data distributions, they still unavoidably lead to a blurred model whenever data are concentrated on a diven-dimensional manifold. This is the case in Figure (a)), where the ground truth distribution is
2

---

Published as a conference paper at ICLR 2021
supported on I-D line and embedded in a 2-D space. The EBM in Figure ((d) learns to give higher den- sity to halo surrounding the data, and thus provides a blurred representation. That is a consequence of EBM having density defined over the whole space, and can result in blurred samples for image models.
An implicit generative model (IGM) is a family of probability distributions Go parametrized by a learnable generator function G: 2 + X' that maps latent samples from fixed latent distribution y to the data space X. The latent distribution 7) is required to have a density over the latent space Z and is often easy sample from. Thus, Sampling from G is simply achieved by first sampling from ry then applying G,
x
==), 2~17
(2)
GANs are popular instances of these models, and are trained adversarially (Goodfellow et al., 2014). When the latent space has a smaller dimension than the input space X. the IGM will be supported on a lower dimensional manifold of X, and thus will not possess a Lebesgue density X' (Bottou et al., 2017). IGMs are therefore good candidates for modelling low dimensional distributions. While GANs can accurately learn the oow-iimeniional support of the data, they have limited power for representing the distribution of mass on the support. This is illustrated in Figure 1(b).
A generalized energy-based model (GEBM) Q is defined by a combination of base G and an energy E defined over subset X' of Rd The base component can typically be chosen to be an IGM in (2). The generalized energy component can refine the mass on the support defined by the base. It belongs to a class& & of real valued functions defined on the input space X', and represents the negative log-density of sample from the GEBM with respect the base C.
(oss=-////p(p-/aaaa (esp((E(())(Gdd))))
Ac,E= log
(3)
where Ac, is the logarithm of the normalizing constant of the model w.r.t. G. Thus, a GEBM Q
re- re-weights samples from the base according to the un- an-normalized importance weights exp( - -(x)).
Using the latent structure of the base G, this importance weight can be pulled-back to the latent space to define a posterior latent distribution
v(z)== ( (exp=))xxpp(((aa -(([z))))Aaa))) A,,
(4)
Hence, the posterior latent can be used instead of the latent noise n for sampling from Q, as summarized by Proposition 1:
Proposition 1. Sampling Q requires sampling a latent 2 2 v (4) then applying the map G,
x x~Q
G=)(,
2ND.
In order to hold, Proposition does not need the generator to be invertible. We provide a proof in Appendix C. which relies on a characterization of probability distribution using generalized moments. We will see later in Section how equation (5) can be used to provide practical sampling algorithms from the GEBM. Next discuss the advantages of GEBMs.
Advantages of Generalized Energy Based Models. The GEBM defined by (3) can be related to exponential tilting (re- weighting) (Siegmund, 1976; Xic et al., 2016) of the base G. The important difference over classical EBMs is that the base G is allowed to change its support and shape in space. By learning the base G, GEBMs can accurately learn the diwdeeenniionl support of data, just like IGMs do. They also benefit from the flexibility of EBMs for representing densities using an energy E refine distribution of mass on the support defined by G as seen in Figure ((c).
Compared to EBMs, that put mass on the whole space by construction (positive density), GEBMs have the additional flexibility to concentrate the probability mass on a dimeeneiiional support learned by the base G, provided that the dimension of the latent space Z is smaller than the dimension of the ambient space X: see Figure I(c) vs Figure (d)) In the particular case when the dimension of Z is equal to the ambient dimension and G' is invertible, the base G becomes supported over the whole space X, and GEBM recover usual EBMs. The next proposition further shows that any EBM can be viewed as particular cases of GEBMs, as proved in Appendix C.I.
Proposition 2. Any EBM with energy E (as in (1)) can be expressed as a GEBM with base G given as normalizing flow with density exp(- -r(x)) and a generalized energy E(x)= = E(x) ({x)))(x)) (z). In this partic- ular case, the dimension of the latent is necessarily equal to the data dimension, i.e. dimí 2) = dim(
3

---

Published as a conference paper at ICLR 2021
Compared to IGMs, that rely on fixed pre-determined latent noise distribution 17. GEBMs offer the additional flexibility of learning a richer latent noise distribution. This is particularly useful when the data is multimodal. In IGMs, such a GANs, the latent noise 7) is usually unimodal thus requiring more sophisticated generator to distort a unimodal noise distribution into a distribution with multiple modes, shown by Cornish et al. (2020). Instead, GEBMs allow sample from posterior over the latent noise defined in (4). This posterior noise can be multimodal in latent space (by incorporating information from the energy) and thus can put more less mass specific regions of the manifold defined by the base G. This allows GEBMs to capture multimodality in data, provided the support of the base broad enough to subsume the data support Figure I(c). The base can be simpler, compared to GANs, as it doesn't 't need to distort the input noise too much to produce multimodal samples (see Figure 8 in Appendix G.4). This additional flexibility comes at no additional training cost compared to GANs. Indeed, GANs still require another model during training, the discriminator network, but do not use it for sampling. Instead, GEBMs avoid this waist since the base and energy can be trained jointly, with no other additional model, and then both are used for sampling.
3 LEARNING GEBMS
In this section we describe a general procedure for learning GEBMs. We decompose the learning procedure into two steps: energy learning step and base learning step. The overall learning procedure alternates between these two steps, as done in GAN training (Goodfellow et, 2014).
3.1 ENERGY LEARNING
When the base G is fixed, varying the energy E. leads family of models that all admit density exp(-F--A Ag, w.r.t. G. When the base G admits a density exp(-r) defined over the whole space, is possible to learn the energy E by maximizing the likelihood of the model -EE++rdPP -A.E However, in general G is supported usual notion of likelihood cannot be used. Instead, we introduce a generalized notion of likelihood which does require well defined density exp(-r) for G:
lower-dimensiona manifold so that is ill-defined and the
Definition (Generalized Likelihood). The expected G-log-likelihood under a target distribution P of a GEBM model Q with base G and energy E is defined as
Le,s(B):==) dess(E)=--EL(0 (((r)dP()) ( AG,E
To provide intuitions about the generalized likelihood in Definition 1, we start by discussing the particular case where KLPPI +oo. We then present the training method in the general case where IP and G might not share the same support, i.e. K L(P]GG) +00.
Special case of finite K LP G) When the Kullback- divergence between P and G is well defined, (6) corresponds to the Donsker- Donsker-Varadhan (DV) lower bound on the KL (Donsker and Varadhan, 1975), meaning that GL(P|| 2 Le (E) for all E. Moreover, the following proposition holds: Proposition 3. Assume that K KL(P|| [G) < to and E. If. in addition, E* maximizes (6), then: [T((P|]Q < (L
In addition, we have that K (T((PI)) = when F is the negative log-density ratio of w.r.t. G.
We refer to Appendix 1 for a proof. According to (7), the GEBM systematically improves over the IGM defined by G, with no further improvement possible the limit case when GP Hence as long as there is an error in mass on the common support of P and G. the GEBM improves over the base G.
Estimating the likelihood in the General setting. Definition can be used to learn maximum likelihood energy E by maximizing Le c (E) w.r.t. É even when the (L |G) is infinite and when P and G don't It necessarily share the same support. Such an optimal solution is well defined whenever the set of energies is suitably constrained. This is the case if the energies are parametrized by a compact set V with e Eg continuous over V. Estimating the likelihood is then achieved using i.i.d. samples (X,) ix (Y..) 1.M from IP and G (Tsuboi et al., 2009; Sugiyama et al., 2012; Liu et al., 2017):
1 N E(X,) llo
(r,(
N Σ
exp( (E(())
(8)
M
4

---

Published as a conference paper at ICLR 2021
In the context of mini-batch stochastic gradient methods, however, M typically ranges from to 1000, which can lead to a poor estimate for the log- partition function As, Moreover, (8) doesn' 't exploit estimates of Ag, E from previous gradient iterations. Instead, we propose estimator which introduces a variational parameter A E lK meant to estimate AG, E in an amortized fashion. The key idea is to exploit the convexity of the exponential which directly implies AA - -eepe-xp--r A Aç, . for any A E RR, with equality only when A A= Ac, Therefore, (6) admits lower-bound of the form
(p.c(E) (o(())2-(EEE)) -
where we introduced the functional FEGG for concision. Maximizing Fr.c (E+ over A recovers the likelihood (E). Moreover, jointly maximizing over E and A yields the maximum likelihood energy Ex and its corresponding log-partition function A =AG, F This optimization is well-suited for stochastic gradient methods using the following estimator Kanamori et al. (2011):
N
M
{ = [x
1
+ AA)
1 M
Σ m=1
(9)
3.2 BASE LEARNING
Unlike Section 3. varying the base G does not need to preserve the same support. Thus, generally not possible to use maximum likelihood methods for learning G. Instead, we propose to use the generalized likelihood (6) evaluated at the optimal energy E* as meaningful loss for learning G, and refer to it as the KL Approximate Lower-bound Estimate (KALE),
KALB(IP|
sup (P+ (E,A)EEXR
(10)
From Section 3.1, KALE(P (G) is always a lower bound on KL(P,G). The bound becomes tight whenever the negative log density of P w.r.t. G is well-defined and belongs to & (Appendix A). Moreover, Proposition 4 shows that KALE is reliable criterion for measuring convergence, and is consequence of (Zhang et al., 2017, Theorem B.1), with proof in Appendix C.2.1:
Proposition 4. Assume all energies in & are L-Lipschitz and that any continuous function can be well approximated by linear combinations of energies & (Assumptions (A) and (B) of Appendix C.2), then KALE(TP|]GS (G) 2 with equality only if PP and (P)((P)))")>) iff - P in distribution.
The universal approximation assumption holds in particular when contains feedforward networks. In fact networks with a single neuron are enough, as shown in (Zhang et al., 2017, Theorem 2.3). The Lipschitz assumption holds when additional regularization of the energy is enforced during training by methods such spectral normalization (Miyato et al., 2018) additional regularization ((b) on the energy Egg, such as the gradient penalty (Gulrajani et al., 2017) as done in Section 6.
Estimating KALE. According to Arora al. (2017), accurate finite sample estimates of divergences that result from an optimization procedures (such as in (10)) depend on the richness of the class E; and richer energy classes can result in slower convergence. Unlike divergences such Jensen-Shannon KL and the Wasserstein distance, which result from optimizing over a parameaaetticc and rich class of functions, KALE is restricted a class parametric energies E.g. Thus, (Arora et al., 2017, Theorem 3.1) applies, and guarantees good finite sample estimates, provided optimization is solved accurately, In Appendix B, we provide an analysis for the more general case where energies are not necessarily parametric but satisfy some further smoothness properties; we emphasize that our rates do not require the strong assumption that the density ratio bounded above and below as (Nguyen al., 2010).
Smoothness of KALE. Learning the base is achieved by minimizing ==0))=KALL KALE(PI)G)) over the set of parameters θ of the generator Go using first order methods (Duchi et al., 2011; Kingma and Ba, 2014; Arbel et al., 2019). This requires (C(6) to be smooth enough so that gradient methods converge to local minima and avoid instabilities during training (Chu et al., 2020). Ensuring smoothness of losses that result from an optimization procedure, as in (10), can be challenging. Results for the regularized Wasserstein are provided by Sanjabi et al. (2018), while more general losses are considered by Chu et al. (2020), albeit under stronger conditions than for our setting. Theorem 5 shows that when E, Go and their gradients are all Lipschitz then KC(B) is smooth enough. We provide a proof for Theorem 5 in Appendix C.2.1.
5

---

Published as a conference paper at ICLR 2021
Theorem 5. Under Assumptions (1) (III) of Appendix C.2, sub- gradient methods on K. converge to local optima. Moreover, K. is Lipschitz and differentiable for almost all 0 E θ with:
=x(0)=exp(-AGoo2
V V. TGL
(11)
Estimating the gradient in (11) is achieved by first optimizing over Ey and A using (9), with additional regular- ization I (v)). The resulting estimators
Algorithm 1 Training GEBM
1: Input F.N,M, nig ne Output Trained generator Gg and energy Es.
F. and Å are plugged (12) to esti- mate (I((0) using samples (Zm) 1:M 1:M from 7/ Unlike for learning the en- ergy E*, which benefits from using the amortized estimator of the log-pan partitiot function, we found that using the em- pirical log-partition for learning the base more stable. We summarize
3: Initialize 0, v and A.
for l 1= 1,... do
5:
for j I. do Sample (XnJIN I and J.NNINGG
6:
7:
9 f - FV. +A)
8:
À log M ΣM
_m=1exppp E (Ym)))
9:
9A 9 exp( (A-A)-1
10:
Update vi and A using 9v and gA-
the training procedure in Algorithm 1, which alternates between learning the energy and the base in a similar fashion adversarial training.
11:
end for
12:
Set F. c- Eng and À +- A.
13:
Update using VK.(0) from (12)
14:
end for
VK(0) exp( exp(-A*) -A)
M
V (ox. (G)))))))
(12)
M
SAMPLING FROM GEBMS
simple estimate of the empirical distribution of observations under the GEBM is via importance sam- pling (IS). This consists in first sampling multiple points from the base G, and then re- re-weighting the sam- ples according to the energy E. Although straightforward, this approach can lead to highly unreliable es- timates, well known problem in the Sequential Monte Carlo (SMC) literature which employs IS exten- sively (Doucet et al., 2001; Del Moral et al., 2006). Other methods such as rejection sampling are known to be inefficient in high dimensions Haugh (2017). Instead, we propose to sample from the posterior us- ing MCMC. Recall from (5) that sample x from Q is of the form x G(z) with z sampled from the pos- terior latent of (4) instead of the prior 1). While sampling from 17 is often straightforward (for instance if y is a Gaussian), sampling from v is generally harder, due to dependence of its density on complex func- tions E and G. is still possible use MCMC methods sample from v/, however, since have access to its density up to a normalizing constant (4). In particular, we are interested in methods that exploit the gradient of v. and consider two classes of samplers: Overdamped samplers and Kinetic samplers.
Overdamped samplers are obtained as a time-discretization of the Overdamped Langevin dynamics:
d2 = (o)g)(=)) - - V=E(G(z)))+
(13)
where Wi is a standard Brownian motion. The simplest sampler arising from (13) is the Unadjusted Langevin Algorithm (ULA):
Zk+1 = Zp A T.
Zo 1)~1)
where (Wx) are i.i.d. standard Gaussians and À is the step-size. For large k. Zz is an approximate sample from v (Raginsky al., 2017, Proposition 3.3). Hence, setting X GZ(()) for large enough l provides an approximate sample from the GEBM Q, summarized in Algorithm 2 of Appendix F.
Kinetic samplers arise from the Kinetic Langevin dynamics which introduce momentum variable:
dæ = vydt,
- -eee--oootttuuoooooooo (V 22yudw,
VE(G())))
(14)
with friction coefficienty> 0, inverse mass 2 momentum vector ii and standard Brownian motion
wt. When the mass u becomes negligible compared to the friction coefficient 7, i.c uy 2 xs 0, standard results show that (14) recovers the Overdamped dynamics (13). Discretization in time of (14)

---

Published as a conference paper at ICLR 2021
leads to Kinetic samplers similar to Hamiltonian Monte Carlo (Cheng et al., 2017; Sachs et 2017).
We consider a particular algorithm from Sachs et al. (2017) which we call Kinetic Langevin Algorithm (KLA) (see Algorithm 3 in Appendix F). Kinetic samplers were shown to better explore the modes of the invariant distribution v compared to Overdamped ones (see (Neal, 2010; Betancourt et al., 2017) 7) for empirical results and (Cheng et al., 2017) for theory), also confirmed empirically in Appendix I) for image generation tasks using GEBMs. Next, we provide the following convergence result:
Proposition 6. Assume that logn(z) is strongly concave and has Lipschitz gradient, that E, G and their gradients are all L-Lipschitz Set x; = (Zx), where zt is given by (14) and call P, the probability distribution of xt. Then Pt, converges to Q in the Wasserstein sense,
where and C are positive constants independent of with c= O(exp(-dim(()))
Proposition 6 is proved in Appendix C.1 using (Eberle et al., 2017, Corollary 2.6), and implies that (ox (2o converges at the same speed as (2))120 When the dimension q of Z is orders of magnitude smaller than the input space dimension d. the process (zt)120 converges faster than typical sampling methods on X, for which the exponent controlling the convergence rate of order (exp((-d))
5 RELATED WORK
Energy based models. Usually, energy based models are required to have density w.r.t. to a Lebesgue measure, and do not use learnalee base measure; in other words, models are supported on the whole space. Various methods have been proposed in the literature to learn EBMs. Contrastive Divergence (Hinton, 2002) approximates the gradient of the log-likelihone by sampling from the energy model with MCMC. More recently, (Belanger and McCallum, 2016; Xie et al., 2016; 2017; 2018c; 2019; Tu and Gimpel, 2018; Du and Mordatch, 2019; Deng et al., 2020) extend the idea using more sophisticated models and MCMC sampling strategies that lead to higher quality estimators. Score Matching (Hyvärinen, 2005) calculates an alternative objective (the score) to the log-likelihood which is independent of the partition function, and was recently used in the context non- non-paraettric energy functions to provide estimators of the energy that are provably consistent (Sriperumbudur et al., 2017; Sutherland et al., 2018 88; Arbel and Gretton, 2018; Wenliang et al., 2019). In Noise Contr-Cootrautive Estimation (Gutmann and Hyvärinen, 2012), a classifier is trained to distinguish between samples from fixed proposal distribution and the target P. This provides an estimate for the density ratio between the optimal energy model and the proposal distribution. In similar spirit, Cranmer et al. (2016) uses classifier to learn likelihood ratios. Conversely, Grathwohl et al. (2020) interprets the logits of a classifier as an energy model obtained after marginalization over the classes. The resulting model is then trained using Contrastive Divergence. In more recent work, Dai et al. (2019a;b) exploit a dual formulation of the logarithm of the partition function as supremum over the set of all probability distributions of some functional objective. Yu et al. (2020) explore methods for using general f-divergences, such as -hasnon,, to train EBMs.
Generative Adversarial Networks. Recent work proposes using the discriminator of a trained GAN to improve the generator quality. Rejection sampling (Azadi et al., 2019) and Metropolis- Hastings correction (Turner et al., 2019; Neklyudov et al., 2019) perform sampling directly on the high-dimensional input space without using gradient information provided by the discriminator. Moreover, the data distribution is assumed to admit a density w.r.t. the generator. Ding et al. (2019) perform sampling on the feature space of some auxiliary pre-trained network; while Lawson et al. (2019) treat the sampling procedure as a model on its own, learned by maximizing the ELBO. In our case, no auxiliary model is needed. In the present work, sampling doesn't interfere with training, in contrast to recently considered methods to optimize over the latent space during training Wu et al. (2019b;a). In Tanaka (2019), the discriminator viewed as an optimal transport map between the generator and the data distribution and is used to compute optimized samples from latent space. This is in contrast to the diffusion-basse sampling that we consider. In (Xie et al., 20 8b;a), two independent models, full support EBM and generator network, are trained cooperatively using MCMC. By contrast, in the present work, the energy and base are part of the same model, and the model support is owerr-iimensionll than the target space X. While we do not address the mode collapse problem, Xu et al. (2018); Nguyen et al. (2017) showed that Ka-based losses are resilient to it thanks to the zero-avoiding property of the KI., good sign for KALE which is derived from KI. by Fenchel duality.
7

---

Published as a conference paper at ICLR 2021
The closest related approach appears in a study concurrent to the present work (Che al., 2020), where the authors propose to use Langevin dynamics on the latent space of a GAN generator, but with different discriminator to ours (derived from the -hasnnnn divergence or Wasserstein-basse divergence). Our theory results showing the existence of the loss gradient (Theorem 5), establishing weak convergence of distributions under KALE (Proposition 4), and demonstrating consistency of the KALE estimator (Appendix B) should transfer to the JS and Wasserstein criteria used in that work. Subsequent the present work, an alternative approach has been recently proposed, based on normalising flows, to learn both the low-dimensions support (Brehmer and Cranmer, 2020). This approach maximises the explicit likelihood of data projection onto a learned manifold, and may be considered complementary to our approach.
support of the data and the density on this
6 EXPERIMENTS
6.1 IMAGE GENERATION.
Experimental setting, We train GEBM on unsupervised image generation tasks, and compare the quality of generated samples with other methods using the FID score (Heusel et., 2017) computed on x 10* generated samples. We consider CIFAR-10 (Krizhevsky, LSUN at. 2015), CelebA (Liu al. 2015) and ImageNet (Russakovsky et al., 2014) all downsampled to 32x32 resolution to reduce computational cost. We consider two network architectures for each of the base and energy, a smaller one (SNGAN ConvNet) and larger one (SNGAN ResNet), both of which are from Miyato et al. (2018). For the base we used the SNGAN generator networks from Miyato et al. (2018) with 100-dimensional Gaussian for the latent noise For the energy we used the SNGAN discriminator networks from Miyato et al. (2018). (Details of the networks in Appendix G.1).
Figure 2: Samples at different
iterations of the MCMC chain of
Algorithm 3 (left to right).
We train the models for 150000 generator iterations using Algorithm 1. After training completed. we rescale the energy by 3 = 100 to get a colder version of the GEBM and sample from it using either Algorithm 2 (ULAA) ) or Algorithm 3 (KIAA) with parameters (7 100,u -000 1). This colder temperature leads to an improved FID score, and needs relatively few MCMC iterations, as shown in Figure 6 of Appendix D. Sampler convergence to visually plausible modes at low tempteratures is demonstrated in Figure 2. We perform 1000 MCMC iterations with initial step-size of x= 10 1 decreased by 10 every 200 iterations. As a baseline we consider samples generated from the base of the GEBM only (without using information from the energy) and call this KALE-GAN. More details are given in Appendix G.
Table shows that GEBM outperforms both KALE and standard GANs when using the same networks for the base/generator and energy/critic. Moreover, KALE-GAN matches the performance of standard GAN (with Jensen-Shannon critic), showing that the improvement of GEBM cannot be explained by the switch from largely due to incorporating the energy function into the model, and sampling using Algorithm 3.
Results:
a KALE-based critic. Rather, the improvement is
This finding experimentally validates our claim that incorporating the energy improves the model, and that all else being equal, GEBM outperforms a GAN with the same generator and critic architecture. Indeed, if the critic is not zero at convergence, then by definition it contains information on the remaining mismatch between the generator (base) and data mass, which the GEBM incorporates, but the GAN does not. The GEBM also outperforms an EBM even when the latter was trained using a larger network (ResNet) with supervision (S) on ImageNet, which is an easier task ( (Chen et al. (2019)). More comparisons on Cifar 10 and ImageNet are provided in Table 4 of Appendix D.
SNGAN (ConvNet)
SNGAN (ResNet)
GEBM
KALE-GAN
GAN
GEBM
KALE-GAN
GAN
EBM
Cifar10 23.02
32.03
29.9
19.31
20.19
21.7
38.2
ImageNet
13.94
19.37
20.66
20.33
21.00
20.50
14.31 (S)
Table 1: FID scores for two versions of SNGAN from (Miyato et al., 2018) on Cifar10 and ImageNet. GEBM: training using Algorithm and sampling using Algorithm 3. KALE-GAN Only the base of GEBM is retained for sampling. GAN: training as in (Miyato et al., 2018) with ş= 128 for the latent dimension as it worked best. EBM: results from Du and Mordatch (2019) with supervised training on (S)..
8

---

Published as a conference paper at ICLR 2021
Table 2 shows different sampling meth- ods using the same trained networks (generator and critic), with KALE-GAN as baseline. All energy-exploiting methods outperform the unmodified KALE-GAN with the same architecture. That said, our method (both ULA and KLA) outperforms both (IHM) (Turner et 201, 2019) 19) and (DOT) (Tanaka, 2019), which both use the energy information. information.
Cifarl0
LSUN
CelebA
ImageNet
KALE-GAN
32.03
21.67
6.91
19.37
IHM
30.47
20.63
6.39
18.15
DOT
26.35
20.41
5.93
16,21
GEBM (ULA)
23.02
16.23
5.21
14.00
GEBM (KLA)
24.29
15.25
5.38
13.94
Table 2: FID scores for different sampling methods using the same trained SNGAN (ConvNet): KALE-GAN a baseline w/o critic
In Table 2, KLA was used in the high friction regime y 100 and thus behaves like ULA. This allows to obtain sharper samples concentrated around the modes of the GEBM thus improving the FID score. If, instead, the goal is to encourage more exploration of the modes of the GEBM, then KLA with a smaller y is better alternative than ULA, as the former can explore multiple modes/images within the same MCMC chain, unlike (ULAA): see Figures 3 to 5 of Appendix D. Moving from one mode to another results an increased FID score while between modes, however, which can be avoided by decreasing A.
6.2 DENSITY ESTIMATION
Motivation. We next consider the particular setting where the likelihood of the model is well- and admits a closed form expression. This is intended principally as a sanity check that our proposed training method in Algorithm I succeeds in learning maximum likelihood solutions. Outside of this setting, closed form expressions of the normalizing constant are not available for generic GEBMs. While this is not an issue (since the proposed method doesn't It require closed form expression for the normalizing constant), in this experiment only, we want to have access to closed form expressions, as they enable direct comparison with other density estimation methods.
Experimental setting. To have a closed-form likelihood, we consider the case where the dimension of the latent space is equal to data dimteeiimnsion, and choose the base G of the GEBM to be Real NVP (Ding et al. (2019)) ) with density exp( -r())) and energy F((x) = (xxx) r(x). Thus, in this particular case, the GEBM has a well defined likelihood over the whole space, and are precisely in the setting of Proposition 2, which shows that this GEBM is equal to an EBM with density proportional to exp(-hi We further require the EBM to be second Real NVP so that its density has closed form expression. We consider 5 UCI datasets (Dheeru and Taniskidou, 2017) for which we use the same pre-processing as in (Wenliang al., 2019). For comparison, we train the EBM by direct maximum likelihood (ML) and contrastive divergence (CD). To train the GEBM, we use Algorithm 1, which doesn't 't directly exploit the closed-form expression the likelihood (unlike direct ML). We thus use either (8) (KALE-DV) or (9) (KALE-F) to estimate the normalizing constant. More details are given in Appendix G.2.
Results. Table reports the Negative Likeliiod (NLL) evaluated on the test set and corresponding the best performance on the validation set. Training the GEBM using Algorithm leads to comparable performance to (CD) and (ML). As shown in Figure of Appendix E, (KALE-DV) and (KALE- F) maintain small error gap between the training and test NLL and, discussed in Section 3.1 and Appendix F. (KALE-F) leads to more accurate estimates of the log- part-titiionn function, with a relative error of order 0.1% compared to 10% for (KALE-DV).
RedWine d= N1,N N 10
Whitewine d= 11,N~~10 10
Parkinsons d= 1==
Hepmass d d=22,N 100
Miniboone d= 43,N
NVP MI NVP CD
11.98
13.05
14.5
24.89
42.28
11.88
13.01
14.06
22. 89
39.36
NVP w KALE (DV)
11.6
12.77
13.26
26.56
46.48
NVP w KALE (F)
11.19
12.66
13.26
24.66
38.35
Table 3: UCI datasets: Negative lokeliold computed on the test and corresponding the best performance on the validation set. Best method in boldface.
7 ACKNOWLEDGMENTS
We thank Mihaela Rosca for insightful discussions and Song Liu, Bo Dai and Hanjun Dai for pointing us to important related work.

---

Published as a conference paper at ICLR 2021
REFERENCES
Arbel, M. and Gretton, A. (2018). Kernel Conditional Exponential Family. In International Conference on Artificial Intelligence and Statistics, pages 1337-1346.
Arbel, M., Gretton, A., Li, W., and Montufar, G. (2019). Kernelized Wasserstein Natural Gradient.
Arbel, M., Sutherland, D., Binkowski, M., and Gretton, A. (2018). On gradient regularizers for mmd gans. In Advances in Neural Information Processing Systems 31. Curran Associates, Inc.
Arjovsky, M., Chintala, S., and Bottou, L. (2017). Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learming Research, International Convention Centre, Sydney, Australia. PMLR.
Arora, S., Ge, R., Liang. Y., Ma, T., and Zhang, Y. (2017). Generalization and equilibrium generative adversarial nets (GANs). In Proceedings the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 224-232. PMLR.
Azadi, S., Olsson, C., Darrell, T., Goodfellow, I., and Odena, A. (2019). Discriminator rejection sampling. International Conference on Learning Representations.
Belanger, D. and McCallum, A. (2016). Structured prediction energy networks. In International Conference on Machine Learning, pages 983-992.
Betancourt, M., Byrne, S., Livingstone, S., and Girolami. M. (2017). The geometric foundations of Hamiltonian Monte Carlo. Bernoulli, 23(4A):2257-2299
Binkowski, M., Sutherland, J., Arbel, M., and Gretton, A. (2018). Demystifying MMD GANs. In International Conference Learning Representations.
Bottou, L., Arjovsky, M., Lopez-Paz, D., and Oquab, M. (2017). Geometrical insights for implicit generative modeling. In Braverman Readings in Machine Learning.
Brehmer, and Cranmer, (2020). Flows for simultaneous manifold learning and density estimation. arXiv preprint 2rXiv:2003.139133
Brock, A., Donahue, J., and Simonyan, (2018). Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: /8Xi:: 1096.
Che, T., Zhang, R., Sohl-Dickstein, J., Larochelle, H., Paull, L., Cao, Y., and Bengio, Y. (2020). Your GAN is secretly an energy-based model and you should use discriminator driven latent sampling.
Chen, T., Zhai, X., Ritter, M., Lucic, M., and Houlsby, N. (2019). Self-supervised gans via auxiliary rotation loss. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 12154-12163.
Cheng, X., Chatterji, N. S., Bartlett, P. L., and Jordan, M. I. (2017). Underdamped langevin meme: A non-asymptotic analysis. arXiv preprint 170.:1777
Chu, C., Minami, K., and Fukumizu, K. (2020). Smoothness and stability in gans. In International Conference Learning Representations.
Cornish, R., Caterini, L.,, Deligiannidis, G., and Doucet, A. (2020). Relaxing bijectivity constraints with continuously indexed normalising flows.
Cranmer, K., Pavez, and Louppe, G. (2016). Approximating likelihood ratios with calibrated discriminative classifiers.
Dai, B., Dai, H., Gretton, A., Song, L.., Schuurmans, D., and He, N. (2019a). Kernel exponential family estimation via doubly dual embedding. In Proceedings of Machine Learning Research, volume 89 of Proceedings of Machine Learning Research, pages 2321-2330. PMLR.
Dai, B., Liu, Z., Dai, H., He, N., Gretton, A., Song, L., and Schuurmans, D. (2019b). Exponential Faminl Estimation via Adversarial Dynamies Embedding. arXiv: /2083 (cs, staf]. arXiv: 1904. 12083.
10

---

Published as a conference paper at ICLR 2021
Davis, D. and Drusvyatskiy, D. (2018). Stochastic subgradient method converges at the rate $0(K4-1/44) on weakly convex functions. arXiv:/802.02980 [cs, math).
Del Moral, P., Doucet, A., and Jasra, A. (2006). Sequential monte carlo samplers. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 683))41114333
Deng, Y., Bakhtin, A., Ott, M., Szlam, A., and Ranzato, M. (2020). Residual energy- based models for text generation. arXiv preprint arXiv:2004.1177
Dheeru, D. and Taniskidou, E. K. (2017). Uci machine learning repository.
Ding, X., Wang. Z. J., and Welch, J. (2019). Subsampling Generative Adversarial Networks: Density Ratio Estimation in Feature Space with Softplus Loss.
Dinh, L., Sohl- Sohl-Dickstein, J., and Bengio, S. (2016). Density estimation using real nvp.
Donahue, J. and Simonyan, K. (2019). Large Scale Adversarial Representation Learning. arXiv:1907.02544 [cs, stat]. arXiv: 1907.02544.
Donsker, M. D. and Varadhan, R. (1975). Asymptotic evaluation of cer- tain markov process expectations for large time, i. 28(1):1-47. _eprint: http:///wieeilllaayllllleeoooo// /ttp:////ppppoo..0/o/aaaaaaa1.o0 10020cpa..1100280000
Doucet, A., Freitas, N. d., and Gordon, N. (2001). Sequential Monte Carlo Methods in Practice. Information Science and Statistics. Springer- New York.
Du, Y. and Mordatch, I. (2019). Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems 32. pages 3608-3618. Curran Associates, Inc.
Duchi, J., Hazan, E., and Singer, Y. (201))). Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, 121111):121121599
Eberle, A., Guillin, A., and Zimmer, R. (2017). Couplings and quantitative contraction rates for Langevin dynamics. The Annals of Probability.
Ekeland, and Témam, R. (1999). Corvex Analysis and Variational Problems. Classies in Applied Mathematics. Society for Industrial and Applied Mathematics.
Feydy, J., Séjourné, T., Vialard, F.-X., Amari, S.-i., Trouvé, A., and Peyré, (2019). Interpolating between optimal transport and mmd using sinkhorn divergences. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2681 2681-2690.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances Neural Information Processing Systems 27, pages 2672-2688 Curran Associates, Inc.
Grathwohl, W., Wang. K.-C., Jacobsen, J..H., Duvenaud, D.,, Norouzi, M., and Swersky, K. (2020). Your classifier is secretly an energy based model and you should treat it like one.
Grover, A., Song, Kapoor, A., Tran, K., Agarwal, A., Horvitz, E. J., and Ermon, S. (2019). Bias correction of learned generative models using likelihood- -free importance weighting. In Advances in Neural Information Processing Systems 32. Curran Associates, Inc
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. (2017). Improved training of wasserstein gans. In Proceedings the s1 International Conference Neural Information Processing Systems, Red Hook, NY, USA. Curran Associates Inc
Gutmann, M. U. and Hyvärinen, A. (2012). Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. The Journal of Machine Learning Research, 13(mull::37-36:
Haugh, M. (2017). Meme and bayesian modeling. IEOR E4703 Monte-Carld Simulation, Columbia University.
11

---

Published as a conference paper at ICLR 2021
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler. B., and Hochreiter, S. (2017). Gans trained by two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems 30, pages 6626-6637. Curran Associates, Inc.
Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 1478):1771-1800.
Ho, J. and Ermon, S. (2016). Generative adversarial imitation learning. In Advances neural information processing systems, pages 4565-4573.
Hyvärinen, A. (2005). Estimation of Non-Noomallzed Statistical Models by Score Matching. The Journal Machine Learning Research, 6:695-709.
Kanamori, T., Suzuki, T., and Sugiyama, M. (2011). f-divergence estimation and two-sample homogeneity test under semiparametrie density-ratio models. IEEE Transactions Information Theory, 58(2):708-72)
Kingma, D. P. and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs]. arXiv: 1412.6980.
Kingma, D. P. and Welling, M. (2014). Auto-encoding variational Bayes. ICLR.
Klenke, A. (2008). Probability Theory: Comprehensive Course, World Publishing Corporation.
Kodali, N., Abernethy, J., Hays, J., and Kira, Z. (2017). On Convergence and Stability of GANs. arXiv:1705.07215 [cs]. arXiv: 1705.07215.
Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Technical report, University of Toronto.
Lawson, J., Tucker, G., Dai, B., and Ranganath, R. (2019). Energy-inspired models: Learning with sampler-induced distributions. Advances in Neural Information Processing Systems 32, pages 8501-8513. Curran Associates, Inc.
LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., and Huang, F.-J. (2006). Predicting Structured Data, chapter A Tutorial on Energy-Based Learning, MIT Press.
Li, C.-L., Chang, W.-C., Cheng, Y., Yang. Y., and Poezos, B. (2017). Mmd gan: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems 30, pages 2203-2213. Curran Associates, Inc.
Liu, S., Bousquet, O., and Chaudhuri, K. (2017). Approximation and Convergence Properties of Generative Adversarial Learning.
Liu, Z., Luo, P., Wang, X., and Tang, X. (2015). Deep learning face attributes in the wild.
Milgrom, and Segal, I. (2002). Envelope Theorems for Arbitrary Choice Sets. Econometrica, 70.
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. (2018). Spectral normalization for generative adversarial networks. In International Conference on Learning Representations.
Nagarajan, V. and Kolter, J. Z. (2017). Gradient descent gan optimization is locally stable.
Neal, R. M. (2010). Meme using hamiltonian dynamics. Handbook of Markov Chain Monte Carlo.
Neklyudov, K., Egorov, E., and Vetrov, D. (2019). The implicit metropolis-hastingg algorithm.
Nguyen, T., Le, T., Vu, H., and Phung. D. (2017). Dual discriminator generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2670-2680.
Nguyen, X., Wainwright, M. J., and Jordan, M. I. (2010). Estimating divergence functionals and the likelihood ratio by convex risk minimization. JEEE Transactions on Information Theory, 56(1 16811)5587-5881
12

---

Published as a conference paper at ICLR 2021
Nowozin, S., Cseke, B., and Tomioka, R. (2016). f-gan: Training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing Systems 29, pages 271-279. Curran Associates, Inc.
Oord, A. v. d., Kalchbrenner, N., and Kavukcuoglu, K. (2016). Pixel recurrent neural networks. arXiv preprini arXiv: 1601 (6600.00759)
Ostrovski, G., Dabney, W., and Munos, R. (2018). Autoregressive quantile networks for generative modeling. arXiv preprint arXiv:8806.05555.
Papamakarios, G., Pavlakou, T., and Murray, (2017). Masked autoregressive flow for density estimation. NIPS.
Radford, A., Metz, L.,, and Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 1rXiv:151.006
Raginsky, M., Rakhlin, A., and Telgarsky, M. (2017). Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis.
Retherford, J. R. (1978). Review: J. diestel and j. uhl. jr., vector measures. Bull. Amer. Math. Soc., 84(4):681-685.
Rezende, D. and Mohamed, S. (2015). Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML'CMMII 15, pages 1530-1538. JMLR.org.
Rezende, J.J., Mohamed, S., and Wierstra, D. (2014). Stochastic backpropagation and approximate inference deep generative models. In ICML., pages 1278-1286.
Rockafellar, R. T. (1970). Convex analysis. Princeton Mathematical Series. Princeton University Press, Princeton, J.
Russakovsky, O., Deng. J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, 7., Karpathy, A., Khosla,
A., Bernstein, M., Berg, A. C., and Fei-Fei, L. (2014). ImageNet Large Scale Visual Recognition Challenge. 10% [cs]. arXiv: 1409.0575.
Sachs, M., Leimkuhler, B., and Danos, V. (2017). Langevin Dynamics with Variable Coefficients and Nonconservative Forces: From Stationary States to Numerical Methods. Entropy, 19.
Sanjabi, M., Ba, J., Razaviyayn, M., and Lee, J. D. (2018). On the convergence and robustness of training gans with regularized optimal transport. In Advances in Neural Information Processing Systems 31, pages 7091-7101. Curran Associates, Inc.
Siegmund, D. (1976). Importance sampling in the monte carlo study of sequential tests. The Annals Statistics pages 673-684.
Simon-Gabriel, C.-J. and Scholkopf, B. (2018). Kernel distribution embeddings: Universal kernels, characteristic kernels and kernel metrics distributions. Journal of Machine Learning Research, 19(44):1-29.
Simsekli, U., Zhu, I.,, Teh, Y. W., and Gurbuzbalaban, M. (2020). Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noise. arXiv:2002.055 05668502..6655.. [cs, stat]. arXiv: 2002.05685.
Sriperumbudur, B., Fukumizu, K., Kumar, R., Gretton, A., and Hyvärinen, A. (2017). Density estimation in infinite dimensional exponential families. Journal of Machine Learning Research.
Sugiyama, M., Suzuki, T., and Kanamori, T. (2012). Density ratio estimation in machine learning. Cambridge University Press.
Sutherland, D., Strathmann, H., Arbel, M., and Gretton, A. (2018). Efficient and principled score estimation with Nystrom kernel exponential families. International Conference on Artificial Intelligence and Statistics, pages 652-660.
13

---

Published as a conference paper at ICLR 2021
Tanaka, A. (2019). Discriminator optimal transport. In Advances in Neural Information Processing Systems 32. Curran Associates, Inc.
Thekumparampil, K. K., Jain, P., Netrapalli, P., and Oh, S. (2019). Efficient algorithms for smooth minimax optimization. In Advances in Neural Information Processing Systems 32, pages 12680-12699 12691. Curran Associates, Inc.
Thiry, L., Arbel, M., Belilovsky, E., and Oyallon, E. (2021). The unreasonable effectiveness of patches in deep convolutional kernels methods. International Conference on Learning Representations.
Tsuboi, Y., Kashima, H., Hido, S., Bickel, S., and Sugiyama, M. (2009). Direct density ratio estimation for large-scale covariate shift adaptation. Journal of Information Processing, 17:138-155.
Tu, L.. and Gimpel, K. (2018). Learning approximate inference networks for structured prediction. arXiv preprint arXiv: 1803.03376.
Turner, R., Hung, J., Frank, E.,, Saatchi, Y., and Yosinski, J. (2019). Metropolis- Hastinggs generative adversarial networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6345-6353, Long Beach, California, USA. PMLR.
Villani, C. (2009). Optimal transport: Old and new. Technical report.
Wenliang, L.,, Sutherland, D., Strathmann, H., and Gretton, A. (2019). Learning deep kernels for exponential family densities. In International Conference on Machine Learning, pages 6737-6746.
Wu, Y., Donahue, J., Balduzzi, D., Simonyan, K., and Lillicrap. T. (2019a). LOGAN: Latent Optimisation for Generative Adversarial Networks. /9/1::////.0000 [cs, star]. arXiv: 1912.00953.
Wu, Y., Rosca, M., and Lillicrap, T. (2019b). Deep compressed sensing. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6850-6860, Long Beach, California, USA. PMLR.
J., Lu, Y., Gao, R., and Wu, Y. N. (2018a) 8a). Cooperative learning of energy- baaeyy model and latent variable model via meme teaching. In AAAI, volume 1, page 7.
Xie, J., Lu, Y., Gao, R., Zhu, S.-C., and Wu, Y. (2018b). Cooperative training of descriptor and generator networks. IEEE transactions on pattern analysis and machine intelligence, 42(1):27-45
Xie, J., Lu, Y., Zhu, S.-C., and Wu, Y. (2016). A theory of generative convnet. In International Conference on Machine Learning, pages 2635-2644.
Xie, J., Zheng, Z., Gao, R., Wang, W., Zhu, S.-C., and Nian Wu, Y. (2018c). Learning descriptor networks for 3d shape synthesis and analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 0629-8638.
Xie, J., Zhu, S.-C., and Nian Wu, (2017). Synthesizing dynamic patterns by spatial-temporal generative convnet. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 7093-7101.
Xie, J.. Zhu, S.-C., and Wu, Y. (2019). Learning energy- energy-based spatial-temporal generative convnets for dynamic patterns. IEEE transactions on pattern analysis and machine intelligence.
Xu, K., Du, C., Li, C., Zhu, J., and Zhang, B. (2018). Learning implicit generative models by teaching density estimators. arXiv preprint 1807.:1880.0336
Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., and Xiao, (2015). L.SUN: Construction of large-scale image dataset using deep learning with humans in the loop.
Yu, L., Song, Y., Song, J., and Ermon, S. (2020). Training deep energy-based models with f-divergence minimization.
Zenke, F., Poole, B., and Ganguli, S. (2017). Continual learning through synaptic intelligence. Proceedings of machine learning research, 70:3987.
Zhang. P., Liu, Q., Zhou, D., Xu, T., and He, X. (2017). On the Anseriiinnation-Geeeraiiizza Tradeoff in GANs. arX [cs, stat]. arXiv: 1711.02771.
14

---

Published as a conference paper at ICLR 2021
A KL APPROXIMATE LOWER-BOUND ESTIMATE
We discuss the relation between KALE (10) and the Kullback- Leibler divergence via Fenchel duality. Recall that distribution IP is said to admit density w.r.t. G if there exists real-valued measurable function To that is integrable w.r.t. C and satisfies dl' = rodG. Such a density is also called the Radon-Nikodym derivative of IP w.r.t. G. In this case, we have:
- rolog(ro)dG.
(I((P]))= =
(15)
Nguyen et al. (2010); Nowozin et al. (2016) derived variational formulation for the KL using Fenchel duality. By the duality theorem (Rockafellar, 1970), the convex and lower cemi-continuous function (:u+>ulog(u) that appears in (15) can be expressed as the supremum of concave function:
((u) supwww - (v))
The function (* is called the Fenchel dual and is defined as sup.==uup.w-i( By convention, the value of the objective is set to -oo whenever u is outside of the domain of definition of S*. When ((u) ulog(u), the Fenchel dual (*) admits a closed form expression of the form (*())= eex((x- 1). Using the expression of Ç in terms of its Fenchel dual Ç*, it is possible to express GG. as the supremum of the variational objective (16) over all measurable functions
F(h): hddp- (()==-hhdP-) exp((h))G++1
(16)
Nguyen al. (2010) provided the variational formulation for the reverse KL using a different choice
for Ç: (C(()==li u)). We refer to (Nowozin et al., 2016) for general f f-divergences. Choosing
smaller set of functions H the variational objective (16) will lead to lower bound on the KI. This is the KL Approximate Lower-bound Estimate (KALE):
KALB(PIIG) = sup.F(h)
(17)
NCH
In general, (C( > IA) The bound is tight whenever the negative log-density hy - llgro belongs to H;; however, we do not require To to be well-defined in general. Equation (17) has the advantage that can be estimated using samples from and G. Given i.i.d. samples (X1...,,NN) and (Y,,..,, YM) from and G. we denote by Pe and & the corresponding empirical distributions. A simple approach to estimate KALE(P)) (G) is by optimizing the penalized objective
an M-estimator. This is achieved
argmaxxF()) A -(h)- (1)) BEH
argmax
(18)
where iis an empirical version of F and (h) is a penalty term that prevents overfitting due to finite samples. The penalty (h) acts as regularizer favoring smoother solutions while the parameter determines the strength of the smoothing and is chosen to decrease as the sample size N and M increase. The M- -estimatoor of KALE(PP|(C [G) is obtained simply by plugging in into the empirical objective F(h):
KALE(PIG)==F0M
(19)
We defer the consistency analysis of (19) to Appendix B where provide convergence rates in a setting where the set of functions H is a Reproducing Kernel Hilbert Space and under weaker assumptions that were not covered by the framework of Nguyen al. (2010).
B CONVERGENCE RATES OF KALE
In this section, we provide convergence rate for the estimator in (19) when H is an RKHS. The theory remains the same whether 7. contains constants or not. With this choice, the Representer Theorem allows to reduce the potentially dimenssonll one. We further restrict ourselves to the well-specified case where the density To of P w.r.t. G is well-defined and belongs to H. so that K KLE matches the KI. While Nguyen et al.
optimization problem in (18) convex
15

---

Published as a conference paper at ICLR 2021
(2010) (Theorem 3) provides convergence rate of 1/VN for related M-estimator, this requires the density To to lower-bounded by as well as (generally) upper-bounded. This can be quite restrictive if, for instance, Tn is the density ratio of two gaussians. In Theorem 7, provide similar convergence rate for the estimator defined in (19) without requiring ro to be bounded. We start by briefty introducing some notations, the working assumptions and the statement of the convergence result in Appendix B. and provide the proofs in Appendix B.2.
B.1 STATEMENT OF THE RESULT
We recall that an RKHS H. of functions defined on domain X' C Rd and with kernel k: is Hilbert space with dot product (.,.), such that y y k(x,)) belongs to H for any x X, and
k(z.y)= - (k(..))..(..))) Vx.yEX.
Any function h H satisfies the reproducing property ((x)= for any x EX.
Recall that KALE(PI|G) is obtained an optimization problem
(ALB(PI(G)=sup> - hc#
(20)
where F is given by:
Since the negative log density ratio hg is assumed to belong to H. this directly implies that the supremum of JF is achieved at tg and F(ho) KALE(IP]))) We are interested in estimating KALE(PP| (G) using the empirical distributions P and G,
N
8 OX
n=1
where (Xn)isn<n n and (Nn) are i..d samples from P and G. For this purpose introduce the empirical objective functional,
f(M)==- hoi==-faeefaae f (xpe--/xp()))
The proposed estimator is obtained by solving a regularized empiriaal problem,
sup ((k) - *EW
1 111/3
(21)
with corresponding population version,
sup F(h)
2 |11l1
(22)
kEH
Finally, we introduce (h,,) and 1"(h,)):
D(h,6) = bexp(-h))dG- fode,
r(h,0)= - (1xx)xxe
The empirical versions D(h,8) and (h,,5) are denoted ((h,6) and (h, Later, we will show that D(h,6) D(h,6) are in fact the gradients of F(h) and F(h) along the direction d.
We state the working assumptions:
The supremum of F over H is attained at hg-
16

---

Published as a conference paper at ICLR 2021
i The following quantities are finite for some positive
v M(a,z) di(()),
/ (M(e,r)exp(((Iboll k(z,x)) dG(x),
k(x,z) dG(x).
(ii) For any e H, (h,,)) = for all 5 then h Theorem 7. Fix any 1 1i > 0. Under Assumptions (i) to (iii), and provided that 1 with probability at least 1 - 2ny that
JN holds
(n.ho))
[(h))
VN
for constant M' (n) that depends only 17 and ho-
The assumptions in Theorem essentially state that the kernel associated the RKHS H needs to satisfy some integrability requirements. That is to guarantee that the gradient 8 (o) and its empirical version are well-defined and continuous. In addition, the optimality condition (F)(h) = is assumed characterize the global solution hg. This will be the case if the kernel characteristic Himon-Gabrid and Scholkopf (201 8). The proof of Theorem 7, in Appendix B.2, takes advantage of the Hilbert structure of the set H. the convexity of the functional J and the optimality condition VF(hh- Xh of the regularized problem, all which turn out to be sufficient for controlling the error of (19).
B.2 PROOFS
We state now the proof of Theorem with subsequent lemmas and propositions.
Proof of Theorem 7. We begin with the following inequalities:
2 (11A 11 (v
The first inequality is by definition of h while the second is obtained by concavity of F. For simplicity we write B -lh- ho/ and C= C ((ho))l Using Cauchy-Schware and triangular inequalities, is easy to see that
2 ( hboll -(ho)) SCB.
Moreover, by triangular inequality, it holds that
B h +
Lemma 1l ensures that -(A)=
converges to 0 as \ - Furthermore, by Proposition 12, Now choosing ア and applying Chebychev inequality Lemma 8, it follows that for any > rj > 0, we have with probability greater than 1 2ny that both
have | < D where -
C(1lholln) VN
C(1holl.n)
D(A)
e
VN
where C(IIboll,0) is defined in Lemma 8. This allows conclude that for any 1j 0, it holds with probability at least 1 -2) that
(m,haa where M((n,ho) depends only on v) and ho-
We proceed using the following lemma, which provides an expression for D(h,8) and D(h,8) along with probabilistic bound:
17

---

Published as a conference paper at ICLR 2021
Lemma's 8. Under Assumptions (i) and for, any h E H such that |[b] S [Aaal +G, there exists (hh) in H satisfyyng
-0h.,))(0,D(()),
for any h EH there exists D(h) satisfying
(6,,0))(o,((h)))
Moreover, for any 0 -) < and any h E H such that ||/||| S | +e:= M, holds with probability
greater than 1 - that
ID(h) C(M.)) VN
where ((,,r) depends only on M and 17.
Proof. First, we show that d++ ( a bounded linear operator. Indeed, Assumption (ii) ensures that (x.,.) and b(xp.))xp(-h((r) are Bochner integrable w.r.t. IP and G (Retherford (1978)), hence Ohh,o) is obtained as
D(h,8)== (0.Mesp(-A)G--M
where Mexp(-N)G (k(x, (os [expp (ew.))xxp((((()))) -f(x))dG and = [d.. Defining ((h) to be = =Mexp(-A)G He leads to the desired result. ((h) is simply obtained by taking the empirical version of D(h). Finally, the probabilistic inequality is a simple consequence of Chebychev's inequality.
The next lemma states that F(h) and F(h) are Frechet differentiable.
Lemma 9. Under Assumptions (i) and (ii), h+ F(h) is Frechet differentiable the open ball of radius [[holl + while h+ F(h) is Frechet differentiable on H. Their gradients are given by D(h) and D() as defined in Lemma 8,
VF(k)=D(h), VF(h)=D(h)
Proof. The empirical functional F(h) is differentiable since it is finite sum of differentiable functions, and its gradient simply given by D(h). For the population functional, we use second order Taylor expansion of exp with integral remainder, which gives
By Assumption (ii) we know that T(h,8) Tbll converges to 0 as soon as ||0|| 0. This allows to directly conclude that F is Frechet differentiable, with differential given by d- D(h, By Lemma 8, we conclude the existence a gradient FF)) which fact given by v=(h)= DDhh).
From now on, we will only use the notation (F(h) and F((h) to refer to the gradients of F(h) and F(h). The following lemma states that (21) and (22) have unique global optimum, and gives a first order optimality condition.
Lemma 10. The problems (21) and (22) admit unique global solutions h and hy in H. Moreover, the following first order optimality conditions hold:
Proof. For (21), existence and uniqueness of minimizer is simple consequence of continuity and
strong concavity of the regularized objective. We now show the existence result for (22). Let's introduce (xhh)= -(h))=Fh-- + & || for simplicity. Uniqueness a consequence of the strong convexity G,, For the existence, consider sequence of elements H such that (x(fi) inf If ho
18

---

Published as a conference paper at ICLR 2021
is not the global solution, then it must hold for & large enough that (x(f)) < (ho)). We also know that F(fx) (ho)), hence, it is easy to see that | < |/oo for & large enough. This implies that fa is bounded sequence, therefore admits weakly convergent sub-sequence
weak compactness. Without loss of generality we assume that fK weakly converges to some element /x E H and that | |hol|-- Hence, ||x Sliminf& f& S ||/holl- Recall now that by definition of weak convergence, we have (x) - (a) for all x X. By Assumption (ii), we can apply the dominated convergence theorem to ensure that F(fx) ((x).. Taking the limit of Gxfa the following inequality holds:
sup (h) = limsuppa (fx) s (h(x))
KEW
Finally, by Lemma 9 we know that Fis Frechet differentiable, hence we can Ekeland and Témam (1999) (Proposition 2.1) to conclude that F.hxxx) = Xhx. We use exactly the same arguments for (21).
Next, we show that h, converges towards hg in H. Lemma 11. Under Assumptions (i) to (ii) it holds that:
A(A) = 1 holl holl +00.
Proof. We will first prove that Yt converges weakly towards foo and then conclude that must also converge strongly. We start with the following inequalities:
2 X (111a11 (11001121110111
0 F(hx)
These are simple consequences of the definitions of hy and ho optimal solutions (20) and (21). This implies that |/> is always bounded by ||/oo].- Consider now an arbitrary sequence (Am m converging Since || is bounded by ||hall. follows by ceek-compactteess of balls in that hx,, admits a weakly convergent sub-sequence. Without loss of generality we can assume that hxm. itself weakly converging towards an element ht" We will show now that h² must be equal to hi- Indeed, by optimality of hx.,. it must hold that
hm F(hm))
This implies that (((())) converges weakly to O. On the other hand, by Assumption (ii). we can conclude that V.h())) must also converge weakly towards V.F(h"), hence VF(hh) 0. Finally by Assumption (iii) we know that ho is the unique solution to the equation V.F(h) hence h = hgg We have shown so far that any subsequence of Ax that converges weakly, must converge weakly towards ho. This allows to conclude that /x,,, actually converges weakly towards ho. Moreover, we also have by definition of weak convergence that:
|| s lim inf ||/... Il-
1-0000
Recalling now that ||h |l S ||hol| it follows that ||x converges towards / Hence, we have the following two properties:
h converges weakly towards fog
|سساا || converges towards [/hol--
This allows to directly conclude that [|xx.. -holl converges Proposition 12. We have that:
1 1h-hills hull -V VF(W))
Proof. By definition of h and 4x the following optimality conditions hold:
=hxx)F(
19

---

Published as a conference paper at ICLR 2021
We can then simply write:
-h- -(h-aa)-VV(()) 71
Now introducing 6: -h-h, H1 and E: - (afii)--((hi) for simplicity and taking the squared norm of the above equation, it follows that
VF(hs)
X - -
2x
By concavity of F on H we know that - -ortt-hh-hhh (h-hx, Therefore:
C LATENT NOISE SAMPLING AND SMOOTHNESS OF KALE
C.I LATENT SPACE SAMPLING
Here we prove Proposition 6 for which we make the assumptions more precise:
Assumption
make the following assumption:
logn is strongly concave and admits a Lipschitz gradient.
There exists a non-negative constant L such that for any x.x. E and 2,2/ Z:
- - -(=)((((())
[ VV =|
Throughout this section, we introduce [o)))==)og(((3))++(CC)))) for simplicity.
Proof of Proposition /. To sample from QG.E. we first need to identify the posterior latent distribution Vg, used produce those samples. We rely (23) which holds by definition of QG.E for any test function on X':
(23)
Hence, the posterior latent distribution is given by ===)= (2) (G(()) and samples from GEBM are produced by first sampling from VG,E, then applying the implicit map G,
X X~Q = X=G(Z), Z~v.
Proof of Proposition 2. the base distribution G admits a density on the whole space denoted by and the energy E is of the form E(x) E(xx)(())))) -r(x) for some parametric function E, it easy to see that Q has a density proportional to exp( E.) and therefore equivalent a standard EBM with energy E.
The converse holds as well, meaning that for any EBM with energy E, is possible to construct a GEBM using an importance weighting strategy. This is achieved by first choosing base G, which required to have an explicit density exp(-r) up to a normalizing constant, then defining the energy of the GEBM to be É(x) =
so that:
(Q(x) (lx))dGo x exep( - E(x))dr
(24)
Equation (24) effectively depends only on E(x) and not on G since the factor exp(r) exactly compensates for the density of G. The requirement that the base also admits tractable implicit map G can met by choosing G to be a normalizing flow (Rezende and Mohamed, 2015) and does not restrict the class of possible EBMs that can be expressed as GEBMs.
20

---

Published as a conference paper at ICLR 2021
Proof of Proposition 6. Let ni be the probability distribution of (2),04) at time of the diffusion in (14), which we recall that
d du; -re (ruu+)UU V
We call The its corresponding invariant distribution bisstrbbutiongiven y by
To (2,0) exp -U(2)-
By Lemma 13 we know that U is dissipative, bounded from below, and has Lipschitz gradient. This allows to directly apply (Eberle et al., 2017)( 2.6.) which implies that
W2 (Te S Cexp(-te),
where is a positive constant and C only depends Toc and the initial distribution no. Moreover, the constant c is given explicitly in (Eberle et al., 2017, Theorem 2.3) and of order where q is the dimension of the latent space Z.
We now consider an optimal coupling II, between He and mo. Given joints samples ((z,, 04 (2,)), from II, consider the following samples in input space (2,,2) == ((2)) Since 2 and have marginals म, and xes it is easy to see that 2, ~ P. and x ~ Q. Therefore, by definition of the W22 distance, have the following bound:
(F(F,,Q)
||--==11 =1
/ G(()) (ll((())))
$ 2 / || 211 (ll
- (T C²2 L'exp((222)))
The second line uses the definition of (xx, as joint samples obtained by mapping (z,,2). The third line uses the assumption that I--siischi Finally, the last line uses that IT, an optimal coupling between J. and Too
Lemma 13. Under Assumption 1, there exists A> and X (0. such that
(TUU
(U())
4u 11=11²
2
V
(25)
2
where y and u are the coefficients appearing (14). Moreover, U bounded bellow and has Lipschitz gradient.
Proof. For simplicity, let's call u(2) = log (2),, w(2) = E (2), and denote by M an apper-bounde the Lipschitz constant of and Vw which is guaranteed to be finite by assumption. Hence (2) = (w))+w(2)) Equation (25) equivalent to having
22 |2 22w(2) (z) Vwo (x))))2A4
-u(z)-2Ault 2Au(=)
(26)
2u
Using that w is Lipschitz, we have that w(2) < (0) + || and - T Tww(z) < M||x||. Hence, 2Aw (2w(2)=x (z)-z² TV)))) to hold is
+2X+
2A. Therefore, sufficient condition for (26)
72
V 22ux)))
|
-2A
(27)
2n
We will now rely on the strong convexity of u, which holds by assumption, and implies the existence of positive constant 0 such that
-u(x) 2 - - V |=12
m71
Vu(z)z - [00)
21

---

Published as a conference paper at ICLR 2021
This allows to write the following inequality,
2- 2X) Vu(z)+ X( -
Vu(z) (2x()))
2u
2u
22 2u
Combining the previous inequality with (27) and denoting M = [Vv(O]]] it sufficient to find A and À satisfying
2u
The Lh.s, in the above equation a quadratic function in ||>||| and admits a global minimum when (
2u The global minimum is always positive provided that A is large enough.
To that U is bounded below, it suffice to note, by Lipschitzness of that w(2) 2 w(0) -M|] and by strong convexity of u that
(z)) M|)+++ -1
Hence, dower-bounded by a quadratic function in || with positive leading coefficient m hence it must be power-bounder by constant. Finally, by assumption, u and w have Lipschitz gradients, which directly implies that U has Lipschitz gradient.
Proof of Proposition 3. By assumption KT((P||G) +oo, this implies that IP admits density w.r.t. which we call rx)). As a result admits also a density w.r.t. Q given by:
Zexp(E((z))r())
We can then compute the K (T((PI)) explicitly:
KL(P|QQ = Er(E] + log Z) Eg
=- F= )+ + (..
Since 0 belongs to E and by optimality of E*, we know that Lp (r, The result then follows directly.
C.2 TOPOLOGICAL AND SMOOTHNESS PROPERTIES KALE
Topological properties of KALE. Denseness and smoothness of the energy class & are the key to guarantee that KALE is a reliable criterion for measuring convergence. We thus make the following assumptions on E:
(A) For all E E -E E and there is Cg > 0 such that cE E E for 0 S c S Cg- For any there exists
continuous function g. any compact support K X and any precision a finite linear combination of energies G= -== E, such that supae k fz))-G(at ((
(B) All energies
E are Lipschitz in their input with the same Lipschitz constant Z>0. 0.
Assumption (A) holds in particular when contains feedforward networks with given number of parameters. In fact networks with single neuron are enough, shown in (Zhang et al., 2017, Theorem 2.3). Assumption (B) holds when additional regularization of the energy is enforced during training by methods such spectral normalization Miyato et al. (2015 8) or gradient penalty Guirajani et al. (2017) as done Section 6. Proposition 4 states the topological properties of KALE ensuring that it can be used as criterion for weak convergence. A proof is given in Appendix C.2.1 and is consequence of (Zhang et al., 2017, Theorem B.1).
Proposition 14. Under Assumptions (A) and (B) holds that:
1. KALE(P G) 2 0 with KALE G) 0 and only ifP=
2. KALE(P|| ++ and only if TP under the weak topology.
22

---

Published as a conference paper at ICLR 2021
C.2.1 TOPOLOGICAL PROPERTIES OF KALE
In this section we prove Proposition 4. We first start by recalling the required assumptions and make them more precise: Assumption 2. Assume the following holds:
The set X' is compact.
For all E - E & and there is CE > O such that cE & e CE. For any continuous function 9, any compact support K in X and any precision e > 0, there exists a finite linear combination of energies G= G=E 1i= such that [f(x)) G()-) S on K.
All energies E in E are Lipschitz in their input with the same Lipschitz constant L> O.
For simplicity we consider the set & R, i.e.: is the set of functions h of the form h -= E where E E & and E R. In all what follows P1 is the set of probability distributions with finite first order moments. We consider the notion of weak convergence on P1 as defined in (Villani, 2009, Definition 6.8) which is equivalent convergence in the Wasserstein- distance W1.
of Proposition 4, We proceed proving the separation properties (1*t statement), then the metrization of the weak topology (2thd statement).
Separation. We have by Assumption that 0e E, hence by definition KALE On the other hand, whenever = G, holds that:
(r) = (oa(0))=-((() (exp( ()xp(())+h-1)) 1)dP,
VhE
Moreover, by convexity the exponential, we know that exp( -)) 12- for all ER. Hence, (h) < FF,,(()) = for all h E H. This directly implies that KALE(P 00. For the converse, we will use the same argument as in the proof of (Zhang et al., 2017, Theorem B.1). Assume that KALE(PP(O) G) 0 and let h be in H. By Assumption 2, there exists C. >> 0 such that ch E H and we have:
Now dividing by cand taking the limit it easy see that - I [hdl of f S0. Again, by Assump- tion 2, we also know that E H. hence, /'dPP - fhdG 0.. This necessarily implies that f'dPP- =0 [hdG=0 for all h E H. By the density of 7 in the set continuous functions on compact sets, we can con- clude that the equality holds for any continuous and bounded function, which in turn implics that P-C. G.
Metrization of the weak topology. We first show that for any P and G with finite first moment, it holds that KALE(PIG) < LW, (P, G), where W (P, the Wasserstein-1 distance between IP and G. For any h E H the following holds:
J hhaP cxp(-h)dG++
F(h)
( ((P))
-ʃ -(exp(-h))+ 1)dG
The first inequality results from the convexity of the exponential while the last one is a consequence of l being L-Lipschitz. This allows to conclude that KALE(PI G) LW (IP, G) after taking the supremum over all h H. Moreover, since W1 metrizes the weak convergence on P1 (Villani, 2009, Theorem 6.9), it holds that whenever sequence G" converges weakly towards P in P1 we also have Wi (P, Gr) - and thus (AL)) - 0. The converse a direct consequence of (Liu et al., 2017, Theorem 10) since by assumption X' is compact.
23

---

Published as a conference paper at ICLR 2021
Well-defined learning. Assume that for any <> and any h and h in & there exists f in 2 such that 1 -flles flee I c then there exists a constant C' such that:
KALE(P Q) CKALE(P..)
This means that the proposed learning procedure which first finds the optimal energy E given base G by maximum likelihood then minimizes KALE(PP G) ensures ends up minimizing the distance between the data end the generalized energy- based model Q.
KALE(P,.Q) ssup_,, Qa (h) kEE
=AKALE(P,
(Ssp: c 3Er
28 Let's choose = KALE((,F))) and h 28 such that || th++=--- - f11 Se. We have by concavity of the function (r,8))=<p,c((h1e) - f we have that:
By assumption, we have that ||++ E fl110 c, thus Le (h. - fcl Moreover, we have that
since 1 E. This ensures that:
L (KALE((.C))
Finally, have shown that:
KALE (
Hence, minimizing KALE(P,G) directly minimizes KALE((PQQ)
C.2.2 SMOOTHNESS PROPERTIES OF KALE
We will now prove Theorem 5. We begin by stating the assumptions that will be used in this section:
(I) E is parametrized by compact set of parameters (T,
(1I) Functions in & are jointly continuous w.r.t. (v),)) and are L-lipschitz and L-smooth w.r.t the imput x:
|| (x) Ex) Lell -
[V V, [v. (x -e
(III Go,z)=G((z) is jointly continuous in and z, with 2 Gz(z) uniformly Lipschitz w.r.t. z:
V2,z' E e.
There exists non-negative functions a and b defined from Z to IR such that 0 - Gz(z) are a-Lipschitz and b-smooth in the following sense:
(e]]) 8'11
Moreover, a and integrable in the following sense:
f (to)eexp/2.. <o0, / fexp(L.folltl)hhhh <o,
I ) ((E))ep((Lobolll))dd <0.
24

---

Published as a conference paper at ICLR 2021
To simplify notation, we will denote by Cf(f) the expected Go log- ligellikiihod under P. In other words,
L (E)
|EdP
EdP EdPg log
We also denote by pe,e the density of the model w.r.t. Go.
exp(-E)
ZG..E= f.o.==xxpp--ER exp(-E)ddGe
PE,0
Z
We write KC(0)= = (PA to emphasize the dependence on 0.
Proof of Theorem To show that sub-gradient methods converge to local optima, we only need to show that IC is Lipschitz continuous and weakly convex. This directly implies convergence to local optima for sub-gres gradient methods, according to Davis and Drusvyatskiy (2018); Thekumparampil et al. (2019). Lipschitz continuity ensures that K. is differentiable for almost all e and weak convexity simply means that there exits some positive constant C 2 0 such that + KC(0) + C||00] is convex. We now proceed show these two properties.
We will first prove that O K(()) is weakly convex in 0. By Lemma 15, we know that for any E' E E, the function 0 La (E) M Mssmooth for the same positive constant M. This directly implies that is also weakly convex and the following inequality holds:
M
Co
(E)+ (1--))10-011
Taking the supremum w.r.t. E, it follows that
(((0)) (10()))00)))00))((0)00)) M 1(1 (1-1))10-0111
This means precisely that K. is weakly convex in 0.
To prove that is Lipschitz, we will also use Lemma 15, which states that Lo(E) is Lipschitz in uniformly &. Hence, the following holds:
Lo (BC))
Again, taking the supremum over F, it follows directly that
XC(0) <(0)=<(())++++]0000
We conclude that IC is Lipschitz by exchanging the roles of 0 and 0' to get the other side of the inequality. Hence, by the Rademacher theorem, K. is differentiable for almost all 0.
We will now provide an expression for the gradient of I. By Lemma 16 we know that wh+ Lo (Ep is continuous and by Assumption (I) & is compact. Therefore, the supremum sup Co is achieved for some function Eý. Moreover, we know by Lemma 15 that (x(E) is smooth uniformly on E, therefore the family Co (F)) Eee is giuieiiiferent..ble. We are in position apply Milgrom and Segal (2002)((hherem)) 3) which ensures that (C) admits left and right partial derivatives given by
0. (C(0) fmm0aCo (T e,
(28)
0, (C(0) lim 0g0 (F Te,
t-
where e is a given direction in R". Moreover, the theorem also states that KC(B) is differentiable iff
t Eg is continuous at 0= Now, recalling that KC(0) actually differentiable for almost all 6, must hold that Fijtte t Fi and 0 K(0)= KC(0) for almost all 0. This implies that the two limits in (28) are actually equal to 0g Ctp E T e. The gradient of K. whenever defined, in therefore given by
VolC(0) - Zo,,
V. V.EKGGad))yGGGooooeEEGGGGllllll
25

---

Published as a conference paper at ICLR 2021
Lemma 15. uniformly on E:
Under Assumptions (I) to (III). the functional (o(E) is Lipschitz and smooth in
(B) ||
(
Proof. By Lemma 16, we have that (B( differentiable, and that
0gLo(E)= ( (...)//////oGGGGGGGoo
Lemma ensures that [a,) is bounded by some positive constant that is independent from E and 0. This implies in particular that (E(E) is Lipschitz with constant C. We will now show that it also smooth. For this, we need to control the difference
D - (E)ll-
We have by triangular inequality:
D
V Vo
V. V. Po.
oG
111
The first term can be upper- bounded using smoothhsss E and the fact that Go is Lipschitz
I T. 1 (1 (p
0'1|
-
The last inequality was obtained by Lemma 17. Similarly, using that VaGa is Lipschitz, it follows by Lemma that
II L. 8'1 1b(pp,,0Go)dd
<L.C1000'11.
Finally, for the last term ITT, we first consider a 0, = 10 (1 for and introduce the function s(t): : PE,8, : oGe. We will now control the difference p oGg, also equal to ((1)-s(0). Using the fact that is absolutely continuous we have that s(1) -()) S dt)) The derivative st( is simply given by s'(t) = (0-0') (M, 100--Ma (M;)s(t) where M, ,v Bo, VoGo, and Mj = M.PE,, oGo dn. Hence,
PF,
s(1) (0)=s(0)((8-8))) J. (M, (M)-
also know that M, is upper-bounded by La(=), which implies
+ )3300+09)10-011,
III -11--310-- [((eatt)ssttttteddmeee) hanpB//ccceeooeee(((a(acaaooooo (f(e))stt(e))mmtt) ))
where the last inequality is obtained using Lemma 17. This allows us to conclude that Lo is smooth for any F. E and O
26

---

Published as a conference paper at ICLR 2021
Lemma 16. Under Assumptions (II) and (II1), it holds that 2b Lo (E() is continuous, and that 0 (Eo) is differentiable in with gradient given by
dg (oLo / (o
Moreover, the grudient is bounded uniformly in 0 and E:
[VeCo( (f (eapa//)ss.///((cbccbllllllllll (esp(-Locallelddttt)))
Proof. To show that - Lo( (.y)) continuous, we will use the dominated convergence theorem. We fix the in the interior of and consider a compact neighborhood W of e/p. By assumption, we have that (15,x) + Egga (z) and (1,22) + Es EG (=)(z)) are jointly continuous. Hence, E (0)) and E (G)( are bounded on W by some constant C. Moreover, by Lipschitz continuity of x Eng, we have
(x) S LL exp( - (V-(((((=2))
seppl) (exp(C)exp((LL
Recalling that admits a first order moment and that by Assumption (III), exp L. Lol is integrable w.r.t. 17, follows by the dominated convergence theorem and by composition of continuous functions that Lo ())) is continuous in v/o-
To show that 0 /- Lo Egp) is differentiable in 0, we will use the differentiation lemma in (Klenke, 2008, Theorem 6.28). We first fix 0n in the interior of θ, and consider compact neighborhood V of 0g. Since 0 + (O)] is continuous on the compact neighborhood V admits maximum value C; hence we have using Assumptions (II) and (III) that
(x) ((( ())(((())))xpp((LLolllll (tteespp(Loollllllxxexpp((OppOe [exp(C)exx(LLLLollll))
Along with the integrability assumption in Assumption (III), this ensures that FeaeeepppFFFFaoaaa is integrable w.r.t for all in V. We also have that exp(-E((G is differentiable, differrrtab with gradient
given by
Using that E is Lipschitz in its inputs and Go(z) is Lipschitz in 0, and combining with the previous inequality, follows that
where a(=) is the location dependent Lipschitz constant introduced in Assumption (III). The r.h.s. of the above inequality is integrable by Assumption (III) and is independent of 0 on the neighborhood V. Thus (Klenke, 2008, Theorem 6.28) applies, and it follows that
Vo exp( Fr(Go, / -
We now directly compute the gradient of Lo (F),
VoLo(E)= (f exp(( (xxp(=E(GG,))d N.SCC,,
Since E and Gg are Lipschitz in and 9 respectively, it follows that |[V, (=)) S Lg and (=)]l Hence, we have
[ L. (
Finally, Lemma allows us to conclude that (o is bounded by a positive constant C independently from 0 and E.
Lemma 17. Under Assumptions (II) and (III), there exists a constant C independent from and E such that
/ d())\p,,GG(())))))))
(29)
forie1,2
27

---

Published as a conference paper at ICLR 2021
Model
FID
Cifar10 Unsupervised
PixelCNN Oord et al. (2016)
65.93
PixelIQN Ostrovski et al. (2018)
49,46
EBM Radford al. (2015)
38.2
WGAN- G- Gulrajani et al. (2017)
36,4
NCSN Ho and Ermon (2016)
25.32
SNGAN Miyato et al. (2018)
21.7
GEBM (ours)
19.31
Cifar10 Supervised
BigGAN Donahue and Simonyan (2019)
14.73
SAGAN Zenke et al. (2017)
13.4
ImageNet Conditional
PixeCCNN
33.27
PixelIQN
22.99
EBM
14.31
ImageNet Supervisedd
SNGAN
20.50
GEBM (ours)
13.94
Table 4: FII) scores on ImageNet and CIFAR-10.
Proof. By Lipschitzness of E and Ga. we have exp(-L.LAll=]) S (expLE(G((0--E((Gd))) S exp(LeLol|a thus introducing the factor exp(E(Be, (0)) in (29) we get
/ a a (el)eeeolt)))d)) (espsp/-L----Lal/l)lllll))) (ata://)(d(k/))//aa/aax/////... (tta)/exp((//oolllldmmt))))
f / (fxpp [ipp[-L.,o|l=llddnn())) "(t))))))) /teeeepptt..oollllddddd
The r.h.s. of both inequalities is independent of 0 and E, and finite by the integrability assumptions Assumption (III).
D IMAGE GENERATION
Figures and show sample trajectories using Algorithm with no friction y =0 for the datasets. It is clear that along the same MCMC chain, several image modes are explored. We also notice the transition from mode to another happens almost at the same time for all chains and corresponds to the gray images. This is unlike Langevin or when the friction coefficient y is large as in Figure 5. In that case each chain remains within the same mode.
Table 4 shows further comparisons with other methods on Cifar 10 and ImageNet 32x32.
E DENSITY ESTIMATION
Figure Figure 7 (left) shows the error in the estimation of the log- eeg-pattiiion function using both methods (KALE-DV and KALE-F), KALE-DV estimates the negative log-likellnn bog-likelihood on each batch of size 100 and therefore has much more variance than KALE-F which maintains the amortized estimator of the log- partititnionn function.
Figure Figure (right) shows the evolution of the negative log likelihodd (NLL) both training and test sets per epochs for RedWine and Whitewine datasets. The error decreases steadily in the case of KALE-DV and KALFE-F while the gap between the training and test set remains controlled.
28

---

Published as a conference paper at ICLR 2021
Figure 3: Samples from the GEBM at different stages of sampling using Algorithm 3 and inverse temperature 3 on CelebA (Left), Imagenet (Right). Each row represents sampling trajectory from early stages (leftmost images) to later stages (rightmost images).
Larger gaps observed for both direct maximum likelihood estimation and Contrastive divergence although the training NI... tends to decrease faster than for KALE.
F ALGORITHMM
Estimating the variational parameter. Optimizing (9) exactly over A yields (8), with the optimal
equal to À = log( Nे - ((((()E))))))) However, to maintain an amortized estimator of the
eeg-paritionn we propose to optimize (9) iteratively using second order updates:
A
A A(exp(A:-AAti)--))
Ao=
(30)
where À is learning rate and Àr+1 is the empirical log-partition function estimated from batch of new samples. By leveraging updates from previous iterations, A can yield much more accurate estimates of the log-partition function confirmed empirically Figure of Appendix E.
Tempered GEBM. It can be preferable to sample from a tempered version of the model by rescaling the energy E by an inverse temperature parameter B, thus effectively sampling from Q. High temperature regimes (3 - 0) recover the base model G while low temperature regimes (5 + co) essentially sample from minima of the energy E. As shown Section low temperatures tend produce better sample quality for natural image generation tasks.
Training In Algorithm 1, we describe the general algorithm for training a GEBM which alternates between gradient steps the energy and the generator. An additional regularization, denoted by ((b) used to ensure conditions of Proposition and Theorem 5 hold. I(bb) can include Lę regularization over the parameters vb, a gradient penalty in Gulrajani et al. (2017) Spectral normalization Miyato et al. (2018). The energy can be trained either using the estimator in (8) (KALE-DV) or the one in (9) (KALE-F) depending the C.
29

---

Published as a conference paper at ICLR 2021
Figure 4: Samples from the GEBM at different stages of sampling using Algorithm 3 and inverse temperature 3 = on Cifarl0 and LSUN (Right). Each represents sampling trajectory from early stages (leftmost images) to later stages (rightmost images).
Sampling In Algorithm 3, we describe the MCMC sampler proposed in Sachs al. (2017) which time discretization of (14).
Algorithm 2 Overdamped Langevin Langevin Algorithm
1: Input X, U.m.E.GG
2: Ouput X²:
3: Zu /ample Initial latent from 7).
4: for I=O...... do
5: Y _loggn ~ E.EoB(Z) // Evaluating V=log(v(Ze+1)) using (4).
6:
W ~ (0,7) Sample standard Gaussian noise
7: Z Ze AY
8: end for
9: X + G-((TT)
30

---

Published as a conference paper at ICLR 2021
Figure 5: Samples from the tempered GEBM at different stages of sampling using langevin and inverse temperature 3 == 100, on Cifarl0 (Left), Imagenet (Middle-left), CelebA (Middle-Right Right) and LSUN (Right). Each row represents sampling trajectory from early stages (leftmost images) to later stages (rightmost images).
Algorithm 3 Kinetic Langevin Algorithm
Input 7, F.G.
2: Ouput X7
3: Zo~e7 Sample Initial latent from 7).
4: fort=0.... do
5:
Z,41 - Z,+
6:
Y V-loog V. Exaluating V=log(p(Ze+1)) using (4).
7:
V4+1 1v++++ Y
8:
W411 WH11~((0,7) // Sample standard Gaussian noise
9:
Ve+1+ + exp(->A)V heWV:/+///(leeeeeeeee /uu(1=epp(-22
10:
V4+1+ V++1+ ud Y441
11: Z++1+ Z + 2 Vi+
A
end for
13: X + GG(ZT)
31

---

Published as a conference paper at ICLR 2021
Effect temperature
FID score vs MCMC itearations
100%
100%
imagenat52
effar10
Isan
90%
CelebA
Relative
80%
70%
70%
10-2
10
10a
500 MCMC iterations
Temperature
Figure 6: Relative FID score: ratio between FID score of the GEBM Qc.E E and its base G. (Left) Evolution of the ratio for increasing temperature on the 4 datasets after 1000 iterations (14), (Right) Evolution of the same ratio during MCMC iteration using (14).
Log partition estimation
Negative Log Likelihood
donsker
donsker
10
kale
Negative Log Likellhood
cd
ml kale
Relative error
14
10-2
10-1
Train
Test
500
1000
1500
2000
500
1000
1500
2000
Epochs
Epochs
partition estimation
Negative Log Likelihood
20
18
Negative Log Likelhood
Relative error
10-2
16
kale
10-1
8
Test
Train
mp4
500
1000
1500
2000
0
500
1000
1500
2000
Epochs
Epochs
Figure 7: (Left): Relative error o" by è using either KALE-DV or KALE-F training Epochs RedWine (Top) and WhiteWin (Bottom) datasets. (Right): Negative log likelihood vs training epochs on both training and test set for different learning methods IKALE-DV.KALEFFF CD and ML) on RedWine dataset.
18-e" Tel
the estimation of the ground truth por-ptriiiinn function
G EXPERIMENTAL DETAILS
In all experiments, we use regularization which is combination of Le norm and a variant of the gradient penalty Gulrajani et al. (2017). For the image generation tasks, also employ spectral normalization Miyato al. (2018). This is to ensure that the conditions Proposition and Theorem hold. We pre-condition the gradient as proposed in Simsekli al. (2020) to stabilize training, and to avoid taking large noisy gradient steps due to the exponential terms (8) and (9). We also use the second-order updates (30) for the variational constant whenever it is learned.
G.1 IMAGE GENERATION
32

---

Published as a conference paper at ICLR 2021
RGB image M
3x3, stride= 1 conv 64 IReLU
2 E ~N (0,1)
4x4, stride= conv 64 IReLU
dense + Mg x Mg x512 4x4, stride- 2 deconv. BN 256 ReLU 4x 4, stride 2 deconv. BN 128 ReLU
3x3, stride= 1 conv 128 IReLU
4x4, stride 2 conv 128 IReLU
3x3, stride= conv 256 IReLU
4 4x4, stridd== deconv. BN 64 ReLU 3, stride= 1 conv. 3 Tanh
4 4, stride= 2 conv 256 IReLU
3x3, stride conv 512 IReI.U
dense 1
Table 5: Base/Generator of SNGAN ConvNet: Mn-4.
Table 6: Energy/Discrinintoo of SNGAN ConvNet: M 3232.
RGB image 2 E RR MxX3
ResBlock down 128
LETRI00~NN00 dense, 4 x 4
ResBlock down 128
ResBlock 128
ResBlock up 256
ResBlock 128
ResBlock up 256
ReLu
ResBlock up 256
Global sum pooling
BN. ReLu, 3 x 3 conv, Tanh
dense +1
Table 8:
Base//Generator of SNGAN ResNet.
Table 7: Energy/Discriminator of SNGAN ResNet.
Network Architecture Table 5 and Table 6 show the network architectures used for the GEBM in the case of SNGAN ConvNet. Table 5 and Table 6 show the network architectures used for the GEBM in the case of SNGAN ResNet. The residual connections of each residual block consists of two convolutional layers proceeded by a BatchNormalization and ReLU activation: as in (Miyato et 2018, Figure 8).
Training: We train both base and energy by alternating gradient steps to learn the energy vs 1 gradient step to learn the base. For the first two gradient iterations and after every 500 gradient iterations on base, we train the energy for 100 gradient steps instead of 5. We then train the model up to 150000 gradient iterations on the base using a batch-size of 128 and Adam optimizer Kingma and Ba (2014) with initial learning rate of 10 and parameters (0.5,.999) for both energy and base.
We decrease the learning rate using scheduler that monitors the FID score in a similar way as in Binkkwski ct al. (2018); Arbel et al. (2018). More precisely, every 2000 gradient iterations on the base, we evaluate the FID score on the training set using 50000 generated samples from the base and check the current score is larger than the score 20000 iterations before. The learning rate is decreased by factor of 0.8 if the FID score fails decrease for consecutive times.
Scheduler:
Sampling: For (DOT) Tanaka (2019), we use the following objective:
ッ || - +ell+
EoG(z)
(31)
kefs
where zy is sampled from standard Gaussian, is a perturbation meant to stabilize sampling and keff is the estimated Lipschitz constant of E B. Note that (31) uses flipped sign for the E B compared to Tanaka (2019). This is because plays the role - where is the discriminator in Tanaka (2019). Introducing the minus sign in (31) leads to a degradation in performance. We perform 1000 gradient iterations with a step-size of 0.0001 which is also decreased by a factor of 10 every 200 iterations as done for the proposed method. As suggested by the authors of Tanaka (2019) we perform the following projection for the gradient before applying it:
(g
9 gtgg 9
/q
33

---

Published as a conference paper at ICLR 2021
We set the perturbation to 0.001 and fefs y to which was also shown in Tanaka (2019) to perform well. In fact, we found that estimating the Lipschitz constant by taking the maximum value [E)G(()] over 1000 latent samples according to 17 lead to higher values for kers: ( Cifar 10: 9.4, CelebA: 7.2, ImageNet: 4.9, Lsun: 3.8). However, those higher values did not perform as well as setting kafs =
For (IHM) Turner et al. (2019) simply run the MCMC chain for 1000 iterations.
G.2 DENSITY ESTIMATION
Pre-processing We use code and pre- pre-processing steps from Wenliang et al. (2019) which we describe here for completeness. For RedWine and White we added uniform noise with support equal to the median distances between two adjacent values. That is to avoid instabilities due to the quantization of the datasets. For Hepmass and MiniBoone, we removed ill-conditioned dimensions as also done in Papamakarios et al. (2017). We split all datasets, except HepMass into three splits. The test split consists of 10% of the total data. For the validation set, we use 10% of the remaining data with upper limit of 1000 to reduce the cost of validation at each iteration. For HepMass, used the sample splitting as done in Papamakarios et al. (2017). Finally, the data is whitened before fitting and the whitening matrix was computed most 10000 data points.
We set the regularization parameter to 0.1 and use a combination of L2 norm and a variant the gradient penalty Gulrajani et (2017):
Regularization:
T(w)² = deb
|w
II (
Network Architecture. For both base and energy, we used an NVP Dinh el. (2016) with NVP layers each consisting of a shifting and scaling layer with two hidden layers of 100 neurons. We do not use nach-normalliztton.
Adam optimizer with learning rate of 0.001 and momentum parameters 0..5, For both KALE-DV and KALE-F, we used a batch-size of 100 data samples vs 2000 generated samples from the base in order reduce the variance of the estimation of the energy. We alternate 50 gradient steps on the energy vs step on the base and further perform 50 additional steps on the energy for the first two gradient iterations and after every 500 gradient iterations on base. For Contrastive divergence, each training step is performed by first producing 100 samples from the model using 100 Langevin iterations with a step- size of 10-2 and starting from batch of 100 data- sata-ss The resulting samples are then used to estimate the gradient of the of the loss.
Training: In all cases
For (CD), we used 100 Langevin iterations for each learning step to sample from the EBM. This translates into improved performance at the expense of increased computational cost compared to the other methods. All methods are trained for 2000 epochs with batch-size of 100 (1000 on Hepmass and Miniboone datasets) and fixed learning rate 0.001, which was sufficient for convergence.
G.3 ILLUSTRATIVE EXAMPLE IN FIGURE
We consider parametric functions G.") and Gf from to of the form:
(z)
(itt)=sin(8rr) sin (2)
(x)=
with 0 = (W,B.W",
(3,WW,6), .). we also call 0 = (1,1,1,0). In addition, we consider sigmoid like function from [0.1] to [0,1] of the form:
= tan(T(2= }
1 2 1+exp(-92) h(2)=
Data generation generative model:
To generate data point X (X1, X2), we consider the following simple
Sample a uniform r.v. from [0.1].
Apply the distortion function h get latent sample Y= h()).
34

---

Published as a conference paper at ICLR 2021
Generate point X using X = ()0((() (Y) and X2 -2= (Y).
Hence, the data are supported on the 1-d line defined by the equation X2 G (X1).
GAN For the generator we sample Z uniformly from [0. 1] then generate a sample (G) (Z)). The goal is to learn 8.
For the discriminator, we used an MLP with layers and 10 hidden units.
For the base use the same generator in the GAN model. For the energy the same MLP as discriminator the GAN model.
GEBM
EBM
To ensure tractability of the likelihood, we use the following model:
X1 X1 ( (1)(202))))
MoG((un,a1) (122200))) refers to a Mixture of two gaussians with mean and variances 14s and as. We learn each the parameters (0,00, ot 12220022 by maximizing the likelihood.
Both GAN and GEBM have the capacity to recover the the exact support by finding the optimal parameter 0* For the EBM, when 0 0*, the mean Gg- (X1) of the conditional gaussian X2 X
draws line which matches the data support exactly, iee. X2 = G (X1).
G.4 BASE/GENERATOR COMPLEXITY
To investigate the effect model complexity of the performance gap between GANs and GRBMs, we performed additional experiments using the setting of Figure 1. Now we allow the generator/base network to better model the hidden transformation that produces the first coordinate X1 given the latent noise. We choose G(") to be either a one hidden layer network or MLP with 3 hidden layers both with leaky ReLU activation, instead of simple linear transform as previously done in Appendix G.3. The network has universal approximation capability that depends on the number of units. This provides a direct control over the complexity the generator/base. We then varied the number of hidden units from to 5 10 units for the one hidden layer network and from 10 to 5 10x units per layer for the MLP. Note that the MLP with 5 10 units per layer stores a matrix of size 2.5 10? and thus contains orders of magnitudes more parameters than the widest shallow network with 5 10 units. We then compared the performance of the GAN and GEBM using the Sinkhorn divergence Feydy al. (2019) between each model and the data distribution. In all experiments, we used the same discriminator/energy network described Appendix G.3. Results are provided in Figure 8
shallow network
deep network
GAN
GAN
1.5
GEBM
1.5
GEBM
Sinkhorn
Sinkhorn
1.0
1.0
0.5
0.5
0.0
0.0
100
10
10²
10a
104
102
10a
104
10a
10
107
Number of parameters
Number of parameters
Figure 8: Sinkhorn divergence between data generating distribution and the trained model (either GAN GEBM) number of hidden units in G( The left figure represents the hidden layer network and right one is for the MLP. Each data point represents the average over 20 independent runs for each choice of number of hidden units.
35

---

Published as a conference paper at ICLR 2021
Estimating the Sinkhorn divergence. The Sinkhorn is computed using 6000 samples from the data and the model, with squared euclidean distance as ground cost and using a regularization <= le We then repeat the procedure times and average the result to get the final estimate of the Sinkhorn distance for a given run.
Training. Each run optimizes the parameters the model using Adam optimizer (31= 0 .,,32 .99).
learning rate 1r= 1e -4 for the energy/discriminator and lr= 1e - 5 for the base/generator and weight decay of le - for the base/generator. Training is performed using KALE for 2000 epochs using batch size of 5000 and 10 gradient iterations for the energy/discriminator per base/generator iteration. We use the gradient penalty for the energy/diseriminator with penalty parameter of 0.01. We then perform early stopping and retain the best performing model validation set.
Observations We make the following observations from Figure 8: the GAN generator indeed improves when increase the number of hidden units. The performance of the GEBM remains stable as the number of hidden units increases. The performance of the GEBM is always better than the GAN, although we can see the GAN converging towards the GEBM. GEBM with a simpler base already outperforms the GAN with more powerful generators. The gap between the GEBM and the GAN reduces as the GAN becomes more expressive. Using deeper network further reduces the gap compared to shallow network.
These observations support the prior discussion that the energy witnesses remaining difference between the generator and training samples, as long it is not flat. This information allows the GEBM to perform better than GAN that ignores The performance gap berween the GEBM and the GAN reduces the generator becomes more powerful and forces the energy to be more flat. This consistent with the result in Proposition 3.
36


Reviews:
[{"review_id": "mSyvhlmRiqN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: In this work, a generalized energy-based model (GEBM) is proposed. During the generation, the base distribution and the energy cooperate to combine the strengths of both the energy-based model and the implicit generative model.\n \n+ves: \n \n1. This paper has proposed a framework so that the energy function can be used to refine the probability mass on the learned base distribution. The framework is trained by alternating between learning the energy and the base. Empirically, the framework outperforms GAN with the same complexity. \n \n2. KL Approximate Lower-bound Estimate (KALE) is used for energy training. There is a lot of detail on this derivation. \n\nConcerns:\n1. In this work, a new framework is proposed for training. The whole process seems complicated. Is there is a simple way to refine the probability mass on the learned base distribution? How about  considering exp(-E(x))G(x). Here G(x) is learned base distribution. A simple way to refine the probability mass is by sampling several generated output near the point and weight the outputs with exp(-E(x))G(x)? This seems easier. Could you explain the more possible benefits of your method?\n\n2. During the generation process, two algorithms are proposed: ULA and KLA.  It seems a large variance. It is unclear which one is better.\n\n\n\nQuestions during the rebuttal period: \nPlease address and clarify the cons above:\nMissed related work.\n \nFor the training of energy based models, there are several related work [2][3], etc. \nIn the work [1], the residual energy is used to better generation. MCMC is also used for generations. \n\n[1] Residual Energy-Based Models for Text Generation. ICLR 2020\n\n[2] Structured Prediction Energy Networks, ICML 2016\n\n[3] Learning Approximate Inference Networks for Structured Prediction. ICLR 2018", "extra_scores": {}, "extra_text": {}}, {"review_id": "dt9HJWU85u", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Originality:\nThe paper presents an interesting generative model termed generalized energy-based model (GEBM), which is essentially an exponential tilting of a plain generator model P_G by an energy based model exp(-E).\nThe appearance of such a model in generative modeling looks to be new. But the structure of the model is not surprisingly new.\n\nSignificance:\n\nThe pros:\n1. The purpose of introducing such a model, as stated by the authors, is to make use of the fact that target distribution may concentrate in a low dimensional manifold in the target domain, which can be captured by a low-dimensional generator model P_G. Once tilted by exp(-E), the overall model can generate sharper samples than the plain energy based model.\n2. The paper also presents tractable methods to train and sample from the proposed GEBM.\n3. The authors show that GEBM performs better than GANs on certain image generation experiments.\n\nThe cons:\n\nWhile it may be easier to understand why GEBM can be better than EBM on certain generative modeling tasks, the authors did not provide much explanation or justification on why GEBM can outperform GANs. As mentioned by the authors, the energy based models and functional generators are quite different generative models. By combining them, how to guarante that their pros rather than the cons will be strengthened? In what kind of tasks GEBM can perform better than GANs? Hopefully the authors can give more theoretical comparisons or intuitions to address these questions.\n\n\nQuality:\n\nPros:\nThe overall quality of the paper is good. Nearly all results regarding the properties of GEBM come with theoretical justifications.\nThe proposed alternative training method and sampling methods look reasonable. \n\nCons:\nIt would be better if there is some analytical comparison between GEBM and GAN.\nAlso, in the experiment section, the authors should reveal more detail on how the energy function (class) is chosen.\nAnother concern is that, for Proposition 1 to hold, does it require the generator G to be invertible?\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HWxbfippz1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a framework called GEBM that combine an implicit generator and an EBM to define a probabilistic model on low dimensional manifold. Specifically, the implicit generator defines the base distribution, and the EBM refines the base. Finally, this method is equivalent to define an EBM on the latent space of the implicit generator together with a mapping from the latent space to the data space. The authors propose to use the KALE to train the generator, and provide a theoretical guarantee about the validness of the KALE.\n\nI personally enjoy reading this paper. It provides a good method to solve the problem where the EBM cannot model the distribution whose support is a low-dimensional  manifold in the data space. However, my main concern is about the contribution of this paper, which can be summarized as follows:\n1. The novelty. This paper mainly has two contributions: the generalized ebm framework and the KALE objective function. For the first contribution, the difference between the related work is discussed. But what’s the advantages? When the base distribution is an implicit model, GEBM is still a GAN-like model: cannot estimate density. When the base distribution is a flow model, GEBM is still an EBM: cannot provide tractable partition function.\n2. The validness of the proposed method. If the base distribution is an implicit model, which means that the support is only a subset of the data space. However, most implicit models, especially GANs, suffer from the mode collapse problem. In this case, the distribution defined by GEBM cannot recover the data distribution. Is there any explanation about the mode collapse problem in GEBM?\n\n\nSome minor concerns about this paper are provided as follows:\n1. The experimental results are chaotic. First, the neural architectures are not introduced, which is the key factor in the performance of GAN-like models. Second, Figure 1 needs more explanation. What is the architectures of the models? Which GAN variant is used? Is EBM trained by MLE using MCMC or score matching?\n2. What’s the main purpose of using the flow model as the density estimator? As mentioned in Proposition 2, GEBM is equivalent an EBM. Besides, in the experiments, the energy model is also an NVP. More explanation should be added to improve the clarity. Further, how NVP is trained with CD?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a generative model termed generalized energy-based model (GEBM).

The goal is modelling complex distributions supported on low-dimensional manifolds, while offering more flexibility in refining the distribution of mass on those manifolds. The key idea is presented as parametrizing the base measure (called a generator in the paper) and the density with respect to this base measure separately. Figure 1 of the paper sketches the idea on a very clear toy example.

The pros:
* Flexibility: Decomposing the full problem as learning the support and learning the density on this support 
* Theoretical justification
* Introducing the KALE objective
* Comparative empirical results with GANs show the additional benefits. Empirically, the framework outperforms GAN with the same complexity.
* Clear written paper

The lack of a comparison with GANs has been raised as a concern.  The authors have satisfactorily answered key questions and
others raised during rebuttal and added several new references. They have also improved the narrative and included an additional experiment to contrast GEBM and GANs in response to AnonReviewer2, also provided more detail 
on how the energy function (class) is chosen. 

Decision: Accept (Poster)