Bounds all around: training energy-based models with bidirectional bounds
Authors: Cong Geng
Jia Wang
Zhiyong Gao
Jes Frellsen
Søren Hauberg
Venue: NeurIPS 2021 Poster (2021)
Keywords: energy-based model
density estimation
bidirectional bounds
Jacobi-determinant estimator

Abstract:  Energy-based models (EBMs) provide an elegant framework for density estimation, but they are notoriously difficult to train. Recent work has established links to generative adversarial networks, where the EBM is trained through a minimax game with a variational value function. We propose a bidirectional bound on the EBM log-likelihood, such that we maximize a lower bound and minimize an upper bound when solving the minimax game. We link one bound to a gradient penalty that stabilizes training, thereby provide grounding for best engineering practice. To evaluate the bounds we develop a new and efficient estimator of the Jacobi-determinant of the EBM generator. We demonstrate that these developments stabilize training and yield high-quality density estimation and sample generation.

Bounds all around: training energy-based models with bidirectional bounds
Cong Geng, Jia Wang, Zhiyong Gao Shanghai Jiao Tong University
Jes Frellsen; Søren Hauberg" Technical University of Denmark (jefr, sohau)@dtu.ce dk
[gengcong, jiawang, zhiyong- gao) edu on
Abstract
Energy-based models (EBMs) provide elegant framework for density estimation, but they are notoriously difficult to train. Recent work has established links to generative adversarial networks, where the EBM is trained through minimax game with a variational value function. We propose a bidirectional bound on the EBM log-likelihood, such that maximize lower bound and minimize upper bound when solving the minimax game. We link one bound to gradient penalty that stabilize training, thereby provide grounding for best engineering practice. To evaluate the bounds we develop new and efficient estimator of the Jacobi-determinant of the EBM generator. We demonstrate that these developments stabilize training and yield high-quality density estimation and sample generation.
Energy-based models
Energy-based models (EBMs) are probabilistic models that draw inspiration from physics and have a
long history in machine learning (Hopfield [982] Hinton and Sejnowski] [983] Smolensky (986). An EBM is specified in terms of an energy function Eg X + R that is parameterized by 0 and defines a probability distribution over X from the Gibbs density:
(x) cxp(-Ee(xx
Zo (xp(-EE(x))dxx
(1)
Ze
where Zg is the normalization constant or partition function. In principle, any density can be described this way for suitable choice Eg. EBMs are typically learned using maximum likelihood estimation (MLE), where we wish to find a value of 0 that minimized the negative data log- likeeioo::ood::
L( Ex-puuix) [log Pa (x)) Ex-pon
[x)] + log Zg.
where Pdata is the data generating distribution.
The fundamental challenge with EBMs for non-standard energy functions is the lack of closed-fome expression for the normalization constant, Zo. which hinders exact pointwise density evaluation, sampling and learning. Therefore, these tasks traditionally performed using approximate methods such as Markov chain Monte Carlo (MCMC, Metropolis et al. 1953 Neal et al. 2011). Similar to
the Boltzmann learning rule (Hinton and Sejnowski 1983 Osogami 2017 gradient-based learning for MLE involves evaluating the gradient VoL(O) Ex-pan [ Ve
Ex-rs Ex)], where
the expectation in the last term necessitates MCMC approximation. Recently, series of papers (Kim)
and Bengio| 2016 Kumar et al. 2019; Abbasnejad et al. 2020; Che et al. 2020) have established links between EBMs and Wasserstein GANs (WGANs, Arjovsky et al. 2017). such that EBMs can be approximately learned through a minimax game. Xic et al.
and Wu et al. (2018) also
employed the adversarial game of EBM. This allows for significant improvements in EBM learning over MCMC based methods (Grathwohl et al. 2021)
"qqual contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this paper, we remark that variational bounds on the value function of the minimax game can be problematic as this is both maximized and minimized. We propose both upper and lower bound the negative log-likelihood, and alternate between their respective minimization and maximization to alleviate this concern. Evaluation the bounds requires evaluating the entropy of the generator, and we provide new efficient estimator. We link the upper bound to the use of gradient penalties which are known to stabilize training. Experimentally, we demonstrate that our approach matches or surpasses state-of-the-art on diverse tasks at negligible performance increase.
1.1 Background: Variational bounds and minimax games
Following Grathwohl et al. (201), we can lower bound the log normalization constant using proposal (variational) distribution Pg (x) and Jensen's inequality:
Fx)) (x)
Fx(())) = -xxpp (Eo (x)] + H [p (x(x)
log Zo = log Ex-ps
2 Exxp, log
(3)
where Hppy] = - -Ep, [og Py] is the (differential) entropy of the proposal distribution. This means that we also obtain lower bound on the negative log likelihood function given by
( == Ex-ppan (xx) [x)] Ex-p, (x) [E (x)] + H[p,] S L(6).
(4)
We note that this bound is tight when Pg pg. since the bound on the log normalization constant (3)
can be equivalently expressed as log Zo 2 log Zo KL (PyllPo). To tighten the bound, we seek
proposal distribution Pg that maximises [(0), while for MLE want to find t that minimizes
[()). Accordingly, MLE can be formulated as a minimax game:
min max (C(0)} min max (1xx [Ea (x)] (x) (x) [p,)
(5)
Ey
Pg
Eg
Ps
This gives tractable and MCMC-free approach EBM learning. assuming evaluate the entropy (Grathwohl et al. 2021).
Our key issue is that both minimizing and maximizing a lower bound is potentially unstable. By minimizing a lower bound, we run the risk of finding an optimum, where the bound is loosest, rather than where it is informative about the optima of the true objective. In particular, minima of the lower bound may be -o0, which is rather unhelpful. Good results have been reported from using a
lower bound in the minimax game (Dai et al. 2017 Kumar et al. 2019 Grathwohl et al. 202 but
the conceptual issue remains.
The WGAN loss function is, as noted by Arjovsky et al. 2017). very similar to the lower bound C(0).
LWGAN = Ex-pasn EE (x) - Ex~p, [Eo(x]].
(6)
with the missing entropy term being the only difference. Lessons from the WGAN can, thus, be expected apply to the EBM setting as well. For example, the success of gradient clipping
(Arjovsky et al. (017) and gradient penalties (Gulrajani et al., (017)) in WGAN training, inspired
Grathwohl et al. (2021) to heuristically use gradient penalties in variational inference to stabilize training. Our work will provide justification to such heuristics.
2 Approximate minimax games through bidirectional bounds
Instead of solving minimax game with lower bounded value fune- tion ( we propose to bound L(Ø) from both above and below:
Migimize L(0)
[L(O)] < L(O) s [L(O)].
With this, we can now follow an optimization strategy of alternating
L(
1. minimize [L()]] with respect Eg.
Maximine L(O)]
2. maximize [L)] with respect to Pg-
Figure 1: The bidirectional bounds 'sandwich" the neg- ative data log-likelihood.
This avoids the potential pitfalls of minimizing lower bound.

---

2.1 A lower bound for maximizing Pg
We already have lower bound ((0) on L()) in Equation 4 The first two terms of the bound can readily be evaluated by sampling from Pdata and Pg, respectively. The entropy of Pg is less straight- forward. Following standard practice in GANs, we define P3 through base variable which is passed through "generator' network G R2 RD. i.. x G(z) ~ Pg- This constructs density over d-dimensional manifold IR which by standard change-of-variables is
( I).
log P. (G)))) = Po(z)( 2
log de
(8)
where Jg E IR xxd is the Jacobian of at z. This assume that the Jacobian exist, i.e. that the generator G has at least one derivative. The entropy of Py is then
1 2
H Hpyl Exxx,, [log Pa (x)] = [Po] + Ez-po
log dJT())
(9)
The entropy of pa trivially evaluates to H [po] = 4/2(1 + log(2 To avoid an expensive evaluation of the tereddtemminnnt-term,, we note that it is easily bounded from below:
1 Σ log s 2 dllogs1, 81,
log det dct(J
(10)
2
where sa 2 ... 2 $1 are the singular values of the Jacobian Jz. Note that in general, this bound is not tight. With this, we have our final lower bound:
L(0)= 2 [L))] = Exnprm (x) [ (x)] - (x) + H [po] + Egnpu [d s1 (z)].
(11)
2.2 An upper bound for minimizing Eg
There are many ways to arrive at upper bound for the log-likelihood. Due to the strong ties between the lower bound (4) and the WGAN objective, we take inspiration from how WGANs are optimized. The original WGAN paper (Arjovsky et al. (017)) noted optimization instabilities unless gradient clipping was applicd. Gulrajani et al. (2017) noted that this clipping could be avoided by adding a gradient penalty, such that the loss become
TAGAN GP INGAN + x[xx)(()) [[I] [ (o -1)3].
(12)
where è is uniform interpolation between real data and generated samples. This is generally acknowledged good way to train a WGAN (Gulrajani et al., 2017). We observe that this loss is similar to our lower bound, but the gradient penalty adds positive term, such that one could speculate that we might have an upper bound. With this intuition in hand, we prove the following statement in the supplementary material.
Theorem 1 Suppose f X' + R is L-Lipschitz continuous, g(æ) is a probability density function with finite support, then there exists constants M, m 2 0 0nd p 2 1 such that:
log Ex=ggte) [f(x)] Ex)g(x)
[log f(x)) & M (Ex-g(x) [| log f(x))) m)'4, 'v,
(13)
Note that if Ex= V log (1 + m 2 1, the bound in Equation dropping the power 1/P.
13 can be simplified by
log [xxg((x) [x(x] Exxg((x) [log f(x) I M Mx) V x log fxx) + Mm.
(14)
For our model, we set f(x) exp( (sp(-EE(x)) and gxx) Py (x). Empirically, observe the bound
Po (x)
simplification in Equation 14) holds through-out most of the training, and use the following upper bound:
(15) For given choice of fxx), we can view as constant and the term Mm can be ignored during optimization. This upper bound can directly be interpreted as the lower bound (4) plus a gradient penalty, albeit one of different form than the traditional WGAN penalty, which is derived purely from a regularization perspective. Our upper bound can, thus, be seen as a justification of the regularization from a maximum likelihood perspective.
[L(0)] = L(0) + MExxp,(x) (o + Vx Log [P] + Mm.
Note that Eq. 8 is only the density the spanned manifold, and that the density is zero off the manifold.
3

---

Bound tightness When pe Pg. we have that (x) + Vx (x) = 0. In Theo- rem m is a constant related to the Lipschitz constant of log f (x) satisfying Vx log f(x)]² < Vy log f())| + m. for all y (see proof for details). When po = Pg we also have
Vx log f'()]]
0, such that m = O. Our upper bound then [T(0)] = (()) ((). and hence is tight.
Numerical evaluation of the bounds
Evaluating the lower bound To evaluate the lower bound in Equation [I] need the smallest singular value of the Jacobian = (z). Recall that this singular value satisfy 81 = | =
||Jv We can then evaluate the singular value by finding Vmin with an iterative optimization
millvy0 T[v]l algorithm, where opt use the celebrated angle-vector LOBPCG algorithm (Knyazev 1998).
This method performs an iterative minimization of the generalized Rayleigh quotient,
vTJJJv
p(v)
(16)
vTV
which converges to Vmin- The gradient of p(v) is proportional to r = JJJv p(v)v. To avoid computing the Jacobian J, we use Jacobian-vector products, which can be efficiently evaluated using automatic differentiation. To compute JTJv, we use the following trick (in pytorch-notation);
JJJv ((Jv)JJ) v ([Jv)]detach)) G(z))".
(17)
The optimal learning rate for this iterative scheme can be found by maximizing the Rayleigh quotient (16). Finally, we follow the suggestions of Knyazez (200T) to improve numerical stability and accelerate convergence, which we omit here for brevity.
Evaluating the upper bound There are two challenges when evaluating Equation (5)). The first is to compute Vx (x,),, where we empirically found that existing methods Shi ctt al. 2018 Li and Turner 2018) were too inefficient for our needs. To evaluate the term Exxp,(x) (x) Ex(x Vx log Py [?), we further loosen the bound
Va Fo(G)))) V (Cg))(
(o + Va(a (G(2))
of
Vain V. (G
(18)
where 81 is the smallest singular value of Jg. Detailed derivations are in the supplementary material. If we choose 2. then we can use Hutchinson]s estimator (1989):
V. V. Vx + Vz logp, = EEv [V) ((T
(19)
where v NV Io). This is easily evaluated using automatic differentiation.
The second challenge is to evaluate logp,(x) which needs the Jacobian of the generator G(z) as entropy estimator described above. We could
dictated by Equation (8. Here, we opt to alternatively use Hutchinson's estimator as proposed by Kumar et al. (2020). Experimentally we do not observe much difference between these two estimators.
4 Related work
In machine learning, there has been a long-standing interest in EBMs dating back to Hopfield networks (Hopfield, 1982). Boltzmann machines (Hinton and Sejnowski 1983 Ackley et al. 1985 and restricted Boltzmann machines (Smolensky)) 1986] [986]|Hinton] (2022), see e.g. reviews in the works by LeCun et al. (2006) Scellier (2020). Learning and evaluation of these models are difficult since the normalization constant cannot be efficiently evaluated. MLE-based learning, such as the Boltzmann learning rule, relies on expensive MCMC sampling to estimate the gradient, and more advanced
MCMC methods are used to reliably estimate the normalization constant (see e.g. [Salakhutdinov and
Murray, 2008) Grosse al. 2013] Burda et al., 2015) Frellsen et al., 2016). For images, MCMC-
based learning has been used to learn non-deep EBMs of both textures (Zhu et al. 1998 Zhu and

---

Mumford, 1998) and natural images (Xie et al. 2015 2016). Learning algorithms that avoid the costly MCMC approximation have been heavily investigated. For instance. |inton (2002) proposed k-step Contrastive Divergence (CD-k) to approximate the negative phase ligliikll gradient, and |Hyvärinen (2005) proposed an alternative method to train non-normalized graphical models using score matching. Deep versions of EBMs have subsequently been proposed, such as deep belief networks (Hinton et al. 2006) and deep Boltzmann machines (Salakhutdinov and Hinton| 2009).
In recent years, there have been renewed interest in deep generative EBMs, particularly for image generation. Grathwohl et al. (202T) gave an excellent overview of the current developments and distinct drawbacks of three classes of learning methods, which we summarize here: (1) MLE methods with MCMC sampling are slow and unstable, and while (2) score matching based methods are comparable faster, they are also unstable and do not work with discontinuous nonlincarities. (3) Noise-contraster estimation (Gutmann and Hyvärinen, 20T0) do not have these drawback, but does not scale well with the data dimensionality.
Our proposed method belongs to a fourth class of algorithms that sidestep the costly MCMC sampling by using a simultaneously learned generator or variational distribution. Kim and Bengio (2016) proposed using a generator function and an adversarial-like training strategy similar ours. They update the generator using the same lower bound as us, but their entropy approximation is quite different. Furthermore, their method does not have a gradient penalty like our upper bound when optimizing the energy function. Consequently, the energy function needs to be explicitly designed to prevent it from growing to infinity, limiting its potential.
The method proposed by /Zhai et al. (2016) plays a min-max game to jointly optimize the energy function and generator using the same lower bound as us. However, their method relies on specific designed bounded multi-modal energy function, which limits its potential. Furthermore, their approximation of the generator entropy comes with no theoretical guarantees, and their regularisation of the energy function does not constitute an upper bound.
Dai et al. (2117) proposed an adversarial learning framework for jointly learning the energy function and a generator. They considered two approaches to maximising the generator entropy: One, which maximizes the entropy by minimizing the conditional entropy using variational upper bound, and another, which makes isotropic Gaussian assumptions for the data, which is not suitable for high- dimensional data. Kumar al. (2019) and Abbasnejja et al. (219) also consider adversarial learning but different approaches to estimating the entropy of the generator. Kumar et al. (2019) estimated the entropy through its connection to mutual information, but they need an additional network to measure the entropy term. [bbbasnejad et al. (2019) maximized the entropy by approximating the generator function's Jacobian log-determinant. However, their method is impractical in high dimensions as the Jacobian is computationally expensive. HHan et al.] (2019) use an adversarial learning strategy in their divergence triangle loss, but their training mechanism radically different from ours, and they rely on an extra encoder for learning the generator.
Xie et al. 2018a 2021a b) proposed cooperative learning of the energy function and generator. However, the cooperative learning approach relies on MCMC or Langevin dynamics to draw samples from the EBM, which is expensive and difficult tune. Like our work, VERA by Grathwohl et al. (202T) avoids the use of MCMC. VERA plays a min-max game and uses variational inference to approximate the gradient of the entropy term, which is different from ours. Furthermore, VERA uses gradient penalty as a regularizer for the energy function, which is a heuristic unlike our upper bound, and their method is memory-consuminng and the hyperparameters are difficult adjust.
5 Experiments
To demonstrate the efficiency of our energy-based model with bidirectional bounds (EBM-BB) we compare against range of methods that represent otate-of-the-art. As representatives of the GAN literature, we consider deep convolutional GANs (DCGANs) by Radford et al. (2016), spectrally normalized GANS (SNGANs) by [Myato et al. (2018) and the WGAN with zero-centered gradient penalty (WGAN-OGP) by Thanh- Tung et al. (2019). As representatives of EBMs, we consider the maximum entropy generators (MEG) by Kumar et al. (2019) and the variational entropy regularized approximate maximum likelihood (VERA) estimator (Grathwohl et al.) 2021] As representatives of CoopNets, we consider two similar methods (Xie et al. 20 (00 Sa (021a) and EBM VAA (Xie et al. 2021b). We also consider NCSN by Song and Ermon 2019) and DDPM by Ho et al. (2020)
5

---

(a) Data
(b) WGAN-0GP
(c) MEG
(d) VERA
(c) EBM-OGP (ours) (f) EBM-BB (ours)
Figure 2: Density estimation on the 25-Gaussians and swiss-roll datasets.
representatives of matching model and diffusion model respectively. To investigate the influence of our derived upper bound, we introduce another baseline in which we replace our upper bound with zero-centered gradient penalty. We denote the resulting model EBM-OGP. Our implementation available at https//ggthuh..ommggegggggg00000 26/E/BBBBBB
5.1 Training details
On toy data and MNIST, all models are realized with multi-layer perceptrons (MLPs), while for natural images, we use the convolutional architecture from StudioGAN (Kang and Park 2020). Details of the network architecture
given in the supplementary material. All experiments conducted
single 12GB NVIDIA Titan GPU using a pytorch (Paszke et al. 2017) implementation. To improve training, we use positive margin for our energy function to balance the bounds. Specifically, let
[L(O)] = [L())] + (M [g + V (0gg,()) []] 5]+
(20)
where [+ = max(0. is the usual hinge, In most experiments, Ç = 1 as this allows us to apply the simplified bound in Equation (15). But in general recommend starting with Ç = 0 and trying successively larger values to stabilize training.
5.2 Toy data
Figure shows estimated densities the 25-Gaussians and swiss-roll datasets using both methods, WGAN-0GP and baselines. We observe that WGAN's discriminator is not
Table Number of captured modes and KL diver- gence between the real and generated distributions.
Model
Modest
KL4
suitable density estimator. This is unsur- prising as WGAN is not supposed to provide density estimate. MEG and EBM-OGP can have inaccurate density information in some edge and peripheral areas. This may be due to insufficient or excessive maximization of entropy since the zero-centered gradient penalty is not a principled objective for maximum likelihood. VERA and our EBM-BB can learn a sharp distribution, but our method is more stable in some inner regions of the 25-Gaussians and edge regions of the swiss-roll.
DCGAN
392 +: 7.4
8.012 + 0.056
SNGAN
441 39.0
2.755 + 0.033
WGAN-OGP
1000 + 0.0
0.048 + 0.003
MEG
1000 0.0
0.042 0.004
VERA
989 9.0
0.152 + 0.037
(BM-0GP(ours)
1000 - 0.0
0.039 0.003
EBM-BB (ours)
1000 + 0.0
0.045 + 0.003
5.3 MNIST
Mode counting Mode collapse frequently occurring phenomenon in which the generator function maps all latent input to small number of points in the observation space. In particular, GANs are plagued by this problem. Since the generator is trained to maximize the entropy of generated distribution, several EBM-based models have been shown to capture all the modes of the data distribution. To empirically verify that our model also captures variety of modes in the
data distribution, we follow the test procedure of Kumar et al. (2019). We train generative

---

model on the StackedMNIST dataset, synthetic dataset created by stacking MNIST on different channels. The true total number of modes is 1,000, and they are counted using a pretrained MNIST classifier. The KI. divergence is calculated empirically between the generated mode distribution and the data distribution. The results appear in Table As expected, GAN-based methods suffer from mode collapse except for WGAN-0GP. All the EBM-based methods capture all or nearly all the modes of the data distribution. Our model EBM-OGP captures all modes and report the smallest KL measures. It worth mentioned that WGAN-OGP obtains comparable results, which was also observed by Grathwohl et al. (202I).
Entropy Estimation In Figure 3 we explore the quality of our entropy estimator measured on MNIST (LeCun) 1998). We use networks with fully connected layers the Jacobian is then easily derived in closed-form, giving us a ground truth for the entropy. We compare our estimator (red) with the ground truth (black) and Hutchinson's ' estimator (blue), as was proposed by Kumar et al. (2020). Finally, we only run iterative estimator for few steps, we also compare with an estimator that uses the smallest singular value computed with high precision (green). We observe that Hutchinson's 's estimator is reasonable close to the ground truth but provides an upper bound, making it inapplicable for lower bound. We further observe a noticeable gap between our estimator and the high precision singular value estimator. Finally, we see that all estimators follow roughly the same trend, which suggests that our estimator provides a suitable target for optimization.
11
14
Smalles singalor valke
Ground inuth
Tuichinson's estimator
5-10f
16"
Iteration
Figure 3: Entropy estimators.
5.4 Natural Images
The data studied thus far simple and perhaps do not challenge our upper bound and entropy estimator. We, therefore, train our model on the standard benchmark 32 x 32 CIFAR-10 10 (Krizhessky et al. 2009) dataset and the 64 64 cropped ANIMEFACE dataset, which both represent significant increase in complexity. Similar to recent work on GANs (Miyato et al. 2018), we report Inception Score (IS), Fréchet Inception Distance (FID) scores and two F& scores (Sajjadi et al. 2018). We compare with competitive GANs, EBM baselines, CoopNets, NCSN and DDPM (Table 3). All GANs and EBMs are reproduced using the same network architecture (see supplements for details) except DCGAN and IGEBM, where we respectively borrow the results from StudioGAN (Kang and Park (020)) and the original implementation (Du and Mordatch 2019). We use the DCGAN (Radford et al. 2016) network architecture for CIFAR- 10 and a Resnet archi- tecture (Kang and 2020) for ANEMIFACE. Parameters are chosen as in the original papers. For VERA (rrthhohll et al. 2021), we were unable to reproduce the reported performance, so we choose hyper-parameters according an extensive grid search. Like the original paper, we choose the entropy weight to be 0.0001. For CoopNets and NCSN, we report the results from the original papers. For DDPM, we used a public available implementation
Image generation
From Table 3 we see that our EBM-BB model is the best performing EBM on CIFAR-10, though it is surpassed by NCSN and DDPM. This is not surprising NCSN and DDPM focus on sample quality rather than optimising a data likelihood. Further note that sampling NCSN and DDPM is significantly more expensive than our method. On ANIMEFACE our method is highly competitive. Figures
4 and [ show samples from different models. For ANIMEFACE. our method generates more diverse samples in terms of face parts than the baselines. All models predominantly generate female faces. This suggests that while our approach captures more density modes than the basclines, all approaches still misses several modes. We further draw attention to several corrupt samples generated by VERA, despite extensive parameter search. For CIFAR-10, we observe no immediate differences in the generative capabilities between models.
Capacity usage For the proposal distribution Py to be adaptive, the associated generator network G(z) should be able to use as much of its available capacity as possible. To compare different methods, we consider an implicit measure of capacity usage C₂ that locally measures the intrinsic dimensionality of Gzz). In particular, we use the anisotropy index (Wang and Ponce 2021) that, for
hheps://www.kagghb http:///wwwwwwagggcccoooooooooee howpee/eeeeceeeeeeancccceecteeee htapss/ggitibboommmoooiiillllldp diiffusion-pytore
7

---

(a) WGAN-0GP
(b) MEG
(c) VERA
(d) EBM-BB (ours)
Figure 4: Generated samples on CIFAR-10 10 with our method and various methods.
(a) WGAN-0GP
(b) MEG
(c) VERA
(d) EBM-BB (ours)
Figure 5: Generated samples ANIMEFACE with method and other generative models
given z, measure the standard deviation of the norm of the directional derivative of G(z) along the input dimensions, i.e.
C₂ std ([] d).
(21)
where std(.) computes the standard deviation of the input, and e; denotes the jth standard basis vector. A small C₂ indicates that the different input dimensions contribute equally to the out- put of G. which imply good capacity usage. We measure the mean value of C2 for z ~ ./ (O,0, Ia) and report mean and standard deviation of the result running for several times in Table 2 VERA, perform well on CIFAR-10, while on ANIMEFACE there more diversity. In both cases, GBM-BB has the best capacity usage. The large difference between EBM-OGP and EBM-BB on
Table 2: Anisotropy indices ()).
Model
CIFAR-10
ANIMEFACE
WGAN-0GP
1.0823 = 0.005
2.4366 + 0.129
MEG
0,9467 1 0.009
2.3600 1 0.142
VERA
3.9694 + 0.055
2.2929 + 0.2074
EBM-UGP (ours)
1.0688 0.0226
3.4560 = 0.0521
EBM-BB (ours)
0.9431 = 0.0191
18016 = 0.1345
We observe that most models, with the exception of
8

---

Table 3: Comparison in terms of FID, Inception Score, F's score (weights recall higher than precision) and Fi/s score (weights precision higher than recall).
Model
Inception
FID
Fst
F1/s t
CIFAR-10
DCGAN
6.64
49.03
0.795
0.83 0.95 = 0.010
WGAN-OGP
7.24 + 0.035
29.31 +: 0.185
0.92 = 0.010
CoopNets (2018a)
6.55
36.4
CoopNets 20221a)
33.61
EBM-VAL VAE
6.65
36.2
IGEBM
6.78
38.2
MEG
6.62 + 0.243
34.55 + 1.145
0.88 + 0.001
0.92 0.010 0.79 + 0.005
VERA
5.06 + 0.555
66.38 + 6.635
0.58 + 0.080
NCSN
8.87 0.12
25.32
DDPM
9.03
7.76
0.98
0.99
EBM-OGP (ours)
6.90 + 0.032
35.42 + 0.582
0.90 + 0.004
0.93 0.002
EBM-BB (ours)
7.45 + 0.014
28.63 + 0.290
0.93 = 0.001
0.95 = 0.008
ANIMEFACE
WGAN-0GP
2.22 0.030
9.76 0.674
0.95 0.005
0.98 0.005
MEG
2.20 + 0.020
9.31 + 0.007
0.95 + 0.005
0.98 + 0.001
VERA
2.15 + 0.001
41.00 + 1.072
0.515 + 0.078
0.78 0.013
DDPM
2.18
8.81
0.94
0.98
EBM-OGP (ours)
2.26 = 0.017
20.53 + 0.524
0.889 = 0.008
0.909 = 0.019
EBM-BB (ours)
2.26 + 0.005
12.75 + 0.045
0.94 = 0.001
0.96 = 0.005
Table 4: AUROC1, AUPRC1 and FPR80) for OOD detection for 'train / test' datasets.
CIFAR-10 10/ SVHN
CIFAR-10 10/ CIFAR- 100
ANIMEFACE/ Bedroom
Model
AUROCT
AUPRC1
FPR801
AUROCT
AUPRC1
FPR80
AUROCT
AUPRCT
FPR80J
WGAN-0GP
0,8
0.83
0.39
0.54
0.55
0.77
0.51
0.48
0.72
MEG
0.79
0.81
0.42
0.52
0.53
0.8
0.56
0.53
0.77
VERA
0.64
0.64
0.51
0.525
EBM-DGP (purs)
0.66
0.69
0.67
0.64
0.63
0.64
0.67
0.63
0.53
EBM-BB (ours)
0.88
0,86
0.1997
0.53
0.52
0.7765
0.53
0,505
0.83
ANIMEFACE suggest that our proposed upper bound helps increase the entropy of the proposal
distribution Pg-
ont-of-distribution detection As EBMs are density estimators, they should, in principle, assign low likelihood to out-of-distribution (OOD) observations. OOD detection performance can then be seen as proxy for the quality of estimated density. To test how well our model fares this regard. we follow previous work (Hendrycks and Gimpel [207 Hendrycks et al. 2019 Alemi et al., 2018 Choi et al. 2018 Ren et al. | 2019 et al. 2021) and report the threshold independent evaluation metries of Area Under the Receiver Operator Characteristic (AUROCT). Area Under the Precision Recall Curve (AUPRCT) and False Positive Rate at 80% (FPR80L), where the arrow indicates the direction of improvement of the metrics. The results reported in Table 4 Each column of the table takes the form "In-istribbttion -ftttf--distrbuution and test set, respectively. We observe that on CIFAR-I0 / SVHN, EBM-BB the top performer. On CIFAR-10 CIFAR- 100 the overall performance degraded significantly. EBM-0GP fared noticeable better than other models. The overall degradation may be caused by the strong similarity between CIFAR-10 10 and CIFAR-100. For ANIMEFACE, / Bedroom a similar situation occurs, even if these datasets are highly dissimilar, but we observe EBM-0GP is much better than other models.
reference to the training
No clear winner can be found from this study. We note that our method consistently performs well, but surprisingly, so does WGAN-0GP even if it is not density estimator. As OOD detection is task that comes with many subtle pitfalls (Havtorn et al. 202T), we suggest that these results should be taken with grain of salt even if our model performs well.

---

Running time Table 5 gives an overview of the time required to train the proposed models and the baselines. All models were trained on single 12GB Titan GPU. We observe that our model should be trained with a lower learning rate, and therefore may increase the total number of epochs.
Table 5: Total training time.
Model
Iterations
Runtime
CIFAR-10
WGAN-0GP
100000
4h
MEG
100000
4,5h
VERA
200000
10b
6
Discussion
EBM-OGP
200000
20h
ERM BB
200000
21h
ANIMEFACE
The first observation in our work is that current methods for training energy-based models (EBMs) interchangeably minimize and max- imize a lower bound. As this may be a potential source of training instability, we propose bound the negative log-likelihood from above and below and switch between bounds when minimizing and maximizing. The lower bound, [L(O)], is similar existing ones, but we provide a new algorithm for its realization. Unlike past work, this algorithm does not need additional networks and does not rely on hard-to-tune parameters: our only parameter controls the number of iterations for the Jacobian approximation and represents a trade-off between the accuracy of the bound and computational budget. The upper bound. [L)) is new to the literature but similar to the common regularization practice of introducing gradient penalties. To the best of our knowledge, this is the first time that gradient penalties have been derived from the perspective of bounding the log-likelihood. It is rewarding that current better engineering practice can be justified from probabilistic perspective. Empirically, we find our proposed model generally performs as well or better than some of the current ofate-of-the-art on variety of tasks. The evidence suggests that our bidirectional bounds allow the generator to increase its entropy and capacity usage. We see this both directly and through improved sample quality on diverse datasets.
WGAN-0GP
100000
13h
100000
VERA
200000
44h
EBM-OGP EBM-BB
100000
38h
100000
40b
The current drawbacks of the method are mainly three concerns. First, we find that a smaller-than-tsuaal learning rate helps our model. While we expect this to be matter of implementation rather than a more profound concern, the current implication that training is approximately twice that of EBM baseline. Second, our bounds rely on computing the smallest singular value of the Jacobian of the generator. We provide an efficient implementation a method for this task, but it needs to run for several iterations to guarantee convergence. In practice, we stop after a fixed, low number of iterations, which we found to work well, but technically this violates the bound. This seems to be unavoidable aspect of our approach, but find that the benefits significantly compensate for this issue. The availability of a fast (approximate) entropy estimator that does not require additional networks and parameters is highly valuable when training EBMs. Finally, for the upper bound, we need an estimate of the volume of support M of the proposal distribution Pg- We do not have viable method for this, and, in practice, we treat M hyper-paramen perameer We have not found it difficult to tune this parameter, but it nonetheless constitutes a limitation.
Limitations
Negative societal impact All high-capacity generative models carry the risk of being used for misinformation, and our model is no exception. The value of EBMs over GANs is that they come with likelihood function, which is more valuable in data analysis, than e.g. for creating deepfakes.
Acknowledgments and Disclosure of Funding
This work was funded in part by the Novo Nordisk Foundation through the Center for Basic Machine Learning Research in Life Science (NNF200C0062606), It also received funding from the European Research Council (ERC) under the European Union's 's Horizon 2020 research, innovation programme (757360), National science foundation of China under grant 61771305 and Shanghai Municipal Science and Technology Major Project (2021SHZZZO0I Nordisk Foundation (NNF200C006561)) 1) and the Independent Research Fund Denmark (9131- 00082B). SH was supported in part by a research grant (15334) from VILLUM FONDEN. The authors are grateful to anonymous reviewers and the handling area chair for valuable discussions and feedback carly version of this manuscript.
JF was supported in part by the Novo
10

---

References
M. E. Abbasnejad, Q. Shi, A. v. d. Hengel, and L. Liu. A generative adversarial density estimator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10782-10791, 2019.
M. E. Abbasnejad, Shi, A. van den Hengel, and L. Liu. Gade: generative adversarial approach to density estimation and its applications. International Journal of Computer Vision, 128(10): 2731-2743, 2020.
D. H. Ackley, G. E. Hinton, and T. J. Sejnowski. A learning algorithm for boltzmann machines. Cognitive Science, 9(1): 169, 1985.
A. A. Alemi, I, Fischer, and J. V. Dillon. Uncertainty in the variational information bottleneck. arXiv preprint XXXx:188070009000 2018.
M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
Y. Burda, R. Grosse, and R. Salakhutdinov. Accurate and conservative estimates of MRF log- likelihood using reverse annealing. In G. Lebanon and S. V. N. Vishwanathan, editors, Proceedings of the Eighteenth International Conference Artificial Intelligence and Statistics, volume 38 of Proceedings of Machine Learning Research, pages 102-110, San Diego, California, USA, 09-12 May 2015. PMLR.
T. Che, R. Zhang. J. Sohl-Dickstein, H. Larochelle, L. Paull, Cao, and Y. Bengio. Your GAN is secretly energy-based model and you should use discriminator driven latent sampling. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12275-12287. Curran Associates, Inc., 2020.
H. Choi, E. Jang, and A. A. Alemi. WAIC, but why? Generative ensembles for robust anomaly detection. arXiv preprint arXi::1810.01399 2018
Z. Dai, A. Almahairi, P. Bachman, E. Hovy, and A. Courville. Calibrating energy-based generative adversarial networks. In International Conference Learning Representations, 2017.
Y. Du and I. Mordatch. Implicit generation and modeling with energy based models. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
J. Frellsen, O. Winther, Z. Ghahramani, and J. Ferkinghoff-Borg Bayesian generalised ensemble markov chain monte carlo. In A. Gretton and C. C. Robert, editors, Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pages 408-416, Cadiz, Spain, 09-11 May 2016. PMLR.
W. S. Grathwohl, Kelly, M. Hashemi, M. Norouzi, K. Swersky, and D. Duvenaud. No MCMC for me: Amortized sampling for fast and stable training of energy-based models. In International Conference on Learning Representations, 2021.
R. B. Grosse, C. J. Maddison, and R. R. Salakhutdinov. Annealing between distributions by averaging moments. In C. J. C. Burges, L. Bottou, M. Welling. Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and C. Courville. Improved training of wasserstein GANs. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, pages 297-304. JMLR Workshop and Conference Proceedings. 2010.

---

T. Han, E. Nijkamp. X. Fang. M. Hill, S.-C. Zhu, and N. Wu. Divergence triangle for joint training of generator model, energy- based model, and inferential model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8670-8679, 2019.
J. D. Havtorn, J. Frellsen, S. Hauberg, and L. Maalee. Hierarchical VAEs know what they don't t know. In M. Meila and Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4117-4128. PMLR, 18-24 Jul 2021.
D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations, 2017.
D. Hendrycks, M. Mazeika, and T. Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019.
G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1711-1800, 2002.
G. E. Hinton and T. J. Sejnowski. Optimal perceptual inference. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, volume 448, 1983.
G. E. Hinton, S. Osindero, and Y.-W. Teh. A fast learning algorithm for deep belief nets. Neural computation, 18(7):1527-1554, 2006.
J. Ho. A. Jain, and P. Abbeel. Denoising diffusson probabilistic models. In H. Larochelle. M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1840-6851. Curran Associates, Inc., 2020. URL https: httpps:/prooceedingg neur sourips.oo cc/ httppppper/222/////e /ttpp////////fieee////eee
file/ /ttp////cbbbeb6aa000011000000010
pdf
J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554 2558, 1982. ISSN 0027-8424.
M. F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Staiisiix-Simmlltion and Computation, 18(3):1059-1076, 1989.
A. Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
M. Kang and J. Park. ContraGAN: Contrastive learning for conditional image generation. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21357-21369 3699. Curran Associates, Inc., 2020.
T. Kim and Y. Bengio. Deep directed generative models with energy-based probability estimation. arXiv preprint arXiv: /606.03330 2016.
A. V. Knyazev. Preconditioned eigensolvers-an oxymoron. Electron. Trans. Numer, Anal, 7: 104-123, 1998.
A. V. Knyazev. Toward the optimal preconditioned eigensolver: Locally optimal block preconditioned conjugate gradient method. SIAM journal on scientific computing, 23(2):517-541, 2001.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
A. Kumar, B. Poole. and K. Murphy. Regularized autoencoders via relaxed injective probability flow. In International Conference on Artificial Intelligence and Statistics, pages 4292-4301. PMLR. 2020.
R. Kumar, S. Ozair, A. Goyal, A. Courville, and Y. Bengio. Maximum entropy generators for energy-based models. arXiv preprint urXiv:1901.0880 2019.
Y. LeCun. The MNIST database of handwritten digits. htp://yann. lecun. com/exdb/nnist/, 1998.
Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and Huang. tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
12

---

Y. Li and R. E. Turner. Gradient estimators for implicit models. In International Conference on Learning Representations, 2018.
N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and Teller. Equation of state calculations by fast computing machines. The journal of chemical physics, 106 1953.
Miyato, Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
R. M. Neal et al. MCMC using hamiltonian dynamics. Handbook of markov chain monte carlo, (11):2, 2011.
T. Osogami. Boltzmann machines and energy-based models. arXiv preprint arXiv:1708.06008. 2017.
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang. Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation pytorch. NIPS 2017 Workshop Autodiff. 2017.
A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference Learning Representations, 2016.
Ren, J. Liu, E. Fertig, J. Snoek, R. Poplin, M. Depristo, Dillon, and B. Lakshminarayanan. Likelihood ratios for out-o--distribution detection. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
M. S. M. Sajjadi, O. Bachem, M. Lučié, O. Bousquet, and Gelly. Assessing Generative Models via Precision and Recall. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
R. Salakhutdinov and G. Hinton. Deep boltzmann machines. In D. van Dyk and M. Welling, editors, Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pages 448-455, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16-18 Apr 2009. PMLR.
R. Salakhutdinov and I. Murray. On the quantitative analysis of deep belief networks. In A. McCallum and S. Roweis, editors, Proceedings of the 25th Annual International Conference on Machine Learning (ICML 2008), pages 872 879. Omnipress, 2008
B. Scellier. A deep learning theory for neural networks grounded in physics. PhD thesis, Université de Montréal, Quebec, Canada, 2020.
J. Shi, S. Sun, and J. Zhu. A spectral approach to gradient estimation for implicit distributions. In International Conference on Machine Learning, pages 4644-4653. PMLR, 2018
Smolensky, Information Processing in Dynamical Systems: Foundations of Harmony Theory, page 194-28 191-281. MIT Press, Cambridge, MA, USA, 1986.
Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. In H. Wallach, H. Larochelle, A. Beygelzimer, F. ''Aché-Buuc, E. Fox, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
H. Thanh- Thng, T. Tran, and S. Venkatesh. Improving generalization and stability of generative adversarial networks. In International Conference Learning Representations, 2019.
B. Wang and C. Ponce. The geometry of deep generative image models and its applications. arXiv preprint arXiv:2101.0660 2021.
Y. N. Wu, J. Xie, Y. Lu, and S.-C. Zhu. Sparse and deep generalizations of the frame model. Annals of Mathematical Sciences and Applications, 3(1:2211-24,, 2018
J. Xie, W. Hu, S.C. Zhu, and Y. N. Wu. Learning sparse FRAME models for natural image patterns. International Journal of Computer Vision, 1422))91-112, 2015.
Xie, Y. Lu, S.-C. Zhu, and Y. N. Wu. Inducing wavelets into random fields via generative boosting. Applied and Computational Harmonic Analysis, 41(1):4-25, 2016.
13

---

J. Xie, S.-C. Zhu, and Y. Nian Wu. Synthesizing dynamic patterns by convnet. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 7093-7101,
generative
J. Xie, Y. Lu, R. Gao, and Y. N. Wu. Cooperative learning of energy-based model and latent variable model via MCMC teaching. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018a.
J. Xie, Y. Lu, R. Gao, S.-C. Zhu, and Y. N. Wu. Cooperative training of descriptor and generator networks. IEEE transactions on paltern analysis and machine intelligence, 42(():27-45, 2018b.
J. Xie, Z. Zheng, R. Gao, W. Wang, S.-C. Zhu, and Y. N. Wu. Learning descriptor networks for 3d shape synthesis and analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8629-8638 2018c.
J. Xie, Z. Zheng, X. Fang, S.-C. Zhu, and Y. N. Wu. Cooperative training of fast thinking initializer and slow thinking solver for conditional learning. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2021a.
J. Xie, Z. Zheng, and P. Li. Learning energybased model with variational auto-encoder amortized sampler. In The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI), volume 2, 202
S. Zhai, Y. Cheng, R. Feris, and Z. Zhang. Generative adversarial networks as variational training of energy based models. arXiv preprint vrXiv:/611.0170) 2016.
C. Zhu and D. Mumford. Grade: Gibbs reaction and diffusion equations. Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271), pages 847-854. IEEE, 1998.
S. C. Zhu, Y. Wu, and D. Mumford. Filters, random fields and maximum entropy (FRAME): Towards a unified theory for texture modeling. International Journal of Computer Vision, 1772610 1998.
14


Reviews:
[{"review_id": "obX5bpIAuj", "reviewer": "Reviewer_vzR3", "summary": "This paper proposes a bidirectional bound on the EBM log-likelihood, such that it can maximize a lower bound and minimize an upper bound when solving the minimax game of EBM. The paper demonstrates the effectiveness of the proposed method on density estimation and sample generation. The major contribution of the current paper is to use the bidirectional bounds to train the EBMs.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an important problem of training EBMs by using the bidirectional bounds to train the EBMs. The paper is easy to follow. And the results look promising.  The major contribution of the current paper is the usage of the bidirectional bounds.\n\nHowever, the connection of the paper to the existing prior works and pioneering works is pretty bad. A lot of related works are missing and a necessary comparison with some baselines methods is also missing.  Related works are pointed out below.\n(1)\tAbout deep generative EBM (from the second paragraph in the related work). There has been a long history for generative EBM for image modeling and generation. For example, [1] is the first EBM with MCMC, i.e., Gibbs sampling, for texture images. [2] is the first EBM using Langevin dynamics for texture images. [3] and [4] are EBMs using Gibbs and HMC for object images. The EBMs proposed in [1][2][3][4] are not deep model but can be still considered a two-layer CNN (because convolution or filtering is used). Of course, it is OK for the current paper to not connect them because they are not DEEP generative EBM.  However, the pioneering work that uses a modern deep neural network to parameterize the EBM and uses Langevin dynamics-based MLE to train the EBM is [5], and [6][7] have even generalized to video and voxel domains. The earliest deep generative EBM references cited in the current paper are from 2019, e.g., Du 2019 and Grathwohl 2021. It is not OK to skip the first EBM with deep CNN.   \n\n[1] Filters, random fields and maximum entropy (FRAME): Towards a unified theory for texture modeling. IJCV, 1998.\n\n[2] Grade: Gibbs reaction and diffusion equations. ICCV 1998.\n\n[3] Inducing Wavelets into Random Fields via Generative Boosting. Journal of Applied and Computational Harmonic Analysis (ACHA) 2015\n\n[4] Learning Sparse FRAME Models for Natural Image Patterns. International Journal of Computer Vision (IJCV) 2014\n\n[5] A Theory of Generative ConvNet (ICML 2016)\n\n[6] Synthesizing Dynamic Pattern by Spatial-Temporal Generative ConvNet (CVPR 2017)\n\n[7] Learning Descriptor Networks for 3D Shape Synthesis and Analysis. (CVPR 2018)\n     \n(2)\tAbout adversarial interpretation of EBM learning. The paper mentioned the minimax game of EBM or adversarial interpretation of EBM learning, but failed to cite the right paper or the first paper that pointing out this. [6] in 2017 is the first one to point out that. [7] and [8] also mentioned the adversarial game of EBM.  \n\n[8] Sparse and Deep Generalizations of the FRAME Model. Annals of Mathematical Sciences and Applications 2018\n\n(3)\tAbout learning a generator as an approximate sampler for EBMs (from the third paragraph in the related work). The current paper missed to discuss cooperative learning or CoopNet [9][10][11][12], which is about the joint training of EBM and top-down generator. In the cooperative training, the generator serves as an effective proposal distribution to efficiently evaluate the EBM. The current paper is related to the cooperative learning idea but didn’t cite or discuss it. \n\n[9] Cooperative learning of energy-based model and latent variable model via MCMC teaching. AAAI 2018\n\n[10] Cooperative learning of descriptor and generator networks. PAMI 2018\n\n[11] Learning Energy-Based Model with Variational Auto-Encoder as Amortized Sampler. AAAI 2021\n\n[12] Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning. (arXiv 2019 or PAMI 2021)\n\n(4)\tAbout baselines. Since the proposed model is close to cooperative learning. The cooperative learning [9] that trains an EBM with a generator and the EBM-VAE [11] that trains an EBM with a VAE sampler should be compared in your experiments.   \n\n\nAfter Rebuttal:\n\nIn the rebuttal stage, authors made efforts to fill the gap between the current paper and the prior works. I updated my score from 3 to 5.  I urge the authors to substantially revise their paper by including the discussion and comparison in their reply. To be specific, (1) in your introduction, build a connection with pioneering works about joint training of EBM and generator. (2) complete the related work parts for both single EBM and joint training of EBM and generator. (3) include the quantitative comparison with the baselines, such as CoopNets, etc in experiments.   \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Given the fact that the authors are not aware of the related literature and lack to compare their models to those closely related works. This might lead to a potential negative ethical impact.  I recommend a resubmission with a rejection.\n\nTo better understand the development and the advance of the EBMs, I suggest the authors review a new tutorial on EBMs in CVPR 2021 for a complete list of references, since the references I pointed out here are still not complete. This will be helpful to improve the current manuscript. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "48h"}, "extra_text": {}}, {"review_id": "mUcE8fTFg_c", "reviewer": "Reviewer_rhEU", "summary": "This work proposes a new method for density estimation using Energy-based model. Classical variational bounds replace the negative log-likelihood by a computable lower bound, and phrase the maximum likelihood problem as a minimax game on this lower bound. A possible algorithm to solve the problem is then to alternate between minimization and maximization of this lower bound (with respect to different variables). Although this approach has shown good performance, the authors claim that the concept of minimizing such a lower bound can lead to instabilities. In particular, they claim that the WGAN objective tends to be unstable to optimize, and that while clipping the gradient in this case solves the instability issue, the origin of the instability lies in minimizing an upper bound. To circumvent this issue, the current work constructs both an upper and a lower bound to the negative log-likelihood, and alternate between minimizing the upper bound and maximizing the lower bound. The upper bound is inspired by what is done in WGAN, i.e., by adding a gradient penalty. The algorithm is tested on several benchmarks, and is shown to be rather competitive compared to existing algorithms.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors claim that the reason for adding gradient penalty in WGAN is to stabilize the training. However, the reason for this penalty comes from the problem formulation of learning a distribution by minimizing the Wasserstein distance. By Kantorovich duality, this leads to a minimax game with Lipschitz constraint, which is enforced by using either gradient clipping or gradient penalty. The use of gradient penalty in WGAN is hence very well motivated. In contrast, the use of gradient penalty in the current framework seems much more abstract than in WGAN.\n\nThe proposed algorithm bears strong similarities with WGAN with gradient penalty. The empirical evaluation actually shows quite similar performance, although the proposed method takes about twice more time to train (Table 5).\nHence, I don’t see the benefit of this approach compared to WGAN, both in theory and in practice.\n\n\nTypos:\n\nl.54 tends\n\nThm 1 there exist\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}, {"review_id": "_0V5nFc2PAA", "reviewer": "Reviewer_y6Eh", "summary": "This paper proposes a method to train EBMs with lower and upper variational bounds. Previous approaches used only one side of the bound. This causes significant training instabilities as the training objective involves alternately maximizing and minimizing the same quantity.\n\nThe lower bound is estimated by lower-bounding the log determinant of the jacobian multiplied with itself with the smallest singular value of the jacobian. \n\nThe upper bound is inspired by tricks used to make WGAN and related EBM training approaches stable by regularizing the gradient. \n\nThe minimum singular value, needed for computing the lower bound, is computed using an iterative algorithm run for a small number of steps. Automatic differentiation is used cleverly to ensure full Jacobians don’t need to be instantiated.\n\nThe upper bound is loosened, and Hutchinson’s trace estimator is employed to efficiently compute. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Describe the strengths of the work:\n\nThe authors develop a fast and stable approach to optimize EBMs like a GAN. They compare to strong baselines from both the EBM and GAN literature. They compare models on large image datasets. The authors perform an important ablation where they remove their upper bound and use the simpler gradient penalty method.\n\nThe authors develop a theoretical justification for a previous heuristic that was found empirically to significantly stabilize training. \n\nThe authors investigate the utility of their approach on a number of tasks including toy data, where density modelling can be compared exactly, mode coverage on MNIST, and sample quality and OOD on image data. \n\nThe authors give an explanation of some future work and open questions about their method.\n\nExplain the limitations of this work:\n\nThe authors claim that their approach is better because it doesn’t require additional networks or rely on hard-to-tune parameters. It might be interesting to scale their approach up to larger images, for example 128x128 or even 256x256 images, but I understand this is difficult especially given the compute used as stated in the paper. \n\nCorrectness:\n\nThe authors carefully empirically verify a number of their claims. In Figure 3, they show the quality of their entropy estimator on a tractable problem.\n\nClarity:\nThe writing is clear.\n\nRelation to prior work: \n\nThis work clearly describes how it builds on existing work by training models with a bidirectional bound, while previous approaches used only a bound in one direction, and a gradient regularization.\n\nReproducibility: Are there enough details to reproduce major results of this work\n\nDetails of the architecture, optimization, and other hyperparameters are provided in the appendix.\n\nAdditional feedback, comments, suggestions:\n\nPlease include details of data processing the appendix. For example, is the data normalized to be in [0, 1] or [-1, 1]? Was data dequantization used? What augmentations were used? What were the train-test splits used for optimizing models?\n\nWhy does the rank-order of timings in Table 5 change between the two experimental setups?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "GmIJgaP8giR", "reviewer": "Reviewer_xLGS", "summary": "The authors propose to sandwich the EBM log-likelihood with newly introduced upper and lower bounds, with which a minimax objective is formed to train EBMs. \nSpecifically, based on the variational lower bound from Grathwohl et al. (2021), the authors approximate the entropy term therein to form the presented lower bound, and then adding a gradient-penalty term mimicking WGAN to form the new upper bound. \nThe lower and upper bounds are optimized wrt different groups of parameters, as, intuitively, both minimizing and maximizing a lower bound wrt the same group of parameters is potentially unstable (this is the main motivation for this submission).", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality. Several techniques used to develop the upper/lower bounds are believed sound and novel; they might be useful to audiences from different domains. As for related works, important discussions associated with popular methods for training EBMs (e.g., [1,2]) are believed missing, as well as the empirical comparisons.\n\nQuality. Minor flaws may exist in some presented techniques: (1) Eq. (8) may not hold for general GAN generators; a GAN generator is not an injective function in general, which means the matrix J_z^T J_z may not be invertible. Please elaborate on that. (2) How to guarantee that the condition in Line 86 is satisfied in practice? Please add the related analysis. (3) Is Eq (15) guaranteed to be an upper bound of the log-likelihood? \n\nClarity. The submission is well-written and easy to follow.\n\nSignificance. The training of EBMs is a popular research direction that is of great interest to the ML community. However, I am not fully convinced by the presented method, because of the missing comparisons with [1,2] and the minor flaws mentioned above.\n\nOther comments:\n\nIn Line 49-50, why \"both minimizing and maximizing a lower bound is potentially unstable\"? Also, what's the ''conceptual issue''? These are not clear in the current submission.\n\nIs Eq. (10) \"the new and efﬁcient estimator of the Jacobi-determinant of the EBM generator,\" mentioned in the Abstract? That contribution is not clearly stated. \n\nSince the lower bound in Eq. (11) is not tight, discussions on the gap are necessary. Also, how expensive is the evaluation of Eq. (11)?\n\nI am concerned about the training stability. How stable is the training empirically? \n\nIt seems Eqs. (20) and (15) show different upper bounds respectively. Which one is actually used in the experiment? Why?\n\nIn the experiments, does the proposed method perform better than the popular methods for training EBMs [1,2]?\n\nMinors:\n1. In Line 12, the abbreviation ''EMBs\" is a typo.\n2. In Line 53, the statement \"the missing entropy term being the only difference\" may not be right, since the function E_{\\theta}(x) in Eq. (6) must be a 1-Lipschitz function.\n\n[1] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11895–11907, 2019.\n[2] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33, 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main contribution of the paper is the development of lower and upper bounds for training EBMs in a stable manner, by drawing inspiration from the WGAN objective. The methodology presented in the paper can be useful for the research community. The reviewers raised some concerns about the paper that were adequately addressed during the rebuttal phase. I therefore recommend the paper for acceptance, but strongly encourage the authors to address the following points for the camera-ready version:

+ Rephrase the points about the instability & grad clipping of WGAN, as discussed with reviewer rhEU.

+ Add the experiment details to the appendix, as suggested by reviewer y6Eh.

+ Add some discussion and references in the related work section. In particular, it'd be useful to include as much of the similarities and differences with previous methods (from the response to reviewer vzR3) as possible.

+ Add some clarification about the invertibility of the Jacobian determinant and the change-of-variables formula, as discussed with reviewer xLGS and myself. In particular, the assumptions under which Eq. (8) holds should be clearly stated in the paper - these are pointed out by the authors in their rebuttal, but some additional questions are: Does this limit the architecture of the generative NN? For example, does each hidden layer need to have a number of hidden units between $d$ and $D$, with non-decreasing sizes along the layers? Is any non-linearity valid? (e.g., ReLu's map many input values to 0, so it shouldn't be used). If any assumption is made, it should be clearly stated in the final version. In addition, the text should clarify that $p(G(z))$ in Eq. (8) is *not* the Lebesgue measure on $\mathbb{R}^D$ but it is restricted to some low-dim manifold.

+ Incorporate the experiments that were ran during the rebuttal period.

Besides the items above, I also have two technical questions that should be clarified in the paper:

+ Below Eq. (20), the paper says that $\zeta=1$ allows for the simplified bound from Eq. (15). Shouldn't it be $\zeta=M$ instead, which corresponds to $m=1$ in Eqs. (13)-(15)?

+ Theorem 2 guarantees the existence of constants ($m$, $M$, and $p\geq 1$) such that the bound in Eq. (13) holds, but unless I'm missing something, it doesn't say what values of the constants guarantee the bound in Eq. (13). But, according to the rebuttal, Eq. (15) is a valid bound for any $p\geq 1$ (as long as the "expression$\geq 1$" assumption is met). So, even though it was obtained from Eq. (13), Eq. (15) makes an explicit statement about what values of the constants constitute a valid bound. How is such discrepancy possible? Is the "expression$\geq 1$" assumption the explanation for that? Does Theorem 2 actually say something about the values of the constants that guarantee the correctness of the bound?

Decision: Accept (Poster)