Learning Probabilistic Models from Generator Latent Spaces with Hat EBM
Authors: Mitch Hill
Erik Nijkamp
Jonathan Craig Mitchell
Bo Pang
Song-Chun Zhu
Venue: NeurIPS 2022 Accept (2022)
Keywords: EBM
latent sampling
generator network
MCMC
Langevin

Abstract: This work proposes a method for using any generator network as the foundation of an Energy-Based Model (EBM). Our formulation posits that observed images are the sum of unobserved latent variables passed through the generator network and a residual random variable that spans the gap between the generator output and the image manifold. One can then define an EBM that includes the generator as part of its forward pass, which we call the Hat EBM. The model can be trained without inferring the latent variables of the observed data or calculating the generator Jacobian determinant. This enables explicit probabilistic modeling of the output distribution of any type of generator network. Experiments show strong performance of the proposed method on (1) unconditional ImageNet synthesis at 128$\times$128 resolution, (2) refining the output of existing generators, and (3) learning EBMs that incorporate non-probabilistic generators. Code and pretrained models to reproduce our results are available at https://github.com/point0bar1/hat-ebm.

Learning Probabilistic Models from Generator Latent Spaces with Hat EBM
Mitch Hill OPPO US Research Center mitch. inw.eiintnpaaa.cc...cc com
Erik Nijkamp Salesforce Research er erik frri.aiikkkmppsleeefforooo. com
Jonathan Mitchell
Bo Pang
University of California, Los Angeles j @ciitchell@ucla.
Salesforce Research
b. wwwg@sallesoorre.ccommmmm com
Song-Chun Zhu
University of California, Los Angeles
sczhudstat, cachuustat.ucla.eee.. ucla.edu
Abstract
This work proposes method for using any generator network as the foundation of Energy-Based Model (EBM). Our formulation posits that observed images are the sum of unobserved latent variables passed through the generator network and a residual random variable that spans the gap between the generator output and the image manifold. One can then define an EBM that includes the generator as part of its forward pass, which we call the Hat EBM. The model can be trained with- out inferring the latent variables of the observed data or calculating the generator Jacobian determinant. This enables explicit probabilistic modeling the output dis- tribution of any type of generator network. Experiments show strong performance the proposed method on (1) unconditional ImageNet synthesis at 128 128 reso- lution, (2) refining the output of existing generators, and (3) learning EBMs that incorporate non-probabillstic generators. Code and pretrained models to reproduce our results are available https: htt///ggtttt//t www hamp/oonttbarr....mmmmmm /hat ebm.
1 Introduction
Generator networks [20, 12] which transform a latent distribution to complex observed distribution (e.g., images, videos) are found across different deep generative models. One limitation of generator networks is the difficulty of obtaining an explicit representation the probability distribution defined by the output of the network after transformation of the latent space. For generators from Generative Adversarial Networks (GANs) [12, 32] and Variational Autoencoders (VAEs) [20, 33] where the latent states corresponding realistic images follow trivial distribution (e.g., isotropic Gaussian), the difficulty of obtaining image space probabilities lies in calculating the log determinant of the generator Jacobian which is needed perform density change-of-variabee a deterministic autoencoder, there might not be a natural way to generate probabilistic samples from the latent space that correspond to realistic images. When the latent space has smaller dimension than the image space, there is further difficulty in describing the generator image distribution since the distribution measure must be confined the manifold of the generator output.
For other generator models such as
This work proposes method for using a generator network as the foundation for an Energy-Base -aaed Model (EBM). The generator network concatenated with hat network that takes an image input
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

G
X=Y+G(Z)
(
Energy (xx;0)
Figure 1: Left: The Hat EBM model takes a joint input (Y, Z) where Y is residual image and Z is a latent vector. An image is generated using X = Y G(Z) for a generator GZZ), and the image is passed to the hat network H (X;0) to obtain the energy of the pair (Y, 2). This allows for principled probabilistic learning which can incorporate the latent space of any generator. Right: Unconditional ImageNet 128x x 128 samples generated by Hat EBM.
and outputs a scalar. Before the generator output is fed into the hat network, the generated image is adjusted by adding residual image that spans the gap between the generator output and the image manifold. The total function, including the generator, addition of the residual image, and the hat network, called the Hat EBM. This formulation allows to define an EBM which can incorporate the generator latent space part of its image space density and MCMC sampling process. Figure displays a diagram of the Hat EBM and selected Hat EBM samples for unconditional ImageNet at resolution 128 128. Our method is general and applies to any generator model, in contrast to existing methods converting generator networks to EBMs which only apply to specific generator model like GAN [5] or VAE [38]. Additionally, the learning method does not require inference of latent states of the observed data as used related works [20, 39, 15, 30].
The Hat EBM formulation enables applications with frozen pretrained generator networks such as learning an EBM refine samples from generators with probabilistic latent spaces (e.g. GANs) and synthesizing samples using ooo-pobbaiilistic generators (e.g. deterministic autoencoders). We also propose self-contained learning method that extends cooperative learning EBM and generator networks [39] to achieve high-quality synthesis. Our main contributions are summarized below.
We introduce method for defining a Hat EBM that incorporates a generator network as part of its forward pass. This EBM includes the generator latent space part of MCMC sampling.
We show that our method can refine samples from pretrained GAN generators and sample from the latent space deterministic autoencoders which are originally incompatible with sampling.
We propose self-contained Hat EBM learning method that trains both generator and energy network from scratch. This enables us to achieve an FID score of 29.2 on unconditional ImageNet resolution 128x 128. demonstrating the EBMs competitive with state-of-ther -fte--------tttttt generative models on complex and high resolution datasets.
Related Work
This section highlights important related methods for EBM learning. A more thorough discussion of related work can be found in Appendix
EBM. An EBM defines an unnormalized density or equivalently Hiibs-Boltzmann distribution. Prototypes include exponential family distributions, Boltzmann machines [1, 34], and the FRAME (Filters, Random field, And Maximum Entropy) model [44]. Recent work has introduced the EBM

---

with ConvNet potential [43, 42]. This dramatically increases the model capacity and enables strong image synthesis performance [29, 9] and adversarial robustness [16]. Several works investigate training an EBM in tandem with an auxiliary model. Kim and Bengio [19] jointly train an EBM and generator without MCMC by using samples from the generator as direct approximations of the EBM density. The EGAN [7] builds on this method by introducing a maximum entropy objective to improve generator training. A similar approach is explored by the VERA model [14]. Cooperative learning [41] trains the EBM and generator by using the generator to initialize samples needed to train the EBM and uses reconstruction loss between generator and EBM samples to learn the generator.
The Flow Contrastive EBM [11] learns an EBM using Noise Contrastive Estimation with an auxiliary flow model.
Latent Space EBM. EBMs in the data space can be highly modti, and MCMC sampling can be difficult [40. 29, 9]. Recent works [30. 31] explore learning an EBM in latent space, which is then mapped to the data space with learned generator. The energy landscape in the latent space smoother and less multi-modal because it occupies much lower dimensional space and stands upon an expressive generator. These works define a prior EBM in the latent space correction of the non-informative uniform prior or isotropic Gaussian prior. To learn the model, one needs to infer the posterior of the latent variables. Posterior inference given such complicated model is non-trivial. One needs to either design a sophisticated amortized inference network or run expensive MCMC. Our model also defines an EBM in the latent space, while its learning does not need posterior inference, making the learning much simpler and more scalable. Several works [36, 5, 2] leverage a pretrained GAN to define EBM in the latent space of the generator with correction based on the discriminator, and shows improved synthesis quality over the pretrained GAN. The VAHBM [38] uses a pretrained VAE to facilitate EBM learning. Our model can likewise be utilized to improve the quality of images from pretrained GAN or VAE generators. Our method is however more general since it can be used to obtain realistic samples from any pretrained generator, including non-probabbllisiic generators from deterministic autoencoders.
3 Formulation of Hat EBM
This section presents the formulation of the Hat EBM energy function and the proposed learning procedure. We first review the fundamental equations of EBM learning. Then we introduce two variants of the Hat EBM: one that learns joint distribution over the latent space and residual image space, and one that learns a distribution of residual images conditional on a fixed latent state from a known latent distribution. Finally. we propose method for learning the hat network and generator network of a Hat EBM simultaneously so that our model can be used for self-contained image generation without the need for pretrained generator.
3.1 Review of EBM Learning
We briefly review the main components of EBM learning following the standard method derived from works such as [17, 44, 40]. A deep EBM has the form
p(a; 0) 2(0) (x;(-U(x;00)
(I)
where U(x; 0) deep neural network with weights and Z(0) is the intractable normalizing constant. Given a true but unknown data density g(x)) Maximum Likelihood learning uses the objective arg ming D (9(x) || p(x;6)), which can be minimized using the stochastic gradient
x
VoU(Xt:0) 1 (XU(XX;
VC(0)
(2)
7
where the positive samples {X;+ Ji=1 are a set of data samples and the negative samples {X; } are samples from the current model p(x; 0). To obtain the negative samples for a deep EBM, is common to use MCMC sampling with the K steps of Langevin equation
s2 V (X ;) + eVe,
x(++1) = x (k)
33)
2
where is the step size and V (0, I). The Langevin trajectories are initialized from set of states {X, 1 obtained from certain initialization strategy.
in
3

---

3.2 Hat EBM: Joint Distribution of Latent and Residual Image
This section presents our method for adapting fixed generator network G(2) to be part of an EBM, which we call the Hat EBM. The Hat EBM defines the joint distribution of the random variable Z E IR' in the m-dimensional latent space of the generator network and a random variable Y R in the d-dimensional image space. The joint energy has the form
U(Y, 2;0) = ((G(Z)+Y;0))
(4)
where (x;; 0) is a neural network that takes an image x E IRd as input and returns a scalar output. The weights of H are given by 0. We call the hat network because sits atop the generator G to incorporate the generator latent space directly into the probabilistic model.
The random variable Y is meant accommodate the gap between the output of G(Z) and the image manifold, since in general we expect that G(Z) contains an approximate but imperfect representation of the image distribution which can be refined by the residual state Y. In practice, we find that the majority of the appearance of a sampled image X = G(Z) + Y comes from the generator output (Z) and not the residual image Y. indicating that the majority of the sampling behaviors of our model are determined by the latent space of G (see Figure 3).
An appealing aspect of the Hat EBM formulation is that calculating the log determinant of the Jacobian of Z(Z) as would be required for an energy of the form (Z): 0), or inferring the Z latent vectors associated with observed images X done in existing work on learning latent prior EBMs [30]. This is possible because we define the distribution of observed images X by X G(Z) + Y where the pair (Y, 2) is drawn from a true but unknown density g(v, 2). We can then use the Maximum Likelihood framework in Section 3.1 to learn the weights 0 of the hat network H () by minimizing the objective arg ming ( 2) Ip(y, 2; 0)) where
learn the model without either
1
ply, 2) = Z(0) exp ( +
(5)
One can obtain negative samples using alternating Langevin updates
y(++))_ = =((k)
V HycaH(((22)))+ (k), + VV,
(6)
2 Sh
2(++)= Z(k)_
V 7a ) + (x+1)); + V2.12
(7)
which switch off between updates with respect to and updates with respect to y. This sampling algorithm is essentially Mctropolis- withinn within-Gbsss since the Langevin update can be written as a Metropolis-Hastingg step with Gaussian proposal, in which case (6) and (7) define valid sampler for p(u, z; 0). Finally, updating 9 can be accomplished using
n
n n Σ VOH(G) + +Y;;0)
VC(0)
n Σ (oH(X
(8)
where X are observed samples and the pairs (Y;- ; are obtained via MCMC. In our formulation the observed data X; are sufficient statistics for H (a:; f) and there is no need to infer the (Y;+, 2 pairs for the positive samples when learning the weights of the hat network.
3.3 Conditional Hat EBM: Residual Image Conditional on Latent Sample
Next we present a conditional variant of the Hat EBM. While the previous version of the Hat EBM applicable to any generator network G(z), including generators from deterministic autoencoders which cannot typically be sampled from, the conditional version of the Hat EBM is tailored towards generator networks which map a trivial latent distribution to complex signals like images. For these kinds of generators, one can use the known latent distribution as ancestral distribution and learn conditional distribution of the residual image given a latent sample. We emphasize that the conditional Hat EBM can be used to learn an unconditional distribution of observed images X and that the term conditional refers to the relationship between the latent variables Y and Z. Our experiments focus on modeling only observed images X without conditional information such as labels or captions.

---

Suppose we use trivial marginal distribution p=(2) for Z and generator G trained to produce realistic images from this latent distribution. In our experiments, Po is always NN, 1). We can now define a conditional Hat EBM density p(y]2; 0) = ズ exp (G(()) + y: d)} and a joint density
1 pl, 2) = (0) po -
In this case, we posit that observed images X are generated according to X G(Z) + Y for some distribution q z) (u)=))(u)=). Obtaining the negative samples (Y,)) is done by first drawing Z from Pu and then obtaining Y, 2 by using Langevin updates the conditional probability plyl 2z 9). Note that plyl 2; 8) = ply| (z)) because of the form of (9). We update using the same equation (8) as the joint Hat EBM because X is still a sufficient statistic for learning (2; 0) and because we do not need to infer the z for X;* since the prior po(=) does not contain any model parameters. In practice we initialize Langevin sampling from Yo = 0 and Z ~ (0, I), and perform K steps of (6) to draw a residual sample YK while keeping Z fixed.
3.4 Learning the Hat Network and Generator in Tandem for Conditional Hat EBM
Both formulations of the Hat EBM above assume that a pretrained generator network G(z) is available as the basis for learning the Hat EBM. In order use the Hat EBM as self-contained learning process for image generation, we now propose method to learn the weights & of a generator network G(z; o) and the weights 0 of a hat network H (z; 8) simultaneously. Our method is based on the cooperative learning [39] strategy. We first briefly review the cooperative learning formulation, and then present the learning formulation for the Hat EBM. key difference between the derivations is that the original cooperative learning method requires MCMC inference of the latent variable ž associated with an MCMC sample X to train G(z; o). while in our formulation the latent variable Z is explicitly defined and does not need to be inferred. This enables major computational savings because we can bypass the expensive MCMC inference of Ž.
In the original cooperative learning [39], the generator output G(2; is trained to match the appearance of Langevin chain Xx sampled from the potential Ua:; 8) and initialized from the state Xo = G (o: お) where Zo ~ N (0, I). The model defines the conditional density of an images X given latents Z ~ (O, as X|Z ~ NG( Φ), 1²1) for some sufficiently small r >> O. Given a sampled state Xx. updating Giz;; Φ) requires inferring Z| X|Xx x using the latent Langevin equation
Zx+1 2x
Zx + 272 v.
1 (G( - Xx13 + +
(10)
2
before updating o using the Maximum Likelihood stochastic gradient
(o(()) ~ 1 Σ 1=1
1 2T
Vol
Xx.ll
(1I)
where the index denotes member of a batch with size n. We note that the code released with the cooperative learning method does not infer the latent variable of observed images and Zo ~ N I) is used in place of Z in the objective (11). In accordance with this approach, we find that inferring Zx often hurts model performance and leads to additional complication. The difficulty of inferring Z|Xx k and the omission of this step in practice leave an unresolved gap in the cooperative learning formulation. The Hat EBM generator update allows us to bypass the Langevin update for Z without theoretical complications.
To update the weights of a Hat EBM generator, we propose to train the generator to match the image samples produced by the current Hat EBM. Given the current generator weights oi and hat network weights 6r at step t, we define our learnable model as Z ~ NO,0 T) and
X N(G o), T2)) and we define the true distribution of (X, Z) as Z N( 7) and + (z) where Yx ZZ is drawn from K steps of Langevin sampling with Hat EBM
YK x= density p(y|z; 0z, Φe) x exp(-H(G(( Φx) + y; 0.)]. Langevin updating is only used to obtain Yx while Z remains fixed, as in the method from Section 3.3. Then e can be updated using the Maximum
Likelihood objective
1
Φ++1 arg min
F (epo(c)plll [[[G(Z;)) (Y
(12)
272
5

---

2 ~ N(0
Y2 0
G(Z:o)
K step Langevin p[Y 0):0)
Update wish (12)
x
x
(3(3:
Update 0 with (8)
Figure 2: Visualization of landem training method for hat network and generator. The left side illustrates training for the hat network H (a; 0). Z is drawn from a latent distribution, Y is initialized from the 0 image and updated according p Z: f, Φ) = p>yp(((a (Z; 6): 0). Then data samples X and negative samples X
Y + G(Z; Φ) are used to update the weight A of the hat network. On the right, pairs (X' from past hat network updates will be drawn randomly from a bank of states to update the weight @ of the generator. The bank memory (X', Z') will then be overwritten by new pair (X, 2) from the current model.
Conceptually, this loss function encourages G(Z; to closely match the appearance of samples =- G(Z Φx) created from fixed generator G(Z; Φx) and fixed hat network H (z; 0,). In other words, the generator update should achieve GZZ; 11+1) Y + G(Z; d.) so that the updated generator absorbs the residual image Z from the current Hat EBM. Once the generator absorbs the current Hat EBM residual, the hat network update should learn to synthesize residual images that refine the the generator output to be more similar to the observed data. Like the original cooperative learning method, our generator is trained using only synthetic images and no observed data images used to update Φb
Ideal training of H and would alternate between using the gradient of (12) until generator con- vergence is reached and using the gradient (8) to update the hat network. In practice we implement the minimization in (12) using only one gradient update initialized from e = Φe to obtain é rather than training until full convergence. This is done to increase training efficiency and to avoid maintaining a separate copy of generator weights for the fixed network G(z; Φz).
In our experiments we observe that using single gradient update with the objective (12) has limited success because the generator output can become too closely tethered biases of the current hat network. We find the same problems with the original cooperative learning objective (11) (see Appendix F). To overcome these problems, we choose to train G(2; () at time t ++7 to match the historical distribution of hat networks Hx;; 0,,) and generators G(z; Ø1.) for a selection of past epochs 1,, .2. S instead of training the generator to match the distribution of the current hat network H (x;; 02) and generator G(z; Φr). This simply involves redefining the true distribution (X, Z) by first sampling 4, from (4,..., (I.)) and then generating
N 7) and Xx = Yx + G(Z ).)) where Z follows the energy HGG;Z; sh,,) + Y; 0.2). In this case the loss (12), after replacing oe with &t,, is a stochastic approximation of the Maximum Likelihood gradient defined by the joint distribution (tz, Z, .x) (see Appendix F). In practice,
implement this procedure by keeping persistent bank of 10,000 pairs (X, Z) created from past hat network updates. When updating the generator, draw n 128 pairs from the current hat EBM. This ensures that the selection fti,..., +t) of past epochs remains within close range of the current epoch i with high probability. Saving the generated images x at each EBM update allows us to learn from past generator weights ét, without maintaining a copy of the weights. See Figure 2 for illustration and Appendix G for a code sketch.
128 pairs from the bank and replace it with newly generated batch of
4 Experiments
In the subsequent empirical evaluations, will address the following questions:
1. Refinement: To what extent can method refine samples from pretrained generator model with known prior distribution?
Is our method capable of turning a generator model pretrained as a deterministic autoencoder into generative model for which samples resemble realistic images?
Retrofit:

---

3. Synthesis: Can our method learn a generator from scratch with competitive quality of synthesis on common image datasets? Can our method be scaled up to challenging datasets such as ImageNet with competitive synthesis for unconditional sampling?
4. cot-of-Distribution: Can the Hat EBM be used for Out-of-Distributt (OOD) detection to dis- tinguish between samples from the training distribution and samples from dissimilar distributions?
Figure 3 visualizes some representative sampling paths for the models trained to investigate questions 2, and 3 on the CIFAR- 10 dataset. The appendix contains details such pseudocode, training parameters, and model architectures.
4.1 Refinement
In this section, we examine the problem of refining samples from a pretrained generator using a joint Hat EBM trained according to the method in Section 3.2. A pretrained SN-GAN with generator used baseline generator G. We learn Hat EBM that incorporates the generator network to refine the initial generator samples. The hat network has the exact same structure as the SN-GAN discriminator except that we remove spectral norm layers. We use fixed batch norm statistics for the generator and our energy is deterministic. The experiment is performed for the CIFAR-10 10 32 x 32 and CelebA 64 x 64 datasets. To evaluate our method, we compare with Discriminator Driven Latent Sampling (DDLS) [5], which takes the pretrained discriminator D learned with G' and samples from the potential (z)) = ((G())) + 2||=|[². We train D and G using the Mimicry repository for reproducible GAN experiments [22] to obtain stronger baselines for G(z) than those used in [5].
Table shows our results. The Hat EBM can learn a hat network capable of refining image appearance more successfully than DDLS on both CIFAR- 10 and CelebA. We use the joint version of the Hat EBM where both Y and Z are updated during sampling. Sampling is initialized from Y,- = and Zo NO0, I). The hat networks learns to tilt Z away from its initial normal distribution to find a nearby latent vector with a more realistic appearance.
Table 1: Improvement in FID by refining samples from fixed SN-GAN generator.
Model
CIFAR-10
CelebA
SN-GAN (baselinc) [26]
18.58 + 0.08
6.13 + 0.03
DDLS [5]
14.59 + 0.07
6.06 = 0.01
Hat EBM (Ours)
14.04 1 0.11
5.98 1 0.02
We observe that the majority of the refinement is occurring in the latent space, and that the residual image is essentially imperceptible (see Figure 3). We also notice that training quickly becomes unstable when Y is removed from the Hat EBM (see Appendix H). In our experience, it is essential to incorporate the residual for stable learning. hat network can learn to discriminate between generator images and images not from the generator, whether they are realistic or not. If so, the hat network can assign increasingly high energy to generator samples in the absence of the residual Y. Even an imperceptible Y is sufficient to prevent the hat network from easily distinguishing positive and negative samples so that learning is stable.
possible reason for this phenomenon that the
4.2 Retrofit
In this section, we incorporate non- con-probabillsttc generator network G(z) into a probabilistic Hat EBM model. This essentially allows us to sample from the latent space of G(z) to find latent samples whose mapping corresponds to a realistic image. Like the results in Section 4.1, the residual image has small norm in the image space and most of the appearance of the sampled images comes from G(z). This happens naturally without the need to coerce Y to be close to 0 by including a prior term such as po(y) x exp(- 20 [lvll)). although including prior term can further limit growth of Y.
The autoencoder generator G(z) is pretrained as the second half of a standard inference network and generator network pairing. An image X is fed into the inference network I and converted into a latent state Z I(X), which is then decoded to obtain reconstruction X G(Z). The inference network and generator are learned jointly using the MSE loss || X133. To keep the latent space mapping
7

---

Refinement
Retrofit
Syuthesis
X
X
X
((2)
G(Z)
G
Y
Y
G(Z)
G(2)
G(2)
X
X
((2)
G(Z)
G
Langevin Steps +
Langevin Steps
Langevin Stege
Figure Visualization shortrun Langevin paths used by each kind of Hat EBM. The same trajectories are used for both training and test-time generation. In each grouping, the three rows correspond image X, generator output GZZ), and residual image Y. All paths are initialized with residual Yo = 0 which corresponds to grey image. For visualization purposes, all Y images have been magnified by a factor of 5. The Refinement paths (left) use the joint Hat EBM to refine the appearance of a pretrained SN-GAN generator. The residual Y is barely noticeable, but still is essential for stable learning. The Retrofit paths (center) use the joint Hat EBM to sample using a non-probabilistic autoencoder generator. The residual Y is somewhat noticeable after magnification but most the image appearance comes from G(Z). The Synthesis paths (right) use the conditional Hat EBM to sample refine generator learned in tandem. G(Z) is fixed during sampling. The residual plays a significant role in refining the generator appearance.
of I (X) numerically stable, we project the m-dimensional raw output of the inference network to the sphere around the origin with radius /m so that [|X(X) vm. More sophisticated methods such as perceptual and adversarial loss could have been used to train the autoencoder, but we use
MSE loss to keep our implementation minimal. We observe that when Z is vector-shaped latent state, can be difficult to learn reconstructions X with sharp appearance even for simple datasets like CIFAR-10. To obtain better reconstructions and therefore latent space with more realistic mappings to the image space, we use imagc-shaped latent states Z. The details of our autoencoder networks can be found in Appendix E. When Z is an image shaped latent, treat it exactly the same vector latent in the learning and sampling algorithms.
We experiment with assimilating autoencoder generator into Hat EBM potential for the CIFAR- 10 dataset. Our results are visualized in Figure 3 with additional results the Appendix We train the Hat EBM using shortrun learning in the latent and image space by initializing Yo = 0 and Zo ~ (O, 7) and using = 100 MCMC steps of (6) and (7) from initialization during both training and testing evaluation generate samples. During the Langevin dynamics, image appearance is refined mostly in the latent space. Our best model achieves a solid FID score of 26.01 + 0.09. This demonstrates that Hat EBM can learn probabilistic model over non-probabilistic latent space.
4.3 Synthesis
In this section, and generator network from scratch for self-contained synthesis. We explore synthesis for uncondi- tional CIFAR-I0 10 at resolution 32 32, CclebA at resolution 64x64, and unconditional ImageNet at resolution 128 128. While recent generative models show promising results for class conditional sampling, unconditional sampling with high quality synthesis remains significant challenge. We find especially strong results for unconditional ImageNet synthesis using the Hat EBM. This demonstrates strong potential of our synthesis method for learning with highly diverse unstructured datasets.
the conditional Hat EBM formulation from Section 3.4 to learn hat network
Our Hat EBM models use SN-GAN architectures sizes 32 x 32, 64 x 64, and 128 x 128, where the discriminator architecture is used for the hat network. During learning, we keep the generator batch norm parameters fixed to mean and variance 1. We remove all spectral norm layers from the hat network. Training parameters can be found in Appendix J. For ImageNet models, we found that

---

Table 2: Comparison of FID scores among representative generative models. For CIFAR- and Celeb-A, EBM models are above the dividing line and other generative models are below. For ImageNet, smaller scale nets are above the line and larger scale nets are below the line. (*=EBM, i=eiipleementaiion)
CelebA (64 x 64)
CIFAR-10 (32 32)
Model
FID
Model
FID
Hat EBM (Ours)*
11.57 1: 0.04
Hat EBM (Ours)*
19.30 + 0.15
Divergence Triangle [15]*
31.9
Improved CD EBM [8]*
25.1
SN-GAN NCSNv2[35]
6.1
VERA [14]*
27.5
10.2
GEBM [3]*4
30.1
Cooperative EBM[39]
33.6
ImageNet (128 x 128)
Flow EBM [11]*
37.3
JEM [13]*
38.4
Model
FID
IGEBM [9]*
40.6
Hat EBM (Ours)*
40.24 + 0.18
DDPM [18]
3.2
SN-GAN [26]
65.7
NCSNv2[35]
10.9
SS-GAN [6]
43.9
BigGAN [4]
14.7
InfoMax GAN [23]
58.9
SN-GAN [26]
18.6
Hat EBM, scaled (Ours)*
29.37 + 0.15
SS-GAN, scaled [6]
23.4
annealing the generator and hat network learning rate by factor of 10 after 250K weight updates for each network improved the FID score significantly. See Appendix for uncurated samples from our Hat EBM models. FID results for our model and a representative selction of generative models are shown in Table 2. We were unable reproduce the GEBM FID scores reported in [3] and we report the score obtained from a reimplementation with the official training code. For the SN-GAN FID score, we report the stronger baseline from our reimplementation in Section 4.1.
Results show strong performance of the Hat EBM compared to competing generative models across all datasets, with an especially strong performance for NetgeNet. Our method significantly outperforms other EBM learning methods on CIFAR-10. The Hat EBM synthesis results are on par with the SN-GAN baseline for CIFAR- 10 and CelebA, and the Hat EBM significantly outperforms SN-GAN for ImageNet. With a budget of 8 GPUs and about 2.5 days of computing, our Hat EBM achieves an ImageNet FID score of 40.0, outperforming the scall-scale SS-GAN.
To our knowledge, the current state-of-the-ar model for unconditional ImageNet synthesis at reso- lution 128 x 128 is the large-scale SS-GAN GAN [6]. which achieves FID score of 23.4. This model was trained using BigGAN architecture and 128-core TPUv3 pods. To scale up our Hat EBM, we doubled the number of channel dimensions for both the hat network and generator network from the original SN-GAN architecture and trained on 32-core TPU-v3 pods. Our best FID score for uncondi- tional ImageNet 128x x 128 was 29.2, which comes within a competitive range of sfate-of-the-arr, We believe that further scaling in future work could enable Hat EBM to match or surpass state-of-the-an Our results decisively demonstrate the potential of EBM learning for high-quality synthesis well beyond the scale investigated in any prior EBM work.
4.4 Out-of-Distribation Detection
Experiments in this section assess Hat EBM performance on -ut-Of--iitributtion (OOD) detection. We use the conditional Hat EBM model trained CIFAR- 10 from Section 4.3 and calculate the energy H (X;0) ()) on unseen in-distribution images from the CIFAR-10 test set and images from OOD datasets which include CIFAR- 100, CelebA. and SVHN. We follow standard OOD evaluation from works such as [27] which compute the AUROC metric on the energy scores of the in-distribution and OOD samples. This metric measures the ability of the Hat EBM to distinguish between in-distribution samples not seen during training and OOD samples. Following [13, 38), we expect that the energy of the OOD datasets will be higher than the energy of in in-distribution test images since higher energy samples should appear with lower frequency in the learn density.
Our results are shown in Table 3. The Hat EBM shows strong performance as an OOD detection method. Among methods that are fully unsupervised, our model has the top performance across all three OOD datasets. Our method approaches the results of methods that are trained with labeled

---

Table Comparison of OOD scores (AUROC) among representative models. We separate scores for fully unsupervised models (above the line) and models which used supervised data (below the line).
Model
SVHN
CIFAR-100
CelebA
Ours
0.92
0.87
0,94
IGEBM [9]
0.43
0,54
0.69
VAEBM [38]
0.83
0,62
0.77
Improved CD EBM [8]
0.91
0.83
JEM [13]
0.67
0.87
0.77
HDGE [24]
0,96
0.91
0.80
OOD EBM [25]
0.91
0.87
0.78
OOD EBM (fine-tuned) [25]
0.99
0.94
1.00
data such as HDGE [24] and the fine-tuned OOD EBM [25], although we do not yet match these scores. Overall, there strong evidence that the Hat EBM is naturally an effective method for OOD detection, especially when supervised label information is unavailable.
Conclusion
Maximum Likelihood learning of EBMs poses significant challenge: drawing negative samples from the current density model, which is often highly multi-modal. Prior art addresses this challenge by recruiting approximations of the EBM in the form of an ancestral sampling from generator model, truncated Langevin chains, flow-based models, or lifting the EBM into the induced latent space of generator models. In contrast, our work proposes a method for absorbing any generator as a backbone of an EBM. The formulation assumes that observed images are the sum of unobserved latent variables pushed forward through the generator and a residual random variable which closes the gap between generator samples and image manifold. The hat network sits atop the generator and residual and both nets form the Hat EBM. The generator allows for efficient sampling but may only capture the coarse structure of the images, while the residuals can capture fine-grain details.
The Hat EBM formulation is presented in three variations: (1) joint learning of latent and residual image for adapting any fixed generator, (2) conditional learning for generators with known prior distribution, (3) self-contained learning of both EBM and generator from scratch. Notably, the training requires neither the log determinant of the generator Jacobian or inference of latent variables, making the learning simple and scalable.
Empirical evaluations demonstrate the various capabilities of the Hat EBM: (1) strong performance for the ImageNet synthesis 128x 128 resolution with self-contained learning, (2) significant refinement of the quality of synthesis of pretrained generators on CIFAR- 10 and CelebA with conditional learning, (3) retrofitting pretrained autoencoder generators with means of sampling, (4) out-o--distributtion detection with state-of-the-ar performance for unsupervised models.
Acknowledgements
This work is supported with Cloud TPUs from Google's Tensorflow Research Cloud (TFRC).
References
[1] David H. Ackley, Geoffrey Hinton, and Terrence J. Sejnowski. A learning algorithm for boltzmann machines. Cognitive Science, (1):147-166, 1985.
[2] Abdul Fatir Ansari, Ming Liang Ang, and Harold Soh. Refining deep generative models via discriminator gradient flow. International Conference on Learning Representations, 2021.
[3] Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. In International Conference on Learning Representations, 2021.
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
10

---

[5] Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your gan is secretly an energy-based model and you should use discriminator driven latent sampling. In Advances in Neural Information Processing Systems, volume 33, 2020.
[6] Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised gans via auxiliary rotation loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[7] Zihang Dai Dai, Amjad Almahairi, Philip Bachman, Eduard Hovy, and Aaron Courville. Calibrating energy-based generative adversarial networks. In International Conference Learning Representations, 2017.
[8] Yilun Du, Shuang Li, B. Joshua Tenenbaum, and Igor Mordatch. Improved contrastive diver- gence training of energy based models. In Proceedings of the 38th International Conference Machine Learning, 2021.
[9]
Yilun Du and Igor Mordatch. Implicit generation and generalization with energy based models. In Advances in Neural Information Processing Systems, volume 32, 2019.
[10] Ruiqi Gao, Yang Lu, Junpei Zhou, Song- Chun Zhu, and Ying Nian Wu. Learning generative convnets via multi multi-grid modeling and sampling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[11] Ruiqi Gao, Erik Nijkamp. Diederik P Kingma, Zhen Xu. Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation energy-based models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[12] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, 2014.
[13] Will Grathwohl, Kuan-Chieh Wang. foern-Henrin Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations, 2020.
[14] Will Sussman Grathwohl, Jacob Jin Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud. No [memc) for me: Amortized sampling for fast and stable training of energy-based models. In International Conference on Learning Representations, 2021.
[15] Tian Han, Erik Nijkamp. Xiaolin Fang, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Diver- gence triangle for joint training of generator model, -ased model, and inferential model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
[16] Mitch Hill, Jonathan Craig Mitchell, and Song-Chun Zhu. Stochastic security: Adversarial defense using long-run dynamics of energy- based models. In International Conference on Learning Representations, 2021.
[17) Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1771-1800 2002.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, 2020.
[19] Taesup Kim and Yoshua Bengio. Deep directed generative models with energy based probability estimation. arXiv preprint arXiv:1606. 100.4336... 2016.
[20] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. International Conference Learning Representations, 2013.
[21] Rithesh Kumar, Anirudh Goyal, Aaron C. Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint arXiv: 1901.08508, 2019.
[22] Kwot Sin Lee and Christopher Town. Mimicry: Towards the reproducibility of gan research. arXiv preprini arXiv:2005.
2020.
[23] Kwot Sin Lee, Ngoc-Trung Tran, and Ngai-Man Cheung. Infomax-gan Improved adversarial image generation via information maximization and contrastive learning. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, 2021.
11

---

[24] Hao Liu and Pieter Abbeel. Hybrid perriminativv-geeeratii training via contrastive learning. arXiv preprint arXiv:2007.09070, 2021.
[25] Weitang Liu, Xiaoyun Wang. John Owens, and Yixuan Li. Energy-based out-oo--dstriiie detection. In Advances in Neural Information Processing Systems, volume 33, 2020.
[26] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
[27] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't t know? In International Conference on Learning Representations, 2019.
[28] Erik Nijkamp. Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of MCMC-based maximum likelihood learning of energy-based models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 2020.
[29] Erik Nijkamp, Mitch Hill, Song-Chhn Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run MCMC toward energy-based model. In Advances in Neural Information Processing Systems, volume 32, 2019.
[30] Bo Pang. Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. In Advances in Neural Information Processing Systems, volume 33, 2020.
[31] Bo Pang and Ying Nian Wu. Latent space energy-based model of symbol-vector coupling for text generation and classification. In Proceedings of the 38th International Conference on Machine Learning, 2021.
[32] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference Representa- tion Learning, 2016.
[33] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference Machine Learning, 2014.
[34] Ruslan Salakhutdinov and Geoffrey E. Hinton. Deep boltzmann machines. In Proceedings of the 12th International Conference on Artificial Intelligence and Statistics, 2009.
[35] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Advances Neural Information Processing Systems, volume 33, 2020.
[36] Akinori Tanaka. Discriminator optimal transport. In Advances in Neural Information Processing Systems, volume 32, 2019.
[37] Tijmen Tieleman. Training restricted boltzmann machines using approximations to the like- lihood gradient. In Proceedings of the 25th International Conference on Machine Learning. 2008.
[38] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy- based models. In International Conference on Learning Representations, 2021.
[39] Jianwen Xie, Yang Lu, Ruiqi Gao, and Ying Nian Wu. Cooperative learning of energy-based model and latent variable model via meme teaching. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
[40] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In Proceedings of the 33rd International Conference on Machine Learning, 2016.
[41] Jianwen Xie, Zilong Zheng, Xiaolin Fang, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of fast thinking initializer and slow thinking solver for conditional learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[42] Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying Nian Wu. Learning descriptor networks for 3D shape synthesis and analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
12

---

[43] Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu. Synthesizing dynamic patterns by spatial- temporal generative convnet. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
[44] Song Chun Zhu, Ying Nian Wu, and David Mumford. Filters, random fields and maximum entropy (FRAME): towards a unified theory for texture modeling. International Journal of Computer Vision, 27(2):107-126, 1998.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix A.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Appendix B.
See
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Ap- pendix D.
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either the supplemental material or as a URL)? [Yes] Code and pretrained models are available at ht //t/t/gttttt com hotm/oonttbarr/atttaaa
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See configs in code files and Appendix J.
(c) Did you report error bars (e.g., with respect the random seed after running ex- periments multiple times)? [Yes] Primary numerical results are FID scores. Each experiment was run times and estimated standard deviation was reported.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix C.
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Our code is annotated acknowledge our use of existing assets. In particular, used network structures
based on existing popular models. All other code was created by the authors.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
Code and pretrained models are available at https: // thub. cow wwt.pointtoarra
hat - ebm.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "zgTNZj_jWLj", "reviewer": "Reviewer_9epW", "summary": "The paper proposes “Hat EBM”, an energy-based generative model that incorporates a generator network that maps points from a latent space to the data space into the EBM. The resulting EBM energy is a function of the image generated by the generator and an image residual. The authors propose methods to: train the EBM on top of a fixed generator and train both the generator and the EBM jointly. The authors show the applicability of the proposed method to sample refinement, synthesis and out-of-distribution detection tasks.", "questions": "- What was the original motivation for the image residual formulation of the EBM? It appears that the original motivation was the problem of refinement; however, the proposed method doesn’t refine samples better than the baseline (which does not require additional training).\n- Line 232: How does the incorporation of the residual Y help prevent the assignment of “increasingly high energy to generator samples”?\n- Do the authors have any comments on why hat EBM performs so well against baseline EBMs on image synthesis?", "limitations": "The authors have not discussed limitations of their work. I suggest that they add a few lines to discuss the shortcomings of the proposed work and potential future directions to fix these limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\n- The paper presents an interesting and novel formulation of an energy-based model.\n- The proposed method is technically sound.\n- Experiments have been presented on three different tasks of refinement, generation, and OOD detection. The results show a significant improvement over the baseline EBM models for the image synthesis task.\n\n[Weaknesses]\n\n- While the the key idea of the paper is presented clearly, parts of the paper are difficult to parse. For example, the paragraph beginning at line 180 lacks clarity. Following are some other comments and suggestions:\n    - It appears from the text that training algorithm is not straightforward. Thus, the main text would benefit significantly from a separate algorithm/pseudocode of the proposed method.\n    - Section 2: Instead of saying “This work [D]…” consider using “XYZ et al. [D]”.\n    - Line 110: What is \\sigma?\n    - Line 112: log determinant of **the Jacobian** of G.\n    - Line 139: \\phi has not been defined yet.\n    - Line 158: What does the subscript t indicate in X_t? Is this a typo?\n    - Line 167: propose **to** initialize…\n- From the results in Table 1, it appears that sample refinement is not a strong suit of hat EBM. The FID scores are barely an improvement. This is particularly important because the baseline refinement methods (e.g., DDLS) do not require training of another network unlike hat EBM. Furthermore, the current manuscript doesn’t discuss/compare against other sample refinement baselines, e.g., DOT [1] and DGflow [2]. In particular, DGflow also studies a method to refine samples from all kinds of probabilistic generators.\n- A discussion of and comparison against the very relevant GEBM [3] is missing. GEBM also uses a base generator and an energy function on top of the generator to draw/refine samples. Furthermore, EBM baselines considered in Table 2 have not been properly discussed in the paper. This discussion is important to correctly contextualize this work.\n- While the synthesis results are impressive, the overall significance is marred by a lack of discussion. Why does hat EBM perform so well against these baseline EBMs? Is the comparison fair (e.g., in terms of networks)?\n\n[1] Tanaka, Akinori. \"Discriminator optimal transport.\" *Advances in Neural Information Processing Systems*\n 32 (2019).\n\n[2] Ansari, Abdul Fatir, Ming Liang Ang, and Harold Soh. \"Refining deep generative models via discriminator gradient flow.\" arXiv preprint arXiv:2012.00780 (2020).\n\n[3] Arbel, Michael, Liang Zhou, and Arthur Gretton. \"Generalized energy based models.\" arXiv preprint arXiv:2003.05033 (2020).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bkn_baPtsMX", "reviewer": "Reviewer_oJLY", "summary": "This work proposes a method for using a generator network as the foundation for an Energy-Based Model (EBM). In essence, the paper considers that the output of the generator network is superimposed with a latent variable defined in the image space; this produces the final image. The generator network is presented with a low-dimensional latent variable input. The low dimensionality of this input space renders the modeling problem less multimodal, thus more tractable. \nThe described formulation of the model renders it amenable to MCMC-type sampling (Langevin gradients) for performing training, in a contrastive divergence type of maximum likelihood. ", "questions": "How the method compares to the alternatives in terms of computational costs, both for training and at test time?\nHow do the authors explain the better performance of GAN-type alternatives?\nWhat is the main merit of the proposed approach that adds important value to the state of the art?", "limitations": "n/a", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The model is solid and the training algorithm is straightforward and correct. The authors provide a variety of experiments that deal with different application types. In all cases, they use standard benchmark data to experiment with. They compare to related EBM-type approaches, as well as some state of the art alternatives based on the GAN principle.\n\nWeaknesses: The authors do not explain how the method compares to the alternatives in terms of computational costs, both for training and at test time. In addition, the results are not fully convincing of the usefulness of the method. Specifically, while the method improves over previous EBMs, it seems of lackluster performance compared to GAN-type methods. How do the authors explain this shortcoming? Why would someone prefer their method over the GAN-type alternatives?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PdZspjqmcO", "reviewer": "Reviewer_N2ii", "summary": "This paper proposes the Hat EBM, which is a method to incorporate a sampler (such as a pre-trained generator) into an energy-based model (EBM). The basic idea is to define a joint probability distribution over both the latent space of the pre-trained generator and a “residual space” – the Hat EBM takes as input a synthesized image from the generator combined with a “residual image,” and uses the output of the Hat EBM in an MCMC sampler. The distribution over these residual images is also learned jointly via MCMC. This approach allows for unconditional image generation, image refinement from a pretrained generator such as a GAN, defining implicit samplers for deterministic decoders, and out-of-distribution (OOD) detection.  Hat EBM achieves strong empirical results on complex datasets such as ImageNet 128x128.\n", "questions": "Questions:\n- Clarification: what does it mean on l.107 when the text says that the “majority of the appearance of G(Z) + Y” comes from the generator network? Is that just saying that the generator is responsible for most of the content/variation in the image?\n\nVery minor miscellaneous things:\n- “We propose [to] initialize Langevin sampling in l. 167\n", "limitations": "The authors included a description of the potential negative societal impacts in Appendix B, though it could have been a bit more thorough.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper’s empirical results are strong across a wide range of tasks.\n- The presentation is fairly clear and easy to follow. They also provided good background material on concepts that were important for understanding the paper, such as cooperative learning.\n\nWeaknesses:\n- The proposed approach hinges on some theoretical assumptions and appears to be quite compute-intensive. However, the authors included a thorough description of the computational limitations of their method in Appendix B and D. Appendix E also included a description of the theoretical assumptions required for their method (e.g. correct model specification and convergence of MCMC), as well as precautions necessary for incorporating the residual vector Y during training in Appendix H. I appreciated the care they took here.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-Q7vZXFv462", "reviewer": "Reviewer_45fe", "summary": "This paper shows how one can use existing generator networks within an energy based model. The authors introduce a *hat network* that takes as input the output of the generator plus a residual image, and returns a scalar representing the energy value. The parameters of the hat network (and potentially of the generator) can be trained with Maximum Likelihood learning where negative samples can be obtained with MCMC steps (Gibbs sampler with Langevin updates) \n\n \n\nThe authors explore different learning scenarios and show how this framework can be used to 1) Refine samples of an existing pretrained generator; 2) Generate images by learning the whole EBM model (generator and hat network) from scratch; 3) Allowing sampling over deterministic generators; 4) Use the EBM for anomaly detection. ", "questions": "See questions in the weaknesses section above.", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS \n\n1. There has been an increased focus lately on using EBM for generative learning. This work presents a novel idea that I believe can inspire further research in the neurips community. \n\n2. This is a quite general formulation that can be used for any type of deep generative model \n\n3. Learning does not require posterior inference over the latent states, making the model simpler and more scalable.  \n\n4. The authors show empirically that the hat EMB outperforms competing methods in a number of different tasks. It also generates compelling unconditional Imagenet samples.  \n\n5. The paper is well written and clear \n\n \n\nWEAKNESSES \n\nThe authors should provide more discussions/experimental details, particularly to make this framework more accessible to people not familiar with MCMC (and its pitfalls): \n\n1. In your experiment you chose 50/100 MCMC steps. What is a good strategy to choose this hyperparameter and ensure that the chains have converged well enough for learning to be efficient? \n\n2. In the paper you do not mention the accept-reject step needed when doing the Langevin updates to define a valid Metropolis-Hastings sampler. Do you monitor the acceptance rate during training? \n\n3. What would it take to use Hamiltonian Monte Carlo instead of the Langevin updates to improve the convergence of the chains? \n\n4. How robust is the algorithm to the choice of the size of bank memory? \n\n \n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept