Improved Contrastive Divergence Training of Energy Based Models
Authors: Yilun Du
Shuang Li
Joshua B. Tenenbaum
Igor Mordatch
Venue:  (2021)
Keywords: Contrastive Divergence
Energy Based Modeling

Abstract: We propose several different techniques to improve contrastive divergence training of energy-based models (EBMs). We first show that a gradient term neglected in the popular contrastive divergence formulation is both tractable to estimate and is important to avoid training instabilities in previous models. We further highlight how data augmentation, multi-scale processing, and reservoir sampling can be used to improve model robustness and generation quality. Thirdly, we empirically evaluate stability of model architectures and show improved performance on a host of benchmarks and use cases, such as image generation, OOD detection, and compositional generation.

Under review as a conference paper at ICLR 2021
IMPROVED CONTRASTIVE DIVERGENCE TRAINING OF ENERGY BASED MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose several different techniques improve contrastive divergence training of energy-based models (EBMs). We first show that a gradient term neglected in the popular contrastive divergence formulation is both tractable to estimate and is important to avoid training instabilities in previous models. We further highlight how data augmentation, multi-scale processing, and reservoir sampling can be used to improve model robustness and generation quality. Thirdly, we empirically evaluate stability of model architectures and show improved performance on a host of benchmarks and use cases, such as image generation, OOD detection, and compositional generation.
INTRODUCTION
Figure I: (Left) EBM generated 128x128 unconditional CelebA-HE HQ images. (Right) 128x128 unconditional LSUN Bedroom Images.
Energy-Baad -ased models (EBMs) have received an influx of interest recently and have been applied to realistic image generation (Han et al. 2019: Du & Mordatch| 2019). 3D shapes synthesis (Xie
al. 2018b) out of distribution and adversarial robustness (Lee et al. 2018 Du & Mordatch
2019 Grathwohl al. 2019), compositional generation (Hinton] 1999 Du et al. 2020a), memory
modeling (Bartunov et al. 2019), text generation (Deng et al.) 2020), video generation (Xie et al.)
2017), reinforeement learning (Haarnoja et al. 2017 Du 20119), protein design and folding
(Igrahham et al.; Du et al.) 2020b) and biologically plausible training (Scellier & Bengio (017))
Contrastive divergence is popular and elegant procedure for training EBMs proposed by (Hinton 2002) which lowers the energy of the training data and raises the energy of the sampled confabulations
generated by the model. The model confabulations are generated via an MCMC process (commonly Gibbs sampling or Langevin dynamics), leveraging the extensive body of research on sampling and stochastic optimization. The appeal of contrastive divergence is its simplicity and extensibility, does not require training additional auxiliary networks (Kim & Bengio 2016 Dai et al. 2019) (which introduce additional tuning and balancing demands), and can be used compose models zero-shot
Despite these ad adnatages, training EBMs with contrastive divergence has been challenging due to training instabilities. Ensuring training stability required either combinations spectral normalization and Langevin dynamics gradient clipping (Du & Mordatch 2019), parameter tuning (Grathwohl) cet al., 2019), early stopping of MCMC chains (Nijkamp et al. 2019b), or avoiding the use modern deep learning components, such self-attention layer normalization (Du & Mordatch 2019). These requirements limit modeling power, prevent the compatibility with modern deep learning architectures. and prevent long-running training procedures required for scaling to larger datasets. With this work, we aim to maintain the simplicity and advantages contrastive divergence training, while resolving stability issues and incorporating complementary deep learning advances.

---

Under review as a conference paper at ICLR 2021
Energy Function
Energy Function
Contrastive Divergence
Bock- propagation
Back- propagation
Downsampling
Hallacinasion
Dowssampling
Energy conbination
(almm))
Exengy combermaton
(O)
Energy real data
Downsampling
Enengy of geserated data
Downsampling
Generated Data
Loss Function
Data Angmenntaion
Dato Augmentation
Langevin
Dara Augmenntiion
Langevin
Langevin
Sampling
Sampling
Sampling
Figure 2: Illustration our overall proposed framework training EBMs. EBMs are trained with contrastive divergence, where the energy function decreases energy of real data samples (green dot) and increases the energy of hallucinations (red dot). EBMs are further trained with a KL loss which encourages generated hallucinations (shown solid red ball) to have low underlying energy and high diversity (shown blue balls). Red/green arrows indicate forward computation for while dashed arrows indicate gradient backpropogation.
An often overlooked detail of contrastive divergence formulation that changes to the energy function change the MCMC samples, which introduces an additional gradient term in the objective function (see Section 2.1 for details). This term was claimed to be empirically negligible in the original formulation and is typically ignored (Hinton 2002 Liu & Wang, 2017) or estimated via high- variance likelihood ratio approaches (Ruiz & Titsias 2019). We show that this term can efficiently estimated for continuous data via combination of auto auto- cito-difffrenniattio and nearest-neighhor entropy estimators. We also empirically show that this term contributes significantly to the overall training gradient and has the effect of stabilizing training. It enables inclusion of self-attention blocks into network architectures, removes the need for sppacity-limiting spectral normalization, and allows us train the networks for longer periods. We do not introduce any new objectives or complexity our procedure is simply a more complete form of the original formulation.
We further present techniques improve mixing and mode exploration of MCMC transitions in contrastive divergence. We propose data augmentation as a useful tool to encourage mixing in MCMC by directly perturbing input images to related images. By incorporating data augmentation as semantically meaningful perturbations, we are able to greatly improve mixing and diversity of MCMC chains. We further propose to maintain a reservoir sample of past samples, improving the diversity of MCMC chain initialization in contrastive divergence. We also leverage compositionality of EBMs to evaluate an image sample at multiple image resolutions when computing energies. Such evaluation and coarse and fine scales leads to samples with greater spatial coherence, but leaves MCMC generation process unchanged. We note that such hierarchy does not require specialized mechanisms such progressive refinement (Karras et al. 2017)
Our contributions are as follows: firstly, we show that a gradient term neglected in the popular contrastive divergence formulation is both tractable to estimate and is important in avoiding training instabilities that previously limited applicability and scalability of energy- highlight how data augmentation and multi-scul processing can be used improve model robustness and generation quality. Thirdly, we empirically evaluate stability of model architectures and show improved performance host of benchmarks and cases, such as image generation, OOD detection, and compositional generation.
models. Secondly, we
AN IMPROVED CONTRASTIVE DIVERGENCE FRAMEWORK FOR ENERGY BASED MODELS
Energy based models (EBMs) represent the likelihood of probability distribution for æ E RD as pe(x) expl - Eg (x)) where the function (xx) _ E. is known the energy function, and Z(0)
Z = f exp - (x) is known as the partition function. Thus an EBM can be represented by an neural network that takes input and outputs scalar.
2

---

we obtain the Under review as a conference paper at ICLR 2021
Training an EBM through maximum likelihood (ML) is not straightforward, Z(0) cannot be reliably computed, since this involves integration over the entire input domain of æ. However, the gradient of ligeliielihood with respect to data sample be represented
a log (2))
( ()Eo(z)
(a(''))
E (zx))
(1)
00
00
00
Note that Equation D is still not tractable, as it requires using Markov Chain Monte Carlo (MCMC) to draw samples from the model distribution po po(x)). which often takes exponentially long to mix. As practical approximation to the above objective, (Hinton 2002) proposes the contrastive divergence objective
KL(p(z) || po())) - (p(()((p())) || po())),
where HIg represents a MCMC transition kernel for pa. and H1 (p(z)) represents sequential MCMC transitions starting from p(ar). The above objective can be seen as an improvement operator, where (I(((x) po(ar) 2 (p(())(p((x))) I| (a))) because Th is converging to equilibrium distribution (o(x) Lyu 2011). Furthermore, the above objective is only zero (at its fixed point), when 1lg does not change the distribution of p(z), which corresponds to (x:) p(x))
2.1 A MISSING TERM IN CONTRASTIVE DIVERGENCE
When taking the negative gradient of the contrastive divergence objective (Equation 2 expression
-(EpC)) (p(()))
[a)
E (o
( 1 Oq(a') OKL(ga || pa (x))
(3)
00
00
06
dqo (x)
where for brevity, we summarize (p(x)) = go (x)). The first two terms are identical to those of Equation I [nd the third gradient term (which we refer to the KL divergence term) corresponds to minimizing the divergence between (o()) and pa().. In practice, past contrastive divergence approaches have ignored the third gradient term, which was difficult estimate and claimed to be
empirically negligible (Hinton [999).. These gradients correspond to a joint loss expression L consisting of traditional contrastive loss LCD and new loss expression L-xI- Specifically, we have
[Full I LCD + LxL where LCD is
LCD = Ep(a) [x)(]] Estop qo [x(())),
(4)
and the ignored KL divergence term corresponding to the loss
LKL = B Dssp gradien(())))(()))))) (æ)] + 18 (zx) (oo(g)
(5)
Despite being difficult to estimate, we show that LxL. useful tool for both speeding up and stabilizing training of EBMs. Figure _ illlstraaees the overall effects of both losses. Equation 4
encourage the energy function to assign low energy to real samples and high energy for generated samples. However, only optimizing Equation often leads to an adversarial mode where the energy function learns to simply generate an energy landscape that makes sampling difficult. The KL divergence term counteracts this effect, and encourages sampling closely approximate the underlying distribution (a)), by encouraging samples to be both low energy under the energy function well diverse. Next, discuss our approach towards estimating this KIL divergence, and show that it significantly improves the stability when training EBMs.
2.2 ESTIMATING THE MISSING GRADIENT TERM
Estimating LxL can further be decomposed into two separate objectives, minimizing the energy of samples from qo (œ). which we refer to as Lopt (Equation 6 and maximizing the entropy of samples from go (x) which refer to Lun (Equation)) 7p
Minimizing Sampler Energy. To minimize the energy of samples from (=( we can directly differentiate through both the energy function and MCMC sampling. We follow recent work in EBMs and utilize Langevin dynamics (Du & Mordatch, 2019: Nijkamp et al.) 2019b] Grathwohl et al. 2019) for our MCMC transition kernel, and note that each step of Langevin sampling fully differentiable with respect to underlying energy function parameters. Precisely, gradient of Lop becomes
Öopt
Ô ( (w'_ Vaj_ Ea Fo(x/_1) + w)
Ey (wt .......)))
w ~~0,00 (6)
30
00
3

---

Under review as a conference paper at ICLR 2021
where x represents the 1th step of Langevin sampling. To reduce to memory overhead of this differentiation procedure, we only differentiate through the last step of Langevin sampling (though show it the appendix that leads to the same effect differentiation through Langevin sampling).
Entropy Estimation. To maximize the entropy samples from (0()), we use a -on-paamettric nearest neighbor entropy estimator (Beirlant al. 1997), which is shown to be mean square consistent (Kozachenko & Leonenko 1987) with root-n convergence rate (Tsybakoy & Van der Meulen 1996). The entropy H of distribution p(æ)) can be estimated through different points sampled from p(æ) as H (2)) # Ei= In(n NN (T;; X)) + (() where the
X
22; Un of
function (N; X) denotes the nearest neighbor distance of 24 to any other data point in X. Based off the above entropy estimator, we write Lent as
Lent [q(a:)log((NN/) B))]
where we measure the nearest neighbor with respect set B of 1000 past samples from MCMC chains (see Section 2.5 for more details). We utilize L2 distance the metric for computing nearest neighbors. Alternatively, Stein's identity may also be used to estimate entropy, but this requires considering all samples, opposed to the nearest, becoming computationally intractable. Our entropy estimator serves a simple, quick to compute estimator of entropy, that prevents sampling from collapsing. Empirically, we find that the combination of the above terms in LkL significantly improves both the stability and generation quality of EBMs, improving robustness across different model architectures.
2.3 DATA AUGMENTATION TRANSITIONS
Langevin sampling, our MCMC transition kernel, is prone to falling into local probability modes [Neal) (01T). 201T). In the image domain, this manifests with sampling chains always converging to fixed image (Du & Mordatch 2019). A core difficulty is that distances between two qualitatively similar images can be significantly far away from each in input domain, which sampling is applied. While Lxl encourages different sampling chains to cover the model distribution, Langevin dynamics alone is not enough to encourage large jumps in finite number of steps. It is further beneficial to have individual sampling chain have to ability mix between probability modes.
To encourage greater exploration between similar inputs in our model, we propose to augment chains of MCMC sampling with periodic data augmentation transitions that encourages movement between "similar" inputs. In par- ticular, we utilize combination of color, horizontal flip, rescaling, and Gaussian blur augmentations. Such combi- nations of augmentation has recently seen success applied in unsupervised learning (Chen et al. (020)) Specifically, during training time, we initialize MCMC sampling from data augmentation applied to an input sampled from the buffer of past samples. At test time, during generation, we apply random augmentation
EBM
Energy Addition
Downsampling
EBM
Energy
Downsampling
F
EBM
Illustration of our multi-scale EBM architccture. Our energy function over an image is defined compositionally as the sum of energy functions on different resolu- tions of an image.
Figure 3:
the input after every 20 steps of Langevin sampling. We illustrate this process in the bottom of Figure Data augmentation transitions are always taken.
2.4 COMPOSITIONAL MULTI-SCALE GENERATION
To encourage energy functions to focus on features in both low and high resolutions, we define our energy function as the composition (sum) of a set of energy functions operating on different scales of an image. illustrated in Figure 3 Since the downsampling operation is fully differentiable, Langevin based sampling can be directly applied to the energy function. In our experiments, we utilize full, half, and quarter resolution image as input and show in the appendix that this improves the generation performance.
2.5 RESERVOIR SAMPLING
To encourage qo (z)) match (æ)),, MCMC steps in (x often initialized from past samples from go (x) to enable more diverse mode exploration, a training objective known as persistent contrastive divergence (Tieleman) 2008). Du & Mordatch (2019) propose implement sampling
4

---

Under review as a conference paper at ICLR 2021
Table I: Table Inception FID scores generations of CIFAR-IO, 10, Ce and LSUN bedroom scenes.
denotes our reimplementation of a SNGAN 128x128 model using the torch mimiery GAN library. All others numbers are taken directly from corresponding papers.
Model
Inception
FID
CIFAR-10 Unconditional
PixelCNN Van Oord et al. 2016
4,60
65.93
IGEBM & Mordalen 2019
6.02
40.58
DCGAN [adioni al.
37.11
WGAN GP [Gulrajam
2017
6.50
36.4
Ours
7.58
35.4
SNGAN MMyyat et al. 2018
8.22
21.7
CelebA-HQ 128x128 Unconditional
SNGAN
55.25
Ours
35.06
SSGAN (Chnn 2019
24.36
LSUN Bedroom 128x128 Unconditional
Figure 4: Randomly selected unconditional
SNGAN
125.53
CelebA-HQ samples from our trained EBM
Ours
49.30
T
TH
T
Visualization of Langevin dynamics sampling chains on an EBM trained on CelebA-HQ 128x128. Samples travel between different modes of images. Each consecutive images represents 30 steps of sampling. with data augmentation transitions every 60 steps
Figure 5:
from past samples by utilizing a replay buffer of samples from go (x) interspersed with samples initialized from random noise. By storing a large batch of past samples, the replay buffer is able to enforce diversity across chains. However, as samples are initialized from the replay buffer and added to the buffer again, the replay buffer becomes filled with set of correlated samples from qe (x) over time. To encourage a buffer distribution representative of all past samples, we instead use reservoir sampling technique over all past samples from qo (az). This technique has previously been found helpful in balancing replay in reinforcement learning (Young et al., 2018 Iscle & Cosgun 2018 Rolnick et al., 2019). Under reservoir sampling implementation, any sample from ge x+) has an equal probability of being the reservoir buffer.
3 EXPERIMENTS
We perform empirical experiments validate the following set of questions: (1) What are the effects of each proposed component towards training EBMs? (2) Are our trained EBMs able to perform well on downstream applications of EBMs (generation, compositionallty. -ittof-distribution detection)? We provide ablations of each of our proposed components in the appendix.
3.1 EXPERIMENTAL. SETUP
We investigate the efficacy of our proposed approach. Models are trained using the Adam Optimizer (Kingma & Ba 2015), single 32GB Volta GPU for CIFAR-10 for I day, and for 3 days on 32GB Volta GPUs for CelebaHQ. and LSUN datasets. We provide detailed training configuration details in the appendix.
5

---

Under review as a conference paper at ICLR 2021
Data Augmentation Transitions
No Data Augmentation Transitions
Figure 6: Output samples of running Langevin Dynamies from fixed initial sample (center of square), with without intermittent data augmmentgmnttioonn transitions. Without auttanammenttiion transitions, all samples converge to same image, while data augmentations enables chains to seperate
Our improvements largely built top of the EBMs training framework proposed in (Du & Mordatch] 2019). We use a buffer size of 10000, with a resampling rate of 5% with L.2 regularization on output energies. Our approach is significantly more stable than IGEBM, allowing us remove aspects of regularization in (Du & Mordatch] 2019). We remove the clipping of gradients in Langevin sampling as well spectral normalization on the weights of the network. In addition, we add self- atteetonnnion blocks and layer normalization blocks in residual networks of our trained models. In multi-scale architectures, we utilize 3 different resolutions of an image, the original image resolution, half the image resolution and quarter the image resolution. We report detailed architectures in the appendix. When evaluating models, we utilize the EMA model with EMA weight of 0.999.
3.2 IMAGE GENERATION
We evaluate our approach on CIFAR-10, LSUN bedroom Yu et al. 2015). and CelebA-HQ (Karras) al., (017)) datasets and analyze our characteristies of our proposed framework. Additional analysis and ablations can be found in the appendix of the paper.
Image Quality. We evaluate our approach on unconditional generation in Table D On CIFAR-10 we find that approach, while not being sette-of-theatte significantly outperforms past EBM approaches that based off implicit sampling from an energy landscape (with approximately the same number of parameters), and has performance in the range of recent GAN models on high resolution images. We further present example qualitative images from CelebA-HQ in Figure TTb and present qualitative images other datasets in the appendix of the paper. We note that our reported SNGAN performance on CelebA-HQ and LSUN Bedroom use default hyperparameters from ImageNet models. Gaps in performance with our model are likely smaller with better dataset specific byper-parameters. Inception Score Oplinizaiion Steps
Effect of Data Augmentation. We evaluate the effect of data augmentation on sampling in EBMs. In Figure 5 we show that by combining Langevin sampling with data augmentation transitions, we are able to enable chains to mix across different images, whereas prior works have shown Langevin converging to fixed images. In Figure 6 we further show that given fix random noise initialization, data augmentation transitions enable to reach a diverse number of different samples, while sampling without data augmentation transitions leads all chains to converge to the same face.
Aug
KL/aat Aug
Incoptonn Boors
2
1000
Steps
Figure 7:
Illustration of Inception Score over long run chains with or with- out data augmentation loss.
Mode Convergence. We further investigate high likelihood modes of our model. In Figure 8 we compare very low energy
samples (obtained after running gradient descent 1000 steps on an energy function) for both our model with data augmentation and KL loss and model without either term. Due to improved mode exploration, we find that low temperature samples under our model with data augmentation/KL loss reflect typical high likelihood "modes" in the training dataset, while our baseline models converges to odd shapes, also noted in (Nijkamp et al.) (2119)). Figure / we quantitatively measure Inception scores as steps of gradient descent with without data augmentation and KI. loss. Our

---

Under review as a conference paper at ICLR 2021
Low Temperature Samples (Data Aug KL)
0.5
KL. Terr
Koue
Kort
bifferenes
0.3
0000
200
603
100
020
1000
Treninn Iterations
CIFAR-10
CelebA-QQ
Low Temperature Sumples (No Data Aug No KL.
1L.
Ensag Dillerma
112
0.3
0.0
-0.1
2002
400
603
500
3000
Trasniing Eterations
CIFAR-10
CelebA-HQ
Figure 9: The KL. loss significantly improves the stability of EBM training. Stable EBM train- ing occurs when the energy difference is roughly (indicated by dashed black line). We find that without using the KL loss term EBM training di- verges quickly with different additions to network architecture, while with the KI. loss training is sta- ble. Spectral normalization improves stability of training. but addition of components such layer normalization also destabilizes training.
Figure 8: Illustration of very low temperature samples from our model with KL loss and data augmentation (top) on CIFAR-10 and CelebA-IQ compared to with- out (bottom). After large number of sampling steps, models trained without KL/data augmentation converge to stranges hues in CIFAR- 10 and random textures on CelebA-HO. In contrast, due to better mode exploration. adding both losses maintain naturalistic image modes on both CIFAR-10 and CelchA-HQ
Inception score decreases much more slowly, with some degree of degradation expected since low temperature samples have less diversity.
Stability/KL Loss EBMs are known difficult to train, and to be sensitive both the exact architecture and to various hyper-parameters We found that the addition of a KI. term into our training objective significantly improved the stability of EBM training, by encouraging the sampling distribution to match the model distribution. In Figure 9 we measure the energy difference between real and generated images over the course training when adding normalization and self-attention layers to models. We find that with Lu, the energy difference between both is kept at O, which indicates stable training. contrast, without Lu all models, with the exception of model with spectral normalization, diverge to negative energy different of 11, indicating training collapse. Furthermore, the use of spectral normalization by itself, albeit stable, precludes the addition of other modern network components such as layer normalization. The addition of the KL. term itself is not too expensive, simply requiring an additional nearest neighbor computation during training, which can be relatively insignificant cost compared to the number of negative sampling steps used during training. With intermediate number of negative sampling steps (60 steps) during training, adding the KL term roughly twice slow normal training. This difference is decreased with larger number of sampling steps. Please see the appendix for additional analysis of gradient magnitudes Lu and Led-
Ablations. We ablate each portion of our proposed approach in Table 2 We find that the KL loss is crucial to the stability of training an EBM, and find that additions such multi-scale architecture are not stable without the presence of a KL loss.
Table 2: Effect each ablation on overall image generation on CIFAR-IO. We further report overall stability of training. KL Loss has a significant effect on stability of EBM training.
KL Loss
KL. Loss
Data
Reservoir Sampling
Multiscale
Inception Score
FID
Stability
(Izu)
Aug
No
No
No
No
No
1.46
253.1
No
No
No
Yes
No
No
6.19
61.3
No
No
Yes
No
No
6.28
Yes
Yes
Yes
Yes
No
No
6.53
$4,3
Yes
Yes
Yes
6.78
50.8
Yes
Yes
Yes
Yes
Yes
7.48
35.4
Yes
7

---

Under review as a conference paper at ICLR 2021
Young
Size
Young
AND Female
AND Type
Young
Sizee
AND Female
AND Type
AND Smiling
AND Position
Young
AND Female
AND Type
AND Smiling
AND Position
Wary
Figure 10: Examples of compositionality in face attributes (left) and different renderings of a shape (right). Our
model is able to construct high resolution globally coherent compositional renderings, including fine detail such lighting and reflections (right).
Table 3: Table of out-of-distribation detection scores on unconditional models trained on CIFAR-10 using log(pe (=))). Our approach performs the best out of both likelihood and EBM models. *JEM is not directly comparable supervised labels.
Model
PixeICNN++
Glow
IGEBM
JEM
Ours
SVHN
0.32
0.24
0.63
0.67
0.91
Textures
0.33
0.27
0.48
0.60
0.88
CIFAR Interpolation
0.71
0.51
0.70
0.65
0.65
CIFARI00
0.63
0.50
0.67
0.83
Average
0.50
0.39
0.57
0.65
0.82
3.3 COMPOSITIONALITY
Energy Based Models (EBMs) have the ability to compose with other models generation time (Hinton 1999: Haarnoja et al. 2017 Du al. 2020a). We investigate to what extent EBMs trained under our new proposed framework can also exhibit compositionality. See (Du et al. 2020a) for discussion of various compositional operators and applications in EBMs. In particular, independent EBMs E(x|c1), Ex c2), E(z[ca), that learn conditional generative distribution of concept factors such facial expression object position. We test independent energy functions together to generate images with each concept factor simultaneously, We consider compositions the CelebA-HQ dataset, where we train independent energy functions of face attributes of age, gender, smiling, and wavy hair and high resolution rendered of different objects rendered different locations, where we train an energy function on size, position, rotation, and identity of the object.
train
if we compose
Qualitative Results. We present qualitative results of compositions of energy functions in Figure I0 In both composition settings, our approach is able to successfully generate images with each of conditioned factors, while also being globally coherent. The left image shows that as we condition on factors of young, female, smiling, and wavy hair, images generation begins exhibiting each required feature. The right image similarly shows that as we condition on factors of size, type, position, and rotation, image generations begin to exhibit each conditioned attribute. We note that figures are visually consistent in terms of lighting, shadows and reflections. We note that generations of thee combination of different factors are only specified at generation time, with models being trained independently. Our results indicate that our framework for training EBMs is a promising direction for high resolution compositional visual generation. We further provide visualization of best comparative compositional model from (Vedantam et al. 2018 in the appendix and find that our approach significantly outperforms
3.4 OUT OF DISTRIBUTION ROBUSTNESS
Energy Based Models (EBMs) have also been shown to exhibit robustness both out-of-distributtin and adversarial samples (Du Mordatch 2019; Grathwohl et al.) 2019). We evaluate out-of- distribution detection of our trained energy through log- lokelioel using the evaluation metrics proposed in Hendryycks (impel[(201)) We similarly evaluate dist----dittiiuution detection of an unconditional CIFAR-10 model.
8

---

Under review as a conference paper at ICLR 2021
Results. We present cut-of-distributte results in Table 3 comparing with both other likelihood models and EBMs and using log-likelihood to detect outliers. We find that on the datasets we evaluate, our approach significantly outperforms other baselines, with the exception of CIFAR-10 interpolations. We note the JEM (Grrathwhll et al. 2019) further requires supervised labels to train the energy function, which has shown to improve nut-of-distribution performance. We posit that by more efficiently exploring modes of the energy distribution at training time, we are able to reduce the spurious modes of the energy function and thus improve out-of-distribution performance.
4 RELATED WORK
Our work is related to large, growing body of work different approaches for training EBMs. Our approach is based on contrastive divergence (Hinton 2002), where an energy function is trained contrast negative samples drawn from model distribution and from real data. In recent years, such approaches have been applied to the image domain (Xie et al. [2016] Gao et al. 2018 Du & Mordatch] 2019, Nijkamp et al., 2019b Grathwohl et al.) 2019). (Gao et al. 2018) also proposes multi-scale approach towards generating images from EBMs, but different from our work, uses each sub-scale EBM to initialize the generation of the next EBM. Our work builds on existing works towards contrastive divergence based training of EBMs, and presents improvements in generation and stability.
A difficulty with contrastive divergence training is the difficulty of negative sample generation. To sidestep this issue, a separate line of work utilizes an auxiliary network to amortize the negative
portions of the sampling procedure (Kim & Bengio 2016; Kumar et al. 2019 Han et al. 2019] Xie
ct al. 2018a; Song & Ou 2018). One line of work (Kim & Bengio 20166] Kumar et al. 2 Song & Ou 2018). utilizes separate generator network for negative image sample generations. In contrast, (Xie et al. 2018a), utilizes generator to warm start generations for negative samples and (Han et al.
2019) minimizes divergence triangle between three models. While such approaches enable better qualitative generation, they also lose some of the flexibility of the EBM formulation. For example, separate energy models can no longer be composed together for generation.
In addition, other approaches towards training EBMs seek instead to investigate separate objectives to train the EBM. One such approach is score matching, where the gradients of an energy function are trained to match the gradients of real data (Hyvärinen] (2055: [Song & Ermon 2019). with a related
denoising DSohl-Dickstein et al. 2015 Saremi al.] 2018) Ho et al.] 2020) approach. Additional
objectives include noise contrastive estimation (Gao et al., 2020 and learned Steins discrepancy
(Grathwohl et al. 2020).
Most prior work in contrastive divergence has ignored the KI. term (Hinton 1999. Salakhutdinov & Hinton 2009). A notable exception (Ruiz Titsias 2019), which obtains similar KI. divergence term to ours. Ruiz & Titsias (2019) use high variance REINFORCE estimator to estimate the gradient of the KL term, while our approach relies on auto-differentiation and nearest neighbor entropy estimators. Differentiation through model generation procedures has previously been explored in other models (Finn & Levine 2017]] Metz et al.) 2016). Other related entropy estimators include those based on Stein's 's identity (Liu et al. (017)) and MINE (Belghazi al. 2018). In contrast to these approaches, our entropy estimator relies only on nearest neighbor calculation, and does not require the training of an independent neural network.
CONCLUSION
We propose simple and general framework for improving generation and ease of training with energy based models. We show that the framework enables high resolution compositional image generation and out-of-distribution robustness. In the future, we are interested in further computational scaling of our framework, and applications to domains such as text and reasoning.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Sergey Bartunov, Jack W Rae, Simon Osindero, and Timothy P Lillicrap. Meta-learning deep energy-based memory models. arXiv preprint arxiv: 2019.
Jan Beirlant, E. Dudewicz, L. Gyor, and E.C. Meulen. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 6, 01 1997.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint TrXXv:1801.04062, 2018.
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised gans via auxiliary rotation loss. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 22154-12163, 2019.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020.
Bo Dai, Zhen Liu, Hanjun Dai, Niao He, Arthur Gretton, Le Song, and Dale Schuurmans. Exponential family estimation via adversarial dynamics embedding. In Advances in Neural Information Processing Systems, pp. 10990, 2019.
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Mare' Aurelio Ranzato. Residual energy-based models for text generation. arXiv preprint arXiv:2004.11714, 2020.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy- based models. arXiv preprint ar arXiv: Xxx::19030 103.:0903.08689, 2019.
Yilun Du, Toru Lin, and Igor Mordatch. Model based planning with energy based models. arXiv preprint arXiv:1909.0688 2019.
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. In Advances in Neural Information Processing Systems, 2020a.
Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, and Alexander Rives. Energy-based models for atomic-resolution protein conformations. arXiv preprint 13111 2020b.
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. arXiv:1710. 1622, 2017.
Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Wu. Learning generative convnets via multi-grit -uul modeling and sampling. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. $155-9166 2018.
Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based models. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7518-7522 2020.
Will Grathwohl, Kuan-Chich Wang. fömn-Herrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv:1912.03326. 2019.
Will Grathwohl, Kuan-Chich Wang, Hen-Henrik Jacobsen, David Duvenaud, and Richard Zemel. Cutting out the middle-man: Training and evaluating energy-based models without sampling. arXiv preprint 20Xiiv2002.0661// 2020.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville, Improved training of wasserstein gans. In NIPS, 2017.
Tuomas Haarnoja, Haoran Tang, Picter Abbeel. and Sergey Levine. Reinforcement learning with deep energy-based policies. arXiv preprint arXiv: N0M1:/772..88160 2017.
10

---

Under review as a conference paper at ICLR 2021
Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song-Chun Chun Zhu, and Ying Nian Wu. Divergence triangle for joint training generator model, energy-base baseey model, and inferential model. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8670-8679, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136, 2016.
Geoffrey E Hinton. Products of experts. International Conference on Artificial Neural Networks, 1999.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput. 1478)1777--880,, 2002.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 110000..1239.. 2020.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(Apr):695-709, 2005.
John Ingraham, Adam Riesselman, Chris Sander, and Debora Marks. Learning protein structure with differentiable simulator.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. arXiv preprint orXiv:/802 /0269, 2018.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In ICIR, 2017.
Taesup Kim and Yoshua Bengio. Deep directed generative models with energy-based probability estimation. arXiv preprint arXiv:1606.00 03439, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
LF Kozachenko and Nikolai N Leonenko. Sample estimate of the entropy of random vector. Problemy Peredachi Informatsii, 23(2):9-16, 1987.
Rithesh Kumar, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint 1rXiv:1901.080 2019.
Kwonjoon Lee, Weijian Xu, Fan Fan, and Zhuowen Tu. Wasserstein introspective neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3702-3711, 2018.
Qiang Liu and Dilin Wang. Learning deep energy models: Contrastive divergence vs. amortized mle. arXiv preprint arXiv: 2017.
Yang Liu, Prajit Ramachandran, Qiang Liu, and Jian Peng. Stein variational policy gradient. arXiv preprint arXiv: /ttl://70..2000 02399,, 2017.
Siwei Lyu. Unifying non-maximum likelihood learning objectives with minimum kl contraction. In Advances in Neural Information Processing Systems, pp. 64-72, 2011.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. 11 2016.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:/802.05955, 2018.
Radford M Neal. Meme using hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, (11), 1.
Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of meme- based maximum likelihood learning of energy- based models. arXiv preprint arXiv: 1903.19933.13333 12370, 2019a.
11

---

Under review as a conference paper at ICLR 2021
Erik Nijkamp, Mitch Hill, Song-Chun persistent short-run meme toward energy-based model. In Advances in Neural Information Processing Systems, pp. 2232-52422 2019b.
Chun Zhu, and Ying Nian Wu. Learning non-convergent non-
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, In ICLR, 2016.
David Roinick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, pp. 350-360, 2019.
Francisco JR Ruiz and Michalis K Titsias. A contrastive divergence for combining variational inference and memc. arXiv preprint arXiv:1905. 1955:1905.04062, 2019.
Ruslan Salakhutdinov and Geoffrey E. Hinton. Deep boltzmann machines. In David A. Van Dyk and Max Welling (eds.), AISTATS, volume 5 of JMLR Proceedings, pp. 448 448-455. JMLR.org, 2009. URL http://wwet /wwww jm r. http://pprrcceeeeaapoaaaeeeaeeee
aaktt/t how.nuuudiiov000 html
Saeed Saremi, Arash Mehrjou, Bernhard Schölkopf, and Aapo Hyvärinen. Deep energy estimator networks. arXiv preprint arXiv:1805.08.. 2018. 2018.
Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy- based models and backpropagation. Frontiers in computational neuroscience, 11:24, 2017.
Jascha Dick-eiic,,,in, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. arXiv preprint arXiv: 503.555333333 arXiv:/503.0358:
2015. Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11918-11930, 2019.
Yunfu Song and Zhijian Ou Learning neural random fields with inclusive auxiliary generators. arXiv preprint XXXi::800.00227/ 2018.
Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference on Machine learning, pp. 1064-1071. ACM, 2008
Alexandre B Tsybakov and EC Van der Meulen. Root-n consistent estimators of entropy for densities with unbounded support. Scandinavian Journal of Statistics, pp. 75-83, 1996.
Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In ICML, 2016.
Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative models of visually grounded imagination. In ICLR, 2018
Jianwen Xie. Yang Lu, Song-Chun Chug Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference Machine Learning, pp. 2644-2644, 2016.
Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu. Synthesizing dynamic patterns by spatial-temporal generative convnet. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Jianwen Xie, Yang Lu, Ruiqi Gao, and Ying Nian Wu. Cooperative learning of energy-based model and latent variable model via meme teaching. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018a.
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying Nian Wu. Learning descriptor networks for 3d shape synthesis and analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8629-8638. 2018b.
Kenny J Young, Richard S Sutton, and Shuo Yang. Integrating episodic memory into reinforcement learning agent using reservoir sampling. arXiv preprint arXiv: 00540, 2018
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: / 00366,... 2015.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 MODEL ARCHITECTURES
We list model architectures used in our experiments in Figure 11 When training multi-scale energy functions, our final output energy function is the sum of energy lunctions applied to the full resolution image. half resolution image, and quarter resolution image. We use the architecture reported in Figure I for the full resolution image. The half-resolution models shares the architecture listed in Figure but with all layers before and including the first down-sampled residual block removed. Similarily, the quarter resolution models share the architectures listed, but with all layers before two down-sampled residual blocks removed.
3x3 conv2d, 64
3x3 conv2d, 64
ResBlock 64
ResBlock Down 64
ResBlock Down 64
ResBlock Down 128
ResBlock 64
ResBlock Down 128
ResBlock Down 64
ResBlock 256
Self Attention 64
ResBlock Down 256
ResBlock 128
Self Attention 512
ResBlock Down 128
ResBlock 512
ResBlock 256
ResBlock Down 512
ResBlock Down 256
Global mean Pooling
Global Mean Pooling
Dense +
Dense
(b) The model architecture used for CelebA/LSUN room experiments.
(a) The model architecture used CIFAR-IO ex- periments.
Figure 11: Architecture of models on different datasets.
A.2 EXPERIMENT CONFIGURATIONS FOR DIFFERENT DATASETS
CIFAR-10 For CIFAR-10, use 40 steps of Langevin sampling to generate negative sample. The Langevin sampling step size is set to be 100, with Gaussian noise of magnitude 0.001 at each iteration. The data augmentation transform consists of color augmentation of strength 1.0 from (Chen et al. 2020), as a random horizontal crop, and image resize between 0.3 and 1.0 and a Gaussian blur of 5 pixels.
CelebA/LSUN Bed For CelebA and LSUN bed datasets, we use 40 steps of Langevin sampling to generate negative samples. The Langevin sampling step size set to be 1000, with Gaussian noise of magnitude 0.001 applied at each iteration. The data augmentation transform consists of color augmentation of strength 0.5 from (Chen et al. 2020), as a random horizontal crop, and a image resize between 0.3 and 1.0 and Gaussian blur of 11 pixels.
A.3 COMPARISON OF CD/KL GRADIENT MAGNITUDES
We plot the overall gradient magnitudes of the contrastive divergence and KL loss terms during training of an EBM in Figure 12, We find that relative magnitude of both training remains constant across training, and that the gradient of the KL objective is non-negligible
A.4 ANALYSIS OF TRUNCATED LANGEVIN BACKPROPAGATION
To test the effect of truncating backpropogation through the KL loss to only one sampling step of I Langevin sampling, train seperate models MNIST, one with backpropogation through
13

---

Under review as a conference paper at ICLR 2021
CD vs KL Grad
CD Grad
KL Grad
10-2
Norm
10-3
10
0
200
400
600
800
1000
Training Iterations
Figure 12: Plots of the gradient magnitude of LKL and Len training iterations. Influences and relative magnitude both loss terms stays constant through training.
9
4
8
8
8
8
9
0
7
0
3 9
9
2
3
5
8
8
8
8
8
4
0
8
8
9
8
0
8
8
8
2
8
3
8
0
8
9
0
6
9
6
8
8
7
8
0 7 8
9
0
9
9
6
0
9
9
9
8
9
5
R
9
6
0
8
8 8 8 0
8
9
4
8
9
9
0
0
0
8
8
9
6
9
3
9 8 9 8
8
8
8
0
9
9
0
6
8
1
8
8
8
5 9
0
7
2
8
0
9
9
6
8
1
9
9
0
8
9
6
9
9
8
9
9
0
4
6
6
0
8
8
8
8
6
0
8
2
9
4 /
8 9
0
8
6
9
8
3
/
8
8
9
S
8
8
3
8
8
8
9
0 6 /
6
8
9
0
9
0
5
0
4
8
8
8
8
7
7
8
Generations on MNIST with backpro- pogation through step of Langevin sampling.
Figure 13:
Figure 14: Generations on MNIST backpro- pogation through all steps Langevin sampling.
all Langevin steps, and one with backpropogation through only the last Langevin step. We obtain FIDs of 90.54 with backpropogation through only step of Langevin sampling and FIDs of 94.85 with backpropogation through all steps Langevin sampling. We present illustrations of samples generated with one step in Figure 13 and with all steps in Figure 14
A.5 ANALYSIS OF EFFECT OF KL LOSS ON MODE EXPLORATION
The KL loss adds an additional term to EBM training that encourages EBM training updates to maintain good mode coverage while optimizing the usual contrastive divergence objective. Thus the KL loss serves as a regularizer to prevent EBM sampling from collapsing. In the absence of the KL loss, EBM sampling always eventually collapses and generates samples in Figure 15
14

---

Under review as a conference paper at ICLR 2021
Young (CBM)
Young
(EBM)
Young
AND Female
Smiling (EBM)
Young
AND Fandie
AND Smiling
(CBM)
Figure Illustrations compositional generations from more blurry than our generations.
Vedantam et al, 2018 Generations are significantly
Figure 15: Illustration of collapsed sampling from EBM.
A.6 COMPARISON TO OTHER COMPOSITIONAL GENERATIVE MODELS
To knowledge, there relatively few other models that compositionally combine, with the approach of JVAE (Vedantam et al. 2018) being the closest to our work. We provide comparisons in Figure 16
Our approach is significantly less blurry than JVAE.
A.7 ADDITIONAL. QUALITATIVE IMAGES
We present randomly generated qualitative images on the LSUN dataset in Figure and the CIFAR- 10 dataset Figure T8, both setting, we find that unconditional images appear mostly globally coherent.
15

---

Under review as a conference paper at ICLR 2021
FANe & IEtI
Figure 17: Randomly selected unconditional LSUN bed 128x128 samples from our trained EBM.
16

---

Under review as a conference paper at ICLR 2021
Figure 18: Randomly selected unconditional CIFAR-10 samples from our trained EBM.
17


Reviews:
[{"review_id": "lz7WFS-aydq", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an improved version of contrastive divergence learning of energy-based models by combining a bag of techniques: (1) add back a KL term that is neglected by previous methods (2) data augmentation (3) multi-scale processing (4) reservoir sampling. Experiments demonstrate the effectiveness of the improvements. \n\nPro:\nThe paper is well-written and easy to follow. Various experiments are performed to demonstrate the efficacy of the improved method. \n\nCons:\n1. The advantage of adding the KL term is not quite obvious given the current experiments. The only experiment that isolates the effect of the KL term is Figure 8 (stability of training), which can be accomplished by simply adding spectral normalization. For all the other improvements, I tend to believe they are due to the techniques of (2)(3)(4).\n\n2. For the first term of L_KL, it is not entirely correct to take gradient only over the last step of Langevin sampling. Need more justifications. For the second term, it requires computing on 1000 samples per update, where the efficiency should be discussed. \n\n3. The multi-scale processing of EBMs has been explored in [1], which should be discussed and compared. Besides, [2][3][4] are relevant references of training EBMs that should be discussed. \n\n4. Qualitative speaking, long-run chains in figure 7 still have a trend of degradation from realistic images. Quantitative analysis (e.g., German-Rubin statistics) would be helpful for evaluating the long-run chains clearly. \n\nOverall, the paper proposes effective improvements on contrastive divergence of EBMs, and performs various experiments to demonstrate the efficacy. However, I am concerned about the correctness and the necessity of adding the gradient term (L_KL), which is one of the major contributions that the authors claim. Please address my concern as listed above. \n\n[1] Learning Energy-Based Models as Generative ConvNets via Multi-grid Modeling and Sampling, Gao et al.\n[2] A Theory of Generative ConvNet, Xie et al. \n[3] Flow Contrastive Estimation of Energy-Based Models, Gao et al. \n[4] Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling, Grathwohl et al. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "YUPBCwIdKBq", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a series of new techniques to enhance the training of an energy-based model, and the proposed techniques include: adding the often neglected KL term to the training scope/data augmentation + multi-scale energy function/an experience replay buffer for training.\n\nThe experiments demonstrate the proposed method could generate high-quality images, compositional tasks and perform out-of-distribution detection.\n\nThe main idea and motivation are well and clearly conveyed by the writing.\n\nThe paper would be stronger if the authors could provide the following pieces:\n- how well is the entropy estimation? We all know that estimating the entropy of data distribution from a high-dimenstional space is very difficult. Does this form of nearest neighbor applicable in other areas? It would also be great if the authors could provide some theoretical analysis here.\n- while the main contribution of the paper seems to be the KL term added into the objective, there are a few other tenichques tagging along. It is not clear what role each of these techniques plays in the experiments. I recommend the authors to show an ablation study.\n- Figure 9 would need to compare against other methods. It is not clear to me how the arithmetic results are stronger than the other published results.\n- An important argument in the paper is that the added KL term enhances the mode coverage. Could the authors provide some more evidence on this point?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CQAPCZKs1UP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes several techniques to improve contrastive divergence training of energy-based models (EBMs). \nFirst, the paper proposes to estimate a gradient term, which is neglected in the standard contrastive divergence training method, and show that this correction avoids training instabilities in previous EBM training methods. \nOther techniques include: using data augmentation, defining the energy function as a sum of energies over multi-scales, and using reservoir sampling.\nEffects of each proposed techniques towards training EBMs are evaluated. The performance of the trained EBMs on image generation, OOD detection, and compositional generation are tested.\n\nIn generally, the paper is well written and addresses the important problem of improving EBM training. But I have some concerns.\n\n1. It is not easy for general readers to understand the upper part of Figure 2, which is said to illustrate the overall effects of the losses L_CD and L_KL. What are the meanings of the red balls (dark and light) in the curve?\n\n2. The paper overlooks a class of competitive training methods, which introduce auxiliary generators to train EBMs, including (Kim & Bengio, 2016; Kumar et al., 2019), [a] and so on.\nThe comment in Section 4 (related work), which describes these methods as utilizing pre-trained networks to approximate portions of energy training, is not correct (not capturing the core idea of these methods). Although learning EBMs without auxiliary generators is worthwhile exploring, the paper needs to give the readers an overall picture of the state-of-the-art of learning EBMs and does not give biased comments. \n\nAlthough the proposed method is somewhat new, the results are not strong, which weakens the contribution of this paper. [a] achieved much better results than the proposed method in CIFAR-10 (Table 1). Additionally, computational cost of the proposed method should be given and compared to previous methods.\n[a] Y. Song, Z. Ou. Learning Neural Random Fields with Inclusive Auxiliary Generators. arxiv 1806.00271, 2018.\n\nSNGAN performs much better than the proposed method in CIFAR-10, but much worse in CelebA-HQ and LSUN. This may confuse readers. The \"reimplementation of a SNGAN 128x128 model using the torch mimicry GAN library\" in CelebA-HQ and LSUN may not faithfully reflect the performance of SNGAN.\n\n3. Considering the above comment, the following claim in this paper needs revision.\n\"significantly outperforms past energy based approaches (with approximately the same number of parameters)\"\n\n4. The paper has sporadic writing problems.\n\nTypo in Eq.(3)\n\ndivergenceterm\n\nLSUN bedroom (?)\n\n5. In A.4 (COMPARISON OF CD/KL GRADIENT MAGNITUDES), it is said that \"the gradient of the KL objective is non-negligible\". But it is this non-negligible gradient term that stabilize the EBM training. Need more analysis here.\n\nHow \"Influences and relative magnitude of both loss terms\" are calculated?\n\n--------update after reading the response-----------\n\nThanks for the authors' response, but some non-trivial concerns are still not adequately addressed.\n1) The inconsistent comparison results between SNGAN and the proposed method over CIFAR-10 and LSUN Bedroom datasets.\n2) I can see the benefit such as compositionality from the proposed method of training EBMs. But the paper still seems to overlook the importance of giving the readers an overall picture of the state-of-the-art of learning EBMs. Table 1 should be expanded to include more state-of-the-art results from EBMs, whether using auxiliary generators or not.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6F_2Nz4Hkgi", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review: This paper studies how to improve contrastive divergence (CD) training of energy-based models (EBMs) by revisiting the gradient term neglected in the traditional CD learning. This paper also introduces some useful techniques, such as data augmentation, multi-scale energy design, and reservoir sampling to improve the training of energy-based model. Empirical studies are performed to validate the proposed learning strategy on the task of image generation, OOD detection, and compositional generation.\n\nStrength: \n+ The idea of dealing with the missing term in the traditional CD learning is important and relevant. \n+ The paper is well written. Specifically, the figure illustration and the organization of the paper make me feel quite easy to follow the paper.  \n+ The motivation of the method is clear, and the experimentation looks OK.  \n\nConcerns: \n+ The contribution of the paper is quite limited. Even though this paper tried to estimate the missing term in the CD learning, it lacks a comprehensive analysis of the benefit and advantages of doing so. For example, (1) what is the cost to add such a term? (2) Can you validate theoretically such a missing term can be helpful for MCMC mixing as you claimed in the paper? \n\n+ About motivation. Even though the motivation of the current paper is clear, which is to improve the CD learning. However, the CD learning (in equation 2) is biased compared with MLE (in equation 1). The original motivation for CD learning is to make EBM learning more efficient. Since currently there has been EBM training method without MCMC or with amortized sampling, I am not sure if the current method is still useful for the community.    \n\n+ About synthesis quality: the synthesized images generated by the proposed images are not impressive. Artifacts can be obviously observed in Figure 12.    \n\n+ Missing important references in related works. The current paper missed to cite the pioneering paper about MLE training of ConvNet-EBM [1]. Those EBM papers you have cited from 2019 is based on [1] or its variant. \n\n+ Incomplete narrative of the development of EBMs in the introduction.  Even though the narrative of the development of EBMs is quite comprehensive, it is not complete and even a little bit misleading. For example, since 2016, the EBMs have been applied to realistic image generation (2016-2019)[1, 3, 4, 6, 7], video generation (2017)[2, 4], and 3D generation (2018)[5] in the community of computer vision. Therefore, the current research direction seems not to be novel, given the fact that authors might miss a lot important developments about EBM made by other fields. CD learning is also studied and discussed in [1] for deep EBM. The current papers only discussed and connected EBM development happened recently in ML community (2019, 2020).     \n\n+ typo: in Section 2.1 line 16: KL divergenceterm => KL divergence term\n\nSome references:\n+ [1] A Theory of Generative ConvNet (ICML 2016)\n+ [2] Synthesizing Dynamic Pattern by Spatial-Temporal Generative ConvNet (CVPR 2017)\n+ [3] Cooperative Learning of Energy-Based Model and Latent Variable Model via MCMC Teaching (AAAI 2018)\n+ [4] Cooperative learning of descriptor and generator networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI 2018).\n+ [5] Learning Descriptor Networks for 3D Shape Synthesis and Analysis. (CVPR 2018) \n+ [6] Learning generative ConvNets via multigrid modeling and sampling. (CVPR 2018)  \n+ [7] Divergence triangle for joint training of generator model, energy-based model, and inference model. (CVPR 2019)   \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a bag of techniques to improve contrastive divergence training of energy-based models (EBMs), particularly a KL divergence term, data augmentation, multi-scale energy functions, and reservoir sampling. The overall paper is well written and clearly presented. 

In response to the major concerns from reviewers, the AC recognizes the authors' effort in expanding related work and adding ablation on the effects of the KL loss. However, reviewers remain unconvinced by the significance of the current results. In particular, the quality improvement by adding the KL terms is subtle compared to using reservoir sampling (as evidenced in the contrast of the last two rows in Table 2). Moreover, the authors are also encouraged to compare additionally with recent development in EBM, as pointed out by R2 & R4.

The AC does find the results on downstream tasks such as out-of-distribution quite promising and interesting. Perhaps it's worth expanding the discussion with formal reasoning on why KL loss helps in this case. 

All four knowledgeable reviewers are leaning towards rejection, the AC respects and agrees with the decision. 

Decision: Reject