No MCMC for me: Amortized sampling for fast and stable training of energy-based models
Authors: Will Sussman Grathwohl
Jacob Jin Kelly
Milad Hashemi
Mohammad Norouzi
Kevin Swersky
David Duvenaud
Venue: ICLR 2021 Poster (2021)
Keywords: Generative Models
EBM
Energy-Based Models
Energy Based Models
semi-supervised learning
JEM

Abstract: Energy-Based Models (EBMs) present a flexible and appealing way to represent uncertainty. Despite recent advances, training EBMs on high-dimensional data remains a challenging problem as the state-of-the-art approaches are costly, unstable, and require considerable tuning and domain expertise to apply successfully. In this work, we present a simple method for training EBMs at scale which uses an entropy-regularized generator to amortize the MCMC sampling typically used in EBM training. We improve upon prior MCMC-based entropy regularization methods with a fast variational approximation. We demonstrate the effectiveness of our approach by using it to train tractable likelihood models. Next, we apply our estimator to the recently proposed Joint Energy Model (JEM), where we match the original performance with faster and stable training. This allows us to extend JEM models to semi-supervised classification on tabular data from a variety of continuous domains.

Published as a conference paper at ICLR 2021
No MCMC FOR ME: AMORTIZED SAMPLING FOR FAST AND STABLE TRAINING OF ENERGY-BASED MODELS
Will Grathwohl
Jacob Kelly*
University of Toronto & Vector Institute
University of Toronto & Vector Institute jke lyêcs toronto. edu
Google Research
wgrathwohler @cs. toronto.edu
Milad Hashemi
Mohammad Norouzi & Kevin Swersky Google Research (mnorouzi, kswersky| com
Google Research mi 11ladd@@ooggee.com
David Duvenaud
University of Toronto & Vector Institute duvenaud8 es. toronto. edu
ABSTRACT
Energy-Bass -aagyd Models (EBMs) present a flexible and appealing way to represent uncertainty. Despite recent advances, training EBMs on high-dimensional data remains challenging problem the state-of-the-an approaches ble, and require considerable tuning and domain expertise to apply successfully. In this work we present simple method for training EBMs at scale which uses an entropy regulaarege generator to amortize the MCMC sampling typically used in EBM training. We improve upon prior MCMC-based entropy regularization methods with a fast variational approximation. We demonstrate the effectiveness of our approach by using it to train tractable likelihood models. Next, we apply our estimator to the recently proposed Joint Energy Model (JEM), where we match the original performance with faster and stable training. This allows us to extend JEM models to semi-supervised classification on tabular data from variety of continuous domains.
costly, unsta-
INTRODUCTION
Energy-Based Models (EBMs) have recently regained popularity within machine learning, partly inspired by the impressive results of Du & Mordatch (2019) and Song & Ermon (2020) on large- scale image generation. Beyond image generation, EBMs have also been successfully applied to a wide variety of applications including: outtof-diittriuttin detection (Grathwohl et al., 2019; Du & Mordatch, 2019; Song & Ou, 2018), adversarial robustness (Grathwohl al., 2019; Hill al., 2020; Du & Mordatch, 2019), reliable classification (Grathwohl et al., 2019; Liu & Abbeel, 2020) and semi-supervised learning (Song & Ou, 2018; Zhao et al.). Strikingly, these EBM approaches outperform alternative classes of generative models and rival hand-tailored solutions on each task.
Despite progress, training EBMs is still challenging task. As shown in Table 1, existing train- ing methods are all deficient in at least one important practical aspect. Markov chain Monte Carlo (MCMC) methods are slow and unstable during training (Nijkamp et al., 2019a; Grathwohl al., 2020). Score matching mechanisms, which minimize alternative divergences are also unstable and most methods cannot work with discontinuous nonlinearities (such as ReLU) (Song & Ermon, 2019b; Hyvärinen, 2005; Song et al., 2020; Pang et al., 2020b; Grathwohl et al., 2020; Vincent, 2011). Noise contrastive approaches, which learn energy functions through density ratio estimation, typically don't t scale well to high-dimensional data (Gao et al., 2020; Rhodes al., 2020; Gutmann & Hyvärinen, 2010; Ceylan & Gutmann, 2018).
"qqual Contribution. Code available github. cow www/wgrathwohhn /VERA

---

Published as a conference paper at ICLR 2021
Training Method
Fast
Stable training
High dimensions
No aux. model
Unrestricted architecture
Approximates likelihood
Markov chain Monte Carlo
X
Score Matching Approaches
/
V
Noise Contrastive Approaches
X
X
VERA (ours)
X
Table I: Features of EBM training approaches. Trade-offs must be made when training unnormal-
ized models and no approach to date satisfies all of these properties.
VERA
PCD
Ground Truth
Density
Density
Density
Generator
MCMC
Samples
0.01
0.1
1.0
0.01
0.03
0.1
Entropy Regularization
SOLD
Figure 1: Comparison of EBMs trained with VERA and PCD. We see that as entropy regularization goes the density becomes more accurate. For PCD, all samplers produce high quality samples, but low-quality density models as the distribution of MCMC samples may be arbitrarily far away from the model density.
In this work, present simple method for training EBMs which performs well as previous methods while being faster and substantially easier to tune. Our method based on reinterpreting maximum likelihood the past for EBM training (Dai et al., 2019). This perspective allows us to amortize away MCMC sampling into GAN-style generator which is encouraged to have high entropy. We accomplish this with novel approach to entropy regularization based on a fast variational approximation. This leads the method we call Variational Entropy Regularized Approximate maximum likelihood (VERA).
bi-level variational optimization problem, which has been explored in
Concretely, we make the following contributions:
We improve the MCMC-based entropy regularizer of Dieng et al. (2019) with a paralleliz- able variational approximation.
We show that an entropy- reguuaizee generator can be used to produce variational bound on the EBM likelihood which can optimized more easily than MCMC-based estimators.
We demonstrate that models trained this way achieve much higher likelihoods than meth- ods trained with alternative EBM training procedures.
We show that our approach stabilizes and accelerates the training of recently proposed Joint Energy Models (Grathwohl al., 2019).
We show that the stabilization of our approach allows to use JEM for semi-supervised learning, outperforming virtual adversarial training when little prior domain knowledge available (e.g., g., for tabular data).
2 ENERGY BASED MODELS
An energy- -aaedd model (EBM) is any model which parameterizes a density as
(fo(x)
Pe (x)
Z(0)
where fo : R + R and Z(0) = ef f modeled. Any probability distribution can be represented in this way for some fo. The energy-based
the normalizing constant which is not explicitly
2

---

Published as a conference paper at ICLR 2021
parameterization has been used widely for its flexibility, ease of incorporating known structure, and relationship to physical systems common in chemistry, biology, and physics (Ingraham et al., 2019; Du et al., 2020; Noć al., 2019).
The above properties make EBMs an appealing model class, but because they are unnormalized many tasks which simple for alternative model classes become challenging for EBMs. For example, exact samples cannot be drawn and likelihoods cannot be exactly computed (or even lower- bounded). This makes training EBMs challenging as cannot simply train them maximize likelihood. The most popular approach to train EB is to approximate the gradient of the maximum likelihood objective. This gradient can be written as:
Vo (o)) = VoJo(a) Eoo(en) (VoJo(>))
(2)
MCMC techniques are used approximately generate samples from po(x)) (Tieleman, 2008). Prac- tically, this approach suffers from poor stability and computational challenges from sequential sam- pling. Many tricks have been developed to overcome these issues (Du Mordatch, 2019), but they largely still persist. Alternative estimators have been proposed to circumvent these challenges, in- cluding score matching (Hyvärinen, 2005), noise contrastive estimation (Gutmann & Hyvärinen, 2010), and variants thereof. These suffer from their own challenges in scaling to high dimensional data, and sacrifice the statistical efficiency of maximum likelihood.
In Figure 1 we visualize densities learned with our approach and Persistent Contrastive Diver- gence (Tieleman, 2008) (PCD) training. As we see, the sample quality of the PCD models is quite high but the learned density models do not match the true model. This is due to accrued bias in the gradient estimator from approximate MCMC sampling (Grathwohl et al., 2020). Prior work (Ni- jkamp et al., 2019b) has argued that this objective actually encourages the approximate MCMC samples to match the data rather than the density model. Conversely, we see that our approach (with proper entropy regularization) recovers high quality model.
3 VARIATIONAL MAXIMUM LIKELIHOOD
We seek the energy function which maximizes likelihood given in Equation 1. We examine the intractable component of the log-likel,hood, the log partition- partition-function function log Z(0) log J cfo()) We can re-write this quantity as the optimum of
log Z(0) = max E [f(=)) + H(4)
(3)
where y is distribution and H(g) ( log (xx] denotes its entropy (see the Appendix A. for the derivation). Plugging this into our original maximum likelihood statement we obtain:
Ô- argmaxx Ep [o(()]] max [[(r)) + H(O))
which gives us an alternative method for training EBMs. We introduce an auxiliary sampler de which train online to optimize the inner-loop of Equation 4. This objective was used for EBM training in Kumar et al. (2019); Abbasnejad et al. (2019); Dai et al. (2017), Dai et al. (2019) (motivated by Fenchel Duality (Wainwright & Jordan, 2008)). Abbasnejad et al. (2019) use an implicit generative model and Dai et al. (2019) propose to use a sampler which is inspired by MCMC sampling from (a() and whose entropy can be computed exactly.
Below we describe our approach which utilizes the same objective with a simpler sampler and new approach to encourage high entropy. We note that when training Po(x) and q(x) online together, the inner maximization will not be fully optimized. This leads our training objective for po(x)) to be an upper bound on log Po Po(x). In Section 5.1 we explore the impact of this fact and find that the bound tight enough to train models that achieve high likelihood on nigh-dimensional data.
For continuous spaces, this would be the differential entropy, but we simply use entropy here for hrevity.
3

---

Published as a conference paper at ICLR 2021
4 METHOD
We now present a method for training an EBM pa(x) = e /2(0) to optimize Equation 4. We introduce a generator distribution of the form (z(z) (x(2)=)))( 2) (z)2 such that:
(0(=|2) - (x),
<(>) = 10,
(5)
where Dy is neural network with parameters ià and thus, Φ {th, a2). This is similar to the decoder of a variational autoencoder (Kingma & Welling, 2013). With this architecture easy optimize the first and second terms of Equation with reparameterization, but the entropy term requires more care.
4.1 ENTROPY REGULARIZATION
Estimating entropy its gradients is a challenging task. Multiple, distinct approaches have been proposed in recent years based on Mutual Information estimation (Kumar et al., 2019), variational upper bounds (Ranganath et al., 2016), Denoising Autoencoders (Lim et al., 2020), and nearest neighbors (Singh & Póczos, 2016).
The above methods require the training of additional auxiliary models or do not scale well high dimensions. Most relevant to this work are Dieng et al. (2019); Titsias & Ruiz (2019) which present a method for encouraging generators such as ours to have high entropy by estimating V Thee estimator takes the following form:
V H(s) = V (logs
V Bo (I)(z,,))
(Reparameterize sampling)
(p(e)rte) (V (x((,))
(e V (ogg)(((
(0)(2))) (=),,,)
(Chain rule)
where we have written x(z,e) e) (z) + de. All quantities in Equation can be easily computed
except for the core-function V, log 4o (x). The following estimator for this quantity can be easily derived (see Appendix A.2):
V log = [x V. log (x(2]]
which requires samples from the posterior ( .r) to estimate. Dieng et al. (2019); Titsias & Ruiz (2019) generate these samples using Hamiltonian Monte Carlo (HMC) (Neal al., 201 I), gradient- based MCMC algorithm. As used in Dieng et al. (2019), 28 sequential gradient computations must be made per training iteration. Since key motivation sequential computation of MCMC sampling, this is not favourable solution. In our work propose a more efficient solution that we find works just as well empirically.
this work is to circumvent the costly
4.2 VARIATIONAL APPROXIMATION WITH IMPORTANCE SAMPLING
We propose replace HMC sampling of (=(=))) with variational approximation
{(z =n) R (=(= x) where in is a conditioning variable we will define shortly. We can use this
approximation with self-normalized importance sampling estimate
V z log (x())
E (=)e) (.r log (x =)]
E (2) Vz log ( 2)]
ga(= xx) V. log (x0 10)
E(2 20)
(2)
Ep(z))((=)=))
V (o
(x) 20)
&
Wi
Vn log ao (ar |x] = V. log (2; {2))
(8)
k
j
Wi
where {zi)/_ E(2 20) and w;
gي(=اa)) We use k = 20 importance samples for all experi- [(2)
ments in this work. This approximation holds for any conditioning information would like use.
4

---

Published as a conference paper at ICLR 2021
To choose this, let consider how samples (0(x)) are drawn. We first sample 20 (0 I) and then 2 ~ 4s (æ) 20). In our estimator we want a variational approximation to g0(2 z) and by con- struction, 2p is sample from this distribution. For this reason we let our variational approximation be
(n 20) - N( 50, 10.,)"1),
simply diagonal Gaussian centered at the zo which generated For this approximation to be useful we must tune the variance 17². We do this by optimizing the standard Evidence Lower-Bound every training iteration
CELBO 20, (en())=0)) [log(gs(2) 2)) +
(S, 20))
(10)
We then (n 20) to approximate V, log 90 which plug into Equation 6 to estimate V H (gx) for training our generator. A full derivation and discussion can be found in Appendix A.3.
Combining the tools presented above arrive Entropy Regularized Approximate maximum likelihood (VERA), outlined in Algorithm We found it helpful to further add an fz-regularizer with weight 0.1 to the gradient of our model's likelihood as in Kumar et al. (2019). In some of our larger-scale experiments we reduced the weight of the entropy regularizer as in Dieng al. (2019). We refer to the entropy regularizer weight as
proposed method which call Variational
Algorithm 1: VERA Training
Input EBM p=(x) efo(m), generator (2, z), approximate posterior (,,2222), entropy weight X, gradient penalty = .1
Output: Parameters 0 such that po p
while True do
Sample mini-batch æ, and generate mini-batch Fig 20 qs (2r, Compute CELBO (1; 20 Rg)) and update !)
// Update posterion
Compute log fo()) - log fo(x,) + >/ log fo fo())] and update // Update EBM
Sample { ~ (2((22))
Compute Va log go { {=:)
// Estimate score fn (Eq.
Compute g 3''V 2tg
// Estimate entropy gradient (Eq.6) // Update generator
Update o with V log fo (x=g) + >g
end
5 EBM TRAINING EXPERIMENTS
We present results training various models with VERA and related approaches. In Figure we visualize the impact our generator's entropy on the learned density model and compare this with MCMC sampling used in PCD learning. In Section 5.1, we explore this quantitatively by training tractable models and evaluating with test-likelihodd. In Section 5.2 we explore the bias of our entropy gradient estimator and the estimator's effect on capturing modes.
5.1 FITTING TRACTABLE MODELS
Optimizing the generator in VERA training minimizes a variational upper bound on the likelihood of data under model. If this bound is not sufficiently tight, then training the model maximize this bound will not actually improve likelihood. To demonstrate the VERA bound is tight enough to train large-scale models we train NICE models (Dinh et al., 2014) on the MNIST dataset. NICE is normalizing flow (Rezende & Mohamed, 2015) model a flexible density estimator which enables both exact likelihood computation and exact sampling. We can train this model with VERA (which does not require either of these abilities), evaluate the learned model using likelihood, and generate exact samples from the trained models. Full experimental details² can be found in Appendix B.3 and additional results can be found in Appendix C.
This experiment follows the NICE experiment in Song al. (2020) and was based their implementation.
5

---

Published as a conference paper at ICLR 2021
We compare the performance of VERA with maximum likelihood training as well as number of approaches for training unnor- malized models; Maximum Entropy Genera- tors (MEG) (Kumar al., 2019). Persistent Contrastive Divergence (PCD). Sliced Score Matching (SSM) (Song al., 2020), Denoising Score Matching (DSM) (Vincent, 2011), Cur- al.,
Method
Exact Samples
Generator Samples
VERA
7
3
q 7
(A 1)
7
3
2
3
j
J
VERA
2
4
9
(A 0)
vature Propagation (CP-SM) (Martens 2012), and CoopNets (Xie et al., 2018). As ablation we also train VERA with the HMC- based entropy regularizer of Dieng et al. (2019), denoted VERA-(HMC), Table 2 shows that VERA outperforms all approaches that do not require a normalized model.
4
9
3
3
5
7
3
7
3
MEG
5
15
7
7
7
3
PCD
Figure 2 shows exact samples from NICE models. For PCD we can see (as observed in Figure that while the approximate MCMC samples resemble the data distribution, the true samples from the model do not. This further reflected in the reported likelihood value which falls behind all methods besides DSM. Coop- Nets perform better than PCD, but exhibit the same behavior of generator samples resembling the data, but not matching true samples. We at- tribute this behavior to the method's reliance on MCMC sampling.
CoopNet
0
SSM
Not Applicable
Figure 2: Left: Exact samples from NICE model trained with various methods. Right: Approxi- mate samples used for training. For VERA, MEG, and CoopNet, these come from the generator, for PCD these are approximate MCMC samples.
Conversely, models trained with VERA gen- erate coherent and diverse samples which rea- sonably capture the data distribution. We also that the samples from the learned generator much more closely match true samples from the NICE model than PCD and MEG. When we re- move the entropy regularizer from the generator (A 0.0) observe considerable decrease in likelihood and we find that the generator samples are far less diverse and do not match exact samples all. Intriguingly, entropy-free VERA outperforms most other methods. We believe this is because even without the entropy regularizer we are still optimizing a (weak) bound on likelihood. Con- versely, the score-matee matching methods minimize an alternative divergence which will not necessarily correlate well with likelihood. Further, Figure 2 shows that MEG performs on par with entropy- free VERA indicating that the Mutual Information-bant entropy estimator may not be accurate enough in high dimensions to encourage high entropy generators.
Maximum Likelihood
VERA
VERA (HMC)
MEG
PCD
SSM
DSM
CP-SM
CoopNet
1.0
À 0.0
X 1.0
-791
-1138
-1214
-1165
-1219
-4207
-2039
-4363
-1517
-1465
Table 2: Fitting NICE models using various learning approaches for unnormalized models. Results for SSM, DCM, CP-SM taken from Song et al. (2020).
5.2 UNDERSTANDING OUR ENTROPY REGULARIZER
In Figure 3, we explore the quality of our score function estimator on PCA (Tipping & Bishop, 1999) model fit to MNIST, setting where we can compute the score function exactly (see Ap- pendix B.4 for details). The importance sampling estimator (with 20 importance samples) has some- what larger variance than the HMC-based estimator but has notably lower bias of .12. The HMC estimator using 2 burn-in steps (recommended in Dieng et al. (2019)) has bias of .48. Increasing the burn-in steps 500 reduces the bias to .20 while increasing variance. We find the additional variance of our estimator is remedied by the improved performance in Table 2.
averaging and the reduced bias helps explain

---

Published as a conference paper at ICLR 2021
Further, we compute the effective sample size (Kong, 1992) (ESS) of our importance sampling proposal on our CIFARIO and MNIST models and achieve an ESS of 1.32 and 1.29, respectively using 20 importance samples. When uninformed proposal (A(00, I)) is used, the ESS is 1.0 for both models. This indicates our gradient estimates are informative for training. More details can be found in Appendix B.6.
Next, we count the number of modes captured on a dataset with 1,000 modes consisting of 3 MNIST digits stacked on top of one another (Dieng et al.. 2019; Kumar al., 2019). We find that both VERA and VERA (HMC) recover 999 modes, but training with no entropy regularization recovers 998 modes. We conclude that entropy regularization unnecessary for preventing mode collapse in this setting.
APPLICATIONS TO JOINT ENERGY MODELS
Joint Energy Models (JEM) (Grathwohl et al., 2019)
0.5
are an exciting application of EBMs. They reinterpret
Bias (per dimension)
VERA
0.4
VERA (HMC)
standard classifiers as EBMs and train them as such to create powerful hybrid gencrative/discriinnaty models
0.3
which improve upon qurely-disrimminatvve models at out- of-distribution detection, calibration, and adversarial ro- bustness.
0.2
0.1
Traditionally, classification tasks are solved with a func- tion fo I + E2 which maps from the data to K un-
0
100
200
300
400
500
Burn-in steps / Importance Samples
Std. Dev. (per dimensior
constrained real-valued outputs (where l is the number of classes). This function parameterizes distribution over
10
labels y given data x: (y]]ar) ed(a)ll) /Ey efo(a)lol),
The same function fo can be used define an EBM for the joint distribution over æ and as: pe( y) = efo())lll (Z) The label can be marginalized out to
5
give an unconditional model p=( = Ey efo(z))l)) (lo())ll//(()) (0).
0 100 200 300 400 500 Burn-in steps / Importance Samples
JEM models are trained to maximize the factorized like- lihood:
Figure 3: Bias (top) and standard devi- ation (bottom), both per dimension, of the score function estimator using HMC and our proposed importance sampling scheme.
lo (zz v) log (1)lz)) + logps(z) (11)
where a is scalar which weights the two terms. The first term is optimized with cross-entropy and the second term is optimized using EBM training methods. In Grathwohl et al. (2019) PCD was used to train the second term. We train JEM models on CIFARIO, CIFARIO0, and SVHN using VERA instead of PCD. We examine how this change impacts accuracy, generation, training speed, and stability. Full experimental details can be found in Appendix B.7.
Speed and Stability While the results presented in Grathwohl et al. (2019) are promising, training models as presented in this work is challenging. MCMC sampling can be slow and training can easily diverge. Our CIFAR10 10 models train 2.8x faster than the official JEM implementation with the default paraeeesss With these default parameters JEM models would regularly diverge. To train for the reported 200 epochs training needed be restarted multiple times and the number of MCMC steps needed to be quadrupled, greatly increasing run-time.
Conversely, we find that VERA was much more stable and our models never diverged. This allowed us to remove the additive Gaussian noise added to the data which is very important to stabilize MCMC training (Grathwohl et al., 2019; Nijkamp et al., 2019a; Du & Mordatch, 2019).
Hybrid Modeling In Tables 3 and 4 we compare the discriminative and generative performance of JEM models trained with VERA, PCD (JEM), and HDGE (Liu & Abbeel, 2020). With a = 1 we find that VERA leads to models with poor classification performance but strong generation per- formance. With a = 100 VERA obtains stronger classification performance than the original JEM model while still having improved image generation over JEM and HDGE (evaluated with FID (Heusel et al., 2017)).
'ttpps: www.hh........c.eeeoe.ettt cotpbbbtt////attthc hwtm//ggrathwhhhoooo JJRM

---

Published as a conference paper at ICLR 2021
Model
CIFARIO
CIFARI00
SVHN
Model
FID
Classifier
95.8
74.2
97.7
JEM
38,4
HDGE
37.6
JEM
92.9
72.2
96.6
SNGAN (Miyato et al., 2018)
25.50
HDGE
96.7
74.5
N/A
NCSN (Song Ermon, 2019b)
23.52
VERA 100
93.2
72.2
96.8
VERA a = 100
30.5
VERA a =
76.1
48.7
94.2
VERA =
27.5
Table 3: Classification on image datasets.
Table 4: FID on CIFARIO.
Unconditional samples can be seen from our CIFARIO and CIFARIO0 models in Figure 4. Samples are refined through simple iterative procedure using the latent space of our generator, explained in Appendix B.7.1. Additional conditional samples can be found in Appendix C.5
Figure 4: Unconditional samples CIFARIO (left) and CIFAR100 (right).
Out-of-Distriuution Detection JEM is a powerful approach for out-of-distribution detection (OOD), greatly outperforming tractable likelihood models like VAEs and flows (Nalisnick et al., 2018). In Table 5, reporting AUROC (Hendrycks & Gimpel, 2016), we see that for all but 1 dataset, VERA outperforms JEM with PCD training but under-performs contrastive training (HDGE). In- triguingly, VERA performs poorly on CelebA. This result, along with the unreliable performance of DSM models at this task (Li al., 2019) leads to questions regarding special benefits of MCMC training that are lost in our method as well as DSM. We leave this to future work.
Model
SVHN
CIFARI00
CIFARIO-Interp
CelebA
JEM
0.67
0.67
0.65
0.75
HDGE
0.96
0.91
0.82
0.80
GLOW
0.05
0.51
0.55
0.57 0.86
VERA
0.83
0.73
0.33
Table 5: Out-of-distribution Detection. Model trained on CIFARIO. Values are AUROC (1).
6.1 TABULAR DATA
Training with VERA is much more stable and easy to apply to domains beyond images where EBM training has been extensively tuned. To demonstrate this we show that JEM models trained with VERA can provide benefit semi-supervised classification datasets from variety of domains. Considerable progress has been made in semi-supervised learning but the most impressive results require considerable domain knowledge (Chen et al., 2020). In domains like images, text, and audio such knowledge exists but for data from particle accelerators, gyroscopes, and satellites, such intuition may not be available and these techniques cannot be applied. In these settings there are far fewer options for semi-supervised learning.
We present VERA as such option. We train semi-supervised JEM models on data from a variety of continuous domains. We perform no data augmentation beyond removing redundant features and standardizing the remaining features. To further demonstrate the versatility of VERA we use an identical network for each dataset and method. Full experimental details can be found in Appendix B.8.
In Table 6, we find on each dataset tested, we find that VERA outperforms the supervised baseline and outperforms VAT which is the strongest domain agnostic cemi-supervised learning method we aware of.
RELATED WORK
Kumar et al. (2019) train EBMs using entropy- reguaay-re generators, attempting to optimize the same objective our own. The key difference is how the generator's entropy regularized. Kumar
We MNIST tabular dataset since
convolutional architectures.
8

---

Published as a conference paper at ICLR 2021
Model
HEPMASS
CROP
HUMAN
MNIST
Supervised Baseline
81.2
87.8
81.0
91.0
VAT
86.7
94.5
84.0
98.6
MEG
72.1
87.5
83.5
94.7
JEM
54.2
18.5
77.2
10.2
VERA
88.3
94.9
89.9
98.6
Full-label
90.9
99.7
98.0
99.5
Table 6: Accuracy of semi-supervised learning on tabular data with 10 labeled examples per class.
et al. (2019) utilize Mutual Information estimator to approximate the generator's ' entropy whereas approximate the gradients of the entropy directly. The method of Kumar al. (2019) requires the training of an additional Ml-estimation network, but our approach only requires the optimization of the posterior variance which has considerably fewer parameters. As demonstrated in Section 5.1, their approach does not perform as well as VERA for training NICE models and their generator collapses single point. This is likely due the notorious difficulty of estimating MI in high dimensions and the unreliability of current approaches for this task (McAllester & Stratos, 2020; Song & Ermon, 2019a).
Dai et al. (2019) train EBMs using the same objective as VERA. The key difference here is in the architecture of the generator. Dai et (2019) use generators with a restricted form, inspired by various MCMC sampling methods. In this setting, a convenient estimator for the generator's entropy can be derived. In contrast, our generators have an unconstrained architecture and we focus on entropy regularization in the unconstrained setting.
Abbasnejad al. (2019) train EBMs and generators as well to minimize the reverse KI.-divergence. Their method differs from ours in the architecture of the generator and the method for encouraging high entropy. Their generator defines an implicit density (unlike ours which defines latent-variable model). The entropy is maximized using series approximation to the generator function's 's Jacobian log-determinant which approximates the change-of shange-of-variablee variables for injective functions.
Gao et al. (2020) train EBMs using Noise Contrastive Estimation where the noise distribution is a normalizing flow. Their training objective differs from ours and their generator restricted to having normalizing flow architecture. These architectures do not scale as well as the GAN-style architectures we use to large image datasets.
As well there exist CoopNets (Xie al., 2018) which cooperatively train EBM and generator network. Architecturally, they are similar to VERA but are trained quit differently. In CoopNets, the generator trained via maximum likelihood on its own samples refined using MCMC the EBM. This maximum likleihood step requires MCMC as well to generate posterior samples as in Pang et al. (2020a). In contrast, the generator VERA trained minimize the reverse KL-divergence. Our method requires no MCMC and was specifically developed to alleviate the difficulties of MCMC sampling.
The estimator of Dieng et al. (2019) was very influential to our work. Their work focused on applications to GANs. Our estimator could easily applied in this setting and to implicit variational inference (Titsias & Ruiz, 2019) as well but we leave this for future work.
CONCLUSION
In this work we have presented VERA, a simple and easy-to-tune approach for training unnormal- ized density models. We have demonstrated our approach learns high quality energy functions and models with high likelihood (when available for evaluation). We have further demonstrated the su- perior stability and speed of VERA compared to PCD training, enabling much faster training of JEM (Grathwohl et al., 2019) while retaining the performance of the original work. We have shown that VERA train models from multiple data domains with additional tuning. This enables the applications of JEM to sumi-super classification on tabular data outperforming a strong baseline method for this task and greatly outperforming JEM with PCD training.

---

Published as a conference paper at ICLR 2021
REFERENCES
M Ehsan Abbasnejad, Qinfeng Shi, Anton van den Hengel, and Lingqiao Liu. A generative adver- sarial density estimator. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 10782-10791, 2019.
JE Besag. Comments on "representations of knowledge in complex systems" by u. grenander and mi miller. J. Roy. Statist. Soc. Ser: B, 56:591-592, 1994.
Ciwan Ceylan and Michael U Gutmann. Conditional noise-contrastive estimation of unnormalised models. arXiv preprint arXiv:1806.000
2018.
Tong Che, Ruixiang Zhang, Jascha Sohl- Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your gan is secretly an energy-based model and you should use discriminator driven latent sampling. arXiv preprint arXiv:2003.06000 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint wrXiv:2002.05709 2020.
Bo Dai, Zhen Liu, Hanjun Dai, Niao He, Arthur Gretton, Song, and Dale Schuurmans. Exponen- tial family estimation via adversarial dynamics embedding. In Advances in Neural Information Processing Systems, pp. 10979-10990, 2019.
Zihang Dai, Amjad Almahairi, Philip Bachman, Eduard Hovy, and Aaron Courville. Calibrating energy-based generative adversarial networks. arXiv preprint arXiv: 1702.0/690, 2017.
Adji B Dieng, Francisco JR Ruiz, David M Blei, and Michalis K Titsias. Prescribed generative adversarial networks. arXiv preprint arXiv: 091.//9000040 10.04332, 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv:1410.8516. 2014.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv: 000199993.8999 2019.
Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, and Alexander Rives. Energy- based models for atomic-resolution protein conformations. arXiv preprint arXiv:2004.13167, 2020.
Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7518-7528, 2020.
Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXhv:1912.03263 2019.
Will Grathwohl, Kuan-Chiee Wang, Jörn- Jörn-Henrik Jacobsen, David Duvenaud, and Richard Zemel. Learning the stein discrepancy for training and evaluating energy-based models without sampling. 2020.
Michael Gutmann and Aapo Hyvärinen. Noise- contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distributiin examples in neural networks. arXiv preprint arXiv:1610.02136, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Mitch Hill, Jonathan Mitchell, and Song-Chun Chun Zhu. Stochastic security: Adversarial defense using long-run dynamics of energy-based models. arXiv preprint arXiv:2005. 13525, 2020.
10

---

Published as a conference paper at ICLR 2021
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, ((Apr))695-70%% 2005.
John Ingraham, Adam Riesselman, Chris Sander, and Debora S Marks. Learning protein structure with a differentiable simulator. In ICLR, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 2015.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:/412.6908 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes.
arXiv preprint
arXiv: 1312.61 14, 2013.
Durk P Kingma and Prafulla Dhariwal. Glow:
Generative flow with invertible 1x1 convolutions. In Advances in neural information processing systems, pp- 10215-10224, 2018.
Augustine Kong. A note on importance sampling using standardized weights. University of Chicago,
Dept. of Statistics, Tech. Rep, 348, 1992.
Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprini 1rXiv:1911008508 2019.
Zengyi Li, Yubei Chen, and Friedrich T Sommer. Annealed denoising score matching: Learning energy-based models in high-dimensional spaces, arXiv preprint arXhv:1910.07762. 2019.
Jac Hyun Lim, Aaron Courville, Christopher Pal, and Chin-Wei Huang. Ar-dae: Towards unbiased neural entropy gradient estimation. arXiv preprint arXiv:2006. 05164, 2020.
Hao Liu and Pieter Abbeel. Hybrid discriminative-eeneratie training via contrastive learning. arXiv preprint :tt:::20070090777 2020.
James Martens, Ilya Sutskever, and Kevin Swersky. Estimating the hessian by back backkpropaggting curvature. arXiv preprint arXiv: h 1206.6464, 2012.
David McAllester and Karl Stratos. Formal limitations on the measurement mutual information. In International Conference on Artificial Intelligence and Statistics, pp. 875-884, 2020.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: 1e02.0995.77 2018.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint 1rXiv:1810.09138 2018.
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
Erik Nijkamp, Mitch Hill, Tian Han, Song- Zhu, and Ying Nian Wu. On the anatomy of meme- based maximum likelihood learning of energy-based models. arXiv preprint 19Xiv:1033.11 12370, 2019a.
Erik Nijkamp. Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run meme toward energy-based model. In Advances in Neural Information Pro- cessing Systems, pp.
2019b.
Frank Noé, Simon Olsson, Jonas Köhler, and Hao Wu. Boltzmann generators: Sampling equilibrium states of many-body systems with deep learning. Science, 365(6457) 1477, 2019.
Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. Advances in Neural Information Processing Systems, 33, 2020a.
Tianyu Pang, Kun Xu, Chongxuan Li, Yang Song, Stefano Ermon, and Jun Zhu. Efficient learning of generative models via -nite-difference score matching. arXiv preprint 200%::200. 2020b.
11

---

Published as a conference paper at ICLR 2021
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 2015.
Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In International Conference on Machine Learning, pp. 324-333, 2016.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:/505.0577 2015.
Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. arXiv preprint arXiv:2006. 2020.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Shashank Singh and Barnabás Póczos. Analysis of k-nearest neighbor distances with application to entropy estimation. arXiv preprint 1rXiv:1603.06: 2016.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. arXiv preprint arXiv:1910.06222, 2019a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11918-11930 2019b.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. arXiv preprint arXiv:2006.090/ 1, 2020.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pp. 574-584 PMLR, 2020.
Yunfu Song and Zhijian Ou. Learning neural random fields with inclusive auxiliary generators. arXiv preprini arXiv: 0006::v80000222. 2018
Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In International Conference on Machine Learning, pp. 1064-1071, 2008.
Michael E Tipping and Christopher M Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology), (133)611-6622 1999.
Michalis K Titsias and Francisco Ruiz. Unbiased implicit variational inference. In The 22nd Inter- national Conference Artificial Intelligence and Statistics, pp. 167-176, 2019.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- 2011.
tation,
Martin J Wainwright and Michael Irwin Jordan. Graphical models, exponential families, and vari- ational inference. Now Publishers Inc, 2008
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-I1), pp. 681-688, 2011.
Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of de- scriptor and generator networks. IEEE transactions on pattern analysis and machine intelligence, 42(1):27-45, 2018.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.00114..000 2016.
Wide residual networks.
arXiv preprint
Stephen Zhao, Jörn-Henrik Jacobsen, and Will Grathwohl. Joint energy-based models for semi- supervised classification.
12

---

Published as a conference paper at ICLR 2021
A KEY DERIVATIONS
A.1 DERIVATION OF VARIATIONAL. LOG-PARTITION FUNCTION
Here we show that the variational optimization given in equation 3 recovers log 2(0).
max E (f (())
[. (())
= max
((()))ogg) log((()))d
exp(fo(x))
max
g(>) log
dx
q(x)
exp(fo(a))
max
log
dx
log Z(0) + log 2
q(x)
(xp(Jo(x))/2(0)
max
((x)) log
dx + log Z(0)
((x)
max
(KL((())))po( + log ((0)
= log Z(0).
A.2 SCORE FUNCTION ESTIMATOR
Here derive the equivalent expression for v log (x() given in Equation 7.
Vz lo) Voga(1)
g0(z)
Vx Je 90, 2)d2
(1()))
v (1, z)dz
(0(x))
V V2 2)
dz
(x(x))
Va =)) dz
(x(xx)
(v log qo (x) 2)) 2)))(2)
dz
(I(
(elx)) V log (x(l))))
A.3 ENTROPY GRADIENT ESTIMATOR
From Equation 6 we have
V Voll(go) Ep(zo)p(c) V log (z e)].
13

---

Published as a conference paper at ICLR 2021
Plugging in score function estimator gives
V H (go) ((())
V x log gs (o))VVz
(12)
(()))
E go(l)))
[V qo( 2)]² Vor(20.e)
g0(= (x)
((()))((e)
E (z((zo)
V log 2)
Vor(20,))
En
20)
[0(()))((z =0)) V log (oggg(()) 12) Vartzone)
9 (2,
(p))))((c)
E (=11za)
p())))())
Ve loggo(<> (2))
Voo(zone)
k
2==1 wy
where { E(2 20) and w,
(a(().m) E(2, zu)
A.3.1 DISCUSSION
We discuss when the approximations in Equation 13 will hold. The importance sampling estimator will be biased when (s 2) differs greatly from E(2 20). Since the generator function Iv (z) is smooth, Lipschitz function (as are most neural networks) and the output Gaussian noise is small, the space of z values which could have generated æ should be concentrated near 2o. In these settings, 20 should be useful for predicting (2(/2))
The accuracy of this approximation is based on the dimension of compared and the Lipschitz constant of gei In all settings we tested, dim(2) < dim(x) where this approximation should hold. dim(2) 2) >> dim(x) the curse of dimensionality would take effect and 2u would be less and less informative about (2(xr) In settings such as these, we do not believe our approach would be as effective. Thankfully, almost all generator architectures we are aware of have dim(2) < dim(x). The approximation could also break down if the Lipschitz constant blew up. We find this does not happen in practice, but this can be addressed with many forms of regularization and normalization.
14

---

Published as a conference paper at ICLR 2021
B EXPERIMENTAL DETAILS
B.I HYPERPARAMETER RECOMMENDATIONS
Type
Hyperparameter
Value
Optimization
31 (ADAM)
0
32 (ADAM)
0.9
learning rate (energy)
10-4*
learning rate (generator)
2 10 1-"0" 4*
learning rate (posterior)
2 1010"
Regularization
(entropy regularization weight)
10-41 41
(gradient norm penalty)
0.1
JEM
(classification weight)
{1, 30, 1003
(classification entropy)
{1.0.1 aa,a)
Table 7: Hyperparameters for VERA.
"When > 1, learning rates were divided by a.
Te found 10 to work best on large image datasets, but in general we recommend starting with X = 1 and trying successively smaller values of À until training is stable.
We give some general tips on how to set hyperparameters when training VERA in Table In all VERA experiments, we use the gradient norm penalty with weight 0.1. This was not tuned during our experiments. When using VERA and MEG we train with Adam (Kingma Ba, 2014) and set 31 = 0, f3 .9 as is standard in the GAN literature (Miyato et al., 2018). In general, recommend setting the learning rate for the generator to twice the learning rate of the energy function and equal to the learning rate of the approximate posterior sampler.
B.1.1 IMPACT OF À
Let us rewrite the generator's training objective
L(y; A) = (e(o)[ + XH
(13)
We can easily see that this objective equivalent (up to multiplicative constant)
fo(x)
Ex(x)
+ H(q).
(14)
From this, it is clear that maximizing Equation 14 is the same minimizing the KL-divergence between q and tempered version pe (x) defined as
efo(z)AA
(15)
Z
Tempering like this is standard practice in EBM training and is done in many recent works. Tem- pering has the effect of increasing the weight of the gradient signal in SGLD sampling relative to the added Gaussian noise. In all of Du & Mordatch (2019); Grathwohl et al. (2019); Nijkamp et al. (2019a:b) the SGLD samplers used a temperature of 1/2 = 20,000. This value near the value of 10, 000 that we use in this work.
Thus we can see that our most important hyper- parameer is actually the temperature the sampler's target distribution. Idcally this temperature would be set to 1, but this can lead to unstable training (as it does with SGLD). To train high quality models, we recommend setting as close to as possible, decreasing if training becomes unstable.
B.2 TOY DATA VISUALIZATIONS
We train simple energy functions on a 2D toy data. The EBM is 2-layer MLP with 100 hidden units per layer using the Leaky-Rel.U nonlinearity with negative slope .2. The generator is 2-layer
15

---

Published as a conference paper at ICLR 2021
MLP with 100 hidden units per layer and uses batch normalization (loffe & Szegedy, 2015) with ReLU nonlincarities. All models were trained for 100,000 iterations with all learning rates set to .001 and used the Adam optimizer (Kingma & Ba, 2014).
The PCD models were trained using an SGLD sampler and a replay buffer with 10,000 examples, reinitialized every iteration with 5% probability. We used 20 steps of SGLD per training iteration to make runtime consistent with VERA. We tested a values outside of the presented range but smaller values did not produce decent samples or energy functions and for larger values, training diverged.
B.3 TRAINING NICE MODELS
The NICE models were exactly as in Song et al. (2020). They have 4 coupling layers and each coupling layer had 5 hidden layers. Each hidden layer has 1000 units and uses the Softplus non- linearity. We preprocessed the data as in Song et al. (2020) by scaling the data to the range [0, 1J, adding uniform noise in the range [-1/512, 1/512], clipping to the range [.001, .999] and applying the logit transform log(x:) log(1 - z). All models were trained for 400 epochs with the Adam optimizer (Kingma& & Ba, 2014) with 3 = 0 and 32 = .9. We use a batch size of 128 for all models. We re-ran the score matching model of Song et al. (2020) to train for 400 epochs as well and found it did not improve performance as its best test performance happens very early in training.
For all generator based training methods has a latent-dimension of 100 and 2 hidden layers with 500 units each. We use the Softplus nonlin- earity and batch-normalization (Toffe & Szegedy, 2015) as common with generator networks.
use the same fixed generator architecture. The generator
For VERA the hyper-parameters we searched over were the learning rates for the NICE model and for the generator. Compared to Song et al. (2020) we needed use much lower learning rates. We searched over learning rates in {.0003, .00000000000000 .000003] } for both the generator and energy function. We found .000003 to work best for the energy function and .0003 to work best for the generator. This makes intuitive sense since the generator needs to be fully optimized for the bound on likelihood to be tight. When equal learning rates were used (.0003,.00 .0003) we observed high sample quality from the generator but exact samples and likelihoods from the NICE model were very poor.
For PCD we search over learning rates in {.0003, .000030 0000003), the number of MCMC steps in (20,40), and the SGLDD noise standard-deviation in 0.13 All models with 20 steps and 1.0 quickly diverged. Our best model used learning rate .000003, step- size 0.1, and 40 steps. We tested the gradient-norm regularizer from Kumar et al. (2019) and found it decreased performance for PCD trained models. Most models with 20 MCMC steps diverged early in training.
SGLD
For review the MCMC sampler we use is stochastic gradient Langevin dynamics (Welling & Teh, 201 This sampler updates its samples by
St 3z
2 Vo + ca, 10,
(16)
where is the noise standard-deviation and is a parameter of the sampler.
For Maximum Entropy Generators (MEG) (Kumar et al., 2019) we must choose a mutual- information estimation network. We follow their work and use an MLP with LeakyReLU non- linearities with negative slope .2. Our network mirrors the generator and has 3 hidden layers with 500 units each. We searched over the same hyper-parameters as VERA. We found MEG to perform almost identically to training with no entropy regularization at all. We believe this has to do with the challenges of estimating MI in high dimensions Song & Ermon (2019a).
For CoopNets (Xie et al., 2018) we use the same flow and generator architectures as VERA. Fol- lowing the MNIST experiments in Xie et al. (2018) we train using 10 SGLD steps per iteration. We tried the recommended learning rates of .007 and .0001 for the flow and generator, respectively but found this to lead quick divergence. For this reason, we search over learning rates for the flow and generator from {.0003, .00003, .000003) as we did for VERA and found the best combination to be .000003 for the flow and ,0003 for the generator. Other combinations resulted in higher quality generator samples but much worse likelihood values. We tested the recommended SGLD step-size of .002 and found this to lead to divergence as well in this setup. Thus, we searched over larger values of (.002,-02 1} found... 1 to perform the best, as with PCD.
16

---

Published as a conference paper at ICLR 2021
B.4 ESTIMATION OF BIAS OF ENTROPY REGULARIZER
If we restrict the form of our generator to a linear function
I == Wz + + +atoc, aC.
2,C~N(0,1)
then we have
(<<|2) -
((x)=N(u W +
meaning we can exactly compute log g(z). and v log (())) which the quantity that VERA (HMC) approximates with the HMC estimator from Dieng et al. (2019) and we approximate with VERA. To explore this, we fit a PCA model on MNIST and recover the parameters W, f the linear generator and the noise parameter a. Samples from this model can be seen in Figure 5.
Both VERA and VERA (HMC) have some parameters which are tuned automatically to improve the estimator throughout training. For VERA this is the posterior variance which is optimized according to Equation 10 with Adam with default hyperparameters. For VERA (HMC) this is the stepsize of HMC which is tuned automatically with a learning rate of .01 for 500 iterations with a batch size of 5000. Samples from the estimators during training were taken with the default parameters, in partieular the number of burn-in steps or the number of importance samples was not varied as they were during evaluation.
outlined in (Dieng al., 2019). Both estimators were trained
The bias of the estimators example in the batch, 5000 estimates were taken from the estimator and averaged to be taken as an estimate of the score function for this batch example. This estimate of the score function was subtracted from the true score function and then averaged over all dimensions and examples in the batch and taken as an estimate of the bias per dimension.
evaluated
batch of 10 samples from the generator. For each
For VERA (HMC) we varied the number of burn-in steps used for samples to evaluate the bias of the estimator. We also tried to increase the number of posterior samples taken off the chain, but we found that this did not clearly reduce the bias of this estimator as the number of samples increased. For VERA we computed the bias on the default of 20 importance samples.
4
4
5
0
%
8 6 2
2
5
7
4
4 4
7
?
3
0
3
5
9
4
9
4
6
8
8
2
H
GF
2/
9
3
4
6
R
8
8
41
4 4
7
47
3
30
04
3
1
4
47
Figure 5: Samples from linear model trained with PCA on MNIST.
B.5 MODE COUNTING
We train VERA without labels on the STACKEDMNIST dataset. This dataset consists of 60,000 examples of stacked MNIST images sampled with replacement from the original 60,000 MNIST train set. As in (Dieng al., 2019) we resize the images to 64 x 64 and use the DCGAN (Radford et al., 2015) architecture for both the energy-function and generator. We use a latent code of 100 dimensions. We train for 17 epochs with a learning rate of .001 and batch size 100.
We estimate the number modes captured by taking 5 10, 000 samples as in (Dieng et al., 2019) and classifying each digit of the 3 stacked images separately with pre-trained classifier MNIST.
B.6 EFFECTIVE SAMPLE SIZE
When performing importance sampling, the quality of the proposal distribution has a large impact. If the proposal is chosen poorly, then typically 1 sample will dominate in the expectation. This can
17

---

Published as a conference paper at ICLR 2021
be quantified using the effective sample size (Kong, 1992) (ESS) which is defined as (
w;
g(z))
W/
N
j=1 Wj
1
ESS
(17)
E
where p(x) is the target distribution, ((x) is the proposal distribution and N is the number of sam- ples. If the self-normalized importance weights dominated by one weight close to I then the ESS will be 1. If the proposal distribution is identical to the target, so that the self-normalized im- portance weights are then uniform, then the ESS will be N. When ESS N, importance sampling is as efficient using the target distribution.
To understand the effect of the proposal distribution on ESS we plot the ESS when doing importance sampling with 20 importance samples from 1 228-dimensional Gaussian target distribution with J = and Σ I. We use a proposal which a 128-dimensional Gaussian with f increasing from 0 to 5. Results can be seen in Figure 6. We see when the means differ by greater than 2, the ESS is approximately 1.0 and importance sampling has effectively failed.
ESS as Proposel Degrades
3
Mr.
Figure Effective sample size of importance sampling as the proposal degrades.
B.7
JEM MODELS
Our energy function used the same architecture as in Grathwohl al. (2019); Liu & Abbeel (2020), a Wide ResNet Zagoruyko & Komodakis (2016) 28-10 with batch-norm removed and with dropout. Our generator architecture is identical Miyato al. (2018). As in Grathwohl al. (2019); Liu & Abbeel (2020) we set the learning rate for the energy function equal to 0.0001. We set the learning rate for the generator equal to 0.0002. We train for 200 epochs using the Adam optimizer with 31 = O and B2 = .9. We set the batch size to 64. Results presented are from the models after 200 epochs with no early stopping. We believe better results could be obtained from further training.
We trained models with a E 30, 100} and found classification to be best with = 100 and generation to be best with a =
Prior work on PCD EBM training (Grathwohl et al., 2019; Du & Mordatch, 2019; Nijkamp et al., 2019a;b) recommends adding Gaussian noise to the data to stabilize training. Without this, PCD training of JEM models very quickly diverges. Early in our experiments we found training with VERA was stable without the addition of Gaussian noise so we do not use it.
As mentioned in Dieng al. (2019), when the strength of the entropy regularizer À is too high, the generator may fall into a degenerate optimum where it just outputs high-entropy Gaussian noise. To combat this, suggested in Dieng al. (2019), decrease the strength of À .0001 for all JEM experiments. This value was chosen by decreasing from 1.0 by factors of 10 until learning took place (quantified by classification accuracy).
18

---

Published as a conference paper at ICLR 2021
B.7.1 MCMC SAMPLE REFINEMENT
Our generator go (z) is trained to approximate our EBM pe pe(x). After training, the samples from the generator of high quality (see Figures 9 and 10, left) but they not exactly samples from po(x). We can use MCMC sampling to improve the quality of these samples. We use a simple MCMC refinement procedure based on the Metropolis Adjusted Langevin Algorithm (Besag, 1994) applied to an expanded state-space defined by our generator and perform the Accept/Reject step in the data space.
We can reparameterize a generator sample x ~ Is (z) as a function æ(2, c) = 90(2) + ca, and we can define an unnormalized density over e2 log e fo(gs(2) Ea) log 2(0, Φ) which is the density (under pa (z)) of the generator sample.
Starting with initial sample 2o, c0
NO, I) define the proposal distribution
2t- + ((- =
V logh(=-- h0
2
pla1211, (-)) =
2--1 +
Van log h (2)
2
(12-,,1)) = p( - (1)e
and accept sample with probability
min baa--
Here, is the step-size and is a parameter of the sampler. We tune 8 with burn-in period to target an acceptance rate of 0.57.
A similar sampling procedure was proposed in Kumar al. (2019) and Che et al. (2020) and in both works was found to improve sample quality. In all experiments,
We clarify that this procedure is not valid MCMC sampler for (a due to the augmented variables and the change in density of 9v which are not corrected for. The density of the samples will a combination of (1) and q0 (z). As the focus of this work training and not sampling/generation we leave the development of more correct generator ACMC-sampling to future work. Regardless,
find this procedure improve visual sample quality. In Figure 7 visualize sampling chain using the ahove method applied to our JEM model trained on SVHN.
6
Figure 7: Visualization of our MALA-inspinge sample refinement procedure. Samples come from JEM model trained on SVHN. Chains progress the right and down. Each image is a consecutive step, no sub-sampling is done.
19

---

Published as a conference paper at ICLR 2021
B.7.2 IMAGE QUALITY METRICS
We present results on Inception Score (Salimans et al., 2016) and Frechet Inception Distance (Heusel et al., 2017). These metrics are notoriously fickle and different repositories are known to give very different results (Grathwohl et al., 2019). For these evaluations we generate 12,800 samples from the model and (unless otherwise stated) refine the samples with 100 steps of our latent-space MALA procedure (Appendix B.7.1). The code to generate our reported FID comes from this publicly avail- able repository. The code to generate our reported Inception Score can be found here.
B.8 SEMI-SUPERVISED LEARNING ON TABULAR DATA
B.8.1 DATA
We provide details about each of the datasets used for the experiments in Section 6.1. HEPMASSS dataset obtained from a particle accelerator where we must classify signal from background noise. CROP is a dataseset for classifying crop types from optical and radar sensor data. HUMAN" is a dataset for human activity classification from gyroscope data. MNIST is an image dataset of handwritten images, treated here tabular data.
Dataset
HEPMASS
CROP
HUMAN
MNIST
Features
15
174
523
784
Examples
6,300,000
263,926
6,617
60,000
Classes
2
6
10
Max class %
50.0
26.1
19.4
11.2
Table 8: Basic information about each tabular dataset.
For HEPMASS and HUMAN data we remove features which repeat the same exact same value more than times. For CROP data we remove features which have covariation greater than 1.01. For MNIST we linearly standardize features to the interval [-1, 1].
We take random 10% subset of the data use a validation set.
B.8.2 TRAINING
We use the same architecture for all experiments and baselines. It has 6 layers of hidden units with dimensions [1000, 500, 500, 250, 250, 250) and a Leaky-ReLU nonlinearity with negative-slope .2 between each layer of hidden units. The only layers which change between datasets are the input layer and the output layer which change according to the number of features and number of classses respectively.
The training process for semi-supervised learning is similar to JEM with an additional objective commonly used in semi-supervised learning:
log Po(x, y) = a log po x) + log (x + (H x:)
(18)
where (p(ylzz) is the entropy of the predictive distribution over the labels.
For all models report the accuracy the model converged to the held-out validation set. We report the average taken over three training runs with different seeds.
We equal learning rates for the energy model, generator, and the entropy estimator. We tune the learning rate and decay schedule for supervised models on the full-set of labels and 10 labels per class.
http:/cc...cdc////aaateee///eeee optical-radaridatal htttps://archive. iss.uct httpa///ssaa/a/////////a////a//a Usinge+mmrtppoones
20

---

Published as a conference paper at ICLR 2021
On VERA tune the learning rate and decay schedule, the weighting of the entropy regularization and the weighting of the entropy of classification outputs 3.
For VAT tune the perturbation size (.0 .1, 1, 10). All other hyperparameters were fixed according to tuning on VERA.
For MEG used the hyperparameters tuned according VERA.
For JEM we tune the number of MCMC steps in the range K E (00,40, 80}. We generate samples
using SGLD with stepsize 1 and noise standard deviation 0.01 in Grathwohl et al. (2019).
Dataset
HEPMASS
CROP
HUMAN
MNIST
Learning rate
10
5
4
10-3 5
10 5
Number of epochs
1
20
800
200
Batch size
64
64
64
64
Decay rate
3
.3
.3
.3
Decay epochs
{.05,.1}
{15, 17)
{200, 300, 400, 600)
{150, 175]
2 (latent) dimension
10
128
128
128
A
10-4
10-4
10
10-4
O
1
1
e (VAT)
.1
.1
3
(JEM)
20
20
40
80
Table 9: Hyperparameters for semi-supervised learning on tabular data
21

---

Published as a conference paper at ICLR 2021
C ADDITIONAL RESULTS
C.I TRAINING MIXTURES OF GAUSSIANS
We present additional results training mixure of Gaussian models using VERA and PCD. Each model consits of 100 diagonal Gaussians and mixing weights. Our experimental setup and hyper- parameter search was identical that presented in Appendiex B.2. We Table 10 that VERA outperforms PCD.
VERA
Max. Likelihood
PCD
À = 0.0
==.01
À = 0.1
X == 1.0
Moons
-2.32
-3.87
-3.63
-3.10
-2.58
-3.82
Circles
-3.17
-3.58
-3.64
-3.74
-3.40
-4.24
Rings -2.83
-3.5
-3.44
-3.24
-3.17
-4.13
Table 10: Fitting a mixture 100 diagonal Gaussians using ML, MCMC approximate ML and generator approximate ML.
C.2 SAMPLES FROM SSL MODELS
We present some samples from our semi-supervised MNIST models in Figure 8.
9
8
8
7
7
7
4
5
4 3
8
4
5
0
2
3
7
6
2
9
0
3
6
>
0
7
9
4
1
9
9
2
7
6
0
1
8
5
4
0 4
8
/
4
8
3
3
1
C
9
4
5
6
3
9
9 7
5 5
5
8
7
2
Figure 8: Unconditional MNIST Samples. Left: samples from the generator and right: samples after 100 steps if MCMC refinement using MALA.
C.3 HYBRID MODELING
We present an extended Table 4 with Inception Score Salimans et al. (2016) and which includes more comparisons.
Model
FID
IS
JEM
38.4
8.76
HDGE
37.6
9.19
SNGAN (Miyato et al., 2018)
25.50
8.59
NCSN (Song & Ermon, 2019b)
23.52
8.91
ADE (Dai et al., 2019)
N/A
7.55
IGEBM (Du & Mordatch, 2019)
37.9
8.30
Glow (Kingma & Dhariwal, 2018)
48.9
3.92
FCE (Gao et al., 2020)
37.3
N/A
VERA a 100
30.5
8.11
VERA
27.5
8.00
VERA a = (generator samples)
32.4
7.34
22

---

Published as a conference paper at ICLR 2021
C.4
UNCONDITIONAL IMAGE GENERATION: CIFARIO
Figure 9: Unconditional CIFARIO Samples. Left: samples from the generator and right: samples after 100 steps if MCMC refinement using MALA.
C.5 UNCONDITIONAL IMAGE GENERATION: CIFARI00
Figure 10: Unconditional CIFARI00 Samples, Left: samples from the generator and right: samples
after 100 steps if MCMC refinement using MALA.
23

---

Published as a conference paper at ICLR 2021
C.6
CONDITIONAL IMAGE GENERATION: CIFARIO AND CIFAR100
Figure 11: Class-conditional samples from CIFARIO
24

---

Published as a conference paper at ICLR 2021
Figure 12: Class-conditional samples from CIFARIO
25

---

Published as a conference paper at ICLR 2021
Figure 13: Class-conditional samples from CIFAR10
26

---

Published as a conference paper at ICLR 2021
Figure 14: Class-conditional samples from CIFARIO
27

---

Published as a conference paper at ICLR 2021
Figure 15: Class-conditional samples from CIFARI0
28

---

Published as a conference paper at ICLR 2021
Figure 16: Class-conditional samples from CIFARIO
29

---

Published as a conference paper at ICLR 2021
Figure 17: Class-conditional samples from CIFARIO
30

---

Published as a conference paper at ICLR 2021
Figure 18: Class-conditional samples from CIFAR10
31

---

Published as a conference paper at ICLR 2021
Figure 19: Class-conditional samples from CIFAR1O
32

---

Published as a conference paper at ICLR 2021
Figure 20: Class-conditional samples from CIFARIO
Figure 21: Class-conditional samples from CIFAR100
33

---

Published as a conference paper at ICLR 2021
Figure 22: Class-conditional samples from CIFAR100
Figure 23: Class-conditional samples from CIFAR100
Figure 24: Class-conditional samples from CIFAR 100
34

---

Published as a conference paper at ICLR 2021
Figure 25: Class-condittonal samples from CIFARI00
Figure 26: Class-conditional samples from CIFAR100
Figure 27: cass-conditional samples from CIFAR100
35


Reviews:
[{"review_id": "d5eBVKdgkuH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method on training energy-based models with maximum likelihood. Instead of using MCMC approaches to sample from the EBM, authors follow previous work on training neural generators for faster sample generation. In particular, authors consider a special generator where the output is convolved with Gaussian noise. The score function of this generator can be estimated with self-normalized importance sampling, which is then used to estimate the entropy term through the reparameterization trick. Authors demonstrate that their new method is able to train EBMs efficiently, and improves the stability and performance of JEMs compared to MCMC-based training approaches.\n\n#### Pros\n* The method is more computationally efficient compared to MCMC-based approaches. As the title suggests, no iterative MCMC approaches are needed. Though the technique is inherently similar to Dieng et al. (2019), authors replaces the HMC sub-routine with a carefully designed variational approximation.\n\n* Experiments on JEMs are particularly interesting. Authors demonstrate that their method can train JEMs in a stable way and outperforms baselines on classification accuracy, sample quality, out-of-distribution detection and semi-supervised learning.\n\n* Writing is clear and easy to follow.\n\n#### Cons\n* From my perspective, the biggest disadvantage is related to various additional hyper-parameters. In VERA training, $\\gamma$ controls the gradient penalty and $\\lambda$ controls the contribution of the estimated entropy gradient. Both requires considerable tuning for optimal performance. However, typical MCMC-based approaches do not require gradient penalty, and tuning $\\lambda$ is unsatisfying — shouldn't $\\lambda \\equiv 1$ for real maximum likelihood training? \n\n* The idea of training a neural generator with the dual form of the likelihood objective has been explored before. Authors have compared with MEG, which uses a similar objective. I think adversarial dynamics embedding is also a necessary baseline, since it uses the same objective and has a special design of the neural generator to make entropy computing tractable. It would be better if authors will include it in both NICE and JEM experiments.\n\n* Authors cited (Song & Ermon, 2019a) when pointing out the difficulty of MI estimation. The reference below should also be included as it actually appeared earlier in 2018.\n\nMcAllester, David, and Karl Stratos. \"Formal limitations on the measurement of mutual information.\" International Conference on Artificial Intelligence and Statistics. 2020.\n\n* Authors did not include methods such as MEG in the JEM experiments. Any reason?\n\n----------------\nPost-rebuttal\n\nI appreciate the authors' response and additional comparison against previous work. I do think that proper comparison with previous work is important, as it allows us to know better when and where the proposed approach is beneficial. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KJEb6ZD1cHv", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an improved algorithm to train EBM-based models, called Variational Entropy Regularized Approximate maximum likelihood. The basic idea is to formulate the intractable partition function as an optimization problem with an additional entropy term. To estimate the gradient of the entropy term, the authors then propose a variational formulation for approximation.\n\nThe idea is interesting but not very exciting, and there consists of technical details that need to be carefully deal with. One of my biggest concerns is that since the algorithm relies on importance sampling to estimate the gradient of log marginal likelihood, the variance can be arbitrarily large. Specifically, the importance weight is the ration between a joint distribution and a proposed variational distribution \\xi. One can see that a sample from \\xi is a zero-mean Gaussian distribution, whereas q(z, x) should not. I suspect this would make the ratios to be vary unstable with high variance, which seems to contradict with the goal of the paper. In other words, I am not convinced why the importance sampling based method can work better than existing methods.\n\nIn addition, this importance sampling based method is introduced to avoid sampling the posterior distribution with HMC. I think this step can be considered as an efficient approximation to the  method, I am expecting HMC should perform at least as good as the proposed method but less efficient. The results in Table 2 seems to agree with this (although not completely). I wonder how is the running time comparison? Also, since HMC based method is a competitive method, why don't you consider this in other experiments such as those in Section 6?\n\nAlso, since the proposed method is claimed to outperform the recent SSM method. I think the same experiments as in the SSM paper should be conducted for comparison.\n\nBTW, the generative images in Appendix are too small to be informative.\n\n=========After rebuttal======\nAfter the rebuttal, my main concern remains. Specifically, the paper defines a variational distribution q(z|x0 via a hierarchical construction: z_0 ~ N(0, 1), z ~ N(z_0, \\eta I), which is essentially a zero-mean Gaussian. And I suspect the this is a bad variational distribution and it will induce high variance. The author said they didn't the hierarchical construction to define the variational distribution, because they fix z_0 after sampling. I don't think this is a formal way of defining a variational distribution. One reason is that even if they fix z_0, the proposal distribution will be a z_0-mean Gaussian, and the mean is randomly drawn from N(0, 1), which will not match the true posterior distribution (they only optimize the variance parameter). I think this should be make clear and investigated in more details. I will keep my initial score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IDPXKRxmg9x", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**: This paper presents a method for improving training of energy-based models. Rather than drawing samples using persistent contrastive divergence / MCMC, this approach parameterizes a separate model, which is trained to directly output samples. This effectively adds an additional KL divergence to the objective. The authors use a particular form of sampling model (a latent Gaussian model), borrowing a few tricks for getting entropy estimates out of the model. Results are demonstrated on a few qualitative setups, but most of the results are centered on improved JEM on sample quality, out-of-distribution detection, and semi-supervised learning. The main benefit of the approach seems to be speed and stability, however, the authors also claim that minimal tuning is needed.\n\n**Strong Points**: Overall, the paper is fairly clear in its description of the background, method and experiments. Likewise, the tables, figures, and algorithm box provide clear demonstrations of key aspects of the approach. The descriptions in Eq. 6 are also useful for unfamiliar readers.\n\nFrom my understanding, the technical aspects of the paper appear to be largely correct. Like previous papers, the authors use an objective that includes both the energy function and the approximate sampler. This includes the entropy of the sampler (from the sampler’s KL). The authors opt for a Gaussian latent variable model as the sampler. Evaluating the entropy thus entails an integral over the posterior of this model. The key technical contribution of this paper is to approximate this integral not with MCMC sampling, but instead use an importance-weighted variational approximation. This involves drawing a sample from the prior, then sampling points from a Gaussian around this point.\n\nThe experimental results in the paper are highly thorough. The authors demonstrate their method in settings where exact log-likelihood estimates are feasible (a flow-based model and probabilistic PCA). This allows the authors to demonstrate both improved samples and log-likelihood performance. In these cases, the authors compare with several relevant baselines. In the main experiments of the paper, the authors incorporate their approximate sampling scheme into JEM, a recent hybrid discriminative/generative model. Experimental results demonstrate that using the approximate sampler generally results in (slight) improvements over JEM, with the exception of semi-supervised classification, where the proposed method is substantially better. In all cases, experiments are performed using standard, benchmark datasets, and the authors compare with competitive, recent methods.\n\nThere are substantial details on the experiments in the appendix. This, combined with the experiments across multiple datasets and settings, suggests that these results are reproducible. However, given the fact that this methods requires training multiple models, exactly reproducing the results will likely require the corresponding code (which the authors did not provide).\n\nWhile the method itself is not highly novel, the thorough experimental results somewhat compensate for this aspect. In particular, at least in the “toy” experiments, the authors compare with MEG, a relevant, related baseline. This demonstrates that, although the proposed method is a natural extension of previous works, the contribution is nevertheless useful.\n\n**Weak Points**: One weak point of this paper is with regard to novelty. The main contribution of this paper is in estimating the integral of the sampling model’s entropy with an importance-weighted variational approximation. This is a fairly natural extension of previous works on learning sampling models, simply making the estimation procedure more efficient. On its own, this is not a highly substantial contribution. It’s also not clear that this is the best set of design choices for this problem. For instance, would the issues with the entropy gradient not be obviated with an exact log-likelihood model, such as a flow-based model / autoregressive model?\n\nSome of the theoretical considerations of the setup appear to be skimmed over. The authors present the approximate sampling method with an equality in Eq. 4. However, in practice, the sampling model will not be able to reach the max, due to inherent limitations in the model class. Further, in Algorithm 1, the energy-based model and the sampling model are trained jointly, rather than training the sampling model inside of an inner loop (for the max operation). This is analogous to amortized inference in VAEs, where training the encoder and decoder jointly typically implies that the encoder cannot fully maximize the ELBO for the decoder. This phenomenon is referred to as “lagging inference networks,” (He et al., 2019) and the performance gap is the “amortization gap” (Cremer et al., 2018). Similarly, there will be a “sampling gap” here due to a “lagging sampling model.” While the method still seems to work well in practice, the fact that this is an approximation is almost entirely missing from the paper. Note that, to generate samples, the authors do indeed run additional MCMC steps (see appendix).\n\nFinally, although the experiments are comprehensive, it’s not clear whether the improvements over JEM are highly substantial. In addition, in the case with the largest gap, semi-supervised learning, it’s unclear why the proposed method, VERA, should dramatically outperform JEM. Given that this is the most substantial result, it seems as though further investigation should be performed to analyze the reason for the boost in performance. Further, the experiments on JEM only compare the baseline models (which uses MCMC) with the proposed (amortized) method. A more complete set of experiments would also include other learned sampling methods, such as MEG.\n\n**Accept / Reject**: Although the method is not highly novel, the experiments are fairly comprehensive in demonstrating the proposed method. Analyses are performed in multiple settings, across multiple datasets, comparing with the relevant baselines. At the very least, this paper demonstrates an efficient method that seems to outperform previous methods in simpler tasks and improve JEM on more difficult tasks. For these reasons, I would be in favor of accepting this paper. With a more detailed motivation/explanation of the design choices, discussion of the quality of the approximation, and some additional sampling model baselines on the JEM results, this paper could be further improved.\n\n**Questions**:\n\nCould you elaborate on the issue with the score function estimator in Eq. 6? Is this simply a matter of having a high-variance estimator?\n\nIn the importance weights, should $q_\\phi (x)$ be included in the denominator?\n\nIn Algorithm 1, are you using a mini-batch of data examples but only one sample from the model. If so, why not use a batch of samples?\n\nThe MNIST samples don’t look great, even for VERA. Why is this the case?\n\nIs using NICE in Section 5.1 representative of energy-based models? Flow-based models tend to have their own issues associated with stability.\n\nWhy is VERA better at semi-supervised learning as compared with JEM?\n\n**Additional Feedback**:\n\nMethod:  \nI would not compare Eq. 5 with a GAN decoder, as GANs define an implicit density.  \n“or simple a diagonal Gaussian” —> “or simply a diagonal Gaussian”\n“to the gradient of our model’s likelihood…”: should specify that this is the EBM.  \n\nEBM Training Experiments: \nI would put zeros in front of all decimals to be consistent.  \nNot sure if it’s fair to conclude that entropy regularization is unnecessary for preventing mode collapse from this experiment. This seems like a limited setting.  \n\nSection A.2:  \n“Equation equation 7” —> “Equation 7”\n\nFigure 7:  \n“Unconditional CIFAR10 Samples” —> “Unconditional MNIST Samples”", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors proposed to train an energy based model with a hierachical
variational approximations. The entropy can be tricky in hierarchical
variational approximations.  The authors suggest using the auxillary
samples to guide an importance samples to compute the gradient of the
entropy. They evaluate their approach on a slew of models. The idea is
straightfoward and could potentially be applied to other hierarchical
variational models out side of the energy-based model setting.  The
authors were responsive and clarified many agressive questions. I'd
ask the authors to clean up two things

- Equation 8 would be easier to follow if it kept the expectation from
  equation 6 thereby making z_0 feel like it materialize out of thin
  air


- A more detailed discusion of when the proposal is good and what could
  be missed out	when relying on the generating z to center the proposal
Decision: Accept (Poster)