MCMC Should Mix: Learning Energy-Based Model with Neural Transport Latent Space MCMC
Authors: Erik Nijkamp
Ruiqi Gao
Pavel Sountsov
Srinivas Vasudevan
Bo Pang
Song-Chun Zhu
Ying Nian Wu
Venue: ICLR 2022 Poster (2022)
Keywords: Generative models
energy-based models
MCMC

Abstract: Learning energy-based model (EBM) requires MCMC sampling of the learned model as an inner loop of the learning algorithm. However, MCMC sampling of EBMs in high-dimensional data space is generally not mixing, because the energy function, which is usually parametrized by deep network, is highly multi-modal in the data space. This is a serious handicap for both theory and practice of EBMs. In this paper, we propose to learn EBM with a flow-based model (or in general latent variable model) serving as a backbone, so that the EBM is a correction or an exponential tilting of the flow-based model. We show that the model has a particularly simple form in the space of the latent variables of the generative model, and MCMC sampling of the EBM in the latent space mixes well and traverses modes in the data space. This enables proper sampling and learning of EBMs.

Published as a conference paper at ICLR 2022
MCMC SHOULD MIX: LEARNING ENERGY-BASED MODEL WITH NEURAL TRANSPORT LATENT SPACE MCMC
Erik Nijkamp 1.3++
1.31*
Ruiqi Gao
Pavel Sountsoy² siege@google com
erik. ni jkampsaaeesoorree Kr.k com
ruiqigegoogle..o com
Srinivas Vasudevan2 ervasude@ppogle.com
Bo Pangl:
Song-Chun Zhu
b. w.pangssleeeoorre.comm
sczhuêstat. ucla.
Ying Nian Wu' ywu@stat. ucla. edu
*Department of Statisties, UCLA "Google Research Salesforce Research 'Beijing Institute for General Artificial Intelligence (BIGAI)
ABSTRACT
Learning energy-based model (EBM) requires MCMC sampling of the learned model as an inner loop of the learning algorithm. However, MCMC sampling of EBMs in high-dimensional data space generally not mixing, because the energy function, which is usually parametrized by deep network, is highly multi-modal in the data space. This is a serious handicap for both theory and practice of EBMs. In this paper, propose to learn an EBM with flow-based model (or in general a latent variable model) serving as a backbone, so that the EBM is a correction exponential tilting of the flow-based model. We show that the model has a particularly simple form in the space the latent variables the backbone model, and MCMC sampling of the EBM in the latent space mixes well and traverses modes the data space. This enables proper sampling and learning of EBMs.
INTRODUCTION
The energy- based model (EBM) LeCunn al. 2006 Ngiam al. 2011 Kim & Bengio
2016
Zhao et al. 2016 Xie et al. 2016 Gao et al. [2015 8 Kumar et al. al.| 2019b Nijkamp et al. 2019 Du &
Mordatch| 2019: Finn et al. 2016; Atchadé al. 2017] De Bortoli al. 2021 Song & Ou 2018)
defines an unnormalized probability density function on the observed data such as images via an energy function, that the density is proportional to the exponential of the negative energy. Taking advantage of the approximation capacity of modern deep networks such as convolutional networks (ConvNet) (LeCun al. 1998, Krizhevsky al., 2012), recent papers (Xie et al. 2016 Gao al.) 2018 Kumar et al. 2019b] Nijkamp et al. 2019 Du & Mordatch, (2019) parametrize the energy function by a CenvNet. The CoMyNet-EBM is highly expressive and the learned EBM can produce realistic synthesized examples.
The EBM can be learned by maximum likelihood estimation (MLE), which follows an "analysis by synthesis" scheme. In the synthesis step, synthesized examples are generated by sampling from current model. the analysis step, the model parameters are updated based on the statistical difference between the synthesized examples and the observed examples. The synthesis step usu- ally requires Markov chain Monte Carlo (MCMC) sampling, and gradient-based sampling such as Langevin dynamics (Langevin) 1908) Hamiltonian Monte Carlo (HMC) (Neal 2011) can be veniently implemented on the current deep learning platforms where gradients can be efficiently and automatically computed by back pack-propagation.
Equual contribution. Majority of research was conducted Google.

---

Published as a conference paper at ICLR 2022
However, gradient-based based MCMC sampling in the data space generally does not mix, which is fundamental issue from a statistical perspective. The data distribution is typically highly multi- modal. To approximate such distribution, the density or energy function of the ConVNet-EBM needs to be highly multi-modal as well. When sampling from such a multi-modal density in the data space, gradient-based MCMC tends to get trapped in local modes with little chance traverse the modes freely, rendering the MCMC non-mixing. Without being able to generate fair examples from the model, the estimated gradient of the maximum likelihood learning can be highly biased, and the learned model parameters can be far from the unbiased estimator given by MLE. Even if we can learn the model by other means without resorting to MCMC sampling, e.g., by noise contrastive estimation (NCE) (Gutmann & Hyvärinen 2010; Gao et al.) 2019, Wang & Ou, 2018) or by amortized sampling (Kim & Bengio [016]] Song & Oul 2018 Grathwohl et al. 2020), , is still necessary to be able to draw fair examples from the learned model for the purpose of model checking or downstream applications based on the learned model.
Accepting the fact that MCMC sampling is not mixing, contrastive divergence (Tieleman 2008 initializes finite step MCMC from the observed examples, so that the learned model is admittedly biased from the MLE. Du et al. (2020) improves contrastive divergence by initializing MCMC from augmented samples. Recently, Nijkamp et al.] (2019) proposes to initialize short-run MCMC from fixed noise distribution, and shows that even though the learned EBM is biased, the short-run MCMC can be considered a valid model that can generate realistic examples. This partially explains why EBM learning algorithm can synthesize high quality examples even though the MCMC does not mix. However, the problem of non-mixing MCMC remains unsolved. Without proper MCMC sampling, the theory and practice of learning EBMs is on very shaky ground. The goal of this paper is to address the problem of MCMC mixing, which is important for proper learning of EBMs. The subpar quality of synthesis of our approach is a concern, which we believe may be addressed with recent flow architectures (Durkan et al.) 2019) and jointly updating the flow model in future work. We believe that fitting EBMs properly with mixing MCMC is crucial to downstream tasks detection and feature
that go beyond generating high-quality samples, such learning. We will investigate our model on those tasks in future work.
space
Space
plx)
9/4l
Figure 1: Demonstration of mixing MCMC with neural transport learned from mixture of eight 2D Gaussians. The Markov chains pulled back into data space freely traverse the modes of the mixture of Gaussians. Left: observed examples (black) and trajectories (blue) of Markov chains (red) in data space x and latent space Right: density estimations with exponentially tilted model Po and underlying flow qa-
We propose to Iearn an EBM with a flow-based model (or in general a latent variable model) backbone model (or base model, or core model), so that the EBM is in the form of a correction,
or exponential tilting, of the bowed model. based models have gained popularity in generative modeling (Dinh et al. 2014: 2016: Kingma & Dhariwal 2018 Grathwohl et al.] 2018
Behrmann et al.] 2018; Kumar al. 2019a| Tran et al. (2019) and variational inference (Kingma & Welling, 2013, Rezende & Mohamed 2015) Kingma et al. 2016, Kingma & Welling 2014] Khemakhem et 2019). Similar the generator model (Kingma & Welling 2013 Goodfellow et al., 2014). a flow-based model is based on a mapping from the latent space to the data space. However, unlike the generator model, the mapping the flow-based model deterministic and one-one, with closed-form inversion and Jacobian that can be efficiently computed. This leads to an explicit normalized density via change of variable. However, to ensure tractable inversion and Jacobian, the mapping in the flow-based model has to be a composition of sequence of simple transformations of highly constrained forms. In order to approximate a complex distribution, it is necessary to compose a large number of such transformations, In our work, we propose to learn the EBM by correcting a relatively simple flowe model with a relatively simple energy function parametrized by free-form ConvNet.
2

---

Published as a conference paper at ICLR 2022
We show that the resulting EBM has particularly simple form in the space of the latent variables. MCMC sampling of the EBM in the latent space, which is a simple special case of neural transport MCMC (Hoffman et al. 2019), mixes well and is able to traverse modes in the data space. This enables proper sampling and learning of EBMs. Our experiments demonstrate the efficacy of learn- ing EBM with flow-based backbone, and the neural transport sampling of the learned EBM greatly mitigates the non-mixing problem of MCMC.
2 RELATED WORK AND CONTRIBUTIONS
The following are research themes in generative modeling and MCMC sampling that are closely related to our work.
Neural transport MCMC. Our work inspired by neural transport sampling (Hoffman et al. 2019). For an unnormalized target distribution, the neural transport sampler trains flow-based model as a variational approximation to the target distribution, and then samples the target distri- bution in the space of latent variables of the flow-based model via change of variable. In the latent space, the target distribution is close to the prior distribution of the latent variables of the flow-based model. which is usually unimodal Gaussian white noise distribution. Consequently the target dis- tribution in the latent space is close to be unimodal and is much more conducive to mixing and fast convergence of MCMC than sampling in the original space (Mangoubi & Smith 2017
Our work is a simplified special case of this idea, where we learn the EBM as a correction of a pre-trained flow-based model, so that we do not need to train separate flow-based approximation to the EBM. The energy function, which is a correction of the flow-based model, does not need to reproduce the content of the flow-based model, and thus be kept relatively simple. Moreover, in the latent space, the resulting EBM takes on a very simple form where the inversion and Jacobian in the flow-based model disappear. This may allow for using free-form flow-based models where inversion and Jacobian do not need to be in closed form (Grathwohl et al. 2018 Behrmann et al., 2018], or more general latent variable models.
Energy-based corrections. Our model is based on an energy energy-based correction or an exponen- tial tilting of a more tractable model. This idea has been explored in noise contrastive estimation (NCE) (Gutmann& & Hyvärinen| 2010;; Gao et al. 2019) and introspective neural networks (INN) (Tu. 2007 Jin et al. 2017] Lazarow et al.| 2017). where the correction obtained by discriminative learn- ing. Earlier works include Rosenfeld et al. (2001); Wang & Ou (2018). Recently Xiao et al. (2020) recruits an EBM to correct a variational autoencoder with MCMC-based learning methods. Cor- recting or refining simpler and more tractable backbone model can be much easier than learning an EBM from scratch, because the EBM does not need to reproduce the knowledge learned by the backbone model. It also allows easier sampling of EBMs.
Amortized sampling. Non-mixing MCMC sampling of an EBM is a clear call for latent variables to represent multiple modes of the original model distribution via explicit top-down mapping, so that the distribution of the latent variables is less multi-mode
Earlier works in this direction in- clude Bengio et al.] (2013): Kim & Bengio (0116); Dai et al.] 20177: Song Oul (2018) Brock
iet al. (2018): ; Xie et al. (201 8): Han et al. (2019) Kumar et al. (2019b); Grathwohl et al. 2020)) In this paper, we choose to use flow-based model for its simplicity, because the distribution in the data space can be translated into the distribution in the latent space by simple change of variable, without requiring integrating out extra dimensions as in the generator model.
Proper learning of E.B.s.. Wang & Ou (017) studies the proper learning of EBMs in the modality of languages and recruits Gibbs sampling from the discrete distributions. comparison, our work concerns images in continuous space for which we sample by gradient-based MCMC. Moreover, our work emphasizes the empirical evaluation of the mixing behavior of Markov chains.
Contributions.
This paper tackles the problem of non-mixing MCMC for sampling from an EBM. We propose to learn an EBM with a How-based backbone model. The resulting EBM in the latent space is of simple form that is much more friendly to MCMC mixing. Our work provides strong empirical evidence regarding the feasibility of mixing MCMC sampling in EBMs parametrized by modern ConvNet for the modality of images.
3

---

Published as a conference paper at ICLR 2022
3 MODEL AND LEARNING
3.1 FLOW-BASED MODEL
Let be the input example, such as an image. flow-based model is of the form
qo()), = (2).
where is the latent vector of the same dimensionality æ; and do is known prior distribution such as Gaussian white noise distribution. is a composition of a sequence of invertible trans- formations whose inversions and log-determinants of the Jacobians can be obtained in closed form. As result, these transformations are of highly constrained forms. denotes the model parameters. Let qa (z) be the probability density at r under the transformation r (a()), then according the change of variable,
go(2)dz = (x)dd,
(2)
where dz and dr are understood as the volumes the infinitesimal local neighborhoods around and respectively under the mapping 9m Then for given x. Ia (z:). and
aa (x) (o(=))))))))) no(=)dz/dzz (9 (z)) deg (z)/Ox)l.
(3)
where the ratio between the volumes dz/ /x is the absolute value of the determinant of the Jacobian.
Suppose we observe training examples (I;, i = I,.....) n)) Pdita where Plaza is the data distri- bution, which is typically highly multi-modal. We can learn by MLE. For large n. the MLE of approximately minimizes the Kullback-Leibler divergence Dx (Pdstallqa)) Ça strives to cover most of the modes in Pdata. and the learned ςa tends to more dispersed than Pdata- In order for ga to approximate Pdata closely, it is usually necessary for to be composition of a large number of transformations of highly constrained forms with closed-form inversions and Jacobians. The learned mapping Ja (z) transports the unimodal Gaussian white noise distribution to highly multi-modal distribution Ga in the data space as an approximation to the data distribution Prlata
3.2 EVERGY-BASER MODEL
An energy-basee based model (EBM) is defined as follows:
1 Z(Ø) exp(fo()))<())
po(x)
(4)
where q(2:) is reference measure, such uniform distribution Gaussian white noise distribu-
tion as ineXie et al. (2016). fo is defined by a bottom-up ConvNet whose parameters are denoted by 8.
The normalizing constant the partition function Z(0) J exp(fo(r))((<))d>= = F. (exp((o (x))]
typically analytically intractable.
Suppose observe training examples I; Pdata for 1, .... n. For large the sample average over (xi) approximates the expectation with respect to Pdata- For notational convenience, we treat the sample average and the expectation as the same.
The log-likelihood is
7 log (o;:) - Epani (o.)).
L(O)=
(5)
n
The derivative of the log- likelihoodd is
L'(0) Epaara [e)f(x))] - Epa [off(()) 1
1
vofo(z.)
oz))
(6)
n
where 27 ~ (z) for = 1,..., n are synthesized examples sampled from the current model po (x)).
The above equation leads to the "analysis by synthesis" learning algorithm. At iteration t. let 0 be the current model parameters. We generate x ~ Po. (x) for = 1,.., n. Then we update 0.1 02 q [.'(()))) (0:)), where di is the learning rate.
4

---

Published as a conference paper at ICLR 2022
gradient-based MCMC sampling such as Langevin dynamics (Langevin, 1908) or Hamiltonian Monte Carlo (HMC) (Neal 2011). where fo()(z) can be automatically computed. Since Pdata is in general highly molti-modal, the learned Po or So tends to be multi-modal as well. As a result, gradient-basee MCMC tends to get trapped in the local modes of fo with little chance of mixing between the modes.
To generate synthesized examples from ps,
3.3 ENERGY- BASED MODEL WITH FLOW-BASED BACKBONE
Instead of using uniform or Gaussian white noise distribution for the reference distribution g(æ) in the EBM in 4 we can use relatively simple flow-based model qa as the reference model. qa can be pre-trained by MLE, and serves as the backbone of the model, so that the model is of the following form
po(x) Z(0)
(x
which is almost the same in except that the reference distribution (ar) is a pre-trained flow- based model gax(x) (z)) The resulting model po (I) is a correction or refinement of gas or an exponential tilting of qa (x), and fa()) free-form ConvNet parametrize the correction. The overall negative energy is fa()) + log qo (I).
In the latent space of z, let p(z) be the distribution of under (o()), then 1 (xp(((()))
p(=)dz = po(r)ddr Z(0)
(8)
Recall equation (2), (a(ddr qo(z)dz, we have
1
p(z) = Z(0) exp(fe(9a())))906>
(9)
p(z) is an exponential tilting of the prior noise distribution go(=). It is very simple form that does involve the Jacobian or inversion of ga.
3.4 LEARNING BY HAMILTONIAN NEURAL TRANSPORT SAMPLING
While 2a (xx) is multi-modal (=()) unimodal. Since pa (x) is a correction of ga. p(z) is a correction po(z), and can be much less multi-modal than pa (a(z) in the data space. After sampling = from p(z), we can generate (z)
Instead of sampling Pa(x), we can sample p(=) ) in equation
The above MCMC sampling scheme is a special case of neutral transport MCMC proposed by Hoffman et al. (2019) for sampling from EBM or the posterior distribution of a generative model. The basic idea is to train a flow-based model as a variational approximation to the target EBM, and sample the FBM in the latent space of the flow-based model. In case, since po is correction of fa. we can simply use la directly as the approximate flow-based model in the neural transport sampler. The extra benefit that the distribution p(z) is of an even simpler form than pe (z)), because p(z) does not involve the inversion and Jacobian of Ja. As a result, we may use flow-based backbone model of more free form such one based on residual network (Behrmann al., 2018). and we will further explore this advantage in the future work. We use HMC (Neal 2201] to sample from p(z), and push the samples forward to the data space through Ja- We can then learn / by MLE according to equation (6. Algorithm describes the details.
Algorithm 1:
Learning the correction fa of flow da with Neural Transport (NT-EBM).
input : Learning iterations T, learning rate 7), batch size m, pre-trained parameters a, initial parameters Og. initial latent variables {=i,o _ (o(z), observed examples (x;) ) number MCMC steps K in each learning iteration. output: Parameters }
for t = 0: :
do
1. Update { by HMC with target distribution p(2) in equation 9 for K steps.
Push the z-space samples forward through Ja obtain synthesized examples {,7 }m
3. Draw observed training examples km
Update 8 according to equation 6).
5

---

Published as a conference paper at ICLR 2022
3.5
LEARNING BY NOISE CONTRASTIVE ESTIMATION
We may also learn the correction (a) discriminatively. as in noise contrastive estimation (NCE) (Gutmann & Hyvärinen 2010) introspective neural networks (INN) (Tu 2007 Jin al.) 2017 Lazarow et al. 2017). Let
I...., n. be the training examples, which are treated as pos- itive examples, and let 24 I,.... n be the examples generated from ga (z), which are treated as negative examples. For each batch, let p be the proportion of positive examples, and p be the proportion of negative examples. We have
log
P(+
log
P 1 --p_
log 2(0) fa()) b fo(
(10)
log (0) is treated separate bias parameter. Then we can estimate b and 0 by fitting a logistic regression on the positive and negative examples. Note, that NCE is the discriminator side of GAN. Similar to GAN, we can also improve the flow-based model based
where 6 = log T-p
the value function of GAN. This may further improve the NCE results.
3.6 GENERAL LATENT VARIABLE MODEL
The above latent space exponential tilting formulation applies to general pre-trained latent variable model (o(=), x Ia (z), as long as the dimensionality of is greater than or equal to that of r. the case where of higher dimensionality than æ, we only need to re-define æ to be (ar, =g) where is sub-vector of z, that the mapping between and (æ, zo) is invertible. In the case of generator network x = g(z) + E, we can re-define to be (z, e). See Appendix A.2 for details.
EXPERIMENTS
In the subsequent empirical evaluations, we shall address the following questions:
(1) Is the mixing of HMC with neural transport, both qualitatively and quantitatively, apparent?
(2) In the latent space, does smooth interpolation remain feasible?
(3) Does the exponential tilting with correction term fo(z) improve the quality of synthesis?
(4) terms of ablation, what is the effect of amount of parameters a for flow-based g.,
(5) Is discriminative learning in the form of NCE an efficient alternative learning method?
The primary concern of our work is the mixing of MCMC, which addressed (1) and (2). We refer to Appendix A.3 and A.4 for details on training settings and model architectures.
4.1 MIXING
In the following, we will recruit diagnostics to quantitatively and qualitatively address the question of mixing MCMC. We will first evaluate the famous Gelman- Rubin statistic for Markov chains running in the latent space and contrast those against chains in the data space. Then, we will evaluate auto-correlation as weaker measure of mixing. Finally, provide visual inspection of Markov chains in our model and compare those with a biased model known not to be amenable to mixing.
Gelman-Rubin. The Gelman-Rubin statistic Gelman et al. 1992. Brooks & Gelman [998) mea- sures the convergence of Markov chains to the target distribution. is based on the notion that if multiple chains have converged, by definition, they should appear "similar" to one another, else, one or more chains have failed to converge. Specifically, the diagnostic recruits an analysis of vari- ance access the difference between the netween-chain and within-chaii variances. We refer to the Appendix A.6 for details. Figure 2 a-b) depicts the histograms of R for
64 chains over 2, 000 steps with burn-in time of 400 steps, Icarned from SVHN dataset. The mean I value is 1.13, which we treat as approximative convergence to the target distribution (Brooks & Gelman 1998). We contrast this result with over-damped Langevin dynamics in the latent space and HMC in the data space, both with unfavorable diagnostics of mixing.
Auto-Correlation. MCMC sampling leads to autocorrelated samples due the inherent Markovian dependence structure. The At (sample) auto-correlation is the correlation between samples At steps apart in time. Figure c-d) shows auto-correlation against increasing time lag At, learned

---

Published as a conference paper at ICLR 2022
(a) Gelman-Rubin latent space.
(b) Gelman-Ruie in
(c) Auto-correlation in latent space.
(d) Auto-correlation in data space (HMC).
data space (HMC).
Figure 2:
Diagnostics for the mixing of MCMC chains with n 2, 000 steps of Langevin (blue) and HMC (orange), learned from SVHN dataset. (a-b) Histograms of Gelman-Ruhie Ruban statistic of multiple
long-run Markov chains. R < 1.2 indicates approximative convergence, (c-d) Auto-correlation of single long-run Markov chain over time lag At with mean depicted as line and min/max as bands.
from SVHN dataset. While the auto-correlation of HMC chains with neural transport vanishes within åt = 200 steps, the over-damped Langevin sampler requires At > 1, 000 steps, and the auto-correlation of HMC chains in the data space remains high. The single long-run Markov chain behavior consistent with the Gelman-Rubil statistic assessing multiple chains.
Visual Inspection. Assume Markov chain is run for large numbers of steps with Hamiltonian neural transport. Then, the Markov chains are pushed forward into data space with visualized long trajectories in Figures
4 and 9 where Py is learned the SVHN (32 32 x 3) (Nezzer et al.
2011) and CelebA (64 x 64 x 3) (Liu et al. 2015) datasets, respectively. Figure 3 contrasts the Markov chains that sample the EBM learned with short-run MCMC (Nijkamp al. 2019). which does not mix, against our method in which the pulled back Markov chains mix freely. We observe
the Markov chains are freely traversing between local modes, which we consider weak indication of mixing MCMC.
Figure 3: Long-run Markov chains for learned models without and with mixing. Top: Chains trapped in an over-saturated local mode. Model learned by short-run MCMC (Nijkamp et al. 2019) without mixing. Bottom: Chain is freely traversing local modes. Model learned by Hamiltonian neural transport with mixing.
010
2
0
m 0
Figure 4: A single long-run Markov Chain with n = 2, 000 steps depicted in 5 steps intervals sampled by Hamiltonian neural transport SVHN (32 x 32 3).
4.2 INTERPOLATION
Interpolation allows us to appraise the smoothness the latent space. particular, two samples ø1 and 22 are drawn from the prior distribution ςo- We may spherically interpolate between them in z-space and then push forward into data space to assess qa.
To evaluate the tilted model pe (z), we magnetized form of the over-damped Langevin equation for which we alter the negative energy (zz) (9 +2))+log go to (z) U(z) > with magnetization constant Hill et al. 2019). Note, - $|12||2 = 2/| 12, thus, the magnetization term introduces vector field pointing with uniform strength towards 2* The resulting Langevin

---

Published as a conference paper at ICLR 2022
equation is dz(t)= = (x)()t))
+ + -(t)=2" dt + (t) with Wiener process (t). To find --000==111
low energy path from 21 towards 22.
z2, 21 and perform 1, 000 steps of the
discretized, magnetized Langevin equation with small 7-
3
3 3
4
4
9
8 8
8 9
2 2 2 2
Figure 5: Low energy path between 21 and 22 by magnetized Langevin dynamics over n = 1,000 steps MNIST (28 x 28 x 1). Top: Trajectory in data-space. Bottom: Energy profile over time.
Figure 5 depicts the low-energy path in data-space and energy (=) over time. The qualitatively smooth interpolation and narrow energy spectrum indicate that Langevin dynamics in latent space (with small magnctization) is able to traverse two arbitrary local modes, thus, substantiating claim that the model is amenable to mixing.
4.3 SYNTHESIS
While the emphasis of our work is the mixing MCMC. do evaluate the quality of synthesis on four datasets: MNIST (28 x 28 x 1) (LeCun et al. 2010), SVHN (32 x 32 x 3) (Netzer et al.) 2011), CelebA (64 64 3) (Liu al. 2015), and, CIFAR-10 (32 x 32 3) (Krizhevsky al.).
IS
6
(a) Samples from flow Ga (b) Samples from Po by (c) Samples from flow (d) Samples from Po by hy ancestral sampling. neural transport.
by ancestral sampling. neural transport.
Figure 6: Comparison of generated samples by ancestral sampling from flow da and neural transport sampling from learned by NT-EBM. Left: SVHN (32 x 32 x 3). Right: CelebA (64 x 64 x 3).
The qualitative results are depicted in Figure 6 which contrast generated samples from Glow ga against Markov chains by Hamiltonian neural transport from pa. Table compares the Fréchet In- ception Distance (FID) (Heusel et al.] 2017) with Inception v3 (Szegedy et al., (016) on 50, 000 generated examples. Both, qualitatively and quantitatively speaking, provement in quality of synthesis with exponentially tilting of the reference distribution by the correction fe. However, the overall quality of synthesis is relatively low in comparison baselines (Miyato et al., 20188) Song & Ou 2018) with FID 29.3 and 20.9 on CIFAR-1O, respectively, which do not involve inference of latent variables. We hope advances in flow architectures and jointly learning the flow model may address these issues in future work.
observe significant im-
Table 1: FID scores for generated examples in comparison to VAE (Kingma & Welling 2013 ABP (Han et al. 2017), and Glow (Kingma Dhariwal 2018).
Method
MNIST SVHN
CelebA
CIFAR-10
VAE ABP
32.86
49.72
48.27
106.37
39.12
48.65
51.92
114.13
Glow (MLE)
66.04
94,23
59.35
90.08
NCE-EBM (Ours)
36.52
79.84
51.73
NT-EBM (Ours)
21.32
48.01
46.38
78.12
In Table 2 we show the FID scores for samples obtained from every 1,000 steps of a single long- chain for model learned on SVHN. That is, the first FID score is calculated over the first
8

---

Published as a conference paper at ICLR 2022
Table 2: FID scores for samples collected from a single long-run chain on SVHN
(32 x 32 x 3).
# Samples
1,000
2,000
3,000
4,000
5,000
6,000
7,000
8,000
9,000
10,000
FID
97.51
76.74
67.67
60.41
60.25
56.45
55.34
53.85
52.32
48.72
1,000 consecutive samples, the second FID over the first 2,000 consecutive samples, and so forth. The FID score converges to our reported FID score with multiple sampling chains and fixed number of sampling steps (Table which indicates that obtain set of fair samples of the model by sampling from just a single very long-run HMC chain.
4.4 ABLATION
We investigate the influence of the number of parameters a flow-based da on the quality of synthesis. In Table 3 we show at what number of parameters small flow-based model with small EBM correction outperforms a large flow-based model. Our method with a "medium" sized backbone significantly outperforms the "largest" Glow.
Table 3: FID scores for generated examples for qa and our method with varying sizes of parameters on SVHN (32 32 x 3). Small: depth = 4, width = 128, Medium: deplh 8, width 128. Large: depth 16, width 256, Largest: depth 32. width 512.
Method
Small
Medium
Large
Largest
Glow (MLF)
110.55
94.34 48.01
89.31
86.18
NT-EBM (Ours)
74.77
43.82
4.5 NOISE CONTRASTIVE ESTIMATION
Noise Contrastive Estimation (NCE) is computationally efficient learning procedure which avoids MCMC sampling by re-casting the learning problem into the form of a logistic regression. Hence, wish to learn the correction fo with our NCE-EBM algorithm according equation (TO), while we still sample from the learned model with neural transport MCMC. Table cobparce the learned models with both learning methods. The long-run Markov chains in the energy-based models learned by NCE are conducive to mixing and remain of high visual quality. Figure TT (see Ap- pendix (.8) depicts samples from 9a (left) and samples from pe learned by our NCE algorithm for which sampling is performed using Hamiltonian neural transport (right) for CelebA. Figure I0 (see Appendix (A.7) depicts a long-run Markov chain pushed forward into data space which enjoys re- alistic synthesis with high diversity. This finding indicates the efficacy of learning flow backboned EBM by NCE, while, after learning the model, we may draw samples by HMC with neural transport.
CONCLUSION
This paper proposes to learn an EBM as correction an exponential tilting of flow-based model, or general top-down latent variable model, so that neural transport MCMC sampling in the latent space of the model can mix well and traverse the modes in the data space. From statistical perspective, the mixing of MCMC is a fundamental problem and is crucial for proper learning of EBMs. In future work, will investigate and identify downstream tasks which significantly benefit. from mixing MCMC and properly learned models by our approach.
ACKNOWLEDGEMENT
The work supported by NSF DMS-2015522 2015577, ONR MURI project N00014- 17001416-1110 1-2007 DARPA XAI project N66001-17-2-4029, and XSEDE grant ASC170063. We thank Matthew Hoffman, Diederik P. Kingma, Alexander A. Alemi, and Will Grathwohl for helpful discussions.

---

Published as a conference paper at ICLR 2022
REFERENCES
Christophe Andrieu and Johannes Thoms. A tutorial on adaptive meme. Statistics and computing, 18(4):343-373, 2008.
Yves F Atchadé, Gersende Fort, and Eric Moulines. On perturbed proximal gradient algorithms. The Journal of Machine Learning Research, 18(1):310-342, 2017.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. arXiv preprint arXiv:/811.0090 2018.
Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. Better mixing via deep represen- tations. In International conference machine learning, pp. 552-560, 2013.
Alexandros Beskos, Natesh Pillai, Gareth Roberts, Jesus-Maria Sanz-Serna, Andrew Stuart, et al. Optimal tuning of the hybrid monte carlo algorithm. Bernoulli, 1965A):150-1154, 2013.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 18Xi:: 1006, 2018.
Stephen P Brooks and Andrew Gelman. General methods for monitoring convergence of iterative simulations. Journal of computational and graphical statistics, 7(4):434 1998.
Ricky TQ Chen, Jens Behrmann, and Jörn-Henrik Jacobsen. Residual flows: Unbiased generative modeling with norm-learned i-resnets.
Zihang Dai, Amjad Almahairi, Philip Bachman, Eduard Hovy, and Aaron Courville. Calibrating energy-based generative adversarial networks. arXiv preprint arXiv:1702.01691, 2017.
Valentin De Bortoli, Agnès Desolneux, Alain Durmus, Bruno Galerne, and Arthur Leclaire. Maxi- mum entropy methods for texture synthesis: theory and practice. SIAM Journal on Mathematics of Data Science, 3(1):52-82, 2021.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint 14X.::4410.8166 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 08803, 2016.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903.0866 2019.
Yilun Du, Shuang Li. Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. arXiv preprint arXiv:2012.01316, 2020.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. Ad- vances Neural Information Processing Systems, 32:7511-7522, 2019.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy- based models. arXiv preprint arXiv:1611.03332 2016.
Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Wu. Learning generative con- vnets via multi-grid modeling and sampling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9155-9164, 2018.
Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based models. arXiv preprint arXiv: 1rXiv:19/2.00550 2019.
Andrew Gelman, Donald B Rubin, al. Inference from iterative simulation using multiple se- quences. Statistical science, 7(4):457-472, 1992.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
10

---

Published as a conference paper at ICLR 2022
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-262222 2680,, 2014.
Will Grathwohl, Ricky TQ Chen, Jesse Betterncour, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.013677 2018.
Will Grathwohl, Jacob Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud. No meme for me: Amortized sampling for fast and stable training of energy-based models. arXiv preprint arXiv:2010.04230, 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Tian Han, Yang Lu, Song-Chun
Chun Zhu, and Ying Nian Wu. Alternating back pacck-ropagation for generator network. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, Califormia, USA., pp. 1976-1984, 2017. URL ptaet:::/aaai.oooo ocs/ toss/indx... php / http:///pppppaaaaaeeeeae
Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Diver- gence triangle for joint training of generator model, energy-based model, and inferential model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 8670-8679, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Mitch Hill, Erik Nijkamp, and Song- Chun Zhu. Building a telescope look into high digensinns image spaces. Quarterly of Applied Mathematics, 77(2):29--21,, 2019.
Matthew Hoffman, Pavel Sountsov, Joshua V Dillon, Ian Langmore, Dustin Tran, and Srinivas Vasudevan. Neutra-lizing bad geometry in hamiltonian monte carlo using neural transport. arXiv preprint arXivv:903. 1903.03744. 2019.
Long Jin, Justin Lazarow, and Zhuowen Tu, Introspective classification with convolutional nets. In Advances in Neural Information Processing Systems, pp. 223-833, 2017.
Ilyes Khemakhem, Diederik Kingma, and Aapo Hyvärinen. Variational autoencoders and nonlin- ear ica: A unifying framework. arXiv preprint arXiv: -xXiv:997700480 2019.
Taesup Kim and Yoshua Bengio. Deep directed generative models with energy- probability estimation. arXiv preprint arXiv: 1606.0330 2016.
Diederik Kingma and Max Welling. Efficient gradient-based inference through transformations between bayes nets and neural nets. In International Conference on Machine Learning, pp. 1782- 1790, 2014.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. /tL: /ttp///axvvvv/va////a/eeeeeaaava org/ abs /bs//412 6980
Diederik P Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-cncoding variational bayes. arXiv preprint
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp. 4743-4751. 2016.
11

---

Published as a conference paper at ICLR 2022
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced search). htt.sttp://......ttttt.. woronttoooooooooooooooooooooooo. sduu/=krz/cifaat catm.
www.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A flow-based generative model for video. arXiv preprint arXiv:1903.01434 2019a.
Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint arXiv: 008508, 2019b.
Paul Langevin. On the theory of Brownian motion. 1908.
Justin Lazarow, Long Jin, and Zhuowen Tu. Introspective neural networks for generative modeling. In Proceedings of the IEEE International Conference Computer Vision, pp. 2774-2783 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2324, 1998.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. Predicting structured data, 100), 2006.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: hMpp://yannleeunn.com comiexdb/mmissst 2, 2010.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.
In Proceedings of International Conference on Computer Vision (ICCV), 2015.
Oren Mangoubi and Aaron Smith. Rapid mixing of hamiltonian monte carlo on strongly log-concave distributions. arXiv preprint arXiv: 1708.07 14, 2017
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint 180.05980,., 2018.
Radford M Neal. MCMC using hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, 2, 2011.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Jiquan Ngiam, Zhenghao Chen, Pang W Koh, and Andrew Y Ng. Learning deep energy models. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 1105- 1112, 2011.
Erik Nijkamp, Song-Chun Chag-Chun Zhu, and Ying Nian Wu. On learning non-convergent short-run meme toward energy-based model. arXiv preprint arXiv: arXiv://904: 1904.09770, 2019.
Prajit Ramachandran, Barret Zoph, and Quoc Le. Searching for activation functions. arXiv preprint arXiv: 170000591 2017.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.057700 2015.
Ronald Rosenfeld, Stanley F Chen, and Xiaojin Zhu. Whole-sentence exponential language models: vehicle for linguistic-statio statissical integration. Computer Speech & Language, 15(1))55-74 2001.
Yunfu Song and Zhijian Ou. Learning neural random fields with inclusive auxiliary generators. 2018.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 2818-2826, 2016.
12

---

Published as a conference paper at ICLR 2022
Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference on Machine learning, pp. 1064- 1071,2008,
Dustin Tran, Keyon Vafa, Kumar Krishna Agrawal, Laurent Dinh, and Ben Poole. Discrete flows: Invertible generative models of discrete data. arXiv preprini arXiv: 1055,4770 2019.
Zhuowen Tu. Learning generative models via discriminative approaches. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8. IEEE, 2007.
Dootika Vats and Christina Knudson. Revisiting the gelman-rubin diagnostic. arXiv preprint 1rXiv:1812.09555 2018.
Bin Wang and Zhijian Ou. Language modeling with neural trans-dimensional random fields. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 294-300. IEEE, 2017.
Bin Wang and Zhijian Ou. Learning neural trans-dimensionnl random field language models with noise-contrastive estimation. In 2018 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6134 6138. IEFE, 2018.
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between varia- tional autoencoders and energy-based models. In International Conference Learning Repre- sentations, 2020.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference Machine Learning, pp. 2635-2644, 2016.
Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of de- scriptor and generator networks. IEEE transactions pattern analysis and machine intelligence, 42(1):27-45, 2018.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprint 160..3369 2016.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 CHINGE OF VARIABLE
Under the invertible transformation x g()), let p(z) be the density of 2, and p(r) ) be the density of r. Let D2 be an infinitesimal neighborhood around 2, and let D₂ be an infinitesimal neighborhood around so that g maps 2 to x, and maps D, to D,. Then
Pr(D.) = Pr(D.).
(11)
Pr(D:)= = p[=)| IID and Pr(D,)) = p(z)) D2l ol Dzl). where |De| and D are the volumes of D. and D respectively. Thus we have
D(=)IDal pxx)
(12)
where we ignore D2)) and Dzl) terms. This the meaning of
p(z)dz = p(z)dz,
(13)
where D2] or dz/dz is the determinant the Jacobian of g-
Equation 13 is convenient starting point for deriving densities under change of variable.
A.2 ANERGY-BASED CORRECTION AND CHANGE OF VARIABLE FOR GENERATOR MODEL
The generator model is the form N(O, Ia), and x - (z) e,,e~ N(O, o²Ip), where D the dimensionality of z, and d < D is the dimensionality of the latent vector. Unlike the flow-based model, the marginal distribution of x involves intractable integral.
We shall study exponential tilting of generator model using the simple equation variable, To that end, we let 8 = (z, <), and let ā = (z, x). Then
13 for change of
i = (2. xx) G. (2) = Ga(z, e) = (z, (z) +c).
(14)
Let (=) be the Gaussian white noise distribution of ī under the generator model. Let qo (5) be the distribution y under the generator model. Consider the change of variable between & and ž. In parallel to equation 113 we have
4o (o(3)dā qa (ā)dd.
(15)
The marginal distribution (x) = [ go (T)dz I (z,zx)) which is intractable.
Suppose we exponentially tilt yo (i) to
pa
exp(fo(T)) (Z).
(16)
7(0)
Again this can be translated into the space of 2 so that under po po(ã). (T),
p( po(T)dī.
(17)
Combining equations 15 (6). and we have
1
((3)= Z(0) (x)())(G.())))((3)
(18)
that is, under the tilted model (z))
((z,) e) = exp(Jo(z,gaa3) + +)))g((z+))
(19)
Z(0)
We may let fe be (2) + e), i.e., it only depends on ga(z) + e, that it is data space energy-based correction the intractable (x). In practice we may also set = 0, although this is not entirely theoretically sound.
14

---

Published as a conference paper at ICLR 2022
A.3 MODEL ARCHITECTURES
For Glow model fo. we follow the setting of Kingma & Dhariwal (018) with n. biliss x = 8,
foww_permullaion = 2, [oow_coullngg
For the EBM model fo. we use the following Conv-Net structure.
We use the following notation. Convolutional operation conv(n) with n output feature maps and bias term. We recruit LipSwishhi a * z)gmoid(()) RRamachandran et 2017 as activation function. We set ny E {32, 64..
S Swish(x)) /1111 (Chen et al.) nonlinearity where Swish(æ) =
Specifically, we set use the following hyper-parameter
1. MNIST: For Glow, n.levels 3, depth 8, width 128. For EBM, π, 32.
2. SVHN: For Glow, neveelss 3, depth width 128. For EBM, ny 32.
3. CelebA: For Glow, n.eevels = 3. depth = 16, width - 256. For EBM, Nt = 32.
4.
CIFAR-10: For Glow, levels depth 16, width 512. For EBM, ni 32.
Energy- based Model (32 x 32 x 3)
Layers
In-Out Size
Stride
Input
32 x 32 3
3 x 3 conv(ny), LipSwish
32 x 32 f2
X conv(2 ny), LipSwish 16 x 16 x (2 * ny)
x conv(4 ny), LipSuiish
8 8 x n4)
x conv(4 ny), LipSwish
4 4 (4 ny)
4 x 4 conv(1)
1x1x1
Table 4: Network structures for EBM with data-space (32 32 3).
Energy-based Model (64 x 64 x 3)
Layers
In-Out Size
Stride
Input
64 x 64 x 3
x 3 conv(n,), LipSwish
64 x 64 X nz
X conv(2 ny), LipSwish 32 32 x (2 $ ny)
X conv(4 nj), LipSurish 16 16 (4 ny)
4 x 4 conv(8 n2), LipSurish
8 x 8 (8 ny)
2
4 x 4 conv(8 nf), LipSwish 4 x 4 conv(1)
4 4 (8 my)
2
1x1x1
Table 5: Network structures for EBM with data-space (64 x 64 x 3).
A.4 TRAINING
Data. The training image dataset are resized and scaled to [-1, 1]. We use 60,000, 70,000, 30,000, 50,000 observed examples for MNIST (28 x 28 1), SVHN (32 x 32 3), CelebA (64 x 64 x 3), and CIFAR-10 (32 x 32 x 3), respectively.
Glow. The parameters of the flow model qa are pre- trained following the configuration and ref- erence implementation provided in (Kingma Dhariwal 2018). Note, the models in Table [ and 3 have been trained based on the official reference implementations, To ensure fair comparison
15

---

Published as a conference paper at ICLR 2022
of learning Glow by MLE and our methods, disable learning of the spatial prior and use ad- ditive coupling layers for Glow. We refer to Appendix A.3 for specification of the Glow model configuration.
EBM. The network parameters are initialized with Xavier Gloort & Bengio 2010 and optimized using Adam (Kingma & Ba 2015) with (31, (2) (0.99, 0.999). For NT-EBM, the learning rates used are 5c-5. 5c-5, Ic 5, 5c-5 for MNIST, SVHN, CelebA, CIFAR-10, 10, respectively and a batch-size of 64 examples. For NCE-EBM, the learning rates used are 1e.5, for MNIST, SVHN, and CelebA, respectively, and batch-size of 128 examples. For NT-EBM, in training the maximum number of parameter 0 updates was 40, 000. For NCE-EBM, in training the maximum number of parameter 9 updates was S0, 000.
HMC. We run Hamiltonian Monte Carlo (HMC) with persistent chains (Tieleman 2008 initialized from and 20 steps of MCMC and 3 leapfrog integrator steps per update of parameters of 8. The initial discretization step-size 0.15 with simple adaptive policy multiplicatively increases or decreasing the step-size of the inner kernel based on the value of the Metropolis-Hasting acceptance rate (Andrieu Thoms, 2008). The target acceptance-rate is set to 0.651 (Beskos et al. 2013).
Figure 7epiccts the MH acceptance-rate and adaptive step-size over time.
Www.y www.mma
Figure 7: Metropolis-Hastingg acceptance rate (top) and adaptive step-size (bottom) over time.
FID. The Fréchet Inception Distance (FID) (Heusel et al.; 2017) with Inception v3 classi- fier (Szegedy al.] 2016) was computed on 50, 000 generated examples with 50, 000 observed examples as reference.
A.5 SYNTHESIS
Figure depicts samples from pre-trained flow ga and samples from pe learned by neural transport MCMC for the dataset CIFAR-I0 (32 x 32 x 3).
(a) Samples drawn from flow Ga by ancestral sampling.
(b) Samples drawn from by Hamiltonian neural transport.
Figure Generated samples from model learned by NT-EBM on CIFAR- 10 (32 x 32 3).
16

---

Published as a conference paper at ICLR 2022
A.6 GELMAN- RUBIN STATISTIC
The Gelman-Rubin statistic (Gelman et al. 1992 Brooks & Gelman 1998) measures the conver- gence of Markov chains the target distribution. It is based on the notion that if multiple chains have converged, by definition, they should appear "similar" to one another, else, one or more chains have failed to converge. Specifically, the diagnostic recruits an analysis of variance to access the difference between the between-chain and within-chain variances.
Let p denote the target distribution with mean μ E R. and variance o² DC. Gelman al. 1992) designs two estimators of o² and compares the square root of their ratio to 1. Let X
{Xi.. 1,....,............. m,j I,...,n] n) denote m Markov chains of length n. Let s m n- (X, - x be the within-chain variance. The quantity s underestimates
a² due positive correlation in the Markov chain. Let 82 n=1 o2 be a mixture of within- chain variance o and between-chain variance of - m-1 overestimate o², if an over-dispersed initial distribution for the Markov chains was used (Gelman al. 1992). That is, s underestimates while 8² overestimates a². Both estimators are consistent for a as - сo VVats& & Knudson 2018). In light of this, the Gelman- statistic monitors
(XX,-XX) The quantity 8 will
convergence the ratio Rt = = S.. Hence, It measures the degree to which variance (of the means) between chains exceeds what one would expect if the chains were identically distributed. If all chains converge to p. then as n + oc, R R- 1. Before that, R > 1. The heuristics R < 1.2 indicates approximate convergence (Brooks & Gelman 1998).
A.7
VERY LONG MARKOV CIAIN
In Figure the model pa is learned with NT-EBM on the CelebA (64 x 64 x 3) (Liu et al. 2015)
dataset. In Figure 10 the model po is learned with NCE-EBM the SVHN (3232 3) Netzer et al. 2011 dataset.
Figure 9: A single long-run Markov Chain with 000 steps depicted in 5 steps intervals sampled by Hamiltonian neural transport on CelebA (64 x 64 x 3).
Figure 10: A single long-run Markov Chain with n = 2, 000 steps depicted in 5 steps intervals sampled by HMC neural transport for model learned by NCE on SVHN (32 32 3).
17

---

Published as a conference paper at ICLR 2022
A.8 NOISE CONTRASTIVE ESTIMATION
Figure [1] depicts samples from (left) and samples from pe learned by our NCE algorithm for which sampling is performed using Hamiltonian neural transport (right) for CelebA.
(a) Samples drawn from Go by ancestral sampling.
(b) Samples drawn from po by neural transport.
Figure Generated samples from a model learned by NCE-EBM on CelebA (64 64 3).
18


Reviews:
[{"review_id": "lLvqLyUI5bC", "reviewer": "Reviewer_Kukr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My first suggestion is that the authors explain what the consequences of non-mixing Markov chains could be. \n\nI was a bit worried when you said the quality of synthesis was a second issue to you. Do the authors feel that mixing is more important than good synthesis? I understand that they are related, but mixing is nice only conditioned on good synthesis.\n\nYou write on page 4, \"for notation convenience\". I don't see the convenience, but I see it provoking many analysts.\n\nYou should state how alpha is pretrained. Additional to this, can alpha be jointly learned with theta?\n\nCan the authors please expand what I should conclude from Figure 2(b)?\n\nOn the same line as my second point: it seems that the authors do not think that Table 1 is the main result. If this is true, why not?\n\nWhat is the cause of the missing experiments in Tables 1 and 2?\n\nI would like the authors to tinker more about the limitations of the work. One thing I would like to know more about is the relationship of your \"backbone\" and what Bayesians would call a prior. Is there *some* truth in the following: are you pre-training a prior?\n\nMinor things: \n\nWhen citing multiple papers at once, consider putting them at the end of the sentence. Else, it is hard to read the sentence.", "summary_of_the_paper": "The authors propose to learn energy-based models with a flow-based model as \"backbone\". This is to utilize Neural Transport and ultimately make the Markov Chains mix well in data space. This is an important problem.", "summary_of_the_review": "I like the paper, and think it is well-written in general. The main contribution is a novel combination of existing methodologies. I am not completely familiar with the literature, but if this combination is novel I believe this to be a solid contribution. However, my evaluation is dependent on adequate responses from the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_Gncq44bXf0", "reviewer": "Reviewer_Wtka", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**General Comment**\n\nI very much enjoyed reading this paper. The motivation is stated clearly; the problem of having a biased gradient estimation of MLE in training EBMs is certainly a reasonable problem. The proposed approach certainly makes sense and I think is a very reasonable approach to this problem. While the work is very close to [1] which may raise some concerns regarding novelty, I think it can also be seen as a strength given  that it is showing a very good application of method for posterior inference in training EBMs. Furthermore, the addition of energy-based correction is I think, while trivial, is a very neat trick that enables one to avoid computing the determinant jacobian when running HMC unlike NeuTra. The experiments I think could use some improvements, but overall demonstrate the case that the chains in the proposed approach mix better than SGLD chains. The proposed model does have a small disadvantage in the sense that it requires to train a flow model in advance. However, as the authors show in Section 4.4, even using a small flow results in a significant improvement. Overall, I am happy to recommend acceptance. \n\n**Related Work**\n\nI think this paper is missing some of the related work, namely [2,3]. I understand this work differs vastly from [2,3] but given that both of these papers also proposed a new method to address the problem of using MCMC sampling to compute the gradients of $\\log p(x)$, I think this work should compare against them or at least discuss them within context. \n\n**Clarity**\n\nThis paper is very well-written. There were very few ambiguities. Good job!\n\n**Experiments**\n\nThe experiments are well-designed overall. The one I think I would have liked to see (in addition to comparison against [2,3]) is a more comprehensive ablation study of HMC vs SGLD. Figure (2) already demonstrates this but if I understand correctly, Figure (2) is run on a toy dataset. Would we have seen something similar in the case of vision datasets? In other words, how much of the gain in performance is due to using HMC instead of SGLD and how much of it is due to sampling in $z$-space instead of $x$-space? Also would have been interesting to compare against standard Contrastive Divergence where instead of initializing from noise, we initialize from a flow model for a fair comparison.  \n\n**Minor Comments**\n\n-In the Introduction, the authors state: “Recently, Nijkamp et al. (2019) proposes to initialize short-run MCMC from a fixed noise distribution, and shows that even though the learned EBM is biased, the short-run MCMC can be considered a valid model that can generate realistic examples. This partially explains why EBM learning algorithm can synthesize high quality examples even though the MCMC does not mix.” Could you elaborate on this statement? Why does this explain that CD is able to synthesize high quality samples? Because of initialization from noise instead of data? Could you elaborate abit more on the reasoning here? Thanks\n\n\n**References**\n\n[1] Hoffman, Matthew, et al. \"Neutra-lizing bad geometry in hamiltonian monte carlo using neural transport.\" arXiv preprint arXiv:1903.03704 (2019).\n\n[2] Grathwohl, Will, et al. \"No MCMC for me: Amortized sampling for fast and stable training of energy-based models.\" arXiv preprint arXiv:2010.04230 (2020).\n\n[3] Du, Yilun, et al. \"Improved contrastive divergence training of energy based models.\" arXiv preprint arXiv:2012.01316 (2020).\n\n", "summary_of_the_paper": "Inspired by NueTra [1], in this paper the authors propose a new approach to train deep energy-based models (EBMs). The motivation here is that EBMs are typically trained via contrastive divergence which requires MCMC sampling in a high-dimensional space and from  a mutli-modal distribution. As a result, most MCMC sampling methods do not mix well which results in a biased estimation of MLE. To address this problem, the authors propose to train an energy-based model with a (pre-trained) backbone flow; the EBM can be interpreted as a correction step to the flow model. The flow $q_\\alpha(z)$ defines a latent space $z$ with a base distribution $q_0(z)$. As pointed out in [1], we can use the flow network to additionally yield the following distribution on $z$-space: $p(z) = p(x)\\frac{\\delta x}{\\delta z}$. Similar to NeuTra [1], the authors propose to run HMC to sample from $p(z)$ which geometrically is a simpler target, and then feed $z$ to the flow to get $x$ again. The authors perform some experiments on both toy and some of the common vision datasets to show that unlike SGLD, this approach mixes way better and is able to traverse to different local models.   ", "summary_of_the_review": "I very much enjoyed reading this paper. The motivation is stated clearly; the problem of having a biased gradient estimation of MLE in training EBMs is certainly a reasonable problem. The proposed approach certainly makes sense and I think is a very reasonable approach to this problem. While the work is very close to [1] which may raise some concerns regarding novelty, I think it can also be seen as a strength given  that it is showing a very good application of method for posterior inference in training EBMs. Furthermore, the addition of energy-based correction is I think, while trivial, is a very neat trick that enables one to avoid computing the determinant jacobian when running HMC unlike NeuTra. The experiments I think could use some improvements, but overall demonstrate the case that the chains in the proposed approach mix better than SGLD chains. The proposed model does have a small disadvantage in the sense that it requires to train a flow model in advance. However, as the authors show in Section 4.4, even using a small flow results in a significant improvement. Overall, I am happy to recommend acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RoIZ4pd43jr", "reviewer": "Reviewer_17Wo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThis paper addresses the mixing problem of EBM to some extent. It is also well known that training with NCE is difficult, and it is interesting that NCE can work with a product of flow and EBM.\n\nQuestions:\n1. Contribution. A key technique used in this paper is the neutral transport MCMC. The paper uses the technique for both training and sampling. The main difference from Hoffman et al. (2019) is to train with neutral transport MCMC. Thereby, it is important to verify that it is the training with neutral transport MCMC that solves the non-mixing problem instead of sampling with neutral transport MCMC. Perhaps the author can add an experiment to see what will happen if training with normal MCMC and sampling with neutral transport MCMC.\n\n2. Since the model can generate different samples in a single MCMC chain, it can potentially improve the sampling speed. For example, to generate 100 samples, prior EBMs need to run 100 parallel MCMC, and the model in this paper only needs to run one MCMC and get samples from different timesteps. Thereby, it is worth adding FID results calculated by samples taken from one MCMC chain (using different intervals) and reporting the corresponding sampling speed up.\n\n3. Despite the improvement of mixing, the sample quality is bad. It is worse than the MCMC trained EBM (Du & Mordatch, 2019). The author needs to give some explanation on this phenomenon.\n\n4. Missing related work. [1*] uses a product of EBM and VAE, which has some similarity. It should be discussed in related work section.\n\n5. Missing experimental details. The dataset used for Figure 2 is not specified. What is the y-axis of Figure 2 (a), (b)?\n\n6. Typos. In Appendix A.6, the range of the index $i$ and $j$ is not consistent.\n\n[1*] VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models", "summary_of_the_paper": "This paper proposes a product of EBM and FLOW and samples in latent space to improve mixing. Such a model can be learned using both MLE or NCE. Empirically, the paper shows that a single MCMC chain of such a model can traverse through different modes. ", "summary_of_the_review": "The paper is clear and easy to understanding, but there are some questions to be addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7fHO17ADV2S", "reviewer": "Reviewer_tv7r", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-Strengths\n\nAddressing the problem of non-mixing of MCMC for sampling from an EBM is important.\nThe particular class of EBM model is not new, but considering the EBM in the latent space as shown in Eq.(9) seems to be new and has the benefit that the resulting EBM Eq.(9) may be much less multi-modal than the EBM Eq.(7) in the data space. \nThe authors raise good questions, which are exmained by empirical evaluations.\n\n-Weaknesses\n\nThe results are not strong. In Table 1, the FID scores over CIFAR-10 is far worse than GAN-based models [a] and EBM-based models [b]. The unusually bad FID scores hurt the quality of the paper, which makes the method, though interesting, not so convincing and useful.\n\nSome closely related references and discussions are missed.\nFirst, note that the EBM model in [c] is parameterized by CNNs and its MCMC sampling is mixing. Thus, the claim \"To the best of our knowledge, our work is the first where MCMC sampling is mixing for EBM parametrized by modern ConvNet\" in this manuscript should be revised. The idea of learning EBMs by NCE has also appeared in Wang&Ou (2018) (the reference in the paper). I suggest the authors to highlight the differences between these prior studies and this work and add relevant discussions, instead of making big claims.\n\n[a] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv:1802.05957, 2018.\n\n[b] Y. Song, Z. Ou. Learning Neural Random Fields with Inclusive Auxiliary Generators. arxiv 1806.00271, 2018.\n\n[c] B. Wang and Z. Ou, “Language modeling with neural trans-dimensional random fields,” IEEE Automatic Speech Recognition and Understanding Workshop, 2017.\n\n--update after feedback--\nTo acknowledge the revision and improvement of this paper, I raise the score.", "summary_of_the_paper": "I find that this paper is a re-submission from NeurIPS2020, for which I acted as one of the reviewers. The content almost remains the same.\n\nThis paper studies the learning of a special class of EBMs, which is a correction or an exponential tilting of a flow-based model.\nAn interesting observation that the resulting EBM in the latent space is of a simple form that is much more friendly to MCMC mixing.\nIt is said that HMC sampling of the EBM in the latent space, which is a simple special case of neural transport HMC, mixes well and traverses modes in the data space. Regarding this main claim, the authors raise a number of scientific questions, which are validated by empirical evaluations.", "summary_of_the_review": "See above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors set up a simple combination of an energy based model and a flow based model that corrects the flow based model with an energy based term. The merits of this relative only an energy based model is improved sampling to compute the gradient. The advantage over a only flow based model is that the kinds of transforms that can be used are less limited.
Decision: Accept (Poster)