Thinking fourth dimensionally: Treating Time as a Random Variable in EBMs
Authors: Omer Yair
Tomer Michaeli
Venue: Submitted to ICLR 2023 (2023)
Keywords: energy-based models
markov chain mote carlo
contrastive divergence

Abstract: Recent years have seen significant progress in techniques for learning high-dimensional distributions. Many modern methods, from diffusion models to Energy-Based-Models (EBMs), adopt a coarse-to-fine approach. This is often done by introducing a series of auxiliary distributions that gradually change from the data distribution to some simple distribution (e.g. white Gaussian noise). Methods in this category separately learn each auxiliary distribution (or transition between pairs of consecutive distributions) and then use the learned models sequentially to generate samples. In this paper, we offer a simple way to generalize this idea by treating the ``time'' index of the series as a random variable and framing the problem as that of learning a single joint distribution of "time" and samples. We show that this joint distribution can be learned using any existing EBM method and that it allows achieving improved results. As an example, we demonstrate this approach using contrastive divergence (CD) in its most basic form. On CIFAR-10 and CelebA ($32\times 32$), this method outperforms previous CD-based methods in terms of inception and FID scores.

Under review as a conference paper at ICLR 2023
THINKING FOURTH DIMENSIONALLY: TREATING TIME AS A RANDOM VARIABLE IN EBMs
Anonymous authors Paper under double-blind review
ABSTRACT
Recent years have seen significant progress in techniques for learning high- dimensional distributions. Many modern methods, from diffusion models to Mhergy-Based-Modee (EBMs), adopt a coarse-to-fine approach. This is often done by introducing a series of auxiliary distributions that gradually change from some simple distribution (e.g., white Gaussian noise).
the data distribution Methods in this category separately learn each auxiliary distribution (or transi- tion between pairs of consecutive distributions) and then use the learned models sequentially to generate samples. In this paper, we offer a simple way to gen- eralize this idea by treating the "time" index of the series a random variable and framing the problem as that of learning single joint distribution of "time" and samples. We show that this joint distribution can be learned using any exist- ing EBM method and that it allows achieving improved results. As an example, we demonstrate this approach using contrastive divergence (CD) in its most basic form. On CIFAR-10 and CelebA (32 x 32), this method outperforms previous CD-based methods in terms of inception and FID scores.
INTRODUCTION
Probability density estimation is among the most fundamental tasks in unsupervised learning. It is used wide array of applications, from image restoration and manipulation (Nichol et al. 202T
Du et al. [20TT; Lugmayr et al.] 22020; Kawar et al. 2021 2022) to out-of-distribution detection (Du& & Mordatch| 2019 Grathwohl et al. 2019] Zisselman Tamar 2020). However, directly fitting explicit probability model to high-dimensional data is hard task, particularly when the data samples concentrate around a ciw-dimensional manifold, as is often the case with visual data. One way to circumvent this obstacle is by using coarse-to- coarse-toffine approaches. In fact, in one form or another, coarse-to-fine strategies have been used with great success in most types of generative models (both implicit and explicit), including generative adversarial networks (GANs) (Karras al.; 2018), variational autoencoders (VAEs) [Vahdat & Kautz, 2020), emergy-based models (EMBs) (Gao al. 2018 Zhao et al. 2020). score matching (Song & Ermon 2019 Li al. 2019 and diffusion
models DSohl-Dickstein et al. 2015 Ho et al. 2020).
The coarse-to-fine idea is commonly implemented through the introduction of series of auxiliary distributions that gradually transition from the data distribution to some simple known distribution that is smoothly spread in space (e.g., standard normal distribution). This construction is illustrated in Fig. [aa two-dimensional data. The index running over the series of distributions is typically referred to as "time". This to reflect either the diffusion-like sequential manner in which samples generated for training (from fine to coarse) (Sohl-Dickstein et al.) 2015 Ho al. 2020) or the annealing-like like sequential order in which samples are generated from the model at test time (from coarse to fine) (Song Ermon 2019). Methods that this construction attempt learn each of the distributions in the series (or each transition rule between pairs of consecutive distributions) separately of the other distributions
In this paper, we explore a more general approach for exploiting the coarse-to-fine structure, which can be used in conjunction with almost any explicit distribution learning algorithm and leads to
It is common to represent all models by single neural network that accepts the "time" index input. But for each "time" step, the network exposed only samples from the corresponding distribution.

---

Under review as a conference paper at ICLR 2023
Palt..)
Po(()
Pg
P.(0)
Pola)
Po(0)
(a) The standard approach
(h) Our approach
Figure I: (a) Coarse-to- distribution learning methods introduce a series of auxiliary distribu- tions that gradually transition from the data distribution (2D spiral in this example) to some simple distribution (a Gaussian here). These methods learn each auxiliary distribution (or pair of consecu- tive distributions) separately. (b) Here treat the "time" index of the series as random variable, and the samples from all distributions as samples from a single random vector z. We then train the model learn the joint distribution (t, z) using samples (f, z).
improved results. The key idea is to gather the samples from all auxiliary distributions and view them as coming from a single joint distribution. More specifically, we treat the "time" index of the series as random variable, and the samples from all auxiliary distributions samples of random vector z. This allows learning a single model for the joint distribution Pz,( t), using pairs of samples (z, t) (see Fig. lb).
To understand the benefit of this joint modeling, it is important to note that many of the individual distributions (=]( commonly occupy only small regions of the space. Thus, when training separate model for each f. each model is accurate over different region in space, which can lead to inaccuracies at test time when switching between models. In contrast, here we learn the joint distribution Ps.1 (z, f), either directly (Sec. (.44 by breaking the problem in a reverse way and learning (=) and Ptiz (1 (Sec. (.3). Thus, during training, our unified model is exposed to samples from the entire space, leading to better stitching of the different parts.
Once model trained using our approach, can be used similarly to existing methods by extracting the auxiliary distributions (=(() and sampling from them one after the other, from coarse to fine. can also be used in alternative ways, as we discuss in Sec. 3.5
To illustrate the strength of our approach, we apply it together with the vanilla contrastive divergence (CD) method (Hinton 2002) on the CIFARIO (Krzhhvssy et al.) 2009) and CelebA (Liu et al.) (32 32) datasets. It is important to note that although the vanilla CD method is theoretically justified (Yir & Michaeli 2020). it fails when directly applied to high dimensional visual data (Gao) al. 2018). This is because it provides good estimates only nearby the data manifold. To date, good results have been obtained only with persistent contrastive divergence (PCD) (Tilleman) 2008 Du & Mordatch 2019), which maintains buffer of past samples. With our approach, the other hand, plain CD not only succeeds in learning the distribution, but it also improves upon all previous PCD-based techniques in terms of Inception Score (IS) and Fréchet Inception Distance (FID).
2015)
RELATED WORK
The idea of learning an explicit generative model by using an auxiliary coarse-to- fine series of distributions, has been used in many works. We briefly mention its within popular models.
Song & Ermon 2019) constructed a series distributions by adding increasing amounts of white Gaussian noise to the training samples. They learned the gradients of the distributions using denois- ing score matching /Vincent 2011). and used the trained model to solve various generative tasks using gradient based simulated annealing.
2

---

Under review as a conference paper at ICLR 2023
Diffusion models (Sohl-Dicksterl et al.) 2015 Ho et al. 2020; Song et al. 2020 Dhariwal & Nichol 2021 are currently the state-of-the-a- methods for image generation. In this approach, the series of distributions is defined through diffusion process that begins with samples from the training set and gradually transforms the distribution into white Gaussian noise. A series of models (denoisers) is then trained capture the reverse process, allowing to generate samples from each distribution given the preceding one. This process can also be viewed as starting from a noisy image and repeatedly removing small portions of the noise until reaching a clean image (Ho al. 2020
Gao et al. (020bb) used a construction similar to that of diffusion models, but with much fewer dis- tributions. They employed a conditional version of maximum likelihood for training an EBM for each distribution in the series. This was done using short MCMC chains for generating adversarial samples from a selected distribution based on samples from consecutive distributions. The adver- sarial samples are then used along with true samples from the selected distribution for calculating the optimization objective.
Rhodes et al. (2020) address an arbitrary series of distributions that transitions between the dataset and some reference distribution. A series of classifiers is then trained to discriminate between each pair of consecutive distributions. The fact that the ratio between the distributions can be extracted from these classifiers is then used for computing the target distribution in a telescopic manner.
3 METHOD
3.1 INTRODUCING THE JOINT DISTRIBUTION
Given a dataset of i..dd samples { we would like to learn a model for the distribution px from which the samples were drawn. To do so, we first select some known distribution Pn as a reference. A possible choice can be white Gaussian noise. We then introduce series of T auxiliary distribu- tions {Pv, [[-1 which begin with the reference distribution, Pv. Pn. and end with the distribution that we know how to of the data, Pvr-1 = Px- The main requirements from those distributions distributions be small. draw samples from them and that the effective overlap between consecutive
One common construction relies on linear combinations of samples from Px and Pn- That is, given sample from the dataset, x, and sample from the reference distribution, n, an intermediate sample Vt can be generated as
0: = 04 + B,n
with some predefined coefficients {oz, 8: T-) that monotonically transition from (oo, (0) = 10) to (aT-1. 11 = (1, 0).
Up this point, this is the standard coarse-to-fine construction. Here, however, we proceed to view the samples from all distributions [pv, ) as coming from single joint distribution. Specifically, we introduce two random variables. The first is a random "time" index select an auxiliary distribution from the series. A simple choice would be a uniform distribution over [0, T' 1] (in See. 3.5 we discuss how pi can be modified test time to aid the sample generation process). Using t, we define a second random variable, z = un Namely, z is random draw from one of the auxiliary distributions, where the index is randomly chosen according to t, so that
Pt. which is used to
Par (z/t) = (z).
(2)
Given these two variables, we define our auxiliary problem as that of learning the joint distribution Pz,1 based on samples of (z,
Learning the joint distribution can be done in several ways:
1. Using the decomposition Pa.((z, t) Pat (t). Since pt is known, this only requires learning Px]t- which is the standard approact Particularly, for each E {O...., - 1}, the distribution (-I)) (t) is learned using an optimization problem that is defined only in terms of samples corresponding to that t. This approach is illustrated in Fig. la
Sttictly speaking, when using EBB--based methods, each Pajt is leareed up to unknown normalization constant, preventing computation of Ps.-- Methods using this technique, only use the individual Paje at test time.
3

---

Under review as a conference paper at ICLR 2023
2. Using the decomposition Pa. (2, (2) Prjzl (t)))) Here need to learn both terms, which are illustrated in Figs. 2b and 2a
(a) Learning Ptjz amounts solving classification problem (i.e., learning predict the discrete "time" variable {0, (0,..., T 1--1 1} from sample z). This can be done using the cross-entropy loss.
(b) The distribution Pz can be learned e.g., using any EBM learning method. Importantly, this is significantly simpler task than learning Pxjt (especially for the finer t's), as Pz a smoother function that is more spread in space.
3. Directly learning Pz,c- which appears in Fig. 1b This requires slightly adapting existing methods as this is joint distribution over two domains with different properties.
It should be noted that although each of these options suffices on its own for extracting Pz,., it is also possible to use combinations of these losses in order further improve the training (e.g., loss on Pije in addition to the standard loss on Pxji).
Our key observation that substantially improved results are obtained when using the losses of Option 2 and/or 3 in conjunction or instead of the standard loss on Pajt (Option 1). This is because the losses on Pijz= Px, and Pz, are optimized using samples from all auxiliary distributions. This is while the standard approach of training a model per using a loss Pztt- involves samples only from that particular í and these samples come from a very restricted region in space.
In Sections 3.3 and 3.4 we show how options 2 and 3, respectively, can be used in conjunction with the CD method. Option leads to better results than Option 3, and improves upon all previous CD-based methods.
3.2 THE PARAMETRIC MODEL
As opposed to the common approach in which / is fed as input to the model (Song & Ermon 2019
Rhodes et al. 2020 Ho al. 2020). here we propose have the model output vector of length T that contains the values of log Pz. (z, t) for all E {l...., T - 1}. Namely, we use a parametric model fe (a neural network in our experiments). which accepts z as input and outputs
fe (z) [ogg P..c (z.t 0) log Pr.c, (z.t 1)..... ... log Pr.c (z,t= = T 1))
(3)
Here أ denotes the model's estimate of the probability density p. This is illustrated in Fig.
In this design, each element of the output vector is basically an EBM for the corresponding auxiliary distribution (here we define the log probabilities without the minus sign). This is related to the observation in (Grathwohl al. 2019), which draws the connection between classifiers and EBMs. However, as opposed to EBMs, here we strive to learn normalized distribution. This can be done
pdle)
Pq |x (0.6, 1.9))
0.06
0.04
0.02
0
150
250
Ppd xx= (..1
0.015
0.01
0.005
6
50
100
150
200
250
Ptyl (-0.5, 1.9)
0.006
0.004
0.002
0
50
100
150
200
250
(b)
Figure 2: (a) Samples from the marginal distribution Pz. (b) The conditional distribution Pil for the three red points of in (a).
4

---

Under review as a conference paper at ICLR 2023
(x,
Neural Network
logPu: 0)
lop
3
2-3 1-0
(-2 2
(s,
Figure 3:
Our parametric model consumes a sample and outputs a vector log probabilities indicating the joint likelihoods of z and t for all values of t O...., 1.
by using an optimization process that mixes values along with using the fact that Pati(2|0) Pn(2) in order to fix the first element of the output vector of fe(z) to be logPn(z).
Defining the model this way allows efficiently computing log P2:0 (z) and log P (1|=), as
log P2:9 (z) logSumExp (fe (=))
(4)
logıuz: (4]z) = logSoftmax (Je (z)),
(5)
These terms can be used both during training (for applying losses on these distributions) and, as we discuss in Sec. 3.5 at test time (for generating samples from the model).
In order aid the training of the model, we make use of an observation made by Rhodes et al. (2020), stating that it is beneficial to represent a ratio of distributions as a product of many small ratio terms. We apply this idea to our model by adding an additional cumulative-summation layer (g(x); = Ej= zy) before the output of the network. This additional layer does not impact the representation power of the network as is an invertible linear operation that can, in theory, be absorbed into the preceding linear layer. However, this layer does affect the optimization of the model, as modifies the initialization of the effective linear operation preceding the output, as well the optimization dynamics. In practice, we found this implicit bias to be crucial for the success of the model when applied to high-dimensional distributions.
3.3 TRAINING THE MODEL USING CD+CE
In this section we describe how suggested model can be trained using CD loss the marginal distribution together with cross-entropy (CE) loss on Pizz- The overall loss in this method is therefore (+) + (cx(O), and we refer to it as CD+CE (see Alg. [
Contrastive divergence The CD method uses MCMC process generate contrastive samples at each training iteration. Specifically, let us denote by Ti. the transition operator of an MCMC process (an operator that performs single MCMC step) designed draw samples from P2:0- Note that in each training iteration, the MCMC process operates with respect to the current estimate the joint distribution. To generate contrastive sample, the MCMC process is initialized with sample from the dataset, 00 2, and is run for K steps, sk+1
The (z*). This results in a contrastive
sample غ k The CD loss is then defined as
LCD (0) = E [log2.:0 (ž) - log P2:0 (z)]
E [logSumExp (f (z)) logSumExp (f (z))],
where the expectation overdraws of contrastive samples (first term) and samples from the dataset (second term). A popular choice for an MCMC process over a continuous distributions is Langevin dynamics in which the transition operator is given by:
Thu (2 = zk
+
V log Pn: (Et) + μr
2
4
V Vz (s (z)) + μe,
5

---

Under review as a conference paper at ICLR 2023
PSRO
P500)
15
1-2
F17
Figure 4: We present three MCMC processes rumning the joint distribution of the toy model. Note that the intermediate values are used neither in CD+CE nor in JointCD, and are shown here only for completeness. Particularly, in JointCD the MCMC runs over the marginal distribution of z, so that plays no role, and in JointCD we draw only once for the final point.
where N (0, I) and l is the step size. To keep the MCMC process accurate, we use a Setropolis-Hastinge rejection step (Hastings 1970) as part of the transition operator.
Cross entropy Learning Ptjz can be achieved by minimizing the standard cross-entropy loss over the outputs of our model,
LCE (0)
log Prizao (t/z)
[logSoftmxx (f (z)).]-
This is equivalent training a classifier to predict the discrete "time" index t given a sample *.
3.4 TRAINING THE MODEL. USING JOINTCD
Our suggested model can also be trained using a CD loss on the joint distribution Pa.1- We refer to this method as JointCD (see Alg. 2. The distribution Pa,1 is visualized in Fig. Ib
To apply CD on Pæ,,, need a transition operator T of an MCMC process that designed to draw samples from P2.10- Having such an operator, we can initialize the process with sample from the dataset, (2 2 = (z, t), and run (2*+1, .x+1) T (** _ for K steps to generate a con- trastive sample (ž, f) (=k ==,,)). The problem is that popular MCMC techniques, like Langevin Dynamics and Hamiltonian Monte Carlo (HMC), are relevant only for continuous distributions, whereas in we show in in A any continuous MCMC process can be extended to work on our joint mixed distribution simply by performing step using the continuous MCMC operator Tiu and then sampling i from Piz- Namely, a single MCMC step in our case takes the form
Pz,1 is mixed distribution (z is continuous and t is discrete). Nevertheless, as
ggh+1
(x
Teno (# 00,
Jh-1 ~ Pi (-1xht)).
We show in App. App.t this process obeys the detailed balance criterion and thus its stationary dis- tribution is indeed Pi, as desired. Note that the intermediate t values are not required for sampling the next step. Therefore, in practice, we draw only the last one, EK We illustrate this joint MCMC process in Fig.

---

Under review as a conference paper at ICLR 2023
Algorithm 1: CD CE
Algorithm 2: JointCI)
while not converged do Sample ~ Pr. Pe/t=t
while not converged do Sample t ~ Pt. ~ Pale-t
f- for 1 to do
3 for to K do + Tiuo (z)
I +- Timo (2)
end
end
Sample i from P1ix=2:0 Take gradient step on
Take gradient step on
log22.0 (z) -log Pr.o (z)-log Pilz: (f/z),
log P2.1. (z, f) - log Pz.1. (z, t),
computing the densities using
computing the densities using (3)
end
end
3.5 SAMPLING FROM THE MODEL
Sampling from the trained model can be done via simulated annealing, In this approach, one runs an MCMC process while constantly replacing the underlying distribution, starting from a simple smooth distribution and gradually refining it into the target distribution. In our context, this is commonly done by running through the series of learned auxiliary distributions, starting with P=]== and gradually increasing t until reaching Px|t-T-1 (which equals px). This algorithm outlined in Alg. in App. B and illustrated in Fig. [aa
We note that our approach allows viewing simulated annealing special case of more general sampling scheme. Specifically, we can interpret simulated annealing as running through a series of distributions p where (z)tt P=tr(=(t) for all n and pe " (t) 5(t n) (here 8(.) denotes kronecker's s delta function). We can therefore generalize this method by using any sequence of distri- (t)) whose centers of mass gradually move from the small values of to the larger ones. In this generalized setting, we have Pz (z, () Pat (2)((=))))))
butions (x(")) (t))
P2, t)pp""() (t)/p((), (t), from which pzz can be extracted by summation over t. The resulting generalized simulated annealing al- thiform[t(n... T
(t) (z)
(1)
gorithm is outlined in Alg. iin App. One particular choice of p. is Pr (n) essentially the mean where ((n) a linear function growing from 0 to T 1. For this choice, Px
(")
(n)
of all the auxiliary distributions, from ((n) to t = T 1. We refer to this variant as soft simulated annealing and exemplify it in Fig. 5b
[=1
F51
F101
F153
F200
F255
(a) Simulated annealing
e1
esi
P101
BS153
r2203
1825
(b) Soft simulated annealing
(c) Ground-truth
Figure 5: We depict intermediate samples along with the underlying distribution from the basic simulated annealing (a) and the soft simulated annealing (b) processes applied to the learned spiral toy model. The ground truth distribution is shown in pane (c).
7

---

Under review as a conference paper at ICLR 2023
(a) Sample generation using simulated annealing from model trained with CD+CE.
(b) Images generated from models trained on CelehA and CIFARIO using CD+CE.
Figure Results on CIFAR10 and CelebA.
4 EXPERIMENTS
We now illustrate the efficacy of our method on toy problem as well as on the CFIARI0 and CelebA datasets. Code for all experiments will be released upon acceptance of the paper.
4.1
TOY MODEL.
The toy problem appearing in Fig. 5c involves data lying 2D shifted spiral, where the shifting has been introduced to aid the visualization. To apply our method, we selected a Gaussian dis- tribution centered at the origin as reference distribution and defined 256 auxiliary distributions, according to (I). We then used JointCD to train a neural network. For the MCMC process, we used Metropolis- Hastinggs adjusted Langevin dynamics with K steps. The details of the network and the training process can be found in App. The resulting learned model is shown in Fig. 5a
4.2 CELEB A & CIFAR10
We trained models using both CD+CE and JointCD on CIFARIO and on CelebA at 32 x 32 reso- lution. Here have used 1024 auxiliary distribution, also according to the linear interpolation in (T] As our parametric model, we used ResNet18 (He et al.) 2016) with minor changes. Here as well, used adjusted Langevin dynamics, but during training gradually increased the lengths of the
8

---

Under review as a conference paper at ICLR 2023
Table FID and inception scores for methods trained unconditionally on CIAFRIO
Model
Inception t
FID
GAN-based
SNGAN (Miyato et al. 2018)
8.22 9.83
21.7
StyleGANZ-ADA Karras et al.
2020)
2.92
Normalizing flows
Residual flow (Chen et al.
2019
46.37
FCE Gao et al. 2020a
37.3
Score based
NCSN-v2 (Song & Ermon 2020)
8.4
10.87
DDPM (Ho et al. 2020
9.46
3.17
EBMs (ML based)
CoopNets (Xie et al. 2018)
6.55
33.61
Multi-grid EBM (Gao et al.
2018
40.01
6.56
CF-EBM ZZao et al.
2020
16.71
EBM-DRL Gao et al.
2020b
8.3
9.58
EMBs (CD based)
EBM-IG (Du & Mordatch 2019
6.78
38.2
Improved-EBie (Du et
202T
7.85
25.1
JointCD (Our)
9.09
26.8
CD+CE (Our)
8.29
23.7
chains, from K to K 100. worth noting that due the need to take large number of MCMC steps before each gradient step, the training is slow and took days on 4 RXT-2080Ti GPUs. The full details of the network and the training process can be found in App. C
For generating samples, we used simulated annealing. We found that on CIFAR10, 10, the CD+CE method performs better than JointCD, The generation process visualized in Fig. 6a and generated samples are shown in Fig. (6b These results are all from the CD+CE model. Results from the JointCD method can be found in App. D
We used the inception score (Salimans al. 2016) and FID (Heusel al., 2017) evaluate the CIFAR results With the CD+CE method, we achieved an inception score of 8.5 and an FID score of 23.7. As be seen in Table I these results improve upon previous CD based techniques.
CONCLUSION
We presented methods for harnessing coarse-to- coarse-to-fine series of distributions for learning EBMs. Our approach views the "time" index of the series as a random variable and defines an auxiliary task of learning the joint distribution of "time" and samples. We illustrated how using this approach in conjunction with the CD method, leads to substantially improved results. One limitation of our method is that it requires relatively long training times. However, believe that with further hyper- parameter tuning and correct architectural choices, this can be somewhat alleviated in the future. Our joint modelling approach can principle be used within other generative models, but leave those directions for future research.
Using the PyTorch implementation from https:/pyyi.org//peeet wwwp://wwwwee/ittoooaaaaaaaaaame -eatrrrcc.gga-mtircll....l which has been shown reproduce the scores of the original implementation with smaller then 0.2%.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Henrik Jacobsen, Residual flows for invertible generative modeling. Advances in Neural Information Processing Systems, 32, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-888 8794, 2021.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, 2019.
Yilun Du, Shuang Li, Joshua B Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. In Energy Based Models Workshop-ICLR 2021, 2021.
Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chur Cang-Chun Zhu, and Ying Nian Wu. Learning generative con- vnets via -uld modeling and sampling. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 9155-9164, 2018.
Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy- -ased models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020a.
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy -bssed models by diffusion recovery likelihood. In International Conference on Learning Representa- fions, 2020b.
Will Grathwohl, Kuan-Chich Wang, Hbern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. International Conference on Learning Representations, 2019.
WK Hastings. Monte carlo sampling methods using markov chains and their applications. Biometrika, pp. 97-109, 1970.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800 2002.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33: 2104-12114, 2020.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochas- tically. Advances in Neural Information Processing Systems, 34:21757-21769, 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In ICLR Workshop Deep Generative Models for Highly Structured Data, 2022.
10

---

Under review as a conference paper at ICLR 2023
P Kingma Diederik and Jimmy Ba Adam. A method for stochastic optimization. arXiv preprint arXiv:1412.6999, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Computer Science Depariment University ofTorunto Tech. Rep, 2009.
Zengyi Li, Yubei Chen, and Friedrich T. Sommer. Learning energy-based models in high- dimensional spaces with multi-scal denoising score matching. 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- on Learning Representations, 2018.
Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte. Srflow: Learning the super- resolution space with normalizing flow. In European conference computer vision, pp. 715-732. Springer, 2020.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 12. 12.
2021.
Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. Ad- vances in neural information processing systems, 33:4905-4911,, 2020.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33: 12238-12444 2020.
Tijmen Ticleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference on Machine learning, pp. 1064- 1071,
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- lation, 23(7):1661-1177 2011.
Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of de- scriptor and generator networks. IEEE transactions on pattern analysis and machine intelligence, 42(1))27-45, 2018.
Omer Yair and Tomer Michaeli. Contrastive divergence learning is a time reversal adversarial game. In International Conference on Learning Representations, 2020.
11

---

Under review as a conference paper at ICLR 2023
Yang Zhao, Jianwen Xie, and Ping Li. Learning energy-based based generative models via searse-to-fine expanding and sampling. In International Conference on Learning Representations, 2020.
Ev Zisselman and Aviv Tamar. Deep residual flow for out of distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13994-14003, 2020.
12

---

Under review as a conference paper at ICLR 2023
A WHY JOINTCD OBEYS DETAILED BALANCE
A sufficient condition for an MCMC process to converge is that it maintains a relation that is know as detailed balance between the conditional distribution of the transitions and the underlying dis- tribution from which we would like to sample. Specifically given an underlying distribution P²(2) and an MCMC process with a conditional probability of PŽ|u (ž]=) step form ž, then detailed balance criteria is given by:
(Zlz) Pa (z) [z)
Pž
(10)
Pz
That exist various MCMC process that operate on continuous random variables and obey (usually approximately) the detailed balance criteria. In order to apply one of these existing MCMC process to the joint distribution of Pa. we suggest the following way take step from (2, to (122, f). In each step we begin by ignoring and generate & using the continuous MCMC process on with regard to the marginal distribution Pz. We then sample based on غ according to the conditional distribution Pt|z- I.e.:
Pa.iz. (2, tz, Pr/z (8 2) Ps/z (z|z)
(11)
As long as the continuous MCMC process obeys the detailed balance criteria than so does the sug- gested process with respect to Pz.t- This can be seen from following derivation:
2. (z, a|. t) Pz, (z, t)
x (7 2 P22z (7|z) Px,1 (z, ()
(12)
P, n, [z, 2/2, t) Pa.1 (2,
Pt x (t z)) Paix ( Pz.1 [z. f)
Pix x (f 2) Pala (z|z) Pija (t|z) Ps (z)
(13)
Pt a (1 (z) Pala (z|²) Pila (7 Ps (2)
Palz (7) (2) Ps (z)
(14)
Paja (z) [2) Ps. (z)
1.
(15)
B THE SAMPLING ALGORITHM
We outline in alg. 3 the process for generating samples from the model using the common simulated annealing, and in alg. the process of using the generalized simuulated annealing described in section 3.5
Algorithm 3: Basic Simulated Annealing
Algorithm 4: General Simulated Annealing
Draw a sample z from Pn = Pz|t=0
Draw sample z from Pn = P2/t=0 for n = 0 to N do
for t = 0 to T - do for n to N do
1 - T (2)
P210
[m) = ΣT- T-1
Pr.o
t) Pt (e) m(())
5 T [(()]
(2)
end
Polo
end
end
C EXPERIMENT DETAILS
C.I TOY MODEL
For the toy model, we used a network of 4 residual blocks containing fully connected layers with a width of 256. As a reference distribution we used white Gaussian distribution with zero mean and STD on = 0.3. We found it beneficial to have the STD of the reference distribution slightly larger then that of the data. We trained the model using JointCD (Alg. 2 with Metropolis-Hastingg adjusted Langevin dynamics. The Langevin step size was been adaptively adjusted during the run to maintain an average acceptance rate of 60% in the Metropolis- Hasttiol adjustment stage. This was done by keeping an array of an individual step size for each value of t. We have found the step size of each to converge in an early state of the training about 0.4,3, where
13

---

Under review as a conference paper at ICLR 2023
We have used Adam (Kingma Diederik & Adam 2014) optimizer with learning rate of 10-3 and the default momentum values of 31 = 0.9, 32 was trained for 400 epochs (30 minutes RTX 2080Ti GPU).
0.999. We used a batch size of 256. The model
C.2 CIFARIO AND CELEBA 32x32
As our model we have used ResNet18 with the following modifications:
We have removed the BatchNorm (loffe Szegedy 2015) layers which, makes the network nondeterministic par input.
2. We have replaced the ReLU activation with LeakyReLU.
3. We have added cumulative-summation layer before the output of the network as described
in section 3.2
As reference distribution, have again used white Gaussian distribution with zero mean and STD of on = 0.5 and we have defined a series of 1024 distribution according to ( When working with the images have subtracted value of 0.5 from all the pixels in order roughly center the data around
We have used the same MCMC process in the toy model along with the same method for adjusting the step size. We have used Adam W (Loshchilov & Hutter 2018) as an optimizer with a weight decay of 10-3, We have used an initial learning 4000 steps. We have reduced the learning rate by factor of 10 every 200,000 steps over the full training process which consisted of 300,000 steps. We used batch size of 512.
of 10 with linear warm-up during the first
The number of steps Langevin dynamic has been set to 5 during the first 40000 and was linearly increased up to 100 over the course of the next 40000 steps. From there on the number of steps has been fixed to 100. Due to this large number of Langevin steps, the training time was relatively long and took 7 days on RTX 2080Ti GPUs.
14

---

Under review as a conference paper at ICLR 2023
Figure 7: Generated images from CelebA using a model trained using CD+CE
D ADDITIONAL RESULTS
15

---

Under review as a conference paper at ICLR 2023
Figure 8: Generated images from CIFAR10 using a model trained using CD+CE
16

---

Under review as a conference paper at ICLR 2023
Figure 9: Generated images from CelebA using model trained using JointCD
17

---

Under review as a conference paper at ICLR 2023
Figure 10: Generated images from CIFARIO using model trained using JointCD
18


Reviews:
[{"review_id": "rKW4YE9yxX", "reviewer": "Reviewer_KRWa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a generative model that draws on the diffusion-based and score-based models that generate samples through a sequence of intermediate distributions, but proposes different parameterisation, training criteria and sampling procedure. The model is motivated largely by the fact that prior models treat time (that parameterises intermediate distributions) as discrete and do not benefit from parameter sharing between these distributions. Limited model evaluation is presented.", "summary_of_the_review": "Interesting well-motivated generative model that needs stronger empirical results.", "strength_and_weaknesses": "**Strengths**\n* The paper proposes an interesting view on diffusion and score-based models that leads to a different formulation and training criteria.\n* The paper is well-written and clearly motivated.\n* Empirical evaluation appears to be sound.\n\n**Weaknesses**\n* The motivation of lack of parameter sharing between models of $p_z(z|t)$ is not convincing. For example, this does not seem to take into account the existence of continuous time diffusion- and score-based models, or the fact that in practice $p_z(z|t)$ are often implemented with neural networks that share most of the parameters.\n* Computation efficiency of the proposed model is not discussed. For example, taking K (which runs up to 100) steps of MCMC in each training iteration seems rather expensive, especially when comparing this to diffusion-based models that can generate a sample from an intermediate distribution required for training in a single step. This aspect of the model should be discussed.\n* Despite clear motivation the benefits of the proposed method are not obvious form the empirical evaluation. For example, the proposed model has one of the worst FID scores among the competing methods and is far behind the SOTA methods in terms of Inception and FID scores. Why is that?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mqmSzooNXLL", "reviewer": "Reviewer_qaem", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to train EBMs that can model the distribution defined by uniformly sampling a linear interpolation, called time, between a noise distribution and the data distribution. The joint distribution of the interpolated image distribution and the time random variable is the target density, and samples from the target density can be obtained by jointly sampling from noise, data, and the interpolation proportion. Two methods are proposed to learn the joint distribution. The first uses joint MCMC sampling to obtain the negative samples and standard CD loss. The second uses a JEM-like objective where the EBM parameterizes the marginal of the image distribution via a log sum exp over the output of the EBM output, which is a vector of the conditional densities for each class. One can draw samples of the model by initializing states from noise and performing a few Langevin updates on at each level of time until the time step corresponding to the data distribution is reached. Experiments evaluating the method are performed on CIFAR-10 and Celeb-A.", "summary_of_the_review": "Overall, I found this work interesting but not exceptional. The core idea of the proposed method is the biggest strength of the paper, and it could prompt further design of diffusion-like sequences of distributions from a reference distribution to the data. The experimental results are reasonable but not extremely compelling and the gap between EBMs and diffusion models remains wide. The high computational cost could be an obstacle towards experiments with more complex datasets. I am on the borderline but inclined to recommend against accepting.", "strength_and_weaknesses": "STRENGTHS:\n\n1. The technical innovations in this work are an interesting application of design principles in diffusion models to EBM learning. The CD+CE method is an elegant way of training the joint density that relates nicely to existing work. \n\nWEAKNESSES:\n\n1. I am not sure that the claims about the method being the first successful CD-based method are valid (paragraph right before Section 2). The Multigrid method (Gao et al. 2018) and Diffusion Recovery (Gao et al. 2020) use transformed data samples for both initial MCMC samples and for the positive samples. One could make the argument that these works use CD if we broaden the definition of CD to include samples that can be created by transforming the data, as done in the present work.\n2. The empirical results in terms of FID score are solid for EBMs but not exceptional. I have some doubt about the reported Inception scores of 8.4 and 8.5, which are much higher than the Inception scores of models with much better FID. Although I realize that Inception and FID can be at odds, this is not generally the case on simple datasets like CIFAR-10. Further experiments at a higher resolution beyond 32x32 or with more diverse datasets would improve the experimental evaluation.\n3. EBM training times are already a critical issue, and further increases in runtime are a significant drawback.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ez7llmHGEB3", "reviewer": "Reviewer_k6Mq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to treat \"time\" in an energy-based model as a random variable and learns a joint distribution of time and samples. Concretely, this results in a parameterized function that outputs the joint log probability of the samples and time indices. Two methods of training such a model have been presented: a) using product of conditional and marginal where training is achieved by the optimization of the contrastive divergence and cross entropy losses; b) using the joint distribution directly. The proposed model offers slight improvements over baseline models trained via constrastive divergence.", "summary_of_the_review": "The key proposal of treating time as a random variable lacks clarity and motivation. The overall technical contribution and significance of this work is limited. ", "strength_and_weaknesses": "[Strengths]\n* I found the idea of modeling a joint distribution by making the neural network output log probabilities for every time index interesting.\n\n[Weaknesses]\n\nMotivation/Claims/Significance:\n* The proposed approach of treating time as a random variable is not well-motivated in the introduction. It is unclear why such a treatment is needed or even useful. Furthermore, it is unclear what treating the time as random variable even means in the context of energy-based models. Sampling methods, such as Langevin Monte Carlo, follow the steepest descent curve in the Wasserstein space (given by the Fokker-Plank equation) to draw from the Boltzmann distribution. This curve of probability densities can be indexed by time but I am not sure what treating time as a random variable would imply.\n* The only potential benefit that's mentioned in the introduction is that it leads to \"improved results\". However, the improvement is over a specific class of EBM and is marginal.\n* It is claimed that the method can be used \"in conjunction with almost any explicit distribution learning algorithm\"; however, only a specific variant has been studied in the paper. It is unclear how it would be applicable to other algorithms/models and how that would help.\n\nWriteup:\n* Some things in the paper are unclear:\n    * Is Eq. (1) just an example that motivates the discussion or is it the form that's used in the experiments. I am not sure if previous works have studied intermediate distributions that are linear combinations in the data space.\n    * What is meant by \"which the standard approach\"? As per my understanding, EBMs learn energy of the Boltzmann distribution and are not indexed or conditioned on time. \n    * Why did the authors choose to initialize the MCMC process \"with a sample from the dataset\"? Can the authors explain what they meant by \"vanilla contrastive divergence\"?\n\nTechnical Contribution and Empirical Evaluation:\n* The overall technical contribution of this work is limited and it doesn't offer any new insights. \n* The sampling procedure now has an inner loop which appears to increase the computational complexity. \n* The authors state that the method results in \"substantially improved results\". However, scores have been reported only for one dataset and the improvement is marginal, that too over the specific class of CD-based EBMs. CelebA scores are conspicuously missing. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject