Learning Energy-Based Generative Models via Coarse-to-Fine Expanding and Sampling
Authors: Yang Zhao
Jianwen Xie
Ping Li
Venue: ICLR 2021 Poster (2021)
Keywords: Energy-based model
generative model
image translation
Langevin dynamics

Abstract: Energy-based models (EBMs) parameterized by neural networks can be trained by the Markov chain Monte Carlo (MCMC) sampling-based maximum likelihood estimation. Despite the recent significant success of EBMs in image generation, the current approaches to train EBMs are unstable and have difficulty synthesizing diverse and high-fidelity images. In this paper, we propose to train EBMs via a multistage coarse-to-fine expanding and sampling strategy, which starts with learning a coarse-level EBM from images at low resolution and then gradually transits to learn a finer-level EBM from images at higher resolution by expanding the energy function as the learning progresses. The proposed framework is computationally efficient with smooth learning and sampling. It achieves the best performance on image generation amongst all EBMs and is the first successful EBM to synthesize high-fidelity images at $512\times512$ resolution. It can also be useful for image restoration and out-of-distribution detection. Lastly, the proposed framework is further generalized to the one-sided unsupervised image-to-image translation and beats baseline methods in terms of model size and training budget. We also present a gradient-based generative saliency method to interpret the translation dynamics. 

Published as a conference paper at ICLR 2021
LEARNING ENERGY-BASED GENERATIVE MODELS VIA COARSE-TO-FINE EXPANDING AND SAMPLING
Yang Zhao, Jianwen Xie, Ping Li
Cognitive Computing Lab Baidu Research
10900 NE 8th St. Bellevue, WA 98004, USA fyangzhao.eric, jianwen. kenny, pingl 98) gma i com
ABSTRACT
Energy-based models (EBMs) parameterized by neural networks can be trained by the Markov chain Monte Carlo (MCMC) sampling-based based maximum likelihood estimation. Despite the recent significant success of EBMs in image generation, the current approaches to train EBMs are unstable and have difficulty synthesizing diverse and high-fidelity images. In this paper, we propose to train EBMs via a mul- tistage coarse-to to-sie expanding and sampling strategy, which starts with learning a coarse-level EBM from images at low resolution and then gradually transits to learn a finer-level EBM from images at higher resolution by expanding the energy function as the learning progresses. The proposed framework is computationally efficient with smooth learning and sampling. It achieves the best performance on image generation amongst all EBMs and is the first successful EBM to synthe- size high-fidelity images at 512 x 512 resolution. can also be useful for image restoration and out-of- of-disstibuution detection. Lastly, the proposed framework is further generalized to the one-sided unsupervised image-to-image translation and beats baseline methods in terms of model size and training budget. We also present a gradient-base generative saliency method to interpret the translation dynamics.
INTRODUCTION
E(X)
magge+ gyple
Finpigh
dog
Coarse-Level
Line-Level
(x)())
Figure 1: Highlights our contributions. (a) Coarse-to-fine EBM with gradient-based MCMC sampling. (b) High-fidelity image generation (512 x 512), image denoising, and image inpainting from top to bottom. (7: corrupted image, É: restored image, æ: ground truth). (c) Energy-basee unsupervised image-to-image translation (256 256) interpreted by the generative saliency.
Recently, energy-based models (EBMs) (Zhu et al., 1998; LeCun et al., 2006) parameterized by modern neural networks have drawn much attention from the deep learning communities. Successful applications with EBMs include generations of images (Xie al., 2016; 2018b; Du & Mordatch, 2019), videos (Xie et al., 2017; 2019), 3D volumetric shapes (Xie et al., 2018c; 2020), unordered point clouds (Xie et al., 2021a), texts (Deng al., 2020), molecules (Ingraham al., 2018; Du

---

Published as a conference paper at ICLR 2021
et al., 2019), etc., as well image-to-image translation (Xie al., 202 lb:c), out-of-distribution detection (Liu et al.. 2020) and inverse optimal control (Xu et al., 2019). EBMs are characterized by (i) Simplicity: The maximum likelihood learning of EBMs unifies representation and generation in single model, and (it) Explicitness: EBMs provide an explicit density distribution of data by training an energy function that assigns lower values to observed data and higher values unobserved ones.
However, it is still difficult to train an EBM to synthesize diverse and high-fidelity images. The maximum likelihood estimation (MLE) of EBMs requires the Markov chain Monte Carlo (MCMC) (Liu, 2008; Barbu & Zhu, 2020) to sample from the model and then updates the model parameters according to the difference between those samples and the observed data. Such an "analysis by synthesis" (Grenander et al., 2007) learning scheme is challenging because the sampling step is neither efficient nor stable. In particular, when the energy function is multimodal due to the highly varied or high resolution training data, it is not easy for the MCMC chains to traverse the modes of the learned model. Fortunately, is common knowledge that the manifold residing in downsampled low-dimensional image space is smoother than that in the original high-dimensional counterpart. Thus, learning an EBM from low-dimensional data is much stabler and faster than learning from high-dimensional data in terms of convergence (Odena et al., 2017; Gao et al., 2018).
Inspired by the above knowledge, we propose to train EBMs via a multistage coarse-to torsseto--fin expanding and sampling strategy (CF-EBM). As shown in Figure (a)), the approach starts with learning coarse- level EBM on low resolution images and then smoothly transits to learn the finer-level EBM by adding new layers that take into account the higher resolution information as the learning progresses. The gradient-basse bbased short-run MCMC (Nijkamp et al., 2019), e.g., Langevin dynamics (Neal et al., 201 I), used for sampling. From the modeling aspect, the coarse-level training can be useful for exploring the global structure of image, while the fine-level training will then gradually refine the image details. Recent works have demonstrated the advantages of this incremental learning (Karras et al., 2018: 8;; Wang et al., 2018). However, there have been no works focusing on the incremental learning of FBMs that incorporates bottom-up representation and top-down sampling in a single net. Besides, as shown in Figure (a). the top-down gradient information for synthesis flows from coarse-level layers towards fine-level layers. Thus, during the coarse-to-fin fine expanding, we can use the coarse-level synthesis to help the fine-level synthesis to stabilize the sampling. Such a coarse-to-fine fine expanding and sampling scheme is useful for high-fidelity synthesis in several vision tasks. See Figure (bb).
Furthermore, we propose a one-sided energy-basee unsupervised mage-to-imagee translation method and scale it up to high resolution. The approach is immediately available with the FC-EBM by using its iterative Langevin dynamics without the need of the cycle consistency (Zhu et al., 2017) or geometry constraints (Fu et al., 2019). Specifically, we learn an EBM of target domain with Langevin dynamics initialized by the examples from source domain. The resulting translator is the short-run MCMC. Compared with those prior works (Zhu et al., 2017; Huang et al., 2018; Park et al., 2020) that learn black-box encoder-decoder networks between domains, our method is much more interpretable in the sense that ours can be explained by a visualization method (Simonyan et al., 2014; Adebayo et al., 2018) that uses gradients to visualize the most essential regions, i.e., the generative saliency, when translating an image from the source domain to the target domain. See Figure ((c).
The contributions of our paper can be summarized as:
To the best of our knowledge, this is the first work that trains EBMs under the "analysis by synthesis" scheme via multistage coarse-to- fine expanding and sampling strategy. Besides, we propose several essential techniques for improving EBM, egg., smooth activations. Particularly, our work is the first to train a pure EBM for synthesizing 512 x 512 images.
We propose novel energy-based unsupervised image-to-image translation approach, which is essentially different from all other existing GAN-based approaches. We demonstrate noticeable results in terms of both translation quality and efficiency of time and memory.
We conduct extensive experiments to validate our approach. including image generation, denoising, inpainting, out-of-distribution detection and unsupervised image translation. Strong results show that our method outperforms or is competitive with the prior art.
The rest of the paper is organized as follows. Section summarizes related works and how the proposed method is different from the prior art. Section 3 introduces the proposed methodology in detail. Section 4 presents extensive experiments to test our method. Section concludes our paper and discuss some future research directions.
2

---

Published as a conference paper at ICLR 2021
2
RELATED WORK
2.1
ENERGY-BASED GENERATIVE MODELS
The main challenge to train EBMs via MLE lies in drawing fair samples from the model, especially when the energy function is parameterized by a highly non-linear ConwNet. The contrastive divergence (CD) (Hinton, 2002; Tieleman, 2008), with MCMC chains initialized from data distribution, is an efficient but biased way to train EBMs. Another direction is to adopt the idea of energy-based correction of a more tractable model train EBMs. Noise contrastive estimation (NCE) (Gutmann & Hyvärinen, 2010; Gao et al., 2020) and introspective neural networks (INNs) (Lazarow et al., 2017; Jin et al., 2017; Lee al., 2018b) belong to this theme. Generative cooperative networks (CoopNets) (Xie et al., 2018b; 2021d;b) train an EBM with generator or a variational auto-encoder (VAE) (Kingma& & Welling, 2014) as amortized sampler by MCMC teaching (Xie et al., 8a). Triangle divergence (Han et al., 2019) trains an EBM without MCMC by amortizing the MCMC via VAE. However, these frameworks still struggle to scale up and model multimodal data. There have been several strategies to improve the EBM training. Gao et al. (2018) adopts a multi-grid method that trains multiple EBMs at different grids simultaneously, where the EBM at coarser grid is used to initialize the image generation by EBM finer grid. However, optimizing and sampling from multiple EBMs will result in low efficiency of both time and memory. To stabilize the training, Nijkamp al. (2019); Grathwohl al. (2020) add the Gaussian white noise to the observed data, resulting in noisy syathesized images. In contrast, our paper proposes to train a single EBM via a coarse-to-fing growing strategy, along with some improved techniques. With smooth parameter training and image sampling, our model can preserve EBM's 's compatibility and synthesize high-fidelity images.
2.2 UNSUPERVISED IMAGE-TO TO-IMAGE TRANSLATION
Unsupervised image-to- to-iaae translation aims at learning two directions of mappings between two unpaired domains. Recent successes are all based on adversarial learning, e.g., CycleGAN (Zhu et al., 2017), UNIT (Liu et al., 2017), MUNIT (Huang et al., 2018), DRIT (Lee et al., 2018a) and U-GAT- IT (Kim et al., 2020). These methods typically train two GANs with two levels of learning objectives: (i) Distribution level: Two adversarial losses are used to capture style discrepancy between source and target domain; (ii) Instance level: To tackle the difficulty of unpaired setting, they adopt cycle consistency loss for content preservation. This loss enables an instance-level supervision to regularize the training of two mappings by enforcing them be bijective function between two domains. Except for works about two-sided translation, efforts on research about one-sided unsupervised image translation have also been made, e.g., DistanceGAN (Benaim & Wolf, 2017), GeGAN (Fu al., 2019) and CUT (Park et al., 2020), which apply geometric or contrastive constraints. We solve this problem from the prospective of EBM, which is different from GAN-based methods. The proposed concise EBM solution only relies on its built-in objective, which is a distribution-level statistics matching, to accomplish the one-sided image translation. It transfers the style and preserves the source content by MCMC without using the cycle-consistency loss. The model demonstrates better performances with less time and memory. Another distinction between our method and GAN-based methods is the natural interpretability of Langevin dynamics. It provides gradient-based saliency map to visualize those key regions that make the two domains distinct, as illustrated in Figure ((c).
3 METHOD
In this section, we first present the EBM learning framework, and then the proposed CF-EBM approach. After that, we generalize our model to the unsupervised image-to-image translation.
3.1 MCMC-BASED MAXIMUM LIKELIHOOD LEARNING OF ENERGY-BASED MODEL
Let E R be the observed example, e.g., image. An energy- based model is defined follows:
Pe (x) Z()) exp( (=)())))
(1)
where Ex(x) RD + R is the energy function defined by a bottom-up ConvNet parameterized by 0. Z(0) f exp(-Ex(x) Ex(x)) dx is the intractable normalizing constant or the partition function.
Given N observed examples (x; V ~ (x), where (xma(x) denotes the unknown data distri-
bution, the model can be trained by maximizing the log-likelihood T.()) NV Em log po (x:) x
N
3

---

Published as a conference paper at ICLR 2021
(x-xpo((x) (x)( The derivative of the negative oogliikelhood is given by
VoI(0) x (x) Vo F.o (x) (x) Eo()),
where the second expectation term under (x) is intractable and can be approximated via MCMC. Given that, the EBM is updated by gradient descent. To sample & pa (x) via MCMC, we rely on gradient-based Langevin dynamics that recursively computes the following step
(x, + ViTEe, G ~~~ ~N,,I,,I) 1It X = x 2
where T/s is the step size of Langevin step and also the variance of Gaussian noise et. Theoretically, to ensure convergence, the MCMC is typically performed with infinite steps and an infinitesimal stepsize (Welling& & Tch, 2011). However, it is impractical for training EBMs. In this paper. we follow Nijkamp et al. (2019) to use short-run MCMC, which always starts from a fixed noise distribution and runs a fixed number 7 of Langevin steps in both training and testing stages. The training with short-run MCMC might result in a biased estimation of EBM but the learned short-run MCMC is still a valid generator, which enables us to synthesize realistic images and efficiently train the model, as seen in most well-established EBM works (Nijkamp et al., 2019; Grathwohl et al., 2020; Pang et al., 2020). In this paper, we keep the step size constant and linearly decay the noise variance till 0.
3.2 CF-EBM: COARSE-TO-FINE EXPANDING AND SAMPLING OF ENERGY-BASEE MODEL
Our primary contribution is the mul- tistage coarse-to-fing fine expanding and sampling methodology for training EBMs. The key idea is to incremen- tally grow the EBM from a low resolu- tion (coarse model) to high resolution (fine model) by gradually adding new layers to the energy function. In this way, both stability and time efficiency in training EBMs benefit. And even- tually, we only keep the EBM at the highest resolution for image generation using the short-run MCMC sampling.
Algorithm 1: CF-EBM Training
Input: Muliirresolution data {x I...,, N:s I...., S}. numbers Langevin steps (T()) Ji-1 and batch sizes (r) 1
Output: Energy function ES3)
(X Ø
for 1::5
m = 0, 8 =0
Expand model: E <- Ela-1) + Expand(
while (m S N) do
Draw real data and initial MCMC samples
Draw observed (x]
~ Pxma(x)
(g Draw initial {x] J==1 21(0)
Smooth MCMC sampling
(8 < and s 1) then
Figure 2 (a) illustrates the proposed multistage coarse-to-fine EBM train- ing strategy, which is accompanied by the pseudocode in Algorithm 1. Let S denote the total number of training stages and (x(*),
Draw initial (x] p
Jim1 ~ ((11)
Update {x] pm
Ji-1 for T 1/ (3) steps
Eg. (3) with E6-1)
V P1-Bupsamlee(x
3x
end
Update )o for T steps of Lqq (3) with g(s)
1, ...,. S) denote the multi-resolution versions of an im- age x, with x()) being the minimal res- olution version of x, and x(5) x. For each x(*), we can easily generate x(=-1) by average pooling with a pool-
Optimize energy function
Update Etal with gradient descent in Eq. (2)
m <- m + n0())
max(1, 2N/a(o)
end
end
Multistage Leazning
E(x)
Smooth Sampling
Elx)
Tronslation
vy0(y)
noise
Br8
x00
Stage1
forword
16x16
16c16
Sx
C
St
S
32x32
32X92
(3)
Stage3 3
y
x
(b)
[c]
Figure 2: Illustrations of (a) multistage CF-EBM, (b) smooth sampling, and (c) image translation.
4

---

Published as a conference paper at ICLR 2021
ing size. At stage s, we associate the model with an energy function Ex*) to learn from x(*). The model starts from an initial minimal resolution at stage 1. When the training proceeds to the next stage + 1, we will add randomly initialized layers at the bottom to expand the resolution, and obtain new energy function E(++1) For example, in Figure 2 (a), when the training stage switches from stage 2 to 3, we expand E which is an energy function of resolution 16 x 16, by adding a bottom block accounting for resolution 32 x 32 to get E(3), After that, we only need to train E at stage 3.
Compared with PGAN (Karras al., 2018) which proposes progressive growing strategy train GAN, our CF-EBM integrates both the learning (Eq. (2)) and the sampling (Eq. (3)) into a single neural network Ex(x), which is essentially different from PGAN that applies discriminator to guide the learning and generator to produce samples. Therefore, training an EBM via multistage growing strategy is more challenging and non-trivial. With the newly expanded energy function E and the enlarged training examples x(++1, the stage transition must be taken care of avoid instability issue. The CF-EBM deals with this issue from the following two aspects:
(i) Smooth EBM Learning: Let Expand(-) denote an expanding block. To expand the resolution of the EBM at each stage, we add an Expand(-) to the bottom of the energy function to double its resolution. We fade in the block smoothly to prevent sudden shocks to the well-trained coarse-level energy function. Specifically, Expand(-)) )) is composition of a primal block Primal() (-) and a fading block (.)) We define Expand(x (x) = 3P Merima, 1 (x) + (1 Fade (x), where 3 is a fading factor that gradually increases from 0 to I as more data are consumed in the training. As a result, the model will first rely on the well-trained coarse-level layers and gradually shift the attention to the newly added fine-level layers. The auxiliary fading block will be abandoned when 3 reaches 1. In this paper, Prima (.) consists of two convolution layers and F Fade(-) is a convolution layer followed by 2 x 2 average pooling to link the previous coarse-level layers.
(ii) Smooth MCMC Sampling: The smooth sampling is realized both implicitly and explicilly.
(a) Implicitly: Considering the resolution transition from stage 1 to stage s, we expand the energy ((x="((xppnd(xx sampling in Eq. (3) can be unfolded as:
function by (x(())
The gradient V Ext) (x(*)) for Langevin
Exxpand(x(*) ( (- () (Expand(x!")) x P + (1 - x
Initially, because the fading factor 3 is small, the newly added primal block Primal() (.), whose weights are randomly initialized, would not largely affect the sampling immediately. Only the coarse- level function F(+-1) and the auxiliary fading block Fade(.) make major contributions to the image synthesis. As 3 increases, the Pr Primall (x())((()))))))))))))))))))))( becomes increasingly well-trained. Meanwhile, the term Vx Fade fades away and X Primal 1 (x(+)) gradually takes the lead in the sampling, thus the synthesized images would become sharper and sharper.
(b) Explicitly: To mitigate the impact of the sudden model expansion on sampling, we utilize the well- trained coarse model E(+11) to initialize the MCMC sampling of the newly expanded model E(s), Specifically, at stage s, the algorithm first generates low resolution samples from E(s-1) by running (( - B) Langevin steps. Those samples are then upsampled (2x) by Upsampled e-) and mixed with a uniform noise to initialize the MCMC sampling of E(*), As the fading factor B increases, such an MCMC initialization assistance from the coarse model would fade away. Eventually, when 3 reaches 1, we can directly sample from E(s) with a purely noise- initializeidd MCMC. Figure 2 (b) illustrates the smooth sampling process at stage 3. We use different numbers of Langevin steps at different stages. For example, we only run MCMC with T<) = 15 steps at stage 1, and gradually increase the number of steps as the model grows. The maximal number of steps is 60 in our paper.
After the training, we obtain the target EBM with E(5) The sam- pling procedure by the learned model is presented in Algorithm 2. The MCMC starts from uniform noise distribution and then runs ((S) steps of Langevin updates to generate samples. Note that, both the initial distribution and the number of Langevin steps are the same in both training and sampling. If we run MCMC with more steps than that used at the training stage, the oversaturation phenomenon will occur (Nijkamp et al., 2019).
Algorithm 2: CF-EBM Sampling
Input: model {(S) steps Output: samples x(()
Draw initial x(9) ~ (-1,1)
for tort=1 = 1: ((9) de
Update (55 hy using Eq. (3)
end
5

---

Published as a conference paper at ICLR 2021
3.3 ENERGY-BASER UNSUPERVISED IMAGE-TO- IMAGE TRANSLATION
We then generalize the proposed approach to the task of unpaired image-to-image translation and scale it up to high resolution image datasets. Given two image domains, X and y, endowing two ground-truth distributions p(x): x X and p(y) y E Y the goal is to learn two energy functions Ex(x) X' + R and Ey(y) y - R. and use their short-run Langevin dynamics for image-to-ing -mage translation. Here, we only explain how to learn Ey for one-sided image translation from X' to y since the other side is straightforward. To be specific, we train the target distribution p(y) exp(-Ey(y)) with the following short-run Langevin dynamics starting from the examples from source domain 2'
V4+1 = Yc Th V Ey(r) + VIG & ~ N(O,))), Fo = x ~ Pdua(()),
(4)
which is also translation process. The underlying assumption of applying EBM for image translation is that the two do- mains (X)) share the same ambient space where each data can decomposed into a content code e and domain specific style code xx of domain X' or sy do- main Y' respectively. This assumption is somewhat similar to the partially shared latent space assumption in Huang et al. (2018); Choi et al. (2020); however, it is directly defined in the ambient space and is more efficient without resorting to complementary models. Consequently, as shown in Figure 2(c) and Figure 3, the above dynamic aims to evolve the style code x towards sy, where the gradient descent steps into lower energy area with content e unchanged as much as possible.
Figure 3: Evolution of energy- based image translation
With the proposed method, visualize the image translation dynamics in Figure 3. Three datasets are selected for demonstration because they contain different kinds of generative saliency, e.g., color, shape, and texture. To make the translation process interpretable, we adopt the gradient- gradient-based saliency maps (Simonyan et al., 2014) to highlight the essential aspects that lead the translation, which we call generative saliency map. It computed the aggregated gradients in Eq. (4), i.e., >. V (Fe). J. It quantifies the magnitude of the change of each pixel that contributes to the image translation. Figure displays different types of generative saliency maps.
Generative Saliency Map:
(a) dog
(b) photo Vangogh
orange->
apple
Figure 4: Unsupervised souueeto-immge translations with gradient- brased generative saliency maps. Two examples are displayed for each of the three datasets. Each example a triplet, including an input image, the corresponding saliency map, and its translated image, from left to right.
3.4 DATA PERTURBATION AND ACTIVATION FUNCTION
Most well-established EBMs add Gaussian noise to training data for stabilizing the training (Nijkamp et al., 2019; Grathwohl et al., 2020). Similarly, score-based models (Song & Ermon, 2019; 2020) inject decayed noise to improve the score estimation. However, the additive noise will slightly change the data distribution, thus resulting in foggy synthesized examples (Grathwohl et al., 2020; Song & Ermon, 2020). See Appendix A.5 for more discussions. With the proposed coarse-to-fin expanding and sampling strategy, our EBM doesn't 't resort to data perturbation for stable training.
We further study the effect of the activation function used in the energy function. When the data is continuous, the smoothness of the derivative of the activation function will substantially affect the Langevin sampling process (because the derivative is inside V F(x) in Eq. (3)), therefore applying

---

Published as a conference paper at ICLR 2021
an activation function with continuous gradients everywhere stabilize the sampling. In this regard, we analyze and compare different activation functions, including ReLU (Nair & Hinton, 2010), Lea (Maas et al., 2013), CELU (Barron, 2017), Swish (Ramachandran et al., 2016), and GELU (Hendrycks & Gimpel, 201 6b). We observe that the non-smooth activations like ReLU and LeakyReI.U may cause divergence of learning. See Appendix A.6 for details.
4 EXPERIMENTS
We showcase the results on various tasks, including image generation, image restoration (inpainting and denoising), -iitt---iitritttionn (OOD) detection and unpaired image translation. Most of them are implemented with the same network architecture and run in the TITAN X (12GB) platform. Detailed architectures and experimental setting are provided in Appendixes A. and A.2. We adopt Fréchet Inception Distance (FID) (Heusel et al., 2017) and Kernel Inception Distance (KID) (Bińkowski al., 2018) for quantitative evaluation and Amazon Turker Platform (AMT) for human perceptual evaluation (detailed in Appendix A.3). KID score is more reliable when there are fewer testing images available (e.g. image translation). More synthesis results can be found in Appendix A.4. In Appendix A.6, we conduct detailed ablation study for activation functions, normalization, etc. Finally, we scale our model up to 512 x 512 resolution on the CelebA-HQ dataset in Appendix A.8.
4.1 IMAGE GENERATION
Datasets: (i) CIFAR-10 (Krizhevsky, 2009) is a dataset containing 60k images at 32 x 32 resolution in classes; (ii) CelebA (Liu et al., 2015) a celebrity facial dataset containing over 200k images. For fair comparison with former EBM works, the 64 x 64 resolution is used for quantitative evaluation; (iii) Ce (Karras et al., 2018) contains 30k high resolution (512 x 512) facial images.
Approach
Models
FID
airplane
VAE
VAE (Kingma & Welling. 2014)
78.41
hird
PixcICNN (Van den Oord et al., 2016)
65.93
Autoregressive
PixelION (Ostrovski et al., 2018)
49.46
Car
WGAN-GP (Gulrajani et al., 2017)
36.40
GAN
SN-GAN (Miyato al., 2018)
21.70
StyleGAN2-ADA (Karras et al., 2020)
2,92
Glow (Kingma Dhariwal, 2018)
45.99
doxr
Flow
Residual Flow (Chen al., 2019a)
46.37
Contrastive Flow (Gao 2020)
37.30
dog
MDSM
2020)
30.93
Score-based
NCSN (Song Ermon, 2019)
25.32
frog
NCK-SVGD (Chang al.. 2020)
21.95
Short-run EBM (Nijkamp al., 2019)
44.50
horse
Multi-grid (Guo et al., 2018)
40.01
EBM
EBM (ensemble) (Du & Mordatch, 2019)
38.20
ship
CoopNets (Xie et al., 8b)
33.61
RBM+VAD (Xie et al., 2021d)
39.01
CF-EBM
16.71
bnuc
Table 1: FID on CIFAR-10
Figure Image denoising
Models
FID
VAE (Kingma& & Welling, 2014)
79.09
&
DOGAN (Radford el al., 2016)
32.71
NCSNv2 (Song & Ermon, 2020)t
28.90
Shurt-run EBM (Nijkamp et al., 2019)
23.02
WGAN-GP (Gulrajani al., 2017)
21.40
mr-Langevin (Block et al., 2020)!
19.54
CF-EBM
14.35
CF-EBM²
10.80
Table FID on CelebA
Figure 6: Image inpainting
The quantitative results for unconditional image generation on CIFAR-10 and CelebA are given in Table and Table 2. In Table 2, the result marked with i is experimented
Quantitative Results:
7

---

Published as a conference paper at ICLR 2021
on the center-cropped (140 x 140) and resized images of CelebA whereas others are trained on the resized CelebA. In all cases, the CF-EBM outperforms other methods by a wide margin. In Table 3, compare our method with some representative open-sourced EBMs and the geore-based model in terms of the number of parameters and the relative computational time. All of the methods rely on Langevin dynamics for sampling. We use 50 Langevin steps for sampling while all other three methods require 60, 100 and even more steps, respectively. We observe that our model consistently stands out in terms of memory and computation efficiency while keeping the lowest FID on CIFAR- 10.
Model Analysis: Once the EBM is learned, it can be directly applied to image restoration tasks. As in Du & Mordatch (2019), we conduct two experiments on image restoration to demonstrate the model's abilities of mode coverage and generalization to unseen data. (i) Image denoising: we randomly select images from the testing dataset of CIFAR-10, and then pollute each image with an additive Gaussian noise E ~ 0.2). Figure shows the denoised results x, which are obtained by Langevin dynamics initialized with x c, all classes of CIFAR-10, implying a good generalization ability. (ii) Image inpainting: We mask 25% areas of each image, and use the Langevin dynamics of the learned EBM to recover the missing areas, which can be viewed as associative memory (Hopfield, 1982). As shown in Figure 6, each recovered image exhibits a meaningful but different recovered pattern than the ground truth. We also provide diverse inpainting results in Figure 11 of Appendix A.4, indicating good mode coverage. We also provide nearest neighbors retrieved from training data for the generated images in Figure 12 of Appendix A.4 to show that our model doesn't 't memorize the training images but synthesizes novel ones.
Models
#Params(M)
Time
EBMrensemble) NCSN
18.7
21.3
7.6
14.2
Short-run EBM
2.8
2.2
CF-EBM
2.7
1.0
Table 3: Comparison of efficiency
Qualitative Samples: We have presented qualitative samples of CelebA-HQ 512 x 512 Figure 1. Please refer to Appendix and see Figure 9 for CIFAR-10 10 samples, Figure 16 for how the noise level affects the samples' visual effect in the evaluation and Figures 17, 18 for more high resolution samples.
4.2 OUT-OF- OUT-OF--DSTRIIUTTER DISTRIBUTION DETECTION
diittt-diitiiiutionn (OOD) detection is a binary classification problem. Du & Mordatch (2019) has shown that the likelihoods of EBMs can be useful for OOD detection. The EBM is expected to output a likelihood that is higher for an in-distribution example and lower for an out-of-distributioo example. According to this principle, we conduct an OOD detection experiment and compare our model with the other three generative models, including Glow (Kingma & Dhariwal, 2018), conditional EBM (Du & Mordatch, 2019) and JEM (Grathwohl et al., 2020). All models are trained on CIFAR- 10, which is considered the in-distribution dataset. We use SVHN (Netzer et al., 2011), uniform distribution (Uniform), constant distribution (Constant), interpolations of CIFAR- 10 images (Interp) and CIFAR- 100 (Krizhevsky, 2009) as OOD datasets. We use the Area Under the Receiver Operating Characteristics curve (AUROC) (Hendrycks & Gimpel, 201 6a) as a metric for evaluation. Unlike JEM (Grathwohl et al., 2020), our method doesn't incorporate any label information at the training stage, but it still shows better or competitive results, as shown in Table 4.
Models
SVHN
Uniform
Constant
Interp
CIFAR-100
Average
Glow
.07
1.0
.00
.45
.51
.41
EBM
.63
1.0
.30
.70
.50
.63
JEM
.67
1.0
.65
.67
.70
CF-EBM (Ours)
.65
1.0
.78
.81
.77
Table 4: Comparison of AUROC scores in OOD detection. The higher the score, the better the performance. The other three models are all conditional generative models except for our CF-EBM.
4.3 UNSUPERVISED IMAGE- T-- IMAGE TRANSLATION
We use four unpaired image translation datasets for evaluation, including cat2dog, Yosemite sum- mer2winter, vangogh2photo and apple2orange. All images are resized to 256 x 256 pixels. More details are provided in Appendix 1.9.1. We compare our approach against CycleGAN (Zhu et al., 2017), CUT (Park et al., 2020) and two latest of methods U-GAT-IT (Kim et al., 2020) and FQ-GAN (Zhao al., 2020). CycleGAN is popular GAN-based image translation framework
8

---

Published as a conference paper at ICLR 2021
and use it to demonstrate basic comparison between the GAN-based model with cycle consistency constraint and the proposed energy- energy-based model without cycle consistency constraint. CUT (Park et al., 2020) is the state-of-the- the-art one-sided unpaired image translation framework which applies the contrastive loss to preserve the content. U-GAT-IT and FQ-GAN are current leading models on cat2dog and vangogh2photo. The baseline results are from the published papers checkpoints.
Quantitative Results: Results are shown in Table 5. In most cases, our model obtains better KID scores and human perceptual scores. Meanwhile, the training time and the model size are incredibly small. The efficiencies mainly come from two aspects: model design and optimization process.
(i) Model design: The EBM only consists of image-to-scalar energy function instead of an image- to lo-image mapping as in GAN-based methods. It can implicitly transfer styles and preserve contents without cycle consistency constraint by Langevin dynamics, which is guided by the energy function. Thus, the model size is much smaller than that of GAN-based model which contains pair of GANs.
(ii) Optimization process: GANs use the adversarial loss match styles and the cycle loss to preserve the contents. In practice, the two losses are partially decoupled and optimized alternatively. In contrast, our approach only applies MLE on one neural network (Eq. (2)) and practically, we find that the training takes less iterations to converge. Although we need to run additional Langevin steps for sampling, the overall optimization cost still low.
Models
cul doy AMT
dog KID
cal AMT
photo KID|
Vangugh AMT
Vangugh KID,.
photo AMT1
#Params
#Days
KID
CycleGAN
8.92
1.1
9.94
5.46
22.1
4.68
28.3
28.3M
U-GAT-IT
7.07
16.0
8.15
40.9
4.28
28.7
5.61
10.5
671M
6.0
FQ-GAN
6.44
31.3
8.90
35.3
6.54
15.9
5.21
20.4
671M
6.0
CF-EBM
6.20
51.6
9.21
22.9
4.25
33.3
4.49
40.8
649K
0.6
Table 5: Quantitative results of different methods on different image translation datasets
Qualitative Results: As can be seen in Figure our model can generate sharper images with superior visual quality than the baselines. It better preserves the source content while evolving the style from the source domain the target domain. We put the comparison between our model and one-sided translation model CUT (Park et al., 2020) in Appendix A.9.3. More translation results on photo t> V Vangogh and Yosemite summer <> winter can also be found in Appendix A.9.4.
Inputs
CycleGAN
CUT
Ours
Inpess
PQ-GAN
Ours
Inputs
PO-GAN
Ours
(a) orange
apple
(b) dog
(c) phote - Vangogh
Figure Qualitative comparison of different methods on different image translation datasets.
CONCLUSION
To tackle the training instability and the multimodal MCMC sampling difficulty of the EBMs, we propose multistage coarse-to- to-see expanding and sampling strategy, coined CF-EBM. Extensive experiments demonstrate the superior performance of the proposed framework, compared with various generative models, in terms of sample quality, computation and memory efficiency. The success of the CF-EBM is due to the proposed progressive expanding and sampling strategy, architecture design of energy functions, and the selective smooth activation. To the best of our knowledge, CF-EBM is the first pure EBM that can synthesize high-fidelity images and also be competent in the unsupervised image-to to-image translation. Future work may include investigating the intrinsic mechanism of EBMs, using larger architectures for better performance, and exploring other energy-based applications.

---

Published as a conference paper at ICLR 2021
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems (NeurIPS), pp. 9505-9515, 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Interaational conference on machine learning (ICML), pp. 214-223, 2017.
Adrian Barbu and Song-Chun Zhu. Monte Carlo Methods. Springer, 2020.
Jonathan T Barron. Continuously differentiable exponential linear units. arXiv preprint arXiv:1704.07444 07443,, 2017.
Sagie Benaim and Lior Wolf. One-sided unsupervised domain mapping. In Advances in Neural Information Processing Systems (NIPS), pp. 752-762, 2017.
Mikolaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations (ICLR), 2018.
Adam Block, Youssef Mroueh, Alexander Rakhlin, and Jerret Ross. Fast mixing of multi-scale langevin dynamics underthe manifold hypothesis. arXiv preprint arXiv:2006.11166 2020.
Wei-Cheng Chang, Chun-Liang Li, Youssef Mroueh, and Yiming Yang. Kernel stein generative
modeling. arXiv preprint arXiv:2007.0300 2020.
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Henrt Jacobsen. Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems (NeurIPS), pp. 9916-9926, 2019a.
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised GANs via auxiliary rotation loss. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12154-12163, 2019b.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Woo Ha. StarGAN v2: Diverse image synthesis for multiple domains. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 8188-8197, 2020.
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Mare' Aurelio Ranzato. Residual energy- based models for text generation. In International Conference on Learning Representations (ICLR), 2020.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.85 2014.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems (NeurIPS), pp. 3608-3618, 2019.
Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, and Alexander Rives. Energy-based models for atomic-resolltion protein conformations. In International Conference on Learning Representations (ICLR), 2019.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, Kun Zhang, and Dacheng Tao. seometryy-cnsisteee generative adversarial networks for one-sided unsupervised domain mapping. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp-
2427-2436 2436, 2019.
Ruiqi Gao, Yang Lu, Junpei Zhou, Song- Chun Zhu, and Ying Nian Wu. Learning generative convnets via gulti-grid modeling and sampling. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9155-9164, 2018.
Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy- based models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7518-7528, 2020.
10

---

Published as a conference paper at ICLR 2021
Will Grathwohl, Chan-Chheh Wang, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference Learning Representations (ICLR), 2020.
Jacobsen, David Duvenaud, Mohammad Norouzi,
Ulf Grenander, Michael Miller, Michael Miller, et al. Pattern Theory: From Representation to Inference. Oxford university press, 2007.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein GANs. Advances Neural Information Processing systems (NIPS), pp. 5767-5777, 2017.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 297-304, 2010.
Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Divergence triangle for joint training of generator model, energy-basee model, and inferential model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.
8670-8679, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations (ICLR), 2016a.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415 2016b.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems (NIPS), pp. 6626-6637, 2017.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1771-1800, 2002.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences (PNAS), (954):2554, 1982.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 172-189, 2018.
John Ingraham, Adam Riesselman, Chris Sander, and Debora Marks. Learning protein structure with differentiable simulator. In International Conference on Learning Representations (ICLR), 2018.
Long Jin, Justin Lazarow, and Zhuowen Tu. Introspective classification with convolutional nets. In Advances in Neural Information Processing Systems (NIPS), pp. 823-833, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference Learning Representations (ICLR), 2018.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Junho Kim, Minjae Kim, Hyeonwoo Kang, and Kwang Hee Lec. U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation. In International Conference on Learning Representations (ICLR), 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference Learning Representations (ICLR), 2014.
11

---

Published as a conference paper at ICLR 2021
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions, In Advances in Neural Information Processing Systems (NeurIPS), pp. 10215-10224, 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Justin Lazarow, Long Jin, and Zhuowen Tu. Introspective neural networks for generative modeling. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp- 2774-2783, 2017.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. Predicting Structured Data, 1(0), 2006.
Hsin-Ying Hsin- Ying Lee, Hung- Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsual Yang. Diverse image-to-image translation via disentangled representations. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 35-51, 2018a.
Kwonjoon Lee, Weijian Xu, Fan Fan, and Zhuowen Tu. Wasserstein introspective neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp- 2018b.
Zengyi Li, Yubei Chen, and Friedrich T. Sommer. Annealed denoising score matching: learning energy based model high-dimensional spaces. arXiv preprint arXiv: 1910.07777 2020.
Jun S Liu. Monte Carlo Strategies in Scientific Computing. Springer Science & Business Media, 2008
Ming-Yu Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Advances in Neural Information Processing Systems (NIPS), pp. 700-708, 2017.
Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based out-of-distribution detection. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pp. 3730-3738, 2015.
Andrew I. Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural network acoustic models. In in ICML. Workshop on Deep Learning for Audio, Speech and Language Processing. 2013.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICLR), 2018.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Internatinal Conference on Machine Learning (ICML), 2010.
Radford M. Neal. Annealed importance sampling. Statistics and Computing, 11(2):125-139, 2001.
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, 2(1 1):2, 2011.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Erik Nijkamp, Mitch Hill, Cong-Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run meme toward bnsrgy-based model. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5232-5242, 2019.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier GANs. In International Conference Machine Learning (ICML., pp. 2642-2651, 2017.
12

---

Published as a conference paper at ICLR 2021
Georg Ostrovski. Will Dabney, and Remi Munos. Autoregressive quantile networks for generative modeling. In International Conference on Machine Learning (ICML), pp. (936-3945, 2018.
Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
Taesung Park, Alexei A. Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-to-image translation. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Alec Radford, Luke Metz. and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference Learning Represen- tations (ICIR), 2016.
Prajit Ramachandran, Barret Zoph. and Quoe V Le. Searching for activation functions. In Interna- tional Conference on Learning Representations (ICLR) Workshops, 2016.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In International Conference on Learning Representations (ICLR) Workshops, 2014.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances Neural Information Processing Systems (NeurIPS), pp. 11918-11930, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score- basee generative models.
arXiv preprint arXiv:2006.09011, 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 2818-2826, 2016.
Tijmen Ticleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In International Conference Machine Learning (ICML), pp. 1064-1071, 2008
Ngoc-Trung Tran, Viet-Hung Tran, Bao-Ngoc Nguyen, Linxiao Yang, al. Self-supervised GAN: Analysis and improvement with multi-class minimax game. In Advances in Neural Information Processing Systems (NeurIPS), pp. 13253-13264, 2019.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. In Advances Neural Information Processing Systems (NIPS), pp. 4790-4798, 2016.
Yifan Wang, Federico Perazzi, Brian McWilliams, Alexander Sorkine- -oriinng Olga Sorkine- Hornung, and Christopher Schroers. fully progressive approach to single-image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 864-873, 2018.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In International Conference on Machine Learning (ICML), pp. 681-688, 201
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning (ICML), pp. 2635-2644 2016.
Jianwen Xie, Chhn Zhu, and Ying Nian Wu. Synthesizing dynamic patterns by spatial-temporal generative convnet. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7093-7101, 2017.
Jianwen Xie, Yang Lu, Ruiqi Gao, and Ying Nian Wu. Cooperative learning of energy-based model and latent variable model via meme teaching. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 32, 2018a.
Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chur Chun Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 42(1):27-455 2018b.
13

---

Published as a conference paper at ICLR 2021
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying Nian Wu. Learning descriptor networks for 3D shape synthesis and analysis. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 8629- 86688, 2018c.
Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu. Learning energy- spatial-temporal generative convnets for dynamic patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019.
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Chun Zhu, and Ying Nian Wu. Generative VoxelNet: learning energy-based models for 3D shape synthesis and analysis. IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 2020.
Jianwen Xie, Yifei Xu, Zilong Zheng, Song-Chur Zhu, and Ying Nian Wu. Generative PointNet: energy-based learning on unordered point sets for 3D generation, reconstruction and classification. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2021a.
Jianwen Xie, Zilong Zheng, Xiaolin Fang. Song-Chun Zhu, and Ying Nian Wu. Cooperative training of fast thinking initializer and slow thinking solver for conditional learning. IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 2021
Jianwen Xie, Zilong Zheng, Xiaolin Fang. Song-Chun Zhu, and Ying Nian Wu. Learning cycle- consistent cooperative networks via alternating meme teaching for unsupervised cross-domain translation. In Proceedings of The Thirty- Fifth AAAI Conference on Artificial Intelligence (AAAI),
2021c.
Jianwen Xie, Zilong Zheng, and Ping Li. Learning energy-based model with variational auto-encoder as amortized sampler. In The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI), 2021d.
Yifci Xu, Jianwen Xic, Tianyang Zhao, Chris Baker, Yibiao Zhao, and Ying Nian Wu. Energy-based continuous inverse optimal control. arXiv preprint 100v:5104..05433, 2019.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.033: 03365, 2015.
Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee, Consistency regularization for generative adversarial networks. In International Conference on Learning Representations (ICLR), 2020.
Yang Zhao, Chunyuan Li, Ping Yu, Jianfeng Gao, and Changyou Chen. Feature quantization improves gan training. In International Conference Machine Learning (ICML), pp. 11376-11366 2020.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2223-2232, 2017.
Song-Chun Cong-Chun&en Zhu, Yingnian Wu, and David Mumford. Filters, random fields and maximum entropy (FRAME): Towards unified theory for texture modeling. International Journal of Computer Vision (IJCV,, 1772)1 1998.
14

---

Published as a conference paper at ICLR 2021
A
APPENDIX
A.I ARCHIITECTURE DESIGN
The neural network architecture of the energy function for the model at 256 256 resolution given in Table 6. Each expanding block Expand(() consists of a primal block Pr primal()) (-) and fading block Fade(-). The Primal(-) consists of convolutional layers. where the first layer does not change the spatial size whereas the second one comes with a stride 2 for down-sampling. The kernel size 3 3. The top Primal(:) (without down- (awn--ampling) uses kernel size and kernel size 4 X 4 for the second layer. We also have residual connection in P Primal(( (-), which a single convolutional layer followed by an average pooling of size x We don't apply the residual connection in the last Pr imal(-) block. FromRGB(-) is special block that transforms 3-channel RGB image to a z-channel feature map, where 2 is number of channels of each kernel at the bottom convolutional layer, in order connect the input image the bottom layer of the energy function. We illustrate the expansion process from resolution 16 x 16 to 32 x 32 in Figure 8.
for the first layer
Block Type
Output shape
Flatten & Linear
1
Primal
4c x 1
Primal
4e x 4
Primal
4c x 8
Primal
4c x 16 x 16
Primal
4c x 32 x 32
Primal
2c x 64 X 64
Primal
128 x 128
FromRGB
c/2 256 256
RGB image DR3x256x266
256 256
Table 6: The neural network architecture for energy function at resolution 256 256. i the channel multiplier and we set c = 32 for all experiments.
E(x)
E(x)
Primal
Primal
8x8
8x8
Primal
Expansion
Primal
16x16
16x16
FromRGB
Fade
Primal
32x32
FromRGB
Figure 8: Illustration of the expansion from resolution 16 16 to 32 32. We first remove the FromRGB block at the bottom of the energy function at resolution 16 and then add an expanding block, which consists of Primal block and a Fading block, and a new FromRGB block, which is used to connect the input RGB image the newly added Primal block.
15

---

Published as a conference paper at ICLR 2021
A.2 EXPERIMENTAL SETTING
We train the network using Adam (Kingma & Ba, 2014) optimizer with 31 = 0.5 and 8g = 0.999. We set the learning rate schedule
{8 x 8 0.001, 16 x 16: 0.001, x 32 0.001, 64 x 64
2.0012, 128 x 128 0.0015) and the data feeding schedule as N = {8 x 8 50k, 16 x 16 75k, 32 32: : 100k, 64 64 125k, 128 128 150k). J. As to the sampling hyperparameters, we set the schedule of the number of Langevin steps T = {8 x 8 15, 16 x 16 : 30, 32 x 32 50, 64 x 64
50, 128 x 128 60), the Langevin step size as 1.0, and the variance of the Langevin noise term as Th 2e 2e 3 2/(T i + 1) in most experiments.
We preprocess CelebA to produce CelebA-64 by using two different settings. (1) Each image is firstly center-cropped into 140 x 140 and then resized to 64 x 64. In Table 2, NCSN (Song & Ermon, 2019), ma-gevnnvin (Block et al., 2020), and WGAN-GP (Lee et al., 2018b) use this setting. The FID computation protocol is based on Song & Ermon (2019), where the FID score is calculated between 10k generated images and all test images. (2) Each image is directly resized to the resolution 64 x 64. The FID is computed on 40k generated samples. In Table 2, VAE, DCGAN, and short-run EBM follow this setting.
A.3 EVALUATION METRICS
We consider three commonly used measures for evaluating the quality of the synthesized images.
(i) Fréchet Inception Distance (FID) (Heusel et al., 2017) compares the distribution of generated images with the distribution of training images. Instead of directly comparing images, the FID compares the mean and standard deviation of one of the deeper layers in Inception v3 (Szegedy et al., 2016), which is a convolutional neural network trained for classification. The FID is consistent with human perceptual judgement. Lower FID scores indicate that the model generate images with higher quality. We follow the dataset split protocol in Zhang et al. (2020) to compute the FID.
(ii) Kernel Inception Distance (KID) (Bińkowski et al., 2018) is an unbiased metric that computes the squared maximum mean discrepancy (MMD) between Inception representations, making it more reliable than FID especially when the number of test images small. For evaluating the image translation performance, we compute KID between the translated images from test images in the source domain and the test images the target domain. Lower KID scores indicate the better translation results. We follow Kim et al. (2020) to compute KID.
(it) For human perceptual evaluation using Amazon Turker Platform, we follow the setting in Zhao et al. (2020). Each testing image is judged by six participants, who are asked to select the best translated images by taking image visual quality and source content preservation into consideration. The participants are informed the information of the target domain, e,g., six examples of images in the target domain are shown to them for reference.
A.4 MORE IMAGE SYNTHESIS RESULTS
Figure shows the generated images of the EBM learned from the CIFAR-10 10 and ImageNet- Ik x pixels) datasets. Table 7 presents the quantitative results of image generation on ImageNet- lk. The proposed CF-EBM achieves competitive results terms of FID with much less parameters.
In Figure 10, we visualize the short-run Langevin dynamics initialized from the uniform noise distribution for generating CelebA images. Figure 11 illustrates the results of image inpainting on test images of CelebA at 64 x 64 resolution. Running Langevin dynamics on the occluded images, we can observe diversity of image completion. It demonstrates that our model generalizes well to unseen test data and the MCMC of the learned model has good property of mode coverage. Figure 12 shows the generated samples and their nearest neighbors retrieved from the training data. We find that the synthesized images not identical to the training data, which means that our model learns to generate new image patterns instead of reconstructing the existing training images.
A.5 THE EFFECT OF DATA PERTURBATION
Almost all energy-basee based related generative models add Gaussian noise [/ I to perturb the training data for stabilizing the training. We investigate the effect of such data perturbation. Nijkamp
16

---

Published as a conference paper at ICLR 2021
(d)
Figure 9: Qualitative results of image synthesis. (a) ground truth images of CIFAR-IO. (b) generated images of CIFAR-I0. (c) ground truth images of ImageNet-IK (32 x 32 pixels). (d) generated images of ImageNet- Ik (32 x 32 pixels).
Approach
Models
FID
#Params
PixeICNN (Van den Oord et al., 2016)
40.51
35M
PixelIQN-small
37.62
>35M
Unconditional
PixelIQN-big
26.56
500
CF-EBM (ours)
26.31
5.82M
PixeICNN
33.27
Conditional
PixelIQN (Ostrovski al., 2018)
22.99
EBM (Du & Mordatch, 2019)
14.31
SS-GAN (Chen et al., 2019b)
17.10
Self-supervision
MS-GAN (Tran et al., 2019)
12.30
Table 7: FID on ImageNet-Ik (32 x 32). The channel multiplier is ch 48.
Figure 10: Illustration of short-run MCMC generative sequences on CelebA (50 Langevin steps)
17

---

Published as a conference paper at ICLR 2021
Figure 11: Illustration of diverse image inpainting. The top row shows the occluded face images and the bottom row shows the original images for reference. Other rows show different inpainting results. Each column corresponds to one example of image inpainting. Our model can complete the occluded areas with different meaningful facial expressions.
Figure 12: Nearest neighbors of synthesized images on CelebA (64 x 64 pixels). Generated samples displaced in the left column whereas their top 10 nearest neighbors retrieved from the training set are shown the other columns.
18

---

Published as a conference paper at ICLR 2021
et al. (2019) have already reported that larger perturbation would lead to lower FID score, and the minimum standard deviation of the additive noise they use is o = 0.03. Also, Song & Ermon (2019) apply perturbation with decayed standard deviation from the EBM learned form the training data with additive noise will memorize the noise information even if they very small. As result, the synthesized images will look noisy and foggy. Figure 13 shows the effect of the data perturbation by illustrating the synthesized images generated by the models learned with different levels of data perturbations. Note that, even if the standard deviation of the additive noise is as small as = 0.01, the effects reflected in the synthesized images are obvious. The same observation can be found from the synthesized images shown Figure 2 of Nijkamp et al. (2019), Figure of Song & Ermon (2020) and Figure 2 of Grathwohl et al. (2020), all of which use data perturbation during training. Our model doesn't rely on the additive noise during training, thus leading to better synthesis quality.
1.0 to 0.01. We claim that
shutters ck
shutters tktck
shutters ckick
shutter tAtck
a=
0.01
σ = 0.02
0.03
Comparison of different levels of data perturbations, with red rectangle areas zoomed in. Images are from LSUN ( (Yu et al., 2015) church at 128 x 128 resolution.
Figure 13:
A.6 ABLATION STUDY
We mainly examine the activation functions, normalization and layer connections.
A.6.1 ACTIVATION FUNCTIONS
We compare the following activation functions by presenting their formulas and derivatives:
Rectified Linear Unit (ReLU) (Nair & Hinton, 2010): f(x) max(0, ær), and the deriva-
tive:
0f(x)
1
if > otherwise
(5)
Ox
0
Leaky Rectified Linear Unit (LeakyRel.U) (Maas et al., 2013): f(x) = max(0, (0, + a
min(0, .)) and the derivative:
0f(x)
{
if 0 a otherwise
1
(6)
where 0.2 is the common setting.
19

---

Published as a conference paper at ICLR 2021
Continuously Differential Exponential Linear Unit (CELU) (Barron, 2017): f(z) max(0, z) + min(0, a (exp(x/a) - 1), where a a shape parameter and the derivative is:
af(=) -
{ { a- (exp(z/a) -
if otherwise
Swish (Ramachandran et al.. 2016): f(z) = x o where a(-) is Sigmoid function and 3 is learnable parameter. The derivative is:
0f(x) Bx ox()) + (Bx) - Br o(32))
(8)
d
We set B in this paper.
Gaussian Error Linear Unit (GELU) (Hendrycks Gimpel, 2016b): f(x) (ar) where (-)) is the cumulative distribution function for Gaussian distribution. It can be
approximated by f()) = 0.5x(1 + tanh( (//nh/////(/////// + 0.044715z"))..
Figure 14 visualizes the above activation functions and their derivatives. Obviously, both ReLU and LeakyRel.U are non-smooth around 0. Quantitative results of image synthesis with different activation functions CIFAR-10 are shown in Table 8. We find that models using LeakyReLU or ReLU activation function diverge early. These phenomenons are also observed in the CelebA experiment. Therefore, smoothness of the activation improve the training stability and synthesis quality EBMs.
Activation function
FID
ReLU LeakyReLU
N/A
78.21
GELU
28.71
CELU (a -a==] 1)
21.50
Swish
16.71
Table Effects of different activation functions
Activations
Derivatives
LeakyReLU
1.0
LeakyReLU
CELU
CELU
Swish
0.8
Swish
0.8
0.4
Figure 14: Visualization of different activation functions and their derivatives
A.6.2 NORMALIZATION
Since the objective in Eq. (2) is similar to that in Wasserstein GAN (Arjovsky et al., 2017), and the bottom-up energy function in the EBM acts as a discriminator, we thus naturally consider spectral normalization (Miyato et al., 2018) to improve the performance and stability. The spectral normalization constrains the Lipschitz constant of the learnable neural network parameters, which is widely used to stabilize the training of the discriminator network of GAN.
20

---

Published as a conference paper at ICLR 2021
The batch normalization is not appropriate here, because each Langevin step relies the running mean/variance of x ~ Pasta (x)s(x).. Even if the statistics are not updated in Eq. (3), the distribution of
Po(x;) keeps changing all the time along chain. As result, the pre-computed statistics of the batch normalization are not suitable for X₁ at different step L. Other normalizations, e-g., instance normalization and layer normalization, not appropriate experimentally.
A.6.3 ANALYSIS OF DIFFERENT COMPONENTS
We first examine how each component affects the model performance. They include progressive growing, residual connection, and spectral normalization. Results are shown in Table 9. From (a)-(c), find that both spectral normalization and residual connection can improve the image synthesis quality. In (d), we disable the coarse-to-fine training but still keep residual connection and spectral normalization, the performance drops. Besides, we also compare the time efficiency regarding the training with and without the coarse-to-fing strategy. It is observed that the model without using cearse-to-fine learning requires approximately 3x more time than the counterpart to converge. (e), we sequentially train multiple EBMs at different resolutions in coarse-to- fine manner. We try to fix the coarse EBMs when sequentially training model at higher resolution, and use the coarse EBM to initialize the sampling of the fine EBM. We keep all EBMs after training. We find that the FID of (e) worse than that of learning single fine EBM as (c).
Configuration
FID
(a) Basic CF-EBM
32.01
(b) + Spectral normalization
20.83
(c) + Residual connection
16.71
(d) Coarse-to- to-see
24.67
(e) Fix coarse EBM
21.25
Table 9: Ablation study on CIFARIO.
A.7 LIKELIHOOD EVALUATION
We conduct an experiment on the continuous MNIST for likelihood evaluation by following the setting in Du & Mordatch (2019). Its open source implementation is at https / /g thub. com/ openai/ / abm__oode ltt___odddeeeeeeeeeeeeeeee release. It applics the Annealed Importance Sampling (AIS) (Neal, 2001) to obtain a bound of the partition function. Figure 15 illustrates comparison of the generated examples on MNIST between our model and the one in Du & Mordatch (2019). As can be seen, our model produce more realistic digits. Table 10 shows comparison of the log-likelihood among three likelihood- models.
6
4
6
0
7
3
9
8
8
0
2
9
1
1
2
7
4
4
6
4
0 7 $ 4
7 8
2 6
0
6
8
2
2
6
4
9
0
9
4
9
78615811
8
6
0
/
6 4 2
8
0
0
5
5
6
5
9
0
8
3
2
04022114 6
0
2
4
3 5
J 5
1 4
7
7
7
7
6 3
6
9
1
6
5
3
8 6
3 9 0
3
2
Figure 15: Image synthesis on MNIST. Left: model. Right: EBM in Du & Mordatch (2019)
21

---

Published as a conference paper at ICLR 2021
Models
Lower Bound
Upper Bound
NICE (Dinh et al., 2014)
1980.0
1980.0
EBM (Du & Mordatch, 2019)
1925.0
2218.3
CF-EBM (Ours)
2197.7
2342.0
Table 10: Log-likelihood in Nats on Continuous MNIST.
A.8 SCALING UP THE CF-EBMS FOR HIGI RESOLUTION SYNTHESIS
To scale up EBMs for sampling high resolution images, we integrate intermediate CF-EBMs that have been learned in Algorithm 1. When the training enters into modeling higher resolution, fix the coarse-level CF-EBM, which only produces initial points for fine-level CF-EBM samplers, In this scenario, the number of Langevin steps can be reduced 15 expedite sampling at each scale, meaning that generating a 256 x 256 images requires 90 steps. On single TITAN X GPU (12GB), the total training time (200k iterations) of CelebA-HE HQ 128 128 and 256 x 256 are about 120 hours and 235 hours, respectively. On single TITAN V V01 GPU, these costs can be reduced to 55 hours and 100 hours (we tested once).
In this experiment, we fix the initial variance do = 0.03 for the noise term of Langevin dynamics in (3) during training. We observe that, at the testing stage, choosing a different value than that used for training will largely affect the synthetic results. As shown in Figure 16, larger no for the noise term tends to make the synthesized images not only more realistic but also more noisy. Figure 17 shows more generated samples
CelebA-HQ 256 256 resolution. In this experiment, the channel multiplier& e = 32 and train it for 5 days on a single GPU. Figure 18 shows generated samples on CelebA-HQ at 512 512 resolution where set the channel multiplier e = 48 and the model is trained for 10 days on a single GPU. Table 11 compares the FID scores of different models on CelebA-HO at 128 x 128 resolution. Our model obtains competitive performance without using any regularization term.
set
Models
FID
Glow (Kingma & Dhariwal, 2018) 8)
68.93
SN-GAN (Miyato et al., 2018)
25.95
WGAN-GP (Gulrajani et al., 2017)
22.57
CR-GAN (Zhang al., 2020)
16.97
CF-EBM (ours)
23.50
Table 11: FID for image generation on CelebA-HQ (128 128). Results of GAN-based approaches are taken from Zhang et al. (2020).
A.9 UNPAIRED IMAGE-TO-IMAGE TRANSLATION
A.9.1 DATASET
In this paper, we experiment on five popular image-to-image translation datasets: (i) selfie2anime This dataset is first introduced in Kim et al. (2020), and contains a selfie domain and an anime domain, each of which has 3400 training images and 100 testing images. (ii) photo2vangogh This dataset is used in Zhu et al. (2017) for collection style transfer. It has 6,287 photo images and 400 Van Gogh painting images in the training set and 751 photo images and 400 Van Gogh painting images in the test set. (iii) apple2orange This dataset is used in Zhu et al. (2017) for object transfiguration. It has 995 training images and 266 test images for apple category and 1019 training images and 248 test images for orange category. (iv) Yosemite summer2winter This dataset is used in Zhu et al. (2017) for season transfer. The summer category has 1273 training images and 309 test images, while the winter category has 854 training images and 238 test images. (v) cat2dog This dataset is used in DRIT (Lee et al., 201 8a). The numbers of images for cat and dog are 871 and 1,364, respectively. Following Kim et al. (2020), we use 100 cat images and 100 dog images as test data.
22

---

Published as a conference paper at ICLR 2021
A.9.2 EXPERIMENTAL SETTING
For the unpaired image-to-imaae translation task, we only made changes compared with the setting used in image generation: (i) the channel multiplier is c 16; (ii) the learning rate is multiplied by 10 only for dog -> cat.
A.9.3 COMPARISON WITH CUT
We implement CUT based on its open-source code contrase ve-unpa red-trrere transtannsl at ion... We then select the most realistic translated results by CUT eat dog. In Figure 19, compare them with the proposed CF-EBM. As be seen, CUT sometimes hallucinates a tongue, resulting in unfaithful translation results. This observation is also mentioned in the original paper (Park et al., 2020). In contrast, our CF-EBM produces more faithful translation results.
com/ htmp/aassugg////p
A.9.4 MORE QUALITATIVE RESULTS ON UNPAIRED IMAGE-TO-IMANE TRANSLATION
See Figures 20 ,221, 22, 23, 24, 25, 26, and 27 for more results of image-to-image translation.
0.
0.01
0.02
0.025
0.03
Figure 16: Effect changing the initial variance тo for noise term e of the Langevin dynamics at the testing stage
23

---

Published as a conference paper at ICLR 2021
2
29
6
Figure 17: Generated examples on CelebA-HQ at 256 x 256 resolution, with a linear decay of me for e from 0.025 to O.
24

---

Published as a conference paper at ICLR 2021
IZ
O
Figure 18: Generated examples CelebA-HQ at 512 x 512 resolution, with a linear decay of for the noise term c from 0.015 to 0.
Input
CUT
CF-EBM
Figure 19: Qualitative comparison between our CF-EBM and CUT.
25

---

Published as a conference paper at ICLR 2021
Figure 20: Translation dynamics of orange - apple
Figure 21: Translation dynamics of cat + dog
26

---

Published as a conference paper at ICLR 2021
Figure 22: Translation dynamics of photo - vangogh.
Figure 23: Translation dynamics of sel selfie fie anime
27

---

Published as a conference paper at ICLR 2021
Figure 24: Translation results on photo - V Vangogh
Figure 25: Translation results on Vangogh + photo
28

---

Published as a conference paper at ICLR 2021
Figure 26: Translation results on Yosemite summer + winter
Figure 27: Translation results on Yosemite winter summer
29


Reviews:
[{"review_id": "Pn7R05EXRC3", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper presents a number of methods to scale up training and sampling of EBMs on image data. The main contribution consists of an approach for progressively growing the model by increasing the image resolution as training progresses. This approach echos similar approaches used for scaling up GAN training. The approach involves slowly annealing in new blocks to the model during training which processes the image at increasing resolutions. This allows training of image EBMs at notably larger resolutions than published in prior work.\n\nThe authors also present a few additional architectural improvements such as using smooth nonlinearities over the currently-popular leaky ReLU. They find that this change allows us to remove the commonly-used Gaussian noise added to the input, improving sample quality. The authors demonstrate the performance of their approach, focusing on image generation. They present results with FID and compare to other generative models and EBMs. They also examine the quality of the learned energy functions with inpainting and denoising experiments. \n\nBeyond unconditional image generation, the authors also propose a simple method for image-translation based on EBMs. In this approach, unconditional EBMs are trained for each domain and an image is translated by taking a sample from 1 domain and running a Langevin MCMC sampler in the target domain EBM starting from the sample from the source domain. The resulting image retains many of the high level characteristics of the source image but adds low level characteristics from the target domain. Surprisingly this simple approach outperforms more involved methods based on adversarial training. \n\nStrong areas:\n\nThe empirical results in this paper are impressive, made more impressive by the simple nature of the approach. It appears that the proposed method gives a notable improvement in image generation quality over prior EBM models while also giving an improvement in run-time and parameter-efficiency. Particularly interesting are the very strong results in the image-translation task. I am not familiar with this space so I cannot say how much of an improvement is reported here over the adversarial methods, but it is quite shocking to me that this very simple approach would outperform two recent state-of-the-art methods for this task while also having no cycle consistency objective. This should certainly bring attention to EBMs for this task.\n\nWeaknesses:\n\nI felt that many of the experimental details in this paper could have been made more clear. I found myself having to re-read the paper to find out that the authors used short-run MCMC instead of PCD. As these training procedures are quite different, this should be made more clear.\n\nI also found the proposed architecture hard to understand. I could not tell if the full energy function continues to take in lower-resolution versions of the image as we add levels or if those inputs are replaced by the outputs of the added higher-resolution layers. The figures are not very clear on this. When sampling high-resolution images, do we first generate lower resolution samples, and use these to seed the high-resolution samples? Figure 2a makes it look like this is what is happening. But then algorithm 1 makes it seem like we generate samples directly from noise for the current training scale. This is important for understanding the method and it could be made much more clear. I would recommend the authors split apart algorithm 1 into an algorithm for training and one for sampling. I think that would make the method much more clear. \n\nWhile the experimental results were strong, I am curious why the authors did not present any results on out-of-distribution detection (aka anomaly detection). These are common nowadays and I am curious as to how this approach will perform given its improved scalability over many EBM models which we find perform very well at this task. \n\n\nMy recommendation:\n\nThis is an interesting work that presents a simple method that allows image EBMs to scale up considerably. The proposed approach notably improves image generation and allows models to be trained on much larger images than has been possible by EBMs in the past. Further, the image-translation application is intriguing and the strong results should make EBMs a standard baseline for that application. \n\nBecause of these strong empirical results, I am recommending to accept this paper, but I am not advocating for a strong-accept.  While the results are strong, the work does not improve our understanding of these models, or introduce any particularly novel techniques. \n\nI think the contributions of this work could be made greater with some experiments on some other applications such as OOD. I also think the work could be greatly improved by some rewrites to section 3.2. This is the main contribution of the paper and the authors should take more care to ensure that their effort is easily understood. \n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8V1eV1lgcz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a method to improve generation with energy based models. The work further shows how energy based models can also be extended to cross class image translation.\n\nStrong Points:\nGenerated samples appear to look good\n\nWeak Points:\nMy most major concern is that the overall technical novelty of the work is rather limited, and I do not think passes the bar of acceptance for ICLR. Past work [2] has shown that EBMs can composed with multi-scale sampling, while [1] notes that the smooth activation function Swish significantly improves EBM generation. Similarly, cross domain generation with EBMs have been previously demonstrated such as in [1]. While [2] uses a separate hierarchy of multi-scale EBMs, this work uses the sampling scheme in [3].\n\nMy second most major concern is other the theoretical framing of the proposed approach and well is issues with statements made in the paper. W\n\nThe proposed sampler linearly decays the noise schedule to 0, but what are the theoretical implication of such a behavior? For Langevin sampling to valid, the step size of both the model and the noise must decay to 0.\n\nThe proposed approach generates samples by running a fixed number of Langevin steps to generate a sample. But also the approach claims to train a maximum likelihood objective, where Langevin steps are used to approximate the energy landscape. A good energy landscape should be able to support an arbitrary number of MCMC sampling steps. What happens when a larger of number of steps of Langevin is run?\n\nThe paper states that past approaches such as [1] have relied on adding white noise to samples. I do not think this is the case. Can the authors elaborate?\n\nMy next major concern is that overall text of the paper is somewhat confusing to read, with some awkward sentences in the text. For example:\n\nSection 3.2:\nAssuming the total number of stages is S and the training starts from the min => Given a total of S stages of training \nIn what follows, be aware of the only difference => The only difference between\nIn this way, both stability and time efficiency in training EBMs are benefited. => In this way, both stability and time efficiency in training EBMs benefit.\n\nSection 3.3:\nWe name the style that can differentiate different domains discriminative saliency. => We call this approach discriminative saliency.\n\nConclusion:\nWe owe the success to the newly proposed network architecture, smoothing activations, the powerful interpretability of EBM =>\nThis success is due to our newly proposed architecture, smoothing activations, and the flexibility of EBM.\n\nTo improve the clarity of the text, it would be good to perhaps add an algorithm block on the precise sampling procedure used to train EBMs. For example, the term short-run samples was never defined and I had to look at referenced papers to figure out the meaning.\n\nThe related work seems to incomplete and misses a lot of recent work towards training EBMs. Some examples of additional approaches towards training EBMs including score matching and minimum probability flow learning.\n\nThere are also some issues with the empirical evaluation of the proposed approach:\n\nTo showcase mode coverage of the EBM, it's important to show diverse inpaintings, as opposed to a single inpainting of an image.\n\nFurthermore, as deep likelihood based model it is important to show some density/likelihood evaluation -- using some variant of AIS.\n\nMinor Comments:\n\nWhy is the proposed approach much more efficient than past approaches for image to image translation? It seems Langevin sampling steps would take more computational time.\n\n \n[1] Yilun Du and Igor Mordatch, Implicit Generation and Generalization with EBMs\n\n[2] Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Wu. Learning generative convnets\nvia multi-grid modeling and sampling. \n\n[3] Tero Karras et al. Progressive Growing of GANs for Improved Quality, Stability, and Variation\n\n\n===== Post Rebuttal Updates\n\nI thank the authors for responding to my comments. The work shows improved generative performance by using a multi-scale architecture, however the approach is the same that used in previous GAN works. Furthermore, the generation quality is not as good those of recent GAN works and I don't believe the added EBM benefits are significant.  In addition, other contributions, such as the use of the Swish activation and domain transfer has also been noted as used in previous work [1]. I also have additional empirical concerns over the experiments.\n\nI list additional comments below:\n\n1) The diverse inpaintings (Figure 11) do not really look very diverse to me and seems to suggest that the model is not learning a likelihood. To evaluate diverse inpaintings, it would be good to follow past work and evaluated on ImageNet images where only the top half of an image is conditioned. [3]\n\n2) The likelihood evaluation are hard to interpret in A.5. An issue with evaluating AIS based likelihood sampling on MNIST for upper and lower bounds of likelihood depends heavily on the large number of steps of sampling required.  Upper and lower bounds depend heavily on the number of steps of sampling run (with unrealistically high likelihoods obtained when running only a few steps of AIS). It seems unlikely to me that the proposed model obtains a significant boost to log-likelihood compared to past approaches, and it would be good to report both the number of AIS transition distribution (and ensure that it is same used in [1]). In particular, I believe that this approach is likely to perform poorly with a large number of gradient steps (as the rebuttal response noted), which is required for proper evaluation of likelihood. \n\n3) The related work is still missing older work in the area of EBM training. Instead of adding additional references to recent work on score based generative modeling, I think the others should cite past works that have used score matching to training energy models. For examples, such works include [4, 5, 6].  \n\n4) I didn't find the out-of-distribution results to be a particularly compelling application of the model (although its good that it performs similarly to past approaches). The only results that appear to be better are uniform (which in my experience performance across all models fluctuates) and interpolation. Furthermore, the model from [1] used in section 4.2 is not conditional. \n\n5) I wouldn't say this paper is the first to generate 512x512 samples with EBMs. For example see [3].\n\n6) It's difficult to evaluate an open source code release, since the code is not provided at the time.\n\n7) Regarding the approach in [1], when doing source translation images are initialized from ground truth images from a seperate class.\n\n\n[1] Yilun Du and Igor Mordatch, Implicit Generation and Generalization with EBMs\n\n[2] Aaron van den Oord, Nal Kalchbrenner, Koray Kavukcuoglu, Pixel Recurrent Neural Networks\n\n[3] Tian Han et al. Divergence Triangle for Joint Training of Generator Model, Energy-based Model, and Inferential Model\n\n[4] Jascha Sohl-Dickstein et al. Minimum Probability Flow Learning \n\n[5] Saeed Saremi. Deep Energy Estimator Networks\n\n[6] Hyvärinen, Aapo. Estimation of non-normalized statistical models by score matching.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8HzDPuzm9e", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an energy-based generative model for image-to-image generation that differs from previous methods by incorporating a progressive learning scheme that gradually increases the resolution of images being trained on.\nStrengths:\n-- The presence of a saliency map in image-to-image translation is a benefit that is harder to get out of other methods.\n-- The ability to infill missing parts of an image with the same model that does unsupervised translation is a nice benefit.\n-- The smoothing along resolutions during training seems like a good way to incorporate the progressive training technique into this model.\nWeaknesses:\n-- The novelty here is very limited. The difference between this and previous energy-based models is the scheme of progressively generating at 8x8, 16x16, 32x32, etc... which itself is a well-established technique from the Progressive GAN. Given neither of these things is new, the novelty lies in just using the one with the other.\n-- The authors claim the saliency map is a benefit of their method but offer no comparisons with other models, e.g. the marginal gradient methods for any CNN model including CycleGANs.\n-- The quantitative scores for the GAN methods are significantly worse than models can actually achieve now. The Big-GAN, which is several years old now and not even SOTA on CIFAR has a better score than the one proposed here.\n-- The qualitative results from the GANs are much poorer than can actually be achieved. For example, the orange->apple not changing shape is a result of a poor architecture with too high of a cycle-consistency coefficient.\n-- The characterization of GANs in the previous work section is very poor. The authors claim, without evidence or citation, that the invertibility of the two directions in a CycleGAN \"may intensify long-standing instability issues\" when it very likely does the *opposite*. They also criticize GANs for requiring an \"elegant design\" like instance normalization, when this is a standard and easy-to-implement part and is no more problematic than any aspects of the authors' proposed framework. Also masquerading as a \"critique\" is that GANs have a \"fancy architecture\": I don't think the authors could devise a definition of fancy that excludes their model. If this section is designed to motivate their energy-based approach, it can be done without using arbitrary, subjective, inaccurate insults of other methods.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "89rVHsl6FIf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Much like progressive growing of GANs two years ago, this paper adopts a similar coarse-to-fine procedure for scaling EBMs to higher resolutions. In particular, the approach starts from learning EBMs on low-resolution images and then smoothly transitions to higher resolution by carefully designing an expand layer and a smooth sampling procedure. Authors were able to obtain competitive FID scores on CIFAR-10, and demonstrate the first set of 256x256 image samples from EBMs. In addition, authors demonstrate successful application of EBMs to unpaired image-to-image translation.\n\n#### Pros\n* Strong experimental results. The FID scores obtained are among the best achievable results with maximum likelihood training. Scaling to 256x256 images is a great advancement of the field. Various experiments on denoising and inpainting demonstrates the versatile applications of EBMs.\n\n* Applications in unpaired image-to-image translation demonstrate strong potential of the approach in image editing\n\n* Sampling typically takes 50 Langevin steps. This is much faster than denoising score matching with Langevin dynamics or denoising diffusion probabilistic models, which typically need thousands of Langevin steps.\n\n#### Cons\n\n* FID computation is done using a PyTorch implementation. Although the results should be close to the original FID implementation based on TensorFlow, they are not exactly comparable to previous results. The numbers in tables are therefore not rigorous.\n\n* There is no widely agreed protocol on how to compute FIDs on CelebA 64. The processing methods can be quite different in different papers, and the number of samples used for FID computation can also be different. Please include your settings explicitly in the paper, otherwise the comparison in the table is not that meaningful.\n\n* The progressive growing procedure requires multiple stages of separate training and also need careful tuning of model architecture and sampling to make transition smooth.\n\n* There is a factual error in Appendix A.3.1.The work of Song & Ermon (2020) does not apply a sequence of decayed perturbation from 1 to 0.01. Quite the reverse, starting from noise scales different from 1 is one main point in that paper.\n\n-------------\nPost-rebuttal\n\nReporting PyTorch-computed FIDs risk the fairness when comparing against previous work. The repo quoted by the authors had a well-known issue, leading to much lower numbers compared to those computed by the original TensorFlow repo. Although this issue has been alleviated following some code update this year, the numbers still won't exactly match those of TensorFlow. Therefore, from a scientific perspective, the FID numbers have to be recomputed with the original TensorFlow code for a rigorous publication.\n\nIn addition, the FID number reported by the authors for their model is computed between 10k samples and the test dataset, while most other FID numbers in Table 1 are computed between 50k samples and the training set (following the original settings of [1]). This also makes the comparison not fair.\n\nI completely agree that FID is a flawed metric and lower FID scores do not necessarily indicate better sample quality. However, since the authors choose to report FIDs and compare against those directly ported from previous work, they have to follow the convention and ensure a fair comparison. Since I didn't get a satisfying response from the authors (authors claimed to \"have corrected it\", they just changed phrasing of their response but didn't re-compute scores), I decide to lower my scores from 7 to 6. I am still marginally inclined to acceptance, but will leave it to the discretion of the AC on whether this paper should be rejected due to flawed FID computation.\n\n[1] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B. and Hochreiter, S., 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems (pp. 6626-6637).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes to train EBMs using multi-stage sampling. The EBMs are then used for generating high dimensional images, performing image to image translation, and out-of-distribution detection. The reviewers are impressed with the results, but indicate that the novelty is limited. While I agree that the work can be seen as a combination of previously proposed techniques, demonstrating that this combination can be made to work well is still a significant contribution to the field. In addition, the paper demonstrates strong results in using Langevin dynamics to translate between images, which I do think is novel. I therefore recommend accepting the paper for a poster presentation.
Decision: Accept (Poster)