EBM Life Cycle: MCMC Strategies for Synthesis, Defense, and Density Modeling
Authors: Mitch Hill
Jonathan Craig Mitchell
Chu Chen
Yuan Du
Mubarak Shah
Song-Chun Zhu
Venue: ICLR 2022 Submitted (2022)
Keywords: energy-based model
MCMC sampling
Langevin sampling
generative modeling
unsupervised learning
image synthesis
adversarial defense
density estimation

Abstract: This work presents strategies to learn an Energy-Based Model (EBM) according to the desired length of its MCMC sampling trajectories. MCMC trajectories of different lengths correspond to models with different purposes. Our experiments cover three different trajectory magnitudes and learning outcomes: 1) shortrun sampling for image generation; 2) midrun sampling for classifier-agnostic adversarial defense; and 3) longrun sampling for principled modeling of image probability densities. To achieve these outcomes, we introduce three novel methods of MCMC initialization for negative samples used in Maximum Likelihood (ML) learning. With standard network architectures and an unaltered ML objective, our MCMC initialization methods alone enable significant performance gains across the three applications that we investigate. Our results include state-of-the-art FID scores for unnormalized image densities on the CIFAR-10 and ImageNet datasets; state-of-the-art adversarial defense on CIFAR-10 among purification methods and the first EBM defense on ImageNet; and scalable techniques for learning valid probability densities.

Under review as a conference paper at ICLR 2022
EBM LIFE CYCLE: MCMC STRATEGIES FOR S SYNTHE- SIS, DEFENSE, AND DENSITY MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
This work presents strategies to learn an Energy-Based Model (EBM) according to the desired length of its MCMC sampling trajectories. MCMC trajectories of different lengths correspond to models with different purposes. Our experiments cover three different trajectory magnitudes and learning outcomes: 1) shortrun sam- pling for image generation; 2) midrun sampling for classifer--gnnosic adversarial defense; and 3) longrun sampling for principled modeling of image probability densities. To achieve these outcomes, we introduce three novel methods of MCMC initialization for negative samples used in Maximum Likelihood (ML) learning. With standard network architectures and an unaltered MI. objective, our MCMC initialization methods alone enable significant performance gains across the three applications that we investigate. Our results include state-of-the-ar FID scores for unnormalized image densities on the CIFAR-I0 10 and ImageNet datasets; state-of- the-art adversarial defense on CIFAR- 10 among purification methods and the first EBM defense on ImageNet; and scalable techniques for learning valid probability densities.
INTRODUCTION
Generative modeling of complex signals is one of the fundamental challenges of computational cognition. The importance of studying generative models is often justified by the Analysis-by by- Synthesis hypothesis, which posits that human cogmition an interplay of bottom-up information from world states and top-down synthesis of imagined states. Powerful generative models could provide the foundation of the imaginative capabilities of future machine intelligence. One approach to generative modeling is to posit the existence of density g(x) of signals x that generates data samples, and to learn q(x) using a flexible model p(x; 0), where 0 is model parameter. This approach is taken by Energy-Based Models (EBMs) (Xie al. 2016). normalizing flows (Kingma & Dhariwal
2020 and auto-regressive models (Oord et al. 2016), well as by Variational Auto-encoders (VAEs) (Kingma & Welling 2013) using joint model p(x, 2; In this work we consider modeling ((x) using an EBM density p(æ; 0) for image signals See Appendix for for brief review of Maximum Likelihood learning with an EBM.
2018), score-based models (Song & Ermon 2019
The primary goal of most deep generative modeling is to generate realistic images. is well-known that shortrun sampling with an EBM is effective method for image generation, but synthesis results still lag behind GANs (Goodfellow et al. 2014), score models, and diffusion models (Ho
al.
2020). Other directions of EBM research include learning valid densities (Nijkamp al.
2020). combining discriminative and generative learning via the Joint Energy Model (Grathwohl
al. 2020) and relatives, and using an EBM for defense against adversarial attacks (Hill al.
2021).
In this work, we focus on unconditional learning where EBMs are trained exclusively on unlabeled images. We explore the tasks of image synthesis, adversarial defense, and density modeling to examine breadth of capabilities for the unconditional EBM. Image synthesis is shared goal across all generative models, while defense and density modeling are tasks that are especially suitable for EBMs. See Appendices B and [ for thorough comparison of EBM and other generative models from the perspective of synthesis, defense, and density estimation.
Each task is naturally associated with certain length of MCMC trajectory. Image synthesis is most effective with shortrun trajectories (about 20 to 200 steps) that can rapidly generate new images. Adversarial defense requires midrun trajectories (about 200 to 2000 steps) that can preserve the class

---

Under review as a conference paper at ICLR 2022
FDD Score vs. Number of Langevin Steps
Sheetrun BBM
Midrom ERM
Longrum BBM
Shatrun Visiiu 70-
Longrux
32-
122 107
207
number Langonin
Figure 1: Illustration of the sampling trajectories that we study in this work. The shortrun samples are initialized from generator that is trained in tandem with the EBM because the goal is self-contained synthesis. Midrun and longrun samples are initiated from a high-quality starting image obtained from pre-trained SNGAN, and we study the ability of the EBMs to preserve the quality of the input image from defense and density estimation points of view. The plots show the FID score (Heusel et al. (017)) of 5,000 samples across Langevin steps. The shortrun samples improve on the generator initialization to achieve high-quality synthesis around 250 steps. The midrun samples achieve reasonably low FID in critical range of about 2k steps where defense can be achieved. The longrun sample maintains reasonable synthesis across the entire trajectory, and much further. The shortrun and midrun samples eventually produce defective results outside of their tuned window of stability.
features while sampling removes adversarial signals. Density modeling requires longrun trajectories (50K steps or more) to ensure proper calibration of probability mass for the model steady-state. It is known that EBMs have near universal tendency to learn misaligned steady-state focusing unrealistic images kNij et al. 2020), and there are very few existing solutions to correct this. We intuitively refer to the spectrum of trajectories as the life cycle of MCMC sample, from youth through middle age to maturity. Young samples have the highest quality visual appearance, middle age samples are useful for securing classifiers, and mature samples represent grounded knowledge of the data density (or lack thereof in the widespread case of a misaligned steady-state). Figure illustrates the sampling paths that we study in this work.
Ideally, a valid density estimator would also be good synthesizer and a good defender. However, synthesis quality and density modeling are goals that tend to be odds with each other. High-quality synthesis is easier to achieve using shortrun sampling with a defective density rather than long-run sampling with a stable density (Nijkamp et al. (020).. Unfortunately, models with high-quality shortrun synthesis lack the stability needed for defense. The EBM defense from Hill et al. (2021) advocates for the use of a valid density approximation to stabilize trajectories, but this requirement might be too strict given the difficulty of density estimation. Learning midrun sampling trajectories sufficient for defense is much more feasible for complex datasets such as ImageNet than full density estimation, even if longrun samples from the defensive model are not realistic. Our work is the first to explore this possibility. The overarching purpose of this paper is to discuss techniques for building sample paths from shortrun longrun, with the hope that these techniques will eventually enable both high quality synthesis and defense to be accomplished with a model that is also valid density. For now, the difficulty of valid density modeling leads us restrict our focus to separate time scales of the sampling regime necessary for each task. Interestingly. EBM training naturally accommodates learning at different trajectory lengths.
Strategies for improving EBM learning beyond the standard framework (e.g. Xie et al. 2018): Du Mordatch (2019); Nijkamp et al. (2020)) can broadly be divided into methods that focus on the initialization of MCMC samples (Xie et al. 2018, Gao et al. [018 Nijkamp et al. (2019) and methods that focus on the ML learning objective (Yu et al. 2020b). Some works explore both (Gao et al.,
2020a: Du et al.) 2020 Our novel learning methods focus MCMC initialization, and we retain the standard ML objective and use conventional network architectures. We introduce three new MCMC initialization strategies which are tailored to the three different trajectories lengths explore. During training we exclusively use shortrun MCMC to ensure computational feasibility. Learning models with midrun and longrun trajectories is accomplished by simulating longer trajectories via well-
2

---

Under review as a conference paper at ICLR 2022
chosen initialization and optimizer annealing. Our initialization strategies are able to significantly improve the state-of-the-at across the tasks we investigate. We summarize our contributions below.
We propose a hybridization of persistent (Tilleman, 2008) and cooperative (Xie et al. 2018) initialization to overcome limitations of each. The proposed method yields state-of-the-art FII) scores for unconditional unnormalized image densities for the CIFAR-10 and ImageNet datasets. The ImagetNet results surpass GANs trained with similar resources. See Section
We show that persistent initialization with appropriately tuned rejuvenation from in-distribution states can be used to train EBMs with stable trajectories of several thousand MCMC steps. This allows us extend the method of Hill et al. (202T) to obtain state-of-the-art purification-based defense for CIFAR-10 10 and to scale the EBM defense to ImageNet. See Section 3
We propose a method for principled density estimation that allows incorporation of a rejuvenation step for persistent states. Incorporating rejuvenation allows to learn well-formed EBM densities at a greater scale than previously possible. See Section
Our initialization methods will primarily build upon persistent Tieleman, 2008 Du & Mordatch 2019) and cooperative (Xie al. 2018) initialization. All of our methods will use a generator network as the source of rejuvenation for persistent states. Our shortrun experiments will learn the generator in tandem with the EBM so the synthesis process is self-contained, while our midrun and longrun experiments will use pretrained generators since we will apply sampling paths from in-distribution initial images rather than synthesizing from scratch.
HYBRID PERSISTENT COOPERATIVE LEARNING FOR IMAGE SYNTHESIS
In this section, we focus on on the conventional task of learning an EBM for high quality synthesis with shortrun sampling. We restrict our attention to achieving high quality synthesis without use of pretrained models, in contrast with works such as [he et al. (2020); (2020)[Alayra al. (209)). Our method involves hybridizing persistent [Tieleman] 2008) and cooperative (Xic al.) [018)) initialization. We first briefly cover the strengths and weaknesses of these methods, then discuss our hybridization, and finally present experimental results.
2.1 MOTIVATION: LIMITATIONS OF PERSISTENT AND COOPERATIVE INITIALIZATION
A common framework for learning synthesis with an EBM is to use persistent initialization for MCMC samples with a certain rate of rejuvenation (e.g. 5% chance) from noise distribution (Du & Mordatch 2019). This approach can cause instability because shortrun samples used to update the model include a mix of higher-energy burn-in samples and lower-energy realistic images, which can destabilize training by increasing the variance of the gradient of the negative samples in (4). On the other hand, removing rejuvenation can decrease the quality of the learned images because persistent images often become stuck in local modes and develop defects that linger for many updates. Persistent banks also scale poorly to large datasets such as ImageNet because the bank cannot efficiently represent the diversity of the data. Initialization from cooperative generator network is an appealing alternative because could enable efficient in-distribution generation of highly diverse appearances. Using generator for rejuvenation allows samples to begin much closer the correct energy spectrum, thereby avoid the instability of noise rejuvenation. However, we observe a major limitation of cooperative learning that affects the generator ability to produce diverse initial states. In particular, generator trained with the cooperative learning objective has difficulty breaking the symmetry of its activations. This happens because shortrun EBM samples are unable to provide novel diversity if the generator initialization already lacks diversity. This is illustrated in Figure 5 Learning quickly becomes unstable without auxiliary techniques such as batch normalization break generator symmetry. The lack of diversity of shortrun EBM samples limits the results of cooperative learning even when training succeeds.
The strengths and limitations of these methods are complementary. Persistent initialization can reliably provide a diverse set of initial images that represent prior samples of several different model snapshots. Even if a single EBM update is biased to certain image defect (e.g. MCMC samples are blue-tinged or too bright), samples across previous iterations have the correct diversity on average. On the other hand, the generator can quickly propose in-distribution states for rejuvenation
3

---

Under review as a conference paper at ICLR 2022
instead of relying on out-of-distribution noise samples aggressively augmented persistent samples. Furthermore, training the generator in tandem with the EBM provides an efficient way of generating new samples from scratch after training that match the samples used during training. In contrast, models trained with persistent initialization often cannot produce good samples after training without an in-distribution initialization (Nijkamp et al. (020). Learning generator and EBM jointly enables self-contained and reproducible FID implementation to evaluate the learned model.
2.2 HYBRID PERSISTENT COOPERATIVE INITIALIZATION
We visualize our proposed initialization technique in Figure The method features two banks of persistent states. One bank is for persistent images and the other bank is for persistent latent vectors. There will always be a one-to-one pairing between persistent images and persistent latents, meaning that paired states will be drawn from the banks, return to the banks, and experience rejuvenation at the same time. All samples are rejuvenated from the current generator. Other than the pairing and generator rejuvenation source, sampling from banks and rejuvenation with fixed probability is conducted in the same way as persistent initialization. This pairing is necessary for the cooperative
learning loss used train the generator. Given a batch of paired samples {Z and {x,,)) ) :=1 drawn from the latent and image bank respectively and generator network g2; Ø), we learn ø using gradient descent on the reconstruction loss
72
(())
-TXX--XXX T(X
where T(X,, 0) = X; represents a shortrun Langevin trajectory with the EBM p(z; 6). This loss can be de- rived in the MI. framework as way to teach the output of g(2; Φ) to match the distribution of p(z; 0). A full implementation of cooperative learning requires an ad- ditional sampling process Z,, but we use straight- through estimator (Bengio et al. (0133) and approximate
Cooperativ-Perssitenn
Generazor
Snew
Data
Image Dank
x Bank
Tegrad
latent sampling with the identity function originally done in the official MATLAB implementation from Xie
Jusi lipille
Genecntor Loss
et al. (0018). We review further details of the coopera-
tive learning loss in Appendix C
Intuitively, the loss 1 encourages the output of the generator to match the outcome of the Langevin sam- pling process (5). Cooperative learning always uses X.o (Z, Φ), which corresponds to rejuvenating with probability 1 in our method. This is the root of the limitation of cooperative learning. If g(2; Φ) has little diversity, T(g(z; Φ); 0) will also have little diver- sity. This causes extreme oscillation in both the EBM and generator output as p(a; 8) attempts to cover the modes of g(z) with shortrun samples from low diversity initialization. Despite oscillation in appearance across σ. generator samples tend to remain nearly identical for any fixed é, thereby perpetuating the instability and preventing further learning. By drawing samples from the image bank, we are effectively choosing X, = (2((;2 σ')) where d' represents a past generator parameter and (Txx) represents the composition of Langevin sampling with past EBMs. This allows us to learn e using appropriately diverse initializations spanning samples of several past models. We observe that this simple adjustment has dramatic effect for improving EBM learning with generator initialization.
Lass
By 1011-1"""" slill
Figure 2: Dooperative-persistee initializa- tion uses paired latent and image states that drawn from persistent banks to learn the EBM and generator.
2.3 EXPERIMENTS: IMAGE SYNTHESIS WITH SHORTRUN MCMC
We present the results of our new learning process applied to the CIFAR-10, Celeb-A, and ImageNet benchmark datasets, using the image sizes 32 32, 64 x 64, and 128 x 128 respectively. A sketch of the hybrid learning algorithm is given in Appendix D Besides standard deep learning techniques for stable optimization, framework is fully described by the MI objective and MCMC initialization
4

---

Under review as a conference paper at ICLR 2022
Table Comparison of FID scores among representative generative models. (*=EBM, f=conditional)
Celeb-A 64 x 64
CIFAR-10 32 x 32
Model
FID
Model
FID
Ours#
16.3
Ours
22.1
Miyato et al. (2018)
5.7
Du et al. 2020
25.1
Song & Ermon 2020)
10.2
Grathwohl et al. (2021
27.5
Han ct al. 2019)
31.9
XIe ct al. (2018)
33.6
Gao ct al.
(0220a)
37.3
Grathwohl ct al.
(2220)
38.4
ImageNet 128 x 128
Du Mordatch
(2119
40.6
Model
FID
et al. (20200a**
30.9
Ours
38.9
Du & Mordaich (2019
37.9
Chen et
2019
43.9
Ho et al. (2020)
3,2
Song & Earmon (2020
10.9
Lee el al.
2021
58.9
Miyato et al.
[2018
65.7
Brock el al. 2019)
14.7
Miyato 2018
27.6
Miyato et al.
2018
21.7
Du & Mordatch
(2019)*1
43.7
above. We use the SNGAN Miyato et al. 2018) architectures for all models, where EBMs use the SNGAN discriminator with no normalization. The generator has batch normalization for the CIFAR- 10 and Celeb-A experiments only. Table I displays the FID scores achieved by our model in comparison with prior methods.
When calculating FID scores, samples are first initialized from the generator then updated with the EBM using number of Langevin updates tuned provide optimal synthesis quality. As mentioned before, appealing aspect of EBM learning with generator initialization is the ease of generating new images from scratch, contrast with persistent initialization. We use 50K samples with the official FID code from Heusel et al. (2017) to caleulate all FID scores. In particular, our scores and framework are consistent with the GAN replication library from Lee & Town (2020). We publicly release the representative of what achievable within an ordinary run of the code provided. Our CIFAR-10 10 results show a significant improvement over prior EBM synthesis and contribute to closing the gap between EBMs and other generative models. Surprisingly, our ImageNet results surpass the results of GANs such as SNGAN (Miyato et al. 2018) and SSGAN (Chen al. 2019) on unconditional synthesis using similar magnitude of computational resources.
checkpoints, learning code, and FID code for each model. The checkpoints
MIDRUN SAMPLERS FOR ADVERSARIAL DEFENSE
This section presents method for learning EBMs that are capable of preserving the appearance of an in-distribution initial state across several thousand MCMC steps. Such models are useful for the purpose of adversarial defense. Our defense framework is based on the approach in Hill al. (2021), which uses an EBM to defend an independent naturally trained classifier. Appendix D briefly reviews the EBM defense and compares this approach with other defense methods. We then present our proposed method for learning a defensive EBM B3) which is based on persistent initialization using fixed pretrained generator as a source rejuvenation. Finally, we apply our defense achieve state-of-the-ar performance for purification- baaed defense on CIFAR-10 and ImageNet.
One limitation of prior EBM defense is the reliance on persistent initialization with no rejuvenation to learn the defensive model. This is done to ensure that defensive sampling trajectories remain stable for arbitrary numbers of steps. However, removing rejuvenation from persistent learning has drawbacks discussed in Section 2.1 In particular, it becomes very difficult to learn meaningful EBMs for large and complex datasets such as ImageNet cannot represent the diversity of the dataset and the quality of persistent images without rejuvenation quickly degrades. Methods for efficient learning of defensive EBMs at a greater scale are needed to extend the EBM defense
this framework because the persistent bank
realistic situations.
5

---

Under review as a conference paper at ICLR 2022
We propose overcome this obstacle by building on the observation that fully stable sampling paths are not required for successful defense. While the defense from Hill et al. (202T) uses EBMs with stable sampling for 100K steps or more, the defense results require less than 2000 steps. A natural question is whether it is possible to learn stable MCMC trajectories for only a predefined midrun range to achieve the defensive benefits without fully stabilizing samples over longrun trajectories. Defining learning procedure to obtain such models is the goal of this section. Efficiently learning EBMs with stable midrun trajectories allows us scale up the EBM defense significantly more challenging domains.
3.1 IN- IN-DISTRIBTTION REJUVENATION FOR LEARNING A DEFENSIVE EBM
The initialization method for our midrun sampler is similar to standard persistent initialization with the adjustment that persistent states are rejuvenated from frozen pretrained generator rather than noise. We use a generator in our experiments so that the EBM and generator could be used after training to sample from scratch. This choice is not necessary and rejuvenation from data samples, or another efficient in in-distribution initialization, is also effective for learning defensive EBMs. We note that MCMC initialization from a trained generator or from data samples is not explored in recent work because the goal of most current EBM learning is image synthesis, which becomes trivial if EBM trajectories are always initialized from high-quality samples. From the perspective of synthesis our learning process is nearly invisible but from the perspective of defense its utility becomes concrete.
Learning an EBM for defense involves tuning the length of the sam- pling trajectory via the number of shortrun training steps and the reju- venation rate, and tuning the annealing schedule of the EBM optimizer. Given desired number of MCMC steps K for defensive update and a shortrun trajectory K 2 100, we simply set the rejuvenation rate to Prejuv = K/ K/Kan Keef to ensure that on average samples will travel Kdef steps before rejuvenation. While in practice we use Kaet 2000 and Prejuv 0.05, we have found that this method can yield stable paths for at least Kaer 50K MCMC steps when Prejuv is low.
Persistent
Data
Bank
To
Logeadi
x
Upcas
Redormakee
Initialization alone is insufficient to stabilize MCMC pathways when model weights are changing quickly. Using low learning rate late in training is a key aspect of stabilizing sampling paths (Nijkamp et al. 2020 Hill et al. 2021). Intuitively, if the EBM optimizer has a suf- Figure 3: Persistent ini- tialization. Positive sam- ples are from data samples and negative samples are MCMC samples initialized from a batch from an image bank. Some states are ran- domly rejuvenated.
EBM Loss
BBM Tans
ficiently low learning rate then MCMC trajectories in the persistent image bank can function as approximate trajectories from the current model, since weights change very little as the persistent states are up- dated. By annealing in tandem with our initialization, are effectively using midrun trajectories of length Kaer initialized from the generator to update the EBM while we are actually using shortrun trajectories of length K from the persistent bank. Annealing is a crucial component for stabilizing both midrun and longrun trajectories. Without annealing, sampling paths are not able to maintain realism for large Kaet-
3.2 EXPERIMENTS: DEFENDING NATURAL CLASSIFIERS WITH AN EBM
We train our EBMs using the persistent initialization described above in tandem with pretrained generator on both CIFAR- 10 and ImageNet. We use the same SNGAN models as before for our CIFAR-10 experiments, with the exception that the generator is pretrained instead of learned. For our ImageNet experiments, we use the BigGAN (Brock et al.) (219) discriminator architecture for our EBM modified for input size 224 x 224 and a pretrained BigGAN Generator. Our naturally trained classifier f(x) is a pretrained WideResNet 28-10 (Zagoruyko ko & Komodakis 2016) for CIFAR-10 and pretrained EfficientNetB-7 architecture (Tan & Le 2020) for ImageNet.
We evaluate our models using the attack gradient g and Algorithm 2 For CIFAR-10 we use 8 and
Kast 500 Langevin steps with loo adversarial parameters of the Lox ball and a is the gradient step size. For ImageNet we used Kaef = 200 Langevin steps for defense with lo adversarial parameters
2 where is the size
255
25G
255 and a 255 We attack ImageNet for 50 attacks

---

Under review as a conference paper at ICLR 2022
Table 2: Defense vs. whitehox attacks with so perturbation E 8//55 for CIFAR-10.
Defense
f(x) Natural
Train Ims.
T(2) Method
Attack
Nat.
Adv.
Ours
Langevin Langevin Gibbs Update
BPDA+EOT
0.866
0.676
Hill
2021
Natural
BPDA+BOT
0.8412
0.5490
Song ei a..
2018
Natural
BPDA
0.95
0.09
S Srinvasan et al
2019)
Natural
Langevin
PGD
0.0048
Yang cf 2019)
Transformed
Mask + Recon.
BPDA+BOT
0.94
0.15
armon
2019
Adversarial
PGD
0.897
0.625
ZZhang ct al.
2019)
Adversarial
PGD
0.849
0,5643
Shalahi et al. 2019
Adversarial
PGD
0.859
0.4633
Madry et al.
2018)
Adversarial
PGD
0.873
0.458
steps across 10K val samples. For CIFAR-10 we perform the same number of attacks across 4K validation samples.
On CIFAR-10, we surpass the robustness of the existing EBM defense using much more re- liable learning framework. The importance of midrun learning is Adv.
Table 3: Defense loc whitebox attacks for ImageNet.
Defense
f(x)
Train Ims.
Nat.
clearly demonstrated by our suc- cessful application of EBM de- fense to ImageNet at the resolu- tion 224 224. The robustness of a naturally trained classifier se- cured by our EBM is comparable with adversarial training. While the ImageNet results for EBM defense are not yet on par with state-of-the the----tthharr art adversarially trained models, our experiments are an important proof of concept that the method can be scaled. See Appendix F for diagnostics and further discussion.
Ours
Natural 2
0.684
0.418
2g6
WWog et
2020
Adversarial
0.609
0.4339
2g5
Shatahi ct al
2019
Adversarial
0,644
0.4339
216
Qun 2019)
Adversarial
0.822
0.427
255
Xie et al.
2021
Adversarial
0.822
0,586
255
LONGRUN SAMPLING FOR DENSITY ESTIMATION
Our final objective is to introduce scalable tools for learning a valid image density with an EBM. Nijkamp al. 2020) revealed that, in the absence of careful implementation, EBM learning always results in an unexpected outcome where steady-state samples from the learned density p(r; 0) have oversaturated and unrealistic appearance that differs drastically from shortrun samples used during learning. This outcome affects all EBMs that are not specifically trained to overcome this defect, as well as related models such Score Matching (Song & Ermon 2019 2020) shown in Figure g To our knowledge, across different generative models, the only way to learn a well-formed potential energy surface approximating a complex density g(z) that is compatible with efficient MCMC sampling is using an EBM.
Despite the theoretical formulation of the EBM as a potential surface, the problem of principled density estimation has received relatively little attention. Successful image synthesis is sometimes used misleading evidence of successful density estimation, but we emphasize these outcomes not equivalent, In this section, we address the lack scalable methods to learn valid densities of complex signals by introducing an MCMC initialization can incorporate rejuvenation while still simulating extremely long trajectories. Learning valid density is fundamental computational problem that is important in its own right, and we further hope that our learning method leads to EBM clustering techniques based on the potential energy basins of well-formed density (Nijkamp et al. 2020).
4.1 INCORPORATING REJUVENATION IN DENSITY ESTIMATION
Prior work has suggested that persistent learning is the most effective method for learning a valid EBM density. Furthermore, works that learn valid density have avoided rejuvenation because
7

---

Under review as a conference paper at ICLR 2022
the incorporation of newly rejuvenated samples into the persistent bank ensures that EBM updates will always include samples that are not at the steady-state. However, persistent learning without rejuvenation has shortcomings mentioned in Section 3 We present hypothesized conditions for learning a valid density that motivate the design of our MCMC initialization:
After certain point in training, all samples used to update the EBM must be approximate steady- state samples of the current model p(2; 0).
Persistent samples that are newly rejuvenated (up about 50K Langevin steps since rejuvenation. and possibly many more) cannot be approximate steady-state samples for any known rejuvenation sources, including data, generators, and noise.
Persistent samples that have undergone sufficiently many lifetime Langevin updates for model whose weights are changing very slowly can be approximate steady-state samples.
Both the second and third point are corroborated by prior work (Nijkamp et al.) 2020 Hill et al.] (02T) as well as our observations. The third point means that persistent states updated with shortrun Langevin can eventually act as longrun Langevin samples the optimizer learning rate is small, because the EBM samples in previous timesteps are essentially samples from the current EBM.
Longrun
Convergent
Update Bank
Burn-in Bank
Data
Iness
Togred
lank Suackan
Dejoraaios
Learning valid density that includes rejuvenation while satisfying the conditions above requires separating the newly rejuvenated samples from samples that are used to update the EBM. This leads us to introduce two persistent image banks: one for newly rejuvenated samples, and one for samples that will be used to update the EBM. Samples in the newly rejuvenated bank that have been updated sufficiently many times will eventually replace samples from the bank used to update the EBM, at which point newly rejuvenated states will be added to the burn-in bank. Figure(4shor shows a visualization of the MCMC initialization method. As in Section we will use pre-trained generator to efficiently obtain high-quality rejuvenated samples so that the generation process is fully synthetic, although data samples could be used as well. Our goal is to preserve the sample quality for an arbitrary number of MCMC steps. We note that this is a sufficient but not necessary condition for learning a valid EBM density. Nonetheless, stable sampling is an important step towards rigorous probabilistic EBMs.
Ø
x
La
FBM Loss
B
B Pyycent
Figure 4: Visualization of our longrun initializatin procedure. Newly reju- venated samples must remain the the burnin bank until they have approach the model steady-state, which point they move to the update bank to be used for model gradients.
Even with our improved initialization, we find that extremely lon- grun trajectories of million or more MCMC steps still tend to oversaturate, although to much lesser degree. To further stabilize the appearance of extremely longrun Langevin samples, we include prior energy terms in the model. Our longrun EBMs have the form
Table 4: FID for 5K samples after 100K Langevin and IM Langevin steps. FID remains stable over long trajectories.
U0o o (x; 0) = U' U" + 20 ||x|
(2)
Data
100K
1M
where Uxx 0) is the model whose weights are updated, (z 0o) is prior EBM with fixed weights Gg and is a parameter controlling the strength of a Gaussian prior. We used a prior EBM shortrun manner. The role of the prior EBM is to provide some stability but also to provide a tendency to oversaturate longer trajectories so that the current EBM learns to correct oversaturation. The Gaussian prior is meant to discourage unbounded activations outside of the image hypercube. Further discussion in Appendix H We find that including both of these terms significantly improves the ability to learn quality synthesis over long trajectories.
CIFAR-10
49.2
51.7
Celeb-A
37.4
45.9
ImageNet
82.3
77.8
4.2 EXPERIMENTS: EBM DENSITY ESTIMATION
In these experiments, we learn EBM models that have stable MCMC trajectories for an arbitrary number of steps. We apply the longrun learning method in Algorithm to CIFAR-10, Celeb-A, and
8

---

Under review as a conference paper at ICLR 2022
ImageNet the resolutions 32 32, 64 x 64, and 64 x 64 respectively, Pre-trained SNGANs are used to rejuvenate images for CIFAR- 10 and Celeb-A, while a resized images from a pre-trained BigGAN are used to rejuvenate ImageNet samples. We require that samples have been updated for 50K to 75K Langevin steps since the last rejuvenation before states are moved from the burn-in bank to the update bank. Our EBMs all use the discriminator architecture of the SNGAN model without spectral normalization. We use pretrained midrun samples for our prior EBM terms in the learned energy 22. See Appendix K for an algorithm sketch. Figure [ shows the evolution of FID score over Langevin steps starting from the generator network for our CIFAR- 10 model. Table shows the FID scores of 5K samples for each dataset. Visualizations of longrun samples with 100K steps and extremely longrun samples using million Langevin steps are shown in Appendix M See Appendix ]
for a discussion of our evaluation comparison to log likelihood.
5 RELATED WORK
Energy-Based Models. Early forms of EBMs include the exponential family distribution, the FRAME model (Zhu et al.) 1998) and Restricted Boltzmann Machines (Hinton 2002). Recent work
has introduced the EBM with a ConvNet potential (Xie et al.) [016] Du & Mordatch] 2019). This
dramatically increased the learning capacity of the model which led to many follow-up works on
image synthesis (Gao et al. 2018 Lee et al. 2018 Nijkamp et al.) 2019), adversarial robustness (Hill
al. (202T), and joint learning of discriminative and generative models (Grathwohl al. 2020).
Several works investigate training and EBM in tandem with an auxilary model. Kim & Bengio (2016)
train an EBM and generator and tandem without MCMC by using samples from the generator as direct approximations of the EBM density and training the generator using a variational objective. similar approach explored by Grathwohl et al. (02TT) Cooperative learning (Xie et al. 2018) trains the EBM and generator by using the generator to initialize samples needed to train the EBM and uses reconstruction loss between generator and EBM samples to learn the generator. Gao et al. (2020a) learn an EBM using Noise Contrastive Estimation with auxiliary flow model. Xiao et al. (2021) use pretrained VAË to facilitate EBM learning. Our work builds on cooperative learning by identifying and resolving symmetry breaking problems in early training. leading to state-of-ther -rrt EBM synthesis for unconditional ImageNet. Despite the formulation of the EBM as an unnormalized density, it has been shown that most EBMs have strong misaligned steady-state distributions (Nijkamp) cet al. 2020). Our work introduces new methods to learn a model with correct steady-state alignment.
Adversarial Robustness. Adversarial Training (AT) (Madry al. 2018), which trains a classifier using PGD-generated adversaries, is the most popular and studied adversarial defense. Many varia- tions and improvements have been introduced, including optimizing the training loop by recycling gradients of past adversaries (Shatahi et al.] 2019). combining single step FGSM with random ini- tialization to achieve similar robustness Wong et all 2020), learning with auxiliary unlabeled data Carmon et al. (2019). local linearization (Qin et al. 2019), and the use of smooth activation functions Xie et al. (2021). An alternative approach to adversarial training involves the use of preprocessing transformations. Randomized smoothing (Cohen et al. 2019) and related methods (Salman et al. 2020) add noise to the input signal to remove adversarial signals. Many other preprocessing defenses have been proposed (Guo et al.) 2018 [Song et al.) 2018 Yang et al. (019),, but nearly all of these methods can be broken by adaptive attacks that are aware of the preprocessing method (Athaale) al. (2018). A notable exception is the EBM defense (Hill et al., 2021). which uses midrun MCMC trajectories to purify images. We ease the restriction of learning EBMs with stability for arbitrary MCMC runs in the EBM defense by introducing a midrun sampler that enables faster learning of defensive EBMs and allows the EBM defense to scale to more complex datasets.
6 CONCLUSION AND FUTURE WORK
We have described three unique MCMC initializations for EBM using different sampling trajectories: shortrun for synthesis, midrun for defense, and long-run for density estimation. Furthermore, we have elaborated on different MCMC initialization strategies used to stabilize these models for different sampling lengths. We have demonstrated the flexibility of these mechanisms by using similar architectures, data, and training platforms to create different EBMs for different applications. We hope that future research incorporates these new training initialization schemes to improve their generative models for wide variety of tasks.
9

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY
We have made our results reproducible by releasing the anonymized github repository https wttp://ceeeccneeeeeeeeeeeeeeeeee
7/ //anooymouus.
All hyperparameters can be found in the configuration files, all checkpoints for the various EBMs are also under the release section and there is a README.
describing how use this repository to reproduce the results described herein. We will release a non-anonymous repository at a later date, All our results were trained on the standard CIFAR-10, ImageNet, and Celeb-A datasets which are publicly available. Our computational resources were primarily 5 TPUv2-8 machines and 5 TPUv3-8 machines made available by the Tensorflow Research Cloud (TRC) program sponsored by Google. These resources are widely available to researchers who apply to the program from accredited institutions. We are very grateful to Google for granting us access to the resources that made this work possible.
REFERENCES
Jean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? In Advances in Neural Information Processing Systems, volume 32, 2019.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In Proceedings of the 35th International Conference on Machine Learning, pp. 274-283, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint OrXiiv:3308 3432, 2013.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference Learning Representations, 2019. URL
htttps://openzeet htt.eeeeeeeee/toteeeeeeeaae
net/
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy Liang. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems, volume 32, 2019.
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your gan is secretly energy- -ased model and you should use discriminator driven latent sampling. arXiv preprint wxXiv:2033.66060 2020.
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucie, and Neil Houlsby. Self-supervised gans via auxiliary rotation loss. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12154-12166 2019.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In Proceedings of the 36th International Conference on Machine Learning, pp. 1310-1320, 2019.
Aurelien Decelle, Cyril Furtlehner, and Beatriz Scoane. Equilibrium and non-equilibrium regimes in the learning of restricted boltzmann machines. In Advances in Neural Information Processing Systems, 2021.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, volume 32, 2019.
Yilun Du Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. arXiv preprint arXiv:2012.013// 2020.
Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Wu. Learning generative convnets via multi-grid modeling and sampling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9155-9164, 2018.
Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy-basee models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7518-7528, 2020a.
10

---

Under review as a conference paper at ICLR 2022
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy based models by diffusion recovery likelihood. arXiv preprint arXiv:2012.08125, 2020b.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Warde-Farley, Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations, 2020.
Will Sussman Grathwohl, Jacob Jin Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud. No {meme) for me: Amortized sampling for fast and stable training of energy-based models. In International Conference on Learning Representations, 2021. URL
htttps://ppenreviie netp://wwweeeeeeeeeeeee..e..ee forum?id=1 (0grrmmmidixxxs550911333
xp5x09f1k3
Cheng-En Guo, Song-Chun Cang-Chun Zhu, and Ying Nian Wu. Modeling visual patterns by integrating descriptive and generative methods. International Journal of Computer Viston, 53:5-29, 2003.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. In International Conference Leurning Represeniations, 2018.
Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Divergence triangle for joint training of generator model, energy- based model, and inferential model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
8670-8679, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter, Gans trained by time-scale update rule converge a local nash equilibrium. In Advances in Neural Information Processing Systems, volume 33, pp. 6629-6640, 2017.
Mitch Hill, Jonathan Craig Mitchell, and Song- Song-Chun Zhu. Stochastic security: Adversarial defense using long-run dynamics of energy-based models. In International Conference on Learning Representations, 2021. URL
Geoffrey Hinton. Training products experts by minimizing contrastive divergence. Neural Computation, 14(8)11771-1800, 2002.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020.
Taesup Kim and Yoshua Bengio. Deep directed generative models with energy-based probability estimation, 2016.
Diederik Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. arXiv preprint 1807.0/807..00 0887.03039, 2018.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 2013.
arXiv preprint
Kwonjoon Lee, Weijian Xu, Fan Fan, and Zhuowen Tu. Wasserstein introspective neural networks, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3702-3711, 2018.
Kwot Sin Lee and Christopher Town. Mimicry: Towards the reproducibility of gan research. arXiv preprint arXiv:2005.02494 2020.
Kwot Sin Lee, Ngoc-Trung Tran, and Ngai-Man Cheung. Infomax-gan: Improved adversarial image generation via information maximization and contrastive learning. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3942-3952, 2021.
Kyungmin Lee and Seyoon Oh. Efficient randomized smoothing by denoising with learned score function, 2021. URL.

---

Under review as a conference paper at ICLR 2022
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. International Conference on Learning Representations, 2018. URL
net/ wtwp://wwwwweeeeeennne...... Loruunii
Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run MCMC toward energy-based model. In Advances in Neural Information Processing Systems, volume 32, 2019.
Erik Nijkamp, Mitch Hill, Tian Han, Long-Chun Zhu, and Ying Nian Wu. On the anatomy of MCMC-based maximum likelihood learning of energy-based models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, 2020.
Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Ko- ray Kavukcuoglu. Conditional image generation with pixelenn decoders. arXiv preprint 1606v:6006.055 2016.
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization, 2019.
Hadi Salman, Mingjie Sun, Greg Yang, Ashish Kapoor, and J. Zico Kolter. Denoised smoothing: A provable defense for pretrained classifiers. Advances in Neural Information Processing Systems, volume 33, pp. 1994521957, 2020.
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free!, 2019.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint 190ivv190..05660 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models, 2020.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Lever- aging generative models to understand and defend against adversarial examples. In International Conference Learning Representations, 2018.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
Vignesh Srinivasan, Arturo Marban, Klaus-Robert Muller, Wojeiech Samek, and Shinichi Nakajima. Defense against adversarial attacks by langevin dynamics. arxiv preprint arXiv: 12017, 2019.
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks, 2020.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. CoRR, abs/1511.01844, 2016.
Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference on Machine learning, pp. 1064-1071, 2008.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. In Advances Neural Information Processing Systems, volume 33, pp. 1633-1645, 2020.
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations, 2020, htttps:::ppppeeiiii... pet//oorum2io=a0x000E55
12

---

Under review as a conference paper at ICLR 2022
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. { A symbiosis between variational autoencoders and energy-based models. In International Conference on Learning Representations, 2021. http://ss/opprrreeeeeeee///rrrrr
Cihang Xie, Mingxing Tan, Boqing Gong, Alan Yuille, and Quoe V. Le. Smooth adversarial training, 2021.
Jianwen Xie, Yang Lu, Song- Chug-Chunn Zhu, and Yingnian Wu. A theory of generative convnet. In Proceedings of the 33rd International Conference on Machine Learning, pp. 2635-2644, 2016.
Jianwen Xie, Yang Lu, Ruiqi Gao, and Ying Nian Wu. Cooperative learning of energy-based model and latent variable model via meme teaching. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Yuzhe Yang, Guo Zhang, Dina Katabi, and Zhi Xu. Me-net: Towards effective adversarial robustness with matrix estimation. In Proceedings of the 36th International Conference on Machine Learning. pp. 7025-7034, 2019.
Jongmin Yoon, Sung Ju Hwang, and Juho Lee. Adversarial purification with score-based generative models. arXiv preprint arXiv:2 0022200600004 2021.
Lantao Yu, Yang Song. Jiaming Song, and Stefano Ermon. Training deep energy- baaegy models with f-divergence minimization, 2020a.
Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon. Training deep energy-based models with
I-divergence minimization. In International Conference on Machine Learning, pp. 10957-10967. PMLR, 2020b.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.0711 2016.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In Proceedings of the 36th International Conference Machine Learning, pp. 7472-7482, 2019.
Long-Chun Zhu. Statistical modeling and conceptualization of visual patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25:691-712, 2003.
Song-Chur Zhu, Ying Nian Wu, and David Mumford. Filters, random fields and maximum entropy (FRAME): towards a unified theory for texture modeling. International Journal of Computer Vision, 1772)1077-16, 1998.
13

---

Under review as a conference paper at ICLR 2022
A REVIEW OF EBM LEARNING AND MCMC INITIALIZATION
We briefly review the main components of EBM learning following the standard method derived from works such as Hinton 2002); Zhu et al. (1998;; : Xie al. (2016). Throughout experiments, we will use this framework and focus on exploring different possibilities for MCMC initialization. A deep EBM has the form
p(z; Z(0) exp(-U(x;;0) 0)}
(3)
where U (z; 0) is ConvNet with weights 0 and Z(0) is the intractable normalizing constant. Max-
imum Likelihood (ML) learning uses the objective arg ming (x=(((=)]lp(x;;) 0)), which can be minimized using the stochastic gradient
VC(1)
VoU(x;:0)
(XU(X;
(4)
where the positive samples {X," Ji=1 are set of data samples and the negative samples {X7 ]==1 are samples from the current model p(æ; 0). To obtain the negative samples for a deep EBM, it is common to use MCMC sampling with the K steps of Langevin equation
x((+1) x(x)
Xx UX nZk
2
where v) is the step size and Z& (O,, 1). The Langevin trajectories are initialized from set of states { obtained from a certain initialization strategy. The method of MCMC initialization determines many of the properties of the learned model.
There are a number of different methods of MCMC initialization for EBM learning. Early Restricted Boltzmann Machines use data samples as the initial states in every iteration (Hinton (002). Persistent initialization (Zhu et al. 1998 Tieleman (208)) maintains bank of MCMC samples from previous model updates as the source of MCMC states in the current iteration. Persistent states can be rejuvenated with certain probability instead of returning to the image bank. Potential sources for rejuvenation include noise samples (Du & Mordatch, 2019). data augmentation applied to persistent states (Du et al. (0200), data samples, samples from generator network. Cooperative learning initializes MCMC states from a generator learned in tandem with an EBM (Xie et al., 2018). The Multi-grid model (Gao al., 2018) initializes samples from lower resolution images. Noise initialization is explored by Nijkamp et al. (2019). The VAEBM (Xiao et al. (021)) initializes samples from pretrained VAE generator. EBMs can learn stable shortrun synthesis from virtually any initialization distribution as long as the initialization distribution remains fairly consistent across training updates. EBM sampling can produce realistic images after training when the testing initialization closely matches the training initialization, while synthesis can fail for test initializations unseen during training. (Nijkamp al. 2020).
B COMPARISON OF EBM AND OTHER GENERATIVE MODELS
In this section, we briefly compare the EBM with other kinds of generative models. Besides the probabilistic model families mentioned in the introduction, we consider two other popular approaches. One approach is design an objective that promotes realistic synthesis by producing fake samples and distinguishing these from real samples, as done by the Generative Adversarial Network (GAN) model (Goodfellow et al. 2014) and its many variations. Another approach comes from recent diffusion models that define a sequence of intermediate distributions to transport signal between reference density (x) and ((x)) (Ho et al. 2020, Song et al. 2020).
Among probabilistic models, the synthesis quality is typically the highest for score matching models that only try to learn distribution gradients and typically lower for models like normalizing flows and auto-regressive objectives that model the full normalized density. EBM synthesis usually lies somewhere in between. Score-mare matching does not require MCMC sampling during learning and as a consequence learning tends to be more straightforward than learning for EBMs. Yet we believe that MCMC sampling is not yet fully optimized for EBM learning and that EBMs can match or surpass score-matching models. Our shortrun initialization method is effort in this direction.
14

---

Under review as a conference paper at ICLR 2022
Coop. (w/ BN)
Coop. (w/o BN)
Ours (w/ BN)
Ours (w/o BN)
Figure 5: Comparison of cooperative learning (Xie et al. 2018) and our hybrid cooperative-persiste learning using appearance of shortrun samples after 500 updates of the EBM using a generator with and without batch norm. The cooperative models have difficulty achieving diversity in the shortrun samples because the EBM is unable significantly change the appearance of initial generator images. In fact, we find cooperative learning is not possible at all without generator batch normalizatin. Our hybrid initialization increases the diversity of shortrun samples by including persistent samples from previous EBM updates. This accelerates learning and dramatically improves synthesis results.
GANs and diffusion models differ fundamentally from probabilistic models of the form p(=; the sense that the former are explicitly formulated from the perspective of synthesis while the latter are formulated for density modeling and synthesis is byproduct. In the literature, the former types of models are sometimes referred to generative models (in more specific sense of the term than typical) while the latter are called descriptive models (Guo et al., 2003 Zhu 2003). Generative models map set to another (e.g. generator mapping latent vector models define a concept in terms of the set of states that have a certain descriptive property (e.g. steady-state samples of an EBM). A descriptive model can be used for generative purposes (e.g. Langevin sampling with an EBM) but this is not the direct goal of descriptive learning.
image), while descriptive
From this perspective, it is perhaps unsurprising that generative methods such GANs and diffusion models surpass EBMs in terms of synthesis, since that the model formulations are geared directly towards generation. Nonetheless, we believe that EBMs have unique properties that distinguish them from generative models and other descriptive models. In particular, EBM learning offers a control of the MCMC sampling paths from a learned model that p(a;; 0) is, to our knowledge, is unobtainable under other current frameworks. Unlike GAN models which map trivial distribution to a complex signal distribution without actually describing the signal distribution, EBMs directly model the complex signal distribution via a potential energy surface. Normalizing flow models are too large for efficient MCMC sampling. VAE models cannot be efficiently marginalized. Score-based models exhibit the same oversaturation behavior miscalibrated EBMs when sampling at constant noise level for many steps (see Figure B). and one cannot adjust MCMC stability during learning model. Diffusion-basee methods do not learn an approximation p(r; 0), but rather can be formulated as learning a conditional probability p(x]=' () that describes state x given state x' rung above or below the ladder of distributions between (z( and ((x) (Gao et al. 20000b). Auto-regressive models depend on certain sequence of components for generation, and are not compatible with joint updates of all pixels. On the other hand, MCMC sampling with an EBM is a natural and relatively efficient process. Uniquely among all methods, the desired stability and length of MCMC sampling trajectories for an EBM can easily be controlled by adjusting the sampling phase of Maximum Likelihood (ML) learning. This can yield model capabilities not obtainable by other kinds of unsupervised models.
we do for EBMs, because sampling not used when training
C COOPERATIVE LEARNING FOR GENERATOR NETWORK
In this section we briefly review the Maximum Likelihood framework for cooperatively learning the generator in tandem with the EBM (Xie al. 2018). The generator model assumes joint distribution (X, 2) of images X and latent signals Z given by
7 N(O,T) T) and X|Z 7(1(Z;0),~21)
15

---

Under review as a conference paper at ICLR 2022
for generator network with weights @ and Gaussian parameter controls the spread of X around g(Z; (). Given iid. pairs {X,, Zz fa the observations have a joint negative log probability
- lo 2 20 = |x 9(2: ((2:0))1//22)) + 2 2 +
In practice, the X; are shortrun MCMC samples from an EBM and the Z₁ are unobserved. The full formulation of cooperative learning involves latent MCMC using the conditional distribution p[ 6) to infer each Z, In practice, since the X; are shortrun samples that are paired with Z; through the process of rejuvenation, one can roughly approximate the latent MCMC process with an identity function and obtain gradient ( We find that this works as well or better than including a latent sampling step on Z, and is consistent with the original implementation from Xie et al. (2018).
Figure 5 iillustrae the importance of including a persistent image bank in the cooperative learning framework. Shortrun samples from the generator in standard cooperative learning have low diversity and it is difficulty for the generator to break the symmetry of its activations early in training. This can lead catastrophic instability without auxiliary techniques like batch norm help break the symmetry. Incorporating persistent states from prior EBM updates when updating the generator helps the shortrun distribution achieve health diversity that enables stable and effective learning.
D COOPERATIVE- ALGORITHM FOR SHORTRUN LEARNING
Algorithm Cooperaaiiv--Persiteen Hybrid Learning
Require:
EBM (a;;)), generator g(2; Φ) Langevin noise 7), number of shortrun steps K, EBM optimizer hy, generator optimizer hy. initial weights Og and do. rejuvenation probability p. max update rounds w. number of training iterations T.
Natural images { 3M [
Ensure: Learned weights 0r and ér.
Initialize bank of random latent states (2.)
Initialize image bank (X, J from generator using X, =
for 1 < 1 < 1'' do
Select batch [X,+] Jim from data samples )
Get paired batches (7,) and 1 from [7.)-= and {x, )
Update ( with K Langevin steps to obtain negative samples (X, k
Get learning gradient 1 (t) using (4) with samples {Xxt Jem and (8, Ho..
Update O2 using gradient (,) and optimizer fug.
Get learning gradient A (1) using 1 with samples {2 and (8, 1
Update de using gradient A! and optimizer hg.
Rejuvenate each 24 from latent distribution with probability p. Also rejuvenate states 2, for which Xs has been updated more than times.
If 7 is rejuvenated, rejuvenate X, = g( Zs: sbe).
Return B end for
{= to [ and (X, ) to [X, JM by overwriting previous states.
E BACKGROUND ON EBM DEFENSE
The most popular method for adversarial defense is adversarial training (AT) (Madry et al. 2018 Wong et al. 2020; Shafahi et al. 2019) which aims to train classifier to correctly predict adversarial samples within small ball around a natural input. Another popular defense method is randomized smoothing (Cohen et al. 2019). which adds Gaussian noise to images to remove adversarial signals before classification. Both of these approaches modify classifier training. Although many methods have been proposed to secure a naturally trained classifier, most have been broken by stronger attacks. A recent method that has been shown to secure a natural classifier is Langevin sampling with an EBM (Hill et al. 202T).
The EBM defense uses a classifier trained with labelled natural images and an EBM trained with unlabelled natural images. The two networks are trained independently, which is key advantage of
16

---

Under review as a conference paper at ICLR 2022
EBM defense adversarial training and randomized smoothing. Since the EBM is independent of the classifier, EBM defense has the potential to secure many classifiers across diverse tasks with single defensive model, while existing methods typically tailored to single method. Starting with naturally classifier trained on natural images (a)), we define its robust counterpart as
F(x) (T(O))(((())
where (xx) a random variable representing K steps of the Langevin transformation 5 initialized from state We cannot evaluate Fx)) directly approximate it using
1 Fn( H Σ ((x) where in T(x) i.i.d.
where f() is forward pass of our classifier return logits and where the accuracy of approximation is driven by the number of replicates Н. Mcaningful evaluation of adversarial defenses must be based on adaptive attack methods which are aware of both f(x) and T(x). For our attack we use the BPDA+EOT formulation from (Athalye et al. 2018
Tramer al. 2020) to obtain the attack gradient
Mat
Mat
ABPDA-BOT (I,
Σ VoL
Σ f(a)).
in ~ ((x) i.i.d.
(8)
Hady h=11
Hdv
1=1
which we use in the standard PGD) framework to generate adversarial examples. Algorithm gives sketch of the defense evaluation.
Algorithm/ 2 EBM Defense Algorithm
Require: Natural images (Ith 1m-1: EBM ((;;), classifier f. Langevin noise 7), attack replicates Hadv defense replicates Hart, loc radius e, attack step size, a, Langevin steps K
Ensure: Defense record (Dm Jm=1 for each image initialized as ones.
for < < M do
select( Xx, y4 from batch
Randomly initialize adversary Xo inside Loo ball around X; do
for S
arg max, | [F (8G-1)Je
A, ARDAAFOT (X--1, M1)) if cy * yı then
é = arg max, [F (&
if c F then
D. 0
end
end
X, = PGD(XX-1,AA,..0
end for
end for
F EBM DEFENSE EXPERIMENT DETAILS AND DIAGNOSTICS
Table 5: Defense for Lo0 against high-power whitebox attacks ImageNet and CIFAR-10.
Dataset
Nat
Adv
Hadv
Haef
samples
ImageNet
0.683
0.38
32
64
1600
CIFAR-10
0.858
0.64
48
128
5120
To verify the integrity of our results and CIFAR- 10 compared to our standard evaluation. While using these resources for all attacks is infeasible in practice, want ensure defense maintains robustness attacker resources
attack with heavily increased resources for ImageNet
17

---

Under review as a conference paper at ICLR 2022
Algorithm 3 Training midrun sampler for EBM defense
Require: Natural images { EBM (2; 0), frozen pre-trained generator g(z), Langevin noise Langevin steps K, EBM optimizer hu, initial weights 80. rejuvenation probability p. number of training
iterations T'
Ensure: Weights 0r defensive EBM.
Initialize image bank {X; hi=1 from generator using X, = 92)
for S t S 7 do
Select hatch {X²+) from data samples {orth Fm=11
Get negative sample batch {-K,,u from {x, hi=1-
Update 18 with K Langevin steps 5) to obtain negative samples (X, lo
Get learning gradient (1) using 4 with samples (X; {X;]) J6=1 and [X,] ]om
Update 6g using gradient A and optimizer hy.
Rejuvenate each Xx. from a pretrained generator g with probability p. Return (X, 1 to {X; ) by overwriting previous states.
end for
increase. As shown in Table our benchmark accuracy remains consistent when we increase the number of attack steps (from 50 to 200) and EOT attack replicates (Hadv).
We also demonstrate results over varying numbers of langevin steps K. We can see in Fig 6 that our sampling trajectory for defending imagenet at K 200 is reasonable to achieve high natural image classification as well as robustness. On CIFAR- 1-10 we are able reduce the number of langevin steps from 1500 in Hill et al. (2021) to K 500, which greatly reduces our compute overhead and demonstrates that a non-convergent model can outperform a convergent model (+ (+13% robustness) while using less steps.
EBM Defense Accuracy
EBM Defense Accuracy
Langevin Steps (Imagenet, /. norm, == 2/255)
Langevin Steps (CIFAR-10, (. norm, z 8/255)
0.8
0.8
0.5-
accuracy
0.6
0.3
0.4
0.1
natural accuracy J_ robustness
0.2
natural accuracy la robustness
0.0
200
600
año
250
1000
1750
Langevin steps K
Langevin steps K
Figure 6: Accuracy over varying numbers of langevin steps K for both the CIFAR-10 and Imagenet experiments. For CIFAR-10 10 we used Hadv = 24 in these experiments to compare against Hill et al. (2021) so the accuracy reported here is slightly higher than in table 2 that uses H 48
G IMPORTANCE OF LEARNING RATE ANNEALING
This section demonstrates the importance of learning rate annealing for learning a robust energy landscape. We repeat the midrun and longrun learning experiments for CIFAR-10 except that we never anneal the learning rate. We then sample with the models for 1,000 steps for the model trained with the midrun method and 100K steps for the model trained with the longrun method. The results in Figure 7 show that lcarning rate annealing is essential for stabilizing both midrun and longrun trajectories.
The importance of annealing can be understood as follows. If the EBM is being updated with a very low learning rate, then samples from recent EBM snapshots can function samples from the current EBM. the case of midrun trajectory, annealing allows the model to robustify trajectories that are approximately long as the lifetime of persistent sample between rejuvenation. In the case of
18

---

Under review as a conference paper at ICLR 2022
Figure 7: Ablation study showing the importance of annealing. Left: Samples from non non-annealed model trained with the midrun method after 1500 MCMC steps. Right: Samples from a non-annealed model trained with the longrun method ater 100K MCMC steps. MCMC samples were initialized from data. This shows that rejuvenation of the midrun trajectories from data and the separation of longrun samples into burn-in and update banks alone is not enough. Annealing ensures that samples from past EBMs function as approximate samples from the current EBM, since the weights are changing very slowly.
longrun learning. annealing allows the burnin samples to approximately reach the model steady- state before they are included in the update bank. This allows the persistent samples in the update bank to function as approximate steady-state samples from the current EBM, leading to proper modeling of probability mass.
H PRIOR EBM FOR LONGRUN LEARNING
This section discusses the choice of training longrun EBMs by using a fixed shortrun EBM as prior distribution. Intuitively, the prior EBM learns reasonable but imperfect approximation of the energy landscape that we know will have energy basins which leak low-energy states. Using a fixed misaligned prior EBM allows the EBM which is being actively updated to focus on learning landscape features which were not correctly learned by the prior EBM. In particular, our longrun allows the actively updated EBM to focus on sealing the leaky energy basins of the prior EBM so that the full model learn the correct distribution of probability mass. By separating the burn-in and update banks and longrun learning, we guarantee that the prior EBM alone would lead to extreme oversaturation by the time a state reaches the update bank unless the actively updated EBM is countcracting the oversaturation tendency to preserve realism. This additional effect of directly driving states towards the oversaturated regions in the absence of correcting force leads to improved stability for longrun models learned with a prior EBM compared to models learned without a prior EBM. The technique has precedent in ResNets. In both cases, it is easier to learn an approximate direct model first and then to approximate the residual that remains.
I COMPARISON TO SCORE-BASED DENOISERS, NORMALIZING FLOWS, AND DIFFUSION MODELS
The score-based model from (Song & Ermon 2020) and its annealed Langevin dynamics process
has recently been used for purifying adversarial signals (Lee & Oh 2021 Yoon al. 2021 l) One approach is add noise and the using the score model to denoise (Lee Oh 202 l). The robustness of this method is upper-bounded by standard randomized smoothing (Cohen al. 2019). Another approach is to initiate the Langevin process of the score model from a natural image as done in the FBM defense. A score model can be used in langevin process T(x) that is direct analogue the
19

---

Under review as a conference paper at ICLR 2022
Score-based Generative Model Defense Accuracy vs. Langevin Steps (Cifar-1o. 8/255)
natrual accuracy L robustness
k=0
k=10
k 25
ACCuraCy
06
k= 50
04-
100
02-
k =
k =
200
Langevin steps
Figure 8: Score-based Langevin experiment the CIFAR- dataset. Left: Accuracy of natural and adversarial images resulting from a BPDA+EOT defense using a score-based model with an annealed langevin purification method for 125 samples over varying steps. Right: Samples received from this annealed langevin diffusion process over the same sampling lengths.
EBM langevin process in 5 and (8). Given a score model Sz) for a low noise value a, one can define the Langevin equation
7 X, = Xx-1 (Xi-)) ZZ
where So(x) ~ V log ((x) since a is low. In both cases, the score-based model is trained with the loss:
1 Σ E (Blm)
|
+ 3 o,
(())= 2L
where 4o, (ā|z) represents adding Gaussian noise of standard deviation o, an image r sampled from
We experimented with this process as a defense mechanism by selecting the smallest trained noise value a = 0.01 and using the Langevin process as a method to purify adversaries. We evaluated this method using our BPDA+EOT attack framework over different numbers of Langevin steps during purification. In contrast to reports from Yoon et al. (202T), we were unable to obtain any significant defense using a pretrained score model when initializing sampling directly from adversarial or natural images. In Figure one can see that the score-based based model drives natural images toward saturation quickly, leading to a sharp decrease in natural classification that undermines the possibility of robustness from sampling. Since the score-based model does not perform sampling during training, and one cannot adjust the stability of its sampling process as we do in this work. While it is not immediately clear how to overcome this problem, we believe that defense with score model is possible and we hope that our observation lead to efforts to stabilize the sampling paths of score models as we do for EBMs in this work.
To underscore our claims about the difficulty of calibrating the probability mass a density model, we further investigate longrun MCMC samples from a normalizing flow and diffusion model. We find that the normalizing flow from the GLOW model (Kingma & Dhariwal 2018) and the recovery likelihood diffusion model (Gao et al., 2020b have misaligned steady-states
well. This shows that the problem of improper density estimation extends well beyond the EBM. Tractable density modeling with a normalizing flow does not prevent steady-state misalignment. These experiments corroborate our claim that log likelihood experiments in previous works are not able to detect the misaligned probability mass of many prior models. We believe that the calibration of the model steady-state is currently best diagnosed with longrun MCMC sampling because the distribution of longrun MCMC samples represents the probability mass of the model.
Figure 9 (left) displays initial and final states from GLOW model density after 100K sampling steps. Despite the fact that the GLOW model has a fully tractable density, is unable to learn a valid distribution of probability mass. 9 (right) shows initial and final samples from the Recovery Likelihood T6 model after 100K steps of the conditional model at the lowest noise value. We observe the same oversaturation for the conditional density as for the unconditional density of standard
20

---

Under review as a conference paper at ICLR 2022
Figure 9: Left: MCMC samples after 100K steps using GLOW model (Kingma & Dhariwal 2018) trained on CIFAR-10 10. Right: MCMC samples after 100K steps using conditional recovery likelihood model (Gao et al. 2020b) trained on CIFAR-10. 10. MCMC samples were initialized from data samples. Neither model can correctly approximate the distribution of probability mass for the data density. The problem of steady-state misalignment extends beyond EBMs to many other generative density models. We tried several different temperatures close to for the GLOW model and found equivalent results.
EBM. Code for the TIK model that the authors evaluate in their longrun experiments is not released so we have not yet been able to directly test their results. The TIK model is equivalent an EBM version of the score model in Figure [ which we have shown has a misaligned steady-state. Further, note the longrun experiments with the TIK model very misleading because the experiments use 100 steps with 1000 distinct conditional models and claim this a longrun evaluation of 100K steps. The correct evaluation is to use 100K steps single conditional model. We strongly believe that the Recovery Likelihood model as originally presented has a misaligned steady-state like many other methods. We hope that the observations in our work can lead efforts to stabilize the sampling trajectories of many existing models.
J REGARDING DENSITY ESTIMATION AND LOG LIKELIHOOD
In this work, we are primarily interested in learning density models that assign the majority of probability mass in realistic regions of the image space. This goal is, surprisingly, distinct from the goal of likelihood maximization. In particular, one can achieve high likelihood with mixture models where only an infinitesimal portion of mass is assigned a mixture component that approximates the true density, while the majority of probability mass is assigned to a degenerate distribution (see Theis et al. (2016), "Great Log Likelihood but Poor Samples"). The steady-state of MCMC sampling with such mixture distribution would concentrate the degenerate distribution even though the structures of the true density exist in high-energy regions. Therefore, log likelihood cannot detect if a model has assigned probability mass in realistic region of the image space. Recent observations have shown that this situation is not just hypothetical (Nijkamp et al. 2020). Non-convergent EBMs are practical examples, since these models consist of mixture of partially formed high-energy energy basins enabling effective shortrun synthesis in realistic regions of the image space and much lower- energy basins in unrealistic image regions that dominate the probability mass. The same misaligned landscape structure extends beyond EBM to RBMs (Decelle et al.) (202]). score models, normalizing flows, and diffusion models (see Appendix D. Log likelihood can only indicate the presence of energy landscape features that are similar to the ground truth landscape, but it cannot detect whether these features will leak into lower-energy basins that represent the true mass distribution. Like shortrun sample quality, high log likelihood is often misleading false evidence that model density concentrates on realistic images.
Overall, we believe that FID using true samples from the EBM and data samples is an appropriate, if rough, measure successful density modeling. The subtlety involves generating true samples from the EBM. We believe that longrun MCMC sampling is the most practical and principled tool for obtaining approximate samples from the EBM steady-state, and our investigations show that density modeling with properly calibrated probability mass is currently most effectively accomplished using MCMC- bbsce EBM training. Though we use the FID metric, our density modeling experiments are best described as realism preservation experiments rather than image generation experiments.
21

---

Under review as a conference paper at ICLR 2022
K LONGRUN LEARNING ALGORITHM
Algorithm 4 Training longrun sampler for density estimation
Require:
Natural images { 1 FBM (x;; 0), frozen pre-trained generator g(=) Langevin noise Langevin steps K, EBM optimizer lus, initial weights On. update threshold D, number of training iterations T.
Ensure: Weights Or for EBM with stable longrun samples. Initialize burn-in image bank {X; ) and update image bank {X;+) Ji=1 from generator using X, = =
Initialize update counts {de 1 using di Unif(((.....
for t do
Select batch {Xxt 18=1 from data samples { mm=
Get initial sample batch 1... from {X;)a
Jim and {Xp,utom from {X, hmm
Get counts d, corresponding to samples {Xto)
1B
Update {X with K Langevin steps to obtain negative samples (X, 1e-
Update {X6.01==1 with K Langevin steps
to obtain updated burn-in samples { (X; 8
Get learning gradient A using {4 with samples (X2) and (X, 10=1- Update 8, using gradient A and optimizer hv. Update the burn-in count de +- do +1
for 1 & b S B do
if de D then
Randomly overwrite one X5 using Xç-
Rejuvenate Xu from generator and do end end for Return {Xi,1em from (X;) and (8, 1o to (X, aa by overwriting previous states. end for
22

---

Under review as a conference paper at ICLR 2022
L SHORTRUN SYNTHESIS RESULTS
Figure 10: Shortrun samples from CIFAR-10 10 EBM at resolution 32 32.
23

---

Under review as a conference paper at ICLR 2022
120
Figure 11: Shortrun samples from Celeb-A EBM resolution 64 64.
24

---

Under review as a conference paper at ICLR 2022
Figure 12: Shortrun samples from ImageNet EBM at resolution 128 128.
25

---

Under review as a conference paper at ICLR 2022
M
LONGRUN SYNTHESIS RESULTS
CIFAR-10, 100K steps
CIFAR-10, 1M steps
Celeb-A, 100K steps
Celeb-A, 1M steps
ImageNet, 100K steps
ImageNet, 1M steps
Figure 13: Longrun samples at 100,000 steps and extremely longrun samples at million steps for EBMs trained on three datasets. Our initialization is able preserve high degree of realism over the first 100K steps, and a reasonable degree of realism over very long trajectories. While oversaturation and distortion is noticeable for some samples using IM steps, many samples have reasonable appearance and there is high diversity. Our method makes significant progress towards aligning longrun samples with high-quality samples from training ensure that the model is valid density.
26

---

Under review as a conference paper at ICLR 2022
N TABLES OF EXPERIMENT PARAMETERS
This section gives a full list of experiment parameters. Some notable minor additions beyond what are discussed in the text fixed MCMC temperature parameter regulate gradient strength the beginning of training, a gradient clipping parameter to restrict the magnitude of EBM and generator weight updates. We only use gradient clipping for shortrun experiments.
We use the annealing schedule
Vanneal [110-* 0), (0000,)
50000), (10-6, 75000), (10-², 100000), (10- 125000)]
in all experiments that involve annealing, where the pairs denote learning rate and model update step at which that learning rate is first used. We denote this schedule Tanneal in the tables.
Shortrun Training
Dataset
Celeb-A
CIFAR-10
ImageNet
Training Steps
150000
100000
300000
Data Epsilon
lc-2
1e-2
1e-2
EBM LR
le-4
le-4
le-4
EBM Optimizer
Adam
Adam
Adam
EBM Gradient Clip
50
Langevin Epsilon
3e-3
5e-3
3e-3
MCMC Steps
75
100
50
Rejuvenation Probability
0.5
0.5
0.5
MCMC Temperature
le-6
le44
1e-7
Max Update Rounds
Persistent Bank Size
10000
10000
10000
Generator L.R
le-4
le-4
5e-5
Generator Optimizer
Adam
Adam
Adam
Generator Gradient Clip
Generator Batch Norm
No
Yes
No
Shortrun Evaluation
Dataset
Celeb-A
CIFAR-10
ImageNet
Langevin Epsilon
3e-3
5e-3
3e-3
MCMC Steps
300
350
320
MCMC Temperature
le-6
le-4
1e-7
27

---

Under review as a conference paper at ICLR 2022
Defense Training
Dataset
CIFAR-10
ImageNet
Training Steps
150000
150000
Data Epsilon
2.5e-2
3e-2
EBM LR
Janneal
aannal
EBM Optimizer
Adam
Adam
Langevin Epsilon
1.25e-2
2e-2
MCMC Steps
100
100
Rejuvenation Probability
0.05
0.05
MCMC Temperature
le-4
le-5
Persistent Bank Size
25000
10000
Defense Evaluation
Dataset
CIFAR-10
ImageNet
Adversarial Steps
50
50
Adversarial Epsilon
255
255
Adversarial Eta
255
255
EOT Attack Reps
48
16
EOT Defense Reps
128
64
Langevin Steps
500
200
Langevin Epsilon
1.25e-2
2c-2
MCMC Temperature
1c-4
1e-5
Longrun Training
Dataset
Celeb-A
CIFAR-10
ImageNet
Training Steps
250000
250000
250000
Data Epsilon
2e-2
2c-2
2e-2
EBM LR
Tanneal
Yanneal
Yanneal
EBM Optimizer
Adam
Adam
Adam
Langevin Epsilon
le-2
le-2
le-2
MCMC Steps
100
100
100
Burn-in Update Rounds
750
750
750
MCMC Steps Burn-in
100
100
100
MCMC Temperature
lc-4
lc-4
le-5
Tau
1.5e-1
1.5e-1
1.5e-1
Persistent Bank Size
15000
10000
15000
Burn-in Bank Size
1000
1000
1000
Prior EBM
Dataset
Ccleb-A
CIFAR-10
ImageNet
Training Steps
150000
150000
150000
Data Epsilon
2e-2
2e-2
1.5e-2
EBM LR
lc-4
lc-4
lc-4
EBM Optimizer
Adam
Adam
Adam
Langevin Epsilon
le-2
1e-2
1e-2
MCMC Steps
100
50
100
Rejuvenation Probability
0.2
MCMC Temperature
le-4
1e-4
1e-5
Tau
1.5e-1
1.5e-1
1.5e-1
Persistent Bank Size
10000
10000
10000
28


Reviews:
[{"review_id": "ukZgSJNnh5h", "reviewer": "Reviewer_8SkN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\nThe paper investigates training heuristics for energy-based models.\n\n**Weaknesses**\n\nI am not convinced by the argument that the performances are \"state-of-the-art\". \n- In Table 1, the proposed EBM method clearly does not outperform GANs and recently proposed diffusion / score-based generative models. \n- In Table 2, the number of *ours* has a typo of 0.0.566 in CIFAR10 and has worse natural accuracy than the EBM method in Hill et al., and ImageNet results are clearly not outperforming baselines (although the state-of-the-art claim avoids this). \n- There are no density estimates on the test set, only FID, since the method does not discuss how to estimate the partition function.\n- We have yet to discuss the amount of compute and time needed to perform training and inference with EBMs, which are quite slow as well.\n\nThe paper does not have many technical novelties. Most of the discussion is about the usage (or not) of persistent initialization and pretrained generator for \"rejuvenation\". The \"rejuvenation model\" is so important in all three cases, that a reasonable alternative is to use a good \"rejuvenation model\" and avoid EBM entirely. The heuristics themselves might be helpful in EBM learning, but it is questionable why we would adopt an EBM in the first place if it does not perform well on image generation/density estimation compared to other models (GANs, flow models, diffusion models). \n- For example, a pretrained SNGAN is used to rejuvenate EBMs (section 4), yet the model itself outperforms EBMs in terms of FID (see Table 1).\n\nIf the method claims \"density estimation\", then we should expect to see results other than FID (since that is image generation), even if partition function is not possible to compute, it can still help to see if density estimation have other uses, such as out-of-distribution detection.\n\n## Post rebuttal response\nI appreciate the authors taking the time to discuss and address my concerns. Unfortunately, I will keep my scores as is.\n\n- Similar to reviewer pepN, I am not entirely convinced about using FID in the \"density modeling\" section. Sure, density modeling does not require partition functions, and both FID and likelihood are flawed as measurements of image quality, but the argument used in this paper can also be used in GANs (it is also modeling some kind of density, and you can evaluate FID with it)? EBMs are more expressive than GANs in the sense that they also give un-normalized densities, but the paper never used these densities for any meaningful tasks (like out-of-distribution detection). The rebuttal also mentions that \"(ours) represent a significant departure from mainstream approaches for learning and evaluating density models\"; I don't see how evaluating FID is a significant departure, most EBM-based work after 2019 already report FIDs? \n- Existing work has shown that diffusion models can beat GANs on ImageNet (even with higher resolutions): https://github.com/openai/guided-diffusion; this includes unconditional and conditional models.\n- The \"technically novel\" bit in this paper seems to be using pretrained generators for rejuvenation, and if there are models that already beats EBM in image generation, why would we train and use another EBM instead? The rebuttal mentions that \"Alternatives such as variational approximation yield much worse distributional approximations than MCMC\", which might be true in principle, but can be quite far from what happens in practice (since MCMC can have slow mixing speed). Again, diffusion models are inspired by variational inference and seem to have quite good FIDs.\n\nAll in all, I find that the paper and rebuttal make some claims in favor of EBMs (and the paper itself) that I cannot fully agree with.\n", "summary_of_the_paper": "The paper discusses learning strategies for energy-based models, short sampling for image generation, midrun sampling for adversarial defense, and longrun sampling for density estimation. The paper claims these methods achieve significant performance gain across the three applications and achieved state-of-the-art performances.", "summary_of_the_review": "The paper spans various different problems, but makes relatively little contribution to each of them. The claims over empirical performances are inflated. The EBM-based methods do not have an advantage over score-based / diffusion generative models in terms of generation and likelihood evaluation (which also avoids estimating partition functions).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VDbyp85m-o", "reviewer": "Reviewer_PEpN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThis paper addresses a large issue in the EBM space. Current methods for training these models are difficult to tune, unstable, and slow compared to other generative modeling approaches. The proposed fix combining ideas from CoopNets and PCD may provide a best-of-both-worlds approach to training EBMs while adding no overhead on top of either of these training methods. \n\nFurther, this paper correctly brings attention to many real issues in the EBM space such as the sample quality / density model quality trade-off. As well, the paper addresses adversarial robustness which appears to be a powerful and promising of EBMs. While I am not an expert in adversarial robustness (and cannot make super-strong claims about the correctness of these experiments), it appears to me that the robustness results give a solid improvement over [6] and many other techniques purely meant for this task.\n\n\nWeaknesses:\n\nWhile the authors provide some high-level intuition for their proposed modifications to CoopNets/PCD, they do not provide any theoretical justification for their proposed approach. Have you considered other ways to improve the diversity of the CoopNet samples? Could this be due to the approximations that are made with the standard CoopNet learning algorithm (which the authors mention)? You could regularize entropy as in [5, 4]. I feel the paper would be made much stronger if more care was placed on the details and more theoretical justification was given.\n\nThe experimental details in the paper are quite scant. The authors do not mention anything about the optimizers used, the learning rates, the batch sizes, or training training time. As such, the results in this paper are not reproducible. I am aware the authors have included code, but these details should be accessible from the paper (at least in the appendix). Further, from an investigation of the code, there appear to be a number of additional experimental hyper-parameters (such as gradient clipping and energy tempering) which are not once mentioned in the paper. These modifications to the training objective can be very responsible for the success/failure of EBMs when training in this way and there has been a good deal of work discussing it.\n\nThis work is attempting to improve stability and success of EBM training. Many of these tricks (or hacks depending on your point of view) are there to accommodate for many of the issues that this paper proposes to address (such as high variance of negative sample gradients from training on newly resampled examples) so it should be made clear which, if any, of these tricks are no longer needed. \n\nI have some issues with the evaluation used in the long-run sampling for density estimation section. The authors claim early in the paper that sample quality can be a misleading indicator of model quality (where quality is evaluated by maximum likelihood), which I completely agree with. I was then disappointed to find the only quantitative result in this action was FID which is a qualitative measure of sample quality. If the authors want to argue that their training procedure learns a better density model then there are many alternative evaluations which could be used -- and have been used in recent EBM work. You could use AIS/RAISE to estimate upper/lower-bounds on likelihood as in [1, 2]. Or, you could use your training procedure to train a tractable likelihood model and then evaluate using the model’s known likelihood as in [3, 4].\n\n\nMinor: Typo in Table 2. \n\n[1] Du, Yilun, and Igor Mordatch. \"Implicit generation and generalization in energy-based models.\" arXiv preprint arXiv:1903.08689 (2019).\n[2] Gao, Ruiqi, et al. \"Learning energy-based models by diffusion recovery likelihood.\" arXiv preprint arXiv:2012.08125 (2020).\n[3] Song, Yang, et al. \"Sliced score matching: A scalable approach to density and score estimation.\" Uncertainty in Artificial Intelligence. PMLR, 2020.\n[4] Grathwohl, Will, et al. \"No MCMC for me: Amortized sampling for fast and stable training of energy-based models.\" arXiv preprint arXiv:2010.04230 (2020).\n[5] Dieng, Adji B., et al. \"Prescribed generative adversarial networks.\" arXiv preprint arXiv:1910.04302 (2019).\n[6] Hill, Mitch, Jonathan Mitchell, and Song-Chun Zhu. \"Stochastic security: Adversarial defense using long-run dynamics of energy-based models.\" arXiv preprint arXiv:2005.13525 (2020).\n\n----------Post author response------------\n\nI appreciate the authors responding to my feedback and thank them for the changes they have made to the paper. Unfortunately, I do not find their arguments convincing regarding density estimation and the use of FID to evaluate the method. As well, if the authors were able to demonstrate that their method simplifies EBM training while offering similar quality results, this would be compelling, but I do not feel this was done in the work. The proposed method is more complicated than CoopNets and PCD and provides additional hyper-parameters to tune. If the authors were able to make a compelling argument that their method allows us to learn a better density model, then I would support acceptance of this work, but I do not believe this was done. For this reason, I will keep my score the same. There are some interesting ideas presented in this work, but I do not feel the results presented demonstrate that they are a sufficient contribution to the field for acceptance. \n", "summary_of_the_paper": "This paper discusses 3 applications for EBMs: image generation, adversarial robustness, and density modeling. The authors discuss how the standard methods for training EBMs do not adequately address the latter two of these applications. For this reason, they propose a new method which combines ideas from 2 popular EBM training methods; CoopNets, and PCD, to alleviate some issues with both methods. The authors present results on the 3 tasks mentioned and their method appears favorable compared to other EBM variants.", "summary_of_the_review": "This paper presents a new method for training EBMs which combines features of two popular training approaches: CoopNets and PCD. While the proposed method makes sense, and appears to work, there are considerable issues with the method’s evaluation, experimental details, and theoretical justification. Thus, in its current form, I do not advocate for its acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PZnZomD5GHp", "reviewer": "Reviewer_oMaP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to follow. The empirical performance looks good compared to the baseline models (although the fairness requires further investigation).\nThe weakness of the paper is that the contribution seems to be incremental, as most of the techniques used in this paper have been proposed somewhere else. Below are some of my concerns.\n\n1. For image generation, why not use a pretrained generator? And if the generator already provide high quality samples (e.g., SNGAN), what is the purpose of using EBM after it? Did the baseline EBMs use generator initialization? Also, I did not see a comparison to Xie et al. (2018).\n\n2. How does the pretained generator initialization compared with data samples? How important if annealing? Is annealing applied to other baseline methods? Lack Ablation studies here.\n\n3. Where is the density estimation result for longrun EBM? ", "summary_of_the_paper": "This work presents different MCMC initialization techniques for training EBM with differerent lengths, for the purposes of image generation, adversarial defense, and density estimation respectively. More specifically, for shortrun image generation, the author proposed a hybrid persistent cooperative initialization that mitigate the lack of diversity issue of EBM learning with generator initialization. For midrun adversarial defense, the author proposed a pretrained generator rejuvenation to scale up EBM defense. For longrun density estimation, the author proposed rejuvenation methods and regularization trick to correct oversaturation.", "summary_of_the_review": "Overall, the paper provide some interesting empirical findings for EBMs with different MCMC lengths. However, current experiment results and lack of novelty make it a bit under the bar of ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6N9PUwHWMzq", "reviewer": "Reviewer_gMv3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Image synthesis section comments:\n- It should be more clearly explained why batch-normalization in the generator isn’t enough to fix the issue.\n- Section 2.2 is confusing. Isn’t the persistent bank just over the distribution (x, z) factorized as p(x|z)p(z), where z is the latent vector and x is the generated image?\n\nAdversarial defense\n- “Annealing” in the final paragraph of 3.1 refers to “annealing the learning rate”, right?\n- Section 3.2 it’s unclear how the hyperparameters are specified. As I understand it, two of K, K_def, and p_rejuv must be specified. Which of the two are specified, and to what values?\n- Are the experiments justifying the reasoning given in the last paragraph of 3.1? How similar are the trajectories of length K late in training to actual trajectories of length K_def, when the learning rate is annealed?\n- In Figure 6, why does robustness decrease for large K? Shouldn’t larger K used during training lead to better performance when using K_def at test-time?\n- How do the results compare when using data samples instead of a pre-trained generator?\n\nDensity Estimation\n- First paragraph, Section 4.1\n   - “However, persistent learning without rejuvenation has shortcomings mentioned in Section 3” -> Section 2?\n- “Persistent samples that are newly rejuvenated (up to about 50K Langevin steps since rejuvenation, and possibly many more) cannot be approximate steady-state samples for any current known rejuvenation sources, including data, generators, and noise.”\n- I think this sentence needs more elaboration / definition of terms. My understanding is as follows: Samples in a bank that have undergone less than 50k langevin updates cannot be steady-state samples\n- What is 50k defined relative to? Is it supposed to be half of 100k, which is the number of steps being used to estimate the density?\n- What is the definition of “lifetime” Langevin updates? Does this refer to the number of Langevin updates applied to a sample between different training iterations (e.g. by being sampled from the persistent bank and its updated version back into the persistent bank)\n- “Samples in the newly rejuvenated bank that have been updated sufficiently many times will eventually replace samples from the bank used to update the EBM, at which point newly rejuvenated states will be added to the first bank.”\n  - “the first bank” refers to the bank for newly rejuvenated samples, correct?\n- How do the results compare when using data samples instead of a pre-trained generator?\n- Are there experiments measuring the quality of the EBM density?\n  - My understanding is that these experiments show that EBMs can be trained so that long-run MCMC samples (which are closer to samples from the true EBM density) are high-quality. Are there experiments measuring how good the density is directly? For example, experiments on out-of-distribution detection? A small tractable model on MNIST where the normalizing constant can be estimated, and compare the log-likelihood to exact density models, as well as exact samples to long-run MCMC samples?\n- The prior EBM used in eqn. 2 seems like a little bit of a hack. To me it sounds like it could just be “slowing-down” how fast MCMC converges before oversaturation. Are there any plots showing that MCMC achieves a wide-region of stability where the sample quality and diversity stays relatively constant?\n\nTypos:\n- Section 1 (Introduction)\n  - “an misaligned” (3rd paragraph)\n- Section 2.2 (Hybrid Persistent Cooperative Initialization)\n  - “uses paired latent and image states that a drawn from” (Figure 2 caption)\n- Section 4 (Longrun Sampling for Density Estimation)\n  - “these outcomes not equivalent” (2nd paragraph)\n  - “the lack scalable methods” (2nd paragraph)\n  - “by introducing an MCMC initialization can incorporate” (2nd paragraph)\n- Section 4.1 (Incorporating Rejuvenation in Density Estimation)\n  - “initializatin” (Figure 4 caption)\n  - “remain the the” (Figure 4 caption)\n  - “burnin bank” (Figure 4 caption) \n  - “until they have approach” (Figure 4 caption)\n  - text under eq. (2) does not explain U_0 term", "summary_of_the_paper": "The paper describes techniques to train EBMs according to the length of MCMC trajectories required. The paper explores three applications of EBMs: image synthesis, adversarial defense, and density estimation. Each of these applications uses a different regime of MCMC sampling length. An approach is described for modifying the initialization of MCMC both during training and at test-time to improve the performance in each of these applications.\n\nFor image synthesis, the paper proposes using a combination of persistent chains and cooperative learning. Persistent chains with rejuvenation can cause instability since negative samples can vary between freshly rejuvenated samples and longer run samples. However, no rejuvenation often leads to lack of sample diversity. Persistent banks don’t scale well to large datasets since they cannot capture all the variability of the larger dataset. Instead, chains can be rejuvenated from a fast generator model, which is trained jointly with the EBM, known as cooperative learning. The authors identify and fix a key issue with cooperative learning. The issue is that generators early in training are unable to generate a diverse set of samples that the EBM can refine. One fix is to use batch normalization in the generator. Instead, the authors propose using a persistent bank over the joint distribution of latent noise and generated images (where the latent noise is input to the generator), which the sampling process is rejuvenated from with the usual process.", "summary_of_the_review": "Overall the paper is well-structured and clear, and presents unique and interesting ideas for training unconditional EBMs for different applications. There are some places where clarity could be improved, and some additional experiments which I think might improve some of the points in the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a strategy to train EBMs according to the length of MCMC trajectories required. The paper covers three settings with the different length of MCMC: image synthesis, adversarial defense, and density estimation. The reviewers generally find that there are interesting ideas and promising results in the paper, but the paper is not ready to publish at its current stage. The argument regarding density estimation and FID evaluation is not convincing. The proposed method is also more complicated than the baseline methods (CoopNets and PCD), and we would need a stronger argument for the added complexity.
Decision: Reject