Guiding Energy-based Models via Contrastive Latent Variables
Authors: Hankook Lee
Jongheon Jeong
Sejun Park
Jinwoo Shin
Venue: ICLR 2023 notable top 25% (2023)
Keywords: energy-based model
contrastive representation learning

Abstract: An energy-based model (EBM) is a popular generative framework that offers both explicit density and architectural flexibility, but training them is difficult since it is often unstable and time-consuming. In recent years, various training techniques have been developed, e.g., better divergence measures or stabilization in MCMC sampling, but there often exists a large gap between EBMs and other generative frameworks like GANs in terms of generation quality. In this paper, we propose a novel and effective framework for improving EBMs via contrastive representation learning (CRL). To be specific, we consider representations learned by contrastive methods as the true underlying latent variable. This contrastive latent variable could guide EBMs to understand the data structure better, so it can improve and accelerate EBM training significantly. To enable the joint training of EBM and CRL, we also design a new class of latent-variable EBMs for learning the joint density of data and the contrastive latent variable. Our experimental results demonstrate that our scheme achieves lower FID scores, compared to prior-art EBM methods (e.g., additionally using variational autoencoders or diffusion techniques), even with significantly faster and more memory-efficient training. We also show conditional and compositional generation abilities of our latent-variable EBMs as their additional benefits, even without explicit conditional training. The code is available at https://github.com/hankook/CLEL.

Published as a conference paper at ICLR 2023
GUIDING ENERGY-BASED MODELS VIA CONTRASTIVE LATENT VARIABLES
Hankook Lee Jongheon Jeong Sejun Park Jinwoo Shin
LLG Al Research KKAIST Korea University
wew.ooo.eee@gggeeeeeeeee {jongheonj, j | (kaist.ac.kttttroooeeeeae www.nnparku0o0000..
ABSTRACT
An energy-based model (EBM) is a popular generative framework that offers both explicit density and architectural flexibility, but training them is difficult since it is often unstable and time-consuming. In recent years, various training techniques have been developed, e.g., better divergence measures or stabilization in MCMC sampling. but there often exists a large gap between EBMs and other generative frameworks like GANs terms of generation quality. In this paper, we propose novel and effective framework for improving EBMs via contrastive representation learning (CRL). To be specific, we consider representations learned by contrastive methods as the true underlying latent variable. This contrastive latent variable could guide EBMs to understand the data structure better, so it can improve and accelerate EBM training significantly. To enable the joint training of EBM and CRL, we also design a new class of latent-variable EBMs for learning the joint density of data and the contrastive latent variable. Our experimental results demonstrate that our scheme achieves lower FID scores, compared to prior-art EBM methods (e.g., additionally using variational autoencoders diffusion techniques), even with significantly faster and more memory-efficient training. We also show conditional and compositional generation abilities of our latent-variable EBMs as their additional benefits, even without explicit conditional training. The code is available https: : / /gthhb.. www/hankoooeeeceeeeeeoo http//haaokkkc.......
INTRODUCTION
Generative modeling is a fundamental machine learning task for learning complex high-dimensional data distributions Paata (x). Among a number of generative frameworks, energy-based models (EBMs, LeCun et al., 2006; Salakhutdinov et al., 2007), whose density is proportional to the exponential negative energy, i.e., pe (x) x exp(- (x) have recently gained much attention due to their attractive properties. For example, EBMs can naturally provide the explicit (unnormalized) density, unlike generative adversarial networks (GANs, Goodfellow al., 2014). Furthermore, they are much less restrictive in architectural designs than other explicit density models such as autoregressive (Oord et al., 201 6b;a) and flow-based models (Rezende & Mohamed, 2015; Dinh et al., 2017). Hence, EBMs have found wide applications, including image inpainting (Du & Mordatch, 2019), hybrid sooriminnativegeneraate
models (Grathwohl et al., 2019; Yang & Ji, 2021), protein design (Ingraham al., 2019; Du et al., 2020b), and text generation (Deng et al., 2020).
Despite the attractive properties, training EBMs has remained challenging; e.g., it often suffers from the training instability due to the intractable sampling and the absence of the normalizing constant. Recently, various techniques have been developed for improving the training stability and the quality of generated samples, for example, gradient clipping (Du & Mordatch, 2019), short MCMC runs (Nijkamp et al., 2019), data augmentations in MCMC sampling (Du et al., 2021), and better divergence measures (Yu al., 2020; 2021; Du et al., 2021). To further improve EBMs, there are several recent attempts to incorporate other generative models into EBM training, e.g., variational autoencoders (VAEs) (Xiao et al., 2021), flow models (Gao et al., 2020: Xie et al., 2022), or diffusion techniques (Gao et al., 2021). However, they often require a high computational cost for training such an extra generative model, or there still exists a large gap between EBMs and ofate-of-the-aar generative frameworks like GANs (Kang et al., 2021) 1) or score-based models (Vahdat et al., 2021).
Work done KAIST.

---

Published as a conference paper at ICLR 2023
Spherical Latert Latint-Variable EBM
Projection
X ~
px 2)
Add negative variables
~ Pdsta
2 n~ (2)
f0
go
(a] Learning joint density
DKt. (ptata (x, 2) [|po(xx, x))
Contrastive Latent Encoder
ag(
9g()
V1 (x))
Pilata X
Pdata (2|x)
LXT-Xem (Zi, Z2, (2') (2))
V2 = (x))
(b) Contrastive learning
f2
ho
Figure 1: Illustration of the proposed Contrastive Latent-guided Energy Learning (CLEL) framework. (a) Our spherical latent-variable EBM (fe. learns the joint data distribution Pdata (x, z) generated by our contrastive latent encoder hg. (b) The encoder he is trained by contrastive learning with additional negative variables ă ~ p2) Here, z, = has (t. a random augmentation, and sg(-) denotes the stop-gradient operation.
where i; ~ T denotes
Instead of utilizing extra expensive generative models, in this paper, we ask whether EBMs can be improved by other unsupervised techniques of low cost. To this end, we are inspired by recent advances in unsupervised representation learning literature (Chen et al., 2020; Grill et al., 2020; He et al., 2021), ) especially by the fact that the discriminative representations can be obtained much easier than generative modeling. Interestingly, such representations have been used to detect out-of- distribution samples (Hendrycks et al., 2019a;b), so we expect that training EBMs can benefit from good representations. In particular, we primarily focus on contrastive representation learning (Oord et al., 2018; Chen et al., 2020; He et al., 2020) since it can learn instance discriminability, which has been shown to be effective in not only representation learning, but also training GANs (Jeong & Shin, 2021; Kang et al., 2021) 1) and -uitof-dissributtin detection (Tack et al., 2020).
In this paper, we propose Contrastive framework for improving EBMs via contrastive representation learning (CRL). Our CLEL consists of two components, which are illustrated in Figure 1.
guided Energy Learning (CLELL), a simple yet effective
Contrastive latent encoder. Our key idea is to consider representations learned by CRL as an underlying latent variable distribution Pamaa (z) x). Specifically, we train an encoder hG via CRL, and treat the encoded representation z == h xx) as the true latent variable given data x, i.e., P (-) This latent variable could guide EBMs to understand the underlying data structure more quickly and accelerate training since the latent variable contains semantic information of the data thanks to CRL. Here, we assume the latent variables are spherical, i.e., ||z||| = 1, since recent CRL methods (He et al., 2020; Chen et al., 2020) use the cosine distance on the latent space.
Spherical latent-variable EBM. We introduce new class of variable EBMs pe (x,z) z) for modeling the joint distribution Pdaaa (x, z) generated by the contrastive latent encoder. Since the latent variables are spherical, we separate the output vector f == fo(x) into its norm ||/l|2 and direction f/|l/'l> 2 for modeling po (x() and po (z(x), respectively. We found that this separation technique reduces the conflict between pa (x) and pa (z) x) optimizations, which makes training stable. In addition, we treat the latent variables drawn from our EBM, ~ pe (z), as additional negatives in CRL, which further improves our CLEL. Namely, CRL guides EBM and vice versa.'
We demonstrate the effectiveness of the proposed framework through extensive experiments. For example, our EBM achieves 8.61 FID under unconditional CIFAR-10 generation, which is lower than those of existing EBM models. Here, we remark that utilizing CRL into our EBM training increases training time by only 10% in our experiments (e.g., 38 +41 GPU hours). This enables to achieve the lower FID score even with significantly less computational resources (e.g., we use single RTX3090 GPU only) than the prior EBMs that utilize VAEs (Xiao al., 2021) or diffusion-basee recovery likelihood (Gao et al., 2021). Furthermore, even without explicit conditional training, our
'The representation quality of CRL for classification tasks is not much improved in our experiments under the joint training of CRL and EBM. Hence, we only report the performance of EBM, that of CRI..
2

---

Published as a conference paper at ICLR 2023
latent-variabl EBMs naturally can provide the latent-conditional density po(x'z); verify its effectiveness under various applications: out-of-distribution (OOD) detection, conditional sampling. and compositional sampling. For example, OOD detection using the conditional density shows superiority over various likelihood-based models. Finally, we remark that our idea is not limited to contrastive representation learning and show EBMs be also improved by other representation learning methods like BYOL (Grill et al., 2020) or MAE (He et al., 2021) (see Section 4.5).
PRELIMINARIES
In this work, we mainly consider unconditional generative modeling: given a set i.i.d. samples (x(") Fi=1 (=1 drawn from an unknown data distribution Paata (x), our goal is to learn model distribution (x(x) parameterized by 0 to approximate the data distribution Paata (x). To this end, we parameterize
(x))) using energy-based models (EBMs), and incorporate contrastive representation learning (CRL)
into EBMs for improving them. We briefly describe the concepts of EBMs and CRL in Section 2.1
and Section 2.2, respectively, and then introduce our framework in Section
2.1 ENERGY-BASED MODELS
An energy-based model (EBM) is probability distribution on Rdx, defined as follows: for Rd
exp(-Eo(x))
/ exp(-Ea (x)))xx
Pa (x)
Ze = JR
Za
where (x) is the energy function parameterized by 0 and Zo denotes the normalizing constant, called the partition function. An important application of EBMs to find parameter 0 such that pe is close to Pdata- A popular method for finding such 0 is to minimize Kullback-Leibe (KL) divergence between Pdata and pe via gradient descent:
DKL. (Pasta (po) = - -xx [log Po (x)) + Constant,
(2)
DKL (Péata (pe) = E Ex- (x)] Ex~pp [x))
(3)
Since this gradient computation (3) is NP-hard in general (Jerrum & Sinclair, 1993), is often approximated via Markov chain Monte Carlo (MCMC) methods. In this work,
the stochastic gradient Langevin dynamics (SGLD, Welling & Teh, 2011), a gradient-based MCMC method for approximate sampling. Specifically, at the (t + 1)-th iteration, SGLD updates the current sample (()) to (((+1) using the following procedure:
x (1+1)
x(1) + 2
Vx Fxlogpo(<()))+e +E5(1), 8(1) ~N(0 I),
(4)
(x(()
where >> is some predefined constant, (0) denotes an initial state, and V denotes the multivariate normal distribution. Here, it is known that the distribution of x (weakly) converges to pe with small enough E and large enough T under various assumptions (Vollmer et al., 2016; Raginsky et al., 2017; Xu al., 2018; Zou al., 2021).
Latent-variable energy-based models. EBMs can naturally incorporate latent variable by specify- ing the joint density Po (x, 2) x exp(- Eo(x 2)) of observed data x and the latent variable z. This class includes number of EBMs: e.g., deep Boltzmann machines (Salakhutdinov & Hinton, 2009) and conjugate EBMs (Wu et al., 2021). Similar to standard EBMs, these latent-variable EBMs can be trained by minimizing KL divergence between Pdata (x) and pa pa(x) as described in (3):
V DEL (Psual Ipp) Ex-pan (x) (x)) Ex-po (x) L (x))
(5)
(x) V (x, 2) where (x) is the marginal energy, i.e., Eg (x) log ʃ exp(-Eol z) dz.
Ex (2) Eo 2)),
(6)
2.2 CONTRASTIVE REPRESENTATION LEARNING
Generally speaking, contrastive learning aims to learn a meaningful representation by minimizing
distance between similar (i.e., positive) samples, and maximizing distance between dissimilar (ie..
3

---

Published as a conference paper at ICLR 2023
negative) samples on the representation space. Formally, let hes R + Rdd be o-parameterized encoder, xx+ and (x, x_) be positive and negative pairs, respectively. Contrastive learning then maximizes sim ( (x), has (x+)) and minimizes sim(h,(x), (x.))) where sim(;,) :) is similarity metric defined the representation space Rd
Under the unsupervised setup, various methods for constructing positive and negative pairs have been proposed: e.g.. data augmentations (He et al., 2020; Chen al., 2020; Tian et al., 2020), spatial or temporal co-oceurrence (Oord et al., 201 8), and image channels (Tian et al., 2019). In this work, mainly focus popular contrastive learning framework, SimCLR (Chen al., 2020), which constructs positive and negative pairs via various data augmentations such as cropping and color jittering. Specifically, given a mini-batch B = ) SimCLR first constructs two augmented
views {v5) (3) =),) 1 (,2(1. for each data sample x(i) via random augmentations 155) TT
(i) Then, it considers (vl") V2
positive pair and (v (3) (R)) as negative pair for all l: # The SimCLR objective LSimCLR is defined as follows:
SSmCLL (B;
Σ
Σ
XeT (No(v)")) ( (hes (v(" (b((v()))))(ee122)77) (7)
2n
1-2
exp(sim 2, 2+)/T)
LNT.Xem (2, 24 {z- 7; log
exp(sim(z, (2))))) Σ. exp(sim(z zz )/T/
where v) u / /||/||////vl| is the cosine similarity, 7 is a hyperparameter for temperature scaling, and Xem denotes the normalized temperature-scales cross entropy (Chen al., 2020).
3
METHOD
Recall that our goal is to learn an energy-based model (EBM) Pa (x) x exp(-Eg(x)) to approximate a complex underlying data distribution Pdata In this work, we propose Contrastive Latent-guides Energy Learning (CLEL), simple yet effective framework for improving EBMs via contrastive representation learning. Our key idea is that directly incorporating with semantically meaningful contexts of data could improve EBMs. To this end, consider the (random) representation z (Z))( of x, generated by contrastive learning, as the underlying latent variable." Namely, we model the joint distribution Pdau (x, z) Pama (x) Pdma (z (z)x via latent-variable EBM po po(xx, z). Our intuition on the benefit of modeling Paatn (x, Z) is two-fold: (i) conditional generative modeling Pdata (x|z) given some good contexts (e.g., labels) of data is much easier than unconditional modeling Paua (x) (Mirza & Osindero, 2014; Van den Oord et al., 2016; Reed et al., 2016), and (ii) the mode collapse problem of generation can be resolved by predicting the contexts Paam xx) (Odena et al., 2017; Bang & Shim, 2021). The detailed implementations of Paata (z|x), called the contrastive latent encoder, and the latent-variable EBM are described in Section 3.1 and 3.2, respectively, while Section 3.3 presents how to train them in detail. Our overall framework is illustrated in Figure 1.
3.1 CONTRASTIVE LATENT ENCODER
To construct a meaningful latent distribution Paau Pz/x(z for improving EBMs, we use contrastive representation learning. To be specific, first train latent encoder hs Rd + Rd,, which is deep neural network (DNN) parameterized by Φ using a variant of the SimCLR objective LSimLLR (7) (we describe its detail in Section 3.3) with random augmentation distribution T. Since our objective only measures the cosine similarity between distinct representations, one can consider the encoder hs maps randomly augmented sample to a wnit vector. We define the latent sampling procedure z ~ Pamt(Z (z|x) as follows:
Pdata (2|x)
= )=
3.2 SPHERICAL LATENT-VARIABLE ENERGY-BASER MODELS
We use a DNN fo & - rh parameterized by 0 for modeling po (x, z). Following that the latent variable Pdaaa (dlaa(z/)) (z|x) is the unit sphere, utilize the directional information fo(x) fl
"Chen al. (2020) shows that the contrastive representations contains such contexts under various tasks.
4

---

Published as a conference paper at ICLR 2023
for modeling po(z)x)) while the remaining information (l is used for modeling pa (x). We empirically found that this norm corm-direction separation stabilizes the latent-variable EBM training. For better modeling pa (z(xx) we additionally apply directional projector go S
S to
fo(x)/ fol (x) which is constructed by a two-layer MI., followed by le normalization. We found that it is useful for narrowing the gap between distributions of the direction fxxx)/)lf((x))) mmiformly-distributt latent variable Pdata 2)) (see Section 4.5 and Appendix E for detailed discussion). Overall, we define the joint energy Lgg (x, z) as follows:
[x)]l2 and the
[1 - Ago ( 1 )
T
Ex(x, z) 2
(1fo(x))12
2,
(10)
Ex(x) = - log f
2)) = 2 = + Constant,
(11)
g4-1
where B 2 is hyperparameter. Note that the marginal energy Ex(x) only depends on (x)
since fgu exp(
z)dz is independent of x due to the symmetry. Also, the norm-based design does not sacrifice the flexibility for energy modeling (see Appendix F for details).
3.3 TRAINING
Remark that Section 3.1 and 3.2 define Paata(X, (x. z) and (x z), respectively. We now describe how to train the contrastive latent encoder h& and the spherical latent-variable EBM pe via mini-batch stochastic optimization algorithms in detail (see Appendix A for the pseudo-code).
Let ) be real samples randomly drawn from the training dataset. We first generate n samples (x(0) J==1 ~ (x) using the current EBM via stochastic gradient Langevin dynamics (SGLD) (4). Here, reduce the computational complexity and improve the generation quality of SGLD,,
two techniques: replay buffer to maintain Markov chains persistently (Du & Mordatch, 2019), and periodic data augmentation transitions encourage exploration (Du et al., 2021). We then draw latent variables from Pdam and pa: (0)) Pduta (z x()) and g(() po p x (*) for all i. For the latter case, we simply the mode of po (x)) instead of sampling, namely, ( go (fo(x) (fo(x)/|ll(( (x)]l2) Let and B =- (xx(1, ) be real and generated mini-batches, respectively.
B ===(6 (x((), )
Under this setup, we define the objective LERM for the EBM parameter 0 as follows:
CEBM (a, (B,
(x
Eo (x( a (Eo(x(0)2 ()())))))) (12)
where the first two terms correspond the empirical average of DKL (Pdata ( and is a hyperpa- rameter for energy regularization to prevent divergence, following Du & Mordatch (2019). When training the latent encoder he via contrastive learning, we use the SimCLR (Chen et al., 2020) loss
LSSCCLR (7) with additional negative latent variables (2(0)) ) To be specific, we define the objective LIE for the latent encoder parameter d as follows:
CLE(B,
1
LNT-Xeat
2 (1) (1)
3-j (z) (k)
(13)
2n
1=1.2
where LNT the normalized temperature-scalee cross entropy defined (8), is hyperparameter
for temperature scaling, (1) == (t, (x) and {1,50)] T are random augmentations. We found that considering (z()) as negative representations for contrastive learning increases the latent diversity, which further improves the generation quality in our CLEL framework.
To sum up, our CLEL jointly optimizes the latent encoder hig and the latent-variable EBM (fe. ge) from scratch via the following optimization: ming.e E [E B; 0, a, B) + LIF(B, B; Φ, 7)]. After training, we only utilize our latent-variable EBM (Jo- go) when generating samples. The latent encoder he is used only when extracting a representation of a specific sample during training.
For example, we found multi-head architecture for modeling (x)) and po (z|x)) makes training unstable. We provide detailed discussion and supporting experiments in Appendix D. Here, is unnecessary since (4))) Vo (xx x)] = V Fg (x).
5

---

Published as a conference paper at ICLR 2023
Table FID scores for unconditional generation on CIFAR-IO (Krizhevsky al., 2009). i denotes
EBMs that utilize auxiliary generators, and denotes hybrid discriminative-genernaa models.
Method
FID
Method
FID
Energy-based models (EBMs)
Other likelihood models
Short-run EBM (Nijkamp et al., 2019)
44.50
PixeICNN (Oord al., 2016b)
65.93
JEM (Grathwohl et al., 2019)
38.40
NVAE (Vahdat Kautz, 2020)
$1.67
IGEBM (Du & Mordatch, 2019)
38.20
Glow (Kingma Dhariwal, 2018)
48.90
FlowCE! (Gao al., 2020)
37.30
NCP-VAE (Aneja et al., 2021)
24.08
VERAHI (Grathwohl et al., 2021)
27.50
Score based models
Improved CD (Du ct al., 2021)
25.10
BiDVL (Kan et al., 2022)
20.75
NCSN (Song & Ermon, 2019)
25.30
GEBM (Arbel et al., 2021)
19.31
NCSN22 (Song & Ermon, 2020)
10.87
CF-EBM (Zhao al., 2021)
16.71
DDPM (Ho al., 2020)
3.17
CoopFlow (Xie et al., 2022)
15.80
NCSN+ (Song et al., 2021)
2.20
CLEL-Base (Ours)
15.27
GAN-based models
VAEBM (Xiao et al., 2021)
12.19
SyyleGANZ-Diffuugeee (Zhao et al., 2020)
5.79
EBM-Diffusion (Gao et al., 2021)
9.58
StyleGAN2-ADA (Karras al., 2020)
2.92
CLEL-Large (Ours)
8.61
Table 2: FID scores for un- conditional generation on Ima- geNet x2x
Table 3: FID improvements via different configurations with training time and GPU memory footprint on single RTX3090 GPU of 24G memory. Underline is based on our estimation as the model cannot be trained on the single GPU.
Method
FID
Method
Params (M)
Time
Memory
FID
IGEBM
62.23
Baseline w/o CLEL
6.96
38h
6G
23.50
PixcICNN
40.51
+ CLEL (Base)
6.96
41h
7G
15.27
Improved CD
32.48
+ multi-scale architecture
19.29
74h
8G
12.46
CF- (F-EBM
26.31
+ CRL with a batch size of 256
19.29
76h
10G
11.65
CLEL-Bass Base (Ours)
22.16
more channels (Large)
30.70
133h
11G
8.61
CLEL-Large (Ours)
15.47
(N blocks)
9.06
163h
10G
17.34
EBM-Diflusion (N blocks)
34.83
652h
40G
9.58
VAEBM
135.88
414h
129G
12.19
EXPERIMENTS
We verify the effectiveness of our Contrastive Latent-guided Energy Learning (CLEL) framework under various scenarios: (a) unconditional generation (Section 4.1), (b) out-of-distribution detection (Section 4.2), (c) conditional sampling (Section 4.3), and (d) compositional sampling (Section 4.4).
All the architecture, training, and evaluation details are described in Appendidx B.
4.1 UNCONDITIONAL IMAGE GENERATION
An important application of EBMs is to generate images using the energy function (x).). To this end, we train our CLEL framework on CIFAR-10 (Krizhevsky et al., 2009) and ImageNet 32x32 (Deng al., 2009; Chrabaszez et al., 2017) under the unsupervised setting. We then generate 50k samples using SGLD and evaluate their qualities using Fréchet Inception Distance (FID) scores (Heusel al., 2017; Seitzer, 2020). The unconditionally generated samples are provided in Figure
Table and 2 show the FID scores of our CLEL and other generative models for unconditional generation on CIFAR-10 and ImageNet 32x32. respectively. We first find that CLEL outperforms previous EBMs under both CIFAR-10 and ImageNet 32x32 datasets. As shown in Table 3, our method can benefit from a multi-scale architecture as Du et al. (2021) did, contrastive representation learning (CRL) with a larger batch, more channels at lower layers in our EBM fo- As a result, we achieve 8.61 FID on CIFAR-IO, 10, which lower than that the prior-art EBM based on diffusion recovery likelihood, EBM-Diffusion (Gao et al., 2021), even with 5x faster and 4x more memory- efficient training (when using the similar number of parameters for EBMs). Then, we narrow the gap between EBMs and state-of-ther tof-the-art frameworks like GANs without help from other generative models. We think our CLEL can be further improved by incorporating an auxiliary generator (Arbel al., 2021; Xiao al., 2021) or diffusion (Gao et al., 2021), and we leave it for future work.

---

Published as a conference paper at ICLR 2023
(a) CIFAR-I0
(b) ImageNet 32x32
Figure 2: Unconditional generated samples from our EBMs on CIFAR-10 (Krizhevsky et al., 2009) and ImageNet 32x32 (Deng et al., 2009; Chrabaszez et al., 2017).
Table 4: AUROC scores in OOD detection using explicit density models on CIFAR-10. Bold and underline entries indicates the best and second best, respectively, among unsupervised methods, where JEM and VERA are supervised methods.
Method
SVHN
Textures
CIFARIO Interp.
CIFAR100
CelebA
PixeICNN+ (Salimans al., 2017)
0.32
0.33
0.71
0.63
GLOW (Kingma Dhariwal, 2018)
0.24
0.27
0.51
0.55
0.57
NVAE (Vahdat & Kautz, 2020)
0.42
0.64
0.56
0.68
IGEBM (Du & Mordatch, 2019)
0.63
0.48
0.70
0.50
0.70
VAEBM (Xiao al., 2021)
0.83
0.70
0.62
0.77
Improved CD (Du et al., 2021)
0.91
0.88
0.65
0.83
CLEL-Base (Ours)
0.9848
0.9437
0.7248
0.7161
0.7717
JEM (Grathwohl al., 2019)
0,67
0.60
0.65
0.67
0.75
VERA (Grathoohl et al., 2021).
0.83
0.86
0.73
0.33
4.2 OUT-OF-DISTRIBUTION DETECTION
EBMs can be also used for detecting out-of-distribution (OOD) samples. For the OOD sample detection, previous EBM-based approaches often use the (marginal) unnormalized likelihood pa(x) x (xp(-EE(x)) In contrast, our CLEL is capable of modeling the joint density p= x Eg (x, z). Using this capability, we propose an energy-basee OOD detection score: given X.
1 sxx " [l)(()) Age
Fo((x)
[le(())) )
(x) [xx((x)
(14)
We found that the second term in (14) helps to detect the semantic difference between in- and out-of-distribution samples. Table 4 shows our CLEL's superiority over other explicit density models in OOD detection, especially when OOD samples are drawn from different domains, e.g., SVHN (Netzer et al., 2011) and Texture (Cimpoi et al., 2014) datasets.
4.3 CONDITIONAL SAMPLING
One advantage of latent-variable EBMs is that they can offer the latent-conditional density px(x(z) oc exp(-Ei (x, z)). Hence, our EBMs can enjoy the advantage even though CLEL does not explicitly train conditional models. To verify this, we first test instance-conditional sampling: given real sample x, we draw the underlying latent variable z ~ Paaal (z|x) using our latent encoder h and then perform SGLD sampling using our joint energy Ex z) defined in (10). We here CIFAR- 10 model. As shown in Figure 3a, the instance-ondiitionlly generated samples contain similar information (e.g., color, shape, and background) to the given instance.
7

---

Published as a conference paper at ICLR 2023
Query
Inslance-conditionnd Samples
Cls-conditiooed Samples
Predictions
plain
0.09
car
0.59
Conditionas Oass
deer
frog
ship
Loc
trunt
(a) Instance-conditional sampling
(b) Class-condition sampling
(c) Confusion matrix
Figure 3: (a, b) Instance- and classcoondi generated samples using our CLEL in CIFAR-10
(c) Confusion matrix for the class-conditionally generated samples computed by external classifier.
Young Female Smiling
Multi-atrriuute-coditiie conditioond Samples
Young Female Smillng
0.50
0.97
0.88
0.77
0.90
0.08
0.93
(a) culti-attribute-eee conditional
attribute-condiioonaa sampling
(b) Attribute predictions
Figure 4: Compositional generation results in CelebA. (a) Samples are generated by conditioning on checked attributes. (b) Attribute predictions of generated samples computed by an external classifier.
This successful result motivates us extend the sampling procedure: given a set of instances (x(*) can we generate samples that contain the shared information in {x(²) } To this end, we first draw latent variables (f)) Pdass (-Ix(9) for all and then aggregate them by summation and normalization: = E1 / z 1122 To demonstrate that samples generated from pe (x z) contains the shared information in (x(()) ). we collect the set of instances (xl²) for each label y in CIFAR-10 10, and check whether X, ~ (-.() has the same label yy. Figure 3b shows the class-conditionally generated samples (X and Figure 3c presents the confusion matrix of predictions for (X,, computed by external classifier c. Formally, each (i, entry is equal to Px. (c(X,) = j). We found that x, is likely to be predicted as the label y, except the sometimes look like semantically similar class, cat. These results verify that our EBM can generate samples conditioning on instance class label, without explicit conditional training.
when y is dog: the generated dog images
4.4 COMPOSITIONALITY VIA LATENT VARIABLES
An intriguing property of EBMs is compositionality (Du et al., 2020a): given two EBMs E'(((1) and E(x(c2) that are conditional energies on concepts e and c2, respectively, one can construct new energy conditioning on both concepts: po (x|c and 62) x exp(- ((x(=)(((c)) - E(x [x())) As shown Section 4.3, our CLEL implicitly learns E(x|z), and a latent variable z can be considered as a concept, e.g., instance or class. Hence, in this section, we test compositionality of our model. To this end, we additionally train our CLEL CelebA 64x64 (Liu et al., 2015). For compositional sampling. we first acquire three attribute vectors for A == {Young, Female, Smiling) did in Scction 4.3, then generate samples from a composition conditional energies as follows:
1 (x(x = - 11.f((x) 33 Σ sim(gg((o(())//]f(xx))) Z.,,
(15)
2
aEAA
where sim(..) :) is the cosine similarity. Figure 4a and 4b show the generated samples conditioning on multiple attributes and their attribute prediction results computed by external classifier, respectively. They verify our compositionality qualitatively and quantitatively. For example, almost generated faces conditioned by (Yunng, Female) look young and female (see the third row in Figure 4.)
8

---

Published as a conference paper at ICLR 2023
Table 5: Component ablation experiments.
Table 6: B sensitivity.
Table 7: Compatibility.
Projection de
Negative (x(
FIDJ
O0Dt
8
FIDI
00D1
SSRL
FIDI
00D1
(a)
Baseline (3
42.46
0.8532
0
42.46
0.8532
42.46
0.8532
(b)
MLP
36.29
0,8580
0.001
37.44
0.8485
SimCLR
35.73
0.8723
(c)
MLP
<
35.73
0.8723
0.01
35.73
0,8723
BYOL
36.31
0.8792
(d)
Identity
86.02
0.8474
0.1
56.39
0.7559
MAE
37,67
0.8561
(c)
Linear
37.35
0.8540
4.5 ABLATION STUDY
Component analysis. To verify the importance of our CLEL's components, we conduct ablation experiments with training a smaller ResNet (He et al., 2016) in CIFAR-10 (Krizhevsky et al., 2009) for 50k training iterations. Then, we evaluate the quality of energy functions using FID and OOD detection scores. Here, we use SVHN (Netzer et al., 2011) 1 as the OOD dataset. Table 5 demonstrates the effectiveness of CLEL'S components. First, we observe that learning po(z)x) to approximate Pdata (2|xx plays a crucial role for improving generation (see (a) vs. (b)). In addition, using generated latent variables
(-(-) as negatives for contrastive learning further improves not only generation, but also OOD detection performance (see (b) vs. (c)). We also empirically found that using an additional projection head critical; without projection go (i.e., (d)), our EBM failed approximate Paua (x). but an additional projection head (i.e.. (c) or (e)) makes learning feasible. Hence, we use 2-layer MLP (c) in all experiments since better than a simple linear function (e). We also test various 3 E 10.1, 0.01, 0.001} under this evaluation setup (see Table 6) and find , = 0.01 is the best.
Compatibility with other self-supervised representation learning methods. While we have mainly focused on utilizing contrastive representation learning (CRL), our framework CLEL is not limited to CRL for learning the latent encoder hgg To verify this compatibility, we replace SimCLR with other self-supervised representation learning (SSRL) methods, BYL (Grill et al., 2020) and MAE (He et al., 2021). See Appendix C for implementation details. Note that these methods have several advantages compared to SimCLR: e.g.. BYOL does not require negative pairs, and MAE does not require heavy data augmentations. Table 7 implies that any SSRL methods can be used to improve EBMs under our framework, where the CRL method, SimCLR (Chen et al., 2020), is the best.
RELATED WORKS
Energy-based models (EBMs) can offer an explicit density and are less restrictive in architecture design, but training them has been challenging. For example, it often suffers from the training instability due to the time-consuming and unstable MCMC sampling procedure (e.g., a large number of SGLD steps). To reduce the computational complexity and improve the quality of generated samples, various techniques have been proposed: a replay buffer (Du & Mordatch, 2019), short-run MCMC (Nijkamp et al., 2019), augmentation--asse researchers have also attempted to incorporate other generative frameworks into EBM training e.g., adversarial training (Kumar al., 2019; Arbel et al., 2021: Grathwohl et al., 2021), flow-based models (Gao et al., 2020; Nijkamp et al., 2022; Xie et al., 2022), variational autoencoders (Xiao al., 2021), and diffusion techniques (Gao et al., 2021). Another direction is on developing better divergence measures, e.g., f-divergence (Yu et al., 2020), pseudo-spherical scoring rule (Yu et al., 2021), 1). and improved contrastive divergence (Du et al., 2021). Compared the recent advances in the EBM literature, we have focused orthogonal research direction that investigates how to incorporate discriminative representations, especially of contrastive learning, into training EBMs.
MCMC transitions (Du al.. 2021). Recently,
6 CONCLUSION
The early advances in deep learning was initiated from the pioneering energy-based model (EBM) works, e.g., restricted and deep Boltzman machines (Salakhutdinov et al., 2007; Salakhutdinov & Hinton, 2009), however the recent accomplishments rather rely on other generative frameworks such as diffusion models (Ho et al., 2020; Song et al., 2021). To narrow the gap, in this paper, we suggest to utilize discriminative representations for improving EBMs, and achieve significant improvements. We hope that our work would shed light again on the potential of EBMs, and would guide many further research directions for EBMs.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
In recent years, generative models have been successful in synthesizing diverse high-fidelity images. However, high-quality image generation techniques have threats be abused for unethical purposes, e.g.. creating sexual photos of an individual. This significant threats call us researchers for the future efforts on developing techniques to detect such misuses. Namely, we should learn the data distribution p(x) more directly. In this respect, learning explicit density models like VAEs and EBMs could be an effective solution, As we show the superior performance in both image generation and -ittof-distribution detection, we believe that energy-baset models, especially with discriminative representations, would be an important research direction for reliable generative modeling.
REPRODUCIBILITY STATEMENT
We provide all the details to reproduce our experimental results in Appendix B. Our code avail- able at https: htap::://gtthiie /ggthub..... com/ Baakkock/LEE) In our experiments, mainly use NVIDIA GTX3090 GPUs.
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
This work was mainly supported by Institute of Information & communications Technology Planning Evaluation (ITTP) grant funded by the Korea government (MSIT) Intelligence Innovation Hub; (KAIST)). This work was partly supported by KAIST-NAVER Hypercreative AI Center.
Artificial
Artificial Intelligence Graduate School Program
REFERENCES
Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. Advances in neural information processing systems, 2021.
Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. International Conference Learning Representations, 2021.
Duhycon Bang and Hyunjung Shim. Mggan: Solving mode collapse using manifold -unidold-guied training.
In Proceedings of the IEEE/CVF international conference computer vision, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011.10566, 2020.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the eifar datasets. arXiv preprint arXiv: /707.088/// 2017.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pottern Recognition, pp- 248-28 255, 2009.
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc' Aurelio Ranzato. Residual casrg models for text generation. In International Conference on Learning Representations, 2020.
Laurent Dinh, Jascha Sohk-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference Learning Representations, 2017.
10

---

Published as a conference paper at ICLR 2023
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy- -ased models. In Advances in Neural Information Processing Systems, 2019.
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. In Advances Neural Information Processing Systems, 2020a.
Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, and Alexander Rives. Energy baargy-based models for atomic-resolution protein conformations. In International Conference on Learning Representations, 2020b.
Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. In International Conference on Machine Learning, 2021.
Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of mnergy-based models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7518-7528, 2020.
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy-based models by diffusion recovery likelihood. In International Conference on Learning Representations, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, 2014.
Will Grathwohl, Kuan-Chich Wang, Joern-Henric Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations, 2019.
Will Sussman Grathwohl, Jacob Jin Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud. No meme for me: Amortized sampling for fast and stable training of emergy-based models. In International Conference on Learning Representations, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap Your Own Latent: new approach self-supervised learning. arXiv preprint arXiv:2006.0773: 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International Conference Machine Learning, pp.
PMLR, 2019a.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019b.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
11

---

Published as a conference paper at ICLR 2023
John Ingraham, Adam Riesselman, Chris Sander, and Debora Marks. Learning protein structure with a differentiable simulator. In International Conference on Learning Representations. 2019.
Jongheon Jeong and Jinwoo Shin. Training {gan}s with stronger augmentations via contrastive discriminator. In International Conference on Learning Representations, 2021.
Mark Jerrum and Alistair Sinclair. Polynomial-time approximation algorithms for the ising model. SIAM Journal on Computing, 22(5):1087-1116, 1993.
Ge Kan, Jinhu Lü, Tian Wang. Baochang Zhang, Aichun Zhu, Lei Huang, Guodong Guo, and Hichem Snoussi. Bi-level doubly variational learning for messgy-based latent variable models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Minguk Kang, Woohyeon Shim, Minsu Cho, and Jaesik Park. Rebooting acgan: Auxiliary classifier gans with stable training. In Advances in Neural Information Processing Systems, 2021.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations 2015.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Advances in neural information processing systems, 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint 190100808, 2019.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy- learning. Predicting structured data, 10)), 2006.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In International Conference Computer Vision, December 2015.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint
arXiv:1411.178. 2014.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Erik Nijkamp, Mitch Hill, Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run meme toward energy-based model. In Advances in Neural Information Processing Systems, 2019.
Erik Nijkamp, Ruiqi Gao, Pavel Sountsov, Srinivas Vasudevan, Bo Pang. Song-Chun Zhu, and Ying Nian Wu. MCMC should mix: Learning energy-based model with neural transport latent space MCMC. In International Conference on Learning Representations, 2022.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference machine learning, 2017.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw 2016a.
audio. arXiv preprint arXiv:1609.03:
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, 201
12

---

Published as a conference paper at ICLR 2023
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1tXiv:187.003755 2018.
Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotie analysis. Conference on Learning Theory, pp. 1674-1703. PMLR, 2017.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text image synthesis. In International conference on machine learning, pp. 1060-1069, 2016.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, 2015.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pp. 448 455, 2009.
Ruslan Salakhutdinov, Andriy Mnih, and Geoffrey Hinton. Restricted boltzmann machines for collaborative filtering. International Conference on Machine Learning. 2007.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcn+++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517, 2017.
Maximilian Seitzer. pytorch-fide sarrch-fid:aaa FID Score for PyTorch. httpp:://gi.huub..mmmmaattttts pytorch- fid, August 2020. Version 0.2.1.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Advances in neural information processing systems, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
Jihoon Tack, Sangwoo Mo. Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint /006::9900008449 2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In Advances Neural Information Processing Systems, 2020.
Arash Vahdat and Jan Kautz. Nvae:
deep hierarchical variational autoencoder. In Advances in Neural Information Processing Systems, 2020.
Arash Vabdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Advances in Neural Information Processing Systems, 2021.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. In Advances in neural information processing systems, 2016.
Sebastian J Vollmer, Konstantinos C Zygalakis, and Yee Whye Teh. Exploration of the (non-) asymptotic bias and variance stochastic gradient langevin dynamics. The Journal of Machine Learning Research, 17(1):5004-5555 2016.
13

---

Published as a conference paper at ICLR 2023
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning. 2020.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient Langevin dynamics. In International Conference on Machine Learning, 2011.
Hao Wu, Babak Esmaeili, Michael Wick, Jean- Saan-Baptiste Tristan, and Jan- lal-Willem van de Meent.
Conjugate energy-based besegy-based models. In International Conference on Machine Learning. 2021.
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy- baaeed models. In International Conference on Learning Representations, 2021.
Jianwen Xie, Yaxuan Zhu, Jun Li, and Ping Li. A tale of two flows: Cooperative learning of langevin flow and normalizing flow toward energy- model. In International Conference on Learning Representations, 2022. URL https: appereeiiees net: forum ? - 3 31d5RLCUUXC.
Pan Xu, Jinghui Chen, Difan Zou, and Quanquan Gu. Global convergence of langevin dynamics based algorithms for nonconvex optimization. Advances Neural Information Processing Systems, 2018.
Xiulong Yang and Shihao Ji. Jem++: Improved techniques for training jem. In International Conference on Computer Vision, 2021.
Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon. Training deep energy-based models with f-divergence minimization. In International Conference on Machine Learning, 2020.
Lantao Yu, Jiaming Song, Yang Song, and Stefano Ermon. Pseudo-spherical contrastive divergence. In Advances in Neural Information Processing Systems, 2021.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for
data-efficient gan training. In Advances in Neural Information Processing Systems, 2020.
Yang Zhao, Jianwen Xie, and Ping Li. Learning energy maaey generative models via coarse-to tourse-to-fine expanding and sampling. In International Conference on Learning Representations, 2021.
Difan Zou, Pan Xu, and Quanquan Gu. Faster convergence of stochastic gradient Langevin dynamics for non-log-concave sampling. In Uncertainty in Artificial Intelligence, 2021.
14

---

Published as a conference paper at ICLR 2023
A TRAINING PROCEDURE OF CLEL
Algorithm 1 Contrastive Latent-guided Energy Learning (CLEL)
Require: latinl-variable EBM (fe, ge), latent encoder his augmentation distribution T. hyperparameters a, 3, r > O, and the stop cop-grraient operation sg(-).
1: for training iterations do
2: // Construct batches B and B
3: Sample {xx(9) )==1 ~ Pdata(X)
) 4: Sample (xx10 Ji=ı ~ px(x) using stochastic gradient Langevin dynamics (SGLD)
5: 20)) + sg (2)
{(0)
6: 2 + sg (/1)l(())))/lll(((())))))))
7: // Compute the EBM loss, LEAM
8:
LEAM + Ei ,
200)) - Eo (x(1)) + a. (Fo(x(()) + (x(
9:
// Compute the encoder loss, LL.
10: 2j + ho(t,) (x. 13) ~ T
LLE 1. Σ. E5m1, LNT-Xent (z) (2)
(1) 66 (zj JaAL)E(112) U12(0) (=ziT
2n
Update 0 and @ to minimize LRAM LI.F.
13: end for
B TRAINING DETAILS
Architectures. For the spherical latent-variable energy-based model (EBM) fo. we use the 8-block ResNet (He et al., 2016) architectures following Du & Mordatch (2019). The details of the (a) small, (b) base, and (c) large ResNets are described Table 8. We append 2-layer MLP with a output dimension of 128 to the ResNet, i.e,, : 3x332x32 - R128 Note that ablation experiments
the small model for Section 4.5. To stabilize training, we apply spectral normalization (Miyato al., 2018) to all convolutional layers. For the projection ge. use a 2-layer MLP with output dimension of 128, the leaky-ReLU activation, and no bias, i.e., (u) = W2 u) E R 128, For the latent encoder h we simpy use the CIFAR variant of ResNet-18 (He al., 2016), followed by 2- 2-layer MLP with a output dimension of 128.
Table 8: Our EBM fo architectures. For our large model, we build three independent ResNets and resize input image 3xx32x32 three resolutions: 32 32, 16 x 16, and 8 8. We use each ResNet for each resolution image, concatenate their output features, and then compute the final output feature (x) R128 using single ML..
Small
Base
Large
Input
(3,32, 32)
(3, 32, 32)
(3,32, 32). (3, 16, 16). (3, 8,888 8)
Conv(3 3, 64)
Conv(3 x 3, 128)
Conv(3 x 3, 256)
ResBlock(64)
x1
ResBlock(128)
x2
ResBlock(256)
x2
AvgPool(2 x 2)
AvgPool(2 2)
AvgPool(2 x 2)
ResBlock(64)
ResBlock(128)
ResBlock(256)
x2
EBM fo(x)
AvgPool(2 x 2)
AvgPool(2 2)
AvgPool(2 x 2)
x
ReaBlock(128)
x1
ReaBlock(256)
x2
ReaB1ock(256)
x2
AvgPool (2 x 2)
AvgPool(2 (2
AvgPool (2 x 2)
(eeBlock(128)
x1
ResBlocx(256)
x2
ResBlock(256)
x2
GlobalAvgPool
GlobalAvgPood
GlobalArgPool
MLP(128, 2048, 128)
MLP(256, 2048, 128)
Concat + MLP(768, 2048, 128)
Training, For the EBM parameter 6, we use Adam optimizer (Kingma & Ba, 2015) with 31 = 0. 32 0.999, and a learning rate of 10-4. We the linear learning rate warmup for the first 2k training iterations. For the encoder parameter ø, we use SGD optimizer with a learning rate of 3 x 10-2, weight decay of 5 10-1, and momentum of 0.9 as described in Chen & He (2020).
15

---

Published as a conference paper at ICLR 2023
For all experiments, we train models up 100k iterations with batch size of 64, unless otherwise stated. For data augmentation T, we follow Chen et al. (2020), i.e., 7 includes random cropping. flipping. color jittering. and color dropping. For hyperparameters, Mordatch (2019), and / = 0.01 (see Section 4.5 for /--seniitivity experiments). For our large model, large batch size of 256 only for learning the contrastive encoder h After training, utilize exponential moving average (EMA) models for evaluation.
1 following Du &
SGLD sampling. For each training iteration, we use 60 SGLD steps with a step size of 100 for sampling & ~ pa. Following Du et al. (2021), we apply random augmentation ~ T for every 60 steps. We also use a replay buffer with a size of 10000 and resampling rate 0.1% % for maintaining diverse samples (Du & Mordatch, 2019). For evaluation, we run 600 and 1200 SGLD steps from uniform noises for our base and large models, respectively.
C IMPLEMENTATION DETAILS FOR BYOL AND MAE
We here provide implementation details for replacing SimCLR (Chen al., 2020) with BYOL (Grill et al., 2020) and MAE (He et al., 2021) under our CLEL framework as shown in Section 4.5.
BYOL. Since BYOL also learns its representations on the unit sphere, the method can be directly incorporated with our CLEL framework.
MAE. Since MAE's representations do not lie on the unit sphere, we incorporate MAE into our CLEL framework by the following procedure:
1. Pretrain a MAE framework and remove its MAE decoder. To this end, we simply use a publicly-available checkpoint of the ViT-tiny architecture.
2. Freeze the MAE encoder parameters and construct learnable 2-layer MLP the top of the encoder.
3. Train only the MI.P via contrastive representation learning without data augmentations using our objective (13) for the latent encoder.
16

---

Published as a conference paper at ICLR 2023
D TRAINING STABILITY WITH NORM-DIRECTION SEPARATION
(2/x)
269
R
Rd
W
Mutihaad
de
150
fo(x)
[/o(x]]l
100
[[fo()]
E
(v(x)
M
f0
fo
4xn0"
Training
(a) Multi-hear head architecture
(b) Norm- direction separation
(c) FID scores during training
Figure 5: (a) A multi-head architecture design and (b) norm-direction separation scheme for modeling p(x) and p(z]x). (c) FID scores with various energy design choices.
At the early stage of our research, we first tested multi-head architecture for modeling p(x) and p(z]x). To specific, E(x) = (x))) R and E(z]x) - 2 (xx)) where f is a shared backbone and y and y' are separate 2-layer MLPs, as shown in Figure 5a. We found that, with this choice, learning p(x) causes a mode collapse for fx() because all samples should be aligned with specific direction. In contrast, learning p(z/x) encourages fe (x) to be diverse due to contrastive learning. Namely, modeling p(x) and p(z[x) with the multi-head architecture makes some conflict during optimization. We empirically observe that the multi-head architecture is unstable in EBM training, as shown in Figure 5c. To remove such a conflict, we design the norm-direction separation for modeling p(x) and p(zix) simultaneously (i.e., Figure 5b), which leads to training stability, as shown in Figure 5c.
E
THE ROLE OF DIRECTIONAL PROJECTOR PRORECTOO
30-
36-
30
30-
59
108
006
1338
(a) fx()
(b) (f((()))
(c) hx
Figure 6: Cosine similarity distributions using (a) fo(x), (b) go (f)()), and (c) (x(x) features.
Our directional projector go is designed for narrowing the gap between the EBM feature direction (o(x)//l[(xx)) and the Specifically, the contrastive latent variable Pdata (z) known be uniformly distributed (Wang & Isola, 2020), but we observed that it is difficult to optimize the feature direction fe be uniform along with learning our norm-based EBM (x x exp(-Il.oox) empirical supports, we analyze the cosine similarity distributions using fx)x,, (x))) and h (x) features on CIFAR-10, as shown in Figure 6. This figure shows that fo(x) tends learn similar directions (see Figure 6a) while (x) tends to be uniformly distributed (see Figure 6c). Hence, it is necessary to employ a projection between them. found that our projector ge successfully narrows the gap as shown in Figure 6b, which significantly improves training EBMss.
-iitrmyldiistiiutted" latent variable Pdma(z) (i.e., (x)x)/ h////////x))))///
to
at the same time. As
17

---

Published as a conference paper at ICLR 2023
F FLEXIBILITY OF NORM-BASED ENERGY FUNCTION
Our norm-based energy parametrization does not sacrifice the flexibility compared to the vanilla parametrization of EBMs. We here show that any vanilla EBM pl (x) x exp(fi (x)), fi(x R. can be formulated by a norm-based EBM p2(x) x (x 2 (x) E IR on compact input space X' (e.g., an image x lies on the continuous pixel space X [0. 255 HWC)..
Let == minxex (x) be the minimum value of f1. Then, (x(x) exp (x) b) is identical to Pi due the normalizing constant. Furthermore, P3 can formulated norm-based EBM p2: for example, if the first component of f2 is the same v (x) b and other components are zero, then P1 and P2 model the exactly same distribution. Therefore, energy-mode modeligg with our norm-based design is not much different from that with the vanilla form.
special case of the
18


Reviews:
[{"review_id": "iWT3IoY7284", "reviewer": "Reviewer_zYAi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a latent variable energy-based model by introducing an energy form for the joint model p(x,z).\nIntuitively, they augment data point x with z that comes from a separate encoder (that is trained using conservative representation learning) and train the joint energy model using contrastive divergence. They also use the z that minimizes E(x,z) as a negative sample to train the encoder.  \nThis setup enforces that the energy model assigns lower energy to the latent representation that is aligned with the external decoder. \n\nThe authors show the capability of their methods using experiments on OOD detection, conditional and unconditional generation, and compositional generation. ", "summary_of_the_review": "The paper is relevant to the community, novel, and shows an interesting connection between SSL representation learning and EBM training. \nThe experimental results are strong and the authors did an extensive analysis of their methods. The main weakness is the lack of sufficient discussion regarding the used energy form and ablation study. For example, the authors show that the identity function is not a useful projector but not discussing why?\n\n", "strength_and_weaknesses": "The idea is novel and experimentally they show a great performance (w.r.t. to FID score) on CIFAR-10.  Although the performance is not better than the SOT in the field, it is impressive regarding other EBMs training methods. There are many applications in that EBMs have an advantage over other probabilistic methods such as score models, so better EBM training algorithms are important to the community.\n\nThe main weakness of the paper is that the authors didn't justify the joint energy model that they are using. \nFor example, they say that \"the remaining information in $\\|| f_\\theta(x)\\||_2$ is used for modeling $p_\\theta(x)$. What do you mean by that?\nor when you say that \"we separate f into direction and norm and we found that this separation reduces the conflict between $p_\\theta$ and p\\theta(z|x) optimization\" what exactly do you mean? Are the direction of the gradient inconsistent? What else did you try?\nWhy does only a particular form of g work well? Why does using the identity function as g drastically reduce the performance of the model? The FID score when using the identity function is twice higher as using beta=0!\n\nIn Table 5, setting b does not use the negative example from p_theta for training the encoder. Does this mean that one can use a pre-trained encoder and distill the representation knowledge into E(x,z)? Have you tried something like that?\n\n\nTypo: \"into our EMB training\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MnFTCXmcM3o", "reviewer": "Reviewer_eaGu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to improve the performance of energy-based models by incorporating latent variables to the model, assuming the positive samples of which come from an encoder optimized by contrastive representation learning. The latent variable EBM is parametrized in a way such that $p(x)$ and $p(z|x)$ can be easily separated. The performance can be further improved by updating the encoder with sampled $z$ from the EBM as auxiliary negative samples in CRL. Experimental results show it leads to state-of-the-art generative performance within EBMs. The learned model is also capable of conditional and compositional generation. ", "summary_of_the_review": "The paper proposes an interesting way for parametrizing latent-variable EBM and combining it with CRL. Experimental results are convincing that it is a promising approach to improving EBMs. It will be even strong if the paper could show the joint estimation method can also improve the learned features of the encoder. ", "strength_and_weaknesses": "### Strength\n- The paper presents an interesting way to parametrize latent-variable EBMs, such that $p(x)$ and $p(z|x)$ are in closed-form. \n- Given the proposed model formulation, the paper proposes a wise way to combine it with contrastive representation learning. Results indicate that such combination can boost the performance of EBMs reasonably well.\n\n### Weaknesses and questions\n- The parametrization of $E_\\theta(x) = \\frac{1}{2} \\|f_\\theta(x)\\|_2^2 + C$ seems to be a constrained set of functions. I'd like to see more discussions in terms of why it could serve as a general set for modeling energy functions. \n- I don't find information in the paper about whether the SimCLR encoder is pretrained or jointly estimated with the EBM starting from scratch. If it is pretrained, I'd like to see another ablation where the encoder is fixed as the pretrained one and only EBM is learned. \n- In the ablation study, all model have only been updated for 50k iterations. Do all the models really converge with such a small amount of iterations? \n- Given the proposed framework also provides a new objective for the encoder of contrastive representation learning, does it also lead to better learned features $z$ compared to the original CRL in terms of downstream tasks such as classification? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JrJdQWZDv6", "reviewer": "Reviewer_hpmQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a very neat idea of embedding contrastive learning into energy-based model learning. On top of the normal latent-variable energy-based models, the authors introduce augmentation and contrastive representation learning to regularize the latent space. The general idea is to still keep the empirical KL divergence, regularized by energy regularization terms. Besides, this work adds contrastive learning, inspired by SimCLR, on the latent space modeling. The experiments give both qualitative and quantitative results. This work can generally outperform other energy or autoencoder-based methods.", "summary_of_the_review": "This is a good idea with many experimental validations. Some claims need to be further supported. And the gap with other type of models should be better explained. Some modules should be expanded to further facilitate understanding.", "strength_and_weaknesses": "Pros:\n1. This is a very interesting idea to embed contrastive learning into EBM. Indeed, contrastive learning has been widely used in a lot of scenarios. Employing it in EBM is quite novel.\n2. In experiments, the paper demonstrates unconditional generation, conditional generation, out-of-sampling detection and conditional sampling. It's thus convincing that this work is versatile, and potentially can be used in many scenarios.\n\nCons:\n1. Could you further explain the P in Eq. 9?\n2. The directional projector lacks explanation or intuitions. It's better to explain the motivation and some insights behind this module.\n3. I think one thing you claim in your paper is that your method can make the EBM training more stable. Maybe you can give more examples to illustrate that.\n4. Some numbers are still worse than GAN-based numbers. Do you have any insights or ideas?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "698YT-O_GAS", "reviewer": "Reviewer_nYLf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to improve EBM training via contrastive representation learning (CRL).\n\nIn particular, it uses SimCLR framework to train the latent variable and implement latent-variable EBM with joint probability. The empirical results show that the proposed framework is more efficient and achieves lower FID scores and better OOD detection on visual datasets. ", "summary_of_the_review": "In general, I think CLEL is a decent but incremental empirical work that verifies the benefits of contrastive latent variables in EBM training. The claims are justified by the downstream tasks and some ablation studies. But I do believe that there should be some results beyond the heuristic motivation and plain numerical results to justify this work so that it can be significant enough to convince the community. Thus, I encourage the authors to repackage the work and find out more merits about CLEL.", "strength_and_weaknesses": "Strength:\n\nThe introduction of latent variable is important in statistical inference tasks. It is exciting to see that the latent variables help EBM in practice.\n \nI appreciate that the endeavor of the authors to adapt EBM models to visual datasets, and the resulting generation quality and OOD detection have been improved over previous EBMs.\n\n\nWeakness:\n\nAlthough the introduction of latent variable is important, the CLEL just combine contrastive learning and EBM in a brute way, which is intuitive but incremental. What I am concerned about the most is the significance and novelty of this work, which is fully heuristic and direct in my opinion.  It is good to see that CLEL can improve image generation and OOD detection but I will really appreciate it if you can provide more insights about the “latent variables” rather than the downstream performance (which is also not strong enough). I must say that the paper can be significantly improved if it can provide enough evidence to show that what is the “latent variable learned by contrastive learning” and provide more statistical logics about the benefits.\n\n\nThe authors claim that the proposed model benefits from the introduction of true underlying latent variable. I believe that it is necessary to justify the why it is the true latent variable? Does it have explainable representations? Or can you prove that in what sense it is true.\n\nI must say the generation quality is not the core issue of EBM’s training, thus I don’t think improving generation quality is significant enough, since we have many other choices.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%