Energy-Based Contrastive Learning of Visual Representations
Authors: Beomsu Kim
Jong Chul Ye
Venue: NeurIPS 2022 Accept (2022)
Keywords: Energy-Based Models
Contrastive Learning

Abstract: Contrastive learning is a method of learning visual representations by training Deep Neural Networks (DNNs) to increase the similarity between representations of positive pairs (transformations of the same image) and reduce the similarity between representations of negative pairs (transformations of different images). Here we explore Energy-Based Contrastive Learning (EBCLR) that leverages the power of generative learning by combining contrastive learning with Energy-Based Models (EBMs). EBCLR can be theoretically interpreted as learning the joint distribution of positive pairs, and it shows promising results on small and medium-scale datasets such as MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100. Specifically, we find EBCLR demonstrates from $\times 4$ up to $\times 20$ acceleration compared to SimCLR and MoCo v2 in terms of training epochs. Furthermore, in contrast to SimCLR, we observe EBCLR  achieves  nearly the same performance with $254$ negative pairs (batch size $128$) and $30$ negative pairs (batch size $16$) per positive pair, demonstrating the robustness of EBCLR to small numbers of negative pairs. Hence, EBCLR provides a novel avenue for improving contrastive learning methods that usually require large datasets with a significant number of negative pairs per iteration to achieve reasonable performance on downstream tasks. Code: https://github.com/1202kbs/EBCLR

Energy-Based Contrastive Learning of Visual Representations
Beomsu Kim
Jong Chul Ye Kim Jaechul Graduate School of AI KAIST jong.ye@kaist.ac
Department of Mathematical Sciences KAIST beomsu. aaa.maaistt......
Abstract
Contrastive learning is a method of learning visual representations by training Deep Neural Networks (DNNs) increase the similarity between representations of positive pairs (transformations of the same image) and reduce the similarity between representations of negative pairs (transformations of different images). Here we explore Energy- -ased Contrastive Learning (EBCLR) that leverages the power of generative learning by combining contrastive learning with Energy- Based Models (EBMs). EBCLR can be theoretically interpreted as learning the joint distribution of positive pairs, and it shows promising results on small and medium-scale datasets such as MNIST, Fashion-MNIST.. CIFAR-16 10, and CIFAR- 100. Specifically, we find EBCLR demonstrates from x4 up x20 acceleration compared to SimCLR and MoCo v2 in terms of training epochs. Furthermore, in contrast to SimCLR, we observe EBCLR achieves nearly the same performance with 254 negative pairs (batch size 128) and 30 negative pairs (batch size 16) per positive pair, demonstrating the robustness EBCLR to small numbers of negative pairs. Hence, EBCLR provides novel avenue for improving contrastive learning methods that usually require large datasets with a significant number of negative pairs per iteration to achieve reasonable performance downstream tasks. Code: https: /ttp:/gitttttt com/ 2222202ks//ete
1202kbs/BBLLR /BBCLR
Introduction
In computer vision, supervised learning requires a large-scale human-annotater dataset of images to train accurate deep neural networks (DNNs). However, acquiring labels for millions of images be difficult or impossible in practice. This has led to the rise of self-supervised learning, which learns useful visual representations by forcing DNNs to be invariant equivariant to image transformations. Among self-supervised learning algorithms, contrastive methods are rapidly gaining popularity for their superb performance.
Specifically, contrastive learning methods between representations of positive pairs (transformations of the same image) and decreasing the similarity between representations of negative pairs (transformations of different images). The negative pairs prevent DNNs from collapsing to the trivial constant function. There numerous contrastive learning methods, such as SimCLR [4, Momentum Contrast (MoCo) 5]. etc.
train DNNs by increasing the similarity
Despite this flurry of research in contrastive learning, contrastive methods require large datasets and a large number of negative pairs per positive pair to achieve reasonable performance on downstream tasks. Although there are recently proposed non-contrastive methods such BYOL [6] and SimSiam 7 that do not rely on negative pairs, they require heuristic techniques such as stop-gradient to avert collapsing the trivial solution. There has been an effort explain the dynamics of non-contrastive methods with linear neural networks but is unclear how the analyses generalize to DNNs.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

p(v.st) 20 p(o.
802
Evattatoonn
8
8
Mppourr
15
MoCo V2 Batch Size 25G
Sim Barch Stee:266
Betch
45
Shored
Projeciios
Projection
Farms
Figure 1: Left: An illustration of EBCLR. Here, means "is monotonically increasing function of". We use p(u, v'), the joint distribution of positive pairs, as a measure of semantic similarity of images. Specifically, p(u, v') will be high when and v' are semantically similar, and low otherwise. DNN So is trained such that the distance in the projection space is controlled by 1/p(u, v'). Right: Comparison of EBCLR, SimCLR, and MoCo v2 CIFAR10 in terms of linear evaluation accuracy. EBCLR at epoch 10 beats MoCo v2 at epoch 100, and EBCLR at epoch 20 beats SimCLR and MoCo v2 epoch 100. Moreover, EBCLR shows identical performance regardless of whether we use 254 negative pairs (batch size 128) or 30 negative pairs (batch size 16) per positive pair.
In this paper, we explore novel avenue in visual representation learning: Energy- Based Contrastive Learning (EBCLR) which leverages the power of generative learning [ 9 10] by combining contrastive learning with energy-based models (EBMs). EBCLR complements the contrastive learning loss with a generative loss, and it can be interpreted as learning the joint distribution of positive pairs. In fact, demonstrate that the existing contrastive loss is special case of the EBCLR loss if the generative term is not used. Although EBMs are notorious for being difficult to train due their reliance on Stochastic Gradient Langevin Dynamics (SGLD) [. another important contribution of this work is that we overcome this by appropriate modifications to SGLD.
Extensive experiments robust to small numbers of negative pairs, and outperforms SimCLR and MoCo v2 12 in terms of sample efficiency and linear evaluation accuracy. Hence, EBCLR opens up a research direction for alleviating the dependence of contrastive methods on large datasets and large batches.
variety of small and medium-scale datasets demonstrate that EBCLR is
Our contributions can be summarized as follows:
We propose novel contrastive learning method called EBCLR which learns the joint distribution of positive pairs. We show that EBCLR loss is equivalent combination of contrastive term and generative term (Section 3}. To the best our knowledge, this the first work to apply EBMs to contrastive learning of visual representations.
We show that EBCLR offers two advantages over conventional contrastive learning methods: EBCLR is several times more sample efficient (Section 4.. and robust small batch sizes (Section 4.2) These factors lead to non-trivial performance gain for EBCLR.
We perform thorough ablation studies of the components EBCLR: effect of changing the weight of the generative term (Section 4.3), effect of projection space dimension (Section and the effect of the proposed SGLD modifications (Section 4.4).
Related Works
In this section, go related works necessary for understanding EBCLR. In Appendix A give a more extensive review of relevant works for those not familiar with EBMs, contrastive learning, generative models.
2.1 Contrastive Learning
For a given batch images { and two image transformations 1, contrastive learning methods first create two views Un t(Ix), o4 (z(x.) of each instance In. Here, the pair (Un, 2/m

---

is called positive pair if n = m and negative pair if o m. Given DNN fe. the views are then embedded into the projection space by passing the views through fo and normalizing.
Contrastive methods train fo to increase agreement hetween projections of positive pairs and decrease agreement between projections of negative pairs. Specifically, fo is trained to maximize the InfoNCE objective []] After training, outputs from the final layer or an intermediate layer of fe are used for downstream tasks.
There are numerous variants contrastive methods. For instance, SimCLR 4 uses a composition of random cropping, random flipping, color jittering. color dropping, and blurring the image transformation. Negative pairs are created by transforming different images within a batch. On the other hand, MoCo [12] maintains queue of negative samples, so negative samples not limited to views of images from the same batch.
2.2 Energy-Based Models
Given a scalar-valued energy function (v(v) with parameter 0, an energy-based model (EBM) defines a distribution by the formula
13
1 Z(0) exp(-Es(v)}
(o(u)
(1)
where (0) is the partition function which guarantees qo integrates to 1. Since there are essentially no restrictions on the choice of the energy function. EBMs have great flexibility in modeling distributions. Hence, EBMs have been applied to a wide variety of machine learning tasks, such as dimensionality learning generative classifiers
reduction via autoencoding 14]
TO 15 generating images [16.
Wang et al. 19 have explored connections between EBMs and InfoNCE to enhance generative performance of EBMs. However, the best of our knowledge, this paper is the first to combine EBMs with contrastive learning for representation learning.
and training regression models 11 18
Given target distribution, EBM can be used to estimate its density p when we only sample from p. One way of achieving this is by minimizing the Kullback-Leibler (KI.) divergence between qe and p that maximizes the expected log-likeli likeliihohod of qo under p [20]:
max EG, [log qo (v)].
9
Stochastic gradient ascent can be used to solve ( 20 Specifically, the gradient of the expected log-likelihood with respect to the parameters can be shown to be
Ee Log (o (v)) = E.ge V (v)) (v)
Hence, updating 0 with (3) amounts to pushing up the energy for samples from qa and pushing down on the energy for samples from p. This optimization method is also known as contrastive divergence (21)
While the second term in (3) can be easily calculated as we have access to samples from p. the first term requires sampling from qo. Previous works [16, 22 I0 [5] have used Stochastic Gradient
Langevin Dynamics (SGLD) [1]] to generate samples from go. Specifically, given a sample vo from
some proposal distribution go. the iteration
at V., Eo(v:) + Et:
4 ~(0,0)
Ut+1 = 0t
2
guarantees that the sequence } converges to a sample from go assuming {ar) decays at polyno- mial rate
However, SGLD requires an infinite number of steps until samples from the proposal distribution converge samples from the target distribution. This is unfeasible, so in practice, only a finite number of steps along with constant step size, i.c.
and constant noise variance = o² are used [6 22 T0 [5]. Moreover, Yang and Ji [5] noted SGLD often generates samples with extreme pixel values that cause EBMs to diverge during training. Hence, they have proposed proximal SGLD which clamps gradient values into an interval [-8, 8) for threshold > Then, the update equation becomes
#t-1 vt a clamp V. Fo.:), +
for O,..., 1, where o0) and clampf:, d) clamps each element of the input vector into 5, 8). In our work, we introduce additional modifications to SGLD which accelerate the convergence of EBCLR.
3

---

Shared
Parameters
zí
tng
Contrastive Loss
Zn
Fa
A Ri (Phr
Sum
Generative Loss
p(v)
22 Eg(An; (ata))
Optinication Procedure
fo
Sample with pubb.
Replay Buffer
Update
Update
Update bufler
Update 9, SGLD using Vs, Ep(aii
Calculate Jomes & upčate
Figure 2: An illustration the learning process EBCLR.
Theory
3.1 Energy-Based Contrastive Learning
Let D distribution of images and distribution of stochastic image transformations. Given D and i.i.d. t,t' T, our goal is to approximate the joint distribution of the views
p(v,"'),
where == t(x), v' f((x)
using the model distribution
go(u, v/) Z(0) -xp(-1l==211///) //1
(6)
where ((0) is a normalization constant, > is a temperature projections computed by passing the views and e' through the DNN fo and then normalizing to have unit norm. We now explain the intuitive meaning of matching qe to
and x and 2' are
Our key idea is use p(u, v') measure of semantic similarity of and v'. If two images and vf are semantically similar, they are likely to transformations of similar images. So, p(u, v') will be high when and o' are semantically similar and low otherwise. Suppose qe successfully approximates If we equate p(u, o') to qo (U, r') in (6) and solve for ||| 2/11, we see that the distance between and z' will become a monotone increasing function of 1/p(u, t'), which is the inverse of semantic similarity of v and v'. So, semantically similar images will have nearby projections, and dissimilar images will have distant projections. This idea is illustrated in Figure []]
To approximate p using go. we train So maximize the expected log-likelihood of qo under p:
max E [og qo (v, e²)].
In order to solve this problem with stochastic gradient ascent, we could naively extend to the setting of joint distributions to obtain the following result.
Proposition The the joint distribution
6 can be formulated as an EBM
go(u. v')
Z(0) exp(-Eg(z,2)))
Eo(v, v') = ||2 2/12/112/T
(8)
and the gradient of the objective of given Eg go (u, E.q. (o
Vo,
(9)
However, computing the first expectation in 9 requires sampling pairs of views (e, e²) from go (v, v') via SGLD, which could be expensive. To avert this problem, we use Bayes' rule decompose
E, flog go (v, o')] [lol (o [)]] + [pllog where go(u) = go(u, du'. (10)

---

In the first equation of 100. the first and second terms at the RHS will be referred to as discriminative and generative terms, respectively, throughout the paper. A similar decomposition was used by Grathwohl et al. [0] in the setting of learning generative classifiers.
Furthermore, add hyper-paramettr generative term. The advantage of this modification will be discussed in Section 4.3 This yields our Energy-Based Contrastive Learning (EBCLR) objective
halance the strength of the discriminative term and the
(()) Ep [log go(v' [)]] + XE, llog go(v)].
(11)
The discriminative term can be easily differentiated since the partition function Z(0) cancels out when go (u, o²) is divided by (v)). However, the generative term still contains (0). We now present key result, which is used to maximize (TT). The proof is deferred to Appendix C.
Theorem 2. The marginal distribution in (0) can be formulated
EBM
(v) = Z(0) expl Eu())), Ea(u) == -log / --1l===1 dul
1
(12)
where Z(0) is the partition function in 66 and the gradient of the generative term is given by
V VoR go (v)) (v [Vo
(13)
Thus, the gradient of the EBCLR objective is
V VoL(0) Ey log do (v') [b] AB go Vg AE V Fs())]
(14)
Theorem 2 suggests that the EBM for the joint distribution can be learned by computing the gradients of the discriminative term and the EBM for the marginal distribution. Moreover, we only need to sample from (v(v) to compute the second expectation in (14)
3.2 Approximating the EBCLR Objective
To implement EBCLR, we need to approximate expectations in [II with their empirical means. Suppose samples {Un,, v/ " from p(u, v²) are given, and let {(Zn,²/) > be the corresponding projections. As the learning goal is to make (",, v²,)) approximate the joint probability density function p(u,, v,,,, the empirical mean qo (v,) can defined as:
To (vn)
Σ
Uns v'm)
(15)
N
of Si #t
where the sum is over the collection of s/ defined
{1/m s/m # 2 = {ux (W [2,}
(16)
and N' = He o% # 2n = 2N 1. One could also use simpler form of the empirical mean:
1 Σ
(v))
go(un )
(17)
Similarly, go (e'le) in [T], which should approximate the conditional probability density p(u'lu). can be represented in terms of qo (Un; 1/)) Specifically, have
go (Vn, v/h)
go (Un 1/2) NO Eve Su, Go (Uay V'r))
e 11 1|,1==1,11// 21,11//
go(v,, un))
N Ev vol tv. e 11mn-=11113 (18)
go (v,)
It is then immediately apparent that the empirical form of the discriminative term using (18) is a particular instance the contrastive learning objective such as InfoNCE and SimCLR. Hence, FBCLR can be interpreted as complementing contrastive learning with a generative term defined by an EBM. We will demonstrate in Section 4.1 that the generative term offers significant advantages over other contrastive learning methods.
5

---

For the second term, we use the simpler form of the empirical mean in (7
N
N
(o(wx)
Σ
4o (un vmmm)
1
1
Σ expl
-
2 ///
(19)
N
(
N
m/=
m=1
We could also Appendix F.3) So, we have found 15 to be not worth the additional complexity, and have resorted
15
as the empirical mean, but either choice showed identical performance (see
to the simpler approximation [17p instead. In Appendix C.2 we theoretically justify that EBCLR will
work as intended even with the approximations (5) or (17). y. If we compare (19) with (12) we can
see that this approximation of qo (o()) yields the energy function (after ignoring the constant log N)
Ea (1; [m [ == log
Σ
e
11
(20)
3.3 Modifications to SGLD
According Theorem 2
we need samples from the marginal go(v) to calculate the second expectation
in 14. Hence, we apply proximal SGLD (5) with the energy function (20) to sample from (o(v) as
Uz+1 = 0z - a clampp v. Lig (ū;; ft - 6} +
(21)
for f = O,..., T 1, where c ~ (N00 o²). We make three additional modifications to proximal SGLD to expedite the training process, From here on, we will be referring to proximal SGLD in (5)
when we say SGLD.
First, we initialize SGLD from generated samples from previous iterations, and with probability p, we reinitialize SGLD chains from samples from proposal distribution qo- This is achieved by keeping a replay buffer B of SGLD samples from previous iterations. This technique of maintaining a replay buffer has also been used in previous works and has proven to be crucial for stabilizing and accelerating the convergence of EBMs 16 10 15).
Second, the proposal distribution do is set to be the data distribution p(v). This choice differs from those of previous works Gaussians as the proposal distribution.
which have either used the uniform distribution mixture of
Finally, we use mulli-stage SGLD (MSGLD), which adaptively controls the magnitude of noise added in SGLD, For each sample in the replay buffer B, we keep count K2 of number of times it has been used as the initial point of SGLD. For samples with low count, we use noise of high variance, and for samples with a high count, we use noise of low variance. Specifically, in (5), we set
σ Tmin + (Amax min)) [1 -ks/K]]++
(22)
where |-, = max(0, +.. o and on are the upper and lower bounds on the noise variance, respectively, and K controls the decay rate of noise variance. The purpose of this technique is to facilitate quick exploration of the modes of qo and still guarantee SGLD generates samples with sufficiently low energy. The pseudocodes for MSGLD and EBCLR are given in Algorithms and 2
respectively, in Appendix B and the overall learning flow of EBCLR is described in Figure
Experiments
We now describe the experimental settings. A complete description is deferred to Appendix D
Baseline methods and datasets. The baseline methods SimCLR, MoCo v2, SimSiam, and BYOL. The hyper-parameters are chosen closely following the original works 4 12
7
6 We use
four datasets: MNIST 23), Fashion MNIST (FMNIST) 24), CIFAR1O, and CIFARI00
25
DNN architecture. We decompose fe = ne o dhe where de is the encoder network and тæ is the projection network. Rather than using the output of fo for downstream tasks, we follow previous
7 and use the output of Φo instead. In our experiments, we set de to be ResNet- 18 26 up the global average pooling layer and Ty to be 2-layer MLP with output dimension 128. However, we remove batch normalization because batch normalization hurts SGLD [6]. We also replace ReLU with leaky ReLU to expedite the convergence of SGLDD For the baselines, we use settings proposed in the original works while keeping the backbone fixed to be 188
works 44, 5 2
g
6
Evaluation. We evaluate the representations by training linear classifier top of frozen de.

---

Dataset
MNIST
FMNIST
CIFAR1O
CIFARI00
Statistic
Accuracy
Rel. Eff.
Accuracy
Rel. Eff.
Accuracy
Eff.
Accuracy
Rel. Eff.
SimSiam
98.6
0.1
87.4
0.1
70.4
0.25
38.3
0.1
BYOL
99.3
0.4
89.0
0.2
70.9
0.25
41.7
0.2
SimCLR
99.0
0.1
88.5
0.15
68.0
0.15
43.1
0.25
MoCo v2
98.1
0.05
87.8
0.1
64.0
0.1
38.2
0.1
EBCLR 99.3
90.1
77.3
49.1
Table Linear evaluation accuracy and efficiency relative to EBCLR. Efficiency of method relative to EBCLR is calculated by the following formula: (number of epochs used by EBCLR reach the final accuracy the method) (total number of training epochs).
4.1 Comparison with Baselines
We use batch size 128 for EBCLR and batch size 256 for the baseline methods following Wang et. al (27 and train each method for 100 epochs. Table shows the result of training each method for 100 epochs. Observe that EBCLR consistently outperforms all baseline methods in terms of linear evaluation accuracy. Moreover, relative efficiency indicates EBCLR is pable of achieving the same level of perfor- mance as the baseline methods with much fewer training epochs, Concretely, we ob- serve least x4 acceleration terms of epochs compared to contrastive methods. Hence, EBCLR is a much more desirable choice than SimCLR or MoCo v2 for learn- ing visual representations when we have a small number of training samples.
Direction
M - FM
FM M
C10 -> C100
C100 - C10
SimSiam
86.9
97.2
39.5
6100
BYOL
87.3
97.8
42.3
70.2
SimCTR
86.9
97.4
39.9
67.3
MoCo
85.3
EBCLR
87.4
98.5
46.9
72.4
Table 2: Comparison of transfer learning results in the linear evaluation setting. Left side of the arrow the dataset than the encoder was pre-trained on, and right side of the arrow is the dataset that linear evaluation was performed on. We use the following abbreviations. MNIST, FM FMNIST, C10: CIFARIO, C100: CIFAR100.
We also investigate the transfer learning performance of EBCLR. Table 2 compares the transfer learning accuracies. EBCLR always outperforms the baseline methods, and the performance gap is especially large on CIFARIO and CIFARI00. This indicates EBCLR learns visual representations that generalize well across datasets. Repeating the above experiments with longer training or KNN classification led to similar conclusions (see Appendixes E. and E.4 respectively).
4.2 Effect of Reducing Negative Pairs
We compared the performances of EBCLR and SimCLR as we reduced the number of negative pairs per positive pair. For MoCo v2, the negative samples are provided by queue updated by momentum encoder. On the other hand, for EBCLR and SimCLR, negative samples come from the same batch the positive pair. So, did not have way of fairly comparing EBCLR and SimCLR with MoCo v2. Hence, we excluded MoCo v2 from this experiment.
We note that, according to (18), given batch of size N, we obtain 2N - 2 negative pairs for each positive pair. SimCLR also has 2N - 2 negative pairs for each positive pair. Hence, we can conveniently compare the sensitivity of EBCLR and SimCLR to the number of negative pairs by varying the batch size.
Table 3 shows the result of training each method for 100 epochs with batch sizes in {16, 64, 128). We make three important observations. First, EBCLR consistently beats SimCLR terms of linear evaluation accuracy for every batch size. Second, EBCLR is invariant to the choice of batch size. This contrasts with SimCLR whose performance degrades as batch size decreases. Consequently, EBCLR with batch size 16 beats SimCLR with batch size 128. Finally, as a byproduct of the second observation, the efficiency of EBCLR relative to SimCLR increases as batch size decreases. These properties make EBCLR suitable for situations where cannot use large batch sizes, e.g., when
7

---

Dataset
MNIST
FMNIST
CIFARIO
CIFARI00
Batch Size
16
64
128
16
64
128
16
128
16
61
128
SimCLR
98.7
99.1
99.1
87.1
88.0
88.2
63.2
67.6
69.0
36.9
39.1
43.0
EBCLR
99.4
99.3
99.3
89.6
90.4
90.1
77.6
78.2
77.3
48.8
49.8
49.1
Rel. Eff.
0.05
0.1
0.15
0.05
0.15
0.1
0.1
0.15
0.2
0.1
0.15
0.25
Table 3: Linear evaluation accuracies and efficiencies relative EBCLR with various batch sizes. Efficiency of SimCLR relative to EBCLR is calculated by the following formula: (number of epochs used by EBCLR with the same batch size to reach the final accuracy SimCLR) / (total number of training epochs).
have a small number of GPUs. Repeating the experiments with longer training or KNN classification again led to similar conclusions (see Appendixes B.2 and E.4 respectively).
4.3 Effect of X and Projection Dimension
We explored the effect of changing the hyper- parameter À which controls the importance of the generative term relative to the discriminative
term (see Equation (II).. Figure 3a shows the performance EBCLR with various values as training progresses. We observe that naively using À = 1.0 leads to poor results. The perfor- mance peaks at 0.1, and then degrades as we further decrease X.
Rob Det
(a) Effect of X.
(b) Effect of proj. dim.
This result has two crucial implications. First, the generative term plays a non non-trivial role in EBCLR. Second, we need to strike a right bal- ance between discriminative term and the generative term to achieve good performance downstream t
Figure 3: Effect of and projection dimension (output dimension of Te, demonstrated on Cl- FAR10.
We also investigated the effect of varying the output dimension of тæ. Figure 3b shows linear evaluation results for projection dimensions {128, 256, 512]. We observe that the projection dimension has essentially no influence on the training process. In this respect, EBCLR resembles SimCLR which is also invariant to the output dimension (see Figure 8 in the work by Chen et.
4.4 Effect of SGLD Modifications
We now study the roles of the three SGLDD modifications proposed Section 3.3 Figure shows the results of varying one parameter of MSGLD while keeping the others fixed.
Effect of reintialization frequency p. Figure 4a displays linear evaluation results for p (0.0, 0.2, 1.0]. We note that setting = 1.0 is equivalent to removing the replay buffer. Also, setting p 0.0 is equivalent to never reinitializing SGLD chains.
Initially, p = 0.0 shows the best performance, as SGLD quickly reaches samples of lower energy. However, learning then slows down because of the lack of diversity of samples in the replay buffer B. This implies that it is necessary to set > 0 in order to learn good representations.
On the other hand, p = 1.0 shows slow convergence in the beginning because samples in the replay buffer are not given enough iterations to reach low energy. Although does beat p = 0.0 at latter epochs, it still often performs worse than - = 0.2. Moreover, it is not sample-efficient compared to p = 0.2 since we have to provide an entire batch of new samples for reinitializing SGLD chains at each iteration.
'Interestingly, we observed similar phenomenon when we used models trained with EBCLR to generate images. For more details, refer the readers Appendix E.5
8

---

(for more Given the above observations. it is clear why the intermediate value 0.2 is the best choice out of p E {0.0, 0.2, 1.0]. p = 0.2 allows enough time for samples the replay buffer reach low energy while still maintaining the diversity of samples in B. Also, it is sample-efficient compared to p 1.0.
Effect of proposal distribution go- Figure 4b) compares línear evaluation accuracies with 4o the uniform distribution and go = p(v). We observe prominent acceleration in the initial epochs for Qo -v)). Hence, we can conclude that this choice of proposal distribution is crucial for the high efficiency of EBCLR compared to the baseline methods in Tables T andd
We believe this acceleration effect can be explained by the work of Hinton [21]. Specifically, let us ob- serve that the EBM update equation (3) pushes up the energy on the model distribution qo- In the im- plementation of EBCLR with to p(r), however, qo is replaced by the distribution of samples created by finite number of (noisy) gradient steps on real data points (see Section 3.3 Hence, the modified EMB update equation contains the curvature informa- tion of the data manifold. This curvature information may expedite the training process of EBCLR. For a detailed discussion on this, we refer the readers to Section 3 of the work by Hinton (2T]
(a) Effect of varying (b) Effect of varying qo-
27.3
35.0
22.9
Comparison of SGLD and MSGLD, Figure 4c shows results with SGLD with E (0.01, 0.05) and MSGLD with min 0.01 and max = 0.05. We note that setting min max reduces MSGLD SGLD. We observe = 0.01 initially shows fast con- vergence but then saturates due the lack of diversity of generated samples. On the other hand, o = 0.05 initially has the worst performance but eventually beats o = 0.01 since a 0.05 quickly explores the modes of go- MSGLD inherits the best of both set- tings. Specifically, MSGLD is as fast as a 0.01 in
87.
11.0
7-0.0
Mooub
60.0
90
100
310 000 130 540
150
(c) SGLD with o e {0.01, 0.05) and MSGLD.
Figure 4: Ablation study of SGLD modifica- tions on CIFARIO.
the beginning, and it does not suffer from the saturation problem.
Limitations and Societal Impacts
Limitations. The main limitation of our work is of scale. While EBCLR demonstrates superior sample efficiency, it requires inner SGLD iterations (which cannot be parallelized) and a replay buffer B. These two components increase the computational burden of EBCLR. So, found it difficult to apply EBCLR to large-scale data such as ImageNet. However, we note that inner SGLD iterations and the replay buffer are particular limitations of EBCLR, but limitations of EBMs in general. Given the increasing efforts to overcome these limitations such as discussion, see Appendix F. we believe EBCLR will eventually be applicable to larger data.
Social Impacts. We generally expect positive outcomes from this research. Further development of EBCLR can mitigate the need for large amount of data and large batch sizes to learn good representations and ultimately lead to a reduction in resource consumption.
6 Conclusion
In this work, we proposed EBCLR which combines contrastive learning with EBMs. This amal- gamation of ideas has led to both theoretical and practical contributions. Theoretically. EBCLR associates distance in the projection space with the density of positive samples. Since the distribution of positive samples reflects the semantic similarity of images, EBCLR is capable of learning good visual representations. Practically, EBCLR several times more sample-efficient than conventional contrastive and non-contrastive learning approaches and is robust to small numbers of negative pairs. Hence, EBCLR applicable even in scenarios with limited data or devices. We believe that EBCLR makes representation learning available to wider range of machine learning practitioners.

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Research Foundation of Korea under Grant NRF- D12ORA2B5BBRR KAIST Key Research Institute (Interdisciplinary Research Group) Project, and Field-oriented Technology Development Project for Customs Administration through National Research Foundation of Korea(NRF) funded by the Ministry of Science & ICT and Korea Customs MM1
References
[1] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv: 03748, 2018.
[2] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, arXiv:1906.05849, 2019.
[3] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views contrastive learning? NeurlPS, 2020.
[4] Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In ICML, 2020.
[5] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[6] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Flena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: new approach to self- supervised learning. urXiv:2006.07733 2020.
[7] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
[8] Julia A. Lasserre, Christopher M. Bishop, and Thomas P. Minka. Principled hybrids of generative and discriminative models. In CVPR, 2006.
191 Hugo Larochelle and Yoshua Bengio. Classification using discriminative restricted boltzmann machines. In ICML, 2008
[10] Will Grathwohl, Kuan-Chich Wang. Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swerwky. Your classifier is secretly and energy based model and you should treat it like one. In ICIR, 2020.
[11] Max Welling and Yee Whye Tch. Bayesian learning via stochastic gradient langevin dynamics. In ICML, 2011.
[12] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv:2003.0423 042997 2020.
Huang. tutorial on [14] Mare' Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energy-based framework for unsupervised learning. 2007.
[13] Yann LeCun. Sumit Chopra, Raia Hadsell, Marc' Aurelio Ranzato, energy-based learning. Predicting Structured Data, 2006.
[15] Xiulong Yang and Shihao Ji. Jem++: Improved techniques for training jem. In ICCV, 2021.
[16] Yilun Du and Igor Mordateh. Implicit generation and generalization in energy based models. In NeurlPS, 2019.
[17] Fredrik K. Gustafsson, Martin Danelljan, Radu Timofte, and Thomas B. Schön. How train your energy-based model for regression. In BMVC, 2020.
[18] Fredrík Gustafsson, Martin Danelljan, Goutam Bhat, and Thomas B, Schön. Energy-based models for deep probabilistic regression. In ECCV, 2020.
[19] Yifei Wang, Yisen Wang, Jiansheng Yang. and Zhouchen Lin. A unified contrastive energy based model for understanding the generative ability of adversarial training. In ICLR, 2022.
10

---

[20] Yang Song and Diederik P. Kingma. How to train your energy-based models. arxiv preprint urXiv:2101.03288, 2021.
[21] Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computa- tion, 2002.
[22] Erik Nijkamp, Mitch Hill, Tian Han, Song- Chun Zhu, and Ying Nian Wu. On learning non con-convergent short-run meme toward energy-hasee hasse model. In NeurlPS, 2019.
[23] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. of the IEEE, 86(11):227) 2324, 1998.
[24] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist a novel dataset for benchmarking machine learning algorithms. arxiv preprint 17080:770..07777, 2017.
[25] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[27] Xudong Wang, Ziwei Liu, and Sella X. Yu. Unsupervised feature learning by cross-level instance-group discrimination. In CVPR, 2021.
[28] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.
[29] Roland S. Zimmermann, Yash Sharma, Steffen Schneider. Mathhis Bethge, and Wieland Brendel. Con- trastive learning inverts data generating process. In JCML, 2021.
[30] Hao Chen, Yaohui Wang, Benoit Lagadec, Antitza Dantcheva, and Francois Bremond. Joint generative and contrastive learning for unsupervised person re-identification. In CVPR, 2021.
[31] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 2011.
[32] Bradley Efron. The efficiency of logistic regression compared to normal discriminant analysis. JASA, 1975.
[33] Andrew Y. Ng and Michael Jordan. On discriminative vs. generative classifiers: comparison of logistic regression and naive bayes. In NewrIPS, 2001.
[34] Yoshua Bengio, Pascal Lamblin, Dan Popovaci, and Hugo Larochelle. Greedy layer-wise training of deep networks. NeurlPS, 2006.
[35] Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? JMLR, 2010.
[36] Stephen Zhao, Jörn-Henrik Jacobsen, and Will Grathwohl, Joint energy-based models for semi-supervised classification. In ICMI. Workshop, 2020.
[37] Wenzheng Zhang and Karl Stratos. Understanding hard negatives in noise contrastive estimation. 2021.
[38] Diedrik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[39] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand
Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[40] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. 2010.
[41] Will Sussman Grathwohl, Jacob Jin Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud. No meme for me: Amortized sampling for fast and stable training of energy-based models. In /CLR, 2021.
[42] Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. JMLR, 2005.
[43] Benjamin Rhodes, Kai Xu, and Michael U. Gutmann. Telescoping density-ratio estimation. 2020.
[44] Ruiqi Gao, Erik Nijkamp, Diederik Kingma, Zhen Xu, Andrew M. Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based models. In CVPR, 2020.
11

---

Checklist
1. For all authors...
(a) Do the main claims made the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations your work? [Yes] Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report bars (c.g., with respect the random seed after running experiments multiple times)? [No] Each EBM was trained once due to computation costs.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix [
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) your work uses existing assets, you cite the creators? [Yes]
(h) Did you mention the license of the assets? [N/A]
(c) Did you include any
either in the supplemental material URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're 'r us-
ing/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable informa- tion offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) you describe any potential participant risks, with links to Institutional Review Board (IRB)
approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "Wj9mt30oWCd", "reviewer": "Reviewer_wreL", "summary": "The paper proposes a representations learning method, Energy-Based Contrastive Learning (EBCLR), that combines contrastive learning with energy-based models (EBMs). The authors not only provide theoretical formulation of the proposed framework but also back up all the design choices empirically by providing adequate ablations. \n\nThe efficiency of the method is demonstrated on MNIST, CIFAR-10 and CIFAR-100 where it outperforms contrastive learning baselines in sample efficiency and linear evaluations.", "questions": "- How do you initialise SGLD in the very first step when there are no previous iterations for generated samples (or replay buffer is empty)?\n- I know the paper focuses on EBCLR’s representation learning capability, however, since there is a generative loss in the objective function, what’s the generative performance of the model? Does contrastive learning help it or deteriorate it?\n- Given the scaling difficulties associated with EBMs, could the contrastive learning be paired with other generative loss to a similar effect?", "limitations": "The main limitations seem to be around experimental evaluation. I would be happy to increase my score if authors could demonstrate the benefits on one of the large scale dataset.\n\nThere doesn't seem to be any obvious negative societal impact.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The paper proposes a simple extension of contrastive learning frameworks to energy based models for learning better representations efficiently.\n- The proposed method outperforms contrastive baselines and reach convergence (4x to 20x) faster than the baselines.\n- The authors provide theoretical formulation of their newly proposed framework.\n- Detailed ablations are provided to justify all the design choices.\n\n**Weaknesses:**\n- Currently, the related work section is more about explaining the inner workings of contrastive and energy based methods instead of focusing on the prior or similar works. Although I am not highly familiar with literature on energy based models but some insights from contrastive divergence in energy based models [2] (or differences with the present work [1]) would go a long way in helping new readers.\n- I would appreciate if the authors could position their work and contributions appropriately w.r.t to the prior work [1].\n- Although it's not a weakness, but I would have appreciated if the authors demonstrated benefits of EBCLR on one of the large-scale datasets.\n\n**Minor Comments:**\n- In abstract, you used terms positive and negative pairs without introducing them. It’s okay for the readers who know about contrastive learning, but someone who is new to the domain wouldn’t get it.\n- Missing appendix reference in line 132.\n- Could you please add subscript $\\textit(t)$ in Eq. 21 similar to update steps in Sec. 2.2, Eq. 5?\n- Line 193, repetition of $\\pi_{\\theta}$.\n\n[1] https://openreview.net/forum?id=XhF2VOMRHS\n[2] http://yann.lecun.com/exdb/publis/pdf/ranzato-unsup-07.pdf", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UoZVuUdOYt", "reviewer": "Reviewer_VTiU", "summary": "The paper proposes a new perspective to leverage contrastive self-supervised learning by combining energy-based models (EBMs). The code is to approximate the joint distribution of a pair of positive samples in contrastive learning by EBMs. Following the idea, the authors build a new contrastive learning method called EBCLR. To support the proposed method, the authors provide both theoretical and empirical evidence.", "questions": "I have put some of my concerns in the analysis of \"Weaknesses\" above. The clarification or improvements over them would be helpful for me to better justify the significance of this paper and to make my final rating more fairly and confident.", "limitations": "The authors have made the limitations analysis in the draft. I don't recognize the additional potential negative societal impact of this work.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors provide both theoretical and empirical evidence to support their claim.\n2. The experiment design is good, especially how it provides support for the efficiency of the proposed method given the limited number of negative pairs. Instead of the absolute performance advantage shown in Table 1, which is relatively not significant, the results in Table 3 are more impressive.\n3. The paper is well written and easy to follow. It is well organized in most parts.\n4. I would put it as a strength the limitation analysis by the authors which answers some of my concerns. \n\nWeaknesses:\n1. I have some concerns about the idea of connecting EMBs and contrastive learning has been studied for long, even not necessarily under the framework created by recent deep-contrastive-self-supervised learning, such as MoCo and SimCLR. Some related works might be missing for the overview of this area, such as:\n* A Unified Contrastive Energy-based Model for Understanding the Generative Ability of Adversarial Training, ICLR 2022\n* A Unified Energy-Based Framework for Unsupervised Learning, Artificial Intelligence and Statistics. PMLR, 2007.\n2. Some parts of the paper is not well organized. For example, at the end of the Introduction section, the authors claim five Contributions, but not all of them are well organized in the experiment section and following discussions. Besides this, it would be not convincing to put some minor points about of the proposed EBCLR and independent contributions.\n3. Some details should be polished, such as missing reference at L132 and the writing of Section 3, where some sentences are not fluent to follow. \n4. As the theoretical part in Section 3 mainly contains intuitive and very straightforward steps, I would recommend the authors to put some more efforts to backup the claim of contributions in following sections.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5KIsqJEfeBv", "reviewer": "Reviewer_oTyr", "summary": "Summary:\n\n\nThe paper proposes an energy based contrastive learning framework that outperforms the existing contrastive learning methods such as SimCLR, MoCo, BYOL, SimSiam on four datasets, MNIST, FMNIST, CIFAR10, CIFAR100, using linear evaluations. In essence, the energy-based contrastive learning adds an energy-based generative term to the existing contrastive learning objective. Subsequently, the paper focusses on proposing relevant challenges to Stochastic Gradient Langevin Dynamics (SGLD) for improved representation learning. Overall, the paper performs some relevant experiments along with the ablation studies that indicate the usefulness of the EBCLR framework. However, the mathematical formulation has some obvious mistakes that the authors would need to clarify. Additionally, the paper is not well-written and requires proper narrative to better pitch the idea along with additional experiments.\n\n\nStrengths:\n\n- It was interesting to see that EBCLR boils down to standard contrastive learning when solved for the KL divergence between the joint distribution of the positive pairs modeled via EBM.\n- The results of the proposed framework showcase that EBCLR outperforms all the existing methods on four datasets. Additionally, the authors show that EBCLR is more sample efficient as it achieves good performance with smaller batch sizes.\n- The authors study the various aspects of the framework through ablation studies. \n\nWeaknesses and Suggestions:\n\n- The abstract and introduction present this work with a motivation to reduce the batch size used for improved representation learning through contrastive methods. However, it seems that being able to perform well with small batch size is a byproduct of EBCLR framework. I don’t see any connection between the formulations discussed in the paper towards making EBCLR sample-efficient. Unless the authors present relevant works which have shown EBMs to be sample-efficient or prove it themselves, I don’t buy their motivation directly.\n- My another major concern is with the core observation itself. Years of previous works has shown that more data should improve the performance more. A good representation learning method should be able to perform better when encountered with more data. Line 235 claims that EBCLR is invariant to batch size which to me is a double edged sword — good performance with low batch size is a good thing but not improving much with more batch size is not good.\n- Equation 20 is wrong. It is calculated by solving Eq 12 and Eq 19. The authors cancel out the normalization terms from Eq 12 and Eq 19 but they are not actually equal! \n- Most of standard contrastive learning methods benchmark themselves on ImageNet dataset which paper does not do. The authors may use the latest FFCV library to fasten their ImageNet training pipeline.\n- The paper uses linear probing accuracy to evaluate the model accuracies. To further understand the representation quality, the authors should perform K-NN classification in the representation space too.\n- Further insights or discussions as to why EBCLR is more sample-efficient are definitely required.\n\n\nEdits:\n- Figure 1 left is not very clear. log 1/ p(v, v’) threw me off at the first place. You may need to explain it more.\n- Eq.4 should have N(0, \\sigma) \n- Line 193 has repetitions", "questions": "Mentioned in the main review", "limitations": "Mentioned in the main review", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Mentioned in the main review", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept